How To Create A Sitemap For Crawlers and Visitors in 2021
A sitemap is a resource for your site that helps users navigate your website and easily find the information they are looking for, or shares metadata to Google and other search engines. This will make it easier for your content to be promoted through the search results.
In this post, we will consider what sitemaps are and the reasons for them along with tutorials on how to make one specifically for Google search purposes.
When search engine web crawlers index your content, they will examine the other pages that link to you and retrieve any website content relevant to your posts or pages. A concern for these crawlers is “can I access this content?” and “is there any more content available beyond what’s immediately accessible?”. Crawlers retrieve content from your site in order to build their index, which is what they use to show search results.
Relying on these ‘natural’ means of discovery can be enough if you have a simple website and don’t rely on users discovering your site in search results, but not if you want to rank high in the search engine results pages (SERPs). Taking such a passive approach can cause problems for new content that you are eager to see in search results, and obscure content you mark up especially for inclusion in rich results.
A sitemap provides search engine crawlers with a list of URLs to your content. This increases the crawler’s ability to reach and index all of your pages, improving its crawling efficiency because it no longer relies solely on your page’s relationship to other referring pages within your site and on the wider web.
What Is a Sitemap?
There are two types of Sitemaps: HTML and XML. An HTML Sitemap is a file that lists all the webpages that should be indexed by search engines in human-readable form, while an XML Sitemap creates an instruction set for robots to follow when they’re crawling your website to index important pages.
Ready to take your website SEO to the next level?
Sitemap Examples
XML: It’s a list of all the URLs (web page addresses) that you want indexed for your site with a few XML tags. This can include URLs of web pages, images, videos and other content files on the site.
HTML: A way to code content that makes it easy for a website’s visitors to locate key parts of your web page. This page is crawled by search engine spiders, and it’s believed that a page in the HTML sitemap is important for ranking well on search engines.
Learn how to create a sitemap for your website using HTML and XML.
Creating sitemaps will help search engines index all the pages on your site.
XML Sitemaps
An XML Sitemap is specifically designed to help web crawlers (also known as spiders or robots) index your website for search engines. Googlebot, for example, will use your Sitemap to discover pages on your site that otherwise may have been missed. XML Sitemaps work best in conjunction with a robots.txt file, which is used to identify the pages to be skipped by web crawlers.
In terms of search engine optimisation (SEO), a sitemap can be vital to your website’s success. By helping search engines to properly index your site, you enable potential visitors to discover its content. Working in conjunction with other SEO components, a properly coded sitemap can help your site achieve a higher ranking in the search engine results.
Once your XML Sitemap is ready, post it on your site and submit its URL to the major search engines. Then keep it up-to-date, as search engines will periodically check the sitemap for new information regarding your site. There are different ways to create and update your XML Sitemap which we will discuss later.
HTML Sitemaps
If your website is extensive, it should include an HTML-based Sitemap designed to help visitors navigate the site and locate desired content. Essentially, this is a “table of contents” page that can be used for quick reference. The basic design layout can and should reflect the rest of your site.
While there is no need for the average visitor to access your XML Sitemap, the HTML Sitemap ought to be accessible to everyone. To that end, you should identify prominent locations on your website where you can include a link to its URL. For example, you can link to your HTML Sitemap from your footer, “help” page, 404 error page, and “contact” page. The HTML Sitemap is intended to be used by visitors, so make sure they can find it.
By using your HTML Sitemap to assist visitors with navigation, you may also reap benefits indirectly in terms of SEO. Because improving navigation will increase visitor satisfaction, it will help those visitors find what they are seeking, encourage them to stay on your website longer, and lure them back periodically. All of these can affect your ranking in the SERPs.
Many website designers don’t incorporate Sitemaps into their website. Accordingly, you can take advantage of their oversight by adding Sitemaps to your site. By doing so, you can gain the edge that may propel you ahead of the competition.
Why Do You Need a Sitemap?
If your website pages are properly linked, search engine crawlers can easily access the majority of it without one. But, sitemaps can help search engine crawlers find pages more easily by providing a map of the site.
If your site has a large amount of content and is not well-linked or built for crawlability like multimedia sites or those that use difficult to parse formatting, you may want to include a sitemap so it can be crawled better.
To find out what Google sees from your site, perform the following Google search:
Site:yourwebsite.com
By including “site:” in front of your domain name you will be requesting Google to list the pages indexed for your site. When you see fewer results from Google search than expected, it could mean something about your site architecture is blocking Googlebot.
Creating a sitemap also applies to situations where there are certain pages on your website that you do not want to be indexed by the search engine. Pages that may be spammy, duplicate content or compete with the homepage on search can be excluded. If you don’t create a site map for your website, search engines will hunt through all the content on your sites unless you specifically exclude them and instruct them not to index these pages. The contents of your robots.txt file also tell crawlers which parts of a website they should avoid.
Ready to take your website SEO to the next level?
How to Create an XML Sitemap for Your Website
You can create a sitemap manually or by using tools that generate it for you. We’ll talk about the benefits and cons of each below so decide which approach is best for you.
Manually Create an XML Sitemap
If you’d like to craft your own sitemap manually, here are the key things to consider:
A site map in XML should include all the key pages that you want search engines to crawl and index.
You will need to follow these steps to create your sitemap.
You’ll prepare the document in a text editor. Then save it as .XML.
Finally, you should always validate your website’s sitemap to ensure it is error-free. You can use a free tool like the XML Sitemap Validator for this purpose. Simply enter your sitemap’s URL and press “validate” to check for errors you need to fix on your map before submission.
If you don’t have technical skills, our suggestion is to:
Use an Automatic an XML Sitemap Generator
There are plenty of free and affordable tools that you can use to automatically generate a site map for your website.
For instance, if you have a small website, you can use the free XML Sitemap Creator. Simply enter the domain of your website, click “Start” and the tool will automatically generate a site-specific sitemap.
If you’re using WordPress, the Rank Math SEO Plugin or Yoast SEO plugin can automatically submit your XML sitemap to Google and Bing after every new post or page that is published, so site crawlers never miss any of your content.
How to Create an HTML Sitemap for Your Website
You can create a sitemap manually or by using tools that generate it for you. We’ll talk about the benefits and cons of each below so decide which approach is best for you.
Create HTML Sitemap for Website Navigation
Creating a HTML sitemap of your website to help visitors find their way around is something you would only do with websites that have a lot of content. If your site has less than 100 content pages, just relying on good menus and navigation is fine.
A HTML sitemap is just a long, orderly list of links. No rocket science needed – just make sure all the links are on one page and ordered correctly.
How to create a sitemap – html sitemap example
The Smart Way to Create an HTML Sitemap
Unfortunately some platforms (like Squarespace) don’t provide HTML sitemaps.
So you’ll need to add a new page, and add the content of your HTML sitemap to it.
Now let’s be smart about this and take advantage of free tools that crawl your website and provide you with a list of pages.
There are two options to choose from when you want to build an HTML sitemap using your website. One option is manually adding all of the information, which can be tedious and time-consuming. The second option is by using a service such as check domains.
Tip: Research how to create a HTML sitemap on your specific platform – someone has likely already done this!
Use an Automatic an HTML Sitemap Generator
Other platforms with a plugin/extension ecosystem will always have an HTML site map option.
For example, WordPress:
Install and activate the plugin Hierarchical HTML Sitemap or the plugin WP Sitemap Page.
- Make a new page in your WordPress installation called ‘Sitemap’.
- Follow the plugin instructions to add the shortcode to it for use on pages that need mapping entries, and then save changes.
- Publish the page, check to see your new HTML sitemap.
How to Submit Your XML Sitemap to Search Engines
Submit your XML site map to online search engines such as Google and Bing by using the sitemaps features in their webmaster tools. Your agency should either do this for you as part of your SEO campaign or teach you how to do it. This will encourage Google’s crawlers to reference your site map when they visit.
To submit it to Google, you need to go through Google Search Console.
Some of you may already have this set up. If not, follow the steps to set up Google Search Console (previously Google Webmaster Tools).
Once you’re logged in:
- Go to your Search Console dashboard
- Click on Sitemaps from the menu
- In the “Add New Sitemap” field add your sitemap
- Click “Submit”.
To submit your sitemap to Bing, you’ll first need to sign up to Bing Webmaster Tools.
Once you’re logged in:
- Go to My Sites and click on your website
- Use the left-hand sidebar menu to go to Configure My Site -> Sitemaps
- Now simply add the URL to your sitemap and click “Submit”.
Google supports a few different sitemap formats, such as XML, RSS, HTML and text files, but it must be UTF-8 encoded. You can submit more than one sitemap, which is especially useful if your site has a lot of pages.
Having a sitemap won’t guarantee that search engines have crawled and indexed your site as you’ve asked, but it will certainly make it easier for them to do so.
Conclusion
Sitemaps are an important tool for webmasters. They’re easy to create and submit, which means they’re worth doing if you want your site indexed more effectively by search engines.
