The first stage of SEO-optimization of the site is to remove all unnecessary. Rarely does it work right the first time. Here, drafts and duplicate pages accumulate on sites – all of them can be indexed by search engines, becoming ballast for further promotion.
The same applies to technical pages (registration / authorization, my account, shopping cart, etc.). We cannot delete them, but we must remove them from sitemap.xml and close them from indexing in robots.txt.
Free services and plugins for generating sitemap.xml and robots.txt files often do not take these points into account – a lot has to be adjusted manually.
Since all my sites mostly on WordPress, previously, I was limited to installing the plugin Yoast SEO. It allows you to generate sitemap.xml and robots.txt. And, if, robots.txt can be edited at your discretion, then sitemap.xml, unfortunately, is created completely automatically, without the possibility of making changes and includes links to all pages of the site (including technical ones).
The solution to this problem was the plugin Kaboom XML/HTML Sitemap Generator. It generates a dynamic sitemap and contains an editor that allows you to exclude unnecessary pages:
The free version of the plugin is suitable for small sites with up to 250 pages. Kaboom XML/HTML Sitemap Generator perfectly complements Yoast SEOand can be paired with it.
As for robots.txt, the file generated in Yoast SEO looks like this:
To close maximum technical pages from indexing, add the following lines:
Disallow: /cgi-bin
Disallow: /?
Disallow: /wp-
Disallow: *?s=
Disallow: *&s=
Disallow: /search/
Disallow: /author/
Disallow: /users/
Disallow: */trackback
Disallow: */feed
Disallow: */rss
Disallow: */embed
Disallow: */wlwmanifest.xml
Disallow: /xmlrpc.php
Disallow: *utm*=
Disallow: *openstat=
Allow: */uploads
Sitemap: http://yoursite.com/sitemap.xml
Host: www.yoursite.ru
You can learn more about setting up the robots.txt file for WordPress sites here.
Everything is ready – the excess was removed. The first stage of SEO-optimization of the site structure is completed. Now you can make an effort to promote the key pages of the site.
Recent Comments