One of the most important aspects of SEO is optimizing the crawl ability of your site. Search engines have near-endless resources, so they have the power to crawl everything they find — and they will. But, that is not the way it should be. Almost every CMS outputs URLs that don’t make sense and that crawlers could safely skip. With Yoast SEO Premium 18.6, we’re starting a series of additions to clean up those unnecessary URLs, feeds, and assets so that the more critical stuff stands a better chance of being crawled.
Making your site easier to crawl
Google and other search engines crawl almost everything they can find — as Yoast founder Joost de Valk proves in a post on his site. But it can be hard to get them to crawl what you want them to Netherlands Mobile Number crawl. Moreover, crawlers can come by many times each day and still not pick up the important stuff. There’s a lot to gain for every party involved — from the crawlers, site owners, and environment — to make this process more sensible. SEO Premium will help search engines crawl your site more efficiently.
In SEO Premium we’re introducing the first addition to our crawl settings, allowing you to manage better what search engines can skip on your site. In this release, we’re starting with those RSS feeds of post comments in WordPress, but we have a long list of stuff that we want to help you manage.
Head over to our new Crawl settings section in the General settings of Yoast SEO Premium and activate the Netherlands Mobile Number first addition to preventing search engines from crawling the post comment feeds. Let’s all start cleaning up the crawling on our sites — it’s better for you, your visitors, search engines, and the environment. All with a little help from Yoast SEO Premium. Let’s go!
This feature is available to all Yoast SEO Premium subscribers in beta form, and we’ve selected not to activate this for every site. In some cases, there still might be sites that use this in a way we can’t anticipate. We’re rolling out more crawling options — big and small — in the coming releases.