: URL blocked by robots.txt in webmaster for Blogger site Remove: Disallow: /search From where can I remove this line? I am using Blogger and in webmaster it shows many of my pages are blocked.

@XinRu657

Your can change/add custom robots.txt on Blogger. Go to Dashboard > Settings › Search preferences > Custom robots.txt >, click edit to enable and edit.

To remove Disallow: /search only. Fill robots.txt format below to text-area (change yourdomain.com to your homepage URL):

User-agent: Mediapartners-Google
Disallow:

User-agent: *
Allow: /

Sitemap: yourdomain.com/feeds/posts/default?orderby=UPDATED

INFO: if you remove Disallow: /search from your robots.txt your website will experience possible problems with duplicate meta titles and descriptions which can be bad for SEO.

Vote Up Vote Down

@Kristi941

Blogger now provides the ability to edit the robots.txt file - although it doesn't feel like you are editing a file.

Log into the Blogger dashboard, and go to Options > Search. There is an edit link beside robots.txt in that screen.

Vote Up Vote Down

@Eichhorn148

From forums.digitalpoint.com/threads/how-to-edit-robots-txt-in-blogspot-blog.422059/

You cant change the robots.txt file in Blogger unless you host your own blog in your server.


As a general rule, "search" pages should be disallowed in robots.txt. Google does not want to include pages of search results in the index. If they did so, users would be clicking from one set of search results to another. Google does not consider that good user experience. In fact, Google has been known to de-index (penalize) entire sites that allow their site search results to be indexed.

Vote Up Vote Down

@Frith620

The robots.txt file is always found in the root folder of your site.

You can usually edit it using a file-manager which cPanel nearly always has. If your site does not have cPanel, there is probably something similar but I cannot guess without knowing more.

The nearly universal way is via FTP. You connect to your server, download the file. Edit it and then upload it back. If you use an IDE such as DreamWeaver for your web-development that is what it usually does. You can use any FTP package though, including command-line ones bundled with most operating systems.

There are two things to note:


When using someone else's server, they can block or override access to pretty much anything. If for some reason, you cannot do the change yourself you will have to file a ticket request with your hosting provider.
Google does not want to index anybody else's search results. For this reason, you will often find search pages blocked in the robots.txt. This may be why its there in the first place. If it just happens to be called search because you have pages about search rather than generated by a search feature, then you should remove it.

Vote Up Vote Down