: Robots.txt got changed to disallow crawling by accident Our robots.txt mistakenly got changed to: User-agent: * Disallow: / This occurred about 2 weeks ago and was changed back yesterday to

Our robots.txt mistakenly got changed to:

User-agent: *
Disallow: /

This occurred about 2 weeks ago and was changed back yesterday to allow crawling.

Our listed pages in the Googles have decreased substantially, though it looks like Google has already started adding more search results for the site in question since changing the robots.txt back.

Are there long term repercussions of this happening, or will everything get back to normal once the site is crawled again?

10.01% popularity Vote Up Vote Down

: Why does my funnel look like that? I've set up a goal with a funnel on Google Universal Analytics. The funnel starts at one domain and the second step is in another domain. I'm using linker

@Turnbaugh106

Posted in: #Conversions #Goals #GoalTracking #GoogleAnalytics

1 Comments

: Multiple IPs/webservers, single domain Our main website is on a Windows server. We're migrating our Blogger blog to WordPress and I think I want to use a Linux-based webhost instead of our Windows

@Turnbaugh106

Posted in: #Domains #Redirects #UrlRewriting

2 Comments

: How long before legacy 302s are recognised as 301s? We have some legacy 302 redirects in place on our site that should have always been 301s. We've now changed the redirects to 301s but various

@Turnbaugh106

Posted in: #301Redirect #302Redirect #Cache #Seo

1 Comments

: How does Bing know when a result is clicked to use in website ranking? I know that Google uses redirect links to count clicks ie: https://www.google.com/url?q=http://html5test.com/&sa=U&ei=xlezU6DPIsqysQS824HwAg&ved=0

@Turnbaugh106

Posted in: #Bing #Ctr

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Kevin317

If Google has started adding pages again, then you are fine. We had this happen with a few sites that were pushed from development into production and their robots.txt was not changed.

In fact, unless you are trying to block pages, I recommend removing robots.txt completely. There's no point in saying allow everything in robots.txt when that is the default action.

10% popularity Vote Up Vote Down

Feed

: Robots.txt got changed to disallow crawling by accident Our robots.txt mistakenly got changed to: User-agent: * Disallow: / This occurred about 2 weeks ago and was changed back yesterday to

More posts by @Turnbaugh106

: Why does my funnel look like that? I've set up a goal with a funnel on Google Universal Analytics. The funnel starts at one domain and the second step is in another domain. I'm using linker

: Multiple IPs/webservers, single domain Our main website is on a Windows server. We're migrating our Blogger blog to WordPress and I think I want to use a Linux-based webhost instead of our Windows

: How long before legacy 302s are recognised as 301s? We have some legacy 302 redirects in place on our site that should have always been 301s. We've now changed the redirects to 301s but various

: How does Bing know when a result is clicked to use in website ranking? I know that Google uses redirect links to count clicks ie: https://www.google.com/url?q=http://html5test.com/&sa=U&ei=xlezU6DPIsqysQS824HwAg&ved=0

Login to post a comment!

1 Comments

Back to top | Use Dark Theme