Mobile app version of vmapp.org
Login or Join
Gonzalez347

: What is better for SEO robots.txt file or dynamically generated robots.txt URL? I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was

@Gonzalez347

Posted in: #Indexing #RobotsTxt #WebCrawlers #Wordpress

I was relying on our custom WordPress package's dynamically generated robots.txt URL for a few sites but it was not being honoured by Google or Bing (plugin URLs were getting indexed despite the rule Disallow /wp-content in robots.txt). I waited for a couple of weeks and uploaded an actual file in the root with the same rules and it worked.

Does this mean that an actual file is better than dynamically generated robots.txt file or is it just me associating some coincidental events?

10.02% popularity Vote Up Vote Down


Login to follow query

More posts by @Gonzalez347

2 Comments

Sorted by latest first Latest Oldest Best

 

@Sue5673885

No difference at all as far as crawlers are able to access them (with all your valid rules) with HTTP 200 response.

10% popularity Vote Up Vote Down


 

@BetL925

There is no difference to search engines between those two methods. You can generate it like you want.

Moreover, you need to know blocking a page with robots.txt is not the perfect way to prevent indexing of pages.
You can read this question for more information.

10% popularity Vote Up Vote Down


Back to top | Use Dark Theme