: Do I need robots.txt? I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need

I want to allow all bots to crawl everything, do I need to create a robots.txt?

If the file is missing will it have a negative influence on some crawlers?

If I need to create it, what should it look like?

10.01% popularity Vote Up Vote Down

: Nofollow doesn't stop crawling? In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" href="/link/share/linkedin/Dance+Combo+%28Jazz%2C+Tap%2C+C

@Rivera981

Posted in: #Googlebot #GoogleIndex #GoogleSearchConsole #Nofollow #RobotsTxt

3 Comments

: Retrieve Facebook Like count after changing URL structure for entire website I just changed the URLs for my entire website. I have set up 301 permanent redirects for all old URLs to new URLs.

@Rivera981

Posted in: #301Redirect #Facebook #Like

2 Comments

: Migrating the source of a FeedBurner podcast feed from one WordPress to another I'm using FeedBurner as the public-facing feed for the Ask Different Podcast. Recently Stack Exchange has set up

@Rivera981

Posted in: #Feedburner #Rss #Wordpress

2 Comments

: Personal site/blog, my desired .com/.net are taken I want to start to have a place on the internet I can call my own. Stack Exchange, Google+, and Twitter are nice, but they're all services

@Rivera981

Posted in: #Domains

8 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Pope3001725

You do not need one as not having one is interpreted as meaning you want to have all of your content crawled. But I recommend using a blank one just to prevent the accumulation of unnecessary 404 errors in your stats.

10% popularity Vote Up Vote Down

Feed

: Do I need robots.txt? I want to allow all bots to crawl everything, do I need to create a robots.txt? If the file is missing will it have a negative influence on some crawlers? If I need

More posts by @Rivera981

: Nofollow doesn't stop crawling? In my website, most of the pages has a linkedin sharing button written in this way: <li> <a id="share_linkedin" href="/link/share/linkedin/Dance+Combo+%28Jazz%2C+Tap%2C+C

: Retrieve Facebook Like count after changing URL structure for entire website I just changed the URLs for my entire website. I have set up 301 permanent redirects for all old URLs to new URLs.

: Migrating the source of a FeedBurner podcast feed from one WordPress to another I'm using FeedBurner as the public-facing feed for the Ask Different Podcast. Recently Stack Exchange has set up

: Personal site/blog, my desired .com/.net are taken I want to start to have a place on the internet I can call my own. Stack Exchange, Google+, and Twitter are nice, but they're all services

Login to post a comment!

1 Comments

Back to top | Use Dark Theme