: Is this "robots.txt" file really preventing all crawling of our website? I'm trying to find out why our SEO is so poor So I've been assigned to take a look at our SEO (an area I have some,

@Smith883

I've put this sort of robots.txt in place when first developing a site because I don't want it to be indexed by Google and others before it's ready.

I've also forgotten to edit that after the site has gone live. *facepalm*

Vote Up Vote Down

@Lee4591628

Maybe someone didn't want to pay for spider traffic?

Regardless, you are reading it correctly:
www.robotstxt.org/robotstxt.html

Web site owners use the /robots.txt file to give instructions about
their site to web robots; this is called The Robots Exclusion
Protocol. It works likes this: a robot wants to vists a Web site
URL, say www.example.com/welcome.html. Before it does so, it
firsts checks for www.example.com/robots.txt, and finds:


User-agent: *
Disallow: /



The "User-agent: *" means this section applies to all robots. The
"Disallow: /" tells the robot that it should not visit any pages on
the site.

Vote Up Vote Down