: Bots would simply ignore this. I'd expect them to look for specific user agents that they need to take note of. So Googlebot for example would look for User-agent: * or User-agent: Googlebot.

@Sent6035632

Bots would simply ignore this.

I'd expect them to look for specific user agents that they need to take note of. So Googlebot for example would look for User-agent: * or User-agent: Googlebot.

So as those two are missing then the'd probably go ahead and crawl the entire site.

We all know that adding user-agents is essential in robots.txt so there's no point trying to second guess them.

Vote Up Vote Down

Login to post a comment!