# Crawlers

I have the following in my robots.txt file:

# Block Googlebot-Image from cache
User-agent: Googlebot-Image
Disallow: /media/catalog/product/cache/

# Crawlers Setup
User-agent: *

# Directories
Disallow: /404/
Disallow: /app/

Disallow: /cgi-bin/
Disallow: /downloader/

Disallow: /errors/
...

But I am still seeing this in my logs:

66.249.66.189 - - [23/Jul/2016:06:30:12 -0700] "GET /media/catalog/product/cache/1/image/1800x/040ec09b1e35df139433887a97daa66f/n/d/nd01120a10g.jpg HTTP/1.1" 200 106745 "-" "Googlebot-Image/1.0"

Am I doing something wrong?

The thing is, I don't want those cached images indexed because they get deleted and I don't want 404 penalties.

10.01% popularity Vote Up Vote Down

: Hosting a Wordpress blog subdomain on another server I wish to add a blog to my online store, but due to security issues with Wordpress and just for peace of mind to keep it away from my

@Odierno851

Posted in: #AddOnDomain #SharedHosting #Subdomain #WebHosting #Wordpress

1 Comments

: Is it SEO safe to embed the H2 or H3 tag in a hyperlink element? I'm making a new layout for our website, and I want to make a list using the unordered list tag (ul) to show our offers.

@Odierno851

Posted in: #Heading #Html #Hyperlink #Seo

3 Comments

: Do embedded YouTube videos on my website consume my bandwidth or not? I would like to know if embedding YouTube videos on my website consumes my bandwidth when the videos are being viewed.

@Odierno851

Posted in: #Video #WebHosting #Youtube

1 Comments

: Product position in Google analytics The instructions for product position pr<productIndex>ps is the following. What is the use case of the product position parameter in the google measurement

@Odierno851

Posted in: #Ecommerce #GoogleAnalytics

1 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Debbie626

I've noticed that Google does the same thing to some of my sites, but they don't actually index the content they've crawled.

Read here: support.google.com/webmasters/answer/35308?hl=en
According to Google, using that in the robots.txt prevents them from indexing the content, it doesn't say that it will stop crawling it entirely.

10% popularity Vote Up Vote Down

Feed

: Google-Image bot not respecting robots.txt I have the following in my robots.txt file: # Block Googlebot-Image from cache User-agent: Googlebot-Image Disallow: /media/catalog/product/cache/ # Crawlers

More posts by @Odierno851

: Hosting a Wordpress blog subdomain on another server I wish to add a blog to my online store, but due to security issues with Wordpress and just for peace of mind to keep it away from my

: Is it SEO safe to embed the H2 or H3 tag in a hyperlink element? I'm making a new layout for our website, and I want to make a list using the unordered list tag (ul) to show our offers.

: Do embedded YouTube videos on my website consume my bandwidth or not? I would like to know if embedding YouTube videos on my website consumes my bandwidth when the videos are being viewed.

: Product position in Google analytics The instructions for product position pr<productIndex>ps is the following. What is the use case of the product position parameter in the google measurement

Login to post a comment!

1 Comments

Back to top | Use Dark Theme