: Google is displaying a URL in my site's index with the cache of that result being completely different Ok, this may turn out to be a silly question but I have observed Google displaying a

@Berumen354

You have an error in your robots.txt file.

On line 11 you have Allow: /, a robots.txt file doesn't say what files and directories you can allow, only what you can disallow. The only supported commands for the robots.txt file are "User-agent" and "Disallow".

As the Disallow: /random command is after the invalid command it is possible the Google Searchbot detected an invalid command and because it couldn't process it stopped processing the entire robots.txt file as if it didn't exist at all.

You can validate your robots.txt file using a tool such as the one located at tool.motoricerca.info/robots-checker.phtml
As for why the cacheed version is different to the live version the cached version it what Google see's at the time the spider went through which in the case of your cached link was 6 April 2016 at 16:05:27 GMT.

A new version of your robots.txt file which you could use is...
#The date is August 29th, 1997. #Robots have taken over the world and documentaries cease to be created by humans. #what will happen next?

#Want to join the Docur team?
#E-mail jonbonsilver//at//gmail//dot//com

#Full access for the internet archive.

User-agent: ia_archiver
Disallow: /random

#Every robot that honours the robots.txt standard:

User-agent: *

#Request file from Docur once every second:

Crawl-delay: 1

#Disallowed urls:

#Lets not send bots on a random documentary mission:

Disallow: /random

Disallow: /new-documentaries
#Above is a temp line due to indexing problems.
Disallow: /?page
Disallow: /live-search
Disallow: /vote
Disallow: /favourite
Disallow: /watch-later
Disallow: /save-list
Disallow: /comment
Disallow: /commentlike
Disallow: /commentdislike
Disallow: /add-review
Disallow: /submit-review
Disallow: /add-to/*
Disallow: /post-list
Disallow: /edit-list
Disallow: /documentary-search
Disallow: /new-list-item
Disallow: /settings
Disallow: /notificationread
Disallow: /documentary/*/l
Disallow: */newest
Disallow: */oldest
Disallow: */highest
Disallow: */lowest

Vote Up Vote Down