: What headers/META headers should I be using for an infinite link system like a calendar? I have a web calendar, all my own work. It works fine. But I see in the logs a bunch of search indexers
I have a web calendar, all my own work. It works fine. But I see in the logs a bunch of search indexers go round and round and round trying to chase every link. It could go forward to the year 3000 month by month and day by day. I don't think the indexers should follow this as it doesn't serve much purpose. What HTTP headers or META headers should I apply to the calendar to let them know it is a calendar, but they shouldn't follow every link on those pages?
More posts by @Harper822
2 Comments
Sorted by latest first Latest Oldest Best
Use rel="nofollow" on all links that you don't want spiders to follow or index (i.e. days, months, etc.).
Use robots.txt to specify paths that you want them to ignore.
If your site is a web application, consider using robots.txt to disallow indexing of the whole app with:
User-agent: *
Disallow: /
Probably simply <a rel='nofollow' ... on the calendar navigation links (as opposed to the events that link to specific events) would be fine.
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.