: Specifying a website to appear in certain search engines and not others? If you want a website to appear in DuckDuckGo for example and not Google, is this possible? Can you specify which search
If you want a website to appear in DuckDuckGo for example and not Google, is this possible? Can you specify which search engines for a website appear in?
More posts by @Berumen354
1 Comments
Sorted by latest first Latest Oldest Best
You can block Google and other search engines by using a robots.txt file and meta name no index. You can use one or the other but Google recommends that you use both robots and meta.
To block a specific bot you use something like this within the robots.txt file:
User-agent: bot-name
Disallow: /
To block a specific bot using HTML then include this meta name within the <head></head>
<meta name="bot-name" content="noindex, nofollow">
You can find the user agents listed on Google's Webmaster Help, below you can find all types of bot user agents that Google uses. Note that Adsbot, and Media partners may not crawl your site nor index.
Googlebot
Googlebot-News
Googlebot-Image
Googlebot-Video
Googlebot-Mobile
Mediapartners-Google
Mediapartners
AdsBot-Google
You can also find Microsoft's Bing crawler user agents on their website
Bingbot
Adidxbot
MSNBot
BingPreview
DuckDuckGo bot uses the user agent duckduckbot and even lists the ip addresses it uses.
duckduckbot
You can pretty much find any user agent using any good search engine, but to save your time its useful to bookmark this website that lists the majority of crawler user agents.
You can block all search engines by using noindex or user-agent:* i.e
Robots.txt
User-agent: *
Disallow: /
Meta Name
`<meta name="robots" content="noindex, nofollow">
Terms of Use Create Support ticket Your support tickets Stock Market News! © vmapp.org2024 All Rights reserved.