: URL is indexed but description not available due to "this site's robots.txt" I have a similar URL indexed by Google https://example.com/companies/company?utm_source=xxx And this output: A description

@Margaret670

Remove this line

Disallow: /


And you're good to go.

User-agent: *
Disallow: /


Here * means all polite bot including Googlebot, BingBot, YandexBot will follow your robots.txt rules, and / meaning all the directory is restricted to crawl.

You should use only this rule.

User-agent: *
Disallow: /static/*
Disallow: /templates/*
Disallow: /translations/*
Sitemap: example.com/sitemap.xml

This robots.txt tell all polite bot to don't crawl all files inside static, templates and translations folder. So again if your sitemap contain these URL's
example.com/static/some-javscript.js https://example.com/static/some-photos.png example.com/templates/some/index.html https://example.com/templates/translations/de/index.html


Then Google will still display that line in search results because you've block it for crawling, so understand robots.txt concept, it's not hard to understand at all.

Vote Up Vote Down