page2 ... Now I don't get how to whitelist the root page at robots.txt

I need to allow only specific pages at some site. Let's say:

/
/page1
/page2
...

Now I don't get how to whitelist the root page at robots.txt

Disallow: / # block everything
Allow: /page1
Allow: /page2
Allow: ? # how to allow the / ???

And I'm stuck here, because Allow: / will allow everything back

Is it even possible?

10.01% popularity Vote Up Vote Down

: .htaccess RedirectMatch not working when some users include "www." in URL I have recently moved a site to a new domain, and at the old domain, I created a .htaccess file that includes one

@Megan663

Posted in: #Htaccess #Redirects

2 Comments

: How to make search engines understand website sections and tags? I have a website to sell products. It basically divides the products based on tags. For Example; if you access tag page for

@Megan663

Posted in: #Microdata #Seo #StructuredData #Tags

1 Comments

: Multiple domains with one google analytics account We developed a Magento website for a client which has 3 stores, and each store has 1 domain name, for example: Store 1 has store1.com Store

@Megan663

Posted in: #GoogleAnalytics #MultipleDomains

1 Comments

: I have found that you get the best performance by using a combination. I tend to use: A core combined JavaScript file that contains a concatenation of all the JavaScript that is required

@Megan663

0 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@Rivera981

You should be able to use a $ sign which indicates the end of the string.

For example

Allow: /$

If you go to the bottom of Google's robots.txt help, you can see a similar example.
developers.google.com/webmasters/control-crawl-index/docs/robots_txt

10% popularity Vote Up Vote Down

Feed

: Explicitly whitelist every page using robots.txt I need to allow only specific pages at some site. Let's say: / /page1 /page2 ... Now I don't get how to whitelist the root page at robots.txt

More posts by @Megan663

: .htaccess RedirectMatch not working when some users include "www." in URL I have recently moved a site to a new domain, and at the old domain, I created a .htaccess file that includes one

: How to make search engines understand website sections and tags? I have a website to sell products. It basically divides the products based on tags. For Example; if you access tag page for

: Multiple domains with one google analytics account We developed a Magento website for a client which has 3 stores, and each store has 1 domain name, for example: Store 1 has store1.com Store

: I have found that you get the best performance by using a combination. I tend to use: A core combined JavaScript file that contains a concatenation of all the JavaScript that is required

Login to post a comment!

1 Comments

Back to top | Use Dark Theme

: Explicitly whitelist every page using robots.txt I need to allow only specific pages at some site. Let's say: / /page1 /page2 ... Now I don't get how to whitelist the root page at robots.txt

More posts by @Megan663

: .htaccess RedirectMatch not working when *some* users include "www." in URL I have recently moved a site to a new domain, and at the old domain, I created a .htaccess file that includes one

: How to make search engines understand website sections and tags? I have a website to sell products. It basically divides the products based on tags. For Example; if you access tag page for

: Multiple domains with one google analytics account We developed a Magento website for a client which has 3 stores, and each store has 1 domain name, for example: Store 1 has store1.com Store

: I have found that you get the best performance by using a combination. I tend to use: A core combined JavaScript file that contains a concatenation of all the JavaScript that is required

Login to post a comment!

1 Comments

Back to top | Use Dark Theme

: .htaccess RedirectMatch not working when some users include "www." in URL I have recently moved a site to a new domain, and at the old domain, I created a .htaccess file that includes one