: Sitemap.xml on separate server (can't use robots.txt) We have several domains managed by the same code base. They share content and assets between each other but are separate entities online.

We have several domains managed by the same code base. They share content and assets between each other but are separate entities online.

I have started generating sitemaps for these sites and hosting them on S3, this works and is a sane method for us as we have no shared file system as such.

But, I obviously cannot submit the sitemap through the UI as it requires the sitemap be on the same domain, which it isnt.

Another option is to use robots.txt to specify your sitemap (as discussed here; Can I host a sitemap on another domain?) - but as all our sites share the same docroot, we can't do that either.

Maybe we could do some fancy URL rewriting to map the sitemaps? I'm not sure what the best option is for us. Any help is much appreciated.

EDIT
I'm trying to leave the sitemap.xml.gz files on s3 but serve the xml index using php on the sites. So you can visit /xml/site1.xml and get the sitemap index which points to s3. I'll report back with how it goes.

EDIT 2
I have added an xml doc on each domain with the index of sitemaps. This is at /xmlsitemap/sitemap.domain.xml.
The index contains the externally hosted sitemap (on s3).
I then added 7 x 'sitemap:' entries into robots.txt - so this should validate the external sitemaps. It doesn't, all sitemap indexes are returning errors that the sitemap is blocked by robots.txt.

The robots.txt is not blocking the sitemap, but it keeps reporting that it is.

Any suggestions?

EDIT 3
Some links ...
[Removed]

EDIT 4 & RESOLUTION
So, to resolve this issue I am generating the xml (both sitemap with delta and index) and storing it in the db. Then i'm just serving the generated xml using nginx/php through registered paths. As we're behind Varnish we can cache the output for 24h or so. Its working well so far.

I also noticed that my s3 bucket did not have a robots.txt so the robots message may have been caused by that and not the local one.

So, resolved by using none of my original options.

10.01% popularity Vote Up Vote Down

: Send pageview with Analytics.js to a second account Using Analytics.js: <script> (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function

@Pope3001725

Posted in: #GoogleAnalytics #Javascript #PageViews #Tracking

1 Comments

: Google Webmaster Tools reports first page presence on major search term but no sign of website on Google On a major search term Webmaster shows that it has achieved an average first page presence

@Pope3001725

Posted in: #GoogleSearch #GoogleSearchConsole #SearchEngines #SearchResults #Seo

1 Comments

: Free web hosting with python script I plan to display my Python script run result to share with others. thus, I plan to do a android apps and also a websites. I have use Pythonanywhere to

@Pope3001725

Posted in: #Python #WebHosting

0 Comments

: Best practice with visual responsive html restructuration I have to follow a sketch of the main webpage of a website I'm building. The main page is essentially blocks rendered as rows, but one

@Pope3001725

Posted in: #DuplicateContent #Layout #Magento #ResponsiveWebdesign #Seo

0 Comments

Login to post a comment!

1 Comments

Sorted by latest first Latest Oldest Best

@LarsenBagley505

You can easily achieve this using the same doc_root without needing to worry about uploading the sitemap to the S3 bucket.

Using the following rewrite code

RewriteEngine On
RewriteCond %{HTTP_HOST} ^(domain1.com)$
RewriteRule ^sitemap.xml$ /domain1-sitemap.xml [L]
RewriteCond %{HTTP_HOST} ^(domain2.com)$
RewriteRule ^sitemap.xml$ /domain2-sitemap.xml [L]
RewriteCond %{HTTP_HOST} ^(domain3.com)$
RewriteRule ^sitemap.xml$ /domain3-sitemap.xml [L]

You place this code in a single .htaccess file in the site root and generate your sitemap files using the naming convention domain-sitemap.xml and the rewrite rules will rewrite sitemap.xml to the appropriate domain to suit the host header.

10% popularity Vote Up Vote Down

Feed

: Sitemap.xml on separate server (can't use robots.txt) We have several domains managed by the same code base. They share content and assets between each other but are separate entities online.

More posts by @Pope3001725

: Send pageview with Analytics.js to a second account Using Analytics.js: <script> (function (i, s, o, g, r, a, m) { i['GoogleAnalyticsObject'] = r; i[r] = i[r] || function

: Google Webmaster Tools reports first page presence on major search term but no sign of website on Google On a major search term Webmaster shows that it has achieved an average first page presence

: Free web hosting with python script I plan to display my Python script run result to share with others. thus, I plan to do a android apps and also a websites. I have use Pythonanywhere to

: Best practice with visual responsive html restructuration I have to follow a sketch of the main webpage of a website I'm building. The main page is essentially blocks rendered as rows, but one

Login to post a comment!

1 Comments

Back to top | Use Dark Theme