: How soon Google is supposed to include new pages to search I'm using Google CSE on my site. As soon as new entity is published, it's immediately added to sitemap.xml rendering script too. How

@Alves908

You have to read this question carefully.

It is about the Google Custom Search Engine which is not the Google Search Engine. It is a feature whereby a site can have a custom search engine for the site. This, at least, is a site search feature.

These are two different things though there is some relationship between the two.

Assuming that a site owner opts for and implements the Google CSE feature and not all of the sites pages have been indexed into the Google Search Engine, what happens?

Google, for CSE customers, guarantees that all of a sites pages will be indexed quickly. This is providing that there is not a penalty for the site that gets in the way. Things have changed enough recently with all of the updates in 2015. It is more possible that a penalty will get in the way and therefore paramount that a site make sure that a penalty does not exist.

Google will completely refetch pages and fetch previously unknown pages it has not yet seen. It will use the sitemap as an audit to ensure that all of the pages are fetched. Google will attempt to do this very quickly though, depending upon the site size and speed, can still take weeks or months.

Why does Google do this?

To ensure that the CSE is complete. The Google Search Engine and Custom Search Engine work using the same index. If the site is not fully indexed, being a CSE customer will ensure that Google will, at least, attempt to index the entire site.

Once this process is complete, then what happens?

We go back to how Google behaves normally and much of that depends upon you.

Ask yourself:


How fresh is my site overall?
How fresh is any particular page?
It that page linked?
How important does the link to the page appear?
How big is my site?
How popular is my site?
How does my site rank with inbound links, social media engagement,
citations, CTR (click-through rate) from the SERPs (search engine
result page), etc.?


Assuming all is good, then Google will see new pages very quickly. If not, then it can take quite a while depending upon all of the above factors. In short, it can take minutes or weeks. On rare occasions, months.

Does Google's reading of my sitemap mean finding pages more quickly?

No. Not always. Google may chose to use your sitemap only to audit whether it can properly fetch your sites pages adequately. This is more common than not. Google is surprisingly old-school. It will prefer and continue to index your site as it has always done for all sites from the beginning of time if it can. Having a sitemap most of the time adds no benefit to the site short of pages behind a login or paywall or a site that is enormous.

So adding a page to your sitemap may do little to nothing. It is important that there be a link to any new page in a place of importance, such as, the home page.

But does that mean that pages that are not linked will not be found by Google?

No. If there are pages listed in a sitemap that are not linked, Google will fetch them and index them. However, Google does not like this. Google believes that all pages should be made available through the site itself. Google also believes that large numbers of pages found in the sitemap that are not linked is a signal for a spam site. They cannot be more wrong on this. For this reason, pages found following links on the site is indexed first and pages found in the sitemap second. This may not be strictly the case, but definitely close enough to be generally true.

You and I know that it is not possible to link a huge number of pages and Google also does not like navigational pages. As well, these pages can be cumbersome and bad for user experience (UX). I dropped mine a long time ago and intended to use the CSE as a valid replacement.

Does the CSE replace linking to a large quantity of pages?

Not very well. Which is a shame. It is the perfect opportunity for a webmaster to use the sitemap to signal to Google all of the pages within a site and use the CSE as a mechanism for a user to find content that cannot be effectively linked to.

Google seems to be the grumpy bear in this otherwise perfect process they created. It beats you up for having pages in your sitemap that is not linked on the site, it beats you up if you are frustrated and attempt to link these pages through navigational pages, and it takes it's sweet time indexing pages found in the sitemap and not found through links.

In addition, site search is not such a popular feature and seems to be under utilized in most sites. It is difficult to convert a user into a site search user. It is a sad fact. Most will go back to the SERPs and continue searching even if their next click is back on your site. Crazy huh?

So what are you to do?


Do everything you can to increase site search through conversion.
Make sure that all new pages are linked and signal importance even
temporarily.
Find a replacement for navigational pages other than the CSE (not to
replace CSE).
Keep your site fresh and popular and promote it on social media.
Let God and Google sort it out from there. In otherwords, do not
worry beyond what you can control and do not try and control Google.

Vote Up Vote Down

@Eichhorn148

If you want your pages indexed faster you need to do more than add them to your sitemap. A sitemap doesn't pass any link juice to the pages and Google usually doesn't index pages that have no link juice. See The Sitemap Paradox for more information.

You need to link new pages from other pages on your site. It might be worth having a "recently published" list on your home page for example. Your home page is a high PageRank page that can usually pass enough link juice to get the items linked there indexed.



Another way to let search engines know about your new content is to use ping services that alert them. The blogging software WordPress has this functionality built in and have a wiki page explaining how it is done. They rely mostly on Ping-O-Matic to spread the word to the various search engines about your updated content.

Vote Up Vote Down

@Jessie594

There is no fixed time from when Google crawls a site to when it is available on the index. A few years ago it would have taken anywhere up to 6 weeks for content to be added due to Google rebuilding the main index in batches, now with the live updating it is meant to take longer but it could be added to the index any time from a few days to a few weeks after the site has been crawled. It depends on a large number of factors, many of which Google won't release as it classes them proprietary.


Once you submit your URL, you might need to wait some time for Google to process your request as well as crawl and index the page. Also, understand that we can't guarantee that Google will index all your changes as Google relies on a complex algorithm to update indexed materials.- From Google (https://support.google.com/webmasters/answer/6065812?hl=en)

Vote Up Vote Down