How Google Crawls and Indexes a Website, and Why Your Page Might Not Show

Google finds pages by crawling, stores them by indexing and orders them by ranking. A page can stall at any one of these three steps, and each step needs a different fix.

If your page is not showing up, the first job is to work out which step it is stuck at. This guide explains what each step means, how to check where a page stands in Search Console and what to do next.

Decoding Google's View

What Crawling, Indexing, and Ranking Actually Mean 

Google doesn’t just find your website and put it on the results page. It moves through three distinct steps first.

Crawling is discovery

Google uses an automated program called Googlebot to visit your pages and read what’s on them, following links from page to page like a visitor clicking around your site.

Indexing is storage

Once Googlebot has read a page, Google decides whether to save it in its index, the massive database it pulls search results from. A page has to be indexed before it can ever appear in search results.

Ranking is placement

For any given search, Google looks through the indexed pages that match the query and decides which order to show them in, based on relevance, quality, and hundreds of other signals.

Here’s the part that trips up most site owners: passing one stage doesn’t guarantee the next. A page can be crawled and never indexed. A page can be indexed and still rank on page nine. Each stage has its own separate set of reasons for failing.

How does Google crawl, index and rank a website?

Google explains the full process in How Google Search works. In plain terms, the three steps look like this:

StepWhat Google doesHow to check itCommon blockers
CrawlingGooglebot follows links and sitemaps to find and fetch your pagesURL Inspection, Crawl Stats reportrobots.txt rules, server errors, pages with no links pointing to them
IndexingGoogle processes the page, picks a main version and stores itPage indexing report, URL Inspectionnoindex tags, duplicate or thin content, wrong canonical
RankingGoogle orders indexed pages for each search using its ranking systemsPerformance report in Search ConsoleWeak relevance, little authority, poor page experience

Why a Crawled Page Might Not Get Indexed

This is the single most common source of confusion in Google Search Console, and it’s worth understanding in detail. Most SEO guides skip straight past it and jump to backlinks and content tips, which is exactly why so many site owners get stuck here without knowing what to check first.

If you open the Pages report in Search Console, you’ll often see a status called “Crawled currently not indexed.” This one confuses people more than any other, because it sounds like an error. It isn’t.

That status means search enginebot successfully visited the page, found no technical problems, and still chose not to add it to the index. In other words, the page passed its technical checks and got rejected anyway, usually for one of these reasons:

  • The content is too thin or too similar to other pages already in the index.
  • The page has little to no authority signal pointing to it, such as internal links or backlinks.
  • Google judged the content not valuable enough to compete for the query it targets.

A related status, “Discovered currently not indexed” means something different. Google knows the URL exists but hasn’t crawled it yet at all, usually because of limited crawl budget or low priority.

The fix for both statuses is rarely a technical one. Adding internal links from stronger, already indexed pages, strengthening the content itself, and earning backlinks are what usually move a stuck page forward. Our full breakdown of crawl errors and how to read the Pages report covers this status in more depth, but the real fix is almost always about content quality and link signals, not a setting you flip.

What do the Search Console indexing statuses mean?

The Page indexing report groups your URLs by status. These are the ones site owners ask about most:

StatusWhat it meansCheck first
Discovered, currently not indexedGoogle knows the URL but has not crawled it yetInternal links to the page, server speed
Crawled, currently not indexedGoogle fetched the page and has not added it to the indexContent value, duplication, internal links
Duplicate, Google chose a different canonicalGoogle treats another URL as the main versionCanonical tags and similar pages
Blocked by robots.txtGoogle is not allowed to fetch the pageRules in your robots.txt file
Excluded by noindex tagThe page tells Google not to index itPage head, SEO plugin settings, HTTP headers

Site owners on Reddit describe the same patterns. Some saw indexed pages fall after a site move, others had a whole site stuck as “crawled, currently not indexed”. A leftover noindex setting and weak internal links are the first things worth ruling out. Our guide to finding and fixing crawl errors in Search Console covers the error side in detail.

robots.txt vs noindex vs canonical: which should I use?

ToolUse it toWatch out for
robots.txtControl which pages Google may crawlIt is not a reliable way to keep a page out of results
noindexKeep a page out of the indexGoogle must be able to crawl the page to see the tag, so do not also block it in robots.txt
Canonical tagPoint to the preferred version of duplicate pagesIt is a hint, and Google can choose a different page

How do I check whether Google has indexed a page?

  1. Open Search Console and paste the URL into the URL Inspection tool.
  2. Read the verdict, then open the live test if you changed the page.
  3. For a wider view, check the Page indexing report for groups of affected URLs.
  4. Use a quick site: search as a rough check only, because it is not exact.

After you fix a problem, ask Google to look again by following its guide to asking Google to recrawl URLs.

Why an Indexed Page Might Still Not Rank

Getting indexed is not the finish line. It’s the starting line.

Once a page is indexed, it’s eligible to appear in search results, but eligibility isn’t ranking. Google still has to decide it’s a better answer than every other indexed page competing for the same search.

A handful of the biggest ranking factors:

  • Content relevance and depth: Does the page actually answer the question as well as, or better than, the pages already ranking?
  • Backlinks: Links from other trusted, relevant sites are still one of the strongest trust signals Google prefer.
  • Core Web Vitals: Google measures loading speed, interactivity, and visual stability, and slow or unstable pages get pushed down.
  • Mobile-first indexing: Google mainly evaluates the mobile version of your site, so a page that looks fine on desktop but breaks on a phone can still lose ranking ground.
  • HTTPS: Secure sites are treated as a baseline trust requirement, not a bonus.
  • Structured data: Marking up your content with schema helps Google understand exactly what a page is about, which supports both classic rankings and features like rich results.

A new page can be fully indexed within days and still take months to rank competitively, especially for a newer site with fewer backlinks. That’s a normal part of the process, not a sign that something is broken.

How to Check Where Your Own Pages Stand

You don’t need to guess. Google Search Console shows you exactly what’s happening at each stage, for free.

  1. Open the URL Inspection tool and enter a specific page. It will tell you whether that page is indexed right now, and if not, why.
  2. Check the Indexing → Pages report for a site-wide view, grouped by status: indexed, crawled but not indexed, discovered but not indexed, blocked by robots.txt, and more.
  3. Review your XML sitemap under Sitemaps, and confirm it only lists pages you actually want indexed.
  4. Check your robots.txt file directly at yoursite.com/robots.txt to make sure it isn't accidentally blocking pages you want found.
  5. Look for canonical tag conflicts. If a page has a canonical URL pointing to a different page, Google may index that other URL instead of the one you meant to rank.

If a page shows as indexed but you’re not seeing it rank for the terms you expected, that’s a ranking problem, not an indexing problem, and the fixes above won’t touch it. That’s when it’s time to look at content quality, internal linking, and backlinks instead.

Practical Example: A Stuck Blog Post

Step 1
Page published
→
Step 2
Search Console shows "Crawled – not indexed"
→
Step 3
No internal links point to it
→
Step 4
Add 2–3 internal links → indexed within days

Does a sitemap guarantee indexing?

No. A sitemap helps Google discover your pages, but it does not make Google index them. Internal links from pages that are already indexed are just as important, and they tell Google a page matters.

Can Google index pages that rely on JavaScript?

Yes, but rendering is an extra step. Google explains the basics in its JavaScript SEO guide. Keep your key text and links in the rendered page so Google can read them, and test with URL Inspection.

Where do AI Overviews and AI Mode fit in?

Google says the same SEO basics apply to its AI features. A page needs to be indexed and eligible to show a snippet, which is covered in its AI features documentation. In other words, a page that is not indexed cannot appear in AI Overviews either.

Is your mobile version holding back indexing?

Google crawls with its smartphone crawler, so content missing from your mobile pages can be missed. Our guide to mobile-first indexing and Googlebot Smartphone explains how to test it.

Getting Started With a Technical Check

If you’ve never done a Search Console audit, a good starting point is checking your mobile-first indexing readiness, since that affects how Google evaluates every page on your site, not just one. From there, the Pages report will show you exactly which URLs need attention.

If this feels like more than you want to manage on your own, our SEO services include a full crawl, index, and ranking audit as a starting step, so you know exactly where each page on your site actually stands before anything else gets changed.

Conclusion

Crawling, indexing, and ranking are three separate jobs, and Google can stop at any one of them without warning you why. A page can be perfectly written and still sit uncrawled. It can be crawled and rejected from the index. It can be indexed and still lose to a hundred other pages competing for the same search. Knowing which stage you’re actually stuck at, rather than guessing, is what separates a five-minute fix from weeks of changing the wrong thing.

The fastest way to find out where your own site stands is to open Search Console and check the Pages report today, rather than waiting for traffic to drop before looking.

Want a second set of eyes on it? SoftCrust runs a full crawl, index, and ranking audit for every new client, so you get a clear answer on exactly which pages are stuck, why, and what to fix first, instead of a generic checklist. Get a free digital marketing audit and we’ll walk you through what Google is actually doing with your site right now.

Frequently Asked Questions

Crawling is Google visiting and reading a page. Indexing is Google deciding to store that page in its search database. A page can be crawled without ever being indexed.

Indexing only means the page is eligible to appear in search results. Ranking depends on separate factors like content quality, backlinks, and how well the page matches what searchers are looking for compared to competing pages.

Some pages get indexed within a day or two. Others, especially on newer sites with few internal or external links, can take several weeks. Adding internal links from established pages usually speeds this up.

Accordion Yes. A misconfigured robots.txt file, an accidental noindex tag, or a canonical tag pointing to the wrong URL can all block indexing without any obvious error message. Checking these directly is the fastest way to rule them out.

Make sure the page is not blocked, link to it from pages that are already indexed and include it in your sitemap. Then use the URL Inspection tool to request indexing.

It means Google fetched the page but has not added it to the index. It is not an error. Improve the page content, add internal links from stronger pages and check for duplicates.

No. Robots.txt controls crawling, while noindex keeps a page out of the index. Google must be able to crawl a page to see its noindex tag.

It is not required, but it helps Google discover pages, especially on new sites. A sitemap does not guarantee indexing.

Written by

Tauseef Shah

Digital Marketing Expert at SoftCrust with hands-on experience running SEO audits and technical fixes for client websites.