SEO SOP: Validate the Not-Indexed Bucket Before You Fix It

Search Console parks a pile of pages in Crawled – currently not indexed and everyone assumes the same thing: Google is refusing to index good pages. AJ Kohn of Blind Five Year Old went and checked. He posted a spreadsheet on LinkedIn titled “Crawled – currently not indexed Validation”: one row per client, how many pages sat in the bucket, and what share of them turned out to be indexed anyway. About 24 clients. The results ran from 99.8% actually indexed down to 0.5%. Half his book could have ignored the report entirely. A quarter of it was sitting on a real problem. The framing here is his. What follows is a way to run it on your own sites.

Objective

Take every URL Search Console has parked in Crawled – currently not indexed or Discovered – currently not indexed, check each one’s real index status through the URL Inspection API, and produce a single number: the share of that bucket that is actually indexed. Then split the genuinely not-indexed remainder into pages that should be indexed and pages that should never have been submitted, and fix each group at the source. Budget half a day for a small site. Budget by the quota for a big one, because URL Inspection is capped at 2,000 queries per day per property.

Run it when a client points at the bucket count and asks what you are doing about it, when the bucket grows by a noticeable chunk in 30 days, after any migration or new post type, or before you spend money on an indexation project you have not proven you need.

Key Steps

1. Do the quota math before you plan anything. You need Owner or Full user on the property and OAuth credentials for the Search Console API with the webmasters.readonly scope. URL Inspection allows 2,000 queries per day and 600 per minute, per property, not per project. A 6,000 URL bucket is a three day job. Work that out now rather than on day two.
2. Pull the URL list, then stop trusting it. Indexing → Pages → click the bucket → Export. Keep Crawled and Discovered as separate columns, because they are different failure modes. Then read Google’s own caveat: the example list “is limited to 1,000 items, and isn’t guaranteed to show all URLs in a given status.” If your bucket says 40,000, that export is 2.5% of it, chosen by a process you do not control.
3. Build your own URL universe instead. Your sitemap index is the list of URLs you told Google to index, which is exactly the population you care about. Walk every child sitemap, collect every loc, and tag each URL with the child sitemap it came from. That one column is what later surfaces “all seven tag archives are stuck.” On WordPress, join post type and status from /wp-json/wp/v2/. Union in the UI export, and flag anything present there but absent from your sitemap, because that is a URL Google found some other way.
4. Inspect every URL in bulk. The Page indexing report has no API. The URL Inspection API does, and it returns coverageState as the same display string the UI shows: Submitted and indexed, Crawled - currently not indexed, URL is unknown to Google. That is the whole trick. You cannot download the report, but you can rebuild it from the bottom up over a list you supply, which means you pick the population instead of accepting a sample. Keep verdict, coverageState, googleCanonical, userCanonical, pageFetchState, robotsTxtState, indexingState, lastCrawlTime and sitemap. Throttle to about eight requests a second and write results to disk as you go, so a crash at URL 1,400 does not cost you the quota twice.
5. Score it. Percent actually indexed equals the URLs whose coverageState is Submitted and indexed or Indexed, not submitted in sitemap, divided by the URLs in the bucket you inspected. Everything else counts as not indexed. Score Crawled and Discovered separately as well as combined. Crawled means Google fetched the page and passed. Discovered means Google never fetched it at all, which on a small site usually points at the sitemap. If you scored 1,000 of a 40,000 bucket, print that denominator on the same line as the percentage.
6. Triage the remainder into two piles. Every URL that really is not indexed either should be indexed and is not, or should never have been submitted. Apply these tells in order and stop at the first match: userCanonical differs from the URL, so you declared it a duplicate yourself; googleCanonical differs, so Google picked something else; it is not a 200; indexingState is blocked by meta or header; robots disallows it; the URL is a thin archive type such as a tag, author, paged or generated permalink. Anything left over is real content that should be indexed. Before any of that, group the stuck URLs by source sitemap. If one child sitemap is 100% stuck, you just found the fix in a single query.
7. Fix at the source, then segment and re-check. For the never-submit pile the fix is the sitemap, not Search Console. Exclude the post type or taxonomy and add noindex to the pages themselves. In Yoast that is Content Types or Categories & Tags, set “Show X in search results” to off, which removes it from the sitemap and adds noindex in one switch. In All in One SEO it is Sitemaps → Sitemap Settings. On core WordPress with no SEO plugin, filter wp_sitemaps_add_provider to drop the provider and use the wp_robots filter for noindex, rather than echoing a second robots tag into wp_head. For the should-be-indexed pile, put those URLs in a small standalone sitemap of their own, submit it separately, and re-inspect the same list in seven days. Search Console reports indexation per sitemap, so that cohort gives you a clean before and after.

Cautionary Notes

• Test Live URL does not answer this question. Open a URL from the drilldown, press Test Live URL, and you get “URL is available to Google” with a green check. That is a crawlability result. It means Googlebot can fetch the page right now. The index status never moved and still reads “URL is not on Google” on the left panel. That green check is probably why so many people conclude the report is broken.
• The Indexing API is not the fix. It is restricted by policy to pages carrying JobPosting or BroadcastEvent markup. It will accept an ordinary URL, return success and burn quota, and the metadata lookup will still 404. Do not sell it as a solution. To nudge a sitemap, resubmit it.
• coverageState is a display string, not an enum. Google can reword the labels without notice. Match on normalised substrings and log anything you do not recognise instead of switching on an exact list.
• Do not blanket-noindex every archive. Tag archives and author archives on a single-author site are almost always junk. Category archives often are not. Check each type against the data before you switch it off.
• You can only ask about URLs you already know. If Google discovered a URL you have never seen, a stray parameter or an old slug, it will not appear in your rebuild. Keep the occasional manual UI export purely to harvest those, then feed them in.

Tips for Efficiency

• Group by source sitemap before anything else. A child sitemap that is 100% stuck is the strongest single signal you will get, and it takes one query.
• Use searchanalytics.query with the page dimension as a cheap prefilter. Any URL with impressions is almost certainly indexed, which saves you inspection quota for the ambiguous ones.
• Checkpoint the run and store results keyed on URL plus inspection date. After the second run you own a week over week diff, which is the thing the UI has never been able to show you.
• Score every property you manage, not just the one that prompted the question. The whole point of AJ’s spreadsheet is the spread between clients.

Prompts to run each phase

Replace the bracketed placeholders. These are written to hand straight to an AI assistant with file and shell access.

Build the URL universe

You are helping me audit indexation for <DOMAIN>.

1. Fetch the sitemap index at <SITEMAP_URL> and recursively expand every child
   sitemap into a flat list of URLs.
2. For each URL record: url, source_sitemap (the child sitemap filename), lastmod.
3. If the site is WordPress, also query /wp-json/wp/v2/ for posts, pages, every
   registered custom post type, categories, tags and users. Join post_type,
   post_status and title onto matching URLs.
4. I am attaching the Search Console exports for "Crawled - currently not indexed"
   and "Discovered - currently not indexed". Union them in. Add a column
   gsc_bucket with values crawled, discovered or none, and a column in_sitemap.
5. Output <DOMAIN>-url-universe.csv. Print row counts: total, per source_sitemap,
   per gsc_bucket, and how many bucket URLs are in no sitemap at all.

Only include URLs you actually retrieved. Do not guess at any URL.

Run the inspections and score the bucket

Inspect every URL in <DOMAIN>-url-universe.csv against the Search Console
URL Inspection API.

Property: <GSC_PROPERTY>   (exact form, e.g. sc-domain:example.com)
Endpoint: POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect
Scope:    https://www.googleapis.com/auth/webmasters.readonly

Hard cap 2000 requests per day and 600 per minute PER PROPERTY. Throttle to
~8 req/sec. Checkpoint so tomorrow resumes rather than restarts. Retry 429 and
5xx with backoff, never retry 403 or 404. Append each result to
<DOMAIN>-inspection.csv immediately.

Capture from inspectionResult.indexStatusResult: verdict, coverageState,
robotsTxtState, indexingState, pageFetchState, lastCrawlTime, googleCanonical,
userCanonical, sitemap, crawledAs. Plus url, source_sitemap, gsc_bucket and
http_status from a plain HEAD request.

Then score it. Treat "Submitted and indexed" and "Indexed, not submitted in
sitemap" as INDEXED, everything else as NOT indexed. Output:
  1. bucket | URLs inspected | actually indexed | % actually indexed, with one
     row for Crawled, one for Discovered, one combined.
  2. The denominator stated plainly: how many URLs GSC claims are in each bucket
     versus how many I inspected. Label it a sample if it is one.
  3. Every distinct coverageState value with counts.
  4. The same breakdown grouped by source_sitemap, sorted by percent stuck.
     Call out any child sitemap where 100% of its URLs are stuck.

Classify the remainder

Classify every NOT-indexed row in <DOMAIN>-inspection.csv into exactly one
bucket. Apply in order, stop at the first match:

  SELF_DUPLICATE       userCanonical set and != url
  GOOGLE_PICKED_OTHER  googleCanonical set and != url
  NOT_200              pageFetchState in (NOT_FOUND, SOFT_404, REDIRECT_ERROR,
                       SERVER_ERROR, ACCESS_DENIED) OR http_status != 200
  BLOCKED_META         indexingState in (BLOCKED_BY_META_TAG,
                       BLOCKED_BY_HTTP_HEADER)
  BLOCKED_ROBOTS       robotsTxtState = DISALLOWED
  THIN_ARCHIVE         url matches /tag/, /author/, /page/<n>/, a date archive,
                       or <CUSTOM_PATTERNS>
  SHOULD_BE_INDEXED    everything else

Output counts per class, the headline split of "should be indexed" versus
"should never have been submitted", and for the never-submit pile group by
source_sitemap so I can see which sitemap to fix. List the SHOULD_BE_INDEXED
URLs in full, because that is the only pile I act on page by page.

Why this one matters

The spread in AJ’s spreadsheet is the whole argument. Twelve of his 24 clients came in above 90% actually indexed, which means the report was almost entirely wrong and the sensible move was to stop worrying and go spend the budget on rankings. Five came in under 50%, where the report was telling the truth and there was real work to do. The median was 89.95%. Nobody in that spreadsheet knew which group they were in until somebody counted.

We ran the same method across three of our own properties and landed at the wrong end of it. On seoarcade.com, 123 URLs in Discovered – currently not indexed, every one inspected, zero actually indexed. On unscriptedseo.com, 48 URLs across both buckets and one indexed. On unscriptedsmallbusiness.com, five and none. One out of 176 across the network.

Which sounds like a disaster until you look at what the URLs were. Sixty of the 123 on seoarcade should never have been submitted in the first place. Fifty were answer permalinks that each carried a canonical pointing at their parent question, so we had declared them duplicates ourselves and a sitemap was submitting every one anyway. Five were emoji slugs redirecting to the wrong post. Two were hard 404s still listed in the page sitemap. Google was not failing. Google was correctly declining pages we asked for by mistake.

On unscriptedseo the same shape turned up faster: all seven tag archives in the tag sitemap were stuck. Seven out of seven. That is not an indexing problem, that is a sitemap that keeps submitting thin archives. Score the bucket first and you find out which conversation you are actually having.

Sources & Relevant Episodes

• The idea and the validation exercise are AJ Kohn‘s, posted on LinkedIn. More from AJ at Blind Five Year Old.
• Google on the report’s limits: the Page indexing report documents the 1,000 example URL cap. Search Console API usage limits documents 2,000 queries per day per property for URL Inspection.
• urlInspection.index.inspect is the endpoint, and the Indexing API quota page is where the JobPosting and BroadcastEvent restriction is written down.
• Part of the SEO SOP library.