SEO SOP: Mine Your 404 Report for AI-Hallucinated URLs

Your 404 report is not just a list of things you broke. Some of those URLs never existed on your site at all, because ChatGPT invented them and sent real people there. This SOP turns that into a build list. It comes out of a Unscripted SEO conversation with Stephanie Long, founder of Junction Jack and Mrs. SEO, where she and I traded evidence that the machines have to be checked. It takes an afternoon, and the pages you get out of it already have traffic pointed at them.

Objective

Separate your 404s into two piles: links that used to work and are now broken, and URLs that never existed on your site because an AI assistant made them up. The second pile is a demand signal. Each invented URL is a page a model decided you should have, published as an answer to a real person, who then clicked through to nothing. You finish with a ranked build list of pages that already have traffic aimed at them, plus a set of redirects that stops the near-misses leaking. Budget an afternoon for the first pass and an hour a month after that.

Key Steps

1. Open Google Search Console and go to Pages, then Not indexed, then “Not found (404)”. This is the fastest source because it is already deduplicated and already counted. If you also have server logs or a 404 plugin, pull those too, since they capture hits GSC never sees. Set the window to the last 90 days.
2. Sort by hits and export the list. Volume is the whole ranking signal here. A URL that got invented once is noise. A URL that keeps getting handed out in answers is a page with recurring demand behind it. Work top down and stop when the counts go flat.
3. Split the list into “used to exist” and “never existed.” For each URL, ask whether it was ever a real page on the site. Check your CMS, your sitemap history, and the Wayback Machine if you are unsure. Anything that was never published is a candidate invented URL. (inferred: the split is the mechanic of the play, though the sourcing method is standard 404 triage.)
4. Fix the “used to exist” pile the ordinary way and set it aside. Redirect the ones with an obvious successor, restore the ones that should not have gone, and leave the genuinely dead ones dead. This is routine housekeeping, and it is not what this SOP is for. Getting it out of the list stops it obscuring the interesting half.
5. Read the invented URLs as a topic list. Group the slugs. The model was not choosing at random. It picked a path that fits your site’s structure and a topic it believed a company like yours would cover. Repeated shapes tell you what buyers are being told you offer. (inferred: grouping is a practical step; the source claim is that the model thought the page was good enough to exist.)
6. Build the invented page at that exact URL where it is one you would actually want. The path matters as much as the content, because the traffic is already addressed to it. Judge each candidate on merit first. If it is a resource you would have built anyway, build it. If it misrepresents what you do, do not manufacture it just because a model asked.
7. Redirect the near-misses to the closest real page. Plenty of invented URLs are a slug variant of something you already published. Those get a 301 to the real page, not a new page. Only build when there is nothing genuine to point at, otherwise you end up with duplicates competing with your own content.
8. Re-pull the report monthly and keep score. The invented URLs move as the models retrain and as your site changes. Track hits before and after on the pages you built, so you can show that a URL a machine imagined turned into real sessions. (inferred: the monthly cadence and measurement are execution detail, not something stated on the episode.)

Cautionary Notes

• Do not build every invented URL. The model is guessing. Some of those guesses describe services you do not offer, and publishing them to catch traffic is how you end up ranking for work you cannot do. The filter is “would I want this page anyway,” not “did a machine ask for it.”
• A 301 beats a thin new page every time. If the invented slug is a variant of something live, redirect it. Building a second page for the same intent splits your own signals.
• GSC does not see everything. Its 404 list is sampled from what Google crawled. People arriving from a chat answer often never touch Google at all, so server logs and a 404 plugin will show hits the Search Console report misses entirely.
• Hit counts here are small by SEO standards. This is not a traffic play at scale. It is a high-intent play: the person clicking arrived because an assistant told them your site had the answer, and they hit a wall instead.
• Some invented links are not on your domain at all. Stephanie clicked through every citation on a single AI query and found all of them broken. Fabricated citations pointing at other people’s sites are the same failure from the other side, and you cannot fix those from your own 404 report.
• Do not report this as an AI ranking win. Building the page captures traffic that was already being sent. Whether the model then starts citing the real page is a separate question you should measure separately.

Tips for Efficiency

• Filter your server logs by AI assistant referrers and user agents first. It gives you a much cleaner starting list than raw 404s and takes one grep.
• Ask the assistants directly. Prompt ChatGPT and the others for a resource you would plausibly publish and watch which URL they hand back. It surfaces the invented paths before anyone hits them.
• Keep the built pages in one tracked group in analytics so the before-and-after is one screenshot, not a rebuild each month.
• Run the same pass on a competitor’s domain during a pitch. Invented URLs on their site are pages the market is being told exist in the category, and nobody has built them yet.
• Batch it with your normal broken-link cleanup. You are already in the 404 report; sorting it into two piles instead of one costs almost nothing extra.

Why this one matters

The usual reason to look at a 404 report is guilt. Something broke, go fix it. The reframe is that the report is now partly a record of what machines are telling people about you.

My side of it on the episode:

“I just saw a post from, I think it was Tim Soulo, or it might have been Patrick Stox out of Ahrefs, literally recommending: hey, check your 404s on your site. Not because it’s broken, but because it was imagineered by ChatGPT, and they sent people to pages that don’t exist on your site. Use that as reverse engineering. They thought that it was good enough to exist, so you might as well add that resource, to plug in the traffic that they’re going to send to imaginary land.”

Stephanie’s answer was one word: “Exactly.” She had come at the same problem from the other end, as the person doing the clicking:

“I wrote a whole post about how I clicked through every single link on a query I did, and they were all broken links. So I don’t even know where that came from.”

That is the loop. Models invent URLs, real people follow them, and the only party who can see the demand is the site owner staring at a 404 count. Her larger point in that episode is that the fundamentals did not stop mattering because AI showed up. “SEO is like the batter to your AI. You have to have it together, or you’re going to fail.” A 404 report is about as fundamental as it gets, and it just quietly became a demand research tool.

Sources & Relevant Episodes

• Jeremy Rivera, SEO Arcade – the play is mine, citing an Ahrefs post by Tim Soulo or Patrick Stox recommending you read your 404s as reverse engineering rather than repair work.
• Stephanie Long, Mrs. SEO / Junction Jack – endorsed it on the spot and supplied the evidence from the other side, having clicked every citation on one AI query and found all of them broken. Full interview: Stephanie Long: SEO Is the Batter to Your AI.
• Listen: SEO Is the Batter to Your AI on Unscripted SEO.
• More from Stephanie: junctionjack.co · mrsseo.co · LinkedIn.
• Related reading: SEO SOP: AI Bot Access-Log Analysis – the log-file pass that pairs with this one, and SEO SOP: Get Cited by ChatGPT with Answer Engine Optimization.
• Part of the SEO SOP library.