SEO SOP: Query the Raw LLM API to See What Retrieval Adds

Martin McGarry’s top tip for anyone learning SEO right now is not a tool. It is an experiment you can run from a terminal in an afternoon: ask the bare model the same question you would type into ChatGPT or Gemini, then compare the two answers. He is a freelance consultant at Search Assistance in Leeds, twenty years into SEO and ten of them clean of black hat, and on the Unscripted SEO episode he gave this as his answer to “is SEO dead.” This SOP is that experiment, written so a team can run it against its own money queries this week.

Objective

A side-by-side record, per query, of what the raw model says on its own and what the chat product says with its retrieval layer switched on. The difference between the two answers is the retrieval step, and the sources that step pulled from are the pages your site has to be on, or be cited by, to show up in AI search. You finish with a target list of surfaces for each money query, plus a thirty-second demonstration for clients of why the site still needs positioning.

Key Steps

1. Get an API key for one vendor and open a terminal. VS Code’s built-in terminal is enough, and so is curl or a ten-line Python script. Martin: “You can pop up Visual Studio, and you can access the API in your terminal.” Pick OpenAI or Gemini, whichever chat product your clients actually use.
2. Write down five real questions your buyers ask. Use the questions that map to your money pages, not “what is SEO”. Include at least one fact question (a spec, a price range, a lead time) and one recommendation question (“which X should I buy for Y”). Save them in a text file so the run is repeatable. (inferred – Martin says “a raw question”; the money-query framing is the operational version.)
3. Send each question to the raw completion endpoint with no tools and no web search. No system prompt beyond the question, no browsing flag, no retrieval plugin. Save every reply verbatim with the date and the model name. Martin: “just ask the raw API a question. So that’s the OpenAI LLM, the Gemini LLM, whatever else you use. And just ask a raw question. What you get back is junk.”
4. Ask the identical question in the vendor’s chat product. Same wording, same day. Save the answer, and save every citation, link or “searched the web” indicator it shows you. Those are the evidence for step 6.
5. Diff the two answers. Everything the chat product knew that the raw model did not came from somewhere else. Martin: “then you realize that there’s some background process augmenting the chat results. So when you use ChatGPT or Gemini, there’s something else working in the background.” That delta is the retrieval layer, and it is the only part of the answer you can influence.
6. Trace where the retrieval fetched from. Start with the citations, then run the same query in Google and Bing and compare the top results to the cited set. Martin: “OpenAI has been called Googling.” The overlap tells you which surfaces feed the answer, and that is the whole point of the exercise: “you still learn where you’re needed and you still learn where your site needs to be positioned for that RAG or that augmentation to happen.”
7. Check whether your site is in the fetched set, and build for the gaps. For each query, mark present or absent. Absent means a page to write, a listing to claim, or a citation to earn on the page that was fetched. Re-run the whole file monthly; retrieval sources move. (inferred – Martin describes the diagnosis; the monthly cadence is the operational version.)

Cautionary Notes

• The junk is the point, not a verdict on the vendor. The raw model is stale and unsourced by design. Martin: “the standalone LLM on its own is rubbish, in my opinion. And there’s definitely some, we know there’s RAG in the background.” Treat the raw answer as the baseline you measure retrieval against.
• Do not switch on web search, tools or a retrieval plugin in the API call. Then you are comparing retrieval with retrieval and the delta disappears. (inferred)
• Date every sample. Chat products change what they fetch without announcing it. An undated comparison is an anecdote, not a finding. (inferred)
• The API key is a credential. Keep it in an environment variable, never in the spreadsheet you share with the client. (inferred)
• Do not stop at “SEO isn’t dead”. That is the headline. The deliverable is the list of surfaces per query. Martin’s own framing: “we know SEO is not dead first of all, because these LLMs are having to go out and get extra information from somewhere.”
• Ranking is not the same as being fetched. A page can sit third in Google and never be cited. The citations are the primary evidence; the SERP is the secondary. (inferred)

Tips for Efficiency

• Script the loop once. A file of questions in, a table out: question, raw answer, chat answer, sources. The second run costs nothing, and the monthly re-check in step 7 becomes a five-minute job.
• Set temperature to 0 for the raw call. You want the same junk twice, so the delta you record is retrieval and not sampling noise. (inferred)
• Run two vendors on the same question file. If OpenAI and Gemini fetch from different surfaces, you have two target lists, not one.
• Put the side-by-side in the client’s onboarding deck. A screenshot of the raw answer next to the chat answer explains why the site still needs positioning faster than any slide about AI search.
• While you are in the machine-readable layer, read your own RSS feed and REST API output. Martin’s untested hunch from the same episode: “there’s a lot of talk about .md files and llms.txt files at the minute and it may be simpler than that. Maybe we just all build RSS feeds”. Treat it as a hypothesis, but a clean feed costs nothing while you are already looking at what machines fetch.

Sources & Relevant Episodes

• Martin McGarry, Search Assistance – he framed it as his top tip for anyone learning SEO right now, and as the proof that SEO is not dead: the chat products have to go and fetch, so learn where they fetch from.
• Full interview recap: Words, links and a grace period: Martin McGarry on why black hat is swamping the SERPs
• Episode page: Martin McGarry: an ex-black hat on the spam flooding Google, WordPress exploits and perspective in AI search
• More procedures: the full SEO SOP library