SEO SOP: AI Bot Access-Log Analysis

When clicks stop being attributable, the measurement has to move to somewhere the AI traffic actually shows up — and that place is a file you already own. This SOP distills the strongest executable play from Chris Green (Technical Director and Senior Consultant, Torque Partnership) on the Unscripted SEO Podcast: read your raw server access logs, split the AI bot hits into training, search and user-triggered, and report the user-triggered ones as a leading indicator of AI visibility. As Chris put it — “one of the other treasure troves of information is access logs.”

Objective

Produce a defensible, repeatable measure of AI visibility out of data you already own. You count AI bot hits per content segment, classify each hit by what the bot was actually doing — crawling for training data, crawling for a search index, or fetching the page because a live human asked an assistant a question — and then track that third category over time. The third number is the deliverable. It is not revenue and you must never present it as revenue. It is the closest thing to a visibility signal you can generate yourself, from your own server, without buying a prompt-tracking tool and without arguing about whose share-of-voice methodology is right.

Why this one is worth building

Chris’s case for it is a positioning argument as much as a technical one. The logs are already on the server, nobody is charging you for them, and almost nobody is packaging them into a client deliverable: “in my experience, very few people are bringing that kind of value.” He notes the large tool vendors are moving on the same ground, but that “trying to land that as an external tool vendor is difficult” — the logs are the client’s, and so is the relationship.

The method he is arguing against is prompt tracking: “If all of the other metrics you have will be share of voice or visibility and they’ll be based on prompt tracking and then it gets horribly complex and easy to argue about.” A bot hit is a log line with a timestamp. That is a much shorter argument to have in a boardroom.

Key Steps

1. Settle the data-access question before you promise the deliverable. Find out what you are actually allowed to do with the logs, and specifically whether you may put them in front of an AI tool. This is the step that kills the project after you have sold it, and Chris flags it from his own client base: “The thing I’m reluctant to do at the moment, and that often depends on my MSAs and my security arrangements and stuff, is giving direct access to your data. Some of my clients I just, can’t do. Some I can only do within a certain environment.” Get the answer in writing, then design the workflow around whichever tier you got.
2. Segment your content before you look at a single log line. Group URLs into meaningful segments — templates, product categories, the documentation tree, the money pages — because the segment, not the individual URL, is the unit of analysis. Chris starts here every time: “take an average of like bot hits to a segment of pages”. A regex per segment is enough; write it down and reuse the same one forever, or your comparisons will not be comparisons.
3. Baseline the average bot hits per segment. Pull a fixed window of logs and record the average hit count per segment, per bot, per day. This is the boring step everybody skips and it is the one that makes every later number mean something. Chris calls it exactly that: “That’s probably one of the most simple things you can do.”
4. Run it as a split test rather than a before-and-after. Change one set of pages and deliberately leave a comparable set untouched, then read both. In Chris’s words: “Say you run like a split test. So you say I’ve updated some pages and not others segment those pages. Take a grab of your log files at a certain date and then compare it.” The untouched segment is your control against seasonality and against a bot operator simply changing its own crawl budget.
5. Classify every hit by what the bot was doing, not just by which company sent it. Three buckets: training crawl, search crawl, and a bot acting on a live user’s behalf. That third bucket is the whole point — “So you can at least understand is this bot just crawling for training data or is this bot acting on the user’s behalf?” The vendors differ wildly in how much they help you here: OpenAI, Perplexity and Anthropic publish what Chris calls “fairly good detailed levels” of documentation, while “Google are the least helpful with how they designate AI bots”. Build the mapping table from each vendor’s published bot documentation and keep it in version control, because it moves.
6. Verify the user agent against the published IP ranges before you trust any of it. A user-agent string is a claim, not a fact, and a spoofed one will quietly corrupt the entire analysis. This is the single most important line in the play: “Is it a valid chat GPT hit or is it someone that’s spoofing it? And without telling Claude to check the IP range and the other network conditions, you may start analyzing bot activity on the wrong bot. It could just be a spoofer.” Do the reverse-DNS and published-IP-range check as a mandatory filter step in the pipeline, before aggregation — not as a spot check afterwards (inferred execution detail — Chris specified checking “the IP range and the other network conditions,” not a particular verification method).
7. Move the logs into a queryable store once the volume gets real. “Log files can be enormous.” At that size you cannot eyeball them or paste them anywhere. Chris’s options, in his order: “you need to use like a log file analysis tool or a tool like Splunk or you need to push it into a database and use BigQuery or SQL or to query it.” His preferred route is BigQuery with reporting on top — “You can kind of use Data Studio as like a front end” — and once the data is there the AI assist is a SQL-writing job, not a data-access job. The prompt shape he gives is worth copying verbatim: tell it your field names, then ask it to “segment my pages on this regex and then I want to understand what my running average for bot hits are. I want to find who my outliers are. Like help me write a SQL query that can help identify that”.
8. Report the user-triggered hits as a correlation, in the client’s language, and say the quiet part out loud. The framing Chris hands you is the deliverable’s whole value: “acting on the user’s behalf is a proxy for visibility or being surfaced at least by an AI tool.” So the report line reads like his: “we made some changes over this six months to strengthen your AI search and the, you know, the average daily visits from the user triggered AI bots has gone up.” Then immediately bound the claim, exactly as he does: “that’s not attached to revenue, but you can say, well, this correlates with our involvement. We are making your site more findable potentially. And that’s a good thing to ground yourself on.” Volunteering the limitation is what makes the number survive scrutiny.

Cautionary Notes

• An unverified user agent will point you at the wrong bot entirely. This is the failure mode that invalidates the whole report rather than just adding noise, which is why IP verification is a step and not a tip. “It could just be a spoofer.”
• The model will find a trend that is not there. Chris is blunt about handing log analysis to an assistant: “It’s very, very easy to start hallucinating things or it might start seeing a trend in the data that’s not really there.” Ask for the query and the raw counts, and sanity-check the counts yourself before anything reaches a slide.
• Most of a log file is noise. Chris’s warning: “depending on your log format, you can have a lot of data in there that’s just noise.” Decide what you are excluding — static assets, health checks, your own monitoring — and write the exclusion into the query so it is applied identically every run.
• The bot taxonomy moves under you. On how many bot types OpenAI runs, Chris said: “I think there’s three now. I think there might even be more. There’s user search and then training for chat gpt. I say that with confidence it changes all the time.” Re-read the vendors’ published bot lists on a schedule; a classification map you built six months ago is quietly wrong.
• Google will not make this easy. “Google are the least helpful with how they designate AI bots”. Expect a coverage gap on the largest player and label it as a gap in the report rather than papering over it.
• Never let this number get attached to revenue. It is a proxy and a leading indicator. The moment it is presented as attributable performance you have built exactly the argument-generating metric Chris is trying to get away from — and you will lose that argument the first time it dips.
• BigQuery is not free. “Obviously it costs, you have to pay data processing.” Partition by date, select the columns you need, and price the recurring query before you promise a monthly report.
• Client security terms may simply forbid the AI-assisted version. Some environments will allow the analysis but not the tooling. Know which one you are in before the kickoff call, not after.

Tips for Efficiency

• Know how you would do the analysis without the AI first. Chris’s rule for anything analyst-shaped: “if you kind of know how you would do it without Claude, you can get Claude to do it much more effectively for you”. If you cannot describe the query in words, you cannot review the one you get back.
• Give it your field names, then ask for the query — not the answer. The failure mode is asking a model to interpret data. The working pattern is asking it to write SQL you can read, run and re-run. That is exactly the prompt Chris describes, and it keeps the raw data out of the tool entirely when your MSA requires that.
• Ask for outliers, not only averages. Chris asks for both in the same breath — the running average tells you the trend, the outliers tell you which segment to go look at. The outlier list is a free ranked to-do list.
• Store the segmentation regex, the exclusion rules and the query as one saved artifact. Every month you re-run the same thing. If any of the three drifts, the comparison is worthless and nothing will warn you.
• Put a dashboard on it once, not a deck every month. Data Studio (Looker Studio) over BigQuery turns this from a recurring analysis into a link you send. The build cost is one-off; the reporting cost after that is close to zero.
• Pair the chart with the sentence. The number does not sell itself — the framing does. Keep “acting on the user’s behalf is a proxy for visibility” and “this correlates with our involvement” on the same slide as the trend line, every time.

Sources & Relevant Episodes

• Chris Green, Torque Partnership — Technical Director and Senior Consultant, running global SEO, GEO, measurement and AI search programmes for Fortune 500 brands. He framed access logs as the measurement layer almost nobody is selling — “one of the other treasure troves of information is access logs” — and prefers user-triggered AI bot hits over prompt-tracking share of voice precisely because the log line is harder to argue with. Every quote in this SOP is his, from the episode below; the caveats are his too, and they are here because he volunteered them.
• Read the full interview: Chris Green: Do Google’s Patents Still Hold Up in an Agentic Web? — Unscripted SEO Podcast, hosted by Jeremy Rivera.
• Related SOP: SEO SOP: Markdown Companion for AI Crawlers — the companion play. That one changes what the bots can read; this one measures whether they came back.
• More episodes: Unscripted SEO — 150+ interviews with SEO practitioners.

Server logs cover the gap, because there is a lot of what Search Console cannot show you.