SEO SOP: Publishing a Machine-Readable Companion File for AI Crawlers

The debate over whether to serve markdown to AI crawlers has an answer hiding in a habit SEOs have had for twenty years. This SOP distills the flagship play from Dan Kurtz (founder, Miss Pepper AI) on the Unscripted SEO Podcast: publish a stripped, machine-readable companion file for the pages whose facts you most need repeated back correctly, then point the crawlers at it. As Dan framed it — “Bots have a preferred content type. Technically, they’re just an additional customer.”

Objective

Make the facts you care most about — what you do, what you charge, who you serve — cheap and unambiguous for an AI crawler to read, by publishing a plain-text/markdown companion version of your highest-stakes pages at stable URLs and linking to them from the canonical HTML. This is not a replacement for your website and not a ranking hack; it is one more format in the multi-purposing rotation you already run, aimed at the audience that now does most of the crawling. The downside is bounded and Dan says so out loud: “the worst thing that can happen is they pass it. They fly over it.”

Why this is worth an afternoon

Cloudflare’s Matthew Prince reported on 3 June 2026 that automated traffic had crossed over to become the majority of requests on the internet — roughly 57.5% bots. That part is independently verified. Dan’s own numbers are narrower and he attributes them himself: reading his own server logs across sites he controls, he sees traditional Googlebot crawling down around 60% while AI bot crawling and retrieval is up around 80%. Treat that as one practitioner’s telemetry, not an industry benchmark — and then go read your own logs, which is the point of Step 6 anyway.

His argument for acting on it is the least exotic thing in the episode: “As SEOs and content marketers, we’ve been doing that for decades. We call it multi-purposing.” Blog post to infographic to YouTube video to reel to short to press release — because different humans have different preferred formats. Bots have one too.

Key Steps

1. List the pages whose facts you most need repeated back correctly. Not your whole site — the handful where a wrong answer costs you money. Pricing, services, coverage area, who you serve, the “what we actually do” page, key product specs. Five to ten URLs is a real first pass. These are the pages an assistant will paraphrase to a buyer who never visits you.
2. Produce a stripped, structured text version of each. Clear headings, bullet lists, plain declarative sentences, no design, no nav, no modals, no cookie banner — markdown or clean plain text. Jeremy’s framing of the format trade-off on the episode: a full HTML page costs a bot roughly three times longer to load than the tokens and time markdown costs, and markdown is already organized the way the machine likes it. Keep every fact identical to the canonical page.
3. Publish each file at a stable URL, like any other asset. Pick one convention and never move it — the simplest is the canonical path plus a .md extension (/pricing/ → /pricing.md). Serve it as plain text, leave it un-redirected, and don’t block it in robots.txt. (inferred execution detail — Dan specified “these separate markdown files,” not a naming scheme.)
4. Link the file from the canonical page so crawlers discover it. This is the step people skip, and Dan is explicit that it’s the whole mechanism: “that’s just a matter of directing them to it next time they come to the website and saying, hey, this file exists.” A plain visible link in the footer or below the content is enough. Add the set to your sitemap and an index file listing them so there is more than one discovery path (inferred).
5. Baseline your bot traffic before anything changes. Pull the last 30–90 days of server logs and split requests by user agent into two buckets — traditional crawlers (Googlebot, Bingbot) and AI crawlers/retrievers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and friends). Write the ratio down. Without the “before” number, Step 6 tells you nothing.
6. Verify pickup by watching the log ratio, not by watching rankings. After a few weeks, re-run the same split and check three things: are AI agents requesting the .md URLs at all, is the AI-vs-traditional ratio moving, and which files get hit most. That last one tells you which pages to do next. Dan does exactly this — “it’s just looking at server logs and stuff like that, you know, got my own sites that I can see those kinds of tests on.”
7. Put the companion file into your publishing checklist. When the canonical page changes, the companion changes in the same ticket. A stale machine-readable file is a machine-readable wrong answer, and it is more confidently wrong than no file at all.
8. Expand deliberately, page by page, on evidence. Add the next tier only when the logs show the first tier being read. This is multi-purposing, and the same discipline applies: you are adding a format for an audience you have confirmed exists, not generating files at scale for their own sake.

Cautionary Notes

• This is a companion, never a replacement. Google’s public guidance is not to serve markdown in place of your pages. Keep the HTML page canonical and fully intact; the .md file is an additional format sitting alongside it.
• Do not let it drift into cloaking. Same facts, same claims, same prices as the page it mirrors. The moment the machine-readable version says something the human version doesn’t, you have built a cloaking problem instead of a convenience.
• Accept that it might do nothing. That is the honest close on this play and it is Dan’s own: “the worst thing that can happen is they pass it. They fly over it.” Budget it as an afternoon, not a quarter.
• Don’t generalize one practitioner’s log data. The ~60% down / ~80% up figures are Dan’s sites, read from Dan’s logs. The Cloudflare majority-bot crossover is the verified industry-level number. Keep those two things separate when you pitch this internally.
• Stale beats nothing only until it’s wrong. An outdated price in a file built specifically to be quoted verbatim is worse than never having published it.

Tips for Efficiency

• Generate the files from the pages you already have. Don’t rewrite anything by hand — convert the canonical HTML to markdown programmatically, then edit down. The whole first tier is an afternoon, not a project.
• Pick the convention before you build the first file. One naming scheme, one directory rule, one link placement. Retrofitting a convention across thirty files after the fact is where this play dies.
• Save the log query, not the result. Store the user-agent split as a reusable command or saved report so the “before” and “after” comparisons are genuinely identical.
• Sort the AI-bot bucket by file, not just by count. Which companion files get read is a free ranked to-do list for which page to convert next.
• Add it to the multi-purposing checklist you already run. If a post already spawns an infographic, a short and a press release, adding “machine-readable companion” to that list costs nothing organizationally — which is exactly Dan’s argument for why this isn’t a new discipline.
• Watch answers, not just logs. Periodically ask the major assistants what your company charges and who it serves. Whether the answer improves is the outcome the crawl ratio is only a proxy for.

Sources & Relevant Episodes

• Dan Kurtz, Miss Pepper AI — framed serving machine-readable files as ordinary multi-purposing rather than a new tactic, on the grounds that “bots have a preferred content type — technically, they’re just an additional customer,” and argued the investment is worth making because the downside is simply being ignored. The ~60% down / ~80% up crawl figures are his, from his own server logs.
• Read the full interview: Dan Kurtz on bots as a customer segment, markdown, and the agentic flywheel — Unscripted SEO Podcast, hosted by Jeremy Rivera.
• More episodes: Unscripted SEO — 150+ interviews with SEO practitioners.
• Cloudflare’s majority-bot-traffic report (Matthew Prince, 3 June 2026, ~57.5% automated) is the independently verified figure referenced above.