We Published Six Markdown Files For AI Crawlers. Then We Found Our Own Site Was Blocking GPTBot.

We published six markdown companion files this week, one for each of six podcast episodes. Then we found out our own site had been refusing OpenAI’s crawler for who knows how long. Here is the whole thing, including the part that went wrong.

What we actually built

Every one of these six episodes now has a plain-text version at the canonical path plus a .md extension. So the Patrick Stox episode lives at https://unscriptedseo.com/patrick-stox-on-building-in-the-geo-era/ and its companion lives at https://unscriptedseo.com/patrick-stox-on-building-in-the-geo-era.md.

Each file contains a machine-readable header, a one paragraph abstract, the key claims with a named speaker and a verbatim quote for each, a table of entities, the questions the episode answers phrased as questions, cross-links to related episodes, and a citation block. The order is deliberate. The citable material sits in the first couple of thousand tokens so that anything which truncates keeps the argument and loses only the evidence.

The objection everybody raises, and why it does not apply here

The standard argument against this is duplicate content. Two URLs, same words, wasted crawl budget.

That is not what these are. The companion file is denser than the page it accompanies. The HTML post is a recap written for a person. The markdown file has structured claims with attribution, an entity table, the questions answered, and verbatim quotes that do not appear in the post at all. It is a different artifact about the same conversation.

That distinction matters for what you do next. We considered adding a canonical header pointing the markdown at the HTML page, and decided against it. A canonical asserts that one thing is a copy of another. That would be false here, and it would suppress a document that might legitimately stand on its own.

Three guests, three positions, no consensus

The funny part is that the people we interviewed do not agree about this at all.

Dan Kurtz says do it. “As SEOs and content marketers, we’ve been doing that for decades. We call it multi-purposing. Bots have a preferred content type. Technically, they’re just an additional customer.” His view on risk is that the downside is bounded: “the worst thing that can happen is they pass it. They fly over it.”

Malte Landwehr says do not. “As an SEO, I don’t want the same content on two different URLs. It wastes crawl resources. If humans land on the .md version, there are no links to click. There’s nothing for them to do. It’s a horrible experience.” He would serve markdown by user agent at the same URL instead, and he is candid that this is a form of cloaking he would only do for LLM crawlers.

Chris Green says it depends on your stack, and nobody is checking. On a simple site it is “almost a non task, you can kind of do it automatically, never think about it again.” On a million page site held together with duct tape it is a governance problem. And his sharpest point is the one nobody has an answer to: “there isn’t a single bot or service out there that will say, hey, your markdown files are wrong.”

Three practitioners, three defensible positions. That is usually a sign that the question is open rather than solved, which is exactly when running your own test beats reading another opinion.

The thing we found by accident

While testing whether the files were reachable, we found that unscriptedseo.com had been returning 429 Too Many Requests to GPTBot. Every page. The homepage included.

Not blocked in robots.txt, which explicitly allows everything. Not a plugin, not an .htaccess rule, not a security setting we could find anywhere. Three sibling sites on the same hosting account serve GPTBot a clean 200. Only this one refuses.

The decisive clue was a missing header. The 429 response carries no X-Powered-By: PHP header, and the 200 does. PHP never runs. The request is refused at the LiteSpeed edge before WordPress is ever reached, which points at a per-site rate ceiling rather than a rule. It is the largest site on the account and GPTBot crawls harder than any other agent.

Two things follow. First, go and check your own logs, because you can be doing everything right at the content layer and still be invisible at the transport layer. Second, notice that the static markdown file was reachable the whole time, because a text file does not execute PHP. Serving bots something cheap is not only a courtesy. It is a way of not hitting the ceiling.

What we are actually testing

Here is the honest state of it. We do not know whether Google will index these files. We do not know whether it will treat them as competing with the HTML page or as a separate document. There is a widely repeated line that Google does not want markdown versions of your pages, but that is a statement about preference, not a documented behaviour, and nobody we can find has published what actually happens.

So we left them indexable. No noindex, no canonical, listed in their own sitemap at https://unscriptedseo.com/sitemap-markdown.xml and referenced from robots.txt. If the answer is that Google ignores them entirely, that is worth knowing. If the answer is that they rank, that is worth knowing a great deal more.

We will report the result either way.

Run the same test yourself

Five prompts. They go in order, and the third one is the one people skip.

1. Find out whether AI crawlers can even reach you

Before anything else, check you are not invisible. Ask your assistant:

Fetch https://YOURSITE.com/ using a normal browser user agent, then tell me what a server would return to GPTBot, ClaudeBot, PerplexityBot and Google-Extended. If you cannot vary the user agent yourself, write me the exact curl commands to run and tell me what each status code would mean. I am looking for any agent that gets a different response from a browser.

2. Build a companion file that is richer than the page, not a copy of it

The point is not to duplicate. The point is to write the version a machine would want:

Read this page: [your URL]. Produce a markdown file that is denser than the page itself. Include: a machine-readable header with the canonical URL and publish date, a one paragraph abstract, the key claims each with a named source and a verbatim quotation, a table of every entity mentioned with disambiguating detail, and a list of the questions this page actually answers, phrased as questions. Do not paraphrase anything into quote marks. Order it so the most citable material comes first, because a fetcher that truncates will keep the top and lose the bottom.

3. Baseline your bot traffic before you publish anything

Without a before number, the after number tells you nothing:

I am going to publish machine-readable companion files and I need a baseline. Using my server access logs from the last 30 to 90 days, split requests into two buckets: traditional crawlers such as Googlebot and Bingbot, and AI crawlers such as GPTBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended. Give me the ratio, tell me how to verify a crawler is genuine rather than spoofed by checking its IP range, and tell me what size of change over the next month would actually be meaningful.

4. Check whether the search engines index it

This is the open question. Run it in a few weeks:

Search for site:YOURSITE.com filetype:md and also search for a distinctive sentence that appears only in my markdown companion file and not on the HTML page. Tell me whether the markdown file appears in the index at all, whether it ranks for anything, and whether it is competing with or complementing the HTML page it accompanies.

5. Stress-test the file for attribution errors

The failure mode that matters most in an interview file:

Read https://YOURSITE.com/your-file.md and answer these three questions: who said each of the key claims, where did the host disagree with the guest, and is there anything in this file you would attribute to the wrong person if you were skimming. I want to know if the structure is doing its job.

What we would do differently already

Baseline the logs before publishing, not after. We have an after number and a rate-limited crawler, which is a worse experiment than it needed to be.
Add a copy button to the prompt blocks. Selecting text out of a code block on a phone is exactly the friction that stops anyone trying.
Put a last-updated stamp in every companion file. A file that silently drifts out of sync with its page is worse than no file, and Chris Green is right that nothing external will catch it.

If you run this on your own site, tell us what happened. Especially if it contradicts what we find.