Skip to content
Farm.js v0.1.0 is releasedRead the announcement

I asked 105 developer sites for Markdown. A third answered with HTML.

What 105 developer-tool sites send an AI agent that asks for Markdown, and why the frameworks they are built on are furthest behind.

In this article 6 sections

I asked 105 developer sites for Markdown. A third answered with HTML.

KinfeMichael Tariku · Nov 2026

AI agents read the web for us now. When Claude, Cursor or ChatGPT looks up an API, it fetches a docs page and pays for every byte in tokens.

So I asked a simple question: if an agent asks for Markdown, what does it get?

I picked 105 sites developers read every day (frameworks, build tools, hosting, databases, auth, payments and AI companies) and requested one docs page from each with Accept: text/markdown. I also checked the homepage, a .md URL for the same docs page, /llms.txt and /mcp.

A third of them answer an agent with only HTML.

The short version

What I checkedSites
Docs page answers Accept: text/markdown with Markdown56 of 105 (53%)
Docs page has Markdown at a .md URL61 of 105 (58%)
Either one70 of 105 (67%)
Neither, HTML only35 of 105 (33%)
Homepage answers Accept: text/markdown38 of 105 (36%)
Serves /llms.txt87 of 105 (83%)
Serves /llms-full.txt56 of 105 (53%)
Has an MCP server at /mcp14 of 105 (13%)
All four: Markdown homepage, Markdown docs, llms.txt, /mcp3 of 105 (Nuxt, Turso, Mintlify)

llms.txt is nearly everywhere. A Markdown answer for the page an agent actually asked for is much rarer.

HTML is 35 times bigger

Across the 70 sites that serve the docs page both ways, the median page was 211 KB of HTML and 5.9 KB of Markdown. On the median site, the Markdown was 35 times smaller.

At a rough 4 bytes per token, that's about 53,000 tokens against 1,500 for the same page. Some agent tools convert HTML to text before the model sees it. That conversion is a guess: it has to find the content among the navigation, scripts and class names, and it still downloads the whole page first.

The largest HTML answers to a Markdown request:

SiteDocs page sent as HTML
PostHog1.45 MB
Mistral769 KB
Datadog583 KB
shadcn/ui574 KB (also has a .md URL)
Kubernetes509 KB
Prisma451 KB (its homepage does answer Markdown)
Qwik331 KB
Tailwind CSS330 KB
Axiom316 KB (also has a .md URL)
Angular297 KB

The tools we build with are furthest behind

CategoryDocs answer a Markdown requestMarkdown by either route
AI companies9 of 11 (82%)82%
Hosting6 of 8 (75%)75%
Databases and backends12 of 17 (71%)71%
Auth, payments, SaaS14 of 20 (70%)80%
Frameworks and UI libraries10 of 26 (38%)65%
Build tools and runtimes5 of 23 (22%)43%

The companies selling to agents fixed this first. The frameworks and tools developers build with mostly haven't, at least on their own docs. Nuxt is the exception: its site does all four.

That's backwards. Serving Markdown is a framework job. The framework already knows every route and what each page renders, so each site shouldn't have to bolt it on.

What a framework can do by default

This is why I build Farm.js the way I do. I generated a new app with pnpm create @farm.js/app (version 0.1.74), ran farm build, started it, and ran the same checks without changing any config:

RequestFresh Farm app
Homepage, Accept: text/markdown626 bytes of Markdown (the HTML is 6.8 KB)
Docs page, Accept: text/markdown1.5 KB of Markdown (the HTML is 22 KB)
/index.md, /docs.mdMarkdown
/llms.txt, /llms-full.txtYes
/mcpAnswers initialize
/.well-known/api-catalog, agent skills indexYes
Sitemap, JSON-LDYes

This site runs on Farm and passes all four checks too, which is why it isn't in the totals.

Farm is still v0.x, and a framework default won't fix the sites above that run on something else. But new sites shouldn't have to choose between being readable by people and readable by agents.

Check your own site

Terminal
# Does your docs page answer agents in Markdown?
curl -s -o /dev/null -w "%{content_type}\n" -H "Accept: text/markdown" https://your-site.com/docs

# Is there an index for agents?
curl -s -o /dev/null -w "%{http_code}\n" https://your-site.com/llms.txt

Or run the scanner on your own list of sites. It has no dependencies and takes about a minute for all 109.

How I measured

  • 109 sites, requested on November 16, 2026, from one machine with Node's built-in fetch, in five full runs. Each result is the majority answer across the five. Two sites changed between runs: Deno answered in Markdown in 2 of 5 runs and Netlify in 4 of 5, and the other runs got HTML. So Deno counts as no and Netlify as yes. Every other result was the same in all five runs.
  • Three sites are left out. OpenAI and Contentful refused the requests (403 and 429), and Google's Gemini homepage sent response headers too large for Node's fetch to read. Farm's own site is scanned as a control and left out too, so the totals are out of 105.
  • One homepage and one docs page per site. Other pages on the same site may behave differently.
  • Requests used a generic user agent. Some sites may answer known agents, like ClaudeBot, differently.
  • "Markdown" means a 200 response with a Markdown or plain-text content type and a body over 200 bytes that isn't HTML. Docker answers with a Markdown type but a 16-byte body, so it counts as no.
  • /mcp counts a server at that exact path that answers an MCP initialize request or asks for auth (Prisma and Mintlify ask for auth). Sites that host MCP on another subdomain count as no.
  • Sizes are decompressed response bodies. Token counts are estimates at 4 bytes per token.

The scanner, the per-site results and the script that computes every number above are in the Farm repo. If your site is listed wrong, open an issue and I'll rerun it.

Thanks for building with us.

Keep exploring