Lockzone

llms.txt returns HTML: check content before calling the docs usable

By wicketwarden, with an AI assistant ยท Published

A successful HTTP response can contain the wrong resource. If /llms.txt returns your app's HTML shell, the URL is reachable, but it has not supplied the documentation an agent expected.

What the file is supposed to provide

The llms.txt proposal describes a Markdown overview of a site or path, with context and links to useful material. It is a proposal for helping clients navigate content. Publishing the filename alone establishes neither usable documentation nor search visibility.

Check the returned bytes as well as the headers. A text/html response beginning with a document shell, an empty app mount and script bundles is a different result from a readable Markdown overview. A text content type can also be mislabeled, so it is evidence to inspect rather than a sufficient pass condition.

Make one small, reproducible check

  1. Read the site's access policy and use an honest client identity. Request the documented URL once with a finite timeout; record the UTC time, original URL and any redirect.
  2. Retain the status, content type and response body or its digest. Inspect the body for the expected project heading, useful instructions and documentation links. Distinguish a shell from a substantive HTML document: HTML may contain useful information even when it is the wrong format for this route.
  3. Classify the result explicitly: readable overview, HTML shell, challenge page, missing file, authentication required, server error or no HTTP response. A challenge served with 200 remains a challenge. Stop there rather than changing clients to get through.
  4. If the overview is readable, follow at most one relevant, policy-permitted documentation link and check whether it answers the intended question. That tests one path through the docs; it does not validate every link or the service itself.

Fix the route that owns the response

A common cause is a single-page application's fallback serving index.html for paths that lack a matching static file. Check the deployed artifact and route order. Serve the actual llms.txt file before the app fallback, with a suitable text content type. Then check through the public proxy or CDN, where a cached fallback may survive a correct origin change.

For a missing documentation resource, return a clear missing-resource response rather than the app shell. For an existing resource, include enough context to identify the product and a small set of relevant links. Do not copy a template claiming endpoints that the deployment does not expose.

A dated example, with its limits

In our November 7, 2026 field follow-up, two of 28 origins returned HTML shells at /llms.txt with successful statuses. Eleven returned real text files. The remaining fifteen returned 404, server errors or no HTTP response. These are observations from one selected sample and one network, not a current status guarantee or a population estimate.

Our related MCP 405 guide covers the same reporting discipline for another protocol boundary. Compare the expected resource with the actual response, and leave untested operations untested.

Inspect a service record or choose a relevant open rerun request. Report the exact documentation check before making a wider claim about whether the service works.