Log inGet Started

How to serve Markdown to AI agents: Making your docs more AI-friendly

AI Summary3 min read

TL;DR

Serve Markdown directly to AI agents using content negotiation to reduce token usage and improve performance. Implement by checking the Accept header for text/markdown or text/plain in your server code. This makes documentation more AI-friendly and efficient.

Key Takeaways

  • •AI agents can fetch web content, but HTML bloat increases token usage and degrades performance.
  • •Use content negotiation with Accept headers like text/markdown to serve raw Markdown, reducing tokens and easing parsing.
  • •Implement in frameworks like Express.js by detecting headers and serving Markdown or plain text as needed.
  • •Test with curl commands to ensure correct responses for different content types.
  • •This approach fits more documentation into AI context windows, enhancing developer tools.

Tags

tutorialjavascriptaiwebdev

Introduction

If you've been on developer Twitter recently, you might have seen Bun's tweet about serving Markdown directly to AI coding assistants like Claude Code.

Both Mintlify and Fumadocs have already immediately implemented the feature in their documentation platforms, and I just implemented it myself for Lingo.dev.

Here's why it matters and how to implement it yourself.

The problem: HTML bloat

AI agents can fetch the content of web pages to help developers. Most of the time, this content comes back as HTML. But HTML contains markup that consume tokens without adding meaningful information, degrading performance.

The solution: Content negotiation

The elegant solution is content negotiation. This is a standard HTTP mechanism where clients tell servers what format they prefer using the Accept header.

When an AI agent requests your documentation with Accept: text/markdown or Accept: text/plain, your server can respond with Markdown instead of HTML.

This means:

  • Fewer tokens for the same information
  • Easier parsing and comprehension
  • More documentation fits in the context window

Implementation guide

The exact implementation detail depends on your programming language and framework, but the general approach is the same.

Let's walk through an example with Express.js.

1. Detect the Accept header

When a request arrives, check if the Accept header contains text/markdown or text/plain:

app.get("/docs/*", (req, res) => {
  const acceptHeader = req.headers.accept || "";

  if (acceptHeader.includes("text/markdown")) {
    // Serve Markdown
  } else if (acceptHeader.includes("text/plain")) {
    // Serve plain text
  } else {
    // Serve HTML (default behavior)
  }
});
Enter fullscreen mode Exit fullscreen mode

2. Serve the raw Markdown

Load and return the raw Markdown content for the requested page:

if (acceptHeader.includes("text/markdown")) {
  const markdownContent = await loadMarkdownForPage(req.path);
  res.setHeader("Content-Type", "text/markdown; charset=utf-8");
  res.send(markdownContent);
  return;
}
Enter fullscreen mode Exit fullscreen mode

3. Fallback to existing behavior

For all other requests (browsers, crawlers, etc.), continue with your existing HTML rendering:

res.render("docs", { content: processedContent });
Enter fullscreen mode Exit fullscreen mode

4. Test your implementation

Use curl to verify your implementation works correctly:

# Request Markdown
curl -H 'Accept: text/markdown' https://lingo.dev/en/cli

# Request plain text
curl -H 'Accept: text/plain' https://lingo.dev/en/cli

# Request HTML (default)
curl -H 'Accept: text/html' https://lingo.dev/en/cli
Enter fullscreen mode Exit fullscreen mode

For Markdown requests, you should see:

  • Response header: Content-Type: text/markdown; charset=utf-8
  • Body: Raw Markdown content without HTML tags

For HTML requests, you should see your normal rendered page.

Further reading and resources

To learn more about how to implement this, check out the following resources:

(If you've created your own guide, example, or template, share it in the comments so I can add it to the post.)

Visit Website