现在阅读这个博客的很多“读者”已经不是人类，而是 Agent（智能体）：Claude、ChatGPT、Perplexity 等，它们往往是因为有人提问而来抓取页面。本周我抽出了一个晚上，对 Deallocated Objects 进行了改造，让它们（顺便也让搜索引擎）能够更舒适地阅读。

这个博客是一个由 Nginx 驱动的静态站点，因此这里的所有内容要么是构建时生成的文件，要么是几行 Nginx 配置，完全不需要任何框架。

每篇文章均提供 Markdown#

对 Agent 来说，解析 HTML 存在大量噪音干扰。现在，每篇文章都同时提供了纯 Markdown 版本。在任何文章 URL 后加上 .md 即可访问：

https://deallocatedobjects.com/posts/the-art-of-the-reveal.md

每个文件开头都包含一段简短的 YAML frontmatter（标题、canonical URL、作者、日期和简短描述），紧接着是正文。较早的文章本来就是用 Markdown 写的，因此构建步骤只需对它们稍作清理，并将图片链接指向自托管副本。

更巧妙的技巧是内容协商（Content Negotiation）。如果客户端在其 Accept 请求头中声明请求 Markdown，它就可以直接通过普通 URL 获取 Markdown 内容，无需额外添加 .md。在 Nginx 中，这只需配置一个 map 以及每个 location 的一行规则：

nginx

http context

map $http_accept $deallocated_md { default ""; "~*text/markdown" ".md"; }

server block

location ~ ^/posts/[^/.]+$ { if ($deallocated_md) { rewrite ^ $uri.md last; } try_files $uri.html =404; }

可以测试一下：

shell curl -H "Accept: text/markdown" https://deallocatedobjects.com/posts/the-art-of-the-reveal

这里有两个关键细节。每个响应都会返回 Vary: Accept，这样中间缓存就绝不会把 HTML 误发给 Agent，或者把 Markdown 误发给浏览器。此外，.md 响应会发送指向对应 HTML 页面的 Link: <...>; rel="canonical" 标头，并附带 X-Robots-Tag: noindex，以防止 Google 将它们视作每篇文章的重复副本。

llms.txt#

/llms.txt 是位于网站根目录的一个纯 Markdown 文件，用于告诉 Agent 它当前正在浏览什么。我的文件包含了博客的一句话简介、如何获取 Markdown、Feed 订阅源地址、我期望的文章引用方式，以及每篇文章的列表（附带其 Markdown 链接和一句话摘要）。

此外还有 /llms-full.txt，它是整站所有文章的全文集合文件（约 225 KB），以及作为 Markdown 版主页的 /index.md。每个响应还会附带一个指向 llms.txt 的 Link 标头，方便 Agent 无需猜测即可直接找到它。

robots.txt#

在 robots.txt 中保持沉默在技术上意味着“允许”，但我更倾向于明确声明。我的配置允许所有人抓取，随后显式列出各大主流爬虫（GPTBot、ClaudeBot、PerplexityBot、Google-Extended 以及其他十几个），并添加了一行 Content-Signal：

User-agent: * Allow: / Content-Signal: search=yes, ai-input=yes, ai-train=yes

对于博客来说，这是正确的决定。我希望被引用。但如果是产品类网站，可能会做出不同的选择。

Feeds#

位于 /feed 的旧 Atom feed 依然可用，且每条条目现在都包含摘要以及指向 text/markdown 的备选链接。我还增加了一个 JSON Feed，其中每项都带有 _markdown_url（JSON Feed 允许使用带有下划线开头的自定义键名）。

旧站点几乎没有任何这方面的优化，因此现在每个页面都配置齐全：

• 真实的 meta description、canonical URL、robots 标签和 lang="en"
• Open Graph 和 Twitter 卡片标签，包含 article:published_time 和 modified_time
• JSON-LD 结构化数据：每篇文章都包含 BlogPosting 和面包屑导航（breadcrumb），首页则包含 WebSite、Blog 以及代表我个人的 Person

摘要（description）的处理比我预想的要棘手一些。旧数据库里为每篇文章预存了摘要，但段落之间连在了一起（例如：“…your app.The RevealWhat do I mean…”），还有少部分包含原始的 <img> HTML 标签。现在，构建流程改为直接从渲染后的文章正文生成摘要。

还有针对 2014 年老旧标记的两处修复。viewport 标签原本包含 maximum-scale=1，这会导致手机端无法双指缩放，现在已将其移除。此外之前完全没有 og:image，因此当我在 X 和 Threads 分享文章时，平台会默认抓取页面上的第一张图片（恰好是顶部的 Little Bites of Cocoa 横幅）。现在每篇文章都有了规范的 1200x630 预览卡片。

IndexNow#

最后，添加了一个 IndexNow key 和一个脚本，每当有新文章发布时就向 Bing 等搜索引擎发送 ping 通知。Google 虽然没有参与该协议，但 Bing 的索引是众多 AI 搜索工具的数据来源，因此花 30 行脚本实现它非常值得。

这些改动都不耗费太多时间，而且大部分只是纯文件。如果你也有自己的博客，从 llms.txt 和文章的 .md 版本开始改造成本低且效果好。你可以试试这篇文章的抓取效果：

shell curl -H "Accept: text/markdown" https://deallocatedobjects.com/posts/making-my-blog-readable-by-ai-agents

有关本站构建方式的其余细节，请参见版本说明（colophon）。

A lot of the people reading this blog now aren’t people. They’re agents: Claude, ChatGPT, Perplexity and friends, fetching a page because somebody asked them a question. This week I spent an evening making Deallocated Objects nice to read for them (and for search engines while I was in there).

The blog is a static site served by nginx, so everything here is either a file generated at build time or a few lines of nginx config. No framework required.

Markdown For Every Post#

HTML is a lot of noise for an agent to wade through. Every post now also exists as plain markdown. Add .md to any post URL:

https://deallocatedobjects.com/posts/the-art-of-the-reveal.md

Each file starts with a little YAML frontmatter (title, canonical URL, author, dates, a short description), then the post. The older posts were written in markdown to begin with, so the build just cleans those up and points their images at the self-hosted copies.

The nicer trick is content negotiation. If a client asks for markdown in its Accept header, it gets markdown from the normal URL, no .md needed. In nginx that’s a map plus one line per location:

nginx

# http context
map $http_accept $deallocated_md {
    default "";
    "~*text/markdown" ".md";
}

# server block
location ~ ^/posts/[^/.]+$ {
    if ($deallocated_md) { rewrite ^ $uri.md last; }
    try_files $uri.html =404;
}

Try it:

Shell

curl -H "Accept: text/markdown" https://deallocatedobjects.com/posts/the-art-of-the-reveal

Two details matter here. Every response sends Vary: Accept, so a cache in the middle never hands HTML to an agent or markdown to a browser. And the .md responses send a Link: <...>; rel="canonical" header pointing at the HTML page, plus X-Robots-Tag: noindex, so Google doesn’t treat them as duplicate copies of every post.

llms.txt#

/llms.txt is a plain markdown file at the root of the site that tells an agent what it’s looking at. Mine has a one-line summary of the blog, how to get markdown, where the feeds are, how I’d like posts cited, and a list of every post with a link to its markdown and a one-line summary.

There’s also /llms-full.txt, which is the full text of every post in one file (about 225 KB), and /index.md, which is the homepage as markdown. Every response also carries a Link header pointing at llms.txt, so an agent can find it without guessing.

robots.txt#

Silence in robots.txt technically means “allowed”, but I’d rather be explicit. Mine allows everyone, then names the big ones explicitly (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and about a dozen more) and adds a Content-Signal line:

User-agent: *
Allow: /
Content-Signal: search=yes, ai-input=yes, ai-train=yes

That’s the right call for a blog. I want to be cited. A product site might choose differently.

Feeds#

The old Atom feed at /feed still works, and every entry now has a summary and a text/markdown alternate link. I also added a JSON Feed, where each item carries a _markdown_url (JSON Feed allows custom keys that start with an underscore).

The old site had almost none of these, so every page got the full set:

• a real meta description, a canonical URL, a robots tag and lang="en"
• Open Graph and Twitter card tags, including article:published_time and modified_time
• JSON-LD structured data: BlogPosting and a breadcrumb on every post, with WebSite, Blog and a Person for me on the index pages

The descriptions were sneakier than I expected. The old database stored a pre-made description for each post, but it had run the paragraphs together (“…your app.The RevealWhat do I mean…”) and a few contained raw <img> HTML. Now the build makes its summaries from the rendered post text instead.

Two more fixes from the old 2014 markup. The viewport tag had maximum-scale=1, which stops people from pinch-zooming on their phones, so that’s gone. And there was no og:image at all, so when I shared a post, X and Threads grabbed the first image on the page, which happens to be the Little Bites of Cocoa banner at the top. Every post has a proper 1200x630 preview card now.

IndexNow#

Last, an IndexNow key and a script that pings Bing and friends whenever a post is published. Google doesn’t participate, but Bing’s index feeds a lot of AI search tools, so it’s worth a 30-line script.

None of this took long, and most of it is just files. If you have a blog, llms.txt and .md versions of your posts are a nice place to start. Here’s this post:

Shell

curl -H "Accept: text/markdown" https://deallocatedobjects.com/posts/making-my-blog-readable-by-ai-agents

The rest of how this site is built is in the colophon.