David Hoang — Writing — agents

Optimizing a Personal Website for AI Agents

Tue, 11 Aug 2026 00:00:00 GMT

For most of the web's life, the question behind "who is my site for?" had one answer: people, and the search crawlers that help people find it. That's changing. A growing share of visits now come from AI agents — answer engines summarizing a page, assistants fetching a bio, retrieval bots gathering context for a model. They don't scroll, they don't hover, and they don't care about a hero animation. They want structure, stable endpoints, and clear permissions.

I recently went through davidhoang.com and treated "agent experience" as a first-class design surface, the same way I'd treat mobile or accessibility. This post is a tour of what I found, what I changed, and — just as importantly — what I deliberately chose not to do.

The mental model: discovery, comprehension, action

Before touching any code, it helped to split "agent experience" into three needs:

Discovery — can an agent find what exists on the site without scraping every page?
Comprehension — once it has a page, can it extract meaning reliably instead of guessing from markup?
Action — if it wants to do something (search, subscribe), can it do so safely and predictably?

Almost every improvement below maps to one of those three. The site already had a strong classic-web foundation — a permissive robots.txt, an auto-generated sitemap, full-text RSS, semantic <article> markup, canonical URLs, and some JSON-LD. That's great for search engines and RSS readers. It's only implicitly good for agents. The goal was to make the implicit explicit.

Discovery

A generated `llms.txt`

llms.txt is an emerging convention: a plain-text, Markdown-flavored manifest at /llms.txt that tells language models what a site is about and where its important content lives. Think of it as a curated "start here" rather than a full crawl.

The key decision was to generate it from the site's own content collections and navigation data rather than hand-maintaining a file that silently rots. It lists the site identity, primary pages, published writing and notes, the RSS feeds, the sitemap, and the search index — and it excludes drafts automatically, because it's built from the same source of truth the rest of the site uses.

A caveat worth stating plainly: llms.txt is a convention, not a guaranteed ranking or ingestion mechanism. I treat it as low-cost, high-clarity signaling, not a magic funnel.

A machine-readable discovery contract

Separately from llms.txt, I published a small versioned document at /.well-known/agent.json. Where llms.txt is prose for a model to read, this is structured JSON for a program to parse. It declares:

site identity and canonical origin
the discovery resources that already exist (sitemap, RSS feeds, search index, robots)
the human-facing actions available (subscribe, read, etc.)
attribution expectations
an explicit version number

The rule I held myself to: never advertise an endpoint that doesn't exist. A discovery contract that lies is worse than none at all, so it only points at things that are actually live.

A deliberate crawler policy

The site's existing stance was already permissive — allow public content, disallow /api/. Rather than change behavior, I made the stance explicit and documented. The robots.txt now spells out how it treats search, retrieval, and training crawlers, and there's a short policy doc explaining the retrieval-vs-training distinction and how to update the bot list over time.

The honest limitation here: robots.txt cannot actually distinguish "retrieve to answer a question" from "retrieve to train a model." Pretending otherwise would be theater. So the policy documents intent and known gaps instead of implying control it doesn't have.

Comprehension

Richer structured data

I strengthened the Schema.org JSON-LD across the site: writing posts are now BlogPosting, notes are modeled as CreativeWork (they're garden notes, not articles, and it felt wrong to overclaim), and both link back to a stable Person/WebSite graph with consistent entity IDs. Publication and modification dates, language, and keywords are included where the underlying data actually supports them — and only where it does. I didn't want to invent a "last modified" date just to look fresh.

An enriched search index — without breaking anything

The site already exposed /search-index.json to power the ⌘K palette. I enriched each entry with canonical URLs, excerpts, dates, tags, and note stages, and made sure the index covers the discoverable static pages rather than silently omitting destinations that are in the sitemap.

This one had a subtle trap. The obvious move was to wrap the array in a versioned envelope ({ schemaVersion, items }). But that endpoint is already public, and an unknown consumer could be relying on the bare-array shape. So instead of a breaking change, the response stays a bare array and the schema version is advertised through an X-Search-Index-Schema-Version header. The command palette tolerates both shapes. Backward compatibility for an endpoint you don't control the consumers of is worth a little inelegance.

Clean Markdown representations

Rendered HTML is noisy for a machine: navigation, theme toggles, shader backgrounds, and view-transition wrappers all surround a few hundred words of actual prose. So every published writing post and note now has a Markdown twin at /{slug}.md, serving the original body with its front matter and a canonical link back to the HTML page. Drafts and internal-only fields never leak. HTML pages advertise the Markdown alternative via <link rel="alternate" type="text/markdown">.

The nice property here is that an agent can grab the content without parsing the chrome.

Action

Experimental WebMCP tools

WebMCP is a much newer, browser-native idea: a page can register callable "tools" through navigator.modelContext, so an in-browser agent can invoke a real function instead of simulating clicks. It's early — an evolving spec with preview-level browser support — so I treated it as strictly progressive enhancement:

feature-detected, so it's a complete no-op in browsers that don't expose the API
no mandatory dependency or polyfill
tools are registered and cleaned up across view transitions so they don't leak
read-only tools (site search, indexed navigation) are clearly separated from anything with side effects

The most important guardrail: the newsletter tool fills and focuses the form but never submits it. An agent can help a person get to the point of subscribing; it cannot subscribe them. Side effects stay behind an explicit human action.

Measuring whether any of this matters

Finally, instrumentation — because optimizing for agents without measuring outcomes is just vibes. I added privacy-conscious analytics that classify known answer-engine referrals and record whether search and newsletter interactions succeed. Crucially, it never sends query text, email addresses, page content, or personal data, and query strings are stripped from analytics payloads.

There's a limitation I wrote down rather than hid: browser analytics fundamentally can't see server-side crawler traffic. Bots don't run your client JS. So the measurement doc is explicit that crawler and citation volume has to be read from server-side logs (Vercel's firewall/analytics), not the browser layer. Measuring the wrong thing confidently is worse than admitting the gap.

What I chose not to do

A few deliberate non-goals, because scope discipline is part of the design:

No full MCP server. This is an editorial site, not an app with an API surface worth operating and securing. A lightweight machine-readable layer beats a server I'd have to babysit.
No unauthenticated content API that could burn quota. The existing static JSON and RSS cover the read cases without opening a new abuse vector.
No pretending emerging standards are settled. llms.txt and WebMCP are both promising and both early. I shipped them as clearly-scoped, low-risk enhancements, not load-bearing infrastructure.

Takeaways

If you want to make your own site friendlier to agents, a reasonable order of operations:

Publish a generated llms.txt and a /.well-known/agent.json contract — cheap, honest signaling.
Enrich the structured data you already have, without overclaiming facts you don't have.
Offer clean content representations (Markdown or documented JSON) so agents skip the chrome.
Make your crawler policy explicit, including its limitations.
Only then experiment with interactive standards like WebMCP — feature-detected and side-effect-safe.
Measure outcomes, and be honest about what your measurement can and can't see.

The throughline across all of it: be explicit, be honest about limits, and never let an agent take an action a human didn't ask for. Good agent experience turns out to look a lot like good API design and good accessibility — clarity, structure, and respect for the consumer on the other end.

The formlessness of AI agents

Sun, 12 Apr 2026 00:00:00 GMT

In The Avengers #54, a mysterious villain called the Crimson Cowl is orchestrating attacks against the team. It's not until the last page that the cowl drops and we see what's underneath: a robot. The next issue reveals him fully—Ultron-5, a living automation built by Hank Pym using his own brain patterns.

By the time anyone sees Ultron, he's already on version five. He was created, gained sentience, rebelled against his creator, hypnotized Pym into forgetting he ever existed, and then quietly iterated through four bodies in secret as a self-upgrade. Each time Ultron is defeated, he rebuilds to a better enclosure for the same intelligence. Like a hermit crab moving through shells, Ultron wasn't becoming something new but finding the optimal fit.

Our AI tools are having their Ultron moment. In a few years of the scaling era of AI, capabilities have evolved rapidly and are compounding. They are breaking out of the chat boxes with tiny context windows and are now taking actions, running in the background, and using similar tools as us. It's not AGI, but it turns out serendipitous cron jobs invoke a sentient connection to many human beings.

The question in the Ultron moment is, "what is the right body for this intelligence?"

New capabilities begin in borrowed form

Steve Jobs made this observation at the 1983 International Design Conference in Aspen. Early television shows were like radio shows with a camera pointed at them. Because the capability is new, people used existing patterns of what they already knew—a host behind a desk, reading to an audience, and filmed it. The breakthrough was you could visually tell a story of War of the Worlds instead of reading it to them.

Early websites were digital brochures and mobile apps were shrunken desktop software. Early iPhone prototypes ran a cut‑down version of OS X's core, adapted to fit a phone. AI agents talking through chat interfaces are like radio shows with a camera pointed at them. Chat is frictionless not because it's the right interface but because it's a familiar one; a turn-by-turn conversational text box.

In Silicon Valley, we're obsessed with pushing the edges of technology and need to be reminded of global behaviors. I recently ran rapid research comparing site-building tools for SMBs (yes I do this for fun). Compared to classic site builders such as Framer, Webflow, and Squarespace, people preferred using Lovable or Replit to build a website because of the conversational interface.

Giving agents a body

OpenClaw is an open‑source framework for controlling your computer through natural‑language agents. Its primary interaction is a chat agent: you text it on messaging platforms such as Telegram or WhatsApp. It runs tasks on your computer, manages your calendar, clears your inbox. That wasn't enough for some people.

Tom saw a fundamental problem: how do you get OpenClaw to control actual motors in real time? His answer was ClawBody, a software bridge that connects OpenClaw to physical hardware so you can train agents in simulation, then deploy them to real robots. Chris asked a different question: what if robots could actually understand spatial depth? He integrated Intel RealSense cameras with OpenClaw and a vision‑language model so robots can track a person through space, maintain distance, and navigate obstacles, all driven by natural language.

Then there's ClawStage, a device built by HooRii Technology that turns OpenClaw into a companion robot — a Raspberry Pi 5 with a transparent display that creates a hologram-like effect for an animated AI persona, with a camera and microphone array that let it perceive its surroundings and respond to voice directionally. Hackster users can design custom personalities for it, download characters from a shared hub, and make the intelligence feel like someone.

A project called Dimensional integrated OpenClaw with the Unitree G1 humanoid robot and gave it Spatial Agent Memory. The agent understands physical space and temporality, knows the layout of rooms, the location of objects, where people tend to be.

Many of these projects started as DIY projects because they wanted to create an enclosure for their sentient tools; like a hermit crab between shells.

We know how to relate to things that have form. A hammer looks like something you swing and a doorknob affords turning. When something is formless, such as an intelligence in machine-form that sees the world in lines of code, we don't have the same intuitive relationship with it. Human and agentic behaviors are not the same. It's a miss for us to replicate ourselves in the form of a new intelligence.

This is why we must explore the right enclosures and interfaces for agents. We'll soon realize all these CLI tools are not primarily for the humans—they're for people giving the formless thing a shape so they can understand what it does.

The shell keeps changing.

The trust tension

Humans need visibility of agents to trust them; to understand what they are doing and where the boundaries are. Formlessness creates anxiety for humans. If one can't see the agent, how is it held accountable?

The current challenge is the interfaces are designed in the borrowed form of chat, sidebar, and copilots. We're in the uncanny valley of agent design. The form is close enough to be usable but wrong enough to be limiting. A chat-based agent can answer your question but can't show you it's already working on three other things in parallel. A copilot can suggest edits but can't demonstrate that it understands the broader context of your project.

The interface doesn't match the capability. As a result, humans and agents are working on the same interfaces and it's not visibly clear who is doing what.

This is the enclosure problem. Not just how agents look, but what kind of container can hold something that is, by nature, without container. That's a harder design question than it sounds. Many of us haven't designed for things that can think, plan, reason, and use the same tools we do.

And if you want to go somewhere uncomfortable: we haven't solved this problem for ourselves either. The relationship between consciousness and body, between the mind and the form it inhabits, is the oldest unsolved question in philosophy. We're spiritual beings who never fully made sense of our own enclosures. Now we're being asked to design one for something else.

The form is still being found

We're doing what humans always do when faced with something formless: we reach for the containers we already have. We put radio on television. We put the legal pad in the Notes app. We put intelligence in a chat box.

We're the ones designing the body for the new medium. That reframes every choice. A chat interface is a claim about what kind of enclosure intelligence should have. A sidebar copilot is a claim. A robotic arm is a claim. Each enclosure shapes what the intelligence can do, how people relate to it, and what it becomes. The form isn't neutral.

It's tempting to settle with the borrowed form, but that's not what we do as designers. Understanding what form formless experiences need is what we should explore. The fact that we haven't found the native form of agents yet means the field is genuinely open. This is the equivalent of being a filmmaker in 1910, a web designer in 1995, a mobile designer in 2008. The early chaotic moments that feel disorienting are what we live for as makers.

We are at the beginning of thinking about different enclosures for agents. The shell keeps changing—and that's the work.

Originally posted on Proof of Concept