Source: https://jimbir.ch/blog/is-your-website-ai-agent-ready/ [Back to blog](https://jimbir.ch/blog) AI # Is your website ready for AI Agents? By Jim Birch Oct 17, 2026 9 min read ![Is your website ready for AI Agents?](https://jimbir.ch/assets/blog/is-your-website-ai-agent-ready.avif) Every week, I host AI Office Hours at Kanopi. Anyone can join to share what they’re working on, demo something, ask questions, or just listen. This week, Wes, one of our developers, shared a tool called [IsAgentReady](https://isagentready.com). He made a point of clarifying that it wasn’t [IsItAgentReady](https://isitagentready.com), Cloudflare’s similarly named tool. Naturally, we started running both against our websites. These tools scan a website and evaluate how well it supports AI agents. I was curious how they worked, what they were checking, and whether their recommendations would make a meaningful difference. So I spent some time testing four tools and making changes to my personal site. At Kanopi, we offer a [Trusted AI Visibility Index (TAVI) audit](https://kanopi.com/services/trusted-ai/ai-visibility-assessment) that examines how an organization’s content is discovered, understood, trusted, and recommended by AI systems. That includes content quality, citations, and technical considerations. This post focuses on the technical side: what can we do to make a website easier for AI agents and large language models (LLMs) to discover, understand, and read? ## The tools I tried four free tools. Two provided detailed technical checks and actionable recommendations. The other two were useful for a quick assessment. ### IsAgentReady [IsAgentReady](https://isagentready.com) scans a website, identifies failing checks, and explains what needs to change. It also provides an MCP server, which meant I could run scans directly through Claude Code using Sonnet 5.5. The project has a [Claude Code plugin with skills for fixing issues](https://github.com/bartwaardenburg/isagentready-skills). I installed it, and Claude read its `llms.txt` guide, but it never actually used any of the skills. I probably won’t keep that plugin installed. ### Cloudflare’s IsItAgentReady [IsItAgentReady](https://isitagentready.com) groups websites into readiness levels, ranging from a basic web presence to an agent-native site. For each failing check, it provides an example prompt describing what needs to be added or changed. I tried several of those prompts with Claude Code, and they worked well. Cloudflare also provides a public API, so I could run scans from the command line. That let me test many changes against preview environments before deploying them. My site is hosted on Cloudflare Pages, with DNS managed through Cloudflare. I could also make configuration and DNS changes using Wrangler, Cloudflare’s command-line tool. ### GEO Metrics and Frase Gemini suggested two other tools when I asked about similar services: - [GEO Metrics](https://www.trygeometrics.com/agent-readiness-score) - [Frase](https://www.frase.io/tools/agent-readiness) Both run in the browser and check a smaller set of requirements. They’re fine for a quick assessment, but they seem designed more for lead generation than ongoing technical testing. Of the four, IsAgentReady and Cloudflare’s tool were the most useful for actually making changes. ## What I changed The scanners work like checklists. They identify what’s missing, explain why it matters, and sometimes show how to fix it. Some recommendations were straightforward. Others introduced protocols and conventions I hadn’t worked with before. Here’s what I changed on my site. ### Let crawlers in My `robots.txt` had a `Disallow` rule for my 404 page under `User-agent: *`. That group applies to all crawlers, including AI crawlers. The scanner interpreted the rule as a restriction for each of them. I removed it. I also added explicit rules for the AI crawlers I want to allow, including GPTBot, ClaudeBot, and PerplexityBot. This was a useful reminder to review crawler rules rather than assume an existing `robots.txt` configuration does what you intend. ### Tell AI systems how they can use your content I added two directives to `robots.txt` to express my preferences for AI training, search, and use in AI-generated answers. ``` User-agent: * Content-Usage: train-ai=y, search=y Content-Signal: ai-train=yes, search=yes, ai-input=yes Allow: / Sitemap: https://jimbir.ch/sitemap.xml ``` These signals communicate how I want my content used. They’re not a guarantee that every crawler will honor them, but they make my preferences explicit. ### Make content easier to read My site is built with Astro and serves static HTML, so most of the content was already accessible without JavaScript. The scanners still found a few things worth fixing: - A page title that was one character too long. - A missing contact page. - A navigation menu that wasn’t exposing its expanded or collapsed state. I also made three changes specifically for machine-readable content. **Structured data:** Some of my JSON-LD entities were missing descriptions. I added them. **Markdown:** My build now generates a Markdown version of every page. When a client requests `text/markdown`, the server returns that version instead of HTML. You can try it: ``` curl -H "Accept: text/markdown" https://jimbir.ch/about/ ``` **`llms.txt` and `llms-full.txt`:** I added two files that provide another way for AI systems to discover the site’s content. The first contains a short directory of pages with descriptions. The second contains the full text of the site in one file. A simplified example of `llms.txt`: ``` # Jim Birch > Personal site of Jim Birch, Director of Engineering and AI Practice Lead at Kanopi Studios. ## Blog - [Post title](https://jimbir.ch/blog/post-slug/): A one-line description ``` These files are emerging conventions, not requirements for appearing in AI search results. There is little information out there that LLMs actually use these files, but it definitely can’t hurt. ### Help agents find those files I added HTTP `Link` headers to the homepage that point to the machine-readable resources. My site uses Cloudflare Pages, so the configuration looks like this: ``` / Link: ; rel="describedby"; type="text/plain" Link: ; rel="service-doc"; type="text/markdown" Link: ; rel="alternate"; type="text/markdown" ``` Apache and nginx can send the same headers using their respective configuration directives. I also added an `AGENTS.md` file describing the resources available on the site, including an AI catalog and an index of agent skills. ### Going further than I probably needed to This is where the experiment became more interesting. Most personal blogs don’t need an API, an MCP server, or agent discovery records in DNS. Mine probably doesn’t either. But I wanted to understand what these technologies do, how they fit together, and which might eventually be useful for our clients. I added a small, read-only API for my blog posts, along with an OpenAPI specification and an API catalog. I also created an MCP server with tools to list, search, and read posts. I exposed similar functionality through WebMCP on the site. Finally, I added a DNS record for agent discovery and configured DNSSEC through my registrar and Cloudflare. Some of these capabilities are still emerging. I wouldn’t recommend adding all of them to every website. For me, the value was in building them, testing them, and understanding what problems they’re intended to solve. ### What about Drupal and WordPress? I made these changes on a static Astro site, but the underlying concepts aren’t specific to Astro. Drupal and WordPress can support the same kinds of capabilities. Depending on the requirement, that might involve a CMS setting, a module or plugin, custom development, or server configuration. The implementation will vary, but the questions are the same: Can agents access your content? Can they understand its structure? Can they discover the resources you want to make available? ## Did it work? After each round of changes, I ran the scans again. I deployed changes to a preview environment first, tested what I could, and then merged them. Some checks required testing the live site because they depend on the production hostname or DNS configuration. The final results were: - **IsAgentReady:** 100/100, A+ - **Cloudflare:** 80/100, Agent-Native ### IsAgentReady [![IsAgentReady results for jimbir.ch: a score of 100 out of 100 and an A+ grade, with all five categories at 100 percent](https://jimbir.ch/assets/blog/is-your-website-agent-ready-isagentready.avif)](https://isagentready.com/en/scan/jimbir.ch) ### Cloudflare [![Cloudflare's agent readiness results for jimbir.ch: a score of 80 and the Agent-Native level, with the first three groups of checks at 100 and the API, auth, MCP, and skill discovery group at 63](https://jimbir.ch/assets/blog/is-your-website-agent-ready-cloudflare.avif)](https://isitagentready.com/jimbir.ch) ## Why I stopped at 80 Cloudflare still reports four failing checks: - **OAuth discovery:** Tells an agent how to authenticate with a protected API. My site doesn’t have one. - **OAuth protected resource metadata:** Describes how to obtain an access token for private resources. My site has no private content. - **`auth.md`:** Describes how to register for an account. My site doesn’t have accounts. - **Agent card:** Advertises an agent running on the site. My site doesn’t run an agent. I could add files to satisfy those checks, but they would describe functionality that doesn’t exist. That wouldn’t make my website more useful. It would just improve the score. So I stopped at 80. ## How much do these scores matter? I found the tools useful, but I wouldn’t treat their scores as a definitive measure of how well a website works with AI. They helped me discover unfamiliar protocols, review existing configurations, and validate technical changes. They also disagreed with each other. One tool reported that my homepage had no structured data, even though it did. Another couldn’t find my sitemap at the standard location, even though it was there. Some checks only apply to websites with accounts, private content, or their own agents. And a passing score doesn’t prove that an LLM will understand your content, cite it correctly, or recommend your website. Those are different questions. I still like having a checklist, though. When you’re building websites for clients, a repeatable process helps ensure that important details aren’t overlooked. These tools gave me another set of things to check, even if I wouldn’t implement every recommendation. ## An AI agent readiness checklist I put together a checklist based on what the four tools tested and what I learned while updating my site. This is where I’d start for a typical content-focused website. I’ve left out most requirements specific to ecommerce, authentication, and sites that operate their own agents. Not everything here is required for AI visibility. Some items are established web best practices, while others are emerging conventions worth evaluating. Start with a preview environment, then test the live site. Some checks depend on the production hostname or DNS records. ### 1\. Make sure crawlers can access your site - Crawlers you want to allow aren’t blocked by firewall rules or bot challenges. - `robots.txt` exists and doesn’t contain unintended `Disallow` rules. - Pages only use `noindex` when you intend to exclude them from search. - Missing pages return a proper HTTP 404 response. - Your XML sitemap is available at `/sitemap.xml` and referenced in `robots.txt`. ### 2\. Define your AI content usage preferences - Review `Content-Signal` and `Content-Usage` directives for training, search, and AI-generated answers. - Identify the AI crawlers you want to allow or block, such as GPTBot, ClaudeBot, and PerplexityBot. - Make sure crawler-specific groups contain the appropriate rules. A crawler matching its own group doesn’t simply inherit the default group’s rules. - Optional: Evaluate other content usage signals, including TDMRep and `ai.txt`. ### 3\. Make your content accessible and readable - Important content is available in server-rendered HTML without requiring JavaScript. - Pages use a clear heading hierarchy and semantic landmarks. - Pages include a language attribute, descriptive image alt text, and concise titles. - Links have descriptive text, form fields have labels, and buttons have accessible names. - Navigation and interactive elements work with a keyboard and expose their state using attributes such as `aria-expanded`. - Optional: Serve Markdown when requested with `Accept: text/markdown`. - Optional: Link to Markdown versions of pages using `rel="alternate"`. - Optional: Publish `llms.txt` and `llms-full.txt` to provide another way to discover and read your content. ### 4\. Describe your content and its authors - Pages have useful meta descriptions and Open Graph metadata. - Structured data uses appropriate JSON-LD types and includes relevant descriptions. - Person or Organization and WebSite entities are defined and connected using `@id`. - Blog posts use `BlogPosting` or `Article` schema with author, publisher, and publication dates. - Author information is available and linked where appropriate. - About, contact, and privacy pages contain useful information and are easy to find. ### 5\. Check your HTTP security headers - The site uses HTTPS and sends HTTP Strict Transport Security (HSTS). - A Content Security Policy is configured. - Appropriate frame protection, `X-Content-Type-Options`, and `Referrer-Policy` headers are present. These are general web security practices rather than AI-specific requirements, but several scanners include them. ### 6\. Help agents discover available resources - Optional: Use HTTP `Link` headers to advertise machine-readable resources. - Optional: Publish an `AGENTS.md` file describing available resources. - Optional: Provide an AI catalog or agent skills index. - Optional: Evaluate DNS-based agent discovery and DNSSEC. ### 7\. If your website provides APIs or agent tools - Publish an OpenAPI specification with operation IDs, descriptions, and response and error models. - Provide an API catalog so clients can discover available interfaces. - Return appropriate JSON errors for invalid API requests and handle CORS preflight requests where needed. - If you operate an MCP server, document its capabilities and support the appropriate protocol version. - If you expose WebMCP tools, define their names, descriptions, input schemas, and execution functions. **Skip what doesn’t apply.** OAuth metadata, `auth.md`, agent cards, Web Bot Auth, and payment protocols are only useful when your site provides the corresponding functionality. ## What I took away I started this because a developer shared a tool during AI Office Hours. I ended up spending considerably more time on it than I expected. Some changes were basic website maintenance. Others gave me a chance to work with technologies I’d only read about. I don’t know how many AI agents will actually use the Markdown endpoints, MCP server, or discovery files I’ve added. That’s part of what I want to learn. What I do know is that the experiment gave me a better understanding of what’s available today, what’s still emerging, and which recommendations make sense for different kinds of websites. And I have a better checklist for the next time I need to evaluate a site. What other tools are you using to test how AI agents interact with websites?