Cloudflare Agent Readiness examined: thirteen checks, three levels and an important distinction between AI-readable websites and agent-operated services.
Field notes / Investigation 02 / Production evidence
Cloudflare Agent Readiness: When Does a Website Need to Become an Agent?
Thirteen readiness checks. Three different architectural questions. One live production website. The useful discovery was not how to make every indicator green — it was understanding which capabilities a business website actually needs.
01 / The question
The seductive simplicity of a readiness score
Cloudflare Agent Readiness presents a progression from foundation checks to advanced integration. It looks straightforward: complete the checks, improve the score, become more AI-ready.
But a business website that needs to be found, retrieved and understood is not necessarily an application that needs to receive instructions from external AI agents. Nor is it automatically an authentication provider, an agent-to-agent service or a tool-execution environment.
We examined the thirteen checks against a real Sydney Business Web production site. The objective was deliberately narrower than maximising a dashboard score: identify what each check actually enables, test what we could independently observe, and decline infrastructure that served no present requirement.
These figures describe the dashboard state during this investigation, not an independently established measure of AI search visibility, answer inclusion or commercial readiness.
02 / Foundation
Level 1: prepare the website to be retrieved and read
Level 1 covered practical discoverability measures: crawler preferences, sitemap availability, AI crawler access rules, Content Signals and Markdown negotiation. For Sydney Business Web, all five Quick Wins were reported complete.
Access and discovery
We retained the existing, deliberately managed robots.txt policy and sitemap rather than letting a dashboard setting rewrite our crawler arrangements. A content-use signal was added explicitly:
Content-Signal: search=yes, ai-input=yes, ai-train=yesThe choices were intentional. They express the site's selected policy; they do not guarantee that a crawler will visit, index, cite or recommend the business.
Representation and delivery
Cloudflare Markdown for Agents was scoped to the homepage through a configuration rule rather than enabled indiscriminately across the site.
A request for text/markdown returned Markdown; a normal browser request continued to return HTML. Earlier testing of the homepage also examined the treatment of six JSON-LD objects in the converted representation.
A GET request on the homepage returned HTTP 200, Content-Type: text/markdown; charset=utf-8, a Cloudflare cache HIT and:
Content-Signal: ai-train=yes, search=yes, ai-input=yesA previous HEAD request did not show the same header. We recorded the difference rather than asserting that we had identified its cause.
Evidence boundary: successful content negotiation and appropriate access settings demonstrate a functioning delivery path in the requests tested. They do not establish that a particular AI system followed the content, indexed it or used it in a recommendation.
The specific Markdown conversion experiment is documented separately in Investigation 01 — Cloudflare Markdown for Agents.
03 / One measured Level 2 implementation
An HTTP Link Header: useful signal, awkward diagnostic
Level 2 introduced three additional mechanisms: an API catalogue, HTTP Link Headers and an auth.md document. We found no present reason to advertise a public API catalogue or publish login instructions for AI agents. Both would imply capabilities that had not been deliberately offered.
The Link Header check was different. Sydney Business Web already maintained a substantive llms.txt file describing the organisation, its services, AI Visibility technologies and supporting pages. We therefore had a real destination worth advertising — without manufacturing an API.
Using a homepage-only Cloudflare Response Header Transform Rule, we added:
Link: </llms.txt>; rel="describedby"; type="text/plain"The choice uses the HTTP Web Linking mechanism in RFC 8288. It is a generic descriptive link, not an api-catalog declaration under RFC 9727.
What we independently confirmed
/llms.txtreturned HTTP 200 withContent-Type: text/plain.- The homepage returned HTTP 200 in HTML and Markdown forms.
- The Link Header was delivered in both representations.
- An initial duplicate header was eliminated by changing the rule operation from Add static to Set static.
- A subsequent Markdown GET showed one Link Header and a cache HIT.
What the scanner reported
The Level 2 audit did not return a normal pass or fail for the header itself. Instead, its audit details repeatedly reported:
FETCH — GET target page
Network error: The operation was abortedThat is an incomplete automated check. It does not override our successful independent response tests, and it does not prove where the failed request was interrupted.
We considered an isolated cache purge and, if necessary, a support enquiry. We did not loosen origin security, rewrite WordPress, alter the Observatory Worker or deploy a fictional API endpoint in pursuit of a green indicator.
04 / Level 2 decisions
A catalogue and an authentication guide are not universal prerequisites
| Check | What it is for | SBW decision |
|---|---|---|
| API Catalog | Discover published APIs and information about their use. | Defer Do not advertise internal endpoints or invent a public catalogue. |
| Link Headers | Advertise related resources through HTTP response metadata. | Implemented Homepage describedby link verified; scanner aborted. |
| Auth.md | Describe how agents register or sign in to a service. | Defer No present agent login workflow to document. |
For a public services website, the absence of an API catalogue may be entirely correct. A site that offers information, advice and enquiries does not become deficient simply because it declines to expose a machine-callable application interface.
05 / Advanced Integration
Eight checks, eight different engineering obligations
Level 3 is where the distinction becomes unmistakable. It evaluates whether the organisation can publish agent skills, operate agent-facing services, support agent authentication or advertise executable capabilities. These are legitimate technologies. They are not interchangeable with crawler access or structured business identity.
| Advanced check | Actual engineering purpose | Decision |
|---|---|---|
| OAuth Discovery | Advertise a real authorisation server and supported authentication flows. | Defer No relevant public authorisation service. |
| OAuth Protected Resource | Identify authorisation servers and access requirements for protected resources. | Defer No protected agent-facing API in scope. |
| A2A Agent Card | Let other agents discover and communicate with an operating A2A agent. | Defer No A2A service to advertise. |
| Skills Index | Distribute discoverable, verifiable skill artifacts that agents can load. | Research A product description is not a downloadable skill. |
| MCP Server Card | Describe an actual MCP server and its reachable tool interface. | Future option Relevant if an authorised tool service is built. |
| Web Bot Auth | Enable cryptographic verification of signed requests made by outbound bots. | Research Distinct from authenticating inbound Observatory traffic. |
| WebMCP | Expose supported browser-page actions as tools to compatible agents. | Prototype candidate Consider a bounded Baby Gorilla workflow. |
| DNS-AID | Advertise genuine agent service endpoints through DNS records. | Defer No agent endpoint to publish today. |
The eight checks may be appropriate for an agent software platform. But an ordinary business website can operate entirely legitimately without any of them. A zero in this section should not be interpreted as proof that AI systems cannot read, resolve or recommend the business.
06 / Where the value may be
Three opportunities we would investigate further
01 / WebMCP for Baby Gorilla
Baby Gorilla already performs a bounded website analysis. A future browser-based tool might allow a compatible agent to submit a URL through a controlled workflow and receive structured results.
Any prototype would preserve existing validation, timeouts, rate limits and abuse controls. It would not bypass them, and it would need to account for evolving browser support and WebMCP interfaces.
02 / MCP as a commercial interface
An authorised MCP service could eventually expose deliberately selected capabilities from the Gorilla product family. Unlike a static server card, that requires a live endpoint, tool contracts, resource limits, logging and appropriate security.
For AI Observatory, any customer-facing data interface would also require clear customer boundaries and access control.
03 / Cryptographic crawler identity
Web Bot Auth is relevant to the wider engineering problem of distinguishing an authentic machine requester from a spoofed user-agent claim. But the Level 3 check asked whether SBW's own outbound agent can sign its requests. Publishing our keys would not authenticate inbound GPTBot, ClaudeBot or other crawlers automatically.
That distinction is essential for AI Observatory, which separates machine retrieval evidence from unverified claims and from downstream indexing or recommendation outcomes.
07 / Engineering judgement
The security cost of chasing a perfect score
Publishing an authentication discovery document implies an authorisation service. Publishing an A2A Agent Card implies a working agent endpoint. Publishing an MCP Server Card implies an available server. Advertising agents through DNS implies real services worth discovering.
These are not harmless decorative additions. They increase the set of interfaces that must be documented, secured, operated and supported. Even where a metadata file itself is public and low-risk, the capability it describes carries engineering obligations.
Our deployment rule
Never advertise an interface that does not exist, expose a service that has not been designed for external use, or loosen security to appease an automated readiness audit.
Our existing WordPress, Cloudflare and Observatory arrangements remained in place. The only new Level 2 deployment described in this investigation was the scoped homepage Link Header. That restraint is part of the result, not a lack of progress.
08 / Findings and limitations
What this investigation proves — and what it does not
Observed
- The dashboard showed all five foundational checks complete.
- The homepage served HTML and negotiated Markdown successfully in the tests performed.
- The chosen content signal was present in a Markdown GET response.
- The
llms.txtresource returned HTTP 200 andtext/plain. - A homepage Link Header was added, tested and de-duplicated.
- The automated Link Header audit repeatedly aborted its request.
Not established
- That every AI crawler or agent processes these signals.
- That these changes increase citations, rankings or recommendations.
- The precise cause of the aborted scanner request.
- That a higher advanced-integration score improves business discoverability.
- That WebMCP, MCP or A2A should be deployed without a defined product requirement.
Our tests were limited to the endpoints and request variants we actually examined. They were not a comprehensive audit of every site URL, every agent implementation or every Cloudflare response path.
09 / Conclusion
AI-ready is not the same thing as agent-operated
The most important finding was architectural, not numerical.
There are at least three distinct questions in modern machine-facing web engineering. Can systems find and retrieve the content? Can they discover well-defined interfaces? And can they securely execute authorised tasks through those interfaces?
A business may need the first, benefit from parts of the second and have no legitimate requirement for the third. Another business may be building an agent platform and need all three. Those are different architectures and different commercial commitments.
For Sydney Business Web, the immediate engineering result is straightforward: retain the verified retrieval foundations; keep the narrowly scoped Link Header; document the scanner's inconclusive audit; and pursue advanced agent technologies only where they can support an actual product or customer workflow.
That is a stronger result than an artificial 13/13.
10 / Frequently asked questions
AI Agent Readiness: Common Questions
Does a higher Cloudflare Agent Readiness score improve AI search visibility?
Not necessarily. Agent Readiness assesses technical capabilities and configurations, including discovery, authentication and integration mechanisms. A higher score does not establish that AI systems retrieve, understand, cite or recommend a business. Those outcomes require separate evidence.
Does every business website need an MCP server or AI agent interface?
No. MCP servers and agent interfaces are appropriate where external AI systems need to interact with defined tools or services. A conventional business website can be discoverable and machine-readable without providing executable agent capabilities.
What is the difference between AI-readable and agent-operated?
An AI-readable website makes information available for machine retrieval and interpretation. An agent-operated service goes further by exposing interfaces through which authorised agents can perform tasks. These are different architectural requirements, with different security and maintenance responsibilities.
Does publishing llms.txt help AI systems understand a business?
An llms.txt file can provide a structured reference describing a business, its services and relevant resources. However, publishing the file does not guarantee that AI systems will discover, retrieve or use it. Its practical value depends on whether particular systems consume that information.
Can a website be AI-accessible without passing every Agent Readiness check?
Yes. A website may provide successful machine retrieval, appropriate crawler access and coherent business information without offering OAuth, A2A, MCP or other advanced integration mechanisms. Readiness should be assessed against the website's actual purpose, not simply the number of completed checks.