Technical Solution Problem 16: Why can’t AI read my Website
A Real-World Cloudflare and AI Retrieval Case Study
Every day, we solve problems for eCommerce website owners. We had a think about how we might use this activity to help others, and came up with this idea: Every week or three, we'll take the trickiest problem and publish our solution.
Important! - Some of these solututions involve adding code to your website (WordPress and Woocommerce mostly), so please ALWAYS be careful. We are not in any way responsible, directly or indirectly for any impact or consequences our code or advice has on your website, nor are we liable for any damage arising from such use.
Always back up your website before changing or adding code and/or editing the database, This is critically important!!!
A live AI assistant may be unable to read a public website when its request arrives without a verifiable identity.
The Contradiction: Recognised Crawlers Succeeded— But Gemini’s Live Fetch Failed
Our own monitoring was proving that recognised search and AI crawlers were reaching the Sydney Business Web website and successfully retrieving business information. Then we asked a leading AI assistant to read one particular public page—and it said that it could not.
This was not a website receiving no machine attention. During the relevant rolling 24-hour period, AI Observatory recorded eight qualifying search and AI systems and 108 successful business-information retrievals. The observed systems included Googlebot, OAI-SearchBot, ClaudeBot, PerplexityBot, Applebot and Bingbot.
“Gemini could not read the page” did not mean that AI crawlers could not crawl or retrieve it. It meant only that the particular live fetcher used during that conversation failed to obtain the page.
The answer begins with the word “read.” An AI assistant may know about a website through Search, use information collected previously, request current information through an AI search crawler or attempt to fetch one particular page during a live conversation.
Those are different technical activities. They may use different infrastructure, IP addresses, request methods and identities—even when they appear to the user under the same familiar AI brand.
The Website Was Machine-Accessible
Our edge evidence showed recognised autonomous systems successfully retrieving substantive website information. This was not an assumption based only on robots.txt or crawler permissions.
A Live Request Is Different
Asking an AI assistant to inspect a URL may trigger a separate, user-directed fetch rather than using the autonomous crawler responsible for Search or regular discovery.
One AI Brand Uses Several Systems
ChatGPT, Gemini and Grok are not single bots knocking on the website’s door. Their search crawlers, training crawlers and conversational tools can have different purposes and identities.
The Firewall Cannot See the Conversation
Website security sees an IP address, user-agent, request method and destination. It cannot simply trust that an anonymous request is helpful because somebody initiated it inside an AI conversation.
To resolve that contradiction, we examined the Cloudflare network-edge evidence to identify what actually arrived when the assistant attempted the live fetch.
Network-Edge Evidence Confirmed Successful Retrieval
Before blaming the website, its content or its crawler configuration, we examined the live network-edge evidence. It showed that recognised search and AI systems were already reaching the site and retrieving substantive business information successfully.
This was not a speculative crawler list. AI Observatory only included qualifying machine activity that passed its identity and public-evidence filters. The snapshot recorded a 77.7% successful business-content retrieval rate during that rolling period.
The evidence eliminated a site-wide access failure: recognised autonomous crawlers were receiving successful responses, while one particular live conversational request could not access one particular page.
Cloudflare Was Filtering Requests— Not Blocking Recognised Crawlers
Sydney Business Web operates behind Cloudflare. Requests encounter the Cloudflare network edge before they reach WordPress or the origin server, allowing suspicious traffic to be filtered before it can interact with the website.
We have custom security rules designed to block traffic from high-risk hosting networks, anonymous automation and other request patterns that do not justify access. These protections exist because public websites are continuously probed by scripts, scanners, scrapers and malicious bots.
The rules were not preventing legitimate Search or AI discovery; the preceding network-edge evidence shows recognised crawlers passing through them successfully.
Cloudflare Sees the Request First
The network edge evaluates the source IP, network owner, request method, user-agent, destination and available identity signals before the request reaches the website application.
Recognised Crawlers Can Pass
Legitimate crawlers can identify themselves through established user-agents, published network ranges, reverse DNS, verified-bot classifications or emerging cryptographic methods.
Cloud Hosting Is Not Automatic Trust
AWS, Google Cloud and other large hosting networks carry both legitimate services and enormous volumes of unwanted automation. Their ownership alone cannot establish who initiated a particular request.
Security Rules Need Evidence
A helpful purpose described inside a private conversation is invisible to the firewall. The request itself must present enough trustworthy information to distinguish it from ordinary network noise.
We will not publish the detailed expressions used by our security rules. The relevant point is simpler: verified machine systems were being admitted, while unidentified traffic from higher-risk infrastructure remained subject to blocking.
We could now examine the failed request without guessing: Cloudflare had recorded exactly what arrived, where it came from and why it was stopped.
What Actually Arrived at Cloudflare?
At the precise time Gemini was asked to read the newly published article, Cloudflare recorded two identical attempts to access that article’s URL. Both were stopped before reaching WordPress.
The timing and requested path closely associate these events with the live Gemini test. However, the request did not identify itself as Gemini, Google-Agent or another recognisable Google fetcher. That absence of identity is not a minor technicality—it is the reason the firewall could not safely distinguish it from ordinary automated traffic.
Amazon AS14618
The request came from an AWS address in the United States. Large cloud networks host legitimate services, but they also carry extensive scanning, scraping and abusive automation.
axios/1.19.0
Axios is a general-purpose HTTP client. This user-agent disclosed the software library making the request, but nothing about the organisation, AI platform or person responsible for it.
HEAD
A HEAD request asks for response headers without requesting the page body. Even if permitted, it would not by itself prove that the article’s substantive content had been retrieved.
No Verifiable AI Identity
The request supplied no referrer and no recognised crawler identity. From the firewall’s perspective, it was an anonymous script arriving from a blocked high-risk hosting network.
The request may have been legitimate in intention, but legitimate intention is not a trustworthy network identity.
Cloudflare therefore behaved exactly as configured. It admitted recognised Search and AI crawlers while rejecting an unidentified request from infrastructure covered by a high-risk network rule.
Weakening the firewall for all AWS traffic—or trusting any request using the word “axios”—would not solve an AI visibility problem. It would remove a security boundary for countless unrelated scripts.
Gemini’s Recommendation Was Sound— Its Explanation Was Partly Speculative
We asked Gemini whether Sydney Business Web should weaken its Cloudflare protection so that its live conversational fetcher could reach the page. Gemini advised us to leave the security rules intact.
Gemini subsequently acknowledged that live AI tools arriving with the same signatures as generic scrapers create a genuine engineering conflict. It suggested disposable cloud environments, off-the-shelf HTTP clients and separate product teams as possible explanations.
Those explanations are plausible, but our Cloudflare evidence cannot prove Gemini’s internal architecture. They remain hypotheses supplied by Gemini, not facts established by the request log.
What Gemini Got Right
- Do not whitelist an entire AWS network for one conversational fetcher.
- Do not treat a generic user-agent as proof of a legitimate AI identity.
- Prioritise verified Search crawlers for broad public discovery.
- Keep unidentified traffic out of public AI retrieval evidence.
- The Cloudflare rule behaved correctly on the evidence available to it.
Where the Explanation Went Too Far
- Live user-directed agents are not commercially irrelevant; they can affect the answer being produced for a user now.
- Verifiable user-directed agents already exist across several major AI platforms.
- Dedicated user-agents, published IP ranges and reverse-DNS verification are already established practices.
- Cryptographic Web Bot Auth is emerging, although implementation remains incomplete.
- The reason this particular Gemini pathway arrived anonymously is not established by our evidence.
The wider industry is not waiting for one future standard before attempting trustworthy identification. Different platforms already separate autonomous Search crawlers from user-directed retrieval agents.
Google-Agent
Google documents a user-triggered agent with published network ranges and experimental Web Bot Auth support. That does not prove every Gemini retrieval pathway uses it—and this request plainly did not identify itself that way. Google documentation.
ChatGPT-User
OpenAI separately documents ChatGPT-User for certain user-initiated actions and publishes IP ranges for verification. OAI-SearchBot remains the separate crawler used for ChatGPT Search visibility. OpenAI documentation.
Claude-User
Anthropic distinguishes Claude-User from Claude-SearchBot and ClaudeBot. It warns that blocking Claude-User may reduce visibility during user-directed web searches. Anthropic documentation.
Cloudflare now similarly classifies machine traffic by behaviour, including separate categories for Search, Agent and Training. Its verified-bot framework supports IP validation and Web Bot Auth rather than relying on a self-declared user-agent alone. Cloudflare’s verified-bot guidance.
The correct conclusion was not that live AI agents do not matter. It was that live AI agents must present a trustworthy identity if they expect security systems to distinguish them from anonymous automation.
What Should You Do When an AI Cannot Read Your Page?
Do not begin by disabling security or randomly allowing bots. First establish which part of the chain failed: the website, Search discovery, the AI tool or the live request at the network edge.
-
Confirm That the Page Is Public
Open the exact URL in a private browser window. Confirm that it loads without authentication, returns the intended page and does not depend on a logged-in session, location permission or consent dialogue.
-
Separate Search Visibility from Live Retrieval
Check whether Google or Bing has indexed the page. An indexed page proves that a Search crawler has previously accessed it; it does not prove that the AI assistant’s live conversational fetcher can retrieve it now.
-
Record the Exact Test
Note the URL, AI platform, wording of the request, exact time and local time zone. Without a precise timestamp, finding the corresponding request among thousands of security and access events becomes unnecessarily difficult.
-
Inspect the Edge Evidence
In Cloudflare, examine Security Events around the recorded time and filter by the requested path, IP address or action. Record the request method, user-agent, network owner, referrer, response and matched security rule. For another CDN or WAF, inspect its equivalent logs; WordPress may never see a request blocked at the network edge.
-
Verify the Identity Before Changing Anything
Do not trust a user-agent string by itself. Check Cloudflare’s verified-bot classification or the platform’s published IP ranges, reverse-DNS requirements or supported Web Bot Auth identity. Only if a verified agent was incorrectly blocked should you adjust the conflicting rule or create a tightly scoped exception—never allow an entire cloud provider, country, ASN or generic HTTP library merely to make one test succeed.
-
Repeat the Test and Preserve the Evidence
Run a fresh test after any justified correction. Confirm that the intended request succeeds without opening access to unrelated automation, and record the before-and-after result.
The Logs Usually Produce One of Three Answers
No Request Arrived
The AI may not have attempted a live fetch, may not have had browsing available or may have answered from previously indexed information. There is no website-side rule to repair.
A Verified Agent Was Blocked
A genuine access conflict exists. Review the precise WAF or CDN action, create the narrowest safe exception and test again without weakening unrelated protections.
An Anonymous Request Was Blocked
The firewall had no trustworthy identity to evaluate. Leave the protection intact. The agent operator—not the website owner—must provide a verifiable retrieval pathway.
“The AI could not read my page” is a symptom. Do not alter security until the network evidence identifies the actual cause.
Cloudflare recommends using Security Events to identify what blocked a request and checking upstream WAF rules when an intended crawler is not reaching the site. Cloudflare Security Events guidance.
Website Owners and AI Platforms Have Different Responsibilities
A website owner can make public information accessible, permit recognised crawlers and maintain sensible security controls. An AI platform must identify its request well enough to be safely recognised; the website owner cannot manufacture trust for somebody else’s anonymous fetcher.
The Website Owner’s Responsibility
- Keep important public pages accessible without authentication.
- Keep robots.txt, crawler permissions and CDN or WAF rules consistent.
- Investigate failed requests and create only narrow, evidence-based exceptions.
- Continue protecting administration, forms, customer data and transactional systems.
The AI Platform’s Responsibility
- Provide a stable user-agent and a verifiable identity through published networks, reverse DNS or signed requests.
- Distinguish Search, training and user-directed agent traffic clearly.
- Use an appropriate retrieval method and report access failures accurately.
- Do not force website owners to trust entire shared cloud networks.
What This Production Case Actually Proves
One incident should not be stretched into a universal theory about every AI assistant or every website. The useful conclusion is the one supported by the observed evidence.
What the Evidence Proves
- Recognised Search and AI crawlers were successfully retrieving the website; it was not generally closed to machine access.
- A live request targeted the exact article at the time of the Gemini test.
- The request arrived from Amazon AS14618 using the generic user-agent axios/1.19.0.
- It used HEAD and supplied no verifiable Google or Gemini identity.
- Cloudflare blocked it under an existing high-risk network rule, so no broad security change was justified.
What the Evidence Does Not Prove
- It does not prove that Gemini can never retrieve the website or that every Gemini pathway uses anonymous AWS infrastructure.
- It does not establish Gemini’s internal architecture or show that ChatGPT, Claude and Grok behave identically.
- It does not mean that all AWS traffic is malicious or that every AI access failure is caused by Cloudflare.
- It does not justify blocking every user-directed AI agent.
- It does not indicate a failure of the page’s content, schema or Search indexing.
When an AI Says It Cannot Read Your Website, Inspect the Evidence
Do not immediately blame the page, disable the firewall or assume that every AI crawler has been excluded. Determine whether the failure involved Search discovery, autonomous crawling or one live conversational request.
Our production evidence showed recognised AI and Search systems successfully retrieving Sydney Business Web while one anonymous live request was blocked. The security rules therefore remained unchanged, and that unidentified request was correctly excluded from our public retrieval evidence.
Reliable AI visibility begins by distinguishing what machines are permitted to do, what they actually retrieved and what the evidence genuinely proves.
AI Website Access: Common Questions
A failed live AI request can have several causes. These answers separate conversational retrieval from ordinary Search and AI crawling.
Why can’t ChatGPT, Gemini or Grok read my website?
The AI may not have live browsing available, may be answering from previously indexed information or may have attempted a request that was blocked by a CDN, firewall or hosting-security system. The page may also require authentication, JavaScript or interaction that the fetcher cannot complete.
Does this mean AI crawlers cannot crawl my website?
No. A failed request inside one AI conversation does not prove that autonomous Search or AI crawlers cannot access the website. In our case, recognised crawlers were successfully retrieving business information while one separate live conversational request failed.
Should I disable Cloudflare or my firewall so the AI can read the page?
No. First inspect the security event and establish what was blocked. If the request was verified and incorrectly stopped, make the narrowest justified correction. Do not disable protection or allow an entire cloud network merely to accommodate one anonymous request.
How can I tell whether a live AI request reached my website?
Record the exact test time and URL, then inspect your CDN, WAF or server logs. Look for the requested path, source IP, ASN, user-agent, request method, response and security action. Requests blocked at the network edge may never appear in WordPress logs.
What is the difference between an AI crawler and a live AI agent?
An autonomous crawler visits websites to support Search, discovery, model development or other platform functions. A live agent makes a request in response to a particular user action. The two systems may use different identities, networks, permissions and retrieval methods.
Does robots.txt control live conversational AI fetchers?
Not consistently. Autonomous crawlers commonly use robots.txt to determine crawling permissions, while some user-triggered fetchers treat those requests differently. Robots.txt also cannot override a CDN or firewall that blocks the request before it reaches the website.
What AI traffic should a security-conscious website allow?
Allow systems that support your business objectives and can be verified through trusted bot classification, published IP ranges, reverse DNS or supported cryptographic identity. Treat unidentified traffic according to your normal security policy rather than trusting a claimed purpose that the request itself does not prove.
References and Supporting Material
Official platform documentation supports the distinctions between Search crawlers, user-triggered agents, verification methods and network-edge security. Sydney Business Web’s production screenshots provide the direct evidence for the incident examined on this page.
AI Platforms and Agent Identity
- Google User-Triggered Fetchers Google-Agent, published agent ranges and user-initiated retrieval.
- Verify Requests from Google Crawlers and Fetchers Published IP ranges, reverse DNS and verification procedures.
- Google Web Bot Auth Guidance Experimental cryptographic authentication for automated agents.
- Overview of OpenAI Crawlers OAI-SearchBot, GPTBot and the user-directed ChatGPT-User agent.
- Anthropic Web Crawler Guidance ClaudeBot, Claude-SearchBot and Claude-User distinctions.
Cloudflare and Security Evidence
- Cloudflare Verified Bots Search, Agent and Training classifications with identity validation.
- Cloudflare Security Events Investigating mitigated requests, false positives and matched actions.
- Allow Traffic from Verified Bots Using verified-bot status within Cloudflare security rules.
- AI Crawl Control with Cloudflare WAF How crawler controls and upstream WAF rules interact.
- Fake Bot Detection and Verification Why user-agent claims require source verification.
Sydney Business Web Evidence
- AI Observatory Managed network-edge retrieval monitoring and live dashboard.
- Verified AI Retrieval Evidence Public evidence boundaries and observed machine retrieval.
- AI Crawler Monitoring at the Cloudflare Edge Engineering methodology for observing crawler requests before origin.
- AI Visibility and SEO — Search Has Changed The live article targeted during the Gemini retrieval test.
- Making a Business Website Visible to AI Technical access, machine-readable identity and supporting evidence.
- AI Visibility Services and Pricing Retrieval monitoring, identity analysis and engineering remediation.
- AI Visibility Glossary Definitions for retrieval, crawlers, entity identity and AI visibility.
Sources reviewed 29 August 2026. Official documentation establishes the published platform and verification mechanisms. The association between the anonymous AWS request and the Gemini test is based on matching time, destination and observed Cloudflare events; the request itself supplied no verifiable Gemini identity.