AI Crawler Blocked by ModSecurity — While the Website Looked Completely Healthy

A client website was operating normally. Human visitors could browse it, the site was online, Cloudflare was serving traffic correctly, and there was no obvious indication that anything was wrong.

But the AI Observatory was showing something very different.

In the Observatory evidence window shown below, ClaudeBot had made 14 autonomous accesses, but only 1 of 7 business retrieval attempts succeeded. Its measured business success rate was therefore just 14.3%.

This is the kind of failure that is easy to miss. The website itself was not down. There was no obvious WordPress error, no broken page, and no visible problem for ordinary users.

The issue existed specifically in the machine retrieval path.

ClaudeBot Was Being Rejected with HTTP 418

The Observatory told us that ClaudeBot was reaching the site, but its retrieval success rate was abnormally low. The next step was to reproduce the behaviour outside the Observatory and see exactly what the server returned.

We sent controlled requests to the website using the ClaudeBot user agent.

Instead of the expected successful response, the request was rejected with:

HTTP 418 is not a normal response for a public webpage. In this context it was effectively a signal that a security rule was intercepting the request before the page could be delivered normally.

That narrowed the problem considerably, but it still did not tell us which layer was responsible.

So the problem stopped being simply “Claude cannot retrieve the site”.

It became a much more useful engineering question: where, exactly, is the request being stopped?

Was Cloudflare Blocking ClaudeBot — or Was the Problem at the Origin?

The next job was to separate the delivery layers.

The site sat behind Cloudflare, so an HTTP 418 response did not automatically tell us whether the rejection was happening at the edge or on the hosting server behind it.

We compared normal public requests through Cloudflare with direct-origin testing using the same ClaudeBot user agent.

That was the decisive result.

If the same failure could be reproduced at the origin, then Cloudflare was not creating the block. The request was being rejected further downstream.

From there, the investigation moved into the hosting security layer, where the behaviour was traced to ModSecurity.

This distinction mattered. Without isolating the layers first, it would have been easy to start changing Cloudflare rules, crawler permissions or WordPress configuration unnecessarily.

Instead, the evidence pointed directly at the correct part of the stack.

Once the Fault Was Isolated, the Host Could Fix the Right Thing

With the failure reproduced at the origin and the security layer identified, the problem could be handed to the hosting provider with useful evidence rather than a vague report that “ClaudeBot seems to be blocked”.

The hosting environment was managed by Synergy Wholesale, and once the issue had been narrowed to ModSecurity they were able to correct the behaviour at the server side.

After the change, the same controlled ClaudeBot request was repeated.

The retest confirmed that Cloudflare was serving the request normally and that the origin was no longer rejecting the ClaudeBot user agent.

That turns a suspected AI visibility problem into a measurable infrastructure incident with a clear before-and-after result.

A Healthy Website Can Still Have an AI Retrieval Problem

This incident is a useful reminder that AI visibility is not only a content, schema or search problem.

A website can look completely normal to its owner, its customers and even conventional monitoring systems while an AI crawler is encountering a very different experience.

That matters because no amount of good content can help if the system trying to retrieve it cannot get the page.

This is why we treat AI visibility as a systems problem rather than a single SEO task. Content, structured data, identity, corroboration and infrastructure all have to work together.

In this case, the content was not the problem. The crawler permissions were not the problem. Cloudflare was not the problem.

The failure existed deeper in the stack, at the origin security layer.

The value of the Observatory was not that it produced another score. It exposed a real retrieval failure that could be investigated, reproduced and fixed.

From “Something Looks Wrong” to a Reproducible Technical Diagnosis

Without the Observatory, this problem could easily have remained invisible.

The website was online, the pages loaded normally, and there was no obvious customer-facing failure to investigate. The first useful signal came from the crawler evidence itself.

That sequence is the important part.

That is a much stronger basis for technical decision-making than relying on assumptions about whether an AI crawler should be able to access a site.

It also means changes can be kept focused. We did not need to alter unrelated Cloudflare rules, rewrite content, change schema, or make speculative WordPress changes.

AI Visibility Needs Retrieval Evidence, Not Assumptions

This case was not about a dramatic website outage. It was about something much quieter: one recognised AI crawler was being obstructed while the site continued to look healthy to almost everyone else.

That is exactly why retrieval evidence matters.

The result was a complete evidence chain from detection through to verification.

For us, that is the practical role of the AI Observatory: not to predict whether an AI system will recommend a business, but to establish whether recognised machine systems are actually reaching the site, retrieving its pages successfully, and encountering infrastructure problems that might otherwise remain invisible.

The Observatory forms one part of Sydney Business Web's wider AI visibility methodology, alongside entity structure, technical diagnostics and external corroboration.

You can read more about the AI Observatory here .

Technical Background and Related Sydney Business Web Resources

This case sits within a wider body of work on AI crawler access, retrieval evidence, entity structure and infrastructure reliability.

External references are provided for technical context. The crawler behaviour, origin testing and remediation described in this article are based on direct observations from the client incident.

Frequently Asked Questions

Do You Know Whether AI Crawlers Can Actually Reach Your Website?

A website can look healthy while recognised AI systems are being blocked, rejected or failing somewhere in the delivery path.

Sydney Business Web uses the AI Observatory to measure real crawler activity, identify abnormal retrieval behaviour and provide evidence that can be investigated at the correct technical layer.

The AI Observatory is available as part of Sydney Business Web's AI visibility diagnostic and monitoring work.

Learn more about the AI Observatory  or  talk to us about your website's AI visibility .