AI Observatory measured the impact of a rare hosting fault that most human visitors would barely notice, while search and AI crawlers repeatedly failed to retrieve business information.
The Hidden Failure
A Website Can Look Fine While Machines Are Failing to Retrieve It
Intermittent faults are easy for people to dismiss. A visitor may occasionally see a 502, 503, 504 or similar error, refresh the page, receive the content normally and continue browsing.
The incident may last only seconds. There may be no obvious outage, no complaint from a customer and no reason for that visitor to think about it again.
Search engines and AI systems behave differently. Their requests arrive on their own schedules. If one of those requests fails, that retrieval has failed: the machine cannot use business information it did not receive.
This was a retrieval-layer failure, not an identity or structured-data problem. Tools such as Schema Gorilla and the AI Identity Diagnostic answer a different question: once information can be retrieved reliably, does it resolve into a coherent business identity?
Successful retrieval is therefore the first prerequisite. Search and AI systems cannot analyse, index, interpret or reuse information they were unable to obtain in the first place.
The Evidence
The Failures Were Not Random Noise
Once the Observatory data was examined across its rolling evidence window, the occasional errors stopped looking occasional. A measurable pattern of failed machine retrievals emerged.
The impact was not uniform. Because the infrastructure fault was intermittent, different systems encountered it at different times according to their own retrieval activity.
In this snapshot, Googlebot recorded a 78% business-information retrieval success rate, Bingbot 47%, OAI-SearchBot 50%, and GPTBot only 12.5%.
PerplexityBot and ClaudeBot recorded 100% during the same rolling window. This does not mean the underlying infrastructure treated those systems differently. It shows why intermittent faults can produce very different observed outcomes depending on when individual crawler requests arrive.
These percentages are therefore evidence-window observations, not ratings of the crawlers themselves. What mattered was that recognised machine systems attempting to retrieve business information were experiencing materially different success during the same infrastructure incident.
“The server fault was important. The additional evidence was knowing what it was doing to real search and AI retrieval.”
— Keith Rowley, Sydney Business WebFinding the Cause
The Evidence Pointed Beyond WordPress
Before escalating the issue, we worked through the normal application and hosting-account checks to determine whether the failures were being generated by the website itself.
The dominant error was HTTP 522: the connection between Cloudflare and the origin server was timing out. Importantly, many of those failed requests never appeared in the normal website access log.
Taken together, the evidence pointed to a failure occurring upstream of normal WordPress and PHP processing rather than an application-level fault inside the website.
We escalated the evidence to the hosting provider with the hourly failure distribution, status-code counts and measured crawler impact.
AI Visibility Engineering
Being Online Is Not the Same as Being AI Visible
A technically good website can contain excellent information and still fail at the first stage of machine visibility: successful retrieval.
Sydney Business Web treats AI visibility as an engineering problem. Machines must first be able to retrieve the website reliably, then resolve what its entities represent, and finally find enough consistent evidence to trust and reuse that information.
Can search engines and AI systems consistently obtain the information?
Can they determine which business, person, service and entities the information describes?
Can they find consistent technical and external evidence supporting those claims?
Provider Confirmation
The Hosting Provider Confirmed a Rare Infrastructure Problem
The website is hosted by a reputable, high-quality hosting provider. This was not a case of an overloaded customer account or an obvious WordPress application failure. The evidence instead pointed to an unusual infrastructure fault affecting a small number of servers.
After we supplied the failure data, the provider confirmed that it was already working with its server-platform supplier to identify and resolve the problem.
The issue was known to produce intermittent Cloudflare 520, 522 and 525 errors. A corrective vendor patch had been supplied and was being tested for stability before deployment to affected servers.
“The interesting part wasn't that a rare hosting fault occurred. Rare faults happen. The interesting part was that the Observatory measured its effect on machine retrieval before ordinary website use made the scale of the problem obvious.”
— Keith Rowley, Sydney Business Web
What This Incident Demonstrated
Machine Retrieval Has to Be Measured, Not Assumed
From a human perspective, Sydney Business Web remained largely available. The occasional transient error was easy to dismiss because a refresh usually restored the page immediately.
The Observatory revealed the part ordinary browsing could not: recognised search and AI systems were repeatedly encountering failed retrievals, while some observed crawler success rates had fallen dramatically.
This incident demonstrated one specific layer of AI visibility: retrieval. It did not establish whether a machine understood the information, resolved the business identity correctly, cited it or reused it in an answer. Those are separate questions.
The AI Observatory can operate independently. In this incident it was the correct diagnostic instrument; Schema Gorilla and the AI Identity Diagnostic were outside the scope of the fault because the problem was retrieval, not business-identity resolution. Those tools become relevant once reliable retrieval has been established.
That distinction matters. Before questions of entity coherence, interpretation or downstream visibility can be assessed meaningfully, the underlying information first has to be reliably obtainable by the systems attempting to retrieve it.
Sydney Business Web's AI Observatory provides verified evidence of real search and AI crawler retrieval activity: which recognised systems arrived, what business information they attempted to retrieve and whether those requests succeeded.
Explore the AI ObservatoryFrequently Asked Questions
AI Retrieval Failure and Website Availability
The incident illustrates an important distinction between a website appearing available to people and being reliably retrievable by automated search and AI systems.
Why did most human visitors not notice the problem?
The failures were intermittent. A visitor might encounter a brief 502, 503, 504 or similar error, refresh the page and receive it normally. That makes isolated failures easy to overlook even when they are occurring repeatedly across a 24-hour period.
What is an AI retrieval failure?
An AI retrieval failure occurs when a search or AI crawler attempts to obtain information from a website but does not successfully receive the requested content. The machine cannot analyse or use information that was not retrieved.
Can intermittent 5xx errors affect AI visibility?
Repeated failures can reduce opportunities for search and AI systems to discover, revisit and process website information. A single failed request does not establish a visibility problem, but persistent retrieval failures are technically significant and should be investigated.
Was WordPress responsible for this incident?
The evidence did not support that conclusion. Account resource monitoring showed no corresponding limits, WordPress recorded no fatal failures capable of explaining the pattern, and many of the failed requests did not reach the normal webserver access log. The hosting provider subsequently confirmed a rare infrastructure-level issue.
What did the AI Observatory reveal?
It showed the cumulative effect of failures that were difficult to recognise through ordinary browsing. During the incident, qualifying business-information retrieval success fell to 60.3%, with markedly different success rates across individual search and AI crawler systems.
Does this mean the hosting provider is unreliable?
No. Rare infrastructure faults can occur even on high-quality hosting platforms. What matters is detecting the fault, identifying the correct technical layer and applying an appropriate fix. In this case the provider was already working with its server-platform supplier and testing a patch.
References and Further Reading
External References
| Reference | Why it matters |
|---|---|
| Cloudflare — Error 522: Connection Timed Out | Cloudflare's technical explanation of HTTP 522, including failures establishing or maintaining communication with the origin web server. |
| Google Search Central — Technical Requirements | Google explains that a page must be accessible to Googlebot and return an HTTP 200 success response to meet its basic technical requirements for indexing. |
| Google Search Central — Troubleshoot Crawling Errors | Google's guidance on crawler availability problems and the effect persistent server errors can have on crawling. |
Internal Sydney Business Web References
| Page | Related information |
|---|---|
| AI Observatory — Verified AI Retrieval Monitoring | How Sydney Business Web measures verified search and AI crawler retrieval activity at the network edge. |
| AI Retrieval Evidence | Live and recorded evidence showing real machine systems retrieving information from Sydney Business Web. |
| AI Crawler Monitoring at the Cloudflare Edge | Technical background on observing crawler activity without adding client-side scripts or website performance overhead. |
| AI Visibility Services & Pricing | Sydney Business Web's AI Visibility Engineering services, diagnostics and technical implementation options. |
AI Visibility Engineering
Do You Know Whether AI Systems Are Actually Reaching Your Website?
A website can look perfectly healthy to you while search and AI systems are seeing something very different.
Sydney Business Web's AI Observatory provides direct evidence of real crawler retrieval activity — showing which machine systems arrive, whether they successfully obtain your business information, and where retrieval failures are occurring.