AI Observatory detected a rare hosting fault that most human visitors would barely notice, while search and AI crawlers repeatedly failed to retrieve business information.
The Hidden Failure
A Website Can Look Fine While Machines Are Failing to Read It
Intermittent faults are easy for people to dismiss. A visitor may occasionally see a 502, 503, 504 or similar error, refresh the page, get the content normally and continue browsing.
The incident may last only seconds. There may be no obvious outage, no complaint from a customer and no reason for that visitor to think about it again.
Search engines and AI systems behave differently. When an automated request fails, that particular retrieval has failed. The machine cannot use business information it did not receive.
Repeated failures of this kind matter because search and AI systems depend on successful retrieval before they can analyse, index, understand or reuse the information published on a website.
The Evidence
The Failures Were Not Random Noise
Once the Observatory data was examined over time, the occasional errors stopped looking occasional. A clear pattern of failed machine retrievals emerged.
The impact was not uniform. Some systems happened to retrieve everything they requested successfully, while others repeatedly encountered the intermittent fault.
In this snapshot, Googlebot achieved a 78% business-retrieval success rate, Bingbot 47%, OAI-SearchBot 50%, and GPTBot only 12.5%.
PerplexityBot and ClaudeBot happened to achieve 100% during the same rolling window. That contrast is important: an intermittent infrastructure fault does not necessarily affect every crawler at the same moment.
“Without continuous evidence, this could easily have been dismissed as a handful of unrelated website errors.”
— Keith Rowley, Sydney Business WebFinding the Cause
The Evidence Pointed Beyond WordPress
Before escalating the issue, we worked through the normal application and hosting-account checks to determine whether the failures were being generated by the website itself.
The dominant error was HTTP 522: Cloudflare was unable to establish a connection with the origin server. Importantly, many of those requests never appeared in the normal website access log.
That combination strongly suggested that the failure was occurring before WordPress or PHP had an opportunity to process the request.
We escalated the evidence to the hosting provider with the hourly failure distribution, status-code counts and crawler impact.
AI Visibility Engineering
Being Online Is Not the Same as Being AI Visible
A technically good website can contain excellent information and still fail at the first stage of machine visibility: successful retrieval.
Sydney Business Web treats AI visibility as an engineering problem. Machines must first be able to retrieve the website reliably, then resolve what its entities represent, and finally find enough consistent evidence to trust and reuse that information.
Can search engines and AI systems consistently obtain the information?
Can they determine which business, person, service and entities the information describes?
Can they find consistent technical and external evidence supporting those claims?
The Resolution
The Hosting Provider Confirmed a Rare Infrastructure Problem
The website is hosted by a reputable, high-quality hosting provider. This was not a case of poor hosting or an overloaded customer account. It was an unusual infrastructure fault affecting a small number of servers.
After receiving our evidence, the provider confirmed that it was already working with its server-platform supplier to identify and resolve the problem.
The issue was known to produce intermittent Cloudflare 520, 522 and 525 errors. A vendor patch was already being tested, with permanent deployment expected once stability had been confirmed.
“The interesting part wasn't that a rare hosting fault occurred. Rare faults happen. The interesting part was that the Observatory detected its effect on machine retrieval before ordinary website use made the scale of the problem obvious.”
— Keith Rowley, Sydney Business Web
What This Incident Demonstrated
You Cannot Measure AI Visibility by Looking at the Website
From a human perspective, Sydney Business Web remained largely available. The occasional transient error was easy to dismiss because a refresh usually restored the page immediately.
The Observatory revealed the part that ordinary browsing could not: search and AI systems were repeatedly encountering failed retrievals, and some crawler success rates had fallen dramatically.
That is the larger lesson. AI visibility cannot be inferred from the fact that a website looks healthy in a browser. Machine access has to be observed and measured directly.
Sydney Business Web's AI Observatory provides verified evidence of real search and AI crawler retrieval activity, including which systems arrived, what they attempted to retrieve and whether those requests succeeded.
Explore the AI ObservatoryFrequently Asked Questions
AI Retrieval Failure and Website Availability
The incident illustrates an important distinction between a website appearing available to people and being reliably retrievable by automated search and AI systems.
Why did most human visitors not notice the problem?
The failures were intermittent. A visitor might encounter a brief 502, 503, 504 or similar error, refresh the page and receive it normally. That makes isolated failures easy to overlook even when they are occurring repeatedly across a 24-hour period.
What is an AI retrieval failure?
An AI retrieval failure occurs when a search or AI crawler attempts to obtain information from a website but does not successfully receive the requested content. The machine cannot analyse or use information that was not retrieved.
Can intermittent 5xx errors affect AI visibility?
Repeated failures can reduce opportunities for search and AI systems to discover, revisit and process website information. A single failed request does not establish a visibility problem, but persistent retrieval failures are technically significant and should be investigated.
Was WordPress responsible for this incident?
The evidence did not support that conclusion. Account resource monitoring showed no corresponding limits, WordPress recorded no fatal failures capable of explaining the pattern, and many of the failed requests did not reach the normal webserver access log. The hosting provider subsequently confirmed a rare infrastructure-level issue.
What did the AI Observatory reveal?
It showed the cumulative effect of failures that were difficult to recognise through ordinary browsing. During the incident, qualifying business-information retrieval success fell to 60.3%, with markedly different success rates across individual search and AI crawler systems.
Does this mean the hosting provider is unreliable?
No. Rare infrastructure faults can occur even on high-quality hosting platforms. What matters is detecting the fault, identifying the correct technical layer and applying an appropriate fix. In this case the provider was already working with its server-platform supplier and testing a patch.
References and Further Reading
External References
| Reference | Why it matters |
|---|---|
| Cloudflare — Error 522: Connection Timed Out | Cloudflare's technical explanation of HTTP 522, including failures establishing or maintaining communication with the origin web server. |
| Google Search Central — Technical Requirements | Google explains that a page must be accessible to Googlebot and return an HTTP 200 success response to meet its basic technical requirements for indexing. |
| Google Search Central — Troubleshoot Crawling Errors | Google's guidance on crawler availability problems and the effect persistent server errors can have on crawling. |
Internal Sydney Business Web References
| Page | Related information |
|---|---|
| AI Observatory — Verified AI Retrieval Monitoring | How Sydney Business Web measures verified search and AI crawler retrieval activity at the network edge. |
| AI Retrieval Evidence | Live and recorded evidence showing real machine systems retrieving information from Sydney Business Web. |
| AI Crawler Monitoring at the Cloudflare Edge | Technical background on observing crawler activity without adding client-side scripts or website performance overhead. |
| AI Visibility Services & Pricing | Sydney Business Web's AI Visibility Engineering services, diagnostics and technical implementation options. |
AI Visibility Engineering
Do You Know Whether AI Systems Are Actually Reaching Your Website?
A website can look perfectly healthy to you while search and AI systems are seeing something very different.
Sydney Business Web's AI Observatory provides direct evidence of real crawler retrieval activity — showing which machine systems arrive, whether they successfully obtain your business information, and where retrieval failures are occurring.
