AI Visibility Verification — Part Two: AI Retrieval Evidence from Real Machine Access
Part One defined the measurement problem. Part Two puts the measurement system into production and examines what the machines actually do.
This is Part Two of a two-part technical series on AI Visibility Verification. In Part One , we examined a basic problem in AI Visibility measurement: businesses can test what AI systems say about them, but it is much harder to establish what those systems are actually attempting to retrieve from the underlying website.
To investigate that layer properly, Sydney Business Web built a lightweight Cloudflare Worker and deployed it at the network edge in front of this website.
The purpose is simple. When recognised search or AI crawler traffic reaches Sydney Business Web, the Worker records selected technical characteristics of the request and writes the resulting observation to Cloudflare Analytics Engine. Those observations can then be queried, grouped and analysed rather than being lost inside ordinary web traffic.
Cloudflare Worker
A small edge-computing application observes relevant machine requests as they pass through the Cloudflare network. It identifies recognised crawler claims, records retrieval characteristics and allows the website request to continue normally.
Cloudflare Analytics Engine
Selected observations are written into a structured analytics dataset so that crawler identity, request purpose, requested resource, HTTP outcome and other useful retrieval signals can later be examined together.
It is important to be precise about what this system does not do.
The Worker does not create AI visibility. It does not improve rankings, tell an AI system what to say, manufacture citations or present different content to crawlers. It is not an SEO trick and it is not a crawler invitation mechanism.
It is a measurement instrument.
The basic observation path
Which identifiable machine systems are actually reaching Sydney Business Web, what are they attempting to retrieve, and what response do they receive?
That sounds straightforward. In practice, it immediately creates a second engineering problem: not every request should be trusted at face value.
A User-Agent calling itself Googlebot, GPTBot or Applebot is an identification claim; it is not automatically proof that Google, OpenAI or Apple made the request. Likewise, a successful retrieval of a stylesheet or image is not equivalent to the successful retrieval of a substantive article, service page or other information-bearing resource.
So the useful work begins after the request has been observed. We need to distinguish crawler identity claims from stronger attribution, substantive resources from ordinary assets, successful retrieval from failed access, and real production traffic from our own commissioning tests.
This is why the objective of the project is not to produce the largest crawler number possible. It is to produce the smallest defensible body of evidence that still answers the measurement question truthfully.
Measure what can actually be observed, state what that observation proves, and stop precisely where the evidence stops.
Want the engineering background first? Read AI Visibility Verification — Part One: Measuring Machine Retrieval → for the measurement rationale, evidence boundaries and commissioning approach behind this production system.
From Raw Requests to Defensible AI Retrieval Evidence
Detecting a crawler request is easy. Deciding what that request actually tells us is considerably harder.
If the objective were simply to count automated requests, ordinary server logs would already give us plenty of numbers. But a large crawler number is not the same thing as useful AI retrieval evidence.
Before an observation can contribute meaningfully to our measurement of AI Visibility, we need to ask several separate questions about it.
Who does the request claim to be?
The User-Agent may identify the request as Googlebot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot or another recognised system. That gives us a useful starting point — but it remains a claim.
Can that identity be corroborated?
Where suitable signals are available, we compare the crawler claim with additional evidence such as the network organisation carrying the request, Cloudflare verification information or other recognised request indicators.
What was the machine trying to retrieve?
A substantive HTML page, PDF, robots file, sitemap or structured-data resource can be relevant to machine discovery and retrieval. A stylesheet, font or ordinary image request does not tell us the same thing.
What actually happened?
We record whether the request succeeded, redirected, failed or was denied. We also distinguish our own deliberate commissioning tests from genuine production observations so that testing does not inflate the result.
These distinctions matter because the same raw crawler name can represent very different evidence.
A request claiming to be a recognised AI crawler but arriving from an unrelated network should not automatically be attributed to that AI company. Conversely, a crawler identity supported by additional provider-network evidence deserves greater confidence.
We therefore retain an attribution confidence alongside the crawler observation rather than pretending that all identification signals are equally strong.
Attribution is treated as a confidence problem
Depending on what evidence is available for a particular request, the observatory can distinguish between several levels of attribution rather than reducing everything to “bot name = company”.
Cloudflare provides a usable verified-bot signal for the request.
The crawler identity is also consistent with the network organisation carrying the request.
The request contains another useful platform-specific signal supporting the claimed identity.
The crawler name is observed, but no stronger attribution signal is available.
Resource classification requires similar care.
Something being machine-readable does not automatically make it useful business content. The reporting layer therefore needs to distinguish substantive information retrieval from ordinary assets, irrelevant technical requests and other traffic that would exaggerate the apparent level of useful machine access.
Recognised crawler claim + stronger attribution where available + meaningful resource + known HTTP outcome + synthetic-test exclusion = far more defensible retrieval evidence.
We need measurement if we want to measure progress.
The ultimate objective is not to become good at monitoring crawlers. The objective is to make a business website easier for search and AI systems to discover, access, understand and use accurately.
If we change site architecture, crawler access, entity information, structured data, evidence pages or machine-readable resources as part of an AI Visibility programme, we need some means of determining whether relevant machine systems are actually reaching the resulting information.
Successful meaningful retrieval does not prove that an AI system understands the business or will cite or recommend it. But it does measure one of the necessary upstream conditions: can the machine reach and successfully retrieve the information we have engineered for it to find?
That gives us a much cleaner basis for analysis than a simple crawler counter.
With the collection layer running, the classification rules in place and our own tests identifiable, we could finally begin examining the first real production observation window.
That is where the experiment became considerably more interesting.
The First Production Observation Window
Once the measurement system began receiving real production traffic, we froze a defined observation window so that every figure in this analysis could be compared against the same dataset.
Search engines and AI systems do not stop crawling while somebody writes a report. The dataset continues changing as machines revisit the website. So rather than mix figures collected at different times, we selected a fixed cutoff for this first analysis.
The observation window begins on 8 August 2026 at 18:50:14 AEST and ends on 9 August 2026 at 14:06:35 AEST — approximately 19 hours and 16 minutes of initial production data.
18:50:14 AEST
14:06:35 AEST
Production requests matching one of the crawler or machine identities monitored by the Worker, excluding our explicitly marked synthetic tests.
Requests classified by the collection layer as targeting information-bearing or machine-readable resources rather than ordinary site assets.
Machine-readable requests receiving a successful HTTP 2xx response within the frozen observation period.
These are not 151 verified AI content retrievals. They are the result of the collection-layer classification before the more demanding reporting filters for attribution, suspicious traffic and genuine business-content relevance are applied. Production traffic quickly demonstrated why that distinction is necessary.
What the collection layer observed
The crawler names below are identities presented by the requests. They should not all be interpreted as independently verified statements that the named company originated every request. Attribution is examined next.
| Family | Claimed crawler | Purpose | Observed | Readable | 2xx |
|---|---|---|---|---|---|
| OpenAI | ChatGPT-User | User-requested fetch | 49 | 46 | 39 |
| Googlebot | Search | 88 | 46 | 32 | |
| Perplexity | PerplexityBot | Search | 34 | 29 | 23 |
| Anthropic | ClaudeBot | Crawler / model use | 32 | 29 | 22 |
| Microsoft | Bingbot | Search | 66 | 30 | 14 |
| OpenAI | OAI-SearchBot | Search | 16 | 14 | 7 |
| Apple | Applebot | Search / assistant | 25 | 24 | 5 |
| GoogleOther | Other crawler | 18 | 7 | 5 | |
| Google-InspectionTool | Search testing | 88 | 2 | 2 | |
| Anthropic | Claude-User | User-requested fetch | 3 | 3 | 2 |
| OpenAI | GPTBot | Crawler / model use | 11 | 10 | 0 |
| Google-CloudVertexBot | Vertex / Agent crawl | 8 | 6 | 0 |
This is about measuring progress in AI Visibility
The purpose of collecting these observations is not to boast about how many bots visited a website.
If we are engineering a business website to become easier for AI and search systems to discover, access and use, we need feedback telling us whether those systems are actually reaching the information we are improving.
One 19-hour observation window cannot establish a trend. But repeated measurements can begin to show whether successful machine access to substantive business information is improving, declining or being obstructed as technical changes are made.
That gives AI Visibility work a measurable upstream signal — rather than relying entirely on whatever an AI answer happens to say when somebody types a prompt.
At first glance, the production data is encouraging. Multiple major search and AI crawler identities appear in the dataset, and many requests received successful responses.
But almost immediately, the telemetry also showed why those numbers cannot simply be accepted at face value.
Some machines were claiming identities that the surrounding evidence did not support.
And that takes us from counting crawlers to the much more important problem of determining who — or what — actually made the request.
When a Crawler Name Is Not a Crawler Identity
One of the first lessons from production traffic was also one of the most important: a machine can call itself almost anything.
Web crawlers normally identify themselves using a User-Agent. That is useful operational information, but it is not an identity certificate.
Anyone making an HTTP request can place a recognised crawler name into that field. A request can therefore arrive describing itself as Googlebot, GPTBot, ChatGPT-User, ClaudeBot or PerplexityBot without that fact alone proving that Google, OpenAI, Anthropic or Perplexity originated it.
User-Agent is an identification claim, not proof.
“I am GPTBot”
The User-Agent gives us a useful claimed identity and allows the request to be classified for further examination.
“What else supports that claim?”
Network ownership, provider-consistent infrastructure, Cloudflare verification where available and other request signals can materially change how much confidence we place in the attribution.
That distinction became very real in the first production dataset.
We observed multiple requests carrying the names of major search and AI crawlers while the surrounding network evidence did not corroborate the claimed provider. A substantial group of these requests originated from general-purpose DigitalOcean hosting infrastructure, rather than networks consistent with the companies named in their User-Agents.
The requested resources provided another clue. Some of these machines were not attempting to retrieve Sydney Business Web articles, services or public machine-readable resources at all.
Some “crawler” traffic was actually probing technical targets
Requests carrying recognised crawler names included attempts to reach paths commonly associated with configuration files, credentials, source-control data and other sensitive technical resources.
Those requests were correctly denied with HTTP 403 responses. More importantly for this project, they should not be counted as evidence that the AI or search companies named in their User-Agents were retrieving Sydney Business Web content.
Record the claim
The crawler name remains useful telemetry. We do not throw the observation away simply because attribution is uncertain.
Reduce attribution confidence
If the surrounding evidence does not support the provider identity, the observation stays uncorroborated rather than being promoted to verified AI retrieval.
Exclude irrelevant noise
Credential probes and unrelated technical targets should not inflate the measure of meaningful business-content retrieval.
This also revealed a refinement needed in our own reporting model.
During commissioning, we classified some JSON, text and other information-bearing resources as potentially meaningful because machines can legitimately consume those formats. Production traffic showed that this category was still too broad.
A file can be technically machine-readable without representing meaningful retrieval of the business. A credential probe is the obvious example.
So the reporting layer now needs a stronger distinction between machine-readable resources and meaningful business-information retrieval.
Bad measurement can make progress look better than it is.
Imagine improving a business website for AI discovery and then reporting that “AI crawler activity increased by 40%”. If part of that increase consists of unrelated machines wearing recognised crawler names, the measurement is largely useless.
What we actually want to know is whether credible search and AI systems are successfully reaching useful information about the business.
That means removing as much noise as reasonably possible before using retrieval activity as a progress signal. The smaller, cleaner number is often considerably more valuable than the impressive-looking one.
And then the dataset produced the opposite case.
One crawler was arriving from infrastructure consistent with the organisation it claimed to represent. Its requests looked like normal retrieval of public Sydney Business Web resources.
But those requests were repeatedly failing.
That crawler was Applebot — and following the evidence ultimately uncovered a real server-side access fault.
The Observatory Found Something We Had Missed
The most useful result of the first production window was not a large crawler count. It was discovering that a legitimate crawler was being denied access despite the care we had already taken to allow genuine search and AI systems through.
Sydney Business Web already had carefully configured Cloudflare bot and security controls. Those controls had been developed through several iterations specifically to balance two competing requirements: keep hostile automated traffic out while allowing legitimate search and AI crawlers to reach the site.
Testing suggested that arrangement was working.
Production measurement showed that there was still a problem.
During the frozen observation window, 14 Applebot requests were corroborated by Apple Inc. network infrastructure and received HTTP 403 responses while attempting to access Sydney Business Web.
This mattered because Applebot was very different from the uncorroborated crawler claims described in the previous section. The surrounding telemetry supported the Apple attribution: the requests were associated with Apple network infrastructure and behaved like ordinary crawler retrieval rather than credential probing.
So a 403 here was not simply security software doing its job against another machine wearing a borrowed crawler name. It was evidence of a potential accessibility fault affecting a credible search and assistant crawler.
Following the evidence through the stack
Worker telemetry
The retrieval observatory recorded Applebot requests receiving HTTP 403 responses.
Network corroboration
The source network information was consistent with Apple rather than unrelated cloud-hosting infrastructure.
Cloudflare corroboration
Cloudflare AI Crawl Control independently showed unsuccessful Applebot requests and HTTP 4xx outcomes.
Origin logs
Raw hosting access logs showed genuine Applebot requests reaching the server and being returned as HTTP 403.
Root cause identified
GreenGeeks investigated the server and confirmed that its ModSecurity configuration was blocking Applebot.
Targeted remediation
Applebot was specifically whitelisted for the account without disabling ModSecurity globally.
The problem was not where we first expected it to be
Because considerable care had already gone into the Cloudflare configuration, it would have been easy to assume that a crawler access problem must still be located there.
The evidence showed otherwise. Applebot was getting through Cloudflare and reaching the hosting server. The rejection occurred further downstream.
GreenGeeks confirmed the root cause
After being supplied with a specific Applebot request, source IP, UTC timestamp and requested path, GreenGeeks examined the server logs and confirmed that ModSecurity was blocking the requests.
GreenGeeks then whitelisted Applebot specifically for the Sydney Business Web account. That was the right type of correction: narrow, attributable and targeted at the diagnosed fault rather than weakening server security globally.
A controlled post-change test subsequently retrieved the same public article successfully with HTTP 200. The response was reported as DYNAMIC, rather than simply being an old cached response.
Many years ago, my late colleague Gerard de Jong, an RF engineer, used to say:
“Om te meet, is om te weet.” To measure is to know. — Gerard de Jong, RF EngineerConfiguration tells us what we intended the system to do. Measurement tells us what it actually did.
We had configured crawler access carefully. We had tested it. We had iterated the controls. On the evidence then available, legitimate search and AI crawler access appeared sound.
Yet a real production crawler was still being obstructed by another security layer we did not directly control.
Without measurement, that fault could have remained invisible. The configuration would still have looked correct while the machine trying to retrieve the website experienced something entirely different.
This is precisely why retrieval measurement matters to AI Visibility engineering. It provides feedback about the operational result, not merely the intended configuration.
Methodology note: the controlled post-change Applebot test was deliberately marked as synthetic and is excluded from the production evidence totals. Because that test originated from our own connection, its User-Agent identified itself as Applebot but its network did not. It proves the access fault was remediated; it does not prove Apple attribution. Genuine Applebot attribution is established separately from the provider-consistent production traffic.
There is a broader point here.
AI Visibility cannot be reduced to schema markup, crawler settings, an occasional AI prompt or a dashboard number. Each of those can provide evidence, but only about a particular part of the system.
The useful question is always narrower: what does this measurement actually prove?
The Applebot case proved that our observatory could do more than count machine traffic. It could expose an operational retrieval problem, help locate the failing layer and provide a way to verify the correction.
The next question is where that leaves the measurement of AI Visibility itself — and what retrieval evidence can, and cannot, legitimately tell us.
So What Does AI Retrieval Evidence Actually Prove?
The Applebot case shows why retrieval measurement is useful. It also shows why we need to be disciplined about what conclusions we draw from it.
AI Visibility is not a single event. A business must pass through several stages before an AI or search system can accurately represent it, cite it, recommend it or use its information in an answer.
Retrieval sits relatively early in that sequence. It is important — but it is not the end of the sequence.
The AI Visibility verification chain
Discoverability
Can the system discover that the resource or entity exists?
Access
Can the machine actually reach the resource without being obstructed?
Retrieval
Did it request and successfully receive useful information?
Understanding
Did the system correctly interpret the business and its information?
Citation / Representation
Does the business appear accurately in generated answers or results?
Recommendation / Outcome
Is the business selected, recommended or otherwise surfaced?
Our retrieval observatory provides direct evidence primarily about access and retrieval. It does not magically tell us what happened inside a search engine or language model after the information was received.
What happened at the website boundary
- A recognised machine identity requested the website.
- The identity claim had a particular level of attribution confidence.
- The machine requested a particular resource or class of resource.
- The website returned a measurable HTTP outcome.
- A public article, machine file or other resource was successfully served.
- A crawler-access fault exists when legitimate requests are repeatedly obstructed.
What happened after retrieval
- That the machine understood the content correctly.
- That the information entered or changed a model's internal knowledge.
- That the business will be cited in an AI-generated answer.
- That the business will be recommended for a particular query.
- That one crawler visit caused a subsequent AI answer.
- That higher crawler traffic automatically means greater AI Visibility.
Successful retrieval does not prove AI Visibility success. But failed retrieval can certainly prevent it.
If a legitimate machine cannot reach a public resource, it cannot retrieve the information contained in that resource through that request. Whatever happens further along the AI Visibility chain is therefore constrained before it even begins.
Conversely, a successful HTTP 200 response tells us only that the machine received the resource. It does not prove that the system understood, retained, cited or recommended the information.
That makes retrieval evidence a necessary upstream measurement rather than a complete measure of AI Visibility.
The real value appears when measurement is repeated
A single observation window tells us what happened during that period. Repeated observation begins to give us something more useful: evidence of change.
Establish the baseline
Record which credible machines are reaching meaningful business resources and what responses they receive.
Improve the website
Strengthen entity clarity, evidence, structured information, crawler access and the technical architecture supporting AI Visibility.
Look for operational change
Compare subsequent periods to determine whether successful retrieval of useful information is improving or whether new access problems have appeared.
We are trying to replace guesswork with evidence.
A business owner does not need to care about every Worker field, ASN or HTTP header involved in this experiment.
The practical question is much simpler: when we improve the website for machine discovery and retrieval, can we measure whether important search and AI systems are actually reaching the information?
Retrieval evidence gives us part of that answer. Representation testing gives us another part. Neither should be confused with the other.
Used together, they give us a considerably more defensible way of evaluating AI Visibility than relying on rankings, crawler counts or isolated prompts alone.
That distinction also suggests the next logical development of this work.
The raw observatory is an engineering instrument. A client or business owner should not need to interrogate production telemetry to understand whether machine access is healthy.
What they need is a smaller, filtered evidence layer showing credible machine retrieval, access health and change over time.
Not another crawler dashboard — but a public AI Retrieval Evidence facility.
Building a Public AI Retrieval Evidence Facility
The production observatory has already demonstrated that retrieval can be measured. The next challenge is turning that engineering telemetry into evidence that a business owner can actually use.
Raw crawler telemetry is useful to engineers because it preserves detail. It lets us investigate individual requests, attribution signals, response codes, network information, resource classes and anomalies.
That same detail makes it a poor public reporting product.
A useful AI Visibility measurement layer should do the opposite: remove noise, reduce complexity and publish only what can be explained and defended.
The public facility should expose evidence — not expose the observatory.
Raw retrieval observatory
Detailed production telemetry used for diagnosis, attribution, classification, investigation and validation. Valuable internally, but deliberately too granular for public interpretation.
Filtered public reporting
Aggregated evidence showing credible machine access, successful retrieval of useful resources, access health and change over time, with clear explanations of what each measure means.
What should the public facility actually show?
Not everything we can collect. Only the information needed to answer useful AI Visibility questions.
Credible machine families
Search and AI systems observed accessing the website, with attribution quality taken into account rather than relying on crawler names alone.
Successful useful access
Successful retrieval of substantive public resources, separated from site assets, security probes and other irrelevant machine traffic.
What machines reached
Selected articles, service information, machine files and other useful public resources that were actually requested and successfully served.
Failures that matter
Credible crawler failures, recurring HTTP errors or other conditions that may prevent legitimate systems from retrieving public information.
Change over time
Comparable 24-hour, 7-day or longer periods showing whether successful retrieval is becoming healthier, weaker or materially different.
Evidence quality
Clear distinction between strongly corroborated machine activity, partially supported attribution and observations that remain only User-Agent claims.
Keep the reporting architecture separate
The public website should not query raw telemetry directly from browser-side code, and sensitive credentials should obviously never be exposed to the page.
A separate read-only reporting layer can perform the filtering, aggregation and caching, then return only the small body of data intended for publication.
We do not intend to publish raw IP addresses, complete request logs, internal security rules, sensitive diagnostic fields or every automated request that touches the website.
Nor should the public page become a live vanity counter announcing that “AI visited us 4,327 times”. A large number without classification and attribution would contradict almost everything this experiment has taught us.
The public number should be smaller than the raw number because the public claim should be stronger than the raw observation.
Ultimately, AI Visibility needs two different evidence planes
Retrieval evidence answers one class of question. What AI systems subsequently say about the business answers another.
Machine Retrieval Evidence
Which credible machines reached the site, what useful information they requested, whether access succeeded, and whether technical barriers were detected.
AI Representation Evidence
How major search and AI systems describe, identify, cite and represent the business when relevant questions are asked.
Neither evidence plane should pretend to prove what the other does.
Retrieval evidence can show that machines reached information. Representation evidence can show what selected systems subsequently say about the business. Over time, observing both gives us a much richer picture than either can provide alone.
That also fits the wider engineering sequence behind Sydney Business Web's AI Visibility work: engineer, corroborate, verify retrieval, then test representation.
Make AI Visibility improvement observable.
The ultimate point is not to give clients another dashboard. Businesses already have plenty of dashboards.
The point is to create a disciplined feedback mechanism: make technical and entity improvements, observe whether credible machines can retrieve the resulting information, identify access faults when they occur, and compare that evidence with how AI systems actually represent the business.
That does not make AI Visibility perfectly measurable. No serious measurement system should claim that.
But it moves us substantially closer to something much more useful: observable evidence of whether the machinery required for AI Visibility is actually functioning.
This project began with a deceptively simple question: how do we know whether search and AI systems are actually retrieving the information we have engineered for them?
The first production window has already answered part of that question — and exposed weaknesses in both the website environment and our initial measurement assumptions along the way.
That is a good result.
A measurement system that only confirms what we already believe is not doing much measuring.
The Same Crawler Name Can Mean Very Different Things
The HTTP 403 responses in our first production dataset provide a useful demonstration of why crawler names cannot be interpreted without looking at the evidence surrounding them.
DigitalOcean is a general-purpose cloud hosting provider. Businesses, developers, applications and automated systems can all operate from its infrastructure.
There is therefore nothing inherently suspicious about a request originating from DigitalOcean.
What matters here is the mismatch between the identity being claimed and the network evidence we can observe.
For example, a request may announce itself as Googlebot, GPTBot or ClaudeBot, while the network information available to our measurement system identifies only generic DigitalOcean infrastructure rather than an organisation consistent with Google, OpenAI or Anthropic.
That does not prove the request is malicious, spoofed or fraudulent. It means something much narrower: the claimed crawler identity has not been corroborated by the network evidence available to us.
Applebot + Apple Inc.
The request says Applebot and the observed network organisation is Apple Inc. Two pieces of evidence point in the same direction, so attribution confidence is materially stronger.
Googlebot + DigitalOcean
The request says Googlebot, but the surrounding network evidence does not corroborate Google. We retain the observation, but do not treat the User-Agent alone as proof that Google made it.
Fourteen requests identified as Applebot were also associated with Apple Inc. network infrastructure and recorded as a provider organisation match.
Sixty-three blocked requests claimed crawler identities belonging to major search and AI providers but came from DigitalOcean infrastructure without a corresponding provider organisation match.
What the frozen dataset actually recorded
Attribution labels below are produced from the evidence available to our collection system. “UA only” means the crawler name was observed, but the provider identity was not independently supported by our network test.
| Claimed Crawler | Attribution Confidence | Network Evidence | Observed Organisation | 403s |
|---|---|---|---|---|
| Applebot | UA + provider network | Provider organisation match | Apple Inc. | 14 |
| Bingbot | UA only | No provider organisation match | DigitalOcean, LLC | 13 |
| GPTBot | UA only | No provider organisation match | DigitalOcean, LLC | 9 |
| Googlebot | UA only | No provider organisation match | DigitalOcean, LLC | 8 |
| OAI-SearchBot | UA only | No provider organisation match | DigitalOcean, LLC | 7 |
| ClaudeBot | UA only | No provider organisation match | DigitalOcean, LLC | 7 |
| ChatGPT-User | UA only | No provider organisation match | DigitalOcean, LLC | 7 |
| PerplexityBot | UA only | No provider organisation match | DigitalOcean, LLC | 6 |
| Google-CloudVertexBot | UA only | No provider organisation match | DigitalOcean, LLC | 6 |
| Applebot | UA only | No provider organisation match | Linode | 5 |
| GPTBot | UA only | No provider organisation match | Microsoft Limited | 1 |
The requested paths gave us another reason to be cautious
Some of the uncorroborated traffic carrying well-known crawler names was attempting to access resources such as:
These are not ordinary Sydney Business Web articles, service pages or public AI Visibility resources. They are technical targets often associated with configuration, credentials or application internals.
Their presence does not by itself prove malicious intent either. But combined with a crawler identity that lacks provider-network corroboration, they give us very little justification for counting those requests as genuine retrieval by the named AI or search company.
A blocked request is not automatically an AI Visibility problem. In much of the uncorroborated traffic above, HTTP 403 was exactly the sort of response we would expect a security system to produce.
The Applebot observations were different.
During the frozen observation window, 14 Applebot requests had supporting Apple Inc. network evidence and were still receiving HTTP 403 responses. The requested resources were legitimate public website resources, rather than credential or configuration probes.
That combination changed the interpretation completely. Instead of seeing an unidentified machine being correctly blocked, we had evidence that a credible crawler was being prevented from accessing information we intended it to reach.
That was the signal that justified further investigation — and ultimately led us to the ModSecurity fault described in the next section.
The crawler name tells us what the request claims to be. The surrounding evidence determines how much confidence we should place in that claim.
Frozen observation cutoff: 9 August 2026, 14:06:35 AEST. Synthetic SBW test traffic excluded. Figures represent request observations, not unique crawler processes or unique visitors. One additional Applebot observation associated with Apple Inc. did not contain populated attribution fields in this result and has deliberately not been added to the provider-corroborated total.
From AI Visibility Claims to AI Visibility Evidence
This two-part project began because we wanted a better answer to a simple question: how do we know whether the technical work intended to improve AI Visibility is actually working?
Asking an AI system what it knows about a business is useful. Search visibility is useful. Citations are useful. Schema validation is useful. Crawler-control testing is useful.
But each measures a different part of the system.
What was missing was direct evidence from the website boundary: which machines were actually arriving, what they attempted to retrieve, whether their identities could be supported, and whether the requested information was successfully served.
Define the measurement problem
Part One established the verification chain and the limits of what machine-retrieval telemetry can legitimately prove.
Put the measurement into production
Part Two collected real traffic, froze a production dataset, challenged crawler identities, refined the reporting model and used the evidence to expose a genuine access fault.
We moved one part of AI Visibility from something we could configure and assume to something we could observe and test.
And the measurements immediately taught us things.
Crawler names are not identities
User-Agent tells us what a request claims to be. Attribution requires supporting evidence.
Machine-readable is not automatically meaningful
Production traffic showed why technical probes and irrelevant resources must be separated from useful business-information retrieval.
Correct configuration is not enough
Carefully designed crawler controls did not reveal the Applebot failure. Production measurement did.
My late colleague Gerard de Jong, an RF engineer, expressed the underlying engineering principle rather more elegantly:
“Om te meet, is om te weet.” To measure is to know.Retrieval evidence is not the whole of AI Visibility. It was never intended to be.
Successful retrieval does not tell us that an AI system understood the business, cited it, represented it correctly or recommended it. Those require different evidence.
But retrieval is an important upstream condition. If useful information cannot reliably reach legitimate machines, later stages of the AI Visibility chain may never get the chance to operate.
The sensible approach is therefore not to search for one magical AI Visibility score.
It is to build a body of evidence across the chain: engineer the information, corroborate the entity, verify retrieval, test representation and observe the resulting outcomes over time.
Not the largest number we can publish. The smallest number that still answers the question truthfully.
AI Visibility should not become another branch of digital marketing built primarily on claims that are difficult to test.
Engineer it. Corroborate it. Measure it. Then show the evidence.
Internal and External References
The retrieval measurements reported in this article come from Sydney Business Web's own production observatory. The references below provide the surrounding methodology, system documentation and first-party crawler information used to interpret that evidence.
Internal References
AI Visibility Verification — Measuring Machine Retrieval
The first part of this two-part series establishes the measurement problem, verification chain and evidentiary boundary between retrieval and later AI outcomes.
Read Part One →Engineering Business Websites for AI Visibility
Our broader engineering approach to making business information discoverable, accessible, understandable and verifiable by search and AI systems.
Read the AI Visibility guide →The Intelligent Entity Skeleton
Sydney Business Web's framework for establishing a coherent machine-readable structure connecting the business, people, services, evidence and published content.
Explore the framework →The AI Credibility Footprint
Our framework for strengthening the consistency, provenance and external corroboration surrounding a business entity across the wider web.
Explore the framework →AI Visibility Glossary
Definitions for crawler access, retrieval, grounding, structured data, entities, GEO, RAG and other terminology used throughout Sydney Business Web's AI Visibility work.
Open the glossary →AI Visibility Services & Pricing
How Sydney Business Web applies entity engineering, technical accessibility, structured evidence and verification to commercial business websites.
View AI Visibility services →External References
Workers Analytics Engine
Cloudflare's official documentation for the analytics platform used by the SBW retrieval observatory to record and query custom machine-access telemetry.
Cloudflare documentation →Googlebot and Crawler Verification
Google's official crawler documentation, including its warning that User-Agent strings can be spoofed and its guidance for verifying whether requests genuinely originate from Google.
Google Search Central →OpenAI — Publishers and Developers
OpenAI's first-party guidance on public website discoverability and OAI-SearchBot access for content that may be surfaced and cited through ChatGPT search.
OpenAI documentation →About Applebot
Apple's official description of Applebot, its crawler behaviour and its use across Apple search experiences, Siri, Safari and AI-related products and services.
Apple Support →Anthropic Web Crawlers
Anthropic's first-party documentation describing ClaudeBot, Claude-User and the different purposes for which its automated systems access public web content.
Anthropic documentation →Perplexity Crawlers
Perplexity's official crawler documentation covering PerplexityBot, Perplexity-User, their respective purposes and published crawler-network information.
Perplexity documentation →Verifying Bingbot
Microsoft's official guidance for distinguishing genuine Bingbot requests from traffic that merely uses the Bingbot User-Agent identity.
Bing Webmaster Tools →Querying Analytics Engine from a Worker
Cloudflare's official implementation guidance for querying Analytics Engine through the SQL API from a Worker — the model used for the reporting layer described in this project.
Cloudflare implementation guide →The production measurements and crawler-attribution tables in this article are derived from Sydney Business Web's own retrieval observatory and are therefore primary operational evidence rather than figures taken from the external sources above.
The diagnosis that genuine Applebot requests were being blocked by ModSecurity, and the subsequent account-specific Applebot whitelist, were confirmed directly to Sydney Business Web by GreenGeeks support following examination of the server logs.
External documentation is used to support crawler definitions, verification methodology and platform behaviour — not to substitute for the measurements recorded on this website.
We Are Building the Measurement Panel Into This Website
The retrieval observatory described in this series is currently an engineering instrument. Our next step is to turn its filtered output into a public AI Retrieval Evidence panel on Sydney Business Web itself.
The panel will not be a raw bot counter. It will be designed to show a much smaller and more useful body of evidence: credible machine systems observed accessing the site, successful retrieval of meaningful public information, access-health warnings and change over time.
In other words, the website will not merely describe our approach to AI Visibility measurement. It will begin publishing the measurement evidence itself.
Want to know what the machines can actually discover and retrieve about your business?
Sydney Business Web approaches AI Visibility as an engineering problem: establish the entity clearly, strengthen the evidence around it, make the information accessible to machines, and then measure what can actually be observed.
Engineer it. Corroborate it. Measure it. Then show the evidence.