LIVE SEARCH & AI CRAWLER EVIDENCE
See Which Search & AI Crawlers Are Reading Your Website
Watch machine retrieval as it happens. This live Observatory measures which search and AI-related crawlers reach our website, whether their identities are corroborated, and whether they successfully retrieve useful public information.
Not a demonstration dataset. These are actual requests to Sydney Business Web, measured over a rolling 24-hour window.
And this is not just about Sydney Business Web.
This Observatory can be installed on suitable business websites to demonstrate what real machines request, which claims can be corroborated and where retrieval succeeds or fails. The diagnostic population remains visible separately, without inflating the evidence.
Contract 1.5: a claimed crawler User-Agent must match a published operator IP range to qualify. This is source-network corroboration, not cryptographic proof of a particular crawler or evidence of endorsement.
LIVE / ROLLING 24-HOUR EVIDENCE
Who Has Actually Been Here?
A live record of search and AI-related machine requests reaching Sydney Business Web. Contract 1.5 distinguishes IP-corroborated autonomous crawlers from other recognisable but separately classified machine activity.
Distinct autonomous crawlers whose claimed identities match published operator IP ranges. This is an observed count, not a fixed list.
Qualifying GET requests for current registered business resources that returned a successful 2xx response.
The proportion of scored business-information attempts that succeeded. Shows N/A when there are no qualifying attempts.
Qualifying successful requests for approved discovery resources, including robots.txt and sitemaps.
Below are the systems behind those figures. Every crawler in the first table contributes to the corroborated evidence above. The second table shows additional observed activity without adding it to those totals.
CORROBORATED IDENTITY + AUTONOMOUS ACTIVITY
Which Crawlers Produced the Evidence?
Only qualifying systems observed during the current rolling 24-hour window appear here. The population expands or contracts with real activity; there is no preset system count.
| Crawler / provider | Role | Autonomous accesses |
Business retrievals |
Business success rate |
Discovery retrievals |
|---|---|---|---|---|---|
| Loading corroborated crawler evidence… | |||||
A green marker indicates source-IP corroboration against an operator-published range. It is not cryptographic verification of a specific crawler or an endorsement.
VISIBLE, BUT NOT COUNTED AS CORROBORATED EVIDENCE
The Broader Machine Picture
These requests carried recognisable crawler identities but did not qualify for the autonomous, IP-corroborated headline population. Some are unverified identity claims; others may be corroborated but non-autonomous. We show what was observed without treating a User-Agent claim as proof.
| Claimed crawler / provider | Classification | Recorded accesses | Headline eligibility |
|---|---|---|---|
| Loading other observed crawler activity… | |||
Important: An unverified crawler identity means the public corroboration threshold was not met, not that traffic was absent. These accesses are excluded from all corroborated crawler and retrieval metrics above. Multiple crawler names may belong to one provider; counts represent crawler identities, not necessarily separate companies.
Systems appear when activity meets the relevant classification rules and disappear when it leaves the rolling window. The table may show more or fewer names tomorrow. That is measurement doing its job.
What qualifies as public retrieval evidence?
More traffic does not automatically mean better evidence. The Observatory keeps these distinctions explicit.
Synthetic commissioning or diagnostic requests do not enter production evidence.
User-requested or owner-triggered fetches are not counted as autonomous crawling.
Claims without published-IP corroboration are visible separately, not promoted into headline numbers.
Only approved business resources contribute to business retrieval scores; discovery files are scored separately.
Evidence over appearances. The public dashboard deliberately distinguishes corroborated autonomous retrievals from separately observed machine requests. Neither activity alone establishes AI understanding, citation or recommendation.
03 / WHAT IT MEANS FOR YOUR BUSINESS
What Could This Tell You About Your Website?
The live evidence above happens to come from Sydney Business Web because this is where we built and commissioned the system. The principle is not limited to our website. A suitable business website can be instrumented to observe its own search and AI-related machine retrieval activity.
“Are Google, Bing and AI-related machine systems actually reaching my website — and can they retrieve what I want them to see?”
Instead of assuming the answer is yes, the Observatory provides evidence of what is actually happening. Corroborated crawlers and other observed requests are reported separately.
FOUR PRACTICAL QUESTIONS
From crawler activity to useful answers.
Not another vague visibility score. Four distinct things a business can check.
Who is reaching the site?
See which crawler systems meet the published-IP corroboration standard, alongside other crawler-labelled requests that remain explicitly unverified — rather than a theoretical list of bots that might visit someday.
Are they retrieving business information?
Separate meaningful retrieval of public business content from requests for images, scripts, fonts, diagnostic resources and other background website traffic.
Is retrieval succeeding?
Measure whether qualifying requests for business information actually succeed. A crawler arriving at the door is not the same thing as successfully retrieving the information inside.
Has something changed?
Because the evidence is time-based, changes become visible. A machine system can appear, disappear, increase its activity or begin encountering retrieval failures.
Website analytics usually asks:
“What are my human visitors doing?”
Sessions, page views, conversions, traffic sources and visitor behaviour remain extremely useful business measures.
This Observatory asks:
“What are machine systems actually retrieving?”
It adds a different layer of visibility: observable retrieval activity from search and AI-related machine systems.
Suppose an OpenAI-related crawler appears in your corroborated evidence today, successfully retrieves several business pages, and then disappears from tomorrow's rolling window.
That does not mean ChatGPT has suddenly forgotten your business. It means exactly what the instrument says: qualifying activity from that machine system was observed during one measurement window and was not observed during the other.
The dashboard reports evidence — not a story invented around the evidence.
WHY THIS MATTERS
AI visibility should not begin and end with asking an AI system what it thinks about your business.
Those tests can be useful, but they examine the result from the outside. Retrieval evidence gives us another view entirely: whether corroborated machine systems are actually accessing the website and successfully retrieving its information.
That is where the engineering comes in. We distinguish corroborated autonomous evidence from other diagnostic activity before making any public claims.
04 / HOW WE MEASURE
How Do We Know These Numbers Are Real?
The public dashboard does not count a request simply because it calls itself a crawler. AI Observatory records selected production request and response evidence, then applies explicit identity, activity and resource rules before including it in a public measurement.
When a claimed crawler identity cannot be sufficiently corroborated, or an otherwise corroborated request is not autonomous, that activity may be reported separately as diagnostic evidence. It does not inflate the qualifying crawler or retrieval figures.
THE MEASUREMENT PIPELINE
Five decisions between a request and a public figure.
Each stage narrows what we can responsibly claim, rather than manufacturing a single AI visibility score.
Observe production traffic
The collector records selected technical characteristics of requests and responses at the live website edge: claimed crawler, requested resource and returned HTTP status.
REAL REQUESTS, NOT SIMULATED CLAIMSCorroborate the identity
A Googlebot, ClaudeBot or other User-Agent label is not proof of origin. Headline eligibility requires the claimed operator's published source-IP range to match.
USER-AGENT ALONE IS NOT ENOUGHSeparate autonomous activity
Owner-triggered diagnostics and user-requested fetches are not independent autonomous crawling. When recorded, they remain outside the autonomous headline population.
AUTONOMOUS MEANS AUTONOMOUSQualify the resource
Business-content scoring requires a current URL in the explicitly approved resource registry. A separate small registry defines which machine-discovery files count.
ONLY APPROVED RESOURCES ARE SCOREDScore the HTTP outcome
Qualifying GET requests returning 2xx count as success; 4xx and 5xx count as failure. Redirects (3xx) remain observed but are neutral in success-rate denominators.
THE DENOMINATOR IS DEFINEDA CRITICAL DISTINCTION
Two visible populations. Two different claims.
Eligible for headline measurements
Autonomous crawler activity with a claimed identity matched against the operator's published IP ranges. Qualifying business and discovery requests are scored according to their own resource and response rules.
Included in verified system and retrieval totals.Visible, but not counted as verified
Requests carrying recognisable crawler labels that do not satisfy the headline criteria. These include unsupported identity claims and, where present, corroborated non-autonomous fetches.
Reported separately, never added to headline totals.What corroboration means: A matching operator-published IP range supports the claimed provider identity, but is not cryptographic proof of the particular crawler. An unverified label means we could not meet that threshold — not that the request did not happen.
FILTERING MATTERS
What does not inflate the numbers?
Different exclusions apply to different measurements. Not every request is a business-information retrieval, and not every crawler-labelled request qualifies as autonomous.
Our synthetic tests
Commissioning and test requests identified as synthetic are excluded from Contract 1.5 production evidence, including diagnostics.
Triggered or requested fetches
Owner-triggered inspection and user-requested fetches do not qualify as autonomous crawling. When observed, they are classified separately.
Unsupported crawler identities
A familiar User-Agent does not establish origin. Requests without the required IP corroboration stay outside the verified population.
Supporting assets
Images, CSS, JavaScript, fonts and video are not registered business resources; their retrieval cannot inflate the business-content success count.
Security and configuration probes
Requests for secrets, configuration paths and unrelated files do not count as successful retrieval of substantive business information.
HEAD checks and old records
HEAD requests do not retrieve a resource body. Historical Contract 1.4 records are also excluded from Contract 1.5 measurements.
TWO TYPES OF RESOURCE
Business information is not machine discovery.
Did a qualifying crawler retrieve substantive public content?
Only GET requests for current, approved business-resource URLs enter this score. A 2xx response succeeds, 4xx/5xx fails, and 3xx is neutral.
Did a qualifying crawler successfully reach discovery files?
Only GET requests for the five approved discovery paths are scored: robots.txt, llms.txt, wp-sitemap.xml, sitemap.xml and sitemap_index.xml. The same HTTP outcome rules apply.
READING THE DASHBOARD
Four headline figures. And one separate view.
Qualifying machine systems
How many distinct, IP-corroborated autonomous crawler systems were observed?
Business retrievals
How many qualifying GET requests for current business resources returned 2xx?
Business success rate
Of the scored business attempts (2xx, 4xx and 5xx), what percentage succeeded? No attempts means N/A.
Discovery retrievals
How many qualifying requests for approved discovery resources returned 2xx?
Separate diagnostic view: other observed crawler systems, their request counts and why their activity was not eligible for the verified population.
FOR THOSE WHO WANT THE ENGINE ROOM
The Technical Architecture
The deployed Observatory collects selected request and response data at the Cloudflare edge. Analytics Engine stores the measurements. A separate reporting Worker applies the Contract 1.5 aggregation rules and serves the public evidence endpoint consumed by this dashboard.
The public API reports a rolling 24-hour window. Contract 1.4 historical records are not silently merged into the new Contract 1.5 figures.
Raw telemetry is evidence material. It is not yet a conclusion.
Measurement rules turn the activity we can observe into a smaller set of claims we can explain and defend.Retrieval proves retrieval — not understanding or recommendation.
Next: what the evidence supports, and what it cannot establish about an AI system's internal decisions.
05 / THE EVIDENCE BOUNDARY
What This Proves — and What It Does Not
Good measurement becomes less useful the moment we claim more than the evidence actually shows. This observatory has a deliberately narrow job: to measure observable machine access and retrieval.
Retrieval proves retrieval. It does not automatically prove what happened inside an AI system after the information was retrieved.
What We Can Say
Observable production traffic shows that the website received qualifying machine requests during the measured period.
The claimed identity was supported by the relevant operator-published source-IP range, not simply a familiar crawler name.
The observatory can distinguish requests for substantive business information from discovery files, support assets and other traffic.
The website response allows us to measure whether retrieval attempts were successfully completed.
Retrieval of resources such as robots.txt and sitemaps can be measured separately from business-content retrieval.
What We Cannot Say
Successful retrieval tells us the information was accessible, not how an AI model interpreted it internally.
Retrieval is a necessary precursor to many forms of representation, but it is not itself evidence of citation.
A crawler retrieving a page does not prove that an AI assistant later selected the business for a recommendation.
Machine access is technical activity. It must never be represented as endorsement or approval by the provider.
Leads, enquiries, sales and other business outcomes are separate downstream measurements.
WHERE THIS MEASUREMENT SITS
AI Visibility Is a Chain, Not a Single Event
Retrieval is important because a machine cannot reliably work with information it cannot first access. But retrieval is only one part of the larger path from being discoverable to producing a business outcome.
Highlighted stages: the principal part of the chain measured by this retrieval observatory.
Retrieval evidence shows observable machine access to this website. It does not by itself prove AI understanding, citation, recommendation or endorsement.
WHY DRAW THE LINE SO CLEARLY?
Because a measurement is only useful if you know what it measures.
We could make the story sound more impressive by blending crawler activity, AI answers, rankings, citations and recommendations into one vague “AI visibility score”.
We deliberately do not.
Access is access. Retrieval is retrieval. Representation is representation. Outcomes are outcomes. Each deserves its own evidence.
The methodology and evidence are published.
The final part of this page links to the supporting technical material, related AI Visibility Verification work and the next step for businesses interested in measuring their own machine retrieval.
06 / SUPPORTING WORK & VERIFICATION
Don’t Just Take the Dashboard at Face Value.
The live display is the visible end of a larger measurement system. We have published the reasoning, measurement boundaries and real-world retrieval evidence behind it so that the method can be examined rather than treated as a black-box marketing claim.
Measuring Machine Retrieval
The first part of our AI Visibility Verification work explains why crawler activity must be measured carefully and why a crawler name alone is not sufficient evidence of identity.
Read Part One → PART 02Evidence from Real Machine Access
The second part examines actual production retrieval evidence, including what the observations can establish and where their evidential limits lie.
Read Part Two → REFERENCEAI Visibility Glossary
A practical reference for the terminology used across AI search, machine retrieval, entities, structured data, verification and Sydney Business Web’s AI Visibility framework.
Open the Glossary →The figures displayed above are not hard-coded into this page. The page requests its current measurements from the Sydney Business Web evidence reporting endpoint. It reports corroborated autonomous crawler evidence and separately identifies observed diagnostic activity that does not qualify for the verified headline measurements.
View the Raw Public Evidence Feed ↗NOW BRING IT BACK TO YOUR BUSINESS
Could We Measure This on Your Website?
Potentially, yes.
This Sydney Business Web implementation is our working production example. The same measurement principles can be applied to suitable business websites where the required access and technical environment allow reliable observation of machine traffic.
The aim is simple: instead of wondering whether search and AI-related systems are reaching your website, give the business a way to observe and measure what is actually happening.
Which IP-corroborated autonomous search and AI-related crawler systems were observed
Whether they successfully retrieved substantive business information
Whether machine-discovery resources were successfully accessible
How machine activity changes across a rolling observation period
Which observations qualified for defensible public evidence — and which remain diagnostic
Verified autonomous crawlers contribute to the public evidence figures. Other observed crawler-labelled requests appear in a separate diagnostic population, with their verification limits plainly stated. Neither population implies AI understanding, citation or endorsement.
NOT EVERY WEBSITE IS IDENTICAL
We Would Check the Environment First.
This is instrumentation, not a plugin badge. Before promising a dashboard, we need to know that the website and its delivery stack give us the technical access required to measure the right things.
Technical Suitability
We examine the website, hosting, DNS/CDN arrangement and the available observation points.
Measurement Design
We determine which resources and machine behaviours matter for the business and define what should — and should not — count.
Observatory Deployment
The measurement layer is implemented, tested and separated from synthetic commissioning traffic.
Baseline & Reporting
Real production observations establish the baseline from which machine retrieval can be monitored over time.
AI VISIBILITY SHOULD BE OBSERVABLE
Want to Know What the Machines Are Actually Doing on Your Website?
Talk to Sydney Business Web about whether a machine-retrieval observatory is technically appropriate for your website and how it could fit into a wider AI Visibility measurement program.
No claims of guaranteed AI citation, recommendation or ranking. We measure what the available evidence allows us to measure.Engineer → Corroborate → Verify Retrieval → Test Representation
Sydney Business Web — AI Visibility VerificationPRACTICAL QUESTIONS
Live AI Retrieval Evidence FAQ
The measurement is deliberately narrow. These are the practical questions we would expect a business owner to ask after seeing the live evidence above.
01 Can this system be installed on any website? +
Not necessarily. The website, hosting, DNS/CDN arrangement and available technical access need to be assessed first. The measurement system requires a reliable observation point where relevant machine requests and website responses can be recorded without interfering with normal website operation.
02 Does seeing an AI crawler mean my business is being recommended by AI? +
No. It can provide evidence that a qualifying machine system reached the website and successfully retrieved information. It does not by itself prove that an AI system understood, cited, represented, recommended or endorsed the business.
03 Why do the crawler names change from day to day? +
Because the public dashboard uses a rolling 24-hour observation window. A crawler system appears when qualifying activity from it is observed inside that window and can disappear again when that activity falls outside it.
The displayed population is therefore evidence of what was actually observed during the current period — not a fixed list of crawler brands.
04 Why are some machine requests excluded from the public figures? +
Because counting everything would produce a larger number, but a less useful one.
Synthetic tests, owner-triggered diagnostics, insufficiently corroborated crawler identities, support assets, security and configuration probes, HEAD-only requests and other non-qualifying activity are prevented from inflating the headline retrieval evidence.
05 Can the dashboard show more than five crawler systems? +
Yes. There is no five-system limit and no fixed crawler list. The dashboard is generated from the qualifying systems actually observed during the rolling measurement window.
It may therefore display three systems, five systems, eight systems or more as real machine activity changes.
06 Can this tell me whether ChatGPT, Claude or another AI system has actually read my website? +
It can show qualifying retrieval activity attributable to relevant machine systems when that activity is actually observed and meets the required corroboration and measurement rules.
That is strong evidence of machine access and retrieval. It does not reveal what an AI model subsequently understood, retained, cited or did with the retrieved information.
We measure what happened at the website. We do not pretend that this reveals everything that happened afterwards inside somebody else's AI system.
EVIDENCE & TECHNICAL SOURCES
Internal Research & External References
The live retrieval evidence above sits on top of published Sydney Business Web research and first-party technical documentation from the companies operating the infrastructure, search engines and machine systems being measured.
Internal Research & Evidence
Our published work explaining the retrieval methodology, evidence boundaries, terminology and wider AI Visibility framework.
AI Visibility Verification — Measuring Machine Retrieval
Why machine retrieval should be measured separately, why crawler activity needs careful interpretation and why a crawler name alone is not sufficient evidence of identity.
AI Retrieval Evidence from Real Machine Access
Real production retrieval evidence from search and AI-related machine systems, together with the limits on what those observations legitimately allow us to conclude.
AI Visibility Glossary
Practical definitions covering AI Visibility, retrieval, entities, structured data, verification, GEO, AEO and other machine-search concepts used throughout this work.
AI Visibility Services & Pricing
How entity engineering, corroboration, technical accessibility, retrieval measurement and representation testing fit within our wider AI Visibility work.
External Technical Sources
Wherever practical, we reference the organisations actually operating the infrastructure and crawler systems rather than secondary marketing or SEO commentary.
Workers Analytics Engine
Cloudflare documentation for the analytics platform used to record and aggregate measurements from the production retrieval observatory.
Analytics Engine SQL API
Documentation for querying Analytics Engine datasets through the SQL API used by the separate reporting layer behind the public evidence dashboard.
Verified Bots
Cloudflare's documentation covering verified automated systems and the distinction between an asserted crawler identity and independently verified bot traffic.
Googlebot & Crawler Verification
Google's first-party documentation describing Googlebot, crawler identification and methods for verifying whether a request genuinely originates from Google.
Verify Bingbot
Microsoft's guidance for determining whether traffic claiming to be Bingbot actually originates from Bing infrastructure.
OpenAI Publishers & Developers
OpenAI guidance covering web discovery, crawler controls and OAI-SearchBot access for content that may be surfaced through ChatGPT search.
Anthropic Web Crawlers & Site Controls
Anthropic's first-party documentation describing its web crawler activity and the controls available to website owners.
About Applebot
Apple's documentation describing Applebot, its uses, crawler identification, verification methods and website access controls.
We deliberately favour documentation published by the organisations operating the infrastructure and machine systems being discussed.
NOW PUT THE INSTRUMENT ON YOUR WEBSITE
Want to Know Which Search & AI Crawlers Are Actually Reaching Your Business?
Sydney Business Web can assess whether your website is suitable for machine-retrieval monitoring and, where technically appropriate, build an observatory around your own production traffic.
Stop relying entirely on crawler lists, assumptions and occasional AI searches. Measure what is actually happening at your website.
We assess technical suitability first. Retrieval monitoring measures observable machine access and retrieval — it does not guarantee AI citation, recommendation, ranking or endorsement.
Engineering Implementation Note How raw machine traffic becomes defensible retrieval evidence Implementation detail for developers, infrastructure teams and technical reviewers
Production telemetry separated from public reporting
The AI Observatory measures machine activity reaching the production website at the Cloudflare edge. A production Cloudflare Worker records selected request and response characteristics into Cloudflare Analytics Engine.
Public reporting is handled separately. A read-only reporting Worker queries the telemetry dataset, applies the Observatory's evidence rules, aggregates the qualifying observations and supplies the live dashboard through a dedicated evidence endpoint.
Evidence is qualified, not merely counted
Raw automated traffic is not treated as retrieval evidence. A crawler-labelled User-Agent alone is insufficient. Observations must satisfy the applicable identity, autonomy, resource and retrieval-state rules before they can contribute to the public measurements.
Synthetic commissioning requests, owner-triggered diagnostics, insufficiently corroborated crawler identities, support assets, security probes and HEAD-only requests are prevented from inflating substantive business-information retrieval figures.
The evidence boundary is deliberate
The resulting telemetry establishes observable evidence of machine access and retrieval. It does not claim to expose the internal reasoning, model weights or downstream decision processes of Google, OpenAI, Anthropic, Microsoft or other AI providers.
Successful retrieval therefore proves that qualifying machine activity reached and retrieved information from the website. It does not by itself prove subsequent understanding, citation, representation, recommendation or endorsement.