AI Crawlers May Be Visiting Your Website. Is Their Retrieval Actually Working?
Don’t just count bots. Investigate the evidence.
The Problem
A crawler visit is not proof of successful retrieval. Access rules, hosting security and failed responses can prevent machines from obtaining useful business information — even when your website appears healthy.
The Sydney Business Web AI Observatory is an Australian-developed managed AI retrieval monitoring system. It records and classifies observable AI and search crawler activity, applies defined filters to business-information retrievals, and provides evidence engineers can use to investigate access failures. The dashboard supplies the evidence; detailed technical investigation and remediation are separately scoped and quoted.
Personal onboarding included: one 15-minute walkthrough of your results with the engineer who developed Observatory, normally 7–14 days after commissioning.
Standard installation for eligible Cloudflare-based websites • No minimum monitoring term • Cancel anytime • Other configurations, investigation and remediation quoted separately
Developed and operated in Australia by Sydney Business Web, led by Keith Rowley, BSc (Hons) Electronic Engineering, MBA (Bond). Nearly 40 years of hardware, software and systems development experience; formerly Pr.Eng. (SA) and MSAIEE.
Actual Observatory dashboard, captured earlier on 6 November 2026 using Measurement Contract 1.5. This is a dated example, not a live feed; your website's results will differ. Select the image to view it full size, or inspect our live retrieval evidence. Observed retrieval does not establish AI recommendations or endorsement.
What You’ll Know
Three Questions. Real Answers.
Who Came?
Which qualifying AI and search crawler systems were observed reaching your website.
What Did They Retrieve?
Whether they reached real business information and machine-discovery resources.
Did It Work?
Whether those qualifying retrieval attempts succeeded or failed.
Professional AI Retrieval Monitoring
Put Your Website Under Observation
Commission a specialist monitoring system that records filtered, observable evidence of recognised AI and search crawler retrieval activity reaching your website.
Managed Monthly Monitoring
Billed monthly in advance from commissioning. No minimum term; cancel anytime.
Standard installation is for technically suitable Cloudflare-based business websites. Other configurations may be quoted separately. If more extensive technical checks are needed to establish suitability, we'll explain the scope before proceeding.
All prices exclude GST • Standard offer subject to website suitability • Monitoring is month-to-month with no minimum term
Request a Website Suitability Assessment
Tell us where to find your website. We’ll assess whether it’s suitable for the Sydney Business Web AI Observatory.
Stage 1 of AI Visibility
Start Measuring Now. Build From There.
Available now:
Discoverability → Access → Retrieval
Sydney Business Web is exploring complementary measurement approaches for downstream AI Visibility activity, including representation, citation, recommendation and outcomes. These capabilities are not included in the current Observatory monitoring service.
Retrieval evidence does not by itself prove understanding, citation or recommendation. Those stages require their own evidence.
Technical Engineering Note How the AI Observatory tracks machine retrieval Infrastructure detail for developers, IT teams and systems specialists
Edge-based machine retrieval telemetry
The Sydney Business Web AI Observatory does not depend on browser analytics, tracking pixels or client-side JavaScript to detect autonomous crawler activity. Instead, a production Cloudflare Worker observes selected request and response characteristics at the website edge as machine systems reach the production website.
Qualified telemetry is recorded into Cloudflare Analytics Engine, separating the measurement system from the website's WordPress, PHP and application database environment. A separate read-only reporting Worker queries those measurements, applies the public evidence rules and supplies the Observatory dashboard through a dedicated evidence endpoint.
Multi-stage evidence filtering
Production reporting uses Measurement Contract 1.5. A crawler name in a User-Agent is not accepted as proof of identity: headline autonomous measurements require the claimed crawler User-Agent to match an operator-published source IP range. Provider ASN alone and generic bot-verification flags do not qualify. Earlier-contract records are excluded from Contract 1.5 totals.
This deliberately produces a smaller evidence set than simply counting raw crawler traffic. The purpose is not to maximise a bot count; it is to produce machine-retrieval measurements that can be defended technically.
What the instrument measures
The AI Observatory measures principally the access and retrieval stages of this chain. It can provide observable evidence that a system whose claimed identity met Contract 1.5's published-IP corroboration rule reached the website, what class of resource it requested, and whether qualifying retrieval succeeded. The dashboard reports a rolling 24-hour observation window; it does not prove downstream AI model behaviour.
It does not claim to inspect the internal reasoning of Google, OpenAI, Anthropic, Microsoft or another AI provider. Successful retrieval does not by itself prove subsequent understanding, citation, representation, recommendation or endorsement.
The architecture therefore keeps raw telemetry, evidence qualification and public reporting as distinct stages. Raw machine activity becomes public evidence only after it satisfies the defined measurement rules.
Public Methodology & Historical Archive
Current production measurement uses Contract 1.5. The linked public GitHub methodology currently documents Contract 1.4, while the Zenodo v1.0.0 archive preserves an earlier technical release. Both are useful historical references, but neither is a complete specification of the current Contract 1.5 identity-corroboration rules.
GitHub methodology — earlier Contract 1.4:
github.com/Sydney-Business-Web/ai-observatory
Historical archived release — v1.0.0:
DOI: 10.5281/zenodo.22040578
The Engineering Behind the Observatory
Inspect the Method. Check the Evidence.
AI Observatory did not begin as a marketing dashboard. It grew from a technical measurement project designed to establish what can—and cannot—be proved about machine retrieval of a business website. The earlier engineering work and current production evidence are available to inspect.
Measuring Machine Retrieval
The engineering problem: separating crawler permission, machine visits and genuine retrieval into things that can actually be measured.
Read Part One Part TwoReal Retrieval Evidence
Earlier production observations, including crawler attribution, successful retrievals and access problems that demonstrated why stricter evidence qualification matters.
Read Part Two Live EvidenceSee the Observatory Running
View Sydney Business Web’s filtered public evidence, including observed machine systems, retrieval activity and successful access to meaningful website resources.
Open the Live ObservatoryParts One and Two document earlier development and historical observations. The live evidence page presents current reporting under Measurement Contract 1.5. Retrieval evidence demonstrates observable machine access; it does not by itself prove understanding, citation, recommendation or endorsement.