Operational • Managed AI Retrieval Monitoring

AI Crawlers May Be Visiting Your Website. Is Their Retrieval Actually Working?

Don’t just count bots. Investigate the evidence.

The Problem

A crawler visit is not proof of successful retrieval. Access rules, hosting security and failed responses can prevent machines from obtaining useful business information — even when your website appears healthy.

The Sydney Business Web AI Observatory is an Australian-developed managed AI retrieval monitoring system. It records and classifies observable AI and search crawler activity, applies defined filters to business-information retrievals, and provides evidence engineers can use to investigate access failures. The dashboard supplies the evidence; detailed technical investigation and remediation are separately scoped and quoted.

Identify corroborated crawlers Measure qualifying retrievals Detect unsuccessful retrievals
Standard installation A$595 + GST Managed monitoring A$29/month + GST

Personal onboarding included: one 15-minute walkthrough of your results with the engineer who developed Observatory, normally 7–14 days after commissioning.

Standard installation for eligible Cloudflare-based websites • No minimum monitoring term • Cancel anytime • Other configurations, investigation and remediation quoted separately

Developed and operated in Australia by Sydney Business Web, led by Keith Rowley, BSc (Hons) Electronic Engineering, MBA (Bond). Nearly 40 years of hardware, software and systems development experience; formerly Pr.Eng. (SA) and MSAIEE.

Operational Technology. Observable Evidence. See the real crawler-level reporting from our website.

Actual Observatory dashboard, captured earlier on 6 November 2026 using Measurement Contract 1.5. This is a dated example, not a live feed; your website's results will differ. Select the image to view it full size, or inspect our live retrieval evidence. Observed retrieval does not establish AI recommendations or endorsement.

What You’ll Know

Three Questions. Real Answers.

01

Who Came?

Which qualifying AI and search crawler systems were observed reaching your website.

02

What Did They Retrieve?

Whether they reached real business information and machine-discovery resources.

03

Did It Work?

Whether those qualifying retrieval attempts succeeded or failed.

Not just a visit count. Filtered evidence of which qualifying machine systems reached your website and whether their retrieval attempts succeeded.

Professional AI Retrieval Monitoring

Put Your Website Under Observation

Commission a specialist monitoring system that records filtered, observable evidence of recognised AI and search crawler retrieval activity reaching your website.

A$595 + GST one-time standard installation and commissioning

Managed Monthly Monitoring

A$29 + GST / month

Billed monthly in advance from commissioning. No minimum term; cancel anytime.

Website suitability assessment
Observatory installation and configuration
Retrieval verification and commissioning
Access to your private Observatory dashboard
Ongoing AI and search retrieval monitoring
Monitoring-system maintenance
One personal 15-minute results walkthrough, normally 7–14 days after commissioning
What's included — and what's separate: your installation includes one 15-minute walkthrough of the initial evidence with the engineer who developed Observatory. The monthly fee covers ongoing evidence collection, dashboard access and maintenance of the monitoring system. Detailed technical investigation, bespoke reporting and remediation are separately scoped and quoted.

Standard installation is for technically suitable Cloudflare-based business websites. Other configurations may be quoted separately. If more extensive technical checks are needed to establish suitability, we'll explain the scope before proceeding.

All prices exclude GST • Standard offer subject to website suitability • Monitoring is month-to-month with no minimum term

Request a Website Suitability Assessment

Tell us where to find your website. We’ll assess whether it’s suitable for the Sydney Business Web AI Observatory.

0 of 350

Stage 1 of AI Visibility

Start Measuring Now. Build From There.

Available now:
Discoverability → Access → Retrieval

Sydney Business Web is exploring complementary measurement approaches for downstream AI Visibility activity, including representation, citation, recommendation and outcomes. These capabilities are not included in the current Observatory monitoring service.

Retrieval evidence does not by itself prove understanding, citation or recommendation. Those stages require their own evidence.

Technical Engineering Note How the AI Observatory tracks machine retrieval Infrastructure detail for developers, IT teams and systems specialists

Edge-based machine retrieval telemetry

The Sydney Business Web AI Observatory does not depend on browser analytics, tracking pixels or client-side JavaScript to detect autonomous crawler activity. Instead, a production Cloudflare Worker observes selected request and response characteristics at the website edge as machine systems reach the production website.

Qualified telemetry is recorded into Cloudflare Analytics Engine, separating the measurement system from the website's WordPress, PHP and application database environment. A separate read-only reporting Worker queries those measurements, applies the public evidence rules and supplies the Observatory dashboard through a dedicated evidence endpoint.

01 Cloudflare Edge Observe production requests and responses
02 Analytics Engine Store selected telemetry measurements
03 Reporting Worker Filter, classify and aggregate evidence
04 Evidence Endpoint Supply the rolling public dashboard

Multi-stage evidence filtering

Production reporting uses Measurement Contract 1.5. A crawler name in a User-Agent is not accepted as proof of identity: headline autonomous measurements require the claimed crawler User-Agent to match an operator-published source IP range. Provider ASN alone and generic bot-verification flags do not qualify. Earlier-contract records are excluded from Contract 1.5 totals.

Identity Corroboration Headline evidence requires both the claimed crawler User-Agent and a matching operator-published source IP range. This is network corroboration, not cryptographic proof of identity.
Autonomous Activity Synthetic commissioning traffic, internal tests, user-requested fetchers and owner-triggered diagnostics are excluded from autonomous headline measurements.
Resource Classification Business retrievals qualify against an explicit registry of current approved business paths. Discovery resources have their own whitelist and are measured separately; unrelated assets and requests do not inflate the score.
Retrieval State Qualifying GET responses: 2xx = success; 4xx/5xx = failure; 3xx = neutral and excluded from success-rate denominators. With no scored attempts, the rate is N/A, not 0%.

This deliberately produces a smaller evidence set than simply counting raw crawler traffic. The purpose is not to maximise a bot count; it is to produce machine-retrieval measurements that can be defended technically.

What the instrument measures

Discoverability  →  Access  →  Retrieval  →  Understanding  →  Citation  →  Recommendation

The AI Observatory measures principally the access and retrieval stages of this chain. It can provide observable evidence that a system whose claimed identity met Contract 1.5's published-IP corroboration rule reached the website, what class of resource it requested, and whether qualifying retrieval succeeded. The dashboard reports a rolling 24-hour observation window; it does not prove downstream AI model behaviour.

It does not claim to inspect the internal reasoning of Google, OpenAI, Anthropic, Microsoft or another AI provider. Successful retrieval does not by itself prove subsequent understanding, citation, representation, recommendation or endorsement.

The architecture therefore keeps raw telemetry, evidence qualification and public reporting as distinct stages. Raw machine activity becomes public evidence only after it satisfies the defined measurement rules.

Public Methodology & Historical Archive

Current production measurement uses Contract 1.5. The linked public GitHub methodology currently documents Contract 1.4, while the Zenodo v1.0.0 archive preserves an earlier technical release. Both are useful historical references, but neither is a complete specification of the current Contract 1.5 identity-corroboration rules.

GitHub methodology — earlier Contract 1.4:
github.com/Sydney-Business-Web/ai-observatory

Historical archived release — v1.0.0:
DOI: 10.5281/zenodo.22040578

The Engineering Behind the Observatory

Inspect the Method. Check the Evidence.

AI Observatory did not begin as a marketing dashboard. It grew from a technical measurement project designed to establish what can—and cannot—be proved about machine retrieval of a business website. The earlier engineering work and current production evidence are available to inspect.

Parts One and Two document earlier development and historical observations. The live evidence page presents current reporting under Measurement Contract 1.5. Retrieval evidence demonstrates observable machine access; it does not by itself prove understanding, citation, recommendation or endorsement.