An AI crawler reaches a website gateway only to be stopped by an HTTP 429 “Too Many Requests” barrier, illustrating how rate limiting can disrupt AI retrieval.
Your Website Is Online. So Why Can't AI Retrieve It?
A website can be online, indexed by Google, technically sound and supported by structured data — yet still fail when an AI crawler attempts to retrieve its content.
An AI Visibility problem hiding in plain sight
At Sydney Business Web, we were monitoring AI crawler activity on our own website when an unexpected pattern emerged: a substantial proportion of business retrieval attempts were unsuccessful.
For ordinary human visitors, the website appeared to be operating normally. But our AI Observatory was recording something different — repeated unsuccessful HTTP responses to requests identifying as AI crawlers.
The investigation moved from edge telemetry through Cloudflare security analytics, crawler impersonation and HTTP response codes, eventually reaching the raw Apache HTTPS access logs.
There, we found origin-log evidence of HTTP 429 responses affecting requests using the GPTBot user-agent. The responses were concentrated in particular sessions and included requests for genuine business pages. A user-agent string alone does not authenticate the requester.
Being visible to a search engine does not automatically mean being reliably accessible to an AI retrieval system.
The first warning
AI Observatory records observed crawler-labelled requests and their HTTP response outcomes rather than inferring access from indexing or search visibility. A successful HTTP status is not, by itself, proof that the expected page content was delivered: a security challenge could return a successful status. The GPTBot-labelled results raised the first question about retrieval reliability.
AI Visibility is also a delivery problem
Technical SEO provides essential foundations: crawlability, indexing, site performance, canonical URLs and accessible content. AI Visibility adds a further operational consideration — can AI systems consistently retrieve the information they need?
A coherent Intelligent Entity Skeleton can help machines understand relationships within a business. An AI Credibility Footprint can support external corroboration. Neither removes the need for reliable content delivery.
Following the Evidence: Where Were the Requests Failing?
The first task was to establish whether Cloudflare was rejecting legitimate AI crawler traffic, whether the origin server was failing to respond, or whether the statistics were being distorted by malicious automated requests.
Cloudflare reported 380 allowed requests and 508 requests in other, unsuccessful classifications across OpenAI-labelled crawler traffic. These are Cloudflare analytics categories, not a count of HTTP failures or proof that 508 content retrievals failed.
GPTBot, OAI-SearchBot and ChatGPT-User have different functions. Their activity must not be treated as interchangeable, and an unsuccessful request cannot automatically be interpreted as lost AI Visibility.
Not every GPTBot is really GPTBot
The Cloudflare security-event export delivered an unexpected finding. Some requests carrying a GPTBot user-agent string were probing for sensitive configuration files and attempting operations unrelated to ordinary web crawling.
The requests included attempts to access development configuration files and endpoints associated with vulnerability scanning. Cloudflare's managed security rules blocked several of these requests.
A user-agent string is a declaration, not proof of identity. Requests claiming to be GPTBot must be distinguished from verified OpenAI crawler traffic before drawing conclusions about AI Visibility. That requires checking the source against OpenAI's published crawler IP ranges, rather than relying on the label alone.
GET /.npmrc
GET /local_settings.py
POST /api/auth
POST /dashboard
POST /_next
Separating HTTP responses
Filtering Cloudflare HTTP traffic for requests containing the GPTBot user-agent identifier produced a more precise picture. Of 323 recorded requests, only two received an HTTP 5xx response. The larger issue lay among the HTTP 4xx responses. These figures describe HTTP status codes, not authenticated crawler identities or verified content delivery. An HTTP 2xx status could, for example, accompany a security challenge instead of the requested page.
The evidence that changed the investigation
Unlike the suspicious configuration-file probes, these rate-limited requests included genuine business pages and WordPress resources. They were concentrated in a short period rather than distributed evenly across the day.
HTTP 429 means that a request has been rejected because a rate-limiting threshold was reached. It does not, by itself, identify which component applied the restriction.
The key distinction for AI Visibility is that the affected requests included meaningful website content. A rate-limited retrieval may prevent a crawler from obtaining the requested information during that particular attempt. Whether it retries successfully, or whether this changes subsequent AI-generated answers, requires separate evidence.
The three monitoring layers were not measuring identical populations or periods. AI Observatory's 41 unsuccessful business-page requests came from a rolling 24-hour measurement restricted to its qualifying business retrievals. Cloudflare's 112 HTTP 429 responses covered GPTBot-labelled requests across a wider range of website paths in a separate 24-hour snapshot. Apache's 93 HTTP 429 responses in the second incident came from origin access-log records during a particular burst.
Business pages, images, WordPress REST endpoints and other paths are represented differently across these measurements. The counts are therefore complementary observations, not three totals that should be numerically equal. A request-level reconciliation would require matching timestamps, paths and identifying data across the underlying records. That reconciliation has not been published.
The next question: Cloudflare or the origin?
Cloudflare established that HTTP 429 responses were occurring, but the analytics alone did not identify their source. Inspecting the website's Apache HTTPS access logs supplied the next layer of evidence: the origin was also recording those responses. The hosting provider subsequently identified an Imunify360 rate-limiting rule as the responsible mechanism, documented in the following sections.
The Origin Logs: Where the HTTP 429s Became Visible
Cloudflare had shown us that requests identifying as GPTBot were receiving HTTP 429 responses. But where were those responses coming from? The decisive next step was to examine what the hosting server itself had recorded.
The same failure, now visible in the Apache log
The October 2026 HTTPS access log for Sydney Business Web contained 423 requests carrying a GPTBot identifier in the user-agent. Of these, 130 were recorded with HTTP status 429 — Too Many Requests.
Why the HTTPS log mattered
Our initial examination of the conventional HTTP access log did not reveal the relevant traffic. Sydney Business Web uses Cloudflare extensively for content delivery, performance and edge security, with encrypted HTTPS connections to its hosting infrastructure.
Examining the separate Apache SSL access log changed the investigation. It revealed both the affected requests and the HTTP 429 response codes recorded by the origin.
This provided additional evidence that rate limiting was occurring along the delivery path. It also narrowed the investigation: the failures were not solely requests rejected at Cloudflare's edge before reaching Apache.
The rate-limiting timeline
All 130 HTTP 429 responses associated with GPTBot-labelled requests in the examined log occurred in two clusters, five days apart.
What Apache recorded
The wider response distribution confirms that the requests were not uniformly rejected. HTTP 200 and HTTP 429 responses occurred within the same monthly dataset. An HTTP 200 status does not, by itself, verify that the intended content rather than a security challenge was delivered.
Not simply nonexistent files
The origin log revealed 429 responses affecting normal website resources, including service pages, technical articles, image files and WordPress REST API endpoints.
This distinction matters. A scanner attempting to retrieve a hidden configuration file is not equivalent to a crawler being denied a genuine business page.
For AI Visibility, the latter is a practical delivery concern: requested information may be unavailable during an otherwise legitimate retrieval attempt. Whether any individual request came from the genuine crawler requires verification.
Examples from the Apache 429 records
What we know — and what remains to verify
Established by the recorded evidence
Apache recorded 130 HTTP 429 responses to GPTBot-labelled requests, concentrated in two episodes. These included legitimate business pages and supporting resources. The origin records demonstrate unsuccessful HTTP retrieval attempts affecting meaningful website content. Following examination of these findings, the hosting provider identified an Imunify360 WAF rate-limiting rule as the responsible mechanism.
Still requiring verification
The provider's diagnosis has not been independently substantiated in this article by a published Imunify360 rule ID or matching WAF event records. The user-agent labels do not prove that every request originated from OpenAI. Nor do HTTP response codes alone establish that successful requests received the intended page rather than a challenge. Any effect on ChatGPT answers or business recommendations also remains unproven.
Engineering the Complete Hosting Environment
Root Cause Identified — Monitoring ContinuesSydney Business Web deliberately operates its own business website on a high-quality shared hosting platform. For customers whose applications, traffic or operational requirements demand greater dedicated resources and control, we also provide advanced VPS hosting solutions.
Our approach is to select infrastructure appropriate to each business, then complement it with professional configuration, Cloudflare integration and ongoing engineering supervision.
This combination allows customers to benefit from established hosting infrastructure and sophisticated cloud-edge capabilities, supported by the technical experience needed to make those systems work effectively together.
The investigation illustrates that engineering approach. Having identified the HTTP 429 responses in Apache's HTTPS logs, we referred the specific evidence to the infrastructure provider for examination.
The provider reported that an Imunify360 Web Application Firewall rule was responsible. According to its technical diagnosis, the rule limited each domain-specific GPTBot session to 30 requests within a 10-second window, returning HTTP 429 responses when that threshold was exceeded.
The provider subsequently whitelisted the responsible rule for our hosting account. It also confirmed that the hosting account had not reached its resource limits. This was therefore diagnosed as a specific security restriction, rather than an exhausted hosting capacity problem.
The intervention was targeted at the identified rule, not a wholesale removal of website security. AI crawler accessibility must remain balanced against protection from malicious automated traffic.
Early Cloudflare monitoring recorded zero GPTBot-labelled HTTP 429 responses in a later 24-hour snapshot. This was encouraging, but the traffic volume was lower and there were no qualifying GPTBot business retrievals in the corresponding AI Observatory window. The evidence is therefore insufficient to declare the correction fully verified under renewed genuine crawler activity.
The provider's diagnosis and corrective action are documented here. The follow-up section presents the initial measurements and the remaining verification work. Additional findings will be published as evidence becomes available.
If a website is indexed, structured correctly and functioning for human visitors, how much confidence can a business place in its AI Visibility without measuring whether its intended content is actually reaching retrieval systems?
AI Visibility Is More Than Being Found. It Is a Connected System.
The HTTP 429 investigation illustrates a wider engineering principle: discoverability, reliable retrieval, coherent business identity and independent credibility all contribute to effective AI Visibility. A weakness in one can undermine the value of the others.
Discoverability
Can AI crawlers locate the website, its important pages and the information describing the business?
Retrieval Reliability
When content is requested, can the delivery infrastructure return the intended content successfully and consistently, rather than merely a successful HTTP status?
Entity Coherence
Do the website and its structured data communicate consistent relationships between the business, services and people?
External Credibility
Is the business identity supported by independent evidence beyond its own website?
Retrieval is the connection between discovery and understanding
Conventional SEO has long recognised the importance of making website content discoverable and accessible. AI Visibility builds on those foundations, while introducing additional questions about how AI retrieval systems obtain and interpret business information.
A crawler may discover a page successfully and still encounter a failed request when attempting to obtain its contents. Equally, a successful HTTP status does not conclusively demonstrate that the intended page was delivered: a security challenge may return an HTTP 200 response.
In our investigation, AI Observatory exposed the HTTP response failure pattern. Cloudflare revealed concentrated HTTP 429 responses, and the Apache HTTPS logs independently recorded rate limiting affecting genuine business resources. The hosting provider subsequently attributed the behaviour to an Imunify360 WAF rule.
The central lesson is not that every crawler should be unrestricted. It is that retrieval reliability must be observable, measurable and managed alongside conventional website security.
When information arrives, does it form a coherent identity?
Successful delivery of the intended content is necessary for a crawler to obtain that information, but it does not establish that the business will be understood correctly. Websites contain pages, names, service descriptions, locations, people and relationships that must form a consistent picture.
The Intelligent Entity Skeleton
Sydney Business Web uses the concept of an Intelligent Entity Skeleton to describe a coherent framework of connected business information.
Structured data can express relationships between an organisation, its services, products, people and other relevant entities. Those connections must also remain consistent with the visible website.
For AI Visibility, the objective is not simply to create valid schema markup. It is to present an internally coherent business identity that retrieval systems can interpret.
The AI Credibility Footprint
A business can describe itself accurately on its own website. But external sources provide another dimension of evidence.
The AI Credibility Footprint describes the wider collection of independent information that can corroborate a business identity, its activities and its expertise.
Relevant citations, professional references, publications and other independently maintained sources can support that wider identity when they are accurate and genuinely relevant.
This is a matter of corroboration, not a guarantee of inclusion in AI-generated answers.
Valid structured data does not guarantee coherent identity. Coherent identity does not guarantee retrieval. And successful retrieval does not guarantee recommendation.
What the HTTP 429 investigation adds to AI Visibility practice
The incident demonstrated that a website may possess the technical foundations for strong AI Visibility while still experiencing delivery failures under particular request conditions.
Simply validating structured data, checking robots.txt or confirming search-engine indexing would not have revealed this pattern.
It required evidence from multiple infrastructure layers: AI Observatory telemetry, Cloudflare analytics and the origin server's access logs.
Equally important was the distinction between requests claiming to represent recognised AI crawlers and independently authenticated crawler traffic. A user-agent identifier alone cannot establish that identity.
AI Visibility needs evidence, not assumptions
Discoverability, reliable retrieval, entity coherence and external corroboration form complementary aspects of AI Visibility. The investigation began with measurable HTTP response failures and progressed through Cloudflare and Apache evidence. The infrastructure provider subsequently identified an Imunify360 WAF rate-limiting rule and applied a targeted correction. The remaining engineering tasks are to verify the outcome during renewed authenticated crawler activity and distinguish successful HTTP responses from confirmed delivery of the intended website content.
The Cause Was Identified. Now We Measure the Correction.
Our infrastructure provider identified the technical mechanism behind the HTTP 429 failures: an Imunify360 Web Application Firewall rule restricting GPTBot sessions when a specific request-rate threshold was exceeded. The diagnosis is attributed to the provider; independent WAF rule records have not been published in this case study.
Provider Diagnosis: A Security Rule, Not Resource Exhaustion
The hosting infrastructure provider reported that an Imunify360 WAF rule enforced a limit of 30 requests within a 10-second window per domain-specific GPTBot session.
According to that diagnosis, exceeding the threshold resulted in temporary HTTP 429 — Too Many Requests responses. The provider also examined account resource usage and reported no evidence that hosting resource limits had been reached.
This distinction matters. The failures were attributed to a particular security policy rather than insufficient shared hosting capacity. The provider's finding complements our Apache evidence, but the detailed WAF rule ID and matching event records have not been published.
AI Observatory
Identified an unusually low HTTP-status success rate among GPTBot-labelled business-page requests, prompting further investigation.
Cloudflare & Apache
Revealed concentrated HTTP 429 responses and confirmed that affected requests were recorded in the origin HTTPS logs. The user-agent label alone did not authenticate crawler identity.
Targeted WAF Whitelist
The hosting provider reported that it whitelisted the identified Imunify360 rule for the affected hosting account, rather than disabling website security generally.
What Cloudflare Shows Now
Following the reported WAF adjustment, we returned to Cloudflare and repeated the HTTP traffic analysis using GPTBot-labelled requests and HTTP status filters.
The two snapshots are not matched experiments: they contain different traffic volumes and may include requests from before the WAF adjustment. GPTBot-labelled traffic also included suspicious requests whose identity was not authenticated. Zero observed HTTP 429 responses in the later snapshot is encouraging, but does not establish reliable retrieval during another substantial, verified GPTBot crawling session.
Retrieval Activity in the Follow-Up Window
AI Observatory's follow-up rolling 24-hour snapshot recorded successful HTTP responses for qualifying business-page requests associated with several major crawler user-agent categories. These status-based measurements do not independently verify response-body content or authenticate every requester.
Bingbot: 16/16 business requests
The same snapshot recorded Googlebot at 11/12 business requests with successful HTTP status (91.7%), Applebot at 6/6 and PerplexityBot at 1/1.
GPTBot showed one successful discovery request, but no qualifying business-page requests in that rolling window (0/0). A post-correction GPTBot business-request success rate therefore could not be calculated.
During a user-initiated website fetch by the Claude AI assistant, the requester received an HTML verification page rather than the intended website content. That observation is not explained by the GPTBot HTTP 429 evidence or by the hosting provider's reported whitelist correction. It may involve a different security mechanism and requires its own investigation.
An HTTP 200 response can sometimes contain a challenge page rather than the requested business content. Where monitoring measures response status rather than response-body integrity, such a request could appear successful without proving that useful information was delivered. We have not established whether this occurred within the Observatory measurements in this case.
Our next engineering task is to determine whether AI Observatory can distinguish an HTTP 200 response containing the intended website content from one containing a security verification page. We will examine the available telemetry, test suitable detection methods and document the outcome, whether or not changes to our monitoring methodology are required.
Provider Diagnosis Received. Corrective Action Reported. Verification Continues.
This investigation documented repeated HTTP 429 responses affecting GPTBot-labelled requests in Cloudflare analytics and the origin HTTPS logs. The hosting provider identified an Imunify360 rate-limiting rule as the cause and reported applying a targeted whitelist correction.
Early follow-up measurements show no HTTP 429 responses in the later sampled Cloudflare window. However, that observation cannot establish whether the change will hold under renewed, independently verified GPTBot business retrieval activity.
Further engineering checks include confirming crawler source identities, obtaining supporting WAF rule evidence where available, and examining whether successful HTTP responses delivered the intended pages rather than security challenges.
We will revisit these findings and update the article with further measured results. The distinction between an observed HTTP response, a provider diagnosis and an independently verified retrieval outcome will remain explicit.
For AI Visibility, the wider lesson remains clear: reliable content delivery cannot be assumed merely because a website is indexed, operational and well structured. It needs to be measured.
Verification in progress — 21 October 2026AI Crawler HTTP 429 Errors: Questions & Answers
What do HTTP 429 errors mean for AI Visibility, and how can website operators distinguish legitimate crawler activity from security threats? These questions summarise the practical lessons from our investigation.
What does HTTP 429 mean for an AI crawler?
HTTP 429 means "Too Many Requests". It indicates that a server or an intermediary is rate-limiting requests.
For an AI crawler, this may prevent requested pages or resources from being retrieved during that attempt. The crawler may retry later, but successful retrieval is not guaranteed.
Can HTTP 429 errors affect AI Visibility?
Yes. Repeated retrieval failures can reduce the opportunities for AI crawler systems to obtain current business information.
However, an HTTP 429 response does not prove that a business has been excluded from ChatGPT or another AI-generated answer. Crawling, retrieval and recommendation are distinct processes.
How can genuine GPTBot traffic be distinguished from impersonation?
The GPTBot user-agent string alone is insufficient to prove that a request originates from OpenAI.
Operators should verify source IP addresses against OpenAI's published crawler IP information, examine request behaviour and correlate security events with access logs.
In this investigation, requests claiming to be GPTBot attempted to access sensitive configuration files, indicating malicious scanning rather than ordinary content discovery.
What caused the HTTP 429 errors in this case?
The hosting infrastructure provider identified an Imunify360 Web Application Firewall rule limiting GPTBot requests to 30 requests within a 10-second window per domain-specific GPTBot session.
Once the threshold was exceeded, requests were temporarily rejected with HTTP 429 responses. The hosting provider reported that it whitelisted the responsible rule for the affected hosting account.
Does this mean shared hosting is unsuitable for AI Visibility?
No. In this case, the infrastructure provider found no evidence of hosting resource limits being reached. The hosting provider attributed the issue to a specific security rule.
Properly selected and technically supervised shared hosting can be entirely appropriate for many business websites. VPS hosting may be preferable where workload, configuration or application requirements justify it.
The important factors are appropriate configuration, effective security, performance and reliable monitoring.
How can businesses measure AI crawler retrieval reliability?
Measuring AI Visibility requires more than checking robots.txt, schema validation or search-engine indexing.
Sydney Business Web uses AI Observatory to measure crawler activity and retrieval outcomes, supported by Cloudflare analytics and origin-server logs where deeper investigation is required.
Initial post-correction monitoring showed zero GPTBot-labelled HTTP 429 responses in a 24-hour Cloudflare snapshot. Further monitoring is needed to assess performance under renewed, verified GPTBot business retrieval activity.
Examine the Evidence. Explore the Engineering.
This investigation draws on production retrieval telemetry, Cloudflare analytics, Apache HTTPS access logs and the hosting infrastructure provider's technical diagnosis. The resources below provide supporting methodology, related investigations and independent technical documentation.
Related AI Visibility Resources
Production monitoring of recognised AI and search crawler access, business retrieval outcomes and machine discovery.
sydneybusinessweb.com.auA previous investigation of ClaudeBot retrieval failures caused by an origin-side security restriction.
sydneybusinessweb.com.auHow connected entities, identifiers and relationships contribute to a coherent machine-readable business identity.
sydneybusinessweb.com.auThe role of independently supported business information in strengthening identity and credibility.
sydneybusinessweb.com.auAnalyse machine-readable business identity across a complete website rather than validating isolated schema fragments.
sydneybusinessweb.com.auInspect the Methodology
Public architecture, evidence qualification, retrieval methodology and measurement boundaries of AI Observatory.
github.com/Sydney-Business-Web/ai-observatoryWhole-site entity graph methodology, identity analysis and structured-data verification documentation.
github.com/Sydney-Business-Web/schema-gorillaStandards, Crawlers & Security
Formal definition of HTTP 429, rate limiting and the optional Retry-After response header.
rfc-editor.orgOfficial documentation of GPTBot, OAI-SearchBot and ChatGPT-User, including crawler identification and published IP information.
developers.openai.comOfficial documentation covering AI bot classification, crawler rate limiting and HTTP 429 responses.
docs.imunify360.comExplanation of rate-limit responses, HTTP 429 behaviour and Cloudflare's related diagnostic information.
developers.cloudflare.comThe HTTP request counts, retrieval measurements and incident findings in this article originate from Sydney Business Web's own monitoring and access-log analysis, supplemented by the hosting provider's technical investigation. External documentation is supplied to explain the technologies and standards involved; it does not independently verify the individual production measurements.