Technical Solution 17: How ModSecurity and Hosting Security Can Block AI Crawler Access
A Real-World AI Retrieval Case Study When Security Rules Blocked Access
Every day, we solve problems for eCommerce website owners. We had a think about how we might use this activity to help others, and came up with this idea: Every week or three, we'll take the trickiest problem and publish our solution.
Important! - Some of these solututions involve adding code to your website (WordPress and Woocommerce mostly), so please ALWAYS be careful. We are not in any way responsible, directly or indirectly for any impact or consequences our code or advice has on your website, nor are we liable for any damage arising from such use.
Always back up your website before changing or adding code and/or editing the database, This is critically important!!!
ModSecurity and other hosting security layers can affect AI crawler access even when a website appears healthy to human visitors. AI Observatory measures the actual retrieval outcomes.
The Website Worked Normally— But AI Crawlers Were Receiving 403 Errors
For this case study, we will call the affected shared-hosting site “A Website”. To its owner and normal visitors, the website appeared to be working. But AI Observatory was measuring something that ordinary browsing could not show: recognised AI crawlers were sometimes being refused access.
The problem was not theoretical. During the investigation, AI Observatory recorded genuine autonomous crawler requests reaching the hosting environment and receiving HTTP responses from the server. Some succeeded normally. Others returned 403 Forbidden.
A website being online does not prove that AI crawlers can reliably retrieve it.
Earlier in the investigation, the measured business-information retrieval success rate for A Website had fallen to approximately 46%. After hosting-security changes were made, the measured rate recovered through 66.7%, 77.2%, 85%, 86.4% and eventually 93.3%.
That recovery was important because it demonstrated that AI retrieval could be affected by the hosting environment even though the website remained available to ordinary visitors. It also showed why crawler accessibility needs to be measured rather than assumed.
The investigation later produced another warning. Legitimate OpenAI crawler traffic began showing intermittent 403 responses again. GPTBot and OAI-SearchBot could succeed at one time and be denied at another, even while other recognised crawlers continued retrieving the website successfully.
The Website Looked Healthy
Human browsing continued to work. A business owner checking the website in a browser could reasonably conclude that there was no access problem at all.
The Failures Were Measurable
AI Observatory recorded actual crawler requests and HTTP outcomes. Retrieval success was measured from production traffic rather than inferred from robots.txt, crawler permissions or a successful browser visit.
Security Can Act Selectively
Shared-hosting security can allow one request and deny another without taking the website offline. ModSecurity, reputation controls, bot filtering, rate controls and other hosting-layer systems can all influence the result.
A 403 Does Not Identify the Culprit
HTTP 403 tells us that access was refused. By itself, it does not prove whether the decision came from ModSecurity, Imunify360, LiteSpeed, Wordfence or another security layer. That requires further evidence.
The next step was therefore not to ask whether the website was up. It was to determine which layer of the hosting stack was deciding whether legitimate AI crawlers were allowed through— and to compare those decisions with the retrieval evidence recorded by AI Observatory.
AI Observatory Exposed a Failure That Human Browsing Could Not See
The first useful number was not page speed, uptime or a successful browser visit. It was the proportion of qualifying AI and search crawler requests that could actually retrieve the website’s current public business information.
At one stage, A Website was recording only about 46% successful retrieval. In other words, more than half of the qualifying retrieval attempts being measured were failing even though the website itself remained available to normal visitors.
Rolling measurements from AI Observatory during the hosting investigation. Each percentage represents the observed state at that point in the recovery, not a laboratory benchmark.
Security Changes Coincided With a Major Recovery
During the investigation, a hosting-side ModSecurity adjustment was made for the website. Retrieval performance subsequently improved markedly. A separate hosting problem involving CloudLinux was later acknowledged and corrected by the provider, after which the measured recovery continued.
That distinction matters. The measurements demonstrated that hosting infrastructure and security controls were materially relevant to crawler retrieval, but they do not justify assigning every failure to one product or one rule.
Then OpenAI Crawlers Began Showing Intermittent 403 Responses
Later evidence revealed a more specific pattern. Verified OpenAI crawler requests were reaching the hosting environment, but some were being answered with HTTP 403 Forbidden. The failures were intermittent: the same crawler families could be accepted at one time and rejected at another.
OAI-SearchBot requested /robots.txt from one verified OpenAI source address and received 403 — 787 bytes.
One second later, GPTBot requested / from a different verified OpenAI source address and also received 403 — 787 bytes.
OAI-SearchBot requested /robots.txt and received 403 — 5015 bytes.
At the same second, GPTBot requested / from another verified OpenAI address and received 403 — 5015 bytes.
Those paired events were significant. Different OpenAI crawler systems, using different source addresses and requesting different resources, were denied at almost exactly the same time and received matching response sizes. That pattern is difficult to explain as a simple missing page or an isolated crawler mistake.
The Block Was Not Permanent
At 22:01 UTC the pattern changed again. OAI-SearchBot successfully retrieved /robots.txt with a 200 response, while GPTBot requested the website root and received a normal 301 redirect.
Earlier records also contained successful GPTBot retrievals. The evidence therefore did not support a simple permanent ban on GPTBot or OAI-SearchBot. The restriction was intermittent.
AI Observatory changed the question from “Is the website online?” to “When a legitimate AI crawler asks for public business information, what does the hosting stack actually return?”
A 403 Can Be Generated Before WordPress Serves the Page
On shared hosting, a request does not simply travel from an AI crawler directly to WordPress. It may pass through several web-server and security layers first, and any one of those layers can potentially refuse the request before the website application has a chance to respond.
This is why an HTTP 403 Forbidden response is evidence of refusal, but not evidence of which product made the decision. ModSecurity may be involved, but so may provider-level filtering, Imunify360, LiteSpeed or Apache controls, rate limiting, reputation systems or application-level security such as Wordfence.
“403 Forbidden” tells us what happened to the request. It does not, by itself, tell us which security layer caused it.
Simplified request path for explanatory purposes. The exact order and products vary between hosting environments.
ModSecurity
ModSecurity is a web application firewall engine that evaluates HTTP requests against security rules. A rule can reject a request before WordPress handles it, returning a 403 even though the website itself remains fully operational for other traffic.
Imunify360 and Provider Security
Shared hosts may operate additional reputation, malware, firewall and behavioural controls outside the customer’s WordPress installation. These systems can make decisions using information or logs that are not visible inside the hosting account.
LiteSpeed or Apache Controls
The web-server layer can enforce access rules, throttling, connection limits or other restrictions. A request can therefore be rejected before the PHP application or a WordPress plugin records anything about it.
WordPress and Wordfence
Application-level security can also block requests, but it operates later in the chain. If the refusal occurs upstream, WordPress and Wordfence may have no record of the denied request at all.
That last distinction became important in the investigation of A Website. The OpenAI requests were visible in the raw hosting access log, and the server returned 403 responses, but the corresponding source addresses did not appear in the Wordfence evidence we examined.
We checked Wordfence's own logs and database records, including its hit history, blocked-IP records and active block table. The OpenAI addresses involved in the paired 403 events were not present.
The account-visible WordPress and PHP error logs were also silent for those requests, while the hosting account did not expose the underlying ModSecurity or Imunify security-event logs.
That shifted the investigation upstream. The evidence increasingly pointed away from WordPress and toward a hosting or server-security decision occurring before the application layer.
The Same Legitimate AI Crawlers Were Allowed, Blocked and Allowed Again
The raw hosting access log provided the clearest evidence. GPTBot and OAI-SearchBot were not simply absent. Their requests reached the server, were recorded, and received different HTTP outcomes at different times.
The crawler identities were corroborated against published provider network information. These were not random scripts merely claiming an OpenAI user-agent.
The number following each 403 above is the response size recorded by the server. That detail became unexpectedly useful. At 17:29, two different OpenAI source addresses requesting two different resources received the same 787-byte 403 response. At 18:21, another pair received matching responses of approximately 5015 bytes.
Verified OpenAI Traffic
The requests were not accepted merely because the user-agent contained GPTBot or OAI-SearchBot. The observed addresses were corroborated against published provider network information before entering the Observatory evidence.
The Block Was Intermittent
The same crawler families produced successful responses, 403 failures and normal redirects at different times. That does not resemble a simple permanent robots.txt or user-agent ban.
The Failures Were Synchronized
Two separate OpenAI crawler requests failed within one second at 17:29, and another two failed at exactly the same second at 18:21. Different source addresses were affected together.
Two Distinct 403 Sizes Appeared
Failed responses clustered around two body sizes: 787 bytes and roughly 5 KB. That suggests different denial templates or handling paths may have been involved, but the response size alone cannot identify the security product responsible.
The evidence did not show an AI crawler that simply “could not reach the website”. It showed legitimate crawler requests reaching the hosting environment and being treated differently at different moments.
This distinction matters for diagnosis. A crawler configuration problem tends to be repeatable. A missing resource tends to remain missing. A permanent block tends to remain a block.
What we observed instead was state-dependent access: verified OpenAI traffic succeeded, was denied, and later succeeded again. That behaviour directed the investigation toward security and hosting controls capable of making request-by-request decisions.
The remaining question was therefore very specific: which layer was producing those 403 responses?
WordPress Was Not Showing the Blocks— So We Kept Moving Up the Stack
Once the raw access log had confirmed genuine 403 responses, the next task was to identify which layer produced them. We began with the security system we could inspect directly: Wordfence.
The affected OpenAI source addresses were checked against Wordfence's available logs and database records. We searched the plugin's event files, hit history, blocked-IP records and active block table.
The failed OpenAI addresses did not appear in those records. We also searched the account-visible WordPress and PHP error logs for the affected addresses, ModSecurity references, access-denied messages and 403 events. Those searches were also clean.
Absence from an application log does not prove that no block occurred. It may mean the request was rejected before the application ever saw it.
What the Evidence Established
- The OpenAI requests reached the shared-hosting environment.
- The web server recorded genuine HTTP 403 responses.
- The crawler source addresses were corroborated as OpenAI infrastructure.
- The failures were intermittent rather than permanent.
- Wordfence showed no matching block records for the affected addresses.
- The account-visible PHP and WordPress error logs showed no corresponding denial events.
What We Could Not Yet Establish
- A 403 response alone does not prove that ModSecurity generated it.
- The account did not expose the provider's ModSecurity audit log.
- The relevant Imunify360 security-event records were not available to the customer account.
- We could not independently inspect every LiteSpeed or provider-level decision.
- The two different 403 response sizes suggest differing handling paths, but do not identify them.
- Only the hosting provider could inspect the upstream security logs required to attribute the decision conclusively.
This is a common difficulty with shared hosting. The business owns the website, but does not necessarily control or even see every layer through which an incoming request passes.
Application Evidence
WordPress, Wordfence and PHP logs are visible to the website operator. In this investigation they contained no corresponding records showing Wordfence blocking the affected OpenAI source addresses.
Web-Server Evidence
The raw hosting access log showed the requests themselves and the HTTP outcomes returned by the server. This established that the 403 responses were real rather than an Observatory interpretation or browser error.
Provider Security Evidence
ModSecurity, Imunify360 and other infrastructure-level security logs may be accessible only to the hosting provider. That final attribution therefore required escalation beyond the website account.
The hosting provider was supplied with exact UTC timestamps, source IP addresses, crawler identities, requested paths, status codes and response sizes. Two paired failure windows were highlighted: 17:29 UTC and 18:21 UTC on 19 September 2026.
The request was deliberately specific. Rather than asking support whether “AI bots are blocked”, we asked them to identify which server or security layer generated those exact 403 responses, and which rule or security decision was responsible.
At this stage, the evidence strongly supported a hosting-layer security decision upstream of WordPress. It did not yet justify claiming that ModSecurity itself caused the current OpenAI failures.
What Should You Do When an AI Crawler Receives a 403?
Do not begin by disabling ModSecurity, turning off Wordfence or asking the hosting company to whitelist every AI-related address. First establish whether the crawler is genuine, what it requested and which layer of the hosting stack actually refused it.
-
Confirm That the Resource Really Is Public
Check the exact URL in a private browser session. Confirm that the resource is current, publicly accessible and does not require login, cookies, a session or some other human interaction. This establishes that the intended resource itself is available—but it does not prove that an AI crawler can retrieve it.
-
Measure the Crawler Request, Not the Browser
Human browsing and crawler retrieval are different observations. Examine real server or edge evidence showing the crawler request, requested path, method and HTTP response. AI Observatory performs this continuously for qualifying machine traffic rather than assuming that browser availability means crawler availability.
-
Record Exact Times and HTTP Outcomes
Preserve the UTC timestamp, crawler identity, source address, requested resource, response status and—where available—response size. A precise timestamp turns a vague complaint such as “GPTBot was blocked” into an event that a hosting provider can actually locate in its security logs.
-
Verify the Crawler Identity
Do not trust a crawler name in the user-agent alone. Corroborate the source using the provider's published network information or another appropriate verification method. Security should not be weakened for an unidentified script simply because it calls itself GPTBot, Googlebot or another recognised crawler.
-
Work Through the Security Layers
Check the evidence available at each layer: WordPress plugins such as Wordfence, PHP and application logs, the web-server access log, and any exposed ModSecurity or hosting-security records. If the request appears in the server log but not in WordPress security records, the denial may have occurred upstream of the application.
-
Do Not Assume Every 403 Is ModSecurity
ModSecurity is capable of returning a 403, but so are other systems. Shared hosting may include Imunify360, LiteSpeed or Apache controls, reputation filtering, bot protection, connection limits and other provider-managed mechanisms. Attribute the block only when the evidence identifies the responsible rule or layer.
-
Escalate With Evidence, Not a General Complaint
If the required security logs are controlled by the hosting provider, give support the exact UTC time, source IP, user-agent, URL, status code and any other distinguishing evidence. Ask which layer generated the response and, where applicable, for the exact security rule or rule ID.
-
Make the Narrowest Safe Correction
If a legitimate crawler is being incorrectly blocked, correct the specific rule or create a tightly scoped exception. Do not disable the entire WAF, permit a whole cloud network or broadly whitelist traffic merely to improve an AI crawler success rate.
-
Measure Again After the Change
A configuration change is not evidence that the problem is fixed. Continue measuring real crawler retrieval. Confirm that legitimate requests now succeed without weakening protection against unrelated automated traffic.
The Investigation Usually Leads to One of Three Places
The Application Blocked It
WordPress or an application-level security product recorded the denial. The website operator can usually inspect the relevant rule or event and determine whether a tightly scoped correction is justified.
The Hosting Layer Blocked It
The request reached the server but was refused before WordPress handled it. ModSecurity, Imunify360, web-server controls or another provider security mechanism may be responsible. Server-side evidence is needed to identify which one.
The Evidence Is Incomplete
A 403 is visible, but the customer account does not expose the log that explains it. At that point the correct action is provider escalation, not guesswork or indiscriminate security changes.
Do not switch off ModSecurity merely because an AI crawler received a 403. First prove that ModSecurity generated the 403—and identify the specific rule that needs attention.
The objective is not to create a website that accepts every automated request. It is to maintain normal security while ensuring that legitimate autonomous crawlers can reliably retrieve the public business information they are intended to access.
That requires measurement. A robots.txt file can state what crawlers are allowed to request. A security configuration can state what should be permitted. Only the retrieval evidence tells you what actually happened.
AI Crawler Access Is a Shared Responsibility
A website owner can make public information available and permit legitimate crawlers. But on shared hosting, the owner does not necessarily control every security decision made before a request reaches WordPress.
Equally, a hosting provider should not be expected to remove security simply because a request claims to be an AI crawler. The crawler itself must present enough trustworthy identity information to be distinguished from ordinary automated traffic.
Reliable AI crawler access depends on three things working together: a correctly configured website, a hosting stack that does not unintentionally reject legitimate traffic, and an AI crawler whose identity can be verified.
The Website Owner
- Keep important public business information genuinely public.
- Maintain sensible robots.txt and crawler permissions.
- Do not disable security merely to improve crawler statistics.
- Measure real retrieval rather than assuming browser access is sufficient.
- Preserve exact timestamps and technical evidence when failures occur.
- Escalate unexplained hosting-layer failures with specific evidence.
The Hosting Provider
- Operate security controls without unnecessarily denying legitimate public retrieval.
- Investigate ModSecurity, Imunify360, LiteSpeed and other server-level events when evidence warrants it.
- Identify the actual rule or mechanism responsible for a 403 response.
- Create narrowly scoped corrections rather than weakening wider protection.
- Provide meaningful technical answers when the relevant logs are unavailable to the customer.
- Recognise that crawler accessibility can fail even while human browsing remains normal.
The AI Platform
- Use stable and clearly documented crawler identities.
- Publish the network information required for independent verification.
- Separate autonomous crawling, Search and user-directed retrieval where their purposes differ.
- Avoid forcing website owners to trust entire cloud networks or unverifiable traffic.
- Use normal, standards-compliant retrieval behaviour wherever practical.
- Make it possible for legitimate traffic to be distinguished from impersonation.
What Happened With A Website?
The website operator had already allowed legitimate crawler access and could demonstrate successful retrieval by recognised autonomous systems. Human visitors could use the site normally.
AI Observatory nevertheless recorded intermittent failures affecting verified OpenAI crawler traffic. Application-level checks did not identify Wordfence as the source of those failures, while the security logs needed for final attribution were controlled by the hosting provider.
The evidence was therefore escalated with exact timestamps, source addresses, crawler identities, requested paths, response codes and response sizes so that the provider could determine which upstream security layer generated the 403 responses.
What This Shared-Hosting Case Actually Proves
A production incident can reveal a real engineering problem without proving every possible explanation for it. The value of this case lies in separating what AI Observatory and the hosting evidence established from what still required provider-side investigation.
What the Evidence Establishes
- A Website remained usable by human visitors while qualifying crawler retrieval performance had fallen to approximately 46%.
- Following hosting and security remediation, measured retrieval performance recovered through 66.7%, 77.2%, 85%, 86.4% and 93.3%.
- A ModSecurity adjustment formed part of the earlier remediation, and the hosting provider later acknowledged and corrected a separate CloudLinux problem.
- Verified GPTBot and OAI-SearchBot requests later reached the hosting environment and received genuine HTTP 403 Forbidden responses.
- The OpenAI failures were intermittent: the same crawler families also received successful responses and normal redirects at other times.
- Separate OpenAI source addresses failed almost simultaneously, with paired responses showing matching 403 body sizes.
- The affected source addresses were not found in the Wordfence evidence examined, including its hit records, blocked-IP records and active block table.
- The account-visible WordPress and PHP logs contained no corresponding denial event for those requests.
What the Evidence Does Not Establish
- It does not yet prove that ModSecurity generated the later OpenAI 403 responses.
- It does not prove that Imunify360, LiteSpeed, Apache or any other specific provider-side component generated them either.
- It does not prove that every shared-hosting environment restricts AI crawlers in the same way.
- It does not mean that OpenAI crawlers are permanently blocked from A Website; the recorded behaviour was explicitly intermittent.
- It does not establish that every 403 received by an AI crawler is a hosting-security problem.
- It does not indicate that the website's content, schema or robots.txt caused these particular 403 responses.
- It does not justify disabling ModSecurity or broadly whitelisting AI, cloud-hosting or automated traffic.
- It does not prove why two distinct 403 response sizes appeared; they are useful forensic clues, not identification of a specific product.
The Defensible Conclusion
The evidence supports a narrower and more useful conclusion: legitimate AI crawler retrieval can be impaired by conditions in the hosting and security stack even while the website appears normal to human visitors.
In this production case, the later OpenAI 403 pattern increasingly pointed upstream of WordPress. Determining the exact component responsible required access to provider-side security logs that were not available inside the shared-hosting account.
The engineering lesson is not “turn off ModSecurity”. It is: measure retrieval, identify the failing layer, correct the specific conflict, and measure again.
If AI Crawler Access Matters, Measure What the Server Actually Returns
A successful browser visit does not prove that GPTBot, OAI-SearchBot or another legitimate machine system can reliably retrieve the same website. Shared-hosting security may make decisions that are invisible to the business owner and leave the human-facing website apparently untouched.
In this production case, AI Observatory exposed a retrieval problem that ordinary website checks did not reveal. The measured success rate fell to approximately 46%, later recovered as high as 93.3%, and subsequent monitoring exposed intermittent HTTP 403 responses affecting verified OpenAI crawler traffic.
ModSecurity was relevant during the earlier remediation, but the later OpenAI failures demonstrate why technical attribution matters. A 403 can originate from ModSecurity, another hosting-security layer, the web server or the application itself. The correct response is not to disable protection—it is to identify the layer responsible.
Good AI crawler access is not achieved by trusting every bot. It is achieved by verifying legitimate traffic, measuring real retrieval, correcting specific conflicts and confirming the result afterwards.
ModSecurity and AI Crawler Access: Common Questions
AI crawler failures can originate in several different parts of the hosting stack. These answers explain what a 403 means, where ModSecurity fits and why real retrieval evidence matters.
Can ModSecurity block AI crawlers such as GPTBot?
Yes. ModSecurity is capable of rejecting HTTP requests before they reach WordPress, including requests from legitimate AI crawlers. However, seeing a 403 Forbidden response does not by itself prove that ModSecurity caused it. Other hosting-security layers can return the same status.
Why can an AI crawler receive a 403 when my website works normally?
Human visitors and autonomous crawlers do not necessarily encounter identical security decisions. A hosting stack may evaluate source networks, request patterns, reputation signals, rate limits or crawler identity before allowing a request through. The website can therefore appear completely healthy in a browser while some machine requests are being refused.
Does a 403 error prove that ModSecurity blocked the request?
No. A 403 proves that the request was refused. On shared hosting, the response could potentially originate from ModSecurity, Imunify360, LiteSpeed or Apache controls, provider-level filtering, Wordfence or another security mechanism. The responsible layer should be identified from logs or provider-side evidence before changes are made.
Can Wordfence block GPTBot or OAI-SearchBot?
Yes, Wordfence can block automated traffic at the WordPress application layer. In the production case described on this page, however, the affected OpenAI source addresses were not found in the Wordfence logs, hit history, blocked-IP records or active block table that we examined. That evidence pushed the investigation further upstream.
Should I disable ModSecurity to improve AI crawler access?
Not simply because an AI crawler received a 403. ModSecurity provides an important security function. First determine whether ModSecurity actually generated the failed response and, if possible, identify the specific rule involved. A narrow correction is preferable to disabling an entire security layer.
How can I prove that an AI crawler is really reaching my website?
Use network, server or edge evidence showing the actual request. Record the crawler identity, source address, requested path, HTTP method, timestamp and response status. Where possible, corroborate the crawler identity against published provider network information rather than trusting the user-agent string alone.
Why is server-log evidence better than checking the website in a browser?
A browser test shows that one human-style request succeeded. It does not show what happened when GPTBot, OAI-SearchBot, Googlebot or another machine system requested the website. Server evidence records the actual machine request and the actual HTTP outcome returned to it.
What does AI Observatory measure?
AI Observatory records qualifying search and AI crawler retrieval activity against a production website and measures the HTTP outcomes of those requests. It distinguishes successful retrieval from failures and redirects using defined evidence rules rather than assuming that a crawler was successful merely because it appeared in a log.
Does AI Observatory prove that an AI system understood or used my content?
No. Retrieval evidence establishes that a qualifying machine system requested a resource and records what the website returned. It does not prove that the content was understood, indexed, cited, recommended, reused in an answer or used for model training.
What should I give my hosting provider when an AI crawler is being blocked?
Provide the exact UTC timestamp, source IP address, crawler user-agent, requested URL or path, HTTP response code and any useful response-size evidence. Ask the provider to identify which security layer generated the response and, where relevant, the specific rule or rule ID involved.
References and Supporting Material
Official technical documentation supports the discussion of web application firewalls, crawler verification and security-layer behaviour. The measured figures and HTTP events described in this case come from Sydney Business Web's AI Observatory, raw production access logs and the hosting investigation conducted on A Website.
Web Application and Hosting Security
- OWASP ModSecurity Open-source web application firewall engine used to inspect and act on HTTP traffic.
- OWASP Core Rule Set Documentation Rule-based web application firewall protection, anomaly detection, tuning and false-positive management.
- Wordfence Firewall Documentation Application-level WordPress firewall behaviour and blocking.
- Shared-Hosting Provider Security Logs ModSecurity, Imunify360, web-server and other provider-managed security logs may exist outside the customer's WordPress account and require hosting-provider investigation.
AI Crawler Identity and Verification
- OpenAI Crawlers Published information about GPTBot, OAI-SearchBot, ChatGPT-User and crawler identification.
- Cloudflare Verified Bots Verification principles for distinguishing recognised automated systems from self-declared bot traffic.
- Google Crawler Verification Published crawler ranges and verification procedures illustrate why a user-agent string alone is not sufficient proof of identity.
Sydney Business Web Evidence
- AI Observatory — Verified AI Retrieval Monitoring Overview of Sydney Business Web's production crawler-retrieval measurement system.
- Verified AI Retrieval Evidence Live public evidence showing qualifying search and AI crawler retrieval activity and measured outcomes.
- AI Crawler Monitoring at the Cloudflare Edge Technical background to crawler observation, corroboration and retrieval monitoring.
- A Website — Production AI Observatory Evidence Rolling retrieval measurements recorded during the hosting investigation, including approximately 46%, 66.7%, 77.2%, 85%, 86.4% and 93.3% observed retrieval success states.
- Raw Shared-Hosting Access Logs Production server records used to correlate GPTBot and OAI-SearchBot requests with HTTP 200, 301 and 403 outcomes, exact UTC timestamps and response sizes.
- Wordfence and Application-Level Checks Investigation included Wordfence event files, hit records, blocked-IP records, active block records and account-visible PHP and WordPress error logs.
Evidence boundary — 20 September 2026: the production evidence establishes intermittent 403 responses affecting verified OpenAI crawler traffic and supports investigation of an upstream hosting-security decision. A ModSecurity adjustment was relevant during the earlier remediation of A Website, but at the time of publication the specific provider-side mechanism responsible for the later OpenAI 403 events had not yet been conclusively identified. The hosting provider was supplied with exact timestamps, source addresses, requested paths, response codes and response sizes for investigation.