Technical Solution 20: Are AI Bots Slowing Down Websites? A Real Case Study

A Real-World Case: Our VPS investigation reveals crawler workloads, CPU saturation and practical performance improvements.


Every day, we solve problems for eCommerce website owners. We had a think about how we might use this activity to help others, and came up with this idea: Every week or three, we'll take the trickiest problem and publish our solution. 

AI crawler activity and server performance illustrated through the AI Observatory dashboard and WordPress infrastructure.

Managing AI crawler demand without sacrificing website performance or valuable AI visibility — a real-world Sydney Business Web engineering investigation.

Executive Summary | A Three-Minute Read

AI Crawling, Website Performance and the New Hosting Challenge

AI bots can contribute to website performance problems, but they are not necessarily the only cause. Our investigation of a live production VPS revealed twelve hours of near-total CPU saturation, ten Apache restart cycles in a single day and concentrated website retrieval failures.

A substantial increase in corroborated AI crawler activity coincided with the initial escalation. However, the server remained saturated after that crawler's activity declined, revealing a second, intensive distributed retrieval workload whose identity could not be established.

What Happened?

CPU utilisation reached approximately 96–99%. One website recorded 170 HTTP 503 responses and four HTTP 500 responses within ten minutes.

What Improved Performance?

Full-page caching and more efficient Apache redirects. Selected repeat product-page responses became 17–30 times faster without additional server capacity or new AI crawler blocks.

What Should Businesses Check?

Whether their website provider actively monitors crawler activity, verifies caching effectiveness, investigates service failures and manages automated access selectively.

The investigation demonstrated substantial request-delivery improvements. Longer-term monitoring is required to establish the sustained reduction in overall server resource consumption.

The business lesson: choose a website provider that understands bot management, infrastructure engineering and AI discoverability — not simply hosting.

Important! - Some of these solutions involve adding code to your website (WordPress and Woocommerce mostly), so please ALWAYS be careful. We are not in any way responsible, directly or indirectly for any impact or consequences our code or advice has on your website, nor are we liable for any damage arising from such use.

Always back up your website before changing or adding code and/or editing the database. This is critically important!!!
Technical Solution 20 | Section 01 | The Emerging Challenge

The Changing Infrastructure Cost of AI Discovery

Are AI bots slowing down websites? As AI systems become increasingly active consumers of online information, website providers must consider not only machine discoverability, but also the infrastructure cost of delivering it.

A website does not need to be under attack to experience serious resource exhaustion. Legitimate automated retrieval can introduce additional processing demands without any corresponding increase in human visitors or paying customers.

During November 2026, Sydney Business Web conducted a detailed forensic investigation of a managed six-core VPS hosting multiple WordPress and WooCommerce websites. Historical server measurements revealed a sharp departure from its established operating baseline, followed by prolonged CPU saturation and website service failures.

Building on our earlier investigation into AI crawling and large WooCommerce catalogues , we examined archived access logs, crawler identities, request distribution, HTTP response codes and application-level caching.

The investigation extended beyond identifying traffic sources. We implemented and tested practical engineering improvements designed to reduce unnecessary processing while maintaining access for valuable search and AI discovery systems.

Automated traffic does not have to be malicious to become an infrastructure problem. The emerging responsibility for website providers is to determine which machine consumers warrant access, what that access costs and how efficiently it can be delivered.

01. A New Infrastructure Consideration

AI systems introduce additional demand for publicly accessible information. Large product catalogues, dynamically generated content and uncached pages can make automated retrieval computationally expensive. Website engineering must increasingly account for machine consumers alongside conventional visitors.

02. Visibility Requires Observability

Our AI Observatory measures recognised AI crawler activity and retrieval outcomes. Combined with server diagnostics, these measurements can support informed decisions about crawler access, infrastructure efficiency and the commercial objectives of AI visibility.

Our objective is not to eliminate automated discovery. It is to make that discovery sustainable, measurable and commercially justified.

Technical Solution 20 | Section 02 | Historical Evidence

Establishing the Server's Historical Baseline

Before investigating individual crawlers, we needed to establish whether the server's workload had genuinely changed, or whether we were simply observing normal variations in website traffic.

We examined historical Linux system activity records retained on the VPS, using the sar utility to extract daily and hourly CPU utilisation figures. These records extended back to 22 October 2026, providing a useful operating baseline before the November incident.

Infrastructure Managed VPS
Processing 6 vCPUs
Memory 8 GB RAM
Applications WordPress / Woo

Between 22 October and 3 November, daily CPU utilisation remained relatively stable, averaging approximately 29%. On 4 November, a substantial departure from that baseline became apparent. By 5 November, average daily utilisation had risen to 88.4%, with extended periods of near-total CPU saturation.

Historical VPS CPU Utilisation
Daily CPU busy percentage | 22 October – 20 November 2026 | Six-core production VPS
Historical daily CPU utilisation, October to November 2026 CPU utilisation remained between approximately 25 and 33 percent from 22 October to 3 November. It increased to 44 percent on 4 November and peaked at a daily average of 88.4 percent on 5 November. Subsequent daily utilisation declined but remained variable. The final measurement on 20 November is a partial day. 100% 75% 50% 25% 0% 4 Nov: workload escalation 88.4% 5 November Pre-incident operating range 22 Oct 27 Oct 1 Nov 5 Nov 10 Nov 15 Nov 20 Nov*
Daily CPU busy Historical baseline Partial day
Figure 1. Actual historical CPU measurements extracted from Linux sysstat records during the investigation. CPU busy was calculated as 100% minus reported CPU idle. This aggregate measure does not attribute CPU consumption to individual applications or crawlers. *20 November represents an incomplete day. All measurements precede the page-cache changes implemented on 21 November.

Pinpointing the Turning Point

Daily averages established that a significant change had occurred, but did not tell us precisely when it began. We therefore examined hourly CPU measurements for 3, 4 and 5 November.

The decisive change appeared at approximately 16:00 UTC on 4 November 2026. Average hourly CPU utilisation increased from 37% to 59% and remained elevated throughout the remainder of the day.

From 08:00 to 19:59 UTC on 5 November, hourly CPU utilisation remained between approximately 96% and 99%.

This was not simply a momentary burst of activity. The server was operating with very little spare CPU capacity for an extended period, leaving limited headroom for additional application requests and genuine website visitors.

Critical Incident Timeline

Date and hour (UTC) CPU busy
4 Nov — 15:00 37.0%
4 Nov — 16:00 59.0%
4 Nov — 20:00 71.7%
4 Nov — 23:00 74.8%
5 Nov — 05:00 87.9%
5 Nov — 08:00 96.9%
5 Nov — 10:00 99.1%
5 Nov — 19:00 98.8%
5 Nov — 20:00 80.3%

Each value represents average CPU utilisation for the indicated hour. UTC timestamps were retained throughout the investigation to permit direct comparison with archived Apache access logs.

We now had something considerably more useful than a general complaint about slow websites: a measurable departure from an established operating baseline, with a specific time window in which to investigate the underlying workload.

Technical Solution 20 | Section 03 | Crawler Identification

Identifying the AI Crawler Surge

Having established when the server's operating conditions changed, we examined archived Apache access logs to determine whether a corresponding change in automated traffic had occurred.

The archived records covered four production websites, with several million recorded HTTP requests. Rather than examining only current traffic, we could reconstruct the workload around the precise time at which CPU utilisation began to escalate.

On Catalogue A, a large WooCommerce website, requests identifying themselves as GPTBot/1.4 increased from 266 to 1,729 between consecutive hours on 4 November 2026.

During the 16:00 UTC hour, 1,728 requests originated from one source IP address and one request from a second address. Both addresses matched OpenAI's published GPTBot network ranges when examined during the investigation.

During the same period, aggregate VPS CPU utilisation increased from 37% to 59%. GPTBot-labelled requests subsequently remained elevated for several hours.

This was our first strong indication that legitimate AI crawling could be a contributing factor in the changing infrastructure workload.

GPTBot Activity During the Initial CPU Escalation

Requests per hour | One WooCommerce catalogue | 4 November 2026 (UTC)
15:00 266
16:00 1,729
17:00 1,754
18:00 1,963
19:00 1,810
Figure 2. Actual hourly GPTBot/1.4-labelled request counts extracted from archived Apache HTTPS access logs. Source addresses were separately examined during the critical onset period.

Establishing Crawler Identity

A user-agent string alone is insufficient evidence of crawler identity. Any automated client can present itself as a recognised search engine, AI crawler or conventional browser.

We therefore compared the source IP addresses responsible for the initial GPTBot surge against OpenAI's published GPTBot IP address ranges . Both observed addresses fell within the published network ranges consulted during the investigation, providing independent corroboration of their declared crawler identity.

This corroboration was based on source-IP range matching, not reverse DNS, forward-confirmed DNS or cryptographic authentication. Published crawler network ranges can change over time, and historical identity assessments should therefore retain the date and basis of corroboration.

This distinction is fundamental to the methodology behind Sydney Business Web's AI Observatory. Crawler classification should incorporate identity corroboration rather than relying exclusively on self-declared user agents.

A legitimate AI crawler can generate considerable retrieval activity without exhibiting malicious behaviour. The operational question is whether the website infrastructure can accommodate that activity efficiently.

Testing the Initial Hypothesis

It would have been tempting to conclude that GPTBot was responsible for the entire server incident. However, examining the following day's traffic revealed an important contradiction.

As GPTBot activity on this website declined, overall CPU utilisation continued to increase. By the late afternoon of 5 November, GPTBot requests to the examined catalogue had ceased, yet the VPS remained almost completely saturated.

Date and hour (UTC) GPTBot requests VPS CPU busy
4 Nov — 15:00 266 37.0%
4 Nov — 16:00 1,729 59.0%
4 Nov — 18:00 1,963 62.5%
5 Nov — 08:00 338 96.9%
5 Nov — 10:00 397 99.1%
5 Nov — 16:00 197 96.8%
5 Nov — 17:00 0 96.2%
5 Nov — 18:00 0 98.7%
5 Nov — 19:00 0 98.8%

GPTBot counts relate to the examined website, whereas CPU measurements relate to the entire VPS. Their temporal association does not establish direct attribution of CPU consumption.

The evidence supported an association between intensive AI crawling and the initial escalation of server workload. It did not support attributing the entire incident to a single AI crawler.

Another workload was clearly worth investigating. We extended our analysis to the other production websites sharing the same infrastructure. What we discovered was a very different form of automated traffic, involving thousands of source IP addresses and repeated retrieval of individual WooCommerce products.

Identifying a crawler is only the first step. Understanding its processing cost, its purpose and its relationship to other workloads is the real engineering challenge.

Technical Solution 20 | Section 04 | Distributed Traffic

Beyond AI Bots: Distributed Catalogue Crawling

The continuing CPU saturation after GPTBot's activity declined indicated that the investigation needed to extend beyond recognised AI crawlers. A second WooCommerce website revealed a substantially different automated traffic pattern.

During 5 November 2026, requests carrying two generic Linux Chrome browser user-agent strings increased dramatically. Unlike the earlier GPTBot workload, these requests originated from thousands of different IP addresses, with very few requests attributable to any individual address.

4,941 Browser-labelled requests during the 09:00 UTC hour on 5 November
3,193 Individual product-page requests within that same hour
5 Maximum requests from any single IP within either examined browser-version group at 17:00 UTC

A Second Workload Emerges

During comparable afternoon hours on 3 and 4 November, these browser-version identifiers accounted for relatively little traffic. By 5 November, they dominated requests to the examined website.

The increase became particularly pronounced at 08:00 UTC, coinciding with the transition into sustained CPU utilisation above 96%.

Distributed Traffic and VPS CPU Utilisation

5 November 2026 | Hourly observations (UTC) | Browser-labelled requests on one website compared with aggregate VPS CPU utilisation
Distributed browser-labelled traffic and VPS CPU utilisation on 5 November 2026 Combined Chrome 156 and Chrome 152 requests increased from 469 at 07:00 to 3,539 at 08:00 and 4,941 at 09:00. CPU utilisation remained between approximately 96 and 99 percent until 19:59. At 20:00 browser-labelled requests declined to 70 and CPU utilisation fell to 80.3 percent. 5,000 3,750 2,500 1,250 0 100% 75% 50% 25% 0% 4,941 requests Traffic declines 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00
Chrome-labelled requests (left axis) VPS CPU busy (right axis)
Figure 3. Combined hourly request counts matching the examined Chrome 156 and Chrome 152 version identifiers, compared with aggregate VPS CPU utilisation. The series use independent vertical axes and represent different measurement scopes. Temporal alignment does not establish direct attribution of CPU consumption.

One particularly revealing observation occurred at 20:00 UTC. Combined requests matching the two browser-version identifiers fell from 2,719 in the preceding hour to just 70. Aggregate VPS CPU utilisation also declined, from 98.8% to 80.3%.

This was further evidence of an association between automated retrieval workloads and changing server pressure, although substantial CPU demand persisted after the traffic reduction.

How Distributed Was the Traffic?

We examined source-IP distribution during the 17:00 UTC hour, when GPTBot traffic to the first catalogue had already ceased.

Observed metric Chrome 156 Chrome 152
Total requests 2,664 593
Distinct source IPs 2,484 584
IPs making exactly one request 2,331 575
Maximum requests per IP 5 2

Source addresses were counted separately within each browser-version group. The groups may contain overlapping addresses. User-agent version identifiers do not establish the software or organisation responsible for the requests.

Why IP-Based Rate Limiting May Be Insufficient

Conventional rate limiting often identifies excessive traffic by counting requests from individual source IP addresses.

In this instance, however, the aggregate request volume was substantial while individual source-address request counts remained extremely low.

No individual address within either examined group exceeded five requests during the hour. A conventional per-IP request threshold would therefore be unlikely to identify the aggregate pattern.

This suggests that effective traffic management may also require examination of requested endpoints, behavioural patterns, identity corroboration and total application workload.

Repeated Retrieval of Identical Products

The investigation also revealed repeated requests for the same WooCommerce product URLs from different source addresses within short periods.

One product received 15 requests from 12 different IP addresses between 17:39:52 and 17:42:56 UTC. Every request returned HTTP 200 and used the same product URL without query-string parameters.

Examining three consecutive clock-minute intervals within this period revealed that product-page retrieval accounted for almost all requests to the website.

Time (UTC) Requests Product requests Distinct product URLs Distinct IPs
17:40 77 74 29 61
17:41 88 87 27 69
17:42 67 65 26 55
Combined 232 226 — —

Distinct product paths and source addresses were counted independently for each minute and cannot simply be added to establish three-minute unique totals.

When identical public product pages are repeatedly retrieved from different source addresses, effective full-page caching may provide a more productive first response than attempting to block every individual requester.

What We Could and Could Not Establish

The observed behaviour was consistent with distributed automated catalogue traversal. However, the access records did not establish whether the source addresses belonged to a coordinated network, a commercial scraping service, browser automation infrastructure or another form of distributed retrieval.

Nor did the generic browser user-agent strings establish any connection to an AI provider. Classifying this workload as verified AI crawling would therefore have been incorrect.

The important engineering finding was independent of the requesters' identities: repeated retrieval of public product pages presented a potentially significant application workload, particularly if those pages were being regenerated through WordPress rather than served from cache.

The investigation had now identified two distinct retrieval patterns. The next question was whether the increased workload had translated into measurable failures of website infrastructure and service availability.

Technical Solution 20 | Section 05 | Infrastructure Reliability

From CPU Saturation to Service Failure

High CPU utilisation is not necessarily a service failure. A server can operate efficiently under substantial demand. The situation becomes materially different when that demand is accompanied by failed website requests, service-monitoring timeouts and repeated web-server restarts.

Our investigation uncovered evidence of all three. By combining archived Apache access logs, Linux system activity records, systemd journal entries and cPanel service-monitoring logs, we could reconstruct the operational consequences of the November incident.

174 HTTP 500 and 503 responses recorded on one website within ten minutes
10 Apache restart cycles recorded during 5 November 2026
15 Apache restart cycles recorded between 4 and 11 November inclusive

Product-Page Retrieval Begins to Fail

During the 09:00 UTC hour on 5 November, the two browser-version groups examined in Section 04 generated 3,193 requests for individual WooCommerce product pages.

Of those requests, 162 received HTTP 503 Service Unavailable responses. None of these 503 responses occurred on the JavaScript assets retrieved by the same groups.

Examining the failures at five-minute resolution revealed that they were concentrated within relatively short periods rather than evenly distributed throughout the hour.

Period (UTC) Product requests HTTP 503 503 rate
09:15–09:19 110 6 5.5%
09:45–09:49 308 59 19.2%
09:50–09:54 384 97 25.3%
09:55–09:59 322 0 0%

Selected five-minute intervals on 5 November 2026. Request counts relate to product URLs retrieved by the two examined Chrome-version groups. The percentages represent recorded HTTP 503 responses divided by product requests within each interval.

During one five-minute interval, approximately one in four product requests from the examined automated traffic groups received HTTP 503 Service Unavailable. This was a measurable failure of content delivery, not merely an increase in response time.

Were the Failures Server-Wide?

Because the affected website shared its VPS with several other production applications, we examined whether comparable errors occurred on those websites during the same ten-minute interval.

Website Requests HTTP 503 HTTP 500
Catalogue A 554 0 0
Catalogue B 428 0 0
Catalogue C 1,034 170 4
Catalogue D 22 0 0
Total 2,038 170 4

All recorded HTTPS requests across four examined websites between 09:45 and 09:54 UTC on 5 November 2026. Unlike the preceding analysis, these figures include all recorded user agents. Website identities have been anonymised.

All 174 observed HTTP 500 and 503 responses occurred on Catalogue C, representing 16.8% of that website's recorded requests during the examined interval.

This suggested that the incident involved more than aggregate VPS resource consumption. Individual applications could experience different failure conditions while operating on the same heavily utilised infrastructure.

Apache Restart Cycles During the Incident

We subsequently examined the systemd journal for the Apache HTTPD service. The records revealed ten stop-and-start cycles on 5 November alone.

Every one occurred during the twelve-hour period in which average hourly CPU utilisation remained between approximately 96% and 99%. Four pairs of Apache restart events occurred approximately 11–16 minutes apart.

Apache Restart Event Timeline

5 November 2026 | UTC | Recorded systemd service-stop events during sustained CPU saturation
Ten Apache restart cycles on 5 November 2026 Apache restart events began at 09:18, 09:34, 12:21, 12:32, 15:56, 16:07, 18:02, 18:32, 19:05 and 19:21 UTC. All occurred during a period of sustained high CPU utilisation. Sustained CPU utilisation: approximately 96–99% 08:00 10:00 12:00 14:00 16:00 18:00 20:00 Each gold marker represents one recorded Apache restart cycle
Figure 4. Actual Apache service restart chronology extracted from the Linux systemd journal. Markers represent the beginning of recorded stop-and-start cycles, not measured outage durations. The shaded region identifies the twelve-hour period of sustained CPU saturation.

Repeated Recovery, Not Necessarily Crashing

The journal showed Apache being stopped, subsequently started and reported as running. The complete stop-to-start transitions generally took approximately 10–12 seconds on 5 November.

These were recorded service restart cycles. They did not, by themselves, establish that Apache had crashed or that every hosted website was unavailable throughout each transition.

We also examined cPanel's chkservd monitoring records. These contained repeated HTTPD service-check timeouts followed by explicit instructions to restart the web server.

That evidence established that automatic recovery was being invoked in response to failed service health checks during the investigated records.

Independent Evidence from Service Monitoring

The cPanel service-monitoring records contained the following diagnostic messages, presented here as selected extracts.

Source: cPanel chkservd service-monitoring logs
httpd [Service check failed to complete

Timeout while trying to get data from service

[socket failure threshold:3/3]

Restarting httpd....

[notify:failed service:httpd]

Repeated entries of this nature demonstrated that the service monitor was unable to obtain an expected response from HTTPD and was initiating recovery attempts.

Although the extracted monitoring records confirmed automatic restarts, they were not individually timestamp-matched against all fifteen systemd restart events. Consequently, we cannot assign every recorded restart to the same trigger.

Restarting a service can restore responsiveness, but it does not necessarily reduce the workload that made the service unresponsive. Recovery is not the same as remediation.

Interpreting the Failure Mechanism

The accumulated evidence pointed towards an infrastructure operating under sustained pressure, with individual application failures and recurring web-server recovery activity.

In a WordPress environment, CPU-intensive PHP processing, occupied PHP-FPM workers, database contention and Apache request-capacity limitations can interact to increase response times and reduce service availability.

However, historical aggregate CPU measurements and Apache access logs cannot independently establish which internal processing constraint generated each HTTP 503 response. Request-level application timing and contemporaneous worker telemetry would be required for that attribution.

Established by the Evidence

  • Sustained aggregate VPS CPU saturation.
  • Intensive, corroborated AI crawler activity coinciding with the initial escalation.
  • Substantial distributed product-page retrieval during continued saturation.
  • Concentrated HTTP 500 and 503 failures.
  • Repeated Apache restart cycles.
  • HTTPD health-check timeouts and automatic recovery attempts.

Not Yet Established

  • The exact CPU cost attributable to each crawler or traffic group.
  • The identity of the distributed browser-labelled requesters.
  • The precise internal cause of each HTTP 503 response.
  • The exact availability impact of every Apache restart.
  • Whether a single configuration change could eliminate the entire problem.

The Commercial Consequences

Automated retrieval competes for infrastructure capacity with the delivery of content to genuine customers. On shared hosting infrastructure, resource-intensive activity on one website may also affect the headroom available to other hosted applications.

Failed responses can interrupt machine discovery, while slow or unavailable websites may undermine customer confidence and commercial activity. The investigation did not quantify lost sales or lost AI visibility, but it established a credible operational risk to both.

For a website provider, the appropriate response is therefore not simply to increase server capacity or repeatedly restart affected services. It is to examine whether the incoming workload can be processed more efficiently and whether every category of automated access warrants the resources allocated to it.

Having established the infrastructure consequences, we turned to practical engineering remediation: eliminating unnecessary application processing, improving page caching and retaining selective crawler-access policies.

Technical Solution 20 | Section 06 | Engineering Remediation

Reducing the Infrastructure Cost of Automated Retrieval

Having identified the traffic patterns and their association with infrastructure pressure, we turned to a practical engineering question: could the existing server deliver the same content with substantially less processing?

Our initial objective was not to increase server capacity or introduce additional crawler restrictions. Instead, we examined whether unnecessary application processing could be eliminated from ordinary website requests.

The remediation work concentrated on two areas: full-page caching of publicly accessible content and moving suitable redirects out of WordPress and into the Apache web-server layer.

17–30× Faster tested repeat product-page responses after enabling caching on two websites
16–56 ms Measured response-time range across the selected optimised Apache redirects
0 Additional crawler blocks or VPS capacity upgrades required for these improvements

01. Identifying Unnecessary WordPress Processing

On two of the WooCommerce websites associated with the incident, we discovered that W3 Total Cache was installed but its full-page caching functionality was disabled.

Repeated requests for identical public product pages continued to take approximately 1.6–2.1 seconds to complete.

Examination of the W3 Total Cache configuration confirmed that page caching was not active. This provided a concrete opportunity to reduce repeated application processing without changing access policies for automated clients.

Why Full-Page Caching Matters

Ordinary WordPress page generation can involve PHP execution, database queries, plugin processing, product metadata retrieval and template rendering.

Full-page caching stores a generated version of a suitable page so that subsequent equivalent requests can reuse the resulting content.

This is particularly relevant to the distributed catalogue traffic described earlier. Numerous source IP addresses were requesting identical product URLs within minutes.

Subject to cache validity, session handling and other exclusions, the server should not need to regenerate the same public product page for every anonymous requester.

However, caching is less effective when crawlers retrieve thousands of previously uncached URLs. The initial GPTBot workload demonstrated why caching must be complemented by appropriate crawl management.

02. Choosing the Appropriate Caching Method

We selected Disk: Basic rather than Disk: Enhanced for the affected WooCommerce applications.

Disk: Enhanced can deliver cached pages directly through Apache rewrite rules, potentially avoiding PHP involvement altogether. This can provide substantial performance benefits, but may require additional attention to application-generated security headers and other request-handling behaviour.

Disk: Basic was selected as a conservative operational choice because of previous experience with security-header compatibility. Our subsequent measurements demonstrated that substantial performance improvements were achievable without adopting the more aggressive caching method.

03. Measured Product-Page Response Improvements

Following the configuration changes on 21 November 2026, we repeated the original product-page requests.

The first request still required approximately two seconds. Subsequent requests, however, demonstrated a dramatic reduction in measured response time, consistent with successful full-page cache reuse.

Before and After Page-Cache Activation

Measured HTTP response completion times | Identical public product URLs | Normal anonymous GET requests
Catalogue C — approximately 17 times faster
Before 1,916 ms
After 114 ms
Catalogue A — approximately 30 times faster
Before 1,603 ms
After 54 ms
Figure 5. Actual before-and-after HTTP response times measured using curl on 21 November 2026. Both comparisons use repeat requests to the same product URL. All requests shown returned HTTP 200. The bars share a common 2,000-millisecond scale. These are individual measurements, not statistical averages or direct CPU-consumption measurements.
Test condition Catalogue C Catalogue A
Before — first request 2.070 s 1.736 s
Before — repeat request 1.916 s 1.603 s
After — initial request 2.118 s 2.041 s
After — second request 0.114 s 0.068 s
After — third request — 0.054 s
Approximate repeat-response improvement 94.1% 96.7%

Percentage improvements compare the previous repeat response with the fastest recorded repeat response after activation. Tests measured HTTP request completion time, including connection overhead, rather than full browser rendering time.

The first request still incurred the cost of ordinary page generation. The substantial improvement appeared when subsequent requests reused the generated content. This is precisely the behaviour required to reduce the cost of repeated catalogue retrieval.

04. Protecting WooCommerce Transactional Behaviour

Full-page caching must not be enabled indiscriminately on an eCommerce website. Customer-specific information, cart contents, checkout sessions and account details require appropriate separation from publicly cached content.

Before activating page caching, we verified the actual WooCommerce page assignments and configured additional exclusions for transactional pages and customer-session cookies.

Essential Cache Exclusions

The implementation retained the existing application exclusions and added the verified Cart, Checkout and My Account URL patterns, including their subordinate endpoints.

WooCommerce cart and session cookie identifiers were also added to W3 Total Cache's rejected cookies configuration.

Caching remained disabled for authenticated users, and unrestricted query-string caching was not introduced.

These settings must be adapted to each website's actual configuration. Additional payment, personalisation or integration endpoints may require their own exclusions.

Following activation, we tested both the HTTP response headers and ordinary shopping behaviour using anonymous browser sessions.

Verified

Cart and Customer Sessions

Products could be added to the cart and the expected quantities and prices were displayed correctly.

Verified

Checkout and Payment Options

Checkout loaded with the expected cart details and payment options, including Afterpay on the applicable website.

Verified

Transactional Response Headers

Cart, Checkout and My Account returned private, no-cache response directives. Empty-cart checkout redirects also behaved as expected.

Verified

Public Product Retrieval

Anonymous product requests continued to return HTTP 200, with substantially faster responses on repeat retrieval.

Functional testing extended to checkout and payment option presentation. No live payment transaction was submitted. These checks established the tested shopping behaviour but did not constitute exhaustive testing of every possible customer-session pathway.

05. Eliminating Expensive Application-Level Redirects

A second optimisation opportunity involved redirects that were being generated through WordPress rather than handled directly by Apache.

Redirecting an incorrectly formatted product URL or a non-canonical hostname is normally a simple operation. There is often little justification for invoking an entire WordPress and WooCommerce application stack merely to return HTTP 301.

We implemented carefully scoped Apache rewrite rules for selected product URLs, product-category URLs and a canonical hostname redirect.

The changes were backed up before implementation and tested individually. Existing application routing and security rules were retained.

Redirect optimisation Before After
Product URL trailing slash ~1,450 ms ~20 ms
Product-category trailing slash 1,370–1,580 ms 16–24 ms
Canonical hostname redirect 1,412–2,296 ms 31–56 ms

Recorded HTTP 301 response times from selected redirect tests conducted on 20–21 November 2026. The canonical hostname implementation was also tested for preservation of the requested path and query string.

A redirect that previously required approximately one and a half seconds could be delivered in around twenty milliseconds. The underlying improvement was not a faster WordPress application, but the removal of unnecessary WordPress processing.

06. Can AI Crawlers Benefit from Page Caching?

To examine whether requests presenting an AI crawler identity could also benefit from the newly enabled page cache, we conducted a controlled user-agent simulation on Catalogue C.

Two successive GET requests were made using a GPTBot-style user-agent string. Both received HTTP 200 responses.

Simulated GPTBot Retrieval Performance

Same product URL | 21 November 2026 | Controlled user-agent simulation
Initial 2,043 ms
Repeat 79 ms
Figure 6. Controlled requests carrying a GPTBot user-agent string demonstrated an approximately 26-fold difference between initial and repeat response times. The bars share a common 2,100-millisecond scale. These were simulated requests from our test environment, not actual retrieval events from verified OpenAI infrastructure.

The result demonstrated that the tested crawler-style request could benefit substantially from cache reuse.

Historical access logs from 4 November 2026 recorded successful GPTBot-labelled retrieval on Catalogue A. However, controlled testing on 21 November returned HTTP 403 because an existing Apache ModSecurity rule restricted GPTBot access.

Inspection of the server configuration confirmed that the same restriction applied to Catalogue B. Both restrictions were retained. The available evidence does not establish precisely when the restriction began affecting these requests.

That restriction was retained deliberately. The business's customer base and commercial objectives did not provide a compelling reason to reintroduce model-training crawler demand during the investigation.

This illustrates an important distinction: technical optimisation and crawler-access policy are separate engineering decisions. A website can be configured to deliver content efficiently while still selectively determining which automated systems may retrieve it.

07. Verification Across Other Hosted Websites

We extended the review to other production websites on the same VPS to determine whether additional page-cache deficiencies existed.

Several websites already had effective page caching. Their configurations were left unchanged. There was no engineering justification for altering working systems simply for the sake of standardisation.

Website Cache assessment Repeat product response
Catalogue A Page cache newly enabled 54–68 ms
Catalogue B Existing page cache verified 42–66 ms
Catalogue C Page cache newly enabled 114 ms
Catalogue D Existing page cache verified 82–93 ms

Selected anonymous product-page tests across four WooCommerce websites. These measurements describe the tested URLs and do not imply that every catalogue page achieves equivalent performance.

08. Initial Server Conditions Following Remediation

A subsequent VPS health check on 21 November showed substantially more comfortable operating conditions than those recorded during the original incident.

Server metric Observed value
Available CPU cores 6
Load average — 1 minute 1.10
Load average — 15 minutes 1.41
Available memory 4,950 MiB
Swap in use 319 MiB
Active swap-in / swap-out None observed
CPU I/O wait 0% throughout the sample

Snapshot recorded at approximately 03:11 UTC on 21 November 2026 using uptime, free and vmstat. These measurements describe the server's condition at the time of testing, not its sustained performance under an equivalent historical crawling workload.

These initial results were encouraging. However, traffic volumes and application workloads vary over time. A short observation period cannot establish the long-term reduction in aggregate CPU consumption attributable to the changes.

Continued monitoring is required to determine whether daily CPU utilisation returns towards its historical baseline, whether service failures decline and whether the improvements remain effective during subsequent intensive crawling activity.

Before investing in additional infrastructure or restricting valuable machine access, website providers should first establish whether their existing infrastructure is processing incoming requests efficiently.

We had demonstrated substantial improvements in individual request delivery. The remaining challenge was to make crawler identification, retrieval monitoring and access-policy decisions part of an ongoing engineering process rather than an occasional response to server failure.

Technical Solution 20 | Section 07 | AI Retrieval Intelligence

AI Observatory: Turning Crawler Activity into Actionable Evidence

One of the most significant challenges exposed by this investigation was not simply the volume of automated traffic. It was the difficulty of determining which machines were retrieving website content, whether their identities could be corroborated and whether their retrieval attempts succeeded.

Conventional website analytics are primarily concerned with human visitors, engagement and conversions. They do not necessarily provide a reliable account of autonomous machine retrieval.

Server access logs contain much of the raw evidence needed for this analysis, but extracting meaningful information from millions of individual records can require substantial engineering effort.

This is one of the reasons Sydney Business Web developed its AI Observatory: to make recognised search and AI crawler activity, identity corroboration and retrieval outcomes measurable on an ongoing basis.

Why Crawler Identification Matters

During the VPS investigation, we identified two very different forms of automated activity. One involved a declared GPTBot identity supported by published source-IP ranges. The other involved generic browser user agents and thousands of distinct source addresses.

Treating both workloads as equivalent AI crawler activity would have produced a misleading interpretation of the evidence and potentially inappropriate access-control decisions.

AI Observatory applies an evidence-classification methodology that distinguishes between claimed identity, corroborated identity and autonomous retrieval eligibility.

Qualifying Autonomous Crawler Evidence
Genuine AI Observatory dashboard | Sydney Business Web | Rolling 24-hour reporting window
AI Observatory qualifying crawler evidence table showing seven recognised systems, their crawler roles, autonomous accesses, business retrievals, success rates and discovery retrievals.
Figure 7. Genuine AI Observatory dashboard capture showing seven qualifying systems within a rolling 24-hour reporting window. The table distinguishes crawler roles, autonomous accesses, business-content retrieval outcomes and discovery retrievals. Green markers indicate source-IP corroboration against operator-published ranges, not cryptographic verification or endorsement. This is a contemporary Observatory snapshot, not historical telemetry from the VPS incident.
View full-size evidence table

The evidence table above demonstrates why a simple count of bot visits is insufficient. Different systems perform different functions, and successful access to a discovery resource does not necessarily establish successful retrieval of business content.

In this example, GPTBot recorded a successful discovery retrieval but no business-content retrieval during the reporting window. OAI-SearchBot, by contrast, recorded both discovery and business retrieval activity.

These are materially different observations when evaluating the purpose and potential business value of machine access.

A crawler visiting a website is not the same as a crawler successfully retrieving its commercially important content. Reliable AI visibility analysis must distinguish between these events.

Evidence That Does Not Qualify

Equally important is the handling of access records that do not meet the requirements for inclusion in headline autonomous crawler evidence.

AI Observatory retains visibility of recognised crawler claims that could not be corroborated, rather than silently treating them as verified crawler activity or removing them from view.

It also distinguishes corroborated non-autonomous accesses, such as qualifying user-initiated retrieval activity, from autonomous crawling.

Excluded Evidence and Identity Classification
Genuine AI Observatory evidence classification | Identity-unverified and non-autonomous accesses
AI Observatory excluded evidence table showing 681 identity-unverified accesses and six corroborated non-autonomous accesses, separated from headline crawler evidence.
Figure 8. Genuine AI Observatory dashboard capture showing access records excluded from headline autonomous crawler evidence. The table preserves claimed crawler identities, provider names, classification and recorded access counts. Exclusion from headline evidence is an evidential decision, not a decision to block network access.
View full-size evidence table
7 Qualifying systems in the illustrated autonomous crawler reporting window
681 Recorded identity-unverified accesses in the excluded evidence table
6 Corroborated non-autonomous accesses excluded from headline evidence

Figures are taken from the genuine dashboard captures reproduced above. They describe different evidence classifications and should not be interpreted as measurements of the historical VPS incident or as direct measures of infrastructure consumption.

An unverified identity does not necessarily indicate malicious behaviour. It indicates that the available evidence was insufficient to corroborate the declared identity.

Similarly, excluding a non-autonomous access from headline crawler statistics does not mean that the access lacks business value. It means that it represents a different type of retrieval event and should be interpreted accordingly.

Evidence classification is not traffic classification for security enforcement. An unverified request may be legitimate, while a corroborated crawler may still impose an unacceptable infrastructure workload.

Connecting AI Retrieval Evidence with Server Diagnostics

AI Observatory and conventional infrastructure monitoring address complementary engineering questions.

Observatory identifies recognised machine retrieval activity and recorded delivery outcomes. Server diagnostics establish how the underlying infrastructure is responding to its aggregate workload.

Used together, these measurements can help a website provider investigate relationships between crawler activity, application processing, resource consumption and service reliability.

Engineering question Evidence source Operational purpose
Which recognised machines are visiting? AI Observatory Identify declared crawler roles and corroborated identities.
Are retrieval attempts succeeding? AI Observatory and HTTP access records Identify successful retrievals and delivery failures.
Is server demand increasing? CPU, memory and historical system metrics Identify resource pressure and departures from established operating baselines.
Where is processing capacity being consumed? PHP-FPM, database and application diagnostics Investigate expensive execution pathways and worker-capacity limitations.
Are corrective actions effective? Controlled performance tests and ongoing monitoring Measure improvements and detect unintended retrieval failures.
Should a particular crawler retain access? Technical evidence and business objectives Determine an appropriate crawler-access policy.

AI Observatory is not a CPU profiler, and it does not independently calculate the processing cost of every crawler request. Request-level infrastructure attribution requires complementary application and server instrumentation.

Nor is successful machine retrieval proof that the retrieved content has been indexed, cited or recommended by an AI system. Retrieval is a measurable part of the wider AI visibility process, not a guarantee of its eventual commercial outcome.

An Engineering Methodology for Sustainable AI Access

The investigation suggests a practical operating methodology for businesses and website providers managing increasing automated retrieval demand.

Stage 01

Observe

Identify recognised crawler activity, corroborate identities where possible, and measure retrieval outcomes. Establish what is actually accessing the website before deciding how to respond.

Stage 02

Diagnose

Compare retrieval activity with historical infrastructure measurements, application performance and service failures. Identify potentially expensive request pathways and unnecessary processing.

Stage 03

Optimise

Improve content delivery, caching and request handling. Apply selective access policies where justified, then measure the consequences for both website performance and machine retrieval.

This is particularly relevant to providers responsible for multiple production websites. Different clients have different commercial objectives, different content architectures and different reasons for permitting machine access.

A large online retailer may benefit from discovery through AI-assisted product search. A specialist supplier serving an established institutional customer base may assign a different value to model-training retrieval.

The appropriate operating policy should emerge from measured activity, infrastructure capabilities and the client's objectives, rather than from a universal assumption that every AI crawler must be allowed or blocked.

Sydney Business Web | AI Visibility Engineering

AI Observatory

Understand which recognised AI and search crawlers are accessing your business website, distinguish qualifying retrieval evidence from unverified activity, and identify whether important content is being successfully delivered.

Combined with infrastructure diagnostics, AI Observatory provides an evidence base for more informed decisions about website performance, crawler management and AI visibility.

Explore AI Observatory →

Before deciding which AI crawlers deserve access to a business website, we must first understand what they are doing, whether they are succeeding and what accommodating their activity may cost.

Technical Solution 20 | Section 08 | Commercial Access Policy

Which AI Crawlers Should Businesses Allow?

The expansion of AI-assisted information discovery introduces a new commercial responsibility for website providers. We must increasingly determine not only whether machine access is technically possible, but whether accommodating that access serves the interests of the business paying for the infrastructure.

Historically, website owners have generally welcomed legitimate search-engine crawling because of its relationship with search visibility and the potential to attract customers.

AI introduces a broader population of machine consumers. Some retrieve content to support search and discovery. Others collect material for model development. Others access websites in response to specific user requests.

These activities may all be legitimate, yet their commercial value to the website owner can differ considerably.

Legitimacy is not the same as commercial benefit. A genuine, identifiable AI crawler can consume significant infrastructure resources without necessarily delivering a measurable return to the business.

Not All AI Retrieval Serves the Same Purpose

Several major AI providers now document separate crawling and retrieval mechanisms for different functions. These distinctions provide website owners with opportunities to make more selective access decisions.

Category 01

Search and Discovery

Crawlers that support search results, AI-assisted discovery and the retrieval of information used to answer search queries.

Potential business value: improved opportunities for discovery, referrals and exposure to prospective customers.

Category 02

Model Training

Crawlers that collect publicly available information potentially contributing to AI model development and training.

Potential business value: less directly measurable, with no guaranteed relationship between collection and future customer referrals.

Category 03

User-Directed Retrieval

Website access initiated in response to a particular user's interaction with an AI assistant rather than an autonomous catalogue crawl.

Potential business value: immediate relevance to a user's information request or commercial investigation.

Understanding the Available Provider Controls

The following examples illustrate why restricting one crawler does not necessarily require excluding an entire AI provider.

Provider / identifier Documented purpose Business consideration
OpenAI — OAI-SearchBot Discovery through ChatGPT search features. Generally worth accommodating where AI search visibility is a business objective.
OpenAI — GPTBot Collection of content potentially used for model training. Can be restricted independently of OAI-SearchBot.
OpenAI — ChatGPT-User User-initiated website retrieval. Consider customer-directed access separately from autonomous crawling.
Anthropic — Claude-SearchBot Search-related content discovery. Relevant to businesses seeking visibility through Claude's search capabilities.
Anthropic — ClaudeBot Collection supporting model development. Evaluate training access independently of search and user-directed retrieval.
Anthropic — Claude-User Retrieval in response to user requests. Consider opportunities for immediate business-content discovery.
Google — Googlebot Crawling for Google Search, including its integrated AI search features. Restrictions may compromise conventional and AI-assisted Google Search visibility.
Google — Google-Extended Content-use preference for certain Gemini training and grounding activities. Not a separate HTTP crawler identity. Restrictions do not affect inclusion in Google Search.

Provider classifications reflect published documentation consulted in November 2026. These mechanisms can evolve. Consult the current OpenAI crawler documentation, Anthropic crawler documentation and Google crawler documentation before implementing access policies.

Different Businesses Require Different Policies

It would be inappropriate to prescribe the same AI crawler policy for every website maintained by a hosting or website engineering provider.

The decision should reflect the client's customer-acquisition strategy, content, infrastructure and tolerance for automated resource consumption.

Commercial scenario A

Consumer Retail and Large Product Catalogues

A retailer may derive considerable potential value from product discovery through conventional search engines and AI-assisted shopping research.

Efficient product-page caching, carefully controlled dynamic endpoints and reliable machine retrieval may therefore be preferable to broad crawler restrictions.

However, unrestricted retrieval of thousands of catalogue pages may impose significant processing costs, particularly where cache misses dominate.

Commercial scenario B

Government and Institutional Suppliers

A specialist supplier serving established government or institutional customers may depend more heavily on existing procurement relationships than on broad consumer discovery.

Such a business may reasonably retain conventional and relevant AI search access while restricting intensive model-training collection that offers no identifiable immediate commercial benefit.

This was a relevant consideration in retaining existing crawler restrictions on two catalogues examined during our investigation.

Commercial scenario C

Professional and Local Services

Businesses acquiring customers through reputation, expertise and local discovery may benefit from reliable retrieval of their service descriptions, business identity and supporting credentials.

Their smaller content repositories may also be less expensive to crawl than large transactional catalogues.

The emphasis may therefore be on reliable discovery, identity corroboration and efficient delivery rather than intensive crawler restriction.

Commercial scenario D

Publishers and Original Content Producers

Businesses producing original research, specialist information or creative content may place a different commercial value on the material being collected.

Their access policies may depend on both infrastructure considerations and preferences concerning the permitted uses of published intellectual material.

For these organisations, the purpose of retrieval may be just as important as the volume of traffic.

Declaring a Policy Is Different from Enforcing It

Website providers have several mechanisms available for communicating and enforcing automated access preferences.

The robots.txt file provides a machine-readable means of communicating crawler-specific access preferences.

Responsible crawler operators document how their systems interpret these preferences. However, robots.txt is not an authentication mechanism, and it does not independently prevent a request from reaching the server.

Where actual enforcement is necessary, Apache rules, ModSecurity or appropriate edge-layer controls can reject selected traffic before it initiates expensive application processing.

Cloudflare's AI Crawl Control provides another mechanism for managing recognised AI crawler access on suitably configured websites.

These mechanisms should operate together as a coherent policy rather than as unrelated layers of configuration.

User-directed retrieval may operate differently from autonomous crawling. For example, OpenAI documents that robots.txt directives may not apply to certain ChatGPT-User requests.

Website providers should therefore understand both the documented behaviour of the individual system and the actual enforcement capabilities of their infrastructure.

A crawler identity is not a security credential. Access controls based exclusively on self-declared user-agent strings cannot reliably distinguish genuine crawlers from clients impersonating them.

Why Blanket Crawler Blocking Is Not Our Preferred Response

Our investigation demonstrated that substantial performance improvements could be achieved by reducing unnecessary application processing.

On two websites, activating full-page caching reduced measured repeat product-page response times by approximately 94–97%. Selected Apache redirect optimisations produced similarly substantial improvements.

These changes did not require additional crawler restrictions or an increase in allocated server capacity.

Nevertheless, caching cannot eliminate every infrastructure cost. Requests still consume network resources and server capacity, and previously uncached pages may continue to require expensive application processing.

Where legitimate crawler demand remains disproportionate to its potential commercial value, selective restriction may be an entirely appropriate engineering decision.

A Practical Crawler-Access Decision Framework

We suggest that website providers assess automated access through five related engineering and commercial questions.

01

Can the Identity Be Corroborated?

Establish whether the request comes from a recognised and corroborated crawler, a self-declared identity or an unidentified automated client. Do not assume that a familiar user-agent string proves identity.

02

What Is the Purpose of Retrieval?

Distinguish search discovery, model training, user-directed retrieval and other automated collection where the available evidence permits that classification.

03

What Does the Activity Cost?

Examine request volumes, cache behaviour, application execution, CPU utilisation and service failures. Identify the incremental demands of the activity where sufficient instrumentation exists.

04

Does the Activity Support Business Objectives?

Consider the potential contribution to relevant search visibility, customer discovery, referrals and other commercial objectives. The existence of successful retrieval alone does not establish value.

05

Can the Workload Be Managed More Efficiently?

Examine caching, redirect handling and application efficiency before introducing broader restrictions. Where restrictions remain appropriate, apply them selectively and monitor their consequences.

The appropriate outcome may be to allow, optimise, limit or restrict a particular form of machine access. The decision should be based on evidence and business objectives, not simply on whether the requester is an AI system.

The Continuing Role of AI Observatory

An access policy should not be regarded as a permanent decision that never requires further examination.

Crawler populations change. Automated retrieval patterns evolve. New search and agentic services emerge, while a business's commercial objectives and infrastructure capabilities can change over time.

Using AI Observatory alongside infrastructure monitoring allows website providers to examine recognised crawler activity and retrieval outcomes as part of an ongoing operational review.

This makes it possible to revisit previous access decisions when retrieval behaviour, service reliability or commercial priorities change.

AI visibility requires more than unrestricted accessibility. It requires deliberate, measurable and sustainable management of machine access to business information.

Technical Solution 20 | Section 09 | Conclusions

Conclusions: Engineering Websites for Sustainable Machine Access

Can AI bots slow down a business website? Yes. Automated retrieval consumes real infrastructure resources, and legitimate AI crawling can contribute to the workload that website providers must accommodate.

Our investigation also demonstrated why that apparently straightforward answer requires careful engineering interpretation.

Corroborated AI crawler activity coincided with the initial escalation in server demand. However, sustained CPU saturation continued after the identified GPTBot workload had substantially declined, revealing the importance of other automated retrieval activity and shared infrastructure workloads.

Most importantly, we identified concrete opportunities to deliver public website content more efficiently, with substantial measured improvements in repeat request response times.

6.5× Initial increase in hourly GPTBot-labelled requests on one catalogue
96–99% Approximate sustained hourly CPU utilisation during the most severe period
10 Recorded Apache restart cycles on 5 November 2026
94–97% Reduction in selected repeat product-page response times after cache activation

Summary of selected measurements from the historical investigation and subsequent controlled remediation tests. These figures represent different measurement scopes and should not be interpreted as a direct calculation of AI-related CPU consumption.

What the Evidence Supports

The value of a production infrastructure investigation lies in establishing findings that can inform practical engineering decisions while remaining clear about the limits of the available evidence.

Principal Findings

  • Corroborated AI crawler activity increased substantially during the initial escalation in server workload.
  • Distributed browser-labelled traffic continued to generate intensive product retrieval during sustained CPU saturation.
  • Individual website retrieval failures and repeated Apache restart cycles were recorded during the incident.
  • Full-page caching substantially reduced selected repeat product-page response times.
  • Moving suitable redirects into Apache eliminated unnecessary WordPress processing.
  • These performance improvements were achieved without increasing server capacity or introducing additional AI crawler blocks.

Outstanding Research Questions

  • What proportion of aggregate CPU consumption can be attributed to individual automated traffic groups?
  • How do large-scale cache-hit and cache-miss patterns vary between different types of crawler?
  • What long-term reduction in server resource consumption will follow the implemented optimisations?
  • How can infrastructure cost be related to measurable commercial outcomes from machine discovery?
  • Which combinations of caching, crawler policy and traffic management provide the best balance for different business models?

These outstanding questions provide a foundation for further investigation. They do not diminish the value of the measured performance improvements already achieved.

Technical Recommendations for Website Providers

Based on the experience gained during this investigation, we recommend the following operational approach to managing automated retrieval workloads.

01

Establish Historical Operating Baselines

Retain sufficient CPU, memory, load and service-monitoring history to identify abnormal operating conditions and reconstruct the onset of an incident. A live server snapshot alone cannot establish when or why a workload changed.

02

Make Machine Retrieval Measurable

Record recognised AI and search crawler activity, identity corroboration, requested content and retrieval outcomes. Keep unverified and non-autonomous access distinguishable from qualifying autonomous crawler evidence.

03

Examine Traffic Beyond Recognised AI Bots

Investigate requested endpoints, repeated URL retrieval, source-IP distributions and temporal traffic patterns. Significant automated workloads may present themselves as ordinary web browsers rather than declared crawlers.

04

Verify Full-Page Caching

Do not assume that the presence of a caching plugin establishes that effective page caching is operational. Test actual initial and repeat responses, particularly on large public product catalogues.

05

Protect Transactional Functionality

Verify cache exclusions for authenticated users, shopping carts, checkout, customer accounts and application-specific sessions. Performance improvements must not compromise customer privacy or transactional correctness.

06

Eliminate Unnecessary Application Execution

Where appropriate, handle straightforward redirects and request-routing decisions at the web-server or edge layer. Avoid invoking an entire application stack for operations that can be completed safely at a lower level.

07

Implement Client-Specific Crawler Policies

Distinguish training collection, search discovery and user-directed retrieval. Align access decisions with the client's commercial objectives, content-use preferences and available infrastructure resources.

08

Measure the Consequences of Every Change

Monitor application response times, crawler retrieval outcomes, resource utilisation, service health and customer-facing functionality. Confirm sustained improvements under representative production workloads rather than relying exclusively on isolated performance tests.

Adding more infrastructure capacity may sometimes be necessary. But capacity expansion should be an informed engineering decision, not an automatic response to processing inefficiency or unmanaged crawler demand.

A Continuing Engineering Investigation

At the time of publication, the selected website and Apache optimisations had been implemented and their immediate performance improvements verified.

The longer-term effects on aggregate VPS CPU utilisation, service reliability and resource demand remain subjects for continued observation.

Future investigations will examine whether these improvements remain effective during subsequent intensive automated retrieval periods and whether additional instrumentation can improve attribution of infrastructure consumption to individual crawler workloads.

Sydney Business Web | Engineering-Led Website Management

Understand What Is Accessing Your Website — and What It Means for Your Business

Sydney Business Web approaches business websites as managed information systems, not simply as hosted collections of pages.

Our engineering work combines website infrastructure management, performance optimisation, automated retrieval diagnostics and AI visibility analysis.

AI Observatory provides measurable evidence of recognised AI and search crawler activity, helping businesses and website providers make informed decisions about machine access and content delivery.

The objective is not to keep machines away from business websites. It is to ensure that the right machines can retrieve the right information, efficiently and reliably, without compromising the infrastructure on which the business depends.

Technical Solution 20 | Section 10 | Research and References

Research Methodology and Technical References

This article documents an original Sydney Business Web engineering investigation conducted on live production website infrastructure, supported by historical system records, archived HTTP access logs and controlled remediation tests.

Historical traffic, CPU utilisation and service reliability measurements were obtained directly from production infrastructure and archived operating records.

Subsequent remediation measurements used controlled HTTP requests to production websites, including the explicitly identified GPTBot user-agent simulation presented in Figure 6. This simulation tested response behaviour rather than the activity of genuine OpenAI crawler infrastructure.

Primary Evidence Sources

Multiple independent measurement sources were used to reconstruct the incident and evaluate the subsequent engineering changes.

Evidence source Measurement and purpose
Linux sysstat / sar Historical aggregate CPU utilisation, operational baselines and hourly escalation patterns. CPU busy was calculated as 100% minus recorded idle time.
Apache HTTPS access logs Request volumes, timestamps, HTTP response codes, requested URLs, user-agent strings and observed source-IP distributions.
Published crawler IP ranges Corroboration of declared GPTBot identities against operator-published network ranges.
systemd journal Timestamped Apache stop-and-start events and recorded service restart chronology.
cPanel chkservd Service health-check failures, socket timeouts and automatic recovery attempts.
Process and resource diagnostics Live PHP-FPM and database activity, system load, available memory, swapping and I/O wait observations.
W3 Total Cache configuration Verification of page-cache activation, selected cache engine and configured transactional exclusions.
curl performance tests Controlled HTTP GET requests, response-code verification, header inspection and total request completion times.
Browser functional testing Verification of public product retrieval, cart behaviour, checkout presentation and available payment options.
AI Observatory Contemporary examples of recognised crawler identification, source corroboration, retrieval outcomes and evidence classification.

Interpretation and Research Limitations

HTTP request counts represent recorded access events, not unique human visitors, individual AI systems or direct measurements of processing cost.

Where requests are described as non-static, the classification is based on exclusion of selected static file extensions. It does not establish that every remaining request invoked PHP or database processing.

CPU measurements represent the aggregate VPS workload. Temporal relationships between traffic activity and CPU utilisation are therefore evidence of association rather than definitive request-level CPU attribution.

Caching and redirect measurements represent selected HTTP request completion times under the observed test conditions. They are not full-browser rendering measurements, statistical performance benchmarks or direct calculations of sustained infrastructure cost reduction.

The AI Observatory screenshots are genuine contemporary dashboard captures presented to illustrate the applicable monitoring methodology. They are not retrospective Observatory records of the 4–5 November incident.

The research distinguishes between directly measured observations, corroborated identities, engineering interpretations and outstanding questions. This distinction is maintained throughout the investigation to avoid attributing conclusions that the available evidence cannot independently establish.

Technical Glossary

The following definitions explain the principal infrastructure and crawler-management terminology used throughout this investigation.

Technical term Meaning
VPS Virtual Private Server. A hosting environment with allocated computing resources.
CPU saturation A condition in which almost all available processor capacity is being utilised.
Apache / HTTPD Web-server software responsible for receiving HTTP requests and delivering website responses.
PHP-FPM PHP FastCGI Process Manager. Manages the worker processes that execute PHP application code, including WordPress and WooCommerce.
MariaDB Database software used to store and retrieve website content, product information and application data.
chkservd A cPanel service-monitoring component that checks service responsiveness and can initiate automatic restart attempts.
sar System Activity Reporter. A Linux diagnostic utility used to examine recorded CPU and other system performance measurements.
ModSecurity A web application firewall capable of inspecting and restricting HTTP requests before they reach application code.
I/O wait A CPU accounting metric representing idle time associated with outstanding input or output operations.
Full-page caching Storing generated website pages for reuse rather than regenerating them for every equivalent request.
Disk: Basic A W3 Total Cache page-caching method that delivers stored content through a PHP-based request pathway.
Cache hit / cache miss A cache hit occurs when suitable stored content is available. A cache miss occurs when the requested content cannot be served from that cache.
HTTP 301 A permanent redirect directing the requester to another URL.
HTTP 403 Forbidden. The server refuses to fulfil the request.
HTTP 500 / 503 Server-side error responses indicating an internal server error or temporary service unavailability, respectively.
User agent A self-declared HTTP client identifier. It may describe a browser or crawler but does not independently prove identity.
Source-IP corroboration Comparing observed source IP addresses against authoritative network information to support a claimed crawler identity.
Distributed crawling Automated content retrieval originating from numerous source addresses rather than a single identifiable network endpoint.
Per-IP rate limiting Restricting request frequency according to individual source IP addresses.
Machine retrieval Automated access to website content by search engines, AI systems, agents or other software clients.

External Technical References

The following authoritative documentation supports the technical concepts, crawler classifications and infrastructure mechanisms discussed in this article. The original production measurements are Sydney Business Web's own research.

[01]
OpenAI — Overview of OpenAI Crawlers

Documentation of GPTBot, OAI-SearchBot, ChatGPT-User and independent crawler access preferences.

[02]
OpenAI — Published GPTBot IP Address Ranges

Operator-published network ranges used for source-IP corroboration. This is a maintained resource whose contents may change over time.

[03]
Anthropic — Web Crawlers and Site Owner Controls

Provider documentation distinguishing ClaudeBot, Claude-SearchBot and Claude-User.

[04]
Google — Common Crawlers

Google crawler identifiers, crawling preferences and the Google-Extended product token.

[05]
Google — AI Features and Your Website

Technical eligibility and content controls for AI Overviews and AI Mode within Google Search.

[06]
Cloudflare — Manage AI Crawlers

Edge-layer management of recognised AI crawler activity and selective access-control mechanisms.

[07]
WooCommerce — Configuring Caching Plugins

Official guidance concerning dynamic transactional pages, customer sessions and WooCommerce cache exclusions.

[08]
Apache HTTP Server — Redirecting and Remapping

Official examples and technical guidance for web-server-level redirects and URL rewriting.

[09]
cPanel — TailWatch and chkservd

Documentation of connection-based service monitoring and automatic service restart mechanisms.

[10]
IETF — RFC 9309: Robots Exclusion Protocol

Internet standards documentation defining the robots exclusion protocol and its relationship to crawler access preferences.

External technical documentation consulted on 21 November 2026. Provider capabilities and crawler-access mechanisms may evolve after publication.

Related Sydney Business Web Research and Resources

[11]
AI Crawling of Large WooCommerce Catalogues

Related engineering research examining the infrastructure implications of intensive catalogue retrieval.

[12]
AI Crawlers Blocked by ModSecurity

A technical investigation into HTTP retrieval failures caused by infrastructure security controls.

[13]
AI Crawler HTTP 429 Errors

Related diagnostic research into automated retrieval restrictions and request-rate failures.

[14]
AI Observatory

Sydney Business Web's machine retrieval monitoring system, providing recognised crawler evidence and recorded content-delivery outcomes.

[15]
AI Retrieval Evidence

Supporting information concerning measurable machine retrieval and evidence-based AI visibility.

[16]
AI Visibility Analysis and Remediation

Engineering-led assessment and remediation of business website identity, accessibility and machine retrieval issues.

Frequently Asked Questions

Can AI bots slow down my website?

Yes. AI crawlers consume network, server and potentially application processing resources. Intensive retrieval can contribute to reduced performance, particularly on websites that generate complex pages dynamically.

The extent of the impact depends on request volume, application efficiency, caching, available infrastructure capacity and other concurrent workloads.

Can AI crawlers cause a website to become unavailable?

Intensive automated retrieval can contribute to resource exhaustion and service unavailability. In this investigation, intensive machine retrieval coincided with sustained CPU saturation, HTTP 503 responses and repeated Apache restart cycles.

However, the evidence did not establish that AI crawling alone caused the entire infrastructure incident.

Are AI crawlers particularly demanding on WooCommerce websites?

Large WooCommerce catalogues can expose thousands of individually generated product and category pages. Retrieving these pages may involve substantial PHP processing, database activity and plugin execution.

Effective page caching, efficient redirects and appropriate crawler policies can help reduce unnecessary application processing.

Should I block all AI bots to protect website performance?

Generally, we recommend evaluating crawler identities, documented purposes and observed behaviour before imposing broad restrictions.

Blanket blocking may prevent access by systems that support valuable search discovery and customer-directed retrieval. Selective restrictions combined with infrastructure optimisation can provide a more appropriate balance.

Can I block GPTBot without blocking ChatGPT search?

Yes. OpenAI documents independent access preferences for GPTBot and OAI-SearchBot.

A website owner can decline GPTBot model-training collection while retaining OAI-SearchBot access for ChatGPT search discovery. Existing firewall and other access controls must also be configured consistently with that decision.

Does robots.txt actually prevent AI crawlers from accessing my website?

Robots.txt communicates crawler-access preferences to systems that observe the robots exclusion protocol. It does not itself enforce network or HTTP access restrictions.

Actual enforcement may require web-server rules, a web application firewall or suitable edge-layer controls.

Will ordinary IP-based rate limiting stop distributed automated traffic?

Not necessarily. During this investigation, one browser-labelled traffic group generated 2,664 requests from 2,484 distinct source IP addresses in one hour.

More than 93% of those addresses made exactly one request. Conventional per-IP thresholds may fail to identify such a distributed aggregate workload.

Can page caching reduce the infrastructure impact of AI crawling?

Yes, particularly when automated systems repeatedly retrieve equivalent public URLs. Full-page caching can avoid much of the processing required to regenerate application content.

In this investigation, selected repeat product-page responses became approximately 17–30 times faster after page-cache activation. A separate simulated GPTBot request sequence demonstrated an approximately 26-fold difference between initial and repeat response times.

Is full-page caching safe for WooCommerce?

It can be, provided that customer-specific and transactional pathways are correctly excluded from public caching.

Cart, checkout, customer account pages, session cookies and application-specific personalised content require particular attention. Correct configuration and functional testing are essential before considering an implementation complete.

How can I identify which AI crawlers are accessing my website?

Server access logs provide raw evidence of HTTP requests, declared user agents, source addresses and delivery outcomes.

Sydney Business Web's AI Observatory provides a structured monitoring system for recognised AI and search crawlers, available identity corroboration and recorded retrieval outcomes.

Unlike a simple bot-visit counter, it distinguishes different retrieval roles and evidence classifications.

Does AI Observatory measure how much CPU each crawler consumes?

No. AI Observatory measures recognised machine access and retrieval outcomes. It is not a request-level CPU profiler.

Infrastructure-cost attribution requires complementary server monitoring, application diagnostics and appropriate request-level instrumentation. These evidence sources are most valuable when interpreted together.

Should I upgrade my hosting if AI crawlers are consuming too many resources?

Additional capacity may be appropriate, but we recommend first establishing whether existing resources are being used efficiently.

Ineffective caching, expensive redirects and unmanaged automated workloads can create unnecessary infrastructure demand. Addressing these issues may substantially improve performance before a hosting upgrade becomes necessary.

What should businesses look for when choosing website hosting in the AI era?

Businesses should look beyond storage, bandwidth, price and conventional uptime commitments.

A capable website management provider should understand crawler identification, selective bot control, infrastructure monitoring, application optimisation and the preservation of valuable AI search access.

Business owners should also establish who is responsible for these activities. Buying a hosting package does not necessarily mean that automated retrieval workloads are actively managed.

Does successful AI crawling guarantee AI visibility or recommendations?

No. Successful retrieval establishes that content was delivered to the requesting system. It does not guarantee indexing, citation, recommendation or commercial referrals.

Reliable retrieval is one measurable component of a broader AI visibility engineering strategy.

Original Sydney Business Web Research

Are AI Bots Slowing Down Your Website?
Technical Solution 20
Keith Rowley | Sydney Business Web
November 2026

This publication documents original production infrastructure observations, forensic diagnostic analysis and controlled engineering remediation tests. Raw production logs and confidential client identifiers are not reproduced.

The investigation is observational and has not undergone independent peer review. Findings should be interpreted within the measurement conditions and limitations described throughout the article.

Sydney Business Web develops engineering-led website infrastructure, machine retrieval monitoring and AI visibility analysis systems for businesses.