Technical Solution 19: AI Crawling Large Woocommerce Catalogues
A Real-World Case: AI Crawling and a 17,000-Product WooCommerce Catalogue
Every day, we solve problems for eCommerce website owners. We had a think about how we might use this activity to help others, and came up with this idea: Every week or three, we'll take the trickiest problem and publish our solution.
Important! - Some of these solutions involve adding code to your website (WordPress and Woocommerce mostly), so please ALWAYS be careful. We are not in any way responsible, directly or indirectly for any impact or consequences our code or advice has on your website, nor are we liable for any damage arising from such use.
Always back up your website before changing or adding code and/or editing the database, This is critically important!!!

Managing AI crawler traffic across large WooCommerce catalogues requires a balance between AI visibility, server performance and website availability.
A WooCommerce catalogue containing approximately 17,000 products presents a very different engineering challenge from a conventional business website.
Product discovery, deep pagination, filtering and cart operations can generate expensive PHP and database requests. When automated crawlers repeatedly explore these pathways, the workload can grow substantially without any corresponding increase in genuine customers.
During a November 2026 investigation of a managed VPS hosting multiple WordPress and WooCommerce websites, we encountered PHP-FPM worker exhaustion, request queues and HTTP timeout errors.
Server logs also exposed automated catalogue exploration, unusual add-to-cart requests and traffic from numerous source IP addresses. These findings required us to examine both the cost of individual requests and the infrastructure's ability to handle them.
The engineering challenge is not simply to block crawlers. It is to keep valuable content discoverable by search engines and AI systems without allowing automated requests to compromise website availability.
01. The Large-Catalogue Problem
Thousands of product URLs, pagination paths and dynamic WooCommerce endpoints create a large processing surface. Not every request can be served cheaply from cache, particularly when catalogue queries or cart operations are involved.
02. The AI Visibility Problem
Search and AI discovery depend on reliable access to website content. Blanket crawler restrictions can undermine that objective. The better approach combines selective access controls, efficient page delivery and measurable server performance.
Our objective was to preserve machine discoverability, protect genuine shoppers and identify which requests were placing unnecessary demands on the VPS.
Why Large WooCommerce Catalogues Change the Rules
The problem with a large product catalogue is not necessarily the number of products. It is the number of ways those products can be requested, filtered, paginated and processed.
A catalogue containing 17,000 products may expose thousands of individual product pages, category archives, search results and pagination combinations. Product variations and filters can expand the available URL space considerably.
Search engines and AI crawlers may explore this content to discover products and understand the business. Other automated clients may repeatedly request deep catalogue pages without providing meaningful discovery value.
The important distinction is that not every catalogue request has the same processing cost. A cached product page might require very little origin processing, while a dynamic catalogue query can involve WordPress, WooCommerce and MariaDB.
Automated Access
A crawler requests a product, category, pagination or dynamic catalogue URL.
Origin Processing
If the request cannot be satisfied by cache, Apache forwards it for PHP processing.
WordPress + Database
WooCommerce executes application logic and may query products, metadata, categories or sessions.
Worker Occupancy
A PHP-FPM worker remains occupied until processing finishes. Concurrent requests consume additional capacity.
This illustrates the typical processing path for uncached PHP requests. Actual behaviour depends on the site's caching, hosting and WooCommerce configuration.
The Catalogue Multiplier
Catalogue size creates an important distinction between a conventional business website and a large eCommerce application.
A brochure website may contain dozens of relatively static pages. A large WooCommerce store can generate many more request paths through product archives, filtering, sorting and pagination.
The resulting workload depends on which URLs are requested, whether they are cached and how efficiently the application processes them. Product count alone does not determine server capacity.
Where the Cost Accumulates
Requests for distant catalogue pages may require expensive database queries, depending on the query structure and indexes.
Cart updates, fragments and AJAX operations often require PHP execution rather than a simple cached response.
Several slow requests can occupy available PHP-FPM workers. Further requests may queue and eventually time out.
The risk is not that an AI crawler reads a product page. The risk is that automated exploration repeatedly triggers expensive application processing faster than the infrastructure can comfortably complete it.
There is also an important attribution problem. A user-agent string can claim to belong to an AI crawler, a search engine or an ordinary browser. That claim alone does not establish the requester's identity. Crawler identification and traffic behaviour therefore need to be assessed separately.
The next question: What did the server logs actually reveal when we investigated a large WooCommerce catalogue experiencing PHP-FPM saturation and repeated automated requests?
Automated Catalogue Requests: Following the Server Evidence
A production incident involving large WooCommerce catalogues exposed the relationship between application processing, automated requests and available server capacity.
The investigation concerned a managed VPS hosting multiple WordPress websites, including substantial product catalogues. The immediate problem was genuine: PHP-FPM workers became saturated, requests accumulated and several websites experienced service errors.
How the Investigation Developed
PHP-FPM Requests Accumulated
Both affected WooCommerce pools reached their five-child limits. A PHP-FPM socket backlog was observed at 129 against a configured backlog of 128. Apache FastCGI waits reached the 300-second timeout interval, with HTTP 503 and 504 errors observed during the incident.
Unusual Catalogue Exploration
Apache access logs revealed requests against obsolete catalogue structures, deep pagination and dynamic WooCommerce endpoints. In one subsequent one-hour sample, 137 GET requests matched an unusual add-to-cart and pagination pattern, originating from 136 distinct source IP addresses.
Measuring PHP Processing
PHP-FPM access timing was introduced to measure individual request duration and CPU consumption. Subsequent dynamic WooCommerce requests were recorded taking approximately one to two seconds.
These measurements were collected after the original incident. They do not establish how long the requests responsible for the earlier saturation took.
GET /?add-to-cart=[product-id]&product-page=[deep-page]
POST /?wc-ajax=get_refreshed_fragments
POST /wp-admin/admin-ajax.php
The cart/pagination GET sequence and AJAX POSTs were observed in the logs. They are shown together as request types, not as a proven sequence generated by the same clients.
Were These Requests Actually AI Crawlers?
This distinction matters. Conventional search and AI crawlers generally retrieve published content. Automated clients that interact with cart operations, JavaScript-driven interfaces or administrative AJAX endpoints can display behaviour more consistent with browser automation.
The unusual cart/pagination requests originated from many different IP addresses and followed a recognisable sequence. This was consistent with automated exploration, but did not identify the software or organisation responsible.
Separately, the investigation recorded traffic claiming crawler identities such as GPTBot. These user-agent claims were not independently verified as originating from the named providers.
We therefore distinguish verified automated request patterns from verified AI crawler identities. The former were demonstrated; the latter were not established for the suspicious cart activity.
Evidence Before and After Corrective Action
| Measurement | Earlier observation | Subsequent verification |
|---|---|---|
| PHP-FPM socket backlog | 129 queued against a configured backlog of 128. | Later check recorded 0 queued against 128. |
| PHP-FPM pool capacity | Five children per affected WooCommerce pool; both reached their limits. | No pool-capacity increase was required for the verified targeted blocking tests. |
| Abusive cart/pagination URL | Matching requests were recorded returning HTTP 200 before the targeted rule. | Controlled test returned HTTP 403 in approximately 14 milliseconds. |
| Apache internal redirects | AH00124 errors recorded with conflicting error-page directives present. | No new AH00124 errors found in the subsequent sampled log interval after correction. |
| Ordinary website access | HTTP 503/504 errors occurred during the production incident. | A later homepage test returned HTTP 200. This was not a complete performance benchmark. |
Measurements were collected at different stages of the investigation. They demonstrate specific conditions and corrective behaviour, not a controlled before-and-after server performance test.
What We Demonstrated
The PHP-FPM saturation, request backlog, FastCGI timeouts and HTTP errors were genuine production conditions.
Automated-looking request patterns were documented in the access logs.
Targeted ModSecurity restrictions rejected selected unwanted requests quickly.
Conflicting Apache ErrorDocument directives were corrected, with no recurrence of the earlier redirect errors in subsequent checks.
What Remains Unresolved
The original worker saturation could not be conclusively attributed to a particular AI crawler or request pattern.
The 137 matching requests represented approximately 2.3 requests per minute. That volume alone does not explain the severe saturation without knowing actual request costs and concurrency.
Slow database queries, application blocking, external service waits and shared dependencies remained possible contributing factors.
Later intermittent load spikes were also not conclusively explained.
The Additional Apache Configuration Problem
A separate investigation identified conflicting Apache ErrorDocument directives. Their effective configuration could direct error handling towards unavailable error documents, resulting in repeated internal redirects and AH00124 errors.
Correcting these directives addressed a demonstrated server configuration problem independently of the crawler investigation.
This was particularly relevant because security rules designed to reject requests cheaply should not trigger further application processing simply to display an error response.
The most important distinction was between observing suspicious automated behaviour and proving its contribution to a production failure. Effective engineering requires evidence for both.
The immediate response established targeted protections, corrected an Apache configuration defect and improved request-level visibility. It did not establish a single definitive cause for all observed server load.
The next engineering task was to preserve legitimate customer and AI discovery access while restricting unnecessary dynamic catalogue processing.
Protecting the Server Without Disappearing from AI Search
The simplest response to excessive crawler traffic would be to block automated visitors. For a business investing in search visibility and AI discoverability, that would be a potentially damaging approach.
Search engines and AI retrieval systems need access to relevant business and product information. Meanwhile, unrestricted access to expensive catalogue operations can place unnecessary demands on the server.
Our approach was therefore selective: preserve useful content discovery, restrict unwanted automated behaviour and reject abusive requests before they reached WordPress wherever possible.
Preserving Useful Discovery
We deliberately preserved access for ordinary customers and important search discovery clients, including Googlebot, Bingbot and OAI-SearchBot.
Search-oriented AI crawlers and general-purpose model crawlers do not necessarily serve the same purpose. Their access policies can therefore differ.
The goal was to keep relevant product and business information available for legitimate discovery, subject to appropriate verification and server capacity.
Restricting Unnecessary Processing
We restricted selected crawler user-agent patterns where their activity offered no sufficiently established discovery benefit to justify the processing cost.
Separate controls addressed suspicious legacy catalogue requests and a specific combination of add-to-cart operations with deep product pagination.
These were targeted restrictions against defined request patterns, not an attempt to make the entire catalogue inaccessible.
Four Principles Behind the Protection
Understand Who Is Requesting What
Examine user-agent claims, request frequency, URL patterns and processing cost. Where crawler identity matters, use the provider's documented IP or DNS verification methods. A user-agent string alone is not proof that a request came from a particular search or AI platform.
Stop Unwanted Requests Before PHP
We used narrowly scoped Apache ModSecurity rules to reject selected requests early in processing. The intention was to prevent unnecessary WordPress execution, database queries and PHP-FPM worker occupancy.
Make Sure a 403 Really Ends the Request
A security rejection should not trigger an expensive PHP-rendered error page. We also corrected conflicting Apache ErrorDocument directives associated with internal redirect errors. Error handling must remain efficient when the application is under load.
Test Both Blocked and Allowed Traffic
We checked the responses to targeted test requests and ordinary website requests, confirming that specific unwanted patterns were rejected while normal page access remained functional. Broader shopping and checkout regression testing is still important when deploying such rules.
Verification: Early Rejection
One test of the targeted cart/pagination restriction returned HTTP 403 in approximately 0.014 seconds.
A separate ordinary homepage request returned HTTP 200, confirming that general page access had not been blocked.
The tests confirmed the immediate rule behaviour. They did not, by themselves, establish a measurable improvement in overall server performance or prove the original incident's cause.
AI visibility does not require unrestricted access to every expensive application endpoint. It requires dependable access to the content and evidence that machines need to discover, understand and corroborate the business.
Robots.txt Is Not a Firewall
Robots.txt provides crawling instructions to compliant bots. It cannot reliably stop clients that ignore those instructions or deliberately impersonate other user agents.
Enforced access controls belong at the appropriate security layer. Where an edge WAF is available, unwanted traffic may be rejected before reaching the origin. Otherwise, carefully configured server-level rules can provide a similar early-rejection function.
Protecting the Catalogue Itself
Crawler restrictions are only one part of the solution. Large catalogues also benefit from efficient product queries, appropriate database indexes, sensible pagination, safe page caching and correctly managed WooCommerce background activity.
The objective is to reduce the cost of legitimate access, not merely reduce the number of visitors reaching the server.
This approach also requires regular review. Crawler identities, business requirements and AI discovery mechanisms can change. A restriction that is justified today should not automatically become a permanent policy without examining its consequences.
The remaining question is how large WooCommerce catalogues should be engineered and monitored to remain responsive as automated discovery continues to expand.
Engineering Large WooCommerce Catalogues for AI Crawling and VPS Resilience
A large WooCommerce catalogue needs more than a powerful hosting account. It needs an application architecture that can accommodate legitimate machine discovery without allowing automated requests to consume disproportionate server resources.
With approximately 17,000 products, the challenge is not simply handling more visitors. It is managing the potentially large number of product, category, pagination and dynamic application requests that automated clients can generate.
Our investigation focused on the managed VPS, Apache, ModSecurity, PHP-FPM and WooCommerce processing environment. The corrective controls documented in this case were implemented directly on that infrastructure.
The following engineering priorities describe how large catalogues should be evaluated and maintained. They include longer-term recommendations beyond the immediate changes performed during the investigation.
Conceptual model for a managed VPS. Actual request handling depends on Apache configuration, security rules, cache eligibility and application behaviour.
Six Engineering Priorities for Large Product Catalogues
Make Important Product Pages Easy to Retrieve
Search engines and AI discovery systems need reliable access to published product information, relevant categories and supporting business content.
Important destinations should have stable URLs, useful content, consistent canonical references and efficient responses.
Where suitable, public product and category pages can be served from a properly configured page cache, reducing repeated PHP and database work. Personalised responses must remain correctly excluded.
Reduce Low-Value Automated Exploration
Deep pagination, product filters, sorting parameters and obsolete catalogue structures can expose large numbers of request combinations.
These URLs do not necessarily contribute useful information to search or AI retrieval. Some may trigger expensive application processing.
Evaluate which URL patterns genuinely deserve crawler access. Use appropriate indexing, canonicalisation and security controls without blocking important product discovery.
Identify Expensive WooCommerce Queries
Large product and metadata tables can make particular queries costly, especially when combined with filters, sorting or deep pagination.
Database slow-query analysis should establish where processing time is being spent before changing indexes or configuration.
Persistent object caching may help repeated queries in suitable environments, but inefficient queries must still be investigated rather than concealed by caching.
Prevent Worker Exhaustion
PHP-FPM pools have finite processing capacity. A small number of slow concurrent requests can occupy every available worker.
Our investigation encountered WooCommerce pools reaching their five-child limits, with queue accumulation and prolonged request timeouts.
Worker limits should be reviewed alongside memory consumption, request timing and database performance. Simply increasing the number of workers can create additional resource pressure.
Keep Background Jobs Under Control
WooCommerce background activity, scheduled actions, imports and maintenance tasks can compete with customer requests for server resources.
The combined workload matters particularly when several WordPress installations share the same VPS.
Review WP-Cron behaviour, scheduled actions and task execution frequency. Where justified, controlled server cron scheduling can improve predictability without disabling essential processing.
Separate Useful AI Discovery from Unwanted Traffic
Not every automated client has the same purpose. Search discovery, AI retrieval, model training and commercial scraping are different activities.
Crawler policies should consider the client's documented purpose, verified identity, request behaviour and the business value of allowing that access.
Our approach preserved important discovery access while restricting selected unwanted crawler patterns and specific abusive catalogue request combinations.
Apply Different Policies to Different Request Types
A large catalogue should not treat every URL as equally valuable or equally expensive.
Preserve useful discovery, optimise response times and maintain accessible, consistent published content.
Examine actual processing costs and search value. Avoid allowing unnecessary combinations to expand the crawlable URL space without practical limits.
Protect dynamic functions from unnecessary automated execution. These operations are generally not required to discover published product information. Preserve legitimate shopping and application functionality.
Maintain useful redirects where appropriate, but reject invalid or abusive patterns efficiently before unnecessary WordPress processing begins.
Define access according to crawler purpose and business requirements. Preserve important discovery pathways while reviewing excessive or unsuitable request behaviour.
Monitoring: What Should Website Operators Measure?
Look Beyond the Server Load Average
Our investigation also demonstrated the limitations of relying on a single load figure to assess server health.
Load averages describe work over time. They do not independently identify which website, application request or process created that workload.
A server may have available CPU capacity while an individual PHP-FPM pool remains constrained by slow requests or insufficient workers.
The useful question is not simply whether the VPS occasionally reports a high load average.
It is whether customers experience slow responses, requests accumulate, application workers become exhausted or avoidable HTTP errors occur.
Could a CDN Help a Large Catalogue?
A suitably configured CDN can reduce origin-server requests by delivering cached public product pages and static resources. Some CDN services also provide crawler controls at the network edge.
However, a CDN does not automatically resolve expensive WooCommerce cart operations, AJAX requests, database queries or PHP-FPM capacity problems. Dynamic and personalised responses still require appropriate application handling.
No CDN-based mitigation was used to address the behaviour described in this investigation. The documented protections were implemented directly within the managed VPS environment using Apache and ModSecurity.
The objective is not to stop machines discovering a large product catalogue. It is to make useful discovery inexpensive, unnecessary processing avoidable and legitimate customer activity dependable.
These principles also highlight the importance of separating infrastructure management from application engineering.
On a managed VPS, the hosting provider may be responsible for maintaining core server services, while the website operator is responsible for WooCommerce behaviour, application configuration and appropriate request policies.
Diagnosing performance correctly requires examining both sides. A crawler restriction cannot repair an inefficient database query, just as increasing PHP-FPM capacity cannot correct an unnecessary application request.
Large-catalogue resilience is an ongoing engineering discipline: measure the workload, identify its cost, protect critical functions and verify the result.
What We Fixed. What We Learned.
The investigation demonstrated that automated catalogue requests, application capacity and Apache configuration must be examined together when a large WooCommerce website encounters performance problems.
Several problems were identified and corrected. Others required continued measurement rather than speculative intervention.
Verified Corrective Results
Targeted request protection: Selected unwanted crawler patterns and abusive catalogue requests were rejected using Apache ModSecurity rules.
Early rejection: A representative suspicious cart/pagination request returned HTTP 403 in approximately 14 milliseconds.
Apache error handling: Conflicting ErrorDocument directives were corrected. Subsequent checks found no recurrence of the previously observed internal redirect errors.
Application availability: Later checks showed the previously congested PHP-FPM socket queue clear, with normal website access responding successfully.
Questions Still Under Investigation
Original incident causation: The precise contribution of each crawler and dynamic request pattern was not established.
Intermittent server load: Occasional elevated load averages continued, although later sampling showed available processing capacity. Their causes remained unidentified.
Long-term optimisation: Database query efficiency, caching behaviour, background processing and combined PHP-FPM capacity remain relevant areas for ongoing assessment.
These outstanding questions should be resolved through measurements and evidence, not assumptions about AI crawlers or server configuration.
Five Lessons for Large-Catalogue Operators
Catalogue Size Changes the Risk
Thousands of products create numerous discovery paths, dynamic operations and database workloads. Hosting decisions must account for processing behaviour, not merely storage or product count.
Measure Request Cost, Not Just Traffic Volume
A relatively small number of expensive requests can exhaust a constrained PHP-FPM pool. Access logs and request timing reveal more than simple visitor counts.
Restrict Behaviour Selectively
Effective crawler policies distinguish useful public discovery from repeated, unnecessary application operations. Blanket blocking is not the only option.
Security Controls Must Be Efficient
Rejected requests should terminate quickly. Incorrect error handling can create additional server processing and undermine the protection that security rules were intended to provide.
Preserve AI Discoverability
Protecting server resources should not unnecessarily remove access to published products, business information or other evidence that search and AI systems need to retrieve.
The objective is not to make a large WooCommerce catalogue invisible to machines. It is to ensure that useful machine discovery remains dependable while unnecessary processing is controlled.
The Connection to AI Visibility Engineering
At Sydney Business Web, we approach AI visibility as a combination of accessible website information, coherent business identity, corroborating evidence and reliable technical infrastructure.
Our Intelligent Entity Skeleton and AI Credibility Footprint frameworks address how a business and its supporting evidence are represented and connected.
Crawler management addresses a different but complementary requirement: ensuring that important information remains technically retrievable without compromising the systems delivering it.
A website may contain excellent structured data and credible business information, but these assets become less useful to automated retrieval systems when important pages are unavailable or repeatedly time out.
Conversely, making every application endpoint unrestricted does not guarantee better AI visibility. Accessibility must be balanced against processing cost, operational reliability and business value.
Sustainable AI visibility depends on both discoverable information and infrastructure capable of delivering it.
AI Crawling and Large WooCommerce Catalogues
Practical answers about crawler behaviour, server performance and protecting large product catalogues without unnecessarily restricting AI discovery.
Does having 17,000 products automatically make a WooCommerce website slow?
No. Catalogue size alone does not determine performance. The important factors include database query efficiency, caching, product filters, pagination, server capacity and the number of concurrent dynamic requests.
A well-engineered large catalogue can perform efficiently, while a smaller catalogue with expensive queries or unsuitable configuration may experience serious performance problems.
Can AI crawlers overload a large WooCommerce catalogue?
Automated crawlers can contribute to server pressure when they repeatedly request uncached product pages, expensive pagination combinations or dynamic application endpoints.
However, unusual automated traffic should not automatically be attributed to a verified AI crawler. Request behaviour, identity and processing cost must be examined separately.
Should large WooCommerce websites block AI crawlers?
Not indiscriminately. Search and AI discovery systems can provide commercially valuable access to published product and business information.
The better approach is to distinguish crawler purposes, verify identity where necessary, preserve useful discovery and restrict unnecessary or abusive processing.
For example, OAI-SearchBot supports OpenAI's search discovery activities, while GPTBot serves a different purpose. Website owners may reasonably adopt different access policies for those crawlers.
Is robots.txt enough to protect a large product catalogue?
No. Robots.txt communicates crawling instructions to compliant automated clients. It is not an enforced security boundary.
Clients that ignore crawler instructions can still request expensive URLs. Appropriate server-level restrictions, application controls and efficient request handling may also be needed.
These protections should be tested carefully to avoid interfering with genuine shoppers or useful search and AI retrieval.
What should operators monitor when AI crawling increases?
Monitor PHP-FPM worker utilisation, request duration, listen queues, database activity, response times, HTTP 5xx errors and automated request patterns.
Pay particular attention to the processing cost of dynamic requests. A modest number of expensive requests can be more disruptive than a much larger number of efficiently cached page requests.
A server load average is useful context, but it cannot independently identify the application or request responsible for a problem.
Supporting Research and Engineering Context
This investigation forms part of Sydney Business Web's wider work on crawler accessibility, machine retrieval, structured business information and resilient website infrastructure.
Sydney Business Web References
A separate real-world case examining how security rules prevented an AI crawler from successfully retrieving website information.
Sydney Business Web's system for observing machine retrieval, crawler activity and access outcomes on instrumented websites.
Why crawler access and successful retrieval must be measured rather than assumed from website availability.
Our framework for connecting business identity, services, people, content and supporting information into a coherent machine-readable structure.
How accessible business information, coherent identity and corroborating evidence contribute to stronger machine understanding.
External Technical References
Google's guidance on crawl capacity, server response behaviour, URL inventory and efficient crawling of large websites.
Explains how product filters and URL parameters create large crawl spaces and additional server workloads.
Describes the different purposes of OpenAI's crawlers, including OAI-SearchBot and GPTBot, and the controls available to website operators.
Technical reference for PHP-FPM process management, worker limits, slow-request logging and related configuration.
Evidence note: The incident observations and corrective actions described in this article come from direct VPS investigation and testing in November 2026. The references provide supporting technical context and related Sydney Business Web research. They are not presented as independent verification of the incident's unresolved causes.