Welcome to Sydney Business Web Technical Solutions Problem 6 - Fast Servers, Slow Websites and Bots!
Servers and Bots: Welcome to Sydney Business Web Technical Solutions Problem 6 - Why Is Your Business Website Slow Even with Powerful Hosting?
Your Server May Be Fast, but Bots can slow dowmn EVERY site it runs - and that's Not ALL!!!
Servers and bots: here’s how they can make a “fast” VPS feel slow.!!! - Every day, we solve problems for eCommerce website owners. We had a think about how we might use this activity to help others, and came up with this idea: Every week or three, we'll take the trickiest problem and publish our solution. Today we look at a problem that every online business owner shouild be aware of - that even fast website hosting can be very slow if not managed competently. Recently, we noticed a distinct slowing of siotes on a Virtual Private Server we use (a VPS). We build our sites carefully and the hosting servers we lease are absolutely top quality and managed by me personally. We see the same pattern again and again: servers and bots decide who gets the CPU. So what was the issue? This is a complee breakdown, but as an appetiser, here is how our server utilization looked when we finished:
Taming servers and bots is what keeps headroom consistent!
As you can see, the server is purring along with lots of spare capacity - and this is consistent. In the sites on this server we have two woocommerce sites each serving 17,000 products. Here are various screenshots taken before we resolved the final issue and while we were fixing PHP workers and SQL issues.
The CPU load under pressure was rising to x11 at times under severe bot pressure. This was really a DoS attack! When servers and bots collide, honest traffic loses.
On this shot you see how the cpu is slightly overloaded but the SWAP disk is completely used up. This is a typical 'servers and bots' issue! When swap is used up, the server runs out of both physical memory and its “overflow” disk-based memory, leaving no room to keep processes alive. At that point the kernel can no longer page out inactive memory, so every running process competes directly for scarce RAM. This leads to stalls, thrashing (excessive disk I/O as the system desperately swaps in and out), and eventually the out-of-memory (OOM) killer terminating processes to free space. It usually happens because too many worker processes (like PHP-FPM or MySQL threads) hold more memory than the system has available, or because memory-hungry tasks stack up faster than they are completed, filling RAM and then swap until both are exhausted. at times we saw server load rising t x11 or even x12!!!
Why Is Your Business Website Slow? (Servers and Bots)
Is it the server? A poor site build? Or something else that few business owners ever think of? Even if your website sits on a “fast” 6-CPU VPS with plenty of RAM, it can still crawl. One of the hidden culprits is bots. And here’s the sting: those bots don’t even have to be attacking your site directly. If they’re pounding away at other sites on the same server, they’re draining the same CPU, memory and I/O pool that your site relies on. Good primer on bot load: Cloudflare on bots to learn more about servers and bots.
So Who’s to Blame?
A big part of the problem is amateur hosting providers who don’t actively manage their servers. They’ll sell “fast” plans, but they don’t watch what’s really happening behind the scenes. Uncontrolled bots, noisy neighbour sites, unthrottled cron jobs — all of these can turn a well-built site into a slow, unreliable one. That’s the difference between hobby hosting and professional management.
What Makes Sydney Business Web Different?
At Sydney Business Web, we don’t just throw hardware at the problem and hope it holds. We ask the right questions:
- Is your slowdown caused by the server itself?
- Is it poor build or unoptimised code?
- Or is it the bots and background processes that others ignore?
By digging deeper, managing resources properly, and using the right tools and bots ourselves, we keep business websites stable, fast, and resilient — even when others on the same hardware stumble.
Stage One: The First Suspicions
Our first thought was simple: could it be the big store? With more than 17,000 products in WooCommerce, it seemed obvious that one customer’s site was dragging the whole VPS down. But was that really the case — or just an easy scapegoat?
Stage Two: Checking the Obvious
We started where anyone would: the server dashboard. CPU spikes, memory usage, swap filling up — all looked suspicious. So we asked the next questions:
- Was PHP-FPM being overloaded by too many simultaneous requests?
- Were MySQL queries piling up, locking tables and starving other sites?
- Was the swap file covering for memory leaks and slowing disk I/O to a crawl?
- Were cron jobs firing all at once, causing sudden surges in load?
Each test gave us clues — but never the final answer. Something else was at play.
Stage Three: Down the Rabbit Holes of Servers and Bots
If not “just the big store,” then what? We dug in — one subsystem at a time.
Could it be PHP-FPM?
- Queue depth rising while workers idle on I/O — classic starvation.
- Per-site pools competing for RAM; pm.max_children too generous for shared memory.
- Slow logs flagging a few endpoints taking the whole pool hostage.
Could it be MySQL?
- Bursty SELECT * ... ORDER BY without supporting indexes.
- Table-level contention from admin screens and batch jobs colliding.
- Buffer pool not sized for hot sets; I/O wait creeping up.
Could it be Cron Collisions?
- Multiple sites scheduling “every minute” tasks — a thundering herd.
- Long-running jobs overlapping with themselves and with backups.
Could it be Cache/CDN Weirdness?
- After deploy, inconsistent variants: some users fast, others hit stale or 404 edges.
- Cache purge fixed symptoms — which hinted at coordination issues, not raw CPU limits.
Each fix helped — none explained the recurring slowdowns across sites.
Stage Four: Following the Noise and Finding the Bots
When database, PHP, and cron tuning didn’t explain the slowdowns, we asked the next question: who is actually using the CPUs?
The answer came from the logs. Access samples showed sharp, regular spikes that had nothing to do with customer activity. The user-agents told the story: heavy crawlers, scrapers, and “harmless” bots. The traffic patterns were too uniform, too relentless, and — most important — they weren’t even hitting our big 17,000-product store. They were pounding other sites on the same VPS, and consuming the same pool of CPU, RAM, and I/O that our client depended on.
Measuring the Impact
- Access log analysis: IP and user-agent tracking revealed spikes every 5–15 minutes, aligned with crawler sweeps.
- Per-vhost profiling: noisy neighbours correlated perfectly with our customer’s slowdowns.
- Correlation tests: throttling those vhosts instantly restored normal response times on the store.
Blocking and Controlling the Bots
- Robots rules and CDN filters: friendly crawlers were rate-limited, aggressive ones were blocked or challenged.
- WAF rules: bad actors were filtered before they could reach PHP or MySQL.
- Resource guardrails: PHP-FPM and cron caps ensured one site couldn’t collapse the whole server under bot load.
That was the breakthrough. The problem wasn’t “just the big store” at all — it was unmanaged bots hammering neighbour sites. Once we measured and controlled them, the 6-CPU VPS finally behaved like the fast machine it was meant to be.
Stage Five: Proving the Real Cause
Once the bots were identified and controls were in place, the proof came quickly. We didn’t need theory — we needed results.
- Latency normalised: page loads on the 17,000-product store dropped back to expected response times.
- Stability returned: no more random stalls or unexplained 404s during crawler surges.
- CPU behaviour: instead of chaotic spikes, utilisation tracked predictably with real customer traffic.
In short, the evidence was undeniable: it wasn’t “too many products” or “too little hardware.” The root cause was unmanaged bot traffic on neighbour sites. By controlling that factor, the server — and the businesses depending on it — performed the way they should.
Stage Six: The Fix That Stuck
Finding the bots was only half the battle. The real test was: could we stop the problem coming back? Quick wins don’t mean much if the same issues creep back a week later. We needed lasting guardrails.
What We Put in Place
- Bot controls: rate-limits and firewall rules to stop “friendly” crawlers from becoming unfriendly, and block bad actors outright.
- Resource guardrails: per-site PHP-FPM and cron caps so one vhost couldn’t monopolise CPU, RAM, or I/O again.
- Cache hygiene: aligned CDN and application cache TTLs with surgical purges after deploys — no more stale or conflicting variants.
- Database tuning: composite indexes and query clean-ups that removed pressure points discovered during the hunt.
- Monitoring bots of our own: small scripts watching latency, traffic spikes, and job runtimes so we knew about trouble before customers did.
These weren’t glamorous fixes — but they were solid, technical measures that turned a fragile, overloaded VPS into a reliable workhorse. The difference wasn’t in buying more hardware; it was in managing what we had with discipline. That’s where professional hosting and site management pays off — and where amateur providers so often fall short.
Stage Seven: The Takeaway for Business Owners
So what does this mean if you run a business website? It means you need to ask harder questions than “how many CPUs do I get?” or “is it the cheapest plan?” Because a “fast” server on paper can still run a slow site in practice — and that can quietly kill your online business.
Questions Every Business Owner Should Ask
- Is my slowdown really the server, or is it noisy neighbours and their bots?
- Does my hosting provider actively manage PHP, MySQL, cron, and crawler load? Or do they just sell plans and hope?
- Are bots being measured and controlled? Or are they free to hammer away until customers give up waiting?
The Warning
Cheap or amateur hosting never tells you this part of the story. They’ll promise “fast servers” and “unlimited plans” while ignoring the very things that slow your site to a crawl. They don’t manage the resource balancing, the cron storms, the crawling bots — and you’re the one who pays when your customers hit the back button.
The Difference with Sydney Business Web
At Sydney Business Web, we manage the whole picture. We don’t just throw hardware at problems; we control the variables that make websites truly fast and reliable. That’s what separates professional business hosting from amateur promises.
Don’t Let Cheap Hosting Kill Your Business
If your website feels slow, don’t assume you just need a “bigger server.” The real danger often comes from unmanaged bots, noisy neighbours, and amateur hosting providers who never look below the surface. That’s how businesses lose customers without ever knowing why.
At Sydney Business Web, we dig deeper. We find the real cause, fix it properly, and keep it fixed — so your website stays fast, stable, and ready to win business.
Before cheap hosting quietly kills your online presence, talk to us. We’ll make sure your site performs the way your customers expect, not the way amateurs hope.
As a final note, a poor server can also badly affect your Email deliverability.Because raw server speed isn’t the only factor. Poor management, noisy neighbours, and unmanaged bot traffic can make even powerful servers crawl. or of course it could just be a 'Home Made' site designed by somebody that has no grasp of the technical aspects of web design - and there are m,illions opf those lol!
A noisy neighbour is another site on the same server that consumes excessive resources, often due to traffic spikes, bad code, or bot activity. This affects your site too.
Yes. Bots don’t have to attack your site directly. If they hammer other sites on the same server, they still drain CPU, RAM, and I/O resources your site depends on
Not always. In many cases, the issue is poor configuration, unoptimised code, or unmanaged bots. Upgrading blindly can waste money without solving the root cause.
We don’t just build and hand over sites. For clients on our managed servers, we actively monitor PHP, MySQL, cron jobs, and bot traffic to ensure consistent speed.
We only provide hosting for our website clients. Our focus is building, managing, and optimising business websites — hosting is part of the package, not a standalone service.
CONTACT SYDNEY BUSINESS WEB NOW!
