Crawler Overload and DDoS

What crawler overload is

Crawler overload is what happens when sustained request volume pushes a site into slow responses or errors, and search engine crawling reacts to that unavailability. The volume can be an application-layer denial-of-service attack - a flood of ordinary-looking HTTP requests, as opposed to a network-layer bandwidth flood - or an abusive scraper, or a botnet hammering the most expensive URLs on the site. The target is not a ranking algorithm. The target is availability, and Google's crawling infrastructure reacts to availability in a documented, deterministic way.

Four terms, defined once so the rest of the page can move quickly:

• HTTP - the request-and-response protocol a browser or crawler uses to ask a server, the machine hosting your website, for a specific URL, the address of one page or file on it.
• Application-layer, or Layer 7, flood - the variant that matters here, because it exhausts application and database resources at volumes far too low to saturate a network connection. A few hundred requests a second aimed at an uncached search page can take down a site a bandwidth attack would not touch.
• Crawl capacity limit - the ceiling Google places on how much it will fetch from a given host, set by that host's observed health.
• CDN - a content delivery network, a layer of distributed cache servers placed in front of your origin so that most requests are answered without your own machine being consulted at all.

Most websites that suffer this are not chosen for their prominence. They are chosen, or simply found, for being reachable and slow.

The threshold, because it is the whole page

The difference between "ignore this" and "act now" is duration, and Google has published enough to draw the line with some precision.

On crawl rate. Google states that if a site slows down or responds with server errors or rate-limiting signals, the crawl capacity limit goes down and Google crawls less; if the site responds consistently and its response times stay stable or improve, the limit goes back up. That part is self-correcting and carries no lasting cost.

On individual status codes. A 429 - "Too Many Requests", the standard rate-limiting response - is "treated as a signal that the server is overloaded, and it's considered a server error". For a 500, "Google decreases the crawl rate for the site. The decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error", and 503 is handled the same way. Then the sentence that matters most on this page: "For Google Search, Google's indexing pipeline removes from the index URLs that persistently return a server error." (Google, How HTTP status codes, and network and DNS errors, affect Google Search, last updated 4 February 2026.)

On duration. The clearest number comes from the page that tells owners how to deliberately slow Google down. Google's crawling infrastructure reduces a site's crawl rate when it meets a significant number of URLs returning 500, 503 or 429 - followed by the warning: "We don't recommend that you do this for a long period of time (meaning, longer than 1-2 days) as it may have a negative effect on how your site appears in Google products", and, if those codes persist for multiple days on the same URL, "the URL may be dropped from Google's index" (Google, Reduce the Google crawl rate, last updated 18 December 2025).

John Mueller of Google put the same boundary in plainer words in a Search Central office-hours session recorded 10 December 2021, asked how long a 503 keeps working as a temporary signal: "after a couple of days we think this is a permanent result code, and we think your pages are just gone", adding that recovery from an extended outage brings "really strong fluctuations" before rankings settle (reported by Search Engine Journal, 13 December 2021).

So the operative reading, and I would put it on a sticky note: crawl rate falls within hours to days of sustained errors and recovers on its own; index removal follows after roughly a couple of days of persistence, and that part does not repair itself on the same timescale. A four-hour outage is an operations incident. A four-day outage is a search incident. Google gives no precise hours-to-deindexing figure, and neither will I - "longer than 1-2 days" and "a couple of days" are the only durations on the record.

Nothing to devalue, and nothing to appeal

Most of the attacks catalogued on this site are attempts to make a ranking system draw a false conclusion from something happening on somebody else's server. Since Penguin 4.0 in 2016 the answer to that class of attack has been devaluation: Google's stated behavior is to ignore unearned links rather than act on the site they point at. The attacker's message is delivered to a system that has stopped reading it.

There is nothing equivalent here, because there is nothing to devalue. Google cannot rank a page it cannot fetch, and no amount of algorithmic skepticism changes that. The attacker is not asking Google to believe anything. They are triggering a published behavior of Google's own systems on purpose, and the instrument is your server.

That is the pattern this reference keeps arriving at from different directions. The attacks that genuinely work are the ones that reach your own infrastructure, not the ones that argue with a link algorithm. Injected spam works because Google correctly attributes your server's output to you. A malware flag works because it measures what your server hands a browser. Overload works because it stops your server answering. In all three, Google is behaving exactly as designed - which is precisely why there is no appeal, no form, and no penalty to lift.

Are you actually being attacked?

Most incidents that arrive described as "we were DDoSed and it killed our rankings" turn out not to be attacks, and starting from the assumption of sabotage wastes the first and most valuable day. Rule these out first:

• An accidentally infinite URL space. Calendars that generate a page per day forever, faceted filters that combine into millions of permutations, session parameters appended to every link. Genuine Googlebot then crawls it hard, and the load looks exactly like an attack because in effect it is one - self-inflicted.
• A firewall or CDN rule that started returning 403 to Googlebot. This produces a crawl collapse with no traffic event at all, which is why it is so often misdiagnosed as a penalty. It was Mueller's own first suggestion when a site owner reported a sudden crawl drop, in an exchange on Reddit reported by Search Engine Journal on 15 August 2025.
• A bad deploy, a saturating scheduled job, or a noisy neighbor on shared hosting.

In that same 2025 exchange Mueller was specific about which signals produce a fast crawl reaction: "I'd only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts." He also noted that 404s alone do not cause a rapid crawl drop, and that crawl rates normalize on their own once server problems resolve - though with "no defined time" for it. If your errors are 404s, you are looking at a different problem.

What the evidence looks like

The Crawl stats report in Search Console, under Settings, is the primary instrument, and it is the one exhibit that speaks for both the technical and the commercial side of an incident. It reports the total number of crawl requests issued for URLs on your site, whether successful or not; total download size; average response time for all resources fetched; a host status indicator; and crawl responses broken down by type, including 5xx and timeouts, with a split by Googlebot type.

Host status covers three things - robots.txt fetching, DNS resolution and server connectivity - and is green when there have been no significant issues in 90 days, amber when problems were resolved more than a week ago, and red for problems in the last week. A red host status, a rising 5xx share and a rising average response time, all aligned to the same window, is the evidence. Nothing else you can produce is as persuasive to a client, an insurer or a court, because it is Google's own record rather than yours.

Alongside it:

• The Pages report in Search Console. Watch for URLs moving into "Server error (5xx)" and, later, out of the indexed set entirely. That second movement is the deindexing arriving.
• Your own server metrics - requests per second, error rate, and 95th-percentile response time - plotted against the same window. Overlaying your response-time curve on the average response time in Crawl stats is the single most convincing exhibit available.
• Access logs. Look for concentration on a handful of expensive endpoints rather than a normal distribution across the site; source addresses spread thinly across many networks; and one useful tell - requests for HTML with no accompanying requests for that page's CSS, JavaScript or images. A flood does not render pages. A real browser always fetches the assets.
• External uptime monitoring with per-region checks, so you know what the outside world saw rather than what your dashboard says.

Getting it back: availability first

The sequence here is unusual, because the search remedy and the operations remedy are the same thing and the operations one comes first.

1. Restore availability. That is the entire SEO remedy. Crawl rate recovers on its own once the server answers reliably - Mueller's word is that it normalizes, with no defined time. Nothing you can do inside Search Console substitutes for a server that responds.
2. Put a CDN and a caching layer in front of the origin, and cache aggressively enough that the flood never reaches the database. Rate-limit the expensive endpoints specifically rather than throttling the whole site. This is also the permanent fix: a site whose pages are served from cache is not a viable target.
3. Verify Googlebot before you write any bot rule. Every mitigation control you switch on is a chance to block the real crawler. Use reverse-then-forward DNS or Google's published address ranges, and exempt verified crawlers from challenges and rate limits.
4. If you must shed load, do it with 503 and a Retry-After header, and watch the clock. Google's own ceiling is longer than 1-2 days, and past that, in Mueller's words, your pages are just gone. A 503 is a short-term instrument and never a posture.
5. Never serve a 200 "site under attack" holding page. A soft error returns success, so Google indexes the apology in place of the product page, and that damage outlives the incident by weeks.
6. Do not add noindex or a robots.txt disallow during an incident. Both take longer to reverse than the outage lasts, and a robots.txt block stops Google discovering that you have recovered.
7. If the excess crawling turns out to be genuine Googlebot, Google's documented channel is a special request through Search Console reporting an unusually high crawl rate and specifying your optimal rate. That form exists for the real crawler, not for attacks.
8. After recovery, watch host status return to green, and expect ranking movement before things settle. Mueller described really strong fluctuations following extended downtime; that is the shape of a normal recovery, not evidence of a penalty.

Doing nothing is frequently correct. If availability came back within hours, host status never went red and the 5xx share stayed low, there is probably nothing to fix and no lasting effect to recover from. Short spikes are absorbed by design. The urge to "do something" after a brief incident is exactly how sites end up with a forgotten noindex in a template six months later.

What does not help: disavow files; reconsideration requests, since no manual action exists here; rewriting content; migrating to a new domain; and blocking by user agent or by country.

Legal recourse - the one category that gets prosecuted

The federal hook in the United States is the Computer Fraud and Abuse Act, 18 U.S.C. 1030, and specifically 1030(a)(5), which reaches knowingly causing the transmission of a program, information, code or command and intentionally causing damage to a protected computer. State computer-crime statutes generally parallel it.

Unlike most of the vectors on this site, this is a category prosecutors actually pursue. The Department of Justice has repeatedly charged operators of attack-for-hire services and seized their domains - for example, the U.S. Attorney's Office for the Central District of California announced on 8 May 2023 the seizure of 13 internet domains associated with services that offered denial-of-service attacks for hire, describing attacks that flood targeted computers with information and prevent them from accessing the internet, and stating that hundreds of thousands of registered users had used such services to launch millions of attacks against millions of victims. Charges included conspiracy and the Computer Fraud and Abuse Act. I describe the category deliberately and name no service; naming one is a referral, and this site does not make them.

The practical reporting route for a victim in the United States is the FBI's Internet Crime Complaint Center. Your host and CDN matter more in the short term: abuse reporting, mitigation, and above all log preservation before rotation. Those logs are the only record of the attack window, and they are what later proves to a client, an insurer or a court that a ranking loss had an external cause.

Set expectations honestly. Attribution to an individual is rare. And Google offers no remedy at all - there is no form that restores rankings lost to downtime, because nothing was penalized. The recovery is technical, and it is finished when your server has been answering reliably for long enough that Google has recrawled everything.

Where owners make the damage permanent

• Blocking Googlebot along with the flood, usually through a blanket firewall challenge or a user-agent rule, which converts a temporary crawl slowdown into a sustained crawl collapse. This is the most expensive mistake on the page and it is made under pressure, at 2am, by someone trying to help.
• Leaving a 503 up for days because it stopped the load, well past the point where Google reads it as permanent.
• Serving a 200 status on a holding page, so the outage itself gets indexed.
• Adding noindex or a robots.txt disallow during the incident and forgetting to remove it - the single most common cause of a recovery that never arrives.
• Redirecting the whole site to a status page on another domain.
• Diagnosing the crawl drop as a penalty and spending three months on content quality when the cause was a firewall rule.
• Letting logs rotate away before the attack window is preserved, which forecloses both the forensic and the legal question.
• Assuming an attack without ruling out a URL space the site made infinite by accident.

My work on incidents like this is the search half, not the mitigation half: establishing from Crawl stats and logs what Google actually experienced and when, distinguishing a crawl collapse caused by errors from one caused by a blocking rule, and rebuilding indexing after the outage ends. I have been working in search since 1996 and the pattern has not changed - the sites that recover quickly are the ones that resisted the temptation to reach for a search-side lever during the incident.

Frequently asked questions