Block bots and scrapers without blocking Google: a practical guide

Open the server statistics of any website and you'll find that a significant share of requests doesn't come from people. There are search engines, monitoring services, social media previews, feed readers. And then there are the others: bots copying articles and prices, looking for vulnerable pages, trying passwords. The problem isn't blocking bots: it's blocking the right ones.

Good bots and bad bots

The first mistake: trusting the user agent

Every request says who it is with its user agent. The catch is that the requester chooses it: writing "Googlebot" in your user agent is trivial, and it's exactly what many scrapers do, because they know almost no site blocks Google.

So:

• blocking by user agent only works against the most naive bots (those introducing themselves as python-requests or curl);
• trusting anyone claiming to be Googlebot opens the door to anyone dressed up as Google.

How to recognise the real Googlebot

Google and Bing offer two ways to check that a request really comes from them.

1. Reverse DNS verification. Take the IP, look up its name and check that it ends in googlebot.com or google.com (for Bing, search.msn.com); then look up the IP of that name and check it matches the original one.

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

2. Official address lists. Google and Bing publish the IP ranges of their crawlers in regularly updated files. It's the easiest method to automate.

1PrimeCDN Shield downloads Google's and Bing's official lists every day: verified addresses never get checks or blocks, not even with «under attack» mode on. Anyone pretending to be Google without being Google is treated like everyone else.

robots.txt is not protection

The robots.txt file tells bots which pages not to visit. Serious search engines respect it, and so do many AI crawlers. But it's a request, not a block: the bots you really want to stop ignore it, and sometimes read it precisely to discover the pages you'd rather hide.

Use robots.txt to manage indexing, not for security.

How to stop bad bots without hitting people

Look at behaviour, not identity

A bot is recognised by what it does more than by how it introduces itself:

• it makes far more requests per second than a person;
• it only requests HTML pages, never images or CSS;
• it produces lots of errors (missing pages, denied access);
• it uses ever-changing parameters to dodge the cache;
• it tries paths typical of vulnerabilities that don't exist on your site.

1PrimeCDN Shield gives every IP a score based on these behaviours, window after window. Above a threshold the IP goes through a check; beyond that, it's blocked for a limited time that doubles if it keeps going. Scanners, which go slowly to avoid notice, are counted over a longer period.

Use an invisible check, not a CAPTCHA

Image CAPTCHAs lose you customers. 1PrimeCDN's check asks the browser to solve a small calculation: a person sees a screen for about a second, then browses normally for hours. A bot making thousands of requests would have to repeat the calculation for each one, and it stops being worth it.

Limit requests per IP

A per-address requests-per-second limit doesn't bother anyone browsing normally and cripples the most aggressive scrapers. On sensitive pages (login, search, cart) the limit can be tighter.

Block by country only when it makes sense

If your audience is in one country and you get thousands of requests from faraway ones, blocking or checking those countries cuts the noise instantly. Don't do it blindly, though: you might block travelling customers or services you use, such as your payment provider's servers.

Don't forget the bots you need

Before tightening the rules, list the automated services that must reach the site: monitoring, the payment gateway confirming orders, the shipping service, backups, any apps. Add them to the always-allowed addresses.

On 1PrimeCDN your site's origin server is automatically allowed on its own zone: if the site reads its own pages through the CDN, Shield doesn't mistake it for a bot.

In short

1. don't trust the user agent: verify crawlers with the official lists or reverse DNS;
2. judge bots by how they behave;
3. use invisible checks instead of CAPTCHAs;
4. limit requests per IP, more on sensitive pages;
5. keep a list of the automated services that must always get through.

If bots are already causing trouble, read how to tell if your site is under attack or write to us from the emergency page.