What is Cloudflare and how to bypass it when web scraping?

Cloudlfare Anti-Bot Protection is a popular anti-bot protection service that is used by many websites to prevent automated access.

Much of automated web access can be malicious though WAF systems like Cloudflare also block harmless web scraper bots.

In this quick intro, we'll take a look at how Cloudflare WAF works and how to bypass it when web scraping.

What is Cloudflare WAF?

Cloudflare offers a Web Application Firewall service that is middleware between the website and the user.

This service determines which incoming connections come from bots and which come from humans.

Web scrapers while being mostly harmless are automated bots and get caught in this indifferent filter.

There is one major thing that differentiates harmful bots and scrapers though - the interaction with the website.

Web scrapers just collect public data and don't interact with the website's direct elements so it's possible to create web scrapers that bypass Cloudflare anti-bot protection. For that, we need to take a look at how Cloudflare works.

How to bypass Cloudflare?

Bypassing Cloudflare when web scraping involves many different techniques and resources that are already implemented by most web scraping APIs.

Here are the best web scraping APIs for Cloudflare anti-bot protected targets.

• indeed.com
• stockx.com

	Service	Success %	Speed	Cost $/1000	🔗
1	Scrapingdog	100% =	7.5s +0.1	$2.2 =	code
2	Scrapfly	99% -1	3.0s -1.7	$3.81 +=	code
3	WebScrapingAPI	99% +1	12.1s -1.6	$2.71 =	code
4	Scraperapi	98% +31	6.3s -0.7	$9.8 =	code
5	Zenrows	81% -6	3.9s -0.2	$2.76 =	code
6	Scrapingbee	7% +3	1.7s -0.5	$3.27 =	code
7	Scrapingant	0%	-	-	code

Data range Nov 11 - Nov 18

Job listing pages on Indeed.com

more on indeed.com

	Service	Success %	Speed	Cost $/1000	🔗
1	Zenrows	100% =	6.0s +1.0	$6.9 =	code
2	Scrapingdog	100% +1	8.8s +0.7	$2.2 =	code
3	Scraperapi	99% -1	5.9s -1.2	$9.8 =	code
4	Scrapfly	96% =	1.7s -0.7	$0.15 =	code
5	WebScrapingAPI	84% +4	38.1s -2.3	$2.71 =	code
6	Scrapingant	84% +2	38.2s -1.0	$1.9 =	code
7	Scrapingbee	56% +5	2.0s -0.2	$3.27 =	code

Data range Nov 11 - Nov 18

Stockx.com product and other public pages

more on stockx.com

However, to bypass Cloudflare WAF manually the web scraper needs to be fortified with a few key features that resist Cloudflare identification methods and to implement that we must really take a look at how Cloudflare works.

How does the Cloudflare anti-bot works?

Cloudflare inspects each incoming connection and generates a trust score which determines the likelihood of the connection of coming from a real human being.

To generate this score Cloudflare is using various fingerprinting and data point metrics. Let's take a quick look at some.

IP Address

The first metric is the IP address of the connecting client and each client has one.

There's a limited amount of IPs on the internet and they each has distinct features that let Cloudflare to assign probabilities to them.

🗺️🗺️🗺️

For example, a user connecting from a home internet connection is significantly less likely to be a robot than a user connecting from a datacenter.

With this IP's are separated into several categories:

• Datacenter - assigned to all data centers like AWS, Google Cloud hosts and so on.
• Residential - assigned to all home connections.
• Mobile - assigned to mobile cell towers, satelites and so on.

To combat IP Address analysis, scrapers should high quality use residential or mobile proxies that haven't been identified by Cloudflare yet.

Javascript fingerprinting and challenges

The second metric is the client's ability to execute Javascript. As most bots don't execute javascript an easy way to identify them is to serve a javascript challenge.

These challenges are often simple mathematical puzzles that use token distributed at other parts of the website. Reverse engineering this behavior can be tricky without using a real web browser.

To solve javascript challenges scrapers need to use real web browsers through headless browser automation. Most commonly through libraries like Puppeteer, Playwright or Selenium.

👣👣👣

Headless browsers can be identified through javascript fingerprinting techniques as they are just slightly different from user browsers in many different ways. Tiny details add up to a full evaluation profile which can be used to identify robots very successfully.

To combat fingerprinting, headless browsers need to be patched with fingerprint resistance and randomization.

HTTP fingerprinting and analysis

Real user web browsers browse and connect in a few different ways that can be used to identify robot connections.

Most robots still use HTTP1.1 connections which are not sure by real browsers at all in 2024. Meaning, scrapers should use newer versions of the HTTP2 or HTTP3 protocol.

🔎🔎🔎

While advanced HTTP clients like libcurl support HTTP2 pretty well they are susceptible to HTTP2 fingerprinting. This type of fingerprinting measure slight differences between the HTTP2 implementation of the client and the real browser.

Another fingerprinting technique related to HTTP is TLS fingerprinting. In this type of fingerprint, the TLS handshake is analyzed for slight differences between the client and the real browser.

To combat HTTP and TLS fingerprinting, scrapers need to use advanced HTTP2 capable clients that are TLS and HTTP fingerprint resistant.

Behavior and technical analysis

While scrapers usually just collect data without interacting with the website, they can still be identified through their behavior.

Most commonly this is done through scraper implementation mistakes that just don't happen with real users. Some examples:

• Forgetting custom request headers.
• Sending requests in different formats or encodings.
• Missing expected cookies or browser-specific headers like User-Agent.
• Visiting pages that aren't visited by real users which are known as honey pots.

These slight irregularities are tracked by Cloudflare in its trust score calculation so, scrapers need to be diligent and replicate requests as close to real user behavior as possible.

🪤🪤🪤

Cloudflare logs every connection and use behavior and pattern analysis to identify robots. This means that scrapers that behave in an unusual pattern, connect in bursts or have any connection irregularities can be identified through AI-based analysis.

To combat behavior analysis, scrapers need to implement a human-like behavior pattern that is indistinguishable from a real user.

What are some websites that use Cloudflare?

Cloudflare is by far the most widely adopted anti-bot on the web and there are many popular websites encountered in web scraping that use it. Here's a quick list:

• Stockx.com
• Indeed.com
• G2.com
• Glassdoor.com
• Hapag-lloyd.com
• Instacart.com
• Kickstarter.com
• Zoopla.co.uk

And many others, though, as most big targets often rotate and use multiple anti-bot technologies based on needs and performances.

Many other targets implement Cloudflare anti-bot protection temporarily when experiencing high traffic or bot attacks which affects web scraping as well.

Summary

Cloudflare uses a variety of fingerprinting and analysis techniques to identify robots and scrapers by generating a trust score. This trust score can be raised and lowered through web scraper enhancement techniques.

Unfortunately for web scraping, Cloudflare is constantly evolving and improving its anti-bot protection so it's a constant battle to keep up with it.

For this we highly recommend using a web scraping API service that is constantly updated and maintained - see the benchmarks for the most up-to-date results!