Bot Protection for Your Website: What to Deploy and How to Implement It When You're Being Scraped

Introduction: When Your Website Becomes Someone Else's Database

Picture this: you spent six months building a catalog, verifying prices, writing product descriptions, and fine-tuning filters. Then you notice a competitor updating their prices twenty minutes after you do, and clone sites with your descriptions word for word are popping up in search results. Meanwhile, your server is groaning under the load, your hosting bill is climbing, and your analytics show strange traffic spikes at night from regions where you don't have a single customer. Sound familiar? That's scraping. And it's not a hypothetical threat — it's a daily reality for any website that has anything of value on it.

According to industry reports from recent years, automated traffic accounts for roughly half of all requests on the internet, and about a third of the total volume comes from outright malicious bots. For e-commerce, classifieds, travel projects, and content platforms, the share can be even higher. And the 2026 trend is obvious: classic scrapers have been joined by large language model-based agents that browse websites just like humans, read content, and make decisions. The line between a "real" and an "automated" visitor is blurring before our eyes.

The mobileproxy.space audience knows this topic from both sides. Marketers and affiliate specialists collect data themselves, developers write scrapers, and business owners monitor competitors. But each of you almost certainly has your own website, landing page, catalog, or API that someone is already pulling data from right now. So in this article, we're switching perspectives: today you're the one being scraped, and we're breaking down how to build anti-bot protection that actually works without getting in the way of real users.

One important caveat. We're not comparing vendors or ranking commercial services — that's covered in a separate blog post. Here we're talking about implementation: what levels of protection exist, in what order to deploy them, how to measure the impact, and what pitfalls await at every step. Here's what you'll get by the end:

• an understanding of how modern anti-bot systems "see" a visitor and why IP blocking stopped being a solution long ago;
• a diagnostic methodology: how to figure out in a single day who's scraping you and how;
• three lines of defense with step-by-step instructions — from server-side rate limiting to behavioral analysis and data traps;
• a phased implementation framework that won't tank your conversion rate;
• checklists, a tool list, and real case studies with numbers.

Get comfortable. This is a big article, but it's designed as a working bookmark you'll come back to at every stage of the project.

The Basics: What Anti-Bot Protection Is and Who It Protects Against

Bots Come in Many Flavors, and Not All of Them Need to Be Blocked

The first thing to accept: "bot" is not a dirty word. Search engine crawlers index your pages and bring you traffic. Uptime monitors check that your site is alive. Payment systems fire webhooks. Partner aggregators pull feeds under contract. All of this is automated traffic that you need and benefit from. The goal of anti-bot protection is not to destroy automation as a phenomenon, but to separate useful and permissible requests from harmful ones — and make the harmful ones economically pointless.

Undesirable bots are typically divided into several categories:

• Content and price scrapers — they pull your catalog, descriptions, stock levels, prices, and reviews. The main villain of this article.
• Aggregators without a contract — they take your data and sell access to third parties.
• Credential stuffing bots — they attack login forms, testing leaked username-password pairs.
• Scalpers and stock snipers — they buy up scarce products or book slots in fractions of a second.
• Spam bots and inflators — they flood forms, register fake accounts, and skew your metrics.
• AI model crawlers — they collect text for training or for real-time responses. Each business decides for itself whether these are acceptable.

Note that scraping itself rarely breaks a site. But it creates parasitic load (sometimes up to 60-80% of all requests to your catalog), distorts analytics, exhausts the limits of external APIs you call during page rendering, and most importantly, robs you of the competitive advantage you collected that data for in the first place.

How Scraping Protection Differs from DDoS Protection

Many site owners equate these two tasks and are surprised when their anti-DDoS filter lets a scraper through. The difference is fundamental. DDoS is a volume attack: the attacker's goal is to exhaust your bandwidth or server resources. Such an attack is easy to spot in metrics and is filtered at the network level. A scraper, by contrast, tries to be invisible: it spreads requests over time, distributes them across tens of thousands of addresses, imitates a browser, and visits the same URLs as real people. A classic DDoS filter sees smooth, "healthy" traffic and lets it through. That's why bot protection requires fundamentally different signals — we'll get to those in the next section.

Three Lines of Defense: The Model We'll Use

To avoid drowning in technology, it helps to picture anti-bot protection as three concentric layers. The analogy is a medieval fortress: a moat and outer wall, a gate with guards, and finally the inner chambers where even a guest who made it through the gate is under surveillance.

1. Network and server level. Rate limiting, network reputation filtering, WAF, basic header rules. Cheap, fast, stops the most primitive bots.
2. Client verification level. JavaScript challenges, TLS and browser fingerprint analysis, captcha where appropriate, proof-of-work tasks. Stops scripts without a browser and poorly configured headless tools.
3. Behavioral level and data protection. Action sequence analysis, session scoring, honeypot traps, obfuscation, and watermarks in the data itself. Works against advanced bots that made it through the first two rounds.

The key idea we'll keep coming back to: no single layer is sufficient on its own. The strength lies in forcing the bot to bypass all three simultaneously, and each additional obstacle raises the cost of its work. When scraping you becomes more expensive than buying data legally or collecting it another way, scraping stops. It's economics, not magic.

Basic Terms You'll Encounter Below

• Rate limiting — limiting the number of requests from a single source per unit of time.
• WAF (Web Application Firewall) — an application-level firewall that filters HTTP requests by rules.
• Fingerprint — a set of client characteristics that allows you to identify its type or specific instance.
• Headless browser — a full browser without a graphical interface, controlled programmatically.
• Challenge — a verification task the client must solve to gain access.
• False positive — a situation where the protection blocks a real user.
• Honeypot — a trap: an element invisible to humans but noticeable to automated clients.

Deep Dive: How a Modern Anti-Bot System "Sees" a Visitor

To deploy protection consciously, you need to understand what signals it relies on. This is where things get interesting: over the past five years, the industry has gone from "ban the IP" to multi-layered scoring, where decisions are made based on hundreds of attributes. Let's break down the main groups of signals.

Network Signals and Why IP Is No Longer the Main One

Historically, the first sign of a bot was the IP address. Requests from a data center? Probably a script. A thousand requests from one address? Definitely a script. This approach still works against lazy scrapers that hit you directly from cloud servers, and it's worth applying. But it has two fundamental limitations.

First: residential and mobile addresses. The readers of this blog know better than most that a single mobile carrier IP can have several thousand real subscribers behind it thanks to NAT. By blocking such an address because of one suspicious client, you cut off an entire city district from your site. Reputation databases for mobile networks are nearly useless: an address a scraper used yesterday belongs to your customer today. That's why smart anti-bot protection uses IP only as one weak signal, not as grounds for a verdict.

Second: distribution. A modern scraper rotates addresses so that each one gets 2-3 requests per hour. The rate limiting threshold sufficient to cut off such a client will inevitably hit real people who open ten catalog tabs.

So what's still useful at the network level? ASN signals (whether a network belongs to a hosting provider), geographic anomalies (traffic from a country where you don't operate), mismatches between IP geolocation and browser language, and statistics by subnet rather than by individual addresses.

Transport and Protocol-Level Fingerprints

Here's a signal many underestimate. When a client establishes a secure connection, it sends a hello message with a list of supported ciphers, extensions, and their order. This set depends on the library the client uses. Chrome on Windows forms one hello, Firefox another, and a Python HTTP library a third, completely unlike a browser's. The hash of this set is called a TLS fingerprint (common schemes are JA3 and the newer JA4).

The magic is that a scraper can fake the User-Agent header in a second, but faking a TLS fingerprint to match the claimed browser is significantly harder. If a client claims to be the latest Chrome but its TLS hello matches a console library, we're looking at a script. Similar fingerprints exist for HTTP/2 (the order and values of SETTINGS frames, stream priorities) and HTTP/3. The combination of "claimed browser + TLS + HTTP/2" already gives decent classification accuracy without a single line of JavaScript.

Browser Fingerprint and Automation Signs

When a client executes JavaScript, the analysis field expands many times over. The protection script collects hundreds of parameters: screen resolution and its ratio to window size, font list, the result of rendering graphics via Canvas and WebGL, audio context characteristics, timezone, language settings, and the presence and behavior of environment-specific objects. Automation signs are checked separately: properties that headless mode leaves in the environment, mismatches between the claimed platform and actual capabilities, and values that are too "perfect" where a real device always has noise.

It's important to understand: the goal is not to definitively identify a human (that's both legally sensitive and technically unreliable), but to find internal contradictions. A real iPhone doesn't report 32 logical processor cores. A real Chrome on Android doesn't have a 1920x1080 screen resolution with no touch input. Each contradiction is a plus to the suspicion score.

Behavioral Signals

The most forgery-resistant class of signals is behavior over time. A human moves the cursor with microscopic jitters and accelerations, scrolls unevenly, pauses to read, and clicks not on the exact center of a button. A human lands on the homepage, goes to a category, looks at three or four products, then goes back. A bot opens five hundred product cards in a row by ascending ID, each exactly 1.2 seconds apart, never loads images, and never requests CSS. Even the most advanced scrapers on managed browsers give themselves away through session structure: it's impossible to dump an entire catalog while behaving like a single shopper.

Scoring Instead of Black-and-White Decisions

Modern systems don't answer the question "bot or human" in binary. They assign each request or session a risk score, and then a policy is applied: low risk — let through, medium — show an invisible challenge or slow the response, high — captcha or block, critical — serve plausible but distorted data. This approach dramatically reduces false positives: a questionable client isn't shown the door but given a chance to prove it's alive.

What's Changing in 2026

Several trends already shaping protection design:

• AI agents as a new class of clients. LLM-based agents control real browsers, read pages, and behave almost like humans. Some are useful (they bring buyers), others are just smarter scrapers. Sites will have to formulate explicit policies for this kind of traffic.
• Standardization of "ethical" bot identification. Mechanisms are being developed where a legitimate automated client cryptographically confirms its affiliation with a specific company. This allows useful bots to be let through without challenges.
• Device attestation and trust tokens. Operating systems and browsers are experimenting with mechanisms that let a site get confirmation that "this is a real device with a real human" without collecting fingerprints. In the long run, this will simplify protection, but coverage is still limited.
• Growth of HTTP/3 and metadata encryption. Some familiar network signals are becoming less accessible, shifting the emphasis to behavioral analysis.
• Regulatory pressure on user data collection. Browser fingerprints and session storage require careful attention to personal data laws. Protection must be designed to collect the minimum necessary.

The takeaway from this section is simple and important: reliable scraping protection is built on a combination of weak signals, not on one "silver bullet" attribute. Keep this in mind as we move to practice.

Method 1. Diagnostics: Finding Out Who's Scraping You and How

Deploying protection blindly is a surefire way to waste your budget and block the wrong people. Before implementing anything, spend a day or two on reconnaissance. The good news: you don't need expensive tools — web server logs and the ability to ask them the right questions are enough.

Step 1. Collect Complete Logs for a Representative Period

Minimum seven days to capture the weekly cycle. Logs should include: time, IP, request method and path, response code, response size, User-Agent, Referer, and processing time. If your web server only writes the standard format, expand it. For a project with millions of requests per day, it's convenient to dump logs straight into an analytics database or at least a tabular format.

Step 2. Find Anomalies in the Traffic Structure

Ask your logs the following questions and record the answers — this will be your baseline.

• Distribution by hour. Real traffic has a daily cycle. A flat "shelf" at night or a spike at 4 AM is the first marker.
• Resource type ratio. How many requests to HTML pages are there per request to CSS, JS, and images? For a browser, this ratio is roughly 1 to 10-30. For a scraper — 1 to 0. Find the addresses and subnets that only request HTML or only JSON.
• Crawl depth. Who opens more than 100 product cards per session? Who sequentially iterates through IDs, pagination pages, and filter combinations?
• Headers. Outdated browser versions, empty or library User-Agents, missing Accept-Language, the same User-Agent across thousands of addresses.
• Response codes. Mass 404s on non-existent IDs are a sign of enumeration. Mass 429s or 403s — someone is already hitting your limits.
• Autonomous systems. Group IPs by ASN. A share of hosting providers in catalog traffic above 5-10% is almost certainly bots.

Step 3. Estimate the Damage in Dollars

Translate your findings into business language, or protection won't get a budget. Calculate: the share of parasitic requests and their infrastructure cost; the load on paid external APIs (exchange rates, geocoding, search) generated by bots; analytics distortion (fake sessions ruin ad attribution, which directly affects marketers); and lost revenue from copied prices and content. Even a rough estimate like "bots cost us $2,000 a month in infrastructure plus an unknown amount in competitive position" changes the conversation with management.

Step 4. Set Traps to Confirm Your Hypotheses

Logs show that someone is there, but not what they want. Supplement the analysis with active reconnaissance:

• Canary records. Add a non-existent product to your catalog with a unique name that isn't indexed by search engines and isn't shown in navigation, but is accessible via a direct link from pagination. If that name shows up on someone else's site, it proves scraping and points to the source.
• Hidden links. A link invisible to humans (off-screen, zero dimensions) but present in the HTML. A human will never click it. Every click is a bot. Don't forget to close it off from search engines via exclusion rules.
• Unique spaces and small variations. Small unique formatting quirks in descriptions that let you later identify copied text.

Step 5. Build Scraper Profiles

The output of diagnostics should be a table of three to five profiles. For example: "Profile A: cloud script without JS, 40,000 requests to product cards overnight, old Chrome User-Agent, one subnet." "Profile B: distributed client on residential addresses, 2-3 requests per address, executes JS, only crawls the price filter." "Profile C: aggregator, identifies itself honestly, pulls a feed once a day." For each profile, it's immediately clear which defense layer will stop it. Profile A will fall to the first line. Profile B will require behavioral analysis. Profile C might be worth offering a partner API — sometimes that's more profitable than war.

Diagnostics Checklist

1. Logs for 7+ days with a full set of fields are collected.
2. Traffic distribution by hour and resource type is built.
3. Top 20 subnets and ASNs by number of requests to "valuable" URLs are identified.
4. Sessions with abnormal depth and regularity are found.
5. The share of parasitic traffic and its cost is calculated.
6. Canary records and hidden links are placed.
7. Scraper profiles are built with the target defense layer indicated.
8. Legitimate bots that must not be touched are identified: search engines, payment webhooks, partners, monitoring.

Method 2. The First Line: Server and Network Level

The first line is the foundation. It won't stop an advanced scraper, but it will cut off 50-70% of primitive bots literally in a day's work and almost for free. Skipping this stage in favor of expensive solutions is a mistake: why pay for analyzing every request if half of them can be discarded by a simple rule?

Rate Limiting Done Right

Rate limiting is the most famous tool and the most frequently misconfigured. The typical mistake: one global per-IP limit for the whole site. The result is either a limit that's too soft and useless, or one that's too strict and hits offices, universities, and mobile networks with NAT. The right approach is differentiated limits:

• By resource type. Static assets and the homepage — generous limits. Product cards, search results, catalog JSON endpoints — noticeably stricter. Login and registration forms — very strict.
• By key. Not just IP. Combine: IP + path, IP + User-Agent, session cookie, and for authenticated users — account ID. For mobile networks, a limit by /24 subnet or by ASN with a high threshold is better than a strict limit per individual address.
• By time window. Two windows: a short one (say, 20 requests in 10 seconds — against bursts) and a long one (say, 600 requests to product cards per hour — against slow crawling). A slow but persistent scraper will hit the second window; a real human won't.
• By reaction type. Exceeding the limit doesn't have to mean blocking. The first stage is a 2-5 second response delay. The second is a 429 code with a Retry-After header. The third is a temporary ban for 10-15 minutes. Gradual escalation barely affects people and slows bots down significantly.

Here's an example of a basic zone configuration for a popular web server that you can adapt to your needs (values are illustrative):

limit_req_zone $binary_remote_addr zone=catalog_burst:20m rate=3r/s;limit_req_zone $binary_remote_addr zone=login:10m rate=10r/m;location /catalog/ { limit_req zone=catalog_burst burst=15 delay=8; }location /login { limit_req zone=login burst=3 nodelay; limit_req_status 429; }

Pay attention to the delay parameter: it allows part of the burst to be served immediately, and the rest with a delay, which is comfortable for a human who opened ten tabs and unpleasant for a script.

Filtering by Network Attributes

Create a policy for autonomous systems. Hosting providers and clouds are the sources of most primitive scrapers. Blocking them entirely isn't always appropriate (corporate proxies and some legitimate services come from there too), but applying strict limits and a mandatory JavaScript challenge to them is reasonable. For countries where you don't do business — same approach: not a block, but an elevated level of verification. Remember the "weak signal adds points" approach, not "one signal decides everything."

Header-Level Rules

A few cheap but effective rules for WAF or server configuration:

• Reject requests with an empty User-Agent or with known HTTP library signatures to non-service URLs. Legitimate integrations should use a dedicated API with a key, not public pages.
• Require header consistency: a claimed Chrome without the characteristic client hint headers is suspicious.
• Return 404 or 410 for outdated URL formats that scrapers keep hitting from old sitemaps.
• Hide server and framework versions in headers — not for scraping protection, but for general hygiene.

Robot Rules and Explicit Policy

The robots file is not a means of protection — a malicious bot ignores it. But it matters for two reasons. First, it lets you explicitly allow useful bots and explicitly disallow unwanted crawlers, including data collectors for AI models — respectable operators follow these rules. Second, it codifies your position: combining technical rules with website terms of use makes unauthorized data collection legally more vulnerable for whoever is doing it. Be sure to include a prohibition on automated collection without written permission in your user agreement.

Caching as Protection

A non-obvious technique: aggressive caching of "valuable" pages at the CDN or reverse proxy level doesn't stop a scraper from taking data, but it removes the main pain — the load on the application and database. If a bot dumps the catalog from cache, your infrastructure barely cares. It's not a victory, but it buys you time to build the next lines without firefighting.

First Line Checklist

1. URL groups with different value levels are defined and their own limits are configured for each.
2. Two limit windows are implemented: short and long.
3. The reaction to exceeding is escalation: delay, 429, temporary ban.
4. Limits for mobile and residential networks are relaxed or counted by subnet.
5. Hosting ASNs and non-target regions are moved to enhanced verification mode, not ban.
6. Header rules are configured to cut off library clients.
7. A robot policy and a legal prohibition on data collection are written.
8. Caching of expensive pages is enabled.
9. Exceptions for search engines, payment systems, and partners are verified and documented.

Method 3. The Second Line: Client Verification and Challenges

The first line works on the request's own attributes. The second goes further and asks the client: "Prove you're a browser, and prove there's a human behind you." This is where scripts without JavaScript execution and hastily configured headless tools get cut off. Let's break down this layer's tools in ascending order of "cost" to the user.

TLS and HTTP/2 Fingerprints: A Seamless Filter

Start with what the user doesn't notice at all. On the load balancer or reverse proxy, compute the connection's TLS fingerprint and compare it against the claimed User-Agent. Typical logic:

• The fingerprint matches a known HTTP library, but the User-Agent claims a browser — high risk.
• The fingerprint matches a browser, but not the family or version in the header — medium risk.
• The fingerprint appears on your site for the first time and is present on thousands of addresses at once — suspicion of a new scraping tool.

An important implementation detail: don't block on a TLS fingerprint on day one. Collect statistics for a week, figure out which fingerprints correspond to your real audience (including browsers inside mobile apps and messenger webviews, which have specific fingerprints), and only then switch the rules to active mode. Otherwise you risk cutting off users who open your site from Telegram or an email client.

JavaScript Challenge: Verifying the Execution Environment

A basic JS challenge is a small script the client must execute to get an access cookie. Without the cookie, the server serves a stub instead of content. Simple scripts on HTTP libraries don't execute JavaScript and stay outside the door. A more advanced version collects environment characteristics and checks their consistency, as we discussed in the signals section.

Practical implementation recommendations:

1. Don't show the challenge to everyone. Apply it to requests with a medium risk score and to valuable sections. The homepage and landing pages that ads link to should open instantly — every extra second before the first screen costs conversion, and affiliate specialists know this better than anyone.
2. Make the challenge invisible. The script should run in the background in 100-300 milliseconds, without a white screen and "checking your browser" messages. For an honest client, the process is imperceptible.
3. Bind the result to the session, not the IP. The access cookie should be signed, have a lifespan, and contain a hash of key client characteristics. Then it can't just be copied and distributed to a hundred scraper threads.
4. Rotate the challenge regularly. A static check will be reverse-engineered within a week. Obfuscate the code, change the checked parameters, rotate the signing keys.
5. Check consistency between JS data and server data. Timezone from the script doesn't match IP geolocation? Claimed platform doesn't match the TLS fingerprint? Every mismatch is a risk point.

Proof of Work: An Economic Barrier

An elegant technique gaining popularity as a captcha alternative. The server gives the client a computational task (e.g., find a value whose hash starts with a certain number of zeros). A phone browser solves it in 200-500 milliseconds, imperceptible to the human. But a scraper making 100,000 requests has to spend hours of CPU time on it. The task difficulty can be increased proportionally to the risk score: a suspicious client gets a task ten times heavier. A human doesn't notice the difference; a bot very much does.

Captcha: Where It's Appropriate and Where It Backfires

Captcha is the most visible and most annoying tool. User experience studies consistently show that every captcha on the path to purchase reduces conversion, with higher losses for mobile users. So the rule is simple: captcha is the last argument, not the first check. Appropriate use cases:

• login forms after several failed attempts;
• registration and password recovery;
• submitting feedback and review forms;
• situations where the session risk score is already high and the client is still trying to browse the catalog;
• protecting critical operations: booking, applying a promo code, adding a scarce product to the cart.

Inappropriate use cases: the homepage, product cards, search, and ad campaign landing pages. If you have to put a captcha on your catalog, it means the previous layers aren't working, and that's what needs fixing. Also: choose modern captcha options with invisible mode and adaptive difficulty, not distorted-letter images from the 2000s.

API and Mobile App Protection

Often scrapers don't even crawl the site — they find the JSON endpoints your frontend uses and take clean data without parsing HTML. Check: is your catalog API accessible without a session cookie and a signed request? If so, all site protection is pointless. The minimum set of measures: short-lived signed tokens issued after passing a challenge; binding the token to the session; limiting the response size and pagination depth; separate, stricter limits on the API. For mobile apps — platform-level app authenticity verification and certificate pinning.

Second Line Checklist

1. TLS/HTTP2 fingerprints are computed, and weekly statistics on your audience are collected.
2. Fingerprint rules are switched to active mode only after analysis.
3. The JS challenge is implemented invisibly and applied adaptively by risk score.
4. The access cookie is signed, bound to the client, and has a lifespan.
5. Proof of work with adaptive difficulty is implemented.
6. Captcha is only on forms and critical operations, not on the catalog.
7. The catalog API is protected with tokens and limits, inaccessible without a session.
8. All checks are tested in messenger webviews and mobile browsers.

Method 4. The Third Line: Behavior, Traps, and Protecting the Data Itself

Suppose the scraper made it through the first two rounds: it uses residential addresses, controls a real browser with a plausible fingerprint, and passes challenges. What now? Now we move from the question "who are you" to "what are you doing" and, ultimately, to devaluing the loot itself. This is the most intelligent layer of anti-bot protection, and it's what separates a mature system from a set of rules.

Behavioral Session Scoring

The foundation of the third line is accumulating risk points per session over time. Attributes worth scoring:

• Navigation depth and structure. More than 50-100 product cards per hour, crawling pagination to the end, iterating through all filter combinations, moving through ascending IDs.
• Rhythm. Intervals between requests that are too uniform, or conversely, intervals with a perfectly random distribution (humans are never perfectly random).
• No interaction. No mouse movement, scroll, touch, window resize, or focus events. The page is loaded, but no one is looking at it.
• Atypical paths. Not a single visit to the homepage, cart, or contact page. Only target pages with data.
• Ignoring resources. Images, fonts, and analytics scripts aren't requested.
• Multi-session signs. The same browser fingerprint on hundreds of addresses, dozens of sessions with the same navigation pattern starting in the same minute.

The reaction to a rising score is the same escalation: first slower responses, then a harder challenge, then captcha, then session blocking. And it's worth blocking the session and fingerprint, not the IP — remember mobile networks.

Honeypot Traps in Production

The traps we used during diagnostics work great as a permanent protection element too. Every hidden link, every invisible form field filled in by a client, every request to a URL not in the navigation — a signal with an almost zero false positive rate. A human physically cannot click an element with zero dimensions. Safety rules: hide traps from search engines, don't use traps that screen readers for users with disabilities might hit, and rotate trap locations and types regularly.

Devaluing the Loot: Working with the Data Itself

Here's a strategy almost no one talks about, and it's often more effective than any block. If you can't stop a bot from taking data, make what it takes useless or treacherous.

• Data poisoning for suspicious sessions. A client with a high risk score gets not a 403 error but a normal page with distorted prices and stock: price plus or minus 7%, random stock, altered rating. The scraper doesn't realize it's been exposed and keeps collecting garbage that ends up in the competitor's database. One such incident, discovered after a repricing, kills the desire to scrape for a long time. Use with caution: only with very high confidence, so you don't show a wrong price to a real buyer, and never for legally significant information.
• Data watermarks. Unique micro-features per session: word order in descriptions, invisible formatting variations, unique "canary" products. They let you later prove where the data leaked from and which client was involved.
• Dynamic markup. Regularly changing CSS class names and HTML structure on valuable pages breaks scrapers tied to selectors. It's an arms race, but each iteration costs you one release and the attacker a rewrite of their logic.
• Rendering critical values unconventionally. Price can be assembled from several elements, injected via CSS pseudo-elements, served as an image, or lazy-loaded after interaction. Each method has a cost for accessibility and SEO — weigh them.
• Limiting output. Don't give the entire catalog to anyone: a hard pagination depth limit, mandatory filter narrowing, no "all products in one list" endpoint.

A Legitimate Channel Instead of War

A paradoxical but effective technique. If aggregators, partners, or researchers systematically need your data, give them an official API with a key, limits, and terms of use. Part of the load moves from the "gray" zone to a controlled one, and you get a legal and technical basis to deal harshly with those who keep scraping public pages.

A Framework for Phased Implementation of All Three Lines

A separate note on process, because most failures happen not because of technology but because of haste. The recommended sequence:

1. Observation mode (1-2 weeks). All rules and scoring work, but nothing is blocked — only decisions are logged. You see who you would have blocked and check against reality.
2. Calibration. You manually review the top "would-have-been-blocked" sessions. Among them you'll find your partners, your own monitoring, a messenger webview, and users with strange but real browsers. You adjust thresholds and add exceptions.
3. Soft action (1-2 weeks). You enable only slowing and invisible challenges. You monitor conversion, bounce rate, and support tickets.
4. Full action with gradual coverage. You enable captchas and blocks first on 10% of traffic, then 50%, then fully. Metrics are checked at each step.
5. Operation. Weekly false positive review, monthly threshold revisiting, challenge and trap rotation, scraper profile updates.

Key Metrics to Monitor

• Share of traffic classified as bot, by site section.
• Number of support tickets about captcha or inaccessibility.
• Conversion and bounce rate before and after, separately for the mobile audience.
• False positive rate from a manually reviewed sample of sessions (target: under 0.5%).
• Load on the application and database, infrastructure cost.
• Time between your price change and its appearance at a known competitor.

Common Mistakes in Anti-Bot Protection Implementation

We've covered what to do. Now about what not to do — based on the experience of dozens of projects, these mistakes repeat with surprising consistency.

Mistake 1. IP Blocking as the Main Measure

The most common and most harmful. The site owner sees a thousand requests from an address, bans it — and cuts off an office center or a mobile carrier segment with thousands of subscribers behind it. The scraper switches to the next address in the meantime. IP is a weak signal, a reason for enhanced verification, but not for a ban.

Mistake 2. Captcha on Everything

A reaction like "let's put a captcha on every page and let the bots suffer." The ones who suffer aren't the bots but your conversion: real people leave, ads waste budget on users who never reached the content, and search engine crawlers get a stub. Captcha is a precision tool for forms and critical operations.

Mistake 3. Enabling Blocks Without Observation Mode

Rules are written in the evening, enabled at night, and by morning support is drowning in complaints. Always observation first, then calibration, then action.

Mistake 4. Forgotten Legitimate Bots

A blocked search engine crawler means lost rankings in two weeks. A blocked payment webhook means unpaid orders. A blocked own monitoring means false alarms. Before enabling any rule, make and check your exception list.

Mistake 5. Ignoring API and Mobile Endpoints

The site is protected exemplarily, but the JSON that the frontend calls is open to the world without a token. The scraper is grateful for clean structured data. Protection must cover all content delivery channels.

Mistake 6. Static Challenges and Traps

Protection configured once is reverse-engineered and bypassed a month later. Anti-bot protection is a process, not a project: rotation, updates, threshold revisiting.

Mistake 7. Data Poisoning Without High Confidence

The temptation to show "spoiled" prices is strong, but one wrong price shown to a real customer, let alone an order placed at it, is a reputational and legal risk. Apply only to sessions with maximum scores and never to legally significant information.

Mistake 8. Forgotten Accessibility

Traps and unconventional rendering can break the site for screen readers and users with disabilities. Test protection with accessibility tools, not just in the latest Chrome.

Mistake 9. Collecting Excessive User Data

Browser fingerprints and detailed behavioral logs are information subject to personal data laws. Collect the minimum needed for the decision, limit retention periods, and reflect the processing in your privacy policy.

Mistake 10. No Success Metrics

Protection is on, but no one can answer "is it better now?" Without a diagnostic baseline and conversion monitoring, you can't tell victory from defeat.

Tools and Resources for Implementation

We promised not to compare vendors and we're keeping our word. Below are tool categories and what each is responsible for, so you can assemble a stack for your scale and budget. You'll easily find specific solution names yourself; what matters here is the architecture.

Web Server and Reverse Proxy Level

Nginx, Apache, Caddy, HAProxy, Envoy. What they provide: rate limiting by flexible keys, header rules, serving different responses by condition, computing TLS fingerprints via modules and extensions. This is your first line, and for a project up to several million requests per day, it may be enough to start with.

WAF and CDN with Protection

Cloud CDNs with built-in WAF and bot filtering modes, or self-hosted WAFs based on open rule sets. What they provide: filtering by ASN reputation and geography, managed challenges, basic scoring, caching. Pros — quick deployment. Cons — limited customization for your business logic.

Specialized Bot Management Platforms

Commercial and open systems that collect client fingerprints, run behavioral scoring, apply adaptive challenges, and provide an analytics dashboard. They're justified when the bot traffic share is high, the data is valuable, and you don't have a team for in-house development. When choosing, look at the observation mode, policy granularity, support for your mobile apps and webviews, and solution transparency — you should understand why a specific client was blocked.

Libraries for In-House Development

Open libraries for collecting browser fingerprints, implementing proof of work, computing TLS fingerprints server-side, and detecting automation signs. Suitable for teams with developers who want full control. Remember the need for constant updates: headless mode signs change with every browser version.

Captchas and Alternatives

Modern verification services with invisible mode and adaptive difficulty, as well as proof-of-work solutions without user interaction. Selection criteria: accessibility for people with disabilities, operation in your region without delays, minimal collection of user data.

Analytics and Observability

A stack for collecting and analyzing logs (ELK and analogues, ClickHouse, cloud analytics systems), dashboards for protection metrics, and alerts for traffic anomalies. Without them you're blind: it's impossible to calibrate thresholds without seeing the system's decisions broken down by audience segment.

Testing Tools

A separate recommendation: test your protection with the same means scrapers use. An automated browser in headless mode, an HTTP client in a popular language, network profiling tools. If your own simple script freely dumps the catalog, the work isn't done. For a team working with mobile proxies, it's useful to test another scenario too: whether the protection blocks honest users from mobile networks, where addresses are shared among thousands of subscribers.

Internal Resources You'll Need

• A person responsible for protection with the authority to change thresholds.
• A false positive review procedure involving support.
• Legally vetted terms of use and a privacy policy.
• An official channel for partners who need your data.

Case Studies: How This Works in Real Projects

Three anonymized stories from implementation practice, with numbers and conclusions. Details are changed; the essence is preserved.

Case 1. An Electronics E-Commerce Store: 71% of Catalog Requests from Bots

Situation. A mid-sized store with about 40,000 products noticed competitors changing prices within half an hour of it. Log diagnostics showed: 71% of product card requests at night and 44% during the day were automated. Main profiles: two cloud scripts without JS and one distributed client on residential addresses crawling only price pages. The infrastructure cost of parasitic traffic was about $1,800 a month, plus paid geocoding API limits were exhausted by mid-day.

Solution. First line: differentiated section limits, strict mode for hosting ASNs, product card caching. Second line: TLS fingerprints with weekly observation, an invisible JS challenge for the catalog, protecting the frontend JSON endpoint with tokens. Third line: session depth scoring, hidden links, canary products, and for maximum-score sessions — price distortion within 5-8%.

Result after two months. The bot traffic share to the catalog dropped to 9%. Cloud scripts disappeared in the first week. The distributed client lasted three weeks, then based on canary products began collecting distorted prices, and a month later its activity stopped. Application load dropped by 58%, infrastructure costs by $1,200 a month. Conversion didn't change statistically significantly; support tickets about captcha were 11 over two months with an audience of hundreds of thousands of visitors. The false positive rate from a manual sample was 0.3%.

Lesson. Data poisoning turned out to be the decisive factor against the most persistent scraper, but it only worked because the previous layers provided high classification confidence.

Case 2. A Services Classifieds Platform: Protecting Contacts and Listings

Situation. A platform with listings from private specialists suffered from mass contact data collection: contractor phone numbers went into spam databases, users complained and left. Collection was done through managed browsers with plausible fingerprints from residential and mobile addresses; the first line barely helped.

Solution. Emphasis on the third line and data protection. Contacts were removed from HTML and served only on click, via a signed request with a session token, with an hourly disclosure limit per session and increased proof-of-work difficulty as the risk score rose. For authenticated users the limit was higher, for guests lower. Session scoring factoring in interaction: revealing a contact without first scrolling to it and without pointer movement added points. Captcha only on the fifth and subsequent disclosure per hour for guests.

Result. The number of contacts revealed per session for suspicious clients dropped from hundreds to 3-4, after which sessions were blocked. Contractor spam complaints fell by 80% over the quarter. The "listing view — contact" conversion for real users rose by 4% — likely because pages got faster from the reduced load.

Lesson. When a scraper is indistinguishable from a human by fingerprint, what saves you is changing the data delivery model itself: valuable data is served on action, dosed, and with a rising price.

Case 3. A B2B Catalog: False Positives as the Main Problem

Situation. An industrial components supplier enabled cloud bot protection in strict mode without observation. Two days later — a 35% drop in leads. The review showed: the company's clients sit on corporate networks with shared addresses and outdated browsers, and a significant portion of traffic comes through a corporate messenger webview with a non-standard TLS fingerprint. The system classified them as bots.

Solution. Rollback to observation mode, two weeks of calibration, building a whitelist of fingerprints and subnets for key clients, and moving corporate segments to "slowing and invisible challenge only, never captcha" mode. Additionally — a partner API for three distributors who previously scraped the catalog for their systems.

Result. Leads recovered within a week after calibration. Bot traffic dropped by 62% due to distributors moving to the official API and primitive scripts being cut off. False positives — under 0.2%.

Lesson. Observation mode is not a formality. And some "scrapers" are actually your partners, whom it's easier to give an API to than to fight.

FAQ: Answers to Tough Questions

Can scraping be stopped completely?

The honest answer is no — if data is publicly accessible to a human, it's technically accessible to a sufficiently motivated bot. The goal of protection is to make collection expensive, slow, and unreliable enough that it loses its economic sense. In most commercial scenarios, that's enough: a competitor getting data with a day's delay and a percentage of distortion gains no advantage.

Will anti-bot protection hurt SEO?

It will if done wrong: blocking search engine crawlers, serving them a challenge instead of content, or rendering prices as images. Properly built protection recognizes legitimate bots (by declared address ranges and reverse DNS checks), lets them through without checks, and doesn't change content for them. Verify page visibility with webmaster tools after every policy change.

How do I protect the site without blocking mobile network users?

Never ban an individual IP from a mobile network; count limits by subnet or ASN with high thresholds; rely on session and behavioral signals, not network ones; apply invisible challenges and slowing to questionable clients, not captcha. Be sure to test protection from real mobile devices on major carriers' networks.

Should I build protection in-house or get a ready-made platform?

It depends on scale and team. The first line and basic challenges can be implemented by one developer in a couple of weeks and cover most primitive bots. Behavioral scoring with constant updates is essentially a separate product, and for projects without a dedicated security team, a ready-made platform is usually cheaper over a year. The reasonable path: start with your own basic measures, measure the remaining problem, and only then decide whether a specialized tool is needed.

How should I treat AI model crawlers?

As a business decision, not just a technical one. If your presence in AI assistant answers brings you an audience — allow crawling via robots rules and don't interfere. If your content is your product and its retelling by a model deprives you of traffic — disallow it explicitly and back it up with technical measures. Respectable operators declare their crawlers and follow the rules; disreputable ones fall under general anti-bot protection alongside other scrapers.

Is it legal to serve scrapers distorted data?

Technically, you're entitled to show different clients different content on your resource, but you must rule out situations where distorted information becomes the basis for a transaction with a real consumer, and you must not distort information that the law requires you to disclose accurately. Apply poisoning only to sessions with maximum classification confidence, disable it for authenticated users, and consult a lawyer about your industry.

How often should protection be updated?

Thresholds and exceptions — weekly in the first months, then monthly. Challenges and traps — rotation every 4-8 weeks or when a bypass is detected. Scraper profiles — revisit with every noticeable change in the traffic picture. Browser automation signs — with every major browser release. Protection without maintenance degrades within one or two quarters.

What if a major competitor with a big budget is scraping me?

Shift the emphasis from blocking to devaluation and evidence: canary records and watermarks give you legal arguments, data poisoning makes the loot unreliable, and output depth limits prevent the catalog from being taken entirely. In parallel, record incidents and assess the prospects of a legal claim — with evidence of systematic collection in violation of terms of use, it's often more effective than a technical arms race.

Where do I start if budget and time are limited?

One day on log diagnostics. Then: differentiated rate limiting with two windows, caching of expensive pages, closing JSON endpoints with tokens, hidden trap links, and canary products. This set can be implemented in a week at almost no cost and cuts off most of the problem. After that — measure and decide whether the next layer is needed.

Conclusion: Protection as a Process, Not a Purchase

If we had to boil everything down to one thought, it would be this: anti-bot protection is not a box you set up and forget, but a system of three coordinated layers and a process of constant calibration. The network level cuts off primitive bots almost for free. Client verification catches scripts and carelessly configured automated browsers. Behavioral analysis and working with the data itself make even advanced scrapers' efforts pointless. And what ties it all together is the scoring principle: don't slam the door on a single signal, but accumulate weak signals and respond proportionally.

The process matters just as much. Diagnostics before implementation, observation mode before enabling blocks, gradual coverage, conversion metrics alongside protection metrics, regular rotation — this is what separates successful projects from those that roll back a week later due to customer complaints. And remember the people behind the addresses: mobile network users, corporate proxies, and messenger webviews are your real customers, and protection must pass the friendliness test for them.

What to do next, starting today:

1. Pull a week's worth of logs and answer the diagnostic questions. It's free and takes a few hours.
2. Calculate how much bot traffic costs you in infrastructure and lost revenue. That number will determine your budget.
3. Implement the first line per the checklist: limits, cache, header rules, API protection.
4. Set up traps and canary records — they'll work for you for months.
5. Plan the second and third lines in phases, with observation mode at every step.
6. Set up a protection maintenance calendar and assign a responsible person.

The data you collect and create is an asset. Like any asset, it deserves protection proportionate to its value and not getting in the way of those it was created for. We hope this guide becomes that bookmark you return to at every stage — from the first look at the logs to a mature, smoothly running system.