# Emacom_BotShield

Protects Magento category and search pages against filter-combination enumeration by residential
botnets. When a request activates more filters than the configured threshold, the module challenges
the visitor with Cloudflare Turnstile and issues an HMAC-signed cookie on success.

Built for traffic that IP, ASN and User-Agent blocking cannot stop: hundreds of addresses from
retail ISPs, a few requests per address, filters appearing both in the query string and inside the
URL path (SEO modules).

## How detection works

Evaluation runs in the category and search controllers, cheapest check first. The first hit ends
evaluation.

| # | Check | Result |
|---|---|---|
| 1 | Module disabled | pass |
| 2 | Context outside `advanced/contexts` | pass |
| 3 | Client IP in the infrastructure whitelist | pass |
| 4 | `X-Verified-Bot: true` and `trust_cf_header = yes` — see [verified-bot header](#optional-verified-bot-header) | pass |
| 5 | Valid signed cookie | pass |
| 6 | Reverse DNS confirms a bot (only when the UA claims one) | pass |
| 7 | Filter count `>=` threshold for that context | **challenge** |
| 8 | Default | pass |

### How filters are counted

The filter count is the number of the store's filterable attributes carried by the request:

| Step | Source |
|---|---|
| which attributes are filterable | `Layer\Category\FilterableAttributeList` — EAV, one query per request |
| which of them the request carries | `RequestInterface::getParams()` |
| empty values dropped | `?color=` narrows nothing |

`Layer\State::getFilters()` is not used. Filters land there in
`LayeredNavigation\Block\Navigation::_prepareLayout()`, which runs while blocks render — after the
predispatch observer. `getParams()` is the same array `Layer\Filter\Attribute::apply()` reads, so a filter absent
from it would not be applied by core layered navigation either.

Request params cover both URL forms because SEO modules rewrite path segments into request
params during routing. Confirm this per store in `log_only` mode: for a URL with filters in the
path, `filters=` in the log must count them.

## Requirements

- Magento 2.4.6–2.4.9 / MageOS, PHP 8.2–8.5
- [`emacom/bot-shield-core`](https://github.com/emacom/bot-shield-core) 1.x
- Cloudflare account with Turnstile (free, independent of plan)
- The real client IP must reach PHP — see [Client IP](#client-ip)

## Installation

```bash
composer require emacom/bot-shield-core:^1.0
bin/magento module:enable Emacom_BotShield
bin/magento setup:upgrade
bin/magento cache:flush
```

The module ships enabled in `log_only` mode, so it starts collecting calibration data immediately
without blocking anything.

Mode, thresholds, cookie TTL and the whitelists live under **Stores → Configuration → Emacom → Bot
Shield**, scope website — config path prefix `emacom_botshield`.

Category pages and search results carry their own threshold:

| Field | Default | Meaning |
|---|---|---|
| `thresholds/category` | 3 | challenge at three active filters on a category page |
| `thresholds/search` | 2 | search results are narrowed less often, so the bar sits lower |
| either set to `0` | — | every request in that context is challenged |

To leave a context alone entirely, uncheck it in `advanced/contexts`.

## Core and adapters

### `emacom/bot-shield-core`

Framework-agnostic BotShield core: challenge decision, Turnstile verification, signed cookie and bot verification via rDNS. It has no dependency on any shop platform — only PHP 8.2–8.5 and the PSR interfaces `psr/log`, `psr/simple-cache` and `psr/clock`.

The same core serves several platforms (Magento, WooCommerce, PrestaShop). Each platform ships a thin adapter that implements the ports and hooks the decision into its request cycle; this module is the Magento adapter. A fix in the core reaches every platform with one release.

| Lives in the core | Lives in the adapter |
|---|---|
| decision order and threshold check — `ChallengeGuard` | integration point — here the `controller_action_predispatch` observer |
| Turnstile `siteverify`, fail-open on network error — `Turnstile\Verifier` | configuration storage and admin UI |
| HMAC-signed `ts_verified` cookie bound to a `/24` or `/64` prefix — `CookieSigner`, `Verification\Cookie` | secret derivation from the platform key — here `crypt/key` |
| `X-Verified-Bot` header, forward-confirmed rDNS — `Verification\Bot` | cookie, HTTP client, cache and log backends |
| challenge page, 302 / 403 responses, AJAX detection — `ChallengePage`, `ChallengeResponse`, `Resolver\AjaxDetector` | which request parameters are filters — `Port\FilterCounter` |
| `assets/challenge-interceptor.js` — Turnstile modal for AJAX requests | including the script on listing pages, platform AJAX markers |

The module wires the core ports to Magento in `etc/di.xml`.

| Core port | Magento implementation |
|---|---|
| `Port\Config` | `Model\Config` — website scope, HMAC key derived from `crypt/key` |
| `Port\Request` | `Model\Port\Request` — `Request\Http`, `RemoteAddress` |
| `Port\CookieJar` | `Model\Port\CookieJar` — `CookieManagerInterface` |
| `Port\HttpClient` | `Model\Port\HttpClient` — `HTTP\Client\Curl` |
| `Port\FilterCounter` | `Model\Resolver\ActiveFilters` |
| `Psr\SimpleCache\CacheInterface` | `Model\Port\Cache` — Magento cache, tag `EMACOM_BOTSHIELD` |
| `Psr\Clock\ClockInterface` | `Model\Port\Clock` — UTC |
| `Psr\Log\LoggerInterface` | `Logger\Logger` — `var/log/botshield.log` |

PSR interfaces are bound per core class through `<type>` arguments. A global preference would change them for every other module.

## AJAX layered navigation

A challenged request that comes from a script gets 403 with the modal payload instead of a 302.

| Signal | Source |
|---|---|
| `X-Requested-With`, `Sec-Fetch-Dest: empty`, `Accept: application/json` | `Core\Resolver\AjaxDetector` |
| `ajax` or `isAjax` query parameter | `Request\Http::isAjax()` |

`challenge-interceptor.js` from the core package is inlined on `catalog_category_view` and `catalogsearch_result_index` through `SecureHtmlRenderer`. The return address drops `ajax`, `isAjax` and `shopbyAjax`, so the visitor lands on the page and not on its JSON.

`js/jquery-challenge-guard.js` (RequireJS themes) keeps the `error` handler of a jQuery AJAX call away from a challenge response. Layered navigation modules leave the page on any failed request, which would close the modal.

The decision runs in `controller_action_predispatch` (`Observer\ListingObserver`), not in a controller plugin. A challenged request never reaches `execute()`, so `afterExecute` plugins of other modules cannot replace the 403 with their own 200 — which then lands in the full page cache.

## Amasty Improved Layered Navigation

Supported without a Composer dependency - detected by module name `Amasty_Shopby`.

| Difference | Handling |
|---|---|
| AJAX navigation (`shopbyAjax=1`) renders its JSON in `afterExecute` | decision moved before `execute()`, the JSON is never built for a challenged request |
| multi-select joins values with a comma (`color=49,50`), SEO URLs too (`black-blue.html`) | with `Amasty_Shopby` enabled every comma-separated value counts |
| failed AJAX request sends the visitor to the unfiltered listing | suppressed for challenge responses by `jquery-challenge-guard.js` |

## ElasticSuite

Supported without a Composer dependency - nothing references Smile classes, so a store without
ElasticSuite runs unchanged.

| Difference | Handling |
|---|---|
| `catalog/navigation_filter/ajax` returns product counts per option for any filter combination | guarded in `controller_action_predispatch_catalog_navigation_filter_ajax`, `q` present → search, otherwise category |
| multi-value filters (`color[]=a&color[]=b`) are applied | with `catalog/search/engine = elasticsuite` every value counts; core ignores array values, so there it counts once |
| `cat` on the AJAX endpoint is the current category | not counted there |

---

## Client IP

PHP must receive the visitor's real address in `REMOTE_ADDR`. Every proxy hop — Cloudflare, load
balancer, Varnish, container network — overwrites it unless the stack is configured to pass it
through. Setting that up is the server's job, not the module's.

### Verify

The module records the address it sees on every evaluated request in `var/log/botshield.log`:

```
[2026-09-22T11:04:17+00:00] 203.0.113.10 INFO botshield_evaluate context=category reason=threshold_exceeded challenged=true filters=3 threshold=3 cookie=false ua="Mozilla/5.0"
```

An address after the timestamp from `10.x`, `172.16-31.x`, `192.168.x` or `127.x`, or an
`ip_degraded=true` pair, means the chain rewrites the address.

### Without it

| Mechanism | Effect |
|---|---|
| Cookie bound to /24 | every visitor shares one prefix — a single cookie works for the whole botnet |
| IP whitelist | own infrastructure is indistinguishable from external traffic |
| Reverse DNS | lookup of a private address always fails |

The module detects the private address itself, logs `botshield_private_ip_detected` at critical
level and stays in `log_only` for every request over the threshold, writing
`botshield_enforce_degraded` on each one.

## Cloudflare Turnstile account

Turnstile is free and works on any Cloudflare plan, including the free one. The domain does not
have to be proxied through Cloudflare for Turnstile itself to work.

| # | Step | Note |
|---|---|---|
| 1 | In the account sidebar: **Protect & connect → Application security → Turnstile**, or directly [dash.cloudflare.com/?to=/:account/turnstile](https://dash.cloudflare.com/?to=/:account/turnstile) | Turnstile belongs to the account, not to a domain: inside a domain view it is missing from the sidebar, go back to **Account home** first |
| 2 | **Add widget**, **Widget name** after the store — `store-botshield` | |
| 3 | **Hostname Management → Hostnames** — every domain the store answers on, staging included | A missing hostname fails verification with `hostname-mismatch` |
| 4 | **Widget mode** — **Managed** | Cloudflare decides per visitor whether an interaction is needed |
| 5 | **Skip future security rule challenges for verified visitors** (pre-clearance) — off | The module issues its own signed cookie |
| 6 | Copy the **Site Key** and **Secret Key** | |
| 7 | Paste both under **Stores → Configuration → Emacom → Bot Shield → Cloudflare Turnstile**, save, `bin/magento cache:flush` | Keys are scoped per website, so stores keep separate widgets and separate Turnstile analytics |

### Optional: verified-bot header

Search engine crawlers are cheaper to recognise from Cloudflare than from reverse DNS. Cloudflare
does not send `X-Verified-Bot` on its own — two Request Header Transform Rules add it. The first
strips any copy the client sent, the second sets it for verified bots only. Without the first rule
a client forges the header and passes.

| # | Step | Note |
|---|---|---|
| 1 | Select the zone in the [Cloudflare dashboard](https://dash.cloudflare.com/) → **Rules → Overview → Create rule → Request Header Transform Rule** | Available on the free plan |
| 2 | Rule name `strip-verified-bot`, **Custom filter expression → Edit expression**: `(http.host eq "shop.example.com")` | One rule per store hostname, or `http.host in {...}` |
| 3 | Action **Remove**, header name `X-Verified-Bot`, **Deploy** | Runs for every request, bots included |
| 4 | **Create rule** again, name `set-verified-bot`, expression: `(http.host eq "shop.example.com" and cf.client.bot)` | Bot Management plans: `cf.bot_management.verified_bot` |
| 5 | Action **Set static**, header name `X-Verified-Bot`, value `true`, **Deploy** | |
| 6 | Check the order under **Rules → Transform Rules**: `strip-verified-bot` above `set-verified-bot` | Rules run top to bottom — reversed, the strip removes the header again |
| 7 | Set **Stores → Configuration → Emacom → Bot Shield → Whitelist → Trust X-Verified-Bot header** to **Yes**, `bin/magento cache:flush` | |

The header travels from Cloudflare to the origin only — browser DevTools never show it. Verify in
`var/log/botshield.log`:

```
[2026-09-23T19:31:46+00:00] 66.249.66.1 INFO botshield_evaluate context=category reason=cf_verified_bot challenged=false cookie=false ua="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
```

The module trusts the header as delivered, so enable it only where the origin accepts traffic from
Cloudflare alone (firewall to Cloudflare ranges, Authenticated Origin Pulls, or a Cloudflare
Tunnel with no public port). A request reaching the origin directly skips both rules, and one
header carries any client through.

Without the rule the module falls back to reverse DNS, at 20–80 ms on the first request per address.
