Your AI agent tries to read a Medium article. Cloudflare returns 403. The agent silently fabricates an answer from search snippets and never tells you. On the pages it can reach, it burns 80,000 tokens parsing ad-tech and cookie banners to find one paragraph of prose. Two open-source Python libraries fix both problems in about 40 lines of code: curl_cffi gets through the CDN, and trafilatura strips the page down to the article itself.
My agent’s http_request tool was returning 403s on Medium, and on the pages it could still reach, a single article was burning 80,000 tokens of context. I run BlueClaw, an open-source AI agent runtime with structured execution tracing, so I had per-call records of exactly which fetches failed, where the tokens went, and what each turn cost. The code change turned out to be tiny. The reasoning behind it is what made the broader token-cost picture for an agent runtime finally make sense.
TL;DR: Cloudflare blocks Python’s default TLS fingerprint, and raw HTML burns 80k tokens to deliver 2k of prose. Use
curl_cffito impersonate Chrome’s handshake, then run the response throughtrafilaturato keep only the article body. No headless browser, no API key, ~90% fewer tokens.
The Common Failure Pattern
Most AI agent web tools are wired up like this:
This works on Wikipedia. It works on docs.python.org. It works on anything that does not run behind a modern CDN.
It fails on Medium. It fails on Substack. It fails on most news sites. And the failure mode is not always loud. Sometimes you get a 403. Sometimes you get a Cloudflare interstitial that the LLM cheerfully summarizes as “the page asks you to verify you are human.” Anthropic’s own Claude Code tracks this as issue #39896: WebFetch fails because Cloudflare’s bot protection blocks the headless verification request. The Cloudflare community forum has version after version of the same complaint: “Claude can’t fetch my site, returns 403.”
Worse, Claude Code issue #45070 documents the silent variant: the agent fabricates an answer from search snippets when WebFetch fails, instead of reporting the failure. The model does not know it failed. You do not know it failed. You ship the wrong answer. This is the failure mode I argued for handling explicitly in AI agent error handling patterns. A tool that fails silently is worse than one that fails loudly.
The naive pipeline has a second problem even when it works. A typical news article’s HTML carries about 6% semantic content by byte count. You are paying premium API rates to tokenize navigation menus, ad slots, GDPR banners, and inline tracking scripts so the model can find one paragraph.
The Fetch-Extract Pattern
Here is the mental model: fetch and extract are two different problems. Solve them with two different libraries, both at browser grade.
The network layer’s job is to look like a real browser to the server. The content layer’s job is to look like a real reader to the model.
Each library is specialized for its layer, and stacking them gives you something neither does alone.
This pattern matters because the typical alternatives miss one layer or the other. A hosted scraping API (Scrapfly, ZenRows, Bright Data) solves the network layer and ignores the content layer. A library like readability-lxml solves the content layer but does nothing for the 403. An HTML-to-markdown converter cuts tokens 80% but still ships 16,000 of them when 3,000 would do. The fetch-extract pattern gives you both layers, locally, with no API key.
Why urllib Fails on Cloudflare
The default Python HTTP stack is honest about being Python. When urllib or requests opens a TLS connection, the resulting TLS and HTTP/2 fingerprint looks nothing like Chrome’s. Modern bot detection fingerprints all of it. There is no User-Agent header gymnastics that fixes it, because the fingerprint is in the bytes of the handshake itself, before any header is sent.
The fix is curl_cffi. It is curl compiled against Chrome’s BoringSSL library, exposed as a Python module, configured to closely emulate Chrome 124’s TLS fingerprint. From the server’s point of view, the connection looks like a real browser at the network layer.
from curl_cffi import requests
resp = requests.get(url, impersonate="chrome124", timeout=30)
resp.raise_for_status()
html = resp.text
Three things to notice. First, the API is the same as requests, so you swap one import. Second, the impersonation target is explicit: you pick a browser version, and curl_cffi ships the matching handshake. (If you want freshness over reproducibility, use the unversioned impersonate="chrome" target, which auto-tracks the latest profile curl_cffi ships.) Third, this is not a headless browser. There is no Chromium binary, no Playwright dependency, no JavaScript engine. The process stays small, fast, and stateless. That matters when your agent runs the tool five times per turn.
What curl_cffi cannot do: solve a JavaScript challenge. Cloudflare Turnstile, hCaptcha, or any other JS-based challenge stops here. For those you need a real browser. But JavaScript challenges are a small minority of what blocks AI agents in practice. In practice, many “Cloudflare 403” failures against AI agents turn out to be TLS fingerprint rejections, and TLS fingerprint rejections are exactly what curl_cffi exists to defeat.
Like any fingerprint-emulation approach, staying current matters: browsers ship new TLS profiles every few months, and an older impersonation target eventually drifts from real Chrome traffic. Pinning to a recent version (or using the unversioned chrome target so curl_cffi tracks the latest) is part of keeping the bypass working over time.
Why HTML Costs 10x More Than the Article
You can have the HTML. You still cannot afford to read it.
Cloudflare ran the numbers themselves when they shipped Markdown for Agents earlier this year: a blog post that requires 16,180 tokens in HTML shrinks to about 3,150 tokens as Markdown. That is a 5x reduction from format conversion alone. MindStudio’s token-reduction guide puts the wider range at “12,000–40,000 tokens for a single page” of raw HTML, and reports a 90% reduction is reachable with selective extraction.
I see the same shape in BlueClaw’s traces. A typical article fetched with urllib.urlopen lands anywhere from 25,000 to 150,000 tokens once you tokenize the response body, depending on how much chrome, JSON-in-script payload, and embedded media the page ships. The same article, fetched and then run through trafilatura’s extract, typically lands at a few hundred to ~10,000 tokens. The reduction is not from compression. It is from throwing away everything that was never going to be useful to the model: nav links, sidebar ads, cookie banners, recommendation widgets, inline SVG icons, JSON-LD blobs, and the GDPR modal markup.
Why does this matter for an agent specifically? Because agents loop. A chatbot reads one page per turn. An agent reads three pages, decides to refine its query, reads five more pages, then summarizes. The HTML cost compounds at every step. By the time the agent answers, the same bloated page has been re-injected into context six or seven times. This is the agentic token tax that the dev.to “Stop feeding raw HTML” post describes: not the cost of one wasteful read, but the quadratic cost of accumulating wasteful reads. It is also why, for structured sources, agents should prefer APIs over search. For the long tail of articles where no API exists, fetch-extract is the floor.
It also matters for local models. A 14B local model running through Ollama has a 32k context window if you are lucky. An 80k-token raw HTML response simply does not fit. The model never sees the article. With trafilatura, the same article fits in the prompt with room left over for the question, the tool history, and the answer.
The Content Layer: trafilatura
There is a small graveyard of article-extraction libraries in Python. newspaper3k was the most popular for years and has not shipped a release since 2018. readability-lxml, the Mozilla Readability port, is well-maintained and works fine on well-formed HTML but breaks on aggressive layouts. boilerpipe is a Java port that nobody wants to depend on.
Trafilatura is the one that has held up. The SIGIR 2023 benchmark combining eight evaluation datasets scored it as the best single open-source extractor at 0.883 mean F1, ahead of Readability and Resiliparse. Only multi-tool ensembles ranked higher, and they pay for it in dependency weight. It is pure Python, has no browser dependency, and ships with a useful escape hatch: if its primary extractor returns suspiciously short output, it falls back to readability-lxml automatically and combines the two strategies. You get the best of both without writing the fallback yourself.
The API surface that matters is two functions:
import trafilatura
text = trafilatura.extract(
html,
include_comments=False,
include_tables=False,
)
That is it. No CSS selectors. No site-specific rules. No DOM parsing in your application code. Trafilatura’s heuristics handle Medium, Substack, the New York Times, IEEE Spectrum, Hacker News, and most personal blogs without configuration. When it does not handle a site well, the failure is graceful: you get back a shorter extraction or None, never a crash, never a 30-second hang.
Putting It Together
Here is the fetch-extract pattern as a standalone function. BlueClaw’s actual http_request tool wraps this in a factory with a domain allowlist and a few extraction tweaks (title prepending, favor_recall=True), but the bones are the same:
from curl_cffi import requests
import trafilatura
def http_request(url: str, extract_main: bool = True) -> str:
"""Fetch a URL with Chrome TLS impersonation.
Optionally runs trafilatura on HTML responses.
"""
resp = requests.get(
url,
impersonate="chrome124",
timeout=30,
allow_redirects=True,
)
resp.raise_for_status()
html = resp.text
# Skip extraction on non-HTML (JSON, text, etc).
content_type = resp.headers.get("content-type", "").lower()
is_html = "text/html" in content_type
if not (extract_main and is_html):
return html
extracted = trafilatura.extract(
html,
include_comments=False,
include_tables=False,
)
# Fall back to raw HTML if extraction returned nothing.
return extracted or html
Five things are worth pointing out.
The impersonate="chrome124" parameter is the entire Cloudflare bypass. There is no proxy configuration, no rotating User-Agent list, no header manipulation. You picked a browser version, curl_cffi shipped its handshake.
The content-type check matters. If your agent calls the tool against a JSON API or a plaintext robots.txt, you do not want trafilatura mangling it. Run extraction only when the server sent HTML.
Treat extract_main as a config knob, not a per-call argument the agent decides at runtime. In BlueClaw, for example, it lives in blueclaw.yaml as http_extract_main and is closed over at tool-registration time, so the agent never sees the toggle. Your config does.
Falling back to raw HTML means extraction failures degrade gracefully instead of silently dropping content. If extraction fails, the model still gets the page, just at the higher token cost. Better expensive than empty.
Dependencies: curl-cffi>=0.7, trafilatura>=1.12. Wheels exist for every platform that matters.
What the Numbers Look Like in a Trace
The reason I trust the 90% number is that BlueClaw writes a structured JSON trace for every run, including before-and-after token counts on every tool call. After turning on http_extract_main, the same fetch through the fetch-extract pipeline looks like this:
Single fresh fetch per URL on 2026-05-19, via curl_cffi.requests.get(impersonate="chrome124"). Tokens counted with tiktoken cl100k_base. Extraction uses trafilatura.extract(include_tables=True, favor_recall=True) with the metadata title prepended. The script and raw JSON live in the companion repo.
| Site | Raw HTML tokens | After trafilatura | Reduction |
|---|---|---|---|
| Medium long-read | 28,506 | 339 | 98.8% |
| Substack long-form (ACX) | 63,722 | 8,358 | 86.9% |
| IEEE Spectrum feature | 147,706 | 4,134 | 97.2% |
| Indie blog (text-heavy) | 22,525 | 9,000 | 60.0% |
| NYT article | — | — | HTTP 403 |
NYT still 403s even with chrome124 impersonation. That is not a TLS-only block, and the escalation path is below.
The shape is consistent: the more aggressively designed the page, the more dramatic the reduction. IEEE Spectrum ships an enormous DOM (148k tokens) because every figure caption, share widget, and recommended-article block lands in the HTML; trafilatura keeps only the body. A clean indie blog gets a smaller cut (60%) because there was less noise to throw away to begin with. Substack falls in the middle: long article body, real reader chrome.
The dollar number that matters is the Sonnet input rate at $3 per million tokens. A single 80,000-token fetch costs $0.24 just to read the input. The same fetch through the fetch-extract pattern costs $0.01 to $0.02. An agent that fetches five pages per turn over a 20-turn session swings from $24 to about $1.50 on web reading alone. That is not a marginal optimization. That is the difference between an agent you can leave running and an agent whose costs surprise you.
What This Pattern Does Not Fix
Two honest limits.
Cloudflare Turnstile and other JavaScript challenges still stop curl_cffi. If a site you need throws a JS challenge, you have to escalate to a real browser. The conventional escalation path is to pair curl_cffi with nodriver or Playwright: solve the challenge once in a headless browser, extract the cf_clearance cookie, then hand the cookie to curl_cffi for subsequent requests. This is real engineering, not a one-liner, and most agent use cases never need it.
Aggressive single-page applications still defeat trafilatura. If a site renders its content client-side from a JSON API, the HTML you receive has nothing to extract. The fix here is to call the JSON API directly when one exists, or to render the page in a real browser when one does not. For SPAs the fetch-extract pattern degrades to “fetch only,” and your tokens go back up.
Neither limit kills the pattern. The 80% of sites that block Python’s default TLS but render real HTML are the 80% where the fetch-extract pattern is decisive. That includes Medium, Substack, GitHub, Stack Overflow’s blog, most news sites, and most documentation sites that hide behind Cloudflare.
Wiring It Into Your Stack
The pattern is framework-agnostic. If you are on:
LangChain or LangGraph, replace the body of your custom WebBaseLoader or the implementation behind your WebSearchTool. The interface stays the same. Only the fetcher changes.
OpenAI Assistants or function calling, this is the implementation behind your fetch_url function. The schema the model sees is unchanged.
Claude Code, the cleanest path is a fetch MCP server that calls this code, because Anthropic’s built-in WebFetch has the architectural Cloudflare issue tracked in #39896. Several community MCP servers already wrap curl_cffi and trafilatura together; you can also wrap your own in about 80 lines using the MCP Python SDK.
Strands Agents SDK, replace the body of http_request in your tool registry. Strands’ stock http_request already does HTML-to-markdown conversion, which gets you part of the way; swapping in trafilatura for the content step and curl_cffi for the network step gets you the rest.
A custom agent, write the function once, register it as a tool, and move on. The whole implementation including type hints and a few error paths is under 50 lines.
What I Watch in Traces
Once the pattern is in production, the trace metrics that matter are not the ones you would guess. For the broader picture of which signals to wire up, see monitoring AI agents in production and the leaner observability-without-a-dashboard setup BlueClaw uses today.
Token-saved-per-call is the obvious one. Watch it on a per-domain basis. A site that suddenly stops saving tokens is a site that either changed its layout (trafilatura’s heuristics still work but are recovering less) or moved its content into a client-rendered SPA (extraction is now returning empty and you are falling back to raw HTML).
Time-to-fetch is the less obvious one. curl_cffi is slightly slower than urllib because of the impersonation cost. Trafilatura adds its own latency on the parsing step. The combined overhead in BlueClaw’s traces is consistently under 400ms on Medium-class pages. If you see that climbing, look for very large pages that are blowing past trafilatura’s default time budget; you can bump it.
403-rate-per-domain is the third. After turning on impersonation, the 403 rate on Cloudflare-protected sites in my traces dropped to under 2% across thousands of fetches. Knowing which is which is what lets you stop blaming the network when the problem has moved.
A 403 today is almost always a JavaScript challenge or a real ban, not a TLS fingerprint problem.
Closing
You do not need a hosted bypass API. You do not need a headless browser. You do not need site-specific selectors. You need two pure-Python libraries doing exactly one thing each: curl_cffi impersonating Chrome at the network layer, and trafilatura reading the article at the content layer. Wire them into your agent’s existing web tool, keep the interface unchanged, and you stop paying for both walls at once.
If your agent reads the web and your token bill is climbing, this is the lowest-effort, highest-impact change you will make this quarter. If your agent reads the web and silently invents answers when fetches fail, this is the change that lets you trust the tool again.
The fetch-extract pattern is not glamorous. It is two imports, one impersonation flag, and one function call. That is the whole point: the cheapest engineering is the kind that does its job and then disappears into a trace you stop looking at.
The curl_cffi + trafilatura web fetcher ships in BlueClaw as part of the http_request tool. The implementation lives in the open-source repo.
