[{"data":1,"prerenderedAt":757},["ShallowReactive",2],{"navigation":3,"/blog/how-i-built-a-token-efficient-crawler":42,"/blog/how-i-built-a-token-efficient-crawler-surround":752},[4],{"title":5,"path":6,"stem":7,"children":8,"page":41},"Blog","/blog","blog",[9,13,17,21,25,29,33,37],{"title":10,"path":11,"stem":12},"ESLint Without Prettier: One Tool for Formatting and Linting in TypeScript","/blog/eslint-best-practice","blog/eslint-best-practice",{"title":14,"path":15,"stem":16},"Why Your Startup Should Go Headless CMS from Day One","/blog/headless-cms-startups","blog/headless-cms-startups",{"title":18,"path":19,"stem":20},"How I Built a Token-Efficient Crawler for 900+ Shops with Crawl4AI","/blog/how-i-built-a-token-efficient-crawler","blog/how-i-built-a-token-efficient-crawler",{"title":22,"path":23,"stem":24},"Migrating a Live B2B Platform to AWS Without Stopping the Business","/blog/migrating-live-b2b-platform-aws","blog/migrating-live-b2b-platform-aws",{"title":26,"path":27,"stem":28},"Nuxt 4 WebSockets Are Elegant. Here's Why They Break on Vercel.","/blog/nuxt-websockets-vs-vercel","blog/nuxt-websockets-vs-vercel",{"title":30,"path":31,"stem":32},"Host Your Docker Container on Railway in 5 Minutes (8GB RAM for $5)","/blog/railway-smart-docker-host","blog/railway-smart-docker-host",{"title":34,"path":35,"stem":36},"Supabase vs Custom Auth: What I Recommend for Early-Stage SaaS","/blog/supabase-vs-custom-auth","blog/supabase-vs-custom-auth",{"title":38,"path":39,"stem":40},"How to Migrate WordPress to Strapi Headless CMS (With 1,000+ Posts)","/blog/wordpress-to-strapi-migration","blog/wordpress-to-strapi-migration",false,{"id":43,"title":18,"author":44,"body":48,"date":744,"description":745,"extension":746,"image":747,"meta":748,"minRead":584,"navigation":202,"path":19,"seo":749,"sitemap":750,"stem":20,"__hash__":751},"blog/blog/how-i-built-a-token-efficient-crawler.md",{"name":45,"avatar":46},"Niklas Grieger",{"src":47,"alt":45},"/profile.jpg",{"type":49,"value":50,"toc":728},"minimark",[51,56,68,71,74,77,81,86,106,111,133,135,139,142,154,168,179,187,193,209,211,215,223,234,238,245,249,265,273,276,280,283,290,294,317,336,356,364,372,396,399,425,429,432,458,464,468,475,667,674,676,680,690,693,713,716,724],[52,53,55],"h2",{"id":54},"my-journey","My journey",[57,58,59,60,67],"p",{},"I'm building ",[61,62,66],"a",{"href":63,"rel":64},"https://gospendl.com/",[65],"nofollow","GoSpendl"," — a product search platform for the Bitcoin community. Users can find Bitcoin-related products across hundreds of shops, and if they have a Nostr account, they can earn satsback cashback on purchases.",[57,69,70],{},"To show products, I need to crawl many different e-commerce websites dynamically — triggered by user search queries, not upfront. Traditional crawling requires you to define a CSS schema per website, and the moment a shop redesigns their page, your schema breaks silently.",[57,72,73],{},"I started researching AI-powered crawling and found a ton of tools. It can be overwhelming. This post documents what I tried, what didn't work, and how I ended up with a two-phase approach that keeps token costs under control across 900+ shops.",[75,76],"hr",{},[52,78,80],{"id":79},"the-challenges","The challenges",[82,83,85],"h3",{"id":84},"_900-shops-10k-products-each-i-cant-save-everything","900+ shops, 10k+ products each — I can't save everything",[57,87,88,92,93,96,99,100,102,105],{},[89,90,91],"strong",{},"Problem 1:"," Crawling and storing every product from every shop upfront is not feasible at this scale.",[94,95],"br",{},[89,97,98],{},"Problem 2:"," Every shop has a different design and HTML structure.",[94,101],{},[89,103,104],{},"Problem 3:"," Not all of them are standard e-commerce shops — some sell e-SIMs, domains, VPN subscriptions, etc.",[57,107,108],{},[89,109,110],{},"My solutions:",[112,113,114,127,130],"ul",{},[115,116,117,118,121,122,126],"li",{},"Instead of crawling product detail pages, I only crawl the ",[89,119,120],{},"search results page"," of each shop (e.g. ",[123,124,125],"code",{},"https://webshop.de/search?q=ipad","). Smaller payload, triggered on demand.",[115,128,129],{},"For the HTML structure problem: I found a way to use AI once per shop to generate a reusable CSS schema — more on that below.",[115,131,132],{},"For non-standard shops: still an open challenge. The AI correctly declines these pages most of the time, which at least avoids bad data.",[75,134],{},[52,136,138],{"id":137},"tools-i-explored","Tools I explored",[57,140,141],{},"Before landing on a solution, I tried three tools. The short version: all of them had dealbreakers for my specific case.",[57,143,144,147,148,153],{},[89,145,146],{},"Thunderbit"," (",[61,149,152],{"href":150,"rel":151},"https://thunderbit.com",[65],"thunderbit.com",") worked surprisingly well in their Chrome Extension — no CSS schema needed, just define fields with a prompt, and it handles pagination too. But it has no API. For dynamic crawling across hundreds of shops on demand, that's a hard stop. I'd recommend it for manual, small-scale exports.",[155,156,159,160],"div",{"className":157},[158],"max-w-xs","\n  ",[161,162],"img",{"alt":163,"src":164,"className":165},"Thunderbit AI Crawler","/blog/explore-ai-crawling/thunderbit.png",[166,167],"w-full","object-contain",[57,169,170,147,173,178],{},[89,171,172],{},"Browse.ai",[61,174,177],{"href":175,"rel":176},"https://www.browse.ai",[65],"browse.ai",") extracted data but missed price fields, and the pricing doesn't scale to 900+ shops.",[155,180,159,182],{"className":181},[158],[161,183],{"alt":184,"src":185,"className":186},"Browse AI missing data","/blog/explore-ai-crawling/browser-ai-first-robot-missing-data.png",[166,167],[57,188,189,192],{},[89,190,191],{},"Firecrawl"," was the most promising — developer-friendly REST API, LLM extraction, highly customizable. But in practice I got missing data (lazy loading issues), timeouts, and accidentally burned all my credits in one API call. JSON array extraction also had quirks in the playground. I might revisit it, but it wasn't reliable enough at the time.",[155,194,159,197],{"className":195},[196],"max-w-sm",[198,199,203,204,159],"video",{"className":200,"controls":202},[166,201],"h-auto",true,"\n    ",[205,206],"source",{"src":207,"type":208},"/blog/explore-ai-crawling/firecrawl-playground.mp4","video/mp4",[75,210],{},[52,212,214],{"id":213},"crawl4ai-my-solution","Crawl4AI — my solution",[57,216,217,222],{},[61,218,221],{"href":219,"rel":220},"https://github.com/unclecode/crawl4ai",[65],"Crawl4AI"," is an open-source crawler built for LLM integration. It runs a real Chromium browser via Playwright, handles JavaScript rendering, and supports multiple extraction strategies. Self-hostable via Docker, no API keys required.",[57,224,225,226,229,230,233],{},"What sold me: it separates ",[89,227,228],{},"how you crawl"," (browser config) from ",[89,231,232],{},"how you extract"," (strategy). That distinction turned out to be the key to solving my token cost problem.",[82,235,237],{"id":236},"the-core-problem-llm-extraction-doesnt-scale-to-900-shops","The core problem: LLM extraction doesn't scale to 900 shops",[57,239,240,241,244],{},"Crawl4AI's ",[123,242,243],{},"LLMExtractionStrategy"," sends the page HTML to an LLM on every crawl request. That works fine for one or two shops. For 900+ shops, triggered multiple times per user search query — it's a token cost nightmare.",[82,246,248],{"id":247},"my-solution-two-phase-crawling","My solution: two-phase crawling",[57,250,251,254,256,257,260,261,264],{},[89,252,253],{},"Phase 1 — Schema Discovery (once per shop, offline):",[94,255],{},"\nUse the LLM to analyze a shop's search results page and generate a ",[123,258,259],{},"JsonCssExtractionStrategy"," schema. This happens once, the result is stored in ",[123,262,263],{},"store-overrides.json",".",[57,266,267,270,272],{},[89,268,269],{},"Phase 2 — Production Crawl (every user query, zero LLM calls):",[94,271],{},"\nUse the stored CSS schema directly. Pure CSS extraction — no LLM involved, no token cost per request.",[274,275],"crawl-architecture-diagram",{},[82,277,279],{"id":278},"why-search-results-pages-not-product-detail-pages","Why search results pages, not product detail pages",[57,281,282],{},"Product detail pages have too much variation — sale layouts, out-of-stock states, variant pages, related product carousels. The LLM often grabs the wrong region.",[57,284,285,286,289],{},"Search result listing cards are different: they follow one repeating pattern in the HTML (a ",[123,287,288],{},"\u003Cli class=\"product-tile\">"," repeated 24 times). Schemas generalize across queries on the same shop and even across shops built on the same CMS (Shopify, for example).",[82,291,293],{"id":292},"what-i-learned-the-hard-way","What I learned the hard way",[57,295,296,299,301,302,305,306,309,310,312,313,316],{},[89,297,298],{},"Use raw HTML, not cleaned HTML.",[94,300],{},"\nCrawl4AI's ",[123,303,304],{},"cleaned_html"," strips many CSS class names that the LLM relies on for pattern recognition. Switching one shop (",[123,307,308],{},"baur.de",") from ",[123,311,304],{}," to raw ",[123,314,315],{},"result.html"," went from 0 extracted cards to 72. Always use the raw field when feeding HTML to the LLM.",[57,318,319,325,327,328,331,332,335],{},[89,320,321,324],{},[123,322,323],{},"scan_full_page"," is non-negotiable.",[94,326],{},"\nMany shops lazy-load product images. Without ",[123,329,330],{},"scan_full_page: true",", the crawler returns cards without ",[123,333,334],{},"imageSrc",". Enabling it took one shop from 14% image coverage to 100%.",[57,337,338,341,343,344,347,348,351,352,355],{},[89,339,340],{},"Give the LLM an explicit opt-out.",[94,342],{},"\nWithout a clear \"if you can't find a pattern, return ",[123,345,346],{},"{ baseSelector: null, fields: [] }","\" clause, the LLM hallucinates plausible-looking CSS selectors (",[123,349,350],{},".product-tile",", ",[123,353,354],{},".price-current",") that don't exist in the actual HTML. These produce schemas that look valid but extract nothing. With the opt-out, the LLM correctly declines shops it can't handle — and we skip them cleanly.",[57,357,358,361,363],{},[89,359,360],{},"List anti-patterns explicitly in the prompt.",[94,362],{},"\n\"Do NOT target autocomplete dropdowns, navigation menus, breadcrumbs, related-products carousels, footer widgets, recently-viewed widgets, filter facets, or pagination links.\" Without this, the LLM cheerfully grabbed Shopify's visually-hidden accessibility helper as the listing container.",[57,365,366,369,371],{},[89,367,368],{},"Add a plausibility guard before writing to Algolia.",[94,370],{},"\nEven with a good prompt, some schemas slip through targeting the wrong HTML region. I run three checks after extraction:",[373,374,375,378,389],"ol",{},[115,376,377],{},"If >50% of extracted product URLs duplicate each other → reject (matched a non-card element)",[115,379,380,381,384,385,388],{},"If most URLs look like ",[123,382,383],{},"/search?q=…"," or ",[123,386,387],{},"/category/…"," → reject (matched navigation)",[115,390,391,392,395],{},"If ≥5 cards were extracted but every ",[123,393,394],{},"name"," field is empty → reject (wrong region)",[57,397,398],{},"These three checks caught several batches of bad data before they reached Algolia.",[57,400,401,408,410,411,414,415,417,418,421,422,424],{},[89,402,403,404,407],{},"Shopify's ",[123,405,406],{},"imageAlt"," fallback.",[94,409],{},"\nShopify themes often render the product title only as the image's ",[123,412,413],{},"alt"," attribute on listing cards. The LLM extracts an empty ",[123,416,394],{}," field, then the plausibility guard fires. Both the schema generator and the webhook now copy ",[123,419,420],{},"imageAlt → name"," when ",[123,423,394],{}," is empty. This alone salvaged several Shopify-based shops.",[82,426,428],{"id":427},"real-numbers-from-gospendl","Real numbers from GoSpendl",[57,430,431],{},"On a random sample of 30 shops:",[112,433,434,440,446,452],{},[115,435,436,439],{},[89,437,438],{},"~33%"," produced working schemas on the first attempt",[115,441,442,445],{},[89,443,444],{},"~43%"," the LLM correctly declined (no usable pattern found)",[115,447,448,451],{},[89,449,450],{},"~7%"," failed the plausibility guard — bad data caught before reaching Algolia",[115,453,454,457],{},[89,455,456],{},"~17%"," infrastructure errors (Crawl4AI timeouts, Cloudflare blocks on heavily protected shops)",[57,459,460,461,264],{},"With hand-picked shop variety (filtering out known anti-bot shops upfront), the success rate jumps to ",[89,462,463],{},"60–70%",[82,465,467],{"id":466},"the-crawler-config-that-matters","The crawler config that matters",[57,469,470,471,474],{},"Both the schema generator and the production crawler must use the ",[89,472,473],{},"exact same Crawl4AI config",", otherwise schemas validated in generation break in production:",[476,477,482],"pre",{"className":478,"code":479,"language":480,"meta":481,"style":481},"language-ts shiki shiki-themes github-light github-dark","{\n  cache_mode: 'BYPASS',         // always see fresh DOM\n  scan_full_page: true,         // lazy-load fix\n  scroll_delay: 0.5,\n  wait_for_images: true,\n  delay_before_return_html: 2.5,\n  simulate_user: true,\n  override_navigator: true,\n  headers: {\n    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',\n    'Sec-CH-UA': '\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"120\"',\n    'Sec-CH-UA-Mobile': '?0',\n    'Sec-CH-UA-Platform': '\"Windows\"',\n  }\n}\n","ts","",[123,483,484,493,514,531,545,557,570,582,594,603,616,629,642,655,661],{"__ignoreMap":481},[485,486,489],"span",{"class":487,"line":488},"line",1,[485,490,492],{"class":491},"sVt8B","{\n",[485,494,496,500,503,507,510],{"class":487,"line":495},2,[485,497,499],{"class":498},"sScJk","  cache_mode",[485,501,502],{"class":491},": ",[485,504,506],{"class":505},"sZZnC","'BYPASS'",[485,508,509],{"class":491},",         ",[485,511,513],{"class":512},"sJ8bj","// always see fresh DOM\n",[485,515,517,520,522,526,528],{"class":487,"line":516},3,[485,518,519],{"class":498},"  scan_full_page",[485,521,502],{"class":491},[485,523,525],{"class":524},"sj4cs","true",[485,527,509],{"class":491},[485,529,530],{"class":512},"// lazy-load fix\n",[485,532,534,537,539,542],{"class":487,"line":533},4,[485,535,536],{"class":498},"  scroll_delay",[485,538,502],{"class":491},[485,540,541],{"class":524},"0.5",[485,543,544],{"class":491},",\n",[485,546,548,551,553,555],{"class":487,"line":547},5,[485,549,550],{"class":498},"  wait_for_images",[485,552,502],{"class":491},[485,554,525],{"class":524},[485,556,544],{"class":491},[485,558,560,563,565,568],{"class":487,"line":559},6,[485,561,562],{"class":498},"  delay_before_return_html",[485,564,502],{"class":491},[485,566,567],{"class":524},"2.5",[485,569,544],{"class":491},[485,571,573,576,578,580],{"class":487,"line":572},7,[485,574,575],{"class":498},"  simulate_user",[485,577,502],{"class":491},[485,579,525],{"class":524},[485,581,544],{"class":491},[485,583,585,588,590,592],{"class":487,"line":584},8,[485,586,587],{"class":498},"  override_navigator",[485,589,502],{"class":491},[485,591,525],{"class":524},[485,593,544],{"class":491},[485,595,597,600],{"class":487,"line":596},9,[485,598,599],{"class":498},"  headers",[485,601,602],{"class":491},": {\n",[485,604,606,609,611,614],{"class":487,"line":605},10,[485,607,608],{"class":505},"    'User-Agent'",[485,610,502],{"class":491},[485,612,613],{"class":505},"'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...'",[485,615,544],{"class":491},[485,617,619,622,624,627],{"class":487,"line":618},11,[485,620,621],{"class":505},"    'Sec-CH-UA'",[485,623,502],{"class":491},[485,625,626],{"class":505},"'\"Not_A Brand\";v=\"99\", \"Google Chrome\";v=\"120\"'",[485,628,544],{"class":491},[485,630,632,635,637,640],{"class":487,"line":631},12,[485,633,634],{"class":505},"    'Sec-CH-UA-Mobile'",[485,636,502],{"class":491},[485,638,639],{"class":505},"'?0'",[485,641,544],{"class":491},[485,643,645,648,650,653],{"class":487,"line":644},13,[485,646,647],{"class":505},"    'Sec-CH-UA-Platform'",[485,649,502],{"class":491},[485,651,652],{"class":505},"'\"Windows\"'",[485,654,544],{"class":491},[485,656,658],{"class":487,"line":657},14,[485,659,660],{"class":491},"  }\n",[485,662,664],{"class":487,"line":663},15,[485,665,666],{"class":491},"}\n",[57,668,669,670,673],{},"The ",[123,671,672],{},"Sec-CH-UA"," client hints helped with some Cloudflare-fronted shops. They're not a silver bullet — a handful of shops still block the crawler — but they noticeably improved coverage.",[75,675],{},[52,677,679],{"id":678},"whats-next","What's next",[57,681,682,683,686,687,264],{},"The crawl pipeline for GoSpendl is live and running. The current crawlable set covers around 14 shops with validated schemas. The schema generator (",[123,684,685],{},"pnpm gen:schema",") makes it straightforward to add more — run it against a shop's search URL, review the summary, persist it with ",[123,688,689],{},"pnpm save:schema",[57,691,692],{},"Remaining challenges:",[112,694,695,701,707],{},[115,696,697,700],{},[89,698,699],{},"Cloudflare-heavy shops"," need a residential proxy or stealth Playwright integration — not built yet",[115,702,703,706],{},[89,704,705],{},"Non-standard shops"," (e-SIM providers, domain registrars) are still hard to handle with CSS schemas",[115,708,709,712],{},[89,710,711],{},"Category inference"," is currently keyword-based; an LLM-backed classifier would be more robust at scale",[57,714,715],{},"If you're building something similar — crawling product data across many shops without blowing your LLM token budget — the two-phase approach (LLM once for schema discovery, CSS extraction for every real crawl) is worth considering. It's the pattern that made GoSpendl work at scale.",[57,717,718,719],{},"The project is open source: ",[61,720,723],{"href":721,"rel":722},"https://github.com/devonik/go-spendl",[65],"github.com/devonik/go-spendl",[725,726,727],"style",{},"html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sScJk, html code.shiki .sScJk{--shiki-default:#6F42C1;--shiki-dark:#B392F0}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sJ8bj, html code.shiki .sJ8bj{--shiki-default:#6A737D;--shiki-dark:#6A737D}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":481,"searchDepth":495,"depth":495,"links":729},[730,731,734,735,743],{"id":54,"depth":495,"text":55},{"id":79,"depth":495,"text":80,"children":732},[733],{"id":84,"depth":516,"text":85},{"id":137,"depth":495,"text":138},{"id":213,"depth":495,"text":214,"children":736},[737,738,739,740,741,742],{"id":236,"depth":516,"text":237},{"id":247,"depth":516,"text":248},{"id":278,"depth":516,"text":279},{"id":292,"depth":516,"text":293},{"id":427,"depth":516,"text":428},{"id":466,"depth":516,"text":467},{"id":678,"depth":495,"text":679},"2025-11-14T00:00:00.000Z","I needed to crawl 900+ e-commerce shops dynamically for GoSpendl. Here's how I combined Crawl4AI's LLM Strategy for one-time schema discovery with CSS Extraction for production crawls — and why that matters for token costs.","md","/blog/explore-ai-crawling/hero.png",{},{"title":18,"description":745},{"loc":19},"HXVjGqye_E9sAnkCM0RtMLpZhrwmUTn-ezCSAzWfOFw",[753,755],{"title":14,"path":15,"stem":16,"description":754,"children":-1},"Most startups pick WordPress or a monolithic CMS because it's fast. 18 months later, they're paying to undo that decision. Here's why headless is the smarter choice from the start.",{"title":22,"path":23,"stem":24,"description":756,"children":-1},"A four-year migration moved a veterinary industry webshop off a legacy monolith onto an orchestrated AWS architecture, service by service, without ever pausing the live business.",1784038599876]