Build Night: Web Research Agents That Don’t Break in Prod x HackerSquad
In 3 hours, build a deep‑research agent that returns structured, citation-backed JSON then challenge it with the problems that kill agents in production (JS pages, blocks, rate limits) and run a small scale burst. Starter repo + credits included.
What we’re building
Project: Deep Research → Structured JSON (with citations)
Input: a company product / market question
Output: schema‑validated JSON + citations pulled from multiple public web sources
Then: flip on scale mode (concurrency + bounded retries + basic metrics) and see if it holds up
This is a build night, not a talk. You’ll ship something you can fork and deploy.
The real problems we’ll solve (hands-on)
If you’ve built agents/RAG before, you already know the hard part isn’t the model—it’s the web + ops.
We’ll work through the failure modes that make “it works on my laptop” die in production:
JS-heavy pages where HTML fetch returns placeholders
Inconsistent results across session/geo (locale, consent, interstitials, A/B variants)
Blocks + friction that appear only at volume
Rate limits + burstiness (works at 1 rps, fails at 20)
Bad retry behavior (duplicate work, runaway costs, thundering herd)
Silent extraction breakage (you get wrong structured data that “looks” right)
No observability (can’t explain failures or cost per task)
You will see how the same agent goes from single-run demo → repeatable pipeline with measurable success rate.
What you’ll walk away with
A deployable repo (Node/Python starter) that includes:
Multi-source retrieval + citation capture
Schema-first extraction (Zod/Pydantic) + validation checks
“Fast path → browser fallback” strategy
Session/geo controls (when needed for consistency)
Concurrency + queue pattern (safe parallelism)
Retry/backoff that’s bounded and measurable
Basic metrics output (success rate / latency + failure reasons)
Plus: credits to keep running it after the event.
Who should come
Advanced devs building agents, research workflows, monitoring/enrichment, competitive intel
AI engineers, full-stack devs, founders shipping agentic features
If you’ve said “it worked yesterday” about a web pipeline, you’ll fit right in
Not intended for “Scraping 101.”