Needham Navigator
An open-source RAG engine for municipal content — starting with Needham, MA.
February 2026
Live AppRAG SystemOpen Source
~70K
Pages Indexed
$11
Total Build Cost
~14
Days to Build
30K
Lines of Code
Live App — Try It
The Build Story
And I thought: What if there were just a clean, AI-native search engine for Needham?
That was it. We (aka Claude and I) started by scraping the town website. Then the school district site. Then the library. Then parks & rec and public documents — 5 content sources total, processed through a 4-tier ingestion pipeline with 38 custom boilerplate-stripping patterns. The search system combines semantic similarity, full-text search, recency weighting, and source-tier ranking to surface answers with confidence scores.
Then Claude suggested scaling the ingestion. Then I blinked. And realized we nearly scraped the entirety of Mass.gov. 49,547 out of 50,000 pages — 99.1%.
Out of thin air, Claude created: structured ingestion pipelines, semantic chunking, vector embeddings, hybrid search, answer synthesis, and fully automated deployment.
Total cost to process and summarize ~70,000 total pages? About eleven dollars. That's less than a Sweetgreen bowl.
Technical Deep Dive
System Architecture
At a high level, Needham Navigator is a 5-layer system: user browser → Next.js on Vercel → AI services → data stores → external municipal sources.
System Architecture
User Browser (Resident)
HTTPS → Next.js App (Vercel)
HTTPS
Vercel Hosting (Next.js App Router)
Pages & UI Component Server
Chat API Route (/api/chat)Search API (/api/search)Daily Cron RouteIngest & Monitor
Streaming / JSON
AI Services
Embed Query · Generate Completion · Reranking
OpenAI API (Embeddings & Chat)Cohere (Reranking — Optional)
Vector Search / Store
Supabase (Postgres DB)
Documents & Chunks · Content Items · Generated Articles · Cached Answers · Telemetry & Costs
Upstash Vector DB
Chunks Namespace · Content Namespace · 1536-dim vectors
Ingest & Monitor
External Sources
Town Sites · RSS Feeds · iCal · PDF Scrapers
needhamma.govneedham.k12.ma.usmass.gov40+ allowed domains
The Claude Development Flywheel
The development process became an almost-automated flywheel. Claude CoWork critiques the UX, generates structured prompts, Claude Code implements features, GitHub Actions validates, and Vercel auto-deploys. Rinse. Lather. Repeat.
The Claude Development Flywheel
CONTINUOUS CYCLE
Claude CoWorkUX Critique
Structured Prompt Generated
Claude CodeImplements Feature
Pull Request Opened
GitHub ActionsLint, Typecheck, Build, Security
Auto Merge + DeployVercel
Error Feedback Loop (if Failed)
📊 Pendo (Analytics)
How Long Would This Take a Human?
I asked GPT, Gemini, and Claude to estimate how long a traditional full-stack engineer + QA would need to architect, build, secure, and deploy this system. Then I normalized everything into person-days and weeks.
| Model | Team | Person-Days | Calendar | FTE Months | Key Constraint |
|---|---|---|---|---|---|
| 🟢GPT | 1 Eng + QA | 320 PD | ~48 weeks | ~11 months | Serial engineering bottleneck; long integration/regression tail |
| 🟢GPT | 2 Eng + QA | 320 PD | ~28-30 weeks | ~6-7 months | Parallelism helps, but coordination + QA gates final stretch |
| 🔵Gemini | 1 Eng + QA | 320 PD | ~43-52 weeks | ~10-12 months | Context-switching across RAG, UI, DevOps; burnout risk |
| 🔵Gemini | 2 Eng + QA | 340 PD | ~26-30 weeks | ~6-7 months | Optimal parallel streams; ~15 PD coordination overhead |
| 🟠Claude | 1 Eng + QA | 139 PD | ~14-16 weeks | ~3.5-4 months | Partial QA overlap; mostly serial |
| 🟠Claude | 2 Eng + QA | 92 PD | ~9-11 weeks | ~2-2.5 months | Strong parallelism; ~15% context overhead |
Not one model said “two weeks.”
The average single-person estimate hovered around a year.
I built it in roughly fourteen days.
Not because I typed faster. Because I orchestrated.
Scraper Flow
A custom web scraper built with Cheerio + Mozilla Readability processes each URL through 6 stages — from raw HTTP fetch to structured storage. 38 CivicPlus boilerplate patterns are stripped during the Clean + Normalize step.
Scraper Flow
Fetch URLHTTP GET / POST requests
Retry LogicParse DOMHTML parsing & traversal
Extract ContentSelects text, images, data
Rate LimitingClean + NormalizeRemove noise, format data (38 regex patterns)
Attach MetadataTimestamp, source, tags, department
DeduplicationStore Structured + RawDatabase & file storage
Hybrid Search Flow
Every user query goes through a hybrid search pipeline that combines semantic vector search (pgvector cosine similarity) with PostgreSQL full-text search, then reranks results using a 4-factor formula before generating a confidence-scored answer via GPT streaming.
Hybrid Search Flow
Overview
User Query
Keyword Filter
Vector Similarity Search
Merge + Rank
LLM Synthesis
Structured Answer
Detailed Flow
User QuerySubmit to /api/chat
▲ Next.js App (Vercel)Query Processing (Rewrite/Expand)
🤖 OpenAI APIEmbed Query → Vector
Hybrid Retrieval Enginepgvector semantic + PostgreSQL FTS
📊 Upstash VectorVector Search (filter: town_id)
⚡ Supabase (Postgres DB)Fetch Text & Metadata
Rerank & Assemble Contextscore = 0.5×semantic + 0.3×FTS + 0.1×recency + 0.1×tier_boostOptional Cohere cross-encoder reranking
Generate Chat Completion (Stream)GPT-5 Nano · Vercel AI SDK
Stream to UserConfidence badge + Source pills
Cache Answer + Log Telemetry7-day TTL · Cost tracking
CI/CD Pipeline
Every push triggers a full CI pipeline: lint, type-check, build verification, and static code analysis via GitHub Actions. Three scheduled workflows handle daily article generation, nightly health checks, and weekly security scans.
CI/CD Pipeline
Overview
Push to Branch
Lint + Type Check
Build Verification
Static Code Analysis
If Pass → Merge
Auto Deploy to Vercel
Detailed GitHub Actions Workflow
CI Workflow (ci.yml)Triggered on PR to main
1. Build, Test, Lint, Audit2. CodeQL & Semgrep Scans3. Auto-Merge & Delete Branch (if green)
✅ Pass & Merge→ Vercel Deployment → Production App
❌ FailComment & Issue (on-failure)
Scheduled Workflows
🕐 Generate Articlesgenerate-articles.yml · Daily
🕐 Nightly Health Checknightly-health-check.yml · Daily
🕐 Weekly Scans & Cleanupcodeql.yml · semgrep.yml · branch-cleanup.yml
Full Tool Stack
Frontend
Next.js (Vercel deployment)React 18Tailwind CSSshadcn/ui
Backend / API
Node.js / TypeScriptVercel AI SDKVercel Serverless Functions
Database
Supabase (Postgres)pgvector (1536-dim)Upstash Vector
AI
GPT-5 Nano (Chat)OpenAI Embedding-3-LargeCohere Reranker (optional)
Search Strategy
Hybrid (keyword + vector similarity)Multi-factor rerankingConfidence scoring
CI/CD
GitHub ActionsCodeQL scanningSonar static analysisDependency vulnerability checks
Hosting
Vercel (auto-deploy on merge)Squarespace (domain registrar)
Read the Full Story
The Human in the Loop: Or, So I Thought
More Like This
ReplitSubstack
Menu Labs
Prompt-to-menu generator, built for $2.48 on Replit.
AppNo-CodeReplit
View project 
ReplitSubstack
SwiftieGPT
Notes from inside the Swiftie Industrial Complex — an AI app that explores the gap between Taylor Swift fandom, parasocial relationships, and what we project onto pop stars. Part satire, part genuine product.
AppData VizTaylor SwiftAI
View project 
ChatGPTSubstack
ChartSmithGPT
An AI-powered micro app that converts any table into styled charts with PNG/SVG export. Paste data, pick a palette, download instantly.
AI Micro AppData VisualizationChartsClaude API
View project