Inkdown
Start writing

Study

70 filesยท12 subfolders

Shared Workspace

Study
AI eng

03-Search

Shared from "Study" on Inkdown

Search Architecture

Overview

The search system provides multi-provider web search with automatic fallback, geo-location support, and intelligent query processing. It prioritizes speed while ensuring results through a cascade of search providers.


Provider Cascade (Priority Order)

Plain text
basic-ques
core
Revision w/ Whiteboard
CN Basics - 1
CN Basics - 2
DNS
Event loop
programming-language-concepts.md
zero-language-explanation.md
DB
Quick
databases-deep-dive.md
01-introduction.md
02-relational-databases.md
03-database-design.md
04-indexing.md
05-transactions-acid.md
06-nosql-databases.md
07-query-optimization.md
08-replication-ha.md
09-sharding-partitioning.md
10-caching-strategies.md
11-cap-theorem.md
12-connection-pooling.md
13-backup-recovery.md
14-monitoring.md
15-database-selection.md
README.md
JS
core topics
Event loop
Merlin Backend
01-Orchestration.md
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft
Mobile
Build Alternative
Bundling
metro-bundler-deep-dive.md
OpenAI Agents Python
00_OVERVIEW.md
01_AGENT_SYSTEM.md
02_RUNNER_SYSTEM.md
03_TOOL_SYSTEM.md
04_ITEMS_SYSTEM.md
05_GUARDRAILS.md
06_HANDOFFS.md
07_MEMORY_SESSIONS.md
08_MODEL_PROVIDERS.md
09_SANDBOX_SYSTEM.md
10_TRACING.md
11_RUN_STATE.md
12_CONTEXT.md
13_LIFECYCLE_HOOKS.md
14_CONFIGURATION.md
15_ERROR_HANDLING.md
16_STREAMING.md
17_EXTENSIONS.md
18_MCP_INTEGRATION.md
19_BEST_PRACTICES.md
20_ARCHITECTURE_PATTERNS.md
opencode-study
context-handling
core
Python
Alembic
Basics
sqlalchemy - fastapi
SQLAlchemy overview
tweets
system_design_for_agentic_apps.md
Agent Loop

Core Function: webSearch

File: src/server/services/webSearch.ts:670

TypeScript

Search Flow Executor

File: src/server/services/webSearch.ts:643

TypeScript

Why This Matters:

  • Fast First: SerpAPI (2s) before Google (2s)
  • Automatic Failover: If one fails, next takes over
  • Empty Check: Only return if results exist
  • Logged: Every failure is tracked

Individual Search Providers

1. SerpAPI Search

File: src/server/services/webSearch.ts:515

TypeScript
2. Google Custom Search (with Key Rotation)

File: src/server/services/webSearch.ts:433

TypeScript

Key Rotation System:

TypeScript

Why Key Rotation:

  • Google limits: 10,000 requests/day per key
  • 5 keys = 49,500 daily capacity
  • Redis tracks usage, expires at midnight PST
  • Automatic failover to next key
3. Google + Retext (Keyword Extraction)

File: src/server/services/webSearch.ts:483

TypeScript

Why Retext:

  • Extracts key terms from long/complex queries
  • Better results for natural language questions
  • "What are the benefits of meditation for anxiety" โ†’ "meditation anxiety benefits"
4. Bing Search

File: src/server/services/webSearch.ts:547

TypeScript

Bing Features:

  • News results included (marked with isNews: true)
  • Geo-location headers for local results
  • User agent passthrough

Result Normalization

All providers convert to common TOrganicResult format:

TypeScript

YouTube Video Extraction: All providers extract YouTube metadata:

TypeScript

Query Filtering

File: src/server/services/webSearch.ts:204

TypeScript

Why Exclude Reddit:

  • Variable quality, often not authoritative
  • Can be excluded via -site:reddit.com/r/
  • Keeps reddit.com (main site) but removes subreddits

Firecrawl Web Search (Alternative)

File: src/server/services/webSearch.ts:723

For deep research, an alternative AI-powered search:

TypeScript

Used in deep research for AI-curated results.


Search Engine Enum

File: src/server/services/webSearch.ts:160

TypeScript

Web Search Tool Integration

The search system is exposed as a tool:

File: src/server/endpoints/unified/tools/webSearch.tool.ts

TypeScript

Focus Modes:

  • YOUTUBE: Video-focused search
  • SOCIAL: Reddit/Twitter discussions
  • ACADEMIC: arXiv, Google Scholar
  • DEFAULT: General web

Key Design Decisions

1. Speed First
  • SerpAPI (2s) before Google (2s)
  • Fast timeouts with retries
  • Fail fast, fail over
2. Automatic Failover
  • No single point of failure
  • 4 providers in cascade
  • Empty results trigger next provider
3. Geo-Location
  • IP-based country detection
  • Language headers passed through
  • Local results when available
4. Rate Limit Management
  • Google: Redis-based key rotation
  • SerpAPI: No limits (paid tier)
  • Bing: Single key, monitored
5. Result Quality
  • Reddit exclusion
  • YouTube metadata extraction
  • News result integration
  • Normalized format across providers

Integration with Orchestrator

Search is a tool called by the orchestrator:

TypeScript

The search service is stateless - each call is independent. Results are immediately streamed to the client and added to the conversation context.