Inkdown
Start writing

Merlin Backend

12 files·0 subfolders

Shared Workspace

Merlin Backend
01-Orchestration.md

03-Search

Shared from "Merlin Backend" on Inkdown

Search Architecture

Overview

The search system provides multi-provider web search with automatic fallback, geo-location support, and intelligent query processing. It prioritizes speed while ensuring results through a cascade of search providers.

Provider Cascade (Priority Order)

Plain text
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft

Core Function: webSearch

File: src/server/services/webSearch.ts:670

TypeScript

Search Flow Executor

File: src/server/services/webSearch.ts:643

TypeScript

Why This Matters:

  • Fast First: SerpAPI (2s) before Google (2s)
  • Automatic Failover: If one fails, next takes over
  • Empty Check: Only return if results exist
  • Logged: Every failure is tracked

Individual Search Providers

1. SerpAPI Search

File: src/server/services/webSearch.ts:515

TypeScript
2. Google Custom Search (with Key Rotation)

File: src/server/services/webSearch.ts:433

TypeScript

Key Rotation System:

TypeScript

Why Key Rotation:

  • Google limits: 10,000 requests/day per key
  • 5 keys = 49,500 daily capacity
  • Redis tracks usage, expires at midnight PST
  • Automatic failover to next key
3. Google + Retext (Keyword Extraction)

File: src/server/services/webSearch.ts:483

TypeScript

Why Retext:

  • Extracts key terms from long/complex queries
  • Better results for natural language questions
  • "What are the benefits of meditation for anxiety" → "meditation anxiety benefits"
4. Bing Search

File: src/server/services/webSearch.ts:547

TypeScript

Bing Features:

  • News results included (marked with isNews: true)
  • Geo-location headers for local results
  • User agent passthrough

Result Normalization

All providers convert to common TOrganicResult format:

TypeScript

YouTube Video Extraction: All providers extract YouTube metadata:

TypeScript

Query Filtering

File: src/server/services/webSearch.ts:204

TypeScript

Why Exclude Reddit:

  • Variable quality, often not authoritative
  • Can be excluded via -site:reddit.com/r/
  • Keeps reddit.com (main site) but removes subreddits

Firecrawl Web Search (Alternative)

File: src/server/services/webSearch.ts:723

For deep research, an alternative AI-powered search:

TypeScript

Used in deep research for AI-curated results.

Search Engine Enum

File: src/server/services/webSearch.ts:160

TypeScript

Web Search Tool Integration

The search system is exposed as a tool:

File: src/server/endpoints/unified/tools/webSearch.tool.ts

TypeScript

Focus Modes:

  • YOUTUBE: Video-focused search
  • SOCIAL: Reddit/Twitter discussions
  • ACADEMIC: arXiv, Google Scholar
  • DEFAULT: General web

Key Design Decisions

1. Speed First
  • SerpAPI (2s) before Google (2s)
  • Fast timeouts with retries
  • Fail fast, fail over
2. Automatic Failover
  • No single point of failure
  • 4 providers in cascade
  • Empty results trigger next provider
3. Geo-Location
  • IP-based country detection
  • Language headers passed through
  • Local results when available
4. Rate Limit Management
  • Google: Redis-based key rotation
  • SerpAPI: No limits (paid tier)
  • Bing: Single key, monitored
5. Result Quality
  • Reddit exclusion
  • YouTube metadata extraction
  • News result integration
  • Normalized format across providers

Integration with Orchestrator

Search is a tool called by the orchestrator:

TypeScript

The search service is stateless - each call is independent. Results are immediately streamed to the client and added to the conversation context.