InkdownInkdown
Start writing

Merlin Backend

12 files·0 subfolders

Shared Workspace

Merlin Backend
01-Orchestration.md

03-Search

Shared from "Merlin Backend" on Inkdown

Search Architecture

Overview

The search system provides multi-provider web search with automatic fallback, geo-location support, and intelligent query processing. It prioritizes speed while ensuring results through a cascade of search providers.

Provider Cascade (Priority Order)

Plain text
Query Input
    ↓
┌─────────────────────────────────────────────────────────────┐
│ 1. SerpAPI (2s timeout, 2 retries)                          │
│    └── Fast, reliable, no rate limits                       │
│         ↓ If fails/empty                                    │
├─────────────────────────────────────────────────────────────┤
│ 2. Google Custom Search (2s timeout, 2 retries)             │
│    └── Standard Google results                              │
│    └── Redis-based key rotation (9900 req/day/key)         │
│         ↓ If fails/empty                                    │
├─────────────────────────────────────────────────────────────┤
│ 3. Google + Retext Keywords (2s timeout, 2 retries)         │
│    └── Extracts key terms from query for better results    │
│         ↓ If fails/empty                                    │
├─────────────────────────────────────────────────────────────┤
│ 4. SerpAPI Slow Mode (5s timeout, 5 retries)                │
│    └── Last resort with extended timeout                   │
└─────────────────────────────────────────────────────────────┘
    ↓
TOrganicResult[] (normalized format)
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft

Core Function: webSearch

File: src/server/services/webSearch.ts:670

TypeScript
export const webSearch = async (query: string): Promise<TOrganicResult[]> => {
	// Get request context for geo-location
	const { req }: TBasicRequestContext = requestContext.get();

	// Extract IP for geo-location
	const ip = isDevEnv ? req.ip : req.header("x-forwarded-for")?.split(",")[0];
	const geo = geoip.lookup(ip || "");

	// Get language preference from browser
	const acceptLanguage = req.headers["accept-language"]
		?.split(",")[0]
		.split(";")[0]
		.trim();

	// Build metadata for all providers
	const webSearchMetadata: TWebSearchMetadata = {
		geo: geo,
		ip: ip,
		acceptLanguage: acceptLanguage,
		googleApiExhausted: false,
	};

	// Filter query (remove reddit from results)
	const filteredQuery = filterQuery(query);

	// Truncate very long queries (1500 char limit for URL safety)
	let truncatedQuery = filteredQuery;
	if (query.length > 1500) {
		truncatedQuery = query.slice(0, 1000) + query.slice(-500);
	}

	// Define the search flow (order matters!)
	const searchEngineFlow: TSearchEngineFlow[] = [
		{
			searchEngine: SearchEngine.SERP_API,
			timeout: WEB_SEARCH_FAST_TIMEOUT, // 2000ms
			retries: MIN_RETRIES_SEARCH_API, // 2
		},
		{
			searchEngine: SearchEngine.GOOGLE,
			timeout: WEB_SEARCH_FAST_TIMEOUT, // 2000ms
			retries: MIN_RETRIES_SEARCH_API, // 2
		},
		{
			searchEngine: SearchEngine.GOOGLE_WITH_RETEXT,
			timeout: WEB_SEARCH_FAST_TIMEOUT, // 2000ms
			retries: MIN_RETRIES_SEARCH_API, // 2
		},
		{
			searchEngine: SearchEngine.SERP_API,
			timeout: WEB_SEARCH_SLOW_TIMEOUT, // 5000ms
			retries: MAX_RETRIES_SEARCH_API, // 5
		},
	];

	// Execute the cascade
	return await executeSearchEngineFlow(
		searchEngineFlow,
		truncatedQuery,
		webSearchMetadata,
	);
};

Search Flow Executor

File: src/server/services/webSearch.ts:643

TypeScript
async function executeSearchEngineFlow(
	flow: TSearchEngineFlow[],
	query: string,
	webSearchMetadata: TWebSearchMetadata,
): Promise<TOrganicResult[]> {
	for (const step of flow) {
		try {
			const handler = searchEngineHandlers[step.searchEngine];
			const results = await handler(
				query,
				step.timeout,
				step.retries,
				webSearchMetadata,
			);

			// Success! Return immediately
			if (results.length > 0) {
				return results;
			}
			// Empty results - continue to next provider
		} catch {
			logger.error(`ERROR/WEB_SEARCH/${step.searchEngine}_SEARCH`);
			// Error - continue to next provider (failover)
		}
	}

	// All providers failed
	logger.error(`ERROR/WEB_SEARCH_FAILED`);
	return [];
}

Why This Matters:

  • Fast First: SerpAPI (2s) before Google (2s)
  • Automatic Failover: If one fails, next takes over
  • Empty Check: Only return if results exist
  • Logged: Every failure is tracked

Individual Search Providers

1. SerpAPI Search

File: src/server/services/webSearch.ts:515

TypeScript
const serpApiSearch = async (
	query: string,
	timeout: number,
	retries: number,
	webSearchMetadata: TWebSearchMetadata,
): Promise<TOrganicResult[]> => {
	const { country: countryCode } = webSearchMetadata.geo || { country: "" };

	return await retryAsyncFunction(async () => {
		const results = await getJson({
			engine: "google",
			api_key: SERPAPI_API_KEY,
			q: query,
			gl: countryCode,
			num: NUMBER_OF_RESULTS + 1, // Bug: API returns 1 less than requested
			timeout: timeout,
		});

		return convertSerpApiResultsToOrganicResults(results.organic_results ?? []);
	}, retries);
};
2. Google Custom Search (with Key Rotation)

File: src/server/services/webSearch.ts:433

TypeScript
const googleSearch = async (
	query: string,
	timeout: number,
	retries: number,
	webSearchMetadata: TWebSearchMetadata,
): Promise<TOrganicResult[]> => {
	const { country: countryCode } = webSearchMetadata.geo || { country: "" };

	return await retryAsyncFunction(async () => {
		if (webSearchMetadata.googleApiExhausted) {
			return [];
		}

		const api_key = await getGoogleApiKey();
		if (!api_key) {
			webSearchMetadata.googleApiExhausted = true;
			return [];
		}

		const results = await google.customsearch("v1").cse.list(
			{
				key: api_key,
				q: query,
				cx: GOOGLE_CX,
				gl: countryCode,
				...(webSearchMetadata.acceptLanguage && {
					hl: webSearchMetadata.acceptLanguage,
				}),
				num: NUMBER_OF_RESULTS,
			},
			{ timeout: timeout },
		);

		return convertGoogleResultsToOrganicResults(results.data.items ?? []);
	}, retries);
};

Key Rotation System:

TypeScript
// 5 API keys for 9900 requests/day each = 49,500 total
const GOOGLE_API_KEY = [
	"AIzaSyCauYABWT_StWp0qTuL__tiC1XeCJtAHu4", // foyer-api
	"AIzaSyB4cG8NzAGEoDTf1GxTNWCNZIsh9ehdm7U", // foyer-dev
	"AIzaSyB701kAXDhrhY3VJl8cHevqvZ0GagDxhc0", // foyer-prod
	"AIzaSyBoKUtl5At8iF64k9ATfIZn8akHc6PVhqI", // notekeeper
	"AIzaSyA4-f6o5pc6uWv2z4UjUGgqbNUqc3JcD9o", // merlin
];

async function getGoogleApiKey(): Promise<string | null> {
	for (const key of GOOGLE_API_KEY) {
		const redisKey = `${GOOGLE_API_KEY_PREFIX}:${key}`;

		// Initialize counter for new day
		if ((await redis.exists(redisKey)) == 0) {
			const now = DateTime.now().setZone("America/Los_Angeles");
			const nextMidnight = now.plus({ days: 1 }).startOf("day");
			const result = await redis.set(redisKey, 0, "NX");
			if (result === "OK") {
				await redis.pexpireat(redisKey, nextMidnight.toMillis());
			}
		}

		// Increment and check
		const count = await redis.incr(redisKey);
		if (count <= GOOGLE_API_SEARCH_LIMIT) {
			return key;
		}
	}
	return null; // All keys exhausted
}

Why Key Rotation:

  • Google limits: 10,000 requests/day per key
  • 5 keys = 49,500 daily capacity
  • Redis tracks usage, expires at midnight PST
  • Automatic failover to next key
3. Google + Retext (Keyword Extraction)

File: src/server/services/webSearch.ts:483

TypeScript
const googleSearchWithRetext = async (
	query: string,
	timeout: number,
	retries: number,
	webSearchMetadata: TWebSearchMetadata,
): Promise<TOrganicResult[]> => {
	const keywords: string[] = [];

	if (webSearchMetadata.googleApiExhausted) {
		return [];
	}

	try {
		const output = await retext()
			.use(retextPos) // Part-of-speech tagging
			.use(retextKeywords)
			.process(query);

		if (output.data.keywords) {
			for (const keyword of output.data.keywords) {
				keywords.push(toString(keyword.matches[0].node));
			}
		}
	} catch {
		logger.warn(`ERROR/WEB_ACCESS/RETEXT_FAILED`);
		return [];
	}

	// Use keywords if found, otherwise original query
	const keywordsQuery = keywords.length > 0 ? keywords.join(" ") : query;
	return await googleSearch(keywordsQuery, timeout, retries, webSearchMetadata);
};

Why Retext:

  • Extracts key terms from long/complex queries
  • Better results for natural language questions
  • "What are the benefits of meditation for anxiety" → "meditation anxiety benefits"
4. Bing Search

File: src/server/services/webSearch.ts:547

TypeScript
const searchBing = async (
	query: string,
	timeout: number,
	retries: number,
	webSearchMetadata: TWebSearchMetadata,
): Promise<TOrganicResult[]> => {
	const { req }: TBasicRequestContext = requestContext.get();
	const {
		country: countryCode,
		area,
		ll,
	} = webSearchMetadata.geo || { country: "", area: 0, ll: [] };

	const latitude = ll[0];
	const longitude = ll[1];
	const radiusInMeters = area * 1000;
	const userAgent = req.headers["user-agent"];
	const xSearchLocation = `lat:${latitude};long:${longitude};re:${radiusInMeters}`;

	// Bing has 1500 char URL limit
	if (query.length > 1300) {
		query = query.slice(0, 1000) + query.slice(-300);
	}

	const encodedParams = new URLSearchParams({ q: query });
	const slicedParams = encodedParams.toString().slice(0, 1300);
	const newParams = new URLSearchParams(slicedParams);
	newParams.append("count", NUMBER_OF_RESULTS.toString());

	if (countryCode && webSearchMetadata.acceptLanguage) {
		newParams.append("cc", countryCode);
	}

	const headers: Record<string, string> = {};
	if (userAgent) headers["User-Agent"] = userAgent;
	if (webSearchMetadata.acceptLanguage)
		headers["Accept-Language"] = webSearchMetadata.acceptLanguage;
	if (webSearchMetadata.ip) headers["X-MSEdge-ClientIP"] = webSearchMetadata.ip;
	if (latitude && longitude) headers["X-Search-Location"] = xSearchLocation;

	return await retryAsyncFunction(async () => {
		const response = await axiosInstance.get(
			`https://api.bing.microsoft.com/v7.0/search`,
			{
				params: newParams,
				headers: {
					"Ocp-Apim-Subscription-Key": BING_API_KEY,
					...headers,
				},
				timeout: timeout,
			},
		);

		const webPages: TWebPage[] = response?.data?.webPages?.value;
		const news =
			response?.data?.news?.value?.map((item: TNewsItem) => ({
				...item,
				isNews: true,
			})) || [];

		return convertBingResultsToOrganicResults(webPages, news);
	}, retries);
};

Bing Features:

  • News results included (marked with isNews: true)
  • Geo-location headers for local results
  • User agent passthrough

Result Normalization

All providers convert to common TOrganicResult format:

TypeScript
type TOrganicResult = {
    url: string;                    // Full URL
    displayed_url: string;         // URL for display
    description: string;            // Snippet/description
    position: number;             // Result rank
    title: string;
    domain: string;
    video?: {                      // YouTube videos
        id: string;
        thumbnail: string;
        author?: string;
        views?: string;
    };
    sitelinks: { inline: Sitelink[] };
    rich_snippet: { top: {...} };
    date: string | null;          // Published date
    date_utc: string | null;
};

YouTube Video Extraction: All providers extract YouTube metadata:

TypeScript
// From URL: youtube.com/watch?v=VIDEO_ID
const videoId = new URL(result.link).searchParams.get("v") ?? "";
const thumbnail = `http://img.youtube.com/vi/${videoId}/maxresdefault.jpg`;

Query Filtering

File: src/server/services/webSearch.ts:204

TypeScript
function filterQuery(query: string): string {
	// Don't double-filter if already has site filter
	if (query.includes("site:")) {
		return query;
	}

	// Exclude reddit from results (quality control)
	const sitesToExclude = ["reddit.com/r/"];
	sitesToExclude.forEach((site) => {
		query += ` -site:${site}`;
	});

	return query;
}

Why Exclude Reddit:

  • Variable quality, often not authoritative
  • Can be excluded via -site:reddit.com/r/
  • Keeps reddit.com (main site) but removes subreddits

Firecrawl Web Search (Alternative)

File: src/server/services/webSearch.ts:723

For deep research, an alternative AI-powered search:

TypeScript
export const firecrawlWebSearch = async (
	query: string,
): Promise<TOrganicResult[]> => {
	const filteredQuery = filterQuery(query);

	const results = await firecrawl.search(filteredQuery, {
		limit: NUMBER_OF_RESULTS,
	});

	return results.data.map((result) => ({
		date: null,
		description: result.description ?? "",
		title: result.title ?? "",
		displayed_url: result.url ?? "",
		domain: "",
		url: result.url ?? "",
		date_utc: null,
		position: 0,
		video: undefined,
		sitelinks: undefined,
		rich_snippet: undefined,
	}));
};

Used in deep research for AI-curated results.

Search Engine Enum

File: src/server/services/webSearch.ts:160

TypeScript
export enum SearchEngine {
	BING = "BING",
	GOOGLE = "GOOGLE",
	GOOGLE_WITH_RETEXT = "GOOGLE_WITH_RETEXT",
	SERP_API = "SERP_API",
}

type TSearchEngineFlow = {
	searchEngine: SearchEngine;
	timeout: number; // Milliseconds
	retries: number; // Retry attempts
};

type TSearchEngineHandlers = {
	[K in SearchEngine]: (
		query: string,
		timeout: number,
		retries: number,
		webSearchMetadata: TWebSearchMetadata,
	) => Promise<TOrganicResult[]>;
};

const searchEngineHandlers: TSearchEngineHandlers = {
	[SearchEngine.BING]: searchBing,
	[SearchEngine.GOOGLE]: googleSearch,
	[SearchEngine.GOOGLE_WITH_RETEXT]: googleSearchWithRetext,
	[SearchEngine.SERP_API]: serpApiSearch,
};

Web Search Tool Integration

The search system is exposed as a tool:

File: src/server/endpoints/unified/tools/webSearch.tool.ts

TypeScript
export const webSearchTool = createTool({
	name: "web_search_tool",
	description: "Search the web for current information",
	parameters: z.object({
		query: z.string(),
		focus: z.enum(["YOUTUBE", "SOCIAL", "ACADEMIC", "DEFAULT"]).optional(),
	}),
	execute: async ({ query, focus }, ctx, progress) => {
		// Adjust query based on focus mode
		let searchQuery = query;
		if (focus === "YOUTUBE") {
			searchQuery += " site:youtube.com";
		} else if (focus === "SOCIAL") {
			searchQuery += " site:reddit.com OR site:twitter.com";
		} else if (focus === "ACADEMIC") {
			searchQuery += " site:arxiv.org OR site: scholar.google.com";
		}

		// Perform search
		const results = await webSearch(searchQuery);

		// Get detailed content from top results
		const detailedResults = await getDetailedResults(results.slice(0, 3));

		return {
			searchResults: results,
			detailedContent: detailedResults,
			citations: results.map((r, i) => `[${i + 1}] ${r.title} - ${r.url}`),
		};
	},
});

Focus Modes:

  • YOUTUBE: Video-focused search
  • SOCIAL: Reddit/Twitter discussions
  • ACADEMIC: arXiv, Google Scholar
  • DEFAULT: General web

Key Design Decisions

1. Speed First
  • SerpAPI (2s) before Google (2s)
  • Fast timeouts with retries
  • Fail fast, fail over
2. Automatic Failover
  • No single point of failure
  • 4 providers in cascade
  • Empty results trigger next provider
3. Geo-Location
  • IP-based country detection
  • Language headers passed through
  • Local results when available
4. Rate Limit Management
  • Google: Redis-based key rotation
  • SerpAPI: No limits (paid tier)
  • Bing: Single key, monitored
5. Result Quality
  • Reddit exclusion
  • YouTube metadata extraction
  • News result integration
  • Normalized format across providers

Integration with Orchestrator

Search is a tool called by the orchestrator:

TypeScript
// In toolOrchestrator.ts run() method
const toolCalls = await getTokenizedToolCalls<TToolCall>(toolCallParsed.data);

for await (const chunk of this.executeRequestedTools(toolCalls, ...)) {
    // chunk.type === "tool:done"
    // result contains web search results
}

The search service is stateless - each call is independent. Results are immediately streamed to the client and added to the conversation context.