InkdownInkdown
Start writing

Merlin Backend

12 files·0 subfolders

Shared Workspace

Merlin Backend
01-Orchestration.md

02-DeepResearch

Shared from "Merlin Backend" on Inkdown

Deep Research Architecture

Overview

Deep Research is Arcane's most sophisticated feature - an autonomous research agent that performs multi-step investigation, synthesis, and reporting. Unlike simple web search, it creates research plans, executes them iteratively, validates findings, and generates comprehensive reports.

Architecture Overview

Plain text
User Query
    ↓
┌─────────────────────────────────────────────────────────────────┐
│              PHASE 0: INITIALIZATION                             │
│  1. Generate conversation summary (understand intent)           │
│  2. Check if feedback needed (clarify ambiguous queries)        │
│  3. Initialize research memory (insights, gaps, contradictions) │
│  4. Get internet context (1 broad search for orientation)        │
└─────────────────────────────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────────────────────────────┐
│              PHASE 1: RESEARCH PLANNING                          │
│  Generate structured plan with 3 step types:                      │
│                                                                  │
│  Independent Steps (Parallel)                                    │
│  └── Can execute simultaneously (no dependencies)                │
│                                                                  │
│  Hybrid Steps (Sequential with Context)                           │
│  └── Need web search + previous step results                     │
│                                                                  │
│  Dependent Steps (LLM Only)                                       │
│  └── Pure analysis, no web search needed                         │
└─────────────────────────────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────────────────────────────┐
│              PHASE 2: EXECUTION (3 Sub-phases)                   │
│                                                                  │
│  2A. Independent Steps → Parallel execution                      │
│       Each: Search → Process Results → Extract Insights          │
│                                                                  │
│  2B. Hybrid Steps → Sequential with accumulated context        │
│       Use independent results + new searches                     │
│                                                                  │
│  2C. Dependent Steps → Sequential LLM-only analysis             │
│       Synthesize all previous findings                           │
└─────────────────────────────────────────────────────────────────┘
    ↓
┌─────────────────────────────────────────────────────────────────┐
│              PHASE 3: SYNTHESIS & REPORTING                        │
│  1. Final validation (check for inconsistencies)                 │
│  2. Aggregate learnings by URL                                   │
│  3. Format for final report                                      │
│  4. Generate report via LLM                                      │
│  5. Create attachments (sources)                                 │
└─────────────────────────────────────────────────────────────────┘
    ↓
Comprehensive Research Report + Citations
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft

Entry Point: handleDeepResearch

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:1036

This 1589-line orchestration function manages the entire research lifecycle:

TypeScript
export async function handleDeepResearch(
	allMessages: TPromptMessage[],
	chatStateManager: ChatStateManager,
) {
	const { eventManager, assistantMessageNode } =
		getDeepResearchRequestContext();

	// Initialize research memory
	initializeResearchMemory();

	// Randomized UI messages for better UX
	const initialMessages = [
		"I'm understanding what you're asking about",
		"I'm figuring out exactly what you need help with",
		"I'm analyzing your question to understand it fully",
		"I'm processing your request to get started",
		"I'm interpreting your question for best results",
	];
	const randomInitial =
		initialMessages[Math.floor(Math.random() * initialMessages.length)];
	const deepResearchEvent = eventManager.createEvent(
		randomInitial,
		"LIGHTBULB",
	);

	// PHASE 0: Get initial internet context
	const internetContext: TLearning = await getInternetContext(allMessages);

	// Check if we need user feedback for ambiguous queries
	const shouldAskFeedback = await feedbackDecider(allMessages);
	deepResearchEvent.end();

	if (shouldAskFeedback) {
		const feedbackEvent = eventManager.createEvent(
			"I'd like to clarify some details",
			"SHAPES",
		);
		const feedbackPrompt = generateFollowUpsPrompt(
			allMessages,
			internetContext.learning,
		);
		feedbackEvent.end();
		return feedbackPrompt; // Early return - need more info from user
	}

	// Mark as using deep research mode
	chatStateManager.addUsedMode("DEEP_RESEARCH");

	// Initialize time tracking
	const startTime = new Date();
	setDeepResearchRequestContext({
		...getDeepResearchRequestContext(),
		isDeepResearch: true,
		startTime,
		globalStepCount: 0,
	});

	// Time limit checker (e.g., 5 minutes max)
	const hasExceededGlobalTimeLimit = () => {
		const elapsedMs = new Date().getTime() - startTime.getTime();
		return elapsedMs >= DEEP_RESEARCH_GLOBAL_TIME_LIMIT_MS;
	};

	// Start research phase
	const deepResearchInitialEvent = eventManager.createEvent(
		"Research Started",
		"SHAPES",
	);

	// Live progress updates every second
	const updateInterval = setInterval(() => {
		const duration = countDeepResearchDuration(startTime);
		streamer.appendEvent("progress", {
			payload: {
				id: deepResearchInitialEvent.id,
				name: `Researching in progress - ${duration}`,
				status: deepResearchInitialEvent.status,
				values: deepResearchInitialEvent.values,
				icon: deepResearchInitialEvent.icon,
			},
		});
	}, 1000);

	try {
		// Generate conversation summary for context
		const conversationSummaryEvent = eventManager.createEvent(
			"Understanding the user's intent, and thinking about how to proceed",
			"BRAIN",
		);
		const conversationSummary = await generateConversationSummary(allMessages);

		conversationSummaryEvent.send([
			{
				type: "TEXT",
				value: JSON.stringify({
					__type: "DEEP_RESEARCH_SUMMARY",
					data: { summary: conversationSummary },
				}),
			},
		]);
		conversationSummaryEvent.end();

		// PHASE 1: Generate research plan
		const planCreationEvent = eventManager.createEvent(
			"Understanding User's Query and trying to create a robust deep research plan",
			"BRAIN",
		);

		const linkLearnings = await getUserLinkLearning(
			chatStateManager,
			conversationSummary,
		);
		const plan = await generateDeepResearchPlan(
			chatStateManager,
			linkLearnings,
			internetContext,
		);
		planCreationEvent.end();

		const allLearnings: TLearning[] = linkLearnings;

		// Store plan in context
		setDeepResearchRequestContext({
			...getDeepResearchRequestContext(),
			deepResearchPlan: plan,
		});

		// Categorize steps
		const independentSteps = plan.filter(
			(s) => s.requiresWebSearch && !s.doesRequirePreviousData,
		);
		const hybridSteps = plan.filter(
			(s) => s.requiresWebSearch && s.doesRequirePreviousData,
		);
		const dependentSteps = plan.filter((s) => !s.requiresWebSearch);

		logger.info(
			{
				independent: independentSteps.length,
				hybrid: hybridSteps.length,
				dependent: dependentSteps.length,
				total: plan.length,
			},
			"DEEP_RESEARCH/STEPS_CLASSIFICATION",
		);

		// PHASE 2A: Independent Steps (Parallel)
		if (!hasExceededGlobalTimeLimit()) {
			const stepsToProcess = [];
			for (const step of independentSteps) {
				if (hasExceededGlobalTimeLimit()) {
					logger.info(
						{ phase: "independent_steps", stepId: step.stepId },
						"DEEP_RESEARCH/GLOBAL_TIME_LIMIT_REACHED",
					);
					break;
				}
				stepsToProcess.push(
					handleWebSearchStep(
						step,
						allMessages,
						allLearnings,
						conversationSummary,
						{
							depth: 0,
							maxDepth: 2,
							parentChain: [],
						},
					),
				);
			}
			await settleAll(stepsToProcess);
		}

		// PHASE 2B: Hybrid Steps (Sequential with context)
		if (!hasExceededGlobalTimeLimit() && hybridSteps.length > 0) {
			const filteredHybridSteps = [];
			for (const step of hybridSteps) {
				if (hasExceededGlobalTimeLimit()) {
					logger.info(
						{ phase: "hybrid_steps", stepId: step.stepId },
						"DEEP_RESEARCH/GLOBAL_TIME_LIMIT_REACHED_DURING_STEP_LAUNCH",
					);
					break;
				}
				filteredHybridSteps.push(
					handleWebSearchStep(
						step,
						allMessages,
						allLearnings,
						conversationSummary,
						{
							depth: 0,
							maxDepth: 2,
							parentChain: [],
						},
					),
				);
			}
			await settleAll(filteredHybridSteps);
		}

		// PHASE 2C: Dependent Steps (LLM-only, sequential)
		if (!hasExceededGlobalTimeLimit()) {
			let currentPlan = [...dependentSteps];
			let previousStepContext = undefined;

			while (currentPlan.some((step) => step.status !== "COMPLETED")) {
				const nextStep = currentPlan.find((step) => step.status === "PENDING");
				if (!nextStep) break;

				if (hasExceededGlobalTimeLimit()) {
					logger.info(
						{
							stepId: nextStep.stepId,
							remainingSteps: currentPlan.filter((s) => s.status === "PENDING")
								.length,
						},
						"DEEP_RESEARCH/GLOBAL_TIME_LIMIT_REACHED",
					);
					break;
				}

				if (nextStep.requiresWebSearch) {
					await handleWebSearchStep(
						nextStep,
						allMessages,
						allLearnings,
						conversationSummary,
						{
							depth: 0,
							maxDepth: 2,
							parentChain: [],
							previousStepContext,
						},
					);
				} else {
					await handleLLMOnlyStep(
						nextStep,
						allMessages,
						allLearnings,
						conversationSummary,
						{
							depth: 0,
							maxDepth: 3,
							parentChain: [],
							previousStepContext,
						},
					);
				}

				updateStepStatus(nextStep.stepId, "COMPLETED");

				// Update context for next dependent step
				const updatedContext = getDeepResearchRequestContext();
				const stepWithContext = updatedContext.deepResearchPlan.find(
					(s) => s.stepId === nextStep.stepId,
				);
				if (stepWithContext?.previousStepContext) {
					previousStepContext = {
						reasoning: stepWithContext.previousStepContext.reasoning,
						keyMatchingElements:
							stepWithContext.previousStepContext.keyMatchingElements,
						scoredElements:
							stepWithContext.previousStepContext.scoredElements?.map(
								(elem) => ({
									content: elem.content,
									relevance: elem.relevance,
									source: elem.source,
									type: elem.type,
								}),
							) || [],
					};
				} else {
					previousStepContext = undefined;
				}

				currentPlan = updatedContext.deepResearchPlan.filter(
					(step) => step.doesRequirePreviousData && step.status !== "COMPLETED",
				);
			}
		}

		// PHASE 3: Final validation
		try {
			const { relevantInsights, relevantGaps, relevantContradictions } =
				await getRelevantContextForStep(
					{
						title: "Final Research Validation",
						task: conversationSummary,
						stepId: "final-validation",
						status: "PENDING",
						description:
							"Determine which elements are most relevant to the current research step",
						requiresWebSearch: false,
						doesRequirePreviousData: false,
					},
					50,
				);

			if (relevantInsights.length > 0) {
				const validation = await performResearchValidation(
					relevantInsights,
					relevantGaps,
					relevantContradictions,
				);
				if (validation?.inconsistencies?.length > 0) {
					setDeepResearchRequestContext({
						...getDeepResearchRequestContext(),
						researchValidation: validation,
					});
				}
			}
		} catch (error) {
			logger.error(error, "ERROR/DEEP_RESEARCH/FINAL_VALIDATION");
		}

		// Aggregate learnings by URL
		const aggregatedLearnings = allLearnings.reduce<TLearning[]>(
			(acc, curr) => {
				const existing = acc.find(
					(l) => l.metadata?.sourceUrl === curr.metadata?.sourceUrl,
				);
				if (existing) {
					existing.learning += `\n${curr.learning}`;
					return acc;
				} else {
					return [...acc, curr];
				}
			},
			[],
		);

		const learnings = aggregatedLearnings.map((learning, index) => ({
			...learning,
			id: index + 1,
		}));

		// Format for report
		const [formattedLearnings, userIntent] = await Promise.all([
			formatLearningsForReport(learnings),
			getUserIntent(chatStateManager),
		]);

		const finalUserIntent =
			userIntent?.length < 1
				? [
						`Understand and research about ${chatStateManager.input.message.content}`,
					]
				: userIntent;

		// Get report framing from LLM
		const reportMessage = await getReportMessageFromLLM(
			finalUserIntent,
			learnings,
			allMessages,
		);
		const reportEvent = eventManager.createEvent(
			reportMessage.mainMessage,
			"BRAIN",
		);
		reportEvent.send([
			{
				type: "TEXT",
				value: JSON.stringify({
					__type: "DEEP_RESEARCH_FINAL_REPORT_MESSAGE",
					data: {
						mainMessage: reportMessage.mainMessage,
						reasoning: reportMessage.reasoning,
					},
				}),
			},
		]);
		reportEvent.end();

		// Generate final report prompt
		const reportPrompt = getGenerateReportPrompt(
			finalUserIntent,
			allMessages,
			formattedLearnings,
		);

		// Add attachments (sources)
		assistantMessageNode.pushAttachments(createAttachments(learnings));
		streamer.appendEvent("attachments", {
			payload: createStreamerPayload(learnings),
		});

		return reportPrompt;
	} catch (error) {
		logger.error(error, "ERROR/DEEP_RESEARCH/PROCESS_FAILURE");
		throw error;
	} finally {
		const finalDuration = countDeepResearchDuration(startTime);
		deepResearchInitialEvent.end();
		clearInterval(updateInterval);

		// Create final event with correct duration
		const finalEvent = eventManager.createEvent(
			`Research completed - ${finalDuration}`,
			"SHAPES",
		);
		finalEvent.end();
	}
}

Web Search Step Handler

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:67

The core iterative research function (handles both independent and hybrid steps):

TypeScript
async function handleWebSearchStep(
	step: TDeepResearchPlanStep,
	allMessages: TPromptMessage[],
	allLearnings: TLearning[],
	conversationSummary: string,
	executionContext: TStepExecutionContext = {
		depth: 0,
		maxDepth: 2,
		parentChain: [],
	},
): Promise<void> {
	// Prevent infinite recursion
	if (executionContext.depth >= executionContext.maxDepth) return;
	if (executionContext.parentChain.includes(step.stepId)) return; // Circular check

	const eventManager = getDeepResearchRequestContext().eventManager;
	let isStepComplete = false;
	let iterationCount = 0;
	const maxIterations = 5;
	let allProcessedResults: TLearning[] = [];
	let serpQueriesEvent: SSEProgressEvent | null = null;
	let stepConfidence = 0;

	// Enhanced context accumulates across iterations
	const enhancedContext = {
		reasoning: executionContext.previousStepContext?.reasoning || "",
		keyMatchingElements:
			executionContext.previousStepContext?.keyMatchingElements || [],
		scoredElements: executionContext.previousStepContext?.scoredElements || [],
	};

	// ITERATIVE RESEARCH LOOP (up to 5 iterations per step)
	while (!isStepComplete && iterationCount <= maxIterations) {
		iterationCount++;

		// Randomized activity message for UI
		const searchActivities = [
			`Searching online for ${step.title} information`,
			`Looking up the latest details about ${step.title}`,
			`Researching key aspects of ${step.title}`,
			`Finding relevant sources about ${step.title}`,
			`Gathering information on ${step.title} from trusted sources`,
		];
		const randomActivity =
			searchActivities[Math.floor(Math.random() * searchActivities.length)];

		try {
			serpQueriesEvent = eventManager.createEvent(randomActivity, "GLOBE");
			updateStepStatus(step.stepId, "IN_PROGRESS");

			// Get relevant learnings from previous steps
			const relevantLearnings = getRelevantLearnings(step, allLearnings);

			// Get context from research memory
			let researchContext = "";
			try {
				const { relevantInsights, relevantGaps, relevantContradictions } =
					await getRelevantContextForStep(step);

				if (relevantInsights.length > 0 || relevantGaps.length > 0) {
					researchContext = formatContextForPrompt(
						relevantInsights,
						relevantGaps,
						relevantContradictions,
					);

					// Add to scored elements for priority
					for (const insight of relevantInsights) {
						enhancedContext.scoredElements.push({
							content: insight.content,
							relevance: insight.confidence,
							source: `insight-${insight.id}`,
							type: "insight",
						});
					}
					for (const gap of relevantGaps) {
						enhancedContext.scoredElements.push({
							content: gap.question,
							relevance: gap.relevance,
							source: `gap-${gap.id}`,
							type: "knowledgeGap",
						});
					}
				}
			} catch (error) {
				logger.error(error, "ERROR/DEEP_RESEARCH/RESEARCH_MEMORY_CONTEXT");
			}

			// Score and prioritize context elements (after first iteration)
			if (iterationCount > 1 && enhancedContext.scoredElements.length > 0) {
				try {
					const result =
						await ResearchMemoryManager.scoreContextElementsForStep(
							enhancedContext.scoredElements.map((e) => ({
								content: e.content,
								relevance: e.relevance,
								source: e.source,
								type: e.type,
							})),
							step,
						);
					enhancedContext.scoredElements = result.map((item) => ({
						content: item.content,
						relevance: item.score || 0.5,
						source: item.metadata?.source || "",
						type:
							item.type === "knowledgeGap"
								? "knowledgeGap"
								: item.type === "contradiction"
									? "contradiction"
									: "insight",
					}));
				} catch (error) {
					logger.error(error, "ERROR/DEEP_RESEARCH/SCORE_CONTEXT_ELEMENTS");
				}
			}

			// For hybrid steps, inject context from previous step
			if (
				step.doesRequirePreviousData &&
				executionContext.previousStepContext
			) {
				const previousContextString = `
Previous research findings:
${executionContext.previousStepContext.reasoning || ""}

Key matching elements from previous research:
${(executionContext.previousStepContext.keyMatchingElements || []).join("\n")}
`;
				researchContext = researchContext
					? `${researchContext}\n\n${previousContextString}`
					: previousContextString;

				step.description = `${step.description}\n\n[Using context from previous step(s): ${executionContext.parentChain.join(",")}]`;
			}

			// Generate SERP queries with context
			let searchResults: TGeneratedSerpQueries;
			try {
				if (researchContext) {
					enhancedContext.reasoning = `${enhancedContext.reasoning}\n\nAdditional Context:\n${researchContext}`;
				}

				searchResults = await generateSerpQueries(
					step,
					conversationSummary,
					relevantLearnings,
					{
						reasoning: enhancedContext.reasoning,
						keyMatchingElements: enhancedContext.keyMatchingElements,
						scoredElements: enhancedContext.scoredElements,
					},
				);
			} catch (error) {
				logger.error(error, "ERROR/DEEP_RESEARCH/SERP_QUERIES_GENERATION");
				searchResults = {
					serpResults: { research_pairs: [] },
					webResults: [{ task: step.task, query: step.title, results: [] }],
				};
			}

			// Process search results (parallel processing of multiple queries)
			let processedResults: TSettledResults<TLearning[][]>;
			try {
				processedResults = await settleAll(
					searchResults.webResults.map((result) =>
						processSearchResult(result, step, conversationSummary),
					),
				);
			} catch (error) {
				logger.error(error, "ERROR/DEEP_RESEARCH/PROCESS_SEARCH_RESULTS");
				processedResults = { fulfilled: [], rejected: [] };
			}

			// Add results with step tracking
			const newResults = processedResults.fulfilled
				.flat()
				.map((learning: TLearning) => ({ ...learning, stepId: step.stepId }));
			allProcessedResults = [...allProcessedResults, ...newResults];

			// Calculate confidence and extract insights
			try {
				stepConfidence = ResearchMemoryManager.calculateConfidence(
					allProcessedResults,
					iterationCount,
				);
				ResearchMemoryManager.updateStepConfidence(
					step.stepId,
					stepConfidence,
					iterationCount,
				);

				if (stepConfidence > 0.6 && newResults.length > 0) {
					// Extract insights and gaps in one LLM call (efficiency)
					const { insights, gaps } =
						await ResearchMemoryManager.extractInsightsAndGaps(
							step,
							newResults,
						);

					// Add to context for next iteration
					for (const insight of insights) {
						enhancedContext.scoredElements.push({
							content: insight,
							relevance: 0.8,
							source: step.stepId,
							type: "insight",
						});
					}
					for (const gap of gaps) {
						enhancedContext.scoredElements.push({
							content: gap.question,
							relevance: gap.relevance,
							source: step.stepId,
							type: "knowledgeGap",
						});
					}

					// Perform meta-analysis
					const analysisResult = await performMetaAnalysis(step, newResults);
					updatePlanWithMetaAnalysis(step.stepId, analysisResult);

					// Deduplicate to prevent memory bloat
					try {
						ResearchMemoryManager.deduplicateInsights(
							getDeepResearchRequestContext().researchMemory,
						);
					} catch (error) {
						logger.error(error, "ERROR/DEEP_RESEARCH/DEDUPLICATION");
					}
				}
			} catch (error) {
				logger.error(error, "ERROR/DEEP_RESEARCH/EXTRACT_INSIGHTS_AND_GAPS");
			}

			// Send learnings to UI
			if (processedResults.fulfilled.length > 0) {
				const learningsToSend = processedResults.fulfilled
					.flat()
					.map((learning, index) => ({
						...learning,
						learning: learning.learning.slice(0, 100),
						id: index + 1,
						stepId: step.stepId,
					}));
				serpQueriesEvent?.send([
					{
						type: "TEXT",
						value: JSON.stringify({
							__type: "DEEP_RESEARCH_LINK",
							data: {
								query: searchResults.webResults[0]?.query || step.title,
								learnings: learningsToSend,
							},
						}),
					},
				]);
			}

			// Check if step is complete (LLM evaluates results)
			for (const result of searchResults.webResults) {
				const formattedResult = {
					task: result.task,
					query: result.query,
					results: (result.results || []).map((r) => ({
						url: r.url || "",
						title: r.title || "",
						description: r.description || "",
						content:
							typeof r.content === "string"
								? r.content
								: JSON.stringify(r.content || ""),
					})),
				};
				const sanitizedResult = sanitizeWebSearchContent(
					JSON.stringify({ webResults: [formattedResult] }),
				);

				let currentStepDoneResponse;
				try {
					currentStepDoneResponse = await currentStepDone(
						sanitizedResult,
						step,
						allMessages,
					);
					isStepComplete = currentStepDoneResponse.isRelevant;
				} catch (error) {
					logger.error(
						error,
						`ERROR/DEEP_RESEARCH/CHECK_STEP_DONE_${step.stepId}`,
					);
					currentStepDoneResponse = {
						isRelevant: false,
						reasoning: "Error evaluating results",
						keyMatchingElements: [],
					};
				}

				if (!isStepComplete) {
					// Not complete - update context for next iteration
					enhancedContext.reasoning = currentStepDoneResponse.reasoning;
					enhancedContext.keyMatchingElements =
						currentStepDoneResponse.keyMatchingElements || [];

					enhancedContext.scoredElements.push({
						content: currentStepDoneResponse.reasoning,
						relevance: 0.9,
						source: "current_iteration",
						type: "reasoning",
					});
					for (const element of currentStepDoneResponse.keyMatchingElements ||
						[]) {
						enhancedContext.scoredElements.push({
							content: element,
							relevance: 0.8,
							source: "current_iteration",
							type: "insight",
						});
					}

					executionContext.previousStepContext = {
						reasoning: currentStepDoneResponse.reasoning,
						keyMatchingElements:
							currentStepDoneResponse.keyMatchingElements || [],
						scoredElements: enhancedContext.scoredElements,
					};
				}

				serpQueriesEvent?.send([
					{
						type: "TEXT",
						value: JSON.stringify({
							__type: "DEEP_RESEARCH_STEP_DONE",
							data: {
								reasoning: currentStepDoneResponse.reasoning,
								keyMatchingElements:
									currentStepDoneResponse.keyMatchingElements,
							},
						}),
					},
				]);

				if (isStepComplete) break;
			}

			// Final meta-analysis if complete or max iterations
			if (isStepComplete || iterationCount >= maxIterations) {
				if (allProcessedResults.length > 0) {
					const analysisResult = await performMetaAnalysis(
						step,
						allProcessedResults,
					);
					updatePlanWithMetaAnalysis(step.stepId, analysisResult);
				}
			}
		} catch {
			// Continue to next iteration instead of failing
		} finally {
			serpQueriesEvent?.end();
		}
	}

	// EVALUATE OUTCOMES & POTENTIALLY CREATE DYNAMIC STEP
	if (allProcessedResults.length > 0) {
		try {
			const evaluation = await evaluateStepOutcomes(
				step,
				allProcessedResults,
				isStepComplete,
			);
			setDeepResearchRequestContext({
				...getDeepResearchRequestContext(),
				stepEvaluations: {
					...getDeepResearchRequestContext().stepEvaluations,
					[step.stepId]: evaluation,
				},
			});

			// Decide if we need more research
			const shouldCreateDynamicStep =
				!isStepComplete &&
				evaluation.informationCompleteness < 0.7 &&
				evaluation.informationQuality > 0.4 &&
				evaluation.suggestedNextSteps.length > 0;

			if (shouldCreateDynamicStep) {
				// Create dynamic follow-up step
				let dynamicStepTask = "";
				let dynamicStepTitle = "";
				let requiresWebSearch = true;

				if (evaluation.suggestedNextSteps.length > 0) {
					const nextStep = evaluation.suggestedNextSteps[0];
					dynamicStepTitle = nextStep.title;
					dynamicStepTask = nextStep.rationale;
					requiresWebSearch = nextStep.requiresWebSearch;
				}

				// Fall back to pipeline if needed
				if (!dynamicStepTitle || !dynamicStepTask) {
					const updatedPipeline = await updatedDeepResearchPipeline(
						step,
						allProcessedResults,
						conversationSummary,
					);
					if (updatedPipeline?.shouldUpdate) {
						dynamicStepTitle =
							updatedPipeline.updatedPlan?.title || "Additional Research";
						dynamicStepTask = updatedPipeline.updatedPlan?.task || "";
						requiresWebSearch =
							updatedPipeline.updatedPlan?.requiresWebSearch || true;
					}
				}

				if (dynamicStepTitle && dynamicStepTask) {
					const newStep: TDeepResearchPlanStep = {
						stepId: `dynamic-${Date.now()}`,
						title: dynamicStepTitle,
						description: `Follow-up research for: ${step.title}`,
						task: dynamicStepTask,
						status: "COMPLETED",
						requiresWebSearch,
						doesRequirePreviousData: true,
						parentStepId: step.stepId,
					};

					const childContext: TStepExecutionContext = {
						depth: executionContext.depth + 1,
						maxDepth: executionContext.maxDepth,
						parentChain: [...executionContext.parentChain, step.stepId],
						previousStepContext: executionContext.previousStepContext,
					};

					// Execute child step
					if (newStep.requiresWebSearch) {
						await handleWebSearchStep(
							newStep,
							allMessages,
							[...allLearnings, ...allProcessedResults],
							conversationSummary,
							childContext,
						);
					} else {
						await handleLLMOnlyStep(
							newStep,
							allMessages,
							[...allLearnings, ...allProcessedResults],
							conversationSummary,
							childContext,
						);
					}

					updatePlanWithDynamicStep(newStep, step.stepId);
				}
			}
		} catch (evaluationError) {
			logger.error(evaluationError, "ERROR/DEEP_RESEARCH/EVALUATE_OUTCOMES");
		}
	}

	// Add to global learnings
	if (allProcessedResults.length > 0) {
		allLearnings.push(...allProcessedResults);
	}
}

Research Memory System

File: src/server/endpoints/unified/features/deepResearch/memory/index.ts

The memory system tracks insights, knowledge gaps, and contradictions across all research:

TypeScript
export const ResearchMemoryManager = {
	// Core operations
	getMemory: () => getMemory(),
	updateMemory: (memory: TResearchMemory) => updateMemory(memory),
	initializeMemory: () => initializeMemory(),
	cleanupMemory: () => cleanupMemory(),

	// Confidence management
	getStepConfidence: (stepId: string) => getStepConfidence(stepId),
	updateStepConfidence: (
		stepId: string,
		confidence: number,
		iteration: number,
	) =>
		updateStepConfidence(stepId, {
			confidence,
			timestamp: new Date().toISOString(),
			iteration,
		}),
	calculateConfidence: (learnings: TLearning[], iteration: number) =>
		calculateConfidence(learnings, iteration),

	// Context analysis
	scoreContextElementsForStep: (
		elements: TScoredContextElement[],
		step: TDeepResearchPlanStep,
	) => {
		const transformedElements = elements.map((e) => ({
			id: e.source || generateId("contextElement"),
			content: e.content,
			type: e.type,
			score: e.relevance,
			metadata: { source: e.source },
		}));
		return scoreContextElementsForStep(transformedElements, step);
	},

	extractInsightsAndGaps: (
		step: TDeepResearchPlanStep,
		learnings: TLearning[],
	) => extractInsightsAndGaps(step, learnings),

	deduplicateInsights: (memory: TResearchMemory) => deduplicateInsights(memory),
};

Memory Structure:

  • Insights: Key findings extracted from sources (with confidence scores)
  • Knowledge Gaps: Questions that need answers
  • Contradictions: Conflicting information across sources
  • Step Confidence: Per-step completion confidence tracking

Usage in Research:

  • Context elements scored by relevance to current step
  • High-relevance insights (0.8+) prioritized
  • Gaps drive additional iterations
  • Contradictions flagged for final validation

LLM-Only Step Handler

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:654

For dependent steps that synthesize without web search:

TypeScript
async function handleLLMOnlyStep(
	step: TDeepResearchPlanStep,
	allMessages: TPromptMessage[],
	allLearnings: TLearning[],
	conversationSummary: string,
	executionContext: TStepExecutionContext = {
		depth: 0,
		maxDepth: 3,
		parentChain: [],
	},
): Promise<void> {
	if (executionContext.depth >= executionContext.maxDepth) return;

	const eventManager = getDeepResearchRequestContext().eventManager;
	const llmProcessingMessages = [
		`Analyzing information about ${step.title}`,
		`Processing data related to ${step.title}`,
		`Synthesizing insights on ${step.title}`,
		`Examining findings about ${step.title}`,
		`Evaluating information on ${step.title}`,
	];
	const randomActivity =
		llmProcessingMessages[
			Math.floor(Math.random() * llmProcessingMessages.length)
		];
	const llmProcessingEvent = eventManager.createEvent(randomActivity, "BRAIN");

	try {
		updateStepStatus(step.stepId, "IN_PROGRESS");

		let stepLearnings: TLearning[] = [];
		let stepConfidence = 0;
		const enhancedContext = {
			reasoning: executionContext.previousStepContext?.reasoning || "",
			keyMatchingElements:
				executionContext.previousStepContext?.keyMatchingElements || [],
			scoredElements:
				executionContext.previousStepContext?.scoredElements || [],
		};

		if (step.doesRequirePreviousData && allLearnings.length > 0) {
			// Get relevant learnings
			const relevantLearnings = getRelevantLearnings(step, allLearnings);

			// Get context from memory
			try {
				const { relevantInsights, relevantGaps } =
					await getRelevantContextForStep(step);
				if (relevantInsights.length > 0 || relevantGaps.length > 0) {
					for (const insight of relevantInsights) {
						enhancedContext.scoredElements.push({
							content: insight.content,
							relevance: insight.confidence,
							source: `insight-${insight.id}`,
							type: "insight",
						});
					}
					for (const gap of relevantGaps) {
						enhancedContext.scoredElements.push({
							content: gap.question,
							relevance: gap.relevance,
							source: `gap-${gap.id}`,
							type: "knowledgeGap",
						});
					}
				}
			} catch (error) {
				logger.error(error, "ERROR/DEEP_RESEARCH/RESEARCH_MEMORY_CONTEXT");
			}

			// Score context elements
			if (enhancedContext.scoredElements.length > 0) {
				const result = await ResearchMemoryManager.scoreContextElementsForStep(
					enhancedContext.scoredElements.map((e) => ({
						content: e.content,
						relevance: e.relevance,
						source: e.source,
						type: e.type,
					})),
					step,
				);
				enhancedContext.scoredElements = result.map((item) => ({
					content: item.content,
					relevance: item.score || 0.5,
					source: item.metadata?.source || "",
					type:
						item.type === "knowledgeGap"
							? "knowledgeGap"
							: item.type === "contradiction"
								? "contradiction"
								: "insight",
				}));
			}

			// Process with LLM
			const processResults = await processStepWithLLM(step, relevantLearnings, {
				depth: executionContext.depth,
				maxDepth: executionContext.maxDepth,
				parentChain: executionContext.parentChain,
				previousStepContext: {
					reasoning: enhancedContext.reasoning,
					keyMatchingElements: enhancedContext.keyMatchingElements,
					scoredElements: enhancedContext.scoredElements,
				},
			});
			stepLearnings = processResults;

			// Calculate confidence (LLM steps usually single iteration)
			stepConfidence = ResearchMemoryManager.calculateConfidence(
				stepLearnings,
				1,
			);
			ResearchMemoryManager.updateStepConfidence(
				step.stepId,
				stepConfidence,
				1,
			);

			// Extract insights if confidence good
			if (stepConfidence > 0.6 && stepLearnings.length > 0) {
				const { insights, gaps } =
					await ResearchMemoryManager.extractInsightsAndGaps(
						step,
						stepLearnings,
					);

				for (const insight of insights) {
					enhancedContext.scoredElements.push({
						content: insight,
						relevance: 0.8,
						source: step.stepId,
						type: "insight",
					});
				}
				for (const gap of gaps) {
					enhancedContext.scoredElements.push({
						content: gap.question,
						relevance: gap.relevance,
						source: step.stepId,
						type: "knowledgeGap",
					});
				}

				const analysisResult = await performMetaAnalysis(step, stepLearnings);
				updatePlanWithMetaAnalysis(step.stepId, analysisResult);

				try {
					ResearchMemoryManager.deduplicateInsights(
						getDeepResearchRequestContext().researchMemory,
					);
				} catch (error) {
					logger.error(error, "ERROR/DEEP_RESEARCH/DEDUPLICATION");
				}
			}

			// Send learnings to UI
			if (stepLearnings.length > 0) {
				const learningsToSend = stepLearnings.map((learning, index) => ({
					...learning,
					learning: learning.learning.slice(0, 100),
					id: index + 1,
					stepId: step.stepId,
				}));
				llmProcessingEvent.send([
					{
						type: "TEXT",
						value: JSON.stringify({
							__type: "DEEP_RESEARCH_LLM_LEARNING",
							data: {
								query: step.title,
								learnings: learningsToSend,
								confidence: stepConfidence,
							},
						}),
					},
				]);
			}

			// Evaluate and potentially create dynamic step
			if (stepLearnings.length > 0) {
				const evaluation = await evaluateStepOutcomes(
					step,
					stepLearnings,
					true,
				);
				setDeepResearchRequestContext({
					...getDeepResearchRequestContext(),
					stepEvaluations: {
						...getDeepResearchRequestContext().stepEvaluations,
						[step.stepId]: evaluation,
					},
				});

				const shouldCreateDynamicStep =
					evaluation.informationCompleteness < 0.7 &&
					evaluation.informationQuality > 0.4 &&
					evaluation.suggestedNextSteps.length > 0;

				if (shouldCreateDynamicStep) {
					// Similar dynamic step creation as web search handler
					// ... (creates child step with depth+1)
				}
			}

			allLearnings.push(...stepLearnings);
		}
	} catch (error) {
		logger.error(error, "ERROR/DEEP_RESEARCH/LLM_STEP_EXECUTION");
		throw error;
	} finally {
		llmProcessingEvent.end();
	}
}

Key Design Decisions

1. Three-Phase Execution
  • Independent: Parallel, fastest results first
  • Hybrid: Sequential but uses accumulated context
  • Dependent: Pure analysis, no external search
2. Iterative Per-Step Research

Each step iterates up to 5 times:

  • Search with progressively refined queries
  • Context from previous iterations
  • Confidence threshold (0.6+) to proceed
  • Dynamic step creation if incomplete
3. Research Memory
  • Tracks insights, gaps, contradictions
  • Scored context elements (relevance 0-1)
  • Deduplication prevents bloat
  • Used across all steps
4. Time Budget Management
  • Global time limit (e.g., 5 minutes)
  • Checked before starting each phase
  • Graceful exit with partial results
  • Progress updates every second
5. Error Resilience
  • Try-catch around every major operation
  • Continue on error, don't fail entirely
  • Log everything for debugging
  • Fallback to empty results if needed

Integration with Main Orchestrator

Deep Research runs as a sub-agent spawned by main orchestrator:

TypeScript
// In webSearch.tool.ts
const deepResearchOrchestrator = new ToolOrchestrator(
    subChatContext,
    AGENT_NAMES.DEEP_RESEARCH_SUPERVISOR,
    deepResearchToolRegistry,
    deepResearchSupervisorConfig,
    50, // 50 tool call limit for deep research
);

const result = await deepResearchOrchestrator.run({
    input: researchQuery,
    assistantMessageNode: subAssistantNode,
    user: chatCtx.user,
    eventManager: subEventManager,
    request,
    response,
});

// Result bubbles up through tool:done chunk
yield {
    type: "tool:done",
    id: toolCallId,
    function: { name: "deep_research_tool" },
    result: {
        result: "DEEP_RESEARCH_COMPLETE",
        toolMetadata: result.toolMetadata,
        toolCallCount: result.toolCallCount,
        usageConfigArray: result.usageConfigArray,
    },
};

The deep research system is a nested orchestrator within the main orchestrator, with its own state, memory, and execution logic.