Inkdown
Start writing

Merlin Backend

12 files·0 subfolders

Shared Workspace

Merlin Backend
01-Orchestration.md

02-DeepResearch

Shared from "Merlin Backend" on Inkdown

Deep Research Architecture

Overview

Deep Research is Arcane's most sophisticated feature - an autonomous research agent that performs multi-step investigation, synthesis, and reporting. Unlike simple web search, it creates research plans, executes them iteratively, validates findings, and generates comprehensive reports.

Architecture Overview

Plain text
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft

Entry Point: handleDeepResearch

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:1036

This 1589-line orchestration function manages the entire research lifecycle:

TypeScript

Web Search Step Handler

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:67

The core iterative research function (handles both independent and hybrid steps):

TypeScript

Research Memory System

File: src/server/endpoints/unified/features/deepResearch/memory/index.ts

The memory system tracks insights, knowledge gaps, and contradictions across all research:

TypeScript

Memory Structure:

  • Insights: Key findings extracted from sources (with confidence scores)
  • Knowledge Gaps: Questions that need answers
  • Contradictions: Conflicting information across sources
  • Step Confidence: Per-step completion confidence tracking

Usage in Research:

  • Context elements scored by relevance to current step
  • High-relevance insights (0.8+) prioritized
  • Gaps drive additional iterations
  • Contradictions flagged for final validation

LLM-Only Step Handler

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:654

For dependent steps that synthesize without web search:

TypeScript

Key Design Decisions

1. Three-Phase Execution
  • Independent: Parallel, fastest results first
  • Hybrid: Sequential but uses accumulated context
  • Dependent: Pure analysis, no external search
2. Iterative Per-Step Research

Each step iterates up to 5 times:

  • Search with progressively refined queries
  • Context from previous iterations
  • Confidence threshold (0.6+) to proceed
  • Dynamic step creation if incomplete
3. Research Memory
  • Tracks insights, gaps, contradictions
  • Scored context elements (relevance 0-1)
  • Deduplication prevents bloat
  • Used across all steps
4. Time Budget Management
  • Global time limit (e.g., 5 minutes)
  • Checked before starting each phase
  • Graceful exit with partial results
  • Progress updates every second
5. Error Resilience
  • Try-catch around every major operation
  • Continue on error, don't fail entirely
  • Log everything for debugging
  • Fallback to empty results if needed

Integration with Main Orchestrator

Deep Research runs as a sub-agent spawned by main orchestrator:

TypeScript

The deep research system is a nested orchestrator within the main orchestrator, with its own state, memory, and execution logic.