Inkdown
Start writing

Study

70 filesยท12 subfolders

Shared Workspace

Study
AI eng

02-DeepResearch

Shared from "Study" on Inkdown

Deep Research Architecture

Overview

Deep Research is Arcane's most sophisticated feature - an autonomous research agent that performs multi-step investigation, synthesis, and reporting. Unlike simple web search, it creates research plans, executes them iteratively, validates findings, and generates comprehensive reports.


Architecture Overview

Plain text
basic-ques
core
Revision w/ Whiteboard
CN Basics - 1
CN Basics - 2
DNS
Event loop
programming-language-concepts.md
zero-language-explanation.md
DB
Quick
databases-deep-dive.md
01-introduction.md
02-relational-databases.md
03-database-design.md
04-indexing.md
05-transactions-acid.md
06-nosql-databases.md
07-query-optimization.md
08-replication-ha.md
09-sharding-partitioning.md
10-caching-strategies.md
11-cap-theorem.md
12-connection-pooling.md
13-backup-recovery.md
14-monitoring.md
15-database-selection.md
README.md
JS
core topics
Event loop
Merlin Backend
01-Orchestration.md
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft
Mobile
Build Alternative
Bundling
metro-bundler-deep-dive.md
OpenAI Agents Python
00_OVERVIEW.md
01_AGENT_SYSTEM.md
02_RUNNER_SYSTEM.md
03_TOOL_SYSTEM.md
04_ITEMS_SYSTEM.md
05_GUARDRAILS.md
06_HANDOFFS.md
07_MEMORY_SESSIONS.md
08_MODEL_PROVIDERS.md
09_SANDBOX_SYSTEM.md
10_TRACING.md
11_RUN_STATE.md
12_CONTEXT.md
13_LIFECYCLE_HOOKS.md
14_CONFIGURATION.md
15_ERROR_HANDLING.md
16_STREAMING.md
17_EXTENSIONS.md
18_MCP_INTEGRATION.md
19_BEST_PRACTICES.md
20_ARCHITECTURE_PATTERNS.md
opencode-study
context-handling
core
Python
Alembic
Basics
sqlalchemy - fastapi
SQLAlchemy overview
tweets
system_design_for_agentic_apps.md
Agent Loop

Entry Point: handleDeepResearch

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:1036

This 1589-line orchestration function manages the entire research lifecycle:

TypeScript

Web Search Step Handler

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:67

The core iterative research function (handles both independent and hybrid steps):

TypeScript

Research Memory System

File: src/server/endpoints/unified/features/deepResearch/memory/index.ts

The memory system tracks insights, knowledge gaps, and contradictions across all research:

TypeScript

Memory Structure:

  • Insights: Key findings extracted from sources (with confidence scores)
  • Knowledge Gaps: Questions that need answers
  • Contradictions: Conflicting information across sources
  • Step Confidence: Per-step completion confidence tracking

Usage in Research:

  • Context elements scored by relevance to current step
  • High-relevance insights (0.8+) prioritized
  • Gaps drive additional iterations
  • Contradictions flagged for final validation

LLM-Only Step Handler

File: src/server/endpoints/unified/features/deepResearch/deepResearch.ts:654

For dependent steps that synthesize without web search:

TypeScript

Key Design Decisions

1. Three-Phase Execution
  • Independent: Parallel, fastest results first
  • Hybrid: Sequential but uses accumulated context
  • Dependent: Pure analysis, no external search
2. Iterative Per-Step Research

Each step iterates up to 5 times:

  • Search with progressively refined queries
  • Context from previous iterations
  • Confidence threshold (0.6+) to proceed
  • Dynamic step creation if incomplete
3. Research Memory
  • Tracks insights, gaps, contradictions
  • Scored context elements (relevance 0-1)
  • Deduplication prevents bloat
  • Used across all steps
4. Time Budget Management
  • Global time limit (e.g., 5 minutes)
  • Checked before starting each phase
  • Graceful exit with partial results
  • Progress updates every second
5. Error Resilience
  • Try-catch around every major operation
  • Continue on error, don't fail entirely
  • Log everything for debugging
  • Fallback to empty results if needed

Integration with Main Orchestrator

Deep Research runs as a sub-agent spawned by main orchestrator:

TypeScript

The deep research system is a nested orchestrator within the main orchestrator, with its own state, memory, and execution logic.