Inkdown
Start writing

Merlin Backend

12 files·0 subfolders

Shared Workspace

Merlin Backend
01-Orchestration.md

Orchestration-2nd-draft

Shared from "Merlin Backend" on Inkdown

Comprehensive Orchestration Guide

Table of Contents

  1. Overview
  2. Core Components
  3. The Orchestration Flow
  4. Agent Configuration System
  5. Tool System
  6. State Management
  7. Streaming Architecture
  8. Multi-Agent System
  9. Data Policies
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft
  • Usage Tracking
  • Error Handling
  • Advanced Features

    • Overview

    The orchestration layer is a custom-built, multi-agent tool orchestration system designed for AI assistant interactions. It manages tool execution, agent coordination, streaming responses, and context window optimization - all without relying on external frameworks like LangChain or AutoGPT.

    Key Design Principles
    • Multi-Agent Architecture: Different agents for different use cases (main thread, deep research, researcher)
    • Streaming-First: Real-time streaming of tool execution and LLM responses
    • Context Optimization: Intelligent token management with the token engine
    • Configurable Behavior: Agent configs allow complete customization of orchestration behavior
    • Policy-Based Data Control: Fine-grained control over what data gets stored
    • Graceful Degradation: Comprehensive error handling and fallback mechanisms
    High-Level Architecture
    Plain text

    Core Components

    1. ToolOrchestrator Class

    Location: src/server/endpoints/unified/orchestrator/toolOrchestrator.ts

    The ToolOrchestrator is the heart of the orchestration system. It's a 1065-line class that manages the entire orchestration lifecycle.

    Constructor
    TypeScript

    Parameters:

    • chatCtx: Chat context containing conversation history, model config, and chat state
    • agentName: Which agent to run (MainThreadAgent, DeepResearchSupervisor, ResearcherAgent)
    • toolRegistry: Registry of available tools for this session
    • agentConfig: Configuration object defining agent behavior
    • customToolCallLimit: Optional override for tool call limits
    Key Properties
    TypeScript
    • chatCtx: Holds conversation state, model config, attachments, etc.
    • registry: Manages which tools are available
    • agentName: Determines which agent behavior to use
    • agentConfig: Defines how this agent behaves (hooks, limits, policies)
    • usageConfigArray: Tracks token usage across all iterations
    • currentToolMetadata: Metadata about tools being executed
    • currentExecutingTool: Currently running tool for error reporting
    Main Method: run()

    The run() method is the main entry point that executes the entire orchestration flow.

    TypeScript
    Tool Execution: executeRequestedTools()

    This is an async generator that executes tools and yields streaming results.

    TypeScript

    Stream Chunk Types:

    • tool:progress: Initial progress event with EventManager
    • tool:start: Tool execution begins
    • tool:stream: Streaming tool result (for tools that return streams)
    • tool:done: Tool execution completed with result
    • tool:error: Tool execution failed
    Single Tool Execution: runSingleTool()
    TypeScript
    2. ToolRegistry

    Location: src/server/endpoints/unified/tools/toolRegistry.ts

    The ToolRegistry manages the lifecycle of tools - registration, retrieval, and filtering.

    TypeScript

    Default Tools:

    • memoryRetrievalTool: Retrieve user memories
    • memoryStorageTool: Store information to user memories
    • craftTool: Generate code/crafts
    • webSearchTool: Search the web
    • imageGenTool: Generate images
    3. Agent Configuration System

    Location: src/server/endpoints/unified/orchestrator/configs/

    Agent configs define the behavior of different agent types. They use a hook-based pattern for maximum flexibility.

    TAgentConfig Interface
    TypeScript
    Main Thread Config

    Location: src/server/endpoints/unified/orchestrator/configs/mainThread.config.ts

    This is the default, user-facing agent that saves all data and has standard behavior.

    TypeScript
    Deep Research Config

    Location: src/server/endpoints/unified/orchestrator/configs/deepResearch.config.ts

    This agent orchestrates deep research by spawning researcher sub-agents and managing report generation.

    TypeScript

    Tool Filter for Deep Research:

    TypeScript
    Researcher Config

    Location: src/server/endpoints/unified/orchestrator/configs/researcher.config.ts

    This is a sub-agent used by the deep research supervisor for individual research tasks.

    TypeScript

    Tool Filter for Researcher:

    TypeScript
    4. Helper Functions

    Location: src/server/endpoints/unified/orchestrator/helpers/baseUtils.ts

    This file contains utility functions used throughout the orchestration system.

    Message Building
    TypeScript
    Tool Choice Configuration
    TypeScript
    Tokenized Tool Calls
    TypeScript
    Tool Result Formatting
    TypeScript
    Special Tool Result Formatting
    TypeScript
    Streaming Index Validation
    TypeScript
    Tool Context Creation
    TypeScript

    The Orchestration Flow

    Step-by-Step Execution
    1. Request Initialization

    When a user sends a request through the unified API:

    TypeScript
    2. State Initialization
    TypeScript
    3. Tool Schema Conversion
    TypeScript
    4. Streaming Index Initialization
    TypeScript
    5. Main Orchestration Loop
    TypeScript
    6. Final Cleanup
    TypeScript

    Tool System

    Tool Interface
    TypeScript
    Tool Registration

    Tools are registered with the ToolRegistry:

    TypeScript
    Tool Execution Flow
    1. Tool Selection: LLM selects tools based on user query
    2. Argument Parsing: Arguments are parsed (with agent-specific preprocessing)
    3. Enable Check: Tool's shouldUse() is called if present
    4. Execution: Tool's execute() is called
    5. Streaming: If tool returns async iterable, results are streamed
    6. Result Processing: Results are processed by agent's handleSpecialTools() hook
    7. Storage: Results are stored based on data policy
    Tool Metadata

    Tools can return metadata about their execution:

    TypeScript

    This metadata controls:

    • Whether to save the result to the database
    • Summary for reduced context
    • Sub-agent tool call counts
    • Sub-agent usage arrays

    State Management

    Orchestrator State
    TypeScript
    State Lifecycle
    1. Initialization: State is created in initializeOrchestrationState()
    2. Agent Initialization: Agent's initializeAgentState() hook is called
    3. Loop Updates: State is updated each iteration
    4. Global Context Sync: State is synced to global request context if enabled
    5. Final Return: Final state is returned with results

    Streaming Architecture

    Streaming Index System

    The orchestration system uses a global streaming index to ensure consistent ordering of streamed content across agents and sub-agents.

    TypeScript
    Stream Types
    TypeScript
    Event Manager

    The EventManager manages progress events for tool execution:

    TypeScript

    Multi-Agent System

    Agent Hierarchy
    Plain text
    Agent Communication

    Agents communicate through:

    1. Tool Results: Sub-agent results are passed back to parent
    2. Tool Metadata: Sub-agent metadata is merged with parent
    3. Usage Arrays: Sub-agent usage is aggregated
    4. Tool Call Counts: Sub-agent tool counts are added to parent iteration
    Sub-Agent Tool

    The researcher_agent_tool is used to spawn researcher sub-agents:

    TypeScript
    Global Context Update

    Main agents update global context, sub-agents do not:

    TypeScript

    This prevents sub-agents from interfering with the main thread's streaming index.

    Data Policies

    Policy Structure
    TypeScript
    Policy Examples

    Main Thread Policy (store everything):

    TypeScript

    Deep Research Policy (only research tools):

    TypeScript
    Policy Application
    TypeScript

    Usage Tracking

    Usage Configuration
    TypeScript
    Usage Collection

    Usage is collected at multiple points:

    1. LLM Response: After each LLM call
    TypeScript
    1. Tool Streaming: During tool result streaming
    TypeScript
    1. Sub-Agent Usage: Merged from sub-agents
    TypeScript
    1. Agent Modification: Agent can modify usage calculation
    TypeScript
    Usage Storage
    TypeScript

    Final usage array is returned and used for billing.

    Error Handling

    Tool Execution Errors
    TypeScript
    Tool Enablement Errors
    TypeScript
    Invalid Tool Name Errors
    TypeScript
    Error Tracking
    TypeScript

    Advanced Features

    Parallel Tool Execution

    Tools can be executed in parallel:

    TypeScript

    Parallel execution is controlled by:

    • Agent config: useParallelTools: boolean
    • User plan: shouldDoParallelToolCalls
    • Model capability: ALWAYS_ALLOW_MULTI_TOOL_CALL_MODELS
    Tool Choice Forcing

    Agents can force specific tools:

    TypeScript
    Message Modification

    Agents can modify messages based on tool results:

    TypeScript
    Context Window Optimization

    The token engine optimizes context usage:

    TypeScript

    The engine:

    • Calculates token costs for different sections
    • Chooses optimal layout (FULL, SUMMARY, TOOL_PROVIDED_SUMMARY_IF_POSSIBLE)
    • Summarizes history if needed
    • Handles tool result summarization
    Few-Shot Examples

    Some models receive few-shot examples:

    TypeScript

    These examples show the LLM how to use tools correctly.

    Tool Result Wrapping

    Some tool results are wrapped to prevent re-generation:

    TypeScript

    This prevents the LLM from re-generating content already sent to the user.

    Custom Tool Limits

    Agents can have custom tool call limits:

    TypeScript

    This is used in deep research to allow more iterations.

    Model Override

    Agents can override the model:

    TypeScript

    This allows switching to cheaper models after retries.

    Summary

    The orchestration layer is a sophisticated, custom-built system that provides:

    1. Multi-Agent Architecture: Different agents for different use cases
    2. Streaming-First Design: Real-time streaming of all content
    3. Context Optimization: Intelligent token management
    4. Configurable Behavior: Hook-based agent configuration
    5. Policy-Based Control: Fine-grained data storage control
    6. Graceful Degradation: Comprehensive error handling
    7. Sub-Agent Support: Hierarchical agent system
    8. Usage Tracking: Detailed token usage collection
    9. Tool Management: Dynamic tool registration and filtering
    10. State Management: Comprehensive state tracking

    The system is production-ready and handles complex scenarios like deep research with sub-agents, streaming tool execution, and context window optimization - all without relying on external orchestration frameworks.