Inkdown
Start writing

Edward

3 files·0 subfolders

Shared Workspace

Edward
Orchestration Layer

Orchestration Layer

Shared from "Edward" on Inkdown

Edward Orchestration Workflow - End to End

Simple Technical KT for Engineers

Table of Contents

  1. What is Orchestration in Edward?
  2. The 5-Minute Overview
  3. Step-by-Step Flow
  4. Deep Dive: Each Layer
  5. Key Files to Read
  6. Common Questions

1. What is Orchestration in Edward?

Orchestration = How Edward coordinates all the pieces to turn a user's chat message into working code.

Overview
Stream Continuation

Think of it like a conductor leading an orchestra:

  • Conductor = Orchestration layer
  • Musicians = LLM, Docker sandbox, file system, package manager, build tools
  • Music = The generated code

The orchestration layer makes sure everyone plays at the right time, in the right order.

2. The 5-Minute Overview

The Big Picture
Plain text
The 3 Main Phases
Plain text

3. Step-by-Step Flow

Step 1: User Sends Message

File: apps/web/stores/chatStream/useStartStream.ts

Plain text
Step 2: API Admission

File: apps/api/services/runs/messageOrchestrator.service.ts

JavaScript

Key Points:

  • Admission control prevents overload (global + per-user + per-chat limits)
  • Run is persisted BEFORE execution (durable)
  • Browser gets runId immediately for tracking
Step 3: Worker Picks Up Job

File: apps/api/services/runs/agent-run-worker/processor.ts

JavaScript

Key Points:

  • Worker is independent (can restart without losing progress)
  • Cancel signal via Redis pub/sub (fast)
  • Events persisted to DB as they happen (resumable)
Step 4: Stream Session Setup

File: apps/api/services/chat/session/orchestrator/runStreamSession.orchestrator.ts

JavaScript

Key Points:

  • Framework resolved before LLM call (better prompts)
  • Token budget checked BEFORE calling LLM (fail fast)
  • Post-generation validation + autofix (quality control)
Step 5: Agent Loop (Multi-Turn)

File: apps/api/services/chat/session/loop/agentLoop.runner.ts

JavaScript

Why Multiple Turns?

Plain text

Loop Continues When:

  • Tools were called but no file output yet
  • No <done> tag received
  • Under turn budget

Loop Stops When:

  • Code/file output detected
  • <done> tag received
  • Tool budget exceeded
  • Max turns reached
  • Client aborted
Step 6: Turn Execution (Stream + Parse)

File: apps/api/services/chat/session/loop/agentLoop.stream.ts

JavaScript

Key Points:

  • Chunks processed as they arrive (not waiting for full response)
  • Parser converts raw text → structured events
  • Events trigger immediate side effects
Step 7: Parser State Machine

File: apps/api/lib/llm/parser.ts

JavaScript

Parser States:

StateTriggerExit
TEXTDefault<thinking>, <edward_sandbox>, <file>
THINKING<thinking></thinking>
SANDBOX<edward_sandbox></edward_sandbox>
FILE<file path="..."></file>
INSTALL<install></install>

Why State Machine?

  • Chunks can split tags across boundaries
  • Need to handle incomplete output safely
  • Can't just regex over full string
Step 8: Event Handler (Side Effects)

File: apps/api/services/chat/session/events/handler.ts

JavaScript

Event Types:

EventAction
SANDBOX_STARTProvision Docker container
FILE_STARTPrepare file path
FILE_CONTENTBuffer to Redis
FILE_ENDSanitize file
SANDBOX_ENDFlush buffers to disk
INSTALL_CONTENTQueue npm install
COMMANDRun shell command
WEB_SEARCHSearch web
Step 9: Sandbox Write Flow (Buffered)

File: apps/api/services/sandbox/write/buffer.ts + flush.ts

Write (Buffered to Redis)
JavaScript
Flush (Redis → Container)
JavaScript

Why Buffer?

Plain text

Benefits:

  • Resilient to partial failures
  • Can batch multiple writes
  • Can replay/repair on failure
Step 10: Install Task Queue

File: apps/api/services/chat/session/loop/agentLoop.runner.ts

JavaScript

Why Serialize Installs?

Plain text
Step 11: Turn Outcome Decision

File: apps/api/services/chat/session/loop/agentLoop.turnOutcome.ts

JavaScript

Decision Tree:

Plain text
Step 12: Finalize

File: apps/api/services/chat/session/orchestrator/runStreamSession.finalize.ts

JavaScript

4. Deep Dive: Each Layer

Layer 1: Message Orchestrator

Purpose: Admission control + queue + stream handoff

Key Functions:

  • unifiedSendMessage() - Entry point
  • createAdmittedRun() - Create run with limits
  • enqueueAdmittedRun() - Queue to worker
  • streamRunEventsFromPersistence() - SSE to browser

What Could Go Wrong:

  • API key decryption fails
  • Model/provider mismatch
  • Run admission rejected (limits)
  • Queue enqueue fails
Layer 2: Stream Session

Purpose: Framework resolve + message prep + token budget + finalize

Key Functions:

  • resolveFramework() - Detect/prefer framework
  • prepareBaseMessages() - Build LLM context
  • composePrompt() - System prompt
  • computeTokenUsage() - Budget check
  • finalizeStreamSession() - Persist results

What Could Go Wrong:

  • Context limit exceeded
  • Framework detection fails
  • Finalize persistence fails
Layer 3: Agent Loop

Purpose: Multi-turn execution + outcome decisions

Key Functions:

  • runAgentLoop() - Main loop
  • executeAgentTurnStream() - Single turn
  • resolveTurnOutcome() - Continue/stop decision

What Could Go Wrong:

  • Turn budget exceeded
  • Max turns reached
  • Abort signal received
  • Continuation prompt fails
Layer 4: Parser

Purpose: Chunk → event conversion

Key Functions:

  • createStreamParser() - State machine
  • process() - Parse chunk
  • flush() - Handle incomplete output

What Could Go Wrong:

  • Tag split across chunks
  • Incomplete output
  • State machine stuck
Layer 5: Event Handler

Purpose: Side effect execution

Key Functions:

  • handleParserEvent() - Dispatch by type
  • handleFileContent() - Buffer writes
  • handleInstallContent() - Queue installs
  • handleCommandEvent() - Run commands

What Could Go Wrong:

  • Sandbox not provisioned
  • File write fails
  • Install conflicts
  • Command timeout
Layer 6: Sandbox Write

Purpose: Buffered writes to container

Key Functions:

  • writeSandboxFile() - Buffer to Redis
  • flushSandbox() - Redis → container
  • scheduleSandboxFlush() - Debounced flush

What Could Go Wrong:

  • Redis unavailable
  • Docker exec fails
  • Lock acquisition fails
  • Container stopped

5. Key Files to Read

Core Orchestration
FilePurpose
apps/api/services/runs/messageOrchestrator.service.tsEntry point
apps/api/services/runs/agent-run-worker/processor.tsWorker execution
apps/api/services/chat/session/orchestrator/runStreamSession.orchestrator.tsStream session
apps/api/services/chat/session/loop/agentLoop.runner.tsAgent loop
apps/api/services/chat/session/loop/agentLoop.stream.tsTurn execution
Parser + Events
FilePurpose
apps/api/lib/llm/parser.tsState machine parser
apps/api/services/chat/session/events/handler.tsEvent side effects
apps/api/services/chat/session/loop/events.tsEvent processing
apps/api/services/chat/session/loop/agentLoop.turnOutcome.tsContinue/stop logic
Sandbox
FilePurpose
apps/api/services/sandbox/write/buffer.tsRedis buffering
apps/api/services/sandbox/write/flush.tsFlush to container
apps/api/services/sandbox/write/flush.scheduler.tsDebounced flush
apps/api/services/chat/file.handlers.tsFile content handling

6. Common Questions

Q: Why multi-turn loop instead of one LLM call?

A: Complex tasks need multiple steps:

Plain text
Q: Why buffer writes to Redis instead of writing directly?

A: Three reasons:

  1. Resilience: If Docker fails, buffer survives in Redis
  2. Performance: One flush vs many small writes
  3. Batching: Multiple chunks → one file write
Q: Why serialize installs?

A: Prevent race conditions:

Plain text
Q: How does cancellation work?

A: Two mechanisms:

  1. Redis pub/sub (fast):

    Plain text

  2. DB polling (backup):

    Plain text
Q: What happens if worker crashes mid-turn?

A: Checkpoint system allows resume:

Plain text
Q: How are tokens budgeted?

A: Multiple levels:

Plain text
Q: How does framework detection work?

A: Three sources:

Plain text
Q: What's the difference between run_event and message?

A: Different purposes:

Plain text

7. Debugging Guide

Trace a Turn
Plain text
Common Failures
SymptomLikely CauseFix
Context limit exceededToo much historyTruncate context
Tool budget exceededToo many tool callsReduce per-turn limit
Turn stuck in loopNo code output detectedCheck parser, tags
Files not writtenFlush failedCheck Redis, Docker
Install conflictsConcurrent installsCheck queue serialization

8. Summary

The Orchestration Flow in One Diagram
Plain text
Key Takeaways
  1. Orchestration is layered - Each layer has a clear responsibility
  2. Multi-turn is essential - Complex tasks need iteration
  3. Buffering matters - Redis buffers make writes resilient
  4. Budgets prevent runaway - Token, tool, turn limits
  5. Events are durable - Persisted for replay/resume
  6. Cancellation is dual - Pub/sub + DB polling

End of Document