Inkdown
Start writing

Merlin Backend

12 files·0 subfolders

Shared Workspace

Merlin Backend
01-Orchestration.md

04-Scraping

Shared from "Merlin Backend" on Inkdown

Scraping Architecture

Overview

The scraping system extracts content from web pages using multiple providers with automatic fallback. It handles regular web pages, JavaScript-rendered content, premium proxies, and even special cases like Twitter/X posts.

Scraping Providers

Plain text
02-DeepResearch.md
03-Search.md
04-Scraping.md
05-Streaming.md
06-MultiProviderLLM.md
07-MemoryAndContext.md
08-ErrorHandling.md
09-RateLimiting.md
10-TaskQueue.md
11-SecurityAndAuth.md
Orchestration-2nd-draft

1. ScrapingBee Integration

File: src/server/services/scrapper.ts

Basic Configuration
TypeScript
Core Scraping Function

File: src/server/services/scrapper.ts:40

TypeScript
API Call Structure

File: src/server/services/scrapper.ts:22

TypeScript

ScrapingBee Parameters:

  • api_key: Authentication
  • url: Target URL to scrape
  • render_js: Execute JavaScript (slower, more expensive)
  • premium_proxy: Use residential proxies (better success rate, more expensive)

Retry Logic (Deep Research):

  • Attempt 1: Standard proxy, no JS
  • Attempt 2: Standard proxy, JS enabled
  • Attempt 3+: Premium proxy, JS enabled

2. Firecrawl Integration

File: src/server/services/firecrawl.ts

For deep research, AI-powered scraping via Firecrawl:

TypeScript

Usage in Deep Research:

TypeScript

Firecrawl Features:

  • Automatic content extraction
  • Markdown conversion
  • PDF text extraction
  • Rate limiting handled by library

3. Deep Research 4-Tier Scraping

File: src/server/endpoints/unified/features/deepResearch/firecrawlSerp.ts

For deep research, a sophisticated multi-tier approach:

Plain text

4. Twitter/X Special Handling

File: src/server/services/scrapper.ts:81

Twitter/X uses a special CDN API instead of scraping:

TypeScript

Why CDN API:

  • Twitter blocks most scrapers
  • CDN endpoint is public but requires token generation
  • Returns clean text (no HTML parsing needed)
  • 4s timeout (fast fail)

Token Algorithm:

JavaScript

This mimics Twitter's frontend token generation to access their CDN.

5. Parallel URL Scraping

File: src/server/endpoints/unified/features/web-access/utils.ts

When multiple search results need scraping, fetch in parallel:

TypeScript

Key Features:

  • Parallel fetching (all at once)
  • 8-second timeout per URL
  • Returns fastest results first
  • Graceful degradation (undefined for failures)

6. HTML to Markdown Conversion

File: src/server/endpoints/unified/features/web-access/utils.ts

TypeScript

Conversion Process:

  1. Scrape HTML (via ScrapingBee)
  2. Parse with JSDOM or similar
  3. Extract main content (remove nav, ads, etc.)
  4. Convert to Markdown (Turndown or similar)
  5. Validate minimum length (100 chars)

7. Retry Architecture

File: src/server/utilities/retry.ts

The retry system used by scrapers:

TypeScript

Usage in Scraping:

TypeScript

Progressive Enhancement:

  • Retry 0: Standard proxy, no JS
  • Retry 1: Standard proxy, JS enabled
  • Retry 2+: Premium proxy, JS enabled

8. Error Handling Strategy

Scraping Error Hierarchy
Plain text
Error Handling in Code
TypeScript

9. Integration with Search

Scraping is the second step after search:

Plain text

In Deep Research:

TypeScript

10. Performance Optimization

Scraping Best Practices

  1. Parallel Execution

    TypeScript

  2. Timeout Management

    TypeScript

  3. Content Validation

    TypeScript

  4. Smart Retries

    TypeScript

Summary

The scraping architecture:

  1. Multi-Tier Fallback: ScrapingBee → Firecrawl → Direct
  2. Progressive Enhancement: Standard → JS → Premium proxy
  3. Parallel Execution: All URLs scraped simultaneously
  4. Smart Retries: Context-aware with exponential backoff
  5. Special Cases: Twitter CDN API for X.com
  6. Content Validation: Minimum length checks
  7. Error Resilience: Graceful degradation to undefined

Key Principle: Never let a scraping failure crash the research. Always have a fallback, always return gracefully.