Shared from "Study" on Inkdown
The scraping system extracts content from web pages using multiple providers with automatic fallback. It handles regular web pages, JavaScript-rendered content, premium proxies, and even special cases like Twitter/X posts.
File: src/server/services/scrapper.ts
File: src/server/services/scrapper.ts:40
File: src/server/services/scrapper.ts:22
ScrapingBee Parameters:
api_key: Authenticationurl: Target URL to scraperender_js: Execute JavaScript (slower, more expensive)premium_proxy: Use residential proxies (better success rate, more expensive)Retry Logic (Deep Research):
File: src/server/services/firecrawl.ts
For deep research, AI-powered scraping via Firecrawl:
Usage in Deep Research:
Firecrawl Features:
File: src/server/endpoints/unified/features/deepResearch/firecrawlSerp.ts
For deep research, a sophisticated multi-tier approach:
File: src/server/services/scrapper.ts:81
Twitter/X uses a special CDN API instead of scraping:
Why CDN API:
Token Algorithm:
This mimics Twitter's frontend token generation to access their CDN.
File: src/server/endpoints/unified/features/web-access/utils.ts
When multiple search results need scraping, fetch in parallel:
Key Features:
File: src/server/endpoints/unified/features/web-access/utils.ts
Conversion Process:
File: src/server/utilities/retry.ts
The retry system used by scrapers:
Usage in Scraping:
Progressive Enhancement:
Scraping is the second step after search:
In Deep Research:
Parallel Execution
Timeout Management
Content Validation
Smart Retries
The scraping architecture:
Key Principle: Never let a scraping failure crash the research. Always have a fallback, always return gracefully.