Spider
/spider
Spider is a concurrency-first crawling engine written in Rust that streams pages the instant they’re fetched, rendering JavaScript only when a page actually needs it — so it crawls fast and scales from a single script to a distributed fleet without code changes. Its core engine is fully open source under a permissive license, and the same code powers Spider Cloud, which layers on managed proxies, headless browsers and anti-bot handling. Transparent per-page pricing and clean markdown output make it a refreshingly developer-friendly choice for high-throughput crawling and AI data collection.
Call Spider through the gateway
Same paths, parameters and responses as Spider's native API — auxiliar injects the upstream key server-side and debits the provider's metered cost from your credits.
curl https://api.auxiliar.ai/spider/<native-path> \ -H "Authorization: Bearer $AUXILIAR_API_KEY"
Exact endpoints and parameters: docs.auxiliar.ai/spider. An empty balance returns HTTP 402 — no silent degradation; see how billing behaves.
Choose if
You want fast crawling, clean markdown and transparent per-page pricing (anti-multiplier).
Avoid if
You need declarative actions (clicks no-op live) or forecastable usage-based billing.
Evals
How Spider performed on the curator-fleet web-access evals — the real measured numbers behind each composite, ranked best-first. Every provider runs the identical task corpus; each scorecard carries its run date. Latest run: 2026-06-22.