Meet Crawlera, a service of the Scrapinghub Platform!

What is Crawlera?

Crawlera is a smart downloader designed specifically for web crawling and scraping. It allows you to crawl quickly and reliably, managing thousands of proxies internally, so you don’t have to.

Crawlera routes requests through a pool of IPs, throttling access by introducing delays and discarding IPs from the pool when they get banned from certain domains, or have other problems.

Accounts provide a standard HTTP proxy API, so you can configure it in your crawler of choice and start crawling.

How does it work?

Crawlera distributes requests among many internal nodes, using a proprietary algorithm to minimize the risks of getting banned, by throttling requests sent to sites from each internal node. If, for whatever reason, any node gets banned, Crawlera will blacklist it and avoid using it for future requests to that domain.

Banned requests typically return a non-200 response (like 403 or 503), or redirect to a captcha page. These responses are detected by Crawlera and the requests are automatically retried from another (clean) node.

Features

  • Instant access to thousands of IPs in our shared pool.
  • IPs from 50+ countries available on request.
  • A ban detection database with over 130 ban types, status codes or captchas.
  • Automatic retrying and throttling to crawl smoothly and prevent bans.
  • HTTP and HTTPS proxy support (with CONNECT).

Only successful downloads are charged.

Pricing

 
  • Price
  • Monthly requests
  • Concurrent requests
  • Custom User agent
  • HTTPS
  • Crawl Assistance
C10
  • Price: $25/month
  • Monthly requests : 150K
  • Concurrent requests: 10
  • Custom User agent: -
  • HTTPS:
  • Crawl Assistance : -
  • Sign Up Now
C50
  • Price: $100/month
  • Monthly requests : 1M
  • Concurrent requests: 50
  • Custom User agent:
  • HTTPS:
  • Crawl Assistance : -
  • Sign Up Now
C100
  • Price: $250/month
  • Monthly requests : 3M
  • Concurrent requests: 100
  • Custom User agent:
  • HTTPS:
  • Crawl Assistance : -
  • Sign Up Now
C200
  • Price: $500/month
  • Monthly requests : 9M
  • Concurrent requests: 200
  • Custom User agent:
  • HTTPS:
  • Crawl Assistance : -
  • Sign Up Now
Enterprise
  • Contact Sales for Custom Settings
  • Contact Sales for Custom Settings
  • Contact Sales for Custom Settings
  • Custom User agent:
  • HTTPS:
  • Crawl Assistance :
  • Sign Up Now