Two hours of scraping, now a few minutes.

One customer's knowledge articles came in through a crawler: an Apify actor that worked through their site three times a day. Each run cost about $10 and took a couple of hours, and the content it scraped wasn't reliable.

The customer had already shared a login with us. I used it to skip the scraping: the actor now only signs in, and the articles come straight from the source's API.

  • Data ingestion
  • APIs
  • Rate limiting
  • Apify
  • Cost

Scraping was slow and costly

Cause.
The actor crawled the customer's knowledge base through its web pages: about $10 and a couple of hours a run, three times a day.
Fix.
The actor now does one thing: it signs in with the shared login and hands back the session token. Ingestion calls the source's API directly with that token.
token = actor.login(user, password)  # all Apify does now
for page in pages(size=200):
    articles += api.articles(page, token)

Scraped content wasn't accurate

Cause.
Pulling articles out of rendered pages wasn't reliable, so what we indexed didn't always match the source.
Fix.
The API returns each article as structured data, as the source stores it, so there's no HTML to pick apart.

The API has a rate limit

Cause.
Fetching everything as fast as possible would run into the API's rate limit.
Fix.
Each request fetches 200 articles, and requests are paced to stay inside the limit, so a full run finishes in a few minutes.

Results

Time per run
a couple of hoursa few minutes
3 runs a day
Crawler cost a year
~$10,950cents a run
$10 a run on Apify
Where the articles come from
scraped pagesthe API
same source

Why the actor is still there

Signing in still goes through the actor, so it stays for that one step. A run that only logs in costs cents; everything after it is plain API calls.

How the numbers were worked out

The yearly figure is the per-run price times three runs a day: $10 × 3 × 365 ≈ $10,950. Run times are as the team saw them, a couple of hours before and a few minutes after, not a formal benchmark.

More case studies

← All case studiesHomeGet in touch