DEV Community

Devil Scrapes profile picture

Devil Scrapes

View all public scrapers @ https://apify.com/devilscrapes

Joined Joined on 
We shelved a finished scraper because the site ignored its own search box. Now we test for it first.
devil_scrapes profile

We shelved a finished scraper because the site ignored its own search box. Now we test for it first.

#api #python #webscraping #crypto
Comments
5 min read
FurnishedFinder Scraper: caps every search at 50, and the fix isn't in the docs
devil_scrapes profile

FurnishedFinder Scraper: caps every search at 50, and the fix isn't in the docs

#webscraping #python #graphql #api
Comments
5 min read
Our QA harness certified this Actor at 4096 MB. It ships on 512.
devil_scrapes profile

Our QA harness certified this Actor at 4096 MB. It ships on 512.

#testing #python #webscraping #devops
Comments
3 min read
A run delivered 0 rows and still got billed $0.30 — the bug was ours, not Bayut's
devil_scrapes profile

A run delivered 0 rows and still got billed $0.30 — the bug was ours, not Bayut's

#webscraping #python #playwright #realestate
Comments
3 min read
The TikTok 'block' that was actually Camoufox running out of memory
devil_scrapes profile

The TikTok 'block' that was actually Camoufox running out of memory

#webscraping #python #api #tiktok
Comments
3 min read
A datacenter IP gets 0 listings; a residential one gets 12. Same build.
devil_scrapes profile

A datacenter IP gets 0 listings; a residential one gets 12. Same build.

#webdev #python #scraping #api
Comments
3 min read
Our scraper went 0-for-19 against Cloudflare. One boolean took it to 9-for-10.
devil_scrapes profile

Our scraper went 0-for-19 against Cloudflare. One boolean took it to 9-for-10.

#webscraping #python #devops #api
Comments
5 min read
We cleared PeoplePerHour 4/4 locally. The first cloud run got HTTP 202 from every URL.
devil_scrapes profile

We cleared PeoplePerHour 4/4 locally. The first cloud run got HTTP 202 from every URL.

#webscraping #python #devops #api
Comments
4 min read
A Dockerfile typo turned 33 real Depop rows into a silent, SUCCEEDED zero
devil_scrapes profile

A Dockerfile typo turned 33 real Depop rows into a silent, SUCCEEDED zero

#webscraping #python #docker #ecommerce
Comments
4 min read
An isinstance(str) check that was always False capped every Facebook Page at exactly 3 posts
devil_scrapes profile

An isinstance(str) check that was always False capped every Facebook Page at exactly 3 posts

#webscraping #python #api #debugging
Comments
4 min read
Alibaba renamed one JavaScript variable and our price field silently dropped to 0%
devil_scrapes profile

Alibaba renamed one JavaScript variable and our price field silently dropped to 0%

#webscraping #python #javascript #ecommerce
Comments
5 min read
A 586 KB HTTP 200 from YouTube that contains zero videos
devil_scrapes profile

A 586 KB HTTP 200 from YouTube that contains zero videos

#webscraping #python #api #youtube
Comments
4 min read
Our test asked for 4 branches, ran 1, and reported PASS
devil_scrapes profile

Our test asked for 4 branches, ran 1, and reported PASS

#testing #python #webscraping #devops
Comments
4 min read
LinkedIn has an /api/ endpoint that needs no login and returns HTML
devil_scrapes profile

LinkedIn has an /api/ endpoint that needs no login and returns HTML

#webscraping #python #api #linkedin
Comments
3 min read
Our release gate read a docstring and blocked the deploy over a word
devil_scrapes profile

Our release gate read a docstring and blocked the deploy over a word

#devops #python #testing #webscraping
Comments
4 min read
No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST
devil_scrapes profile

No introspection, no allowlist: reconstructing Whatnot's GraphQL queries from a compiled AST

#graphql #python #webscraping #api
Comments
4 min read
Zero results, five rows: the Flippa search response that lies about matching nothing
devil_scrapes profile

Zero results, five rows: the Flippa search response that lies about matching nothing

#python #webscraping #api #webdev
Comments
4 min read
A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices
devil_scrapes profile

A 200 that isn't reachability: SpareRoom's soft-redirect and the empty run nobody notices

#python #webscraping #api #webdev
Comments
5 min read
A 404 is not a failure: what Discord's invite endpoint taught us about billing
devil_scrapes profile

A 404 is not a failure: what Discord's invite endpoint taught us about billing

#python #api #webscraping #discord
5
Comments
5 min read
A 25%-failing test is not flaky. It's a bug reporting itself honestly.
devil_scrapes profile

A 25%-failing test is not flaky. It's a bug reporting itself honestly.

#python #testing #webscraping #debugging
5
Comments
4 min read
A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.
devil_scrapes profile

A bare useApifyProxy: true resolves to datacenter. Patreon 403s every one of those.

#webscraping #python #api #cloudflare
Comments
4 min read
grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.
devil_scrapes profile

grep '"aggregate_rating":' finds zero matches on Zomato. The field is there, just double-escaped.

#webscraping #python #javascript #api
Comments
4 min read
?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.
devil_scrapes profile

?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.

#webscraping #python #api #ecommerce
Comments
5 min read
?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.
devil_scrapes profile

?q=chairs on Chairish isn't a search - it's a redirect to a category page wearing a search URL.

#webscraping #python #api #ecommerce
Comments
5 min read
?page=2 on Immowelt returns page 1, byte for byte. It's a button, not a route.
devil_scrapes profile

?page=2 on Immowelt returns page 1, byte for byte. It's a button, not a route.

#webscraping #python #api #realestate
Comments
5 min read
window.__NUXT__ is not JSON. It's a function you have to run first.
devil_scrapes profile

window.__NUXT__ is not JSON. It's a function you have to run first.

#webscraping #python #javascript #api
Comments
4 min read
Our build wave shipped four scrapers that scraped nothing
devil_scrapes profile

Our build wave shipped four scrapers that scraped nothing

#testing #python #webscraping #devops
Comments
4 min read
A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.
devil_scrapes profile

A one-step probe told us YouTube was blocked. It was asking the wrong endpoint.

#webscraping #python #api #youtube
Comments
4 min read
The top-earning product on Indie Hackers reports $10,000,000,000,000,000 a month
devil_scrapes profile

The top-earning product on Indie Hackers reports $10,000,000,000,000,000 a month

#webscraping #python #saas #data
Comments
3 min read
Grailed's __NEXT_DATA__ is empty. The listings come from Algolia, and the key is on the page.
devil_scrapes profile

Grailed's __NEXT_DATA__ is empty. The listings come from Algolia, and the key is on the page.

#webscraping #python #algolia #nextjs
Comments
3 min read
Coursera ships two big JSON blobs on every search page. Neither has the results.
devil_scrapes profile

Coursera ships two big JSON blobs on every search page. Neither has the results.

#webscraping #python #html #api
Comments
4 min read
dict.get(key, [None])[0] is not safe, and our green test suite agreed with us
devil_scrapes profile

dict.get(key, [None])[0] is not safe, and our green test suite agreed with us

#python #testing #webscraping #debugging
Comments
4 min read
Pinterest's ad library endpoint was hiding behind one missing curl flag
devil_scrapes profile

Pinterest's ad library endpoint was hiding behind one missing curl flag

#webscraping #python #api #marketing
Comments
4 min read
Google News RSS has no cursor, so the query IS the cursor
devil_scrapes profile

Google News RSS has no cursor, so the query IS the cursor

#news #webscraping #python #api
Comments
4 min read
Trustpilot needs a browser exactly once — the WAF token replays onto curl
devil_scrapes profile

Trustpilot needs a browser exactly once — the WAF token replays onto curl

#webscraping #python #aws #debugging
Comments
4 min read
Meta's Ad Library 403 is 481 bytes of instructions, not a block
devil_scrapes profile

Meta's Ad Library 403 is 481 bytes of instructions, not a block

#webscraping #python #api #debugging
Comments
4 min read
Indeed 403s Chrome on every proxy tier — and our laptop said it was fine
devil_scrapes profile

Indeed 403s Chrome on every proxy tier — and our laptop said it was fine

#webscraping #python #debugging #devops
Comments
4 min read
LinkedIn Ad Library Caps at 24 Ads — Here's the Real Pagination Endpoint
devil_scrapes profile

LinkedIn Ad Library Caps at 24 Ads — Here's the Real Pagination Endpoint

#webscraping #python #api #apify
Comments
7 min read
Fiverr Gig Scraper: One Listing, Two Rows, Only One Says Sponsored
devil_scrapes profile

Fiverr Gig Scraper: One Listing, Two Rows, Only One Says Sponsored

#webscraping #python #api #apify
Comments
5 min read
Why 43% of Yahoo Finance's 1-Minute Candles Come Back Null
devil_scrapes profile

Why 43% of Yahoo Finance's 1-Minute Candles Come Back Null

#webscraping #python #api #apify
Comments
5 min read
CoinMarketCap's lastUpdated Lies About Your EUR Price
devil_scrapes profile

CoinMarketCap's lastUpdated Lies About Your EUR Price

#webscraping #python #api #apify
Comments
5 min read
Binance's api.binance.com Blocks Every US IP — Its Own Data Mirror Doesn't
devil_scrapes profile

Binance's api.binance.com Blocks Every US IP — Its Own Data Mirror Doesn't

#webscraping #python #api #apify
Comments
5 min read
The Invisible Character Hiding In We Work Remotely's Job Descriptions
devil_scrapes profile

The Invisible Character Hiding In We Work Remotely's Job Descriptions

#webscraping #python #api #apify
Comments
3 min read
Two Thirds Of Stocktwits Messages Have No Sentiment — And The API Says So Explicitly
devil_scrapes profile

Two Thirds Of Stocktwits Messages Have No Sentiment — And The API Says So Explicitly

#webscraping #python #api #apify
Comments
3 min read
Remotive's Salary Field Mixes Two Different Comma Conventions
devil_scrapes profile

Remotive's Salary Field Mixes Two Different Comma Conventions

#webscraping #python #api #apify
Comments
3 min read
Mastodon's acct Field Hides the Domain You Already Know
devil_scrapes profile

Mastodon's acct Field Hides the Domain You Already Know

#webscraping #python #api #apify
Comments
3 min read
Arbeitnow's Pagination Link Lies About Its Own Filter
devil_scrapes profile

Arbeitnow's Pagination Link Lies About Its Own Filter

#webscraping #python #api #apify
Comments
4 min read
UTC Offsets Are Not Integers — A Remote Job Feed With 5.75 In It
devil_scrapes profile

UTC Offsets Are Not Integers — A Remote Job Feed With 5.75 In It

#webscraping #python #api #apify
Comments
4 min read
Airbnb's search payload doesn't carry amenities. Here's what we do instead of guessing.
devil_scrapes profile

Airbnb's search payload doesn't carry amenities. Here's what we do instead of guessing.

#showdev #python #webscraping #api
Comments
4 min read
How we keep 200 web scrapers alive
devil_scrapes profile

How we keep 200 web scrapers alive

#webscraping #apify #python #data
Comments
6 min read
Why your scraper gets blocked at 3 a.m.
devil_scrapes profile

Why your scraper gets blocked at 3 a.m.

#webscraping #apify #python #data
Comments
5 min read
Export a competitor's entire Google Ads library
devil_scrapes profile

Export a competitor's entire Google Ads library

#webscraping #apify #python #data
Comments
5 min read
Pay-per-result scraping: what you actually pay
devil_scrapes profile

Pay-per-result scraping: what you actually pay

#webscraping #apify #python #data
Comments
5 min read
Run an Apify Actor From n8n, Make or Zapier
devil_scrapes profile

Run an Apify Actor From n8n, Make or Zapier

#webscraping #apify #automation #python
Comments
6 min read
Twitch Chat Scraper and 2 More Live-Stream Chat Archives
devil_scrapes profile

Twitch Chat Scraper and 2 More Live-Stream Chat Archives

#webscraping #apify #socialmedia #python
Comments
6 min read
Scrape Car Listings Europe: 22 Marketplaces, One Schema
devil_scrapes profile

Scrape Car Listings Europe: 22 Marketplaces, One Schema

#webscraping #apify #automotive #python
Comments
6 min read
OpenCorporates Alternative: US Business Registry Data
devil_scrapes profile

OpenCorporates Alternative: US Business Registry Data

#webscraping #apify #kyb #leadgeneration
Comments
6 min read
Custom Web Scraper Development vs Building In-House
devil_scrapes profile

Custom Web Scraper Development vs Building In-House

#webscraping #apify #engineering #python
Comments
6 min read
The US sanctions list still ships with a 1980s DOS end-of-file byte
devil_scrapes profile

The US sanctions list still ships with a 1980s DOS end-of-file byte

#python #api #opendata #webscraping
Comments
4 min read
This API returns bank details for IBANs that are invalid
devil_scrapes profile

This API returns bank details for IBANs that are invalid

#api #fintech #python #webdev
Comments
4 min read
loading...