Best Web Scraping APIs for Etsy: July 2026 Benchmark
Benchmarks run by the Scrapeway team ยท
Last updated: August 14, 2026 ยท
How we benchmark
Scrapfly is the best web scraping API for Etsy, with a
99% success rate across 8 web scraping APIs
benchmarked against live Etsy pages in July 2026.
3 of the 8 cleared Etsy reliably enough to recommend.
Etsy is protected by DataDome, so most scraping APIs
either fail it or pay for it in speed and cost. The benchmark is refreshed twice a month, with no affiliate
links and no sponsors.
Ranking history: web scraping APIs for Etsy over time
Etsy target ranking history
The 7 web scraping APIs for Etsy, reviewed
1. Scrapfly: 99% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
99%
5.1s
$3.89
#1 of 7
$30/mo
Etsy puts DataDome in front of its listing, shop, and search pages, and most listing data sits in the server
rendered HTML and embedded JSON. Scrapfly cleared 99% here by
generating a genuine device fingerprint and passing DataDome's checks, so it reaches the listing page instead
of the CAPTCHA interstitial. Because it only bills for successful requests, the block and challenge pages Etsy
returns as a 200 don't quietly run up the cost.
At $3.89 per 1,000 successful requests and
5.1s average response time, the figures are strong for an API capable of
clearing DataDome. The 99% success rate is the headline, and the ranking history above
shows how that rate has held across previous runs.
Pros:
Highest success rate in the test, clearing DataDome's device fingerprinting
Only charges for successful scrapes, so CAPTCHA pages returned as 200s cost nothing
One asp flag plus residential proxies handles Etsy with little tuning
First class SDKs for Python, TypeScript, Go, and Rust, plus a Scrapy extension
Cons:
Credit cost per request rises once ASP, JavaScript rendering, or residential proxies are enabled
The entry (Discovery) plan caps concurrency at 5, so high volume listing crawls need a higher tier
The free tier is a single batch of 1,000 credits, enough to prototype but not to benchmark at volume
2. Firecrawl: 93% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
93%
5.2s
$6.33
#2 of 7
$16/mo
Firecrawl's draw on Etsy is its output format. It returns structured markdown rather than raw HTML, so a
pipeline feeding an LLM or RAG system skips the parsing step on Etsy's listing markup. Rendering a full
browser on every page and generating that markdown carries a cost and latency premium over lighter, HTTP based
options, so that saving has to be worth the premium for your use case. On Etsy it cleared 93%
this run.
Pros:
Returns markdown built for LLM pipelines, saving a parsing step for AI and RAG pipelines
Runs a real browser, which covers pages that need JavaScript execution
Cons:
Cost is the top user complaint in reviews
Renders a full browser on every page, which adds latency
3. WebScrapingAPI: 91% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
91%
34.8s
$2.71
#3 of 7
$19/mo
WebScrapingAPI covers a broad range of language SDKs behind a simple REST interface, so it drops into most
stacks without a client library, and it supports async and batch submission for queued jobs. The scorecard
carries this run's speed and cost. On Etsy it cleared 91% this run.
Pros:
Broad language SDK range behind a simple REST interface
Async and batch submission for queued jobs
Cons:
DataDome's block pages return as 200s, so it's easy to keep paying unless you check content
Support tickets often go unanswered for days, per user reviews
4. Scraperapi: 90% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
90%
11.3s
$4.90
#4 of 7
$49/mo
Scraperapi is built around a fast request path. When it clears a request it tends to return quickly, which
suits latency sensitive lookups that can absorb retries. Etsy's DataDome layer has to be cleared before the
listing JSON is reachable, and the scorecard carries how it did on that this run. On Etsy it cleared
90% this run.
Pros:
Fast when it clears, with a simple integration
Broad language SDK support
Cons:
Login required flows and form filling are off limits
Geotargeting is gated by plan (US and EU only until the Business tier)
5. Zenrows: 88% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
88%
16.0s
$6.90
#5 of 7
$69/mo
Zenrows is a general purpose scraping API with JavaScript rendering and session support, positioned as a
generalist rather than a specialist on any single axis. It renders a real browser and supports sessions, and
on Etsy it cleared 88% this run. It fits when you want one general purpose tool for Etsy plus
other targets rather than optimizing hard for cost, speed, or maximum reliability.
Pros:
Real browser rendering and session support for targets that need JavaScript execution
One general purpose tool spans Etsy and other targets
Cons:
Cost climbs on heavy or large scale usage, the recurring user complaint
Premium proxy geographic coverage is unclear
6. Scrapingant: 37% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
37%
52.2s
$1.90
#6 of 7
$19/mo
Scrapingant bundles JavaScript rendering and session support at a low entry price, with a smaller feature
surface than the larger providers. On Etsy it cleared 37% this run.
Pros:
Low sticker price
JavaScript rendering included
Cons:
Smaller feature surface and fewer integration options than the larger providers
Billing covers requests that return block or challenge pages, so failures still cost credits
Small provider with a thin public track record
7. Scrapingbee: 24% success on Etsy
On Etsy
Speed
Cost/1k
Overall
From
24%
4.2s
$3.31
#7 of 7
$49/mo
Scrapingbee is fast and cheap per request, with JavaScript rendering for lighter targets. Etsy is a hard case
for it, because DataDome scores the device fingerprint and behavior on every request, and every request that
gets challenged partway through still costs credits. On Etsy it cleared 24% this run.
Pros:
Fast response times
Low sticker cost per request
Cons:
DataDome's per request scoring penalizes the partial signals an HTTP first approach tends to produce
Credits burn quickly once JavaScript rendering or premium proxies are enabled
No plan tier between the small and large options
COMPARISON
About scraping Etsy
Etsy is a marketplace for handmade, vintage, and craft goods, and the data people scrape from it is mostly
commercial. That means listing titles, prices, shipping and availability, shop and seller details, ratings and
review counts, favorites, and search result and category rankings. Most of this lives on listing pages (the
/listing/{id} URLs), shop pages, and search results.
Etsy listing pages are largely server rendered, so a plain HTTP request often returns the core listing fields,
with additional structured data available in embedded JSON and JSON-LD blocks in the page. A headless browser
matters more for getting past the antibot layer than for rendering the content itself, and search and shop pages
can paginate through secondary requests.
HTTP ANALYSIS
Etsy is protected by DataDome. See the
DataDome benchmark page for how DataDome detects bots.
The practical detail for scraping Etsy is that its blocks and CAPTCHA interstitials often come back as a 200
rather than a hard error, so success has to be measured on response content, not status codes.
etsy_scraper.py
importjsonfromparselimportSelector# install using `pip install scrapfly-sdk`fromscrapflyimportScrapflyClient,ScrapeConfig,ScrapeApiResponse# create an API client instanceclient=ScrapflyClient(key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.scrape(ScrapeConfig(url=url,headers=headers,asp=True,render_js=render_jsorFalse,cache=False,cache_ttl=900,country=countryor'US',method='GET',))returnapi_result.selectorurl="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# install using `pip install scrapfly-sdk`
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
# create an API client instance
client = ScrapflyClient(key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.scrape(ScrapeConfig(
url=url,
headers=headers,
asp=True,
render_js=render_js or False,
cache=False,
cache_ttl=900,
country=country or 'US',
method='GET',
))
return api_result.selector
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# install using `pip install firecrawl-py`fromfirecrawlimportFirecrawl# create an API client instanceclient=Firecrawl(api_key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.scrape(url=url,headers=headersor{},formats=['rawHtml'],only_main_content=False,skip_tls_verification=False,timeout=150000,store_in_cache=False,max_age=900,)assertapi_result.raw_html,"firecrawl returned no html for this page"returnSelector(api_result.raw_html)url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# install using `pip install firecrawl-py`
from firecrawl import Firecrawl
# create an API client instance
client = Firecrawl(api_key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.scrape(
url=url,
headers=headers or {},
formats=['rawHtml'],
only_main_content=False,
skip_tls_verification=False,
timeout=150000,
store_in_cache=False,
max_age=900,
)
assert api_result.raw_html, "firecrawl returned no html for this page"
return Selector(api_result.raw_html)
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# webscrapingapi has a Python SDK but it's not great, use httpx instead:# `pip install httpx`importhttpx# create an API client instanceclient=httpx.Client(timeout=180)# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.get("https://api.webscrapingapi.com/v2",# the target url is passed as a parameter belowheaders=headers,params={"url":url,"api_key":"YOUR API KEY",# NOTE: add your API KEY here!"timeout":60_000,"render_js":render_jsorFalse,"method":'GET',},)assertapi_result.status_code==200,api_result.reason_phrasereturnSelector(api_result.text)url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# webscrapingapi has a Python SDK but it's not great, use httpx instead:
# `pip install httpx`
import httpx
# create an API client instance
client = httpx.Client(timeout=180)
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.get(
"https://api.webscrapingapi.com/v2", # the target url is passed as a parameter below
headers=headers,
params={
"url": url,
"api_key": "YOUR API KEY", # NOTE: add your API KEY here!
"timeout": 60_000,
"render_js": render_js or False,
"method": 'GET',
},
)
assert api_result.status_code == 200, api_result.reason_phrase
return Selector(api_result.text)
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# install using `pip install scraperapi-sdk`fromscraperapi_sdkimportScraperAPIClient# create an API client instanceclient=ScraperAPIClient(api_key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.make_request(url=url,headers=headers,params={"render":False,},method='GET',)assertapi_result.ok,api_result.textreturnSelector(api_result.text)url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# install using `pip install scraperapi-sdk`
from scraperapi_sdk import ScraperAPIClient
# create an API client instance
client = ScraperAPIClient(api_key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.make_request(
url=url,
headers=headers,
params={
"render": False,
},
method='GET',
)
assert api_result.ok, api_result.text
return Selector(api_result.text)
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# install using `pip install zenrows`fromzenrowsimportZenRowsClient# create an API client instanceclient=ZenRowsClient(apikey="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.get(url,headers=headers,params={})# zenrows answers with the page HTML; "json_response" is only accepted with js_render onassertapi_result.ok,api_result.textreturnSelector(api_result.text)url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# install using `pip install zenrows`
from zenrows import ZenRowsClient
# create an API client instance
client = ZenRowsClient(apikey="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.get(
url,
headers=headers,
params={
}
)
# zenrows answers with the page HTML; "json_response" is only accepted with js_render on
assert api_result.ok, api_result.text
return Selector(api_result.text)
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# install using `pip install scrapingant-client`fromscrapingant_clientimportScrapingAntClient# create an API client instanceclient=ScrapingAntClient(token="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.general_request(url,json=True,proxy_type='residential',method='GET',)# the scrapingant client returns its own Response object: content holds the pageassertapi_result.status_code==200,api_result.textreturnSelector(api_result.content)url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# install using `pip install scrapingant-client`
from scrapingant_client import ScrapingAntClient
# create an API client instance
client = ScrapingAntClient(token="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.general_request(
url,
json=True,
proxy_type='residential',
method='GET',
)
# the scrapingant client returns its own Response object: content holds the page
assert api_result.status_code == 200, api_result.text
return Selector(api_result.content)
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# install using `pip install scrapingbee`fromscrapingbeeimportScrapingBeeClient# create an API client instanceclient=ScrapingBeeClient(api_key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.get(url,headers=headers,params={"json_response":True,"transparent_status_code":True,})assertapi_result.ok,api_result.textdata=api_result.json()returnSelector(data['body'])url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# install using `pip install scrapingbee`
from scrapingbee import ScrapingBeeClient
# create an API client instance
client = ScrapingBeeClient(api_key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.get(
url,
headers=headers,
params={
"json_response": True,
"transparent_status_code": True,
}
)
assert api_result.ok, api_result.text
data = api_result.json()
return Selector(data['body'])
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
importjsonfromparselimportSelector# scrapingdog has no integration but we can use httpx# install using `pip install httpx`importhttpx# create an API client instanceclient=httpx.Client(timeout=180)# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:params={"api_key":"YOUR API KEY","url":url,}api_result=client.get('https://api.scrapingdog.com/scrape',params=params,)assertapi_result.status_code==200,api_result.textreturnSelector(api_result.text,type="html")url="https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"# scrape through US geo-location as etsy is blocked in many countriesselector=scrape(url,country="US")# etsy has product data as JSON in a script elementscript=selector.xpath("//script[contains(text(),'offers')]/text()").get()data=json.loads(script)frompprintimportpprintpprint(pprint(data))
Output$ python etsy_scraper.py
{'@context': 'https://schema.org',
'@type': 'Product',
'aggregateRating': {'@type': 'AggregateRating',
'ratingValue': '5.0',
'reviewCount': 615},
'brand': {'@context': 'https://schema.org',
'@type': 'Brand',
'name': 'WestArtFactory'},
'category': 'Electronics & Accessories < Gadgets',
'description': 'The Python Cheat Sheet is an absolute must for every software '
'engineer, hacker or programmer.\n'
'The product consists of a high-quality printed circuit board '
'(PCB).\n'
'The special thing about the board is the wafer-thin lettering '
'and conductor tracks coated with real gold.\n'
'The gold is applied very thinly using a highly complex '
'process and forms a homogeneous connection with the circuit '
'board.\n'
'This not only creates a unique look, but also a very special '
'feel.\n'
'\n'
'In addition to the excellent quality of the coaster, the '
'carefully selected software examples are also extremely '
'helpful in Python software development.\n'
'The cheat sheet offers helpful support, especially for '
'beginners. Professionals should see the cheat sheet as a sign '
'of belonging to the Python community :)\n'
'\n'
'Dimensions: 10 cm x 10 cm [3.94" x 3.94"]\n'
'Thickness: 1.6mm [0.04"]\n'
'Weight: 30g',
'gtin': 'n/a',
'image': [{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/bf2e1d/4363359320/il_fullxfull.4363359320_696a.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/c/3000/2384/0/96/il/bf2e1d/4363359320/il_340x270.4363359320_696a.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_fullxfull.4363359086_ct24.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/2256d7/4363359086/il_340x270.4363359086_ct24.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_fullxfull.4410691101_mnk9.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/18d781/4410691101/il_340x270.4410691101_mnk9.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_fullxfull.4410691105_tows.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/02a44d/4410691105/il_340x270.4410691105_tows.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_fullxfull.4363305468_2u8t.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/3ce3cd/4363305468/il_340x270.4363305468_2u8t.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_fullxfull.4410691113_1vd7.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/9dd0f1/4410691113/il_340x270.4410691113_1vd7.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_fullxfull.4363305474_g8nj.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/764f10/4363305474/il_340x270.4363305474_g8nj.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_fullxfull.4410691059_tgtk.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/d8f739/4410691059/il_340x270.4410691059_tgtk.jpg'},
{'@context': 'https://schema.org',
'@type': 'ImageObject',
'author': 'WestArtFactory',
'contentURL': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_fullxfull.4410691049_mhx2.jpg',
'description': None,
'thumbnail': 'https://i.etsystatic.com/32739805/r/il/712818/4410691049/il_340x270.4410691049_mhx2.jpg'}],
'logo': 'https://i.etsystatic.com/isla/22edb6/52736152/isla_fullxfull.52736152_gllqny0m.jpg?version=0',
'material': 'Circuit Board/Gold/FR4',
'name': 'Python Cheat Sheet Coasters made from a high quality circuit board '
'for software engineers, hackers and programmers',
'offers': {'@type': 'Offer',
'availability': 'https://schema.org/InStock',
'eligibleQuantity': 416,
'price': '21.49',
'priceCurrency': 'USD'},
'review': [{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Sarah Givens'},
'datePublished': '2024-03-24',
'reviewBody': 'Super fast shipping and really great quality!',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Christina'},
'datePublished': '2024-03-14',
'reviewBody': 'Very pretty and useful, my husband was happy with '
'his gift',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'bob'},
'datePublished': '2024-03-08',
'reviewBody': 'good long delivery due to 3rd party company delay',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}},
{'@type': 'Review',
'author': {'@type': 'Person', 'name': 'Julien Aversano'},
'datePublished': '2024-02-27',
'reviewBody': 'Great quality, will order again',
'reviewRating': {'@type': 'Rating',
'bestRating': 5,
'ratingValue': 5}}],
'sku': '1336771386',
'url': 'https://www.etsy.com/listing/1336771386/python-cheat-sheet-coasters-made-from-a'}
import json
from parsel import Selector
# scrapingdog has no integration but we can use httpx
# install using `pip install httpx`
import httpx
# create an API client instance
client = httpx.Client(timeout=180)
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
params = {
"api_key": "YOUR API KEY",
"url": url,
}
api_result = client.get(
'https://api.scrapingdog.com/scrape',
params=params,
)
assert api_result.status_code == 200, api_result.text
return Selector(api_result.text, type="html")
url = "https://www.etsy.com/sg-en/listing/1336771386/python-cheat-sheet-coasters-made-from-a"
# scrape through US geo-location as etsy is blocked in many countries
selector = scrape(url, country="US")
# etsy has product data as JSON in a script element
script = selector.xpath("//script[contains(text(),'offers')]/text()").get()
data = json.loads(script)
from pprint import pprint
pprint(pprint(data))
How to choose a web scraping API for Etsy
Because Etsy sits behind DataDome, start by narrowing to the providers still clearing it this run (the top of the
ranked table), then choose within that set based on your job. Low volume listing lookups give you more room on
cost. Large listing and search crawls put more weight on reliability and concurrency.
Reliability first.Scrapfly leads the current Etsy ranking, which makes it the default
starting point for production listing and search crawls. The ranking history shows its track record.
Value. Among the APIs still clearing Etsy, sort by cost per successful request. The cheapest
sticker price is rarely the cheapest per usable Etsy page.
Speed. For latency sensitive listing lookups rather than bulk crawls, pick the fastest option
that still clears Etsy reliably.
The key principle is to judge on cost per successful request, not sticker price. DataDome returns block and
CAPTCHA pages as a 200, so a cheap API can look like it's working while delivering empty pages, which makes its
real cost per usable result far higher than the rate card suggests.
PRICING
How we benchmark web scraping APIs for Etsy
We independently benchmark 8 web scraping APIs against live Etsy pages, 1,000+ requests per
service, twice a month. Every API is tested against the same Etsy URLs at the same time, and cost is measured per
1,000 successful requests on entry plan pricing. We pay for the plans ourselves. No affiliate links. No sponsors.
Just data.
8 APIs ยท 1,000+ requests each ยท twice a month ยท no affiliate links, no sponsors
Benchmarking Etsy has one wrinkle worth knowing about. Success is measured on response content, not HTTP status
codes. DataDome's CAPTCHA interstitial and soft blocked responses return a 200, so a test that only checked for a
200 would overstate results. We verify that responses contain the expected listing data before counting them as
successful. Every provider is tested against the same URLs in the same run, so the numbers stay comparable.
Latest data: Jul 31 to Aug 14, 2026.
QUALITY TESTING
Frequently asked questions about scraping Etsy
Is it legal to scrape Etsy?
Scraping publicly available Etsy data such as prices and listing details is generally treated as lower risk than
scraping data behind a login, but Etsy's Terms of Service prohibit automated access, and seller and reviewer data
can fall under privacy laws. Legality depends on what you collect, where you operate, and how you use the data, so
treat this as general information rather than legal advice and check your own situation.
What's the cheapest API that works on Etsy?
Sort the ranked table by cost per successful request and read down to the first provider still clearing Etsy this
run. That's the cheapest option that actually delivers. Lower priced APIs further down often fail too many
requests for their sticker price to be meaningful, and DataDome's soft blocking hides those failures unless you
check response content.
Do I need a headless browser to scrape Etsy?
For the content, not always. Etsy listing pages are mostly server rendered, so an HTTP request with good proxies
often returns the core fields. For getting past DataDome reliably, a genuine browser context helps clear the
device challenge. The APIs at the top of the ranking handle the antibot layer for you.
Why do some APIs score low on Etsy?
Because DataDome scores device signals on every request and blocks the ones it doesn't trust. A request based tool
may return 200 responses that are actually CAPTCHA or block pages, and since we score on response content, those
count as failures rather than successes.
How often is this benchmark updated?
Twice a month against the same live Etsy targets, 1,000+ requests per API each run. We publish after validating
the run and checking failures for configuration or detection errors.
Conclusion
Etsy serves most of its listing data in server rendered HTML but gates access behind DataDome, so the right web
scraping API is one that clears the device challenge reliably rather than one that just renders JavaScript. For
production listing and search crawls, start with Scrapfly, the highest success rate in the current run.
For cost or speed on lighter listing jobs, choose among the providers still clearing Etsy this run.
Whatever you pick, verify results on response content rather than status codes, because DataDome's CAPTCHA and
block pages both return a 200. The benchmark refreshes twice a month, so check the live Etsy results before
committing.