Best Web Scraping APIs for LinkedIn: July 2026 Benchmark
Benchmarks run by the Scrapeway team ·
Last updated: August 14, 2026 ·
How we benchmark
Scrapfly is the best web scraping API for LinkedIn, with a
97% success rate across 8 web scraping APIs
benchmarked against live LinkedIn pages in July 2026.
3 of the 8 cleared LinkedIn reliably enough to recommend.
LinkedIn is protected by Arkose Labs and gates most data behind a login wall, so most scraping APIs either fail it
or pay for it in speed and cost. The benchmark is refreshed twice a month, with no affiliate links and no
sponsors.
Ranking history: web scraping APIs for LinkedIn over time
LinkedIn target ranking history
The 4 web scraping APIs for LinkedIn, reviewed
1. Scrapfly: 97% success on LinkedIn
On LinkedIn
Speed
Cost/1k
Overall
From
97%
32.7s
$6.83
#1 of 4
$30/mo
LinkedIn puts Arkose Labs in front of sessions it flags and shows an authwall to unauthenticated requests,
with public profile and job data served through its internal API rather than the visible HTML.
Scrapfly cleared 97% here by generating a genuine browser
fingerprint and passing the challenge, so it reaches the public profile and job data instead of an authwall.
Because it only bills for successful requests, the authwall and challenge pages LinkedIn returns don't quietly
run up the cost.
At $6.83 per 1,000 successful requests and
32.7s average response time, the figures are strong for a challenge gated site
like LinkedIn. The 97% success rate is the headline, and the ranking history above
shows how that rate has held across previous runs.
Pros:
Highest success rate in the LinkedIn benchmark this run, clearing the authwall and Arkose Labs' challenge on
public pages
Only charges for successful scrapes, so authwall and challenge pages cost nothing
One asp flag plus residential proxies handles LinkedIn with little tuning
First class SDKs for Python, TypeScript, Go, and Rust, plus a Scrapy extension
Cons:
Credit cost per request rises once ASP, JavaScript rendering, or residential proxies are enabled
The entry (Discovery) plan caps concurrency at 5, so large profile crawls need a higher tier
The free tier is a single batch of 1,000 credits, enough to prototype but not to benchmark at volume
2. Scrapingdog: 92% success on LinkedIn
On LinkedIn
Speed
Cost/1k
Overall
From
92%
0.2s
$10.00
#2 of 4
$40/mo
Scrapingdog offers low entry pricing and a simple API, without first party SDKs, and it markets a dedicated
LinkedIn endpoint. LinkedIn shows an authwall to unauthenticated requests, and Arkose Labs serves an
interactive challenge to sessions it flags. On LinkedIn it cleared 92% this run.
Pros:
Low entry pricing
Simple API with a dedicated LinkedIn endpoint
Cons:
No first party SDKs
Reviewers frequently cite slow, email only support with no live chat option
3. WebScrapingAPI: 90% success on LinkedIn
On LinkedIn
Speed
Cost/1k
Overall
From
90%
14.8s
$2.71
#3 of 4
$19/mo
WebScrapingAPI covers a broad range of language SDKs behind a simple REST interface, so it drops into most
stacks without a client library, and it supports async and batch submission for queued jobs. The scorecard
carries this run's speed and cost. On LinkedIn it cleared 90% this run.
Pros:
Broad language SDK range behind a simple REST interface
Async and batch submission for queued jobs
Cons:
LinkedIn's authwall pages make it easy to keep paying unless you check content
Support tickets often go unanswered for days, per user reviews
4. Scraperapi: 88% success on LinkedIn
On LinkedIn
Speed
Cost/1k
Overall
From
88%
16.6s
$14.70
#4 of 4
$49/mo
Scraperapi is built around a fast request path. When it clears a request it tends to return quickly, which
suits latency sensitive lookups that can absorb retries. LinkedIn's authwall and Arkose Labs' challenge sit in
front of the public profile and job data, and the scorecard carries how it did on that this run. On LinkedIn
it cleared 88% this run.
Pros:
Fast when it clears, with a simple integration
Broad language SDK support
Cons:
Login required flows are off limits, and much of LinkedIn sits behind a login
Geotargeting is gated by plan (US and EU only until the Business tier)
COMPARISON
About scraping LinkedIn
LinkedIn is the largest professional network, and the data people scrape from it is mostly career detail. That
means public profile headlines and summaries, current and past positions, skills, education, company pages, and
public job postings with titles, companies, locations, and descriptions. The subset that is publicly reachable
without an account is limited to public profile and job pages.
LinkedIn is a JavaScript heavy app that serves data through its internal (Voyager) API rather than the page HTML,
and it shows an authwall to unauthenticated or suspicious requests. Much of the platform, including full profiles,
connections, and feeds, sits behind a login and is out of scope for public scraping. The benchmark targets public
profile and job data only.
AUTOMATION
LinkedIn is protected by Arkose Labs, which serves an interactive challenge to sessions it flags. Arkose Labs is
not one of the benchmarked antibot pages in this series, so there is no dedicated page to link. The key point for
scraping LinkedIn is that its authwall and challenge pages return without a hard error status, so success has to
be measured on response content, not status codes.
linkedin_scraper.py
importjsonfromparselimportSelector# install using `pip install scrapfly-sdk`fromscrapflyimportScrapflyClient,ScrapeConfig,ScrapeApiResponse# create an API client instanceclient=ScrapflyClient(key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.scrape(ScrapeConfig(url=url,headers=headers,asp=True,render_js=render_jsorFalse,cache=False,cache_ttl=900,method='GET',))returnapi_result.selectorurl="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# install using `pip install scrapfly-sdk`
from scrapfly import ScrapflyClient, ScrapeConfig, ScrapeApiResponse
# create an API client instance
client = ScrapflyClient(key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.scrape(ScrapeConfig(
url=url,
headers=headers,
asp=True,
render_js=render_js or False,
cache=False,
cache_ttl=900,
method='GET',
))
return api_result.selector
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# scrapingdog has no integration but we can use httpx# install using `pip install httpx`importhttpx# create an API client instanceclient=httpx.Client(timeout=180)# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:params={"api_key":"YOUR API KEY",# this endpoint takes a profile id rather than a url"linkId":'pedro-mac-dowell',"type":'profile',}api_result=client.get('https://api.scrapingdog.com/linkedin',params=params,)assertapi_result.status_code==200,api_result.text# this dedicated endpoint answers with structured JSON rather than page HTML,# so read the fields straight off api_result.json() instead of the selector belowreturnSelector(api_result.text,type="html")url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# scrapingdog has no integration but we can use httpx
# install using `pip install httpx`
import httpx
# create an API client instance
client = httpx.Client(timeout=180)
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
params = {
"api_key": "YOUR API KEY",
# this endpoint takes a profile id rather than a url
"linkId": 'pedro-mac-dowell',
"type": 'profile',
}
api_result = client.get(
'https://api.scrapingdog.com/linkedin',
params=params,
)
assert api_result.status_code == 200, api_result.text
# this dedicated endpoint answers with structured JSON rather than page HTML,
# so read the fields straight off api_result.json() instead of the selector below
return Selector(api_result.text, type="html")
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# webscrapingapi has a Python SDK but it's not great, use httpx instead:# `pip install httpx`importhttpx# create an API client instanceclient=httpx.Client(timeout=180)# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.get("https://api.webscrapingapi.com/v2",# the target url is passed as a parameter belowheaders=headers,params={"url":url,"api_key":"YOUR API KEY",# NOTE: add your API KEY here!"timeout":60_000,"render_js":render_jsorFalse,"method":'GET',},)assertapi_result.status_code==200,api_result.reason_phrasereturnSelector(api_result.text)url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# webscrapingapi has a Python SDK but it's not great, use httpx instead:
# `pip install httpx`
import httpx
# create an API client instance
client = httpx.Client(timeout=180)
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.get(
"https://api.webscrapingapi.com/v2", # the target url is passed as a parameter below
headers=headers,
params={
"url": url,
"api_key": "YOUR API KEY", # NOTE: add your API KEY here!
"timeout": 60_000,
"render_js": render_js or False,
"method": 'GET',
},
)
assert api_result.status_code == 200, api_result.reason_phrase
return Selector(api_result.text)
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# install using `pip install scraperapi-sdk`fromscraperapi_sdkimportScraperAPIClient# create an API client instanceclient=ScraperAPIClient(api_key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.make_request(url=url,headers=headers,params={"render":False,},method='GET',)assertapi_result.ok,api_result.textreturnSelector(api_result.text)url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# install using `pip install scraperapi-sdk`
from scraperapi_sdk import ScraperAPIClient
# create an API client instance
client = ScraperAPIClient(api_key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.make_request(
url=url,
headers=headers,
params={
"render": False,
},
method='GET',
)
assert api_result.ok, api_result.text
return Selector(api_result.text)
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# install using `pip install zenrows`fromzenrowsimportZenRowsClient# create an API client instanceclient=ZenRowsClient(apikey="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.get(url,headers=headers,params={})# zenrows answers with the page HTML; "json_response" is only accepted with js_render onassertapi_result.ok,api_result.textreturnSelector(api_result.text)url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# install using `pip install zenrows`
from zenrows import ZenRowsClient
# create an API client instance
client = ZenRowsClient(apikey="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.get(
url,
headers=headers,
params={
}
)
# zenrows answers with the page HTML; "json_response" is only accepted with js_render on
assert api_result.ok, api_result.text
return Selector(api_result.text)
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# install using `pip install scrapingbee`fromscrapingbeeimportScrapingBeeClient# create an API client instanceclient=ScrapingBeeClient(api_key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.get(url,headers=headers,params={"json_response":True,"transparent_status_code":True,})assertapi_result.ok,api_result.textdata=api_result.json()returnSelector(data['body'])url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# install using `pip install scrapingbee`
from scrapingbee import ScrapingBeeClient
# create an API client instance
client = ScrapingBeeClient(api_key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.get(
url,
headers=headers,
params={
"json_response": True,
"transparent_status_code": True,
}
)
assert api_result.ok, api_result.text
data = api_result.json()
return Selector(data['body'])
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# install using `pip install firecrawl-py`fromfirecrawlimportFirecrawl# create an API client instanceclient=Firecrawl(api_key="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.scrape(url=url,headers=headersor{},)assertapi_result.raw_html,"firecrawl returned no html for this page"returnSelector(api_result.raw_html)url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# install using `pip install firecrawl-py`
from firecrawl import Firecrawl
# create an API client instance
client = Firecrawl(api_key="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.scrape(
url=url,
headers=headers or {},
)
assert api_result.raw_html, "firecrawl returned no html for this page"
return Selector(api_result.raw_html)
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
importjsonfromparselimportSelector# install using `pip install scrapingant-client`fromscrapingant_clientimportScrapingAntClient# create an API client instanceclient=ScrapingAntClient(token="YOUR API KEY")# create scrape function that returns HTML parser for a given URLdefscrape(url:str,country:str="",render_js=False,headers:dict=None)->Selector:api_result=client.general_request(url,)# the scrapingant client returns its own Response object: content holds the pageassertapi_result.status_code==200,api_result.textreturnSelector(api_result.content)url="https://www.linkedin.com/in/adammgrant"selector=scrape(url)# big chunk of the dataset can be found in microdata markup:data=json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())# the resulting dataset is pretty big but here are some example fields:person_data=next(dfordindata['@graph']ifd['@type']=="Person")frompprintimportpprintpprint(person_data)
Output$ python linkedin_scraper.py
{'@type': 'Person',
'address': {'@type': 'PostalAddress',
'addressCountry': 'US',
'addressLocality': 'Filadelfia, Pennsylvania, Estados Unidos'},
'alumniOf': [{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2003,
'startDate': 1999},
'name': 'Harvard University',
'url': 'https://www.linkedin.com/school/harvard-university/'},
{'@type': 'EducationalOrganization',
'member': {'@type': 'OrganizationRole',
'endDate': 2006,
'startDate': 2003},
'name': '********** ** ********'}],
'awards': ['100 Most Creative People in Business',
'Class of 1984 Teaching Award',
'#1 New York Times bestseller',
'Thinkers 50 Most Influential Management Thinkers',
'Class of 1984 Teaching Award',
'World Economic Forum Young Global Leader',
'Fellow, Martin Prosperity Institute',
"HR's Most Influential International Thinkers",
'"Goes Above and Beyond the Call of Duty" MBA Teaching Award',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award',
'Excellence in Teaching Award',
'Forbes Most Dynamic Social Innovation Initiatives of 2013',
'Harvard Business Review Ideas that Shaped Management',
'Wall Street Journal Favorite Books of 2013',
'Washington Post Books Every Leader Should Read',
'Amazon Best Books of the Year',
'Financial Times Books of the Year',
'Inc. Best Books of 2013 for Entrepreneurs',
"Fortune's Five Must-Read Business Books",
'Oprah Magazine 15 riveting reads to pick up in May',
'Class of 1984 Teaching Award',
'Excellence in Teaching Award, MBA Curriculum',
'New York Times bestseller',
'Wall Street Journal bestseller',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'BusinessWeek Favorite Professors',
'Class of 1984 Teaching Award',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Cummings Scholarly Achievement Award',
'Distinguished Scientific Award for Early Career Contribution to '
'Applied Psychology',
'Distinguished Early Career Contributions Award – Science',
'Excellence in Teaching Award, MBA Core Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'“Goes Above and Beyond the Call of Duty” MBA Teaching Award',
'World’s 40 Best Business School Professors Under 40',
'Owens Scholarly Achievement Award, Best Publication in I/O '
'Psychology',
'Excellence in Teaching Award, MBA Elective Curriculum',
'Excellence in Teaching Award, Undergraduate Division',
'MBA Teaching All-Star',
'Tanner Award for Excellence in Undergraduate Teaching',
'Rensis Likert Prize, Best Paper from a Dissertation in '
'Organization Studies',
'Weatherspoon Award for Excellence in Undergraduate Teaching',
'Best Published Scholarly Article',
'Early Research Award, Applied Science',
'Graduate Research Fellowship',
'Junior Fellow',
'Manager of the Year'],
'description': "Recognized as Wharton's top-rated professor, and one of the "
"world's 10 most influential…",
'disambiguatingDescription': 'Creator, Top Voice',
'image': {'@type': 'ImageObject',
'contentUrl': 'https://media.licdn.com/dms/image/C4E03AQFGdrbBw3FYhA/profile-displayphoto-shrink_200_200/0/1629123595757?e=2147483647&v=beta&t=D7WsVKwVonUGSGEJKzoEzdJiKBwDKx2zVmkm66I3rCM'},
'interactionStatistic': {'@type': 'InteractionCounter',
'interactionType': 'https://schema.org/FollowAction',
'name': 'Follows',
'userInteractionCount': 5310073},
'jobTitle': ['******, ****** *********, ***** *****, **** *** ****, '
'*********, ****** *',
'*** **** *. ********* ********* ** ********** *** **********',
'************ **-** ******',
'******* *******',
'********* ********* ** **********, **** ******',
'********* ******',
'********* ********* ** **********',
'********* ********* ** ************** ********',
'******** ** ********* & ******** ***********',
'******** ** *********** *****'],
'knowsLanguage': [{'@type': 'Language', 'name': 'English'},
{'@type': 'Language', 'name': 'Spanish'}],
'memberOf': [],
'name': 'Adam Grant',
'sameAs': 'https://www.linkedin.com/in/adammgrant',
'url': 'https://www.linkedin.com/in/adammgrant',
'worksFor': [{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': 'Penguin Publishing Group',
'url': 'https://www.linkedin.com/company/penguin-group-usa'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '*** *** **** *****'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '********** ******** ******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'member': {'@type': 'OrganizationRole'},
'name': '******* ** ********** *******'},
{'@type': 'Organization',
'location': 'Philadelphia, PA',
'member': {'@type': 'OrganizationRole'},
'name': '*** ******* ******'},
{'@type': 'Organization',
'location': 'Chapel Hill, NC',
'member': {'@type': 'OrganizationRole'},
'name': '********** ** ***** ******** ** ****** ****'},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"},
{'@type': 'Organization',
'location': 'Cambridge, MA',
'member': {'@type': 'OrganizationRole'},
'name': "***'* ** ************"}]}
import json
from parsel import Selector
# install using `pip install scrapingant-client`
from scrapingant_client import ScrapingAntClient
# create an API client instance
client = ScrapingAntClient(token="YOUR API KEY")
# create scrape function that returns HTML parser for a given URL
def scrape(url: str, country: str="", render_js=False, headers: dict=None) -> Selector:
api_result = client.general_request(
url,
)
# the scrapingant client returns its own Response object: content holds the page
assert api_result.status_code == 200, api_result.text
return Selector(api_result.content)
url = "https://www.linkedin.com/in/adammgrant"
selector = scrape(url)
# big chunk of the dataset can be found in microdata markup:
data = json.loads(selector.xpath("//script[@type='application/ld+json']/text()").get())
# the resulting dataset is pretty big but here are some example fields:
person_data = next(d for d in data['@graph'] if d['@type'] == "Person")
from pprint import pprint
pprint(person_data)
How to choose a web scraping API for LinkedIn
Because LinkedIn sits behind an authwall and Arkose Labs' challenge, start by narrowing to the providers still
clearing it this run (the top of the ranked table), then choose within that set based on your job. Low volume
public profile lookups give you more room on cost. Large profile or job crawls put more weight on reliability and
concurrency.
Reliability first.Scrapfly leads the current LinkedIn ranking, which makes it the
default starting point for production public profile and job crawls. The ranking history shows its track record.
Value. Among the APIs still clearing LinkedIn, sort by cost per successful request. The
cheapest sticker price is rarely the cheapest per usable LinkedIn page.
Speed. For latency sensitive profile lookups rather than bulk crawls, pick the fastest option
that still clears LinkedIn reliably.
The key principle is to judge on cost per successful request, not sticker price. LinkedIn returns authwall and
challenge pages that can look like a normal response, so a cheap API can burn through requests without returning
usable data, which makes its real cost per usable result far higher than the rate card suggests.
PRICING
How we benchmark web scraping APIs for LinkedIn
We independently benchmark 8 web scraping APIs against live LinkedIn pages, 1,000+ requests per
service, twice a month. Every API is tested against the same public LinkedIn URLs at the same time, and cost is
measured per 1,000 successful requests on entry plan pricing. We pay for the plans ourselves. No affiliate links.
No sponsors. Just data.
8 APIs · 1,000+ requests each · twice a month · no affiliate links, no sponsors
Benchmarking LinkedIn has one wrinkle worth knowing about. Success is measured on response content, not HTTP
status codes. LinkedIn's authwall and Arkose Labs' challenge pages can return without a hard error status, so a
test that only checked the status code would overstate results. We verify that responses contain the expected
public profile or job data before counting them as successful. Every provider is tested against the same URLs in
the same run, so the numbers stay comparable.
Latest data: Jul 31 to Aug 14, 2026.
QUALITY TESTING
Frequently asked questions about scraping LinkedIn
Is it legal to scrape LinkedIn?
Scraping publicly available LinkedIn profile data is sensitive because it is personal data that can fall under
privacy laws such as GDPR and CCPA, and LinkedIn's Terms of Service prohibit automated access. Public data
scraping has been the subject of high profile litigation, so the legal picture is nuanced. Treat this as general
information rather than legal advice and check your own situation before scraping LinkedIn.
What's the cheapest API that works on LinkedIn?
Sort the ranked table by cost per successful request and read down to the first provider still clearing LinkedIn
this run. That's the cheapest option that actually delivers. Lower priced APIs further down often fail too many
requests for their sticker price to be meaningful, and LinkedIn's authwall hides those failures unless you check
response content.
Do I need a headless browser to scrape LinkedIn?
In practice yes. Arkose Labs serves an interactive challenge to sessions it flags, and clearing it reliably needs
a genuine browser context. The profile and job data itself comes from LinkedIn's internal API rather than the page
HTML, so once you're past the challenge you read that API response rather than the DOM. The APIs at the top of the
ranking handle the browser and challenge for you.
Why do some APIs score low on LinkedIn?
Because LinkedIn shows an authwall to unauthenticated or suspicious requests and Arkose Labs challenges sessions
it flags. A client it doesn't trust receives authwall or challenge pages, and since we score on response content,
those count as failures rather than successes.
How often is this benchmark updated?
Twice a month against the same live public LinkedIn targets, 1,000+ requests per API each run. We publish after
validating the run and checking failures for configuration or detection errors.
Conclusion
LinkedIn gates most data behind an authwall and Arkose Labs' challenge, and serves it through an internal API, so
the right web scraping API is one that clears the challenge reliably and reaches the public profile and job data.
For production public profile and job crawls, start with Scrapfly, the highest success rate in the
current run. For cost or speed on lighter lookups, choose among the providers still clearing LinkedIn this run.
Whatever you pick, verify results on response content rather than status codes, because LinkedIn's authwall and
challenge pages don't always carry a hard error status. The benchmark refreshes twice a month, so check the live
LinkedIn results before committing.