10 ms·

Web Scraping via JavaScript Runtime Heap Snapshots (2022)

362 points by djoldman 3y ago

unixfox 3y ago
https://news.ycombinator.com/item?id=31205139 https://news.ycombinator.com/item?id=31205139
1-6 3y ago
Is there something which would allow me to do this with Jupyter Notebook?
- _boffin_ 3y ago
  What exactly are you trying to do?
c0balt 3y ago
Huh, I'm surprised this gets past the symbol mangling done by most js minifiers. Though maybe they don't mangle attribute fields everywhere. However this is a nice hack around "modern" page structures and kudos to the author for making a proper tool out of it.
- bugsliker 3y ago
  It's the JSON data payload that has unminified keys. Though YouTube is one of the few google sites that still use JSON, most use protocol buffers which generate JS interfaces which would indeed be mangled by minifiers.
- madeofpalk 3y ago
  Most/all minifiers won't actually mangle object property names as those often have observable side effects. You want to grab all the keys for an object and do something different depending on the name of the key - you can no longer do that if the minifier has mangled all the object keys. Not to mention I imagine it would be significantly harder track all references to object keys across an application (as opposed to just local variables).
  - btown 3y ago
    As I recall, the only minifier that went this far was Closure Compiler [0] - the caveats documented there still apply! (Fun fact: I believe that Closure Library, and by extension the Closure Compiler, are still used for the Gmail rich text editor! [1]) [0] https://developers.google.com/closure/compiler/docs/api-tutorial3 https://developers.google.com/closure/compiler/docs/api-tuto... [1] https://google.github.io/closure-library/source/closure/goog/demos/editor/editor.html https://google.github.io/closure-library/source/closure/goog...
    - nathell 3y ago
      And ClojureScript.
      - deleted 3y ago
        [deleted]
simonw 3y ago
Has anyone seen a version of this trick that works with Playwright instead of Puppeteer? EDIT: https://github.com/adriancooney/puppeteer-heap-snapshot/blob/master/src/capture.ts https://github.com/adriancooney/puppeteer-heap-snapshot/blob... is the code that captures the snapshot, and it uses createCDPSession() - it looks like Playwright has an equivalent for that Puppeteer API, documented here: https://playwright.dev/docs/api/class-cdpsession https://playwright.dev/docs/api/class-cdpsession
- None4U 3y ago
  The main/only difference here is that Puppeteer only supports Chromium, while Playwright support multiple browsers. CDP is the Chrome DevTools Protocol. Otherwise, as long as you're using Chrome in both, you get the same base protocol with a different API.
bnchrch 3y ago
I love this. Thank you. As I understand it this only works for SPAs or other heavy js frontends and would not work on HTML. I think that’s fine. What I’m really excited is this combined with traditional mark up scanning plus (incoming buzz word) AI. Scraping is slowly becoming unstoppable and that a good thing.
- koromak 3y ago
  Is that true? The DOM is represented somewhere too, probably in the same heap. I'd bet it works on static sites.
  - bnchrch 3y ago
    Oh you may be right!
svdr 3y ago
Wouldn't web scraping be possible by taking screenshots of the rendered pages and then reading them with OCR?
- ekianjo 3y ago
  probably very inefficient as it would depend on layout a lot too
  - cush 3y ago
    As inefficient as parsing heap snapshots?
    - brigadier132 3y ago
      Much more
- deleted 3y ago
  [deleted]
- spaniard89277 3y ago
  You'll be spending resources on LLMs like crazy. Possible but very messy IMO.
  - anamexis 3y ago
    You don't need LLMs for OCR.
    - ekianjo 3y ago
      OCR does not get you the names of the classes in a DOM
    - spaniard89277 3y ago
      No but maybe you want to do something with the ocr output.
ricardo81 3y ago
>In recent years, the web has gotten very hostile to the lowly web scraper. It's a result of the natural progression of web technologies away from statically rendered pages to dynamic apps built with frameworks like React and CSS-in-JS. Dunno, a lot of the time it actually makes scraping easier because the content that's not in the original source tends to be served up as structured data via XHR- JSON usually- you just need to take a look at the data you're interested in and if it's not in 'view-source', it's coming from somewhere else. Browser based scraping makes sense when that data is heavily mangled or obfuscated, laden with captchas and other anti-scraping methods. Or if you're interested in if text is hidden, what position it's on the page etc.
- Raed667 3y ago
  Agreed! multiple times I wasted hours figuring out what selectors to use, but then remembered that I can just look at the network tab and have perfectly structured JSON data.
  - spaniard89277 3y ago
    But what if their backend blocks you? I'm trying to develop a instagram scraper and I find that I'll have to spend money on rotating proxies. It doesn't matter if you scrape the DOM or get some Json. I just need to scrape some public account posts and, I may be dumb, but I dunno how to do that with the official APIs (developers.facebook is hard to understand for me).
    - Raed667 3y ago
      Instagram is notoriously hard to scrape. I was talking more about random SPAs that you can find. For Instagram, here is a link that maybe helps: https://apify.com/apify/instagram-profile-scraper https://apify.com/apify/instagram-profile-scraper
    - blister 3y ago
      Hah, I literally just fought this for the past month. We run a large esports league that relies on player ranked data. They have the data, and as mentioned above, they send it down to the browser in beautiful JSON objects. But they're sitting behind Cloudflare and aggressively blocking attempts to fetch data programmatically, which is a huge problem for us with 6000+ players worth of data to fetch multiple times every 3 months. So... I built a Chrome Extension to grab the data at a speed that is usually under their detection rate. Basically created a distributed scraper and passed it out to as many people in the league as I could. For big jobs when we want to do giant batches, it was a simple matter of doing the pulls and when we start getting 429 errors (rate limit blocking code they use), switch to a new IP on the VPN. The only way they can block us now is if they stop having a website. Give one of the commercial VPN providers a try. They're usually pretty cheap and have tons of IPs all over the place. Adding a "VPN Disconnect / Reconnect" step to the process only added about 10 seconds per request every so often.
johnnyworker 3y ago
A bit of a tangent, but a long time ago I was kicked out of a Facebook group for what I considered to be completely made up reasons -- and what really got to me was that by being banned from it, I couldn't even point to the posts that had been actively misunderstood and distorted. I couldn't find anything in the cache files, so I saved a process dump of the still running Firefox and stitched the posts together from that. I stopped caring the as soon as I had my proof, but was still sheepishly proud that I managed to get it.
- Alifatisk 3y ago
  That's very clever
- wmichelin 3y ago
  When you figured it out, what was the cause?
  - johnnyworker 3y ago
    It's been over 10 years, and not interesting, but because you asked: it was a political group, and I was replying to a group member who was talking about how love is all you need. I tried to make the point that love may be an essential ingredient, but is not all you need, certainly not for political change (I think I used some Chomsky quotes to that effect...), and that group member essentially straw manned that into advocating hate (which I really wasn't). Knowing how I was back then, I probably was being a Dwight Schrute about it, as in "bzzzt, false, love is NOT all you need, because A, B and C". So we were both irritated and talked back back and forth, ceding no ground, and suddenly an admin banned me without warning. It was the suddenness more than anything that got to me; I felt misrepresented and censored, and I just wanted to be able to re-read it all to gain closure, if you will. If I could find the file today I would probably cringe at what I wrote, or at least how I wrote it, but back then I nodded and thought "yup, I'm correct", haha.
    - wmichelin 3y ago
      Totally feel that, I got a life-time ban from a huge programming sub on Reddit for being rude to a person who was obviously just copying-and-pasting their software homework on the forum.
tantalor 3y ago
> These properties were chosen manually by working backward from the visible data on the web page to the data structure containing said data (I'll dive into that process in another blog post) That would seem to be the actually interesting/challenging part.
- jackbeck 3y ago
  I would assume that you could just search the heap by the value shown on the page to find out what the key is.
- ricklamers 3y ago
  Using introspection tools like queryObjects in Chrome Dev Tools should get you quite far: https://developer.chrome.com/docs/devtools/console/utilities/#queryObjects-function https://developer.chrome.com/docs/devtools/console/utilities...
_boffin_ 3y ago
Anyone know of any research on generating HTML differentials against updated webpages and with automatic healing of wrappers / selectors or research on using LLMs with webscraping and how to reduce token usage while retaining context?
- simonw 3y ago
  I've been doing some pretty dumb tricks for reducing token usage and piping to LLMs which have worked really well. I have a strip-tags CLI tool which I can pipe HTML through on its way to an LLM, described here: https://simonwillison.net/2023/May/18/cli-tools-for-llms/ https://simonwillison.net/2023/May/18/cli-tools-for-llms/ I also do things like this: shot-scraper javascript news.ycombinator.com 'document.body.innerText' -r \ | llm -s 'General themes, illustrated by emoji' Output here: https://gist.github.com/simonw/3fbfa44f83e12f9451b58b595451414a https://gist.github.com/simonw/3fbfa44f83e12f9451b58b5954514... That's using https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ to get just the document.body.innerText as a raw string, then piping that to gpt-3.5-turbo with a system prompt. In terms of retaining context, I added a feature to my strip-tags tool where you can ask it to NOT strip specific tags - e.g.: curl -s https://www.theguardian.com/us | \ strip-tags -m -t h1 -t h2 -t h3 That strips all HTML tags except for h1, h2 and h3 - output here: https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd8871f https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd88... Full documentation here: https://github.com/simonw/strip-tags/blob/main/README.md https://github.com/simonw/strip-tags/blob/main/README.md
  - _boffin_ 3y ago
    First, love your work. Sadly, I don't think this path would work for me as for what I do, I need the selectors as part of the workflow. Roughly, my end goal is to do a single or multi-shot with the following information HTML differential (could be selectors, xpaths, data regions, differentials of any of the above, etc...), code stacktrace, related code, and prompt. For this example, let's consider that the flow involves the bot to login to a website. I have selectors for the `.username` and `.password` inputs and then a selector for the login button as `.login-btn`. 1. The site updates their page and changes up all their IDs, but keeps the same structure. 2. The site updates their page and changes up all their IDs, but changes the structure and the form is named something different and is somewhere else in the DOM. 3. many... many other examples. Trying to figure out how to minimize the tokens, but keep the needed context to regenerate the selectors that are needed to maintain the workflow.
topherjaynes 3y ago
I'm not a legal scholar and this isn't my area of expertise, but the final note has links out to a TechCrunch article about LinkedIn vs hiQ Labs Inc, which alludes to web scraping being legal, but the case wasn't decided for a few more months, and the court sided with Linkedin. What's the final verdict on web scraping vs creating fake accounts to get user information (which the case focused on)
jawerty 3y ago
This is very cool. I do a lot of puppeteer scraping and this library would help with a lot of the more complicated DOMs to work with. I have a live coding stream I did the other day scraping Facebook for comments https://www.youtube.com/live/03oTYPm12y8?feature=share https://www.youtube.com/live/03oTYPm12y8?feature=share If you're interested in seeing puppeteer in action I started doing streams last month where I talk through my method. I’ll be posting a lot more since it's been very fun. Overall puppeteer is great because you get to easily inject js scripts in a nice API. Selenium is great too but not as developed of a web scraping interface imo. Also puppeteer is a very optimized headless browser which is a given. What really matters is implementing a VPN proxy and storing your cookies during auth routines which I can get into if you have any questions about that.
solanav 3y ago
Cool! I use selenium to do phishing detection at my company and I use javascript declared variables as a source of data to analyse. It’s specially useful for links that are obfuscated by concatenating two variables into another one.
breatheoften 3y ago
I used a technique like this a few years back in a production product ... We had an integration partner (who we had permission to integrate with) that offered a different api for integration partners than was used for their website but which was horribly broken and regularly gave out the wrong data. The api was broken but the data displayed on their web page was fine so someone on the team wrote a browser automation (using ruby and selenium!) to drive the browser through the series of pages needed to retrieve all the information required. Needless to say, this broke all the time as the page/css changed etc. At some point I got pulled in and ran screaming away from selenium to puppeteer -- and quickly discovered the joy that is scripting the browser via natively supported api's and the chrome debugger protocol. The partners web page happened to be implemented with the apollo graphql client and I came across the puppeteer api for scanning the javascript heap -- I realized that if I could find the apollo client instance in memory (buried as a local variable inside some function closure referenced within the web app) -- I could just use it myself to get the data I needed ... coded it up in an hour or so and it just worked ... super fun and effective way to write a "scraper"! OnDocumentReady -> scan the heap for the needed object -> use it directly to get the data you need
- visarga 3y ago
  Wondering if we can automate the finding part with a LLM. You just tell if what data you want scraped.
  - dontupvoteme 3y ago
    I think the approach would be to classify the property by the values.
- paulddraper 3y ago
  > scripting the browser via natively supported api's Every modern browser has native support for WebDriver (i.e. Selenium) APIs. The advantage to Puppetteer is that the Chrome Dev Tools API is just a better API.
j0hnyl 3y ago
Does JavaScript state live in the heap too? Can we use this technique to pull objects from memory?
xkcd1963 3y ago
My dear fellows this is joyous news
adriancooney 3y ago
Thanks for posting this again! It's a year later and I still haven't touched the web scraper in production which is great to reflect on. It seems running the Youtube command on the post is still producing the exact same data too. $ npx puppeteer-heap-snapshot query \ --url https://www.youtube.com/watch\?v\=L_o_O7v1ews \ --properties channelId,viewCount,keywords --no-headless
- odysseus 3y ago
  Did you ever make another blog post about how to choose properties working backward from the visible data on the web page to the data structure containing said data? Searching the heap manually is not working very well. The data I want is in a (very) long list of irrelevant values within a "strings" key. It might have something to do with the data on the page that I want to scrape being rendered by JavaScript.
lcof 3y ago
[dead]
elwell 3y ago
I'm tempted to scrape this blog page so I can read it without my eyes hurting.
elwell 3y ago
It's still a game of cat and mouse. Next step is for the website to store multiple instances of similarly structured data so you scrape the dummy unknowingly.
- dontupvoteme 3y ago
  Or generate garbage data to poison their scraper.
bonadrag 3y ago
How do we evade the captchas with this method? Asking for a friend...
enson110 3y ago
Thanks for sharing. I think one of the most challenging part of web scraping is dealing with the website's anti-scraping measures, such as it needs to sign in, encountering 403 forbidden error, and reCAPTCHA. Does anyone have more experience in handling that?

362 points by djoldman 3y ago

unixfox 3y ago

https://news.ycombinator.com/item?id=31205139 https://news.ycombinator.com/item?id=31205139

1-6 3y ago

Is there something which would allow me to do this with Jupyter Notebook?

_boffin_ 3y ago
What exactly are you trying to do?

c0balt 3y ago

Huh, I'm surprised this gets past the symbol mangling done by most js minifiers. Though maybe they don't mangle attribute fields everywhere. However this is a nice hack around "modern" page structures and kudos to the author for making a proper tool out of it.

bugsliker 3y ago
It's the JSON data payload that has unminified keys. Though YouTube is one of the few google sites that still use JSON, most use protocol buffers which generate JS interfaces which would indeed be mangled by minifiers.
madeofpalk 3y ago
Most/all minifiers won't actually mangle object property names as those often have observable side effects. You want to grab all the keys for an object and do something different depending on the name of the key - you can no longer do that if the minifier has mangled all the object keys. Not to mention I imagine it would be significantly harder track all references to object keys across an application (as opposed to just local variables).
- btown 3y ago
  As I recall, the only minifier that went this far was Closure Compiler [0] - the caveats documented there still apply! (Fun fact: I believe that Closure Library, and by extension the Closure Compiler, are still used for the Gmail rich text editor! [1]) [0] https://developers.google.com/closure/compiler/docs/api-tutorial3 https://developers.google.com/closure/compiler/docs/api-tuto... [1] https://google.github.io/closure-library/source/closure/goog/demos/editor/editor.html https://google.github.io/closure-library/source/closure/goog...
  - nathell 3y ago
    And ClojureScript.
    - deleted 3y ago
      [deleted]

simonw 3y ago

Has anyone seen a version of this trick that works with Playwright instead of Puppeteer? EDIT: https://github.com/adriancooney/puppeteer-heap-snapshot/blob/master/src/capture.ts https://github.com/adriancooney/puppeteer-heap-snapshot/blob... is the code that captures the snapshot, and it uses createCDPSession() - it looks like Playwright has an equivalent for that Puppeteer API, documented here: https://playwright.dev/docs/api/class-cdpsession https://playwright.dev/docs/api/class-cdpsession

None4U 3y ago
The main/only difference here is that Puppeteer only supports Chromium, while Playwright support multiple browsers. CDP is the Chrome DevTools Protocol. Otherwise, as long as you're using Chrome in both, you get the same base protocol with a different API.

bnchrch 3y ago

I love this. Thank you. As I understand it this only works for SPAs or other heavy js frontends and would not work on HTML. I think that’s fine. What I’m really excited is this combined with traditional mark up scanning plus (incoming buzz word) AI. Scraping is slowly becoming unstoppable and that a good thing.

koromak 3y ago
Is that true? The DOM is represented somewhere too, probably in the same heap. I'd bet it works on static sites.
- bnchrch 3y ago
  Oh you may be right!

svdr 3y ago

Wouldn't web scraping be possible by taking screenshots of the rendered pages and then reading them with OCR?

ekianjo 3y ago
probably very inefficient as it would depend on layout a lot too
- cush 3y ago
  As inefficient as parsing heap snapshots?
  - brigadier132 3y ago
    Much more
deleted 3y ago
[deleted]
spaniard89277 3y ago
You'll be spending resources on LLMs like crazy. Possible but very messy IMO.
- anamexis 3y ago
  You don't need LLMs for OCR.
  - ekianjo 3y ago
    OCR does not get you the names of the classes in a DOM
  - spaniard89277 3y ago
    No but maybe you want to do something with the ocr output.

ricardo81 3y ago

>In recent years, the web has gotten very hostile to the lowly web scraper. It's a result of the natural progression of web technologies away from statically rendered pages to dynamic apps built with frameworks like React and CSS-in-JS. Dunno, a lot of the time it actually makes scraping easier because the content that's not in the original source tends to be served up as structured data via XHR- JSON usually- you just need to take a look at the data you're interested in and if it's not in 'view-source', it's coming from somewhere else. Browser based scraping makes sense when that data is heavily mangled or obfuscated, laden with captchas and other anti-scraping methods. Or if you're interested in if text is hidden, what position it's on the page etc.

Raed667 3y ago
Agreed! multiple times I wasted hours figuring out what selectors to use, but then remembered that I can just look at the network tab and have perfectly structured JSON data.
- spaniard89277 3y ago
  But what if their backend blocks you? I'm trying to develop a instagram scraper and I find that I'll have to spend money on rotating proxies. It doesn't matter if you scrape the DOM or get some Json. I just need to scrape some public account posts and, I may be dumb, but I dunno how to do that with the official APIs (developers.facebook is hard to understand for me).
  - Raed667 3y ago
    Instagram is notoriously hard to scrape. I was talking more about random SPAs that you can find. For Instagram, here is a link that maybe helps: https://apify.com/apify/instagram-profile-scraper https://apify.com/apify/instagram-profile-scraper
  - blister 3y ago
    Hah, I literally just fought this for the past month. We run a large esports league that relies on player ranked data. They have the data, and as mentioned above, they send it down to the browser in beautiful JSON objects. But they're sitting behind Cloudflare and aggressively blocking attempts to fetch data programmatically, which is a huge problem for us with 6000+ players worth of data to fetch multiple times every 3 months. So... I built a Chrome Extension to grab the data at a speed that is usually under their detection rate. Basically created a distributed scraper and passed it out to as many people in the league as I could. For big jobs when we want to do giant batches, it was a simple matter of doing the pulls and when we start getting 429 errors (rate limit blocking code they use), switch to a new IP on the VPN. The only way they can block us now is if they stop having a website. Give one of the commercial VPN providers a try. They're usually pretty cheap and have tons of IPs all over the place. Adding a "VPN Disconnect / Reconnect" step to the process only added about 10 seconds per request every so often.

johnnyworker 3y ago

A bit of a tangent, but a long time ago I was kicked out of a Facebook group for what I considered to be completely made up reasons -- and what really got to me was that by being banned from it, I couldn't even point to the posts that had been actively misunderstood and distorted. I couldn't find anything in the cache files, so I saved a process dump of the still running Firefox and stitched the posts together from that. I stopped caring the as soon as I had my proof, but was still sheepishly proud that I managed to get it.

Alifatisk 3y ago
That's very clever
wmichelin 3y ago
When you figured it out, what was the cause?
- johnnyworker 3y ago
  It's been over 10 years, and not interesting, but because you asked: it was a political group, and I was replying to a group member who was talking about how love is all you need. I tried to make the point that love may be an essential ingredient, but is not all you need, certainly not for political change (I think I used some Chomsky quotes to that effect...), and that group member essentially straw manned that into advocating hate (which I really wasn't). Knowing how I was back then, I probably was being a Dwight Schrute about it, as in "bzzzt, false, love is NOT all you need, because A, B and C". So we were both irritated and talked back back and forth, ceding no ground, and suddenly an admin banned me without warning. It was the suddenness more than anything that got to me; I felt misrepresented and censored, and I just wanted to be able to re-read it all to gain closure, if you will. If I could find the file today I would probably cringe at what I wrote, or at least how I wrote it, but back then I nodded and thought "yup, I'm correct", haha.
  - wmichelin 3y ago
    Totally feel that, I got a life-time ban from a huge programming sub on Reddit for being rude to a person who was obviously just copying-and-pasting their software homework on the forum.

tantalor 3y ago

> These properties were chosen manually by working backward from the visible data on the web page to the data structure containing said data (I'll dive into that process in another blog post) That would seem to be the actually interesting/challenging part.

jackbeck 3y ago
I would assume that you could just search the heap by the value shown on the page to find out what the key is.
ricklamers 3y ago
Using introspection tools like queryObjects in Chrome Dev Tools should get you quite far: https://developer.chrome.com/docs/devtools/console/utilities/#queryObjects-function https://developer.chrome.com/docs/devtools/console/utilities...

_boffin_ 3y ago

Anyone know of any research on generating HTML differentials against updated webpages and with automatic healing of wrappers / selectors or research on using LLMs with webscraping and how to reduce token usage while retaining context?

simonw 3y ago
I've been doing some pretty dumb tricks for reducing token usage and piping to LLMs which have worked really well. I have a strip-tags CLI tool which I can pipe HTML through on its way to an LLM, described here: https://simonwillison.net/2023/May/18/cli-tools-for-llms/ https://simonwillison.net/2023/May/18/cli-tools-for-llms/ I also do things like this: shot-scraper javascript news.ycombinator.com 'document.body.innerText' -r \ | llm -s 'General themes, illustrated by emoji' Output here: https://gist.github.com/simonw/3fbfa44f83e12f9451b58b595451414a https://gist.github.com/simonw/3fbfa44f83e12f9451b58b5954514... That's using https://shot-scraper.datasette.io/ https://shot-scraper.datasette.io/ to get just the document.body.innerText as a raw string, then piping that to gpt-3.5-turbo with a system prompt. In terms of retaining context, I added a feature to my strip-tags tool where you can ask it to NOT strip specific tags - e.g.: curl -s https://www.theguardian.com/us | \ strip-tags -m -t h1 -t h2 -t h3 That strips all HTML tags except for h1, h2 and h3 - output here: https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd8871f https://gist.github.com/simonw/fefb92c6aba79f247dd4f8d5ecd88... Full documentation here: https://github.com/simonw/strip-tags/blob/main/README.md https://github.com/simonw/strip-tags/blob/main/README.md
- _boffin_ 3y ago
  First, love your work. Sadly, I don't think this path would work for me as for what I do, I need the selectors as part of the workflow. Roughly, my end goal is to do a single or multi-shot with the following information HTML differential (could be selectors, xpaths, data regions, differentials of any of the above, etc...), code stacktrace, related code, and prompt. For this example, let's consider that the flow involves the bot to login to a website. I have selectors for the `.username` and `.password` inputs and then a selector for the login button as `.login-btn`. 1. The site updates their page and changes up all their IDs, but keeps the same structure. 2. The site updates their page and changes up all their IDs, but changes the structure and the form is named something different and is somewhere else in the DOM. 3. many... many other examples. Trying to figure out how to minimize the tokens, but keep the needed context to regenerate the selectors that are needed to maintain the workflow.

topherjaynes 3y ago

I'm not a legal scholar and this isn't my area of expertise, but the final note has links out to a TechCrunch article about LinkedIn vs hiQ Labs Inc, which alludes to web scraping being legal, but the case wasn't decided for a few more months, and the court sided with Linkedin. What's the final verdict on web scraping vs creating fake accounts to get user information (which the case focused on)

jawerty 3y ago

This is very cool. I do a lot of puppeteer scraping and this library would help with a lot of the more complicated DOMs to work with. I have a live coding stream I did the other day scraping Facebook for comments https://www.youtube.com/live/03oTYPm12y8?feature=share https://www.youtube.com/live/03oTYPm12y8?feature=share If you're interested in seeing puppeteer in action I started doing streams last month where I talk through my method. I’ll be posting a lot more since it's been very fun. Overall puppeteer is great because you get to easily inject js scripts in a nice API. Selenium is great too but not as developed of a web scraping interface imo. Also puppeteer is a very optimized headless browser which is a given. What really matters is implementing a VPN proxy and storing your cookies during auth routines which I can get into if you have any questions about that.

solanav 3y ago

Cool! I use selenium to do phishing detection at my company and I use javascript declared variables as a source of data to analyse. It’s specially useful for links that are obfuscated by concatenating two variables into another one.

breatheoften 3y ago

I used a technique like this a few years back in a production product ... We had an integration partner (who we had permission to integrate with) that offered a different api for integration partners than was used for their website but which was horribly broken and regularly gave out the wrong data. The api was broken but the data displayed on their web page was fine so someone on the team wrote a browser automation (using ruby and selenium!) to drive the browser through the series of pages needed to retrieve all the information required. Needless to say, this broke all the time as the page/css changed etc. At some point I got pulled in and ran screaming away from selenium to puppeteer -- and quickly discovered the joy that is scripting the browser via natively supported api's and the chrome debugger protocol. The partners web page happened to be implemented with the apollo graphql client and I came across the puppeteer api for scanning the javascript heap -- I realized that if I could find the apollo client instance in memory (buried as a local variable inside some function closure referenced within the web app) -- I could just use it myself to get the data I needed ... coded it up in an hour or so and it just worked ... super fun and effective way to write a "scraper"! OnDocumentReady -> scan the heap for the needed object -> use it directly to get the data you need

visarga 3y ago
Wondering if we can automate the finding part with a LLM. You just tell if what data you want scraped.
- dontupvoteme 3y ago
  I think the approach would be to classify the property by the values.
paulddraper 3y ago
> scripting the browser via natively supported api's Every modern browser has native support for WebDriver (i.e. Selenium) APIs. The advantage to Puppetteer is that the Chrome Dev Tools API is just a better API.

j0hnyl 3y ago

Does JavaScript state live in the heap too? Can we use this technique to pull objects from memory?

xkcd1963 3y ago

My dear fellows this is joyous news

adriancooney 3y ago

Thanks for posting this again! It's a year later and I still haven't touched the web scraper in production which is great to reflect on. It seems running the Youtube command on the post is still producing the exact same data too. $ npx puppeteer-heap-snapshot query \ --url https://www.youtube.com/watch\?v\=L_o_O7v1ews \ --properties channelId,viewCount,keywords --no-headless

odysseus 3y ago
Did you ever make another blog post about how to choose properties working backward from the visible data on the web page to the data structure containing said data? Searching the heap manually is not working very well. The data I want is in a (very) long list of irrelevant values within a "strings" key. It might have something to do with the data on the page that I want to scrape being rendered by JavaScript.

lcof 3y ago

[dead]

elwell 3y ago

I'm tempted to scrape this blog page so I can read it without my eyes hurting.

elwell 3y ago

It's still a game of cat and mouse. Next step is for the website to store multiple instances of similarly structured data so you scrape the dummy unknowingly.

dontupvoteme 3y ago
Or generate garbage data to poison their scraper.

bonadrag 3y ago

How do we evade the captchas with this method? Asking for a friend...

enson110 3y ago

Thanks for sharing. I think one of the most challenging part of web scraping is dealing with the website's anti-scraping measures, such as it needs to sign in, encountering 403 forbidden error, and reCAPTCHA. Does anyone have more experience in handling that?