4 ms·

This is great, thanks a lot. It's my understanding that Playwright is the "new Puppeteer" (even with core devs migrating). I presume this sort of technique wou

by BbzzbB 4y ago

This is great, thanks a lot. It's my understanding that Playwright is the "new Puppeteer" (even with core devs migrating). I presume this sort of technique would be feasible on Playwright too? Do you think there's any advantage or disadvantage of using one over the other for this use case, or it's basically the same (or I'm off base and they're not so interchangeable)? I'm basing my personal "scraping toolbox" off Scrapy which I think has decent Playwright integration, hence the question if I try to reproduce this strategy in Playwright.

mdaniel 4y ago
My understanding of Playwright is that it's trying to be the new Selenium, in that it's a programming language orchestrating the WebDriver protocol That means that if you are running against Chromium, this will likely work, but unless Firefox has a similar heapdump function, it is unlikely to work[1]. And almost certainly not Safari, based on my experience. All of that is also qualified by whether Playwright exposes that behavior, or otherwise allows one to "get into the weeds" to invoke the function under the hood 1 = as an update, I checked and Firefox does have a memory snapshot feature, but the file it saved is some kind of binary encoded thing without any obvious strings in it I didn't see any such thing in Safari
- asabla 4y ago
  Well kind of for Firefox, there is this profiling tool which you could use (semi-built in) https://github.com/firefox-devtools/profiler https://github.com/firefox-devtools/profiler. Which let you save a report in json.gz format
- nmstoker 4y ago
  I had understood that Playwright actually used the DevTools protocol rather than the WebDriver protocol, as mentioned here: https://github.com/microsoft/playwright/issues/4862 https://github.com/microsoft/playwright/issues/4862 And there's a bit of detail about how they're different here: https://stackoverflow.com/q/50939116/142780 https://stackoverflow.com/q/50939116/142780 However that's more a detail and doesn't really undermine your point about Firefox / Safari being handled differently, it's just that Playwright implemented their own versions of the protocol for those two non-Chromium based browsers
- nousermane 4y ago
  > Firefox does have a memory snapshot feature, but the file it saved is some kind of binary encoded thing without any obvious strings in it Those .fxsnapshot files are gzipped binary heaps. There is a 3rd-party decoder for it: https://github.com/jimblandy/fxsnapshot https://github.com/jimblandy/fxsnapshot

by BbzzbB 4y ago

mdaniel 4y ago

My understanding of Playwright is that it's trying to be the new Selenium, in that it's a programming language orchestrating the WebDriver protocol That means that if you are running against Chromium, this will likely work, but unless Firefox has a similar heapdump function, it is unlikely to work[1]. And almost certainly not Safari, based on my experience. All of that is also qualified by whether Playwright exposes that behavior, or otherwise allows one to "get into the weeds" to invoke the function under the hood 1 = as an update, I checked and Firefox does have a memory snapshot feature, but the file it saved is some kind of binary encoded thing without any obvious strings in it I didn't see any such thing in Safari

asabla 4y ago
Well kind of for Firefox, there is this profiling tool which you could use (semi-built in) https://github.com/firefox-devtools/profiler https://github.com/firefox-devtools/profiler. Which let you save a report in json.gz format
nmstoker 4y ago
I had understood that Playwright actually used the DevTools protocol rather than the WebDriver protocol, as mentioned here: https://github.com/microsoft/playwright/issues/4862 https://github.com/microsoft/playwright/issues/4862 And there's a bit of detail about how they're different here: https://stackoverflow.com/q/50939116/142780 https://stackoverflow.com/q/50939116/142780 However that's more a detail and doesn't really undermine your point about Firefox / Safari being handled differently, it's just that Playwright implemented their own versions of the protocol for those two non-Chromium based browsers
nousermane 4y ago
> Firefox does have a memory snapshot feature, but the file it saved is some kind of binary encoded thing without any obvious strings in it Those .fxsnapshot files are gzipped binary heaps. There is a 3rd-party decoder for it: https://github.com/jimblandy/fxsnapshot https://github.com/jimblandy/fxsnapshot