3 ms·

This uses Chrome DevTools Protocol in a pretty clever way. I used it to archive a highly interactive website and it worked like a charm. The README states: "It

This uses Chrome DevTools Protocol in a pretty clever way. I used it to archive a highly interactive website and it worked like a charm. The README states: "It runs connected to a browser, and so is able to access the full-scope of resources (with, currently, the exception of video, audio and websockets, for now)" I wonder what kind of limitations makes it hard to intercept those resources like the rest of the content.

jmaygarden 6y ago
Video and audio is probably just a matter of not having gotten around to it yet. WebSockets are another matter. I’m not sure what one would do with a two-way channel in a general sense. It’s often not an idempotent operation.
- graderjs 6y ago
  This guy is right I just haven't gotten around to the audio and video it's a little bit more work I think. websockets is not a normal request response thing that I think is easy to cache so it's a little bit more complex and I haven't really begun to think about it in a way that I feel will be a good approach... Or any approach that I think could work.
harlanji 6y ago
I could smash in the audio visual, as my platform is all about archiving and being the origin for a CDN-fronted open offline-first platform with minimal resources that can go into a boat etc. tinydatacenter.com, Github / harlanji / ispooge,tinydatacenter, biz@harlanji.com - need $375/wk.
nexuist 6y ago
Video files are massive, so it may just be the case that archiving videos takes so long they didn't want to support it.
- graderjs 6y ago
  That's also a consideration. And I haven't thought of a way to approach it where we get the nice sort of size of the archives that we have currently, and when adding video it would just balloon and I think that could be surprising so I haven't thought of a good approach to gel all that.
  - nexuist 6y ago
    Not to mention things like e.g. livestreams ala Twitch, all it takes is one long stream at high resolution to take up your entire disk and then you can't archive anything else.
imhoguy 6y ago
Video stream is rarely accessed in one request. There may be prelight request to read format structure and of course any seek operation fragments it even more (Accept-Range header). That makes it hard to assemble reslurce for replay later.

jmaygarden 6y ago

Video and audio is probably just a matter of not having gotten around to it yet. WebSockets are another matter. I’m not sure what one would do with a two-way channel in a general sense. It’s often not an idempotent operation.

graderjs 6y ago
This guy is right I just haven't gotten around to the audio and video it's a little bit more work I think. websockets is not a normal request response thing that I think is easy to cache so it's a little bit more complex and I haven't really begun to think about it in a way that I feel will be a good approach... Or any approach that I think could work.

harlanji 6y ago

I could smash in the audio visual, as my platform is all about archiving and being the origin for a CDN-fronted open offline-first platform with minimal resources that can go into a boat etc. tinydatacenter.com, Github / harlanji / ispooge,tinydatacenter, biz@harlanji.com - need $375/wk.

nexuist 6y ago

Video files are massive, so it may just be the case that archiving videos takes so long they didn't want to support it.

graderjs 6y ago
That's also a consideration. And I haven't thought of a way to approach it where we get the nice sort of size of the archives that we have currently, and when adding video it would just balloon and I think that could be surprising so I haven't thought of a good approach to gel all that.
- nexuist 6y ago
  Not to mention things like e.g. livestreams ala Twitch, all it takes is one long stream at high resolution to take up your entire disk and then you can't archive anything else.

imhoguy 6y ago

Video stream is rarely accessed in one request. There may be prelight request to read format structure and of course any seek operation fragments it even more (Accept-Range header). That makes it hard to assemble reslurce for replay later.