4 ms·

I’m curious of why did you go on the path of using chrome debugging functionality instead of implementing an HTTP proxy which would provides the benefit of behi

by Eikon 7y ago

I’m curious of why did you go on the path of using chrome debugging functionality instead of implementing an HTTP proxy which would provides the benefit of behind browser-agnostic too. May you expand on that please?

gear54rus 7y ago
One thing I can imagine is the fact that chrome is using system proxy settings. That means that changing them will affect all other apps on the machine and could save a lot of garbage requests you don't actually need. On the other hand user agent filtering could be implemented to partially solve this.
- jeroenhd 7y ago
  Chrome can accept proxy settings from the command line do all you'd need to do would be to change the shortcut to Chrome. The more dangerous part would be doing TLS MitM to allow HTTPS content to be recorded. This is well documented but still a potential security issue if the certificate somehow gets picked up by malware or something.
stevekemp 7y ago
After reading this I was wondering if it might be fun to write a HTTP-proxy that a) recorded everything, in an SQLite database, and b) presented a localhost-server which would let you search that content. I suspect it would get very very very busy, with tracking-pixels, etc, but if you only made it archive text/plain, text/html, and similar content-types it might be a decent alternative to bookmarks, albeit only on a single host/network. Wouldn't be hard to knock up a proof of concept, perhaps I should do that this evening.
- prox 7y ago
  Could probably strip the tracking codes anyway. Someone linked to wwwoffle, check if that isn’t what you had in mind.
- randrandrand 7y ago
  I did something like this a while ago: https://github.com/nspin/spiderman https://github.com/nspin/spiderman I used the wonderful tool mitmproxy for both recording and serving.
e12e 7y ago
Also wonder about the proxy thing. Sounds like squid in offline mode would be broadly similar? https://www.oreilly.com/library/view/squid-the-definitive/0596001622/re169.html https://www.oreilly.com/library/view/squid-the-definitive/05... You'd have to mitm ssl unfortunately: https://wiki.squid-cache.org/Features/MimicSslServerCert https://wiki.squid-cache.org/Features/MimicSslServerCert As for proxy switching, in addition to command line options, there's https://chrome.google.com/webstore/detail/proxy-switcher-and-manage/onnfghpihccifgojkpnnncpagjcdbjod https://chrome.google.com/webstore/detail/proxy-switcher-and... Very convenient for using ssh as a socks5 proxy, for example.

by Eikon 7y ago

gear54rus 7y ago

One thing I can imagine is the fact that chrome is using system proxy settings. That means that changing them will affect all other apps on the machine and could save a lot of garbage requests you don't actually need. On the other hand user agent filtering could be implemented to partially solve this.

jeroenhd 7y ago
Chrome can accept proxy settings from the command line do all you'd need to do would be to change the shortcut to Chrome. The more dangerous part would be doing TLS MitM to allow HTTPS content to be recorded. This is well documented but still a potential security issue if the certificate somehow gets picked up by malware or something.

stevekemp 7y ago

After reading this I was wondering if it might be fun to write a HTTP-proxy that a) recorded everything, in an SQLite database, and b) presented a localhost-server which would let you search that content. I suspect it would get very very very busy, with tracking-pixels, etc, but if you only made it archive text/plain, text/html, and similar content-types it might be a decent alternative to bookmarks, albeit only on a single host/network. Wouldn't be hard to knock up a proof of concept, perhaps I should do that this evening.

prox 7y ago
Could probably strip the tracking codes anyway. Someone linked to wwwoffle, check if that isn’t what you had in mind.
randrandrand 7y ago
I did something like this a while ago: https://github.com/nspin/spiderman https://github.com/nspin/spiderman I used the wonderful tool mitmproxy for both recording and serving.

e12e 7y ago

Also wonder about the proxy thing. Sounds like squid in offline mode would be broadly similar? https://www.oreilly.com/library/view/squid-the-definitive/0596001622/re169.html https://www.oreilly.com/library/view/squid-the-definitive/05... You'd have to mitm ssl unfortunately: https://wiki.squid-cache.org/Features/MimicSslServerCert https://wiki.squid-cache.org/Features/MimicSslServerCert As for proxy switching, in addition to command line options, there's https://chrome.google.com/webstore/detail/proxy-switcher-and-manage/onnfghpihccifgojkpnnncpagjcdbjod https://chrome.google.com/webstore/detail/proxy-switcher-and... Very convenient for using ssh as a socks5 proxy, for example.