heipei

45 ms·

421.
by heipei 9y ago
Now if only Firefox also implemented the Chrome Remote Debugging Protocol... https://chromedevtools.github.io/debugger-protocol-viewer/to...
422.
by heipei 9y ago
Some services have actually gone down that exact road. pastebin.com is one of them. But in other cases you simply don't want anyone to be able to extract your info automatically. A good example would be e-commerce sites which don'
423.
by heipei 9y ago
It depends on the target audience. For Google (and for most people) the goal of Headless Chrome is to offer an easy and feature-complete way of automatically testing websites, e.g. for performance (PWA are all the craze) and bugs. For those
424.
by heipei 9y ago
TOR is an easy one to detect. Case in point: Try to browse the web with TOR and see how many CloudFlare captchas you have to solve in the first 10 minutes ;) Also TOR has public lists of exit nodes ( https://torstatus.blutmagie.de
425.
by heipei 9y ago
That's like the people pushing Github PRs which will mine $coin in the CI process. But seriously, you can do that, but scrapers will have short timeouts anyway before they abandon the page or consider it loaded, so there's probabl
426.
by heipei 9y ago
True. Then again, the cost for the scraper can be raised significantly by changing your obfuscation / anti-scraping methods frequently. All of the sudden a scraper will need close monitoring to ensure his scripts / regexes are sti
427.
by heipei 9y ago
Like buses and trams put a limit on horizontal movement? ;) Honestly though, as long as the change of direction is somehow indicated in advance, and the acceleration isn't too abrupt, there shouldn't be anything limiting the horiz
428.
by heipei 9y ago
And it's actually getting easier with every new shiny web API. Want to make sure only the latest Chrome can retrieve the content of your website? Why not run a Webassembly computation that will yield the correct URL to fetch. Or what a
429.
by heipei 9y ago
There are multiple ways to hide your IP for free: Public proxies, VPNs which are not so well-known, and lastly there is a wealth of browser extensions which aim at hiding your IP to some extent. This would be one example where your traffic
430.
by heipei 9y ago
If you know how to interface with Chrome headless then you're 95% there already. Just run xvfb, start a Chrome instance on the DISPLAY with the appropriate flags (for exposing the debugging port) and you're done.
431.
How we pay people at Basecamp (DHH)(m.signalvnoise.com)
4 points by heipei 9y ago|1 comments
432.
by heipei 9y ago
My original motivation was: "This website is slow as shit, let me fire up Chrome Devtools to see how many requests it is making." This doesn't scale and doesn't contain additional info such as GeoIP information, so I cre
433.
by heipei 9y ago
If you want to see the percentage of requests from a website that are IPv6, my service https://urlscan.io supports both IPv4 and IPv6. A lot of similar services only scan via IPv4, which I believe is an oversight.
434.
by heipei 9y ago
Range will suffer greatly, but in terms of top speed and acceleration a Tesla does not have to hide. This is one of the better videos of a Tesla accelerating, as it does not show the passengers but just the road / speedometer. The driv
435.
by heipei 9y ago
That's the same motivation that started me to build https://urlscan.io . I wanted an easy way for everyone to visualise the amount, size and destinations of the various HTTP requests that a single page-load triggers. Inciden
436.
by heipei 9y ago
There are many ways to quickly get and use a GeoIP DB like MaxMind, most requiring way less than 20 minutes. This is just one example of something I started using recently: docker run -p 8080:8080 -d fiorix/freegeoip curl localhost:808
437.
by heipei 9y ago
As a Linux user, I mostly go for 'curl ifconfig.co' ;)
438.
by heipei 9y ago
That's not everything: This website contacted 56 IPs in 6 countries across 47 domains to perform 100 HTTP transactions. In total, 3 MB of data was transfered, which is 5 MB uncompressed. It took 4.103 seconds to load this page. 37 cook
439.
by heipei 9y ago
These numbers are interesting, but could non-US folks please stop crying about the crazy high absolute numbers before doing some basic math and adjusting for other metrics besides total salary? I always use Numbeo ( https://www.nu
440.
by heipei 9y ago
I'm not surprised, it's probably the same reason my Roomba keeps bumping into black objects at full speed yet manages to slow down before touching bright objects / walls.
441.
by heipei 9y ago
I haven't looked at Headless closely enough yet, but the biggest pros I can see are: - Potentially less overhead (system resources) - Much simpler setup (compared to something like Xvfb) - Better support for actual automation tasks, e.
442.
by heipei 10y ago
Right off the bat: No. The reason for that is that crawling using a proper browser (i.e. Chrome) is a lot more resource-intensive than using a dedicated tool which only gets the top resource and maybe tries to parse some additional resource
443.
by heipei 10y ago
I'm using more than one Chrome process so I can kill the processes every so often (e.g. after timeout or when they get stuck). Inside each Chrome instance I use 16 tabs, there might be a number of factors at play: - Are you worried abo
444.
by heipei 10y ago
Thanks for that information. I've been putting off moving over to Chrome headless for https://urlscan.io and haven't had the time to do extensive testing yet. Right now Xvfb works fine for me. Still, I'm running 3
445.
by heipei 10y ago
Not just that, also rather large fetch (AJAX) requests to get all of the JavaScript, or so it seems: https://urlscan.io/result/36f1ada1-a7d1-4b4f-8e65-8269135e10... (gotta be a new record for initial page load size)
446.
by heipei 10y ago
Yes, Chrome is the way to go in my opinion (or in general any browser with a proper DevTools API). Zero setup (start the browser, use the API), zero feature-lag, zero deviation from regular user behaviour, all the security features of the r
447.
by heipei 10y ago
Yes, noise and battery runtime are the only two things keeping me with a MacBook. I'd be willing to compromise on other things (resolution, CPU, aesthetics), but there is still no alternative. But currently still happy with my 2015 MBP
448.
by heipei 10y ago
Good config, but if you're not defining the proxy in an upstream {} block, you can't make use of the keepalive parameter, which keeps a number of connections to the backend alive at any time, reducing the RTT for an actual request
449.
by heipei 10y ago
This is just my opinion, but I can see PhantomJS lose a lot of its appeal when the --headless switch for Google Chrome hits the stable channel (quite a way off still, sure). This was already discussed to some extent here: https:/&#x2F
450.
by heipei 10y ago
I'm using Google Chrome in an instrumented fashion on https://urlscan.io/ and I've always thought that using Chrome that way would make a pretty good crawler. Maybe this is me being naive, but it's certainly

More ›