aspyct

44 ms·

481.
by aspyct 7y ago
Hehe, my "fleet of crawlers" is a single machine in a closet so far :) I'll think about that kind of synchronization later. However I do parse and respect the "crawl-delay" now, thanks for pointing it out!
482.
by aspyct 7y ago
Didn't know about that search engine. Thanks a lot! Actually found a few fun datasets, made my day :)
483.
by aspyct 7y ago
I tried the elegant way before, after creating a mobile application to find fuel pumps around the country for a specific brand. My request was greeted with a "don't publish; we're busy making one; we'll sue you anyway&qu
484.
by aspyct 7y ago
I'm a lot more motivated to do data science on topics I actually care about :) Unfortunately those topics (or websites, in this case) don't expose ready-made databases or csv files.
485.
by aspyct 7y ago
That's great advice! Especially the one about response times. I didn't think of that, and will integrate it in my sleep timer :)
486.
by aspyct 7y ago
Yeah that's a must do, but I think most websites don't even bother making a robots.txt beyond "please index us, google". However that wouldn't necessarily mean they're happy about someone vacuuming their whole
487.
by aspyct 7y ago
Cool, didn't know about this. Thanks!
488.
by aspyct 7y ago
Well, having a few websites of my own, I really do think that point 1 is the worst. I can't filter bots that disguise as users from my access logs, and they actually hurt my work (i.e. figuring out what people read). Totally agree with
489.
by aspyct 7y ago
Great recommendations, thanks!
490.
by aspyct 7y ago
That's a good point! So far I'm not planning on publicly disclosing any of my results, but that may come, I guess.
491.
Ask HN: Best practices for ethical web scraping?
270 points by aspyct 7y ago|107 comments
492.
by aspyct 7y ago
Well you sure need a representative dataset to optimize your queries. Often, that means a replica of the production server, with real PII data in there.
493.
by aspyct 7y ago
> writing from his climate controlled room surrounded by concrete for miles in any direction. What does this have to do with genetic engineering? I'm not worried about killer moths, obviously. But as we've seen in the past, era
494.
by aspyct 7y ago
So basically, it's a temporary solution that we will keep reusing until some genes mix and things really go awry. Great.
495.
by aspyct 7y ago
Who gives clearance to that kind of experiment? Oxitec has done the same in the past with mosquitoes, with small to no result. Some say it's even worse now. https://articles.mercola.com/sites/articles/archive&
496.
by aspyct 7y ago
What do you mean, "non issue"? We had timeouts all over the place. It doesn't matter how well you app is designed, if the network is congested, you're gonna have calls from your clients...
497.
by aspyct 7y ago
PST - Potato Standard Time, everybody knows that.
498.
Ask HN: Chrome activating my microphone on OS X
45 points by aspyct 7y ago|15 comments
499.
by aspyct 7y ago
Ha neat! I was actually looking at your product a few hours ago. Might use it in the not-so-far future.
500.
by aspyct 7y ago
Got JavaScript disabled (thanks firefox and noscript), and I have none of that bullshit. Give it a try :)
501.
by aspyct 8y ago
That's absolutely awesome! On a side note, I'd be super wary of having that kind of data in the cloud. A local raspberry pi server would be nice.
502.
by aspyct 8y ago
Note that chroot is not a security feature, it can be compromised in many ways. You probably need containers like lxc/lxd
503.
by aspyct 8y ago
Net neutrality has been dead in many countries for quite some time. In Belgium for example, mobile network is pretty expensive. Except for Facebook (and derivatives), then it's free. Guess who wins?
504.
by aspyct 8y ago
Have you ever run FreeBSD on a laptop? From what I hear it runs nice, but I'm specifically wondering about the battery life. Any experience?
505.
by aspyct 8y ago
My experience is the exact opposite. Osx breaks my dev environment at every update. there's just news on this site saying it dropped support for some nvidia card. In the meantime, I've had the same debian install for 5 years, and
506.
by aspyct 8y ago
Can't tell about original real books, but the fake books definitely are real :) https://en.wikipedia.org/wiki/Fake_book
507.
by aspyct 8y ago
So... the country needs help. How do we help?
508.
by aspyct 9y ago
Compiler errors are a daily issue with Xamarin unfortunately...
509.
by aspyct 9y ago
I find going back to nature every now and then is a nice band-aid on my solitude wound. I'm still alone, mind you, but natured is filled with amazing things you can look at, focus on. Trekking has become a major part of my life in the
510.
by aspyct 10y ago
Well exercise worked for me. Started cycling to work, lost 10kg in 6 months, with no change made to my diet.

More ›