4 ms·
Show HN: I have written a cloud native dark web crawler in Go
- creekorful 6y agoRelease 1.0.0 will be out very soon. Check it out guys!
- lrossi 6y agoCan you share some examples of what it finds? As long as it is safe to post that.
- creekorful 6y agoI have an instance running for about 5 days+ with 1,300,000 docs. It's mostly huge amount of porns, some weird forums & lot of marketplaces. There's still some good finds like a 'sysadmin' forum with huge amount of good posts & tips.
- creekorful 6y agoI'll certainly publish an index of interesting (legal) hostnames on my GitHub account.
- visox 6y agonice, what is actually needed to crawl onion sites ? i am not a go dev, would try it with scala/java
- creekorful 6y agoJust a tor proxy (socks) instance, and a HTTP client configured to use the proxy. Other than that nothing more than what a web crawler should have.
- visox 6y agocool i look into that some time.