11 ms·

Building a Dark Web Crawler in Go

283 points by aadlani 7y ago

zhdc1 7y ago
Crawlers are fun! If you're new to the field and want something that's easy to set up & polite, I strongly recommend Apache Storm Crawler (https://github.com/DigitalPebble/storm-crawler https://github.com/DigitalPebble/storm-crawler).
sbmthakur 7y ago
A well written article with lot of technical details. Well done. However, I'm wondering what would be a good practical purpose of crawling dark web.
- creekorful 7y ago
  Thank you! There's no practical purpose for the crawler. It's more an educational project than anything.
  - warent 7y ago
    Weird, for some reason your comments are being instantly marked as "dead." I think there's some kind of filter that's tripping out for your account since it's new. I vouched for your two comments so hopefully everyone can see them now, but an admin (i.e. dang?) will need to look into this for a longer term solution.
    - creekorful 7y ago
      Thank you sir. Actually my other comments are invisible too. That's weird.
Havoc 7y ago
Sounds like a recipe to score yourself a free FBI visit
- penagwin 7y ago
  Generally the FBI doesn't give a hoot until you start distributing illegal stuff....
  - fishtacos 7y ago
    What does suck is being put on IP blacklists by various providers for merely running a Tor relay, not an exit node. There are several websites I can only access through VPN because of my IP is associated with running a relay.
bureaucrat 7y ago
First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.
- buildbuildbuild 7y ago
  A polite suggestion, but this is not currently possible. The Tor Project recently added a consensus flag which can globally disable single hop client connections as a DDoS mitigation approach. It is currently enabled. (DoSRefuseSingleHopClientRendezvous)
  - tgragnato 7y ago
    If I were to judge by the statistics of the heartbeat messages I have access to.. The number of people trying to create one-hop circuits is huge.
- malux85 7y ago
  > First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?
  - creekorful 7y ago
    Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety
    - sandworm101 7y ago
      It's now just "Tor" and when accessing hidden services it isnt really a proxy. The Onion Router acronym went away back with the vidalia proxy. But i do miss the oldschool torbutton. It was fun.
rolltiide 7y ago
I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope
- Phenomenit 7y ago
  I thought dread was dead?
  - rolltiide 7y ago
    Its not, hit up dark fail for the onion link to dark fail and browse the latest onion links
  - CryptoBanker 7y ago
    So is bitcoin they tell me...
fs111 7y ago
Any http-aware software that supports socks proxies can access information on hidden services, so any crawler can do it. I fail to see what is novel about that, except that it uses k8s and mongo and a catchy blog title.
jmnicolas 7y ago
I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.
- creekorful 7y ago
  You are right. That's why it's an educational project and not a public search engine
  - mellosouls 7y ago
    IANAL but "educational project" won't fly in court, and nor should it.
    - creekorful 7y ago
      I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code. Why should I face legal problems?
      - rndgermandude 7y ago
        Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?
        mhluongo 7y ago
        Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children
        deleted 7y ago
        [deleted]
Hitton 7y ago
Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just absolute ones.
- creekorful 7y ago
  Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!
  - bluesign 7y ago
    I think 75000 comment is coming from stats[1]. [1] https://metrics.torproject.org/hidserv-dir-onions-seen.html https://metrics.torproject.org/hidserv-dir-onions-seen.html
  - deleted 7y ago
    [deleted]
  - akklesed 7y ago
    Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course). [1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...
mschuster91 7y ago
To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.
- creekorful 7y ago
  Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?
  - weatherlight 7y ago
    yes.
goatsi 7y ago
How well does it handle a gzip bomb? https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/762-At...
getpolarized 7y ago
Go is a horrible language in which to write a crawler. The main problem is that NLP and machine learning code simply isn't as prevalent and robust as it is in Java and Python.
- marcrosoft 7y ago
  Go is great for a crawler. What does NLP and ML have to do with crawling?
seisvelas 7y ago
I did the same in Racket when I made a Tor search engine. Here's the source code of the crawler! https://github.com/torgle/torgle/blob/master/backend/torgle.rkt https://github.com/torgle/torgle/blob/master/backend/torgle....
woodandsteel 7y ago
So how well would this thing work? What I am asking is what percentage of all the tor hidden service sites out there would get detected by it?

283 points by aadlani 7y ago

Crawlers are fun! If you're new to the field and want something that's easy to set up & polite, I strongly recommend Apache Storm Crawler (https://github.com/DigitalPebble/storm-crawler https://github.com/DigitalPebble/storm-crawler).

sbmthakur 7y ago

A well written article with lot of technical details. Well done. However, I'm wondering what would be a good practical purpose of crawling dark web.

creekorful 7y ago
Thank you! There's no practical purpose for the crawler. It's more an educational project than anything.
- warent 7y ago
  Weird, for some reason your comments are being instantly marked as "dead." I think there's some kind of filter that's tripping out for your account since it's new. I vouched for your two comments so hopefully everyone can see them now, but an admin (i.e. dang?) will need to look into this for a longer term solution.
  - creekorful 7y ago
    Thank you sir. Actually my other comments are invisible too. That's weird.

Havoc 7y ago

Sounds like a recipe to score yourself a free FBI visit

penagwin 7y ago
Generally the FBI doesn't give a hoot until you start distributing illegal stuff....
- fishtacos 7y ago
  What does suck is being put on IP blacklists by various providers for merely running a Tor relay, not an exit node. There are several websites I can only access through VPN because of my IP is associated with running a relay.

bureaucrat 7y ago

First of all, it’s hidden sevices, not dark web. Second, to anyone crawling hidden services or crawling over tor, please run a relay or decrease your hop. Don’t sacrifice other’s desperate need for anonymity for your $whatever_purpose_thats_probably_not_important. It could be some fun thing to do for you, but some people are relying on tor to use the free, secure and anonymous Internet.

buildbuildbuild 7y ago
A polite suggestion, but this is not currently possible. The Tor Project recently added a consensus flag which can globally disable single hop client connections as a DDoS mitigation approach. It is currently enabled. (DoSRefuseSingleHopClientRendezvous)
- tgragnato 7y ago
  If I were to judge by the statistics of the heartbeat messages I have access to.. The number of people trying to create one-hop circuits is huge.
malux85 7y ago
> First of all, it’s hidden sevices, not dark web For the uninitiated, can you please explain the differences in what they are and how they're accessed?
- creekorful 7y ago
  Author here. The differences are explained in the post. The dark web is a vast groups of services that cannot be accessed without using special software / proxy. The hidden services are service running on the TOR network and accessed using a browser that use the TOR proxy. They are a type of dark web services, but not the entirety
  - sandworm101 7y ago
    It's now just "Tor" and when accessing hidden services it isnt really a proxy. The Onion Router acronym went away back with the vidalia proxy. But i do miss the oldschool torbutton. It was fun.

rolltiide 7y ago

I’ve been pretty surprised at how big hidden services have become Dread, the dark net reddit, is surprisingly vibrant I think its weird that people almost don't want to hear positive stories about dark net. It’ll be funny when news articles and romcoms just start “forgetting” to qualify their plot piece with the “its scary” trope

Phenomenit 7y ago
I thought dread was dead?
- rolltiide 7y ago
  Its not, hit up dark fail for the onion link to dark fail and browse the latest onion links
- CryptoBanker 7y ago
  So is bitcoin they tell me...

fs111 7y ago

Any http-aware software that supports socks proxies can access information on hidden services, so any crawler can do it. I fail to see what is novel about that, except that it uses k8s and mongo and a catchy blog title.

jmnicolas 7y ago

I'd be concerned that the DB is going to contain some pretty nasty stuff that might be hard to explain in front of a judge.

creekorful 7y ago
You are right. That's why it's an educational project and not a public search engine
- mellosouls 7y ago
  IANAL but "educational project" won't fly in court, and nor should it.
  - creekorful 7y ago
    I don't host a Trandoshan instance neither give access to a database of results. I Only provide access to the source code. Why should I face legal problems?
    - rndgermandude 7y ago
      Because some eager police detective or DA might read your article, raid you and find your personal instance/DB full of nasty stuff. Some of the nasty stuff will not only be illegal to distribute, but actually illegal to possess at all. Child abuse stuff for example. I am guessing you have some personal instance you use at least for testing/"education", right?
      - mhluongo 7y ago
        Please stop the FUD, or point to an example of a software dev getting contacted about their software being used by a third party to exploit children
        deleted 7y ago
        [deleted]

Hitton 7y ago

Disclaimer: I have rather small experience with Golang and just skimmed the crawler code. From what I could see, author made effort to make the crawler distributed with k8s (which I don't is needed considering there are only approximately 75 000 onion addresses) using modern buzzword technology, but from what I could see the crawler itself is rather simplistic. It doesn't even seem to index/crawl relative urls, just absolute ones.

creekorful 7y ago
Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!
- bluesign 7y ago
  I think 75000 comment is coming from stats[1]. [1] https://metrics.torproject.org/hidserv-dir-onions-seen.html https://metrics.torproject.org/hidserv-dir-onions-seen.html
- deleted 7y ago
  [deleted]
- akklesed 7y ago
  Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course). [1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...

mschuster91 7y ago

To anyone experimenting with such stuff, take care and don't make your services publically available. Especially the dark web is full with highly illegal content such as child pornography and in some jurisdictions even "involuntary possession" such as in browser caches may be enough to convict you.

creekorful 7y ago
Do you think I should add a license in Github to mention that? To protect me and the users who will use the crawler?
- weatherlight 7y ago
  yes.

goatsi 7y ago

How well does it handle a gzip bomb? https://www.hackerfactor.com/blog/index.php?/archives/762-Attacked-Over-Tor.html https://www.hackerfactor.com/blog/index.php?/archives/762-At...

getpolarized 7y ago

Go is a horrible language in which to write a crawler. The main problem is that NLP and machine learning code simply isn't as prevalent and robust as it is in Java and Python.

marcrosoft 7y ago
Go is great for a crawler. What does NLP and ML have to do with crawling?

seisvelas 7y ago

I did the same in Racket when I made a Tor search engine. Here's the source code of the crawler! https://github.com/torgle/torgle/blob/master/backend/torgle.rkt https://github.com/torgle/torgle/blob/master/backend/torgle....

woodandsteel 7y ago

So how well would this thing work? What I am asking is what percentage of all the tor hidden service sites out there would get detected by it?