4 ms·

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in

Author here. I'm fairly new to Golang too and it's my first project. Regarding the number of onion addresses available you are wrong. Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. Not taken but available. I agree with the fact that the crawler is really simplistic. But the project is new (2 months I think) and has to evolve. You can make a PR If you want to help me to improve it!

bluesign 7y ago
I think 75000 comment is coming from stats[1]. [1] https://metrics.torproject.org/hidserv-dir-onions-seen.html https://metrics.torproject.org/hidserv-dir-onions-seen.html
deleted 7y ago
[deleted]
akklesed 7y ago
Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course). [1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...
- kodablah 7y ago
  > IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course) V3 addresses are just ed25519 pub keys and a couple byte changes. You can use Go libraries like Bine [0] to generate as many V3 (or V2) addresses as you want from keys. 0 - https://godoc.org/github.com/cretz/bine/torutil#OnionServiceIDFromV3PublicKey https://godoc.org/github.com/cretz/bine/torutil#OnionService...
ValleZ 7y ago
I assure you that there are less than 10k unique onion addresses. This is a huge overkill to have a distributed system to crawl something this small. edit: onion services, not addresses
jerf 7y ago
"Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available." As a defense against the parent comment, though, this proves way too much. It doesn't matter how much k8s you throw at that, you're never going to so much as find your first site, if you're looking at the problem that way. That's not really a relevant number here.

bluesign 7y ago

I think 75000 comment is coming from stats[1]. [1] https://metrics.torproject.org/hidserv-dir-onions-seen.html https://metrics.torproject.org/hidserv-dir-onions-seen.html

deleted 7y ago

[deleted]

akklesed 7y ago

Offtopic nitpick: >Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available. I sorta understand what you mean, technically it's 32 characters per position (5 bits), and 16 positions. In v2 .onion addresses, that is. v3 ones [1] are 56 positions, but not all the bits are used for addressing, so the same formula wouldn't quite work to calculate real theoretical capacity. IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course). [1] https://trac.torproject.org/projects/tor/wiki/doc/NextGenOnions https://trac.torproject.org/projects/tor/wiki/doc/NextGenOni...

kodablah 7y ago
> IIRC someone already made site which generates unlimited links to v3 addresses (without having them lead to anywhere, of course) V3 addresses are just ed25519 pub keys and a couple byte changes. You can use Go libraries like Bine [0] to generate as many V3 (or V2) addresses as you want from keys. 0 - https://godoc.org/github.com/cretz/bine/torutil#OnionServiceIDFromV3PublicKey https://godoc.org/github.com/cretz/bine/torutil#OnionService...

ValleZ 7y ago

I assure you that there are less than 10k unique onion addresses. This is a huge overkill to have a distributed system to crawl something this small. edit: onion services, not addresses

jerf 7y ago

"Addresses are encoded in Base32 which means there are 32 characters available. So there are 32^16=1.208925819614629174706176×10^24 addresses available." As a defense against the parent comment, though, this proves way too much. It doesn't matter how much k8s you throw at that, you're never going to so much as find your first site, if you're looking at the problem that way. That's not really a relevant number here.