5 ms·

Researchers Showcase Decentralized AI-Powered Torrent Search Engine

synctext 2y ago
Tribler founding professor here, AMA. repo stuff: https://github.com/Tribler/tribler/wiki#current-items-under-active-development https://github.com/Tribler/tribler/wiki#current-items-under-...
- 19h 2y ago
  I operate several instances of magentico with 150m+ torrents (several years worth). Is there a way I could run Tribler on a server w/o UI?
  - anacrolix 2y ago
    You could try https://www.coveapp.info/ https://www.coveapp.info/. It has a superior DHT indexer built in. It is intended as a end user product tho.
- kozlovsky 2y ago
  Looks interesting. Why it is not popular after these decades?
- sp1ceman 2y ago
  More a general question, what is your strategy for working on topics/technologies that seem to be quite "counter-culture", internet, cyberspace from within an 'established/establishment' academic institution?
  - synctext 2y ago
    Our university is cycling distance from Amsterdam, everything is legal there. But more seriously, it is indeed difficult to work on CyberPunk 2.0 topics within a university. Solid question! As an academic you critically rely on incoming grant money, so my strategy for past decades have been to discover what the real problems are before they become hype. That means you need to need to run your research team as a startup. Very few academic thinkers. We discover what is hard using our deployment-first methodology [1]. We have 2.3 million downloads of our distributed system. Our "Bandwidth as a currency" used a ledger in aug 2007 [2]. That became a big hit in jan 2009 with Bitcoin. Gave us a few years headstart for funding on others. In 2012 we had Internet deployment of fully decentralised machine learning (SGD) [3]. This showed the need for a global web-of-trust. [1] https://arxiv.org/pdf/2301.04508 https://arxiv.org/pdf/2301.04508 [2] http://news.bbc.co.uk/2/hi/technology/6971904.stm http://news.bbc.co.uk/2/hi/technology/6971904.stm [3] http://www.kl.csko.hu/projektek/msc_thesis.pdf http://www.kl.csko.hu/projektek/msc_thesis.pdf
    - inoop 2y ago
      > In 2012 we had Internet deployment of fully decentralised machine learning (SGD) Peer-to-peer training of the next big LLM when? :)
CuriouslyC 2y ago
Interesting stuff. The problem with decentralized search is that it is essentially an adversarial problem. A "friends" list or social graph would go a long way towards providing a trust framework to build decentralized search on, that way trust scores for friends can be propagated out and used to rank results.
- synctext 2y ago
  yes! Spam is real and every stranger could be a spammer. Decentralised trust is the cornerstone of our work. Big topic 25 years ago around the Pagerank patent time. Now we are one of the last remaining academic teams on this topic. We deployed a decentralised trust algorithm based on the interaction graph to 94k people, see [1]. Theoretical foundations are based on proving that the Harvard impossibility result against Sybil attacks made too strict assumptions, [2]. Leading to trust scores with resilience against fake identities: MeritRank [3]. This is being released in first version in this Tribler version. So hopefully this trust framework will help stop spammers a bit. [1] https://research.tudelft.nl/files/89353583/1_s2.0_S1389128621001705_main.pdf https://research.tudelft.nl/files/89353583/1_s2.0_S138912862... [2] https://pure.tudelft.nl/ws/files/96914542/p1263.pdf https://pure.tudelft.nl/ws/files/96914542/p1263.pdf [3] https://arxiv.org/pdf/2207.09950 https://arxiv.org/pdf/2207.09950
wongarsu 2y ago
If you just want the decentralized torrent search there's also Magnetico [1] and Bitmagnet [2]. Tribler tries to be a whole lot more, including its own tor-like network for anonymity and this latest AI powered search to make it easier to find content you might not know by name 1: https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico 2: https://bitmagnet.io/ https://bitmagnet.io/
- synctext 2y ago
  Magnetico is pioneering work. Very simple, very effective. However, with a single 1 Gbps VPS you can take it down. Simply spam at full speed and switch VPN reguarly. Same for Bitmagnet. Usenet, email and now Mastodon show that you need a trust framework. Otherwise you open to abuse and spam. With Tribler we're hoping that after 3 generations of failures/learnings we now have operational decentralised trust. See our public thinking for 11 years on Github [1]. [1] https://github.com/Tribler/tribler/issues/31 https://github.com/Tribler/tribler/issues/31
- anacrolix 2y ago
  Try https://www.coveapp.info/ https://www.coveapp.info/
sktrdie 2y ago
Just read through the article, not sure I get what's the novel thing here. Any insights?
- tinytiger 2y ago
  Torrent tracker websites that provide us with magnet links may be taken down. If there is a search function which can run locally on the computer, there is no vulnerable node in the decentralized network which could be targeted.
  - toomuchtodo 2y ago
    You still need to bootstrap and maintain DHT/torrent corpus state/data in a potentially adversarial environment, correct? Solving for durable storage seems to be a challenge to overcome (unless that itself is served as a mutable torrent? or append only immutable torrents?).
    - tinytiger 2y ago
      The adversarial environment may become friendly if the web of trust idea materializes. Otherwise yeah, the proposed search method would encounter issues.
      - toomuchtodo 2y ago
        Could you perhaps (at a cost) burn magnet links for each torrent into the Ethereum blockchain? This would enable any node to mount and scan the ledger in a way that would be challenging to prevent, considering the financial incentives for the network to persist. Someone else is operating the decentralized, durable storage system for you. It's a hard problem to solve, see SciHub, Libgen, Anna's Archive, etc.
        tinytiger 2y ago
        For sure that could be done. I guess the benefit of having an ML search engine is that it may be able to infer what you're looking for even though you can't exactly name it. But yeah, you're right. Burning magnet links into Ethereum seems like it might work decently, never thought of that.

Tribler founding professor here, AMA. repo stuff: https://github.com/Tribler/tribler/wiki#current-items-under-active-development https://github.com/Tribler/tribler/wiki#current-items-under-...

19h 2y ago
I operate several instances of magentico with 150m+ torrents (several years worth). Is there a way I could run Tribler on a server w/o UI?
- anacrolix 2y ago
  You could try https://www.coveapp.info/ https://www.coveapp.info/. It has a superior DHT indexer built in. It is intended as a end user product tho.
kozlovsky 2y ago
Looks interesting. Why it is not popular after these decades?
sp1ceman 2y ago
More a general question, what is your strategy for working on topics/technologies that seem to be quite "counter-culture", internet, cyberspace from within an 'established/establishment' academic institution?
- synctext 2y ago
  Our university is cycling distance from Amsterdam, everything is legal there. But more seriously, it is indeed difficult to work on CyberPunk 2.0 topics within a university. Solid question! As an academic you critically rely on incoming grant money, so my strategy for past decades have been to discover what the real problems are before they become hype. That means you need to need to run your research team as a startup. Very few academic thinkers. We discover what is hard using our deployment-first methodology [1]. We have 2.3 million downloads of our distributed system. Our "Bandwidth as a currency" used a ledger in aug 2007 [2]. That became a big hit in jan 2009 with Bitcoin. Gave us a few years headstart for funding on others. In 2012 we had Internet deployment of fully decentralised machine learning (SGD) [3]. This showed the need for a global web-of-trust. [1] https://arxiv.org/pdf/2301.04508 https://arxiv.org/pdf/2301.04508 [2] http://news.bbc.co.uk/2/hi/technology/6971904.stm http://news.bbc.co.uk/2/hi/technology/6971904.stm [3] http://www.kl.csko.hu/projektek/msc_thesis.pdf http://www.kl.csko.hu/projektek/msc_thesis.pdf
  - inoop 2y ago
    > In 2012 we had Internet deployment of fully decentralised machine learning (SGD) Peer-to-peer training of the next big LLM when? :)

CuriouslyC 2y ago

Interesting stuff. The problem with decentralized search is that it is essentially an adversarial problem. A "friends" list or social graph would go a long way towards providing a trust framework to build decentralized search on, that way trust scores for friends can be propagated out and used to rank results.

synctext 2y ago
yes! Spam is real and every stranger could be a spammer. Decentralised trust is the cornerstone of our work. Big topic 25 years ago around the Pagerank patent time. Now we are one of the last remaining academic teams on this topic. We deployed a decentralised trust algorithm based on the interaction graph to 94k people, see [1]. Theoretical foundations are based on proving that the Harvard impossibility result against Sybil attacks made too strict assumptions, [2]. Leading to trust scores with resilience against fake identities: MeritRank [3]. This is being released in first version in this Tribler version. So hopefully this trust framework will help stop spammers a bit. [1] https://research.tudelft.nl/files/89353583/1_s2.0_S1389128621001705_main.pdf https://research.tudelft.nl/files/89353583/1_s2.0_S138912862... [2] https://pure.tudelft.nl/ws/files/96914542/p1263.pdf https://pure.tudelft.nl/ws/files/96914542/p1263.pdf [3] https://arxiv.org/pdf/2207.09950 https://arxiv.org/pdf/2207.09950

wongarsu 2y ago

If you just want the decentralized torrent search there's also Magnetico [1] and Bitmagnet [2]. Tribler tries to be a whole lot more, including its own tor-like network for anonymity and this latest AI powered search to make it easier to find content you might not know by name 1: https://github.com/boramalper/magnetico https://github.com/boramalper/magnetico 2: https://bitmagnet.io/ https://bitmagnet.io/

synctext 2y ago
Magnetico is pioneering work. Very simple, very effective. However, with a single 1 Gbps VPS you can take it down. Simply spam at full speed and switch VPN reguarly. Same for Bitmagnet. Usenet, email and now Mastodon show that you need a trust framework. Otherwise you open to abuse and spam. With Tribler we're hoping that after 3 generations of failures/learnings we now have operational decentralised trust. See our public thinking for 11 years on Github [1]. [1] https://github.com/Tribler/tribler/issues/31 https://github.com/Tribler/tribler/issues/31
anacrolix 2y ago
Try https://www.coveapp.info/ https://www.coveapp.info/

sktrdie 2y ago

Just read through the article, not sure I get what's the novel thing here. Any insights?

tinytiger 2y ago
Torrent tracker websites that provide us with magnet links may be taken down. If there is a search function which can run locally on the computer, there is no vulnerable node in the decentralized network which could be targeted.
- toomuchtodo 2y ago
  You still need to bootstrap and maintain DHT/torrent corpus state/data in a potentially adversarial environment, correct? Solving for durable storage seems to be a challenge to overcome (unless that itself is served as a mutable torrent? or append only immutable torrents?).
  - tinytiger 2y ago
    The adversarial environment may become friendly if the web of trust idea materializes. Otherwise yeah, the proposed search method would encounter issues.
    - toomuchtodo 2y ago
      Could you perhaps (at a cost) burn magnet links for each torrent into the Ethereum blockchain? This would enable any node to mount and scan the ledger in a way that would be challenging to prevent, considering the financial incentives for the network to persist. Someone else is operating the decentralized, durable storage system for you. It's a hard problem to solve, see SciHub, Libgen, Anna's Archive, etc.
      - tinytiger 2y ago
        For sure that could be done. I guess the benefit of having an ML search engine is that it may be able to infer what you're looking for even though you can't exactly name it. But yeah, you're right. Burning magnet links into Ethereum seems like it might work decently, never thought of that.