15 ms·
Database-less torrent website
- dfghdfhs 5y agoCould someone more savvy clarify.... On that page I do inspect, got console and do: loadDBAndExec('SELECT * FROM my_table LIMIT 10;') I get Uncaught ReferenceError: loadDBAndExec is not defined Why doesn't this work?
- manigandham 5y agoThat's just example code in the blog post. The live demo uses a webworker with different code: https://boredcaveman.xyz/demo/megacat/database-app.js https://boredcaveman.xyz/demo/megacat/database-app.js
- Hakashiro 5y agoThese projects while technically impressive and interesting, I'm sure wouldn't make much of a difference to copyright holders. I can imagine something like this: - Please take down our IP - I can't because I am not hosting it - But you are displaying it on your website, take your website down or filter what your website displays to avoid showing copyrighted material And that's the end of it. It doesn't matter where the data is stored. It does matter, a lot, where the data is displayed. If your website displays illegal content, you're facilitating access so you're breaching the law. And, honestly, I'm not sure there's an easy solution to that other than using onion or I2P sites which are only marginally more safe against copyright takedown requests because, due to their very nature, they are obscure and difficult to access.
- xialvjun 5y agoSince it has used IPFS, why not host the content of those torrents with IPFS?
- betwixthewires 5y agoYou can do that. And I like the idea. But for now, torrents are the go to method of sharing large pieces of data, and utilizing IPFS to move the centralized aspect of torrents to a decentralized space is a good idea.
- vindarel 5y agorelated: https://torrent-paradise.ml/ https://torrent-paradise.ml/ Also based on IPFS with this HTTP gateway. I find a lot of torrents in there.
- etagate 5y agoLove this website too. But in this case the database and search engine is hosted on the server, it only uses IPFS to distribute a copy of a folder ready out of the box (db included) to be hosted as another instance (a backup).
- mab122 5y agoYou actually should be able to access it using IPFS at/ipns/torrent-paradise.ml or /ipns/12D3KooWB3GY1u6zMLqnf3MJ8zhX3SS1oBj7VXk3xp6sJJiFGZXp but it seems that clicking search throws an JS error :/
- jokoon 5y agoany similar link you know for DHT searches?
- ThalesX 5y agoIf any dev is here, I'm on IPFS and can't seem to search, getting console error: > Uncaught ReferenceError: passQueryToResultpage is not defined
- yonixw 5y agoEnhance it with the fact you can lazy load sqlite query using HTTP Range as demonstrated in: https://news.ycombinator.com/item?id=27016630 https://news.ycombinator.com/item?id=27016630
- Closi 5y agoExactly the thought I had reading this! Combining these projects would be awesome, proper semantic web! Plus it would mean near-zero loading times as the db size scaled (although ultimately some peers would still need the full database).
- lovasoa 5y agoWell, you could reimplement "absurd sql" over ipfs. Possible, but far from trivial.
- samwillis 5y agoAs the SQLite file is being downloaded vie IPFS it wouldn’t be a HTTP Range request. Does IPFS have the equivalent? (My guess would be yes as I believe it breaks up the file for distribution, but have no idea if it’s exposed in the IPFS.js api) EDIT: After a quick scan of the docs, I think you can do this (but I certainly don't know enough). With at least the "The Mutable Files API" which "is a virtual file system on top of IPFS that exposes a Unix like API", you can provide an `offset` and `length` to `ipfs.files.read(path, [options])`. I don't know if that then translates to only downloading that part of the file from IPFS or not. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfsfilesreadpath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI... EDIT2: In fact the `ipfs.cat` api that the OP is using takes `offset` and `length` parameters too. https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FILES.md#ipfscatipfspath-options https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/FI...
- SahAssar 5y agoIt doesn't have to be range requests, you could split the file instead of depending on the range requests. Essentially it's the same as the chunking instructions for hosters who have a maximum file size as is demonstrated here: https://github.com/phiresky/sql.js-httpvfs/blob/master/create_db.sh https://github.com/phiresky/sql.js-httpvfs/blob/master/creat...
- politician 5y agoHow do immutable systems like IPFS manage the problem of updates to a blog?
- yonixw 5y agoYou have "publish ids" which you control using private keys.. like a dns... so you can have your main page address known and all the other posts linked from that page
- Gigachad 5y agoThey have another system called IPNS but unless it’s improved since I checked, it wasn’t very practical.
- lekevicius 5y agoWhat's wrong with IPNS? I'm using it for my decentralized website (via fleek), and it seems to work well.
- nybble41 5y agoIt has a reputation for being slow if you're not using the DNS-based version of the protocol (DNSLink). Though the experimental IPNS-over-PubSub interface helps if both sites are using it.
- tecleandor 5y agoI'm curious about why didn't they compress the SQLite file using gzip. Even in level 1, for superfast compression/decompression halved the file size.
- etagate 5y agoI knew I could do it, I just wanted to make a demo of the whole thing to see if it would have worked in practice. But yes there's space for a lot of improvements like compression and lazy load with sqlite (cited in a comment)
- tecleandor 5y agoGreat! I was honestly wondering because I didn't know if there were any technical limitations/problems, but it's quite interesting nonetheless.
- danka 5y agoI wonder why the author ignored the option of compression in the post. Even with a simple gzip DEFLATE compression, those 10MB of plain text could get as small as a 1MB archive and possibly more, meaning that in a compressed 10MB payload you could fit much much more than 135K records.
- vermilingua 5y agoIt isn't 10MB of plain text though, it's 10MB of binary SQLite database. I agree that compression would be useful here, but I don't think a simple gzip DEFLATE would be. I was curious so I compressed that torrent db with a few different methods: 11.1MB 11116544B dump.sqlite 10.2MB 10155419B dump.csv 6.6MB 6573399B dump.sqlite.gz 6.6MB 6565771B dump.zip 5.6MB 5616842B dump.rar gzip is certainly suitable to be used in this situation, I stand corrected.
- rdubz 5y agoBack of envelope: The 10MB estimated size came from [100 bytes per row] * [100k rows]. 50 of the bytes per row were "description", which should compress well (2-3x, I'd guess). 40 bytes per row were the IPFS ID/hash, IIUC. I assumed this is like a Git hash, 40 hex chars, which is really just 20 bytes of entropy. He also estimated 14 bytes for the size (stored as a string representation of a decimal integer, up to 1e15 - 1, or 1PB?). That's about 50 bits or 6-7 bytes, as a binary integer. Sizes wouldn't be uniformly distributed though so it would compress to even fewer bytes. So if SQLite was smart (or one gzips the whole db file, like you did), it makes sense that a factor of 2 or so is reclaimable.
- thinkloop 5y agoIPFS requires "pinning" content for it to exist, meaning you host it, whoever pins content becomes liable for its distribution, if no one pins it, it disappears. That's why you can't freely host things like child porn on IPFS.
- etagate 5y agoThis is true even for a torrent btw. And (un?)fortunately you can distribute anything on IPFS as long as it's encrypted.
- Gigachad 5y agoNo one is going to pin encrypted data they do not understand. And IPFS doesn’t get users to seed random files, you only seed stuff you downloaded. So really it’s not much different to hosting it on a personal http server except any downloaders can also host.
- mavhc 5y agoSounds very similar to bittorrent itself
- nybble41 5y ago> No one is going to pin encrypted data they do not understand. Select users who have the decryption key (communicated out-of-band) might be willing to pin the encrypted data for the benefit of others who also have the key. There are also for-pay pinning services who probably wouldn't care whether the data was encrypted. And whether it's pinned it or not, as long as it's downloaded the data remains available to be shared (at least until it's GC'd).
- Gigachad 5y agoOk but you could do that on literally any platform. I could encrypt data and stick it on google drive or aws and the hosts wouldn’t have any clue what it was.
- dfghdfhs 5y agoI love the internet.
- lekevicius 5y agoOne unmentioned con: no updates, no new torrents can be added (or, updates require re-deployment of full new .sqlite db, together with a new website). I think there's a space for decentralized database format. Something that would have immutable rows (not the whole db), ranges and search, indexes, etc. Maybe there's something like this already?
- gerwim 5y agoWell there is Dolt: https://www.dolthub.com/ https://www.dolthub.com/
- mosselman 5y ago> (or, updates require re-deployment of full new .sqlite db, together with a new website) Seems like a small price to pay. I can see new stuff living in torrents that then get committed to a sqlite database that is then deployed every now and then.
- chrisjc 5y ago> Something that would have immutable rows Isn't this just a log (or a stream like Kafka or Kinesis)? In fact you might even be able say every database already has this ;) (binlog, oplog, etc) > not the whole db If rows are immutable, what part of the db is left not being immutable? If rows were immutable, doesn't that imply that any existing "ranges, searches, indexes, etc" are immutable too? If you going to the effort of making a decentralized database, why not also decompose all of these parts from one another... no reason the tables (logs), indexes, search, etc have to live in the same database, they could be spun off as completely different parts. Basically a something that indexes logs and then something else that takes those indexes and makes the searchable. In fact this is all the rage right now with centralized databases... all of the work being done with streaming systems just seems to be an effort of decomposing and inverting databases... every old is new again. Anyways, I agree with the idea and don't really know enough about decentralized systems to really understand why such a distributed database can't or hasn't already been built.
- oscargrouch 5y agoMy current project can expose a sqlite database over torrent wrapped by the Chrome cache filesystem. The torrent is merkle-tree based, and for each change in the SQLite memory page it also updates the merkle tree giving you a different torrent infohash at the end. So as databases are published together with the applications the initial torrent info is the same, so theres a swarm of initial peers. Once it gets updated and the torrent "turns into another", the solution is to use the RPC api (that my solution provides) over the initial swarm of peers and use some sort of distributed algorithm like raft or gossip to define how to deal with the subsequent changes over the peers. The important part is that all other changed torrents from all the peers are available to all peers in the network, and with a RPC working as a abstract interface to organize them, it will work for a lot of different scenarios.
- donkarma 5y agoThat's not going to stop takedowns, they don't care about if it's technically on your site or not.
- jtbayly 5y agoExactly. I forget where I just saw this. It seemed like a particularly absurd example, but the examples are numerous. Torrent sites don’t have the copyrighted content either, but they get shut down all the time.
- roywiggins 5y agoBut the torrents themselves tend to stay seeded... if you stick the site itself on IPFS and anyone in any jurisdiction can "seed" it, then it's going to be hard to kill by court order.
- tablespoon 5y ago> That's not going to stop takedowns, they don't care about if it's technically on your site or not. Yep. "The underlying mistake is thinking about this like a game where you can make up rules for the government to follow." - https://news.ycombinator.com/item?id=29913036 https://news.ycombinator.com/item?id=29913036 Also the assumptions here kind of remind me of domain fronting: https://www.zdnet.com/article/amazons-aws-latest-to-give-up-domain-fronting-putting-censorship-apps-in-peril/ https://www.zdnet.com/article/amazons-aws-latest-to-give-up-.... Basically, it's assuming IPFS will protect it from the authorities, but what might actually happen is it makes IPFS a target of the authorities. Now that probably won't happen with this because the authorities don't actually care that much about torrenting/piracy, but the error is still there.
- Chris2048 5y agoThe "rules for the government to follow" are the laws, and these aren't being made up, but rather pre-exist (safe harbour laws) with some pretty strong backers (goog et al)
- ineedasername 5y agoThe headline doesn't make much sense. There's still a database, what changes is simply where it's stored when in use. "Torrent Website w/o DB Server" might work better. That aside, would this method scale? 135,000 torrents doesn't seem comprehensive, so I would expect real world use to have many more. Maybe a different SQLite db for different categories?
- moolcool 5y agoSince it's search is just based on prefixes, you could probably just alphabetically partition the databases.
- senjin 5y agoMakes me wonder if you could host a pouchdb in ipfs
- SuchAnonMuchWow 5y agoReading just the title, I was expecting storing/serving magnet files on DNS, as someone did a few years back if I remember correctly (just as a joke/poc, not as a serious solution). I'm pretty sure it was posted on hw, but I can't find the link anymore.
- ronyfadel 5y agoI'm disappointed that I don't find result for "La vita e bella" but I do find results for "La vita è bella".
- kderbyma 5y agolove these kind of things. I am interested in distribution methods for smarter clients without needing a dedicated server to host from.
- MuchHustle 5y agoTheres already like 10 crypto projects implementing that. The most prominent is probably Bit Torrent Token (BTT). https://www.bittorrent.com/token/btt/ https://www.bittorrent.com/token/btt/