11 ms·
Build your own BitTorrent
- sunbum 3y agoWhy in the world does this want access to my github account, with 0 explanation as to why.
- _steady 3y agoIt does say it only asks for your email (read) privileges tbf, but yeah I didn't bother either after that step.
- lucideer 3y agoThis gives the site read access to your private (hidden) Github email address(es) - I think often developers implementing the Github API think this permission is needed to access your public email, & it gets requested unintentionally.
- captn3m0 3y agoNot every account sets a public email, so this ensures that the app gets a reachable email address for every user.
- HeckFeck 3y agoI don't see why they can't present it as a simple list of blog articles with a link to the repository. It achieves the same thing without anyone knowing who I am.
- aleph_minus_one 3y agoWell, because codecrafters.io would love to see you as a paying customer: https://codecrafters.io/pricing https://codecrafters.io/pricing
- sarupbanskota 3y agoCodeCrafters co-founder here. You're right, we could present it as just the articles. Our overview is publicly accessible and doesn't require a paywall. https://app.codecrafters.io/courses/bittorrent/overview https://app.codecrafters.io/courses/bittorrent/overview All the "content" is also available on our GitHub https://github.com/codecrafters-io/build-your-own-bittorrent https://github.com/codecrafters-io/build-your-own-bittorrent If you'd like us to run tests against your code, show you progress, community examples, hints, and so on, then you can do the interactive experience, which requires signing up.
- aleph_minus_one 3y ago> All the "content" is also available on our GitHub https://github.com/codecrafters-io/build-your-own-bittorrent https://github.com/codecrafters-io/build-your-own-bittorrent In this Git repository, I see various stages of the source code, and Docker files. The claim "all the 'content' is also available on our GitHub" clearly does not hold, because at least the article texts are missing there. EDIT: Sorry, it is there, as was pointed out to me in the answer: https://github.com/codecrafters-io/build-your-own-bittorrent/blob/main/course-definition.yml https://github.com/codecrafters-io/build-your-own-bittorrent...
- rohitpaulk 3y agohttps://github.com/codecrafters-io/build-your-own-bittorrent/blob/main/course-definition.yml https://github.com/codecrafters-io/build-your-own-bittorrent...
- sarupbanskota 3y agoThe course definition file contains all the prompts (including nuances per language) that are viewable on the UI. https://github.com/codecrafters-io/build-your-own-bittorrent/blob/main/course-definition.yml https://github.com/codecrafters-io/build-your-own-bittorrent...
- FatalLogic 3y agoLooks like one reason is Codecrafters is a learning site which uses Github to store users code, instead of hosting it themselves. This post is about one project which is at Codecrafters They should explain why though
- sarupbanskota 3y agoBy default, you're not required to publish your code to GitHub (although you can sync with a couple clicks). By you get a repo for you to work out of, which is hosted on CodeCrafters' git servers.
- lucideer 3y agoLooks like they're just leaning on Github as an auth provider without offering any alternatives. Most sites that do this offer a range of options, including normal email signup.
- sarupbanskota 3y agoAlmost all of our target audience has a GitHub account, so using it for auth keeps things simple and is one less thing to manage :)
- yangikan 3y agoyeah, thanks but no thanks.
- Hrundi 3y agoThis links to the product's application, the main website clears things up: https://codecrafters.io/ https://codecrafters.io/
- sarupbanskota 3y agoIt's required for authentication (the original post links to app.codecrafters.io) — codecrafters.io is the marketing page :)
- sangelxyz 3y agoYou upload the challenge to your git, it's then runs a few tests on your implementation.
- dizzydes 3y agoAs the years go on, I find there are fewer and fewer small project ideas that give me butterflies. But this is one! Hadn't heard of CodeCrafters before but I love how academic their challenge ideas are (eg build a DB). I'd love to see a compiler build in there too.
- deleted 3y ago[deleted]
- sarupbanskota 3y agoBuild your own Interpreter is in the works! https://twitter.com/codecraftersio/status/1688503736086540288 https://twitter.com/codecraftersio/status/168850373608654028...
- mr-karan 3y agoJon Gjengset[1] is currently doing a livestream on the same challenge in Rust, on his YT channel[2]. [1]: https://thesquareplanet.com/ https://thesquareplanet.com/ [2]: https://www.youtube.com/watch?v=jSTkEPPiULs https://www.youtube.com/watch?v=jSTkEPPiULs
- Dragging-Syrup 3y agoThank you for pointing this out. I only got to catch the tail end, but it was really cool to watch.
- codetrotter 3y agoYou can re-watch it using the same URL if you want to see it from the beginning.
- qwertox 3y agoThere's also a copy of the livestream on Twitch [0], for those who are blocked on YouTube. [0] https://www.twitch.tv/videos/1954769913 https://www.twitch.tv/videos/1954769913
- sarupbanskota 3y agoWhat a fun live-stream
- Jonhoo 3y agoOh hey, that's me! A better link is https://www.youtube.com/watch?v=jf_ddGnum_4 https://www.youtube.com/watch?v=jf_ddGnum_4 which has chapter marks and has the power outage in the middle spliced away :p
- faitswulff 3y agoHi Jon, thanks for recording! I'm excited to watch this. Do you happen to know why captions aren't enabled on your videos? Oftentimes the issue is that the video's primary language isn't set. Once this is done, youtube will probably caption the rest, though I'm not sure if that's true of videos of all lengths.
- techn00 3y agoReminds me of building a bittorrent client in go: https://blog.jse.li/posts/torrent/ https://blog.jse.li/posts/torrent/
- tomcam 3y agoThanks! That looks like a treasure and I can’t wait to do my own implementation.
- deleted 3y ago[deleted]
- T3RMINATED 3y ago[dead]
- eimrine 3y agoIs this BitTorrent course free (as beer)? I can see a neighbour course about HTTP server "free during beta".
- gnrlst 3y agovery much paid
- shp0ngle 3y agoThey have some free tier and paid tier and I am not sure what is in what. https://codecrafters.io/pricing https://codecrafters.io/pricing
- orng 3y agoLike GP I was also confused and tried looking for a pricing page but failed. Seems like there isn't a link to it from https://app.codecrafters.io/catalog https://app.codecrafters.io/catalog which is the site you go to if you click the big CodeCrafters logo in the top left of the page. There is a "Subscribe" button which takes you to https://app.codecrafters.io/pay https://app.codecrafters.io/pay but I wasn't savvy enough to notice it or realize what it was. Only after starting a course did I begin to suspect that I needed to subscribe since there were a bunch of locks all over. However, it was not clear that the locks actually did anything since I could still click on those links. I guess I would have found out after completing the first step and not being able to progress. This appears to be by design, which strikes me as slightly dishonest.
- sarupbanskota 3y agoThank you for pointing this out. It's not by design. We actually revised our marketing pages recently, and had a regression that got rid of the tooltip that explained what was free. We have a Linear task for fixing it back, but we weren't expecting this HN post today and so it wasn't the top of our list to fix :) I've prioritised it now tho
- sarupbanskota 3y agoThis is fixed btw
- batch12 3y agoYears ago, one of my previous roles was building and supporting a custom linux live OS that could be used by employees on cheap netbooks (remember those?). To distribute updates, I ended up building our own internal torrent server from scratch and used it to distribute image updates. It was a good learning exercise as one of the first times I had built software to conform to a standard and work with out of the box clients.
- sarupbanskota 3y agoFun times
- shp0ngle 3y agoWow this looks like a course I am willing to pay for. And I wanted to learn Rust anyway :)
- sarupbanskota 3y agoAlso good for other languages :)
- the8472 3y agoOddly the uploading part is missing. To be a peer, an equal, things have to flow both ways. Bittorrent wouldn't work otherwise.
- rohitpaulk 3y agoYep, "Seeding files" is an extension: https://app.codecrafters.io/vote/challenge-extension-ideas?course=bittorrent https://app.codecrafters.io/vote/challenge-extension-ideas?c.... Not supported yet, but will be available soon (we add new extensions based on user votes)
- sarp 3y agoHi, course author here! You're right, this was left out initially to scope down the challenge (it's already quite long with 11 stages). We're planning to add it as a challenge extension, uploading is one of the highly voted challenge extension ideas by the community.
- the8472 3y agoIf it were structured in fashion closer along the protocol startup process or how a real client is structured more people might end up with a functional client rather than a "bittorrent leecher". Up to "peer handshake" most things look fine. From there it rushes towards a half-finished goal and if people stop there you're leaving the internet littered with one-sided examples. After the handshake you'll need state machines to handle multiple peers, piece-sets to track what you've downloaded, the rarest-first algorithm[0], message processing and so on. The final result of having a downloaded file falls out of that almost as a side-product at some point. [0] http://bittorrent.org/bittorrentecon.pdf http://bittorrent.org/bittorrentecon.pdf
- gabipurcaru 3y agoThis is super fun! I did the same back in uni, it was an awesome project: https://github.com/bbpcr/Yomato https://github.com/bbpcr/Yomato
- wonderwonder 3y agoWhat a fun site
- ducktective 3y agoDoes anyone know why BitTorrent-based Linux package managers has not become a thing?
- thinkingkong 3y agoMost linux distributions have a rather robust mirroring operation. This is much faster than bittorrent. Lots of cloud vendors provide a mirror endpoint for traffic within their network, and ISPs typically also have them. The network is fast enough for lots of small files to not really justify it.
- Avamander 3y agoThough with webseeds it shouldn't matter that much, not to add redundancy with BT.
- strangesmells02 3y agoI think that hundreds of thousands of peer machines would be way faster than a few centralized repositories... right? It seems you would have essentially unlimited bandwidth.
- tiluha 3y agoFinding peers is somewhat slow. When downloading via torrent it takes a while to really ramp up in speed. With a centralized repository you can start downloading basically right away. For many small files the latency is more important than the bandwidth, given that most repositories are not short on bandwitdth.
- strangesmells02 3y agoThat seems easy to work around. You could write it in such a way that it would have a live cache of peers ready to go plus also the centralized peers would always be available. I personally suspect that having centralized repos is a legacy technology from pre-torrent days when also disk wasnt cheap.
- deleted 3y ago[deleted]
- ikornaselur 3y agoI really like this idea! I wanted to leant Erlang some time ago and a friend wanted to learn Crystal, so we set out to be able to share files between each other with completely custom clients! It was so much fun when we were able to exchange files with the base protocol and some.. Is it call BEP? Enhancements tk the protocol? It's probably my favourite way of learning a new language, as it's simple enough to understand and implement
- sarupbanskota 3y agoThere's Crystal support on CodeCrafters too! You can Build your own Redis.
- tetsuhamu 3y agoIt seems to be failing to create a repository right now
- sarupbanskota 3y agoWe got a massive inflow of traffic from HN today and that seems to have hit some rate limits with GitHub. We're looking into it.
- BigTuna 3y agoI think it's breaking some other things as well [stage-1] Expected "\"orange\"\n" as stdout, got: "Hello World 2!\n"
- rohitpaulk 3y ago(codecrafters dev here) This should only affect C# repositories, and we're working on a fix - C# support is relatively new, and looks like there are some teething issues with the caching mechanism we use to run tests (we aim to keep median response times <3s).
- rohitpaulk 3y ago(codecrafters dev here) Rate limits issue should be fixed now!
- ahoka 3y agoDoes this include the DHT? BT, the protocol itself is not very interesting, it's just a very bad file access protocol over HTTP without DHT, which makes it really P2P.
- remram 3y agoNo, only uses a tracker.
- rohitpaulk 3y ago(codecrafters dev here) It doesn't, but will soon: https://app.codecrafters.io/vote/challenge-extension-ideas?course=bittorrent https://app.codecrafters.io/vote/challenge-extension-ideas?c.... We release a set of "base" stages first, and then work on extensions based on demand. DHT is one of them, magnet links is another that folks have voted for.
- qingcharles 3y agoThat's awesome. Was about to dive into DHT myself.
- ahoka 3y agoGreat!
- wmf 3y agoI don't think this is a fair description of the BitTorrent protocol. HTTP is only used for discovering peers; the data transfer is P2P using a custom protocol. (I'm ignoring optional extensions like HTTP seeding.)
- Dowwie 3y agoCan anyone comment as to how far one can go before paying for the codecrafters service? The crowd that is interested in these kinds of experiences may also like Protohackers, which is completely free.
- sarupbanskota 3y agoThe Build your own HTTP Server challenge is currently completely free. Otherwise, you can do the first 2 stages of any challenge without paying. You can also check out all the prompts and overview for all challenges without a paywall, and you can attempt beyond stage 2 if you have the membership. You might also find tutorials here useful https://github.com/codecrafters-io/build-your-own-x https://github.com/codecrafters-io/build-your-own-x
- clubm8 3y agoI wish more folks distributed Linux ISOs via Bittorrent since it has an integrity check built into the protocol -- messing with PGP is hard and showing me an MD5 sum over a self signed certificate is... just special.
- Bu9818 3y agoInstead of replacing the md5sum on the download page an attacker could replace the infohash/magnet link/.torrent file.
- waynesonfire 3y agomessing with PGP is hard _FOR YOU_. Your inadequacies are not universal.
- throwaway4aday 3y agowhat's with the sign in required, is this a paid tutorial? Here are some free tutorials: JS - https://allenkim67.github.io/programming/2016/05/04/how-to-make-your-own-bittorrent-client.html https://allenkim67.github.io/programming/2016/05/04/how-to-m... GO - https://blog.jse.li/posts/torrent/ https://blog.jse.li/posts/torrent/ Python - https://markuseliasson.se/article/bittorrent-in-python/ https://markuseliasson.se/article/bittorrent-in-python/
- sarupbanskota 3y agoYou might also find tutorials here useful https://github.com/codecrafters-io/build-your-own-x https://github.com/codecrafters-io/build-your-own-x
- lakomen 3y agoThanks
- forgotpwd16 3y agoIt isn't a tutorial/course but a guided step-by-step project with server-side tests for every step.
- lakomen 3y agoViewing on mobile. Not sure what I see. There is no content, nothing clickable. Just decode bencoded strings and integers and some comments. What's the point?
- sarupbanskota 3y agoYeah the mobile experience could use more work, but tbh it's intended for desktop use (since it requires working with a terminal and code editor)
- bogomipz 3y agoI am curious has anyone subscribed to codecrafters.io and if so what was your impression of the service?
- nindalf 3y agoI subscribed for a few months last year and enjoyed it. I liked it enough that I’m subscribing for a life time now. There’s a few different learning styles out there, but personally for me, I learn best while doing projects. That’s why this service hits the sweet spot for me. If you’re like me in this way, then I’d recommend it.
- zeeshanmh215 3y agoI am life time subscriber, as someone who likes to learn by doing, this platform was perfect to me. I didn't ever think i will look into internals but this one seduced me to dive deep in a fun way.
- sarp 3y agoHi! Sarp here, author of the Build your own BitTorrent challenge on Codecrafters. Some back story: After being laid off from my FAANG job, I found myself very unmotivated to go back. I started looking for interesting programming projects to revive my interest in coding. While nomading, I discovered Codecrafters on Nomadlist and really liked the push code to git and pass different stages interaction. The gamification helped me focus and projects allowed me to go deeper on software I used (SQLite, Git, Redis etc.). I even picked up a new language (Go) to do the challenges with. After completing all the challenges on the site, I ran out of things to do. This is when I decided to build a BitTorrent client which was one of the highly voted ideas on the site. I learned many new things by building a BitTorrent client: the BitTorrent protocol, how torrent files are structured, encoding issues, pipelining network requests, url encoding binary values, using Channels in Go etc. I’d love any feedback on the challenge. Also happy to answer any questions!
- sarupbanskota 3y agoAre we gonna get extensions to the challenge?
- blurbleblurble 3y agoBetter yet, extensions to implement BitTorrent extensions? Such as the mutable torrent BEP and others? It'd be great if this spawned hundreds of top quality BitTorrent clients
- sarupbanskota 3y agoBuild your own Extension
- sarp 3y agoTo limit the scope of the challenge, I had to leave out a lot of cool features you see in modern clients: magnet links, UDP trackers, DHT I'm planning to add them over time starting with magnet links which is the highest voted extension idea right now
- 3y ago
- JyB 3y ago> downloading a file from a single peer The hard part is avoided?
- r3trohack3r 3y agoFor folks who speak JavaScript, Feross built WebTorrent which brought the protocol into a browser tab. The code base is delightful to read. A lot of developers are better at writing code than they are reading it. If you're wanting to flex your code-reading muscles, I haven't found many better places to start than the WebTorrent codebase. I put it up there with redis in being fun to read. https://github.com/webtorrent/webtorrent https://github.com/webtorrent/webtorrent
- rohitpaulk 3y agoAnd for Go, there's https://github.com/chihaya/chihaya https://github.com/chihaya/chihaya & https://github.com/cenkalti/rain https://github.com/cenkalti/rain.
- deleted 3y ago[deleted]
- ashcatchum 3y agoHave been exploring this and it is a pretty fun way to learn some of the intrinsics of some products we use daily! I have also had positive interactions with Sarp so hoping this product takes off!
- r3trohack3r 3y agoFor anyone who is interested in peer-to-peer systems like this, and completes Sarp's course, I have an open interview challenge you can submit it to if you find that you want to continue building in this space as a profession: https://gitlab.com/webai-open/network/interview-challenge https://gitlab.com/webai-open/network/interview-challenge Take the guidelines to heart though. We evaluate you on demonstrating understanding of what you did, not that you completed the course. My advice for standing out would be to continue building on it past the end of the course and do something cool yourself.
- mcgingras 3y agoDo y'all really only work M-Th? If so, that's awesome.
- r3trohack3r 3y agoYes. Our salary bands are competitive outside of FAANG (my previous life was at NFLX), we are fully remote, nomad friendly, and work 4 day weeks. You can come on full time or as a contractor (your choice). We give space to learn and do things right. Are comfortable investing in knowledge today to see compounding returns tomorrow. For example, I spent the first 3 weeks of my employment here sitting on my couch reading research papers. That paid dividens, we collapsed a 1.5 year timeline into a 1 month timeline. 3 weeks of reading research papers and 1 week of building got us to a milestone we didn't plan on reaching until a year+ into the project: we trained a model running on a developers laptop in Grand Rapids Michigan against a dataset sitting on another laptop in Yorkshire, with a fully auditable CI/CD log of what data was fed into the model. Another example, the team decided we should do rust to cross-compile to WASM etc. from day one, so we all took 2 weeks to study the Rust Book and learn together. Now we have a subset of our p2p stuff compiling to WASM and running in headless firefox during our integration tests from day one. Pretty flexible in every respect, just need good people who can help build this.
- mcgingras 3y agoHonestly, sounds awesome.
- rubiktubik 3y agoI love the idea of this site but I don’t understand the pricing model. It says 30$/month but I can only pay 120$ for 3 months. It should be 90$ for 3 month. A monthly payment model would be more accessible. I do I misunderstood something?