12 ms·
BitTorrent vs. HTTP
- inconclusive 10y agoWhy we still use HTTP is beyond me. And I don't mean about the speed issues. Why have a protocol that's so complicated when most of the things we need to build with it are either simpler or reimplement parts of the protocol.
- thrillgore 10y agoBased on my Snort statistics, a small but increasing number of sites and service providers are starting to expose support for the QUIC protocol.
- recrof 10y agoQUIC is not application-layer protocol.
- shakna 10y agoI'm not as familiar with HTTP 2, so I'll only talk about the previous specifications... Which are dead simple to construct, send, receive and parse. Really. For example, let's curl -L (view everything but the body) for the spec: http://www.ietf.org/rfc/rfc7230.txt http://www.ietf.org/rfc/rfc7230.txt HTTP/1.1 200 OK Date: Tue, 24 Jan 2017 12:00:55 GMT Content-Type: text/plain Transfer-Encoding: chunked Connection: keep-alive Set-Cookie: __cfduid=df57c7720b704a40e4c3367bbe248771c1485259254; expires=Wed, 24-Jan-18 12:00:54 GMT; path=/; domain=.ietf.org; HttpOnly Last-Modified: Sat, 07 Jun 2014 00:41:49 GMT ETag: W/"3247b-4fb343e4dcd40-gzip" Vary: Accept-Encoding Strict-Transport-Security: max-age=31536000 X-Frame-Options: SAMEORIGIN X-Xss-Protection: 1; mode=block X-Content-Type-Options: nosniff CF-Cache-Status: EXPIRED Expires: Tue, 24 Jan 2017 16:00:54 GMT Cache-Control: public, max-age=14400 Server: cloudflare-nginx CF-RAY: 326353e6a6a257a7-IAD A bunch of newline, (CRLF), seperated key-value mappings. Some with a DSL (Such as Set-Cookie). It gives you a status message instantly, a date to check against cache, a Content-Type for your parser, acceptable encoding, for your parser, a bunch of other values for your cache. All for free. As for the body of the content? For a gzipped value like this, it's everything outside the header, until EOF. That's not quite as easy as when the content-length parameter is given, but hardly difficult for parsing. HTTP is easy. In fact, HTTP is so easy, that in-complete HTTP servers can still serve up real content, and browsers can still read it. HTTPS is more complicated, but if you simply rely on certificate stores and CAs, it becomes much easier, but HTTPS is a different protocol.
- motoboi 10y ago> As for the body of the content? For a gzipped value like this, it's everything outside the header, until EOF. That's not quite as easy as when the content-length parameter is given, but hardly difficult for parsing. This is chunked and keep- alive. Things get a little trickier
- shakna 10y agoTrue, you keep the connection open, and receive a length of expected bytes, and then said bytes, until 0 is sent. Still simple enough that there are a dozen implementations of less than a page, only a search away.
- ainiriand 10y agoWhy we use HTTP you ask? Because it works. How about that.
- mreithub 10y agoCould you elaborate your issues with HTTP a bit? What kind of protocol would do a better job? Minimal implementations of HTTP (and I'm strictly talking about the transport protocol, not about HTML, JS, ...) is dead simple and relatively easy to implement. Of course there's a ton of extensions (gzip compression, keepalive, chunks, websockets, ...), but if you simply need to 'add HTTP' to one of your projects (and for some reason none of the existing libraries can be used) it shouldn't take too many lines of code until you can serve a simple 'hello world' site. On top of all that, it's dead simple to put any one of the many existing reverse proxies/load balancers in front of your custom HTTP server to add load balancing, authentication, rate limiting (and all of those can be done in a standard way) Furthermore, HTTP has the huge advantage of being readily available on pretty much every piece of hardware that has even the slightest idea of networking. Any new technology would have to fight a steep uphill battle to convince existing users to switch. Have I mentioned that it's standardized and open?
- ue_ 10y agoMy friend once mentioned that FTP would be a good option, I'm not sure why though. I think they regarded HTTP as superfluous for the purpose of what we use the web for.
- protomyth 10y agoAnyone who has to deal with ftp on firewalls would say ftp (and ftps) would do well to disappear from the world.
- mreithub 10y agoIt's not just firewalls. The fact that (unencrypted) FTP is still widely used today when better alternatives like SFTP (via SSH) have existed for years strikes me as odd. (I'm speaking about authenticated connections. For anonymous access - which should be read-only anyway - you're usually better off using HTTP anyway)
- 10y ago
- jaimehrubiks 10y agoBasic http is dead simple, it works, and it also has many addons with backward compatibility (one can still use a basic http client or server in most cases) and even new version fully optimized to nowadays needs (and even in binary form)
- kiliankoe 10y ago> While clients (probably) could ask for chunks in a chronological order and then get the data in a sequential manner [...] Why probably? This is exactly how the immensely popular Popcorn Time worked.
- anonymfus 10y agoMost clients support this in some way. In qBittorrent it's a context menu option, for example.
- masklinn 10y agoLikewise Tixati, on a per-file basis you can ask for a sequential download (prioritise downloading the file in sequential order) and "head and tail" (further prioritise the first and last 500k, as some file formats require a header and a footer to start using).
- geoah 10y agoThe non-random fetching is also how super-seeding works [1]. https://en.wikipedia.org/wiki/Super-seeding https://en.wikipedia.org/wiki/Super-seeding
- fnord123 10y agoIt's literally in the same sentence. In the part that you elided: "it will not at all distribute the load as evenly among the peers as the "random access" method does." If people only stay to download the file and then disconnect then there will be a higher concentration of parts from the beginning of the file than to the end. This reduces the resiliency of the swarm.
- kiliankoe 10y agoI only wanted to bring up that the beginning of the sentence made it sound like a theoretical concept that might work, but no one has ever tried or seen in practice. The concerns are definitely valid.
- na85 10y agoOne thing I always fail to see in these types of comparison articles is that BitTorrent will happily destroy any ability for those sharing your connection to use it if you let it. Probably due to the sorry state of affairs that is the consumer router space, running BitTorrent renders even web browsing painfully slow and sometimes completely nonfunctional. Why BitTorrent does this while normal http transfers do not is not clear to me. Perhaps due to the huge number of connections made. Either way, when given a choice I'll always take a direct HTTP transfer over a torrent, for no other reason than the fact that I'd like to be able to watch cat videos while the download completes.
- anc84 10y agoSimply limit your client's allowed number of connections and limit its allowed bandwidth.
- na85 10y agoOf course that works but why isn't the protocol robust enough to do this automatically?
- j3097736 10y agoBecause bittorrent is designed to be easy to implement to favour adoption, they could have created a more performant DHT and added something like eMule's automatic upload speed sense but that would have been more complex.
- anc84 10y agoWhy should it? I like that the protocol itself favours performance. Traffic shaping/QoS should be done by the maintainers of the pipes (ie routers or OS).
- madgar 10y agoThe most common problem is with asymmetric network connections with limited upload bandwidth. If you don't limit your upload rate, BitTorrent will consume your upload bandwidth so thoroughly that even TCP ACKs for other applications aren't sent in a timely fashion.
- motoboi 10y agoHe forgot to mention a very important aspect of bittorrent: the uTP (micro-TP) protocol. uTP is a very nice bittorrent over UDP protocol implementing LEDBAT congestion control algorithm [1]. 1 - https://en.m.wikipedia.org/wiki/LEDBAT https://en.m.wikipedia.org/wiki/LEDBAT
- JepZ 10y agoEverything there looks very pro HTTP like... But hey, curl doesn't support bittorrent afaik. So why should haxx.se care ;-)
- ryancox 10y agoInteresting link in that Wikipedia article [1] to Apple's LEDBAT implementation used for downloading updates. [1] https://opensource.apple.com/source/xnu/xnu-1699.32.7/bsd/netinet/tcp_ledbat.c https://opensource.apple.com/source/xnu/xnu-1699.32.7/bsd/ne...
- sametmax 10y agoI wish browsers and OS would implement torrent by default. Not only would it make the sharing file experience (downoading, sending, etc) much better, but it would open the door to many cool P2P stuff on the client side. It's a standard, we know it works and how.
- kome 10y agoOpera 12 had torrent... It's incredible how Opera was innovative at the time. Nowadays it just another useless chrome.
- akjainaj 10y agoYes, and it also had an IRC client. Having lots of features unrelated to web browsing did not make it better.
- d33 10y ago> HTTP is an established and formalized protocol standard by the IETF. > Bittorrent is a protocol designed by the company named Bittorrent and there is a large amount of variations and extensions in implementations and clients. Extensions are in many cases standarized. There's a lot of variations in HTTP as well. One important thing that's missed is that a magnet link also contains a hash of the torrent file, which points to a specific file. Over HTTP, everyone across the network connection can change the contents of the file and there's no automated way of detecting that.
- robryk 10y agoHTTP is standardized, but lots of stuff doesn't follow the standard in the areas that didn't use to matter. Then we get problems like this: https://bugs.chromium.org/p/chromium/issues/detail?id=616212 https://bugs.chromium.org/p/chromium/issues/detail?id=616212 that have to be worked around by assuming that other devices might not adhere to the specification.
- LinuxBender 10y agoMy personal preference would be MPTCP, then BitTorrent, Googles idea for UDP HTTP, then plain TCP HTTP. Many things can benefit from MPTCP however, including BitTorrent.
- sleepychu 10y agoNot sure if Daniel reads this but: >only gets a few /s/gets/has/
- discreditable 10y agoBitTorrent can cheat a little with web seeds. It treats an HTTP host as a "seed" in the swarm. https://en.wikipedia.org/wiki/BitTorrent#Web_seeding https://en.wikipedia.org/wiki/BitTorrent#Web_seeding
- derefr 10y agoSpeaking of web seeds, the magnet: URI scheme—with an embedded web seed—has everything it needs to be an all-in-one subresource integrity + CDN-neutral subresource caching mechanism. If browsers could resolve magnet: links natively, you could just link all your page's resources as magnet: resources with the copies on your own servers acting as the web seeds.
- Forbo 10y agoI really hope that things like ZeroNet start to get more attention/integration into browsers. I see amazing potential for the democratization of the web using technologies like this. https://zeronet.io/ https://zeronet.io/
- niftich 10y agoBittorrent has innate support for content addressing, while HTTP is location-addressed: there is simply no way to decouple a content's location from its identity with HTTP. You can try to fake it, of course, with more indirection, but each hop requires communications with a specific location reachable with HTTP. In fortunate circumstances, Bittorrent, it can run fully P2P without needing 'servers' -- depending on whether you have to holepunch a NAT, whether you're okay with peer discovery taking longer in the DHT without a tracker, and whether you're okay with no fallback webseed as a seed-of-last-resort. Magnet links, a distinct concept not only applicable to Bittorrent, are the cherry on top, which enable even the ".torrent" file (or rather, the equivalent information) to be found without having to possess the .torrent file itself, from just a hash. BT's architecture has a nice effect that popular content consumes fewer of the original host's bandwidth than it would with HTTP, but this works best if peer interest roughly coincides in time. This is why it's a good fit for distributing patches [1][2], or newly released episodic content, and a fairly poor fit for anything else. The Bittorrent extensions (BEPs) vary widly in quality, clarity, verbosity, and rigor, and aren't always up to the level of HTTP extensions acknowledged by the IETF. Most BEPs were implemented in only one product and then submitted as a BEP retroactively, while throughout its lifetime most of HTTP's enhancements were discussed and developed in a semi-open, but publicly viewable process in the IETF with more emphasis on consensus and early prototyping, rather than final implementation and retroactive standardization. These days this process has partially been subsumed by prominent vendors implementing a behavior, running it for an extended amount of time as a proprietary enhancement, then submitting a slightly altered version to the IETF for discussion, so admittedly the two enhancement processes are a lot more similar now than they've been in the past. [1] http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/ http://arstechnica.com/business/2012/04/exclusive-a-behind-t... [2] http://wow.gamepedia.com/Blizzard_Downloader http://wow.gamepedia.com/Blizzard_Downloader
- sigi45 10y agoSo much wrong/missing and missleading information. It would be better to delete it.
- RedHatTurtle 10y agoIsnt IPFS the best adaptation of "bittorrent ideas" to the http general use case? On an unrelated issue, it baffles me how people with uncapped internet refuse to help serve content to other people, like enabling the distribution of windows updates. I would love to have that on linux and would never dare to turn it of unless is was reaching my data cap. It's just liek seeding you torrents, hell i have seed ratios of over 100 for some linux distro images.