12 ms·
Justin.tv's Live Video Broadcasting Architecture (2010)
- dang 7y agoDiscussed at the time: https://news.ycombinator.com/item?id=1196264 https://news.ycombinator.com/item?id=1196264
- simlevesque 7y agoWowza is a great product. If you want to build a huge custom VOD + live boradcasting product, it is a great way to do it.
- OedipusRex 7y agoI miss the Justin.Tv days. Made a lot of friends there.
- PaulCroweRule34 7y agoHey friend
- jrs95 7y agoSome parts of Twitch are still kind of like that. The Classic WoW community especially has been amazing. Met a lot of people there that I play games & just hang out in Discord with pretty much every day.
- OedipusRex 7y agoTwitch spun out of Justin.tv and then they killed Justin.tv :(
- aerovistae 7y agoI can barely follow along with this, it's very technical. I can't imagine how Kyle Vogt acquired the necessary knowledge to make this work. Example: > The point of having multiple datacenters is not for redundancy, it's to be as close as possible to all the major peering exchanges. They picked the best locations in the country so they would have access to the largest number of peers. This is the kind of thing where I would have to hire some kind of network engineering expert, and he just figured this stuff out and made it work? I can't fathom other people's intelligence sometime.
- arcticfox 7y agoPretty exceptional indeed. Also impressive that he was able to grow from founder-stage tech to that scale, since they're largely different problems. Especially back in 2010. I feel like I'd have a much better shot of being able to figure out that scale these days than a decade ago. (If I spent my free time studying and not watching Age of Empires 2 on Justin.tv/Twitch).
- drwl 7y agoJustin Kan goes a bit into Kyle's accelerated learning - https://www.youtube.com/watch?v=YzyatiQrQlQ https://www.youtube.com/watch?v=YzyatiQrQlQ
- kd5bjo 7y ago> I can’t imagine how Kyle Vogt acquired the necessary knowledge to make this work. By this point in history, it wasn’t just him anymore and we’d done a few rounds of improvements already out of necessity. As I recall, he got us up and running at PAIX based mostly on research, but most of the other data centers were built out by a network engineer(1) we hired away from YouTube. While he was working on the network engineering and keeping the original system afloat, I did a lot of the software work for the system described here. (1) Name withheld out of courtesy
- kvogt 7y agoTrue. Kd5bjo is an incredible engineer.
- kd5bjo 7y agoI really should thank you for the opportunity— not many people would let a new college hire spend months on a green-field rewrite of something so core to the company.
- scurvy 7y agoCC? Dude's on webcasts with GigaOM. He was in customer sales roles with a VAR. He's not shy.
- Nition 7y agoSlightly off topic but this is my favorite justin.tv video, and also shows something of the website and how early in the days of live streaming we were back then: https://www.youtube.com/watch?v=BqgEm8XWXu8 https://www.youtube.com/watch?v=BqgEm8XWXu8
- 64738 7y agoThanks for that! I hadn't seen it before, that's pretty funny.
- glacials 7y agoI worked at Twitch from 2014 to 2018. I was never on the video team, but here are some details that have changed. Video: - Everything is migrated from RTMP to HLS; RTMP couldn't scale across several consumer platforms - This added massive delay (~30+s) early on, the video team has been crushing it getting this back down to the now sub-2s - Flash is dead (now HTML5) - F5 storms ("flash crowds") are still a concern teams design around; e.g. 900k people hitting F5 after a stream blips offline due to the venue's connection - afaik Usher is still alive and well, in much better health today - Most teams are on AWS now; video was the holdout for a while because they needed specialized GPUs. EDIT: "This isn't quite right; it has more to do with the tight coupling of the video system with the network (eg, all the peering stuff described in the article)" -spenczar5 - Realtime transcoding is a really interesting architecture nowadays (I am not qualified to explain it) Web: - No more Ruby on Rails because no good way was found to scale it organizationally; almost everything is now Go microservices back + React front - No more Twice - Data layer was split up to per-team; some use PostgreSQL, some DynamoDB, etc. - Of course many more than 2 software teams now :P - Chat went through a major scaling overhaul during/after Twitch Plays Pokemon. John Rizzo has a great talk about it here: https://www.twitch.tv/videos/92636123?t=03h13m46s https://www.twitch.tv/videos/92636123?t=03h13m46s Twitch was a great place to spend 5 years at. Would do again.
- arcticfox 7y agoF5 storm is an awesome name for the venue blip -> refresh reaction. I've certainly contributed my fair share to your storms. It's basically automatic.
- joefourier 7y ago"Live video can't be made by pushing video faster, it takes a completely differently architecture." Funnily enough, that's pretty much how HLS, the modern live video standard, works - it's essentially a series of tiny video clips loaded & played one right after the other, distributed through the same CDN as normal video files. Thanks to HLS, live video is actually much worse than it was 10 years ago with RTMP in terms of latency. There's been some recent efforts in getting it down, although they're generally not standardised, hard to scale (e.g. WebRTC) and or a bit awkward.
- spenczar5 7y agoThey're starting to become standardized. Apple announced LHLS, a "low latency extension" to the HLS standard, https://developer.apple.com/documentation/http_live_streaming/protocol_extension_for_low-latency_hls_preliminary_specification https://developer.apple.com/documentation/http_live_streamin.... I don't think anyone really follows Apple's spec for various technical reasons, though. Most do some sort of chunked-transfer encoding, along with pre-signaling segments in playlists, as outlined by the Periscope folks here: https://medium.com/@periscopecode/introducing-lhls-media-streaming-eb6212948bef https://medium.com/@periscopecode/introducing-lhls-media-str...
- eridius 7y ago> I don't think anyone really follows Apple's spec for various technical reasons, though. I would have assumed it's because Apple only announced this 4 weeks ago, and the only clients that support it are beta software.
- slimscsi 7y agoNone follow apples spec because it’s a month old. It’s not for technical reasons. Like it or not (TBH I’m in the fence about it), it WILL be the standard.
- zemnmez 7y agokind of? i think this is more about live replication infrastructure than the video carrier itself. with youtube, you can set up some CDNs, but with livestreaming you need to be continually ingesting and spitting out content at the same time to lots of places in the world at once
- kd5bjo 7y agoI was the primary architect for Usher and the server-side of the video system described here. I’m happy to answer any questions, assuming I still remember the answers.
- loljjkm8 7y agoWhy did you call it Usher and not T-pain
- kd5bjo 7y agoI don’t get your reference, but we chose the name Usher because it’s the software equivalent of the person at the theater who looks at your ticket and shows you where your seat is— it doesn’t actually handle any of the video data, it just knows where you should go to find it.
- deleted 7y ago[deleted]
- spydum 7y agoSurprised Microstrategy didn’t sue you guys (or vice versa). They built some access / authorization tool named Usher which seemed to be a total flop (why a BI company was even toying with this perplexes me)
- kd5bjo 7y agoIt was/is an internal code name that never shows up except in URLs requested by the player and highly technical articles like this one; I’m not a lawyer, but I expect they might have a hard time showing consumer confusion.
- person_of_color 7y agoHow would an IC SWE find a rocket ship startup like this in 2019?
- cbzehner 7y agoCheck out https://breakoutlist.com/ https://breakoutlist.com/, it's designed for answering just that question - with a bias towards Silicon Valley hype. If you're interested in Atrium, reach out to me (see profile). We're hiring!
- person_of_color 7y agoAwesome resource. Time to send some resumes over the weekend. Who's with me?
- augbog 7y agoReally cool reading this when I just started 2 weeks ago and I was perusing the Video Architecture docs. It's cool to see how far Twitch has come :) thanks for sharing
- imjustsaying 7y agoCan anyone recommend some stacks for hosting video streaming? For 10k concurrent stream receivers in the US and Europe, what would you be looking at to get the stream under 30 seconds lag time? I realize this is a hard problem but would like to explore it.
- SaltyBackendGuy 7y ago> They also don't have chats with the 100,000 people watching a channel. What they do is assign people into rooms of 200 people each so you can have a meaningful interaction in a smaller group. This also helps with scaling. I thought this was a pretty clever strategy. Just curious if this is still a thing. I've watched an unhealthy amount of Twitch (not all with chat open) and never noticed this.
- savanaly 7y agoI'm sure it's not. Often, the streamer is reading the chat onscreen while streaming, and it's always identical to the one I'm seeing, except perhaps delayed by some seconds.
- doomjunky 7y agoMaybe twitch creates an illusion by showing the streamer only a subset of people that is shared among all other groups. This allows everyone to see what the streamer sees and to communicate within their group.
- mlex 7y agoThis is definitely untrue for present-day Twitch. When you're in a chat room, you're in the chat room with everyone.
- kvogt 7y agoBuilding this was really fun, and I’m very proud of what kd5bjo, Emmett, and many others did to help turn Justin.tv/Twitch into what it is today. We found a way to make what was fundamentally an unprofitable business (if you relied on CDNs) work by relentlessly focusing on reducing cost to the absolute bare minimum through good technology choices and innovating when necessary. Justin.tv would have died otherwise.
- JoeOfTexas 7y agoThanks for sharing the information! Even if its 9 years old now, its very interesting for us dreamers.
- foobar_ 7y agoI used to think high scalability was something to aspire to. Now I think it's all bullshit. The web is a terrible architecture for videos. Torrents solved this problem. If you trying to make a camel walk with two legs, you don't deserve praise but a tomato.
- olah_1 7y agoIf I started a Twitch (or Discord) competitor today, I would use Janus as a base for web-rtc to share the load https://janus.conf.meetecho.com/ https://janus.conf.meetecho.com/ In fact, Evasyst is a blend of Twitch and Discord and uses Janus for this reason. https://evasyst.com/ https://evasyst.com/
- ec109685 7y agoTwilio is another option to get started since they can handle a lot of the heavy lifting: https://www.twilio.com/video https://www.twilio.com/video
- olah_1 7y agoWhy pay Twilio for web-rtc though? Aren't there enough open source web-rtc wrappers like Janus?
- slimscsi 7y agoYou would look at how much that costs and make a different decision. It can’t scale economically to Twitch size without massive amounts of investment into POPs (source: I built the HLS transcoding stack/infrastructure and HTML5 player at Twitch). A voice only service, sure because it’s 100 times less bandwidth.
- kd5bjo 7y agoGranted, we made the same evaluation about Wowza a decade ago, but it turned out to be useful as a small piece of a large system. Given how much Twitch has grown since then, no off-the-shelf system would be able to handle the scale. A brand-new competitor, though, won’t have to deal with those scaling problems for a while. They have to figure out product-market fit first, which probably can be done off-the-shelf these days. And then they’ll need to figure out how to keep the lights on, which will take a similar amount of engineering work as it did for Justin/Twitch.
- malloy 7y agoVery interesting