8 ms·

I worked at Twitch from 2014 to 2018. I was never on the video team, but here are some details that have changed. Video: - Everything is migrated from RTMP to

I worked at Twitch from 2014 to 2018. I was never on the video team, but here are some details that have changed. Video: - Everything is migrated from RTMP to HLS; RTMP couldn't scale across several consumer platforms - This added massive delay (~30+s) early on, the video team has been crushing it getting this back down to the now sub-2s - Flash is dead (now HTML5) - F5 storms ("flash crowds") are still a concern teams design around; e.g. 900k people hitting F5 after a stream blips offline due to the venue's connection - afaik Usher is still alive and well, in much better health today - Most teams are on AWS now; video was the holdout for a while because they needed specialized GPUs. EDIT: "This isn't quite right; it has more to do with the tight coupling of the video system with the network (eg, all the peering stuff described in the article)" -spenczar5 - Realtime transcoding is a really interesting architecture nowadays (I am not qualified to explain it) Web: - No more Ruby on Rails because no good way was found to scale it organizationally; almost everything is now Go microservices back + React front - No more Twice - Data layer was split up to per-team; some use PostgreSQL, some DynamoDB, etc. - Of course many more than 2 software teams now :P - Chat went through a major scaling overhaul during/after Twitch Plays Pokemon. John Rizzo has a great talk about it here: https://www.twitch.tv/videos/92636123?t=03h13m46s https://www.twitch.tv/videos/92636123?t=03h13m46s Twitch was a great place to spend 5 years at. Would do again.

arcticfox 7y ago
F5 storm is an awesome name for the venue blip -> refresh reaction. I've certainly contributed my fair share to your storms. It's basically automatic.
- ummonk 7y ago
  Yeah, and the problem is that it often does work to fix issues. It's the web equivalent of "have you tried turning it off and turning it back on again?"...
  - spenczar5 7y ago
    Or, the live video equivalent of "retry" :)
  - Lorin 7y ago
    Maybe something as simple as a textual overlay "Don't worry - the stream will be back soon" and a script to internally hammer the only element of the page that's actually needed for a video refresh
    - Sahhaese 7y ago
      text overlays would still be clunky, what's really needed is dynamic video spliced directly into the stream so the viewer understands it's the broadcasters' connection that is poor not the viewer's. This could be achieved if twitch allowed broadcasters to upload a 2 second loop as a 'placeholder' for connection drops, and twitch could mix that in if it detects too many frame-drops. It would need to be a setting though because some streamers just stream over a bad connection (e.g. Hitchhiking streams) and wouldn't want that interruption.
      - hatsix 7y ago
        I guess this is actually being tested right now. I saw something like this last week, though I've also seen streams just drop.
- pulkitsh1234 7y ago
  I initially thought 'F5 storm' refers to people typing "F" in the chat.
diminoten 7y ago
> - No more Ruby on Rails because no good way was found to scale it organizationally; almost everything is now Go microservices back + React front Ugh, I just... I keep trying to pretend I don't need to learn Go, but every highly scalable system I read about that's recently been written about seems to be using it. Maybe I just need to stay away from systems that need to scale? Heh...
- echelon 7y ago
  Go isn't the only language that scales, it just happens to be popular amongst the scripting language crowd as a next step. You're by no means limited in your choice. You could do Java, C#, Rust...
- snaky 7y ago
  The keyword here is organizationally. Technically speaking you can build scalable systems using anything you want. But if you need to hire a couple of hundreds developers, you better go with Java 7 or Go than Ruby, Lisp or Perl. The dumber and more uniform the better.
  - ummonk 7y ago
    The key distinguisher there is static typing. Which is how Facebook manages to make do with PHP, and Dropbox with Python. By adding top annotations for a static type-checker...
    - viraptor 7y ago
      It would be interesting to hear a comparison from someone using https://crystal-lang.org/ https://crystal-lang.org/ at scale. It's basically Ruby + types, which would make it the closest for isolating that one feature. It can't run Rails, but there are very similar web frameworks available.
      - mcpherrinm 7y ago
        It seems Stripe solved this problem with https://sorbet.org/ https://sorbet.org/ which is actually Ruby + types.
wrigby 7y ago
On my first read-through I thought 'F5 Storm' referred to F5 load balancers, not hitting the F5 key to refresh the page.
spenczar5 7y ago
Hi glacials :) Small correction from someone at Twitch today: > video was the holdout for a while because they needed specialized GPUs This isn't quite right; it has more to do with the tight coupling of the video system with the network (eg, all the peering stuff described in the article).
- glacials 7y ago
  Oh, edited! Thanks Spencer :)
- deleted 7y ago
  [deleted]
zemnmez 7y ago
miss you!
- glacials 7y ago
  Miss you too buddy! Hit me up when you're in Seattle next.
  - zemnmez 7y ago
    of course :) will be easier when I've relocated to sf
conover 7y ago
“+ React front” to say the least! Hope you are well, glacials.
- glacials 7y ago
  Hehe, great job on that Chris :)
thomastjeffery 7y ago
> Realtime transcoding I'm curious: Do services like Twitch specify a specific desired codec/bitrate that doesn't get transcoded? Transcoding seems like a lot of effort for lower quality end result. If I were streaming, I would want to avoid transcoding as much as possible. Since we're talking about live broadcasting, there is a unique ability for the streamer to choose the format they upload.
- kd5bjo 7y ago
  In the RTMP days, the highest quality setting in the viewer was always a straight pass through from the broadcaster, and the reduced versions were transcoded in the data center to fit down lower-bandwidth last-mike pipes.
  - thomastjeffery 7y ago
    Gotcha. For whatever reason, I had forgotten about lower-bandwidth copies.
  - slimscsi 7y ago
    Same in the HLS days. What used to be called “source” was just a remux from rtmp to ts.
PullJosh 7y ago
> everything is now Go microservices back Excuse the simple question: When I hear "microservices", I think serverless backend. Is that right, or are they different? If they're the same, how do you stream video with serverless? (Seems like streaming, websockets, etc... shouldn't be possible in a serverless environment...)
- glacials 7y ago
  They're different. Microservices are still stateful applications that run 24/7. They are just really small in scope. e.g. the Friends feature on Twitch is one microservice, running in its own autoscaling group, with internal APIs used by other microservices like Whispers.
- leonidasv 7y ago
  Microservices are about splitting code into different servers instead of a monolithic codebase. You end up with different servers (probably virtualized) for each domain of the application. Like, instead of having the video decoding and the analytics code in the same monolith attached to same DB, you deploy a different server for each one, generally with a new DB for each. When the services need to talk to each other, they do it via network (REST, gRPC, etc.).
- discordance 7y ago
  My team follows microservice patterns, and have deployed services that utilise websockets over both serverless (Azure Functions and Lambda) as well as regular hosted services (on k8s, EC2 and Azure App service etc). Nothing stopping you there. On the streaming video side we did an app that used Azure Media services + Azure functions.. works well enough. Not necessarily a good idea, but one 'feature' of microservices is the ability to pick different stacks, languages and delivery methods on an individual service level.
- 013a 7y ago
  "Microservice" describes the size and scope of each deployment artifact. It answers the question "is the whole system just one big ball, is it broken up, how broken up is it?" It doesn't describe how it is deployed. "Serverless" describes how a deployment artifact is deployed and runs. Generally it refers to a class of technologies in multiple domains whereby intricate knowledge of the underlying host is abstracted behind a cleaner API, with things like scaling, security, patching, etc handled by an infrastructure provider. While the term rose in prominence alongside "functions as a service", which is certainly a technology that generally qualifies as serverless, there are many serverless products out there: AWS Fargate for running containers, DynamoDB for a database, S3 for object storage, all of these are "serverless". A good signal is: if I can SSH into it, its not serverless. A microservice can certainly be deployed serverless (ECS/Fargate or Google Cloud Run comes to mind). A microservice can even refer to one or more logically related functions-as-a-service; the term more-so speaks to how the engineering teams organize their business domain into the code and how the APIs speak to each other, rather than the exact underlying technologies.
cantbecool 7y ago
Didn't Twitch use elemental machines for transcoding?
- grogenaut 7y ago
  No. Elemental is more of a high end encoding system for quality. Twitch is more about bulk cheap transcodes of good quality. Think about it. MLB has maybe 18 concurrent events. Twitch is running minimum in the 10k range.
- Dobbs 7y ago
  No we never had Elementals. In the early days there was no way we could afford them. In the later days I don't think we would want them as we needed to scale so many transcode jobs that it was easier to have a large farm of dumb machines to organise jobs across. There may have been an element machine at one point that was used for testing/playing but I really don't think so, and know there wasn't one between 2010 and 2017.
  - kd5bjo 7y ago
    Transcoding was a relatively late addition to the whole system— for a long time we only passed through the original video bits unchanged and tried to advise broadcasters about picking compromise settings. By the time we decided transcoding was necessary, we had enough in-house video engineering knowledge to build our own system integrated with everything else.
    - Dobbs 7y ago
      We had transcoding as early as 2011. As that is when I made my first commits to the video jobs codebase, specifically to the transcoding jobs. It was quite late when we had the resources ($$$) to provide widespread availability of transcodes to the community.
      - kd5bjo 7y ago
        Yeah; my perspective on “late” is probably pretty skewed, since I left as the rebranding was still being developed. I think the favorite new name when I left was something like Xarth; Twitch was a much better choice.
NightlyDev 7y ago
"F5 storms" are easy to handle. Intercept all keypress combinations for refresh and do what you want with it client side. (spread it out over time, use a high-performance endpoint to check if live or a combination) Most people doesn't use the refresh button in the browser, so only a small amount of traffic will be uncontrolled.
- cortesoft 7y ago
  You have any data on most people not using the refresh button in the browser?
- jrockway 7y ago
  Do you have any data to support that? I personally don't have an F5 key on my keyboard (it requires pressing a modifier), so I pretty much always click the reload button to fix a stream blip. The impression I get from reading Twitch chat is that most people are using mobile. I doubt they have a keyboard plugged in and press F5 to refresh. That said, you certainly don't need your video streaming servers to handle those hundred-thousand refresh requests.
  - squeaky-clean 7y ago
    Yeah anecdotally every non-tech person I know clicks the refresh button. It also doesn't catch people watching on mobile web who refresh. And I don't believe their mobile app has a refresh button, whenever a stream glitches out for me I force kill and reload the mobile app.

arcticfox 7y ago

F5 storm is an awesome name for the venue blip -> refresh reaction. I've certainly contributed my fair share to your storms. It's basically automatic.

ummonk 7y ago
Yeah, and the problem is that it often does work to fix issues. It's the web equivalent of "have you tried turning it off and turning it back on again?"...
- spenczar5 7y ago
  Or, the live video equivalent of "retry" :)
- Lorin 7y ago
  Maybe something as simple as a textual overlay "Don't worry - the stream will be back soon" and a script to internally hammer the only element of the page that's actually needed for a video refresh
  - Sahhaese 7y ago
    text overlays would still be clunky, what's really needed is dynamic video spliced directly into the stream so the viewer understands it's the broadcasters' connection that is poor not the viewer's. This could be achieved if twitch allowed broadcasters to upload a 2 second loop as a 'placeholder' for connection drops, and twitch could mix that in if it detects too many frame-drops. It would need to be a setting though because some streamers just stream over a bad connection (e.g. Hitchhiking streams) and wouldn't want that interruption.
    - hatsix 7y ago
      I guess this is actually being tested right now. I saw something like this last week, though I've also seen streams just drop.
pulkitsh1234 7y ago
I initially thought 'F5 storm' refers to people typing "F" in the chat.

diminoten 7y ago

> - No more Ruby on Rails because no good way was found to scale it organizationally; almost everything is now Go microservices back + React front Ugh, I just... I keep trying to pretend I don't need to learn Go, but every highly scalable system I read about that's recently been written about seems to be using it. Maybe I just need to stay away from systems that need to scale? Heh...

echelon 7y ago
Go isn't the only language that scales, it just happens to be popular amongst the scripting language crowd as a next step. You're by no means limited in your choice. You could do Java, C#, Rust...
snaky 7y ago
The keyword here is organizationally. Technically speaking you can build scalable systems using anything you want. But if you need to hire a couple of hundreds developers, you better go with Java 7 or Go than Ruby, Lisp or Perl. The dumber and more uniform the better.
- ummonk 7y ago
  The key distinguisher there is static typing. Which is how Facebook manages to make do with PHP, and Dropbox with Python. By adding top annotations for a static type-checker...
  - viraptor 7y ago
    It would be interesting to hear a comparison from someone using https://crystal-lang.org/ https://crystal-lang.org/ at scale. It's basically Ruby + types, which would make it the closest for isolating that one feature. It can't run Rails, but there are very similar web frameworks available.
    - mcpherrinm 7y ago
      It seems Stripe solved this problem with https://sorbet.org/ https://sorbet.org/ which is actually Ruby + types.

wrigby 7y ago

On my first read-through I thought 'F5 Storm' referred to F5 load balancers, not hitting the F5 key to refresh the page.

spenczar5 7y ago

Hi glacials :) Small correction from someone at Twitch today: > video was the holdout for a while because they needed specialized GPUs This isn't quite right; it has more to do with the tight coupling of the video system with the network (eg, all the peering stuff described in the article).

glacials 7y ago
Oh, edited! Thanks Spencer :)
deleted 7y ago
[deleted]

zemnmez 7y ago

miss you!

glacials 7y ago
Miss you too buddy! Hit me up when you're in Seattle next.
- zemnmez 7y ago
  of course :) will be easier when I've relocated to sf

conover 7y ago

“+ React front” to say the least! Hope you are well, glacials.

glacials 7y ago
Hehe, great job on that Chris :)

thomastjeffery 7y ago

> Realtime transcoding I'm curious: Do services like Twitch specify a specific desired codec/bitrate that doesn't get transcoded? Transcoding seems like a lot of effort for lower quality end result. If I were streaming, I would want to avoid transcoding as much as possible. Since we're talking about live broadcasting, there is a unique ability for the streamer to choose the format they upload.

kd5bjo 7y ago
In the RTMP days, the highest quality setting in the viewer was always a straight pass through from the broadcaster, and the reduced versions were transcoded in the data center to fit down lower-bandwidth last-mike pipes.
- thomastjeffery 7y ago
  Gotcha. For whatever reason, I had forgotten about lower-bandwidth copies.
- slimscsi 7y ago
  Same in the HLS days. What used to be called “source” was just a remux from rtmp to ts.

PullJosh 7y ago

> everything is now Go microservices back Excuse the simple question: When I hear "microservices", I think serverless backend. Is that right, or are they different? If they're the same, how do you stream video with serverless? (Seems like streaming, websockets, etc... shouldn't be possible in a serverless environment...)

glacials 7y ago
They're different. Microservices are still stateful applications that run 24/7. They are just really small in scope. e.g. the Friends feature on Twitch is one microservice, running in its own autoscaling group, with internal APIs used by other microservices like Whispers.
leonidasv 7y ago
Microservices are about splitting code into different servers instead of a monolithic codebase. You end up with different servers (probably virtualized) for each domain of the application. Like, instead of having the video decoding and the analytics code in the same monolith attached to same DB, you deploy a different server for each one, generally with a new DB for each. When the services need to talk to each other, they do it via network (REST, gRPC, etc.).
discordance 7y ago
My team follows microservice patterns, and have deployed services that utilise websockets over both serverless (Azure Functions and Lambda) as well as regular hosted services (on k8s, EC2 and Azure App service etc). Nothing stopping you there. On the streaming video side we did an app that used Azure Media services + Azure functions.. works well enough. Not necessarily a good idea, but one 'feature' of microservices is the ability to pick different stacks, languages and delivery methods on an individual service level.
013a 7y ago
"Microservice" describes the size and scope of each deployment artifact. It answers the question "is the whole system just one big ball, is it broken up, how broken up is it?" It doesn't describe how it is deployed. "Serverless" describes how a deployment artifact is deployed and runs. Generally it refers to a class of technologies in multiple domains whereby intricate knowledge of the underlying host is abstracted behind a cleaner API, with things like scaling, security, patching, etc handled by an infrastructure provider. While the term rose in prominence alongside "functions as a service", which is certainly a technology that generally qualifies as serverless, there are many serverless products out there: AWS Fargate for running containers, DynamoDB for a database, S3 for object storage, all of these are "serverless". A good signal is: if I can SSH into it, its not serverless. A microservice can certainly be deployed serverless (ECS/Fargate or Google Cloud Run comes to mind). A microservice can even refer to one or more logically related functions-as-a-service; the term more-so speaks to how the engineering teams organize their business domain into the code and how the APIs speak to each other, rather than the exact underlying technologies.

cantbecool 7y ago

Didn't Twitch use elemental machines for transcoding?

grogenaut 7y ago
No. Elemental is more of a high end encoding system for quality. Twitch is more about bulk cheap transcodes of good quality. Think about it. MLB has maybe 18 concurrent events. Twitch is running minimum in the 10k range.
Dobbs 7y ago
No we never had Elementals. In the early days there was no way we could afford them. In the later days I don't think we would want them as we needed to scale so many transcode jobs that it was easier to have a large farm of dumb machines to organise jobs across. There may have been an element machine at one point that was used for testing/playing but I really don't think so, and know there wasn't one between 2010 and 2017.
- kd5bjo 7y ago
  Transcoding was a relatively late addition to the whole system— for a long time we only passed through the original video bits unchanged and tried to advise broadcasters about picking compromise settings. By the time we decided transcoding was necessary, we had enough in-house video engineering knowledge to build our own system integrated with everything else.
  - Dobbs 7y ago
    We had transcoding as early as 2011. As that is when I made my first commits to the video jobs codebase, specifically to the transcoding jobs. It was quite late when we had the resources ($$$) to provide widespread availability of transcodes to the community.
    - kd5bjo 7y ago
      Yeah; my perspective on “late” is probably pretty skewed, since I left as the rebranding was still being developed. I think the favorite new name when I left was something like Xarth; Twitch was a much better choice.

NightlyDev 7y ago

"F5 storms" are easy to handle. Intercept all keypress combinations for refresh and do what you want with it client side. (spread it out over time, use a high-performance endpoint to check if live or a combination) Most people doesn't use the refresh button in the browser, so only a small amount of traffic will be uncontrolled.

cortesoft 7y ago
You have any data on most people not using the refresh button in the browser?
jrockway 7y ago
Do you have any data to support that? I personally don't have an F5 key on my keyboard (it requires pressing a modifier), so I pretty much always click the reload button to fix a stream blip. The impression I get from reading Twitch chat is that most people are using mobile. I doubt they have a keyboard plugged in and press F5 to refresh. That said, you certainly don't need your video streaming servers to handle those hundred-thousand refresh requests.
- squeaky-clean 7y ago
  Yeah anecdotally every non-tech person I know clicks the refresh button. It also doesn't catch people watching on mobile web who refresh. And I don't believe their mobile app has a refresh button, whenever a stream glitches out for me I force kill and reload the mobile app.