4 ms·

From hosting a peertube instance solely for my own stuff for several years, I've come to appreciate just how difficult self hosting a streaming video platform i

by corndoge 10mo ago

From hosting a peertube instance solely for my own stuff for several years, I've come to appreciate just how difficult self hosting a streaming video platform is. As you say, bandwidth and storage requirements are significant; another less obvious one is transcoding. When a user uploads an HD video file, it needs to be transcoded into lower resolutions if you want there to be a hope of people streaming it. While Peertube itself is perfectly happy running on 2-4 vcpu cores on a cheap cloud vm, if you use those cores to handle transcode jobs it can take huge amounts of time (like 20+ hours) to transcode even medium length 1080p videos. So you really need either a lot of CPU that sits mostly idle, or hardware acceleration, both of which are expensive when purchased from cloud providers. Or you can use remote transcoding to offload transcode jobs onto your home gaming pc or whatever, which works well, but can be complicated and a bit touchy to set up properly, and now you have a point of failure dependent on your home network... And then, people watching videos are used to the YouTube experience with its world class CDN infra enabling subsecond first frame latencies even for 4k videos. They go on Peertube and first frame takes like 5 seconds for a 1080p video...realistically, with today's attention spans most of them are going to bounce before it ever plays.

hsbauauvhabzb 10mo ago
What value do you get in transcoding your own stuff? I have plex transcoding disabled on all local network devices that stream it and run into minimal issues (codecs on TV devices, mostly).
- corndoge 10mo ago
  By "my own stuff" I mean that I use my instance to upload videos I would otherwise upload to youtube - videos I made that I intend to share with people. The usual reasons for transcoding apply.
alisonatwork 10mo ago
The funny thing is that YouTube has now enshittified to the point where people routinely DO wait well over 5 seconds to watch the video they actually wanted to watch while interstitials and other commercials are jammed in. Even with adblock enabled, the latest YouTube code won't unlock the first frame of the actual video till some period of ad time has passed so you just sit there looking at a black screen. This on its own definitely isn't enough to get people to leave the platform, but it's still notable how much worse the experience has gotten compared to a few years ago.
- llbbdd 10mo ago
  On what setup? All YouTube videos load and start playing instantly for me. Every time I've experienced otherwise in the last couple years, it's been my first indication that e.g. AWS is exploding that day
  - alisonatwork 10mo ago
    I wonder if it depends what country you are in. I only notice it occasionally when the video won't play in FreeTube or PipePipe (which always has the pause at the start since the last few months) and I'm forced to open an incognito browser tab to watch, and then I realize just how many ads other people are being subjected to before they can even watch the video.
  - MobiusHorizons 10mo ago
    You likely pay for YouTube premium if you aren’t noticing adds
    - llbbdd 10mo ago
      I do pay for premium but my impression of the parent was that this was independent of ads. The test I did in the other comment didn't trigger an ad for some reason even though I was logged out, which may be why it loaded so fast.
      - MobiusHorizons 10mo ago
        Ah. The parent mentioned several frustrations that I am not familiar with (presumably since I also pay for premium and don’t block the ads), but my impression was that the delay was caused by the code refusing to play the video until the time slot for the ad had completed even if the ad failed to load (as would happen when blocking the ad http request)
  - moebrowne 10mo ago
    I bet you're using Chrome. Open a video in Chrome and the video is immediately playable, load the same video on the same machine in Firefox and you can expect to wait 5+ seconds for the video to be playable. I suspect that non-Chrome browsers are being intentionally hobbled
Dylan16807 10mo ago
I shove 1080p mp4s onto a very cheap server and I get 2 seconds of load time there versus somewhere between 1 and 2 seconds on youtube. And looking at network requests, the first chunk of the file loads in well under a second so I'd expect something with the metadata preloaded could start playing at that point. So if peertube takes 5 seconds, I really wonder why. Is it inconvenient to transcode before/during upload?
- corndoge 10mo ago
  If you scale an instance you need to use object storage (s3/b2/etc). Fetch from object storage can occasionally have latency spikes. 5 seconds is somewhat exaggerating, I clicked through 10 or so videos on my instance to check and it's 2-3 seconds most of the time.
  - Dylan16807 10mo ago
    We can exclude rare enough outliers. I've experienced B2 throwing a wrench into the dream of low latency, but some object stores are very fast. And more importantly you only need the first couple megabytes of each video to be on fast storage.
    - corndoge 10mo ago
      I'm using B2, so maybe that's it. I have the instance configured to serve video directly from B2 rather than proxying it. Peertube has no facilities to keep the first few mb of each video local to the server that I am aware of.
catapart 10mo ago
Since you seem like you have practical knowledge here, I hope you don't mind me asking: Would it change the equation, meaningfully, if you didn't offer any transcoding on the server and required users to run any transcoding they needed on their own hardware? I'm thinking of a wasm implementation of ffmpeg on the instance website, rather than requiring users to use a separate application, for instance. Would you think a general user couldn't handle the workload (mobile processing, battery, etc), or would that be fairly reasonable for a modern device and only onerous in the high traffic server environment?
- corndoge 10mo ago
  > Would it change the equation, meaningfully, if you didn't offer any transcoding on the server and required users to run any transcoding they needed on their own hardware? I think the user experience would be quite poor, enough that nobody would use the instance. As an example a 4k video will transcoded at least 2 times, to 1080p and 720p, and depending on server config often several more times. Each transcode job takes a long time, even with substantial hwaccel on a desktop. Very high bitrate video is quite common now since most phones, action cameras etc are capable of 4k30 and often 4k60. > Do you think a general user couldn't handle the workload (mobile processing, battery, etc), or would that be fairly reasonable for a modern device and only onerous. If I had to guess, I would expect it be a poor experience. Say I take a 5 minute video, that's probably around 3-5gb. I upload it, then need to wait - in the foreground - for this video to be transcoded and then uploaded to object storage 3 times on a phone chip. People won't do it. I do like the idea of offloading transcode to users. I wonder if it might be suited for something like https://rendernetwork.com/ https://rendernetwork.com/ where users exchange idle compute to a transcode pool for upload & storage rights, and still get to fire-and-forget uploads?
  - catapart 10mo ago
    Right on. Thanks for the consideration! I really appreciate you walking through that; it's an eye-opener! It seems like you not only deal with a considerable amount of five-minute-or-greater videos, but much higher quality than I was expecting, too. I also like the idea of user-transcoding because, honestly, I think it's better for everyone? I would love if every place I uploaded video or audio content offered an option to "include lower-quality variants" or something. Broadly, it's my product; I should have the final say on (and take responsibility for) the end result. And for high-quality stuff, the people who make it tend to have systems equipped to do that better anyway. So they could probably get faster transcoding times by using their own systems rather than letting the server do it. Seems like a win-win, even outside of the obvious benefits of "make a whole lot of computers do only the work they each need done, instead of making a few computers do the work that everyone needs done". With the only slight downside of the "average user" having some extra options that they don't understand which cause them to use it wrong and then everyone hates your product. Yay, app development.

hsbauauvhabzb 10mo ago

What value do you get in transcoding your own stuff? I have plex transcoding disabled on all local network devices that stream it and run into minimal issues (codecs on TV devices, mostly).

corndoge 10mo ago
By "my own stuff" I mean that I use my instance to upload videos I would otherwise upload to youtube - videos I made that I intend to share with people. The usual reasons for transcoding apply.

alisonatwork 10mo ago

The funny thing is that YouTube has now enshittified to the point where people routinely DO wait well over 5 seconds to watch the video they actually wanted to watch while interstitials and other commercials are jammed in. Even with adblock enabled, the latest YouTube code won't unlock the first frame of the actual video till some period of ad time has passed so you just sit there looking at a black screen. This on its own definitely isn't enough to get people to leave the platform, but it's still notable how much worse the experience has gotten compared to a few years ago.

llbbdd 10mo ago
On what setup? All YouTube videos load and start playing instantly for me. Every time I've experienced otherwise in the last couple years, it's been my first indication that e.g. AWS is exploding that day
- alisonatwork 10mo ago
  I wonder if it depends what country you are in. I only notice it occasionally when the video won't play in FreeTube or PipePipe (which always has the pause at the start since the last few months) and I'm forced to open an incognito browser tab to watch, and then I realize just how many ads other people are being subjected to before they can even watch the video.
- MobiusHorizons 10mo ago
  You likely pay for YouTube premium if you aren’t noticing adds
  - llbbdd 10mo ago
    I do pay for premium but my impression of the parent was that this was independent of ads. The test I did in the other comment didn't trigger an ad for some reason even though I was logged out, which may be why it loaded so fast.
    - MobiusHorizons 10mo ago
      Ah. The parent mentioned several frustrations that I am not familiar with (presumably since I also pay for premium and don’t block the ads), but my impression was that the delay was caused by the code refusing to play the video until the time slot for the ad had completed even if the ad failed to load (as would happen when blocking the ad http request)
- moebrowne 10mo ago
  I bet you're using Chrome. Open a video in Chrome and the video is immediately playable, load the same video on the same machine in Firefox and you can expect to wait 5+ seconds for the video to be playable. I suspect that non-Chrome browsers are being intentionally hobbled

Dylan16807 10mo ago

I shove 1080p mp4s onto a very cheap server and I get 2 seconds of load time there versus somewhere between 1 and 2 seconds on youtube. And looking at network requests, the first chunk of the file loads in well under a second so I'd expect something with the metadata preloaded could start playing at that point. So if peertube takes 5 seconds, I really wonder why. Is it inconvenient to transcode before/during upload?

corndoge 10mo ago
If you scale an instance you need to use object storage (s3/b2/etc). Fetch from object storage can occasionally have latency spikes. 5 seconds is somewhat exaggerating, I clicked through 10 or so videos on my instance to check and it's 2-3 seconds most of the time.
- Dylan16807 10mo ago
  We can exclude rare enough outliers. I've experienced B2 throwing a wrench into the dream of low latency, but some object stores are very fast. And more importantly you only need the first couple megabytes of each video to be on fast storage.
  - corndoge 10mo ago
    I'm using B2, so maybe that's it. I have the instance configured to serve video directly from B2 rather than proxying it. Peertube has no facilities to keep the first few mb of each video local to the server that I am aware of.

catapart 10mo ago

Since you seem like you have practical knowledge here, I hope you don't mind me asking: Would it change the equation, meaningfully, if you didn't offer any transcoding on the server and required users to run any transcoding they needed on their own hardware? I'm thinking of a wasm implementation of ffmpeg on the instance website, rather than requiring users to use a separate application, for instance. Would you think a general user couldn't handle the workload (mobile processing, battery, etc), or would that be fairly reasonable for a modern device and only onerous in the high traffic server environment?

corndoge 10mo ago
> Would it change the equation, meaningfully, if you didn't offer any transcoding on the server and required users to run any transcoding they needed on their own hardware? I think the user experience would be quite poor, enough that nobody would use the instance. As an example a 4k video will transcoded at least 2 times, to 1080p and 720p, and depending on server config often several more times. Each transcode job takes a long time, even with substantial hwaccel on a desktop. Very high bitrate video is quite common now since most phones, action cameras etc are capable of 4k30 and often 4k60. > Do you think a general user couldn't handle the workload (mobile processing, battery, etc), or would that be fairly reasonable for a modern device and only onerous. If I had to guess, I would expect it be a poor experience. Say I take a 5 minute video, that's probably around 3-5gb. I upload it, then need to wait - in the foreground - for this video to be transcoded and then uploaded to object storage 3 times on a phone chip. People won't do it. I do like the idea of offloading transcode to users. I wonder if it might be suited for something like https://rendernetwork.com/ https://rendernetwork.com/ where users exchange idle compute to a transcode pool for upload & storage rights, and still get to fire-and-forget uploads?
- catapart 10mo ago
  Right on. Thanks for the consideration! I really appreciate you walking through that; it's an eye-opener! It seems like you not only deal with a considerable amount of five-minute-or-greater videos, but much higher quality than I was expecting, too. I also like the idea of user-transcoding because, honestly, I think it's better for everyone? I would love if every place I uploaded video or audio content offered an option to "include lower-quality variants" or something. Broadly, it's my product; I should have the final say on (and take responsibility for) the end result. And for high-quality stuff, the people who make it tend to have systems equipped to do that better anyway. So they could probably get faster transcoding times by using their own systems rather than letting the server do it. Seems like a win-win, even outside of the obvious benefits of "make a whole lot of computers do only the work they each need done, instead of making a few computers do the work that everyone needs done". With the only slight downside of the "average user" having some extra options that they don't understand which cause them to use it wrong and then everyone hates your product. Yay, app development.