8 ms·

How are we still talking about this? The original “architecture” was just stupid. Splitting thousands of hours of video into 1-2s chunks, uploading them to s3,

How are we still talking about this? The original “architecture” was just stupid. Splitting thousands of hours of video into 1-2s chunks, uploading them to s3, then processing them in isolation? Does anyone even consider the monster layers of abstraction that these services require? It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea.

fzeindl 3y ago
> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea. Maybe Amazon used this to test step functions and lambdas.
- Cthulhu_ 3y ago
  I think there's a lot of projects within Amazon that get extra marks for dogfooding AWS and for providing materials / whitepapers for use cases of various AWS services. I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible. I'm sure Amazon's internal training is just that.
  - TheSoftwareGuy 3y ago
    > I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible That sounds like an obvious good thing right? The training isn't supposed to teach you how to solve problems, its supposed to teach you how to use a particular set of tools.
    - taeric 3y ago
      Ish? If we are talking about "hammer training," then I can see the argument. Teach me about the different hammers and why I would pay attention to the weight and such. But all too often, the training picks something like "build a clubhouse." And in that, I would expect legit solutions based on specifications of a clubhouse that you are wanting to build. Language advocates fall hard on this. They will "build a TODO application" to show off part of the language. But end up with an application that can't do what folks would want a TODO application to actually do. It is very frustrating.
twic 3y ago
I once worked on a payment processing system. Files came in with about a thousand payments, in a textual format, 10-20 short lines of text each. The geniuses who led the project before me decided that the first step was to split the file into single payment chunks, then put each one into RabbitMQ, from where a different service would pick them up and parse them - then encode the parsed payments in JSON, and put them back in RabbitMQ, for a third process to pick up and handle. This was all for performance, of course. I tried to explain that just parsing the whole file in one go would be much faster than splitting it up, writing it to a queue, and reading it back in. That it probably wouldn't be much slower than the splitting. But nobody was interested. Obviously distributing the work would be much faster. It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.
- tuyiown 3y ago
  Since many have been told repeatedly that rails is slow, maybe some just stopped caring if the code actually got slow.
  - deleted 3y ago
    [deleted]
- renegade-otter 3y ago
  That intuitive feel of "what's faster" has been lost ever since computing power exploded. Back when the resources were at a premium, we were just better engineers. Not all of us had the command of Leetcode algorithm problems, but we rarely missed a database index and knew to stay close to the metal as much as possible, avoiding network trips, avoiding database trips. It still holds true today, and software has become bloated and slower because these simple rules are being ignored. It's all in the cloud, it's all distributed, it's all "free". As for Rails - it's the same problem but it's different. If you stay too high up in the abstraction orbit, it's hard to think of the nitty gritty of performance. What is the cloud doing? What is the ORM doing? Once it's all "magic", you just keep piling on non-performant crap until it becomes an emergency. Then you just add more cache :)
axus 3y ago
No True Microservice would do something that dumb.
Miraste 3y ago
This explains why Prime Video is the only streaming service that can never stay stable on my gigabit connection, ugh. Sometimes it drops resolution until it's 360p.
- scrum-treats 3y ago
  This is true.
dbrueck 3y ago
> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3 Eh, that's a separate thing - that's how modern streaming works, i.e. it's what allowed streaming to become really, really good, and it's how pretty much every major streaming platform works today.
- dylan604 3y ago
  Serving an HLS segmented file is not the same as doing the processing work that enables a good transcode to that HLS.
  - dbrueck 3y ago
    Sorry, who is saying those two are the same?
dylan604 3y ago
I don't know about 1-2s chunks, but have you ever had to process a UHD HDR ProRes master with 24 channels of audio? Processing that on a single machine (depending on the level of processing) can easily take up to 10 hours. I definitely agree the 1-2s chunks makes no sense, but cutting it up into segments is the only way to get realistically acceptable processing times.
throwaway2990 3y ago
> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea. It’s only crazy to you because you don’t know what you’re doing. But it seems the average HNer isn’t smart enough to digest anything beyond “AWS” and “Cost” before raging about stuff they know nothing about.
rewmie 3y ago
> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3, then processing them in isolation? Now that you place it like that, it does sound very dubious. The overhead required by uploading MBs go S3 go afterwards have to download it back to the lamdda's execution context should have the same order of magnitude as processing the payload. I'm sure they were aware of this but we're hoping that things would scale well due to the embarrassingly parallel nature of the problem.
dangoodmanUT 3y ago
that's how HLS works if you switch "processing" to "serving"

fzeindl 3y ago

> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea. Maybe Amazon used this to test step functions and lambdas.

Cthulhu_ 3y ago
I think there's a lot of projects within Amazon that get extra marks for dogfooding AWS and for providing materials / whitepapers for use cases of various AWS services. I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible. I'm sure Amazon's internal training is just that.
- TheSoftwareGuy 3y ago
  > I've done some AWS training, especially the introductory ones are intended to get you to use as many different services as possible That sounds like an obvious good thing right? The training isn't supposed to teach you how to solve problems, its supposed to teach you how to use a particular set of tools.
  - taeric 3y ago
    Ish? If we are talking about "hammer training," then I can see the argument. Teach me about the different hammers and why I would pay attention to the weight and such. But all too often, the training picks something like "build a clubhouse." And in that, I would expect legit solutions based on specifications of a clubhouse that you are wanting to build. Language advocates fall hard on this. They will "build a TODO application" to show off part of the language. But end up with an application that can't do what folks would want a TODO application to actually do. It is very frustrating.

twic 3y ago

I once worked on a payment processing system. Files came in with about a thousand payments, in a textual format, 10-20 short lines of text each. The geniuses who led the project before me decided that the first step was to split the file into single payment chunks, then put each one into RabbitMQ, from where a different service would pick them up and parse them - then encode the parsed payments in JSON, and put them back in RabbitMQ, for a third process to pick up and handle. This was all for performance, of course. I tried to explain that just parsing the whole file in one go would be much faster than splitting it up, writing it to a queue, and reading it back in. That it probably wouldn't be much slower than the splitting. But nobody was interested. Obviously distributing the work would be much faster. It was a weird experience. My colleagues just had no ability to reason about the "physics" of what computers did. I think they mostly had Rails backgrounds, and I wonder if you just don't exercise that muscle much doing Rails.

tuyiown 3y ago
Since many have been told repeatedly that rails is slow, maybe some just stopped caring if the code actually got slow.
- deleted 3y ago
  [deleted]
renegade-otter 3y ago
That intuitive feel of "what's faster" has been lost ever since computing power exploded. Back when the resources were at a premium, we were just better engineers. Not all of us had the command of Leetcode algorithm problems, but we rarely missed a database index and knew to stay close to the metal as much as possible, avoiding network trips, avoiding database trips. It still holds true today, and software has become bloated and slower because these simple rules are being ignored. It's all in the cloud, it's all distributed, it's all "free". As for Rails - it's the same problem but it's different. If you stay too high up in the abstraction orbit, it's hard to think of the nitty gritty of performance. What is the cloud doing? What is the ORM doing? Once it's all "magic", you just keep piling on non-performant crap until it becomes an emergency. Then you just add more cache :)

axus 3y ago

No True Microservice would do something that dumb.

Miraste 3y ago

This explains why Prime Video is the only streaming service that can never stay stable on my gigabit connection, ugh. Sometimes it drops resolution until it's 360p.

scrum-treats 3y ago
This is true.

dbrueck 3y ago

> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3 Eh, that's a separate thing - that's how modern streaming works, i.e. it's what allowed streaming to become really, really good, and it's how pretty much every major streaming platform works today.

dylan604 3y ago
Serving an HLS segmented file is not the same as doing the processing work that enables a good transcode to that HLS.
- dbrueck 3y ago
  Sorry, who is saying those two are the same?

dylan604 3y ago

I don't know about 1-2s chunks, but have you ever had to process a UHD HDR ProRes master with 24 channels of audio? Processing that on a single machine (depending on the level of processing) can easily take up to 10 hours. I definitely agree the 1-2s chunks makes no sense, but cutting it up into segments is the only way to get realistically acceptable processing times.

throwaway2990 3y ago

> It’s crazy to me to think someone actually thought using step functions and lambdas in this way was a good idea. It’s only crazy to you because you don’t know what you’re doing. But it seems the average HNer isn’t smart enough to digest anything beyond “AWS” and “Cost” before raging about stuff they know nothing about.

rewmie 3y ago

> Splitting thousands of hours of video into 1-2s chunks, uploading them to s3, then processing them in isolation? Now that you place it like that, it does sound very dubious. The overhead required by uploading MBs go S3 go afterwards have to download it back to the lamdda's execution context should have the same order of magnitude as processing the payload. I'm sure they were aware of this but we're hoping that things would scale well due to the embarrassingly parallel nature of the problem.

dangoodmanUT 3y ago

that's how HLS works if you switch "processing" to "serving"