7 ms·
Wow, I never realized that Google offers per-minute billing. We have a video decoding, encoding workload, and it has been a major pain point handling it on AWS
by fpgaminer 11y ago
Wow, I never realized that Google offers per-minute billing. We have a video decoding, encoding workload, and it has been a major pain point handling it on AWS. Decoding/Encoding 4K video requires quite a bit of beef, with a hefty minimum memory requirement on the order of 3+ GB for h264 encoding. Less if you use ultrafast preset, but then you pay for larger files and larger egress.
So, we can spin up a beefy EC2 (c4.8xlarge or larger) for each video, and get them handled quickly, but we get charged for the full hour which is quite expensive. All the machines we would target for spot instances have had very rocky pricing lately, so that often doesn't save much. We could spin up less beefy instances, trying to encode/decode in ~1 hour so as to maximize economy, but then, obviously, each video takes much longer to process.
Lambda would be an okay solution, approximately double the price of equivalent EC2 solutions, while providing two very big wins: granularity and 5 minute encoding/decoding time regardless of video size/length. That second point is pretty insane; we could process an entire 4K movie in 5 minutes by unleashing a swarm of Lambdas on it. The problem is that Lambdas have a max memory of 1.5GB which isn't enough for 4K encoding (h264). Also they have limited disk space, so it's challenging to get frames and video in-out (and no FUSE module in the kernel :/). We're still experimenting, but even if we do get it to work it will probably be with the ultrafast preset, which is non-ideal.
Elastic Transcoder doesn't handle decoding/encoding, only transcoding, so its a no-go. We thought to use it in a larger pipeline, like have lambdas sew frames up into a lossless codec, then hand that to Elastic Transcoder to be compressed. But ET won't encode 444 formats, and it incurs the cost of both a lambda swarm and ET, so that's not great.
Per-minute billing would be ideal. Now I have to consider whether it's worth it to port everything over to Google. Oh how I wish these companies didn't use ludicrous egress pricing to enforce vendor lock-in.
- vgt 11y agoYour story is exactly the story of Atomic Fiction: https://cloudplatform.googleblog.com/2015/10/Atomic-Fiction-walks-The-Walk.html https://cloudplatform.googleblog.com/2015/10/Atomic-Fiction-...
- dookahku 11y ago> and no FUSE module in the kernel FUSE typically has a lot of latency for context switches, or so I under stood it. Is it possible to make a native driver?
- Eugr 11y agoWhat's even better, you can use preemptible instances in GCE which are a rough equivalent to Spot instances in Amazon, but have fixed pricing (no bidding nightmare). They can live up to 24 hours, but can be shut down earlier, so your processing pipeline should be able to handle it. But with up to 70% lower prices and the same per-minute billing it's a steal! Keep in mind that GCE has 10 minute minimum, so if you run an instance for less than 10 minutes, you'll be charged for 10 minutes, but after that billing is per-minute.
- jedberg 11y agoWhy not set up a render farm that's always running and then send it job after job to keep the pipeline full? You said that making it take an hour was ok, so I assume the processing time is somewhat flexible. If you don't have enough workload to always keep one machine busy, you could always wait until you did have enough workload and then spin up a one machine render farm, have it work for an hour and then shut it back down until you had enough work. It's true that per minute billing would save you money, but assuming what you're doing is meant to scale up, once you get past enough work to keep a single machine busy, you've gained nothing. However if you put in the work now to separate the jobs from the machines, you're on the path to much better future scalability, because you can size the render farm to the work instead of having the farm and the number of jobs directly linked.
- fpgaminer 11y ago> You said that making it take an hour was ok, so I assume the processing time is somewhat flexible. My comment was unclear. It's a balancing act between running time of a job, time before a job starts, and cost. 1 and 2 are both important and appreciated parameters, but only if 3 allows. > It's true that per minute billing would save you money, but assuming what you're doing is meant to scale up, once you get past enough work to keep a single machine busy, you've gained nothing. Completely true, but at that point we're running a cloud within a cloud ... which seems silly. Per-minute billing would get us what we need, without the complexity of having to engineer a cloud within a cloud. Plus, it would allow scaling horizontally, which improves parameter #2 above, without impacting #3. In other words, a farm of huge machines will need a queue to keep them busy. Having a queue means there's some delay before a job starts. Whereas if we just spin up a per-minute billed machine immediately for each job, there is no delay. That requires less engineering, and handles any workload whether it be heavy or sporadic.
- jedberg 11y ago> Completely true, but at that point we're running a cloud within a cloud You're doing that anyway, even if you run one job per machine, you just loose granularity (one of the big advantages of any "cloud"). > In other words, a farm of huge machines will need a queue to keep them busy. Having a queue means there's some delay before a job starts But that's only true if you can't fill one machine. Once you're filling one machine, you're queue processing time will be the same. Presumably you still have a queue, it's just that the queue drives the start of machines instead of processes on the machine. > That requires less engineering Right, like I said, you can do less engineering now but pay more later as your workloads scale up in both money and maintenance costs.
- illumin8 11y agoLambda is not a good fit for long running processes. Also, the lambda instances that run your code only have a small amount of RAM (I believe 512MB is the default, but it is tunable). In short, I don't believe lambda would be a good fit for your 4K video encoding use case.
- wnevets 11y agoLambda's limits are currently a max 5min run time and ~1.5GB.
- milesward 11y agoLittle pro-tip: use Direct Connect and POOF egress goes down to nearly $.03 per GB. https://aws.amazon.com/directconnect/ https://aws.amazon.com/directconnect/ Disclaimer: I work at GCP, and used to work at AWS, so I've actually seen this save customers $millions
- dclusin 11y agoThe leased line provider will also charge you for bandwidth too. But probably overall cheaper. And you get QOS guarantees too.
- milesward 11y agoNot if you use Direct Peering from Google :D https://cloud.google.com/interconnect/direct-peering https://cloud.google.com/interconnect/direct-peering Disclaimer: I work at GCP, so I know you can only get a few TB/s this way, which might be a bottleneck. YMMV
- wyldfire 11y agoOh, to be constrained by "a few TB/s".... :)
- boulos 11y agoOr even easier (come on Miles!), use CDN Interconnect via Fastly, CloudFlare, etc.: https://cloud.google.com/interconnect/cdn-interconnect https://cloud.google.com/interconnect/cdn-interconnect Disclosure: I work on Compute Engine.
- matt_wulfeck 11y agoAny blog posts you have links for describing how this architecture looks?
- milesward 11y agoAWS---[DX---GCI/DP]---GCP It's sorta like that. Disclaimer: I work at GCP, so feedback on how you'd like to see a more detailed version might actually have an effect!
- deleted 11y ago[deleted]
- boulos 11y agoFor workloads like yours, per-minute billing is amazing. It lets you have a simpler architecture (machine failed? no problem, just re-enqueue the single video or chunk that it was handling) and is economically equivalent to running the work back-to-back. We want to encourage you to do this, because internally we would never expect someone to keep some long running jobs up to run mapreduces. To entice you some more, custom machine types are also a huge differentiator here. The jump in cost and number of cores between fixed instance shapes is huge. When choosing between the c4.4xlarge and 8xlarge, you've got just over a 2x jump in price and performance. With custom machine types, you could nudge yourself upwards to say 18 vCPUs or 24 (and can even tune the type to your resolution). So take a look! We've got a free trial of $300 for 60 days. Disclosure: I work on Compute Engine (and specifically, Preemptible VMs).
- wstrange 11y agoPlease add a preemptible option for Container Engine! (GKE)
- Klinky 11y agoWhat are the benefits of using cloud providers for video encoding, compared to a dedicated server provider, or even a localized server? Dedicated providers can offer 16-core/32-thread Xeon servers for a few hundred dollars a month. If you need to spin up dozens or hundreds of machines at short notice, and then take them down, I could see that cloud use case. If it's a handful or just one, it seems the cloud only complicates the matter.
- extra88 11y agoCouldn't Amazon Snowball or Direct Connect significantly reduce the cost of getting your data off their storage?