5 ms·

Thanks to all the FFmpeg contributors! Fantastic piece of software. On a project I was recently on recently we started hitting the per-region concurrent transc

by djm_ 11y ago

Thanks to all the FFmpeg contributors! Fantastic piece of software. On a project I was recently on recently we started hitting the per-region concurrent transcode limits on Amazon's Elastic Transcoder. [1] Instead of sharding over pipelines or accounts we set up a pipeline with FFMPEG + Lambda functions and it performed fantastically (within the free tier even). It was incredibly simple to write the functions and has given that project a lot more freedom; with the caveat that the any single task you undertake should occur within the timeout window (currently 5 minutes). Having said that, it's also straight forward to split the process into steps and have multiple lambda jobs to make the flow more of a pipeline. [1] http://docs.aws.amazon.com/elastictranscoder/latest/developerguide/limits.html http://docs.aws.amazon.com/elastictranscoder/latest/develope...

douglasfshearer 11y ago
How did the costs compare with this setup? I work on a project where ET isn't flexible enough (MPEG-DASH), and wondered whether Lambda would make for a good alternative to EC2 + SQS + Scaling Groups.
- bagels 11y ago
  EC2 + SQS + ffmpeg is 1/5 the cost of elastic transcoding for me, and that's not even using all the ec2 capacity.
Exuma 11y ago
Wow... I would truly appreciate if you could share a bit more about your specific setup. I found myself working on a new project yesterday that I was really really excited about, until I saw the costs to transcode video. How does doing all this in-house compare price wise (say, per minute), compared to using elastic transcoder? Edit: The ultimate lowest cost I can find is $0.0125-0.015
- djm_ 11y ago
  I wish I could edit my parent comment but alas. The key point was missed: we were dealing with very short, small videos. If you are dealing with longer or large videos, it's simply not feasible on Lambda. As for costings, unfortunately I cannot retrieve them as this project was mid last year and I've since moved on to other clients. They can be calculated though with a few short tests I'm sure.
- deleted 11y ago
  [deleted]
manishsharan 11y ago
Could you share your experience with FFMPEG+ Lambda ? I ran into trouble with this when dealing with large files, especially when some of the files were being pulled off non S3 sources. Also what EC2 cores were you using. ?
- djm_ 11y ago
  Sure: you likely don't want to be dealing with large files on Lambda. Why? * The maximum timeout window for a function invocation is 300 seconds * The maximum available temp disk space per instance is 500MB * Memory is maxed at 1.5GB In the function invocation time window you need to: * retrieve the file (to memory or disk) * transcode the file (outputting to memory or disk) * upload the file (as the disk is not persistent) This along with the following facts make it infeasible: * transfers from S3 are fast, but non-s3 sources are likely to be much slower. * assuming you mean large as in GB - you have nowhere to put the files (disk too small, memory too small). * transfers to S3 are fast, but uploading the transcoded video to a non-s3 source will likely me much slower. Hope that helps.
  - cosmie 11y ago
    I don't know much about video transcoding, but if FFMPEG can utilize streams it's easy to work around the lambda size constraints. You can process several GBs in the 5 minute window by piping your S3 download stream through your transformation steps then directly into an S3 upload stream. Nothing ever persists to disk, so your only worry if anything is managing your stream buffers so you don't run out of memory. As long as any single step of your pipeline doesn't exceed the time limit, you can make really nifty pipelines for large file processing by using the S3 upload as "temp space" then an S3 event to automatically trigger the next step of your pipeline.
    - jdp 11y ago
      ffmpeg can utilize streams, in both input and output. The trouble comes from different codecs and containers, especially on output. Some formats aren't append-only—the prime example being MP4 + h.264—and so ffmpeg needs to be able to write to a seekable output device, ruling out streaming output in those cases.
Rezo 11y ago
Did you try simply asking AWS to raise the limit? It even suggests so on your linked page. In my experience, every limit is immediately relaxed when requested; number of VPCs (I see people do horrible things to work around this all the time! Just ask!), EC2s / region, SES limits (need to send 10 million emails / day? No problem!), API Gateways / account, total ASGs... I believe all of these are there to keep you from shooting yourself in the foot through automation gone wrong or inexperience. I've seen some crazy complicated architectures, where just sending an email or lifting up the phone solves the thing within an hour.
- sithadmin 11y ago
  >number of VPCs (I see people do horrible things to work around this all the time! Just ask!) I'm intrigued.
- brianshaler 11y ago
  I've been surprised/impressed by their quick and painless limit increases. It makes sense to have low default limits so people don't accidentally spin up a thousand instances or send a million emails. It seems the limits are mostly there to protect you from a bug or test in early development costing you a bunch of money.
- SteveNuts 11y ago
  >I believe all of these are there to keep you from shooting yourself in the foot through automation gone wrong or inexperience. Also to prevent you from racking up huge bills in case an api key is compromised, and the attacker is able to spin up tons of instances for a botnet or something on your dime.
- djm_ 11y ago
  No! Thank you for pointing that out; I've always taken "limit" to mean hard but I shall no longer. We had other reasons to move and it did end up working well for that project and others, but I can obviously only say that with hindsight. They need to write that in big bold letters.
- iampims 11y ago
  There are some limits they can't/don't want to increase.
kcorbitt 11y ago
Just want to add my voice to everyone hoping for a writeup. I'm especially interested in the cost comparison.
Veratyr 11y ago
An interesting thing to look into might be using the (apparently Kepler) NVENC capabilities present on EC2 G2 instances. For $2.60 an hour, you get 4x Kepler GPUs that can handle ~4x realtime 1080p encodes each (120fps per GPU), or 16x realtime 1080p encodes total (480fps). To convert this into rather odd units, that works out to ~1.37Tpix/$ (1920x1080 x 480fps x 3600s / $2.6). Put this on reserved instances and that number is pushed up to ~2.2Tpix/$. According to [1], ffmpeg + x264 performance on the most cost effective instances (c3.xlarge) was 20s for a 30s, 960x540 video, or roughly 0.5Mpix at 45fps. That's 83Gpix for $0.21, or 0.399Tpix/$ at spot prices or 0.672Tpix/$ at reserved prices. Depending on how much you care about your compression quality (NVENC isn't quite as good as x264 veryslow but it's definitely usable, particularly with its "two pass" preset), it might be worth a good look at the GPU encoders. [1]: https://github.com/sportarchive/CloudTranscode/blob/master/benchmark/benchmark-aws-ffmpeg.xlsx https://github.com/sportarchive/CloudTranscode/blob/master/b...

by djm_ 11y ago

douglasfshearer 11y ago

How did the costs compare with this setup? I work on a project where ET isn't flexible enough (MPEG-DASH), and wondered whether Lambda would make for a good alternative to EC2 + SQS + Scaling Groups.

bagels 11y ago
EC2 + SQS + ffmpeg is 1/5 the cost of elastic transcoding for me, and that's not even using all the ec2 capacity.

Exuma 11y ago

Wow... I would truly appreciate if you could share a bit more about your specific setup. I found myself working on a new project yesterday that I was really really excited about, until I saw the costs to transcode video. How does doing all this in-house compare price wise (say, per minute), compared to using elastic transcoder? Edit: The ultimate lowest cost I can find is $0.0125-0.015

djm_ 11y ago
I wish I could edit my parent comment but alas. The key point was missed: we were dealing with very short, small videos. If you are dealing with longer or large videos, it's simply not feasible on Lambda. As for costings, unfortunately I cannot retrieve them as this project was mid last year and I've since moved on to other clients. They can be calculated though with a few short tests I'm sure.
deleted 11y ago
[deleted]

manishsharan 11y ago

Could you share your experience with FFMPEG+ Lambda ? I ran into trouble with this when dealing with large files, especially when some of the files were being pulled off non S3 sources. Also what EC2 cores were you using. ?

djm_ 11y ago
Sure: you likely don't want to be dealing with large files on Lambda. Why? * The maximum timeout window for a function invocation is 300 seconds * The maximum available temp disk space per instance is 500MB * Memory is maxed at 1.5GB In the function invocation time window you need to: * retrieve the file (to memory or disk) * transcode the file (outputting to memory or disk) * upload the file (as the disk is not persistent) This along with the following facts make it infeasible: * transfers from S3 are fast, but non-s3 sources are likely to be much slower. * assuming you mean large as in GB - you have nowhere to put the files (disk too small, memory too small). * transfers to S3 are fast, but uploading the transcoded video to a non-s3 source will likely me much slower. Hope that helps.
- cosmie 11y ago
  I don't know much about video transcoding, but if FFMPEG can utilize streams it's easy to work around the lambda size constraints. You can process several GBs in the 5 minute window by piping your S3 download stream through your transformation steps then directly into an S3 upload stream. Nothing ever persists to disk, so your only worry if anything is managing your stream buffers so you don't run out of memory. As long as any single step of your pipeline doesn't exceed the time limit, you can make really nifty pipelines for large file processing by using the S3 upload as "temp space" then an S3 event to automatically trigger the next step of your pipeline.
  - jdp 11y ago
    ffmpeg can utilize streams, in both input and output. The trouble comes from different codecs and containers, especially on output. Some formats aren't append-only—the prime example being MP4 + h.264—and so ffmpeg needs to be able to write to a seekable output device, ruling out streaming output in those cases.

Rezo 11y ago

Did you try simply asking AWS to raise the limit? It even suggests so on your linked page. In my experience, every limit is immediately relaxed when requested; number of VPCs (I see people do horrible things to work around this all the time! Just ask!), EC2s / region, SES limits (need to send 10 million emails / day? No problem!), API Gateways / account, total ASGs... I believe all of these are there to keep you from shooting yourself in the foot through automation gone wrong or inexperience. I've seen some crazy complicated architectures, where just sending an email or lifting up the phone solves the thing within an hour.

sithadmin 11y ago
>number of VPCs (I see people do horrible things to work around this all the time! Just ask!) I'm intrigued.
brianshaler 11y ago
I've been surprised/impressed by their quick and painless limit increases. It makes sense to have low default limits so people don't accidentally spin up a thousand instances or send a million emails. It seems the limits are mostly there to protect you from a bug or test in early development costing you a bunch of money.
SteveNuts 11y ago
>I believe all of these are there to keep you from shooting yourself in the foot through automation gone wrong or inexperience. Also to prevent you from racking up huge bills in case an api key is compromised, and the attacker is able to spin up tons of instances for a botnet or something on your dime.
djm_ 11y ago
No! Thank you for pointing that out; I've always taken "limit" to mean hard but I shall no longer. We had other reasons to move and it did end up working well for that project and others, but I can obviously only say that with hindsight. They need to write that in big bold letters.
iampims 11y ago
There are some limits they can't/don't want to increase.

kcorbitt 11y ago

Just want to add my voice to everyone hoping for a writeup. I'm especially interested in the cost comparison.

Veratyr 11y ago

An interesting thing to look into might be using the (apparently Kepler) NVENC capabilities present on EC2 G2 instances. For $2.60 an hour, you get 4x Kepler GPUs that can handle ~4x realtime 1080p encodes each (120fps per GPU), or 16x realtime 1080p encodes total (480fps). To convert this into rather odd units, that works out to ~1.37Tpix/$ (1920x1080 x 480fps x 3600s / $2.6). Put this on reserved instances and that number is pushed up to ~2.2Tpix/$. According to [1], ffmpeg + x264 performance on the most cost effective instances (c3.xlarge) was 20s for a 30s, 960x540 video, or roughly 0.5Mpix at 45fps. That's 83Gpix for $0.21, or 0.399Tpix/$ at spot prices or 0.672Tpix/$ at reserved prices. Depending on how much you care about your compression quality (NVENC isn't quite as good as x264 veryslow but it's definitely usable, particularly with its "two pass" preset), it might be worth a good look at the GPU encoders. [1]: https://github.com/sportarchive/CloudTranscode/blob/master/benchmark/benchmark-aws-ffmpeg.xlsx https://github.com/sportarchive/CloudTranscode/blob/master/b...