5 ms·

Benchmarking cloud video encoding: Zencoder, Encoding.com, Sorenson, & Panda

43 points by jon_dahl 15y ago

rorrr 15y ago
Sponsored by Zencoder. As much as I want to trust this benchmark, I'm doubtful. If Zencorer knew which account you used, they could've assigned the best dedicated servers to your jobs.
- jon_dahl 15y ago
  Sure, it could be a scam. But for what it's worth, it's not - we absolutely didn't cheat. Cheating in a benchmark might work some places, but it certainly doesn't work for a company that sells to hackers. If you want to verify the results yourself, we're going to post the benchmarking source code to Github this afternoon. You can review the methodology that CloudHarmony used and run the tests yourself, with your own account, without us having any knowledge that you're testing us. That way, we have no way of cheating. You can even change up the files and use different encoding profiles to be sure that we didn't pick specific files or settings that happen to be faster at Zencoder.
  - sologoub 15y ago
    Would be interesting if someone independent could corroborate the results. The sponsorship disclosure is appreciated, but I agree with the original comment - raises red flags.
    - brandonarbini 15y ago
      Here's the source code that anyone can use to independently test and corroborate the results: https://github.com/zencoder/cloudharmony-benchmark https://github.com/zencoder/cloudharmony-benchmark. We chose Cloud Harmony to run the benchmark on the basis of their independent reputation. However, we'd happily sponsor (pay for encoding fees) another third-party to run the tests as well. Feel free to suggest an additional, neutral person/company that could do this. Though the results will be the same, no matter who runs it. :)
  - timc3 15y ago
    And how did you measure the quality of the outputted file? This matters a huge amount, it doesn't matter if you are 4 times as quick if the output is not up to the standards needed by the client.
    - jon_dahl 15y ago
      Definitely - this is important as a control. Every test was done using the same core settings (codecs, profiles, resolution, frame rate, bitrates, two-pass encoding). Beyond that, CloudHarmony used the default settings at each provider. We'll try to publish each video for visual inspection, but suffice it to say that they all look pretty close, especially three of the four, which use the same core encoder (x264) and the same settings. There may be microversion differences and minor variations in settings, but not much. The fourth uses MainConcept, which isn't a bad encoder itself. You'll probably see a bigger spread in audio quality - Encoding.com and Panda rely on faac, I think, which is an abandoned AAC encoder that isn't very good.
  - zimbatm 15y ago
    Thanks for taking the time to do these tests (@panda here). I checked your benchmarks and I can confirm that they are legit on our side. To be fair our customers are not the same type. We mainly have people who trans-code a lot of videos (like DVD libraries) and in that case we are a lot cheaper than zencoder. Our presets also use a much more recent version of ffmpeg since we are not stuck with command-line compatibility issues
- vgurgov 15y ago
  My service relies on Zencoder. before we ended up with them we tested most of these services and build own stuff. result? we are very happy with zencoder as we found similar results. its really fast(you will notice difference once you switch from their competitors) and works all the time. disc: I am not affiliated with zencoder in any way.
  - ceejayoz 15y ago
    Same experience here. My only affiliation is they sent me a t-shirt. Lovely service.
- mikeryan 15y ago
  If you want to prove you're the better service what's your options? Hope someone does the benchmarking for you?
  - wmf 15y ago
    Generally you design a benchmark and pay an "independent" lab to run it and publish the results.
- jread 15y ago
  For what it's worth, I ran the tests and wrote this blog post and can assure you that we made every effort to be fair with the methodology and honest about the results. Take a look at our blog and website. Since 2009, we've released many benchmarks publicly and have a good reputation for providing unbiased, objective analysis.
Jabbles 15y ago
This only measures the speed, not the quality. Encodings may vary enormously in their complexity, you can't treat 2 encoders that produce the same "resolution" as being equal.
- bgentry 15y ago
  That's much harder to do objectively, but I'd love to see some 3rd party evaluations.
- DarkShikari 15y ago
  They (almost) all use the same encoder. Albeit most likely with different settings, for a different speed/quality tradeoff. And some might be using ancient versions from the dawn of the Bronze Age.
spaznode 15y ago
Where's the shout out to ffmpeg? Don't tell me that almost all services involved aren't built on top of it.
- jon_dahl 15y ago
  Here you go: ffmpeg rules. Zencoder, Panda, and Encoding.com use it. Sorenson doesn't. That's part of the reason why our video quality > Sorenson. Also here: http://zencoder.com/opensource http://zencoder.com/opensource. Big shout outs to x264, libavcodec, gpac, mediainfo, lftp, curl, mkvtoolnix, oggz, sox, lame, Ubuntu Linux, C, C++, Ruby, Rake, Postgresql, Redis, Dennis Ritchie, Steve Jobs, vim, emacs, and a hundred others. We couldn't have done it without all of you.
- zimbatm 15y ago
  Big shout out to ffmpeg from Panda. As far as I know, we are the only ones to give you full access to the command-line options in case you want to do funny stuff. The disadvantage is that we have been stuck with ffmpeg 0.6 on custom commands to not break backward-compatibility (which is what they used). If they had used the presets, the encoding would have run with the latest ffmpeg which is also a lot faster.
  - spaznode 15y ago
    Ouch! (regarding getting fsck-ed with command line options) Before current job I built a fairly large more general media management system which included tons of video related work (and encoding ;) ) using ffmpeg for some fairly prominent (something something geographic something / etc) so I definitely appreciate how powerfully awesome ffmpeg is once you get over the initial overwhelming hump of having to learn all the more general video concepts involved outside of ffmpeg. Which is why seeing a "performance comparison" blog post was immediately extremely suspicious sounding for the very valid point that everyone has brought up that speed isn't all that matters. From my experience speed is one of the lowest concerns anyone doing anything professionally with video has. Quality / reliability / tweak-ability are all much more important. If you're writing software you intend to sell for money for anything other than cat videos uploaded to youtube - do yourself a favor and make sure you're informed about all the core concepts before randomly picking someone. You'd be surprised at all the hiccups and encoding issues you run across once you have a large enough sample set of "problem videos" to test against.
gtaylor 15y ago
Bit of a shameless plug, but we wrote our own EC2 + ffmpeg encoder, since we didn't want to pay out the ears for encoding: http://media-nommer.readthedocs.org/ http://media-nommer.readthedocs.org/ Open source (BSD), beta'ish, but very much functional. Save some money! Pull requests are welcome and greatly appreciated, as is feedback on the docs.
- Heff 15y ago
  Very cool. How long did it take you to build that?
  - gtaylor 15y ago
    December 21, 2010 was our start date, so approaching a year now.
- lobster_johnson 15y ago
  A shameless plug on top of your shameless plugin: We also wrote our own encoding system, called Tootsie (think Dustin Hoffman), which is in production use: https://github.com/alexstaubo/tootsie https://github.com/alexstaubo/tootsie Written in Ruby, based on FFmpeg + ImageMagick, handles both video, audio and photos (we use it for photo/thumbnail scaling). JSON API, easy to use. It's designed to run anywhere, EC2 in particular. MIT license.
  - gtaylor 15y ago
    Looks cool, but are there any real central docs to look through (aside from that README)?
    - lobster_johnson 15y ago
      No. The README covers pretty much everything you need to know. I would be happy to explain (and rectify the readme) if there is anything that is unclear, however.

43 points by jon_dahl 15y ago

rorrr 15y ago

Sponsored by Zencoder. As much as I want to trust this benchmark, I'm doubtful. If Zencorer knew which account you used, they could've assigned the best dedicated servers to your jobs.

jon_dahl 15y ago
Sure, it could be a scam. But for what it's worth, it's not - we absolutely didn't cheat. Cheating in a benchmark might work some places, but it certainly doesn't work for a company that sells to hackers. If you want to verify the results yourself, we're going to post the benchmarking source code to Github this afternoon. You can review the methodology that CloudHarmony used and run the tests yourself, with your own account, without us having any knowledge that you're testing us. That way, we have no way of cheating. You can even change up the files and use different encoding profiles to be sure that we didn't pick specific files or settings that happen to be faster at Zencoder.
- sologoub 15y ago
  Would be interesting if someone independent could corroborate the results. The sponsorship disclosure is appreciated, but I agree with the original comment - raises red flags.
  - brandonarbini 15y ago
    Here's the source code that anyone can use to independently test and corroborate the results: https://github.com/zencoder/cloudharmony-benchmark https://github.com/zencoder/cloudharmony-benchmark. We chose Cloud Harmony to run the benchmark on the basis of their independent reputation. However, we'd happily sponsor (pay for encoding fees) another third-party to run the tests as well. Feel free to suggest an additional, neutral person/company that could do this. Though the results will be the same, no matter who runs it. :)
- timc3 15y ago
  And how did you measure the quality of the outputted file? This matters a huge amount, it doesn't matter if you are 4 times as quick if the output is not up to the standards needed by the client.
  - jon_dahl 15y ago
    Definitely - this is important as a control. Every test was done using the same core settings (codecs, profiles, resolution, frame rate, bitrates, two-pass encoding). Beyond that, CloudHarmony used the default settings at each provider. We'll try to publish each video for visual inspection, but suffice it to say that they all look pretty close, especially three of the four, which use the same core encoder (x264) and the same settings. There may be microversion differences and minor variations in settings, but not much. The fourth uses MainConcept, which isn't a bad encoder itself. You'll probably see a bigger spread in audio quality - Encoding.com and Panda rely on faac, I think, which is an abandoned AAC encoder that isn't very good.
- zimbatm 15y ago
  Thanks for taking the time to do these tests (@panda here). I checked your benchmarks and I can confirm that they are legit on our side. To be fair our customers are not the same type. We mainly have people who trans-code a lot of videos (like DVD libraries) and in that case we are a lot cheaper than zencoder. Our presets also use a much more recent version of ffmpeg since we are not stuck with command-line compatibility issues
vgurgov 15y ago
My service relies on Zencoder. before we ended up with them we tested most of these services and build own stuff. result? we are very happy with zencoder as we found similar results. its really fast(you will notice difference once you switch from their competitors) and works all the time. disc: I am not affiliated with zencoder in any way.
- ceejayoz 15y ago
  Same experience here. My only affiliation is they sent me a t-shirt. Lovely service.
mikeryan 15y ago
If you want to prove you're the better service what's your options? Hope someone does the benchmarking for you?
- wmf 15y ago
  Generally you design a benchmark and pay an "independent" lab to run it and publish the results.
jread 15y ago
For what it's worth, I ran the tests and wrote this blog post and can assure you that we made every effort to be fair with the methodology and honest about the results. Take a look at our blog and website. Since 2009, we've released many benchmarks publicly and have a good reputation for providing unbiased, objective analysis.

Jabbles 15y ago

This only measures the speed, not the quality. Encodings may vary enormously in their complexity, you can't treat 2 encoders that produce the same "resolution" as being equal.

bgentry 15y ago
That's much harder to do objectively, but I'd love to see some 3rd party evaluations.
DarkShikari 15y ago
They (almost) all use the same encoder. Albeit most likely with different settings, for a different speed/quality tradeoff. And some might be using ancient versions from the dawn of the Bronze Age.

spaznode 15y ago

Where's the shout out to ffmpeg? Don't tell me that almost all services involved aren't built on top of it.

jon_dahl 15y ago
Here you go: ffmpeg rules. Zencoder, Panda, and Encoding.com use it. Sorenson doesn't. That's part of the reason why our video quality > Sorenson. Also here: http://zencoder.com/opensource http://zencoder.com/opensource. Big shout outs to x264, libavcodec, gpac, mediainfo, lftp, curl, mkvtoolnix, oggz, sox, lame, Ubuntu Linux, C, C++, Ruby, Rake, Postgresql, Redis, Dennis Ritchie, Steve Jobs, vim, emacs, and a hundred others. We couldn't have done it without all of you.
zimbatm 15y ago
Big shout out to ffmpeg from Panda. As far as I know, we are the only ones to give you full access to the command-line options in case you want to do funny stuff. The disadvantage is that we have been stuck with ffmpeg 0.6 on custom commands to not break backward-compatibility (which is what they used). If they had used the presets, the encoding would have run with the latest ffmpeg which is also a lot faster.
- spaznode 15y ago
  Ouch! (regarding getting fsck-ed with command line options) Before current job I built a fairly large more general media management system which included tons of video related work (and encoding ;) ) using ffmpeg for some fairly prominent (something something geographic something / etc) so I definitely appreciate how powerfully awesome ffmpeg is once you get over the initial overwhelming hump of having to learn all the more general video concepts involved outside of ffmpeg. Which is why seeing a "performance comparison" blog post was immediately extremely suspicious sounding for the very valid point that everyone has brought up that speed isn't all that matters. From my experience speed is one of the lowest concerns anyone doing anything professionally with video has. Quality / reliability / tweak-ability are all much more important. If you're writing software you intend to sell for money for anything other than cat videos uploaded to youtube - do yourself a favor and make sure you're informed about all the core concepts before randomly picking someone. You'd be surprised at all the hiccups and encoding issues you run across once you have a large enough sample set of "problem videos" to test against.

gtaylor 15y ago

Bit of a shameless plug, but we wrote our own EC2 + ffmpeg encoder, since we didn't want to pay out the ears for encoding: http://media-nommer.readthedocs.org/ http://media-nommer.readthedocs.org/ Open source (BSD), beta'ish, but very much functional. Save some money! Pull requests are welcome and greatly appreciated, as is feedback on the docs.

Heff 15y ago
Very cool. How long did it take you to build that?
- gtaylor 15y ago
  December 21, 2010 was our start date, so approaching a year now.
lobster_johnson 15y ago
A shameless plug on top of your shameless plugin: We also wrote our own encoding system, called Tootsie (think Dustin Hoffman), which is in production use: https://github.com/alexstaubo/tootsie https://github.com/alexstaubo/tootsie Written in Ruby, based on FFmpeg + ImageMagick, handles both video, audio and photos (we use it for photo/thumbnail scaling). JSON API, easy to use. It's designed to run anywhere, EC2 in particular. MIT license.
- gtaylor 15y ago
  Looks cool, but are there any real central docs to look through (aside from that README)?
  - lobster_johnson 15y ago
    No. The README covers pretty much everything you need to know. I would be happy to explain (and rectify the readme) if there is anything that is unclear, however.