66 Responses to »Video encoder comparison«

• astrange (2010-02-25 23:33)

  libavcodec will encode much faster, and probably not much worse, without “qns=3”. Unfortunately, tuning lavc encodes is really difficult (pretty much all the options need to be changed from defaults, you’ve only changed 75% of them), so I can’t complain too much.
  You might want to try “vb_strategy=2:vmax_b_frames=4:cmp=262:subcmp=262:dia=770” – the last three being magic for rd+chroma+UMH motion search.

• yuvi (2010-02-25 23:38)

  Out of curiosity, why did manually specify x264 settings somewhere between preset slower and veryslow? –slow-firstpass in particular almost doesn’t improve quality at all but is a low slower. And dirac encoders don’t support two pass, so you can’t really compare their speed in a single pass to encoders that do two passes.

• chris (2010-02-26 01:32)

  for theora, you shouldn’t try to use bitrate number at all, use quality number instead and check the quality.

  setting a bitrate using ffmpeg2theora will only produce CBR files, soft-target does not help. Only quality number (1-10) seems to be able to do VBR in theora.

• David Schleef (2010-02-26 03:34)

  You should be using Schroedinger, not dirac-research. The latter is a research encoder and generally doesn’t have sane defaults. At SD sizes, Schroedinger is pretty consistently somewhere between a good MPEG-4 ASP encoder and x264.

  If you had difficulties using Schroedinger, you could have asked.

  (Also, “–mv-prec 1/8” WTF? I’m not surprised you got bad results.)

• Gregory Maxwell (2010-02-26 04:17)

  It’s unfortunate that you didn’t use the most current Theora encoder… The 1.1 release is known to perform poorly on SSIM, the current encoder work results in an across the board SSIM improvement of about 2dB. A marked improvement, certainly, though x264 is still a more advanced encoder (and one extensively tuned against SSIM).

  Would you mind making all the files available (the input file and all your encodes?) I can provide hosting if you’re short on internet-connected disk space.

• KeyJ (2010-02-26 08:19)

  yuvi: That’s simple: I wanted to have a sane amount of reference frames (i.e. not 16, more like 4 ;)

  chris: So you’re saying there’s no way to do proper bitrate-controlled two-pass encoding for Theora, at least not with ffmpeg2theora? That’s a pity.

  David: I tried to use schroedinger (using ffmpeg2dirac --use-schro), but it failed to generate any usable bitstreams. Regarding the Dirac options, I must admit that I don’t have any clue which settings make sense and which don’t, and since diracvideo.org was down when I did the test, I didn’t even have access to the spec. I used -mv-prec 1/8 because for most other codecs, more sub-pel precision is a good thing – again, I didn’t know that this would hurt Dirac. You’re welcome to suggest better settings.

  Gregory: I used the Theora encoder from SVN trunk as of 2010-02-04, not the 1.1 release. I just assumed that this is the latest and greatest version – or should I have used a particular branch instead?

  I’m planning to publish the scripts I used in an update of this post anyway. I also thought about making at least a few example encodes available here, but I’m my disk quota here is low. Do you really think it makes sense to publish all encoded streams?

  @all: Thank you for your feedback (and given that the post has been online for less than half a day, I guess there will be much more soon :). If I got enough input, I might eventually do a re-test with recommended settings from the developers.

• Gregory Maxwell (2010-02-26 10:09)

  I’d somehow missed the part where you said you used the most current versions, but it sounds like I wasn’t incorrect by sheer dumb luck:

  Theora development isn’t on Theora trunk, never has been except for release candidates. Current development is theora-ptalarbvorm. I suppose it’s just as well that you didn’t us it on 2010-02-04 ptalarbvorm had a bug that produced corrupted bitstreams on some inputs.

  Though, it looks like some bad decisions with rate control are holding back the improvements on this clip: the SSIM improvement that I’m seeing is only 0.44dB. (Though this is with an SSIM tool which is more faithful than the AviSynth one; YMMV)

  Now knowing that you’re using the most current x264— you shouldn’t be surprised to see it SSIM nailing libtheora even in baseline mode. The temporal RDO feature in current x264 (I guess it’s called “mbtree” there?) provides a pretty big performance boost which is independent of the profile used. An analogous feature has been on the TODO for Theora for a long time. x264 has about as many lines of x86 assembly as the entire size of the libtheora encoder’s C code— it’s still simply a more advanced encoder, regardless of the format qualities. Technically Theora and baseline h264 are broadly similar from a coding-tools perspective, so we should expect the performance to converge over time.

  A lossless copy of the input clip and one or two representation outputs from each of the formats would be nice, at least then any differences between your results and local results could be isolated to encoding vs measurement related.

  How’d you manage the 800×336 clip? The original is 1920×800. 800×336 is either cropped or distorted.

• KeyJ (2010-02-26 10:23)

  Gregory: Good to know that there’s a more up-to-date Theora branch, even though it’s very well hidden ;) If I repeat my tests at some later time (which is highly probable), I’ll use that instead of the normal Theora trunk.

  Regarding the SSIM tool, I just wanted something that works and is at least a little bit respected in the community. I’m also a bit suspicious about the so-called »improvements« they made, but I didn’t find another implementation, so I just used that one.

  The 800×336 clip is indeed a tiny bit distorted, but the difference in aspect ratio is just 0.8%, which is acceptable in my opinion.

• Dapper (2010-02-26 11:40)

  You said “The [x264] developers claim that it’s easily on par with commercial implementations”. This is a very misleading statement as I think you’ll find that they, and many other people, think it is by far the best H.264 encoder available in software or hardware. The way you state it, it appears to be implied that because it is open source it must naturally be inferior.

  In one recent test done by an x264 developer (on anime material) he found that x264 base profile beat every other encoder in the test including 4 proprietary H.264 encoders using their best quality settings. So saying they think their encoder is on par vastly overstates the quality of the average H.264 implementation and makes it seems odd that you’d be surprised that Theora would be in that same position.

  Obviously this makes it tricky to compare H.264 with another format, when one implementation can be 4x better than another widely used one. I just think it’s a shame that Theora keeps getting called crappy, with the implication that the developers are idiots, just because one single implementation of H.264 happens to be so very good. They’re working within very different constraints, most notably one is treading carefully through the patent minefield where the other isn’t. To overlook that is like reporting on a boxing match and failing to mention that one fighter had a hand tied behind his back.

• KeyJ (2010-02-26 13:35)

  Dapper: I think you misunderstood me. It was not my intention to depreciate x264 or open source software in general. After all, I did this whole test with OSS exclusively, so how could I be against it?

  I also want to distantiate myself from the »Theora is crap« camp. Even though I’m underwhelmed by its performance in my test, I absolutely appreciate the work that guys like Monty and Gregory put into it. I know that it’s very hard to put a (relatively) old format like VP3/Theora in such a good shape over such a short period of time, and the developers have done a tremendous job in doing so. (Oh, and by the way, this has nothing to do with avoiding patents – this has already been taken care of by On2 and Xiph.org in the initial specification of VP3 and Theora. The true problem is living with the restrictions of the format which are a result of patent avoidance.)

• perrabyte (2010-02-26 13:41)

  Energy efficiency of an encoder is much more important when you have achieved quality and compression that is good enough.

  This is important since everything now goes mobile and you want your battery in the phone, video-camera, ipad or laptop to last as long as possible.

• KeyJ (2010-02-26 13:55)

  perrabyte: Correct. But contrary to what you might think, that’s a point for H.264, not against it: When encoded or decoded on a general-purpose CPU, H.264 is indeed slower (and thus, requires more energy) than most other codecs, except the wavelet ones (like Dirac). But hardware acceleration for H.264 is commonplace, which makes encoding/decoding again much more efficient than any pure software encoder/decoder.

• Dapper (2010-02-26 14:42)

  Thanks for the response. Note that patents not only affect codec specifications. Much of the benefit you get after the spec is frozen is due to choosing what to encode and how many bits to use. I’m not a lawyer but I think it’s clear that a patent could easily cover these optimization techniques without it being required to implement a different, but still standards compliant, encoder or a decoder that plays back the result of either. (Apparently the On2 patents for VP3/Theora don’t cover the format itself, just optimizations.)

  I realize this will sound like FUD but a complex and featureful H.264 encoder such as x264 is more likely to step on a non-MPEG-LA patent than an H.264 encoder that sticks to the basics. Do the x264 devs care? I doubt it as they don’t worry about the well known MPEG-LA patents either.

  But the Theora devs have to, because that is the very reason for the codec to exist. I’m fairly certain it was mentioned on their mailing list that they needed to check out a technique they intended to add to the encoder for patent risk within the last few months, though I can’t google up a link at the moment.

• Adam (2010-02-26 17:09)

  Very nice article, would have been great to see vc-1 in this comparison as well. You can use the free version of microsoft expression encoder to do vc1 encoding.

• david espenlaub (2010-02-26 19:50)

  Would you please share what was the length of the Star Trek clip that you used in your testing. It might make some of the other time measurements more interesting.

• perrabyte (2010-02-26 20:09)

  KeyJ:

  My observation is that hardware H.264 encoding/decoding capabilities is not commonplace. My netbook certainly does not have it. Watching H.264 in any reasonable resolution is impossible. Progress is made so fast these days that when true hardware support is commonplace it is no longer relevant. We would then have moved on to a different codec. In the end it will be to costly to implement special hardware support just for a codec in my opinion.

  Im in favor of using both Theora and H.264 in web video so we can have some evidence on how well hardware specific support really works out.

• anon (2010-02-27 07:47)

  Considering the comments here, it would be great if you would update your findings and provide the source video you used as well. It would be especially important if they show considerable difference and would better assist people in fairly judging the value of the free codecs.

• nurbs (2010-02-27 15:12)

  I also think the x264 settings that were used aren’t really useful if you want compare encoding speeds. By leaving away –slow-firstpass alone you almost double the encoding speed with basically no loss in quality (definitely smaller than 1%). –subme 10 can be switched out for 9 and –trellis can be left at the default for even higher speed gain, although at more quality loss. With that you’d get encoding times much closer to XviD and Theora.
  –b-pyramid strict only exists for Blu-Ray compatibility and isn’t normally used for other scenarios, since it has worse quality than –b-pyramid normal. That shouldn’t have much influence on speed though.

  I’m also surprised that you didn’t use either the “film” or “SSIM” tunings of x264. I’d probably have used “film” since I see the subjective quality as more important than metrics in any codec comparison.

• nurbs (2010-02-27 15:46)

  I forgot to mention that this comparison also illustrates how you shouldn’t rely too much on metrics. For instance the 500 kbps x264 baseline encode has much lower metrics than the 1000 kbps Theora encode, but the x264 image looks more detailed. Of course a sample video would be nice how well it holds up in motion, but I guess the difference would still hold.

• creamyhorror (2010-02-27 17:54)

  Honestly, for screenshot comparisons of close quality levels, it’s actually very important to make sure the frame types are the same. A p-frame will look much better than a b-frame due to the higher no. of bits allocated, so a single p-frame (say for x264 Main Profile) in the comparison could throw everything off when it’s compared to a b-frame representing x264 High Profile.

  Anyway, nice test setup and results, although it looks like some refining of the encoder settings might be in order.

• Brett (2010-02-28 18:19)

  If you scale this webpage down so that the images are cell phone sized, then there is little difference between the original and any of the codecs, especially in action scenes. He he.

• KeyJ (2010-02-28 18:43)

  Adam: I’m trying to find a way to include VC-1 in the next round. Microsoft offers the VC-1 Encoder SDK for free, and I guess I’ll just write a simple encoder application with that and run in through WINE.

  David: It’s 3239 frames at 24 fps (or was it 23.976? Please forgive me the 0.1% error ;).

  perrabyte: Somehow your second comment concerns a totally different topic than the first one. The first one was about energy consumption on mobile devices, the second one was about computing power in netbooks. You’re totally right, though – not having proper H.264 acceleration in (relatively) large netbooks while every modern smartphone does have it really sucks :(

  nurbs: Thanks for the input. I might even include two different sets of x264 settings in the next round: A very slow, but very high quality one, and a faster but potentially worse one.

  creamyhorror: You’re right. However, it’s hard to find a frame which is encoded as a P-Frame with seven different encoders, yet is meaningful enough to make the differences visible well enough. Furthermore, P vs. B doesn’t matter all that much: Since P and B frames are (almost) alterating in most cases, encoders do their best to make sure that there are no large quality differences between them – otherwise, they would produce very annoying temporal noise. The only thing to avoid is I-Frames, because these do have a noticeable different quality level. The two frames I picked were likely no I-Frames on any encoder, by the way, because they were very close to the next scene change.

• Esurnir (2010-03-01 16:05)

  protips : X264 already got very sane preset that are very good at determining what’s “good” and what isn’t. Like your slow-firstpass probably doesn’t even bring 1% advantage compared to a fast first pass.
  To use the preset put –preset ultrafast,veryfast,faster,fast,slow,slower,veryslow,placebo. The “medium” so to speak preset is when you don’t enter anything and placebo (which is the only preset using –slow-firstpass) doesn’t bring 1% of an advantage to veryslow, leave it out if you want the comparison to be timely

  use –tune ssim for x264 so that it disable psycho visual optimisation that “may” hurt ssim (but provide a better visual appearance).

  your command line then would work like :
  CMD=”x264 –fps $FPS –preset –tune ssim –stats pass1.tmp”

  I advise you also take the latest git for x264 (and maybe provide the version number in the test).

  If you don’t like preset, just input –full-help and paste what you want.

  Oh one last thing, the difference between high profile and main profile isn’t as big because the only thing deactivated in high profile using your settings is –8x8dct, that’s a very useful settings that is a default and only deactivated in ultrafast because of it’s usefulness, but it’s not a “complete revolution” compared to say disabling CABAC (10% or more bitrate for -free- http://akuvian.org/src/x264/entropy.png ) when you switch to baseline or no bframe etc.

  last, your right about Pframe vs Bframe not mattering, an encoder should look the most consistent as possible over the encode, jump in quality in a P frame or worse an I frame will look horrendous to the viewer (for an example of an horrible encoder with a behavior like that, see the white house website live streaming codec they use, I switched to cspan website instead because my eyes were bleeding).

  If your encoder jump quality between frame type, it’s doing something horribly wrong.

• Esurnir (2010-03-01 16:14)

  If I remember correctly, qpel doesn’t help in xvid save in anime.

  GMC however would probably help since it’s an additional mode to the encoder (which would only be selected if it help)

• KeyJ (2010-03-01 16:20)

  Esurnir: Thanks, I already got that --slow-firstpass was a bad idea ;) Regarding the presets, I’m also convinced now that just picking --preset slow (or --preset slower in combination with a sane amount of reference frames) would have been the right way to go.

  Regarding the differences between the profiles, I’ve already come to the same conclusions as you did (though I didn’t mention them in the article ;).

  Good point about qpel, I should check that in a brief pre-test before doing the next round. GMC is out of the question though, because I see no point in generating files with features that no hardware decoder supports :)

• Manabu (2010-03-02 23:38)

  Very good independent comparison, despite the many flaws already pointed. That is very dificult to come by nowadays, ever since Doom9 stoped his. I’m eargely waiting for an next installment with the corrections sugested.

  It would be interesting that you include two sets of results, one more focused on compression, other in speed. For x264 it would be for example one with –preset slower (to see something close to maximum sane quality), and one with –prest faster, for an idea of what busy encoding farms use. But this will likely double your work and make the graphs too dificult to read… so I don’t know if it is worth for every codec. Maybe only for x264 that has an broader range of speeds.

  But I’m afraid that –tune ssim may give an unfair advantage to x264, as other codecs don’t have an option to artificially tune itself to the ssim quality metric. Most of them are actually tuned based on psnr. I think that just leaving w/o any extra metric specific tune is the best.

  And I don’t thing anyone would opose to dropping main vs high profile comparion in the next installment, as that will not really change from this one.

  And it is likely an good idea to put the Y axis of the graphic in logaritimic scale, as the SSIM itself.

• Esurnir (2010-03-04 07:03)

  If you want a sane ammount of Reference frame (like target a certain ammount of hardware compatibility) add –level 3.1 for SD or –level 4.1 for HD, it’ll reduce the number of ref frame accordingly.

• TravisO (2010-03-12 15:28)

  The amount of detail you went into was great, and the image comparison is amazing in it’s simplicity and it’s usefullness. This is by far the best codec comparison I’ve ever seen in my life. This article is a shiny example of the perfect and fair comparison.

• R.T. Systems (2010-03-12 15:57)

  I’m pretty sure it’s not fair to compare these codecs if the key intervals aren’t set the same. For example, x264 default keyint is 250, but it is 64 in Theora and you set it to 18 in MPEG-2.

• KeyJ (2010-03-13 12:07)

  R.T.Systems: The differences are fully intentional – I wanted to have the bitstream parameters of each codec to be close to the typical settings. For x264 and XviD, this means very long GOPs, for MPEG-2 it’s normal to have short GOPs (in fact, 18 frames is already quite long for MPEG-2). Regarding Theora, who am I to argue with the default settings? :) Anyway, 64 seems to be a sane value for that codec, considering it doesn’t have B frames. Moreover, I doubt that this has any serious influence anyway, because the test clip has mostly short scenes (much less than 2 seconds).

• Ernst (2010-03-13 13:32)

  What about –speedlevel 0 for ffmpeg2theora?

• Calc-Yolatuh (2010-03-17 23:36)

  You could have stood one more graph: quality vs decode-time.

  A properly coded b-frame should look great, since it only builds on the progress of preceding frames. A b-frame alone is a blocky, distorted, discolored mess oddly enough.

  On acceleration: Some of us may still remember when laptops shipped with DVD decoder chips, and graphics cards advertised hardware decoding.

  More and more new netbooks have no trouble up to 720p, some can handle 1080p high-profile with less than 50% CPU.

• Rob (2010-03-18 12:12)

  The VC-1 codec is sorely missing in this list.
  As a popular streaming codec it should really feature.
  As a popular streaming codec it is supposed to combined high quality video with excellent client side decoding performance.

• Esurnir (2010-03-21 00:35)

  @Rob: Since the test are done on linux you would need a VC-1 open source encoder, all VC-1 encoder today are Windows only.

• Esurnir (2010-03-21 01:47)

  Beside the test were done at “offline encoding” speed. If you want a “Streaming test” you would need to have a test done so that every encoder encode at arround 30 fps.

• Mathias Bynens (2010-03-23 12:30)

  Let’s wait and see how VP8 performs.

• Fgd (2010-03-23 17:03)

  In my experience ffmpeg2theora is not suitable for this kind of comparisons as it produces inferior (read: awful) results. Using a Gstreamer pipeline has proved to produce much better results.

• cary Fitzhugh (2010-03-23 18:32)

  I may have missed this, but I think the conclusion misses the real issue in the whole debate.

  There is no debate that an x264 video will most likely look much better than a theora encoded video. It’s more mature, etc. etc.

  But that’s not the issue. It’s that MPEG-4LA reserves the right to charge end users to use their intellectual property. Theora is pretty free, if you can write an encoder/decoder you can use it anywhere. h.264 is not even remotely the same. It is owned by MPEG-4LA, and that group controls the rights. If they decide in 4 years to triple the license price for h264 encoding, then they will have the ‘right’ to.

  A better comparison would be to say, look – here’s theora, and it’s free, and it’s pretty good. Here’s h264, it’s more mature and better in some ways, but at any point in the future they can charge you through the nose.

• KeyJ (2010-03-23 19:06)

  Mathias: I’d like to see that, too. But unless something miraculous happens, I guess On2 won’t make an encoder available for the public again :(

  FGD: Thanks for the hint, I’ll check this. I just hope that you are not right, because I find gstreamer too complex, but that’s a matter of personal taste :)

  Cary: Maybe you didn’t read the first paragraph in the post properly – it clearly states that I don’t give a f*** about that political bullshit. Sorry to be so drastic, but it’s my clear intention to keep this pointless debate out of here.

• Nicholas (2010-03-28 08:46)

  Since it is understandable you are wary about the bandwith, maybe you can upload the test clips to a file hosting site like Megaupload: http://www.megaupload.com/

• DaVince (2010-03-29 15:32)

  When will you do a new comparison? It’s obvious from the comments that the current one isn’t all that accurate anymore…

• KeyJ (2010-03-29 15:40)

  DaVince: I guess it will be around May, but I won’t promise anything :)

• teraflop (2010-04-07 17:06)

  Very nice, about what I expected, except that I am surprised that MPEG2 is still almost competitive relative to XVid… and it seems that benchmarks where Theora can seriously compete with X264 are most likely seriously flawed.
  Btw.,if you want some relatively fast settings for X264 with decent quality, try these (with high profile):

  -b 4 -m 6 -r 4 –no-chroma-me –me dia -A “p8x8,b8x8,i8x8” –no-dct-decimate –no-mixed-refs

  I am not sure how they compare to XVid, but they should be only a bit slower, if at all, but with a much better quality rating at ~120% or so.

• Ross (2010-04-13 22:02)

  The biggest question I have is how do all these compare to VP8 now that Google is likely to open it up, and use it for HTML5.

• KeyJ (2010-04-14 07:49)

  Ross: That’s the big question, no doubt. I will include VP8 in the next round of decoder tests for sure. If the guys at On2 really did their homework, VP8 should be near x264 Baseline Profile in terms of efficiency.

• CableCat (2010-04-25 01:12)

  This was a very nice comparison. I have been doing the same, and I come to the same conclusion as you. Theora requires about 2 times the bandwidth to reach the same quality as x264. Dirac is just useless, except for almost lossless compression.

• pafnucy (2010-04-26 09:37)

  It’s interesting to see how small the SSIM gain becomes at bitrates of over 1500kbps (for x264). I realize that you have used a downscaled sample here, but I wonder how does it look like for 720p or 1080p. People like to toss around 4000kbps as a recommended bitrate for 720p, but is it really necessary?

• nurbs (2010-04-26 11:26)

  You have to be careful when comparing SSIM, because when you get closer to 1 smaller differences in SSIM represent relatively larger differences in quality. For instance 0.98 is twice as good as 0.96, which is twice as good as 0.94. Generally you should use 1/(1-SSIM) if you want the differences on a linear scale.

  About bitrates: I generally encode my blu-rays downscaled to 720p (actually 1280*xxx; depending on the aspect ratio of the movie) with –crf 21 in x264, which lets the codec decide the bitrate based on the complexity of the video. The quality is good enough for me, and while I can tell the difference when I’m looking for it it’s nothing I’d notice when I’m just watching the movie.
  With that setup I get bitrates mostly between 2 and 4 Mbps, so depending on the content and the level of quality you are satisfied with 4 Mbps is probably a reasonable bitrate for 720p if you want rather good quality on most sources. Stuff like 300 or Planet Terror will probably take more than that if you want transparency to the source. Generally grainy or old footage takes more bitrate, while the new stuff shot with digital cameras can be compressed better because there is less noise and grain.

  Edit: Damn it, I meant: For instance 0.98 is twice as good as 0.96, which is twice as good as 0.92.

• KeyJ (2010-04-26 11:47)

  pafnucy: I’m already thinking about doing the next comparison in 720p anyway, so we’ll find out.

  nurbs: In fact, I use --crf for my personal encodes all the time, because it saves you the second pass without sacrificing quality – the only downside is that you don’t have any kind of direct control over the target bitrate, but since I’m usually encoding files for online distribution or hard drive storage, this isn’t an issue. For the codec comparison, however, two-pass encoding is the only way, because I want a specified target bitrate there.

• nurbs (2010-04-26 17:57)

  Yes, –crf is pretty useless for comparisons like this one, but for archiving your DVD collection on a hard disk it’s great. You can just reuse the settings for more or less every encode, maybe with a little higher CRF for high resolutions or when encoding cartoons, you save time and you get the quality you want.

  A common mistake people make is picking a CRF value and then changing an option to see how the filesize changes and then using the size difference to determine how much that option changes compression efficiency. That’s because the simple explanation for CRF encoding is that it produces constant quality, but the options used change the way the rate factor is calculated. So you pick a slower option, get a larger filesize and end up being confused. This is particularly apparent if you turn of the psy-stuff, for instance by using –subme 5 and comparing it to the defaults, where you end up with higher encoding speed and lower filesize.

• pafnucy (2010-04-26 18:39)

  Nurbs: thanks for your answer, it was very informative.
  KeyJ: definitely looking forward to it!

  I am also curious how well animated sources are compressed. I have come across http://x264dev.multimedia.cx/?p=102, but the source used seems to have been SD as well and the output bitrate is at just 250kbps. If it’s not too much of a hassle, could you include a HD cartoon source in the next instalment as well?

• CableCat (2010-04-26 22:09)

  For your next comparison, you should use an uncompressed video as source.

  See: http://media.xiph.org/

• KeyJ (2010-04-27 07:53)

  CableCat: I’d absolutely love to use losslessly compressed video as source material for the test! But alas, there’s no proper material available that I know of. What I need for the test is:

  1. freely available
  2. 1080p/24
  3. running time of a few minutes
  4. good mix of static, high-detail scenes and action scenes
  5. typical content with real, photographed/filmed images

  As cool as »Big Buck Bunny« and »Elephants Dream« might be, they barely fulfill point 4 and they absolutely miss point 5. When comparing codecs, real-world images, CG and cartoons are three completely different worlds. I’d like to focus on real images, but if I was going to include CG imagery as well (I’m not sure yet), I’ll certainly include »Big Buck Bunny«, as it’s just perfect for that case.

• nurbs (2010-04-27 11:01)

  As a lossles real world video you could use Tallship. It was used on the x264 blu-ray demo disk so I guess it’s free to use. Only problem is that it’s 1080i30 so it fails your second criterion, plus there isn’t that much action. The x264 people used avisynth to convert it to 720p60/1.001 for their demo.

  Video: http://video-test-sequences.hexagon.cc/torrents
  Audio: http://cid-bee3c9ac9541c85b.skydrive.live.com/browse.aspx/.Public/LadyWashington
  Demo disk info: http://x264dev.multimedia.cx/?p=328
  Avisynth script: http://forum.doom9.org/showpost.php?p=1394744&postcount=622

• Rob (2010-05-19 13:25)

  This page contains some very high quality source material:
  http://www.microsoft.com/windows/windowsmedia/musicandvideo/hdvideo/contentshowcase.aspx

• Rob (2010-05-19 13:28)

  Here some more:
  http://www.hdgreetings.com/other/ecards-video/video-1080p.aspx

• Patrick (2010-09-10 06:48)

  Very good review. I hope you will update this every half a year but I guess it’s cumbersome. Thanks for all the information though!

• Ross 2 (2010-09-29 10:05)

  Congrats on a good review. libvpx is now available, and so is an updated schroedinger 1.0.9 which would be good to add to the graph. I am more interested in 1 pass encoding for real time streaming applications so I’m trying to build my own test rig for that.

  Something which confuses me about codec authors: If you develop a codec, you want to make sure it performs better than what’s already available. But they don’t seam to do that, at least its not advertised. What would be great is a standard test system where every codec can be run against a set number of ‘useful’ video clips and the output plotted on a graph. Interested?

  Also I use RAW 1080p clips from http://media.xiph.org/video/derf/ which are ok for what I do at the moment. If you need minutes worth of RAW 1080p video it might be worth joining all the clips together?

• Gulfport Chiropractor (2011-01-12 18:51)

  I’d like to see WebM (V8) compared. now that chrome is dropping support for closed source codecs.

• jojo (2011-05-01 14:15)

  Very nice post, I surely love this site, keep it up.

• Stjepan Brbot (2011-05-16 21:36)

  I don’t understand one thing. You said that you were interested in algorithmic complexity of CODEC so you added times from two encoding passes and subtracted decoding time!? Why?

• KeyJ (2011-05-17 07:55)

  Stjepan: That’s not quite the case. What I do is make three encoding runs and take the minimum time. This way, I select the run in which external influences (interruptions from other processes, cold caches, swapping etc.) were minimal, because this is closest to pure CPU time, but still has a relation to real (»wall clock«) time.

  Subtracting the decoding time means subtracting the time required to decode the video input for the decoder, which is the same across all encoders. This has nothing to do with the time required to decode the encoded video.

• lfarkas (2012-05-15 10:02)

  is there any chance to a new updated test? after 2 years it’s be very useful for everyone. what i really like to know which is a reasonable quality with fast encode and decode.

• eXo (2012-12-28 02:29)

  Very nice article. I had not ventured into codec comparisons since the days of divx/XviD
  An update for 2013 would be much appreciated
  Tank you

• Ryan (2016-03-13 07:53)

  Could do this for H.265

• Angel G. (2016-06-29 22:40)

  Now, in 2016, we’ll be pleased to see an update of this excellent article, including the now new kids on the (macro)block – VP8, VP9, HEVC;
  Also we’d like to have hardware h.264 implemantations (VA-API,VDPAU) compared;
  Single pass “low latency” (videoconference) profile encodings also are good to compare, because many people prefer the HW encoding without taking into account how inferior is this for the quality.

Post a Reply