HTML5 video codecs

04 Feb 2011
html5

HTML5 video is an element introduced in the HTML5 draft specification for the purpose of playing videos or movies, partially replacing the object element. The current HTML5 draft specification does not specify which video formats browsers should support in the video tag. User agents are free to support any video formats they feel are appropriate.

365 days ago, I found a nice article from an x264 developer. He successfully pointed out the Adobe mistakes about Flash and analyzed the future of internet video, including Google’s VP8 opening. That article led to another, The first in-depth technical analysis of VP8, and its response, An analysis of WebM and its patent risk – updated.

Short story

H.264/MPEG-4 Part 10 or AVC (Advanced Video Coding) is a standard for video compression, and is currently one of the most commonly used formats for the recording, compression, and distribution of high definition video, broadcasters, independent developers, video artists, hardware, pro and consumer software, they all use/support it.

Google bought On2 and its VP8 codec. Theora is a free and open video compression format from the Xiph.org Foundation and it is being distributed without licensing fees.Theora was created by a group of open-source developers based on the VP7 codec, an implementation also done by On2 Technologies. Theora is an open video codec. VP8 was a commercial product, licensed by On2.

WebM is an audio-video format designed to provide a royalty-free, high-quality open video compression format for use with HTML5 video. The project's development is sponsored by Google. A WebM file consists of VP8 video and Vorbis audio streams, in a container based on a profile of Matroska. The project releases WebM related software under a BSD license. Google is now building a community around WebM (similar to that around Theora), but it hasn't taken any steps to submit WebM to ISO, ITU, or SMPTE for formal open standardization.

HTML5 started to be supported by all the major browsers:

  • Mozilla Firefox (3.5 and later) supports Theora video and Vorbis audio in an Ogg container. Firefox 4 also supports WebM (and H.264 through a Microsoft plugin).
  • Opera supports Theora video and Vorbis audio in an Ogg container and WebM.
  • Google Chrome supports Theora video and Vorbis audio in an Ogg container. Google Chrome 6.0 also supports WebM. (Chrome dropped H.264 support)
  • Safari on Macs and Windows PCs will support anything that QuickTime supports. This is a long list, but it does not include WebM, Theora, Vorbis, or the Ogg container. However, QuickTime does ship with support for H.264 video (main profile) and AAC audio in an MP4 container. I like the Safari approach. Let the users decide. The player can play every installed system-wide.
  • Mobile phones like Apple’s iPhone and Google Android phones support H.264 video (baseline profile) and AAC audio (“low complexity” profile) in an MP4 container.
  • Adobe Flash supports H.264 video (all profiles) and AAC audio (all profiles) in an MP4 container. Adobe Flash will support WebM.
  • Internet Explorer 9 will support H.264 video and AAC audio in an MP4 container.
  • Internet Explorer 8 has no HTML5 video support at all, but virtually all Internet Explorer users will have the Adobe Flash plugin.

Thoughts

Approximately a month ago I read about Google dropping H.264 from Chrome - Ars Technica. Chrome will support only WebM and Theora. I was puzzled and I started on checking which codec every browser would support. Until then I thought everyone would follow the Safari’s approach. It 's not an easy task but it can be done. Only Safari is mainly targeted to Mac OS with QuickTime pre-installed. The OS media playback engine like DirectShow on Windows and GStreamer on Linux would be a solution.

Why my browser should become a video player? For the web to be cross platform consistent, you can say. I agree. Standards serve a purpose. Standards made the web the consistent experience we have now (years after). I don't like the idea my browser to be a video player though. I mean a fully functional 2-3 format video player and me in the middle having to decide which browser to use based on video performance. I want my browser "snappy" and modular.

Browsers are graphics viewers. Well, the case is that we are talking about video here. Image “playback” is a low-resources function opposing to video playback and I think a more fundamental one. Video is not a construction prerequisite to build a website, it is content (I am ready to accept that video can be a “construction” element in the near future). Keep in mind that the supported image formats were a browser-users decision, not an html one; "There is no limit in the Web specifications to the graphical formats that can be used on the Web. You just need a MIME type so that the format is labelled correctly for transfer across the Web, and so that a suitable viewer (if one exists) can be located at the other end". Keep in mind also that jpeg is an open standard, not an open format. Only SVG is a vector graphics format designed at W3C, written in XML and stylable with CSS. We waited years for the svg to be supported…

Users should have the choice of which video codec to install (legally or illegally). Users and media producers will define the standard. Do you remember the DivX web player and the admittedly excellent stage6 website? …

The topic extends also to that the <video> element is not promoting a complete standard; w3c let the companies decide and fight and now we’re facing this situation.
- Did they have to? No, the html part is done and that is part the standard is referring to.
- HTML5 video? Nice.
- Not a codec standard? Sceptic. User-Agents make it easy, let the system-wide installed codecs and media players handle it.
- Some proprietary codec is forcing me to install it? Sorry, I won’t bite (if I don't want or cannot pay), I will choose the next website that streams-uploads “open” codec videos or just royalty-free. We all know that there are many proprietary server technologies, but the LAMP configuration managed, handled and promoted the “low-cost” web evolution (web2 for others).

Furthermore,

the Ars Technica article says that the reason Google has given for this change is that WebM (which pairs VP8 video with Vorbis audio) and Theora are "open codecs" and H.264 apparently isn't.

Openness can't be the issue

This explanation is lacking, to say the least. It appears to be a conflation of several issues: openness, royalty-freedom, and source code availability, among others. In the traditional sense, H.264 is an open standard … in the same way as, for example, JPEG still images, or the C++ programming language, or the ISO 9660 file system used on CD-ROMs. H.264 is unambiguously open.

In contrast, neither WebM's VP8 nor Theora were assembled by a standards body such as ISO. VP8 was developed independently and entirely in secret by the company On2, prior to the company's purchase last year by Google. Theora was created by a group of open-source developers based on early work also done by On2 …

For Google to claim that it is moving to "open codecs" is quite absurd: H.264 is very much an open codec. WebM is not.

It's about (cost) freedom. What H.264 isn't, however, is royalty-free.

If openness is so important that Google is willing to remove features from Chrome, there is no way that the company should be shipping Flash in Chrome.

Other features, too, should be culled. Chrome (currently) supports MP3 and AAC audio when used with HTML5's <audio> tag. Both of these compression algorithms are patent-encumbered, and neither is royalty-free (though both are, like H.264, open standards developed by industry consortia). They should be no more acceptable to Google than H.264. But if the company plans to remove them, it certainly hasn't said so.

Google responds

Why is Google supporting WebM for the HTML <video> tag?

This week’s announcement was solely related to the HTML <video> tag, which is part of the emerging set of standards commonly referred to as “HTML5”. We believe there is great promise in the <video> tag and want to see it succeed. As it stands, the organizations involved in defining the HTML video standard are at an impasse. There is no agreement on which video codec should be the baseline standard. Firefox and Opera support the open WebM and Ogg Theora codecs and will not support H.264 due to its licensing requirements; Safari and IE9 support H.264. With this status quo, all publishers and developers using the <video> tag will be forced to support multiple formats.

Why didn’t you select H.264 as the baseline codec for the HTML <video> tag in Chrome?

We acknowledge that H.264 has broader support in the publisher, developer, and hardware community today (though support across the ecosystem for WebM is growing rapidly). However, as stated above, there will not be agreement to make it the baseline in the HTML video standard due to its licensing requirements. To use and distribute H.264, browser and OS vendors, hardware manufacturers, and publishers who charge for content must pay significant royalties—with no guarantee the fees won’t increase in the future. To companies like Google, the license fees may not be material, but to the next great video startup and those in emerging markets these fees stifle innovation.

But it's not just the license fees; an even more important consideration is the pace of innovation and what incentives drive development…

Does this mean I will no longer be able to play H.264 videos in Chrome?

H.264 plays an important role in video and the vast majority of the H.264 videos on the web today are viewed in plug-ins such as Flash and Silverlight. These plug-ins are and will continue to be supported in Chrome. Our announcement was only related to the <video> tag, which is part of the emerging HTML platform. While the HTML video platform offers great promise, few sites use it today and therefore few users will be immediately impacted by this change.

Isn’t this just an effort by Google to control the web video format?

WebM is backed by many in the web community. Google views its role like any other community member and has no desire or intent to control the WebM format. Our goal is to see the HTML <video> tag become a first-class video platform. As with many other web platform efforts, we expect the majority of organizations and individuals contributing to WebM won’t be affiliated with Google or any single entity.

Developers have already created high-quality alternative (yet compatible) implementations of WebM, and we think that kind of choice is great for everyone.

Won’t this decision force publishers to create multiple copies of their videos?

Some have expressed concern that our announcement will force publishers and developers to maintain multiple copies of their content when they otherwise would not have had to. Google is among the largest publishers of video content in the world, and as such we are sympathetic to this concern. Remember, Firefox and Opera have never supported H.264 due to its licensing requirements, they both support WebM and Ogg Theora. Therefore, unless publishers and developers using the HTML <video> tag don’t plan to support the large portion of the desktop and mobile web that use these browsers, they will have to support a format other than H.264 anyway (which is why we are working to establish a baseline codec for HTML video)…

Bottom line, we are at an impasse in the evolution of HTML video. Having no baseline codec in the HTML specification is far from ideal. This is why we're joining others in the community to invest in WebM and encouraging every browser vendor to adopt it for the emerging HTML video platform (the WebM Project team will soon release plugins that enable WebM support in Safari and IE9 via the HTML standard <video> tag)…

Posted by Mike Jazayeri, Product Manager

Updated: Clarified that the Safari and IE9 plug-ins to be released by the WebM Project Team enable WebM playback via the HTML standard <video> tag canPlayType interface and not an alternate non-standard method.

Extend your reading: "Decoding the HTML 5 video codec debate" and "HTML 5 and Web video: freeing rich media from plugin prison"

A solution that seems logical on the surface is to simply expose each platform's underlying media playback engine through the HTML 5 video element—DirectShow on Windows, GStreamer on Linux, and QTKit on Mac OS X. This would make it possible for the browser to play any video formats that are supported natively on the user's computer.

From a purely technical perspective, this is not an impossible problem to solve as there are already existing libraries that do this and provide a cohesive abstraction layer on top. One prominent option is Nokia's Phonon library. It could also possibly be done by using the Quicktime and DirectShow plugins for GStreamer.

Mozilla strongly opposes this approach because it would heighten the risk of fragmentation. Allowing content providers to use any codec that is available on the user's computer might undermine the advantages of the HTML 5 media element because there would be no consistency guarantee and content would not be able to work everywhere. That is, however, arguably the situation that already exists as a result of the impasse in the codec debate.

It is not that easy to have a deterministic approach upon the matter. Right now H.264 bests on terms of performance and spread and it 's free for non-commercial playback. I understand the Firefox situation (read bellow also). I love Firefox. I use Firefox. I will just install the plugin to support H.264. As the time passes by I/we will judge which codec serve us the best in terms of quality, cost and audience adoption. If it is WebM/Theora, then it is.

MPEG LA (H.264) made a step further and announced that it will not charge any royalties for streaming H.264 encoded video in the future.

It doesn’t solve the problem that Mozilla and Opera have with H.264 as the default HTML5 video codec. If a browser needs to be able to play video natively, it needs a decoder, which needs to be licensed from the MPEG-LA. That may not be a problem for Microsoft, Google and Apple. They already have a license and adding H.264 to their browsers won’t cost them a dime. Mozilla and Opera are not so lucky – they will have to pay millions to get a license. And the problem is even more complicated for Mozilla, because their browser is open-source and they need to be able to sublicense the H.264 patents to any user of their source code. That is simply not something that the MPEG-LA is going to allow.

There is a way around this though. Both Mozilla and Opera could simply use the media framework provided by the operating system to decode the video. Both Microsoft and Apple have licensed H.264 for use in their media frameworks allowing all applications on that system to decode and encode H.264 video … Only things like commercially distributing video created with the frameworks are not covered by the license … That only leaves Linux users. As far as I know there are no Linux distributions with a properly licensed H.264 decoder. If you want to play H.264 you’ll either need to buy a closed-source binary codec from Fluendo or install an "illegal" decoder. Opera and Mozilla could just ignore this problem and simply let the user worry about this. If a decoder is provided by the operating system, it can use it whether or not it is properly licensed. It’s not their responsibility to ensure that users have properly licensed all the software on their systems.

rakaz.nl

I couldn’t agree more with… By the way, nice site, you can try the HTML5 test.

Microsoft backs up H.264 through its Interoperability project. On December 15, 2010 Claudio Caldato, Port25, posted about a new Windows Media Player Firefox plugin with extended functionalities. It enables web pages to offer video in the H.264 format using HTML5 video.

[update: Google WebM plugin for Internet Explorer 9]

Info
References