G
Guest
Hello!
As most of you know, there are several audio codecs available today: MP3,
Ogg Vorbis, AAC, WMA, Musepack, etc. Often, there are also more encoders
available that encode to a specific format. For example, you can use Nero
Digital, iTunes or Coding Technologies encoders to create AAC bitstreams. MP3
can also be encoded by LAME, Shine, Blade Encoder, etc. Being an audio
enthusiast and wanting to know which codec and encoder delivers the best
sound quality at a given bitrate, I started to run blind listening tests.
Because these tests are ran blindly, the testers cannot rate MP3 higher than
WMA for example just because they are MP3 zealots or WMA haters - the tester
doesn't know which sample he is listening to. Moreover, the tester cannot
pretend to hear artifacts if he isn't able to pass an ABX test, For example,
if he manages to tell apart sample A from sample B 8 times one after another,
we can almost say for sure that there really is an audible difference between
the two samples. If you want to read more about blind tests, especially about
the ABC/HR or ABX method, there are several documents which you can find with
Google or whatever search engine you prefer.
Probably you are asking yourself now what this has to do with Windows Vista.
Windows Vista and Windows Media Player 11 introduced a new WMA encoder - WMA
9.2 and WMA 10. Even before Microsoft launched these two products to the
public, they published the results of a listening test comparing HE-AAC
against WMA:
http://www.microsoft.com/windows/windowsmedia/forpros/codecs/Comparison.aspx
Microsoft claims that the test was also done blindly and that WMA had a
higher rating that HE-AAC. While in my opinion this test was unfair, since
Microsoft instructed NSTL which settings to use and also provided a very
recent encoder, NSTL didn't discuss the settings with the Nero Digital
developers and also didn't ask for the most recent version of the HE-AAC
encoder. Not wanting to believe the results, especially since listening tests
in the past showed that WMA was often worse than AAC, I could hardly wait for
Windows Vista or Windows Media Player 11 to become available in order to run
my own test. However, the EULA of both Windows Vista and WMP 11 state that I
am not allowed to publish benchmark results without having a written
permission from MS to do so. Not wanting to violate the EULA, I started to
get in touch with several MS developers and also with the licensing
department. Half of the e-mails I sent remained unanswered, for a quarter of
them I received some standard replies that MS is checking the issue
internally and that someone would get in touch with me ASAP and another
quarter contained some vague information that the encoder is still beta and
that it can change at any time and might still contain bugs. Well, the given
arguments are very weak. Nero AG, LAME developers, Vorbis developers etc.
also made beta encoders available for such tests and didn't fear that the
outcome of the result might harm their image. I am also very upset that while
I am not allowed to disclose results of my own tests, Microsoft still has the
results of their test online - most likely using the same beta encoder I want
to use and which might be buggy as well.
I think that with this attitude towards audio enthusiasts and possible
future customers Microsoft is doing more harm than by allowing me to publish
the results of the test. Such test are not made to harm a company - no way,
they can also be very helpful. LAME and Nero got tuned because of such
listening tests.
I really hope that someone from Microsoft who is responsible for WMA or for
licenses will get in touch with me.
Regards,
Sebastian
As most of you know, there are several audio codecs available today: MP3,
Ogg Vorbis, AAC, WMA, Musepack, etc. Often, there are also more encoders
available that encode to a specific format. For example, you can use Nero
Digital, iTunes or Coding Technologies encoders to create AAC bitstreams. MP3
can also be encoded by LAME, Shine, Blade Encoder, etc. Being an audio
enthusiast and wanting to know which codec and encoder delivers the best
sound quality at a given bitrate, I started to run blind listening tests.
Because these tests are ran blindly, the testers cannot rate MP3 higher than
WMA for example just because they are MP3 zealots or WMA haters - the tester
doesn't know which sample he is listening to. Moreover, the tester cannot
pretend to hear artifacts if he isn't able to pass an ABX test, For example,
if he manages to tell apart sample A from sample B 8 times one after another,
we can almost say for sure that there really is an audible difference between
the two samples. If you want to read more about blind tests, especially about
the ABC/HR or ABX method, there are several documents which you can find with
Google or whatever search engine you prefer.
Probably you are asking yourself now what this has to do with Windows Vista.
Windows Vista and Windows Media Player 11 introduced a new WMA encoder - WMA
9.2 and WMA 10. Even before Microsoft launched these two products to the
public, they published the results of a listening test comparing HE-AAC
against WMA:
http://www.microsoft.com/windows/windowsmedia/forpros/codecs/Comparison.aspx
Microsoft claims that the test was also done blindly and that WMA had a
higher rating that HE-AAC. While in my opinion this test was unfair, since
Microsoft instructed NSTL which settings to use and also provided a very
recent encoder, NSTL didn't discuss the settings with the Nero Digital
developers and also didn't ask for the most recent version of the HE-AAC
encoder. Not wanting to believe the results, especially since listening tests
in the past showed that WMA was often worse than AAC, I could hardly wait for
Windows Vista or Windows Media Player 11 to become available in order to run
my own test. However, the EULA of both Windows Vista and WMP 11 state that I
am not allowed to publish benchmark results without having a written
permission from MS to do so. Not wanting to violate the EULA, I started to
get in touch with several MS developers and also with the licensing
department. Half of the e-mails I sent remained unanswered, for a quarter of
them I received some standard replies that MS is checking the issue
internally and that someone would get in touch with me ASAP and another
quarter contained some vague information that the encoder is still beta and
that it can change at any time and might still contain bugs. Well, the given
arguments are very weak. Nero AG, LAME developers, Vorbis developers etc.
also made beta encoders available for such tests and didn't fear that the
outcome of the result might harm their image. I am also very upset that while
I am not allowed to disclose results of my own tests, Microsoft still has the
results of their test online - most likely using the same beta encoder I want
to use and which might be buggy as well.
I think that with this attitude towards audio enthusiasts and possible
future customers Microsoft is doing more harm than by allowing me to publish
the results of the test. Such test are not made to harm a company - no way,
they can also be very helpful. LAME and Nero got tuned because of such
listening tests.
I really hope that someone from Microsoft who is responsible for WMA or for
licenses will get in touch with me.
Regards,
Sebastian