I wasn't expected that !

[Carl Daniel [VC++ MVP]]

The Standard Edition is good for learning programming, as it runs in
debug mode all the time - which is just fine for that purpose.

This is not true. There is a difference between Debug and Release builds
even under the Standard Edition (different runtime libraries, most notably).

-cd

 

[Tester]

You might want to raise the priority of your test apps:
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.RealTime;
Thread.CurrentThread.Priority = ThreadPriority.Highest;

SetPriorityClass( GetCurrentProcess(), REALTIME_PRIORITY_CLASS);
SetThreadPriority( GetCurrentThread(), THREAD_PRIORITY_TIME_CRITICAL);


Also note that DateTime/GetTickCount is not high-resolution enough for
precise timing. One should use
QueryPerformanceCounter and QueryPerformanceFrequency like so:

LARGE_INTEGER start={0};
LARGE_INTEGER end={0};
LARGE_INTEGER freq={0};
if(!QueryPerformanceFrequency(&freq))
return 0;

QueryPerformanceCounter(&start);
//time consuming operation
QueryPerformanceCounter(&end);

double elapsedTime = (end.QuadPart - start.QuadPart)*1.0f/freq.QuadPart;

One should also subtract the difference of QueryPerformanceCounter call
itself.

 

[Brandon Bray [MSFT]]

I agree, that this is probably the case. It is still dissapointitng that
competitor versions of C++ and even Microsoft own C# Standard version
does not behave so badly depending on the style of writing code. For me
it was a dissapointment. I understand, that you are trying to "excuse"
Microsoft, but the fact remains, that the stnadard version is inferior
comparing to the competition. Since large majority of the users will buy
standard version, this test will help them to make their mind and clearly
understand the restrictions they are getting into.

I am certainly not trying to make excuses for the product. The observable
differences in the style are as expected and by design. That does not make
it inferior. When debugging an application, it is very important that the
compiler not muck with the code. That is why the compiler does exactly has a
difference between the two programming styles.

Your observation was strictly about debug behavior. The complaint was that
this lead to an inferior product. This is simply not a fair statement. The
entire discussion is summed up with "optimizing compilers are better than
non-optimizing compilers" no matter what the language is. No one will argue
with that. However, that is not what the writeup says. While you have
repeatedly said that you understand that, the feedback is simply that
reading the conclusions page alone or the page about problems with Visual
C++ does not even mention you are using a non-optimizing compiler.

The title of the writeup might better be titled as to what to expect with
free compilers.

I disagree. The test # 2, in which Visual C++ simply stinks, all you use,
is library functions. It seems, that C# math library is much better
written than C++. All three tests do not use any extensive memeory
allocation that would change test outcome significantly. I avoided on
purpose allocating memory. Visual C++ Win32 console version runs really
slow comparing to C# (and presumably .NET C++ which should equal C#). I
suspect, that math run time libraries used with Win32 console are not
that good as the ones in the .NET Math class. Borland is even worse here.
How much "optimization" you can do here with optimizing compiler if you
still are going to use bad libraries ?

Of course, saying that the math libraries in Visual C# are better than the
Visual C++ libraries is highly suspicious. If you really believe that, it
would be far more important to isolate the math library by itself and
benchmark that. As done, the information is highly inconclusive.

I really don't mean to be rude, but I see reports like this fairly
frequently and they're always misinformed. The math libraries that the C++
compiler uses are supplied by Intel and AMD and are highly optimized to the
hardware. If there was truly any competition, it would be a matter or
milliseconds in difference and certainly within a noise threshold.

And they do when you actually use .NET console application in Visual C++.
If you try to use Win32 console application, Visual C++ losses to C# at
least in it's standard version.

Of course it does. The C# compiler doesn't do any optimization -- it relies
on the JIT compiler in the runtime. For C++, there isn't any JIT compiler to
optimize the code. Given that you're working with a non-optimizing compiler,
the result is completely expected.

The thread here started with the statement that the result of this study was
surprising. In reality, it is completely expected that a non-optimizing tool
set will fall behind an optimizing tool set.

On another note, Carl Daniel was studying a curious problem where the
optimizing compiler would print zero for the elapsed time. The problem was
that 'benchmark' is an algebraic function that does not manipulate global
memory. This allows the optimizer to move the call benchmark forward or
backwards. Thus, the two calls to GetTickCount were sequential (which
improves hot code paths, as the code is more likely to be in the instruction
cache). The solution is to make the outcome variable volatile.

double volatile outcome = benchmark(i);

Hope that helps!

 

[Hendrik Schober]

I'm not sure what exactly you mean by this,. The same code compiled in
stanadard version as "debug" runs about twice as slow as the same code
compiled with "release" setting, so clearly there is a difference.
If your statement would be true, changing this setting would have no effect
whatsoever or the setting would not be even made available, wouldn't be ?

Bo's statement might have simplified
things a little bit too much. There's
more differences between release and
debug builds than whether the optimizer
kicks in. For example, debug builds will
use the debug version of the RTL, which
is slower than the release version.

[...]
JM

Schobi

--
(e-mail address removed) is never read
I'm Schobi at suespammers dot org

"Sometimes compilers are so much more reasonable than people."
Scott Meyers

 

[Jack]

I am certainly not trying to make excuses for the product. The observable
differences in the style are as expected and by design. That does not make
it inferior.

I see your point. I guess, my dissapointement has more to do with the
fact, that I did not know prior to my tests, that Microsoft is
selling in fact different product under standard version. Because of
my prior experience with Borland I simply didn't expect that.
From the certain point of view it reminds me the same marketing
scheme of NVIDIA, which sold under the generic name of GeForce 4
two different chips with two different price points. The cheaper
version was actually renamed beefed up GeForce 2 chip, worse than the
older GeForce 3 chip. If one knew that, one could say, that it is not
the big deal, if you did not know that it was a dirty trick from the
certain point of view. Same by the way is now with the USB 2.0 High
speed and USB 2.0 full speed. How many people belive, that this
marketing scheme is crystal clear and fair ? I don't dispute, that the
information is out there, but many customers probably will buy the
product without knowing the fact.

Of course, saying that the math libraries in Visual C# are better than the
Visual C++ libraries is highly suspicious. If you really believe that, it
would be far more important to isolate the math library by itself and
benchmark that. As done, the information is highly inconclusive.

What is suspicious, is rather why C++ square root function in either
Borland or Microsoft version is two to three times slower that C# version
without any gain in accuracy. My concern about the libraries, it is
simply my guess about the cause. Since the code between various high math
functions in test # 2 is identical with the exception of library call,
then the major increase of C# advantage over C++ (either Borland or
Microsoft) for some of the functions can be only attributed to the
differences between the library function implementation. How otherwise
would you explain, that the arctan() test has very similar timing between
C++ and C# versions of the code, but when the single library call in the
code is changed from arctan() to exp() or sqrt(), suddenly C# executes
otherwise the same code two to three times faster than C++ with the same
accuracy ? What exactly changed in optimalization ? There maybe a very
good scientific explanation, that for example C++ libraries are more
stable at extreme exotic input etc., but as a occasional user I'm not
in a position to either prove it or really care about it. It is similar
to the famous Pnetium FDIV flaw, that affected mainly experts but for
everyday occasional programmers had no impact whatsoever (except FUD
of course ).

I disagree also about approach. If my aprroach is to calculate
particular integral (as I'm doing in the test # 2), C# won in most cases
by a large margin over non optimized C++ by Microsoft and over
supposedly highly optimized C++ by Borland. As a end user I don't care
what is the exact cause of C++ being slower. All I care is to have my
program run as fast as it can. In any case either library was bad or
the information exchange mechanism between the library and program was
crippled or something else. You should not expect from the occasional
end user to make a scientific analysis why his program that is written
in a conventional, acceptable way is uderperforming on the theoretically
better platform. What is then better in that platform ? The occasional
user should be advised to run the platform that generates the fastest
code most of the time without tweaking.

For the expert your approach is absolutely right but it is not for
the target user of my test.

JM

 

[Brandon Bray [MSFT]]

How many people belive, that this marketing scheme is crystal clear and
fair ? I don't dispute, that the information is out there, but many
customers probably will buy the product without knowing the fact.

This absolutely something worth complaining about. I've unfortunately had to
deal with too many customers who bought the Standard Edition not knowing
that the compiler was non-optimizing. This certainly makes your analysis
worth while, and demonstrates that Visual C++ isn't competitive at low price
points.

In fact, this is exactly the feedback that led to our decision to stop
building the non-optimizing compiler. In the next version of Visual C++
codenamed "Whidbey", we will only provide the optimizing compiler

Cheerio!

 

[Bonj]

The compiler might even be as clever as to simply *not do* some of the
calculations at all, since it knows the variables don't get used in a
meaningful way. That's why I think as close to real world test scenarios
with genuine program output are better. So you don't really know, I mean
that might be why it showed as doing it really fast, when in fact it was
just not doing it as it knew it didn't need to.

As I said, it's too bad for Microsoft. I'm not a professional programmer

and I cannot, as many engineers and scientists

in similar to mine position, justify cost of more expensive versions of

the software. I mentioned that clearly in the

test page.

Consider this for example:

(www.borland.com)
Borland C++ Builder 6.0 (Personal) $69
Borland C++ Builder 6.0 (Professional) $999
Borland C++ Builder 6.0 (Enterprise) $2999

(www.microsoft.com)
Microsoft Visual C++ .NET (Standard) $109
Microsoft Visual C++ .NET (Professional) $1079

In fact you can download complete, full version of the Borland compiler

(without IDE) for free. Also you can get open

source (gcc) C++ windows compiler freely. This make the price span even

more ridiculous. I'm refusing to buy expensive

versions of the software for occasional programming sessions. My money

making job has nothing to do with computer

programming.


And that makes sense. On my Athlon XP 1800 + (similarly to your timing)

90,000,000 loops of four additions and two

divisions took roughly 2 seconds. For the sake of simplicity assuming that

addition and division takes the same time, it

means six general math operations per loop, which means about 540,000,000

math operations in two seconds or 270,000,000

operations per second. Athlon XP 1800+ runs at 1533 MHz. So in fact it

made each operation in about 5.7 clock cycle

which is pretty good. You can't massively improve this. That is why the

differences between Borland, C# and VC++ were

 

[Bonj]

The good news is that in the future, we are including the optimizing
compiler in all SKUs. This should address the concern of availability.

That IS good news - I don't think you've really got anything to worry about
with GNU (it's horrible and linux-based) but the fact that you can download
a Borland optimizing compiler AND debugger from Borland for free in UNDER 20
MB should worry you, currently.
I certainly hope reliability of the SDK's installer come top of the priority
list aswell. I don't know why but I've only got 28 files in my
SDK\v1.1\Include directory. There is one .cpp, one .def and the others are
..h and .idl files. There is no windows.h. Reinstalling windows, fixing the
disk, and reinstalling the SDK wouldn't make the header files reappear. I
phoned up Microsoft, but they wanted £185 to even attempt to start giving an
answer to my question. This doesn't bode well for the reputation of
Microsoft, certainly in my esteem anyway - and how many other people are
experiencing the same thing? Microsoft have long been successful due to two
main factors - commercial aggressiveness and technical excellence, but if
they're losing the edge on the latter to other outfits.... I now therefore
not only prefer to use Borland, but have no other option.
I would definitely be interested to know if Microsoft have decided to do the
right thing and include an optimizer for free though. I mean it's mainly
businesses that spend the money on the pricey software anyway, and they are
going to buy the IDE no matter what not because they want the optimizer, but
because they want their staff to use the IDE to build applications as
quickly as possible. Keeping the hobbyist satisfied

 

[Bonj]

The title of the writeup might better be titled as to what to expect with
free compilers.

It certainly might aswell! However, simply the inclusion of those few words
would have made it a lot more difficult to dismiss the test as unfair.

 

[William DePalo [MVP VC++]]

There is no windows.h. Reinstalling windows, fixing the
disk, and reinstalling the SDK wouldn't make the header files reappear. I
phoned up Microsoft, but they wanted £185 to even attempt to start giving an
answer to my question. This doesn't bode well for the reputation of
Microsoft, certainly in my esteem anyway

Why should that be so? In 12 years or so of SDK installations <windows.h>

and how many other people are
experiencing the same thing?

People with problems tend to be vocal. Common problems affect lots of
people. The collective roar of people experiencing common problems is hard
to miss. On the other hand, the absence of a roar around an issue is often
evidence that no common problem exists.

I now therefore not only prefer to use Borland, but have no other option.

Way back when C6 was current, Borland's IDE and compiler was a quite
popular option, widely regarded as better. It was the one that I choose,
too. By C7 it wasn't immediately obvious which was better. The introduction
of the "Visual" line was the beginning of the end for Borland.

Just by the way, the free command line compiler that Borland gives away is a
few years old, no?

Regards,
Will

 

[Bonj]

has never disappeared on me. <EG>

Excellent. It definitely has on mine though.

People with problems tend to be vocal. Common problems affect lots of
people. The collective roar of people experiencing common problems is hard
to miss. On the other hand, the absence of a roar around an issue is often
evidence that no common problem exists.

Yep, and likely me more than most. But if you throw enough shit, some
sticks. It doesn't alter the fact that Microsoft's compiler therefore simply
doesn't work on my computer, while Borland's works perfectly. I've got the
VS.NET 2002 IDE which I use if I want to build some utility, but for my
hobbyist-type work I think I'd rather use Borland. Basically, I'll use MS
for things that don't take long to write and I'm not really putting my heart
in it or as much effort, and when the IDE makes things much more convenient
which would be possible otherwise but boring. For my main programming,
though, if I can't really be sure the development software isn't going to
behave like a house of cards after a while, then it just smacks of the
author of such dev tools not putting as much effort into their product as I
intend to put into mine - which I think is profesionally rude, and I can't
stand it. For long-termish projects, then I want to be able to trust the
method by which I'm developing it. And I'm afraid Microsoft's compiler
simply doesn't live up to that requirement for me.

option.

Way back when C6 was current, Borland's IDE and compiler was a quite
popular option, widely regarded as better. It was the one that I choose,
too. By C7 it wasn't immediately obvious which was better. The introduction
of the "Visual" line was the beginning of the end for Borland.

Well, commercially that may be the case, but it is a shame they felt it
necessary to send limos to Borland's offices to take 34 of their executives
and key developers out to lunch then subsequently poach them, in order to
achieve their market-leading position. When I read that it put even less
trust in microsoft in me than I already had, and it puts the 'stupid white
man' tag even more firmly on their shoulders. They may be wrong, but some do
claim there is a growing tide of mistrust towards ms - chiefly centred
around the growing 'locking-in' and 'locking-down' strategies.

Just by the way, the free command line compiler that Borland gives away is a
few years old, no?

Possibly - but the stats speak for themselves - Borland provide a free
optimizing compiler - Microsoft simply don't.

 

[Brandon Bray [MSFT]]

It certainly might aswell! However, simply the inclusion of those few
words would have made it a lot more difficult to dismiss the test as
unfair.

That is pretty much my point. I do think there is value in these kinds of
analyses, as long as the reader is not misled.

 

[Tester]

There are two SDKs. One is Platform SDK and the other one is .NET SDK. You
have .NET SDK, which included standard version of C++/MC++ compiler, but no
win32 headers or libs.

 

[Hendrik Schober]

[...]
Just by the way, the free command line compiler that Borland gives away is a
few years old, no?

It's BCC5.5, the compiler that came with
C++ Builder 5. Regarding std conformance
it is better than VC6 (what wouldn't?).
However, I think that VC7.1 has passed it
in this regard. (From what I know VC's
GC has for long been better than BCC's.
But I have never tested this myself.)

Regards,
Will

Schobi

--
(e-mail address removed) is never read
I'm Schobi at suespammers dot org

"Sometimes compilers are so much more reasonable than people."
Scott Meyers

 

[Guest]

MS .NET Framework EULA

"3.4 Benchmark Testing. You may not disclose the results of any benchmark test of the .NET Framework component of the Software to any third party without Microsoft’s prior written approval.

 

[Hendrik Schober]

MS .NET Framework EULA:

"3.4 Benchmark Testing. You may not disclose the results of
any benchmark test of the .NET Framework component of the Software
to any third party without Microsoft's prior written approval."

LOL!
These clauses spread through software
licenses like viruses.

Schobi

--
(e-mail address removed) is never read
I'm Schobi at suespammers dot org

"Sometimes compilers are so much more reasonable than people."
Scott Meyers