NUNIT vs VSTS ?? Which is better ?

[Parag]

Hi,
I am trying to figure out best testing tool for my project. I have
narrowed down my requirements to two tools NUNIT and VSTS unit. But I have
used neither and I have to use only one of them. Hence can someone who has
used them before share his/her experience on them so that I can get a better
idea and make a proper choice ?
Any link or suggestions are also welcome.

Thanks and Regards,
Parag
MTS,Persistent Systems Private Limited

 

[Peter Rilling]

They both really do the same thing. Here are the reasons that I would
choose NUnit over MSs version.

1) Unit testing in VS can only be done through a high-end version of the
product. If all your developers do not use the team version, then they will
not be able to use the MS testing framework.
2) NUnit has been around for years with a proven track records.
3) NUnit if available for many versions of the framework and also Mono (I
think).
4) NUnit does not couple the testing to just an MS environment.
5) External tools may work better with NUnit such as build systems.

 

[William Stacey [MVP]]

I have used both. nunit is nice, but VS test is suit. I use it in the VSDev
version.

--
William Stacey [MVP]

| Hi,
| I am trying to figure out best testing tool for my project. I have
| narrowed down my requirements to two tools NUNIT and VSTS unit. But I have
| used neither and I have to use only one of them. Hence can someone who has
| used them before share his/her experience on them so that I can get a
better
| idea and make a proper choice ?
| Any link or suggestions are also welcome.
|
| Thanks and Regards,
| Parag
| MTS,Persistent Systems Private Limited
|
|

 

[Guest]

They are pretty equivalent. The VSTS tags are a bit closer to the Attributes
throughout the framework, which is nice. If you are using Team System, I
would go ahead and use VSTS. If not, nUnit is a fine alternative. I use both,
depending on the project location.

--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA

***************************
Think Outside the Box!
***************************

 

[William Stacey [MVP]]

There are two things that I really like more about VS. Better overall
integration with VS. And you can right-click a method and gen the test stub
automatically. You also get code coverage results with integration to show
what lines where not covered. Not sure what nunit offers today in that
regard.

--
William Stacey [MVP]

in message | They are pretty equivalent. The VSTS tags are a bit closer to the
Attributes
| throughout the framework, which is nice. If you are using Team System, I
| would go ahead and use VSTS. If not, nUnit is a fine alternative. I use
both,
| depending on the project location.
|
| --
| Gregory A. Beamer
| MVP; MCP: +I, SE, SD, DBA
|
| ***************************
| Think Outside the Box!
| ***************************
|
|
| "Parag" wrote:
|
| > Hi,
| > I am trying to figure out best testing tool for my project. I have
| > narrowed down my requirements to two tools NUNIT and VSTS unit. But I
have
| > used neither and I have to use only one of them. Hence can someone who
has
| > used them before share his/her experience on them so that I can get a
better
| > idea and make a proper choice ?
| > Any link or suggestions are also welcome.
| >
| > Thanks and Regards,
| > Parag
| > MTS,Persistent Systems Private Limited
| >
| >
| >

 

[John Murray]

While NUnit doesnt offer strong vstudio integration, or coverage there
are a number of 3rd party tools that do this integration .... I
personally prefer TestRunner.

That being said, I really believe that the market positioning of VSTS by
MS puts it out of reach for most independent developers and small
development shops. That being said, IMHO, I personally see NUnit as the
best option.

 

[Jon Skeet [C# MVP]]

There are two things that I really like more about VS. Better overall
integration with VS. And you can right-click a method and gen the test stub
automatically.

Does it assume a "one test method per concrete method" model though?
That's what Eclipse assumes when you ask it to generate a test skeleton
- and it's almost never what I actually want. (I just start with a bare
test class.)

It's also not useful for true TDD, where the tests come before the code
they're calling.

You also get code coverage results with integration to show
what lines where not covered. Not sure what nunit offers today in that
regard.

You can run NUnit over NCover. I haven't used it much myself, but I've
done enough to know it works.

 

[William Stacey [MVP]]

| That being said, I really believe that the market positioning of VSTS by
| MS puts it out of reach for most independent developers and small
| development shops. That being said, IMHO, I personally see NUnit as the
| best option.

Remember you don't need VSTS to use it. You need at lease VSDev IIRC. I
have VSDev with the MSDN. Am I wrong?

I thought same about NUnit before I really used VS. One day I *forced
myself to use VS for good sized library. I would not go back now.
Different strokes. Code coverage is big.

 

[William Stacey [MVP]]

| Does it assume a "one test method per concrete method" model though?
| That's what Eclipse assumes when you ask it to generate a test skeleton
| - and it's almost never what I actually want. (I just start with a bare
| test class.)

Not 100% sure what you mean. By default, if you just have a new class, it
will stub out (pretty nice) each method (w/ overload), constructor and
properties IIRC. And even the private methods if desired. You can then add
any amount of new test methods, delete or change existing ones. You can do
any amount of tests (high, low, range, expected exceptions, etc) inside one
method or break them out as you want. Get nice coverage results, see what
failed. Click on failed tests to jump to that code. Click on uncovered
results to jump to that method and see the uncovered lines in color, etc.
Lot more going on here then just red and green traffic lights.

| It's also not useful for true TDD, where the tests come before the code
| they're calling.

They encourage and support either. Possibly the easiest way is to stub out
your class with exception in the body, then gen your tests over the class.
Then just keep adding code to your class as you go and iterating your tests.
You could also just stub out your tests without any class first, but not
sure how valuable that would be. Its all just code, so you can do what you
want.

| > You also get code coverage results with integration to show
| > what lines where not covered. Not sure what nunit offers today in that
| > regard.
|
| You can run NUnit over NCover. I haven't used it much myself, but I've
| done enough to know it works.

Have nothing against NUnit. I can only suggest trying the test and code
coverage in VS (if you got the MSDN w/ VSDev). I think you will be very
happy with the whole solution.

 

[John Murray]

code coverage is available when you use TestRunner and NUnit as well
..... at a fraction of the cost

 

[Chris Mullins]

[Nunit Vs VSTS]

There are pros and cons to both. The NUnit Stuff is free, and free is
compelling.

The VSTS stuff has integration into Visual Studio. NUnit doesn't. On the
other hand NUnit has some nice VS add-in's that give it integration -
arguably a cleaner integration than VSTS has.

VSTS has Code Coverage metrics, Nunit doesn't. This is a huge win. On the
other hand, NUnit has NCover, which gives you code coverage.

Both Nunit and VSTS have NAnt and MSBuild tasks to run tests and perform
continous integration.

The VSTS test infrastructure is integrated into the Profiler which is very
nice. NUnit has no equilivent that I've been able to find. With Compuware
not offering a Community Edition of their VS 2005 profiler (and the product
being so expensive smaller shops and indiviual consultants can't buy it) ,
this is a compelling win.

The VSTS has the memory profiling stuff built in as well. NUnit has no
equilivent. The SciTech Memory Profiler for $100 is far better than the MS
one, and at a realistic price point.

For 95% of the developers I work with, Team System isn't an option due to
cost. Microsoft's failure to include the Unit Test infrastructure with VS
Pro means almost everyone is stuck with NUnit for testing.

Overall, I like NUnit better - if only I could talk Compuware into giving me
a copy of their DevPartner Pro suite I would be all set.

 

[Chris Mullins]

Remember you don't need VSTS to use it. You need at lease
VSDev IIRC. I have VSDev with the MSDN. Am I wrong?

Unfortunatly, I think so.

http://msdn.microsoft.com/vstudio/products/compare/

The Professional Version I often use doesn't have any of the testing stuff,
and unless you got the very, very expensive MSDN subscription, that's the
version of VS.NET that comes with MSDN.

Code coverage is big.

http://ncover.org/site/

 

[Chris Mullins]

[I've responded to so many of your posts this last week, I feel like I'm
stalking you]

| You can run NUnit over NCover. I haven't used it much myself, but I've
| done enough to know it works.

Have nothing against NUnit. I can only suggest trying the test and code
coverage in VS (if you got the MSDN w/ VSDev). I think you will be very
happy with the whole solution.

It's nice - even nicer, to me, is the performance profiler and memory
profiler integration with the VSTS infrastructure. I also love the way you
can rollup smaller tests into big test runs using the Web Stress tools. Very
slick design by Microsoft.

 

[William Stacey [MVP]]

cool. Have not tried the memory profiler stuff yet.

 

[Jon Skeet [C# MVP]]

| Does it assume a "one test method per concrete method" model though?
| That's what Eclipse assumes when you ask it to generate a test skeleton
| - and it's almost never what I actually want. (I just start with a bare
| test class.)

Not 100% sure what you mean. By default, if you just have a new class, it
will stub out (pretty nice) each method (w/ overload), constructor and
properties IIRC.

What *exactly* do you mean by stubbing out? Creating the methods in the
production class, or creating methods in the test class to match the
ones in the production class.

My point is that if I've got a single method in the production class, I
may well want three or four test methods with names indicating what
aspect of that method they'll be testing. Sometimes two methods are
tested by the same set of tests. In other words, creating a "matching"
set of methods in the test code has never seemed that useful to me.

And even the private methods if desired. You can then add
any amount of new test methods, delete or change existing ones. You can do
any amount of tests (high, low, range, expected exceptions, etc) inside one
method or break them out as you want. Get nice coverage results, see what
failed. Click on failed tests to jump to that code. Click on uncovered
results to jump to that method and see the uncovered lines in color, etc.
Lot more going on here then just red and green traffic lights.

Good to hear VS is finally catching up with Eclipse a bit

| It's also not useful for true TDD, where the tests come before the code
| they're calling.

They encourage and support either. Possibly the easiest way is to stub out
your class with exception in the body, then gen your tests over the class.

No, that's not test-driven development. With test-driven development
you don't know what methods you'll have in your class until you've
written your tests.

Then just keep adding code to your class as you go and iterating your tests.
You could also just stub out your tests without any class first, but not
sure how valuable that would be. Its all just code, so you can do what you
want.

Yes, but different IDE features can certainly help TDD. I seem to
remember that 2005 is capable of creating methods that you call even
when they don't exist just like Eclipse does. That's what really helps
TDD.

| > You also get code coverage results with integration to show
| > what lines where not covered. Not sure what nunit offers today in that
| > regard.
|
| You can run NUnit over NCover. I haven't used it much myself, but I've
| done enough to know it works.

Have nothing against NUnit. I can only suggest trying the test and code
coverage in VS (if you got the MSDN w/ VSDev). I think you will be very
happy with the whole solution.

I'm not entirely clear about what "VSDev" is. I'll have to look through
my MSDN discs to find it. I have VS 2005 Pro installed, and that
*doesn't* have the MS test stuff. Have they released the testing stuff
as a free add-in? (I've been gently prodding MS for this for a
while...)

 

[Jon Skeet [C# MVP]]

For 95% of the developers I work with, Team System isn't an option due to
cost. Microsoft's failure to include the Unit Test infrastructure with VS
Pro means almost everyone is stuck with NUnit for testing.

That's my feeling as well, although William is talking about this
"VSDev" thing which I don't know about, so maybe it's available to
"normal" MSDN subscribers now.

Personally I think they should have included it even in VC# Express.
They'd have instantly grabbed a *huge* proportion of the unit testing
mindshare, as well as it being good PR in terms of "MS believes in unit
testing" being a good thing to persuade developers of

 

[William Stacey [MVP]]

| What *exactly* do you mean by stubbing out? Creating the methods in the
| production class, or creating methods in the test class to match the
| ones in the production class.

Sorry, was not clear. I mean either.



Your class:

public class Class1

{

private string name;

public Class1(string name)

{

this.name = name;

}

public string DoSomething(int num)

{

throw new NotImplementedException();

}

}


The default test generated:
....
[TestMethod()]

public void DoSomethingTest()

{

string name = null; // TODO: Initialize to an appropriate value



Class1 target = new Class1(name);



int num = 0; // TODO: Initialize to an appropriate value



string expected = null;

string actual;



actual = target.DoSomething(num);



Assert.AreEqual(expected, actual, "MessageLib.Class1.DoSomething did not return the expected value.");

Assert.Inconclusive("Verify the correctness of this test method.");

}

....

You can create your class with just its "interface/signatures" with NotImplementedExceptions until you dev each method. You could also add methods to your Tests that throw Inconclusive (or what ever) until you implement the methods in the class.

| My point is that if I've got a single method in the production class, I
| may well want three or four test methods with names indicating what
| aspect of that method they'll be testing. Sometimes two methods are
| tested by the same set of tests. In other words, creating a "matching"
| set of methods in the test code has never seemed that useful to me.

I understand. TMK, it only adds one-for-one by default. Naturally, in DoSomethingTest() above, you could add multiple "tests" inside that one method and each test would need to pass that particular Assert or you get the respective Assert fails. You could call any number of helper methods, or create multiple Test methods with a diff name (i.e. DoSomethingTestNull, etc) as you need. Not sure how a tool would "know" you wanted 2 or 4 test methods for the same overload (or what to call them)?

| No, that's not test-driven development. With test-driven development
| you don't know what methods you'll have in your class until you've
| written your tests.

You could do that above if you wanted. If class1.DoSomething() did not exist yet, do what you need to do in the DoSomething() Test method. Are you thinking of something else?

| Yes, but different IDE features can certainly help TDD. I seem to
| remember that 2005 is capable of creating methods that you call even
| when they don't exist just like Eclipse does. That's what really helps
| TDD.

Forgive me. How would this help you? There is nothing being tested. You want that test to fail anyway until you dev the logic. So why call ghost methods? Just Assert.Inconclusive("Not implemented") in that test method until you hook-up the class logic. I am surely missing something.

|
| > | > You also get code coverage results with integration to show
| > | > what lines where not covered. Not sure what nunit offers today in that
| > | > regard.
| > |
| > | You can run NUnit over NCover. I haven't used it much myself, but I've
| > | done enough to know it works.
| >
| > Have nothing against NUnit. I can only suggest trying the test and code
| > coverage in VS (if you got the MSDN w/ VSDev). I think you will be very
| > happy with the whole solution.
|
| I'm not entirely clear about what "VSDev" is. I'll have to look through
| my MSDN discs to find it. I have VS 2005 Pro installed, and that
| *doesn't* have the MS test stuff. Have they released the testing stuff
| as a free add-in? (I've been gently prodding MS for this for a

Sorry, all those VS versions confuse me. Mine is VSTS Software Developer. With your MSDN pkg, you had to pick if you wanted Developer or Tester - IIRC. You may have it already. If not, contact your lead and they may be able to change things around for you. HTH

 

[Jon Skeet [C# MVP]]

| What *exactly* do you mean by stubbing out? Creating the methods in the
| production class, or creating methods in the test class to match the
| ones in the production class.

Sorry, was not clear. I mean either.

You can create your class with just its "interface/signatures" with
NotImplementedExceptions until you dev each method. You could
also add methods to your Tests that throw Inconclusive (or what
ever) until you implement the methods in the class.

I don't want to write *any* code (not even the signatures) until I've
written a test though. The question is how it makes that.

| My point is that if I've got a single method in the production class, I
| may well want three or four test methods with names indicating what
| aspect of that method they'll be testing. Sometimes two methods are
| tested by the same set of tests. In other words, creating a "matching"
| set of methods in the test code has never seemed that useful to me.

I understand. TMK, it only adds one-for-one by default. Naturally, in
DoSomethingTest() above, you could add multiple "tests" inside that
one method and each test would need to pass that particular Assert or
you get the respective Assert fails.

Eek no - having multiple logical tests within a single test should be
avoided. It makes it harder to work out how much is wrong, because the
first assertion that fails masks later ones which *would* have failed
(unless assertions are handled a very different way in VSTS...)

You could call any number of helper
methods, or create multiple Test methods with a diff name (i.e.
DoSomethingTestNull, etc) as you need. Not sure how a tool would
"know" you wanted 2 or 4 test methods for the same overload (or what
to call them)?

It can't - which is sort of my point. I can't say I find any of the
skeleton-generators helpful, and can't see how they're likely to
*become* helpful. In other words, whether or not a system includes one
doesn't sway me in favour of it.

| No, that's not test-driven development. With test-driven development
| you don't know what methods you'll have in your class until you've
| written your tests.

You could do that above if you wanted. If class1.DoSomething() did not exist yet,
do what you need to do in the DoSomething() Test method. Are you thinking of
something else?

I'm not sure whether or not we're talking past each other - see
below...

| Yes, but different IDE features can certainly help TDD. I seem to
| remember that 2005 is capable of creating methods that you call even
| when they don't exist just like Eclipse does. That's what really helps
| TDD.

Forgive me. How would this help you? There is nothing being tested.
You want that test to fail anyway until you dev the logic. So why call ghost
methods? Just Assert.Inconclusive("Not implemented") in that test method
until you hook-up the class logic. I am surely missing something.

The point of TDD is that you write the test before you write the method
which implements it - including the signature, often. That way, you
design the API in terms of how to make it usable and testable, rather
than trying to imagine how you *might* want to use it. So, I'd write a
test which called imaginary methods, get the IDE to create dummy
implementations for those methods, and then run the test (which would
fail).

The important thing is that the IDE has to make it easy to generate the
dummy methods given a test which calls them - that's what helps TDD.

| I'm not entirely clear about what "VSDev" is. I'll have to look through
| my MSDN discs to find it. I have VS 2005 Pro installed, and that
| *doesn't* have the MS test stuff. Have they released the testing stuff
| as a free add-in? (I've been gently prodding MS for this for a

Sorry, all those VS versions confuse me. Mine is VSTS Software Developer. With
your MSDN pkg, you had to pick if you wanted Developer or Tester - IIRC. You may
have it already. If not, contact your lead and they may be able to change things
around for you. HTH

Ah, so you *do* need VSTS, as I thought. That's the problem - not
everyone (far from everyone) will have VSTS. If it were available in
Pro but not Express, I'd still view that as a pity - but not being
available in Pro? Bad mistake IMO.

Jon

 

[William Stacey [MVP]]

| I don't want to write *any* code (not even the signatures) until I've
| written a test though. The question is how it makes that.

In effect your writing to some abstract interface that does not exist.
Seems like a waste of time. May as well at least write the interface first
so you have some idea of what your doing. That is probably a religious
thing that I should punt on. If it works for you/others - all the best.

| Eek no - having multiple logical tests within a single test should be
| avoided. It makes it harder to work out how much is wrong, because the
| first assertion that fails masks later ones which *would* have failed
| (unless assertions are handled a very different way in VSTS...)

I see your point in general for a large project. Guess it depends on what
is reasonable for your project and how you want to factor your tests.

| It can't - which is sort of my point. I can't say I find any of the
| skeleton-generators helpful, and can't see how they're likely to
| *become* helpful. In other words, whether or not a system includes one
| doesn't sway me in favour of it.

ok. I end up changing the default one too, but find it at least gets me
boot-strapped and off and running on that method. I understand your point.

| The important thing is that the IDE has to make it easy to generate the
| dummy methods given a test which calls them - that's what helps TDD.

If that works for you. I don't see that saving me any time and would
instead confuse me. Interface first, for me anyway, seems more reasonable.
Different strokes.

| Ah, so you *do* need VSTS, as I thought. That's the problem - not
| everyone (far from everyone) will have VSTS. If it were available in
| Pro but not Express, I'd still view that as a pity - but not being
| available in Pro? Bad mistake IMO.

I agree. The version in Developer *should be included in *all versions.
The extra things like Test Manager could be the value-add in the Test
specific version. I think they went version *crazy and gave marketing too
much coffee that month.
--wjs

 

[Jon Skeet [C# MVP]]

| I don't want to write *any* code (not even the signatures) until I've
| written a test though. The question is how it makes that.

In effect your writing to some abstract interface that does not exist.
Seems like a waste of time. May as well at least write the interface first
so you have some idea of what your doing.

The point of TDD is that you know the interface which is best to use
when you've used it.

That is probably a religious
thing that I should punt on. If it works for you/others - all the best.

Many others - I suggest you read up on TDD. It's great, although it
does take a certain shift in perspective.

| Eek no - having multiple logical tests within a single test should be
| avoided. It makes it harder to work out how much is wrong, because the
| first assertion that fails masks later ones which *would* have failed
| (unless assertions are handled a very different way in VSTS...)

I see your point in general for a large project. Guess it depends on what
is reasonable for your project and how you want to factor your tests.

I don't see how the size of the project is relevant - a test should
test one thing and only one thing, where possible.

| It can't - which is sort of my point. I can't say I find any of the
| skeleton-generators helpful, and can't see how they're likely to
| *become* helpful. In other words, whether or not a system includes one
| doesn't sway me in favour of it.

ok. I end up changing the default one too, but find it at least gets me
boot-strapped and off and running on that method. I understand your point.

It's certainly a personal thing.

| The important thing is that the IDE has to make it easy to generate the
| dummy methods given a test which calls them - that's what helps TDD.

If that works for you. I don't see that saving me any time and would
instead confuse me. Interface first, for me anyway, seems more reasonable.
Different strokes.

Indeed. I'm not saying it should do anything without user intervention,
by the way - in Eclipse you click on a lightbulb either in the problem
view or by the side of the line with the compilation error, and that
offers you the option of fixing the problem by creating a new method
(or type, or whatever).

| Ah, so you *do* need VSTS, as I thought. That's the problem - not
| everyone (far from everyone) will have VSTS. If it were available in
| Pro but not Express, I'd still view that as a pity - but not being
| available in Pro? Bad mistake IMO.

I agree. The version in Developer *should be included in *all versions.
The extra things like Test Manager could be the value-add in the Test
specific version. I think they went version *crazy and gave marketing too
much coffee that month.

Definitely.