Visual Basic 2005 Design and Development - Chapter 17 pps

After they write a small piece of code,they often feel that the code is too simple to need testing.. Instead of thinking of a small piece of code as too simple to test, you should think

Previous chapters have mentioned testing in passing, but have not really explained what kinds oftests are possible and what kinds of tests you should perform For example, the section on test-drive development in Chapter 3, “Agile Methodologies,” assumes that you write tests for codebefore you write the actual code, but it doesn’t explain how to do that

This chapter provides some additional information about testing It describes the kinds of testsyou can perform and their purposes It explains when you should test different parts of an appli-cation, and what you should do when tests detect a problem

Testing PhilosophyThe way you think about testing shapes the application’s progress Too many developers think oftesting as an annoying, half-unnecessary process that should be performed at the end of develop-ment and only as time permits The result is an application full of known and unknown bugs thatmay be shipped to meet some preset deadline The users develop an unfavorable perception ofyour development team, and believe the application is frustrating and has low quality (I’m sureyou can think of a few major commercial applications that satisfy that description.)

To produce a stable application, programmers must have the idea from the start that testing is animportant, necessary, and helpful part of development that occurs throughout the application life-cycle They should understand that testing happens for a reason: to find bugs before the users do.Bugs found by tests are generally much easier to fix than those found by users A well-designedtest can give you much better information about what was happening when the error occurredthan the user can A test can often lead you directly to the code that caused the error, whereas auser has no idea what the code is doing internally when the problem surfaces

Finding a bug with a test not only saves you time; it saves your users the time they would otherwisespend reporting and tracking bugs It also improves their perception that your development team buildssolid applications and that you consider the users’ time and needs important.

Agile methods require frequent builds, and help promote early and frequent testing Very few ers would release a program with absolutely no testing whatsoever If you plan for a new release everyfour weeks, you know that at least at a system level the code is tested every month

develop-That’s a start, but it’s not sufficient If the pressure to make a new release every month leads you to defertesting until the monthly build, you may end up with bugs that are deeply embedded in the code Youwill only have added at most four weeks’ work on top of the bug, but digging through that code can putyou significantly behind schedule Remember that the next build is due in only another four weeks

In any development effort, you need to build on a solid foundation You need to check every piece ofcode thoroughly so that you can move on with confidence to the next phase of development Agilemethods emphasize this because incomplete testing leads so quickly to big problems, but the same istrue no matter what development approach you use

Frequent system-level testing is good, but you need to do more

Test Early and Often

Testing should begin as soon as there is something to test Whenever programmers write a piece of codethat does something well-defined, they should write tests to prove that it works No code should betreated as working until it has been tested After you have done a thorough job of testing the code, youcan move on confidently without having to wonder if the code is stable

Developers naturally have a difficult time following this advice After they write a small piece of code,they often feel that the code is too simple to need testing They don’t know of any problems in the code(after all, if they knew of problems they would fix them), so they assume there are none

Developers continue writing code until they have a big enough piece that it won’t easily fit inside theirheads At that point, it is much easier to believe that there may be bugs in the code, so it’s more natural

to start testing (assuming developers don’t face time pressures that make it easy to put off testing evenlonger)

Unfortunately, at this point, it’s also much more difficult to find any bugs Because the developer cannotkeep the whole body of code easily in mind, it’s much more difficult to understand what might be caus-ing a bug once you find it If the code is so complex that you think it’s worth looking for bugs, then it’stoo complex to easily find the bugs

If you had tested the code in smaller pieces as each was written, the same bug would still be there I’mnot talking about changing the way you write code at this point, so you presumably would have madethe same mistake It just would have been easier to find

Instead of thinking of a small piece of code as too simple to test, you should think of it as being easier totest You don’t need to write as many test cases to cover all of the paths through a small piece of code.You don’t need to worry about complex interactions among lots of pieces of code if you’re only working

on one piece

Later, after you test the pieces individually, you can test them together At that point, you should findfewer bugs, because you have already shaken them out during the lower-level tests.

Frequent testing often changes the structure of your code If you test everything in small pieces, you will tend to break the code into small pieces Instead of writing a single long routine, you can write a smaller routine that calls several other small, closely related routines that you write and test individually.

Test Everything

No piece of code is too small or too simple to test Looked at in tiny pieces, particularly right after youhave written it, most code is simple, straightforward, and obviously correct I have probably seen hun-dreds of pieces of code that were so simple they could not possibly be wrong, and yet they contained

a bug

Test everything, no matter how simple it seems You’ll flush out bugs while they are easy to catch andbefore other developers have a chance to find them Simple code is easy to test, so you can get a feeling

of accomplishment with little extra effort

Save Tests Forever

When code is later modified, you need to test it again to see if the modifications broke the code Youshould also test the rest of the application’s code (or at least any code that could possibly have beenaffected by the changes) to see if the changes broke that code, as well Retesting old code to see if new

changes broke it is called regression testing.

Testing the modified code is a lot easier if you still have the original test code lying around If thechanges should not have affected the code’s behavior, you can rerun the tests as they are If the modifica-tions were intended to change the code’s results, you can modify the tests accordingly You may alsoneed to add new tests, but testing the altered code will be easier starting from the old tests than it would

be if you had to write new tests from scratch

Regression testing other code is practically trivial if you save the original tests You start by rerunningthose tests You may need to add new tests to exercise new features in the code, but if the modificationsare not supposed to change a piece of code’s behavior, you can use its old tests as they are

Writing tests can take as much time as writing any other kind of code, so you should treat it accordingly.Save the test code with the code it tests in a source-code control system Then, if you need to retrieve anolder version of the code, you can also retrieve its tests

Chapter 3 says a bit more about testing in its section titled “Test-Driven Development.” A point worth repeating here is that you must ensure that the test is correct If the test is wrong, it may report that the code contains a bug, when really the bug is in the testing code Usually, the bug really is in the code, but it’s worth keeping this issue in the back of your mind If it’s taking more time than you think it should

to find the bug, take a look at the testing code.

It’s also important to remember that tests can rarely guarantee that the code is correct A test cannot say the code is perfect; the test can only say that it did not find any problems.

Don’t Shoot the Messenger

Each developer should test his or her own code as soon as it is written Usually, a developer will also testhis or her own code when it is integrated into other pieces of code written by the same developer Atsome level, however, the application’s code must be tested in larger pieces that contain code written bymore than one developer At that point, egos sometimes step in and make testing argumentative andhostile

When another developer finds a bug in your code, it’s hard not to take it personally You wrote the codeand created the bug, so it’s natural to feel as if your skills are being questioned Instead of becomingdefensive, focus on the end result Someone else who finds a bug in your code saves you the embarrass-ment of having a user find the bug Though a user finding a bug may feel less personal (the user doesn’tknow who added the bug to the code), it reflects badly on the development team as a whole

Thank other developers who find your bugs because they’re doing your work for you Fix the bug, andtry to think of ways you could have caught it earlier Add new tests to the code’s test suite so that bugwould be caught in the future, and try to apply the lesson to future code

Kinds of Tests

There are several kinds of tests you should make during different stages of development You can groupthese according to the scope of the code that you are testing:

❑ Sub-unit level testing looks at the smallest pieces of code that you can effectively test

❑ Unit-level testing covers the smallest self-contained subsystems in the application

❑ Integration tests study the ways units interact

❑ System tests look at the application as a complete tool as the user sees it

Sub-Unit Tests

Sub-unit tests look for bugs in the smallest units of code that are executable They test intermediate

results that may not be intended for use by larger parts of the application These are typically helper tines used to implement tools that are used by other parts of the application

rou-For example, suppose you are writing a dispatch application that routes repair trucks to service calls.The assignment module will need helper routines that match trucks with jobs they can work (does a jobneed special equipment or skills?), calculate distances between repair trucks and the service callsthrough the street network (who is closest?), track appointment windows (when will the customer beavailable?), and so on Those routines will need other helper routines that build the street network, findshortest paths, check for street restrictions (such as a 10-ton limit on certain a bridge), track employeeassignments (who is driving which truck), and so forth

Ideally, you will have sub-unit tests for every routine that you write As soon as you write one of theseroutines, you write a test routine and verify that the routine works (Or, if you’re using test-driven devel-opment, you may write the tests first.) You write a routine to load a street network and then test it Next

you write a routine that checks employee assignments and merges employee skills with the trucks’equipment, and then test it You write a shortest-path function and test it By the time you’ve written themain assignment module, all of the routines that make it up should be tested.

In practice, most developers skip testing at this level Instead, they write all of the routines that make up

a higher-level unit of code and then test at the unit level There are two problems with that technique.First, if the unit test discovers a problem, it may be more difficult to trace back to the sub-unit routinethat caused the problem If the helper routines work relatively independently, it may not be too difficult

to track the problem If the helper routines have complex interactions, then figuring out which one isbroken can take considerable time and effort

The second problem with relying on unit-level testing is that it can be difficult to ensure that you havetested every code path through every helper routine

For example, suppose a major code unit uses five helper routines that can each take five paths of tion If you test the 5 routines separately, you need to use at least 5 * 5 = 25 sets of inputs to follow everypossible path of execution It may take some thinking, but it shouldn’t be too difficult to come up withinputs to test each of those paths

execu-In contrast, suppose you only test the five routines together in their larger unit execu-In that case, there are atotal of 55= 3,125 possible combinations of routes through the 5 routines Manually designing enoughtest cases to cover all of these possible combinations would be very difficult If you can easily generatesets of inputs randomly, you can perform 3,125 test cases easily enough, but there’s no guarantee that therandom cases will follow every possible path of execution Even if the helper routines operate indepen-dently so that you only need to cover the 25 possible paths through the individual routines and not the3,125 combinations, there’s no guarantee that you will hit them all Later, when you make a change tothe code, there’s no guarantee that the random tests will exercise the modified code, so a new bug canremain hidden until much later

If possible, test at the sub-unit level before you test at the unit level You should be able to cover eachpossible path of execution at least once a lot more easily than you can at the unit level

Unit Tests

Unit tests look for bugs in relatively self-contained pieces of code Typically, these are subsystems,

libraries, and other tools used by various parts of an application They should generally be written inseparate modules, possibly in separate class or control libraries

Whenever you have built a self-contained part of a code unit, you should test it The sooner you test thecode, the simpler it will be, so the easier it will be to fix any problems that you find

If you test the unit frequently as you add to it and modify it, you can focus on the code that has changedsince the last tests when you encounter a bug Suppose you add 20 lines to a library containing 10,000lines, test it, and find a new bug that wasn’t there before Chances are good that the new bug is in themost recent 20 lines In contrast, suppose you add 300 new lines before testing Then, if you find a newbug, you have a lot more code to study to isolate the bug

Unit testing is usually your final chance to find any bugs before the other members of the developmentteam find them and embarrass you at the next staff meeting.

And they can embarrass you! I know of one project where they tacked a rubber chicken to the door of

anyone who broke the weekly system build The follow-on project used a stuffed skunk In a third ject, a developer named Fred was so well-known for breaking the weekly build that the other developers eventually spoke of “defredding” the code every Friday afternoon.

pro-Do a thorough job of testing the unit as it was intended to be used, covering as many different situations

as possible Also, test it with abnormal and missing data Sooner or later, another developer will pass thewrong kind of data to your code and you must be prepared to handle it If your public routines do agood job of validating inputs, throwing well-described exceptions when the data is incorrect, then otherdevelopers will understand that their code is causing the problem and they can fix it without botheringyou Some projects spend a lot of time on finger-pointing competitions that are easy to avoid if youcheck the inputs to your code carefully

Integration Tests

Unit tests try to find problems in a single, more-or-less independent unit Integration tests look for bugs

when two or more system units interact

An integration test should make one unit call code in another, and then use the results in some ful way Ideally, the units contain lots of input and output validation, so the called unit validates itsinputs, calculates a result, validates the result, and returns the result to the caller The caller then vali-dates the results to ensure that they make sense If you have done a really good job of unit testing andperformed extensive validations, you should find no surprises during integration testing

meaning-One way to think of integration testing is as the process of integrating one unit with the rest of the cation That may not be strictly accurate, because integration tests often cover changes to one or moreunits, rather than newly adding a unit to the system, but the idea of focusing on a single unit for testing

appli-is useful

Aside from random bugs that should have been caught by more thorough unit testing, the most mon kind of problem discovered by integration testing occurs when two developers have different ideasabout what the interfaces between the units should look like If everyone is doing proper input and out-put validation, one unit rejects another’s input or output The developers who wrote the two units mustthen (hopefully amicably) decide which set of assumptions is correct, and make whatever changes areneeded to bring the code into line

com-System Tests

System tests look for problems with the application as a whole If you have run a rigorous set of sub-unit,

unit, and integration tests, you should find very little at this stage

Any bug that you do find should be caused by unexpected interactions among the application’s units.For example, you may need to follow a long and very specific set of steps to put the data into a state thatcauses an error Any simple operations that cause errors should have been caught during integrationtesting

During system testing, you should step through all of the use cases defined by the specification Youshould also try variations on the test cases, because it is likely that users will use those variations eitherintentionally or accidentally If possible, try executing the same steps in different orders to see what hap-pens The results should match the expected results defined by the use cases.

Try everything that the users will try, and all of the things they shouldn’t try Run tests with missing andinvalid inputs, illegal orders of operations, and anything else you can think of that the users might dowrong Make sure the program doesn’t crash, and gives the user enough information to fix the problem.The system tests are your last chance to catch bugs before the users find them Your credibility sufferswhen the users catch bugs that you should have caught during system testing Customer-reported bugsalso usually come with an extra level of management, requiring additional bug tracking, formal report-ing, and possibly definition and approval by a change committee before you can fix them Catch bugsbefore they reach the users to make things nice and simple

Regression Tests

A regression test repeats previously executed tests to see if any of the application’s behavior has changed

after you have made changes to the code A user reports a bug, you fix the offending code, and then youperform regression tests to see if the fix worked and if it broke anything else

Ideally, you would rerun every test ever written for the application to look for new problems If youhave a well-designed automated testing system, you may actually be able to do that More often, how-ever, developers only run system-level tests that exercise the feature that was modified

Whenever you make a change to the code, you should step through the code in the debugger to verifythat the change actually works It is amazing how many developers change some code and perform afew system-level tests that don’t actually execute all of the new code Developers also often verify thatthe modified code fixes the immediate problem, but don’t run enough tests to be certain that it doesn’tbreak some previously working scenario Then the bug reappears at a later date in a new guise, and youneed to debug the same code again Do a good job of regression testing the first time so that you don’thave to waste time fixing it again

Whenever you find a bug in the code, you should add new tests at every level possible to catch that bug

in the future You should add new validation tests to the code using Debug.Assert You should addnew sub-unit and unit-level tests to your existing library of testing routines If a bug in one unit is visiblefrom another unit, you should add integration tests to detect it Finally, if you have automated systemtests, you should add a new test to look for the bug If you cannot detect the bug manually, you shouldadd new scenarios to the system’s use cases to look for the bug If you add all of these tests, this bugshould never go undetected again

Test TechniquesMany developers perform an ad hoc sort of testing They write some code, throw some obvious sets ofdata at it, and if no errors occur, they assume that the code is correct Though you should not ignoreyour instincts about what data is likely to flush out errors, there are some specific techniques that youcan use to make finding any bugs more likely These techniques test the code with typical and atypicalvalues, values that are very large and very small, and values that don’t make any sense

Exhaustive Testing

In exhaustive testing, you feed every possible combination of data into a routine and you validate the

result For example, suppose your application monitors five input queues, finds the item with the est priority, and processes it If the items in the queues might have priorities ranging from 1 to 10, thereare at most 105= 100,000 combinations of priorities In that case, you could write a test that loopsthrough every possible combination of priorities, runs the code that picks out the highest-priority item,and then verifies that it selected the right item

high-Exhaustive testing guarantees that the code works correctly If you try every possible input and alwaysproduce a correct output, you have proven that the code works Unfortunately, in practice it’s oftenimpossible to exhaustively test every possible input to a routine

Suppose you have a shop-scheduling problem where you have a number of jobs that can be performed

on different pieces of equipment You know how long it takes to change a machine’s setup from one figuration to another, and you want to know the best order in which to work the jobs to minimize setuptime If you only have 5 jobs, you can try all 5 factorial (or 5! = 120) possible combinations to see if yourcode has picked the best one If you have 10 jobs, you can still check all of the 3.6 million or so possiblecombinations If you might have up to 20 jobs, however, there would be more than 2.4E18 possible com-binations, and exhaustive verification is impossible Even if you could examine 1 million combinationsper second, it would take you more than 77,000 years to try every possibility

con-Although exhaustive testing provides the best proof that a routine works, it’s usually impractical In thatcase, you need to use another method such as black box or white box testing

Black Box Testing

In black box testing, you treat the routine that you are testing as an opaque black box You put inputs in

one side and results pop out the other Because the box is black, you can’t see what’s going on inside Allyou can do it enter inputs and see if the result is correct

When you design black box tests, you can use information about the types of inputs and outputs that theapplication will actually need For example, suppose you have a routine that gives bonuses to employ-ees based on the number of sales they make during a year You may know from experience that sales-people generally sell between $1 million and $10 million worth of products per year In that case, youcould test a selection of values between 1 million and 10 million You might expand the range to covervalues between 10 thousand and 100 million just to be sure you cover all of the reasonable inputs.You should also test weird cases that are unlikely to ever occur to make sure that the program raises anappropriate error In this example, you could see what the program does if you enter $1, $0, $-10, or $1trillion You should also see what the code does if you call the routine passing it no inputs at all (if that ispossible for the routine)

These tests cover the normal, expected values, plus strange values that the code might not handle erly, but picking them didn’t require any knowledge of how the code works If you know how the codeworks, you can look for more bugs by using white box testing

prop-White Box Testing

White box testing is similar to black box testing, except that now you get to use information about how

the code works to pick inputs that are likely to cause problems

Suppose you know that the code in the previous example uses a Select Casestatement to calculatebonuses that increase when a salesperson sells $1 million, $2 million, $5 million, and $10 million In thatcase, you could test the code at these breakpoint values and at values one penny greater or less

Random Testing

Random testing is exactly what it sounds like: trying the code with an assortment of randomly generated

values Because you have no way of knowing how likely a random value is to detect a bug, you usuallyneed to run a lot of random tests to have much assurance that the code works correctly

For example, consider the previous sales bonus example and suppose the code has trouble with valuesthat are less than one dollar below the cutoff values $1 million, $2 million, $5 million, and $10 million Ifyou generate random dollar amounts between $0 and $100 million, you will have about a 4 in 100 mil-lion chance of randomly picking a value that will uncover the bug If you run 100 thousand tests, youwill only have about a 33 percent chance of detecting the bug You’ll need to run around 1 million tests

to have a 98 percent chance of finding this bug

Random testing is not a substitute for black box and white box testing It simply gives you a chance tocatch bugs using inputs that you didn’t know would cause problems The best strategy is to use all threemethods: black box, white box, and random testing Test the most typical inputs, strange inputs thatmight give the code problems, and a large number of random inputs

Testing Mechanics

To test a routine, you can make another routine that calls it with various inputs and verifies that the puts are correct To make tracking the test routines easier, use a simple naming convention so that theyare easy to find For example, suppose you have a ComplexNumberclass You could put the routines thattest that class in a module named Test_ComplexNumber.vb

out-You can embed the test routines in the main application, or you can make an external testing application

Testing Inside the Application

To embed test routines in the main application, simply create new testing modules in the application Torun the tests, add one or more testing forms to the project Give these forms meaningful names that aresimilar to the test module names so that they’re also easy to find For example, you might call a test form

If the test form can set up the tests easily by itself, you can change the program’s startup form to the testform and run the program Open Solution Explorer, double-click My Project, click the Application tab,and select the form in the “Startup form” drop-down, as shown in Figure 17-1 Now, when you run theapplication, the test form starts and you can perform the tests.

Figure 17-1: Use the “Startup form” drop-down to select the test form

When you are finished testing, use the “Startup form” drop-down again to change the startup form back

to the application’s main form

Sometimes setting up the data to run the tests is difficult It may take you quite a while working throughthe application’s normal forms before you get the data in a state where you can test it In that case, itmay be easier to display the test form after you have worked the application into the proper state Add amenu item or secret key sequence that displays the test form, and hide this shortcut from the users.One way to hide the test menu is to use a custom constant In Solution Explorer, double-click My Project,click the Compile tab, and click the Advanced Compile Options button to display the dialog shown inFigure 17-2 Enter custom constants in the indicated text box The dialog in Figure 17-2 sets the

Rather than creating your own constant, you can use the pre-defined constant DEBUG Then, the

application will show the test menu in debug builds and not in release builds.

Figure 17-2: Enter custom constants in the Advanced Compiler Settings dialog.

Now, write code to check the constant and show or hide the test menu The following code shows the

the menu mnuTestsvisible and enabled If the constant is missing or False, it hides and disables themenu

Private Sub frmMain_Load(ByVal sender As System.Object, _ByVal e As System.EventArgs) Handles MyBase.Load

‘ Hide the Tests menu unless the SHOW_TEST_MENU

‘ command line argument is defined

#If SHOW_TEST_MENU ThenmnuTests.Visible = TruemnuTests.Enabled = True

#ElsemnuTests.Visible = FalsemnuTests.Enabled = False

#End IfEnd Sub

The following code shows how the program displays the test form when you select the Testmenu’s

Main Test Formcommand It simply displays the test form modally The code is contained in an #If

block, so it is omitted when the SHOW_TEST_MENUconstant is undefined or False

‘ Display the test form.

Private Sub mnuTestsMainTestForm_Click(ByVal sender As System.Object, _

ByVal e As System.EventArgs) Handles mnuTestsMainTestForm.Click

#If SHOW_TEST_MENU Then

Dim frm As New Test_frmMainfrm.ShowDialog()

pro-Because you used a naming convention for the test modules and forms (for example, starting each with

Test), you can easily find and remove those modules from the project The only remaining references tothe testing modules should be in the code that displays the test form In the previous example, that code

is contained in the mnuTestsMainTestForm_Clickevent handler You can hide the code by setting the

Over time, references from the main program to the testing often sneak into the code, so removing the

testing code isn’t as easy as you would hope You can prevent those sorts of dependencies by periodically going through the exercise of removing the testing code to see if you can build the application without it.

If you make this build fairly regularly, you won’t have to make many changes when you need to build a release for the users.

Testing Outside the Application

If you test from within the application, you must be able to hide the test code from users For example,you may need to hide and disable testing menus You will also eventually want to remove the testingfrom the finished application

Another approach is to make a separate testing application to exercise the code This works best if youbreak your code into separate class libraries Make a new executable application and add a reference toone or more of the libraries Now, add whatever code you need to test the routines in the libraries.Because the test application lies outside of the libraries, it can only call their public methods, so it canonly perform tests at the unit level and higher It cannot see the methods private to the libraries, so itcannot perform sub-unit level testing

The test program also cannot see code placed in the application’s main form, Sub Main, and other codethat is not contained in libraries This encourages you to put as much code as possible in libraries Thathelps decouple the different parts of the application, so that usually gives you a more robust design inthe end

This also encourages you to separate the application’s code from its user interface That also generallyleads to more modular designs that are easier to implement, test, and debug The result is a thin userinterface sitting on top of a rich set of libraries