Testing Basics

Although it doesn't get a lot of importance, testing is arguably the most important phase in any software development project. Certainly testing is —the only— way to know whether your project or product is destined for success or doomed to failure before it goes live.


Testing involves operation of a system or application under controlled conditions and evaluating the results. The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn't or things don't happen when they should. It is oriented to 'detection'.


The U.S. Commerce Department has estimated that buggy software costs nearly $60 billion annually, and that at least $22.2 billion worth of those bugs could have been prevented with more thorough testing.


And although you might like to think that your newly coded application is virtually bug-free, we all know by now that it just ain't so. Note that there's simply no such thing as an error-free application. There is no software that is perfect.

Defect can be caused by a flaw in the application software or by a flaw in the application specification. For example, unexpected (incorrect) results can be from errors made during the construction phase, or from an algorithm incorrectly defined in the specification. Testing is commonly assumed to mean executing software and finding errors. This type of testing is known as dynamic testing, and while valid, it is not the most effective way of testing. Static testing, the review, inspection and validation of development requirements, is the most effective and cost efficient way of testing. A structured approach to testing should use both dynamic and static testing techniques.


There are many, many different ways to test software, but most tend to fall into two broad categories: functional testing and performance testing.


Functional Testing

The functional testing focuses on the program's functionality against the specification.
Functional testing covers how well the system executes the functions it is supposed to execute—including user commands, data manipulation, searches and business processes, user screens, and integrations.


Although functional testing is often done toward the end of the development cycle, it can—and should, say experts—be started much earlier. Individual components and processes can be tested early on, even before it's possible to do functional testing on the entire system.


Performance Testing

This term is often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans. Performance testing is perhaps the biggest area of testing.


Performance testing can be defined as the testing conducted to evaluate the compliance of a system or component with specified performance requirements. Often this is performed using an automated test tool to simulate large number of users.


The different types of performance testing that can be done are:

• Classic performance testing, which measures response times and transaction rates
• Load testing, which measures the system's ability to handle varied workloads
• Stress testing, which looks for errors produced by low resources or competition for resources
• Volume testing, which subjects the software to larger and larger amounts of data to determine its point of failure

Key Objectives of Performance Testing

• Plan Approach - What is it that you are trying to measure and why? You need to understand your motive of performance testing your application
• Create a plan and a model - After planning your approach, invest time and effort into creating your capacity and performance model. The approach should include the key inputs , key items to measure and how you plan to measure.
• Implement Approach - Roll out your performance testing approach, including your measurement, profiling and benchmarking tools and processes, to start feeding your capcity model. At this point of your testing, you will most likely iterate for a defined period, inputting the data from profiling and benchmarking into your performance testing model.

How to Report Bugs Effectively

Introduction

Anybody who has written software for public use will probably have received at least one bad bug report. Reports that say nothing ("It doesn't work!"); reports that make no sense; reports that don't give enough information; reports that give wrong information. Reports of problems that turn out to be user error; reports of problems that turn out to be the fault of somebody else's program; reports of problems that turn out to be network failures.

There's a reason why technical support is seen as a horrible job to be in, and that reason is bad bug reports. However, not all bug reports are unpleasant: I maintain free software, when I'm not earning my living, and sometimes I receive wonderfully clear, helpful, informative bug reports.

In this essay I'll try to state clearly what makes a good bug report. Ideally I would like everybody in the world to read this essay before reporting any bugs to anybody. Certainly I would like everybody who reports bugs to me to have read it.

In a nutshell, the aim of a bug report is to enable the programmer to see the program failing in front of them. You can either show them in person, or give them careful and detailed instructions on how to make it fail. If they can make it fail, they will try to gather extra information until they know the cause. If they can't make it fail, they will have to ask you to gather that information for them.

In bug reports, try to make very clear what are actual facts ("I was at the computer and this happened") and what are speculations ("I think the problem might be this"). Leave out speculations if you want to, but don't leave out facts.

When you report a bug, you are doing so because you want the bug fixed. There is no point in swearing at the programmer or being deliberately unhelpful: it may be their fault and your problem, and you might be right to be angry with them, but the bug will get fixed faster if you help them by supplying all the information they need. Remember also that if the program is free, then the author is providing it out of kindness, so if too many people are rude to them
then they may stop feeling kind.

"It doesn't work."

Give the programmer some credit for basic intelligence: if the program really didn't work at all, they would probably have noticed. Since they haven't noticed, it must be working for them. Therefore, either you are doing something differently from them, or your environment is different from theirs. They need information; providing this information is the purpose of a bug report. More information is almost always better than less.

Many programs, particularly free ones, publish their list of known bugs. If you can find a list of known bugs, it's worth reading it to see if the bug you've just found is already known or not. If it's already known, it probably isn't worth reporting again, but if you think you have more information than the report in the bug list, you might want to contact the programmer anyway. They might be able to fix the bug more easily if you can give them information they didn't
already have.

This essay is full of guidelines. None of them is an absolute rule. Particular programmers have particular ways they like bugs to be reported. If the program comes with its own set of bug-reporting guidelines, read them. If the guidelines that come with the program contradict the guidelines in this essay, follow the ones that come with the program!

If you are not reporting a bug but just asking for help using the program, you should state where you have already looked for the answer to your question. ("I looked in chapter 4 and section 5.2 but couldn't find anything that told me if this is possible.") This will let the programmer know where people will expect to find the answer, so they can make the documentation easier to use.

"Show me."

One of the very best ways you can report a bug is by showing it to the programmer. Stand them in front of your computer, fire up their software, and demonstrate the thing that goes wrong. Let them watch you start the machine, watch you run the software, watch how you interact with the software, and watch what the software does in response to your inputs.

They know that software like the back of their hand. They know which parts they trust, and they know which parts are likely to have faults. They know intuitively what to watch for. By the time the software does something obviously wrong, they may well have already noticed something subtly wrong earlier which might give them a clue. They can observe everything the computer does during the test run, and they can pick out the important bits for themselves.

This may not be enough. They may decide they need more information, and ask you to show them the same thing again. They may ask you to talk them through the procedure, so that they can reproduce the bug for themselves as many times as they want. They might try varying
the procedure a few times, to see whether the problem occurs in only one case or in a family of related cases. If you're unlucky, they may need to sit down for a couple of hours with a set of
development tools and really start investigating. But the most important thing is to have the programmer looking at the computer when it goes wrong. Once they can see the problem
happening, they can usually take it from there and start trying to fix it.

"Show me how to show myself."

This is the era of the Internet. This is the era of worldwide communication. This is the era in which I can send my software to somebody in Russia at the touch of a button, and he can send me comments about it just as easily. But if he has a problem with my program, he can't have me standing in front of it while it fails. "Show me" is good when you can, but often you can't.

If you have to report a bug to a programmer who can't be present in person, the aim of the exercise is to enable them to reproduce the problem. You want the programmer to run their
own copy of the program, do the same things to it, and make it fail in the same way. When they can see the problem happening in front of their eyes, then they can deal with it.

So tell them exactly what you did. If it's a graphical program, tell them which buttons you pressed and what order you pressed them in. If it's a program you run by typing a command, show them precisely what command you typed. Wherever possible, you should provide a
verbatim transcript of the session, showing what commands you typed and what the computer output in response.

Give the programmer all the input you can think of. If the program reads from a file, you will probably need to send a copy of the file. If the program talks to another computer over a network, you probably can't send a copy of that computer, but you can at least say what kind of computer it is, and (if you can) what software is running on it.

"Works for me. So what goes wrong?"

If you give the programmer a long list of inputs and actions, and they fire up their own copy of the program and nothing goes wrong, then you haven't given them enough information. Possibly the fault doesn't show up on every computer; your system and theirs may differ in some way. Possibly you have misunderstood what the program is supposed to do, and you are both looking at exactly the same display but you think it's wrong and they know it's right.

So also describe what happened. Tell them exactly what you saw. Tell them why you think what you saw is wrong; better still, tell them exactly what you expected to see. If you say "and then it went wrong", you have left out some very important information.

If you saw error messages then tell the programmer, carefully and precisely, what they were. They are important! At this stage, the programmer is not trying to fix the problem: they're just
trying to find it. They need to know what has gone wrong, and those error messages are the computer's best effort to tell you that. Write the errors down if you have no other easy way to remember them, but it's not worth reporting that the program generated an error unless you can also report what the error message was.

In particular, if the error message has numbers in it, do let the programmer have those numbers. Just because you can't see any meaning in them doesn't mean there isn't any. Numbers contain all kinds of information that can be read by programmers, and they
are likely to contain vital clues. Numbers in error messages are there because the computer is too confused to report the error in words, but is doing the best it can to get the important information to you somehow.

At this stage, the programmer is effectively doing detective work. They don't know what's happened, and they can't get close enough to watch it happening for themselves, so they are searching for clues that might give it away. Error messages, incomprehensible strings of
numbers, and even unexplained delays are all just as important as fingerprints at the scene of a crime. Keep them!

If you are using Unix, the program may have produced a core dump. Core dumps are a particularly good source of clues, so don't throw them away. On the other hand, most programmers don't like to receive huge core files by e-mail without warning, so ask before mailing one to anybody. Also, be aware that the core file contains a record of the complete state of the program: any "secrets" involved (maybe the program was handling a personal message, or dealing with confidential data) may be contained in the core file.

"So then I tried . . ."

There are a lot of things you might do when an error or bug comes up. Many of them make the problem worse. A friend of mine at school deleted all her Word documents by mistake, and before calling in any expert help, she tried reinstalling Word, and then she tried running
Defrag. Neither of these helped recover her files, and between them they scrambled her disk to the extent that no Undelete program in the world would have been able to recover anything. If she'd only left it alone, she might have had a chance.

Users like this are like a mongoose backed into a corner: with its back to the wall and seeing certain death staring it in the face, it attacks frantically, because doing something has to be better than doing nothing. This is not well adapted to the type of problems
computers produce.

Instead of being a mongoose, be an antelope. When an antelope is confronted with something unexpected or frightening, it freezes. It stays absolutely still and tries not to attract any attention, while it stops and thinks and works out the best thing to do. (If antelopes had a technical support line, it would be telephoning it at this point.) Then, once it has decided what the safest thing to do is, it does it.

When something goes wrong, immediately stop doing anything. Don't touch any buttons at all. Look at the screen and notice everything out of the ordinary, and remember it or write it down.
Then perhaps start cautiously pressing "OK" or "Cancel", whichever seems safest. Try to develop a reflex reaction - if a computer does anything unexpected, freeze.

If you manage to get out of the problem, whether by closing down the affected program or by rebooting the computer, a good thing to do is to try to make it happen again. Programmers like problems that they can reproduce more than once. Happy programmers fix bugs faster and
more efficiently.

"I think the tachyon modulation must be
wrongly polarised."

It isn't only non-programmers who produce bad bug reports. Some of the worst bug reports I've ever seen come from programmers, and even from good programmers.

I worked with another programmer once, who kept finding bugs in his own code and trying to fix them. Every so often he'd hit a bug he couldn't solve, and he'd call me over to help. "What's gone wrong?" I'd ask. He would reply by telling me his current opinion of what needed to be fixed.

This worked fine when his current opinion was right. It meant he'd already done half the work and we were able to finish the job together. It was efficient and useful.

But quite often he was wrong. We would work for some time trying to figure out why some particular part of the program was producing incorrect data, and eventually we would discover that it wasn't, that we'd been investigating a perfectly good piece of code for half an hour, and that the actual problem was somewhere else.

I'm sure he wouldn't do that to a doctor. "Doctor, I need a prescription for Hydroyoyodyne." People know not to say that to a doctor: you describe the symptoms, the actual discomforts and aches and pains and rashes and fevers, and you let the doctor do the diagnosis of what the problem is and what to do about it. Otherwise the doctor dismisses you as a hypochondriac or crackpot, and quite rightly so.

It's the same with programmers. Providing your own diagnosis might be helpful sometimes, but always state the symptoms. The diagnosis is an optional extra, and not an alternative to giving the symptoms. Equally, sending a modification to the code to fix the problem is a useful addition to a bug report but not an adequate substitute for one.

If a programmer asks you for extra information, don't make it up! Somebody reported a bug to me once, and I asked him to try a command that I knew wouldn't work. The reason I asked him to try it was that I wanted to know which of two different error messages it would give. Knowing which error message came back would give a vital clue. But he didn't actually try it - he just mailed me back and said "No, that won't work". It took me some time to persuade him to try it for real.

Using your intelligence to help the programmer is fine. Even if your deductions are wrong, the programmer should be grateful that you at least tried to make their life easier. But report the
symptoms as well, or you may well make their life much more difficult instead.

"That's funny, it did it a moment ago."

Say "intermittent fault" to any programmer and watch their face fall. The easy problems are the ones where performing a simple sequence of actions will cause the failure to occur. The programmer can then repeat those actions under closely observed test conditions and watch what happens in great detail. Too many problems simply don't work that way: there will be programs that fail once a week, or fail once in a blue moon, or never fail when you try them in
front of the programmer but always fail when you have a deadline coming up.

Most intermittent faults are not truly intermittent. Most of them have some logic somewhere. Some might occur when the machine is running out of memory, some might occur when another program tries to modify a critical file at the wrong moment, and some might occur only in the first half of every hour! (I've actually seen one of these.)

Also, if you can reproduce the bug but the programmer can't, it could very well be that their computer and your computer are different in some way and this difference is causing the problem. I had a program once whose window curled up into a little ball in the top left corner of the screen, and sat there and sulked. But it only did it on 800x600 screens; it was fine on my 1024x768 monitor.

The programmer will want to know anything you can find out about the problem. Try it on another machine, perhaps. Try it twice or three times and see how often it fails. If it goes wrong when you're doing serious work but not when you're trying to demonstrate it, it might
be long running times or large files that make it fall over. Try to remember as much detail as you can about what you were doing to it when it did fall over, and if you see any patterns, mention them. Anything you can provide has to be some help. Even if it's only
probabilistic (such as "it tends to crash more often when Emacs is running"), it might not provide direct clues to the cause of the problem, but it might help the programmer reproduce it.

Most importantly, the programmer will want to be sure of whether they're dealing with a true intermittent fault or a machine-specific fault. They will want to know lots of details about your computer, so they can work out how it differs from theirs. A lot of these details will depend on the particular program, but one thing you should definitely be ready to provide is version numbers. The version number of the program itself, and the version number of the
operating system, and probably the version numbers of any other programs that are involved in the problem.

"So I loaded the disk on to my Windows . . ."

Writing clearly is essential in a bug report. If the programmer can't tell what you meant, you might as well not have said anything.

I get bug reports from all around the world. Many of them are from non-native English speakers, and a lot of those apologise for their poor English. In general, the bug reports with apologies for their poor English are actually very clear and useful. All the most unclear reports come from native English speakers who assume that I will understand them even if they don't make any effort to be clear or precise.

• Be specific. If you can do the same thing two different ways, state which one you used. "I selected Load" might mean "I clicked on Load" or "I pressed Alt-L". Say which you did. Sometimes it matters.
  
  
• Be verbose. Give more information rather than less. If you say too much, the programmer can ignore some of it. If you say too little, they have to come back and ask more questions. One bug report I received was a single sentence; every time I asked for more
  information, the reporter would reply with another single sentence. It took me several weeks to get a useful amount of information, because it turned up one short sentence at a time.
  
  
• Be careful of pronouns. Don't use words like "it", or references like "the window", when it's unclear what they mean. Consider this: "I started FooApp. It put up a warning window. I
  tried to close it and it crashed." It isn't clear what the user tried to close. Did they try to close the warning window, or the whole of FooApp? It makes a difference. Instead, you could say "I started FooApp, which put up a warning window. I tried to close the
  warning window, and FooApp crashed." This is longer and more repetitive, but also clearer and less easy to misunderstand.
  
  
• Read what you wrote. Read the report back to yourself, and see if you think it's clear. If you have listed a sequence of actions which should produce the failure, try following them
  yourself, to see if you missed a step.
  

Summary

• The first aim of a bug report is to let the programmer see the failure with their own eyes. If you can't be with them to make it fail in front of them, give them detailed instructions so that they can make it fail for themselves.
  
  
• In case the first aim doesn't succeed, and the programmer can't see it failing themselves, the second aim of a bug report is to describe what went wrong. Describe everything in
  detail. State what you saw, and also state what you expected to see. Write down the error messages, especially if they have numbers in.
  
  
• When your computer does something unexpected, freeze. Do nothing until you're calm, and don't do anything that you think might be dangerous.
  
  
• By all means try to diagnose the fault yourself if you think you can, but if you do, you should still report the symptoms as well.
  
• Be ready to provide extra information if the programmer needs it. If they didn't need it, they wouldn't be asking for it. They aren't being deliberately awkward. Have version numbers at your fingertips, because they will probably be needed.
  
  
• Write clearly. Say what you mean, and make sure it can't be misinterpreted.
  
  
• Above all, be precise. Programmers like precision.
  
  

---

Originally written by Simon Tatham, professional and free-software programmer

Disclaimer: I've never actually seen a mongoose or an antelope. My zoology may be inaccurate.

Copyright © 1999 Simon Tatham.

This document is OpenContent.

You may copy and use the text under the terms of the OpenContent Licence.

Free Black Box Testing Course Video

Cem Kaner has posted, free, an evolving online version (with video, course notes, examples, exercises, papers, tests, and more) of his Black Box Software Testing course.

From the course overview:"Black box testing is the craft of testing a program from the external view. We look at how the program operates in its context, getting to know needs and reactions of the users, hardware and software platforms, and programs that communicate with it."This course is an introduction to black box testing. It is a superset of the Software Testing 1 introductory courses that Florida Tech requires in its undergraduate (CSE 3411) and graduate (SWE 5411) software engineering degree programs.

The full set of materials are equivalent to about a two-semester course."...and from the learning objectives for the course:"Testing is a type of technical investigation. If you think of it as a routine task, you'll do crummy testing. So, the essence of what I hopeyou'll learn in the course is how to investigate, how to plan your investigation, how to report the results, and how to tell whether you're doing a good job. Excellence in the course will require you to extend your critical thinking, writing, and teamworking skills. Finally, the work that you do in this course might help you land a job.

Many of the course tasks were designed to be realistic or impressive (to an employer) and to give you a chance to do professional-quality work that you can show off during a job interview.

http://www.testingeducation.org/BBST/index.html

Test Driven Development

TDD is one of the core methodologies of “Extreme Programming”. Kent Beck is founder of Extreme Programming. Extreme Programming (aka XP) has been successful as it emphasis more on Customer Satisfaction. XP encourages to keep the design very simple and straight forward. XP significantly changes the way we code, it drifts from the traditional style of coding. XP is best suited when the requirements change dynamically. We might have a system whos functionality changes every few months. Some times requirements change when the customer is not very clear about what he wants. Lastly the developer’s productivity increases using XP.

Test Driven Development is known as “Test First”. TDD lays more stress on through unit testing. It’s a usual practice to write and execute test cases after the coding is complete. Goal of TDD is to write a clean code. But in TDD, first we write test cases, write the code and execute test cases. So that’s the reason it’s called “Test First”. It requires a different mind set from developers.

In tradition software practice we follow the below steps.
Create the Technical Design Document.
Write the Code.
Create Unit Test Cases.
Test the code based on Unit Test Cases
Create the build and pass on to the testing team.

Pit falls of Existing System.
o Most of the Bugs (which are of course nightmares of developer community) are discovered during the test cycles, rather than at unit test phase.
o Identifying the root cause of bugs becomes very difficult.o Code becomes tough to manage.o We can never be sure that every line of our code is tested.

Implementing TDD.
TDD takes a different approach compared to traditional software development. Here a test case for a specific functionality is written rather than code. The functional code is written only to pass that test cases, one good thing is that duplication of code is avoided.

Rules of TDD are
o Test comes first.
o Write a functional code only if a test case fails.
o Make sure that your code is testing 100%.
o It always better to use Code coverage tools.

Initially, adopting TDD can be pain for developer community. Natural tendency of developers is to write code first and test next. More over they feel it’s too much of writing test cases rather coding. Developers are not used to do write and execute large no. of test cases. To implement TDD there are two requirements which are must.
o A faster complier.
o An automated tool for executing the test cases.
o Always use production data for your unit tests.
o In case if you don’t have production data on time, spend some time to create test data similar to production data.

Automated tool is a must for TDD. Without which executing a unit test case will be very difficult and becomes more time consuming. I have listed available tools for different languages in below section.

Benefits of TDD
o Biggest Benefit is verification. We can have exhausite set of automated unit test cases that can protect our system against defects.
o It is best suited to the systems where requirements are very dynamic.
o We can get simple and clear design.
o Code becomes loosely coupled and much easier to manager.
o Test code provides a complete documentation.
o It ensures that the system is free from defects.
o Avoids duplication of code.
o QA time is saved as many defects as many of bugs are worked out up front.
o Sytem what we develop exacts does it needs to do and no more.
o High quality unit tests can significantly augment the quality of code and can generate a tremendous amount of confidence in the integrity of tested code.

--------------------------------------------------------------

Originally written by
Sudheer Palyam http://spaces.msn.com/members/sudheerpalyam/PersonalSpace.aspx?_c=

Testing is easy, what’s the big deal?

Deepak had come down for an interview at a top MNC. He sat down in a chair, glanced around the conference hall.

“Actually,” He confessed, “even though I’m here for an interview for the testing position, I really want a job in development.” He gave his interviewer, Satish, a conspiratorial smile. “I’ve been in testing for two years,” He explained, “and I think I’ve pretty much done it all at this point. I’m ready to move on.”

Satish, who had been in testing for four times as many years and felt as if he’d just scratched the surface, was taken aback. Hmmm, he thought. Let’s see how good he is. Standing at an empty whiteboard, Satish sketched a dialog box with four fields and a button. He labeled the fields and then stepped back. “Here’s a form from a functional system. Given what you see, what tests would you want to run?” Satish handed him the pen in case he wanted to draw his ideas on the board.

“Well,” Deepak began without moving from his seat, “I’d try some boundary conditions, some negative tests, a use case or two, and I’d be done.” He looked very satisfied with his answer. Satish smiled. “And you’d be done?”

Deepak frowned as he thought for a moment, and then smiled in return. “Yup. I’d be done.” Satish’s smile dissolved. “Deepak, most of the candidates we have through here would have filled that whiteboard with tests. You’ve barely begun to explore the testing problem I posed to you.” Satish paused to let his criticism sink in.

“Want to try again?”

Deepak came up with a few more tests, but not many. He had no idea what Satish was looking for, and He didn’t particularly appreciate his attitude.

Testing is easy, He thought. What’s the big deal?

The big deal isn’t how many tests Deepak surfaced as much as his attitude toward testing. He thinks he already knows all there is to know. Satish, on the other hand, recognizes that professional growth can take a lifetime. He knows that no matter how long he spends in the testing field, he’ll still have more to learn. He takes time to nurture his skills, grow his abilities, and feed his mind. As a result, Satish will continue to progress in his testing career. Deepak’s career stalled after just two short years, while Satish is continuing to grow professionally and personally. How can you assure your continued professional growth?

Testing is almost infinite - Avoid contentment
Deepak was quite satisfied with his answer to Satish’s testing problem. He felt that he’d covered everything. But since testing is an almost infinite problem, no answer would be complete. Deepak had fallen into the trap of complacency. Testers like Satish know that no matter how many clever tests they’ve already thought up, there are more tests they could be running and that testing is infinite.



Consider the above example. How many test cases can you think you can write for this?
If we want to write one test case for each and every path that the above flowchart shows, there are 10 to the power of 14 possible paths! If we execute one test per millisecond, it would take 3.17 years to test this program!!

Whenever there is a lot of testing to be done and not enough time to do the testing, we have to prioritize so that at least the most important things get done. In testing, there's never enough time or resources. In addition, the consequences of skipping something important are severe. In such a scenario, we have to prioritize and make out how to find the most important bugs first.So prioritization has received a lot of attention. The approach is called Risk Driven Testing, where Risk has very specific meaning.
We have to prioritize testing by Risk. That makes sure that we're doing the most important things first. But we also need to allow some time to get some coverage of the less risky features and usage scenarios. Risk drives good testing!!!

Study Bugs
Satish has more than a passing curiosity about bugs. He seeks to understand them:
· Why did the software behave that way?
· How can I search out related bugs in other areas?
· What was the root cause?

It’s good if you study the bugs that have already been found in the software you’re testing. It’s even better if you study other people’s bugs, too. The next time you get together with a group of testers, swap bug stories. Meetings with other team members provide an ideal forum for a friendly chat about bugs with other testers working on different kinds of software.

No matter how long you spend in the testing field, you’ll still have more to learn.

But Do You Know if It's Any Good

Here is a good article I came acrross long time back... Originally written by Elisabeth Hendrickson for the computerworld magazine, titled "But do you know if it's any good"

"This new system is driving me crazy!" Janet, the hotel desk attendant, muttered as she punched at the keyboard buttons. She looked back at me, flashing her best customer smile. "Sorry, it will be a minute."

She returned to scowling at the keyboard. Apparently the system finally accepted her input; she looked up at me with a satisfied expression. There was a pause as we waited for the system to respond. A long pause.

To fill the time, she asked, "So Ms. Hendrickson, what do you do?"

"I work with software development organizations to improve software quality."

Read the complete article here... http://www.computerworld.com/action/article.docommand=viewArticleBasic&articleId=76551

Testing is Development too!

A lot of developers consider testing to be a bit of a necessary evil. They don't want to write test plans. They don't even want to write test cases. Some of them don't view QA as being in the same ballpark as development.

Designing and developing test harnesses and test case code can be every bit as challenging - and rewarding - as designing and developing the product itself.

Some of the challenges include:
figuring out how to automate as much as possible, not only of the actual test case execution but also of the result verification and regression figuring out how to create negative test cases - to test and verify error cases - that run automatically and accurately figuring out the edge cases in the spec, the boundary conditions, the possible off-by-one errors Developers tend to focus on positive execution paths and often don't worry about boundary conditions (yes, they should!).

So, as a developer, take some time to think about life on the other side of the looking glass. Think about test cases. Think about how you would plan all those test cases. Think about automating those tests.

author - seancorfield

Risk Driven Testing

Whenever there is a lot of testing to be done and not enough time to do the testing, we have to prioritize so that at least the most important things get done.
In such a scenario, we have to prioritize and make out How to find the most important bugs first.

How to find the most important bugs first?
In testing, there's never enough time or resources. In addition, the consequences of skipping something important are severe.

So prioritization has received a lot of attention. The approach is called Risk Driven Testing, where Risk has very specific meaning.

Here's how you do it:

Take the pieces of your system, whatever you use - modules, functions, section of the requirements - and rate each piece on two variables, Impact and Likelihood.

Impact is what would happen if this piece somehow malfunctioned. Would it destroy the customer database? Or would it just mean that the column headings in a report didn't quite line up?

Likelihood is an estimate of how probable it is that this piece would fail.

Together, Impact and Likelihood determine Risk for the piece.

For example, suppose we're testing new two wheeler as they come off the production line. We only have one hour to test each vehicle.

We could easily justify spending all that time testing the brakes. After all, if the brakes don't work right, there'll be a really large Impact - literally, in this case. Therefore, while it's important to see that the vehicle runs, it's even more important to be sure that if it does run, it will stop.

Just as in testing software, we say that something "works" if it works every time, under all combinations of conditions.

So we could soak the brakes with water, freeze them, overheat them, and so on - all the standard engineering tests. We could spend the whole time doing this.

But we also have to make sure that the vehicle starts, accelerates, check that the lights, horn etc work.

Unfortunately, in the real world there's never enough time to do all the testing we'd like to do. So, as in the above example, we have to prioritize testing by Risk. That makes sure that we're doing the most important things first. But we also need to allow some time to get some coverage of the less risky features and usage scenarios.

Risk drives good testing!!!

Software Development Life Cycle

Software Development Life Cycle (SDLC) is the overall process of developing information systems through a multistep process from investigation of initial requirements through analysis, design, implementation and maintenance. There are many different models and methodologies, but each generally consists of a series of defined steps or stages.

Once upon a time, software development consisted of a programmer writing code to solve a problem or automate a procedure. Nowadays, systems are so big and complex that teams of architects, analysts, programmers, testers and users must work together to create the millions of lines of custom-written code that drive our enterprises. To manage this, a number of system development life cycle (SDLC) models have been created: waterfall, fountain, spiral, build and fix, rapid prototyping, incremental, and synchronize and stabilize.

The oldest of these, and the best known, is the waterfall: a sequence of stages in which the output of each stage becomes the input for the next. These stages can be characterized and divided up in different ways, including the following:

Project planning, feasibility study: Establishes a high-level view of the intended project and determines its goals.

Systems analysis, requirements definition: Refines project goals into defined functions and operation of the intended application. Analyzes end-user information needs.

Systems design: Describes desired features and operations in detail, including screen layouts, business rules, process diagrams, pseudocode and other documentation.

Implementation: The real code is written here.

Integration and testing: Brings all the pieces together into a special testing environment, then checks for errors, bugs and interoperability.

Acceptance, installation, deployment: The final stage of initial development, where the software is put into production and runs actual business.

Maintenance: What happens during the rest of the software's life: changes, correction, additions, moves to a different computing platform and more. This, the least glamorous and perhaps most important step of all, goes on seemingly forever.

Software Testing Terminology

A few simple definitions of Testing Terminology.

Error - The difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition.

Fault - An incorrect step, process, or data definition in a computer program.

Debug - To detect, locate, and correct faults in a computer program.

Failure - The inability of a system or component to perform its required functions within specified performance requirements. It is manifested as a fault.

Testing - The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software items.

Static analysis - The process of evaluating a system or component based on its form, structure, content, or documentation.

Dynamic analysis - The process of evaluating a system or component based on its behavior during execution.

Correctness - The degree to which a system or component is free from faults in its specification, design, and implementation. The degree to which software, documentation, or other items meet specified requirements. The degree to which software, documentation, or other items meet user needs and expectations, whether specified or not.

Verification - The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Formal proof of program correctness.

Validation - The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.

Rapid Testing

Rapid Software Testing - The style of rapid testing is tester-centric (as opposed to technique-centric) and is a blend of heuristic testing, risk-based testing, and exploratory testing.

How is Rapid Software Testing different from normal software testing?

Software Testing practice differs from industry to industry, company to company, tester to tester and person to person. But there are some elements that most test projects have in common. Let's call those common elements "normal testing". In our experience, normal testing involves writing test cases against some kind of specification. These test cases are fragmentary
plans or procedures that loosely specify what a tester will do to test the product. The tester is then expected to perform these test cases on the product, repeatedly, throughout the course of the project.

Rapid testing differs from traditional testing in several major ways:

• Don’t Waste Time. The most rapid action is no action at all. So, in rapid testing we eliminate any activity that isn't necessary. Traditional testing, by comparison, is a bloated
  mess. It takes some training and experience to know what to eliminate, of course. In general, streamline documentation and devote testing to primarily to risk areas. Don't repeat tests just because someone told you that repetition is good. Make sure that you get good information value from every test. Consider the opportunity cost of each testing activitiy.

• Mission. In Rapid Testing we don't start with a task ("write test cases"), we start with a
  mission. Our mission may be "find important problems fast". If so, then writing test cases may not be the best approach to the test process. If, on the other hand, our mission is "please the FDA auditors", then we not only will have to write test cases, we'll have to write certain kinds of test cases and present them in a specifically approved format. Proceeding from an understanding of our mission, we take stock of our situation and look for the most efficient and effective actions we can take right now to move towards fulfilling that mission.

• Skills. To do any testing well requires skill. Normal testing downplays the importance of skill by focusing on the format of test documentation rather than the robustness of
  tests. Rapid Testing, as we describe it, highlights skill. It isn't a mechanical technique like making microwave popcorn, or filling out forms at the DMV. Robust tests are very important, so we practice critical thinking and experimental design skills. A novice tester will not do RT very well unless supervised and coached by a senior tester who is trained (or self-trained) in the art. We hope the articles and presentations on this site will help you work on those skills.

• Risk. Normal testing is focused on functional and structural product coverage. In other words, if the product can do X, then try X. Rapid Testing focuses on important problems. We gain an understanding of the product we're testing to the point where we can imagine what kinds of problems are more likely to happen and what problems would have more impact if they happened.Then we put most of our effort into testing for those problems. Rapid Testing is concerned with uncovering the most important information as soon as possible.

• Heuristics. We must beware of overthinking the testing problem, so we use heuristics (loose translation: rules of thumb) to help us get out of analysis paralysis and test
  more quickly. Heuristics are essentially reflexes-- biases to respond in a certain way-- that generally help us test the right things at the right time. Rapid Testers collect, memorize and practice using helpful heuristics. In normal testing, heuristics are also used, but testers are often not aware of them and do not fully control them.

• Exploration. Rapid Testing is also rapid learning, so we use exploratory testing. We avoid pre-scripting test cases unless there is a clear and compelling need. We prefer
  to let the next test we do be influenced by the last test we did. This is a good thing, because we're not imprisoned by pre-conceived notions about what we should test, but let ourselves develop better ideas as we go. Other approaches to doing testing quickly, such as extensive test automation, run the risk that you'll end up with a lot of very fast tests that don't help you find the important problems in the product.

• Teaming. Rapid Testers cheat. That is, they do things that our former elementary school teachers would consider cheating: we figure things out in advance where possible, we borrow other people's work, we use every resource and tool we can find. For example,
  one important technique of Rapid Testing is pair testing: two heads, one computer. This idea has proven valuable in the practice of Extreme Programming, and it works for testing just as well. In our experience of normal testing, testers usually work quietly and alone, rather than hunting for bugs like a pack of rapid wolves.

Unit Testing

What is a unit?

Some standards—particularly ANSI/IEEE Std 1008-1987 (IEEE
Standard for Software Unit Testing)—use a lax definition of
software unit. According to IEEE-Std-1008 a software unit "...may
occur at any level of the design hierarchy from a single module to
a complete program".

Unit. The smallest compilable component. A unit
typically is the work of one programmer (At least in principle). As
defined, it does not include any called sub-components (for
procedural languages) or communicating components in general.

Unit Testing. In unit testing called components (or
communicating components) are replaced with stubs, simulators, or trusted components. Calling components are replaced with drivers or trusted super-components. The unit is tested in isolation.

For object-oriented programs this means that the unit is usually
a class. Some classes (like a simple Stack) might be
self-contained, but most call other classes and are in turn called
by yet other classes. In an attempt to reduce confusion when things
go wrong, you should try to test each class in isolation. If you
don't test them in isolation, you are implicitly trusting all
classes used by the class you are testing. You are effectivelly
saying: I think all the other classes already work and if they
don't, I'm prepared to sort out the mess myself. That's what
"trusted" means in the above definition. If you don't think the
other classes work, you should test in isolation. This is normally
more work as writing stubs and drivers is a pain.

When to test?

As the scope of unit testing narrows down from complete programs
to individual classes, so does the meaning of integration testing.
Any time you test two or more already unit-tested classes together
instead of using stubs, you are doing a litle bit of integration
testing.

For some systems integration testing is a big issue, because
they wait to finish coding before they start unit testing. This is
a big mistake, as delaying unit testing means you will be doing it
under schedule pressure, making it all-too-easy to drop the tests
and just finish the code. Developers should expect to spend between
25% and 50% percent of their time writing unit tests. If they leave
testing until they have finished, they can expect to spend the same
amount testing as they spend writing the module in the first place.
This is going to be extremely painful for them. The idea is to
spread the cost of unit testing over the whole implementation
phase. This is sometimes called "incremental glass-box testing"
(see Marc Rettig's article).

If you wait until you've finished coding before you start unit
testing, you'll have to choose an integration strategy. Are you
going to start with low level classes first and work your way up
until you reach the classes that expose some functionality through
an public API or start from the top and write stubs for lower level
classes or will you just test them all in one go?

Code Coverage

The greatest doubt I had when writing the standard, was not only
how much coverage to mandate but also whether to mandate any
coverage at all. It is easy enough to come up with a figure: 85%
seems to be pretty standard. But I have to agree with Brian Marick
[BM] that there is no evidence supporting this number. In my
opinion 100% is reasonable, as anything less means you haven't
tested that particular statements at all. Of course it is difficult
to have automatic unit tests for certain parts of the code.
Hardware interaction and UI code are typical examples. Or panics.
If you have acceptance tests that include tests for these parts of
the code or review them thoroughly, and if you make a sincere
effort to minimise the size and complexity of these parts of the
code, you can normally get away with not unit testing them. But I'd
rather include any known exceptions to the coverage rule in the
standard itself instead of arbitrarily lowering the bar for all the
rest of the code.

Three pitfalls to consider if you mandate coverage:

1. Don't consider tests that don't increase coverage as redundant.
   This is a big mistake. They might not add coverage, but they might
   find bugs. Coverage isn't everything. Brian Marick expressed this
   nicely “Coverage tools don't give commands (make that
   evaluate true), they give clues (you made some mistakes somewhere
   around there)”. Use coverage metrics to improve your test
   design skills.



2. If you mandate 85% coverage, what's the chance of anybody
   actually achieving significantly more coverage than that? Nil?



3. If developers get fixated with achieving coverage, they might
   try to reach 100% from the start and keep it there. This might be
   difficult to achieve and impact the productivity. The standard
   addresses the first point by including guidelines on test case
   design as an appendix. The standard addresses the second point by
   mandating 100% code coverage. The risk here is that people will
   tend to shift code into those areas (external interactions, UI)
   where coverage isn't mandated. It is the responsibility of
   everybody reviewing test code to watch out for this. The standard
   addresses the third point by making coverage measures available
   from the start but only requiring compliance with them in mayor
   milestones of the project. Also designing black-box test cases
   improves the chances of the test cases remaining adequate after
   some implementation change.

Picking a bigger unit

Testing each individual class can be a pain. In OO systems some
classes are very closely related and testing each of them in
isolation might mean much more effort in the sense of writing
stubs, etc. Taking this to the logical conclusion means you test
only the public API of the executable. The main reason against this
is that it is extremely difficult to get good coverage using this
approach, even if you settle for a relatively easy goal like
statement coverage. Also testing individual classes means you
actually only need to worry about stubs for external systems for
those classes that actually interact with them. For all classes in
your component that only interact with other classes in the same
component, you might not need stubs at all.

Automatic and Self-documenting

This standard mandates much less unit test documentation than is
required by older standards such as ANSI/IEEE Std 829-1983. The
main reason for this is that these standards more or less imply
manual execution of tests. A lot of documentation is required in
order to make this execution repeatable by somebody other than the
developer. This is not needed as the standard proposes the creation
of self-documenting automatic unit tests. The unit tests created by
following this standard are self-documenting–because all use
the same command-line arguments–and don't require any manual
procedures. When manual procedures are wanted for more in-depth
testing, the standard also specifies a location for that
information (testharness -h).

Test Results

Another area where standards frequently require more
documentation is test results. Which test case failed? Are you sure
you followed the test procedure? What were you doing when the test
failed? All this is redundant. Tests are automatic and only give
two answers for each test case: OK or Not OK. Sometimes standards
require information like: Information Why it is redundant Inputs
Cannot be chosen. Hard-coded. Expected results For each test: Test
Name…OK Actual results For some test: Test Name…[OK
or Not OK] Anomalies Report as Defect. Date and Time Printed by the
unit test Procedure Step Only one step: run them. Environment No
environment required for compliant tests. Attempts to repeat Tests
are automated. One failure means report as defect. Testers Whoever
reports the defect. Observers Whoever is watching the tester (not
very fun with automatic tests…).

Summing it up: “[with other approaches] extensive test
documentation is developed for each class. In contrast, our
approach minimizes the need to document each test suite by
standarizing the structure of all test suites. In other words, the
test framework is documented once and reused for each test
suite.” [CB].

Beyond unit tests

Apart from unit tests other types of tests are needed. Our
products are mainly reusable components (APIs) in contrast with
other companies that produce finished applications. This will be
even more so as the emphasis of producing GUIs for our components
shifts over to techview. Because we're producing components,
developer testing is easier as we can create programs that use our
components in order to test them. Independent testing and
validation is made more difficult for this same reason, as some
types of testing would require the testing team to be composed of
C++ programmers. This being unlikely, the approach to
validation/acceptance testing usually is: 1. Developers provide a
reference UI (either the developer's of the component or techview),
so that the functionality can be tested interactively. 2.
Developers provide a “acceptance test application”.
This application could be driven by a simple text file, allowing
testers to add their own test cases based on their interpretation
of the requirements. These approaches are still valid for testing.
The unit testing approach proposed in this standard affects them:
1. By lowering defect rates in acceptance testing. 2. By providing
build verification tests. 3. By making sure testers aren't the
first ones to run the unit tests. 4. By providing clear guidance on
which unit test failures should be raised as defect: any failure
means a defect. A reasonable cross-team integration approach might
still be necessary and is beyond the scope of this standard as it
heavily depends on project specific issues. The only recommendation
I would make in that respect is to set-up a cross-team automatic
build (with BVT as outlined in the standard).

Different Types of Testing

Functionality Testing : The functionality of the application ( i.e GUI components ). against the specifications ( eg, if we click " submit" button, the application should display ..... " confirmation dialog box")

Acceptance testing: Formal testing conducted to determine whether a system satisfies its acceptance criteria and thus whether the customer should accept the system.

Regression Testing: Testing the application for checking whether the previous features are working properly or not, after adding new features to the application.

Stress Testing: The idea of stress testing is to find the breaking point in order to find bugs that will make that break potentially harmful.

Load Testing: This is merely testing at the highest transaction arrival rate in performance testing to see the resource contention, database locks etc...

Black-box Testing: Focuses on the program's functionality against the specification.

White-box Testing: Focuses on the paths of logic.

Unit Testing: The most 'micro' scale of testing; to test particular functions or code modules. Typically done by the programmer and not by testers, as it requires detailed knowledge of the internal program design and code. Not always easily done unless the application has a well-designed architecture with tight code; may require developing test driver modules or test harnesses.

Integration Testing - testing of combined parts of an application to determine if they function together correctly. The 'parts' can be code modules, individual applications, client and server applications on a network, etc. This type of testing is especially relevant to client/server and distributed systems.

Incremental Integration Testing: Continuous testing of an application as new functionality is added; requires that various aspects of an application's functionality be independent enough to work separately before all parts of the program are completed, or that test drivers be developed as needed; done by programmers or by testers.

Functional Testing: Black-box type testing geared to functional requirements of an application; this type of testing should be done by testers.

System Testing: Black-box type testing that is based on overall requirements specifications; covers all combined parts of a system.

Sanity Testing: Typically an initial testing effort to determine if a new software version is performing well enough to accept it for a major testing effort. For example, if the new software is crashing systems every 5 minutes, bogging down systems to a crawl, or destroying databases, the software may not be in a 'sane' enough condition to warrant further testing in its current state.

Performance Testing: This term is often used interchangeably with 'stress' and 'load' testing. Ideally 'performance' testing (and any other 'type' of testing) is defined in requirements documentation or QA or Test Plans.

Usability Testing: Testing for 'user-friendliness'. Clearly this is subjective, and will depend on the targeted end-user or customer. User interviews, surveys, video recording of user sessions, and other techniques can be used. Programmers and testers are usually not appropriate as usability testers.

Installation/Uninstallation Testing: Testing of full, partial, or upgrade install/uninstall processes.

Security Testing: Testing how well the system protects against unauthorized internal or external access, willful damage, etc; may require sophisticated testing techniques.

Compatability Testing: Testing how well software performs in a particular hardware/software/operating system/network/etc. environment.

Ad-hoc Testing: Testing the application in a random manner.

Alpha Testing: Testing of an application when development is nearing completion; minor design changes may still be made as a result of such testing. Typically done by end-users or others, not by programmers or testers.

Beta Testing: Testing when development and testing are essentially completed and final bugs and problems need to be found before final release. Typically done by end-users or others, not by programmers or testers.

All Posts

Funny
Testing Flash Movies
Best Practices
SDLC
Unit Testing with Mock Objects
Check List for Test Preparation
Task Based Software Testing
Test Efficiency vs Test Effectiveness
Function vs Non-Functional testing
Test Case
Load Testing
What is a Test Plan?
Best Practices For J2EE Testing
BugZilla
Important Considerations for Test Automation
Educational Seminars On Software Testing
History's Worst Software Bugs
Fundamentals of Software Testing
Thread Based Integration Testing
Testing for Zero bugs
White Box Testing
What is User Acceptance Testing?
Black Box Testing
Testing Without a Formal Test Plan
What is a Nighlty Test Case
Test Plan
Automation Testing versus Manual Testing
QA Check List
Developing a Test Specification
Defect Severity and Defet Priority
Regression Testing
Software Testing Terminology
Software Testing Principles
Unit Testing
Software Test Automation and the Product Life Cycle
Usability Testing
Black Box Testing
Unit, Component and Integration Testing
Testing Glossary
Tasks of a Test Engineer
Introduction to Test Case Writing
Fundamentals of Software Testing - In Brief
Software Errors
Few FAQ's about Testing
The Basics of Automated Testing - 2
The Basics of Automated Testing - 1
Testing Basics
How to Report Bugs Effectively
Free Black Box Testing Course Video
Test Driven Development
Testing is easy, what’s the big deal?
But Do You Know if It's Any Good
Testing is Development too!
Risk Driven Testing
Software Development Life Cycle
Software Testing Terminology
Rapid Testing
Unit Testing
Different Types of Testing