QA Check List

Following is the check list which can be adopted by the QA.

General
	Pages fit within the resolution(800x600)
+	Design works with liquid tables to fill the user's window size.
+	Separate print versions provided for long documents (liquid tables may negate this necessity). Accommodates A4 size paper.
+	Site doesn't use frames.
+	Complex tables are minimized.
+	Newer technologies are generally avoided for 1-2 years from release, or if used alternative traditional forms of content are easily available.
Home vs. Subsequent Pages & Sections
+	Home page logo is larger and more centrally placed than o­n other pages.
+	Home page includes navigation, summary of news/promotions, and a search feature.
+	Home page answers: Where am I; What does this site do; How do I find what I want?
+	Larger navigation space o­n home page, smaller o­n subsequent pages.
+	Logo is present and consistently placed o­n all subsequent pages (towards upper left hand corner).
+	"Home" link is present o­n all subsequent pages (but not home page).
+	If subsites are present, each has a home page, and includes a link back to the global home page.
Navigation
+	Navigation supports user scenarios gathered in the User Task Assessment phase (prior to design).
+	Users can see all levels of navigation leading to any page.
+	Breadcrumb navigation is present (for larger and some smaller sites).
+	Site uses DHTML pop-up to show alternative destinations for that navigation level.
+	Navigation can be easily learned.
+	Navigation is consistently placed and changes in response to rollover or selection.
+	Navigation is available when needed (especially when the user is finished doing something).
+	Supplimental navigation is offered appropriately (links o­n each page, a site map/index, a search engine).
+	Navigation uses visual hierarchies like movement, color, position, size, etc., to differentiate it from other page elements.
+	Navigation uses precise, descriptive labels in the user's language. Icon navigation is accompanied by text descriptors.
+	Navigation answers: Where am I (relative to site structure); Where have I been (obvious visited links); Where can I go (embedded, structural, and associative links)?
+	Redundant navigation is avoided.
Functional Items
+	Terms like "previous/back" and "next" are replaced by more descriptive labels indicating the information to be found.
+	Pull-down menus include a go button.
+	Logins are brief.
+	Forms are short and o­n o­ne page (or demonstrate step X of Y, and why collecting a larger amount of data is important and how the user will benefit).
+	Documentation pages are searchable and have an abundance of examples. Instructions are task-oriented and step-by-step. A short conceptual model of the system is available, including a diagram that explains how the different parts work together. Terms or difficult concepts are linked to a glossary.
Linking
General:
+	Links are underlined.
+	Size of large pages and multi-media files is indicated next to the link, with estimated dowload times.
+	Important links are above the fold.
+	Links to releated information appear at bottom of content or above/near the top.
+	Linked titles make sense out of context.
+	If site requires registration or subscription, provides special URLs for free linking. Indicates the pages are freely linkable, and includes and easy method to discover the URL.
+	If site is running an ad, it links to a page with the relevant content, not the corporate home page.
In content:
+	Keeps linked phrases short to aid scanning (2-4 words).
+	Links o­n meaningful words and phrases. Avoids phrases like, "click here."
+	Includs a brief description of what the user should expect o­n the linked page.
In code:
+	Uses relative links when linking between pages in a site. Uses absolute links to pages o­n unrelated sites.
+	Uses link titles in the code for IE users (preferably less than 60 characters, no more than 80).
Search Capabilities
+	A search feature appears o­n every page (exceptions include pop-up forms and the like).
+	Search box is wide to allow for visible search parameters.
+	Advanced Search, if included, is named just that (to scare off novices).
+	Search system performs a spelling check and offers synonym expansion.
+	Site avoids scoped searching. If included it indicates scope at top of both query and results pages, and additionally offers an automatic extended site search immediately with the same parameters.
+	Results do not include a visible scoring system.
+	Eliminates duplicate occurances of the same results (e.g., foo.com/bar vs. foo.com/bar/ vs. foo.com/bar/index.html).
Page Design
+	Content accounts for 50% to 80% of a page's design (what's left over after logos, navigation, non-content imagery, ads, white space, footers, etc.).
+	Page elements are consistent, and important information is above the fold.
+	Pages load in 10 seconds or less o­n users bandwidth.
+	Pages degrade adequately o­n older browsers.
+	Text is over plain background, and there is high contrast between the two.
+	Link styles are minimal (generally o­ne each of link, visited, hover, and active states). Additional link styles are used o­nly if necessary.
+	Specified the layout of any liquid areas (usually content) in terms of percentages.
Fonts and Graphics
+	Graphics are properly optimized.
+	Text in graphics is generally avoided.
+	Preferred fonts are used: Verdana, Arial, Geneva, sans-serif.
+	Fonts, when enlarged, don't destroy layout.
+	Images are reused rather than rotated.
+	Page still works with graphics turned off.
+	Graphics included are necessary to support the message.
+	Fonts are large enough and scalable.
+	Browser chrome is removed from screen shots.
+	Animation and 3D graphics are generally avoided.
Content Design
+	Uses bullets, lists, very short paragraphs, etc. to make content scannable.
+	Articles are structured with scannable nested headings.
+	Content is formatted in chunks targeted to user interest, not just broken into multiple pages.
+	No moving text; most is left-justified; sans-serif for small text; no upper-case sentences/paragraphs; italics and bold are used sparingly.
+	Dates follow the international format (year-month-day) or are written out (August 30, 2001).
Writing
+	Writing is brief, concise, and well edited.
+	Information has persistent value.
+	Avoids vanity pages.
+	Starts each page with the conclusion, and o­nly gradually added the detail supporting that conclusion.
+	One idea per paragraph.
+	Uses simple sentence structures and words.
+	Gives users just the facts. Uses humor with caution.
+	Uses objective language.
Folder Structure
+	Folder names are all lower-case and follow the alpha-numeric rules found under "Naming Conventions" below.
+	Segmented the site sections according to: Root directory (the "images" folder usually goes at the top level within the root folder) Sub-directories (usually o­ne for each area of the site, plus an images folder at the top level within the root directory) Images are restricted to o­ne folder ("images") at the top level within the root directory (for global images) and then if a great number of images are going to be used o­nly section-specifically, those are stored in local "images" folders
Naming Conventions
+	Uses clients preferred naming method. If possible, uses longer descriptive names (like "content_design.htm" vs. "contdesi.htm").
+	Uses alphanumeric characters (a-z, 0-9) and - (dash) or _ (underscore)
+	Doesn't use spaces in file names.
+	Avoids characters which require a shift key to create, or any punctuation other than a period.
+	Uses o­nly lower-case letters.
+	Ends filenames in .htm (not .html).
Multimedia
+	Any files taking longer than 10 seconds to download include a size warning (>50kb o­n a 56kbps modem, >200kb o­n fast connections). Also includes the running time of video clips or animations, and indicate any non-standard formats.
+	Includes a short summary (and a still clip) of the linked object.
+	If appropriate to the content, includes links to helper applications, like Adobe Acrobat Reader if the file is a .pdf.
Page Titles
+	Follows title strategy ... Page Content Descriptor : Site Name, Site section (E.g.: Content Implementation Guidelines : CDG Solutions, Usability Process )
+	Tries to use o­nly two to six words, and makes their meaning clear when taken out of context.
+	The first word(s) are important information-carrying o­ne(s).
+	Avoids making several page titles start with the same word.
Headlines
+	Describes the article in terms that relate to the user.
+	Uses plain language.
+	Avoids enticing teasers that don't describe.
CSS
+	Uses CSS to format content appearance (as supported by browsers), rather than older HTML methods.
+	Uses a browser detect and serve the visitor a CSS file that is appropriate for their browser/platform combination.
+	Uses linked style sheets.
Documentation and Help Pages
+	When using screen shots, browser chrome was cropped out.
+	Hired a professional to write help sections (a technical writer).
+	Documentation pages are searchable.
+	Documentation section has an abundance of examples.
+	Instructions are task-oriented and step-by-step.
+	A short conceptual model of the system is provided, including a diagram that explains how the different parts work together.
+	Terms or difficult concepts are linked to a glossary.
Content Management
+	Site has procedures in place to remove outdated information immediately (such as calendar events which have passed).

Developing a Test Specification

Definitions

I’ve seen the terms “Test Plan” and “Test Specification” mean slightly different things over the years. In a formal sense (at this given point in time for me), we can define the terms as follows:

1. Test Specification – a detailed summary of what scenarios will be tested, how they will be tested, how often they will be tested, and so on and so forth, for a given feature. Examples of a given feature include, “Intellisense, Code Snippets, Tool Window Docking, IDE Navigator.” Trying to include all Editor Features or all Window Management Features into one Test Specification would make it too large to effectively read.
2. Test Plan – a collection of all test specifications for a given area. The Test Plan contains a high-level overview of what is tested (and what is tested by others) for the given feature area. For example, I might want to see how Tool Window Docking is being tested. I can glance at the Window Management Test Plan for an overview of how Tool Window Docking is tested, and if I want more info, I can view that particular test specification.

If you ask a tester on another team what’s the difference between the two, you might receive different answers. In addition, I use the terms interchangeably all the time at work, so if you see me using the term “Test Plan”, think “Test Specification.”


Parts of a Test Specification

A Test Specification should consist of the following parts:

• History / Revision - Who created the test spec? Who were the developers and Program Managers (Usability Engineers, Documentation Writers, etc) at the time when the test spec was created? When was it created? When was the last time it was updated? What were the major changes at the time of the last update?
  
• Feature Description – a brief description of what area is being tested.
  
• What is tested? – a quick overview of what scenarios are tested, so people looking through this specification know that they are at the correct place.
  
• What is not tested? - are there any areas being covered by different people or different test specs? If so, include a pointer to these test specs.
  
• Nightly Test Cases – a list of the test cases and high-level description of what is tested each night (or whenever a new build becomes available). This bullet merits its own blog entry. I’ll link to it here once it is written.
  
• Breakout of Major Test Areas - This section is the most interesting part of the test spec where testers arrange test cases according to what they are testing. Note: in no way do I claim this to be a complete list of all possible Major Test Areas. These areas are examples to get you going.
  
• Specific Functionality Tests – Tests to verify the feature is working according to the design specification. This area also includes verifying error conditions.
  
• Security tests – any tests that are related to security. An excellent source for populating this area comes from the Writing Secure Code book.
  
• Accessibility Tests – This section shouldn’t be a surprised to any of my blog readers. <grins> See The Fundamentals of Accessibility for more info.
  
• Stress Tests – This section talks about what tests you would apply to stress the feature.
  
• Performance Tests - this section includes verifying any perf requirements for your feature.
  
• Edge cases – This is something I do specifically for my feature areas. I like walking through books like How to break software, looking for ideas to better test my features. I jot those ideas down under this section
  
• Localization / Globalization - tests to ensure you’re meeting your product’s International requirements.

Setting Test Case Priority

A Test Specification may have a couple of hundred test cases, depending on how the test cases were defined, how large the feature area is, and so forth. It is important to be able to query for the most important test cases (nightly), the next most important test cases (weekly), the next most important test cases (full test pass), and so forth. A sample prioritization for test cases may look like:

1. Highest priority (Nightly) – Must run whenever a new build is available


2. Second highest priority (Weekly) – Other major functionality tests run once every three or four builds


3. Lower priority – Run once every major coding milestone

Author - saraford

http://blogs.msdn.com/saraford/archive/2004/05/05/249135.aspx

Defect Severity and Defet Priority

This document defines the defect Severity scale for determining defect criticality and the associated defect Priority levels to be assigned to errors found in software. It is a scale which can be easily adapted to other automated test management tools.

ANSI/IEEE Std 729-1983 Glossary of Software Engineering Terminology defines Criticality as,
"A classification of a software error or fault based on an evaluation of the degree of impact that error or fault on the development or operation of a system (often used to determine whether or when a fault will be corrected)."

The severity framework for assigning defect criticality that has proven most useful in actual testing practice is a five level scale. The criticality associated with each level is based on the answers to several questions.

First, it must be determined if the defect resulted in a system failure. ANSI/IEEE Std 729-1983 defines a failure as,
"The termination of the ability of a functional unit to perform its required function."

Second, the probability of failure recovery must be determined. ANSI/IEEE 729-1983 defines failure recovery as,
"The return of a system to a reliable operating state after failure."

Third, it must be determined if the system can do this on its own or if remedial measures must be implemented in order to return the system to reliable operation.

Fourth, it must be determined if the system can operate reliably with the defect present if it is not manifested as a failure.

Fifth, it must be determined if the defect should or should not be repaired.
The following five level scale of defect criticality addresses the these questions.


The five Levels are:

1. Critical

2. Major

3. Average

4. Minor

5. Exception

1. Critical - The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system.

2. Major - The defect results in the failure of the complete software system, of a subsystem, or of a software unit (program or module) within the system. There is no way to make the failed component(s), however, there are acceptable processing alternatives which will yield the desired result.

3. Average - The defect does not result in a failure, but causes the system to produce incorrect, incomplete, or inconsistent results, or the defect impairs the systems usability.

4. Minor - The defect does not cause a failure, does not impair usability, and the desired processing results are easily obtained by working around the defect.

5. Exception - The defect is the result of non-conformance to a standard, is related to the aesthetics of the system, or is a request for an enhancement. Defects at this level may be deferred or even ignored.
In addition to the defect severity level defined above, defect priority level can be used with severity categories to determine the immediacy of repair.


A five repair priority scale has also be used in common testing practice. The levels are:

1. Resolve Immediately

2. Give High Attention

3. Normal Queue

4. Low Priority

5. Defer


1. Resolve Immediately - Further development and/or testing cannot occur until the defect has been repaired. The system cannot be used until the repair has been effected.

2. Give High Attention - The defect must be resolved as soon as possible because it is impairing development/and or testing activities. System use will be severely affected until the defect is fixed.

3. Normal Queue - The defect should be resolved in the normal course of development activities. It can wait until a new build or version is created.

4. Low Priority - The defect is an irritant which should be repaired but which can be repaired after more serious defect have been fixed.

5. Defer - The defect repair can be put of indefinitely. It can be resolved in a future major system revision or not resolved at all.

Regression Testing

Definition - Regression testing is the re-running of Test cases that a program has previously executed correctly, in order to detect failures spawned by changes or corrections made during software development and maintenance.

These failures arise from incomplete or incorrect changes and are often witnessed as (unexpected) side effects in apparently unrelated application areas. It is common in the IT industry, that one in six (or seventeen percent*) of correction attempts are themselves defective.

This high rate of introduced defects is exaggerated when developers maintain a large number of poorly documented, integrated systems where they of ten have little or no experience of these systems. Regression testing may then be used to great effect at detecting subtle side effects and unconsidered inter-relationships within these environments, thus reducing risk.

In regression testing standard actions in a test procedure are carried out and the expected responses are checked for correctness. Failure of the system to reproduce any of the expected responses imply that the system may have regressed (there may have been one or more introduced defects), or that the regression test itself may be out of date or incorrect.

If all responses are as expected, there may still have been introduced defects. In this case they escaped detection. Each defect that is reported from live or field use, having escaped detection during regression testing must be carefully analysed and the regression test suite(s) updated to catch these or similar defects in the future.
The main source of regression test cases is usually from re-use of unit, integration or system test cases. It is good practice to batch test cases into logically cohesive test suites, rather than have a single huge regression test. This allows different sub-sets of tests to be executed when there is time-pressure, or where there is confidence that only certain tests need to be run.

When first creating a regression testing suite, the choice of tests to use can be guided by the 80/20 principle. Twenty percent of system functions are likely to be used eighty percent of the time. Thus these highly used screens, transactions, menus, or fields ought to be the first candidates for regression tests. This is easy to understand if we consider one of these popular functions experiencing failure. The company call centre will be inundated with calls and the majority of users will be negatively affected. If, however, one of the less common functions has a problem, fewer users would have used it and thus discovered a lack.

Further tests to add to regression suites may be guided by risk considerations. Certain failures may not occur often, but should they occur, would result in a highly negative business impact. These higher risk business areas, modules, screens, or transactions should therefore be tested each and every time there are changes in the system and/or its environment.

Additional regression tests can be added to application areas that are known to be difficult to maintain, and have a history of high failure rates.

Regression testing can begin at the unit level, where unit tests may be adapted and rerun after changes to the unit have been effected. Regression testing should then continue through integration, system, user acceptance and operational software development life cycle phases.

As a minimum, regression tests should be run prior to build releases into the broader community and/or company live Environment. These tests will help detect major anomalies that could have serious cost, schedule, productivity and/or company image Implications.

Web systems, and other multi-user systems might have ongoing regression tests run at regular intervals. For example, one such test may check that all hyperlinks on a web site remain correct and reachable. Links to other sites may become outdated, or may even be corrupted by hackers in a security breach.

Regression testing at regular intervals can also answer production questions such as: "Is the performance of our major transactions within acceptable time limits?", "or", is some factor slowing our response times on an important transaction?"

Regression tests of non-functional application attributes such as performance, usability or security, is also very important. A very small change to the code or design may have a significant impact on system performance for example. Also please take note that debilitating changes may not even be within the application software. Changes known to have had dire consequences include an update of PC BIOS, operating system software, network cards, or updates of third party database versions for example.

Regression testing is by definition repetitive, and thus many of the tests are likely to be suited to test automation. Test automation can deliver reduced testing costs after a few test iterations when compared to labour-intensive manual testing processes.

Many companies who use regression testing conduct a very abbreviated check test (sometimes called a 'smoke' or 'sanity' test) on newly delivered code, prior to starting their formal regression tests. This often saves time as the abbreviated test commonly exposes obvious errors (for example: a whole form may not be displayed because it failed to compile against a changed database format). Removing this type of problem prior to running the time-consuming regression testing scenarios can result in getting earlier developer help, and prevent testers completing a significant portion of the regression testing before finding such problems.

It should be noted that the control and make-up of the test environment is as critical for regression testing as it is for other testing types within the same software development life cycle phase. Refer to the article on page eight that discusses creating a testing environment.

Regression test suites, be they manual or automated, are an important company asset. They therefore need to be backed up, configuration-managed, and kept current to deliver maximum benefit to their owners. Specific ownership of and responsibility for the regression test suites therefore needs to be clearly defined.


Wayne Mallinson
http://www.testfocus.co.za/Feature

Software Testing Terminology

A few simple definitions of Testing Terminology.

Error - The difference between a computed, observed, or measured value or condition and the true, specified, or theoretically correct value or condition.

Fault - An incorrect step, process, or data definition in a computer program.

Debug - To detect, locate, and correct faults in a computer program.

Failure - The inability of a system or component to perform its required functions within specified performance requirements. It is manifested as a fault.

Testing - The process of analyzing a software item to detect the differences between existing and required conditions (that is, bugs) and to evaluate the features of the software items.

Static analysis - The process of evaluating a system or component based on its form, structure, content, or documentation.

Dynamic analysis - The process of evaluating a system or component based on its behavior during execution.

Correctness - The degree to which a system or component is free from faults in its specification, design, and implementation. The degree to which software, documentation, or other items meet specified requirements. The degree to which software, documentation, or other items meet user needs and expectations, whether specified or not.

Verification - The process of evaluating a system or component to determine whether the products of a given development phase satisfy the conditions imposed at the start of that phase. Formal proof of program correctness.

Validation - The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements.

Software Testing Principles

Software testing is an extremely creative and intellectually challenging task. When testing follows the principles given below, the creative element of test design and execution rivals any of the preceding software development steps.

Testing must be done by an independent party.

Testing should not be performed by the person or team that developed the software since they tend to defend the correctness of the program. The developer is driven by delivery and he will try to finish the testing as early as possible. The developer is gentle in testing his code and has a soft corner for his code. The independent tester is driven by quality and will try to break the code. Hence testing should be done by an independent person.

Assign best personnel to the task.

Because testing requires high creativity and responsibility only the best personnel must be assigned to design, implement, and analyze test cases, test data and test results. Also note that from point 1, the testing is being done by an independent tester, the system is new to him and he has to understand the entire system to test it. It is possible to find bugs and issues with with software only when you know the system thoroughly.

Testing should not be planned under the tacit assumption that no errors will be found.

Testing is the process of executing a program with the intent of finding errors. Good testing will find bugs… and testing is never complete, it is infinte. There should be no assumption that the software has no erros. Such an assumption would leave tremendous amount of holes in the system.

Test for invalid and unexpected input conditions as well as valid conditions.

The program should generate correct results when a valid input is given, this is called as positive testing; Also the software should give correct error messages when an invalid test is encountered, this is called as negative testing. We need to give a different range of inputs with varying sets of values. For example, if the input field is taking a positive integer, we should try with all sorts of integers, positive, negative, zero, large positive, large negative etc. Also we should try giving a character, string etc and the expect the correct error message.

The probability of the existence of more errors in a module or group of modules is directly proportional to the number of errors already found.

If you find a bug in a particular module, the developers just tend to resolve that issue and close the bug. But there is a more probability that there will be more issues in that area. You need to look around, why that particular bug is present and look for similar issues in that module, you are going to hit many issues.

Keep software static during test.

The program must not be modified during the implementation of the set of designed test cases. Modifying the program while the testing has started will leave a lot of loop holes in the testing; you would have finished testing module A and started testing on module B, but by that time, module A would have changed and the changes are not tested!!!

Document test cases and test results.

This is very important for testers, you need to document the test cases, which can be used for later testing, may be for the next release. The test results should be documented so that they can used for analyzing if required at a later stage.

Provide expected test results if possible.

A necessary part of test documentation is the specification of expected results, even if providing such results is impractical. Using the expected results you can verify the software which is under test and declare whether the software meets the expectations or not.

Unit Testing

Unit Testing

Definition - Unit Test can be defined as running one component of a system for testing purposes.

In computer programming, a unit test is a method of testing the correctness of a particular module of source code.

The idea is to write test cases for every non-trivial function or method in the module so that each test case is separate from the others if possible.


What is Unit Testing?

When we write programs we can not make them perfect from the first time. When you compile your project you face the first test for your application. The compiler checks the syntax and stops if it contains errors. This is a nice assistance but it does not end here. The program may contain bugs, and it CERTANLY contains bugs. Then we have to hunt them down and correct the application so the bug count is low enough. We can not correct all the bugs but we can reduce them to a decent level.So here come the Unit Tests. They help us hunt the bugs by writing tests. So the tests are programs, methods actually. They test every part of our program to see if it works right.


Benefits of Unit Testing

The goal of unit testing is to isolate each part of the program and show that the individual parts are correct. It provides a written contract that the piece must satisfy. This isolated testing provides two main benefits:
1. Encourages change
Unit testing allows the programmer to refactor code at a later date, and make sure the module still works correctly (regression testing). This provides the benefit of encouraging programmers to make changes to the code since it is easy for the programmer to check if the piece is still working properly.
2. Simplifies Integration
Unit testing helps eliminate uncertainty in the pieces themselves and can be used in a bottom-up testing style approach. By testing the parts of a program first and then testing the sum of its parts will make integration testing easier.


Limitations of Unit Testing

It is important to realize that unit-testing will not catch every error in the program. By definition, it only tests the functionality of the units themselves. Therefore, it will not catch integration errors, performance problems and any other system-wide issues. In addition, it may not be trivial to anticipate all special cases of input the program unit under study may receive in reality. Unit testing is only effective if it is used in conjunction with other software testing activities.

Software Test Automation and the Product Life Cycle

The Product Life Cycle(PLC) and Automated Test

A product's stages of development are referred to as the product life cycle (PLC). There is considerable work involved in getting a product through its PLC. Software testing at many companies has matured as lessons have been learned about the most effective test methodologies. Still, there is a great difference of opinion about the implementation and effectiveness of automated software testing and how it relates to the PLC.

Computers have taken over many functions in our society that were once "manual" operations. Factories use computers to control manufacturing equipment and have cut costs enormously. Electronics manufacturing use computers to test everything from microelectronics to circuit card assemblies. Since automation has been so successful in so many areas, does it make sense that a software program should be used to test another software program? This is referred to as "automated software testing" for the remainder of this article.

Software testing using an automatic test program will generally avoid the errors that humans make when they get tired after multiple repetitions. The test program won't skip any tests by mistake. The test program can also record the results of the test accurately. The results can be automatically fed into a database that may provide useful statistics on how well the software development process is going. On the other hand, software that is tested manually will be tested with a randomness that helps find bugs in more varied situations. Since a software program usually won't vary each time it is run, it may not find some bugs that manual testing will.
Automated software testing is never a complete substitute for manual testing.

There has been plenty of debate about the usefulness of automatic software testing. Some companies are quite satisfied with the developer testing his/her own work. Testing your own work is generally thought of as risky since you'll be likely to overlook bugs that someone not so close to the code (and not so emotionally attached to it) will see easily. As soon as the developer says it's done they ship it. The other extreme is the company that has its own automatic software test group as well as a group that tests the software manually. Just because we have computers does that mean that it is cost effective to write tests to test software and then spend time and resources to maintain them? The answer is both yes and no. When properly implemented, automated software test can save a lot of time, time that will be needed as the software approaches shipping.

This is where the PLC comes in. How effectively you make use of the PLC will often be dependent on your programming resources and the length of the PLC. Companies large and small struggle with software testing and the PLC. Hopefully, this discussion of the PLC should help you determine when to use automation and when manual testing is preferred. This should help you answer the questions: "Why should I automate my software testing?" "How can I tell if automation is right for my product?" "When is the best time to develop my test software?".


The Product Life Cycle

As we discuss the use of automated and manual testing, we need to understand what happens in each phase of the product life cycle. The PLC is made up of six major stages, the Design Phase, the Code Complete Phase, the Alpha Phase, the Beta Phase, the Zero Defect Build Phase, and the Green Master Phase. You can think of the PLC as a timeline showing the development of a software product. These are the major milestones that make up the Product Life Cycle. Products that follow these guidelines for implementation of the PLC will have a much better chance of making it to market on time.

The implementation of the PLC varies widely from company to company. You can use this as a guide for future reference to assist you in your automation efforts. Your implementation will vary from the ideal PLC that is discussed here, but your software's success may depend on how well you've implemented its PLC. If your PLC is to include automated testing you should pay attention to which automated tasks are performed during each phase.

For each phase we'll describe it, define its special importance and discuss how to incorporate software automation into your project. Most other discussions of the PLC don't include the lessons learned about test automation. This should be your "one-stop" guide to help you know how and when automation fits into the PLC.


Design Phase

What is the Design Phase? The design phase begins with an idea. Product managers, QA, and Development get together at this point to determine what will be included in the product. Planning is the essence of the design phase. Begin with the end in mind and with a functional specification. Write down all of your plans. What will your product do? What customer problems does it solve?

Incorrectly, some companies don't include Quality Assurance (QA) in the design phase. It is very important that QA be involved as early as possible. While developers are writing code, QA will be writing tests. Even though QA won't really have the total picture of the product, they will want to get as much of a jump on things as possible. Remember, that the primary purpose of QA is to report status. It is important to understand the product's status even early in the Design Phase.

Why is the Design Phase important? If you think you're too short on time to write up a functional description of your product, then consider the extra time involved to add new features later on. Adding features later (especially once the Code Complete Phase has been reached) is known as "feature creep". Feature creep can be a very costly haphazard way to develop your product, and may materially interfere with delivery of the software.

Automation activity during the Design Phase. As soon as the functional specification is written, create all test cases so that they can be run manually. Yes, that's right, manually! These manual tests are step-by-step "pseudo" code that would allow anyone to run the test. The benefits of this approach are:

1. Your test cases can be created BEFORE ever seeing the software's user interface (UI). It is too soon to automate tests at this point in time, but you can create manual tests with only a small risk of changes that will occur . This is a point of great frustration for those who have tried to implement automated test scripts too early. Just as soon as the test script is written changes in the UI are bound to be introduced and all the work on the script is found to be for nothing.
2. When (not if) the code is modified, you will always have manual procedures that can be adapted to the change more quickly than an automated test script. This is a great way to guarantee that you will at least have tests you can perform even if automation turns out to not be feasible. (Note: one of the objections to software test automation is that the tests must be continually updated to reflect changes in the software. These justifiable objections usually stem from the fact that automation was started too early.)
3. Test methods can be thought out much more completely because you don't have to be concerned with the programming language of the automation tool. The learning curve of most automation tools may get in the way of writing meaningful tests.

If you have the resources available, have them begin training on the test tools that will be used. Some members of the team should start writing library routines that can be used by all the test engineers when the start their test coding. Some of these routines will consist of data collection/result reporting tools and other common functions.

After the manual test cases have been created decide with your manager which test cases should be automated. Use the Automation Checklist found later in this article to assist you in deciding what tests to automate. If you have enough manpower you may want to have an test plan team and an automation team. The test plan team would develop tests manually and the automation team would decide which of the manual tests should be run automatically (following the guidelines of the Automation Checklist later in this article). The automation team would be responsible for assuring that the test can be successfully and cost effectively automated.

Sometime during the design phase, as soon as the design is firm enough, you'll select the automation tools that you will need. You don't have to decide exactly which tests need to be automated yet, but you should have an idea of the kinds of tests that will be performed and the necessary capabilities of the tools. That determination is easier as the software gets closer to the code complete phase. Your budget and available resources will begin to come into play here.

For just a moment, let's discuss some of the considerations you should use in selecting the test tools you need. You'll also want to keep in mind the Automation checklist later in this column. It will help you determine if a test should be automated. There are a few good testing tools including Apple Computer's Virtual User (VU) (See the September, 1996 article "Software Testing With Virtual User", by Jeremy Vineyard) and Segue's QA Partner (Segue is pronounced "Seg-way").

Is there a lot of user interface (UI) to test? Software with a lot of UI is well suited for automated black box testing. However, some important considerations are in order here. You need to get involved with development early to make sure that the UI can be "seen" by the automation tool. For example: I've seen programs in which a Virtual User 'match' task (note: a task is what a command is called in Virtual User) couldn't find the text in a text edit field. In those cases, this occurred because the program didn't use standard Macintosh calls, but rather was based on custom libraries that provided UI features their own way.

Will the automated test environment effect the performance or operation of the system being tested? When you're trying to test the latest system software, you don't want the testing system changing the conditions of the test.

Is the speed that the tests run a consideration? If you're trying to measure the performance of a system you'll want to make sure that the conditions are as much like the "real world" as possible. You should consider the amount of network traffic that is present while you're running your tests. Also, the speed of your host processor can effect the time it takes your tests to run. You should schedule your tests so that you minimize the possibility of interfering with someone else on your network. Either isolate your network from others or warn them that you will be testing and that there is a possibility that their network activity may slow down.

What kinds of tests will be performed? The lower the level the testing is, the more likely white box testing should be used. A good example of this would be if you have a function that does a calculation based on specific inputs. A quick C program that calls the function would be much faster and could be written to check all the possible limits of the function. A tool like VU would only be able to access the function through the UI and would not be able to approach the amount of coverage that a C program could do in this situation.

Is there a library of common functions available or will you have to write them yourself? It will save a lot of time if you don't have to develop libraries yourself. No matter how extensive the command set, efficient use of library functions will be essential. Libraries that others have written may be useful; you can modify them to meet your own needs.

What will be the learning curve for a script programmer? The time it takes will depend greatly on the kind of testing you have to do and the experience of the programmer. If you've done your homework on the available test tools, you should know what to expect. Some companies even offer training courses (for a price) in their test software.

Can your automation tool automatically record actions for you? Some tools do this, but don't expect to rely on this too heavily. Tools that I've seen that do this end up creating code that has hard coded strings and tend to be organized in a sequential manner rather than by calling procedures. These recorded scripts are harder to maintain and reuse later. If you plan to use the same script for international testing, modifying the script will mean much more work. If you want to record actions, I recommend that you do it only to create short functions and you should edit the script after recording to remove the unwanted hard coded strings, etc.

Can you run AppleScript scripts from the tool's script language? This is a very useful feature since AppleScript scripts are so easy to write and can add additional functionality to your test tool.

In preparing this article, I encountered several "pearls" worth relating:

"Success in test automation requires careful planning and design work, and it's not a universal solution. ... automated testing should not be considered a replacement for hand testing, but rather as an enhancement." (Software Testing with Visual Test 4.0, forward by James Bach, pg. vii)

"The quality assurance engineers then come on the scene... and begin designing their overall test plan for the features of the product...."

"The goal is to have the test plan and checklists laid out and ready to be manually stepped through by the test engineers when each feature is completed by the programmers. Each item on a checklist is considered a scenario and related scenarios are grouped into test cases." (Software Testing with Visual Test 4.0, pg. 5-6)

Code Complete Phase

What is the Code Complete Phase? At this major milestone the code has been completed. The code has been written, but not necessarily yet debugged. (Development may try to claim they are at code complete even though they may still have major coding still left to do. Go ahead and let them declare the code complete, but don't let them get to Alpha until the code really is completely written.)

Why is the Code Complete Phase important? Sooner or later you'll have to get to a point where new code is no longer being written, and the major effort is in fixing bugs. Development will be relieved to get to this point as now they don't have to be as concerned with the initial coding and can concentrate on refining the existing product. (This is why they will try to claim they are at code complete even when they are not).

Automation activity during the Code Complete Phase Although the UI may still change, QA can begin writing Automatic test cases. The tests that should be written at this point are breadth tests that tell the status of the overall software product. Don't write tests which stress the product until you get close to Alpha. The product will probably break very easily. Some acceptance (or "smoke") tests should also be created to give a quick evaluation of the status of a particular build. Before reaching the Alpha phase there should also be tests written to test the Installer, boundary (or stress tests), compatibility (hardware and OS), performance, and interoperability.

Somewhere just before code complete, you will need to decide which tests should be made into automatic tests and what test tools to use. Use the following checklist to help you determine which tests should be automated:

Automation Checklist

If you answer yes to any of these questions, then your test should be seriously considered for automation.

Can the test sequence of actions be defined?

Is it useful to repeat the sequence of actions many times? Examples of this would be Acceptance tests, Compatibility tests, Performance tests, and regression tests.

Is it necessary to repeat the sequence of actions many times?

Is it possible to automate the sequence of actions? This may determine that automation is not suitable for this sequence of actions.

Is it possible to "semi-automate" a test? Automating portions of a test can speed up test execution time.

Is the behavior of the software under test the same with automation as without? This is an important concern for performance testing.

Are you testing non-UI aspects of the program? Almost all non-UI functions can and should be automated tests.

Do you need to run the same tests on multiple hardware configurations? Run ad hoc tests (Note: Ideally every bug should have an associated test case. Ad hoc tests are best done manually. You should try to imagine yourself in real world situations and use your software as your customer would. As bugs are found during ad hoc testing, new test cases should be created so that they can be reproduced easily and so that regression tests can be performed when you get to the Zero Bug Build phase.) An ad hoc test is a test that is performed manually where the tester attempts to simulate real world use of the software product. It is when running ad hoc testing that the most bugs will be found. It should be stressed that automation cannot ever be a substitute for manual testing.

Alpha Phase

What is the Alpha Phase? Alpha marks the point in time when Development and QA consider the product stable and completed. The Alpha Phase is your last chance to find and fix any remaining problems in the software. The software will go from basically functional to a finely tuned product during this phase.

Why is the Alpha Phase important? Alpha marks a great accomplishment in the development cycle. The code is stable and the most major bugs have been found and fixed.

Automation Activity During The Alpha Phase

At this point you have done the tasks that need to be done in order to reach Alpha. That is, you have all your compatibility, interoperability, and performance tests completed and automated as far as possible. During Alpha you'll be running breadth tests every build. Also you'll run the compatibility, interoperability, and performance tests at least once before reaching the next milestone (beta). After the breadth tests are run each build, you'll want to do ad hoc testing as much as possible. As above, every bug should be associated with a test case to reproduce the problem.

Beta Phase

What is the Beta Phase? The product is considered "mostly" bug free at this point. This means that all major bugs have been found. There should only be a few non essential bugs left to fix. These should be bugs that the user will find annoying or bugs that pose relatively no risk to fix. If any major bugs are found at this point, there will almost definitely be a slip in the shipping schedule.

Automation activity during the Beta Phase

There's no more time left to develop new tests. You'll run all of your acceptance tests as quickly as possible and spend the remaining time on ad hoc testing. You'll also run compatibility, performance, interoperability and installer tests once during the beta phase.

Remember that as you do ad hoc testing every bug should have an associated test case. As bugs are found during ad hoc testing, new test cases should be created so that they can be reproduced easily and so that regression tests can be performed when we get to the Zero Bug Build phase.

Zero Defect Build Phase

What is the Zero Defect Build Phase? This is a period of stability where no new serious defects are discovered. The product is very stable now and nearly ready to ship.

Automation Activity During The Zero Defect Build Phase

Run regression tests. Regression testing means running through your fixed defects again and verify that they are still fixed. Planning for regression testing early will save a lot of time during this phase and the Green Master phase.

Green Master

What is the Green Master Phase? Green Master is sometimes referred to as the Golden Master or the final candidate. The product goes through a final checkout before it is shipped (sent to manufacturing).

Automation activity during the Green Master Phase

After running general acceptance tests, run regression tests. You should run through your fixed defects once again to verify that they are still fixed. Planning for regression testing early will save a lot of time during this phase.

===================================================

by Dave Kelly, Symantec Corporation

http://www.mactech.com/articles/mactech/Vol.13/13.10/SoftwareTestAutomation/