Black Box Testing

Black box testing or functional testing is used to check that the outputs of a program, given certain inputs, conform to the functional specification of the program.

The term black box indicates that the internal implementation of the program being executed is not examined by the tester. For this reason black box testing is not normally carried out by the programmer.

Black Box Testing is testing without knowledge of the internal workings of the item being tested. For example, when black box testing is applied to software engineering, the tester would only know the "legal" inputs and what the expected outputs should be, but not how the program actually arrives at those outputs. It is because of this that black box testing can be considered testing with respect to the specifications, no other knowledge of the program is necessary.
For this reason, the tester and the programmer can be independent of one another, avoiding programmer bias toward his own work. For this testing, test groups are often used, "Test groups are sometimes called professional idiots...people who are good at designing incorrect data."

The opposite of this would be glass box testing, where test data are derived from direct examination of the code to be tested. For glass box testing, the test cases cannot be determined until the code has actually been written. Both of these testing techniques have advantages and disadvantages, but when combined, they help to ensure thorough testing of the product.
Also, do to the nature of black box testing, the test planning can begin as soon as the specifications are written.


Advantages of Black Box Testing
There are many advantages of Black Box Testing...

1. More effective on larger units of code than glass box testing


2. The tester needs no knowledge of implementation, including specific programming languages


3. The tester and programmer are independent of each other


4. Tests are done from a user's point of view


5. Will help to expose any ambiguities or inconsistencies in the specifications


6. Test cases can be designed as soon as the specifications are complete


7. The test is unbiased because the designer and the tester are independent of each other.


8. The tester does not need knowledge of any specific programming languages.


9. The test is done from the point of view of the user, not the designer.


10. Test cases can be designed as soon as the specifications are complete.

Disadvantages of Black Box Testing

1. Only a small number of possible inputs can actually be tested, to test every possible input stream would take nearly forever


2. Without clear and concise specifications, test cases are hard to design


3. There may be unnecessary repetition of test inputs if the tester is not informed of test cases the programmer has already tried


4. May leave many program paths untested


5. Cannot be directed toward specific segments of code which may be very complex (and therefore more error prone)


6. Most testing related research has been directed toward glass box testing


7. The test can be redundant if the software designer has already run a test case.


8. The test cases are difficult to design.


9. Testing every possible input stream is unrealistic because it would take a inordinate amount of time; therefore, many program paths will go untested.

Testing Strategies/Techniques

1. Black box testing should make use of randomly generated inputs (only a test range should be specified by the tester), to eliminate any guess work by the tester as to the methods of the function


2. Data outside of the specified input range should be tested to check the robustness of the program


3. Boundary cases should be tested (top and bottom of specified range) to make sure the highest and lowest allowable inputs produce proper output


4. The number zero should be tested when numerical data is to be input


5. Stress testing should be performed (try to overload the program with inputs to see where it reaches its maximum capacity), especially with real time systems


6. Crash testing should be performed to see what it takes to bring the system down


7. Test monitoring tools should be used whenever possible to track which tests have already been performed and the outputs of these tests to avoid repetition and to aid in the software maintenance


8. Other functional testing techniques include: transaction testing, syntax testing, domain testing, logic testing, and state testing.


9. Finite state machine models can be used as a guide to design functional tests


10. According to Beizer the following is a general order by which tests should be designed:


11. Clean tests against requirements.


12. Additional structural tests for branch coverage, as needed.


13. Additional tests for data-flow coverage as needed.


14. Domain tests not covered by the above.


15. Special techniques as appropriate--syntax, loop, state, etc.


16. Any dirty tests not covered by the above.

Glass box testing requires the intimate knowledge of program internals, while black box testing is based solely on the knowledge of the system requirements. Being primarily concerned with program internals, it is obvious in SE literature that the primary effort to develop a testing methodology has been devoted to glass box tests. However, since the importance of black box testing has gained general acknowledgement, also a certain number of useful black box testing techniques were developed.
It is important to understand that these methods are used during the test design phase, and their influence is hard to see in the tests once they're implemented.

Advantages

1. Forces test developer to reason carefully about implementation


2. Approximates the partitioning done by execution equivalence


3. Reveals errors in "hidden" code:


4. Beneficent side-effects


5. Optimizations (e.g. charTable that changes reps when size > 100)

Disadvantages

1. Expensive


2. Miss cases omitted in the code