What is a Nighlty Test Case

Why is a nightly called a nightly?

A nightly is a test case that must be run every time there is a new build. Generally, a new build is published every night, and these tests are run nightly, hence the name, nightly tests. Other names are Acceptance or Self-Host, but the concept remains the same.


What should a nightly cover?

A good nightly test case verifies the following:

1. Is the feature usable?


2. Can the user perform basic end-to-end scenarios?


3. Most importantly, if a certain feature/scenario is broken, does the dev need to drop everything and immediately fix the issue?

Think of nightlies as regression tests. The nightlies cover the most important scenarios, so if the test case fails, a regression has occurred that the dev must investigate immediately. (If you’re wondering why don’t devs just run tests before checking in, they do – which is a topic for a later time. Or if you’re wondering why don’t devs just have the testers test the build before checking in, we do – called buddy tests – which is a topic for a later time.)

Consider creating a new text file. Putting the above into practice, a create text file nightly may verify the following:

1. Is a Text File template available on the New File Dialog? Is the Text File created upon ok’ing the New File Dialog with the text file selected?


2. Can the user insert text into the Text File?


3. Can the user perform basic editing scenarios, like cut, copy, paste, and selection?


4. Can the user save the Text File? Does the File – Save As dialog appear upon save?


5. Does the text file appear dirty upon editing it?


6. Can the user close the file?


7. Does the file appear in the Most Recently Used list?

Why doesn’t a nightly cover more?

One might think, “that isn’t very much testing for a text file,” but consider how many different languages we support, how many different OS we support, the different SKUs, and the Express Skus. After a while, the testing matrix really starts to get big, especially given that we’re not only running these tests with every new build, but we’re also analyzing the results for each new build. And, there are more than 5 builds a week, believe me, when you add in all the different flavors of runs I’ve mentioned above.

To reiterate, the point of a nightly is to ensure that the feature is “testable”, also called “Self-Test”, and to find any regressions. If nightlies are passing at 100%, the feature is said to be “Self-Host” or testable for other scenarios. Once QA signs off on a feature / build, the feature / build is said to be shippable. Once nightlies are finished running, we are able to say, “This build or feature is Self-Host." If the build fails (obviously implying no features are testable), the build is considered Self-Toast, a term used by the build lab, which is a great play-on-words.