How many bugs are in 1,000 lines of code? Typical code: 1-10 Platform code: 0.1-1 The best—NASA: 0.01-0.1 Never assume your software doesn’t have bugs. Test-Driven Development Test before you build! Specs are already written We know what the expected behavior is We can write tests for the expected behavior first All tests fail to start We know we are done writing code when all tests pass “NYI” (not-yet implemented) often, writing test exposes gaps in your specs How NOT! not write tests Random Sampling Pick one or two inputs and show your code works on it Why it doesn’t work: there maybe specific inputs that break your code Exhaustive Testing Test for the domain of inputs Why it doesn’t work: tests run forever How DO you write tsets Black-Box Testing Pretend the code implementation is a black box All you know is what the specification; and what the input/output produces White-Box Testing You can see the implementation You test for specific edge cases Off-by-one, running time, specific inputs, etc. Malicious Testing What happens if a user is trying to break your system Sometimes, this is known as “pen-testing” or “white-hack hacking” Take CS340 Compsec How BIG are your tests Unit Testing Backbone of testing Typically, that means one test per function Tests choose representative inputs Idempotent: the state of the testing system should be a the beginning and end of the test (tests should revert) (setup + teardown tests) Subsystem Testing Exercise multiple functions working together in a system Often takes longer OK to run these less frequently End-to-End Integration Exercise the entire workflow May involve external libraries, hardware, etc. Regression Testing Isolate the cause of the bug to the smallest possible test case Write a test assuming the bug is fixed Fix the bug Add the test to your test suite How MUCH do we run tests Ideally, run tests every time code is committed Ideally—run tests that address the function Schedule long tests What to test for see also here equivalence partitioning Come up with one test case per equivalence class. For instance, for a function that uppercases letters, analyze the following: Lowercase letters Uppercase letters Non-alpha letters Non-printable letters Combinations Each group will therefore have nicely the requirements covered boundary value analysis In addition to just testing 1 element per class in equivalence partitioning, try to test boundary values (off-by-one, etc.) cases for each equivalence class if you can come up with them. Arrange, Act, Assert arrange for setup by setting up variables, etc., and define the expected result (yes we do it before to be more readable) act do the thing assert correctness by checking the expected result