Learning Goals

  • Get to know a few common terms of testing
  • Work with the Julia unit testing package Test.jl

Material is taken and modified from the SSE lecture, which builds partly on the py-rse book, and from the Test.jl docs.

1. General Introduction to Testing

What is Testing?

  • Smelling old milk before using it
  • A way to determine if a software is not producing reliable results and if so, what is the reason
  • Manual testing vs. automated testing

Why Should you Test your Software?

  • Improve software reliability and reproducibility
  • Make sure that changes (bugfixes, new features) do not affect other parts of software
  • Generally all software is better off being tested regularly. Possible exceptions are very small codes with single users.
  • Ensure that a released version of a software actually works.

Some Ways to Test Software

  • Assertions
  • Unit testing
  • Integration testing
  • Regression testing


@assert condition "message"
  • Principle of defensive programming
  • Nothing happens when an assertion is true; throws error when false.
  • Types of assertion statements:
    • Precondition
    • Postcondition
    • Invariant
  • A basic but powerful tool to test a software on-the-go

Unit Testing

  • Catching errors with assertions is good but preventing them is better.
  • A unit is a single function in one situation.
    • A situation is one amongst many possible variations of input parameters.
  • User creates the expected result manually.
  • Actual result is compared to the expected result by @test.

Integration Testing

  • Test whether several units work in conjunction.
  • Integrate units and test them together in an integration test.
  • Often more complicated than a unit test and gives higher test coverage.

Regression Testing

  • Generating an expected result is not possible in some situations.
  • Compare the current actual result with a previous actual result.
  • No guarantee that the current actual result is correct.
  • Risk of a bug being carried over indefinitely.
  • Main purpose is to identify changes in the current state of the code with respect to a past state.

2. Unit Testing in Julia with Test.jl

Setup of Test.jl

  • Standardized folder structure:

    β”œβ”€β”€ Manifest.toml
    β”œβ”€β”€ Project.toml
    β”œβ”€β”€ src/
    └── test
        β”œβ”€β”€ Manifest.toml
        β”œβ”€β”€ Project.toml
        β”œβ”€β”€ runtests.jl
        └── setup.jl
  • Singular test vs plural runtests.jl

  • setup.jl for all using XYZ statements, included in runtests.jl

  • Additional packages in [extra] section of ./Project.toml or in new ./test/Project.toml

    • In case of the latter: Do not add the package itself to the ./test/Project.toml
  • Run: ]test when root project is activated

Implement and Structure Tests

  • @test expr: Test whether expression expr is true

  • @test expr broken=true: Explicitly mark test as broken

  • @test_throws exception expr: Test whether expression expr throws exception (test unhappy path)

    julia> @test_throws DimensionMismatch [1, 2, 3] + [1, 2]
    Test Passed
          Thrown: DimensionMismatch
  • @testset: Structure tests

    @testset "trigonometric identities" begin
        ΞΈ = 2/3*Ο€
        @test sin(-ΞΈ) β‰ˆ -sin(ΞΈ)
        @test cos(-ΞΈ) β‰ˆ cos(ΞΈ)
  • @testset for ... end: Test in loop

Further Reading and Watching

3. Test.jl Demo

We use MyTestPackage, which looks as follows:

β”œβ”€β”€ Manifest.toml
β”œβ”€β”€ Project.toml
β”œβ”€β”€ src
β”‚   β”œβ”€β”€ find.jl
β”‚   └── MyTestPackage.jl
└── test
    β”œβ”€β”€ find.jl
    β”œβ”€β”€ Manifest.toml
    β”œβ”€β”€ Project.toml
    β”œβ”€β”€ runtests.jl
    └── setup.jl
  • Look at MyTestPackage.jl and find.jl: We have two functions find_max and find_mean, which calculate the maximum and mean of all elements of a ::AbstractVector.

    • Assertions were added to check for NaN values
  • Look at runtests.jl:

    • Why do we need using MyTestPackage?
    • We include dependencies via setup.jl: Test and StableRNG.
    • Testset β€œfind”
  • Look at find.jl

    • Unit tests for find_max and find_mean
    • test_throws to test unhappy path
    • Test with absolute tolerance
    • Integration test, which tests combination of both methods
  • Run tests:

    ]activate .

4. Exercise

Write tests for your own statistics package 😊