Re-posted from: https://bkamins.github.io/julialang/2022/04/22/testing.html
Introduction
Recently I was invited by Talk Julia to take part in the podcast.
Today it has been released and you can watch it on YouTube.
In the discussion I share my thoughts on DataFrames.jl design principles
and discuss several examples from my upcoming Julia for Data Analysis
book.
One of the things I discuss in the podcast is that in DataFrames.jl development
process we are putting a lot of emphasis on tests so I thought that it is worth
to expand on this topic a bit more in this post.
The codes I use were tested under Julia 1.7.2 and InlineStrings.jl 1.2.2.
Testing and reproducibility of tests
When one develops some software ensuring its proper test coverage is one of the
key practices that help maintaining good quality of code.
My experience with DataFrames.jl is that taking care of proper test coverage
serves three important goals:
- Making sure the functionality we provide follows the contract specified in
its documentation. This is particularly important when testing corner case
situations, since they happen rarely in practice (so there is lower
probability that users spot problems and report them) and also allow us to
make sure our design is logically consistent. - Making sure that as we add new functionalities to the package we do not
accidentally break something. - Making sure that upgrading dependencies of DataFrames.jl to newer versions
does not break something (and the biggest such dependency is Base Julia).
A crucial part of writing tests is ensuring their reproducibility. What I mean
by this is that when you run your test suite twice on you should get the same
results. This intends to avoid situations in which you run your tests and get a
bug report. Then you run the tests again and the bug is not present. Such
situation is unwanted, as it is later hard to locate the root cause of the
reported bug.
Randomized tests
When one implements complex algorithms it is hard to cover all possible
hard testing scenarios by writing them down by hand. In such situations,
one of the possible testing methods is to use randomized tests.
An example, old and already resolved, problem that potentially could have been
caught by randomized tests is an issue related to pasting Unicode
characters in Julia REPL. Let me here present a minimal working example of a
code having a similar problem.
Assume I want to write a code that strips last character from a string. Here
is an attempt to implement it:
julia> mychop(s::AbstractString) = isempty(s) ? s : s[1:end-1]
mychop (generic function with 1 method)
Let us write a simple test set for this function:
julia> using Test
julia> @testset "basic test" begin
@test mychop("") == ""
@test mychop("a") == ""
@test mychop("abc") == "ab"
end
Test Summary: | Pass Total
basic test | 3 3
Test.DefaultTestSet("basic test", Any[], 3, false, false)
All looks good so far. However, let us run some more advanced testing
of mychop using randomized tests:
julia> using Random
julia> @testset "advanced tests" begin
Random.seed!(1234)
for _ in 1:40
len = rand(1:10)
input = rand(UInt8, len) |> String
output = join(collect(input)[1:end-1])
@test output == mychop(input)
end
end
advanced tests: Error During Test at REPL[83]:7
Test threw exception
Expression: output == mychop(input)
StringIndexError: invalid index [9], valid nearby indices [8]=>'˄', [10]=>'�'
Test Summary: | Pass Error Total
advanced tests | 39 1 40
ERROR: Some tests did not pass: 39 passed, 0 failed, 1 errored, 0 broken.
What I do in this test is generateing input string using randoom bits and then
use a slow method to get a desired output string by going through individual
characters of the original string. I ensure reproducibility of this test on a
given version of Julia by setting the seed of random number generator using
Random.seed!(1234).
From the bug report we can see that something is not right with indexing. The
problem occurs if the second character from the end of the string is not ASCII.
Let us check it:
julia> mychop("∀a")
ERROR: StringIndexError: invalid index [3], valid nearby indices [1]=>'∀', [4]=>'a'
The point of randomized test is that guessing the test scenario
second character from the end of the string is not ASCII
is not easy.
Let us fix mychop to resolve this problem:
julia> mychop(s::AbstractString) = isempty(s) ? s : s[1:prevind(s, end)]
mychop (generic function with 1 method)
julia> @testset "basic test" begin
@test mychop("") == ""
@test mychop("a") == ""
@test mychop("abc") == "ab"
end
Test Summary: | Pass Total
basic test | 3 3
Test.DefaultTestSet("basic test", Any[], 3, false, false)
julia> @testset "advanced tests" begin
Random.seed!(1234)
for _ in 1:32768 # pick some round larger number to be sure all works well
len = rand(1:10)
input = rand(UInt8, len) |> String
output = join(collect(input)[1:end-1])
@test output == mychop(input)
end
end
Test Summary: | Pass Total
advanced tests | 32768 32768
Test.DefaultTestSet("advanced tests", Any[], 32768, false, false)
Are we done now? Not yet. We should check our function with some other
AbstractString than String. Let me use InlineStrings.jl:
julia> using InlineStrings
julia> @testset "InlineStrings.jl test" begin
@test "ab" == @inferred mychop(InlineString("abc"))
end
InlineStrings.jl test: Error During Test at REPL[110]:2
Test threw exception
Expression: "ab" == #= REPL[110]:2 =# @inferred(mychop(InlineString("abc")))
return type SubString{String3} does not match inferred return type Union{SubString{String3}, String3}
Test Summary: | Error Total
InlineStrings.jl test | 1 1
ERROR: Some tests did not pass: 0 passed, 0 failed, 1 errored, 0 broken.
As you can see there is still some more work to be done, as our mychop
function is not type stable, which is checked by the @inferred macro. The
problem is that indexing into String3 that happens if the passed string is not
empty produces a SubString. Let us fix it by making sure that in every branch
of code we apply the same operation to our source string (in the code, like in
the codes above, we use the fact that AbstractString is guaranteed to use
1-based indexing).
julia> mychop(s::AbstractString) = s[1:prevind(s, max(1, end))]
mychop (generic function with 1 method)
julia> @testset "advanced tests" begin
Random.seed!(1234)
for _ in 1:32768 # pick some round larger number to be sure all works well
len = rand(0:10) # make sure to cover 0-length strings
input = rand(UInt8, len) |> String
output = join(collect(input)[1:end-1])
@test output == @inferred mychop(input)
@test output == @inferred mychop(InlineString(input))
end
end
Test Summary: | Pass Total
advanced tests | 65536 65536
Test.DefaultTestSet("advanced tests", Any[], 65536, false, false)
Now all works as expected and we are done.
Conclusions
I hope you found my thoughts on writing tests useful. As usual, I have some
additional comments regarding the presented codes:
- I used
Random.seed!explicitly in my tests. However, the@testsetmacro,
before the execution of its body, makes a call toRandom.seed!(seed)where
seedis the current seed of the global RNG. Therefore, if you design a
larger test suite you do not have to set the seed in every@testset. It is
enough to set it once per all tests you run. - As I have already commented, the presented codes are reproducible on a given
version of Julia. If you want them to be reproducible across different
versions of Julia I recommend you to use for example the
StableRNGs.jl package as a source of randomness in your tests.