By: Tamás K. Papp

Re-posted from: https://tamaspapp.eu/post/julia-local-coverage/

While codecov.io and coveralls.io are fine services and very easy to set up for Github repositories with Julia packages, they become more difficult to use under some circumstances. For example, if testing requires large binaries you don’t want to build every time, you need to build a Docker image to run testing and obtain coverage information. Also, if your test scripts take a long time and you have access to a powerful computer, you might want to just run coverage locally when working on the code.

Naturally, Julia has all the facilities for generating and collecting coverage information locally. To make their application even easier, I packaged up a small set of scripts in LocalCoverage.jl that I use to generate and visualize coverage information for Julia packages on my machine.

This is how it works:

1. run generate_coverage(pkg) to

   a. run Pkg.test(pkg; coverage = true),

   b. use Coverage.jl for collecting information to a single file in lcov format,

   c. unless disabled, run the external program genhtml to format everything as a nice HTML file.

2. if you have generated the HTML files, open_coverage(pkg) opens them for you in your default browser.

3. if you want to tidy up, run clean_coverage(pkg).

This is what it looks like for this package:

By: Tamás K. Papp

Re-posted from: https://tamaspapp.eu/post/julia-ci-gitlab/

As an academic, I picked up good practices for programming mostly by osmosis. My approach to “testing” software went through the following stages:

1. See if the code runs (this got me through my undergrad years).

2. Check if the results “look OK”, then forget about testing.

3. Compare results occasionally to known results from papers or other code (eg in a different language).

4. Write some unit tests ex post, as an afterthought after the code is finished (time pressure helps to ensure that overtesting is never a problem).

5. Use unit tests from the very beginning, especially before optimizing and refactoring code.

6. Set up automatic testing, as part of continuous integration.

I think that 1–3 above is a pretty usual path, naturally traversed after the recognition that some testing would be nice, but lacking the knowledge of how to implement it in a consistent manner. This is comparable to using copies of directories as crude attempts at “version control”.

Later, I picked up 4–6 while being exposed to these ideas when learning about new languages. Automated unit testing is one of those things one does not miss until learning about it, then subsequently cannot imagine doing without. In a research context, the two main advantages are scientific integrity — I should make a best effort to ensure that my results are correct — and dealing with bugs early. While the first one is abstract and philosophical, the second is a practical concern: I found that if I skimp on testing, the bugs show up much later, usually at an inconvenient time, and I will have to spent time locating the bug (not always trivial, especially with numerical code) and switch context to something I was working on months ago. It is my experience that while tests can be tedious to write, time spent on them is a very good investment.

I knew about unit tests before coming to Julia, but learned about automated CI in the Julia community. This was because package template libraries “do the right thing” by making it very easy to set up an automated testing framework: for example, PkgDev.generate creates the appropriate test configuration for Travis CI and various coverage services.

I never cease to be amazed by the fact that these services are available for free for public / open source projects, which is very generous of these providers. However, occasionally one would like to keep the project private for a little while. The usual scenario for me is working on code that is related to a paper, which I plan to make public with the latter; in this case one would need the pro (non-free) version of Travis and related tools.

Alternatively, Gitlab offers CI/CD with private repositories. I am exploring this at the moment for various projects, and boiled down the necessary configuration into the repository GitlabJuliaDemo.jl. It has

1. a CI setup for Pkg.test,

2. a coverage summary as a percentage.

While coverage analysis could be automated too with a custom Docker image, I leave his for future work.¹

So far, I am very satisfied with Gitlab. The interface is well-designed, clean, and intuitive; tests complete in a few minutes (just like Travis).

By: Tamás K. Papp

Re-posted from: https://tamaspapp.eu/post/travis-docker-julia-ci/

This post may be useful for maintainers of Julia packages which require a large binary dependencies on CI services like Travis.

I have recently started using Kristoffer Carlsson’s excellent PGFPlotsX for plotting. The package is a thin wrapper which emits LaTeX code for use with pgfplots, which is extremely versatile and well-documented.¹ However, since most of the action happens in LaTeX, unit testing requires a lot of binary dependencies, including the TeXLive suite and some related packages. This is not a problem on one’s own machine where these would need to be installed just once, but when I submitted PRs, tests on Travis timed out more often than not because it had to install all of these for every run using apt-get.

The documentation of Travis suggested that docker may be a solution for such cases, and I have been looking an opportunity to experiment with it anyway. After reading their tutorial it was relatively quick to produce an image based on plain vanilla Ubuntu 17.10, which is available as a docker image to build on, and the required TeXLive and related packages, plus some utilities.

During building the image, I download the binaries for the stable version Julia, while nightly is downloaded on demand. This speeds up CI by 40–50 seconds for stable.

This is how it is run:

1. the directory of the Julia package is mounted in the container at /mnt,

2. Pkg.clone() and testing proceed as usual,

3. coverage results are copied back to /mnt when done.

The resulting image runs in 3–4 minutes consistently. In case someone finds it useful for Julia packages with similarly large binary dependencies, I made it available as texlive-julia-minimal-docker on Github.² Naturally, for projects with other large binary dependencies, one would install different Ubuntu packages or binaries.