Category Archives: Julia

Developing your Julia package

By: DSB

Re-posted from: https://medium.com/coffee-in-a-klein-bottle/developing-your-julia-package-682c1d309507?source=rss-8bd6ec95ab58------2

A Tutorial on how to quickly and easily develop your own Julia package

We’ll show step-by-step how to develop your first package in Julia. To do this, we use the very useful PkgTemplate.jl, and we base our Tutorial on this guide by Quantecon and this video by Chris Rackauckas.

First things first. Start by creating the folder to store your package. In our case, we’ll be creating a package named “VegaGraphs”, which is a package that I’m developing. So, inside this folder, open your Julia REPL and install PkgTemplate.jl.

Installing PkgTemplates

1. Creating your Package Template

First, we need to create a template for our package. This template will include things like Licensing, Plugins, Authors, etc. For our example, we’ll be using a simple template with the MIT License. The plugins used will be GitHub Actions and Codecov. I’ll not be diving into these plugins, but just for the sake of clarity, I’ll briefly explain what they do.

  • GitHub Actions is a plugin that automatically builds a virtual machine and tests your code, hence, you can control the machine configurations necessary to running your package.
  • Codecov is a plugin that analysis your code, and evaluates how much of it is covered in your tests. So, for example, suppose that you wrote 3 different functions, but forgot to write a test for one of them. Then, Codecov will point out that there is no tests for such function.

Still inside the REPL, run the following commands:

t = Template(;user="YourUserNameOnGithub", plugins = [GitHubActions(), Codecov()], manifest = true)
generate("VegaGraphs.jl",t)

The first line of code defines the template, while the second one will generate your package. Note that I didn’t specify a folder, so the package will be created in a default location, which will be ~/.julia/dev (for Linux).

Now, just copy the files from the ~/.julia/dev/VegaGraphs to the folder where you will be working from, and then setup your git repository by running the following commands in the terminal:

# inside the VegaGraphs working folder
git init
git add -A
git commit -m "first commit"
git branch -M master
git remote add origin [email protected]:YOURUSERNAME/VegaGraphs.git
git push -u origin master

Taking a look inside the generated folder, you’ll have two folders and four files:

./src
./test
README.md
LICENSE
Manifest.toml
Project.toml
  • /src : This folder is where you will write the code for your package per se;
  • /test : Here is for storing your tests;
  • Project.toml: This is where you will store information such as the author, dependencies, julia version, etc;
  • Manifest.toml : This is a machine generated file, and you should just leave it be;

The other files are self explanatory.

2. Writing Code

We are ready to start coding our package. Note that inside the /src folder we already have a file named VegaGraphs.jl , which is the main file of our package. We can do all our coding directly inside VegaGraphs.jl , but as our code gets large, this might become messy.

Instead, we can write many different files for organizing our code, and then use VegaGraphs.jl to join everything together. Let’s do an example. We’ll code a very simple function, which we’ll store in another file, called graph_functions.jl , that will also be inside the ./src folder.

Here is some example code:

# code inside graph_functions.jl
function sum_values(x,y)
return x+y
end

The code above is a simple implementation of a function. Below I show how to actually make this function available to users. One just needs to “include” the graph_functions.jl file, and to export the plot_scatter . Once exported, the function is now available to anyone who imports our package.

# code inside VegaGraphs.jl
module VegaGraphs
using VegaLite
export sum_values
include("graph_functions.jl")
end

Note that, besides including our function, I’ve also imported the VegaLite package. Hence, I need to specify VegaLite.jl as a dependency. We’ll do this by using the REPL again.

Go to the root of your package and open the REPL by running the command julia in the terminal.Now, press ] . This will put you on “package mode”. Next, write activate . , which will activate the Julia environment to your current folder. Finally, write add VegaLite , and this will add VegaLite.jl to your dependencies inside the Project.toml file.

Adding dependency to your package

3. Creating and running tests

So we’ve implemented a function and added a dependency to our package. The following step is to write a test, to guarantee that our code is indeed working. The code is self-explanatory, we just write our test inside a testset. You can write as many as you like to guarantee that your function is working properly.

# code inside ./test/runtests.jl
using VegaGraphs
using Test
@testset "VegaGraphs.jl" begin
x = 2
y = 2
@test VegaGraphs.sum_values(x,y) == 4
end

Once the test is written, we have to run it and see if everything is working. Again, go to the root of the project and open your REPL. Guarantee that your environment is activated and run the tests as shown in the image below:

Running tests for your package

This will run your tests and see if everything passes. Once everything passes, we can trust that our code is working properly, and we can move on to more implementations.

4. Workflow – Text editor + Jupyter Notebook

With everything shown up until now, you are already ready to develop your package in Julia. Still, you might be interested on how to develop an efficient workflow. There are many possibilities here, such as using IDEs such as Juno and VsCode. I prefer to use Jupyter Notebooks with Vim (my text editor of choice), and doing everything from the terminal.

To use your developing package in your Notebook, you will need to activate your environment in a similar way that we’ve been doing up until now. As you open a new Notebook, run the following code in the very first cell.

Using Jupyter Notebook for developing packages

Note here that besides activating the environment, we also imported a package called Revise. This package is very helpful, and it should be imported right in the beginning of your notebook, before you actually import your own package, otherwise it won’t work properly!

Every time you modify the code in your package, to load the changes to your notebook you would have to restart the kernel. But when you use Revise.jl, you don’t need to restart your kernel, just import your package again, and the modifications will be applied.

Now, my workflow is very simple. I use Vim to modify the code in my package, and use the Notebook for trying things out. Once I get everything working as it should, I write down some tests, and test the whole thing.

5. Registering/Publishing your Package

Finally, suppose that you’ve finished writing your package, and you are ready to share it with the Julia community. Like with other packages, you want users to be able to write a simple Pkg.add("MyPackage") and install it. This process is called registering.

To do this, first go to Registrator.jl and install the app to your Github account.

In the Registrator.jl Gitub page, click in the “install app”

Next, go to your package Github’s page and enter the Issues tab. Create a new issue, an write @JuliaRegistrator register() , as shown in the image below.

Registering your package

Once you’ve done this, the JuliaRegistrator bot will open a pull request for your package to be registered. Unless you actually want your package to be registered, don’t actually submit the issue.

And that’s all.


Developing your Julia package was originally published in Coffee in a Klein Bottle on Medium, where people are continuing the conversation by highlighting and responding to this story.

Binning your data with Julia

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2020/12/11/binning.html

Introduction

Cutting data into groups (binning) is one of the most common data preprocessing
tasks.

You can easily do binning into groups of equal sizes using the cut function
from CategoricalArrays.jl like this (here we bin a vector of values from 1 to 10
into 2 groups):

julia> using CategoricalArrays

julia> cut(1:10, 2)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 5.5)"
 "Q1: [1.0, 5.5)"
 "Q1: [1.0, 5.5)"
 "Q1: [1.0, 5.5)"
 "Q1: [1.0, 5.5)"
 "Q2: [5.5, 10.0]"
 "Q2: [5.5, 10.0]"
 "Q2: [5.5, 10.0]"
 "Q2: [5.5, 10.0]"
 "Q2: [5.5, 10.0]"

However, the issue becomes more challenging when the number of bins is not a
divisor of vector length or if you have duplicates in the data.

The post is tested under Julia 1.5.3, DataFrames.jl 0.22.1, CategoricalArrays.jl
0.9.0, and FreqTables.jl 0.4.2.

The corner cases of binning

Let us first highlight some potential issues when binning data.

The first problem is when the number of groups is not a divisor of the vector
length. Let us check it out on some examples:

julia> cut(1:10, 3)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 4.0)"
 "Q1: [1.0, 4.0)"
 "Q1: [1.0, 4.0)"
 "Q2: [4.0, 7.0)"
 "Q2: [4.0, 7.0)"
 "Q2: [4.0, 7.0)"
 "Q3: [7.0, 10.0]"
 "Q3: [7.0, 10.0]"
 "Q3: [7.0, 10.0]"
 "Q3: [7.0, 10.0]"

julia> cut(1:10, 4)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 3.25)"
 "Q1: [1.0, 3.25)"
 "Q1: [1.0, 3.25)"
 "Q2: [3.25, 5.5)"
 "Q2: [3.25, 5.5)"
 "Q3: [5.5, 7.75)"
 "Q3: [5.5, 7.75)"
 "Q4: [7.75, 10.0]"
 "Q4: [7.75, 10.0]"
 "Q4: [7.75, 10.0]"

The cut function is deterministic and it uses the quantile function to find
the bin endpoints. This means that in the first example cut(1:10, 3) the third
bin will be always larger than the first and second bin. Similarly in cut(1:10,
4)
the first and the fourth bins are going to be larger deterministically.

The other problem is duplicates in data. Consider the following scenario:

julia> cut([1; fill(2, 8); 3], 2)
10-element CategoricalArray{String,1,UInt32}:
 "Q1: [1.0, 2.0)"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"
 "Q2: [2.0, 3.0]"

We want two bins. Ideally both should have five elements, but since we have
duplicates in our data the first bin has size one and the second size nine.

In some cases you will like what cut produces, but in other cases
one might want to avoid these two problems, that is:

  • always make the bins of equal size and if it is not possible to do so, make
    the decision which bin should be larger and which smaller randomly;
  • allow duplicates to be split between two or more bins (this is then
    unavoidable in some cases), but in such a way that each duplicate has the same
    chance to fall into each bin.

Random binning

Here is the function that performs the binning that has the properties I have
described above:

using DataFrames
using FreqTables
using Random

function binvec(x::AbstractVector, n::Int,
                rng::AbstractRNG=Random.default_rng())
    n > 0 || throw(ArgumentError("number of bins must be positive"))
    l = length(x)

    # find bin sizes
    d, r = divrem(l, n)
    lens = fill(d, n)
    lens[1:r] .+= 1
    # randomly decide which bins should be larger
    shuffle!(rng, lens)

    # ensure that we have data sorted by x, but ties are ordered randomly
    df = DataFrame(id=axes(x, 1), x=x, r=rand(rng, l))
    sort!(df, [:x, :r])

    # assign bin ids to rows
    binids = reduce(vcat, [fill(i, v) for (i, v) in enumerate(lens)])
    df.binids = binids

    # recover original row order
    sort!(df, :id)
    return df.binids
end

Let us now test the binning on the following vector:

julia> Random.seed!(1234);

julia> x = repeat('a':'c', 3)
9-element Array{Char,1}:
 'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
 'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)
 'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)
 'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
 'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)
 'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)
 'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
 'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)
 'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)

julia> binvec(x, 2)
9-element Array{Int64,1}:
 1
 2
 2
 1
 2
 2
 1
 1
 2

As you can see 'b's are split betwen bin 1 and 2 to make them have almost
equal size.

Let us make sure that binvec does the right job in deciding on bin sizes
and splitting 'b's between both bins.

julia> df = reduce(vcat, [DataFrame(x=x, run_id=i, row_id=axes(x, 1),
                                    group_id=binvec(x, 2)) for i in 1:10_000]);

julia> freqtable(df, :group_id, :row_id, :x)
2×9×3 Named Array{Int64,3}

[:, :, x='a'] =
group_id ╲ row_id │     1      2      3      4      5      6      7      8      9
──────────────────┼──────────────────────────────────────────────────────────────
1                 │ 10000      0      0  10000      0      0  10000      0      0
2                 │     0      0      0      0      0      0      0      0      0

[:, :, x='b'] =
group_id ╲ row_id │    1     2     3     4     5     6     7     8     9
──────────────────┼─────────────────────────────────────────────────────
1                 │    0  4941     0     0  5005     0     0  5027     0
2                 │    0  5059     0     0  4995     0     0  4973     0

[:, :, x='c'] =
group_id ╲ row_id │     1      2      3      4      5      6      7      8      9
──────────────────┼──────────────────────────────────────────────────────────────
1                 │     0      0      0      0      0      0      0      0      0
2                 │     0      0  10000      0      0  10000      0      0  10000

Indeed we see that each 'b' falls to group 1 and group 2 with 50%
probability. Also the expected size of group 1 and group 2 is 4.5 as
desired. (both calculations are approximate because we used simulation.)

Conclusions

First – let me comment in what scenario the binning I described is desirable.
Assume you have a set of patients you want to vaccinate against COVID-19. Now
let each of them have assigned a discrete urgency level (typically there will be
3 or 4 such urgency levels). You then have to split them into several batches
of equal size so that each batch gets a vaccine in a different period. If you
want to be fair in assigning people to batches you get exactly the setting I
have described.

Second – as usual I wanted to showcase some features of JuliaData ecosystem. In
particular you have seen reducevcat combo twice (for vectors and for data
frames) and integration of FreqTables.jl with DataFrames.jl that I am very fond
of. Of course this is not the fastest way to get the desired results. I
encourage you to write a faster function that does the same what the binvec
function does.