By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/07/22/juliacon2022.html

Introduction

During JuliaCon 2022 I will run a
tutorial on DataFrames.jl.
In the tutorial I will focus on ways you can write transformation operations
using the select/transform/combine functions and the operation
specification syntax.

In this post I want to give you a preview of the topics I am going to cover
in my tutorial.

The post was written under Julia 1.7.2, DataFrames.jl 1.3.4,
and DataFramesMeta.jl 0.12.

What is operation specification syntax?

If you are new to DataFrames.jl then you probably wonder what
operation specification syntax is. Fortunately it is quite easy.

If you want to transform data using one of the select/transform/combine
functions you can specify the transformation you want to perform using the
following general syntax:

[source columns] => [function] => [output columns]

For example if you write :a => mean => :b you will get a mean of column :a
and store it in output data frame in column :b (visually: input column :a is
passed to the mean function whose output is passed to output column :b).

Additionally, if you prefer assignment style of specifying operations you can
use DataFramesMeta.jl package that allows you to write the same as just
:b = mean(:a). To use DataFramesMeta.jl you need to prefix the appropriate
function name with @ (to turn it into a macro).

Let me show you a minimal working example of such a transformation.
We compute mean of column :val by groups defined by the :id column:

julia> using DataFramesMeta

julia> using Statistics

julia> df = DataFrame(id=[1, 2, 1, 2, 1, 2], val=1:6)
6×2 DataFrame
 Row │ id     val
     │ Int64  Int64
─────┼──────────────
   1 │     1      1
   2 │     2      2
   3 │     1      3
   4 │     2      4
   5 │     1      5
   6 │     2      6

julia> combine(groupby(df, :id), :val => mean => :mean_val)
2×2 DataFrame
 Row │ id     mean_val
     │ Int64  Float64
─────┼─────────────────
   1 │     1       3.0
   2 │     2       4.0

julia> @combine(groupby(df, :id), :mean_val = mean(:val))
2×2 DataFrame
 Row │ id     mean_val
     │ Int64  Float64
─────┼─────────────────
   1 │     1       3.0
   2 │     2       4.0

This is a simple example of what operation specification syntax
can do. In this post let me give you a more complex example (I explain
all the details of how it works in my upcoming tutorial).

The question from StackOverflow

In this StackOverflow question the user wanted to analyze iris data
set and get 25% and 75% quantiles of the Sepal.Length column.

The R code that the StackOverflow user provided was:

> library(dplyr)

> iris %>%
+        group_by(Species) %>%
+        summarise(
+           quantile(Sepal.Length, c(.25, .75)) %>%
+              matrix(nrow = 1) %>%
+              as.data.frame() %>%
+              setNames(paste0("Sepal.Length", c(.25, .75)))
+     )
# A tibble: 3 x 3
  Species    Sepal.Length0.25 Sepal.Length0.75
  <fct>                 <dbl>            <dbl>
1 setosa                 4.8               5.2
2 versicolor             5.6               6.3
3 virginica              6.22              6.9

The question was how to achieve the same with DataFrames.jl and
DataFramesMeta.jl?

Here is a solution. First we need to load the iris dataset (in the code I take
advantage of the fact that this dataset is bundled into DataFrames.jl
installation folders):

julia> using CSV

julia> iris = CSV.read(joinpath(dirname(pathof(DataFrames)),
                                "..", "docs", "src", "assets", "iris.csv"),
                       DataFrame)
150×5 DataFrame
 Row │ SepalLength  SepalWidth  PetalLength  PetalWidth  Species
     │ Float64      Float64     Float64      Float64     String15
─────┼──────────────────────────────────────────────────────────────────
   1 │         5.1         3.5          1.4         0.2  Iris-setosa
   2 │         4.9         3.0          1.4         0.2  Iris-setosa
  ⋮  │      ⋮           ⋮            ⋮           ⋮             ⋮
 149 │         6.2         3.4          5.4         2.3  Iris-virginica
 150 │         5.9         3.0          5.1         1.8  Iris-virginica
                                                        146 rows omitted

Now we are ready to use the combine function and the operation specification
syntax:

julia> combine(groupby(iris, :Species),
               :SepalLength =>
               (x -> [quantile(x, [0.25, 0.75])]) =>
               string.("SepalLength", [0.25, 0.75]))
3×3 DataFrame
 Row │ Species          SepalLength0.25  SepalLength0.75
     │ String15         Float64          Float64
─────┼───────────────────────────────────────────────────
   1 │ Iris-setosa                4.8                5.2
   2 │ Iris-versicolor            5.6                6.3
   3 │ Iris-virginica             6.225              6.9

With DataFramesMeta.jl you would write this using the @combine macro as
follows (I additionally show here how to use operation chaining with @chain):

julia> @chain iris begin
           groupby(:Species)
           @combine($["SepalLength0.25", "SepalLength0.75"] = [quantile(:SepalLength, [0.25, 0.75])])
       end
3×3 DataFrame
 Row │ Species          SepalLength0.25  SepalLength0.75
     │ String15         Float64          Float64
─────┼───────────────────────────────────────────────────
   1 │ Iris-setosa                4.8                5.2
   2 │ Iris-versicolor            5.6                6.3
   3 │ Iris-virginica             6.225              6.9

Conclusions

The operation specification syntax was designed to allow doing simple
transformations in an easy way, but at the same to also support quite complex
operations, like the one we did on the iris data frame.

If you would like to hear a detailed explanation of how to write such code
please join me during the upcoming workshop.

You can find all the examples that I will use during the workshop in the
accompanying GitHub repository.

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/07/22/regex.html

Introduction

During JuliaCon 2022 I gave a tutorial on DataFrames.jl.
You can find its recording on YouTube and all source code on GitHub.

This post is a follow up to one of the questions that I got during
the workshop. The topic of the discussion was applying the same function to
many columns of a data frame. Since the question is quite technical I will
first give you a brief introduction to the topic and next dive deep into the
issue.

I hope that this will be a useful material even for people that do not use
DataFrames.jl as we will explore the consequences of the
avoid type piracy rule that the Julia Manual recommends.

The post was written under Julia 1.7.2, DataFrames.jl 1.3.4.

How can one apply a function to multiple columns in DataFrames.jl?

Let us create a sample data frame first:

julia> using DataFrames

julia> df = DataFrame(a1=1:2, b1=3:4, a2=5:6)
2×3 DataFrame
 Row │ a1     b1     a2
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      3      5
   2 │     2      4      6

Next I define a simple function that allows us to inspect what arguments
it received:

julia> inspect(x...) = Ref(x)
inspect (generic function with 1 method)

In this function I wrap a tuple of arguments in Ref as in DataFrames.jl
Ref protects the wrapped value against being expanded.

Let us check this function in action on some simple examples of combine
transformation (if you do not know the operation specification syntax please
check my tutorial on DataFrames.jl I have linked above for an
introduction):

julia> combine(df,
               :a1 => inspect,
               r"a" => inspect,
               Cols(endswith("1")) => inspect)
1×3 DataFrame
 Row │ a1_inspect  a1_a2_inspect     a1_b1_inspect
     │ Tuple…      Tuple…            Tuple…
─────┼────────────────────────────────────────────────
   1 │ ([1, 2],)   ([1, 2], [5, 6])  ([1, 2], [3, 4])

julia> combine(df,
               AsTable(:a1) => inspect,
               AsTable(r"a") => inspect,
               AsTable(Cols(endswith("1"))) => inspect)
1×3 DataFrame
Row │ a1_inspect      a1_a2_inspect             a1_b1_inspect
    │ Tuple…          Tuple…                    Tuple…
────┼─────────────────────────────────────────────────────────────────────
  1 │ ((a1=[1, 2],),) ((a1=[1, 2], a2=[5, 6]),) ((a1=[1, 2], b1=[3, 4]),)

What we can see in these examples is the following:

• r"a" picks all columns whose names match this regular expression
  (in this case contain "a");
• Cols(endswith("1")) picks all columns whose names meet the endswith("1")
  predicate (that is, end with "1");
• by default the selected columns are passed as multiple positional arguments to
  the executed function;
• if you wrap the selected columns in AsTable then they get passed a single
  positional argument in a NamedTuple.

It is crucially important to understand at this point that r"a" and
Cols(endswith("1")) column selectors do not get resolved before being passed
to combine:

julia> r"a" => inspect
r"a" => inspect

julia> Cols(endswith("1")) => inspect
Cols{Tuple{Base.Fix2{typeof(endswith), String}}}((Base.Fix2{typeof(endswith), String}(endswith, "1"),)) => inspect

What I mean by resolved is that the expression will get its meaning (i.e. will
determine which columns it actually selects, inside the combine function in
the context of the data frame that is passed as a first argument to combine).

Having seen these basic examples, let us check how one can apply a given
function to multiple columns individually. You can do it e.g. like this:

julia> combine(df, :a1 => inspect, :a2 => inspect)
1×2 DataFrame
 Row │ a1_inspect  a2_inspect
     │ Tuple…      Tuple…
─────┼────────────────────────
   1 │ ([1, 2],)   ([5, 6],)

but there is an easier way. You can do it like this:

julia> combine(df, [:a1, :a2] .=> inspect)
1×2 DataFrame
 Row │ a1_inspect  a2_inspect
     │ Tuple…      Tuple…
─────┼────────────────────────
   1 │ ([1, 2],)   ([5, 6],)

The point is that combine (and other functions in DataFrames.jl) accept
vectors and matrices of operation specification syntax expressions and above
I create such a vector using broadcasting with .=>.

Let us check:

julia> [:a1, :a2] .=> inspect
2-element Vector{Pair{Symbol, typeof(inspect)}}:
 :a1 => inspect
 :a2 => inspect

julia> combine(df, [:a1 => inspect, :a2 => inspect])
1×2 DataFrame
 Row │ a1_inspect  a2_inspect
     │ Tuple…      Tuple…
─────┼────────────────────────
   1 │ ([1, 2],)   ([5, 6],)

Having the information I have shared above we are now ready to face the pirates.

Using column selectors to pick columns to which we apply the function

We have seen above that r"a" and Cols(endswith("1")) do not get resolved
before they get passed to combine. Therefore how can we apply the inspect
function to all columns selected by them?

The basic approach is to use the names function like this:

julia> names(df, r"a")
2-element Vector{String}:
 "a1"
 "a2"

julia> names(df, r"a") .=> inspect
2-element Vector{Pair{String, typeof(inspect)}}:
 "a1" => inspect
 "a2" => inspect

julia> combine(df, names(df, r"a") .=> inspect)
1×2 DataFrame
 Row │ a1_inspect  a2_inspect
     │ Tuple…      Tuple…
─────┼────────────────────────
   1 │ ([1, 2],)   ([5, 6],)

This method works, but it is a bit heavy-handed, as it requires us to use
the names function that needs the df as its first argument. This duplication
of information is not optimal.

Therefore we are tempted to skip the names(df, r"a") and just write
r"a" .=> inspect. Let us see what it gives us:

julia> combine(df, r"a" .=> inspect)
1×1 DataFrame
 Row │ a1_a2_inspect
     │ Tuple…
─────┼──────────────────
   1 │ ([1, 2], [5, 6])

julia> r"a" .=> inspect
r"a" => inspect

Unfortunately this is not what we expected. The reason is that, as you can see,
r"a" is treated by broadcasting as a scalar. Here we need to note that
the r"a" .=> inspect is resolved before the value of this expression is
passed to combine, so evaluation of this expression cannot be made by Julia
in the context of the df data frame.

Let us check what happens if we use the same approach with
Cols(endswith("1")) .=> inspect:

julia> combine(df, Cols(endswith("1")) .=> inspect)
1×2 DataFrame
 Row │ a1_inspect  b1_inspect
     │ Tuple…      Tuple…
─────┼────────────────────────
   1 │ ([1, 2],)   ([3, 4],)

This time we got what we wanted. It seems that this expression had to be
resolved only after it got passed to combine. Let us inspect it:

julia> Cols(endswith("1")) .=> inspect
DataAPI.BroadcastedSelector{Cols{Tuple{Base.Fix2{typeof(endswith), String}}}}(Cols{Tuple{Base.Fix2{typeof(endswith), String}}}((Base.Fix2{typeof(endswith), String}(endswith, "1"),))) => inspect

We see some strange DataAPI.BroadcastedSelector type. Its role is exactly to
delay the final resolution of broadcasted operation only after the expression
is processed in combine. When combine sees a value of type
DataAPI.BroadcastedSelector it does its post-processing in the context
of the df data frame to give us the desired result.

So how does this relate to type piracy? The answer is:

• we could have made in DataFrames.jl Cols(endswith("1")) .=> inspect
  to have a delayed broadcasting behavior because the Cols selector
  is defined in DataAPI.jl so we can customize how it is handled in broadcasting;
• we could not do the same for r"a", because the Regex type is defined in
  Base Julia, and it has a defined broadcasting behavior. Packages, like
  DataFrames.jl should not change how r"a" is handled by broadcasting because
  it would be type piracy.

Conclusions

So what do we learn from the today’s post?

• for DataFrames.jl users: remember that using a regular expression as a column
  selector, while convenient, does not have the nice broadcasting support as
  other selectors, like Cols, Not, Between, and All, have;
• for general audience: Julia is really flexible and allows packages to
  customize almost everything (in our case how broadcasting works); the only
  limitation is that such customization should be done on the types that you
  define yourself; changing the behavior of types defined in external packages
  is not recommended and it is called type piracy.

If you want to learn what exactly happens when you pass
Cols(endswith("1")) .=> inspect to combine
you can check it here and here.

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/07/15/main.html

Introduction

Many users are attracted to Julia because it offers good performance of code
execution. It is possible, because Julia programs are compiled to native
machine code before they are executed. At the same time Julia is a convenient
tool in interactive data science sessions.

However, there is a fine line between doing data analysis in an interactive
session and writing a script that is meant to be executed many times after
its creation. The problem is that often interactive session turns into
a script eventually, and there is a temptation to use the code originally
written verbatim in that script.

In this post I will explain why I personally never do it. Instead when
I realize that I will need some code many times I change my mindset
how it should be written.

The points I rise in my post today are commonly known so most likely what I
discuss is not new for many of the readers. However, I decided to write it as
keeping them in mind is especially important when coding in Julia, as opposed
to, for example, R or Python, as I will explain in this post.

The post was tested under Julia 1.7.2 and Julia 1.9.0-DEV.575.

How do I define interactive and script coding style?

There are many aspects in which a well written program is different from a log
of an interactive session. However, in my opinion, the most important one is
that in interactive session one heavily uses global variables, while in programs
preferably no global variables are used. Instead, I typically define main
function taking no arguments and I implement all top-level logic in this
function.

The three most important reasons why this is a desirable practice are:

• performance;
• “forgotten variable” problem;
• Julia’s variable scoping rules.

Let me give examples of these three potential problems.

Are loops in Julia fast?

One of the selling points of Julia is that for loops are fast, as opposed
to R or Python. However, you might ask if this is really always the case.

Let us check (start a fresh Julia session):

julia> s = 0
0

julia> @time for i in 1:10^8
           s += i
       end
  5.766783 seconds (200.00 M allocations: 2.980 GiB, 6.07% gc time)

julia> s
5000000050000000

This is not fast. The reason is clear for any relatively experienced Julia
user. In the loop we refer to s variable that is global. It is one of the
first things one learns when one starts using Julia.

The problem is, in my experience, that in practice it is very easy to forget
to fix such issue when switching from interactive mode to script mode if the
code base is more than a few dozen lines of code.

How would the code with top-level main function look in this case?
(start a fresh Julia session)

julia> function main()
           s = 0
           for i in 1:10^8
               s += i
           end
           return s
       end
main (generic function with 1 method)

julia> @time main()
  0.000001 seconds
5000000050000000

There are other options to make this code run fast in top level scope, here are
some of them.

First is to use let block (start a fresh Julia session):

julia> s = 0
0

julia> @time s = let s = s
           s = 0
           for i in 1:10^8
               s += i
           end
           s
       end

  0.000001 seconds (1 allocation: 16 bytes)
5000000050000000

The other is to add type assertion to global variable s (this requires Julia 1.8,
I used Julia 1.9.0-DEV.575 in the test):

julia> s::Int = 0
0

julia> @time for i in 1:10^8
           s += i
       end
  1.226593 seconds (100.00 M allocations: 1.490 GiB, 10.49% gc time)

julia> s
5000000050000000

As you can see type assertion makes the code run faster, but it is slower
than the other options.

However, even if these options are available I find the main function
approach cleanest.

Why does my program produce strange results?

Here is an example program that can produce surprising results (start a fresh
Julia session):

julia> l = -100
-100

julia> function f()
       x = 1
       for i in l:3
           x += i
       end
       return x
       end
f (generic function with 1 method)

julia> f()
-5043

The problem I show here is standard trap, I have replaced 1 with l in the
for loop specification.

The issue is that if you have more than a few hundreds of lines of code it is
easy to forget what variable names you have already defined in global scope.
Then, if you mistype something in other part of the code, instead of an error
you silently get a wrong result. I have, unfortunately, fallen victim of this
trap so many times that I started to call it “forgotten variable” problem.

The risk of such problem is significantly lower if you wrap all your code in
functions (and preferably the body of the function should be short). In such
cases you most likely will remember all variables that are defined in each
function.

Why does my program change its behavior when I run it as a script?

Save the code we have used above for analyzing for loop performance in
the example1.jl file. The contents of the file should be:

s = 0
@time for i in 1:10^8
    s += i
end
println(s)

Now I try running this code in a script:

$ julia example1.jl
┌ Warning: Assignment to `s` in soft scope is ambiguous because a global
variable by the same name exists: `s` will be treated as a new local.
Disambiguate by using `local s` to suppress this warning or `global s`
to assign to the existing global variable.
└ @ example1.jl:3
ERROR: LoadError: UndefVarError: s not defined

As you can see, because of Julia’s scoping rules, the code that worked in
interactive mode stopped working in script mode. This is bad, but at least
your code threw an error.

Let us try the following code:

x = 0
for i in 1:10
    x = i
end
@show x

First run it in a fresh interactive session:

julia> x = 0
0

julia> for i in 1:10
           x = i
       end

julia> @show x
x = 10
10

Now save this code as example2.jl file and run it as a script:

$ julia example2.jl
┌ Warning: Assignment to `x` in soft scope is ambiguous because a global
variable by the same name exists: `x` will be treated as a new local.
Disambiguate by using `local x` to suppress this warning or `global x`
to assign to the existing global variable.
└ @ example2.jl:3
x = 0

Now the code gave a warning (which is easy to ignore since it is complex, and
even might get lost in a flood of different outputs that the script produced),
but produced a result. The problem is that the result is different than the
one produced in an interactive session.

Conclusions

The problems I described in my post today are well known and most developers
remember about them when writing programs. However, in data science projects one
starts ones work in an interactive session, and only later turns the code into a
script. In such cases it is tempting to keep the code written in invractive mode
unchanged. The problems I have described today show why you should resist this
temptation and refactor your code to use functions in such a case.

What is important to keep in mind that while “forgotten variable” problem is
present in many programming languages, the performance and variable scoping
traps are Julia specific. The reason why performance trap is not so relevant
for R or Python users is that these languages are not compiled so for loops
are slow even when defined inside functions. Similarly, the scoping rule
of how global variables are treated in local soft scope is only an issue
in Julia. (If you are unsure what local soft scope is please refer to the
section on Local Scope in the Julia Manual.)

In summary, when working with Julia, you need to switch your mindset much
earlier from interactive mode to script mode than you would have to in R or
Python.