By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/06/17/missing.html

Introduction

Some time ago I have written a post about
ABC of handling missing values in Julia. Its objective was to give
an introduction to the topic for the newcomers. However, occasionally
users complain that working with missing values in Julia is less convenient
than in e.g. Python or R.

Such opinions are always debatable, so recently I decided to run a small
pool on Julia Discourse about the skipmissing function.
The question was if we want to shorten the skipmissing name into something
that is more convenient to use in interactive work.
To my surprise, a vast majority of voters preferred a verbose and explicit
operation name. This preference regarding handling of missings, to my surprise,
reminded me of several passages from The Zen of Python:

• Explicit is better than implicit.
• Readability counts.
• In the face of ambiguity, refuse the temptation to guess.

So given this preference, how should Julia users retain convenience. Let me
share some of my thoughts on this topic that did not make into my
previous post.

This post was written under Julia 1.7.2, Missings.jl 1.0.2,
MissingsAsFalse.jl 0.1, and StatsBase.jl 0.33.16.

Verbosity

Indeed writing missing, skipmissing, and passmissing (the last one is
defined in Missings.jl), in places where they are needed, might seem verbose.
However, the good news is that most of the time you do not have to type them
fully thanks to completions so in your editor/REPL:

• instead of missing write mis<tab>;
• instead of skipmissing write skipm<tab>;
• instead of passmissing write pas<tab>.

Let us compare. In R:

sum(c(1, NA), na.rm=T)

vs Julia:

sum(skipmissing([1, missing]))

seems longer. However, if you take into account the amount of typing you need to
do (number of keystrokes) it is the same.

Missing values in logical conditions

As I have written in this post I personally strongly recommend using
coalesce to handle missing values in logical conditions. This allows you,
to follow the Explicit is better than implicit. principle by explicitly
showing in the code if missing should be treated as true or as false.

Here is a short example:

julia> c = missing
missing

julia> c ? "true" : "false"
ERROR: TypeError: non-boolean (Missing) used in boolean context

julia> coalesce(c, false) ? "true" : "false"
"false"

However, some users find it more convenient to use @mfalse macro from
the MissingsAsFalse.jl package:

julia> using MissingsAsFalse

julia> @mfalse c ? "true" : "false"
"false"

Correlation matrix with missing values

A common, and relatively complex case of handling missing values, is computing
of correlation matrix of data that contains missings. Let us check what Julia
offers here. We will use the pairwise function from StatsBase.jl.

julia> using Random

julia> using StatsBase

julia> using Statistics

julia> Random.seed!(1234);

julia> x = rand([1:10; missing], 16, 4)
16×4 Matrix{Union{Missing, Int64}}:
  4          missing   2         10
  7         8           missing   8
  3          missing   7          5
 10         9          8         10
  4         2          7          6
  5         6          1           missing
   missing  7          8          9
  9         3          2          6
  6         7         10           missing
  9         3          4          6
  7         2          9          8
  9         7          3          3
  1         8          6          5
  3         9          4          7
  5         7          3          2
  8         5           missing   4

julia> pairwise(cor, eachcol(x))
4×4 Matrix{Union{Missing, Float64}}:
 1.0        missing   missing   missing
  missing  1.0        missing   missing
  missing   missing  1.0        missing
  missing   missing   missing  1.0

julia> pairwise(cor, eachcol(x), skipmissing=:pairwise)
4×4 Matrix{Float64}:
  1.0        -0.218849    -0.0177704   0.0922413
 -0.218849    1.0          0.00973122  0.102969
 -0.0177704   0.00973122   1.0         0.364821
  0.0922413   0.102969     0.364821    1.0

julia> pairwise(cor, eachcol(x), skipmissing=:listwise)
4×4 Matrix{Float64}:
  1.0       -0.229568   -0.100028   0.247283
 -0.229568   1.0        -0.127153  -0.0420058
 -0.100028  -0.127153    1.0        0.67068
  0.247283  -0.0420058   0.67068    1.0

By default pairwise for cor returns missing when at least one of the
columns contains missing values. Use :pairwise value of skipmissing keyword
argument to skip entries with a missing value in either of the two vectors
passed to cor and use :listwise to skip entries with a missing value in
any of the vectors passed to pairwise.

In this example we see that since there are several ways how missing values
should be handled by cor in pairwise computations, instead of using the
skipmissing function, a keyword argument is used allowing to specify what
the data scientist wants exactly.

A similar situation is in the subset and subset! functions from
DataFrames.jl, that also take a skipmissing keyword argument, to simplify
handling of logical conditions specifying which rows from a source data frame
should be kept.

Conclusions

In Julia the approach is that handling of missing values is explicit. This
choice is guided by the fact that then in the code it is explicitly visible what
was the developer’s decision about how they should be treated. This choice makes
code more verbose. Today I have tried to show that, especially with a good
editor/REPL support, in practice it does not introduce a large overhead.

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/06/10/view.html

Introduction

Macros in Julia are a very nice element of the language. You can easily visually
identify macros as they are called using the @ prefix. I most often use the
@assert, @show, @spawn, @edit, and @time macros. However, one needs to
understand how macros work to confidently use them.

Today I want to write about typical situations when using
the @view and @views macros to create views can lead to surprising results.

The post was tested under Julia 1.7.2.

The @view macro and expression range

Assume we have a vector representing 24 months of data and we want to compute
correlation between the first 12 observations and the last 12 observations.

Here is a simple way to do it:

julia> using Random

julia> using Statistics

julia> Random.seed!(1234);

julia> x = randn(24);

julia> cor(x[1:12], x[13:24])
-0.018264675865149734

However, this operation copies data. If we want to avoid this we can use
views. To start let us check the view function:

julia> cor(view(x, 1:12), view(x, 13:24))
-0.018264675865149734

Let us check if indeed this reduces allocations first (using the @allocated
macro):

julia> @allocated cor(x[1:12], x[13:24])
336

julia> @allocated cor(view(x, 1:12), view(x, 13:24))
112

Indeed it does. Of course in our toy example the benefit is minimal.

Now we are ready to get to the main point of my post. Suppose we have
cor(x[1:12], x[13:24]) and want to do use views. As you can see in
codes above turning indexing into view call requires re-writing of the
expressions. This is where the @view macro comes handy.

julia> cor(@view x[1:12], @view x[13:24])
ERROR: LoadError: ArgumentError: Invalid use of @view macro: argument must be a reference expression A[...].

Or does it? We have some problem when using the macro. Let us check the
Julia Manual:

Macros are invoked with the following general syntax:

@name expr1 expr2 ... or @name(expr1, expr2, ...)

Since we invoked @view using the first style Julia eagerly considers
everything that follows it as a single expression. In this case the whole
x[1:12], @view x[13:24] part of code is passed to @view as a single
expression and we get an error.

We need to use the second macro invocation style in this case:

julia> cor(@view(x[1:12]), @view(x[13:24]))
-0.018264675865149734

Now all worked as expected. The only downside is that it feels a bit
inconvenient to write @view(...). Fortunately, Julia’s creators have thought
about it and designed a @views macro which turns every array slicing
(i.e., array[...]) operation in a passed expression to a view. In our case
this would be:

julia> @views cor(x[1:12], x[13:24])
-0.018264675865149734

The @views macro surprises

Let us check using the @macroexpand macro if indeed @views works as I promised:

julia> @macroexpand @views cor(x[1:12], x[13:24])
:(cor((Base.maybeview)(x, 1:12), (Base.maybeview)(x, 13:24)))

What is this strange Base.maybeview function? Is like getindex, but returns
a view for array slicing operations, while remaining equivalent to getindex
for scalar indices and non-array types. So it is almost the same as view.

Let us see the difference between using @view and @views:

julia> @view x[1]
0-dimensional view(::Vector{Float64}, 1) with eltype Float64:
0.9706563288552144

julia> @views x[1]
0.9706563288552144

Most of the time using a 0-dimensional view or a scalar will not make a
difference, however, sometimes it does. Let us have a look:

julia> similar(@view x[1])
0-dimensional Array{Float64, 0}:
1.40721121e-315

julia> similar(@views x[1])
ERROR: MethodError: no method matching similar(::Float64)

So things are, unfortunately, not as simple as you might expect, especially if
you are writing generic code and do not know upfront if you will use a scalar
index or not.

Having learned what I have written above you might think that the following code
will not work:

julia> x, y, z = [1, 2, 3], [2, 3, 4], [4, 5, 6]
([1, 2, 3], [2, 3, 4], [4, 5, 6])

julia> @views x[1] = y[1] + z[1]
6

julia> x
3-element Vector{Int64}:
 6
 2
 3

However, it produces a correct result. What is the reason? Let us check:

julia> @macroexpand @views x[1] = y[1] + z[1]
:(x[1] = (Base.maybeview)(y, 1) + (Base.maybeview)(z, 1))

As we can see @views is smart enough not to apply the view transformation
to the left hand side of the assignment. This feature is not documented in its
docstring, but can be found (and is even commented about) in the source code of
the @views macro (in the _views function to be precise).

Interestingly, this rule is in play even in the example given in
the docsting of the @views macro. Let us check it:

julia> A = zeros(3, 3);

julia> @macroexpand @views for row in 1:3
                    b = A[row, :]
                    b[:] .= row
                end
:(for row = 1:3
      #= REPL[67]:2 =#
      b = (Base.maybeview)(A, row, :)
      #= REPL[67]:3 =#
      b[:] .= row
  end)

We can see that since b[:] is on left hand side of the assignment it is not
touched by @views.

Conclusions

What are the lessons learned?

1. Macros in Julia are powerful. Well designed macros can make developer’s
   life easier and code more readable.
2. Macros can be tricky. The most common problem when using macros is the
   @name expr1 style of invocation, which can process “too much” of your code
   unexpectedly.
3. @views is not equivalent to multiple invocations of @view. There are
   subtle differences between them. Fortunately these differences matter
   mostly in generic code and thus package developers need to be aware of them.

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/06/10/view.html

Introduction

Macros in Julia are a very nice element of the language. You can easily visually
identify macros as they are called using the @ prefix. I most often use the
@assert, @show, @spawn, @edit, and @time macros. However, one needs to
understand how macros work to confidently use them.

Today I want to write about typical situations when using
the @view and @views macros to create views can lead to surprising results.

The post was tested under Julia 1.7.2.

The @view macro and expression range

Assume we have a vector representing 24 months of data and we want to compute
correlation between the first 12 observations and the last 12 observations.

Here is a simple way to do it:

julia> using Random

julia> using Statistics

julia> Random.seed!(1234);

julia> x = randn(24);

julia> cor(x[1:12], x[13:24])
-0.018264675865149734

However, this operation copies data. If we want to avoid this we can use
views. To start let us check the view function:

julia> cor(view(x, 1:12), view(x, 13:24))
-0.018264675865149734

Let us check if indeed this reduces allocations first (using the @allocated
macro):

julia> @allocated cor(x[1:12], x[13:24])
336

julia> @allocated cor(view(x, 1:12), view(x, 13:24))
112

Indeed it does. Of course in our toy example the benefit is minimal.

Now we are ready to get to the main point of my post. Suppose we have
cor(x[1:12], x[13:24]) and want to do use views. As you can see in
codes above turning indexing into view call requires re-writing of the
expressions. This is where the @view macro comes handy.

julia> cor(@view x[1:12], @view x[13:24])
ERROR: LoadError: ArgumentError: Invalid use of @view macro: argument must be a reference expression A[...].

Or does it? We have some problem when using the macro. Let us check the
Julia Manual:

Macros are invoked with the following general syntax:

@name expr1 expr2 ... or @name(expr1, expr2, ...)

Since we invoked @view using the first style Julia eagerly considers
everything that follows it as a single expression. In this case the whole
x[1:12], @view x[13:24] part of code is passed to @view as a single
expression and we get an error.

We need to use the second macro invocation style in this case:

julia> cor(@view(x[1:12]), @view(x[13:24]))
-0.018264675865149734

Now all worked as expected. The only downside is that it feels a bit
inconvenient to write @view(...). Fortunately, Julia’s creators have thought
about it and designed a @views macro which turns every array slicing
(i.e., array[...]) operation in a passed expression to a view. In our case
this would be:

julia> @views cor(x[1:12], x[13:24])
-0.018264675865149734

The @views macro surprises

Let us check using the @macroexpand macro if indeed @views works as I promised:

julia> @macroexpand @views cor(x[1:12], x[13:24])
:(cor((Base.maybeview)(x, 1:12), (Base.maybeview)(x, 13:24)))

What is this strange Base.maybeview function? Is like getindex, but returns
a view for array slicing operations, while remaining equivalent to getindex
for scalar indices and non-array types. So it is almost the same as view.

Let us see the difference between using @view and @views:

julia> @view x[1]
0-dimensional view(::Vector{Float64}, 1) with eltype Float64:
0.9706563288552144

julia> @views x[1]
0.9706563288552144

Most of the time using a 0-dimensional view or a scalar will not make a
difference, however, sometimes it does. Let us have a look:

julia> similar(@view x[1])
0-dimensional Array{Float64, 0}:
1.40721121e-315

julia> similar(@views x[1])
ERROR: MethodError: no method matching similar(::Float64)

So things are, unfortunately, not as simple as you might expect, especially if
you are writing generic code and do not know upfront if you will use a scalar
index or not.

Having learned what I have written above you might think that the following code
will not work:

julia> x, y, z = [1, 2, 3], [2, 3, 4], [4, 5, 6]
([1, 2, 3], [2, 3, 4], [4, 5, 6])

julia> @views x[1] = y[1] + z[1]
6

julia> x
3-element Vector{Int64}:
 6
 2
 3

However, it produces a correct result. What is the reason? Let us check:

julia> @macroexpand @views x[1] = y[1] + z[1]
:(x[1] = (Base.maybeview)(y, 1) + (Base.maybeview)(z, 1))

As we can see @views is smart enough not to apply the view transformation
to the left hand side of the assignment. This feature is not documented in its
docstring, but can be found (and is even commented about) in the source code of
the @views macro (in the _views function to be precise).

Interestingly, this rule is in play even in the example given in
the docsting of the @views macro. Let us check it:

julia> A = zeros(3, 3);

julia> @macroexpand @views for row in 1:3
                    b = A[row, :]
                    b[:] .= row
                end
:(for row = 1:3
      #= REPL[67]:2 =#
      b = (Base.maybeview)(A, row, :)
      #= REPL[67]:3 =#
      b[:] .= row
  end)

We can see that since b[:] is on left hand side of the assignment it is not
touched by @views.

Conclusions

What are the lessons learned?

1. Macros in Julia are powerful. Well designed macros can make developer’s
   life easier and code more readable.
2. Macros can be tricky. The most common problem when using macros is the
   @name expr1 style of invocation, which can process “too much” of your code
   unexpectedly.
3. @views is not equivalent to multiple invocations of @view. There are
   subtle differences between them. Fortunately these differences matter
   mostly in generic code and thus package developers need to be aware of them.