Category Archives: Julia

Share what you've learned!

Re-posted from: https://opensourc.es/blog/2021-03-01-share-what-youve-learned/index.html

Share which tool, workflow, package helped you out with the rest of the community

How to check the version of a package?

Re-posted from: https://bkamins.github.io/julialang/2021/02/27/pkg_version.html

Introduction

I think the most common type of question related to DataFrames.jl package is
that users are reporting that some functionality does not work as documented.

Sometimes it is indeed a bug but in the majority of cases the reason is that
the user does not have a correct version of the package installed. In this post
I discuss several ways of checking the version of the package one has in
a current project environment.

This post was written under Julia 1.6.0-rc1, DataFrames.jl 0.22.5 and Chain.jl
0.4.4 (so as usual you can expect some exercises in using DataFrames.jl).

Elementary methods

If you are in an interactive mode then you have two basic options. The first one
uses package manager mode. Press ] in Julia REPL and then write the following:

(bkamins) pkg> status
      Status `~/Project.toml`
  [4c9194b5] ABCDGraphGenerator v0.1.0 `https://github.com/bkamins/ABCDGraphGenerator.jl#master`
  [8be319e6] Chain v0.4.4
  [9a962f9c] DataAPI v1.6.0 `~/.julia/dev/DataAPI`
  [a93c6f00] DataFrames v0.22.5

or if you are interested in a particular package do:

(bkamins) pkg> status DataFrames
      Status `~/Project.toml`
  [a93c6f00] DataFrames v0.22.5

In the above output it is worth to note two common scenarios:

ABCDGraphGenerator.jl is tracking master branch of a GitHub repository as a
source (so it means it was not installed from Julia registry);
DataAPI.jl is checked out for development (using dev command) and the package
is tracking a local folder.

Alternatively we could have generated the same outputs using API like this:

julia> using Pkg

julia> Pkg.status()
      Status `~/Project.toml`
  [4c9194b5] ABCDGraphGenerator v0.1.0 `https://github.com/bkamins/ABCDGraphGenerator.jl#master`
  [8be319e6] Chain v0.4.4
  [9a962f9c] DataAPI v1.6.0 `~/.julia/dev/DataAPI`
  [a93c6f00] DataFrames v0.22.5

julia> Pkg.status("DataFrames")
      Status `~/Project.toml`
  [a93c6f00] DataFrames v0.22.5

The downside of both approaches is that they produce information to the screen.
However, often one is interested in processing programmatically the installed
packages status.

Working with `Pkg.dependencies`

The Pkg.dependencies function returns a dictionary mapping package UUIDs
to information about them. As you can check in the documentation string
of the function the available information is stored in the following fields:

name: the name of the package
version: the version of the package (this is Nothing for stdlibs)
is_direct_dep: the package is a direct dependency
is_tracking_path: whether a package is directly tracking a directory
is_pinned: whether a package is pinned
source: the directory containing the source code for that package
dependencies: the dependencies of that package as a vector of UUIDs

Using Pkg.dependencies we can easily write a function that returns a version
of the package. Here is an example:

julia> using Chain

julia> get_pkg_version(name::AbstractString) =
           @chain Pkg.dependencies() begin
               values
               [x for x in _ if x.name == name]
               only
               _.version
           end
get_pkg_version (generic function with 1 method)

julia> get_pkg_version("DataFrames")
v"0.22.5"

Here is another example getting summary statistics about installed packages in a
data frame:

julia> using DataFrames

julia> get_pkg_status(;direct::Bool=true) =
           @chain Pkg.dependencies() begin
               values
               DataFrame
               direct ? _[_.is_direct_dep, :] : _
               select(:name, :version,
                      [:is_tracking_path, :is_tracking_repo, :is_tracking_registry] =>
                      ByRow((a, b, c) -> ["path", "repo", "registry"][a+2b+3c]) =>
                      :tracking)
           end
get_pkg_status (generic function with 1 method)

julia> get_pkg_status()
4×3 DataFrame
 Row │ name                version  tracking
     │ String              Union…   String
─────┼───────────────────────────────────────
   1 │ DataAPI             1.6.0    path
   2 │ DataFrames          0.22.5   registry
   3 │ Chain               0.4.4    registry
   4 │ ABCDGraphGenerator  0.1.0    repo

As you see I have selected to provide only the most essential information
about packages in the output: name, version and whether package is tracking
registry, local path, or external repository.

If you would pass direct=false you get information about all available
packages (direct and indirect dependencies of the project). It is usually not
very useful, however, as the list tends to be long, as you can see here:

julia> get_pkg_status(direct=false)
62×3 DataFrame
 Row │ name                         version  tracking
     │ String                       Union…   String
─────┼────────────────────────────────────────────────
   1 │ OrderedCollections           1.4.0    registry
   2 │ LibSSH2_jll                           registry
   3 │ Statistics                            registry
   4 │ ArgTools                              registry
   5 │ Compat                       3.25.0   registry
   6 │ Reexport                     1.0.0    registry
   7 │ SharedArrays                          registry
  ⋮  │              ⋮                  ⋮        ⋮
  56 │ Dates                                 registry
  57 │ MbedTLS_jll                           registry
  58 │ Serialization                         registry
  59 │ IteratorInterfaceExtensions  1.0.0    registry
  60 │ Libdl                                 registry
  61 │ Artifacts                             registry
  62 │ InteractiveUtils                      registry
                                       48 rows omitted

Concluding remarks

I hope you might find these patterns useful in your work with the Julia language.

Before finishing, let me mention one other case that you might occasionally
need. The above examples show you the version of the package in your current
project environment. However, in one Julia session you can change active project
environment many times. If you would be interested in getting information about
a version of the currently loaded package here is the way to do it (this will not
work for packages from stdlib as they are bundled with Julia and have a fixed
version):

julia> Pkg.TOML.parsefile(joinpath(pkgdir(DataFrames), "Project.toml"))["version"]
"0.22.5"

Let us check that indeed the loaded version does not change if we change project
environment:

(bkamins) pkg> status DataFrames
      Status `~/Project.toml`
  [a93c6f00] DataFrames v0.22.5

(bkamins) pkg> add [email protected]
   Resolving package versions...
    Updating `~/Project.toml`
  [a93c6f00] ↓ DataFrames v0.22.5 ⇒ v0.21.8
    Updating `~/Manifest.toml`
  [324d7699] ↓ CategoricalArrays v0.9.3 ⇒ v0.8.3
  [a8cc5b0e] - Crayons v4.0.4
  [a93c6f00] ↓ DataFrames v0.22.5 ⇒ v0.21.8
  [59287772] - Formatting v0.4.2
  [2dfb63ee] ↓ PooledArrays v1.1.0 ⇒ v0.5.3
  [08abe8d2] - PrettyTables v0.11.1
  [189a3867] ↓ Reexport v1.0.0 ⇒ v0.2.0
  Progress [========================================>]  3/3
  ? DataFrames
2 dependencies successfully precompiled in 2 seconds (21 already precompiled)
1 dependency failed but may be precompilable after restarting julia

(bkamins) pkg> status DataFrames
      Status `~/Project.toml`
  [a93c6f00] DataFrames v0.21.8

julia> Pkg.TOML.parsefile(joinpath(pkgdir(DataFrames), "Project.toml"))["version"]
"0.22.5"

and we see that although project version of the package is changed the loaded
version remains the same.

How do loops in Julia handle local variables?

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2021/02/19/binding.html

Introduction

Today I decided to write about a feature of Julia that is well known to people
working with it a lot, but which often triggers questions from experienced
people switching to Julia from other languages.

The topic of this post is why this code fails:

julia> function f()
       for i in 0:3
           if i == 0
               v = 0
           else
               v += 1
           end
       end
       end
f (generic function with 1 method)

julia> f()
ERROR: UndefVarError: v not defined

and what to do to fix the problem.

The post was written under Julia 1.5.3.

The root cause

The reason why people ask this question is that e.g. in Python the following code
runs without any problem:

>>> def f():
...     for i in range(4):
...         if i == 0:
...             v = 0
...         else:
...             v += 1
...     return v
...
>>> f()
3

So what is the root cause of this difference? The reason is that in Julia
for loop creates a new scope for the variables that are not present in the
enclosing scope (i.e. variables local to the for loop do not leak out).

Moreover, as is explained here in the Julia manual such variables
get a new binding in each iteration of the loop. So in our example although we
set v = 0 in the first iteration this value is not retained in the following
iterations of the loop.

Fixing the problem

Fortunately it is easy to fix the problem in case you wanted v to behave
differently. Just define a local variable in the enclosing scope like this:

julia> function f()
       local v
       for i in 0:3
           if i == 0
               v = 0
           else
               v += 1
           end
       end
       return v
       end
f (generic function with 1 method)

julia> f()
3

Why do we need a fresh binding of loop-local variables in each iteration?

If you thought that what Julia did in the topmost example was strange then
consider which of the following examples you find surprising (I use
comprehensions this time).

This is Python:

>>> l = [lambda : i for i in range(4)]
>>> for i in range(4):
...     print(l[i]())
...
3
3
3
3

And this is Julia:

julia> l = [() -> i for i in 1:4];

julia> for i in 1:4
           println(l[i]())
       end
1
2
3
4

Sometimes things can get nasty

And what if we update a variable that is defined local outside the loop?

Here are two examples:

julia> function g1()
       l = []
       j = 0
       for i in 0:3
           push!(l, () -> j)
           j += 1
       end
       return l
       end
g (generic function with 1 method)

julia> l1 = g1();

julia> for i in 1:4
           println(l1[i]())
       end
4
4
4
4

julia> function g2()
       l = []
       j = 0
       for i in 0:3
           push!(l, () -> j)
           j = i + 1
       end
       return l
       end
g (generic function with 1 method)

julia> l2 = g2();

julia> for i in 1:4
           println(l2[i]())
       end
4
4
4
4

So far we see what we would expect. Variable j does not get a new binding
inside the loop as it is defined outside of it, so we have just reproduced the
behavior seen in Python.

However, how would you then explain this:

julia> function g3()
       x = []
       local j
       for i in 0:3
           j = i
           push!(x, () -> j)
       end
       return x
       end
g (generic function with 1 method)

julia> l3 = g3();

julia> for i in 1:4
           println(l3[i]())
       end
0
1
2
3

The reason is that in g1 and g2 Julia is boxing j, while it does not in
g3. Here are the consequences.

Consequence 1: impact on performance

Have a look at this test:

julia> function agg(fun)
       s = 0
       for i in 1:10^6
           s += fun()
       end
       return s
       end
agg (generic function with 1 method)

julia> agg(l1[1])
4000000

julia> @time agg(l1[1])
  0.038266 seconds (999.87 k allocations: 15.257 MiB)
4000000

julia> agg(l3[1])
0

julia> @time agg(l3[1])
  0.000007 seconds
0

And we see that closures created by g1 have a very bad performance, while
g3 gives us super fast closures.

Consequence 2: crazy things you can do

Since Julia is boxing j in the case of g1 and g2 you can do the following:

julia> for i in 1:4
           println(l1[i]())
       end
4
4
4
4

julia> l1[1].j.contents = 100
100

julia> for i in 1:4
           println(l1[i]())
       end
100
100
100
100

Of course I do not recommend doing such things. By this example I just highlight
that indeed j is boxed in this case.

Let us check:

julia> l1[1].j # boxed value
Core.Box(100)

julia> l3[1].j # just an Int
0

How we could have learned about this? You can use @code_warntype to see what
is going on:

julia> @code_warntype g1()
Variables
  #self#::Core.Compiler.Const(g1, false)
  l::Array{Any,1}
  j@_3::Core.Box
  @_4::Union{Nothing, Tuple{Int64,Int64}}
  i::Int64
  #15::var"#15#16"
  j@_7::Union{}

Body::Array{Any,1}
1 ─       (j@_3 = Core.Box())
│         (l = Base.vect())
│         Core.setfield!(j@_3, :contents, 0)
│   %4  = (0:3)::Core.Compiler.Const(0:3, false)
│         (@_4 = Base.iterate(%4))
│   %6  = (@_4::Core.Compiler.Const((0, 0), false) === nothing)::Core.Compiler.Const(false, false)
│   %7  = Base.not_int(%6)::Core.Compiler.Const(true, false)
└──       goto #7 if not %7
2 ┄ %9  = @_4::Tuple{Int64,Int64}::Tuple{Int64,Int64}
│         (i = Core.getfield(%9, 1))
│   %11 = Core.getfield(%9, 2)::Int64
│   %12 = l::Array{Any,1}
│         (#15 = %new(Main.:(var"#15#16"), j@_3))
│   %14 = #15::var"#15#16"
│         Main.push!(%12, %14)
│   %16 = Core.isdefined(j@_3, :contents)::Bool
└──       goto #4 if not %16
3 ─       goto #5
4 ─       Core.NewvarNode(:(j@_7))
└──       j@_7
5 ┄ %21 = Core.getfield(j@_3, :contents)::Any
│   %22 = (%21 + 1)::Any
│         Core.setfield!(j@_3, :contents, %22)
│         (@_4 = Base.iterate(%4, %11))
│   %25 = (@_4 === nothing)::Bool
│   %26 = Base.not_int(%25)::Bool
└──       goto #7 if not %26
6 ─       goto #2
7 ┄       return l

julia> @code_warntype g3()
Variables
  #self#::Core.Compiler.Const(g3, false)
  j::Int64
  x::Array{Any,1}
  @_4::Union{Nothing, Tuple{Int64,Int64}}
  i::Int64
  #19::var"#19#20"{Int64}

Body::Array{Any,1}
1 ─       Core.NewvarNode(:(j))
│         (x = Base.vect())
│   %3  = (0:3)::Core.Compiler.Const(0:3, false)
│         (@_4 = Base.iterate(%3))
│   %5  = (@_4::Core.Compiler.Const((0, 0), false) === nothing)::Core.Compiler.Const(false, false)
│   %6  = Base.not_int(%5)::Core.Compiler.Const(true, false)
└──       goto #4 if not %6
2 ┄ %8  = @_4::Tuple{Int64,Int64}::Tuple{Int64,Int64}
│         (i = Core.getfield(%8, 1))
│   %10 = Core.getfield(%8, 2)::Int64
│         (j = i)
│   %12 = x::Array{Any,1}
│   %13 = Main.:(var"#19#20")::Core.Compiler.Const(var"#19#20", false)
│   %14 = Core.typeof(j)::Core.Compiler.Const(Int64, false)
│   %15 = Core.apply_type(%13, %14)::Core.Compiler.Const(var"#19#20"{Int64}, false)
│         (#19 = %new(%15, j))
│   %17 = #19::var"#19#20"{Int64}
│         Main.push!(%12, %17)
│         (@_4 = Base.iterate(%3, %10))
│   %20 = (@_4 === nothing)::Bool
│   %21 = Base.not_int(%20)::Bool
└──       goto #4 if not %21
3 ─       goto #2
4 ┄       return x

Conclusions

The post has started-off easy, but ended with some surprising behavior.
I hope you found it useful to better understand how Julia works and how to
diagnose things.

The major take aways are:

basic: Julia creates new bindings for loop-local variables on each iteration;
not-basic: if you are creating closures using local variables and need them to
be fast (and you probably do if you use Julia) always check if Julia compiler
was able to prove that it does not have to do boxing as it affects both the
behavior and the performance.

juliabloggers.com

A Julia Language Blog Aggregator

Category Archives: Julia

Share what you've learned!

How to check the version of a package?

Introduction

Elementary methods

Working with `Pkg.dependencies`

Concluding remarks

How do loops in Julia handle local variables?

Introduction

The root cause

Fixing the problem

Why do we need a fresh binding of loop-local variables in each iteration?

Sometimes things can get nasty

Consequence 1: impact on performance

Consequence 2: crazy things you can do

Conclusions

Introduction

Elementary methods

Working with Pkg.dependencies

Concluding remarks

Introduction

The root cause

Fixing the problem

Why do we need a fresh binding of loop-local variables in each iteration?

Sometimes things can get nasty

Consequence 1: impact on performance

Consequence 2: crazy things you can do

Conclusions

Working with `Pkg.dependencies`