Category Archives: Julia

Poor man’s guide to despecialization

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2021/04/02/despecialization.html

Introduction

We are just about to release DataFrames.jl 1.0. It will bring many new
features we are excited about. One of the major additions is support of multiple
threading in many core operations. These changes promise performance
improvements when processing large data frames. The cost is that code complexity
has also grown significantly.

Clearly code-base size is an issue for maintainers of the package, but should
end-users care? The answer is that unfortunately yes, as making code more
complex makes it compile longer.

In this post I summarize the experience we have when trying to reduce compile
time latency.

TLDR: if you have functions that are expensive to compile but are not
performance critical perform argument standardization.

In this post I am using Julia 1.6, MethodAnalysis v0.4.4, SnoopCompile v2.6.0,
and DataFrames.jl main branch state at this commit (in this case it
is relevant as changes in the code base will likely affect the results).

Before we start our experiments disable precompilation (to have a clean ground
for comparisons and simplify things; the conclusions I present hold also when
proper precompilation statements are added). To do so comment-out lines 152 and
153 in src/DataFrames.jl file:

#include("other/precompile.jl")
#precompile()

The context

In DataFrames.jl when one performs transformation of data the following two
areas cause challenges related to run-time compilation:

  • users can pass arbitrary functions as transformations;
  • the ouptut of these functions can be arbitrary values (and the logic of
    processing them depends on their type).

Let us have a look at a simple example:

julia> using DataFrames

julia> gdf = groupby(DataFrame(a=1), :a);

julia> @time combine(gdf, x -> (a=1,));
  2.440215 seconds (8.09 M allocations: 469.718 MiB, 8.16% gc time)

julia> @time combine(gdf, x -> (a=1,));
  0.083986 seconds (550.13 k allocations: 33.481 MiB, 99.73% compilation time)

julia> @time combine(gdf, x -> (b=1,));
  0.364469 seconds (996.17 k allocations: 61.213 MiB, 4.25% gc time, 99.74% compilation time)

julia> @time combine(gdf, x -> (c=1,));
  0.337301 seconds (983.31 k allocations: 60.455 MiB, 1.91% gc time, 99.72% compilation time)

We can see that in each call we pass a new anonymous function to combine.
Additionally, the return value of this function is a NamedTuple that has a
fresh type each time, as the name of the column changes.

As you can see the compilation time in these examples is quite high.

Let us check how to reduce it. We will concentrate on only one method
_combine_multicol that is defined in line 7 of
src/groupeddataframe/complextransforms.jl. Its signature is:

_combine_multicol(firstres, fun::Base.Callable, gd::GroupedDataFrame,
                  incols::Union{Nothing, AbstractVector, Tuple, NamedTuple})

We start witch checking how many method instances were generated for it in our code:

julia> using MethodAnalysis

julia> methodinstances(DataFrames._combine_multicol)
9-element Vector{Core.MethodInstance}:
 MethodInstance for _combine_multicol(::NamedTuple{(:a,), Tuple{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:a,), Tuple{Int64}}, ::var"#1#2", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Type, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:a,), Tuple{Int64}}, ::var"#3#4", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:b,), Tuple{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:b,), Tuple{Int64}}, ::var"#5#6", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:c,), Tuple{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:c,), Tuple{Int64}}, ::var"#7#8", ::GroupedDataFrame{DataFrame}, ::Nothing)

As you can see each call like combine(gdf, x -> (c=1,)) generates two new method instances.

Before we move forward let us check how this code would be run in 0.22 release
(precompilation is turned on here so we are not comparing apples to apples,
but if we disabled precompilation the conclusion would be similar):

julia> using DataFrames

julia> gdf = groupby(DataFrame(a=1), :a);

julia> @time combine(gdf, x -> (a=1,));
  1.412728 seconds (2.69 M allocations: 167.339 MiB, 3.50% gc time, 45.87% compilation time)

julia> @time combine(gdf, x -> (a=1,));
  0.041718 seconds (190.24 k allocations: 11.522 MiB, 20.74% gc time, 99.03% compilation time)

julia> @time combine(gdf, x -> (b=1,));
  0.209238 seconds (481.99 k allocations: 30.126 MiB, 99.58% compilation time)

julia> @time combine(gdf, x -> (c=1,));
  0.194280 seconds (468.45 k allocations: 29.367 MiB, 5.33% gc time, 99.53% compilation time)

julia> using MethodAnalysis

julia> methodinstances(DataFrames._combine_multicol)
13-element Vector{Core.MethodInstance}:
 MethodInstance for _combine_multicol(::DataFrame, ::Function, ::GroupedDataFrame{DataFrame}, ::Tuple{Vector{Bool}})
 MethodInstance for _combine_multicol(::DataFrame, ::Type, ::GroupedDataFrame{DataFrame}, ::Tuple{Vector{Bool}})
 MethodInstance for _combine_multicol(::Any, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Type, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:a,), Tuple{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:b,), Tuple{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:c,), Tuple{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Type, ::GroupedDataFrame{DataFrame}, ::NamedTuple)
 MethodInstance for _combine_multicol(::Any, ::Function, ::GroupedDataFrame{DataFrame}, ::NamedTuple)
 MethodInstance for _combine_multicol(::Any, ::Type, ::GroupedDataFrame{DataFrame}, ::Tuple)
 MethodInstance for _combine_multicol(::Any, ::Function, ::GroupedDataFrame{DataFrame}, ::Tuple)
 MethodInstance for _combine_multicol(::NamedTuple, ::Type, ::GroupedDataFrame{DataFrame}, ::Tuple{Vector{Bool}})
 MethodInstance for _combine_multicol(::NamedTuple, ::Function, ::GroupedDataFrame{DataFrame}, ::Tuple{Vector{Bool}})

So we can see that DataFrames.jl 0.22 produced even more instances, but since the
code was simpler the compilation time was lower.

Despecialization

The first advice one gets in such cases is to use @nospecialize on function
arguments to avoid excessive specialization. In our case we see that problematic
are firstres and fun arguments. Now exit Julia and change the signature of
the method to:

_combine_multicol(@nospecialize(firstres), @nospecialize(fun::Base.Callable),
                  gd::GroupedDataFrame,
                  incols::Union{Nothing, AbstractVector, Tuple, NamedTuple})

Now start your Julia again and run the code we have checked above:

julia> using DataFrames

julia> gdf = groupby(DataFrame(a=1), :a);

julia> @time combine(gdf, x -> (a=1,));
  2.238896 seconds (8.10 M allocations: 470.113 MiB, 5.34% gc time)

julia> @time combine(gdf, x -> (a=1,));
  0.095581 seconds (550.16 k allocations: 33.482 MiB, 9.94% gc time, 99.75% compilation time)

julia> @time combine(gdf, x -> (b=1,));
  0.304849 seconds (982.79 k allocations: 60.359 MiB, 2.95% gc time, 99.71% compilation time)

julia> @time combine(gdf, x -> (c=1,));
  0.297039 seconds (969.93 k allocations: 59.620 MiB, 6.12% gc time, 99.72% compilation time)

julia> using MethodAnalysis

julia> methodinstances(DataFrames._combine_multicol)
7-element Vector{Core.MethodInstance}:
 MethodInstance for _combine_multicol(::NamedTuple{(:a,), Tuple{Int64}}, ::var"#1#2", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Function, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Type, ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:a,), Tuple{Int64}}, ::var"#3#4", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:b,), Tuple{Int64}}, ::var"#5#6", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::NamedTuple{(:c,), Tuple{Int64}}, ::var"#7#8", ::GroupedDataFrame{DataFrame}, ::Nothing)
 MethodInstance for _combine_multicol(::Any, ::Union{Function, Type}, ::GroupedDataFrame{DataFrame}, ::Nothing)

As you can see things got slightly better. GC time is distorting the comparison
a bit, but correcting for it we see an improvement. Also we have reduced the
number of method instances from 9 to 7. But wait – we wanted to disable specialization,
and we still get 7 method instances to be compiled. How to reduce it?

Poor man’s despecialization

The approach I found to work reliably is to aggresively perform
argument standardization. A standard trick to be sure that we break
method specialization for some value is to wrap it in Ref{Any}, and then
immediately unwrap.

Let us try it. Now we change our method signature to:

_combine_multicol((firstres,)::Ref{Any}, (fun,)::Ref{Any},
                  gd::GroupedDataFrame,
                  incols::Union{Nothing, AbstractVector, Tuple, NamedTuple})

we also need to change two lines of code where _combine_multicol is called,
namely lines 268:

idx, outcols, nms = _combine_multicol(Ref{Any}(firstres), Ref{Any}(cs_i), gd, nothing)

and 395:

idx, outcols, nms = _combine_multicol(Ref{Any}(firstres), Ref{Any}(fun), gd, incols)

in file src/groupeddataframe/splitapplycombine.jl.

Apply the changes and run our code in a fresh Julia session again:

julia> using DataFrames

julia> gdf = groupby(DataFrame(a=1), :a);

julia> @time combine(gdf, x -> (a=1,));
  2.262996 seconds (7.70 M allocations: 445.018 MiB, 8.53% gc time)

julia> @time combine(gdf, x -> (a=1,));
  0.047010 seconds (299.33 k allocations: 18.038 MiB, 99.52% compilation time)

julia> @time combine(gdf, x -> (b=1,));
  0.278649 seconds (733.47 k allocations: 45.020 MiB, 2.44% gc time, 99.62% compilation time)

julia> @time combine(gdf, x -> (c=1,));
  0.245550 seconds (720.62 k allocations: 44.284 MiB, 2.73% gc time, 99.61% compilation time)

julia> using MethodAnalysis

julia> methodinstances(DataFrames._combine_multicol)
1-element Vector{Core.MethodInstance}:
 MethodInstance for _combine_multicol(::Base.RefValue{Any}, ::Base.RefValue{Any}, ::GroupedDataFrame{DataFrame}, ::Nothing)

Now that is much better. We compiled only one method instance for
_combine_multicol and the timings have significantly improved.

Conclusions

Let us now check what timings the above code has after applying such techniques
to all relevant functions in source code, as proposed in this PR.
I still keep precompilation disabled so the comparison is again a bit unfair
in reference to 0.22 release:

julia> using DataFrames

julia> gdf = groupby(DataFrame(a=1), :a);

julia> @time combine(gdf, x -> (a=1,));
  2.478645 seconds (8.43 M allocations: 489.011 MiB, 9.96% gc time)

julia> @time combine(gdf, x -> (a=1,));
  0.005444 seconds (7.43 k allocations: 486.211 KiB, 96.44% compilation time)

julia> @time combine(gdf, x -> (b=1,));
  0.199044 seconds (371.68 k allocations: 22.773 MiB, 99.39% compilation time)

julia> @time combine(gdf, x -> (c=1,));
  0.173116 seconds (358.82 k allocations: 22.033 MiB, 4.05% gc time, 99.50% compilation time)

julia> using MethodAnalysis

julia> methodinstances(DataFrames._combine_multicol)
1-element Vector{Core.MethodInstance}:
 MethodInstance for _combine_multicol(::Base.RefValue{Any}, ::Base.RefValue{Any}, ::GroupedDataFrame{DataFrame}, ::Base.RefValue{Any})

And now we are under 0.22 timings. Also note that the whole compilation cost
is now related to generation of methods related to the new type of the return
value (the NamedTuple issue). Let us check:

julia> using SnoopCompile

julia> t = @snoopi_deep combine(gdf, x -> (d=1,));

julia> sort(SnoopCompile.flatten(t), by = x -> x.exclusive_time)
271-element Vector{SnoopCompileCore.InferenceTiming}:
 InferenceTiming: 0.000019/0.000019 on InferenceFrameInfo for convert(::Type{Int64}, 1::Int64)
 InferenceTiming: 0.000019/0.000019 on InferenceFrameInfo for convert(::Type{Int64}, 0::Int64)
 InferenceTiming: 0.000020/0.000020 on InferenceFrameInfo for convert(::Type{Int64}, 2::Int64)
 ⋮
 InferenceTiming: 0.007164/0.022407 on InferenceFrameInfo for DataFrames._combine_rows_with_first!(::NamedTuple{(:d,), Tuple{Int64}}, ::Tuple{Vector{Int64}}, ::Function, ::GroupedDataFrame{DataFrame}, nothing::Nothing, ::Tuple{Symbol}, Val{true}()::Val{true})
 InferenceTiming: 0.096098/0.096305 on InferenceFrameInfo for map(::Type{Tuple}, ::Tuple{NamedTuple{(:d,), Tuple{Int64}}})
 InferenceTiming: 0.136345/0.279824 on InferenceFrameInfo for Core.Compiler.Timings.ROOT()

and we see that the biggest cost is paid by map function applied to NamedTuple,
which is a function in Julia Base.

In summary the approaches I discuss here are most useful when you expect to get
values of very heterogeneous types as arguments to your functions. In DataFrames.jl
the two most common cases of these situations are:

  • anonymous transformation functions;
  • NamedTuples as produced values from transformations.

If you would have any comments on the best strategies to avoid code
specialization please contact me as reducing compilation latency is one
of the priorities of DataFrames.jl 1.0 release.

Before I finish let me add one thing. If you are interested in true performance
guided tips to reduce compilation latency (not just poor man’s ones I have given
in this post) I highly recommend you to check out this, this, and
this post.

Forward model autodiff

By: Oisin Fitzgerald

Re-posted from: https://oizin.github.io/posts/autodiff-forward/index.html

Forward mode automatic differentiation

Oisín Fitzgerald, April 2021

A look at the first half (up to section 3.1) of:

Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18.

https://www.jmlr.org/papers/volume18/17-468/17-468.pdf

Introduction

Automatic differentiation (autodiff) reminds me of Arthur C. Clarke's quote "any sufficiently advanced technology is indistinguishable from magic". Whereas computer based symbolic and numerical differentiation seem like natural descendants from blackboard based calculus, the first time I learnt about autodiff (through Pytorch) I was amazed. It is not that the ideas underlying autodiff themselves are particularly complex, indeed Bayin et al's look at the history of autodiff puts Wengert's 1964 paper entitled "A simple automatic derivative evaluation program" as a key moment in forward mode autodiff history. (BTW the paper is only 2 pages long – well worth taking a look). For me the magic comes from autodiff being this digitally inspired look at something as "ordinary" but so important to scientific computing and AI as differentiation.

Differentiation

If you are unsure of what the terms symbolic or numerical differentiation mean, I'll give a a quick overview below but would encourage you to read the paper and it's references for a more detailed exposition of their various strengths and weaknesses.

Numeric differentiation – wiggle the input

For a function \(f\) with a 1D input and output describing numeric differentiation (also known as the finite difference method) comes quite naturally from the definition of the derivative. The derivative is

\[\frac{df}{dx} = \text{lim}_{h \rightarrow 0}\frac{f(x+h)-f(x)}{h}\]

so we approximate this expression by picking a small enough \(h\) (there are more complex schemes). There are two sources of error here, the first is from approximating the infinitesimally small \(h\) with a plain finitely small \(h\) (truncation error) and the second is from round-off error. Round-off error occurs because not every number is represented in the set of floating point numbers so for a small \(h\) the difference \(f(x+h)-f(x)\) can be quite unstable. Unfortunately these two source of error play against each other (see graph – on the left hand size round-off error dominates whereas on the right hand side truncation error dominates).

using Plots
plotlyjs()
h = 10 .^ range(-15, -3, length=1000)
x0 = 0.2
f(x) = (64*x*(1-x)*(1-2*x)^2)*(1-8*x+8*x^2)^2
df = (f.(x0 .+ h) .- f(x0)) ./ h
plot(log10.(h),log10.(abs.((df .- 9.0660864))),
xlabel="log10(h)",ylabel="log10(|Error|)",legend=false)

However, such small errors are actually not all that important in machine learning! The main issue with numeric differentiation for machine learning is that the number of required evaluations of our function \(f\) scales linearly with the number of dimension of the gradient. In contrast backpropagation (an autodiff method) can calculate the derivatives in "two" evaluations of our function (one forward, one back).

Symbolic – fancy lookup tables

Symbolic differentiation is differentiation as you learnt it in school programmed into software, all the rules \(\frac{d}{dx}\text{cos}(x) = -\text{sin}(x), \frac{d}{dx} x^p = px^{(p-1)}, \frac{d}{dx}f(g(x)) = f'(g(x))g'(x)\) etc… are known and utilised by the software. If you evaluate the derivative of a function f using a symbolic programming language dfdx = derivative(f,x) the object returned dfdx is just whatever function the symbolic program matches as the derivative of f using it's internal derivative lookup and application of the rules of differentiation (chain rule etc). It is manipulation of expressions. The main issue with symbolic differentiation for ML (which anyone who has used Mathematica for a help with a difficult problem can attest to) is expression swell, where the derivative expression is exponentially longer than the original expression and involves repeated calculations.

Automatic differentiation – alter the program

Autodiff is the augmentation of a computer program to perform standard computations along with calculation of derivatives, there is no manipulation of expressions. It takes advantage of the fact that derivative expressions can be broken down into elementary operations that can be combined to give the derivative of the overall expression. I'll be more clear about elementary operations soon but you can think of an elementary operations as being any operation you could give to a node on a computational graph of your program.

Forward mode

To be more concrete about autodiff, let's look at forward mode. Consider evaluating \(f(x_1,x_2) = x_1 x_2 + \text{log}(x_1 ^2)\). We break this into the computational graph below and associate with each elementary operation the intermediate variable \(\dot{v}_i = \frac{\partial v_i}{\partial x}\), called the "tangent". The final "tangent" value \(\dot{v}_5\), which has been calculated as the function evaluates at the input (3,5) is a derivative at the point (3,5). What derivative exactly depends on the initial values of \(\dot{x_1}\) and \(\dot{x_2}\).

Sketching a forward mode autodiff library

It's surprisingly easy to implement forward mode autodiff in Julia (at least a naive form). Below I create a forward model module that creates a new object Dual that is a type of Number, and then proceed to overload common mathematical functions (e.g. sin and *) to account for this new number type. Each instance of Dual with have a prime and tangent slot. If we want the derivative with respect to argument x₁ of the function y = f(x₁,x₂) we simply set x₁.t = 1.0 (leaving x₂.t = 0.0) and check the value of y.t. For more see this video from MIT's Alan Edelman

import Base: +,-,/,*,^,sin,cos,exp,log,convert,promote_rule,printlnstruct Dual <: Number
  p::Number # prime
  t::Number # tangent
end+(x::Dual,y::Dual) = Dual(x.p + y.p, x.t + y.t)-(x::Dual,y::Dual) = Dual(x.p - y.p, x.t - y.t)/(x::Dual,y::Dual) = Dual(x.p/y.p, (x.t*y.p - x.p*y.t)/x.p^2)*(x::Dual,y::Dual) = Dual(x.p*y.p, x.t*y.p + x.p*y.t)sin(x::Dual) = Dual(sin(x.p), cos(x.p) * x.t)cos(x::Dual) = Dual(cos(x.p), -sin(x.p) * x.t)exp(x::Dual) = Dual(exp(x.p), exp(x.p) * x.t)log(x::Dual) = Dual(log(x.p), (1/x.p) * x.t)^(x::Dual,p::Int) = Dual(x.p^p,p*x.p^(p-1)* x.t)# We can think of dual numbers analogously to complex numbers
# The epsilon term will be the derivative
println(x::Dual) = println(x.p," + ",x.t,"ϵ")# deal with conversion, and Dual with non-Dual math
convert(::Type{Dual}, x::Number) = Dual((x,zero(x)))
promote_rule(::Type{Dual},::Type{<:Number}) = Dual;

Lets test on our example \(f(x_1,x_2) = x_1 x_2 + \text{log}(x_1 ^2)\), the derivative at (3,5) should be \(5 \frac{2}{3}\).

x1 = Dual(3.0,1.0)
x2 = Dual(5.0,0.0)
f(x1,x2) = x1*x2 + log(x1^2)
y = f(x1,x2)
# df/dx1
println(y.t)
# direct calculation
println(x2.p + 2/x1.p)
5.666666666666667
5.666666666666667

Conclusion

Autodiff is important in machine learning and scientific computing and (forward mode) surprisingly easy to implement. I'll look at reverse mode autodiff in another post.

Thanks to Oscar Perez Concha who helped with discussions on the content of this post.

First Steps #2: The REPL

By: Josh Day

Re-posted from: https://www.juliafordatascience.com/first-steps-2-the-repl/

Prerequisite

What is the REPL?

First Steps #2: The REPL

REPL stands for Read-Eval-Print-Loop.  When you open Julia (double-click the icon or type julia in a terminal if you have it on your PATH), you'll enter Julia's REPL.  You'll see something like this:

               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0 (2021-03-24)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>
Julia REPL at Startup

When you type something and press enter, Julia will take your input, evaluate it, and print the result, e.g. typing x = "Hello" and hitting enter will result in:

julia> x = "Hello"
"Hello"

julia>
x = "Hello"

REPL Modes

Julia's REPL has amazing modes that affect how Julia interprets your input.  Here are the modes that are built-in:

? ➡ Help Mode 📖

Help Mode does a few things, best seen through example.  Type ? to enter Help Mode.  Then type println and press enter.

First Steps #2: The REPL
Image of ?println Output
  • The top line displays a fuzzy search of what you searched for.  This will help you find items even if you mistype something.
  • If your search term matches an existing identifier, the documentation for that identifier will print.  In this example our search found the function println.

Press delete to go back to Julia mode and type x = 1.  Now type ?x.  

  • Your search x matches with the x you just created.  However, there is no documentation available, so a summary of x (its type information) is printed out instead.  We'll talk about types in a future post.

; ➡ Shell Mode 💻

This lets you run shell commands (as if you opened a terminal but didn't start Julia).  A benefit of Julia's Shell Mode is that you can interpolate values into a command with $, e.g.

shell> echo $x
1
Interpolating a Julia value into a Shell Mode command.

] ➡ Pkg Mode 📦

Pkg Mode lets you run commands with Julia's package manager (we'll cover this in-depth in a future post) .  In the previous post we added packages with:

using Pkg

Pkg.add("StatsBase")

Alternatively, we could have used Pkg Mode and entered add StatsBase instead 🎉!  On a fresh install of Julia 1.6 you'll see something like:

(@v1.6) pkg> add StatsBase
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `~/.julia/environments/v1.6/Project.toml`
  [2913bbd2] + StatsBase v0.33.4
    Updating `~/.julia/environments/v1.6/Manifest.toml`
  [34da2185] + Compat v3.25.0
  [9a962f9c] + DataAPI v1.6.0
  [864edb3b] + DataStructures v0.18.9
	⋮
  [3f19e933] + p7zip_jll
  Progress [========================================>]  10/10
10 dependencies successfully precompiled in 9 seconds
Using Pkg Mode to Install StatsBase

REPL Modes are extendable! 🔌

Julia packages can add their own REPL modes.  Some examples are:

  • $ ➡ RCall (interpret input as R)
  • < ➡ Cxx (interpret intput as C++)

Notable REPL Features

Re-Run a Line

  • Press the up key to display the previously-run command.  Continue pressing up to go through your history.
  • Press ctrl-r to activate a search of your history.  Begin typing and the most recent line that contains your search will appear.  You can continue pressing ctrl-r to move backwards through matching lines in your history and ctrl-s to move forward.  If you've only run the commands seen in this and the previous post, typing x will then find the x = 1 line you entered earlier.  Press enter to choose the matching line.  

The Value of the Previous Line

The Julia REPL stores the value of the previous line in a variable called ans.  

julia> x = 1
1

julia> ans + 1
2
Using ans

Tab-Completion

You can Auto-complete names, file paths, symbols, and emoji.  

  • Try typing prin and pressing tab.  Your input will change to print.  Now press tab again.  Julia will show you that there are more identifiers available that also match with print. 
julia> print
print       println      printstyled
print[TAB]
  • Julia can help you type out file/directory paths when you press tab inside of a string.  For example, suppose you have a file at "/Users/<username>/file.csv". Typing "/Users/<username>/f" and then tab will autocomplete the f to file.csv (if it's the only file/directory that begins with f).  If there are multiple matches, press tab again to display them.
  • Try typing \pi and pressing tab.  Your input will change to π ! 🎉
  • Try typing \:smile: and pressing tab.  Your input will change to 😄!  Fun fact: This feature was introduced as an April Fool's Day joke but won traction in the Julia community.  You can even use emoji as variables!  There's no good reason to do this, but it's neat 🙃.
julia> 😄 = 1
1

julia> 🚀 = 2
2

julia> 😄 + 🚀
3
Emoji variables

Skip Printing

Sometime you may not want to display the return value of a line (e.g. the object prints a wall of text).  You can always skip displaying an object by adding ; to the end of a line:

julia> x = "1234"
"1234"

julia> x
"1234"

julia> x;

julia>
Skip Printing with ;

That's It!

This is a high-level overview of the Julia REPL.  It covers the things you'll need to know to get started with the REPL.  If you want to go further into the weeds, check out the official docs here.

Next up is First Steps #3: Plots.

Additional Resources