Working with matrices in DataFrames.jl 1.0

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2021/04/16/arrays.html

Introduction

DataFrames.jl will have a 1.0 release in a few days. In this post I
want to comment on an issue that I expect might cause the most legacy code
breakage with this release.

The post is tested under Julia 1.6, OrdinaryDiffEq 5.52.3, and
DataFrames.jl 0.22.7 (but I also discuss what changes in 1.0 release).

The past

In the past all matrices could be converted to a DataFrame like this:

~$ julia --banner=no
(@v1.6) pkg> st DataFrames
      Status `~/.julia/environments/v1.6/Project.toml`
  [a93c6f00] DataFrames v0.22.7

julia> using DataFrames

julia> mat = [1 2; 3 4]
2×2 Matrix{Int64}:
 1  2
 3  4

julia> DataFrame(mat)
2×2 DataFrame
 Row │ x1     x2
     │ Int64  Int64
─────┼──────────────
   1 │     1      2
   2 │     3      4

julia> exit()

However, unfortunately this output is deceptive. Try this:

~$ julia --banner=no --depwarn=error
(@v1.6) pkg> st DataFrames
      Status `~/.julia/environments/v1.6/Project.toml`
  [a93c6f00] DataFrames v0.22.7

julia> using DataFrames

julia> mat = [1 2; 3 4]
2×2 Matrix{Int64}:
 1  2
 3  4

julia> DataFrame(mat)
ERROR: `DataFrame(columns::AbstractMatrix)` is deprecated, use `DataFrame(columns, :auto)` instead.

julia> exit()

As you can see DataFrame(mat) is deprecated. The problem is that
Julia 1.6 is hiding deprecation warnings by default. Under
DataFrames.jl 1.0 DataFrame(mat) will error always.

Why is DataFrame(mat) not allowed?

The reason why we decided to disallow DataFrame constructor with a single
Matrix argument is that DataFrames.jl now follows the rule:

If DataFrame constructor is passed a single positional argument this argument
must be a table that is following Tables.jl API.

The point is that Matrix does not support this API.

A person knowing the DataFrames.jl API more thoroughly might point out that the
statement above is not true, and DataFrames.jl allows for some exceptions where
a non-table is accepted in a constructor. This is indeed the case. Here are
the offending cases:

~$ julia --banner=no --depwarn=error
(@v1.6) pkg> st DataFrames
      Status `~/.julia/environments/v1.6/Project.toml`
  [a93c6f00] DataFrames v0.22.7

julia> using DataFrames

julia> DataFrame(:a => 1) # a pair
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     1

julia> df = DataFrame([:a => 1]) # a vector of pairs
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     1

julia> dfr = df[1, :]
DataFrameRow
 Row │ a
     │ Int64
─────┼───────
   1 │     1

julia> DataFrame(dfr) # DataFrameRow
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     1

julia> gdf = groupby(df, :a)
GroupedDataFrame with 1 group based on key: a
First Group (1 row): a = 1
 Row │ a
     │ Int64
─────┼───────
   1 │     1

julia> DataFrame(gdf) # GroupedDataFrame
1×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     1

These four cases are deliberately left for convenience because there is a very
low risk that they would cause confusion. Out of these four cases a vector of
pairs is most problematic, as in Tables.jl it would get the following treatment:

julia> DataFrame(Tables.columntable([:a => 1]))
1×2 DataFrame
 Row │ first   second
     │ Symbol  Int64
─────┼────────────────
   1 │ a            1

However, we have decided that it is extremely unlikely that someone might want
to get this type of a data frame (and in case you wanted it the code above
shows you how to get it reliably).

So why have we not made an exception for matrices? The reason is that there
are many cases where AbtractMatrix actually supports Tables.jl interface
and requires a special way how it should be converted to a DataFrame.

To give some specific example have a look at this issue related
to differential equations solving (I have adapted it a bit):

~/Desktop/Dev/DF_dev$ julia --banner=no
(@v1.6) pkg> st DataFrames
      Status `~/.julia/environments/v1.6/Project.toml`
  [a93c6f00] DataFrames v0.22.7

julia> using OrdinaryDiffEq, DataFrames

julia> function parameterized_lorenz(du, u, p, t)
        du[1] = p[1] * (u[2] - u[1])
        du[2] = u[1] * (p[2] - u[3]) - u[2]
        du[3] = u[1] * u[2] - p[3] * u[3]
       end
parameterized_lorenz (generic function with 1 method)

julia> u0 = [1.0, 0.0, 0.0];

julia> tspan = (0.0, 1.0);

julia> p = [10.0, 28.0, 8/3];

julia> prob = ODEProblem(parameterized_lorenz, u0, tspan, p);

julia> sol1 = solve(prob, Rosenbrock23());

julia> DataFrame(sol1)
3×61 DataFrame
 Row │ x1       x2           x3          x4          x5          x6           ⋯
     │ Float64  Float64      Float64     Float64     Float64     Float64      ⋯
─────┼─────────────────────────────────────────────────────────────────────────
   1 │     1.0  0.999925     0.999178    0.995241    0.989338    0.977794     ⋯
   2 │     0.0  0.000209532  0.00230391  0.0134134   0.0302987   0.0641965
   3 │     0.0  7.83999e-10  9.47873e-8  3.21298e-6  1.63912e-5  7.35689e-5
                                                             55 columns omitted

julia> DataFrame(Tables.columntable(sol1))
61×4 DataFrame
 Row │ timestamp    value1     value2        value3
     │ Float64      Float64    Float64       Float64
─────┼────────────────────────────────────────────────────
   1 │ 0.0           1.0        0.0           0.0
   2 │ 7.48361e-6    0.999925   0.000209532   7.83999e-10
   3 │ 8.23197e-5    0.999178   0.00230391    9.47873e-8
   4 │ 0.000480311   0.995241   0.0134134     3.21298e-6
   5 │ 0.00108852    0.989338   0.0302987     1.63912e-5
   6 │ 0.00232154    0.977794   0.0641965     7.35689e-5
   7 │ 0.00391981    0.963652   0.107499      0.000206214
  ⋮  │      ⋮           ⋮           ⋮             ⋮
  55 │ 0.735315     -7.14414   -8.67351      25.5598
  56 │ 0.771575     -7.66582   -9.02351      25.4699
  57 │ 0.812974     -8.20321   -9.44232      25.6821
  58 │ 0.859509     -8.74036   -9.79982      26.2593
  59 │ 0.908783     -9.18149   -9.8967       27.1128
  60 │ 0.960609     -9.41649   -9.59708      28.0144
  61 │ 1.0          -9.39804   -9.12529      28.5183
                                           47 rows omitted

Now we can see that DataFrame(sol1) produces a wrong result because sol1
is an AbstractMatrix as you can check here:

julia> sol1 isa AbstractMatrix
true

Let us switch to main branch of DataFrames.jl for the remaining of this post
to test the behavior under DataFrames.jl 1.0 that will be released soon:

~/Desktop/Dev/DF_dev$ julia --banner=no
(@v1.6) pkg> activate --temp
  Activating new environment at `/tmp/jl_43Ofes/Project.toml`

(jl_43Ofes) pkg> add DataFrames#main

(jl_43Ofes) pkg> st DataFrames
      Status `/tmp/jl_43Ofes/Project.toml`
  [a93c6f00] DataFrames v0.22.7 `https://github.com/JuliaData/DataFrames.jl.git#main`

julia> using OrdinaryDiffEq, DataFrames
[ Info: Precompiling OrdinaryDiffEq [1dea7af3-3e70-54e6-95c3-0bf5283fa5ed]

julia> function parameterized_lorenz(du, u, p, t)
        du[1] = p[1] * (u[2] - u[1])
        du[2] = u[1] * (p[2] - u[3]) - u[2]
        du[3] = u[1] * u[2] - p[3] * u[3]
       end
parameterized_lorenz (generic function with 1 method)

julia> u0 = [1.0, 0.0, 0.0];

julia> tspan = (0.0, 1.0);

julia> p = [10.0, 28.0, 8/3];

julia> prob = ODEProblem(parameterized_lorenz, u0, tspan, p);

julia> sol1 = solve(prob, Rosenbrock23());

julia> DataFrame(sol1)
61×4 DataFrame
 Row │ timestamp    value1     value2        value3
     │ Float64      Float64    Float64       Float64
─────┼────────────────────────────────────────────────────
   1 │ 0.0           1.0        0.0           0.0
   2 │ 7.48361e-6    0.999925   0.000209532   7.83999e-10
   3 │ 8.23197e-5    0.999178   0.00230391    9.47873e-8
   4 │ 0.000480311   0.995241   0.0134134     3.21298e-6
   5 │ 0.00108852    0.989338   0.0302987     1.63912e-5
   6 │ 0.00232154    0.977794   0.0641965     7.35689e-5
   7 │ 0.00391981    0.963652   0.107499      0.000206214
  ⋮  │      ⋮           ⋮           ⋮             ⋮
  56 │ 0.771575     -7.66582   -9.02351      25.4699
  57 │ 0.812974     -8.20321   -9.44232      25.6821
  58 │ 0.859509     -8.74036   -9.79982      26.2593
  59 │ 0.908783     -9.18149   -9.8967       27.1128
  60 │ 0.960609     -9.41649   -9.59708      28.0144
  61 │ 1.0          -9.39804   -9.12529      28.5183
                                           48 rows omitted

And as you can see this time all worked as expected.

How to move from a matrix to a data frame the old way?

A question is can you get an old behavior easily under DataFrames.jl 1.0?
The answer is yes. It is enough to pass :auto as a second positional argument
to treat any AbstractMatrix the old way. The key point here is that :auto
adds a second argument to a constructor, which allows to disambiguate this call
and make sure we do not try a Tables.jl fallback. So continuing our last example
we have:

julia> DataFrame(sol1, :auto)
3×61 DataFrame
 Row │ x1       x2           x3          x4          x5          x6         ⋯
     │ Float64  Float64      Float64     Float64     Float64     Float64    ⋯
─────┼───────────────────────────────────────────────────────────────────────
   1 │     1.0  0.999925     0.999178    0.995241    0.989338    0.977794   ⋯
   2 │     0.0  0.000209532  0.00230391  0.0134134   0.0302987   0.0641965
   3 │     0.0  7.83999e-10  9.47873e-8  3.21298e-6  1.63912e-5  7.35689e-5
                                                           55 columns omitted

Here are some more examples (still on main):

julia> DataFrame([1 2; 3 4])
ERROR: ArgumentError: `DataFrame` constructor from a `Matrix` requires passing :auto as a second argument to automatically generate column names: `DataFrame(matrix, :auto)`

julia> DataFrame([1 2; 3 4], :auto) # auto generated column names
2×2 DataFrame
 Row │ x1     x2
     │ Int64  Int64
─────┼──────────────
   1 │     1      2
   2 │     3      4

julia> DataFrame([1 2; 3 4], [:c1, :c2]) # passing column names explicitly
2×2 DataFrame
 Row │ c1     c2
     │ Int64  Int64
─────┼──────────────
   1 │     1      2
   2 │     3      4

So as you can see the fix is easy.

Finally let me comment that another common similar case is a vector of vectors
being passed to a DataFrame constructor. It follows the same rules:

julia> DataFrame([1:2, 3:4])
ERROR: ArgumentError: `DataFrame` constructor from a `Vector` of vectors requires passing :auto as a second argument to automatically generate column names: `DataFrame(vecs, :auto)`

julia> DataFrame([1:2, 3:4], :auto)
2×2 DataFrame
 Row │ x1     x2
     │ Int64  Int64
─────┼──────────────
   1 │     1      3
   2 │     2      4

julia> DataFrame([1:2, 3:4], [:c1, :c2])
2×2 DataFrame
 Row │ c1     c2
     │ Int64  Int64
─────┼──────────────
   1 │     1      3
   2 │     2      4

Again – using Tables.jl default behavior would get you something unexpected
(unless you are really deep into Tables.jl mechanics ?):

julia> DataFrame(Tables.columntable([1:2, 3:4]))
2×2 DataFrame
 Row │ start  stop
     │ Int64  Int64
─────┼──────────────
   1 │     1      2
   2 │     3      4

Conclusions

We have tried very hard to make things in DataFrames.jl 1.0 maximally consistent
with the whole Julia package ecosystem while allowing a relatively easy handling
of common data processing tasks.

Conversion from a matrix to a DataFrame is one of common hard corner cases
affected. I hope this post explains you the rationale behind the design
decisions taken in DataFrames.jl 1.0 release in this area and ways how the
DataFrame constructor should be used to give you desired results.

Julia Quickstart

By: Josh Day

Re-posted from: https://www.juliafordatascience.com/quickstart/

Enjoying Julia For Data Science?  Please share us with a friend and follow us on Twitter at @JuliaForDataSci.

Julia Quickstart

This post is something between a FAQ and lightning-fast introduction to Julia.  Think of it as "First Steps #0: I've heard of Julia. What's it Like to Code in It?".  After you've read this, check out our First Steps series to keep on learning!  

This page was last updated June 10, 2021.

🤔 I'm Stuck.  Where Can I Find Help?

1. Try the Julia REPL's help mode.

2. Help mode didn't answer your question?

3. Still stuck? Time to ask for help! 🙋

  • Do you think other people have the same question?
    • Yes: Please post your question on Julia Discourse for posterity! Slack messages disappear after a time and we'd love to keep our shared knowledge searchable.
    • No: Ask on the Julia Slack.

The Julia community is full of people who like to help! We'll note that it's beneficial for everyone if you ask good questions.

Working with Arrays

Creating Vectors

x = [1, 2, 3, 4]

# A "Range" doesn't store the values between 1 and 4.
y = 1:4  

# `1:4` -> `[1, 2, 3, 4]`
collect(y)  

# 1 to 100 with a step size of 3: [1, 4, 7, ..., 94, 97, 100]
1:3:100

Creating Matrices

# Row-vector (1 x 4)
[1 2 3 4] 

# Matrix (2 x 3)
[1 2 3 ; 3 4 5]

# Matrix (100 x 3) of random Normal(0, 1) samples
randn(100, 3)

Indexing (1-Based)

If someone tells you a language is unusable because it uses 1 (or 0)-based indexing, they are just plain wrong.

1-based indexing is a big deal for same reason most other fad topics are a big deal: it’s such a simple idea that everyone can have an opinion on it, and everyone seems to think they can “help” by telling their personal experience about how this arbitrary choice has affected them at one time in their life.

– Chris Rackauckus via Julia Discourse

x = rand(100, 2)

x[3, 2]  # retrieve 3rd row of column 2

Arrays are Column-Major

This means that data in a matrix is stored in computer memory with column elements next to each other.

x = rand(100, 2)

x[105] == x[5, 2]

Working With Strings

  • A big difference from some languages is that " is different from '.
  • Strings are made "like this".
  • Character literals are made like this: 's'.
  • String concatenation is achieved via *:
julia> "Hello, " * "World!"
"Hello, World!"
Hello, World!
  • String interpolation is achieved through $.
julia> x = "World!"
"World!"

julia> "Hello, $x"
"Hello, World!"
Hello Again
  • String macros change the interpretation of a string:
julia> r"[a-z]"  # I'm a regular expression!
r"[a-z]"

julia> html"<div>I'm html</div>"  # I'm HTML!
HTML{String}("<div>I'm html</div>")

📦 How do I Find/Install/Load Packages?

Finding Packages

JuliaHub is a great resource for discovering packages.  We find it's a bit easier to find stuff compared to Googling.

It's hard to know which Julia packages are "the good ones" at first glance.  However, good packages tend to have similar characteristics:

  • Active development.  GitHub's pulse feature shows a summary of package activity.
  • Quality documentation.  It's a good sign when the docs are both understandable and thorough, as they are for DataFrames.
  • Other people are interested in it.  On GitHub, the Watch number is the how many people receive notifications for activity, the Star number is how many people have "liked" it, and Fork is how many people have created their own copy of the package to potentially make changes to it.  It's typically a good sign when these numbers are large.
Julia Quickstart
"Interest" in DataFrames.jl

Installing Packages

The simplest way to add packages is to use Pkg Mode in the REPL by pressing ].  You'll notice the prompt will change to (current environment) pkg>

(@v1.6) pkg> add DataFrames, StatsBase

Loading Packages

using DataFrames, StatsBase

# Only bring certain names into the namespace
using StatsBase: countmap, zscore

Using Environments

Julia lets you use different environments that use different collections of packages/package versions.  The default environment is v1.6 (note the Pkg Mode prompt above).  You can activate a new environment with:

] activate <dir>

If you make changes (e.g. add a package) to an environment, two files will be created: Project.toml and Manifest.toml.

  • What's Project.toml?  How the user tells Julia what they want installed.  Version bounds for packages go here.
  • What's Manifest.toml?  How Julia tells the user what is installed.  

What are Types?

  • Everything in Julia has a type.
julia> typeof(1)
Int64
  • Types can be parameterized by another type.  For example, an Array is parameterized by the type of its elements and number of dimensions.  Therefore, a vector of 64-bit integers is an Array{Int64, 1}.
julia> typeof([1,2,3])
Vector{Int64} (alias for Array{Int64, 1})
  • If we follow Int64 "up the type tree" we'll eventually run into Any, the top level abstract type.  
julia> supertype(Int64)
Signed

julia> supertype(Signed)
Integer

julia> supertype(Integer)
Real

julia> supertype(Real)
Number

julia> supertype(Number)
Any
  • Abstract types "don't exist", but they define a set of concrete types (things that exist).  For example, you can create an instance of Int64, but not Real.  Inside the set of all Real numbers, Int64 is one of many concrete types.
Julia Quickstart
Real Numbers

🎉 What is Multiple Dispatch? 🎉

  • Multiple dispatch is a major part of why people love Julia.  The gist of it is that you can write functions so that different/specialized code is called depending on the types of the arguments.
julia> f(x::Int) = 1
f (generic function with 1 method)

julia> f(x::Float64) = 2
f (generic function with 2 methods)

julia> f(1)
1

julia> f(1.0)
2
Super Simple Multiple Dispatch Example
  • Above we used ::Type to add a type annotation.  Since we only added methods for ::Int and ::Float64, our function f can only be called on Ints and Float64s.  However, type annotations are not necessary:

Automatic Specialized Code

  • Julia uses a Just-in-time compiler, meaning that every time you call a function with new types, Julia compiles a specific method for exactly those types.  Thus the following two functions will have the same performance!
function f(x::Type1, y::Type2, z::Type3)
    # big computation
end

function f(x, y, z)
    # big computation
end

What is Broadcasting?

Broadcasting is a way of applying a function to multiple inputs at once.  

  • For example, there is no mathematical definition for calling the sine function on a vector, but many languages will automatically apply sine to each element.  In Julia, you must explicity broadcast a function over multiple inputs by adding a dot .
julia> sin([1,2,3])
ERROR: MethodError: no method matching sin(::Vector{Int64})

julia> sin.([1,2,3])
3-element Vector{Float64}:
 0.8414709848078965
 0.9092974268256817
 0.1411200080598672
  • You can even fuse broadcasted computations, which removes the need to create temporary vectors:
julia> x = [1,2,3];

julia> y = [4,5,6];

julia> z = [7,8,9];

julia> x .+ (y .* sin.(z))
3-element Vector{Float64}:
 3.6279463948751562
 6.946791233116909
 5.472710911450539
Broadcast Fusion

How do I Code in Julia?

According to the 2020 Julia User & Developer Survey (PDF), Julia programmers use the following editors/IDEs "frequently":

A new coding environment on the scene is Pluto.jl, which we love!  If you are new to Julia or programming in general, we recommend starting with Pluto 🎈.

What are Macros?

Macros (names that start with @) are functions of expressions.  They let you change an expression before it gets run.  For example, @time will record both the time elapsed and allocations generated from an expression.

julia> @time begin
          sleep(1)
          sleep(2)
       end
  3.008873 seconds (8 allocations: 256 bytes)

Metaprogramming (writing code that writes other code) is a pretty advanced topic.  It's also a super powerful tool.

That's it!

Did you like this post?  Have a question?  Did we miss something important?

Ping us on Twitter at @JuliaForDataSci 🚀

Additional Resources