Visualizing Data with Julia using Makie

By: DSB

Re-posted from: https://medium.com/coffee-in-a-klein-bottle/visualizing-data-with-julia-using-makie-7685d7850f06?source=rss-8bd6ec95ab58------2

Plotting in Julia with Makie

A Brief Tutorial on Makie.jl

When starting learning Julia, one might get lost in the many different packages available to do data visualization. Right out of the cuff, there is Plots, Gadfly, VegaLite … and there is Makie.

Makie is fairly new (~2018), yet, it’s very versatile, actively developed and quickly growing in number of users. This article is a quick introduction to Makie, yet, by the end of it, you will be able to do a plethora of different plots.

The Future of Plotting in Julia

When I started coding in Julia, Makie was not one of the contenders for “best” plotting libraries. As time passed, I started to here more and more about it around the community. For some reason, people were saying that:

“Makie is the future” — People in the Julia Community

I never fully understood why that was the case, and every time I tried to learn it, I’d be turned off by the verbose syntax, and, frankly, ugly examples. It was only when I bumped into Beautiful Makie that I decided to put aside my prejudices and get on with the times.

Hence, if you are starting to code in Julia, and is wondering which plotting package you should invest your time to learn, I say to you that Makie is the way to go, since I guess “Makie is the future”.

Number of GitHub Star’s in per repository. I guess indeed Makie is the future, if this trend keeps going.

Starting with Makie… Pick your backend

The versatility in Makie can make it a bit unwelcoming for those that “just want to do a damn scatter plot”. First of all, there is Makie.jl, CairoMakie.jl, GLMakie.jl WGLMakie.jl ?. Which one should you use?

Well, here is the deal. Makie.jl is the main plotting package, but you have to choose a backend to which you will display your plots. The choice depends on your objectives. So yes, besides Makie.jl, you will need to install one of the backends. Here is a small description to help you chose:

  • CairoMakie.jl: It’s the easiest to use of all three, and it’s the ideal choice if you just want to produce static plots (not interactive);
  • GLMakie.jl: Uses OpenGL to display the plots, hence, you need to have OpenGL installed. Once you do a plot and run the display(myplot) , it’ll open an interactive window with your plot. If you want to do interactive 3D plots, then this is the backend for you;
  • WGLMakie.jl: It’s the hardest one to work with. Still, if you want to create interactive visualizations in the web, this is your choice.

In this tutorial, we’ll use CairoMakie.jl.

Your first plot

After picking our backend, we can now start plotting! I’ll go out on a limb and say that Makie is very similar to Matplotlib. It does not work with any fancy “Grammar of Graphics” (but if you like this sort of stuff, take a look at the AlgebraOfGraphics.jl, which implements an “Algebra of Graphics” on Makie).

Thus, there are a bunch of ready to use functions for some of the most common plots.

using CairoMakie #Yeah, no need to import Makie
scatter(rand(10,2))

Easy breezy… Yet, if you are plotting this in a Jupyter Notebook, you might be slightly ticked off by two things. First, the image is just too large. And second, it’s kind of low quality. What is going on?

By default, CairoMakie uses raster format for images, and the default size is a bit large. If you are like me and prefer your plots to be in svg and a bit smaller, then no worries! Just do the following:

using CairoMakie
CairoMakie.activate!(type = "svg")
scatter(rand(10,2),figure=(;resolution=(300,300)))

In the code above, the CairoMakie.activate!() is a command that tells Makie which backend you are using. You can import more than one backend at a time, and switch between them using this activation commands. Also, the CairoMakie backend has the option to do svg plots (to my knowledge, this is not possible for the other backends). Hence, with this small line of code, all our plots will now be displayed in high quality.

Next, we defined a “resolution” to our figure. In my opinion, this is a bit of an unfortunate name, because the resolution is actually the size of our image. Yet, as we’ll see further on, the attribute resolution actually belongs to our figure, and not to the actual scatter plot. For this reason we pass the whole figure = (; resolution=(300,300)) (if you are new to Julia, the ; is just a way of separating attributes that have names, from unnamed ones, i.e. args and kwags).

Congrats! You now know the bare minimum of Makie to do a whole bunch of different plots! Just go to the Makie’s website and see how to use all the different ready-to-use plotting functions! In order to be self contained, here is a small cheat sheet from the great book Julia Data Science.

Of course, we still haven’t talked about a bunch of important things, like titles, subplots, legends, axes limits, etc. Just keep on reading…

Storopoli, Huijzer and Alonso (2021). Julia Data Science. https://juliadatascience.io. ISBN: 9798489859165.

Figure, Axis and Plot

Commands like scatter produce a “FigureAxisPlot” object, which contains a figure, a set of axes and the actual plot. Each of these objects has different attributes and are fundamental in order to customize your visualization. By doing:

fig, ax, plt = scatter(rand(10,2))

We save each of these objects in a different variable, and can more easily modify them. In this example, the function scatter is actually creating all three objects, and not only the plot. We could instead create each of these objects individually. Here is how we do it:

fig = Figure(resolution=(600, 400)) 
ax = Axis(fig[1, 1], xlabel = "x label", ylabel = "y label",
title = "Title")
lines!(ax, 1:0.1:10, x->sin(x))
Plot from code above

Let’s explain the code above. First, we created the empty figure and stored it in fig . Next, we created an “Axis”. But, we need to tell to which figure this object belongs, and this is where the fig[1,1] comes in. But, what is this “[1,1]”?

Every figure in Makie comes with a grid layout underneath, which enable us to easily create subplots in the same figure. Hence, the fig[1,1] means “Axis belongs to fig row 1 and column 1”. Since our figure only has one element, then our axis will occupy the whole thing. Still confused? Don’t worry, once we do subplots you’ll understand why this is so useful.

The rest of the arguments in “Axis” are easy to understand. We are just defining the names in each axis and then the title.

Finally, we add the plot using lines! . The exclamation is a standard in Julia that means that a function is actually modifying an object. In our case, the lines!(ax, 1:0.1:10, x->sin(x)) is appending a line plot to the ax axis.

It’s clear now how we can, for example, add more line plots. By running the same lines! , this will append more plots to our ax axis. In this case, let’s also add a legend to our plot.

fig = Figure(resolution=(600, 400)) 
ax = Axis(fig[1, 1], xlabel = "x label", ylabel = "y label",
title = "Title")
lines!(ax, 1:0.1:10, x->sin(x), label="sin")
stairs!(ax, 1:0.1:10, x->cos(x), label="cos", color=:black)
axislegend(ax)
#*Tip*: if you are using Jupyter and want to display your
# visualization, you can do display(fig) or just write fig in
# the end of the cell.

Ok, our plots are starting to look good. Let me end this section talking about subplots. As I said, this is where the whole “fig[1,1]” comes into play. If instead of doing two plots in the same axis we wanted to create two parallel plots in the same figure, here is how we would do this.

fig = Figure(resolution=(600, 300)) 
ax1 = Axis(fig[1, 1], xlabel = "x label", ylabel = "y label",
title = "Title1")
ax2 = Axis(fig[1, 2], xlabel = "x label", ylabel = "y label",
title = "Title2")
lines!(ax1, 1:0.1:10, x->sin(x), label="sin")
stairs!(ax1, 1:0.1:10, x->cos(x), label="cos", color=:black)
density!(ax2, randn(100))
axislegend(ax)
save("figure.png", fig)

This time, in the same figure, we created two axis, but the first one is in the first row and first column, while the second one is in the second column. We then just append the plot to the respective axis. Lastly, we save the figure in “png” format.

Final Words

That’s it for this tutorial. Of course, there is much more the talk about, as we have only scratched the surface. Makie has some awesome capabilities in terms of animations, and much more attributes/objects to play with in order to create truly astonishing visualizations. If you want to learn more, take a look at Makie’s documentation, it’s very nice. And also, the Julia Data Science book has a chapter only on Makie.

References

This article draws heavily on the Julia Data Science book and Makie’s own documentation.

Storopoli, Huijzer and Alonso (2021). Julia Data Science. https://juliadatascience.io. ISBN: 9798489859165.

Danisch & Krumbiegel, (2021). Makie.jl: Flexible high-performance data visualization for Julia. Journal of Open Source Software, 6(65), 3349, https://doi.org/10.21105/joss.03349


Visualizing Data with Julia using Makie was originally published in Coffee in a Klein Bottle on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to DataFramesMeta.jl

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2021/11/19/dfm.html

Introduction

If you start using Julia for data science you might get overwhelmed by the
number of available options and features. Today I want to write about the
DataFramesMeta.jl package that greatly simplifies one of the most
difficult parts of the DataFrames.jl package to learn, namely – performing
data transformations.

In this post I will omit all advanced features of both DataFramesMeta.jl
and DataFrames.jl and focus on simple issues to help you build a correct
mental model how things should be used.

The post was written under Julia 1.6.3, DataFrames.jl 1.2.2, and
DataFramesMeta.jl 0.10.0.

Setting up the stage

Let us first load the required packages and create some simple data frame we
will want to work with:

julia> using DataFramesMeta

julia> using Statistics

julia> df = DataFrame(x=1:5, y=11:15)
5×2 DataFrame
 Row │ x      y
     │ Int64  Int64
─────┼──────────────
   1 │     1     11
   2 │     2     12
   3 │     3     13
   4 │     4     14
   5 │     5     15

Notice that when we load DataFramesMeta.jl also DataFrames.jl is automatically
loaded to your working environment. Additionally, I have loaded the Statistics
module as soon we will use it in our examples.

Understanding data transformations

When you want to perform some transformation of your data the first thing you
need to answer is if you want to aggregate data or manipulate columns.

Data aggregation is a simple concept – I take a column as input and produce e.g.
its mean, which is a single aggregated value. In DataFrames.jl we call this
operation combine, as we are combining rows.

When I talk about column manipulation I mean operations that we take a column
and produce output that is also a column that has the same number of elements
as the source, e.g. I multiply the column by 2. In DataFrames.jl we call this
operation either select or transform. What is the difference between
select and transform? When you perform a select operation you keep in the
result only the results of the operations you performed. On the other hand,
when you transform a data frame you additionally keep all the columns from
the source data frame.

Let us now have a look at examples of these three operations. Start with
aggregation:

julia> @combine(df, :sum_y = sum(:x), :mean_y = mean(:y))
1×2 DataFrame
 Row │ sum_y  mean_y
     │ Int64  Float64
─────┼────────────────
   1 │    15     13.0

As you can see we used the combine word and prepended it with @ which
signals that this is a DataFramesMeta.jl operation. As a first argument in our
call we passed the source data frame. Next we specified the aggregations we
want to perform. Note that each aggregation is specified just as you would
write normal Julia code using variables. There is only one rule to learn. When
you prefix the variable name with : it means that you are referring to a
column of a data frame.

Now let us perform selection and transformation side by side to see the
difference:

julia> @select(df, :z = :x + :y)
5×1 DataFrame
 Row │ z
     │ Int64
─────┼───────
   1 │    12
   2 │    14
   3 │    16
   4 │    18
   5 │    20

julia> @transform(df, :z = :x + :y)
5×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1     11     12
   2 │     2     12     14
   3 │     3     13     16
   4 │     4     14     18
   5 │     5     15     20

As you can see both operations create a new column :z. The difference is that
@transform also keeps the :x and :y variables, while @select drops them.

Let us write another transformation:

julia> @transform(df, :z = :x * :y)
ERROR: MethodError: no method matching *(::Vector{Int64}, ::Vector{Int64})

This time the operation failed. Most Julia users know why. You cannot multiply a
vector by a vector – this is not a properly defined mathematical operation.
Instead you have to broadcast the multiplication operation like this (this is
often called a vectorized operation):

julia> @transform(df, :z = :x .* :y)
5×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1     11     11
   2 │     2     12     24
   3 │     3     13     39
   4 │     4     14     56
   5 │     5     15     75

In more complex scenarios adding the . for broadcasting can easily get
annoying, e.g.:

julia> @transform(df, :a = 2 .* :x, :b = :x .* :y .^ 2)
5×4 DataFrame
 Row │ x      y      a      b
     │ Int64  Int64  Int64  Int64
─────┼────────────────────────────
   1 │     1     11      2    121
   2 │     2     12      4    288
   3 │     3     13      6    507
   4 │     4     14      8    784
   5 │     5     15     10   1125

On the other hand practice shows that such broadcasted operations are quite
common. Therefore in DataFrames.jl parlance they are called by-row operations.
DataFramesMeta.jl allows an easy way to tell @select and @transform that
all operations that user passes to them should be applied by-row. Just prefix
the name of the transformation function with the r character (r stands for
row). Therefore we have @rselect and @rtransform:

julia> @rselect(df, :a = 2 * :x, :b = :x * :y ^ 2)
5×2 DataFrame
 Row │ a      b
     │ Int64  Int64
─────┼──────────────
   1 │     2    121
   2 │     4    288
   3 │     6    507
   4 │     8    784
   5 │    10   1125

julia> @rtransform(df, :a = 2 * :x, :b = :x * :y ^ 2)
5×4 DataFrame
 Row │ x      y      a      b
     │ Int64  Int64  Int64  Int64
─────┼────────────────────────────
   1 │     1     11      2    121
   2 │     2     12      4    288
   3 │     3     13      6    507
   4 │     4     14      8    784
   5 │     5     15     10   1125

As you can see we got rid of the dots, paying the cost of having all operations
applied by-row to our data.

As an exercise think how you would subtract the mean from column :x in our
data frame. Can we use @rselect or we must use @rselect? You can use both:

julia> @select(df, :x, :x2 = :x .- mean(:x))
5×2 DataFrame
 Row │ x      x2
     │ Int64  Float64
─────┼────────────────
   1 │     1     -2.0
   2 │     2     -1.0
   3 │     3      0.0
   4 │     4      1.0
   5 │     5      2.0

julia> @rselect(df, :x, :x2 = :x - mean(df.x))
5×2 DataFrame
 Row │ x      x2
     │ Int64  Float64
─────┼────────────────
   1 │     1     -2.0
   2 │     2     -1.0
   3 │     3      0.0
   4 │     4      1.0
   5 │     5      2.0

I would say, however, that this time using @select is more natural. Although
we have to use the . in :x2 = :x .- mean(:x) it is pretty easy to understand
what was going on there.

When we used @rselect we had to pass the df.x column to the mean (this is a
value computed as any other Julia code, DataFramesMeta.jl does not touch it as
it does not have : in front). Note that just passing :x would be incorrect,
as mean would be also applied by-row to it so we would broadcast mean over
the :x column and the result would be:

julia> @rselect(df, :x, :x2 = :x - mean(:x))
5×2 DataFrame
 Row │ x      x2
     │ Int64  Float64
─────┼────────────────
   1 │     1      0.0
   2 │     2      0.0
   3 │     3      0.0
   4 │     4      0.0
   5 │     5      0.0

and this is most likely not what we want (unless we wanted to check that
subtracting some number from itself is equal to zero). In summary putting a r
prefix broadcasts the operation with respect to the columns of a data frame
(i.e. parts of the passed expression that contain names with a : prefix).

So now we know that if we prefix select or transform with r we switch to
by-row mode. Is there anything more to learn? Indeed there is one more thing
you need to know. This is a ! suffix that these functions can take. What it
does is that it makes the operation update the passed data frame. Note that
above when we performed transformations we were getting a fresh data frame, but
our df source data frame was untouched. When you suffix ! you get exactly
the same result but it gets stored in the data frame you passed to the
operation. Here are some examples:

julia> df
5×2 DataFrame
 Row │ x      y
     │ Int64  Int64
─────┼──────────────
   1 │     1     11
   2 │     2     12
   3 │     3     13
   4 │     4     14
   5 │     5     15

julia> @transform!(df, :z = :x + :y)
5×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1     11     12
   2 │     2     12     14
   3 │     3     13     16
   4 │     4     14     18
   5 │     5     15     20

julia> df
5×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1     11     12
   2 │     2     12     14
   3 │     3     13     16
   4 │     4     14     18
   5 │     5     15     20

julia> @select!(df, :s = :x + :y + :z)
5×1 DataFrame
 Row │ s
     │ Int64
─────┼───────
   1 │    24
   2 │    28
   3 │    32
   4 │    36
   5 │    40

julia> df
5×1 DataFrame
 Row │ s
     │ Int64
─────┼───────
   1 │    24
   2 │    28
   3 │    32
   4 │    36
   5 │    40

Why might we want such in-place operations? Consider a large data frame
with 10,000 columns. If you perform a @transform of such a data frame adding
one column to it you will copy a lot of data (which takes time and RAM). By
doing @transform! you will be faster and more memory efficient, at the expense
of mutating the source data frame.

Conclusions

Today as a conclusion let me present the following flowchart summarizing
the basic available data transformation options in DataFramesMeta.jl
that I have covered:

Transformations guideline flowchart

There are many more features of DataFramesMeta.jl that I have not covered like:
subsetting rows of a data frame, sorting it, or performing operations on
grouped data. You can find all the details in the documentation of
DataFramesMeta.jl.

So you want to do Google Summer of Code with the Julia Language

By: Logan Kilpatrick

Re-posted from: https://logankilpatrick.medium.com/so-you-want-to-do-google-summer-of-code-with-the-julia-language-bffb8d3b280e?source=rss-2c8aac9051d3------2

With the announcement of expanded eligibility for Google Summer of Code (GSoC), there are likely to be a lot of new folks considering…