By: julia on Abel Soares Siqueira
Re-posted from: https://abelsiqueira.com/blog/2022-01-26-python-and-julia-2/
I worked on this blog post working for the Netherlands eScience Center. Co-authored by Faruk Diblen.
Check the blog post here.
Categorical vs pooled arrays
Re-posted from: https://bkamins.github.io/julialang/2022/01/21/categorical.html
Introduction
When working with data in Julia you most likely have encountered two packages
PooledArrays.jl and CategoricalArrays.jl. They both provide
custom implementations of AbstractArray that are useful when you have low
cardinality data.
In this post I want to highlight major similarities and differences between
these two packages.
The examples were tested under Julia 1.7.0, DataFrames.jl 1.3.1,
PooledArrays.jl 1.4.0, CategoricalArrays.jl 0.10.2, FreqTables.jl 0.4.5,
and BenchmarkTols 1.2.2.
PooledArrays.jl
As I have explained in my recent post you can use PooledArrays.jl when
you have large vectors that have few unique values. The simplest way of thinking
about PooledArray type is that it can be used as a way to compress the data.
Here is an example:
julia> using PooledArrays
julia> v = repeat(["a", "b", "c", "d"], 10^6)
4000000-element Vector{String}:
"a"
"b"
"c"
"d"
⋮
"a"
"b"
"c"
"d"
julia> Base.summarysize(v)
32000076
julia> pv = PooledArray(v)
4000000-element PooledVector{String, UInt32, Vector{UInt32}}:
"a"
"b"
"c"
"d"
⋮
"a"
"b"
"c"
"d"
julia> Base.summarysize(pv)
16000580
As you can see even for short strings we have achieved significant compression.
You can have even better compression by passing compress=true keyword argument:
julia> pv2 = PooledArray(v, compress=true)
4000000-element PooledVector{String, UInt8, Vector{UInt8}}:
"a"
"b"
"c"
"d"
⋮
"b"
"c"
"d"
julia> Base.summarysize(PooledArray(v, compress=true))
4000532
There is one more benefit of using PooledArray when working with DataFrames.jl.
You can expect grouping and join operations to be performed faster. Here are
some examples:
julia> using DataFrames
julia> using BenchmarkTools
julia> df = DataFrame(;v, pv, pv2)
4000000×3 DataFrame
Row │ v pv pv2
│ String String String
─────────┼────────────────────────
1 │ a a a
2 │ b b b
3 │ c c c
⋮ │ ⋮ ⋮ ⋮
3999998 │ b b b
3999999 │ c c c
4000000 │ d d d
3999994 rows omitted
julia> @benchmark groupby($df, :v)
BenchmarkTools.Trial: 61 samples with 1 evaluation.
Range (min … max): 66.280 ms … 170.877 ms ┊ GC (min … max): 0.68% … 58.51%
Time (median): 79.254 ms ┊ GC (median): 0.62%
Time (mean ± σ): 82.754 ms ± 17.021 ms ┊ GC (mean ± σ): 5.53% ± 10.36%
█ ▇
▃▃▃█▅▁▄█▅▄▁▁▃▄█▆▃▁▁▃▃▁▁▁▁▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃ ▁
66.3 ms Histogram: frequency by time 154 ms <
Memory estimate: 125.04 MiB, allocs estimate: 37.
julia> @benchmark groupby($df, :pv)
BenchmarkTools.Trial: 678 samples with 1 evaluation.
Range (min … max): 3.920 ms … 20.284 ms ┊ GC (min … max): 0.00% … 4.13%
Time (median): 4.835 ms ┊ GC (median): 0.00%
Time (mean ± σ): 7.373 ms ± 4.111 ms ┊ GC (mean ± σ): 5.02% ± 10.96%
▃█▇▃▂ ▁▅ ▁ ▄▁▃▂
██████▇▆▇██▇▆▆▆▅▆▇▆█▆▄▅▁▁▆▁▁▆▄▇▅▅▆█▄█████▇▄▅▄▄▄▆▅▅▅▅▁▄▁▅▄▅ ▇
3.92 ms Histogram: log(frequency) by time 18.6 ms <
Memory estimate: 30.52 MiB, allocs estimate: 61.
julia> @benchmark groupby($df, :pv2)
BenchmarkTools.Trial: 710 samples with 1 evaluation.
Range (min … max): 3.462 ms … 19.995 ms ┊ GC (min … max): 0.00% … 8.23%
Time (median): 4.356 ms ┊ GC (median): 0.00%
Time (mean ± σ): 7.035 ms ± 4.189 ms ┊ GC (mean ± σ): 5.63% ± 11.95%
▂▄█▃▁ ▁ ▄▁ ▃ ▃
██████▇▆█▇▇██▇▇▇▆▅▆▇▆▄▅▁▁▁▄▄▆▄▄▇▄▁▄▇██████▇▅▇▅▄▅▄▅▄▁▄▄▅▅▅▇ ▇
3.46 ms Histogram: log(frequency) by time 18 ms <
Memory estimate: 30.52 MiB, allocs estimate: 61.
julia> df2 = df[1:4, :]
4×3 DataFrame
Row │ v pv pv2
│ String String String
─────┼────────────────────────
1 │ a a a
2 │ b b b
3 │ c c c
4 │ d d d
julia> @benchmark innerjoin($df, $df2, on=:v, makeunique=true)
BenchmarkTools.Trial: 26 samples with 1 evaluation.
Range (min … max): 153.954 ms … 281.650 ms ┊ GC (min … max): 1.63% … 34.70%
Time (median): 189.137 ms ┊ GC (median): 1.12%
Time (mean ± σ): 193.827 ms ± 26.318 ms ┊ GC (mean ± σ): 5.15% ± 8.92%
▁ ▁ ▁█ ▄ ▁
▆▁▁▆▁▁▁▁▁█▆▆▁█▁██▆▁█▁█▆▆▆▁▆▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆▁▁▁▁▁▁▁▁▆ ▁
154 ms Histogram: frequency by time 282 ms <
Memory estimate: 143.01 MiB, allocs estimate: 262.
julia> @benchmark innerjoin($df, $df2, on=:pv, makeunique=true)
BenchmarkTools.Trial: 32 samples with 1 evaluation.
Range (min … max): 124.786 ms … 258.973 ms ┊ GC (min … max): 2.46% … 42.85%
Time (median): 152.645 ms ┊ GC (median): 2.56%
Time (mean ± σ): 157.173 ms ± 28.157 ms ┊ GC (mean ± σ): 7.05% ± 9.68%
▄ ██▁ █▁ ▄▄
█▁▆▁▁▆▁▁▁███▆██▁██▁▁▆▁▁▁▁▆▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆▁▁▁▁▆ ▁
125 ms Histogram: frequency by time 259 ms <
Memory estimate: 158.27 MiB, allocs estimate: 274.
julia> @benchmark innerjoin($df, $df2, on=:pv2, makeunique=true)
BenchmarkTools.Trial: 34 samples with 1 evaluation.
Range (min … max): 119.717 ms … 252.698 ms ┊ GC (min … max): 1.92% … 44.18%
Time (median): 143.639 ms ┊ GC (median): 1.02%
Time (mean ± σ): 147.490 ms ± 26.220 ms ┊ GC (mean ± σ): 6.05% ± 10.18%
▁ █ ▃▆▁ ▁
█▁▁▁▁▄█▇▄▁███▁▁█▇▁▄▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▄ ▁
120 ms Histogram: frequency by time 253 ms <
Memory estimate: 169.72 MiB, allocs estimate: 274.
Apart from these differences working with pooled vectors should have no
user visible differences in comparison to working with normal vectors.
CategoricalArrays.jl
The CategoricalArray type is also performing data compression, but its main
use case is when you want to represent your data as categorical in a statistical
sesnse (both ordered and unordered).
However, let me start with showing you the compression effect and grouping
and joining speedups, as they are the same as for PooledArrays.jl.
julia> using CategoricalArrays
julia> cv = categorical(v)
4000000-element CategoricalArray{String,1,UInt32}:
"a"
"b"
"c"
"d"
"a"
⋮
"a"
"b"
"c"
"d"
julia> Base.summarysize(cv)
16000612
julia> cv2 = categorical(v, compress=true)
4000000-element CategoricalArray{String,1,UInt8}:
"a"
"b"
"c"
"d"
"a"
⋮
"a"
"b"
"c"
"d"
julia> Base.summarysize(cv2)
4000564
julia> df = DataFrame(;v, cv, cv2)
4000000×3 DataFrame
Row │ v cv cv2
│ String Cat… Cat…
─────────┼────────────────────
1 │ a a a
2 │ b b b
3 │ c c c
⋮ │ ⋮ ⋮ ⋮
3999998 │ b b b
3999999 │ c c c
4000000 │ d d d
3999994 rows omitted
julia> @benchmark groupby($df, :v)
BenchmarkTools.Trial: 54 samples with 1 evaluation.
Range (min … max): 72.757 ms … 109.888 ms ┊ GC (min … max): 0.00% … 4.32%
Time (median): 92.208 ms ┊ GC (median): 3.27%
Time (mean ± σ): 92.917 ms ± 11.122 ms ┊ GC (mean ± σ): 3.53% ± 2.48%
▆ ▂ ▆ █
▄▄▁▁▁▁▁▄▁▄▄▆▆▁▁▄▄▁▄▁▄▆█▆▁▁▁▁▁▆▁▄▆██▄▁▁▁▄▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▆█▁█ ▁
72.8 ms Histogram: frequency by time 108 ms <
Memory estimate: 125.04 MiB, allocs estimate: 33.
julia> @benchmark groupby($df, :cv)
BenchmarkTools.Trial: 714 samples with 1 evaluation.
Range (min … max): 3.930 ms … 15.026 ms ┊ GC (min … max): 0.00% … 4.58%
Time (median): 4.474 ms ┊ GC (median): 0.00%
Time (mean ± σ): 7.007 ms ± 3.824 ms ┊ GC (mean ± σ): 5.99% ± 12.26%
▃ █▇ ▄▁ ▁ ▅ ▂ ▄
█▇██▆▇▄▁▄▁▁▁▁▄▆▅▄███▁▁▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▆▅▁▆▆▁▁▅▅▁█▁█▇█▇█ ▇
3.93 ms Histogram: log(frequency) by time 14.1 ms <
Memory estimate: 30.52 MiB, allocs estimate: 57.
julia> @benchmark groupby($df, :cv2)
BenchmarkTools.Trial: 634 samples with 1 evaluation.
Range (min … max): 3.506 ms … 22.074 ms ┊ GC (min … max): 0.00% … 0.00%
Time (median): 5.651 ms ┊ GC (median): 0.00%
Time (mean ± σ): 7.881 ms ± 4.817 ms ┊ GC (mean ± σ): 6.49% ± 13.01%
▁█▆▂▂ ▃▃▂ ▄▂ ▁ ▄ ▄▁
████████████▆▇▅▆█▆█▇▅▁▁▅▅▆▄▇▅▆█████▇▄▅▅▆▅▄▆▅▅▁▇▄▅▆▆▇▅▇▆▆▄▄ ▇
3.51 ms Histogram: log(frequency) by time 21.2 ms <
Memory estimate: 30.52 MiB, allocs estimate: 57.
julia> df2 = df[1:4, :]
4×3 DataFrame
Row │ v cv cv2
│ String Cat… Cat…
─────┼────────────────────
1 │ a a a
2 │ b b b
3 │ c c c
4 │ d d d
julia> @benchmark innerjoin($df, $df2, on=:v, makeunique=true)
BenchmarkTools.Trial: 27 samples with 1 evaluation.
Range (min … max): 158.062 ms … 193.518 ms ┊ GC (min … max): 0.46% … 1.62%
Time (median): 187.180 ms ┊ GC (median): 1.66%
Time (mean ± σ): 186.249 ms ± 7.073 ms ┊ GC (mean ± σ): 1.89% ± 0.88%
█▂
▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▄▁▁▁▁▁▁▁▁▁▁▁▁▁▁███▄▄▆▄▄▄▄▁▁▆ ▁
158 ms Histogram: frequency by time 194 ms <
Memory estimate: 143.02 MiB, allocs estimate: 275.
julia> @benchmark innerjoin($df, $df2, on=:cv, makeunique=true)
BenchmarkTools.Trial: 33 samples with 1 evaluation.
Range (min … max): 133.837 ms … 253.376 ms ┊ GC (min … max): 0.54% … 43.65%
Time (median): 152.783 ms ┊ GC (median): 1.83%
Time (mean ± σ): 154.100 ms ± 21.196 ms ┊ GC (mean ± σ): 5.68% ± 8.33%
█▂ ▅ ▅▂
▅▁██▅██▅▅████▁▅▁▁▁▁▁▁▁▁▁▅▁▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅ ▁
134 ms Histogram: frequency by time 253 ms <
Memory estimate: 158.27 MiB, allocs estimate: 284.
julia> @benchmark innerjoin($df, $df2, on=:cv2, makeunique=true)
BenchmarkTools.Trial: 31 samples with 1 evaluation.
Range (min … max): 118.869 ms … 292.135 ms ┊ GC (min … max): 0.63% … 39.85%
Time (median): 149.992 ms ┊ GC (median): 2.45%
Time (mean ± σ): 161.881 ms ± 35.070 ms ┊ GC (mean ± σ): 8.69% ± 10.37%
█
▃▁▁▁▁▃▃▁▁▃█▆▃▃▃▃▁▃▄▃▃▃▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▃▁▁▁▁▃ ▁
119 ms Histogram: frequency by time 292 ms <
Memory estimate: 169.72 MiB, allocs estimate: 284.
So what is the difference?
First, only a limited set of value types can be stored in categorical array:
julia> categorical([nothing])
ERROR: ArgumentError: CategoricalArray only supports AbstractString, AbstractChar and Number element types (got element type Nothing)
Second, when you retrieve values from categorical arrays they are wrapped in
a custom type:
julia> cv[1]
CategoricalValue{String, UInt32} "a"
You can extract out the underlying value using the unwrap function:
julia> unwrap(cv[1])
"a"
Why do we have this behavior? The reason is that values stored in the
categorical array are treated as categories, and the wrapped value is just a
label for some category.
Let us first check the order of levels in the cv vector and try to compare two
of its elements:
julia> levels(cv)
4-element Vector{String}:
"a"
"b"
"c"
"d"
julia> cv[1] < cv[2]
ERROR: ArgumentError: Unordered CategoricalValue objects cannot be tested for order using <. Use isless instead, or call the ordered! function on the parent array to change this
The order of levels will be reflected e.g. if you use FreqTables.jl freqtable
function on such a vector:
julia> using FreqTables
julia> freqtable(cv)
4-element Named Vector{Int64}
Dim1 │
──────┼────────
"a" │ 1000000
"b" │ 1000000
"c" │ 1000000
"d" │ 1000000
Let us make the cv vector ordered and change the order of levels:
julia> ordered!(cv, true);
julia> levels!(cv, ["d", "b", "a", "c"]);
Now try doing the same operations:
julia> levels(cv)
4-element Vector{String}:
"d"
"b"
"a"
"c"
julia> cv[1] < cv[2]
false
julia> cv[1], cv[2]
(CategoricalValue{String, UInt32} "a" (3/4), CategoricalValue{String, UInt32} "b" (2/4))
julia> freqtable(cv)
4-element Named Vector{Int64}
Dim1 │
──────┼────────
"d" │ 1000000
"b" │ 1000000
"a" │ 1000000
"c" │ 1000000
As you can see freqtable respects the ordering of levels. Also when we compared
cv[1] vs cv[2] the order is respected (i.e. since level "a" is in position
3, and level "b" is in position 2 the comparison returns false).
Conclusions
In summary, most of the time using PooledArrays.jl is what you will need if you
only require to save memory when working with large data. However, if you want
to process data that is categorical in statistical sense use
CategoricalArrays.jl. There is some mental overhead of working with categorical
values as there is a special CategoricalValue type that you need to learn how
to work with. However, the benefit is that the information about levels and
their ordering is retained and can be used if you do operations on elements
of categorical vector.
How to call Julia code from Python
Re-posted from: https://blog.esciencecenter.nl/how-to-call-julia-code-from-python-8589a56a98f2?source=rss----ab3660314556--julia
A three-part series on achieving high performance with high-level code
By Abel Soares Siqueira and Faruk Diblen
Target audience
This is the first post in a three-part series about achieving high performance with high-level code. This series is aimed at people working with Python who needs better performance but prefers not to develop a low-level performant library.
Introduction
Having recently joined the Netherlands eScience Center after working with research using the Julia language for seven years, I was excited to highlight some of its cool features. At the eScience Center, many of our engineers use Python, some use C or C++, and in some cases, we see Python calling C++ code, to speed up the code. This was the perfect opportunity to introduce Julia’s interoperability with Python and to investigate whether we could achieve comparable speed by calling Julia code in Python. This means that for situations where Python’s performance is not sufficient, we can speed it up with another high-level language, avoiding the use of a low-level language like C++. This series of three blog posts will investigate these topics.
In this first post, we will learn how to call a Julia code from Python. We will set up the environment and show some examples. In the second post, we will take a problem that was solved by using Python in combination with C++ to speed up the code. We will replace the C++ code with a Julia code and compare the performance. In the third post, we will solve the same problem in Julia, optimize the Julia code to reach its maximum performance and compare it with the implementation in the second post.
What is Julia, and how does it compare to Python?
What is Julia? Julia is a high-performance, high-level programming language. It was created a few years ago with the ambitious goal of being fast with a high-level syntax, and it has been mostly successful. It can, in a few cases, reach the speed of low-level programming languages like C. For more information, the julialang.org site is a great first stop.
One of the most frequently asked questions is: “how does it compare to Python or some other programming language in terms of performance?”. The short answer: Julia is generally faster than Python and many other programming languages.
The performance of a Python code can be optimized, but even the optimized code usually underperforms compared to a Julia version of the same code. The performance increase of a Python code can be achieved in a few ways, but a frequent one is to call a code written in a low-level language, such as C, C++ and Fortran, like NumPy which does its calculations mostly in those low-level languages. What is less common, but also possible, is to call Julia from Python. In this post, we are going to show you how to do that!
Before we forget, all the code used in this post can be found in our GitHub repository. We have also created a Docker image that includes a ready-to-use environment to run both Julia and Python. To run that environment with Python 3.10 and Julia 1.6, install Docker and run the following in your terminal:
$ docker pull abelsiqueira/python-and-julia:py3.10-jl1.6
$ docker run -it exec abelsiqueira/python-and-julia:py3.10-jl1.6 /bin/bash
Preparation
In the following steps, we will configure our system to execute Julia code from Python. To learn more about this topic, the documentation for the packages we describe below is a great starting point. You will need four things:
- Python distribution compiled with shared libpython option. There are workarounds, but this is the most straightforward way.
- Julia, the executable that runs the Julia language.
- PyCall, the Julia package that defines the conversions between Julia and Python.
- PyJulia, the Python package to access Julia from Python.
We are going to go through the installation and configuration of these steps on a Linux system. It will be very similar on MacOS or with WSL for Windows once the required tools are installed.
Step 1: Python with shared libpython
To check whether the Python distribution is compiled with –enable-shared option, we run:
ldd $(which python3) | grep libpython
If the output is something like:
libpython3.10.so.1.0 => /usr/local/lib/libpython3.10.so.1.0 (0x00007f567e548000)
… then we are good to go! If we get nothing, that means that the Python distribution has not been compiled with the desired flags. In this case, we can compile our own Python distribution with the flag –enable-shared, which takes some time but is mostly straightforward. This Dockerfile has the instructions. Remember that if you just want to test it out, you can run the Docker image as mentioned in the previous section.
Step 2: Julia and PyCall
Now, we will install Julia. We recommend using jill, a script I created, which downloads and installs a specific version of Julia, but Julia can also be installed via the official binaries or package managers. In this post, we use version 1.6.5, which is the current Long Term Support version at the time of writing. Most likely this will work with a newer version as well. To install Julia 1.6.5 using jill, we run:
$ wget https://raw.githubusercontent.com/abelsiqueira/jill/v0.4.0/jill.sh
$ sudo bash jill.sh -y -v 1.6.5
Now, we will install PyCall and configure it to use the correct Python version. We start Julia by running julia in the terminal, and then we set the ENV["PYTHON"] variable:
$ julia
julia> ENV["PYTHON"] = "PATH/TO/python"
Here we use the full path to Python’s executable. In our case, it is the Python distribution we compiled from the source code. You could change the path according to your configuration.
Now, we will install PyCall using Pkg, Julia’s package manager:
julia> using Pkg
julia> Pkg.add("PyCall")
Step 3: PyJulia
As the last step, we must install the Python package to talk with Julia. First, use pip, Python’s package manager, to install the package PyJulia — remember to use the same Python passed to ENV["PYTHON"]:
$ python3 –m pip install julia
To finalize configuring the communication between Julia and Python, we run the following in the Python interpreter:
$ python3
>>> import julia
>>> julia.install()
If we had more than one Julia version on our system, we could specify it with an argument:
>>> julia.install(julia='julia-1.6.5')
We test the installation running the following in the Python interpreter run:
>>> from julia import Main
>>> Main.eval('[x^2 for x in 0:4]')
Showcasing PyJulia
Basics
- To use a Julia module, use from julia import MODULE
- To evaluate a command, import Main and use Main.eval("…")
- To create and use variables, use Main.VARIABLE
- To install Julia packages, import Pkg and use Pkg.add("Package")
- Use %load_ext julia.magic to add a IPython’s magic command called %julia. Just prepend %julia to Julia commands. In this case, use $var to access python variables
Example: Linear Algebra
In this short example, we can see one of the strengths of Julia syntax for Linear Algebra. A random linear system is created and solved. The result is checked with NumPy, so we can see the compatibility.
We have chosen to define A, b and x in three different ways, to show the different syntaxes. The definition of A occurs completely inside the eval block. The variable A is created and is available inside the Julia scope, or as Main.A. The definition of b uses the Main.b access directly and uses the result of Main.eval. Finally, %julia is the magic IPython command to simply use Julia syntax directly.
We can quickly compare the timing of solving the system with Julia’s backslash command and Numpy’s linalg.solve:
Example: Automatic differentiation
The next example installs and uses the package called ForwardDiff, which performs automatic differentiation. ForwardDiff defines a Julia type called Dual internally, so we can’t use it with Python functions because Python functions are not compatible with that type. However, we can define Julia functions and use them.
The local minimum of the quadratic occurs at 2.5, so the derivative at 2.5 is 0.0.
Another, more interesting interaction is below, in which we create a function g inside Julia, and define functions for its derivatives there. Then we create a Python function with the Taylor expansion around the value a. Furthermore, we use Matplotlib, Python’s plotting library to visualize the results coming from Julia. Pretty neat, right?
Next episodes
Now that we can call Julia code in Python, we are prepared to move to our next adventure: improve the speed of a Python code by calling Julia from it. Follow our medium account to get notified when Part 2 goes live.
Many thanks to our proofreaders and reviewers, Elena Ranguelova, Jason Maassen, Jurrian Spaaks, Patrick Bos, Rob van Nieuwpoort, Stefan Verhoeven, and Veronica Pang.
How to call Julia code from Python was originally published in Netherlands eScience Center on Medium, where people are continuing the conversation by highlighting and responding to this story.