DataFrames.jl version 0.21 was a major release that introduced a number of
significant changes to DataFrames.jl API. The list is long, so
I briefly summarize here he most significant things in terms of functionality:
we now allow to select columns using strings (selection using Symbols is
still allowed);
a completely new design of API for working with columns of a data frame or
grouped data frame, covered by select/select!, transform/transform!,
and combine functions; it is consistent (so you learn it once and reuse
everywhere), more flexible, and has a better performance than the old one;
in particular two wrappers ByRow and AsTable have been added to API;
major enhancements to push! and append!, which allow an easy way to
digest heterogeneous data (varying element types, varying column sets)
into a data frame;
GroupedDataFrame now supports a fast lookup by grouping columns (so making
a GroupedDataFrame can be now seen as adding an index to a data frame)
filter/filter! are now fast using Pair-interface;
rules for pseudo-broadcasting (spreading single observations across multiple
rows) have been established and are consistently applied in all methods that
allow this operation.
All these changes combined mean that now all operations on data frames can be
expressed via function chaining (and you have a full control if you want to
make copies or perform operations in-place). There are many users who like this
style of expressing transformations made on data. If you want to go this way,
then probably you should consider learning one of the packages that makes
it easier to work with |> operator. There are many excellent alternatives
in the Julia ecosystem. Let me mention two Pipe.jl (easier) and Underscores.jl (more powerful, but harder to master).
After the release I got several questions about showing how things work in
practice. Therefore in this post I list tutorials that are currently available
and have been updated to show how DataFrames.jl v0.21 works.
In the Part II post (that I plan to prepare next week) I will show some new
material that was prepared under DataFrames.jl v0.21.
Tutorials for release 0.21
There are four sources of information about the functionality of DataFrames.jl
0.21 that you can check out (and I maintain them so that they should be
up to date):
Recently updated the materials about DataFrames.jl that I have presented
during JuliaCon2019 workshop. You will be able to find
there two notebooks that include worked examples how you can process
real-life data sets.
I hope these materials will be useful for exploring the latest release
of DataFrames.jl!
When you work on a project using the Julia language most likely you will use
some packages that are available in the Julia ecosystem. In particular, JuliaHub is a great place to look for packages that might be useful
for you.
Most likely the first thing you start doing is adding the package to your
default environment. This is the easiest thing to do, but has one significant
downside — Julia package ecosystem is evolving very fast. This, in particular,
means that during your project life cycle the versions of the packages provided
by their maintainers can go up and introduce breaking changes. In consequence
your code might suddenly stop working for no apparent reason.
In this post I have collected some practices I find useful to avoid such
problems. Even if you do not end up using the functionalities of the Julia
package manager I discuss on daily basis I think it is worth to be aware of
their existence.
All examples were tested under Julia 1.4.1.
For every project keep a separate project environment
This is a basic rule. Unless I do quick-and-dirty interactive calculations
I always create a project environment for my work.
Fortunately this is really easy. Just use generate command in the Julia
package manager. The steps to achieve it are easy:
start julia in the folder in which you want to create a project
press ] character to enter package manager mode
execute generate [target folder name] and press enter
press backspace to leave this mode
write exit()
Here is a screen shot of the session where I executed these steps:
When I use -e command Julia executes the commands that I pass and quits.
In this case I loaded the Pkg module and executed Pkg.generate function that
is a part of Pkg module API.
Let us check what is the contents of the test_project folder:
~$ cd test_project/
~/test_project$ ls -R
.:
Project.toml src
./src:
test_project.jl
~/test_project$
You can see that two files were created: a Project.toml file in the top-level
directory that specifies the dependencies of our project
and src/test_project.jl which is a placeholder for our code.
Let us quickly inspect their contents:
You can change the contents or rename the test_project.jl file in whatever
way you like, but do not touch Project.toml file as it contains the list
of dependencies of your project (currently there are none). Here
you can find the details of the specification of Project.toml file contents.
Now — a crucial step is that whenever you want to work with your project
always activate the project environment specified by Project.toml file
when starting Julia.
The easiest way to do it is to make sure that you are in the folder that
contains Project.toml file and write:
The --project=. part tells Julia to start the interpreter using Project.toml
file from the current working directory as a specification of your dependencies.
I this post I discuss how to make Julia automatically activate the
project environment in your current working directory on startup.
Now we can check that indeed we are in a correct project environment that is
empty:
In the next section I discuss how to add packages to your project.
When adding packages use preserve=PRESERVE_DIRECT keyword argument
You can add dependencies to your project using the Pkg.add function in the
project manager. You can find the list of the available options by running the
help for the Pkg.add function (I omit the output here as it is long).
It is crucial that the Pkg.add function has a preserve keyword argument that
tells Julia what it is allowed to do with the already installed packages. In this post I have discussed potential problems when multiple
packages you install have conflicting dependencies. Therefore my practice is to
use preserve=PRESERVE_DIRECT keyword argument. I do not want the packages that
my code depends on to change versions (and if there is some conflict generated
due to this restriction I prefer to get an error rather than a package version
change). However, I typically allow changing versions of recursive dependencies
as normally it should not affect my code.
Let me give an example how one can run such a command (in this case it does not
really matter if we add the preserve=PRESERVE_DIRECT keyword argument as we do
not have any packages installed):
If you are curious what Pipe.jl package does you can check it out here.
We are informed that Project.toml and Manifest.toml were updated.
Let us inspect their contents from within Julia as an exercise:
julia>print(read("Project.toml",String))name="test_project"uuid="d78710ad-1861-4169-903b-684d2f77c7fa"authors=["Bogumił Kamiński <[email protected]>"]version="0.1.0"[deps]Pipe="b98c9c47-44ae-5843-9183-064241ee97a0"julia>print(read("Manifest.toml",String))# This file is machine-generated - editing it directly is not advised[[Pipe]]git-tree-sha1="f11840ebaf295b39319c2750f158621a96173fc5"uuid="b98c9c47-44ae-5843-9183-064241ee97a0"version="1.2.0"julia>
In general Manifest.toml contains the exact specification of all dependencies
of our project (direct and recursive) and Project.toml lists only essential
information about direct dependencies.
Before we move on let me stress in what cases using preserve=PRESERVE_DIRECT
is most important. Assume you have worked on some project for some time already.
It had several packages as its dependencies. Now you decide that you need to add
some new package to its direct dependencies. The potential problem is that it is
possible (as you have worked on your project already for some time) that the
packages that you have installed previously have new versions available. Most
likely you do not want these packages to change their versions when you add
a new package as your code might stop working. This is exactly what preserve=PRESERVE_DIRECT keyword argument safeguards you against.
When updating packages use level=UPDATELEVEL_PATCH option
However, if you work on a project for some time the packages that you depend on,
might have released patches (e.g. bug fixes or documentation enhancements) that
you want to allow in your project.
You can achieve such an update by running Pkg.update(level=UPDATELEVEL_PATCH).
Here is an example of this command run:
In this case nothing happened as we have just installed that package. To check
out all the options of the Pkg.update function run its help. In particular, it is
OK to allow changing major or minor versions of your dependencies when running
the Pkg.update function, but remember that in this case your code might stop
working correctly, so if you do this please make sure to test that your code
produces the expected results after the update (and if it fails you can use the Pkg.undo() function to undo the latest change to the active project).
Final notes on using package manager
In this post I used the Pkg module API by calling functions. All this
functionality is also available via a package manager, which you enter by
pressing ] in the Julia command line. The names of the commands are the same,
and you can get help on them by writing e.g. help add.
Each package in Julia has a list of other packages it depends on.
They are specified in Project.toml file. Here is an
example from DataFrames.jl pacakge release 0.21.0:
The details how to read this file are given here. For this post
the important part is [compat] section (again here you can find
the details). This section informs Julia package manager which versions of other
packages are allowed to be installed with version 0.21.0 of DataFrames.jl.
Conflicting dependency requirements
The crucial thing is that when you install packages Julia package manager
analyzes the [compat] section of each package required to be installed in the
project and tries to find such a combination of package versions that is
possible to be active at the same time.
Sometimes it is impossible to find such a combination and you get an error
that starts with the words Unsatisfiable requirements detected for package.
This is a problematic situation, and here you have a description
how to read such a message, and here some tips how to fix them.
While getting an error is a critically problematic situation it has one positive
element: you know that something went wrong.
In this post we want to concentrate on a less critical case: you install several
packages, you get no errors, but for some strange reasons your code does not
work as expected. The likely situation is caused by the fact that the package
manager has decided that it is only possible to install some old version of a
package, that e.g. can have a different API than the latest version of the
package that you have implemented your code against.
From my experience such situations are quite common and hard to detect
especially if you have dozens of installed packages.
Let me give a simple example of such a situation. Recently I wanted to install
Plots.jl and GraphPlot.jl packages. Here is a dump of the steps I have made in
a clean environment:
And we see that multiple packages got downgraded to their earlier versions.
In the next two sections we will discuss: (a) how to detect such situations
automatically, and (b) how to diagnose what is the root cause of the problem.
Automatically detecting packages installed in a version that is not latest
Here is a short snippet that lists you the packages that are installed but
are not in their latest versions. It is probably not 100% proof for corner
cases of various configurations (this is what Julia package manager does, and
it is a piece of complex code) but shoul be good enough for typical use cases.
It produces us a dictionary of packages that are not installed in their latest
versions giving us a tuple indicating installed and latest version for each
such package.
As you can see we get an information that Plots.jl currently is not in its
latest version as expected. We already know that it was downgraded when forcing
to install GraphPlot.jl in version 0.4.2.
Let me give a few comments how the presented code works:
we run our function in the registries/General subdirectory of the first
entry of the DEPOT_PATH variable (this is where Julia package manager first
looks for installed packages);
the deps variable is a a dictionary of all dependencies of the current
project;
Registry.toml holds information about available packages in the registry,
we parse it and store dictionary with package information in the general_pkgs variable;
constrained is a dictionary that we will populate with information about
packages that are not in their latest version
we iteratively scan all packages in deps dictionary we are interested only
in packages that are direct dependencies (that were explicitly installed),
have a version (packages from standard library do not have a version but we
do not install them anyway), and that are present in general_pkgs dictionary
(this means that if you are using several depots or sone non-standard way
of installing packages this code is not guaranteed to be 100% correct);
if we have a package that is interesting for us we get a list of all its
available versions from Versions.toml that is stored in its directory,
we extract all the versions, convert them to VersionNumber type and select
a maximum version (note that VersionNumber type has a properly defned order
so this is safe to do);
finally we compare the installed and latest versions of the packages and if
they differ this information is stored and later returned.
Diagnosing the problems with package version constraints
We have learned that upgrading GraphPlot.jl package to version 0.4.2 caused us
dependency problems. Let us then find out what are its copat sepecifications.
Here is the code that does the job:
And we see that ColorTypes.jl and Colors.jl had to be downgraded when we
installed the 0.4.2 version of GraphPlot.jl (we can see in the listing above
that they were downgraded). And downgrading these two packages, in cascade,
made Julia package manager change the versions of other packages, Plots.jl in
particular.
In the code above note that in order to get a path of GraphPlot.jl file from
GraphPlot.jl package we first have to load it with using GraphPlot command.
Then in order to properly construct the toml_loc variable we have to go one
step up in the path with .. as GraphPlot.jl file is located in src
subdirectory in the package sources.
In order to resolve such problems systematically is best to submit a PR to the
packages that have outdated dependencies. This is exactly what I did here (and the PR got merged as of this writing).