Simplifying Julia Package Integration with Extensions

By: Great Lakes Consulting

Re-posted from: https://blog.glcs.io/package-extensions

This post was written by Steven Whitaker.

The Julia programming languageis a high-level languagethat is known, at least in part,for its outstanding composability.Much of Julia’s composabilitystems from its multiple dispatch,which allows functions written in one packageto work with objects from another packagewithout either package needing to depend on or even know about the other.(See another blog post for more details.)

Sometimes, however,it is useful for a packageto be able to extend its functionsto provide additional functionalitywhen given an object of a specific typefrom another package.One way to do sois to add the other package as an explicit dependencyso that its type is availablefor the first package to useto define a specific method for it.

But what if the package can function just finewithout the additional functionality?What if the extra functionalityisn’t integral to what the package doesand only appliesif the userwants to work with objectsof that specific type?In this case,it doesn’t make much senseto make the other package a direct dependency,because then every userpays the price of extra package load timefor functionality that only some users actually want.

The solution is package extensions.A package extension is codethat gets loaded conditionally,depending on what other packagesthe user has explicitly loaded.In other words,when a user loads both the packageand the dependency the extension depends on,the extension gets loaded automatically.This way,users who want to use the packagecan do so without the added dependency,while users who want the extra functionalitycan load the dependency themselves.

In this post,we will learn about some package extensionsthat exist in the Julia package ecosystem.We will also learn how to write a package extensionand how to load the extension.

This post assumes you are familiarwith the structure of a Julia package.If you need to learn more,check out our post on creating Julia packages.

Package Extensions in the Wild

Writing a Package Extension

To create a package extension,one needs to create a modulethat adds method definitionsto functions from one of the packages(either the package being extendedor the package that triggers loading the extension)that dispatch on types from the other package.This module will live in the ext directoryof the package being extended.Additionally,the extended package’s Project.tomlneeds to be updatedto inform the package managerof the existence of the extensionand when to load it.

Let’s look at a concrete example.

Example Package to Extend

This example will build on a custom package called Averages.jlthat we discussed in our blog post on testing Julia packages.The package code is as follows:

module Averagesusing Statistics: meanexport compute_averagecompute_average(x) = (check_real(x); mean(x))function compute_average(a, b...)    check_real(a)    N = length(a)    for (i, x) in enumerate(b)        check_real(x)        check_length(i + 1, x, N)    end    T = float(promote_type(eltype(a), eltype.(b)...))    average = Vector{T}(undef, N)    average .= a    for x in b        average .+= x    end    average ./= length(b) + 1    return a isa Real ? average[1] : averageendfunction check_real(x)    T = eltype(x)    T <: Real || throw(ArgumentError("only real numbers are supported; unsupported type $T"))endfunction check_length(i, x, expected)    N = length(x)    N == expected || throw(DimensionMismatch("the length of input $i does not match the length of the first input: $N != $expected"))endend

Creating the Extension

For this example,we will create an extensionthat implements additional functionality for DataFrames.These are the tasks we need to doto implement the extension:

  1. Create the extensionat Averages/ext/AveragesDataFramesExt.jl.Note that this follows the naming convention for extensions:<PackageName><NameOfPackageThatTriggersExtension>Ext.Inside this file,we create a module called AveragesDataFramesExt(same name as the file)and put the code we want to be includedwhen Averages.jl and DataFrames.jl are loaded together:

    module AveragesDataFramesExtimport Averagesusing Averages: compute_averageusing DataFrames: All, DataFrame, combinefunction Averages.compute_average(df::DataFrame)    @info "Running code in AveragesDataFramesExt!"    df_avg = combine(df, All() .=> compute_average)    return df_avgendend

  2. Add [weakdeps] and [extensions] sectionsto the Project.toml of Averages.jl.(See our previous blog post for the original Project.toml.)In [weakdeps],specify DataFrames.jl and its UUID,and, in [extensions],specify our extension (AveragesDataFramesExt)and its dependency (DataFrames.jl).The UUID of DataFrames.jl can be foundin DataFrames.jl’s Project.toml.

    Here’s the updated Project.toml for Averages.jl:

    name = "Averages"uuid = "1fc6e63b-fe0f-463a-8652-42f2a29b8cc6"version = "0.1.0"[deps]Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"[weakdeps]DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"[extensions]AveragesDataFramesExt = "DataFrames"[extras]Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"[targets]test = ["Test"]

    (Note that,just as compatible versions of the [deps] packagescan be specified in a [compat] section,so too can the compatible versions of the [weakdeps] packagesbe specified.)

Using the Extension

First,let’s see what happensif we try this without the extension:

julia> compute_average(DataFrame(a = [1, 2], b = [3.0, 4.0]))ERROR: ArgumentError: only real numbers are supported; unsupported type AnyStacktrace: [1] check_real(x::DataFrame)   @ Averages /path/to/Averages/src/Averages.jl:34 [2] compute_average(x::DataFrame)   @ Averages /path/to/Averages/src/Averages.jl:7 [3] top-level scope   @ REPL[5]:1

So,now let’s see if the extensionallows this function call to work.

To use the extension,install and load Averages.jl and DataFrames.jl(for Averages.jl, use the dev command,i.e., pkg> dev /path/to/Averages)and then call compute_average:

julia> using Averages, DataFramesjulia> compute_average(DataFrame(a = [1, 2], b = [3.0, 4.0]))[ Info: Running code in AveragesDataFramesExt!12 DataFrame Row  a_compute_average  b_compute_average      Float64            Float64   1                1.5                3.5

Nice, it works!And with that,we have an example package extensionthat illustrates how to implement your own.

And remember,a user of Averages.jlwill only incur the cost of loading AveragesDataFramesExtif they load DataFrames.jl.For more details,see the slide annotationsin this screenshot from JuliaCon 2023:

JuliaCon 2023 package extensions talk

(See also the full talk on package extensionsfor even more details.)

Note: Where Should an Extension Live?

By the way,if you’re wondering why we put the extension in Averages.jlinstead of DataFrames.jl,the answer isthat it doesn’t really matterbecause the user experiencewill be the same regardless.If you still want some rules to follow,I’m not aware of any Julia best-practicesin this regard,but here are some rules that make sense to me:

  • If one of the two packages in questiondefines an interface,the extension should go in the packagethat implements the interface.
  • Otherwise,put the extension in the packagethat owns the functionsthat are being extended.In our example,we extended the compute_average function.Since this function is defined in Averages.jl,we put the extension in Averages.jl.
  • An exception to the previous ruleis if getting the new functionality rightrequires a good understandingof the internals of the new data typethat’s being dispatched on,in which case the extensionshould belong in the packagethat defines the type.For example,if compute_average was super complicatedfor some reasonwhen working with DataFrames,it would make sense for those with the needed expertise(i.e., the developers of DataFrames.jl)to own and maintain the extension.

Summary

In this post,we listed some real Julia packagesthat have their own package extensions.We also demonstrated creating our own extensionfor an example packageand showed how to use the extension’s code.

What package extensions have you found useful?Let us know in the comments below!

Additional Links

]]>