Author Archives: Great Lakes Consulting

How to Integrate Julia Code Within a Python Program

By: Great Lakes Consulting

Re-posted from: https://blog.glcs.io/julia-python-juliacall

This post was written by Steven Whitaker.

Ever wish your Python code could run fasteron heavy calculations or simulations?With juliacall,you can call Julia straight from Pythonand instantly access blazing-fast performanceand powerful scientific libraries,all without rewriting your existing code.Supercharge your Python workflows today andelevate your data science and engineering projectsto new heights!

In this Julia for Devs post,learn step-by-step how to install andutilize juliacall,enabling you toboost critical code performanceeffortlessly,without rewriting your entire project.Unlock the powerful combination ofPythons vast ecosystemand Julias speed,making it easy to experiment,optimize,or gradually migrate key components.

Let’s dig in!

Installing juliacall

Installation is a breeze,all you need is

pip install juliacall

You can test your installationby running the following in Python:

from juliacall import Main as jl

The first time this runs,it will install the Julia packagesneeded for communicatingbetween Julia and Python.

Then you can try it out:

import numpy as npA = np.array(jl.rand(5, 3))x = np.array(jl.randn(3))y = A @ x

Great,Julia-Python interoperabilityworks for this small example!Now let’s seehow we can extend thisto a larger example.

Calling Custom Code

In practice,we might have written some custom code in Juliathat we want to integrateinto our Python workflow.Let’s walk through the processof this integration.

Julia Code

Typically,the Julia code will be organizedinto a package,including its own package environmentand dependencies.

We’ll work with an examplethat runs a simulationusing OrdinaryDiffEq.jl and StaticArrays.jl.The example packagehas the following directory structure:

JuliaExample Project.toml src     JuliaExample.jl

The Project.toml has the following content:

name = "JuliaExample"uuid = "0b6476de-1cea-499f-93be-749bc74a9c07"authors = ["Author Name <[email protected]>"]version = "0.1.0"[deps]OrdinaryDiffEq = "1dea7af3-3e70-54e6-95c3-0bf5283fa5ed"StaticArrays = "90137ffa-7385-5640-81b9-e52037218182"

And JuliaExample.jl contains:

module JuliaExampleusing OrdinaryDiffEq: ODEProblem, Tsit5, solveusing StaticArrays: SVectorstruct Params    ::Float64    ::Float64    ::Float64endfunction f(u, p, t)    (; , , ) = p    dx =  * (u[2] - u[1])    dy = u[1] ( - u[3]) - u[2]    dz = u[1] * u[2] -  * u[3]    return SVector(dx, dy, dz)endfunction simulate(u0, t_start, t_end, , , )    u0 = SVector{3, Float64}(u0)    tspan = (t_start, t_end)    p = Params(, , )    prob = ODEProblem{false}(f, u0, tspan, p)    sol = solve(prob, Tsit5())endend

Python Code

We have our custom Julia code,so now let’s seewhat our Python workflow looks likethat calls out to Julia.

We’ll have our code organizedin the following directory structure:

python_example pyproject.toml scripts    run.py src     python_example         __init__.py         analysis.py

The main functionality of our Python codeis in analysis.py:

from juliacall import Main as jljl.seval("using JuliaExample: simulate")import numpy as npimport matplotlib.pyplot as pltdef simulate(*args):    result = jl.simulate(*args)    t = np.array(result.t)    sol = np.array([np.array(u) for u in result.u])    return t, soldef plot_results(t, sol):    plt.figure(figsize=(10, 6))    labels = ["x", "y", "z"]    colors = ["tab:blue", "tab:orange", "tab:green"]    for i in range(3):        plt.plot(t, sol[:, i], label=labels[i], color=colors[i])    plt.xlabel("Time [s]")    plt.ylabel("Value")    plt.title("Solution Components Over Time")    plt.legend()    plt.grid(True)    plt.tight_layout()    plt.show()

This code provides two functions:one for calling out to Juliato run a simulation,and another for plotting the simulation results.

Let’s break down some of this codeto see how Julia is incorporated:

  • from juliacall import Main as jl
    We saw this earlier;this is how to load juliacall.
  • jl.seval("using JuliaExample: simulate")
    Here,we load our Julia package,specifically bringing the function simulateinto scope.
  • The Python function simulatecalls the Julia simulate,passing along all its inputs: 
    result = jl.simulate(*args)
    The Python functionthen does some processingto convert the Julia resultsinto something more easily utilizedby further Python processing.

This functionality is exercisedin the run.py script:

from python_example import simulate, plot_resultsimport numpy as npu0 = np.array([1, 0, 0])t_start = 0t_end = 100alpha = 10beta = 28gamma = 8/3t, sol = simulate(u0, t_start, t_end, alpha, beta, gamma)plot_results(t, sol)

Finally,for completeness,here’s __init__.py:

from .analysis import simulate, plot_results

Finding Julia

We have all our code set up,so now we need Pythonto be able to find the Julia codeso we can call out to it.In other words,we need the Julia package environmentused by juliacallto have JuliaExample as a dependency.

We can accomplish thisby creating a juliapkg.json filein our Python project directory(i.e., python_example/juliapkg.json).The file should contain the following JSON:

{    "packages": {        "JuliaExample": {            "uuid": "0b6476de-1cea-499f-93be-749bc74a9c07",            "path": "path/to/JuliaExample",            "dev": true        }    }}

Note that the uuid hereneeds to match the uuidin JuliaExample/Project.toml.And the "path" and "dev": true fields are necessarybecause our Julia package exists locally on our machine;it is not a registered Julia package.See the juliacall docsfor more information about juliapkg.json.

Putting It Together

Now we have all the components we need:Python code to run,Julia code to call out to,and juliapkg.json to tell juliacallwhere to find our Julia code.

So, what happens when we run run.py?

The first time it is run,juliacall will set up the Julia package environment,installing the dependencies of JuliaExample.Then, the script proceeds to run the simulation(calling out to Julia to do so)and plot the results:

Simulation results

Awesome,we now have a working exampleshowing how to call out to Juliafrom a Python project!

Summary

In this post,we saw how to install and use juliacallto call out to Julia from within Python.We looked at a trivial exampleas well as a more realistic exampleof integrating custom Julia codeinto a Python project.

What custom Julia codedo you want to integrateinto your Python projects?Contact us, and we can make it happen!

Additional Links

  • PythonCall.jl
    • One cool thing about juliacallis it is maintained in the same GitHub repoas PythonCall,which is the recommended wayto call Python code from Julia.
  • GLCS Modeling & Simulation
    • Connect with us for Julia Modeling & Simulation consulting.
]]>

Simplifying Julia Package Integration with Extensions

By: Great Lakes Consulting

Re-posted from: https://blog.glcs.io/package-extensions

This post was written by Steven Whitaker.

The Julia programming languageis a high-level languagethat is known, at least in part,for its outstanding composability.Much of Julia’s composabilitystems from its multiple dispatch,which allows functions written in one packageto work with objects from another packagewithout either package needing to depend on or even know about the other.(See another blog post for more details.)

Sometimes, however,it is useful for a packageto be able to extend its functionsto provide additional functionalitywhen given an object of a specific typefrom another package.One way to do sois to add the other package as an explicit dependencyso that its type is availablefor the first package to useto define a specific method for it.

But what if the package can function just finewithout the additional functionality?What if the extra functionalityisn’t integral to what the package doesand only appliesif the userwants to work with objectsof that specific type?In this case,it doesn’t make much senseto make the other package a direct dependency,because then every userpays the price of extra package load timefor functionality that only some users actually want.

The solution is package extensions.A package extension is codethat gets loaded conditionally,depending on what other packagesthe user has explicitly loaded.In other words,when a user loads both the packageand the dependency the extension depends on,the extension gets loaded automatically.This way,users who want to use the packagecan do so without the added dependency,while users who want the extra functionalitycan load the dependency themselves.

In this post,we will learn about some package extensionsthat exist in the Julia package ecosystem.We will also learn how to write a package extensionand how to load the extension.

This post assumes you are familiarwith the structure of a Julia package.If you need to learn more,check out our post on creating Julia packages.

Package Extensions in the Wild

Writing a Package Extension

To create a package extension,one needs to create a modulethat adds method definitionsto functions from one of the packages(either the package being extendedor the package that triggers loading the extension)that dispatch on types from the other package.This module will live in the ext directoryof the package being extended.Additionally,the extended package’s Project.tomlneeds to be updatedto inform the package managerof the existence of the extensionand when to load it.

Let’s look at a concrete example.

Example Package to Extend

This example will build on a custom package called Averages.jlthat we discussed in our blog post on testing Julia packages.The package code is as follows:

module Averagesusing Statistics: meanexport compute_averagecompute_average(x) = (check_real(x); mean(x))function compute_average(a, b...)    check_real(a)    N = length(a)    for (i, x) in enumerate(b)        check_real(x)        check_length(i + 1, x, N)    end    T = float(promote_type(eltype(a), eltype.(b)...))    average = Vector{T}(undef, N)    average .= a    for x in b        average .+= x    end    average ./= length(b) + 1    return a isa Real ? average[1] : averageendfunction check_real(x)    T = eltype(x)    T <: Real || throw(ArgumentError("only real numbers are supported; unsupported type $T"))endfunction check_length(i, x, expected)    N = length(x)    N == expected || throw(DimensionMismatch("the length of input $i does not match the length of the first input: $N != $expected"))endend

Creating the Extension

For this example,we will create an extensionthat implements additional functionality for DataFrames.These are the tasks we need to doto implement the extension:

  1. Create the extensionat Averages/ext/AveragesDataFramesExt.jl.Note that this follows the naming convention for extensions:<PackageName><NameOfPackageThatTriggersExtension>Ext.Inside this file,we create a module called AveragesDataFramesExt(same name as the file)and put the code we want to be includedwhen Averages.jl and DataFrames.jl are loaded together:

    module AveragesDataFramesExtimport Averagesusing Averages: compute_averageusing DataFrames: All, DataFrame, combinefunction Averages.compute_average(df::DataFrame)    @info "Running code in AveragesDataFramesExt!"    df_avg = combine(df, All() .=> compute_average)    return df_avgendend

  2. Add [weakdeps] and [extensions] sectionsto the Project.toml of Averages.jl.(See our previous blog post for the original Project.toml.)In [weakdeps],specify DataFrames.jl and its UUID,and, in [extensions],specify our extension (AveragesDataFramesExt)and its dependency (DataFrames.jl).The UUID of DataFrames.jl can be foundin DataFrames.jl’s Project.toml.

    Here’s the updated Project.toml for Averages.jl:

    name = "Averages"uuid = "1fc6e63b-fe0f-463a-8652-42f2a29b8cc6"version = "0.1.0"[deps]Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"[weakdeps]DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"[extensions]AveragesDataFramesExt = "DataFrames"[extras]Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"[targets]test = ["Test"]

    (Note that,just as compatible versions of the [deps] packagescan be specified in a [compat] section,so too can the compatible versions of the [weakdeps] packagesbe specified.)

Using the Extension

First,let’s see what happensif we try this without the extension:

julia> compute_average(DataFrame(a = [1, 2], b = [3.0, 4.0]))ERROR: ArgumentError: only real numbers are supported; unsupported type AnyStacktrace: [1] check_real(x::DataFrame)   @ Averages /path/to/Averages/src/Averages.jl:34 [2] compute_average(x::DataFrame)   @ Averages /path/to/Averages/src/Averages.jl:7 [3] top-level scope   @ REPL[5]:1

So,now let’s see if the extensionallows this function call to work.

To use the extension,install and load Averages.jl and DataFrames.jl(for Averages.jl, use the dev command,i.e., pkg> dev /path/to/Averages)and then call compute_average:

julia> using Averages, DataFramesjulia> compute_average(DataFrame(a = [1, 2], b = [3.0, 4.0]))[ Info: Running code in AveragesDataFramesExt!12 DataFrame Row  a_compute_average  b_compute_average      Float64            Float64   1                1.5                3.5

Nice, it works!And with that,we have an example package extensionthat illustrates how to implement your own.

And remember,a user of Averages.jlwill only incur the cost of loading AveragesDataFramesExtif they load DataFrames.jl.For more details,see the slide annotationsin this screenshot from JuliaCon 2023:

JuliaCon 2023 package extensions talk

(See also the full talk on package extensionsfor even more details.)

Note: Where Should an Extension Live?

By the way,if you’re wondering why we put the extension in Averages.jlinstead of DataFrames.jl,the answer isthat it doesn’t really matterbecause the user experiencewill be the same regardless.If you still want some rules to follow,I’m not aware of any Julia best-practicesin this regard,but here are some rules that make sense to me:

  • If one of the two packages in questiondefines an interface,the extension should go in the packagethat implements the interface.
  • Otherwise,put the extension in the packagethat owns the functionsthat are being extended.In our example,we extended the compute_average function.Since this function is defined in Averages.jl,we put the extension in Averages.jl.
  • An exception to the previous ruleis if getting the new functionality rightrequires a good understandingof the internals of the new data typethat’s being dispatched on,in which case the extensionshould belong in the packagethat defines the type.For example,if compute_average was super complicatedfor some reasonwhen working with DataFrames,it would make sense for those with the needed expertise(i.e., the developers of DataFrames.jl)to own and maintain the extension.

Summary

In this post,we listed some real Julia packagesthat have their own package extensions.We also demonstrated creating our own extensionfor an example packageand showed how to use the extension’s code.

What package extensions have you found useful?Let us know in the comments below!

Additional Links

]]>

Discover the Key Features and Updates in Julia 1.12

By: Great Lakes Consulting

Re-posted from: https://blog.glcs.io/julia-1-12

This post was written by Steven Whitaker.

A new version of the Julia programming languagewas just released!Version 1.12 is now the latest stable version of Julia.

This release is a minor release,meaning it includes language enhancementsand bug fixesbut should also be fully compatiblewith code written in previous Julia versions(from version 1.0 and onward).

In this post,we will check out some of the features and improvementsintroduced in this newest Julia version.Read the full post,or click on the links belowto jump to the features that interest you.

If you are new to Julia(or just need a refresher),feel free to check out our Julia tutorial series,beginning with how to install Julia and VS Code.

Improved Quality of Life with Redefinable Types

Julia 1.12 introduces a major quality of life improvementfor package developmentby allowing types to be redefined.To illustrate,prior to Julia 1.12:

julia> struct A endjulia> struct A           x::Int       endERROR: invalid redefinition of constant Main.AStacktrace: [1] top-level scope   @ REPL[2]:1

But in Julia 1.12,no error is thrown!Note, however,that any objects of the old typewill still be of the old type—they don’t automatically updateto conform to the new type definition.For example:

julia> struct A endjulia> a1 = A()A()julia> struct A           x::Int       endjulia> a2 = A(1)A(1)julia> a1@world(A, 38513:38515)()

Note that after redefining A,a1 is denoted as being of type Afrom a previous so-called “world age”.

Importantly,when defining a method that dispatches on a type,it will be defined for the version of the typethat existed at the time of method definition.Continuing from the previous example:

julia> f(a::A) = println("hello")f (generic function with 1 method)julia> f(a2)hellojulia> f(a1) # Method doesn't exist for the old typeERROR: MethodError: no method matching f(::@world(A, 38513:38515))The function `f` exists, but no method is defined for this combination of argument types.Closest candidates are:  f(::A)   @ Main REPL[6]:1Stacktrace: [1] top-level scope   @ REPL[8]:1julia> struct A # Update the type           x::Float64       endjulia> f(A(2.0)) # Method doesn't exist for the new typeERROR: MethodError: no method matching f(::A)The function `f` exists, but no method is defined for this combination of argument types.Closest candidates are:  f(::@world(A, 38516:38521))   @ Main REPL[6]:1Stacktrace: [1] top-level scope   @ REPL[18]:1

In other words,if updating a type,be sure to update methods as well.

Often, however,when someone is working with a typeand many methods that use the type for dispatch,they are developing a package.Package development works differentlythan just working in the REPL,so let’s see how package developmentis affected by the ability to update types.

First,without proper tooling,Julia package development can be a slog,regardless of whether types (or anything else)can be updated.This is because normally,after a package is loaded,changes made to the package’s source codedon’t take effectuntil Julia is restartedand the package is loaded again.In other words,the whole Julia runtimehas to be started up againand code has to be reloadedeven if just a single function in the package was updated.

Fortunately,Revise.jl exists.Revise.jl provides massive quality of life improvementsfor package developersby allowing changes to the package source codeto take effect immediately,without restarting Julia.

The biggest caveat?Revise.jl couldn’t handle changes to struct definitions.So,if you decide a struct in your package needs an extra field,you’re out of luck,you have to restart Julia.And then if you decide the struct actually didn’t need that field,too bad again,restart Julia.

But that all changes with Julia 1.12!This release adds a mechanismfor redefining typesthat Revise.jl hooks into,removing arguably the most significant remaining pain pointof Julia package development.The number of times a developer needs to restart Juliawill decrease significantlywith Julia 1.12,allowing for greater productivity.

New Way to Fix Function Arguments

Julia 1.12 introduces the Fix structthat can be used to fix function arguments.(Here, “to fix” means “to set”,not “to correct”.)Think of it as another way to define a closure.

One of the main uses of Fixis purely for convenience,particularly when creating closuresof functions with many inputs.For example,suppose we have the following (nonsensical) physical model:

function dynamics(velocity, resistance, gravity, position, friction, length, mass)    return velocity + resistance + gravity + position + friction + length + massend

Now,suppose we have another functionthat computes(or accepts as input)a particular value for massand then creates a new functionthat computes dynamics with mass fixed.Previously,an anonymous function typically would be used,so let’s compare using an anonymous functionto using Fix:

mass = 30 # Some computed or given value# Using an anonymous function.# Note that the `let mass = mass` is essential# to guarantee Julia doesn't box `mass`# (see <https://docs.julialang.org/en/v1/manual/performance-tips/#man-performance-captured>).# (Though also note that sometimes boxing does not occur# even if the `let` block is omitted.)dynamics_with_mass = let mass = mass    (velocity, resistance, gravity, position, friction, length) -> dynamics(velocity, resistance, gravity, position, friction, length, mass)end# Using `Fix`.# The `7` indicates that we want to fix the 7th argument.dynamics_with_mass_fix = Base.Fix{7}(dynamics, mass)

As you can see,using Fix is much more concise.But bothdynamics_with_massand dynamics_with_mass_fixact as a function of six argumentsand compute dynamics with mass fixedto a specific value.

Next,suppose we want to compute the derivative of dynamicsas a function of velocityfor fixed values of the other parameters.In this case,there are six arguments that need to be fixed.Fix allows for fixing just one argument,but Fixes can be nestedto fix multiple arguments.Let’s compare using an anonymous functionto using Fix:

# Fixed values, either computed or given.resistance = 2gravity = 9.8position = 0friction = 0length = 0.5mass = 30# Using an anonymous function.f = let resistance = resistance,        gravity = gravity,        position = position,        friction = friction,        length = length,        mass = mass    velocity -> dynamics(velocity, resistance, gravity, position, friction, length, mass)end# Using `Fix`.f_fix = Base.Fix{2}(Base.Fix{2}(Base.Fix{2}(Base.Fix{2}(Base.Fix{2}(Base.Fix{2}(dynamics, resistance), gravity), position), friction), length), mass)# Using `Fix` with piping for nicer formatting.pipefix(::Val{N}, x) where {N} = Base.Fix{2}(Base.Fix{N}, x)f_pipefix = Base.Fix{2}(dynamics, resistance) |>    pipefix(Val{2}(), gravity) |>    pipefix(Val{2}(), position) |>    pipefix(Val{2}(), friction) |>    pipefix(Val{2}(), length) |>    pipefix(Val{2}(), mass)

In this case,I would argue using an anonymous functionis clearer to read.However,Fix can have some performance advantagesover anonymous functions.For example,using Fix can reduce the need to compileanonymous functions that are used repeatedly.

For a lot more context and discussion,see the pull requestthat introduced Fix.

Progress Towards More Reasonable Executables

Perhaps surprisingly,an experimental featureis probably the most highly anticipated addition in Julia 1.12:a new command-line argument --trimthat enables a (currently experimental) featurethat removes unnecessary codefrom compiled executables.

Wait, Julia can generate standalone executables?Yes, though, again, it’s experimental,this time through a script called juliac.jl.(Surprisingly,the release notes say nothing about this script,though it is mentionedin a few threads on Julia Discourse,of which the most helpful to me was this thread.)

Let’s see how to use juliac.jlwith a simple program:

# main.jlfunction @main(args::Vector{String})::Cint    for arg in args        # Note the use of `Core.stdout` instead of `stdout`        # (which is used by default if an `io` argument is omitted).        println(Core.stdout, arg)    end    return 0end

This program simply prints to the consoleall its given command-line arguments.

To compile it into an executable,just run the following command:

julia ~/.julia/juliaup/julia-1.12.0+0.x64.linux.gnu/share/julia/juliac/juliac.jl --output-exe print_args --experimental --trim main.jl

(Note that I tested thiswith Julia installed via juliaupon Linux.The location of the juliac.jl scriptmay be different on your system.)

To verify it works:

$ ./print_args hello worldhelloworld

So,what exactly is the impact of --trim?Without the new (experimental) feature,the size of the print_args executablewas 206 MB on my computer.With --trim,that number improved by over 100x,dropping to just 1.6 MB!

Summary

In this post,we learned aboutsome of the new featuresand improvementsintroduced in Julia 1.12,including redefinable types,Fix,and --trim.Curious readers cancheck out the release notesfor the full list of changes.

What are you most excited aboutin Julia 1.12?Let us know in the comments below!

Additional Links

]]>