Tag Archives: Julia

advent of code 2025

By: Domenic Di Francesco

Re-posted from: https://allyourbayes.com/posts/AoC/


TLDR

Advent of code is a website created by Eric Wastl, releasing a (increasingly challenging) programming puzzle each day. This year we will not be receiving the usual 25 days worth (I have never made it that far anyway), but it will instead be finishing on 12th December (see announcement on Reddit). This is a welcome change of pace for me.

Spoiler alert: super un-optimised, but hopefully readable Julia solutions ahead, so you may only want to look after you have had a go yourself. Let’s see how we go…

Play along: https://adventofcode.com


Day 1: a password from a decoy safe ?

So we have this safe, with a dial set to 50, and a sequence of turns to make, see below:

seq = ["L68", "L30", "R48", "L5", "R60", "L55", "L1", "L99", "R14", "L82"]

Here an “R” means we are turning the dial to the right, and an “L” means we are turning it to the left, followed by the number indicating how many turns. “L68” means turn the dial left 68 times. The dial wraps around, so if we turn left from 0, we go to 99, 98, and so on.

We are asked:

  • part 1: how many times do we end up on zero after a turn?
  • part 2: how many times do we turn through zero in total?

Floor division fld() will count how many times we lap around zero (and in which direction).

function get_passwords(seq::Vector{String}, dial_start::Int = 50, dial_mod::Int = 100)

    dial = dial_start; pass_zeros = 0; end_zeros = 0

    for turn  seq
        direction = turn[begin]; turns = parse(Int, turn[2:end])
        
        if direction == 'R'

            pass_zeros += fld(dial + turns, dial_mod)
            dial += turns

        elseif direction == 'L'

            # we need to catch the case where the dial starts at 0 and not count that first turn
            pass_zeros -= fld(dial - turns - 1, dial_mod) - fld(dial - 1, dial_mod)
            dial -= turns
            
        end

        dial = mod(dial, dial_mod)

        if dial == 0
            end_zeros += 1
        end
    end

    return (end_zeros=end_zeros, pass_zeros=pass_zeros)
end
get_passwords (generic function with 3 methods)

A nice opener, and I did make a mistake in my first guess for part 2, forgetting the case where the dial starts on zero and we turn left. We shouldn’t count that first turn as going through zero, and so I had to add the start and end points, with that -1 offset.

It appears I wasn’t the only one…

[2025 Day 1 Part 2] Public Service Announcement
byu/StaticMoose inadventofcode

seq |> get_passwords
(end_zeros = 3, pass_zeros = 6)

⭐⭐

Day 2: fake IDs at the gift shop ?

We’re given these ‘product IDs’ from the gift shop, and we need to sort through them to find which ones are invalid. Problematic IDs that we are looking for are made up exclusively of repeating sequences of numbers.

product_ids = ["11-22", "95-115", "998-1012", "1188511880-1188511890", "222220-222224",
        "1698522-1698528","446443-446449","38593856-38593862","565653-565659",
        "824824821-824824827","2121212118-2121212124"]

In part 1, we were looking for a single repeated sequence, so the first half of the ID is the same as the second half. Then in part 2, the length of that sequence could change. Here’s my solution:

function is_invalid(id::Int; part_1::Bool = false)
    
    id_string = string(id); midpoint = length(id_string) ÷ 2

    if part_1
        return id_string[begin:midpoint] == id_string[midpoint+1:end]
    end

    for pattern_length  1:midpoint 
    # a pattern more than half the length can't repeat
        if id_string == id_string[begin:pattern_length] ^ (length(id_string) ÷ pattern_length)
        # is the id made *only* of this repeated pattern?
            return true
        end
    end

    return false
end
is_invalid (generic function with 1 method)

The puzzle itself felt simpler than yesterday, but only after I read the description a few times. I initially thought that we were going to be asking to find IDs that contained some (any) repeated sequence, but knowing that the entire ID had to be a repetition simplified things.

invalid_ids = Int[]
for id  product_ids

    parts = split(id, "-")
    id_start = parse(Int, parts[begin]); id_end = parse(Int, parts[end])

    for id  id_start:id_end
        if is_invalid(id)
            push!(invalid_ids, id)
        end
    end

end 

invalid_ids |> sum
4174379265

BenchmarkTools.jl tells me this runs in 459.768 ms, with lots of allocations 606.35 MiB, presumably due to all of the string splitting, concatenation, and going back and forth to integers. I don’t think I’ll have time to think about smarter ways to do this, but I’m sure I’ll be forced to in the coming days. Happy to be brute-forcing for now.

My young children happened to wake me up super early today, so I will have scored highly on the leaderboard, but I’m in it for the puzzles, not the glory.

⭐⭐

Day 3: maximum joltage ⚡

We find batteries, each with their own joltage rating, arranged in banks (rows).

Going from left to right, which batteries should we connect up to get the maximum joltage possible from each row?

joltages = [
    "987654321111111"
    "811111111111119"
    "234234234234278"
    "818181911112111"
    ]

For instance, if we were asked to connect 4 batteries in the third row, we can’t make a 4 digit number higher than 4478 (selecting batteries 3 (joltage = 4), 6 (joltage = 4), 14 (joltage = 7), and 15 (joltage = 8)).

In part 1 we were asked to select 2 batteries from each bank, and in part 2 we were to select 12. Here’s a solution:

function max_joltage_for_bank(bank::String; n_batteries::Int = 2)

    digits = parse.(Int, collect(bank)) 

    max_joltage = []; index = 1

    for k  1:n_batteries

        digits_needed = n_batteries - k
        search_end = length(digits) - digits_needed

        max_val = -1; max_idx = -1
        for i  index:search_end
            if digits[i] > max_val
                # keep track of the biggest digit in this range and it's index...
                max_val = digits[i]; max_idx = i
            end
        end

        # ...so we can update the new starting point
        index = max_idx + 1
        push!(max_joltage, max_val)
    
    end

    return max_joltage |> join |> x -> parse(Int, x)
end
max_joltage_for_bank (generic function with 1 method)

This is based on selecting the maximum digit we can, whilst still leaving space for however many remaining batteries we need to select.

joltages |> x -> max_joltage_for_bank.(x, n_batteries = 12) |> sum
3121910778619

Don’t forget to then update your starting index accordingly, rather than just adding 1, otherwise you’ll keep selecting the same battery over and over – although seeing this helpded me fix it.

Check out some smarter solutions than mine:

-❄️- 2025 Day 3 Solutions -❄️-
byu/daggerdragon inadventofcode

⭐⭐

Day 4: Dunder Mifflin warehouse ?

There are only so many types of AoC puzzles, and today I recognised bts of code I write every year. Namely, checking where I can safely move in in a grid while staying within bounds.

We’re given the layout of warehouse floor – specifically, where a reams of paper are located @, and where there is an empty space ..

grid = [
    "..@@.@@@@."
    "@@@.@.@.@@"
    "@@@@@.@.@@"
    "@.@@@@..@."
    "@@.@@@@.@@"
    ".@@@@@@@.@"
    ".@.@.@.@@@"
    "@.@@@.@@@@"
    ".@@@@@@@@."
    "@.@.@@@.@."
    ]

We are asked:

  • part 1: If we can only move reams with less than 4 neighbouring reams – how many can we move?
  • part 2: Once we’ve removed them, we might be able to move more. How many are left that can never be moved?

Here’s a solution:

function find_accessible(warehouse::Matrix{Char}; n_aux::Int = 4)

    n_rows, n_cols = size(warehouse); labelled_warehouse = copy(warehouse)
    
    # all 8 directions: (row_offset, col_offset)
    directions = [
        (-1, -1), (-1, 0), (-1, 1),
        ( 0, -1),          ( 0, 1),
        ( 1, -1), ( 1, 0), ( 1, 1)
    ]
    
    for row  1:n_rows
        for col  1:n_cols
            
            warehouse[row, col] != '@' && continue # we're only checking the papers
            
            neighbouring_papers = 0
            for (Δr, Δc)  directions
                check_row, check_col = row + Δr, col + Δc
                
                if 1  check_row  n_rows && 1  check_col  n_cols # stay in the grid!
                    if warehouse[check_row, check_col] == '@'
                        neighbouring_papers += 1
                    end
                end
            end
            
            if neighbouring_papers < n_aux
                labelled_warehouse[row, col] = 'x'
            end
        end
    end
    
    return labelled_warehouse
end
find_accessible (generic function with 1 method)

grid |> 
    lines ->  [lines[r][c] for r  eachindex(lines), c  1:length(lines[1])] |>
    warehouse -> count(==('x'), find_accessible(warehouse))
13

Not super elegeant. I replace the reams that can be moved with an x and then count them. It was important to make this change to a copy of the warehouse grid, otherwise we would be changing the grid as we go, and thinking there are less neighbouring reams than there actually are.

…and for part 2, we can just keep calling this function until there are no more reams that can be moved:

function remove_all_reams(grid::Matrix{Char})
    labelled_warehouse = copy(grid)
    while true
        new_labelled_warehouse = find_accessible(labelled_warehouse)
        new_labelled_warehouse == labelled_warehouse && break
        labelled_warehouse = new_labelled_warehouse
    end
    return count(==('x'), labelled_warehouse)
end

loevely visualisation on the reddit of clearing out the warehouse:

[2025 Day 4 Part 2]
byu/EverybodyCodes inadventofcode

⭐⭐

Day 5: fresh ingredients ?

Part 2 got me today.

Our inputs are a list of ranges, followed by ingredient id’s.

ingredients_db = [
    "3-5"
    "10-14"
    "16-20"
    "12-18"

    "1"
    "5"
    "8"
    "11"
    "17"
    "32"
]

An ingredient is fresh if its id falls within any of the ranges given. After a little pre-processing, we can collect all of the ranges that fresh ingredient fall into

…because we suspect we will need that for part 2, even though they end up going in a different, more complicated direction.

fresh_ingredients = Dict{Int, Vector{UnitRange{Int}}}()
for i  ingredients
    for r  ranges
        if i  r
            push!(get!(fresh_ingredients, i, UnitRange{Int}[]), r)
        end
    end
end

and then returning the number of fresh ingredients.

fresh_ingredients |> length
3

Part 2 instead asks to find the number of possible fresh ingredients i.e. how many unique ingredient IDs fall within any of the given ranges.

…and in the real input data, the ranges were large, and overlapped. So, in cases where there was overlap, I combined them into a single new range (with the smaller of the start values, and larger of the end values).

sorted_ranges = sort(ranges, by = first)

fresh_ingredients_merged = [sorted_ranges[begin]]
for range  sorted_ranges[2:end]
    # start a new range when the lower value is outside the largest value of the merge
    if last(fresh_ingredients_merged[end]) < first(range)
        push!(fresh_ingredients_merged, range)
    else
    # the start of the new range stays the same, but the end needs to be extended if it's bigger for the new range
    new_max_range = [last(fresh_ingredients_merged[end]), last(range)] |> maximum
    fresh_ingredients_merged[end] = first(fresh_ingredients_merged[end]):new_max_range
    end    
end

…and then summed the lengths of the now non-overlapping ranges (without duplicate ingredient IDs):

[length(x) for x  fresh_ingredients_merged] |> sum
14

⭐⭐

Day 6: Cephalopod Maths ?

I googled cephalopod – it means a creature with a head and tentacles.

In the advent of code universe, they get maths homework, and this is what it looks like:

sums = [
    "123 328  51 64 ",
    " 45 64  387 23 ",
    "  6 98  215 314",
    "*   +   *   +  "
]

This is asking us to apply the operator in the last row, to the numbers above it. In part 1, the numbers above the first (leftmost) * are 123, 45, 6. In part 2 we are told the numbers are written in coluns (not rows) so we instead have to multiply 356, 24, and 1.

So the difference is in fiddling the parsing of the numbers. Here’s a solution:

function solve_sums(lines::Vector{String}; part_2::Bool = false)
    
    n_cols = lines[end] |> length
    # need to break until the next sum when there is a column of spaces
    no_sum = [all(line[c] == ' ' for line  lines) for c  1:n_cols]

    results = Int[]; col = 1
    while col <= n_cols
        # skip the separator columns that split the sums
        while col <= n_cols && no_sum[col]
            col += 1
        end
        col > n_cols && break
        
        # otherwise we start a new sum
        start_sum = col
        
        # and keep going until the next separator
        while col <= n_cols && !no_sum[col]
            col += 1
        end

        # the operator is always in the first col of each sum, in the last row
        op = lines[end][start_sum]
        
        if part_2
            # pt.2: each column is a number (read top-down)
            numbers = Int[]
            for nums_col  start_sum:col-1
                digits = [line[nums_col] for line  lines[1:end-1] if line[nums_col] != ' ']
                if !isempty(digits)
                    push!(numbers, parse(Int, String(digits)))
                end 
            end
        else
            # pt.1: each row is a number, but they can have leading spaces
            numbers = [parse(Int, strip(line[start_sum:col-1])) for line  lines[1:end-1]]
        end
        
        # an excuse to use Julia metaprogramming :)
        push!(results, op |> Symbol |> eval |> x -> reduce(x, numbers))
    end
        
    return results
end
solve_sums (generic function with 1 method)

Not completed on the day for the first time, but life is very hectic.

Oh, and an excuse to use Julia metaprogramming! We could have put in some conditional logic to take products when the operator is * and sums when it’s +, but instead we can directly convert to a symbol and evaluate it. Not at all required for this problem, but hey…

And then add up the results:

sums |> solve_sums |> sum
4277556

I’m inferring from the below meme, but I guess some people created matrices for all the numbers in each sum, rather than just strip the whitespace ?

Neat – wish I did it that way ??

[2025 Day 6 (Part 2)] The solution be like…
byu/HotTop7260 inadventofcode

⭐⭐

Day 7: Tachyon beam splitting ⚛️

After yesterday, we’ve got 12 ⭐’s on the board

we’re 1/2 way there [Jon Bon Jovi voice]

A beam of tachyons, | flows from a starting point, S down a grid. When it hits a splitter, ^, two beams emerge (one either side).

tachyon_manifold = [
    ".......S.......",
    "...............",
    ".......^.......",
    "...............",
    "......^.^......",
    "...............",
    ".....^.^.^.....",
    "...............",
    "....^.^...^....",
    "...............",
    "...^.^...^.^...",
    "...............",
    "..^...^.....^..",
    "...............",
    ".^.^.^.^.^...^.",
    "..............."
]

In part 1, we are asked how many times the beam is split, and in part 2 we are asked how many parallel timelines (possible paths) exist for the beam.

Here’s my solution:

function analyse_beams(beam_rows::Vector{String})

    n_rows = beam_rows |> length
    n_cols = beam_rows[end] |> length
    grid = [beam_rows[row][col] for row  1:n_rows, col  1:n_cols]

    # find starting position
    start_pos = findfirst(==('S'), grid)
    start_row, start_col = start_pos[1], start_pos[2]

    # initialise beams and timelines...
    splits = 0; timelines = Dict(start_col => 1)
    
    for row  (start_row + 1):n_rows
        # ...and then keep track of new ones
        new_timelines = Dict{Int, Int}()
        
        for (col, count)  timelines
            if grid[row, col] == '^'
                # for part 1, we count the splits
                splits += 1 
                # for part 2, we "do" the split and count the timelines (add beams to the left and right - after an in-bounds check)
                col - 1 >= 1 && (new_timelines[col - 1] = get(new_timelines, col - 1, 0) + count)
                col + 1 <= n_cols && (new_timelines[col + 1] = get(new_timelines, col + 1, 0) + count)
            else
                new_timelines[col] = get(new_timelines, col, 0) + count
            end
        end

        timelines = new_timelines

    end

    return ("splits" => splits, "timelines" => timelines)
end
analyse_beams (generic function with 1 method)

This is a tidied up version, because I was initially separately tracking a variable beams, which I ended up separately recording in my part 2 dictionary. Also, I’ve got no idea if I needed the in-bounds checks (where I discount any beams that stray off the edges of the tachyon manifold).

tachyon_manifold |> analyse_beams
("splits" => 21, "timelines" => Dict(5 => 10, 13 => 1, 15 => 1, 7 => 11, 11 => 2, 9 => 11, 12 => 1, 3 => 2, 1 => 1))

Day 8: Connecting junction boxes ?

We are given 3-d (x, y, z) coordinates of electrical junction boxes and ask to connect the closest ones (shortest Euclidean distance) together to form circuits.

junction_boxes = [
    "162,817,812",
    "57,618,57",
    "906,360,560",
    "592,479,940",
    "352,342,300",
    "466,668,158",
    "542,29,236",
    "431,825,988",
    "739,650,466",
    "52,470,668",
    "216,146,977",
    "819,987,18",
    "117,168,530",
    "805,96,715",
    "346,949,466",
    "970,615,88",
    "941,993,340",
    "862,61,35",
    "984,92,344",
    "425,690,689"
]

The key insight: We don’t need to track every box in each circuit, just whether two boxes share a common “root”. This is a Union-Find (∪ + ?) problem.

Each box starts as it’s own standalone circuit. When we connect two boxes, one joins the other’s (the root’s) circuit. To check if two boxes are in the same circuit, we can then follow the chain and check if they share the same root.

coords = junction_boxes |>
    boxes -> split.(boxes, ",") |>
    boxes -> [parse.(Int, b) for b  boxes]

# each box starts as its own root
n_boxes = length(coords); parent = collect(1:n_boxes)

# follow the chain upward to find the root
function find_circuit_root(box_index)
    while parent[box_index] != box_index
        box_index = parent[box_index]
    end
    return box_index
end

# connect two circuits by making one root point to the other
function connect_boxes!(box_from, box_to)
    root_from = find_circuit_root(box_from)
    root_to = find_circuit_root(box_to)
    
    # already in same circuit?
    root_from == root_to && return false
    
    # otherwise => merge
    parent[root_from] = root_to
    return true
end

Here I check the distance between every pair of boxes, and arrange shortest to longest:

distances = []
for box  1:n_boxes
    for alt  (box+1):n_boxes  # avoid double-counting pairs
        dist = (coords[box] .- coords[alt]).^2 |> sum |> sqrt
        push!(distances, (dist = dist, from = box, to = alt))
    end
end
sort!(distances, by = first)

Part 1 asks us to attempt to connect the 1000 shortest connections and part 2 asks wants us to keep going until we end up with 1 big circuit and a final connection. Here’s my solution:

function size_circuits(distances; n_attempts::Union{Int, Nothing} = nothing)
    
    # each box starts as is its own circuit
    parent = collect(1:n_boxes)
    last_connection = nothing

    for (attempt, d)  enumerate(distances)
        
        if connect_boxes!(d.from, d.to)
            last_connection = d
        end

        # part 1: stop after N attempts
        if !isnothing(n_attempts) && attempt >= n_attempts
            break
        end
        
        # part 2: stop when everything is connected
        roots = [find_circuit_root(i) for i  1:n_boxes]
        length(unique(roots)) == 1 && break
    end

    roots = [find_circuit_root(i) for i  1:n_boxes]
    circuit_sizes = [count(==(r), roots) for r  unique(roots)]
    sort!(circuit_sizes, rev=true)

    return Dict("circuit_sizes" => circuit_sizes, 
                "last_connection" => last_connection)
end
size_circuits (generic function with 1 method)

There are some lovely visualisations of the growing mega-circuit of part 2:

[2025 Day 8 (Part 2)] Visualisation
byu/Derailed_Dash inadventofcode

size_circuits(distances)["last_connection"] |>
    last -> coords[last.from][1] * coords[last.to][1]
25272

The puzzles are getting tricky. This is possibly the first one that can’t be brute-forced? My naive approach of adding new boxes to sets of circuits was pretty slow (O(n)). This is also the first puzzle I was unable to solve them on the day of release due to other commitments, but I’m not fussed about that.

⭐⭐

Day 9: Red and Green tiles ? ?

We are provided with coordinates (in 2-d today) of red tiles:

red_tiles = [
    "7,1",
    "11,1",
    "11,7",
    "9,7",
    "9,5",
    "2,5",
    "2,3",
    "7,3"
]

We are asked to find the largest (by area) rectangle that can be made, if we took two red tiles as opposite corners. Part 1 didn’t have any further restrictions and so was one of the simplest stars to get so far this year. Part 2 was not…

In part 2, we connect sequential red tiles using green tiles. We then have a border of green tiles since, adjacent tiles are always in the same row or column (thankfully!) and we also fill in the inside of this shape with green tiles. The large rectangles we are trying to form cannot include any coordinates that are not either red or green tiles.

How do we know which tiles are “inside” the loop?

I think there are some clever ways to do this by flooding the outer grid with empty tiles or the inner grid with green tiles, until we hit the border.

I used a simpler approach. For each empty tile, walk left and count how many times you cross vertical boundaries. After an odd number of crossings, you must be inside (green tile). ? Trick: be carefull to ignore horizontal boundaries, as they will just run parallel to our walk (we won’t cross them). This is why I added the above check.

# connect sequential red tiles with green tiles
function draw_green_line!(grid, from, to)
    x_min, x_max = minmax(from[1], to[1])
    y_min, y_max = minmax(from[2], to[2])
    
    for x  x_min:x_max
        for y  y_min:y_max
            grid[x, y] == 0 && (grid[x, y] = 2)  # only fill empties
        end
    end
end
draw_green_line! (generic function with 1 method)

# are we inside the shape?
function is_inside(grid, x, y)
    grid[x, y] != 0 && return false  # already filled
    
    crossings = 0
    for check_x  1:(x-1)
        # counting vertical boundaries
        at_y = grid[check_x, y]  (1, 2)
        above = y > 1 && grid[check_x, y-1]  (1, 2)
        at_y && above && (crossings += 1)
    end
    
    return isodd(crossings)
end
is_inside (generic function with 1 method)

Turns out the grid is enormous – 9655610113 tiles.

Since we only care about red tiles, we can index their sparse coordinates. I eventually remembered to use the original (enormous) grid coordinates to calculate how big the rectangle areas are. Here’s my solution:

# check rectangle contains only red/green tiles
function is_valid_rectangle(grid, c1, c2)
    x_min, x_max = minmax(c1[1], c2[1])
    y_min, y_max = minmax(c1[2], c2[2])
    
    for x  x_min:x_max
        for y  y_min:y_max
            grid[x, y] == 0 && return false
        end
    end
    return true
end
is_valid_rectangle (generic function with 1 method)

function max_area(red_tiles_coords::Vector{Vector{Int}}; part_2::Bool = false)
    
    n_tiles = length(red_tiles_coords)
    max_area = 0
    best_pair = nothing

    if part_2
    # coordinate compression
        all_x = sort(unique(c[1] for c  red_tiles_coords))
        all_y = sort(unique(c[2] for c  red_tiles_coords))
        
        # big grid => compressed index
        x_to_idx = Dict(x => i for (i, x)  enumerate(all_x))
        y_to_idx = Dict(y => i for (i, y)  enumerate(all_y))
        
        # Compress all red tile coordinates
        compressed_coords = [[x_to_idx[c[1]], y_to_idx[c[2]]] for c  red_tiles_coords]
        
        # using reduced grid dims...
        max_x = length(all_x)
        max_y = length(all_y)
        grid = zeros(Int, max_x, max_y)
        
        # ...mark red tiles and connect with greens
        for c  compressed_coords
            grid[c[1], c[2]] = 1
        end
        
        for i  1:n_tiles
            next_i = i == n_tiles ? 1 : i + 1
            draw_green_line!(grid, compressed_coords[i], compressed_coords[next_i])
        end
        
        # and paint the interior green too
        for x  1:max_x
            for y  1:max_y
                is_inside(grid, x, y) && (grid[x, y] = 2)
            end
        end
    end

    for i  1:n_tiles
        for j  (i+1):n_tiles
            
            # part 2: check validity using compressed coords
            if part_2 && !is_valid_rectangle(grid, compressed_coords[i], compressed_coords[j])
                continue
            end

            # ...but then calculate the area on the big grid
            width = abs(red_tiles_coords[i][1] - red_tiles_coords[j][1]) + 1
            height = abs(red_tiles_coords[i][2] - red_tiles_coords[j][2]) + 1
                        
            if width * height > max_area
                max_area = width * height
                best_pair = (i, j)
            end
        end
    end

    return max_area
end
max_area (generic function with 1 method)

which matches the example answer:

red_tiles |>
    tiles -> split.(tiles, ",") |>
    tiles -> [parse.(Int, t) for t  tiles] |> 
    coords -> max_area(coords, part_2 = true)
24

A hodgepodge of a solution, which may have some breaking edge cases, but got me there. This is the first time I have been 3 puzzles away from completing advent of code…

⭐⭐

Day 10: ⏳

Citation

BibTeX citation:
@online{di_francesco2026,
  author = {Di Francesco, Domenic},
  title = {Advent of Code 2025},
  date = {2026-02-20},
  url = {https://allyourbayes.com/posts/AoC/},
  langid = {en}
}
For attribution, please cite this work as:
Di Francesco, Domenic. 2026. “Advent of Code 2025.”
February 20, 2026. https://allyourbayes.com/posts/AoC/.

Simplifying Julia Package Integration with Extensions

By: Great Lakes Consulting

Re-posted from: https://blog.glcs.io/package-extensions

This post was written by Steven Whitaker.

The Julia programming languageis a high-level languagethat is known, at least in part,for its outstanding composability.Much of Julia’s composabilitystems from its multiple dispatch,which allows functions written in one packageto work with objects from another packagewithout either package needing to depend on or even know about the other.(See another blog post for more details.)

Sometimes, however,it is useful for a packageto be able to extend its functionsto provide additional functionalitywhen given an object of a specific typefrom another package.One way to do sois to add the other package as an explicit dependencyso that its type is availablefor the first package to useto define a specific method for it.

But what if the package can function just finewithout the additional functionality?What if the extra functionalityisn’t integral to what the package doesand only appliesif the userwants to work with objectsof that specific type?In this case,it doesn’t make much senseto make the other package a direct dependency,because then every userpays the price of extra package load timefor functionality that only some users actually want.

The solution is package extensions.A package extension is codethat gets loaded conditionally,depending on what other packagesthe user has explicitly loaded.In other words,when a user loads both the packageand the dependency the extension depends on,the extension gets loaded automatically.This way,users who want to use the packagecan do so without the added dependency,while users who want the extra functionalitycan load the dependency themselves.

In this post,we will learn about some package extensionsthat exist in the Julia package ecosystem.We will also learn how to write a package extensionand how to load the extension.

This post assumes you are familiarwith the structure of a Julia package.If you need to learn more,check out our post on creating Julia packages.

Package Extensions in the Wild

Writing a Package Extension

To create a package extension,one needs to create a modulethat adds method definitionsto functions from one of the packages(either the package being extendedor the package that triggers loading the extension)that dispatch on types from the other package.This module will live in the ext directoryof the package being extended.Additionally,the extended package’s Project.tomlneeds to be updatedto inform the package managerof the existence of the extensionand when to load it.

Let’s look at a concrete example.

Example Package to Extend

This example will build on a custom package called Averages.jlthat we discussed in our blog post on testing Julia packages.The package code is as follows:

module Averagesusing Statistics: meanexport compute_averagecompute_average(x) = (check_real(x); mean(x))function compute_average(a, b...)    check_real(a)    N = length(a)    for (i, x) in enumerate(b)        check_real(x)        check_length(i + 1, x, N)    end    T = float(promote_type(eltype(a), eltype.(b)...))    average = Vector{T}(undef, N)    average .= a    for x in b        average .+= x    end    average ./= length(b) + 1    return a isa Real ? average[1] : averageendfunction check_real(x)    T = eltype(x)    T <: Real || throw(ArgumentError("only real numbers are supported; unsupported type $T"))endfunction check_length(i, x, expected)    N = length(x)    N == expected || throw(DimensionMismatch("the length of input $i does not match the length of the first input: $N != $expected"))endend

Creating the Extension

For this example,we will create an extensionthat implements additional functionality for DataFrames.These are the tasks we need to doto implement the extension:

  1. Create the extensionat Averages/ext/AveragesDataFramesExt.jl.Note that this follows the naming convention for extensions:<PackageName><NameOfPackageThatTriggersExtension>Ext.Inside this file,we create a module called AveragesDataFramesExt(same name as the file)and put the code we want to be includedwhen Averages.jl and DataFrames.jl are loaded together:

    module AveragesDataFramesExtimport Averagesusing Averages: compute_averageusing DataFrames: All, DataFrame, combinefunction Averages.compute_average(df::DataFrame)    @info "Running code in AveragesDataFramesExt!"    df_avg = combine(df, All() .=> compute_average)    return df_avgendend
  2. Add [weakdeps] and [extensions] sectionsto the Project.toml of Averages.jl.(See our previous blog post for the original Project.toml.)In [weakdeps],specify DataFrames.jl and its UUID,and, in [extensions],specify our extension (AveragesDataFramesExt)and its dependency (DataFrames.jl).The UUID of DataFrames.jl can be foundin DataFrames.jl’s Project.toml.

    Here’s the updated Project.toml for Averages.jl:

    name = "Averages"uuid = "1fc6e63b-fe0f-463a-8652-42f2a29b8cc6"version = "0.1.0"[deps]Statistics = "10745b16-79ce-11e8-11f9-7d13ad32a3b2"[weakdeps]DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"[extensions]AveragesDataFramesExt = "DataFrames"[extras]Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"[targets]test = ["Test"]

    (Note that,just as compatible versions of the [deps] packagescan be specified in a [compat] section,so too can the compatible versions of the [weakdeps] packagesbe specified.)

Using the Extension

First,let’s see what happensif we try this without the extension:

julia> compute_average(DataFrame(a = [1, 2], b = [3.0, 4.0]))ERROR: ArgumentError: only real numbers are supported; unsupported type AnyStacktrace: [1] check_real(x::DataFrame)   @ Averages /path/to/Averages/src/Averages.jl:34 [2] compute_average(x::DataFrame)   @ Averages /path/to/Averages/src/Averages.jl:7 [3] top-level scope   @ REPL[5]:1

So,now let’s see if the extensionallows this function call to work.

To use the extension,install and load Averages.jl and DataFrames.jl(for Averages.jl, use the dev command,i.e., pkg> dev /path/to/Averages)and then call compute_average:

julia> using Averages, DataFramesjulia> compute_average(DataFrame(a = [1, 2], b = [3.0, 4.0]))[ Info: Running code in AveragesDataFramesExt!12 DataFrame Row  a_compute_average  b_compute_average      Float64            Float64   1                1.5                3.5

Nice, it works!And with that,we have an example package extensionthat illustrates how to implement your own.

And remember,a user of Averages.jlwill only incur the cost of loading AveragesDataFramesExtif they load DataFrames.jl.For more details,see the slide annotationsin this screenshot from JuliaCon 2023:

JuliaCon 2023 package extensions talk

(See also the full talk on package extensionsfor even more details.)

Note: Where Should an Extension Live?

By the way,if you’re wondering why we put the extension in Averages.jlinstead of DataFrames.jl,the answer isthat it doesn’t really matterbecause the user experiencewill be the same regardless.If you still want some rules to follow,I’m not aware of any Julia best-practicesin this regard,but here are some rules that make sense to me:

  • If one of the two packages in questiondefines an interface,the extension should go in the packagethat implements the interface.
  • Otherwise,put the extension in the packagethat owns the functionsthat are being extended.In our example,we extended the compute_average function.Since this function is defined in Averages.jl,we put the extension in Averages.jl.
  • An exception to the previous ruleis if getting the new functionality rightrequires a good understandingof the internals of the new data typethat’s being dispatched on,in which case the extensionshould belong in the packagethat defines the type.For example,if compute_average was super complicatedfor some reasonwhen working with DataFrames,it would make sense for those with the needed expertise(i.e., the developers of DataFrames.jl)to own and maintain the extension.

Summary

In this post,we listed some real Julia packagesthat have their own package extensions.We also demonstrated creating our own extensionfor an example packageand showed how to use the extension’s code.

What package extensions have you found useful?Let us know in the comments below!

Additional Links

]]>

A Beginner’s Guide to DataLoaders in Julia with MLUtils.jl

By: Random blog posts about machine learning in Julia

Re-posted from: https://rkube.github.io/julia/deep-learning/dataloaders/2025/11/08/dataloaders.html

Introduction

Data Loaders are critical tools to work efficiently with deep learning.
They are the glue that bring the data to your machine learning model. Doing this efficiently
and without errors is important for successful training of any model.

Datasets and DataLoaders are concepts that encapsulate the logic for preparing the data to be
input in your model. In their basic form, they prepare small batches of data for ingestion by
reshaping individual samples into matrix form. More involved applications may load
images from files, augment them through flipws, crops, rotations.

The good news is that in Julia, MLUtils has you covered for
all your data loading needs. In this blog post we’ll dive into how to create both Datasets and DataLoaders.
We start out with a simple-as-possible example and then walk through a more realistic example.

Data loaders in easy mode

Let’s start by writing a simple as possible data loader to illustrate the core concepts of data loading
using the DataLoader. The task we are looking at is somewhat related to image classification, but simplified.

Our data may be a bunch of images, like 1,000 of them. Each of the images is assumed to be in one of
ten classes. A mock up of that data is something like

X_train = randn(Float32, 10, 10, 1_000)
Y_train = randn(1:10, 1_000)

Here, X_train are 1_000 samples of 10×10 matrices. These are mocked up as normally distributed Float32
data with shape (10, 10, 1_000), corresponding to (width, height, sample). Julia stores arrays in column-major,
so in our case the data for width is consecutive in memory. The last dimension, the sample dimension varies slowest.
The targets are mocked as 10_000 samples of random integers between 1 and 10, indicated by the 1:10 syntax.

A nice thing about Julia is that you often get away without defining custom classes. In our simlpe case,
we can get away with defining our dataset as

data = (X_train, Y_train)

No custom class definition needed. The dataset is just a tuple of two vectors.

Now we move on to the data loader. The job of the data loader is to sample from our dataset. It should
do batching, i.e. loading multiple (x,y) samples in one go, and random shuffling.
For this simple dataset, we just need to create a DataLoader:

loader = DataLoader(data, batch_size = 3)

DataLoaders are iterable, and we can look at the first item like so:

julia> (x_first, y_first) = first(loader);
julia> size(x_first)
(10, 10, 3)
julia> size(y_first)
(3,)

When we iterate over the dataloader, we can use individual samples. Here is the iteration skeleton:

julia> for (x, y) in loader
            @show size(x), size(y)
       end
[...]
(size(x), size(y)) = ((10, 10, 3), (3,))
(size(x), size(y)) = ((10, 10, 1), (1,))

Note that in the last iteration, the DataLoader only returns vectors with a single sample.
This is the loader watching out for us and stopping at the end of the data. We have 1_000 samples,
and using batchsize=3, the last iteration can only feature a single sample. This is because
1000 / 3 = 333.33333 and the dataset is exhausted after the first sample in the last iteration.

We can confirm this by querying the length of the dataloader:

julia> length(loader)
334

That’s it. In conclusion, for a simple in-memory dataset creating a tuple data = (X, Y) is all you need.
The DataLoader does the rest.

A more realistic example

Now let’s look at a more involved case where we want to read text data from a file and tokenize it.
This sample is inspired by Andrej Karpathy’s NanoGPT video,
github collab.

For this application, it’s convenient to encapsulate information on the DataSet into a struct. This struct
collects the length of the text, a block size for data loading, dictionaries that map characters to tokens
and vice-versa, and a vector of all the tokens. The members important for the data loading logic are
block_size and data. While data holds the data vector itself, block_size defines the length of
an individual sample.

In addition, we give the struct a constructors that reads all lines in the text file, creates an array of
unique characters, and dictionaries to map from character to token and a dictionary to map from token to character.
While the dictionaries are not necessary for the data loading logic to work, they are included here for completeness.

struct NanoDataset
    block_size::Int64               # How long an observation is. This is important for the dataloader
    ch_to_int::Dict{Char, Int64}    # Maps chars to tokens
    int_to_ch::Dict{Int64, Char}    # Maps tokens to chars
    data::Vector{Int64}             # The tokens

    function NanoDataset(filename::String, block_size::Int64)
        lines = readlines(filename)             # Read all lines
        _out = [c for l ∈ lines for c ∈ l]      # Create char array
        chars = sort!(unique(_out))             # Sort all chars occuring in the dataset
        push!(chars, '\n')                      # Add the \n character that we stripped when only looking at lines
        ch_to_int = Dict(val => ix for (ix, val) in enumerate(chars))   # Mapping of chars to int
        int_to_ch = Dict(ix => val for (ix, val) in enumerate(chars))


        all_tokens = [ch_to_int[s] for s in join(lines, "\n")]

        new(block_size, ch_to_int, int_to_ch, all_tokens)
    end
end

To create this dataset we run

ds = NanoDataset(FILENAME, 16)

Great, now moving on to the DataLoader. To make the dataloader work, we have to implement the numobs and
getobs interface as described in the
Documentation. This basically means
that we have to make DataLoader know how to get the length of the dataset and how to get a single observation.

Fortunately, the MLUtils documentation tells us how to implement both. numobs just requires to specialize Base.length for our type. This function returns how
many observations (that is the number of samples) there are in the dataset.

We implement this like

Base.length(d::NanoDataset) = length(d.data) - d.block_size - 1

which makes numobs work for our dataset:

julia> numobs(ds)
1115375

Next, we’ll get getobs working.

For the easy case, an individual sample of X_train can be accessed by X_train[:,:,42]. That is, array
indexing is the access. For NanoDataset we need to define how to get a single observation. We do this
by specializing Base.getindex on our dataset:

function Base.getindex(d::NanoDataset, i::Int)
    1 <= i <= length(d) - d.block_size - 1|| throw(ArgumentError("Index is out of bounds"))
    return (d.data[i:i+d.block_size-1], d.data[i+1:i+d.block_size])

So when we access ds through [] braces, we get two vectors: The input X and the target Y. X is
a sequence of tokens of length block_size and Y is this same sequence, shifted by a single index.

I’d like to note here, that Base.getindex is a good place to get fancy. For image loading, this
function may load data from a file. Or apply augmentations through
Augmentor.jl. The sky (or ChatGPT) is the limit!

Now we can access samples in our dataset like

julia> ds[1]
([18, 47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14], [47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43])

Note that Y is just X shifted by one index, just as implemented in getindex.
Before we define a dataloader, let’s implement one more function that allows us to load minibatchs:

MLUtils.getobs(d::NanoDataset, i::AbstractArray{<:Integer}) = [getobs(d, ii) for ii in i]

This function defines to dispatch calls where requested samples are given by i::AbstractArray{<:Integer}
to multiple calls of getobs. For example, loading the first two samples returns this:

julia> getobs(d, [1,2])
2-element Vector{Tuple{Vector{Int64}, Vector{Int64}}}:
 ([18, 47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14], [47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43])
 ([47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43], [56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43, 44])

Fantastic, now we have everything in place to create a DataLoader. We can create a dataloader that
loads 2 batches at the same time:

julia> dl = DataLoader(d, batchsize=2)
557688-element DataLoader(::NanoDataset, batchsize=2)
  with first element:
  2-element Vector{Tuple{Vector{Int64}, Vector{Int64}}}

julia> (x,y) = first(dl)
2-element Vector{Tuple{Vector{Int64}, Vector{Int64}}}:
 ([18, 47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14], [47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43])
 ([47, 56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43], [56, 57, 58, 1, 15, 47, 58, 47, 64, 43, 52, 10, 65, 14, 43, 44])

The data loader returns a vector of lenth 2, each element of the vector a tuple of vectors.
Ideally for deep learning, we’d like a matrix of size (16,2) though. Remember the first dimension is the sequence
length, dimensions 2 is the batch dimension. The DataLoader can do this by setting collate=true:

julia> dl = DataLoader(d, batchsize=2, collate=true)
557688-element DataLoader(::NanoDataset, batchsize=2, collate=Val{true}())
  with first element:
  (16×2 Matrix{Int64}, 16×2 Matrix{Int64},)

julia> (x,y) = first(dl)
([18 47; 47 56;  ; 65 14; 14 43], [47 56; 56 57;  ; 14 43; 43 44])

julia> x
16×2 Matrix{Int64}:
 18  47
 47  56
 56  57
 57  58
 58   1
  1  15
 15  47
 47  58
 58  47
 47  64
 64  43
 43  52
 52  10
 10  65
 65  14
 14  43

Now the DataLoader returns the desired matrix. In addition, DataLoader supports shuffling and multithreading,
just add shuffle=true, parallel=true to the parameter list.

Where to go from here

In this tutorial we looked at how to use DataLoaders in Julia through two examples. In the first example,
we had in-memory data that can be passed as a tuple into the DataLoader. That just worked.
In the second example, we had to implement the numobs, getobs interface to make it work. This is a
just a bit more work, but allows to work with arbitrary custom data. The getobs implementation is where
to hook in lazy-loading from disk, augmentations, and so on.

Now that we looked at some examples on data loading, here are some things to try next

  • Try building your own Dataloader and use it in a Lux or Flux training loop.
  • Play around with image augmentation using Augmentor
  • Try using threads and measure the speed-up compared to single-threaded loading.