New features in DataFrames.jl 1.3: conclusion

By: Blog by Bogumił Kamiński

Re-posted from: https://bkamins.github.io/julialang/2022/01/07/release13.html

Introduction

This is the last post from the series introducing features added in DataFrames.jl 1.3. There are many changes I have not covered yet. I have selected some
of them that I think are most relevant in typical data wrangling workflows.

The topics I plan to discuss are:

  • ordering of groups in groupby;
  • unstack now supports fill keyword argument;
  • deprecations in deleting rows and sorting API.

The post was written under Julia 1.7.0, DataFrames.jl 1.3.1,
Chain.jl 0.4.10, and FreqTables.jl 0.4.5.

Ordering of groups in groupby

Let me start with highlighting that GroupedDataFrame objects produced by the
groupby function are indexable. This means that you can flexibly subset groups
or re-order them. Here is an example:

julia> using DataFrames

julia> df = DataFrame(a=[1,1,2,2,2,3])
6×1 DataFrame
 Row │ a
     │ Int64
─────┼───────
   1 │     1
   2 │     1
   3 │     2
   4 │     2
   5 │     2
   6 │     3

julia> gdf = groupby(df, :a, sort=true)
GroupedDataFrame with 3 groups based on key: a
First Group (2 rows): a = 1
 Row │ a
     │ Int64
─────┼───────
   1 │     1
   2 │     1
⋮
Last Group (1 row): a = 3
 Row │ a
     │ Int64
─────┼───────
   1 │     3

julia> gdf[[3, 1]]
GroupedDataFrame with 2 groups based on key: a
First Group (1 row): a = 3
 Row │ a
     │ Int64
─────┼───────
   1 │     3
⋮
Last Group (2 rows): a = 1
 Row │ a
     │ Int64
─────┼───────
   1 │     1
   2 │     1

Here the gdf[[3, 1]] operation picked two groups from gdf putting group
with original index 3 first and group with original index 1 next.

This feature is often useful and gives a lot of flexibility to the users. Here
is an example showing how you can sort groups based on non-key column values:

julia> df = DataFrame(a=[1,1,2,2,2,3], x=6:-1:1)
6×2 DataFrame
 Row │ a      x
     │ Int64  Int64
─────┼──────────────
   1 │     1      6
   2 │     1      5
   3 │     2      4
   4 │     2      3
   5 │     2      2
   6 │     3      1

julia> gdf = groupby(df, :a, sort=true)
GroupedDataFrame with 3 groups based on key: a
First Group (2 rows): a = 1
 Row │ a      x
     │ Int64  Int64
─────┼──────────────
   1 │     1      6
   2 │     1      5
⋮
Last Group (1 row): a = 3
 Row │ a      x
     │ Int64  Int64
─────┼──────────────
   1 │     3      1

julia> gdf[sortperm([sum(sdf.x) for sdf in gdf])]
GroupedDataFrame with 3 groups based on key: a
First Group (1 row): a = 3
 Row │ a      x
     │ Int64  Int64
─────┼──────────────
   1 │     3      1
⋮
Last Group (2 rows): a = 1
 Row │ a      x
     │ Int64  Int64
─────┼──────────────
   1 │     1      6
   2 │     1      5

However, this means that one should be careful when considering the ordering
of groups in a GroupedDataFrame. For this reason apart from integer indexing
GroupedDataFrame also supports indexing using values of grouping columns
(in the example I show Tuple indexing, but also NamedTuple and dictionary
indexing is supported):

julia> df = DataFrame(name=["Alice", "Bob"])
2×1 DataFrame
 Row │ name
     │ String
─────┼────────
   1 │ Alice
   2 │ Bob

julia> gdf = groupby(df, :name, sort=true)
GroupedDataFrame with 2 groups based on key: name
First Group (1 row): name = "Alice"
 Row │ name
     │ String
─────┼────────
   1 │ Alice
⋮
Last Group (1 row): name = "Bob"
 Row │ name
     │ String
─────┼────────
   1 │ Bob

julia> gdf[("Bob",)]
1×1 SubDataFrame
 Row │ name
     │ String
─────┼────────
   1 │ Bob

or you can use a special GroupKey object that is produced by the keys
function (this option is fastest):

julia> keys(gdf)
2-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (name = "Alice",)
 GroupKey: (name = "Bob",)

So what is new in DataFrames.jl 1.3? The thing is that previously user was not
able to fully control the initial ordering of groups produced by groupby in
all cases. Now this can be controlled by the sort keyword argument and the
API has been established with the following rules:

  • if you pass sort=true the groups will be sorted by values of grouping columns;
  • if you pass sort=false the groups will be produced in order of their first
    appearance in the source data frame;
  • if you omit passing the sort keyword argument the ordering of groups is
    undefined and will depend on the grouping algorithm used (DataFrames.jl has
    several grouping algorithms and tries to choose the fastest available).

To see that these options matter let me show two examples of grouping on an
integer column:

julia> df = DataFrame(id=[2, 3, 1])
3×1 DataFrame
 Row │ id
     │ Int64
─────┼───────
   1 │     2
   2 │     3
   3 │     1

julia> keys(groupby(df, :id))
3-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (id = 1,)
 GroupKey: (id = 2,)
 GroupKey: (id = 3,)

julia> keys(groupby(df, :id, sort=true))
3-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (id = 1,)
 GroupKey: (id = 2,)
 GroupKey: (id = 3,)

julia> keys(groupby(df, :id, sort=false))
3-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (id = 2,)
 GroupKey: (id = 3,)
 GroupKey: (id = 1,)

julia> df = DataFrame(id=[2, 30, 1])
3×1 DataFrame
 Row │ id
     │ Int64
─────┼───────
   1 │     2
   2 │    30
   3 │     1

julia> keys(groupby(df, :id))
3-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (id = 2,)
 GroupKey: (id = 30,)
 GroupKey: (id = 1,)

julia> keys(groupby(df, :id, sort=true))
3-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (id = 1,)
 GroupKey: (id = 2,)
 GroupKey: (id = 30,)

julia> keys(groupby(df, :id, sort=false))
3-element DataFrames.GroupKeys{GroupedDataFrame{DataFrame}}:
 GroupKey: (id = 2,)
 GroupKey: (id = 30,)
 GroupKey: (id = 1,)

As you can see passing the sort keyword argument produces a consistent
ordering. However, when it is not passed in both examples we got a different
order of groups.

unstack now supports fill keyword argument

The change in unstack is pretty simple, but in many common scenarios will be
useful I think. Now you can specify what value should be used to fill missing
combinations of data.

Let me give a practical example. Assume you have a data frame where you have
several observations of peoples’ hair color and eye color:

julia> df = DataFrame(hair=["brown", "yellow", "brown", "brown"],
                      eyes=["blue", "blue", "green", "blue"])
4×2 DataFrame
 Row │ hair    eyes
     │ String  String
─────┼────────────────
   1 │ brown   blue
   2 │ yellow  blue
   3 │ brown   green
   4 │ brown   blue

You can create a frequency table of this data with the FreqTables.jl package:

julia> using FreqTables

julia> freqtable(df, :hair, :eyes)
2×2 Named Matrix{Int64}
hair ╲ eyes │  blue  green
────────────┼─────────────
brown       │     2      1
yellow      │     1      0

You got a matrix with a desired result. However, what if you wanted to get
a DataFrame instead. In the past you would do:

julia> using Chain

julia> @chain df begin
           groupby([:hair, :eyes], sort=true)
           combine(nrow)
           unstack(:hair, :eyes, :nrow)
       end
2×3 DataFrame
 Row │ hair    blue    green
     │ String  Int64?  Int64?
─────┼─────────────────────────
   1 │ brown        2        1
   2 │ yellow       1  missing

The only problem is that you get missing instead of 0 in the cell where
there were no observations. To get 0 you would write:

julia> @chain df begin
           groupby([:hair, :eyes], sort=true)
           combine(nrow)
           unstack(:hair, :eyes, :nrow)
           coalesce.(0)
       end
2×3 DataFrame
 Row │ hair    blue   green
     │ String  Int64  Int64
─────┼──────────────────────
   1 │ brown       2      1
   2 │ yellow      1      0

Since DataFrames.jl the pipeline is easier as you can pass fill=0 keyword
argument to unstack:

julia> @chain df begin
           groupby([:hair, :eyes], sort=true)
           combine(nrow)
           unstack(:hair, :eyes, :nrow, fill=0)
       end
2×3 DataFrame
 Row │ hair    blue   green
     │ String  Int64  Int64
─────┼──────────────────────
   1 │ brown       2      1
   2 │ yellow      1      0

Deprecations in deleting rows and sorting

The deprecation in row deletion is simple. The delete! function is deprecated
in favor of deleteat! function. This change was made to make the DataFrames.jl
API consistent with the Julia Base API (where delete! is defined to remove a
mapping for the given key in a collection, while deleteat! removes items
from given indices).

The deprecation in sorting API is more subtle. Consider the following data
frame:

julia> df = DataFrame(x=[1, 2, 2, 1], y =[2, 2, 1, 1], z=1:4)
4×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      2      1
   2 │     2      2      2
   3 │     2      1      3
   4 │     1      1      4

If you sort it without passing the list of columns on which it should be sorted
by default a lexicographic sort on all columns is performed:

julia> sort(df)
4×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      1      4
   2 │     1      2      1
   3 │     2      1      3
   4 │     2      2      2

is the same as:

julia> sort(df, All())
4×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      1      4
   2 │     1      2      1
   3 │     2      1      3
   4 │     2      2      2

However, to our surprise, currently also when you ask for sorting on no columns
you also get a data frame sorted on all columns:

julia> sort(df, Cols())
┌ Warning: When empty column selector is passed ordering is done on all colums. This behavior is deprecated and will change in the future.
│   caller = sortperm(df::DataFrame, cols::Cols{Tuple{}}; alg::Nothing, lt::typeof(isless), by::typeof(identity), rev::Bool, order::Base.Order.ForwardOrdering) at sort.jl:579
└ @ DataFrames ~/.julia/packages/DataFrames/BM4OQ/src/abstractdataframe/sort.jl:579
4×3 DataFrame
 Row │ x      y      z
     │ Int64  Int64  Int64
─────┼─────────────────────
   1 │     1      1      4
   2 │     1      2      1
   3 │     2      1      3
   4 │     2      2      2

We think that it is an incorrect behavior and in the future sorting on no
columns will produce the result identical to the input data frame (no sorting
will be performed).

Conclusions

This post concludes a series of reviews of new features in DataFrames.jl release
1.3. I have not covered everything that was introduced, a complete list of
changes can be found in the NEWS.md file.

I hope you will enjoy using the package! Happy data wrangling in year 2022!

Advent of Code 2021 – Day 25

By: Julia on Eric Burden

Re-posted from: https://ericburden.work/blog/2022/01/06/advent-of-code-2021-day-25/

It’s that time of year again! Just like last year, I’ll be posting my solutions to the Advent of Code puzzles. This year, I’ll be solving the puzzles in Julia. I’ll post my solutions and code to GitHub as well. I had a blast with AoC last year, first in R, then as a set of exercises for learning Rust, so I’m really excited to see what this year’s puzzles will bring. If you haven’t given AoC a try, I encourage you to do so along with me!

Day 25 – Sea Cucumber

Find the problem description HERE.

The Input – Echinodermic Perambulation

Here we are! The final advent puzzle, and after a brief jaunt into strange and unusual input formats (that we didn’t actually write parsing code for), here we are again. Parsing input from a text file, how I have missed you!

# No fancy data structures this time, just two `BitMatrix`, one representing the
# locations of east-bound cucumbers, and one representing the locations of
# sounth-bound cucumbers, on a 2D grid. Can you tell I'm ready to be done?
function ingest(path)
    eastlines  = []
    southlines = []
    open(path) do f
        for line in readlines(f)
            chars = collect(line)
            push!(eastlines,  chars .== '>')
            push!(southlines, chars .== 'v')
        end
    end
    eastmatrix  = reduce(hcat, eastlines)  |> transpose |> BitMatrix
    southmatrix = reduce(hcat, southlines) |> transpose |> BitMatrix
    return (eastmatrix, southmatrix)
end

Ah, the tried and true ‘parse the grid to a BitMatrix’ approach! With two matrices!

Part One – The Cucumbers Go Marching

It looks like, at some point, the sea cucumbers will deadlock and stop moving about so much, giving us space to land the submarine. Guess you have to land submarines, then? Never considered that, to be honest, it just hasn’t come up before. Ah well, the good news is that, even here on the ocean floor, traffic is bound to come to a standstill sooner or later. Since they’re apparently quite polite, a sea cucumber will only move if there’s an empty space in front of it at the start of a round, and all the east-bound cucumbers try to move before the south-bound cucumbers. With all that in mind, we’re ready to proceed.


# Helper Functions -------------------------------------------------------------

# For checking on the total state of cucumber migration.
function pprint(east::BitMatrix, south::BitMatrix)
    template = fill('.', size(east))
    template[east]  .= '>'
    template[south] .= 'v'
    for row in eachrow(template)
        for char in row
            print(char)
        end
        println()
    end
end

# Move all the east-bound cucumbers that can be moved
function moveeast(east::BitMatrix, south::BitMatrix)
    occupied   = east .| south      # Spaces occupied by any type of cucumber
    nextstate  = falses(size(east)) # An empty BitMatrix, same size as `east`
    ncols      = size(east, 2)      # Number of columns in the grid
    colindices = collect(1:ncols)   # Vector of column indices

    # For each pair of column index and column index + 1 (wrapped around due
    # to space-defying ocean currents)...
    for (currcol, nextcol) in zip(colindices, circshift(colindices, -1))
        # Identify the cucumbers who moved in `currcol` by the empty spaces
        # in `nextcol`
        moved = east[:,currcol]  .& .!occupied[:,nextcol]

        # Identify the cucumbers who waited in `currcol` by the occupied
        # spaces in `nextcol`
        stayed = east[:,currcol] .& occupied[:,nextcol]

        # Place the updated cucumber positions into `nextstate`
        nextstate[:,nextcol] .|= moved
        nextstate[:,currcol] .|= stayed
    end

    return nextstate
end

# Move all the south-bound cucumbers than can be moved
function movesouth(east::BitMatrix, south::BitMatrix)
    occupied   = east .| south       # Spaces occupied by any type of cucumber
    nextstate  = falses(size(south)) # An empty BitMatrix, same size as `south`
    nrows      = size(south, 1)      # Number of rows in the grid
    rowindices = collect(1:nrows)    # Vector of row indices

    # For each pair of row index and row index + 1 (wrapped around due to
    # intradimensional currents)...
    for (currrow, nextrow) in zip(rowindices, circshift(rowindices, -1))
        # Identify the cucumbers who moved in `currrow` by the empty spaces
        # in `nextrow`
        moved = south[currrow,:] .& .!occupied[nextrow,:]

        # Identify the cucumbers who waited in `currrow` by the occupied
        # spaces in `nextrow`
        stayed = south[currrow,:] .& occupied[nextrow,:]

        # Place the updated cucumber positions into `nextstate`
        nextstate[nextrow,:] .|= moved
        nextstate[currrow,:] .|= stayed
    end

    return nextstate
end

# Move the herd(s)! Get the next state for east-bound and south-bound cucumbers,
# check to see if any of their positions changed, and return the results
function move(east::BitMatrix, south::BitMatrix)
    nexteast  = moveeast(east, south)
    nextsouth = movesouth(nexteast, south)

    eastchanged  = east  .⊻ nexteast
    southchanged = south .⊻ nextsouth
    anychanged = any(eastchanged .| southchanged)

    return (anychanged, nexteast, nextsouth)
end


# Solve Part One ---------------------------------------------------------------

# Solve it! Just keep updating the cucumber positions in a loop until they don't
# change, then report the number of rounds that took.
function part1(input)
    (east, south) = input
    rounds = 0
    anychanged = true
    while anychanged
        (anychanged, east, south) = move(east, south)
        rounds += 1
    end
    return rounds
end

I bet that trip from one edge of the transit zone (?) to the other is wild! Good thing these space-warping ocean currents don’t seem to affect submarines, or we’d be in trouble…

Part Two – Reflection

If you haven’t heard, Day 25 – Part Two is always the “go back and finish the rest of the puzzles” challenge, so no extra coding needed here. Instead, I’d like to use this space to reflect on the Advent of Code puzzles for this year and my experience with them.

This is my second year with Advent of Code, as I was introduced to it last year by a friend on a community Slack channel. Just like last year, this experience created a lot of joy, frustration, annoyance, elation, and satisfaction for me, which is typical of my experience with good puzzles. Since last year, I’ve spent a lot more time honing my knowledge of algorithms and data structures, partly as a way to scratch the itch that AoC helped to create, and partly as a way of sharing this love of puzzles with others. You see, in that community Slack channel I mentioned, some friends and I have starting hosting regular meetups aimed at both spending some time solving puzzles and helping some of the folks who are newer to our community practice live-coding for technical interviews. It’s been a blast, and I look forward to continuing into the new year. (Here’s the GitHub repo if you’re curious about our meetups, anyone is welcome to join).

This year came with it’s own set of challenges, from figuring out how to see family during the holidays (thanks Omicron!) to navigating my own bout of non-COVID related illness (I’m fine, but those were an unpleasant few days). There were a few days of puzzles that set me back on my “solve AoC > write blog post” daily cycle, and I had to consider my holiday priorities as a result. In the end, since this blog post isn’t going up until a few days into January, you can probably guess how that worked out. I really enjoy solving these puzzles, and I can get a bit obsessive when I’ve got an unsolved problem lying around in my head, but time with my family is just too precious to short-change. That’s an area I can get better in, to be honest. Weird how coding puzzles can teach life lessons, too.

I’ve been working through the AoC puzzles in Julia this year, and I’m not sure if it’s because of how similar some of the concepts are to R (my most proficient language), the amount of practice I’ve put in over the year, or just general growth as a programmer, but I found thinking through these puzzles in Julia to be a pretty seamless experience. When I did this with Rust, there was definitely a learning curve related to getting the code to do the things I had in my head, but it just seemed to flow a lot more smoothly in Julia. There’s a lot of really ergonomic syntax in Julia, enabled in part by it being a dynamic language, but I don’t think that’s the whole story. My recent time in Rust encouraged me to add Types in lots of places that, according to the Julia documentation and other talks/articles I’ve seen and read, I really didn’t have to and probably shouldn’t have. Frankly, I like Types, and I find being explicit about data types in my code tends to make it easier for me to reason about and clarifies what I did when I come back to it later. In particular, I found the syntax for single-line function assignment, multiple dispatch, native multidimensional array support, and the breadth and diversity of the Base package to be some of the really compelling features of Julia. The speed gains over R and Python (those operations not directly backed by Rust or C or FORTRAN) are significant, but not always as significant as I would have hoped. There’s probably a lot of room for me to improve in optimizing Julia code, though, so I suspect I’ve left a lot of performance on the table. Guess this means I’ll have to keep writing Julia code, then. As a next step, I’d like to get familiar with the Dataframes.jl and Makie.jl (data frames and plotting) packages, since that’s the kind of work I most often tend to do for my day job. Who knows, maybe Julia will inspire me to finally dabble in a bit of machine learning? I hear that’s a pretty hot topic…

I’ve really enjoyed writing this blog series. As usual, explaining how I did things (even to what I imagine to be, like, five random people on the internet who stumble across this site or these articles wherever they get posted) really helped me to clarify, correct, refactor, and update my own understanding. I recommend coding puzzles regularly to new and aspiring programmers, but I’d also recommend finding someone who could benefit from your knowledge (whatever level you feel like you’re at) and giving teaching/explaining a go. Communication skills are important for growing your career (and being a good human), and I feel like teaching, even a little, is a great way to grow those skills and your understanding of the topic.

Just like last year, I’d like to send out a huge thank you to Eric Wastl and his crew of helpers for putting this event/phenomenon on and keeping it running. If you can, please consider donating to this effort to help keep the magic going, I guarantee there will be a lot of coders (215,955 completed at least one puzzle this year, as of the time of writing) who will be glad you did.

If you’re interested, you can find all the code from this blog series on GitHub, please feel free to submit a pull request if you’re really bored. Happy Holidays!

Advent of Code 2021 – Day 25

By: Julia on Eric Burden

Re-posted from: https://www.ericburden.work/blog/2022/01/06/advent-of-code-2021-day-25/

It’s that time of year again! Just like last year, I’ll be posting my solutions to the Advent of Code puzzles. This year, I’ll be solving the puzzles in Julia. I’ll post my solutions and code to GitHub as well. I had a blast with AoC last year, first in R, then as a set of exercises for learning Rust, so I’m really excited to see what this year’s puzzles will bring. If you haven’t given AoC a try, I encourage you to do so along with me!

Day 25 – Sea Cucumber

Find the problem description HERE.

The Input – Echinodermic Perambulation

Here we are! The final advent puzzle, and after a brief jaunt into strange and unusual input formats (that we didn’t actually write parsing code for), here we are again. Parsing input from a text file, how I have missed you!

# No fancy data structures this time, just two `BitMatrix`, one representing the
# locations of east-bound cucumbers, and one representing the locations of
# sounth-bound cucumbers, on a 2D grid. Can you tell I'm ready to be done?
function ingest(path)
    eastlines  = []
    southlines = []
    open(path) do f
        for line in readlines(f)
            chars = collect(line)
            push!(eastlines,  chars .== '>')
            push!(southlines, chars .== 'v')
        end
    end
    eastmatrix  = reduce(hcat, eastlines)  |> transpose |> BitMatrix
    southmatrix = reduce(hcat, southlines) |> transpose |> BitMatrix
    return (eastmatrix, southmatrix)
end

Ah, the tried and true ‘parse the grid to a BitMatrix’ approach! With two matrices!

Part One – The Cucumbers Go Marching

It looks like, at some point, the sea cucumbers will deadlock and stop moving about so much, giving us space to land the submarine. Guess you have to land submarines, then? Never considered that, to be honest, it just hasn’t come up before. Ah well, the good news is that, even here on the ocean floor, traffic is bound to come to a standstill sooner or later. Since they’re apparently quite polite, a sea cucumber will only move if there’s an empty space in front of it at the start of a round, and all the east-bound cucumbers try to move before the south-bound cucumbers. With all that in mind, we’re ready to proceed.


# Helper Functions -------------------------------------------------------------

# For checking on the total state of cucumber migration.
function pprint(east::BitMatrix, south::BitMatrix)
    template = fill('.', size(east))
    template[east]  .= '>'
    template[south] .= 'v'
    for row in eachrow(template)
        for char in row
            print(char)
        end
        println()
    end
end

# Move all the east-bound cucumbers that can be moved
function moveeast(east::BitMatrix, south::BitMatrix)
    occupied   = east .| south      # Spaces occupied by any type of cucumber
    nextstate  = falses(size(east)) # An empty BitMatrix, same size as `east`
    ncols      = size(east, 2)      # Number of columns in the grid
    colindices = collect(1:ncols)   # Vector of column indices

    # For each pair of column index and column index + 1 (wrapped around due
    # to space-defying ocean currents)...
    for (currcol, nextcol) in zip(colindices, circshift(colindices, -1))
        # Identify the cucumbers who moved in `currcol` by the empty spaces
        # in `nextcol`
        moved = east[:,currcol]  .& .!occupied[:,nextcol]

        # Identify the cucumbers who waited in `currcol` by the occupied
        # spaces in `nextcol`
        stayed = east[:,currcol] .& occupied[:,nextcol]

        # Place the updated cucumber positions into `nextstate`
        nextstate[:,nextcol] .|= moved
        nextstate[:,currcol] .|= stayed
    end

    return nextstate
end

# Move all the south-bound cucumbers than can be moved
function movesouth(east::BitMatrix, south::BitMatrix)
    occupied   = east .| south       # Spaces occupied by any type of cucumber
    nextstate  = falses(size(south)) # An empty BitMatrix, same size as `south`
    nrows      = size(south, 1)      # Number of rows in the grid
    rowindices = collect(1:nrows)    # Vector of row indices

    # For each pair of row index and row index + 1 (wrapped around due to
    # intradimensional currents)...
    for (currrow, nextrow) in zip(rowindices, circshift(rowindices, -1))
        # Identify the cucumbers who moved in `currrow` by the empty spaces
        # in `nextrow`
        moved = south[currrow,:] .& .!occupied[nextrow,:]

        # Identify the cucumbers who waited in `currrow` by the occupied
        # spaces in `nextrow`
        stayed = south[currrow,:] .& occupied[nextrow,:]

        # Place the updated cucumber positions into `nextstate`
        nextstate[nextrow,:] .|= moved
        nextstate[currrow,:] .|= stayed
    end

    return nextstate
end

# Move the herd(s)! Get the next state for east-bound and south-bound cucumbers,
# check to see if any of their positions changed, and return the results
function move(east::BitMatrix, south::BitMatrix)
    nexteast  = moveeast(east, south)
    nextsouth = movesouth(nexteast, south)

    eastchanged  = east  .⊻ nexteast
    southchanged = south .⊻ nextsouth
    anychanged = any(eastchanged .| southchanged)

    return (anychanged, nexteast, nextsouth)
end


# Solve Part One ---------------------------------------------------------------

# Solve it! Just keep updating the cucumber positions in a loop until they don't
# change, then report the number of rounds that took.
function part1(input)
    (east, south) = input
    rounds = 0
    anychanged = true
    while anychanged
        (anychanged, east, south) = move(east, south)
        rounds += 1
    end
    return rounds
end

I bet that trip from one edge of the transit zone (?) to the other is wild! Good thing these space-warping ocean currents don’t seem to affect submarines, or we’d be in trouble…

Part Two – Reflection

If you haven’t heard, Day 25 – Part Two is always the “go back and finish the rest of the puzzles” challenge, so no extra coding needed here. Instead, I’d like to use this space to reflect on the Advent of Code puzzles for this year and my experience with them.

This is my second year with Advent of Code, as I was introduced to it last year by a friend on a community Slack channel. Just like last year, this experience created a lot of joy, frustration, annoyance, elation, and satisfaction for me, which is typical of my experience with good puzzles. Since last year, I’ve spent a lot more time honing my knowledge of algorithms and data structures, partly as a way to scratch the itch that AoC helped to create, and partly as a way of sharing this love of puzzles with others. You see, in that community Slack channel I mentioned, some friends and I have starting hosting regular meetups aimed at both spending some time solving puzzles and helping some of the folks who are newer to our community practice live-coding for technical interviews. It’s been a blast, and I look forward to continuing into the new year. (Here’s the GitHub repo if you’re curious about our meetups, anyone is welcome to join).

This year came with it’s own set of challenges, from figuring out how to see family during the holidays (thanks Omicron!) to navigating my own bout of non-COVID related illness (I’m fine, but those were an unpleasant few days). There were a few days of puzzles that set me back on my “solve AoC > write blog post” daily cycle, and I had to consider my holiday priorities as a result. In the end, since this blog post isn’t going up until a few days into January, you can probably guess how that worked out. I really enjoy solving these puzzles, and I can get a bit obsessive when I’ve got an unsolved problem lying around in my head, but time with my family is just too precious to short-change. That’s an area I can get better in, to be honest. Weird how coding puzzles can teach life lessons, too.

I’ve been working through the AoC puzzles in Julia this year, and I’m not sure if it’s because of how similar some of the concepts are to R (my most proficient language), the amount of practice I’ve put in over the year, or just general growth as a programmer, but I found thinking through these puzzles in Julia to be a pretty seamless experience. When I did this with Rust, there was definitely a learning curve related to getting the code to do the things I had in my head, but it just seemed to flow a lot more smoothly in Julia. There’s a lot of really ergonomic syntax in Julia, enabled in part by it being a dynamic language, but I don’t think that’s the whole story. My recent time in Rust encouraged me to add Types in lots of places that, according to the Julia documentation and other talks/articles I’ve seen and read, I really didn’t have to and probably shouldn’t have. Frankly, I like Types, and I find being explicit about data types in my code tends to make it easier for me to reason about and clarifies what I did when I come back to it later. In particular, I found the syntax for single-line function assignment, multiple dispatch, native multidimensional array support, and the breadth and diversity of the Base package to be some of the really compelling features of Julia. The speed gains over R and Python (those operations not directly backed by Rust or C or FORTRAN) are significant, but not always as significant as I would have hoped. There’s probably a lot of room for me to improve in optimizing Julia code, though, so I suspect I’ve left a lot of performance on the table. Guess this means I’ll have to keep writing Julia code, then. As a next step, I’d like to get familiar with the Dataframes.jl and Makie.jl (data frames and plotting) packages, since that’s the kind of work I most often tend to do for my day job. Who knows, maybe Julia will inspire me to finally dabble in a bit of machine learning? I hear that’s a pretty hot topic…

I’ve really enjoyed writing this blog series. As usual, explaining how I did things (even to what I imagine to be, like, five random people on the internet who stumble across this site or these articles wherever they get posted) really helped me to clarify, correct, refactor, and update my own understanding. I recommend coding puzzles regularly to new and aspiring programmers, but I’d also recommend finding someone who could benefit from your knowledge (whatever level you feel like you’re at) and giving teaching/explaining a go. Communication skills are important for growing your career (and being a good human), and I feel like teaching, even a little, is a great way to grow those skills and your understanding of the topic.

Just like last year, I’d like to send out a huge thank you to Eric Wastl and his crew of helpers for putting this event/phenomenon on and keeping it running. If you can, please consider donating to this effort to help keep the magic going, I guarantee there will be a lot of coders (215,955 completed at least one puzzle this year, as of the time of writing) who will be glad you did.

If you’re interested, you can find all the code from this blog series on GitHub, please feel free to submit a pull request if you’re really bored. Happy Holidays!