Tag Archives: Julia

JuliaCon 2020 in Retrospective

By: Andrew Rosemberg, Chris Davis, Glenn Moynihan, Matt Brzezinski, and Will Tebbutt

Re-posted from: https://invenia.github.io/blog/2020/08/12/juliacon/

We at Invenia are heavy users of Julia, and are proud to once again have been a part of this year’s JuliaCon. This was the first year the conference was fully online, with about 10,000 registrations and 26,000 people tuning in. Besides being sponsors of the conference, Invenia also had several team members attending, helping host sessions, and presenting some of their work.

This year we had five presentations: “Design documents are great, here’s why you should consider one”, by Matt Brzezinski; “ChainRules.jl”, by Lyndon White; “HydroPowerModels.jl: Impacts of Network Simplifications”, by Andrew Rosemberg; “Convolutional Conditional Neural Processes in Flux”, by Wessel Bruinsma; and “Fast Gaussian processes for time series”, by Will Tebbutt.

JuliaCon always brings some really exciting work, and this year it was no different. We are eager to share some of our highlights.

JuliaCon is not just about research

There were a lot of good talks and workshops at JuliaCon this year, but one which stood out was “Building microservices and applications in Julia”, by Jacob Quinn. This workshop was about creating a music album management microservice, and provided useful information for both beginners and more experienced users. Jacob explained how to define the architectural layers, solving common problems such as authentication and caching, as well as deploying the service to Google Cloud Platform.

A very interesting aspect of the talk was that it exposed Julia users to the field of software engineering. JuliaCon usually has a heavy emphasis on academic and research-focused talks, so it was nice to see the growth of a less represented field within the community. There were a few other software engineering related talks, but having a hands-on practical approach is a great way to showcase a different approach to architecting code.

Among the other software engineering talks and posters, we can highlight “Reproducible environments with Singularity”, by Steffen Ridderbusch; the aforementioned “Design documents are great, here’s why you should consider one”, by Matt Brzezinski; “Dispatching Design Patterns”, by Aaron Christianson; and “Fantastic beasts and how to show them”, by Joris Kraak.

But it stays strong in the machine learning community

The conference kicked off with a brief and fun session on work related to Gaussian processes, including our own Will Tebbutt who talked about TemporalGPs.jl, which provides fast inference for certain types of GP models for time series, as well as Théo Galy-Fajou’s talk on KernelFunctions.jl. Although there was no explicit talk on the topic, there were productive discussions about the move towards a common set of abstractions provided by AbstractGPs.jl.

It was also great to see so many people at the Probabilistic Programming Bird of a Feather, and it feels like there is a proper community in Julia working on various approaches to problems in Probabilistic Programming. There were discussions around helpful abstractions, and whether there are common ones that can be more widely shared between projects. A commitment was made to having monthly discussions aimed at understanding how the wider community is approaching Probabilistic Programming.

Another interesting area that ties into both our work on ChainRules.jl, the AD ecosystem and the Probabilistic Programming world, is Keno Fischer’s work. He has been working on improving the degree to which you can manipulate the compiler and changing the points at which you can inject additional compiler passes. This intends to mitigate the type-inference issues that plague Cassette.jl and IRTools.jl. Those issues lead to problems in Zygote.jl (and other tools). We expect great things from changes to how compiler pass injection works with the compiler’s usual optimisation passes.

Finally, Chris Elrod’s work on LoopVectorization.jl is very exciting for performance. His talk contained an interesting example involving Automatic Differentiation (AD), and we’re hoping to help him integrate this insight into ChainRules.jl in the upcoming months.

As well as in the engineering community

This year we saw a significant number of projects on direct applications to engineering, including interesting work on steel truss design and structural engineering. Part of why the engineering community is fond of Julia is the type structure paired with multiple dispatch, which allows developers to easily extend types and functions from other packages, and build complex frameworks in a Lego-like manner.

A direct application of Julia in engineering that leverages the existing ecosystem is HydroPowerModels.jl, developed by our own Andrew Rosemberg. HydroPowerModels.jl is a tool for planning and simulating the operation of hydro-dominated power systems. It builds on three main dependencies (PowerModels.jl, SDDP.jl, and JuMP.jl) to efficiently construct and solve the desired problem.

The pipeline for HydroPowerModels.jl uses PowerModels.jl—a package for parsing system data and modeling optimal power flow (OPF) problems—to build the OPF problem as a JuMP.jl model. Then the model is modified in JuMP.jl to receive the appropriate hydro variables and constraints. Lastly, it is passed to SDDP.jl, which builds the multistage problem and provides a solution algorithm (SDDP) to solve it.

There were several tools for working with networks and graphs

As a company that works on problems related to electricity grids, new developments on how to deal with networks and graphs are always interesting. Several talks this year featured useful new tools.

GeometricFlux.jl adds to Flux.jl the capability to perform deep learning on graph-structured data. This area of research is opening up new opportunities in diverse applications such as social network analysis, protein folding, and natural language processing. GeometricFlux.jl defines several types of graph-convolutional layers. Also of particular interest is the ability to define a FeaturedGraph, where you specify not just the structure of the graph, but can also provide feature vectors for individual nodes and edges.

Practical applications of networks were shown in talks on economics and energy systems.

Work done by the Federal Reserve Bank of New York on Estimation of Macroeconomic Models showed how Julia is being applied to speed up calculations on equilibrium models, which are a classic way of simulating the interconnections in the economy and how interventions such as policy changes can have rippling impacts through the system. Similarly, work by the National Renewable Energy Laboratory (NREL) on Intertwined Economic and Energy Analysis using Julia demonstrated equilibrium models that couple economic and energy systems.

Quite a few talks dealt specifically with power networks. These systems can be computationally challenging to model, particularly when considering the complexity of actual large-scale power grids and not simple test cases. NetworkDynamics.jl allows for modelling dynamic systems on networks, by bridging existing work in LightGraphs.jl and DifferentialEquations.jl. This has, in turn, been used to help build PowerDynamics.jl. Approaches to speed up power simulations were discussed in A Parallel Time-Domain Power System Simulation Toolbox in Julia. Finally, another talk by NREL on a Crash Course in Energy Systems Modeling & Analysis with Julia showed off a collection of packages for power simulations they are developed.

This year the whole event happened online

It may not have been the JuliaCon we envisioned, but the organisers this year did an incredible job in adjusting to extraordinary circumstances and hosting an entirely virtual conference.

A distinct silver lining in moving online is that attendance was free, which opened the conference up to a much larger community. The boost in attendance no doubt increased the engagement with contributors to the Julia project and provided presenters with a much wider audience than would otherwise be possible in a lecture hall.

Even with the usual initialization issues with conference calls (“Can you hear me now?”), the technical set-up of the conference was superb. In previous years, JuliaCon had the talks swiftly available on YouTube and this year they outdid themselves by simultaneously live-streaming multiple tracks. Being able to pause and rewind live talks and switch between tracks without leaving the room made for a convenient viewing experience. The Discord forum also proved great for interacting with others and for asking questions in a manner that may have appealed to the more shy audience members.

Perhaps the most pivotal, yet inconspicuous, benefit of hosting JuliaCon online is the considerably reduced carbon footprint. Restricted international movement has brought to light the travel industry’s impact on the planet and international conferences have their role to play. Maybe the time has come for communities that are underpinned by strong social and scientific principles, like the Julia community, to make the reduction of emissions an explicit priority in future gatherings.

In spite of JuliaCon’s overall success, there are still kinks to iron out in the online conference experience: the digital interface makes it difficult to spontaneously engage with other participants, which tends to be one of the main reasons to attend conferences in the first place, and the lack of “water cooler”-talk (although Gather.Town certainly helped in providing a similar experience) means missed connections and opportunities for ideas to cross-pollinate. Not for a lack of trying, JuliaCon seemed to miss an atmosphere that can only be captured by being in the same physical space as the community. We don’t doubt that the online experience will improve in the future one way or the other, but JuliaCon certainly hit the ground running.

We look forward to seeing what awaits for JuliaCon 2021, and we’ll surely be part of it once more, however it happens.

Alien facehugger wasps, a pandemic, webcrawlers and julia

By: Ömür Özkir

Re-posted from: https://medium.com/oembot/alien-facehugger-wasps-a-pandemic-webcrawlers-and-julia-c1f136925f8?source=rss----386c2bd736a1--julialang

collect and analyze covid 19 numbers for Hamburg, Germany

TL;DR

  1. Build a webcrawler in julia.
  2. Use the data for a simple plot.

Motivation

The web is full of interesting bits and pieces of information. Maybe it’s the current weather, stock prices, or the wikipedia article about the wasp that goes all alien parasite facehugger on other insects, which you vaguely remember from one of those late night documentaries (already sorry you are reading this?).

If you are lucky, that data is available via an API, making it usually pretty easy (not always tho, if API developers come up with byzantine authentications, required headers or other elegant/horrible designs, the fun is over) to get to the data.

A lot of the shiny nuggets are not available via a nice API, tho. Which means, we have to crawl webpages.

Pick something that you are really interested in / want to use for a project of yours, that’ll make it less of a chore and far more interesting!

Local = Relevant

For me, currently, that’s the Covid-19 pandemic. And more specifically, how it is developing close to me. In my case, that means the city of Hamburg in Germany.

Chances are, these specific numbers/case is not relevant to you. But that’s a good thing, you can use what you learned here and mine the website of your home city maybe (or whatever you are interested in).

Nothing helps your brain absorb new things better than generalizing those new skills and using them to solve related problems!

There is the official website of the city, that has a page for the covid-19 numbers, hamburg.de.

The page with the numbers is in german, but don’t worry, that’s what our webcrawler will hopefully help us with — we can get to the numbers without having to understand all the surrounding text. I will try to help out and translate what is relevant, but that will only be a minor detail when we try to find the right text to extract from.

If you like, you can check out the notebook or even code along in binder.

First, let’s get some of the dependencies out of the way:

Aside from HTTP.jl to request the website, we will also use Gumbo to parse html and Cascadia to extract data from the html document via selectors.

using HTTP
using Gumbo, Cascadia
using Cascadia: matchFirst

We need to fetch and parse the website, which is easily done with Gumbo.

url = "https://www.hamburg.de/corona-zahlen"
response = HTTP.get(url)
html = parsehtml(String(response))
# =>
HTML Document:
<!DOCTYPE >
HTMLElement{:HTML}:<HTML lang="de">
  <head></head>
  <body class="no-ads">
    HTTP/1.1 200 OK
    ServerHost: apache/portal5
    X-Frame-Options: SAMEORIGIN
    Access-Control-Allow-Origin: *
    Content-Type: text/html;charset=UTF-8
    Content-Language: de-DE
    Date: Wed, 05 Aug 2020 19:30:23 GMT
    Transfer-Encoding: chunked
    Connection: keep-alive, Transfer-Encoding
    Set-Cookie: JSESSIONID=AAF197B2F1191AACC08B70C4F8DAB18F.liveWorker2; Path=/; HttpOnly
    Set-Cookie: content=13907680; Path=/servlet/segment/de/corona-zahlen/
    Set-Cookie: BIGipServerv5-webstatic-80-12-cm7=201658796.20480.0000; path=/; Httponly; Secure
    <meta content="IE=edge" http-equiv="X-UA-Compatible"/>
    <meta charset="utf-8"/>
    <meta content="text/html" http-equiv="content-type"/>
    <script type="text/javascript">window.JS_LANG='de';    </script>
    <meta content="width=device-width, initial-scale=1.0" name="viewport"/>
...

Alright, we can now start to parse data from the html document, by using query selectors.

You know, they might actually be called css selectors, I don’t know. How precise is the frontend terminology anyways, right?

Oh look, a pack of wild frontenders! Hmm, what are they doing, are they encircling us? They do look kinda angry, don’t they? I guess they just tried to vertically align some div the whole day or something.

Ok… I guess we should leave now.

Seems important

hamburg.de: Visual hierarchy, the topmost information is usually something important

We could start with the first information we see on the page, after all, there must hopefully be a reason that it is at the top of the page.

The three bullet points with the numbers mean confirmed cases, recovered and new cases. Now the trick is to find the best selectors. There are a few plugins for the different browsers that help finding the right selectors quickly.

But it is also pretty easy to do by hand. When right-click/inspecting an element on the page (this requires the developer tools) one can pretty quickly find a decently close selector.

If you want to test it out in the browser first, you can write something like this document.querySelectorAll(".c_chart.one .chart_legend li") in the browser console. Some browsers even highlight the element on the page when you hover over the array elements of the results.

Using the selectors in julia is pretty neat:

eachmatch(sel".c_chart.one .chart_legend li", html.root)
# => 
3-element Array{HTMLNode,1}:
 HTMLElement{:li}:<li>
  <span style="display:inline-block;width:.7em;height:.7em;margin-right:5px;background-color:#003063"></span>
  Bestätigte Fälle 5485
</li>


 HTMLElement{:li}:<li>
  <span style="display:inline-block;width:.7em;height:.7em;margin-right:5px;background-color:#009933"></span>
  Davon geheilt 5000
</li>


 HTMLElement{:li}:<li>
  <span style="display:inline-block;width:.7em;height:.7em;margin-right:5px;background-color:#005ca9"></span>
  Neuinfektionen 25
</li>

Ok, we need to extract the numbers from the text of each html element. Using a simple regex seems like the easiest solution in this case. Check this out, it looks very similar to the previous selector matching:

match(r"\d+", "Neuinfektionen 25")
# =>
RegexMatch("25")

Nice, huh? Ok, but we only need the actual match.

match(r"\d+", "Neuinfektionen 25").match
# =>
"25"

And we need to cast it to a number:

parse(Int, match(r"\d+", "Neuinfektionen 25").match)
# =>
25

We want to do this for each element now, so we extract the text from the second node (the span element is the first, see the elements above).

Then we do all the previously done matching and casting and we got our numbers!

function parsenumbers(el)
    text = el[2].text
    parse(Int, match(r"\d+", text).match)
end
map(parsenumbers, eachmatch(sel".c_chart.one .chart_legend li", html.root))
# =>
3-element Array{Int64,1}:
 5485
 5000
   25

Learning how to Date in german

We should also extract the date when those numbers were published. The selector for the date on the page is very easy this time: .chart_publication.

In the end we want some numbers, that we can use to instantiate a Date object, something like this Date(year, month, day).

We are starting out with this, however:

date = matchFirst(sel".chart_publication", html.root)[1].text
# =>
"Stand: Mittwoch, 5. August 2020"

Oh dear, it’s in german again. We need "5. August 2020" from this string.

parts = match(r"(\d+)\.\s*(\S+)\s*(\d{4})", date).captures
# =>
3-element Array{Union{Nothing, SubString{String}},1}:
 "5"
 "August"
 "2020"

Better, but it’s still in german!

Ok, last bit of german lesson, promised, how about we collect all the month names in a tuple?

Then we can find it’s index in the tuple. That would be the perfect input for our Date constructor.

const MONTHS = ("januar", "februar", "märz", "april", "mai", "juni", "juli", "august", "september", "oktober", "november", "dezember")
findfirst(m -> m == lowercase(parts[2]), MONTHS) # => 8
using Dates
Date(parse(Int, parts[3]),
    findfirst(m -> m == lowercase(parts[2]), MONTHS),
    parse(Int, parts[1]))
# => 2020-08-05

More local = more relevant!

There are a few more interesting nuggets of information, I think the hospitalization metrics would be very interesting, especially to investigate the correlation between when cases are confirmed and the delayed hospitalizations.

But one thing that is especially interesting (and I don’t think such locally detailed information is available anywhere else) are the number of cases in the last 14 days, for each borough.

Speaking of local, this is probably the most local we can get.

List of boroughs, number of new infections aggregated for the last 14 days

By now, you probably start to see a pattern:

  1. find good selector
  2. extract content
  3. parse/collect details
rows = eachmatch(sel".table-article tr", html.root)[17:end]
df = Dict()
foreach(rows) do row
    name = matchFirst(sel"td:first-child", row)[1].text
    num = parse(Int, matchFirst(sel"td:last-child", row)[1].text)
    df[name] = num
end
df
# =>
Dict{Any,Any} with 7 entries:
  "Bergedorf"     => 17
  "Harburg"       => 28
  "Hamburg Nord"  => 26
  "Wandsbek"      => 63
  "Altona"        => 14
  "Eimsbüttel"    => 12
  "Hamburg Mitte" => 41

great, that’s it?

No! No, now the real fun begins. Do something with the data! You will probably already have some idea what you want to do with the data.

How about ending this with something visual?

Visualizations, even a simple plot, can help a lot with getting a feel for the structure of the data:

using Gadfly
Gadfly.set_default_plot_size(700px, 300px)

There are a lot of great plotting packages for julia, I personally really like Gadfly.jl for its beautiful plots.

plot(x=collect(keys(df)), 
    y=collect(values(df)), 
    Geom.bar,
    Guide.xlabel("Boroughs"),
    Guide.ylabel("New Infections"),
    Guide.title("New infections in the last 14 days"),
    Theme(bar_spacing=5mm))
Even such a simple plot already helps understanding the data better, right?

The end! Right?

Ha ha ha ha- nope. Webcrawlers are notoriously brittle, simply because the crawled websites tend to change over time. And with it, the selectors. It’s a good idea to test if everything works, once in a while, depending on how often you use your webcrawler.

Be prepared to maintain your webcrawler more often than other pieces of software.

A few things to check out

Very close to the topic: I created a little package, Hamburg.jl, that has a few datasets about Hamburg, including all the covid-19 related numbers that we scraped a little earlier.

The official julia docs should get you up and running with your local julia dev setup.

One more crawler

Ok, one more thing, before I let you off to mine the web for all its information:

html = parsehtml(String(HTTP.get("https://en.wikipedia.org/wiki/Emerald_cockroach_wasp")))
ptags = eachmatch(sel".mw-parser-output p", html.root)[8:9]
join(map(n -> nodeText(n), ptags))
# =>
"Once the host is incapacitated, the wasp proceeds to chew off half of each of the roach's antennae, after which it carefully feeds from exuding hemolymph.[2][3] The wasp, which is too small to carry the roach, then leads the victim to the wasp's burrow, by pulling one of the roach's antennae in a manner similar to a leash. In the burrow, the wasp will lay one or two white eggs, about 2 mm long, between the roach's legs[3]. It then exits and proceeds to fill in the burrow entrance with any surrounding debris, more to keep other predators and competitors out than to keep the roach in.\nWith its escape reflex disabled, the stung roach simply rests in the burrow as the wasp's egg hatches after about 3 days. The hatched larva lives and feeds for 4–5 days on the roach, then chews its way into its abdomen and proceeds to live as an endoparasitoid[4]. Over a period of 8 days, the final-instar larva will consume the roach's internal organs, finally killing its host, and enters the pupal stage inside a cocoon in the roach's body.[4] Eventually, the fully grown wasp emerges from the roach's body to begin its adult life. Development is faster in the warm season.\n"

…the wasp proceeds to chew off half of each of the roach’s antennae, after which it carefully feeds from exuding…

…what…

…The hatched larva lives and feeds for 4–5 days on the roach, then chews its way into its abdomen…

…the…

…Over a period of 8 days, the final-instar larva will consume the roach’s internal organs, finally killing its host…

…hell mother nature, what the hell…

Alien facehugger wasps, a pandemic, webcrawlers and julia was originally published in oembot on Medium, where people are continuing the conversation by highlighting and responding to this story.