Author Archives: Josh Day

Artifacts!

By: Josh Day

Re-posted from: https://www.juliafordatascience.com/artifacts/

Artifacts!

In this article we'll cover what artifacts are and how to use them, both as a package user and a package developer.

What is an Artifact?

In Julia, an artifact is simply a fixed/immutable something that a package needs (that isn't Julia code).  This could be an executable binary, a text file, or any other kind of immutable data.  

The "rules" for artifacts are the following:

  • Artifacts must be immutable.
  • Artifacts must be a (optionally g-zipped) tarfile (ends with .tar or .tar.gz).
  • Artifacts must be publicly available on the internet.

What's the Point of Artifacts?

To understand the usefulness of artifacts, let's take a look at some alternatives:

❌ Storing binary data directly in the git repo.

This is bad practice because git is only meant for tracking changes in text files.  A change to a binary file will require git to save an entirely new version of the file.

❌ Using a deps/build.jl script.

Before artifact support was added to Pkg, this was how you included artifacts along with a package.  When you install a package, this script will run (if it exists).  The script can include downloading files the package needs.  

Downsides to this approach are:

  • If two packages use the same artifact, they will both download their own copy.
  • Packages that use this method are incompatible with PackageCompiler.jl.
  • It is tricky to get platform-specific dependencies working properly.

In other words, Pkg's artifact support allows you to:

✅ Avoid bloat in git repos.
✅ Make source code immutable (a package directory's state isn't changed by a build.jl script).
✅ Have packages that share dependencies.
✅ Use PackageCompiler.
✅ Install platform-specific dependencies in a more robust way.

Artifacts as a Package User

As a user, you don't need to make yourself concerned with artifacts.  Whenever a package is installed on your machine, Julia's package manager will automatically download the artifacts it needs!

Artifacts as a Package Developer

As a developer, we highly recommend using the ArtifactUtils package.  See the docs for more info, but here we'll walk through creating a package from scratch that needs an artifact.

1) Let's make a package called Hello

using PkgTemplates

t = Template()

generate(t, "Hello")
c

2) Navigate to the root directory of Hello

path = joinpath(homedir(), ".julia", "dev", "Hello")

cd(path)

3) Create what we need

  • We'll write a file called "hello.txt" into a temporary directory with the contents "Hello!  I am an artifact!".
dir = mktempdir()

file = touch(joinpath(dir, "hello.txt"))

open(file, "w") do io 
    write(io, "Hello!  I am an artifact!")
end

4) Get the ID for our artifact

using ArtifactUtils

artifact_id = artifact_from_directory(dir)

5) Upload the artifact somewhere

gist = upload_to_gist(artifact_id)

6) Create an Artifacts.toml in your Hello package directory

add_artifact!("Artifacts.toml", "hello_artifact", gist)

7) Use the artifact in the package

  • Every package can have an __init__ function that runs right after the package is loaded.  In this case, we'll use __init__ to print out the contents of our "hello.txt" artifact.
sourcecode = """
module Hello 

using Artifacts 

function __init__()
    path = joinpath(artifact"hello_artifact", "hello.txt")
    
    println(read(path,String))
end

end #module
"""

open(io -> write(io, sourcecode), joinpath("src", "Hello.jl"), "w")

8) Add the Artifacts dependency and you're done!

using Pkg

# activate the Hello project
Pkg.activate(".")  

# Make sure we add the Artifact dependency.
Pkg.add("Artifacts")  

# get all of Hello's dependencies including artifacts
Pkg.instantiate()     

🎉 Hey it works!

julia> using Hello
Hello!  I am an artifact!


🚀 That's It!

You now know the basics of Julia's artifact system!

Still confused about something?  Did we miss anything important?  Let us know on Twitter at @JuliaForDataSci!


Enjoying Julia For Data Science?  Please share us with a friend and follow us on Twitter at @JuliaForDataSci.

Want to write an article for Julia For Data Science?  Get in touch!  juliafordatascience@gmail.com


Julia For Data Science Numbers:

Hey, this section is new.  We thought it would be fun to include some statistics about Julia For Data Science.  We'll include our member/follower numbers in each post from here on out.

  • Newsletter members: 166
  • Twitter followers: 836

Cheatsheets

By: Josh Day

Re-posted from: https://www.juliafordatascience.com/cheat-sheets/

Enjoying Julia For Data Science?  Please share us with a friend and follow us on Twitter at @JuliaForDataSci.

Cheatsheets

This post is an attempt to aggregate all of the cheatsheet resources that Julia community members have created (last updated Nov 9, 2021).

The Cheatsheets

* = Not a cheetsheet per se, but worthy of inclusion.

Julia and Comparisons to Other Languages

Specific Packages

What have we missed?

Please let us know! Contact us via Twitter @JuliaForDataSci or email us at juliafordatascience@gmail.com.

Using JSON Web APIs from Julia

By: Josh Day

Re-posted from: https://www.juliafordatascience.com/how-to-wrap-a-json-web-api/

Enjoying Julia For Data Science?  Please share us with a friend and follow us on Twitter at @JuliaForDataSci.

Using JSON Web APIs from Julia

A common way to access data is through web APIs that return a JSON response.  In this post we'll look at how the XKCD.jl package implements a user-friendly interface for accessing the xkcd webcomic's JSON interface.

Getting the Data

The xkcd webcomic's JSON interface accepts request of the form:

  1. https://xkcd.com/info.0.json (metadata for most recent comic)
  2. https://xkcd.com/<i>/info.0.json (metadata for i-th comic)

We'll use the HTTP package's get function to retrieve this info:

julia> using HTTP

julia> res = HTTP.get("https://xkcd.com/552/info.0.json");

Interpret Data as JSON

We can then read our result body into JSON using the JSON3 package:

julia> using JSON3

julia> JSON3.read(res.body)
JSON3.Object{Vector{UInt8}, Vector{UInt64}} with 11 entries:
  :month      => "3"
  :num        => 552
  :link       => ""
  :year       => "2009"
  :news       => ""
  :safe_title => "Correlation"
  :transcript => "[[A man is talking to a woman]]\nMan: I used to think correla…
  :alt        => "Correlation doesn't imply causation, but it does waggle its e…
  :img        => "https://imgs.xkcd.com/comics/correlation.png"
  :title      => "Correlation"
  :day        => "6"

Making it User-Friendly

To make using the xkcd JSON interface easier to use, let's create a struct to hold the JSON data and let its constructor do the heavy lifting:

struct Comic 
    json::JSON3.Object
end

Comic() = Comic(JSON3.read(HTTP.get("https://xkcd.com/info.0.json").body))
    
Comic(i::Int) = Comic(JSON3.read(HTTP.get("https://xkcd.com/$i/info.0.json").body))

🎉 Now getting comic metadata is as simple as Comic(552)!

What about Authorization?

The xkcd API does not require authorization.  However, many APIs require that you purchase an API key in order to access them.  In this case, it's best practice to try to ensure API keys won't end up being accidentally saved in git (or other version control).

To do this, we can rely on environmental variables (ENV in Julia).  Suppose the xkcd API requires an apikey parameter.  We would want to change our Comic constructor to something like:

function Comic(i::Int)
    apikey = ENV["MY_API_KEY"]
    url = "https://xkcd.com/$i/info.0.json?apikey=$apikey"
    data = JSON3.read(HTTP.get(url).body)
    Comic(data)
end
Imaginary apikey parameter in xkcd API

Other Neat Stuff

Since we working with images (comics) in the XKCD.jl package, we can add a pretty show method so that the comic image (not just the metadata) will appear in coding environments like Jupyter and Pluto.

function Base.show(io::IO, ::MIME"text/html", c::Comic)
    show(io, MIME"text/html"(), HTML("""
	   <div>
        <h2><code>XKCD.Comic</code> $(c.json.num): $(c.json.title)</h2>
        <img src="$(c.json.img)" alt="$(c.json.alt)" title="$(c.json.alt)">
		<div>
          <a href="https://xkcd.com/$(c.json.num)">
            Link to Original
          </a>
	    </div>
      </div>
    """))
end
Using JSON Web APIs from Julia
XKCD.jl + Pluto.jl = ❤️

🚀 That's It!

Enjoying Julia For Data Science?  Please share us with a friend and follow us on Twitter at @JuliaForDataSci.