projects | juliabloggers.com

Julia is a young programming language.
This means that its native libraries are immature.
We are in a time when Julia is a mature enough as a language that it is out-pacing its libraries.

One way to use mature libraries from a young language is to borrow them from another language.
In this case, we’ll be borrowing from Python.
(Julia can also easily wrap libraries from C or Fortran.
In fact, this capability was important in combination with Python’s great C-interface to make calling Python from Julia do-able.)

Python from Julia: PyCall.jl

Calling Python from Julia requires PyCall.
You can get it using Pkg.add("PyCall").

Let’s start with a “Hello, World” scale example.

using PyCall
pyeval("2+2") #=> 4
pyeval("str(5)") #=> "5"

# doing things by hand, for fun :)
math = pyimport(:math) #=> PyObject <module 'math'>
pycall(math["sin"],Float64,1) #=> 0.8414709848078965

Using Python Libraries from Julia

For a quick practical example of Python libraries filling in Julia’s current gaps, we can use Python’s matplotlib for graphing.
If you also install matplotlib,
you can run this fun code snippet I found on the mailing list.
If you run it, it will pop up a window with a graph of a nice dotted, squiggley, red line.

using PyCall
@pyimport pylab
x = linspace(0,2*pi,1000); y = sin(3*x + 4*cos(2*x));
pylab.plot(x, y; color="red", linewidth=2.0, linestyle="--")
pylab.show()

pylab is the name for the Python module that includes matplotlib and numpy
in a single namespace, according to matplotlib’s docs;
it is the recommended interface to matplotlib.
@pyimport is a Julia macro that is equivalent to a using statement on the Julia equivalent of the Python module.
What this means is that we can now use pylab as if it’s a Julia module, which is what the dot-notation (pylab.plot) is taking advantage of.
We use two functions from pylab: plot and show.
Notice that we don’t have to do anything special when we invoke them:
they look just like Julia functions, right down to the keyword arguments.

Julia from Python: the julia module

This is the less polished direction;
it’s not really intended for public consumption yet.
To get the Python module that you need to call Julia, git clone https://github.com/JuliaLang/IJulia.jl.
Then, inside the IJulia.jl/python/ folder, run python setup.py install. (you may need sudo on that last command)

First, the opening incantation:

import julia
j = julia.Julia()

The second line will take a while to run; it’s starting and setting up Julia.

Now, you’ll be able to call Julia from the Python REPL.
For example:

j.run("2+2") #=> 4
j.run("sin(pi)") #=> 1.2246467991473532e-16
j.run("x = 5") #=> 5
j.run("x += 2") #=> 7
j.run("x") #=> 7

Let’s be best friends: Mutual Recursion.

This section is a code example.
We’ll start at the top, and then wander down through the implementation.
You can find all the code on github.

If you clone that repo, you can open a Python REPL up inside that folder and run the following:

import pyiseven
pyiseven.even(5)

As one would hope, you’ll get False back.
The import line probably took a really long time to load; the even function seems pretty straight forward given its name and behavior.
But, let’s take a look at pyiseven.py anyway.

import julia
j = julia.Julia()

def even(x):
  j.run("using IsOdd")
  return not j.run("IsOdd.odd(" + str(x) + ")")

Oh no!
This looks terribly inefficient: even if Julia is 20x faster than Python, calling another programming language to figure out if your number is odd is a bad plan.

Maybe we should take a look at IsOdd.jl:

module IsOdd

using PyCall
@pyimport pyiseven

function odd(x)
  if x == 1
    true
  elseif x == 0
    false
  else
    pyiseven.even(x-1)
  end
end

end

Oh no! It gets worse!
We make another function call back into Python’s even function on every call to Julia’s odd.
That means that it takes two function calls (and two trips between languages) for each increment of even‘s argument.
(In fact, if you call pyiseven.even(201) (or greater), then the stack explodes, unfortunately.)
If you find that your production code is too slow because you’re using mutual recursion between nine different languages, blame Dan Luu for this terrible idea.

tl;dr

You should really check out PyCall, which does a great job of translating between Python and Julia values and modules.

If you’ve been using IPython notebook, then you should also try IJulia notebook. (Same frontend, different language). You may have noticed that the IJulia project’s repo is the origin of that Python julia module.

Here are some examples of starting and interacting with other programs from Julia.
The official documentation is pretty good, but I want something with more (basic) examples and fewer words.
I do reccommend reading that to see some of the fancier tricks you can pull
(and for up-to-date documentation, when this blog post gets stale).

Running Other Programs

The easiest, most basic way to run a shell command is run.

julia> help(run)
Base.run(command)

   Run a command object, constructed with backticks. Throws an error
   if anything goes wrong, including the process exiting with a non-
   zero status.

You can’t communicate with the process at all,
and run will block until the command finishes running.

julia> run(`echo hello!`)
hello!

Notice that Command literals are written with backticks, not single quotes (').

You can run two commands in parallel using &:

julia> run(`echo hello` & `echo world`)
hello
world

If you try to cat a file that doesn’t exist, you’ll get an error.

julia> run(`cat test.txt`)
cat: test.txt: No such file or directory
ERROR: failed process: Process(`cat test.txt`, ProcessExited(1)) [1]
 in error at error.jl:22
 in pipeline_error at process.jl:430
 in run at process.jl:413

The first line is cat printing an error message.
The rest is Julia throwing an error.
Note that the command is containedin backticks, not normal single quotes.

Redirecting

Base.|>

   Redirect standard input or output of a process.

   Example: run(`ls` |> "out.log")
   Example: run("file.txt" |> `cat`)

Use |> to redirect the STDOUT of a command to a file
or to write a file to a command’s STDIN.

julia> run(`echo hello, world` |> "test.txt")

julia> run(`cat test.txt`)
hello, world

julia> run("test.txt" |> `cat`)
hello, world

You can also use |> to redirect a process’s output to another process.

julia> run(`echo hello` |> `cat`)
hello

julia> run(`echo $("hellonhinhello, world")` |> `cat` |> `grep -n o`)
1:hello
3:hello, world

String Interpolation

You can use $ to interpolate into Command literals, in the same way you can into string literals.

julia> filename = "testnnewline.txt"
"testnnewline.txt"

You can see what the interpolation expands to by just printing the Command literal.
For example, this doesn’t run or do anything, it just shows you what the command looks like.

julia> `cat $filename`
`cat 'test
newline.txt'`

If you run the command without having that file around, you’ll get an error because cat doesn’t exit cleanly.

julia> run(`cat $filename`)
cat: test
newline.txt: No such file or directory
ERROR: failed process: Process(`cat 'test
newline.txt'`, ProcessExited(1)) [1]
 in error at error.jl:22
 in pipeline_error at process.jl:430
 in run at process.jl:413

Creating the file resolves the error:

julia> run(`echo the name is $filename` |> filename)

julia> run(`cat $filename`)
the name is test
newline.txt

The rest of this section’s examples are mostly from the official documentation.

julia> names = ["foo","bar","baz"]
3-element ASCIIString Array:
 "foo"
 "bar"
 "baz"

Command interpolation is more specialized than just plain string interpolation.
Lists will become escaped and space separated, which is more reasonable in shell commands than the normal square-brackets-and-commas notation.

julia> `grep xylophone $names`
`grep xylophone foo bar baz`

If you interpolate a list as part of a word, the resulting space separated list will be versions of the complete word:

julia> `grep xylophone $names.txt`
`grep xylophone foo.txt bar.txt baz.txt`

julia> `grep xylophone hi$names.bye`
`grep xylophone hifoo.bye hibar.bye hibaz.bye`

julia> `grep xylophone hi$(names)bye`
`grep xylophone hifoobye hibarbye hibazbye`

You can do even cooler things if you have two arrays.
If you interpolate two arrays into the same word,
then you’ll get all combinations of elements in the two lists.

julia> exts = ["aux","log"]
2-element ASCIIString Array:
 "aux"
 "log"

julia> `rm -f $names.$exts`
`rm -f foo.aux foo.log bar.aux bar.log baz.aux baz.log`

Getting Output

The easiest way to call a command and get it’s output as a String is readall.

julia> readall(`echo hello?`)
"hello?n"

This words just as well with chains of commands and redirections
because the chain will become a single Cmd value
when the redirectin operators are done.

julia> readall("test.txt" |> `cat`)
"hello, worldn"

If you just want the contents of a file as a string, you don’t need to use cat or any other command. Just use readall on the file:

julia> readall("test.txt")
"hello, worldn"

However, sometimes you’d like to be able to read the output in over time, rather than all at once.
For that, you want readsfrom:

julia> help(readsfrom)
Base.readsfrom(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream reading from the
   process' standard output.

julia> (st,pr) = readsfrom(`cat poem`)
(Pipe(active, 0 bytes waiting),Process(`cat poem`, ProcessRunning))

This use of readsfrom isn’t that interesting, since cat exits right away.

julia> pr
Process(`cat poem`, ProcessExited(0))

You can use any reading stream functions you’d like: readall, readline, etc.

julia> readline(st)
" There is a place where the sidewalk endsn"

The pipe is closed because the process has exited.

julia> st
Pipe(closed, 718 bytes waiting)

This way, you can process the data one line (or whatever increment you like) at a time.

julia> readline(st)
"And before the street begins,n"

julia> readline(st)
"And there the grass grows soft and white,n"

julia> st
Pipe(closed, 646 bytes waiting)

When you run out of data in the pipe, readline will start returning empty strings: "".

Want to ls, but get the result as an array of file and directory names?
You want to use readdir()

julia> help(readdir)
# methods for generic function readdir
readdir(path::String) at file.jl:169
readdir() at file.jl:198

julia> readdir()
6-element String Array:
 "newnline"      
 "poem"           
 "snowman☃snowman"
 "sp ace"         
 "ttab"

Sending Input

Sometimes, you might want to communicate with another process in a more complex way than just commandline args.

Write to STDIN and then wait for it to finish

The writesto function will give you a Pipe hooked up to the STDIN of the process and a Process. Do not read from the Pipe.

julia> (si,pr) = writesto(`cat`)
(Pipe(open, 0 bytes waiting),Process(`cat`, ProcessRunning))

If you don’t give cat any filenames (or if you pass in a -),
then cat will read from STDIN until you send CTRL-D.

julia> write(si,"hello")
5

We can keep writing to cat for as long as we want.
As you can see, the process is still running:

julia> pr
Process(`cat`, ProcessRunning)

When you’re done writing to the process, close the Pipe.

julia> close(si)

This will make cat exit, as it’s done doing it’s work:

julia> pr
Process(`cat`, ProcessExited(0))

If you want to wait for the process to be down doing work,
you can call wait on the Process.
This will block until the process exits.

julia> wait(pr)
0

Read AND Write on the same process

Using the readandwrite function, you can get two Pipes and a Process.
The Pipes are the STDOUT and STDIN of the process.
The Process is a Julia value representing the asynchronous process started to run the command.

You can read from the STDOUT of the process, but don’t write to it.
You can write to the STDIN of the process, but don’t read from it.

julia> (so,si,pr) = readandwrite(`cat`)
(Pipe(active, 0 bytes waiting),Pipe(open, 0 bytes waiting),Process(`cat`, ProcessRunning))

julia> write(si,"hellongoodnbyen")
15

julia> so
Pipe(active, 15 bytes waiting)

julia> si
Pipe(open, 0 bytes waiting)

julia> close(si)

julia> so
Pipe(closed, 15 bytes waiting)

julia> pr
Process(`cat`, ProcessExited(0))

julia> readall(so)
"hellongoodnbyen"

julia> so
Pipe(closed, 0 bytes waiting)

REPL specialness

In the REPL, you can use a ; to run shell commands easily.

julia> ;ls
new?line  NGrep.jl  poem  snowman☃snowman  sp ace  t?ab  test.txt

julia> ;mkdir folder

julia> ;ls
folder  new?line  NGrep.jl  poem  snowman☃snowman  sp ace  t?ab  test.txt

julia> ;cd folder/
/home/leah/src/lambdajam/folder

The name of the directory (folder/) tab completed! 🙂

julia> ;ls

julia> ;touch my_new_file

julia> ;ls
my_new_file

API Summary

Setting up the process

julia> help(readsfrom)
Base.readsfrom(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream reading from the
   process' standard output.

julia> help(writesto)
Base.writesto(command)

   Starts running a command asynchronously, and returns a tuple
   (stream,process). The first value is a stream writing to the
   process' standard input.

julia> help(readandwrite)
Base.readandwrite(command)

   Starts running a command asynchronously, and returns a tuple
   (stdout,stdin,process) of the output stream and input stream of the
   process, and the process object itself.

Interacting with streams

julia> help(write)
Base.write(stream, x)

   Write the canonical binary representation of a value to the given
   stream.

julia> help(readall)
Base.readall(stream)

   Read the entire contents of an I/O stream as a string.

julia> help(readline)
Base.readline(stream)

   Read a single line of text, including a trailing newline character
   (if one is reached before the end of the input).

julia> help(close)
Base.close(stream)

   Close an I/O stream. Performs a "flush" first.

Interacting with processes

julia> help(wait)
Base.wait(x)

   Block the current task until some event occurs, depending on the
   type of the argument:

   * "RemoteRef": Wait for a value to become available for the
     specified remote reference.

   * "Condition": Wait for "notify" on a condition.

   * "Process": Wait for the process to exit, and get its exit code.

   * "Task": Wait for a "Task" to finish, returning its result
     value.

juliabloggers.com

A Julia Language Blog Aggregator

Tag Archives: projects

Julia Calling Python Calling Julia…