juliabloggers.com http://www.juliabloggers.com A Julia Language Blog Aggregator Wed, 26 Apr 2017 00:00:00 +0000 en-US hourly 1 https://wordpress.org/?v=4.7.4 69897842 Wilmott – Automatic Differentiation for the Greeks http://www.juliabloggers.com/wilmott-automatic-differentiation-for-the-greeks/ Wed, 26 Apr 2017 00:00:00 +0000 http://juliacomputing.com/blog/2017/04/26/Wilmott-AD-Greeks Online quantitative finance magazine Wilmott featured Julia yet again.

Julia Computing’s Dr. Simon Byrne and Dr. Andrew Greenwell engage the magazine readers in a solution they built in Julia, that uses Automatic Differentiation (AD) to calculate price sensitivities, also known as the Greeks.

Fast and accurate calculation of these price sensitivities is extremely crucial in understanding the risk of an option position, and using AD in Julia achieves precisely that.

Traditionally, the world is familiar with using finite-difference approximation for the same calculations. Simon and Andrew go on to argue how that solution is numerically unstable, and how their solution will not only shoot up numerical accuracy, but will also eliminate computational overheads.

To put that in context, there are C++ libraries that assist in these calculations too, QuantLib being one of them. However, a simple implementation of a Cox–Ross–Rubinstein tree (for pricing an American put) with AD in Julia fared 3x times faster than with the C++ library. The code for this example is available here.You can also read the article to know more.

At Julia Computing, we curate all this and much more as part of JuliaFin, a suite of Julia packages that simplify the workflow for quantitative finance, including storage, retrieval, analysis and action.

Julia is already solving a variety of use cases. BlackRock, the Federal Reserve Bank of New York, Nobel Laureate Thomas J. Sargent, and the world’s largest investment banks, insurers, risk managers, fund managers, asset managers, foreign exchange analysts, energy traders, commodity traders and others are all using Julia to solve some of their very complex and challenging quantitive computational problems.

]]>
By: Julia Computing, Inc.

Re-posted from: http://juliacomputing.com/blog/2017/04/26/Wilmott-AD-Greeks.html

Online quantitative finance magazine Wilmott featured Julia yet again.

Julia Computing’s Dr. Simon Byrne and Dr. Andrew Greenwell engage the magazine readers in a solution they built in Julia, that uses Automatic Differentiation (AD) to calculate price sensitivities, also known as the Greeks.

Fast and accurate calculation of these price sensitivities is extremely crucial in understanding the risk of an option position, and using AD in Julia achieves precisely that.

Traditionally, the world is familiar with using finite-difference approximation for the same calculations. Simon and Andrew go on to argue how that solution is numerically unstable, and how their solution will not only shoot up numerical accuracy, but will also eliminate computational overheads.

To put that in context, there are C++ libraries that assist in these calculations too, QuantLib being one of them. However, a simple implementation of a Cox–Ross–Rubinstein tree (for pricing an American put) with AD in Julia fared 3x times faster than with the C++ library. The code for this example is available here.You can also read the article to know more.

At Julia Computing, we curate all this and much more as part of JuliaFin, a suite of Julia packages that simplify the workflow for quantitative finance, including storage, retrieval, analysis and action.

Julia is already solving a variety of use cases. BlackRock, the
Federal Reserve Bank of New
York
, Nobel
Laureate Thomas J.
Sargent
,
and the world’s largest investment banks,
insurers,
risk managers, fund
managers
, asset
managers, foreign exchange
analysts
, energy
traders, commodity traders and others are all using Julia to solve some of their very complex and challenging quantitive computational problems.

]]>
3610
Timing in Julia http://www.juliabloggers.com/timing-in-julia/ Mon, 24 Apr 2017 12:10:48 +0000 http://www.pkofod.com/?p=181 Continue reading Timing in Julia ]]> By: pkofod

Re-posted from: http://www.pkofod.com/2017/04/24/timing-in-julia/

Timing code is important when you want to benchmark or profile your code. Is it the solution of a linear system or the Monte Carlo integration scheme that takes up most of the time? Is version A or version B of a function faster? Questions like that show up all the time. Let us have a look at a few of the possible ways of timing things in Julia.

The basics

The most basic timing functionalities in Julia are the ones included in the Base language. The standard way of timing things in Julia, is by use of the @time macro.

julia> function test(n)
           A = rand(n, n)
           b = rand(n)
           @time A\b
       end
test (generic function with 1 method)

Do note, that the code we want to time is put in a function . This is because everything we do at the top level in the REPL is in global scope. It’s a mistake a lot of people do all the time, but currently it is a very bad idea performance wise. Anyway, let’s see what happens for n = 1, 10, 100, and 1000.

julia> test(1);
  0.000002 seconds (7 allocations: 320 bytes)
 
julia> test(10);
  0.000057 seconds (9 allocations: 1.313 KB)
 
julia> test(100);
  0.001425 seconds (10 allocations: 80.078 KB)
 
julia> test(1000);
  0.033573 seconds (10 allocations: 7.645 MB, 27.81% gc time)
 
julia> test(1000);
  0.045214 seconds (10 allocations: 7.645 MB, 47.66% gc time)

The first run is to compile both test, and then we have a look at what happens when the dimension of our problem increases. Elapsed time seems to increase, and we also see that the number of allocations, and the amount of memory that was allocated increases. For the runs with dimension 1000 we see something else in the output. 30-50% of the time was spent in “gc”. What is this? Julia is a garbage collected language. This means that Julia keeps track of current allocations, and frees the memory if it isn’t needed anymore. It doesn’t do this all the time, though. Running the 1000-dimensional problem once more gives us

julia> test(1000)
  0.029277 seconds (10 allocations: 7.645 MB)

We see it runs slightly faster, and there is no GC time this time around. Of course, these things will look slightly different if you try to replicate them.

So now we can time. But what if we want to store this number? We could be tempted to try

t = @time 3+3

but we will realize, that what is returned is the return value of the expression, not the elapsed time. To save the time, we can either use @timed or @elapsed. Let us try to change the @time to @timed and look at the output when we have our new test2 function return the return value.

julia> function test2(n)
           A = rand(n, n)
           b = rand(n)
           @timed A\b
       end
test2 (generic function with 1 method)
 
julia> test2(3)
([0.700921,-0.120765,0.683945],2.7889e-5,512,0.0,Base.GC_Diff(512,0,0,9,0,0,0,0,0))

We see that it returns a tuple with: the return value of A\b followed by the elapsed time, then the bytes allocated, time spent in garbage collection, and lastly some further memory counters. This is great as we can now work with the information @time printed, but we still have access to the results of our calculations. Of course, it is a bit involved to do it this way. If we simply wanted to see the elapsed time to act on that – then we would just use @time as we did above.

Before we move on to some simpler macros, let us consider the last “time*-family” macro: @timev. As we saw above, @timed contained more information about memory allocation than @time printed. If we want the “verbose” version, we use @timev (v for verbose):

julia> function test3(n)
           A = rand(n, n)
           b = rand(n)
           @timev A\b
       end
test3 (generic function with 1 method)

Running test3 on a kinda large problem, we see that is does indeed print the contents of Base.GC_Diff

julia> test3(5000);
  1.923164 seconds (12 allocations: 190.812 MB, 4.67% gc time)
elapsed time (ns): 1923164359
gc time (ns):      89733440
bytes allocated:   200080368
pool allocs:       9
malloc() calls:    3
GC pauses:         1
full collections:  1

If any of the entries are zero, the corresponding lines are omitted.

julia> test3(50);
  0.001803 seconds (10 allocations: 20.828 KB)
elapsed time (ns): 1802811
bytes allocated:   21328
pool allocs:       9
malloc() calls:    1

Of the three macros, you’ll probably not use @timev a lot.

Simpler versions

If we only want the elapsed time or only want the allocations, then we used either the @elapsed or @allocated macros. However, these do not return the results of our calculations, so in many cases it may be easier to just used @timed, so we can grab the results, the elapsed time, and the allocation information. “MATLAB”-style tic();toc()‘s are also available. toc() prints the time, while toq() is used if we want only the returned time without the printing. It is also possible to use time_ns() to do what time.time() would do in Python, although for practically all purposes, the above macros are recommended.

More advanced functionality

Moving on to more advanced features, we venture into the package ecosystem.

Nested timings

The first package I will present is the nifty TimerOutputs.jl by Kristoffer Carlsson. This packages essentially allows you to nest @time calls. The simplest way to show how it works, is to use the example posted at the announcement (so credit to Kristoffer for the example).

using TimerOutputs
 
# Create the timer object
to = TimerOutput()
 
# Time something with an assigned label
@timeit to "sleep" sleep(0.3)
 
# Data is accumulated for multiple calls
for i in 1:100
    @timeit to "loop" 1+1
end
 
# Nested sections are possible
@timeit to "nest 1" begin
    @timeit to "nest 2" begin
        @timeit to "nest 3.1" rand(10^3)
        @timeit to "nest 3.2" rand(10^4)
        @timeit to "nest 3.3" rand(10^5)
    end
    rand(10^6)
end

Basically we’re timing the sleep call in one time counter, all the additions in the loop in another counter, and then we do some nested generation of random numbers. Displaying the to instance gives us something like the following

 ───────────────────────────────────────────────────────────────────────
                                Time                   Allocations      
                        ──────────────────────   ───────────────────────
    Tot / % measured:        6.48s / 5.60%           77.4MiB / 12.0%    
 
 Section        ncalls     time   %tot     avg     alloc   %tot      avg
 ───────────────────────────────────────────────────────────────────────
 sleep               1    338ms  93.2%   338ms    804KiB  8.43%   804KiB
 nest 1              1   24.7ms  6.80%  24.7ms   8.52MiB  91.5%  8.52MiB
   nest 2            1   9.10ms  2.51%  9.10ms    899KiB  9.43%   899KiB
     nest 3.1        1   3.27ms  0.90%  3.27ms   8.67KiB  0.09%  8.67KiB
     nest 3.3        1   3.05ms  0.84%  3.05ms    796KiB  8.34%   796KiB
     nest 3.2        1   2.68ms  0.74%  2.68ms   92.4KiB  0.97%  92.4KiB
 loop              100   6.97μs  0.00%  69.7ns   6.08KiB  0.06%      62B
 ───────────────────────────────────────────────────────────────────────

which nicely summarizes absolute and relative time and memory allocation of the individual @timeit calls. A real use case could be to see what the effect is of using finite differencing to construct the gradient for the Generalized Rosenbrock (GENROSEN) problem from CUTEst.jl using a conjugate gradient solver in Optim.jl.

using CUTEst, Optim, TimerOutputs
 
nlp = CUTEstModel("GENROSE")
 
const to = TimerOutput()
 
f(x    ) =  @timeit to "f"  obj(nlp, x)
g!(g, x) =  @timeit to "g!" grad!(nlp, x, g)
 
begin
reset_timer!(to)
@timeit to "Conjugate Gradient" begin
    res = optimize(f, g!, nlp.meta.x0, ConjugateGradient(), Optim.Options(iterations=5*10^10));
    println(Optim.minimum(res))
end
@timeit to "Conjugate Gradient (FiniteDiff)" begin
    res = optimize(f, nlp.meta.x0, ConjugateGradient(), Optim.Options(iterations=5*10^10));
    println(Optim.minimum(res))
end
show(to; allocations = false)
end

the output is a table as before, this time without the allocations (notice the use of the allocations keyword in the show method)

 ────────────────────────────────────────────────────────────────
                                                   Time          
                                           ──────────────────────
             Tot / % measured:                  33.3s / 100%     
 
 Section                           ncalls     time   %tot     avg
 ────────────────────────────────────────────────────────────────
 Conjugate Gradient (FiniteDiff)        1    33.2s  99.5%   33.2s
   f                                1.67M    32.6s  97.9%  19.5μs
 Conjugate Gradient                     1    166ms  0.50%   166ms
   g!                               1.72k   90.3ms  0.27%  52.6μs
   f                                2.80k   59.1ms  0.18%  21.1μs
 ────────────────────────────────────────────────────────────────

And we conclude: finite differencing is very slow when you’re solving a 500 dimensional unconstrained optimization problem, and you really want to use the analytical gradient if possible.

Benchmarking

Timing individual pieces of code can be very helpful, but when we’re timing small function calls, this way of measuring performance can be heavily influenced by noise. To remedy that, we use proper benchmarking tools. The package for that, well, it’s called BenchmarkTools.jl and is mainly written by Jarrett Revels. The package is quite advanced in its feature set, but its basic functionality is straight forward to use. Please see the manual for more details than we provide here.

Up until now, we’ve asked Julia to tell us how much time some code took to run. Unfortunately for us, the computer is doing lots of stuff besides the raw calculations we’re trying to time. From the example earlier, this means that we have a lot of noise in our measure of the time it takes to solve A\b. Let us try to run test(1000) a few times

julia> test(1000);
  0.029859 seconds (10 allocations: 7.645 MB)
 
julia> test(1000);
  0.033381 seconds (10 allocations: 7.645 MB, 6.41% gc time)
 
julia> test(1000);
  0.024345 seconds (10 allocations: 7.645 MB)
 
julia> test(1000);
  0.039585 seconds (10 allocations: 7.645 MB)
 
julia> test(1000);
  0.037154 seconds (10 allocations: 7.645 MB, 2.82% gc time)
 
julia> test(1000);
  0.024574 seconds (10 allocations: 7.645 MB)
 
julia> test(1000);
  0.022185 seconds (10 allocations: 7.645 MB)

There’s a lot of variance here! Let’s benchmark instead. The @benchmark macro won’t work inside a function as above. This means that we have to be a bit careful (thanks to Fengyang Wang for clarifying this). Consider the following

julia> n = 200;
 
julia> A = rand(n,n);
 
julia> b = rand(n);
 
julia> @benchmark A\b
BenchmarkTools.Trial: 
  memory estimate:  316.23 KiB
  allocs estimate:  10
  --------------
  minimum time:     531.227 μs (0.00% GC)
  median time:      718.527 μs (0.00% GC)
  mean time:        874.044 μs (3.12% GC)
  maximum time:     95.515 ms (0.00% GC)
  --------------
  samples:          5602
  evals/sample:     1

This is fine, but since A and b are globals (remember, if it ain’t wrapped in a function, it’s a global when you’re working from the REPL), we’re also measuring the time dynamic dispatch takes. Dynamic dispatch happens here, because “Julia” cannot be sure what the types of A and b are when we invoke A\b since they’re globals. Instead, we should use interpolation of the non-constant variables, or mark them as constants using const A = rand(n,n) and const b = rand(20). Let us use interpolation.

julia> @benchmark $A\$b
BenchmarkTools.Trial: 
  memory estimate:  316.23 KiB
  allocs estimate:  10
  --------------
  minimum time:     531.746 μs (0.00% GC)
  median time:      717.269 μs (0.00% GC)
  mean time:        786.240 μs (3.22% GC)
  maximum time:     12.463 ms (0.00% GC)
  --------------
  samples:          6230
  evals/sample:     1

We see that the memory information is identical to the information we got from the other macros, but we now get a much more robust estimate of the time it takes to solve our A\b problem. We also see that dynamic dispatch was negligible here, as the solution takes much longer to compute than for Julia to figure out which method to call. The @benchmark macro will do various things automatically, to try to give as accurate results as possible. It is also possible to provide custom tuning parameters, say if you’re running these benchmarks over an extended period of time and want to track performance regressions, but that is beyond this blog post.

Dynamic dispatch

Before we conclude, let’s have a closer look at the significance of dynamic dispatch. When using globals it has to be determined at run time which method to call. If there a few methods, this may not be a problem, but the problem begins to show itself when a function has a lot of methods. For example, on Julia v0.5.0, identity has one method, but + has 291 methods. Can we measure the significance of dynamic dispatch then? Sure. Just benchmark with, and without interpolation (thanks again to Fengyang Wang for cooking up this example). To keep output from being too verbose, we’ll use the @btime macro – again from BenchmarkTools.jl.

julia> x = 0
0
 
julia> @btime identity(x)
  1.540 ns (0 allocations: 0 bytes)
0
 
julia> @btime +x
  15.837 ns (0 allocations: 0 bytes)
0
 
julia> @btime identity($x)
  1.540 ns (0 allocations: 0 bytes)
0
 
julia> @btime +$x
  1.548 ns (0 allocations: 0 bytes)
0

As we can see, calling + on the global x takes around 10 times a long as the single method function identity. To show that declaring the input a const and interpolating the variable gives the same result, consider the example below.

julia> const y = 0
0
 
julia> @btime identity(y)
  1.539 ns (0 allocations: 0 bytes)
0
 
julia> @btime +y
  1.540 ns (0 allocations: 0 bytes)
0
 
julia> @btime identity($y)
  1.540 ns (0 allocations: 0 bytes)
0
 
julia> @btime +$y
  1.540 ns (0 allocations: 0 bytes)
0

We see that interpolation is not needed, as long as we remember to use constants.

Conclusion

There are quite a few ways of measuring performance in Julia. I’ve presented some of them here, and hopefully you’ll be able to put the tools to good use. The functionality from Base is good for many purposes, but I really like the nested time measuring in TimerOutputs.jl a lot, and for serious benchmarking it is impossible to ignore BenchmarkTools.jl.

]]>
3608
Julia at the Intel AI Day, Bangalore 2017 http://www.juliabloggers.com/julia-at-the-intel-ai-day-bangalore-2017/ Mon, 24 Apr 2017 00:00:00 +0000 http://juliacomputing.com/blog/2017/04/24/Intel-AI-Day Bengaluru, India - Julia Computing featured at one of India’s most prominent AI conferences, demonstrating two very powerful Deep Learning use cases the company is trying to solve using Julia on Intel’s hardware.

The two day event was organized and crafted to showcase robust AI-supporting hardware and software solutions from Intel and it’s partners.

Julia Computing Inc, one of Intel’s partners from India, took centre-stage on day two for their demonstrations. The first of the two demos, namely Neural Styles caught the audience’s fancy - building a neural network imposing the style of one image onto another. Our very own Ranjan Anantharaman took a live picture of the audience and applied transforms to it in real-time.

The second demo targeted solving the serious problem of identifying if a person has symptoms of Diabetic Retinopathy by taking only but an image of his retina as an input, without any human intervention or prediction.

The following video holds a glimpse of what the two firms envision as the future of AI.

Julia Computing and Intel - Acelerating the AI revolution

Also see this whitepaper on Intel and Julia Computing working together on an AI stack.

About Julia

Julia is the simplest, fastest and most powerful numerical computing language available today. Julia combines the functionality of quantitative environments such as Python and R, with the speed of production programming languages like Java and C++ to solve big data and analytics problems. Julia delivers dramatic improvements in simplicity, speed, capacity, and productivity for data scientists, algorithmic traders, quants, scientists, and engineers who need to solve massive computational problems quickly and accurately.

Julia offers an unbeatable combination of simplicity and productivity with speed that is thousands of times faster than other mathematical, scientific and statistical computing languages.

Partners and users include: Intel, The Federal Reserve Bank of New York, Lincoln Laboratory (MIT), The Moore Foundation and a number of private sector finance and industry leaders, including several of the world’s leading hedge funds, investment banks, asset managers and insurers.

About Julia Computing, Inc.

Julia Computing, Inc. was founded in 2015 to develop products around Julia such as JuliaFin. These products help financial firms leverage the 1,000x improvement in speed and productivity that Julia provides for trading, risk analytics, asset management, macroeconomic modeling and other areas. Products of Julia Computing make Julia easy to develop, easy to deploy and easy to scale.

]]>
By: Julia Computing, Inc.

Re-posted from: http://juliacomputing.com/blog/2017/04/24/Intel-AI-Day.html

Bengaluru, India – Julia Computing featured at one of India’s most prominent AI conferences, demonstrating two very powerful Deep Learning use cases the company is trying to solve using Julia on Intel’s hardware.

The two day event was organized and crafted to showcase robust AI-supporting hardware and software solutions from Intel and it’s partners.

Julia Computing Inc, one of Intel’s partners from India, took centre-stage on day two for their demonstrations. The first of the two demos, namely Neural Styles caught the audience’s fancy – building a neural network imposing the style of one image onto another. Our very own Ranjan Anantharaman took a live picture of the audience and applied transforms to it in real-time.

The second demo targeted solving the serious problem of identifying if a person has symptoms of Diabetic Retinopathy by taking only but an image of his retina as an input, without any human intervention or prediction.

The following video holds a glimpse of what the two firms envision as the future of AI.

Julia Computing and Intel – Acelerating the AI revolution

Also see this whitepaper on Intel and Julia Computing working together on an AI stack.

About Julia

Julia is the simplest, fastest and most powerful numerical computing language available today. Julia combines the functionality of quantitative environments such as Python and R, with the speed of production programming languages like Java and C++ to solve big data and analytics problems. Julia delivers dramatic improvements in simplicity, speed, capacity, and productivity for data scientists, algorithmic traders, quants, scientists, and engineers who need to solve massive computational problems quickly and accurately.

Julia offers an unbeatable combination of simplicity and productivity with speed that is thousands of times faster than other mathematical, scientific and statistical computing languages.

Partners and users include: Intel, The Federal Reserve Bank of New York, Lincoln Laboratory (MIT), The Moore Foundation and a number of private sector finance and industry leaders, including several of the world’s leading hedge funds, investment banks, asset managers and insurers.

About Julia Computing, Inc.

Julia Computing, Inc. was founded in 2015 to develop products around Julia such as JuliaFin. These products help financial firms leverage the 1,000x improvement in speed and productivity that Julia provides for trading, risk analytics, asset management, macroeconomic modeling and other areas. Products of Julia Computing make Julia easy to develop, easy to deploy and easy to scale.

]]>
3600
Deep Learning on the New Ubuntu-Based Data Science Virtual Machine for Linux http://www.juliabloggers.com/deep-learning-on-the-new-ubuntu-based-data-science-virtual-machine-for-linux/ Tue, 18 Apr 2017 16:00:12 +0000 https://blogs.technet.microsoft.com/machinelearning/?p=11826 Read more]]> By: Cortana Intelligence and ML Blog Team

Re-posted from: https://blogs.technet.microsoft.com/machinelearning/2017/04/18/deep-learning-on-the-new-ubuntu-based-data-science-virtual-machine-for-linux/

Authored by Paul Shealy, Senior Software Engineer, and Gopi Kumar, Principal Program Manager, at Microsoft.

Deep learning has received significant attention recently for its ability to create machine learning models with very high accuracy. It’s especially popular in image and speech recognition tasks, where the availability of massive datasets with rich information make it feasible to train ever-larger neural networks on powerful GPUs and achieve groundbreaking results. Although there are a variety of deep learning frameworks available, getting started with one means taking time to download and install the framework, libraries, and other tools before writing your first line of code.

Microsoft’s Data Science Virtual Machine (DSVM) is a family of popular VM images published on the Azure marketplace with a broad choice of machine learning and data science tools. Microsoft is extending it with the introduction of a brand-new offering in this family – the Data Science Virtual Machine for Linux, based on Ubuntu 16.04LTS – that also includes a comprehensive set of popular deep learning frameworks.

Deep learning frameworks in the new VM include:

  • Microsoft Cognitive Toolkit
  • Caffe and Caffe2
  • TensorFlow
  • H2O
  • MXNet
  • NVIDIA DIGITS
  • Theano
  • Torch, including PyTorch
  • Keras

The image can be deployed on VMs with GPUs or CPU-only VMs. It also includes OpenCV, matplotlib and many other libraries that you will find useful.

Run dsvm-more-info at a command prompt or visit the documentation for more information about these frameworks and how to get started.

Sample Jupyter notebooks are included for most frameworks. Start Jupyter or log in to JupyterHub to browse the samples for an easy way to explore the frameworks and get started with deep learning.

GPU Support

Training a deep neural network requires considerable computational resources, so things can be made significantly faster by running on one or more GPUs. Azure now offers NC-class VM sizes with 1-4 NVIDIA K80 GPUs for computational workloads. All deep learning frameworks on the VM are compiled with GPU support, and the NVIDIA driver, CUDA and cuDNN are included. You may also choose to run the VM on a CPU if you prefer, and that is supported without code changes. And because this is running on Azure, you can choose a smaller VM size for setup and exploration, then scale up to one or more GPUs for training.

The VM comes with nvidia-smi to monitor GPU usage during training and help optimize parameters to make full use of the GPU. It also includes NVIDIA Docker if you want to run Docker containers with GPU access.

Data Science Virtual Machine

The Data Science Virtual Machine family of VM images on Azure includes the DSVM for Windows, a CentOS-based DSVM for Linux, and an Ubuntu-based DSVM for Linux. These images come with popular data science and machine learning tools, including Microsoft R Server Developer Edition, Microsoft R Open, Anaconda Python, Julia, Jupyter notebooks, Visual Studio Code, RStudio, xgboost, and many more. A full list of tools for all editions of the DSVM is available here. The DSVM has proven popular with data scientists as it helps them focus on their tasks and skip mundane steps around tool installation and configuration.


To try deep learning on Windows with GPUs, the Deep Learning Toolkit for DSVM contains all tools from the Windows DSVM plus GPU drivers, CUDA, cuDNN, and GPU versions of CNTK, MXNet, and TensorFlow.

Get Started Today

We invite you to use the new image to explore deep learning frameworks or for your machine learning and data science projects – DSVM for Linux (Ubuntu) is available today through the Marketplace. Free Azure credits are available to help get you started.

Paul & Gopi

]]>
3596
DifferentialEquations.jl v1.9.1 http://www.juliabloggers.com/differentialequations-jl-v1-9-1/ Fri, 07 Apr 2017 01:30:00 +0000 http://juliadiffeq.org/2017/04/07/features.html ]]> By: JuliaDiffEq

Re-posted from: http://juliadiffeq.org/2017/04/07/features.html

DifferentialEquations v1.9.1 is a feature update which, well, brings a lot of new
features. But before we get started, there is one thing to highlight:

]]>
3587
DifferentialEquations.jl Workshop at JuliaCon 2017 http://www.juliabloggers.com/differentialequations-jl-workshop-at-juliacon-2017/ Tue, 04 Apr 2017 11:00:00 +0000 http://juliadiffeq.org/2017/04/04/juliacon.html There will be a workshop on DifferentialEquations.jl at this year’s JuliaCon! The title is “The Unique Features and Performance of DifferentialEquations.jl”. The goal will be to teach new users how to solve a wide variety of differential equations, and show how to achieve the best possible performance. I hope to lead users through an example problem: start with ODEs and build a simple model. I will show the tools for analyzing the solution to ODEs, show how to choose the best solver for your problem, show how to use non-standard features like arbitrary precision arithmetic. From there, we seamlessly flow into more in-depth analysis and models. We will start estimating parameters of the ODEs, and then make the models more realistic by adding delays, stochasticity (randomness), and Gillespie models (discrete stochastic models related to differential equations), and running stochastic Monte Carlo experiments in parallel (in a way that will automatically parallelizes across multiple nodes of an HPC!).

]]>
By: JuliaDiffEq

Re-posted from: http://juliadiffeq.org/2017/04/04/juliacon.html

There will be a workshop on DifferentialEquations.jl at this year’s JuliaCon!
The title is “The Unique Features and Performance of DifferentialEquations.jl”.
The goal will be to teach new users how to solve a wide variety of differential
equations, and show how to achieve the best possible performance. I hope to lead
users through an example problem: start with ODEs and build a simple model. I
will show the tools for analyzing the solution to ODEs, show how to choose the
best solver for your problem, show how to use non-standard features like arbitrary
precision arithmetic. From there, we seamlessly flow into more in-depth
analysis and models. We will start estimating parameters of the ODEs, and then
make the models more realistic by adding delays, stochasticity (randomness), and
Gillespie models (discrete stochastic models related to differential equations),
and running stochastic Monte Carlo experiments in parallel (in a way that will
automatically parallelizes across multiple nodes of an HPC!).

]]>
3582
Continuous-time deterministic dynamic programming in Julia http://www.juliabloggers.com/continuous-time-deterministic-dynamic-programming-in-julia/ Sun, 02 Apr 2017 17:06:04 +0000 http://tpapp.github.io/post/dynamic-programming/ For the past few weeks I have been organizing pieces of code I have used to solve economic models into Julia packages. EconFunctions.jl is a collection of trivial functions that I noticed that I kept recoding/copy-pasting everywhere, occasionally making errors. ContinuousTransformations.jl is a library for manipulating various commonly used homeomorphisms (univariate at the moment), which are useful in functional equations or Markov Chain Monte Carlo. Finally ParametricFunctions.jl is for working with parametric function families.

In this post I use these three to solve a simple, deterministic dynamic programming model in continuous time, known as the Ramsey growth model. If you are not an economist, it is very unlikely that it will make a lot of sense. If you are a student, I added basic references at the end, which are consistent with the methods in this post.

Caveat: these libraries are in development, I am refining the API and changing things all the time. It is very likely that as time progresses, code in this post will not run without changes. In other words, treat this as a sneak peak into a library which is in development.

Theory

This is standard material, I am just repeating it so that this post is self-contained. We solve

$$\max \int_0^\infty e^{-\rho t} u(c_t) dt$$ subject to $$\dot{k}_t = F(k_t) - c_t, k_t \ge 0 \forall t.$$

where $u( c )$ is a CRRA utility function with IES $\theta$, $F(k) = A k^\alpha - \delta k$ is a production function that accounts for depreciation. Our problem is described by the Hamilton-Jacobi-Bellman equation

$$\rho V(k) = \max_c u( c ) + (F(k)-c) V’(k)$$

Notice that once we have $V$, the first-order condition

$$u’(c(k)) = V’(k)$$

yields the policy function $c(k)$, which we are interested in. Combining the envelope condition

$$\rho V’(k) = F’(k) V’(k) + (F(k)-c) V{‘}{’}(k)$$

and using the functional form for CRRA utility, we obtain

$$\frac{c’(k)}{c(k)} (F(k)-c(k)) = \frac{1}{\theta} (F’(k)-\rho)$$

which is a recursive form of the so-called Euler equation.

Also, note that we can characterize the steady state capital and consumption by

$$k_s = \left(\frac{\delta+\rho}{A\alpha}\right)^{1/(\alpha-1)}$$

and

$$c_s = F(k_s)$$

Julia code: solving the Euler equation

Load the libraries (you need to clone some code, as the packages are not registered).

using ParametricFunctions       # unregistered, clone from repo
using ContinuousTransformations # unregistered, clone from repo
using EconFunctions             # unregistered, clone from repo
using Plots; gr()
using Parameters
using NLsolve

It is useful to put model parameters in a single structure.

"""
Very simple (normalized) Ramsey model with isoelastic production
function and utility.
"""
@with_kw immutable RamseyModel{T}
    θ::T                        # IES
    α::T                        # capital share
    A::T                        # TFP
    ρ::T                        # discount rate
    δ::T                        # depreciation
end

This is the key part: we code the residual for the Euler equation. The function should take the model (which contains the parameters), a function $c$ that has been constructed using a function family and a set of parameters, and a scalar $k$, at which we evaluate the residual above. Everything else can be automated very well.

"""
Residual of the Euler equation.
"""
function euler_residual(model::RamseyModel, c, k)
    @unpack θ, ρ, α, A, δ = model
    Fk = A*k^α - δ*k
    F′k = A*α*k^(α-1) - δ
    ck, c′k = c(ValuePartial(k))
    (c′k/ck)*(Fk-ck) - 1/θ*(F′k-ρ)
end

Above, c can be treated like an ordinary function, except that if you call it with ValuePartial(x), you get the value and the derivative.

The steady state will be handy:

"Return the steady state capital and consumption for the model."
function steady_state(model::RamseyModel)
    @unpack α, A, ρ, δ = model
    k = ((δ+ρ)/(A*α))^(1/(α-1))
    c = A*k^α - δ*k
    k, c
end

Let’s make a model object (parameters are pretty standard), and calculate the steady state:

model = RamseyModel(θ = 2.0, α = 0.3, A = 1.0, ρ = 0.02, δ = 0.05)

kₛ, cₛ = steady_state(model)

We will solve in a domain around the steady state capital.

kdom = (0.5*kₛ)..(2*kₛ)

Given the pieces above, obtaining the solution can be done very conscisely: create a residual object, which is basically a mapping from parameters to the function family to the residuals:

res = CollocationResidual(model, DomainTrans(kdom, Chebyshev(10)),
                          euler_residual)

The above say that we want 10 Chebyshev polynomials, transformed to the domain kdom, to be used for constructing the $c(k)$.

We call the solver, providing an initial guess, $c(k) = k\cdot c_s/k_s$, for the policy function $c(k)$. The guess is that consumption is linear in capital, and the line goes through the steady state values. Other reasonable guesses are possible, but note that it is worthwhile thinking a bit about a good one, so that you get fast convergence.

The function below fits a parametric function from the given family to the initial guess, then solves for the residual being $0$ using NLsolve with automatic differentiation under the hood.

c_sol, o = solve_collocation(res, k->cₛ*k/kₛ; ftol=1e-10,
                             method = :newton)

Convergence statistics:

Results of Nonlinear Solver Algorithm
 * Algorithm: Newton with line-search
 * Starting Point: [1.83249,1.09949,7.10543e-16,4.44089e-16,-1.77636e-16,-8
.88178e-17,-8.43769e-16,1.64313e-15,1.33227e-16,3.10862e-16]
 * Zero: [1.57794,0.433992,-0.0360164,0.00624848,-0.00134301,0.000320829,-8
.21347e-5,2.28742e-5,-6.85183e-6,1.48105e-6]
 * Inf-norm of residuals: 0.000000
 * Iterations: 6
 * Convergence: true
   * |x - x'| < 0.0e+00: false
   * |f(x)| < 1.0e-10: true
 * Function Calls (f): 7
 * Jacobian Calls (df/dx): 6

Overall, pretty good, very few iterations. We plot the resulting function:

plot(c_sol, xlab = "k", ylab = "c(k)", legend = false)
scatter!([kₛ], [cₛ])

Notive how the collocation nodes are added automatically (this is done with a plot recipe). It should, of course, go thought the steady state.

It is very important to plot the residual:

plot(k->euler_residual(model, c_sol, k), linspace(kdom, 100),
     legend = false, xlab="k", ylab="Euler residual")
scatter!(zero, points(c_sol))

Note the near-equioscillation property, which you get from using Chebyshev polynomials. You get $10^{-6}$ accuracy, which is neat (but note that this is a simple textbook problem, very smooth and tractable).

Selected reading

Acemoglu, Daron. Introduction to modern economic growth. Princeton University Press, 2008. Chapter 8.

Miranda, Mario J., and Paul L. Fackler. Applied computational economics and finance. MIT press, 2004. Chapters 10 and 11.

]]>
By: Julia on Tamás K. Papp's blog

Re-posted from: http://tpapp.github.io/post/dynamic-programming/

For the past few weeks I have been organizing pieces of code I have used to solve economic models into Julia packages. EconFunctions.jl is a collection of trivial functions that I noticed that I kept recoding/copy-pasting everywhere, occasionally making errors. ContinuousTransformations.jl is a library for manipulating various commonly used homeomorphisms (univariate at the moment), which are useful in functional equations or Markov Chain Monte Carlo. Finally ParametricFunctions.jl is for working with parametric function families.

In this post I use these three to solve a simple, deterministic dynamic programming model in continuous time, known as the Ramsey growth model. If you are not an economist, it is very unlikely that it will make a lot of sense. If you are a student, I added basic references at the end, which are consistent with the methods in this post.

Caveat: these libraries are in development, I am refining the API and changing things all the time. It is very likely that as time progresses, code in this post will not run without changes. In other words, treat this as a sneak peak into a library which is in development.

Theory

This is standard material, I am just repeating it so that this post is self-contained. We solve

$$\max \int_0^\infty e^{-\rho t} u(c_t) dt$$
subject to
$$\dot{k}_t = F(k_t) – c_t, k_t \ge 0 \forall t.$$

where $u( c )$ is a CRRA utility function with IES $\theta$, $F(k) = A k^\alpha – \delta k$ is a production function that accounts for depreciation. Our problem is described by the Hamilton-Jacobi-Bellman equation

$$\rho V(k) = \max_c u( c ) + (F(k)-c) V’(k)$$

Notice that once we have $V$, the first-order condition

$$u’(c(k)) = V’(k)$$

yields the policy function $c(k)$, which we are interested in. Combining the envelope condition

$$\rho V’(k) = F’(k) V’(k) + (F(k)-c) V{‘}{’}(k)$$

and using the functional form for CRRA utility, we obtain

$$\frac{c’(k)}{c(k)} (F(k)-c(k)) = \frac{1}{\theta} (F’(k)-\rho)$$

which is a recursive form of the so-called Euler equation.

Also, note that we can characterize the steady state capital and consumption by

$$k_s = \left(\frac{\delta+\rho}{A\alpha}\right)^{1/(\alpha-1)}$$

and

$$c_s = F(k_s)$$

Julia code: solving the Euler equation

Load the libraries (you need to clone some code, as the packages are not registered).

using ParametricFunctions       # unregistered, clone from repo
using ContinuousTransformations # unregistered, clone from repo
using EconFunctions             # unregistered, clone from repo
using Plots; gr()
using Parameters
using NLsolve

It is useful to put model parameters in a single structure.

"""
Very simple (normalized) Ramsey model with isoelastic production
function and utility.
"""
@with_kw immutable RamseyModel{T}
    θ::T                        # IES
    α::T                        # capital share
    A::T                        # TFP
    ρ::T                        # discount rate
    δ::T                        # depreciation
end

This is the key part: we code the residual for the Euler equation. The function should take the model (which contains the parameters), a function $c$ that has been constructed using a function family and a set of parameters, and a scalar $k$, at which we evaluate the residual above. Everything else can be automated very well.

"""
Residual of the Euler equation.
"""
function euler_residual(model::RamseyModel, c, k)
    @unpack θ, ρ, α, A, δ = model
    Fk = A*k^α - δ*k
    F′k = A*α*k^(α-1) - δ
    ck, c′k = c(ValuePartial(k))
    (c′k/ck)*(Fk-ck) - 1/θ*(F′k-ρ)
end

Above, c can be treated like an ordinary function, except that if you call it with ValuePartial(x), you get the value and the derivative.

The steady state will be handy:

"Return the steady state capital and consumption for the model."
function steady_state(model::RamseyModel)
    @unpack α, A, ρ, δ = model
    k = ((δ+ρ)/(A*α))^(1/(α-1))
    c = A*k^α - δ*k
    k, c
end

Let’s make a model object (parameters are pretty standard), and calculate the steady state:

model = RamseyModel(θ = 2.0, α = 0.3, A = 1.0, ρ = 0.02, δ = 0.05)

kₛ, cₛ = steady_state(model)

We will solve in a domain around the steady state capital.

kdom = (0.5*kₛ)..(2*kₛ)

Given the pieces above, obtaining the solution can be done very conscisely: create a residual object, which is basically a mapping from parameters to the function family to the residuals:

res = CollocationResidual(model, DomainTrans(kdom, Chebyshev(10)),
                          euler_residual)

The above say that we want 10 Chebyshev polynomials, transformed to the domain kdom, to be used for constructing the $c(k)$.

We call the solver, providing an initial guess, $c(k) = k\cdot c_s/k_s$, for the policy function $c(k)$. The guess is that consumption is linear in capital, and the line goes through the steady state values. Other reasonable guesses are possible, but note that it is worthwhile thinking a bit about a good one, so that you get fast convergence.

The function below fits a parametric function from the given family to the initial guess, then solves for the residual being $0$ using NLsolve with automatic differentiation under the hood.

c_sol, o = solve_collocation(res, k->cₛ*k/kₛ; ftol=1e-10,
                             method = :newton)

Convergence statistics:

Results of Nonlinear Solver Algorithm
 * Algorithm: Newton with line-search
 * Starting Point: [1.83249,1.09949,7.10543e-16,4.44089e-16,-1.77636e-16,-8
.88178e-17,-8.43769e-16,1.64313e-15,1.33227e-16,3.10862e-16]
 * Zero: [1.57794,0.433992,-0.0360164,0.00624848,-0.00134301,0.000320829,-8
.21347e-5,2.28742e-5,-6.85183e-6,1.48105e-6]
 * Inf-norm of residuals: 0.000000
 * Iterations: 6
 * Convergence: true
   * |x - x'| < 0.0e+00: false
   * |f(x)| < 1.0e-10: true
 * Function Calls (f): 7
 * Jacobian Calls (df/dx): 6

Overall, pretty good, very few iterations. We plot the resulting function:

plot(c_sol, xlab = "k", ylab = "c(k)", legend = false)
scatter!([kₛ], [cₛ])

Notive how the collocation nodes are added automatically (this is done with a plot recipe). It should, of course, go thought the steady state.

It is very important to plot the residual:

plot(k->euler_residual(model, c_sol, k), linspace(kdom, 100),
     legend = false, xlab="k", ylab="Euler residual")
scatter!(zero, points(c_sol))

Note the near-equioscillation property, which you get from using Chebyshev polynomials. You get $10^{-6}$ accuracy, which is neat (but note that this is a simple textbook problem, very smooth and tractable).

Selected reading

Acemoglu, Daron. Introduction to modern economic growth. Princeton University Press, 2008. Chapter 8.

Miranda, Mario J., and Paul L. Fackler. Applied computational economics and finance. MIT press, 2004. Chapters 10 and 11.

]]>
3574
The DAG of Julia Packages http://www.juliabloggers.com/the-dag-of-julia-packages/ Sun, 02 Apr 2017 00:00:00 +0000 https://juliohm.github.io/science/DAG-of-Julia-packages/

If your package is listed below, please consider fixing it:

Instructions

This interactive visualization made with D3 shows the Directed Acyclic Graph (DAG) of all registered Julia packages up until 02-April-2017.

The size of a node represents its influence (i.e. out degree) in the DAG. The color represents the required Julia version.

Hover the mouse over the elements to get more information.

Data

The data was extracted from METADATA with the following script:

using JSON
using LightGraphs
using ProgressMeter

# find all packages in METADATA
pkgs = readdir(Pkg.dir("METADATA"))
filterfunc = p -> isdir(joinpath(Pkg.dir("METADATA"), p)) && p  [".git",".test"]
pkgs = filter(filterfunc, pkgs)

# assign each package an id
pkgdict = Dict{String,Int}()
for (i,pkg) in enumerate(pkgs)
  push!(pkgdict, pkg => i)
end

# build DAG
G = DiGraph(length(pkgs))
@showprogress 1 "Building graph..." for pkg in pkgs
  children = Pkg.dependents(pkg)
  for c in children
    add_edge!(G, pkgdict[pkg], pkgdict[c])
  end
end

# find required Julia version
juliaversions = String[]
for pkg in pkgs
  versiondir = joinpath(Pkg.dir("METADATA"), pkg, "versions")
  if isdir(versiondir)
    latestversion = readdir(versiondir)[end]
    reqfile = joinpath(versiondir, latestversion, "requires")
    juliaversion = string(get(Pkg.Reqs.parse(reqfile), "julia", "NA"))
    push!(juliaversions, juliaversion)
  else
    push!(juliaversions, "BOGUS")
  end
end

# construct JSON
nodes = [Dict("id"=>pkgs[v],
              "indegree"=>indegree(G,v),
              "outdegree"=>outdegree(G,v),
              "juliaversion"=>juliaversions[v]) for v in vertices(G)]
links = [Dict("source"=>pkgs[u], "target"=>pkgs[v]) for (u,v) in edges(G)]
data = Dict("nodes"=>nodes, "links"=>links)

# write to file
open("DAG-Julia-Pkgs.json", "w") do f
  JSON.print(f, data, 2)
end

Improvements

The DAG can be improved in many ways. Below is a list of issues that I would like to address, feel free to suggest more:

  1. The number of categories (i.e. Julia version strings) in the legend is too big. I wonder what would be a more sensible choice for coloring a version string [vmin,vmax), would it be the minimum vmin or the maximum vmax required Julia version?

  2. It would be great to present the data on a map. If you know how to get an approximate latitude/longitude for the author of a package (perhaps using GitHub API?), please leave a comment.

  3. Nodes could be collapsed by clicking with the mouse. Related visual elements would be updated accordingly.

  4. The evolution of the DAG with time should be interesting. How to extract time information from the commits in METADATA corresponding to new package tags?

]]>
By: Júlio Hoffimann

Re-posted from: https://juliohm.github.io/science/DAG-of-Julia-packages/

If your package is listed below, please consider fixing it:


Instructions

This interactive visualization made with D3 shows the Directed Acyclic Graph (DAG) of all registered Julia packages up until 02-April-2017.

The size of a node represents its influence (i.e. out degree) in the DAG. The color represents the required Julia version.

Hover the mouse over the elements to get more information.

Data

The data was extracted from METADATA with the following script:

using JSON
using LightGraphs
using ProgressMeter

# find all packages in METADATA
pkgs = readdir(Pkg.dir("METADATA"))
filterfunc = p -> isdir(joinpath(Pkg.dir("METADATA"), p)) && p  [".git",".test"]
pkgs = filter(filterfunc, pkgs)

# assign each package an id
pkgdict = Dict{String,Int}()
for (i,pkg) in enumerate(pkgs)
  push!(pkgdict, pkg => i)
end

# build DAG
G = DiGraph(length(pkgs))
@showprogress 1 "Building graph..." for pkg in pkgs
  children = Pkg.dependents(pkg)
  for c in children
    add_edge!(G, pkgdict[pkg], pkgdict[c])
  end
end

# find required Julia version
juliaversions = String[]
for pkg in pkgs
  versiondir = joinpath(Pkg.dir("METADATA"), pkg, "versions")
  if isdir(versiondir)
    latestversion = readdir(versiondir)[end]
    reqfile = joinpath(versiondir, latestversion, "requires")
    juliaversion = string(get(Pkg.Reqs.parse(reqfile), "julia", "NA"))
    push!(juliaversions, juliaversion)
  else
    push!(juliaversions, "BOGUS")
  end
end

# construct JSON
nodes = [Dict("id"=>pkgs[v],
              "indegree"=>indegree(G,v),
              "outdegree"=>outdegree(G,v),
              "juliaversion"=>juliaversions[v]) for v in vertices(G)]
links = [Dict("source"=>pkgs[u], "target"=>pkgs[v]) for (u,v) in edges(G)]
data = Dict("nodes"=>nodes, "links"=>links)

# write to file
open("DAG-Julia-Pkgs.json", "w") do f
  JSON.print(f, data, 2)
end

Improvements

The DAG can be improved in many ways. Below is a list of issues that I would like to address, feel free to suggest more:

  1. The number of categories (i.e. Julia version strings) in the legend is too big. I wonder what would be a more sensible choice for coloring a version string [vmin,vmax), would it be the minimum vmin or the maximum vmax required Julia version?

  2. It would be great to present the data on a map. If you know how to get an approximate latitude/longitude for the author of a package (perhaps using GitHub API?), please leave a comment.

  3. Nodes could be collapsed by clicking with the mouse. Related visual elements would be updated accordingly.

  4. The evolution of the DAG with time should be interesting. How to extract time information from the commits in METADATA corresponding to new package tags?

]]>
3564
Blogging with Hugo, Julia, Weave.jl http://www.juliabloggers.com/blogging-with-hugo-julia-weave-jl/ Thu, 30 Mar 2017 08:02:32 +0000 http://tpapp.github.io/post/blogging-weave-julia-hugo/ ]]> By: Julia on Tamás K. Papp's blog

Re-posted from: http://tpapp.github.io/post/blogging-weave-julia-hugo/

I have made a PR to Weave.jl which Matti Pastell kindly merged recently. This allows a relatively smooth workflow for blogging using the static website generator Hugo, and generating some pages with plots and evaluated Julia results. I made the source for my blog available so that others can use it for their own blogging about Julia. An example is this post.

The gist of the workflow is as follows:

  1. for posts which do not need Weave, just use Hugo. Make sure you read their excellent tutorial. This very fast.

  2. for posts which contain Julia code and generated plots, use a script to generate a skeleton file in a separate directory, and work on that. Call another script to generate the .md file using Weave.jl. This is the slow part, so it is not automated.

The README gives more details. Feel free to ask questions here. If you have a better workflow, I would like to hear about it.

]]>
3576
BlackRock’s Julia-Powered Aladdin Platform Featured in New York Times http://www.juliabloggers.com/blackrocks-julia-powered-aladdin-platform-featured-in-new-york-times/ Wed, 29 Mar 2017 00:00:00 +0000 http://juliacomputing.com/press/2017/03/29/alladin New York, NY – BlackRock’s Julia-powered Aladdin analytics and risk management platform was featured in yesterday’s New York Times in an article titled “At BlackRock, Machines Are Rising Over Managers To Pick Stocks”.

BlackRock is the world’s largest asset manager, with $5.1 trillion under management. BlackRock’s trademark Aladdin platform was built using Julia, the fastest modern high performance open source computing language for data and analytics.

About Julia Computing and Julia

Julia Computing (JuliaComputing.com) was founded in 2015 by the co-creators of the Julia language to provide support to businesses and researchers who use Julia.

Julia is the fastest modern high performance open source computing language for data and analytics. It combines the functionality and ease of use of Python, R, Matlab, SAS and Stata with the speed of Java and C++. Julia delivers dramatic improvements in simplicity, speed, capacity and productivity. With more than 1 million downloads and +161% annual growth, Julia adoption is growing rapidly in finance, energy, robotics, genomics and many other fields.

  1. Julia is lightning fast. Julia provides speed improvements up to 1,000x for insurance model estimation, 225x for parallel supercomputing image analysis and 11x for macroeconomic modeling.

  2. Julia is easy to learn. Julia’s flexible syntax is familiar and comfortable for users of Python, R and Matlab.

  3. Julia integrates well with existing code and platforms. Users of Python, R, Matlab and other languages can easily integrate their existing code into Julia.

  4. Elegant code. Julia was built from the ground up for mathematical, scientific and statistical computing, and has advanced libraries that make coding simple and fast, and dramatically reduce the number of lines of code required – in some cases, by 90% or more.

  5. Julia solves the two language problem. Because Julia combines the ease of use and familiar syntax of Python, R and Matlab with the speed of C, C++ or Java, programmers no longer need to estimate models in one language and reproduce them in a faster production language. This saves time and reduces error and cost.

Julia users, partners and employers looking to hire Julia programmers in 2017 include: Google, Apple, Amazon, Facebook, IBM, Intel, Microsoft, BlackRock, Capital One, PricewaterhouseCoopers, Ford, Oracle, Comcast, DARPA, Moore Foundation, Federal Reserve Bank of New York (FRBNY), UC Berkeley Autonomous Race Car (BARC), Federal Aviation Administration (FAA), MIT Lincoln Labs, Nobel Laureate Thomas J. Sargent, Brazilian National Development Bank (BNDES), Conning, Berkery Noyes, BestX, Path BioAnalytics, Invenia, AOT Energy, AlgoCircle, Trinity Health, Gambit, Augmedics, Tangent Works, Voxel8, Massachusetts General Hospital, NaviHealth, Farmers Insurance, Pilot Flying J, Lawrence Berkeley National Laboratory, National Energy Research Scientific Computing Center (NERSC), Oak Ridge National Laboratory, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, National Renewable Energy Laboratory, MIT, Caltech, Stanford, UC Berkeley, Harvard, Columbia, NYU, Oxford, NUS, UCL, Nantes, Alan Turing Institute, University of Chicago, Cornell, Max Planck Institute, Australian National University, University of Warwick, University of Colorado, Queen Mary University of London, London Institute of Cancer Research, UC Irvine, University of Kaiserslautern.

Julia is being used to: analyze images of the universe and research dark matter, drive parallel supercomputing, diagnose medical conditions, provide surgeons with real-time imagery using augmented reality, analyze cancer genomes, manage 3D printers, pilot self-driving racecars, build drones, improve air safety, manage the electric grid, provide analytics for foreign exchange trading, energy trading, insurance, regulatory compliance, macroeconomic modeling, sports analytics, manufacturing and much, much more.

]]>
By: Julia Computing, Inc.

Re-posted from: http://juliacomputing.com/press/2017/03/29/alladin.html

New York, NY – BlackRock’s Julia-powered Aladdin analytics and risk management platform was featured in yesterday’s New York Times in an article titled “At BlackRock, Machines Are Rising Over Managers To Pick Stocks”.

BlackRock is the world’s largest asset manager, with $5.1 trillion under management. BlackRock’s trademark Aladdin platform was built using Julia, the fastest modern high performance open source computing language for data and analytics.

About Julia Computing and Julia

Julia Computing (JuliaComputing.com) was founded in 2015 by the co-creators of the Julia language to provide support to businesses and researchers who use Julia.

Julia is the fastest modern high performance open source computing language for data and analytics. It combines the functionality and ease of use of Python, R, Matlab, SAS and Stata with the speed of Java and C++. Julia delivers dramatic improvements in simplicity, speed, capacity and productivity. With more than 1 million downloads and +161% annual growth, Julia adoption is growing rapidly in finance, energy, robotics, genomics and many other fields.

  1. Julia is lightning fast. Julia provides speed improvements up to
    1,000x for insurance model estimation, 225x for parallel
    supercomputing image analysis and 11x for macroeconomic modeling.

  2. Julia is easy to learn. Julia’s flexible syntax is familiar and
    comfortable for users of Python, R and Matlab.

  3. Julia integrates well with existing code and platforms. Users of
    Python, R, Matlab and other languages can easily integrate their
    existing code into Julia.

  4. Elegant code. Julia was built from the ground up for
    mathematical, scientific and statistical computing, and has advanced
    libraries that make coding simple and fast, and dramatically reduce
    the number of lines of code required – in some cases, by 90%
    or more.

  5. Julia solves the two language problem. Because Julia combines
    the ease of use and familiar syntax of Python, R and Matlab with the
    speed of C, C++ or Java, programmers no longer need to estimate
    models in one language and reproduce them in a faster
    production language. This saves time and reduces error and cost.

Julia users, partners and employers looking to hire Julia programmers in 2017 include: Google, Apple, Amazon, Facebook, IBM, Intel, Microsoft, BlackRock, Capital One, PricewaterhouseCoopers, Ford, Oracle, Comcast, DARPA, Moore Foundation, Federal Reserve Bank of New York (FRBNY), UC Berkeley Autonomous Race Car (BARC), Federal Aviation Administration (FAA), MIT Lincoln Labs, Nobel Laureate Thomas J. Sargent, Brazilian National Development Bank (BNDES), Conning, Berkery Noyes, BestX, Path BioAnalytics, Invenia, AOT Energy, AlgoCircle, Trinity Health, Gambit, Augmedics, Tangent Works, Voxel8, Massachusetts General Hospital, NaviHealth, Farmers Insurance, Pilot Flying J, Lawrence Berkeley National Laboratory, National Energy Research Scientific Computing Center (NERSC), Oak Ridge National Laboratory, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, National Renewable Energy Laboratory, MIT, Caltech, Stanford, UC Berkeley, Harvard, Columbia, NYU, Oxford, NUS, UCL, Nantes, Alan Turing Institute, University of Chicago, Cornell, Max Planck Institute, Australian National University, University of Warwick, University of Colorado, Queen Mary University of London, London Institute of Cancer Research, UC Irvine, University of Kaiserslautern.

Julia is being used to: analyze images of the universe and research dark matter, drive parallel supercomputing, diagnose medical conditions, provide surgeons with real-time imagery using augmented reality, analyze cancer genomes, manage 3D printers, pilot self-driving racecars, build drones, improve air safety, manage the electric grid, provide analytics for foreign exchange trading, energy trading, insurance, regulatory compliance, macroeconomic modeling, sports analytics, manufacturing and much, much more.

]]>
3584
Using pipes while running external programs in Julia http://www.juliabloggers.com/using-pipes-while-running-external-programs-in-julia/ Mon, 27 Mar 2017 18:12:30 +0000 http://perfectionatic.org/?p=340 ]]> By: perfectionatic

Re-posted from: http://perfectionatic.org/?p=340

Recently I was using Julia to run ffprobe to get the length of a video file. The trouble was the ffprobe was dumping its output to stderr and I wanted to take that output and run it through grep. From a bash shell one would typically run:

ffprobe somefile.mkv 2>&1 |grep Duration

This would result in an output like

 Duration: 00:04:44.94, start: 0.000000, bitrate: 128 kb/s

This works because we used 2>&1 to redirect stderr to stdout which would in be piped to grep.

If you were try to run this in Julia

julia> run(`ffprobe somefile.mkv 2>&1 |grep Duration`)

you will get errors. Julia does not like pipes | inside the backticks command (for very sensible reasons). Instead you should be using Julia’s pipeline command. Also the redirection 2>&1 will not work. So instead, the best thing to use is and instance of Pipe. This was not in the manual. I stumbled upon it in an issue discussion on GitHub. So a good why to do what I am after is to run.

julia> p=Pipe()
Pipe(uninit => uninit, 0 bytes waiting)
 
julia> run(pipeline(`ffprobe -i  somefile.mkv`,stderr=p))

This would create a pipe object p that is then used to capture stderr after the execution of the command. Next we need to close the input end of the pipe.

julia> close(p.in)

Finally we can use the pipe with grep to filter the output.

julia> readstring(pipeline(p,`grep Duration`))
"  Duration: 00:04:44.94, start: 0.000000, bitrate: 128 kb/s\n"

We can then do a little regex magic to get the duration we are after.

julia> matchall(r"(\d{2}:\d{2}:\d{2}.\d{2})",ans)[1]
"00:04:44.94"
]]>
3554
Image Stitching: Part 1 http://www.juliabloggers.com/image-stitching-part-1/ Sun, 26 Mar 2017 00:00:00 +0000 http://learningjulia.com/2017/03/26/image-stitching-part-1 ]]> By: Michele Pratusevich

Re-posted from: http://learningjulia.com/2017/03/26/image-stitching-part-1.html

This is Part 1 of 2 in my posts about how to stitch two images together using Julia. It’s rough around the edges, since I’m learning how to do this myself. In Part 1 I talk about finding keypoints, descriptors, and matching two images together. Next time, I’ll talk about how to estimate the image transformation and how to actually do the stitching.

I’ve included my notebook here. You can see the original on Github if you like.

Note: There are a number of places where I’ve included the Jupyter Notebook widgets in the rendering below. You can click buttons and slide sliders, but it does not affect the output. It’s fun to play with the widgets though!

You can also skip to any of the headers below:



In this series I want to accomplish a simple task: stitch two images together. This is a pretty standard task in computer vision and image processing, and it has a lot of elements with lots of small details – iterating through images, comparing pixel values, interpolation, matrix operations, and more.

When I first started this task I thought it would be a piece of cake, but I’d forgotten how many steps and details there are, so I’m going to split this task across two posts. To break it down, here are the steps required:

  1. Extract feature points (Part 1)
  2. Calculate descriptors (Part 1)
  3. Match points (Part 1)
  4. Calculate transformation (Part 2)
  5. Stitch images (Part 2)

This notebook covers Part 1, everything from feature points to matching. In the next post I’ll take a look at how to actually do the stitching.

Setting up and loading images

There is a package made by the JuliaImages community called ImageFeatures.jl that implements lots of feature extraction methods. It isn’t committed to the Julia METADATA.jl packages directory, but the awesome thing is that you can still clone the package locally by doing:

Pkg.clone("https://github.com/JuliaImages/ImageFeatures.jl");

Or, when you have it installed and want to udpate:

Pkg.update("ImageFeatures");

Pretty cool that in Julia, packages can just be Github repositories!

In [1]:
using ImageFeatures, Images, FileIO;

As a set of sample images to use for the stitching exercise, I took two pictures of the Stata Center at MIT. It is home to CSAIL (the Computer Science and Artificial Intelligence Laboratory), where I and many others spend much of our MIT careers in. I really like using it as a case study for image stitching because as you can see, the building is pretty weird, leaving many opportunities for feature detection to go wrong!

In [2]:
img1 = load("imgs/stata-1.png")
img2 = load("imgs/stata-2.png")
[img1 img2]
Out[2]:
In [3]:
size(img1)
Out[3]:
(500,375)

Extracting Feature Points

The first step to stitching two images together is finding feature points, sometimes called keypoints. The main purpose of feature points are to identify points in an image that are “interesting”, for some value of interesting. When we are doing image stitching, we generally think points are interesting if they correspond to “corners” in the image. Basically, points at which boundaries happen between objects. The idea is that if we can identify points at the corners of objects, they are unique enough to match nicely if they appear in another image. Of course, this does not happen in practice, but it works fairly well, as we shall see later on.

One method to finding image corners is the Harris corner method, which uses the Sobel kernel (which I implemented and talked about in my last post) to find areas in the image that have strong gradients.

Thankfully, the Images.jl package we are familiar with implements imcorner:

In [4]:
?imcorner
Out[4]:
search: imcorner

corners = imcorner(img; [method])
corners = imcorner(img, threshold, percentile; [method])

Performs corner detection using one of the following methods –

1. harris
2. shi_tomasi
3. kitchen_rosenfeld

The parameters of the individual methods are described in their documentation. The maxima values of the resultant responses are taken as corners. If a threshold is specified, the values of the responses are thresholded to give the corner pixels. The threshold is assumed to be a percentile value unless percentile is set to false.

This is the first time I’ve come across the Julia method type, so I assumed that I could just pass the function itself (in this case, the harris function):

In [5]:
# construct keypoints
features_1 = imcorner(img1, harris);
Out[5]:
MethodError: no method matching imcorner(::Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2}, ::Images.#harris)
Closest candidates are:
  imcorner(::AbstractArray{T,N}, ::Any, ::Any; method, args...) at /home/mprat/.julia/v0.5/Images/src/corner.jl:27
  imcorner(::AbstractArray{T,N}; method, args...) at /home/mprat/.julia/v0.5/Images/src/corner.jl:19

But as you can see, it throws an error. Just to confirm, harris is a real Julia function in the Images package:

In [6]:
?harris
Out[6]:
search: harris shareproperties SharedMatrix

harris_response = harris(img; [k], [border], [weights])

Performs Harris corner detection. The covariances can be taken using either a mean weighted filter or a gamma kernel.

So what’s the problem? The problem is that in the imcorner function, method is a keyword argument, not a positional argument. How do you know that? Well, the function signature says:

function imcorner(img::AbstractArray; method::Function = harris, args...)

In case you missed it, there is a ; in between the img argument and the method argument. This means that the img argument is a positional argument, and everything after the ; (starting from method), is a keyword argument. The difference is that for a keyword argument, you always have to specify what variable the argument maps to in the function signature, like this:

In [7]:
features_1 = imcorner(img1, method=harris);

In this case, method is a keyword argument and we explicitly have to specify it when calling imcorner.

Now our function calls work just fine! In Julia, positional arguments come before keyword arguments, and keyword arguments cannot be positional arguments. So once you define an argument after a semicolon, it must always be called with a keyword. You can read more about the differences between positional and keyword arguments in the Julia docs.

Visualizing keypoints

Let’s see what type gets returned by imcorner and see if we can visualize the features.

In [8]:
summary(features_1)
Out[8]:
"500×375 BitArray{2}"
In [9]:
sizeof(features_1)
Out[9]:
23440

You can see that features_1 is a BitArray – this means it is the same as a binary mask the size of our image. A “True” value at a given index means that pixel is a feature, “False” means it is not a feature. The cool thing about BitArrays is that they only store a single bit of information per entry, rather than say an array of booleans, which uses more bits. You can see that this BitArray only takes up 23440 bytes. (aside: there are 8 bits in a byte, so $23440$ bytes $= 23440 \cdot 8 = 1875520$ bits $= 500 \cdot 375$ bits, one bit per entry in the BitArray)

What I want to do is visualize the feature points by drawing a circle around the feature point. I’ll do this by iterating over the BitArray and constructing a new one that stores the points that we will turn yellow.

In [10]:
function draw_points(image::AbstractArray, mask::BitArray{2}; c::Colorant=colorant"yellow")
    new_image = copy(image);
    new_image[mask] = c;
    return new_image;
end
Out[10]:
draw_points (generic function with 1 method)
In [11]:
draw_points(img1, features_1)
Out[11]:

This function works to draw our points, but it is not very generic. The BitArray size is encoded directly. We don’t actually care of the BitArray has exactly 2 channels, all we want to know is that it is the same size as the image.

The way we do this is by defining draw_points parametrized by two values T and N. As far as I can tell by reading Julia package code, by convension T is used to represent “Type” and N is used to represent dimension. What this function header says is that for an image of type T with size N, make sure the BitArray mask has the same size. We do this to make sure the calling of the image on the mask doesn’t yield an out of bound error.

You’ll notice that I also include a color argument as a colorant from the Colors.jl package. Because I specify a default value for this argument, I don’t have to call draw_points by passing a color every time. If I don’t, it will just use yellow.

In [12]:
function draw_points{T,N}(image::AbstractArray{T,N}, mask::BitArray{N}, c::Colorant=colorant"yellow")
    new_image = copy(image);
    new_image[mask] = c;
    return new_image;
end
Out[12]:
draw_points (generic function with 3 methods)
In [13]:
draw_points(img1, features_1)
Out[13]:

Now let’s see if we can use Interact.jl again to visualize the 3 different kinds of corner detectors implemented in ImageFeatures.

In [14]:
using Interact;
In [15]:
@manipulate for m=[harris, shi_tomasi, kitchen_rosenfeld]
    draw_points(img1, imcorner(img1, method=m))
end
Out[15]:

As we can see, the differences are pretty minimal, so let’s pick Harris corners to use as our keypoint locator and keep exploring parameters. We can see from ImageFeatures that the other way to call imcorner is to specify a percentile and a threshold.

If you specify imcorner WITHOUT these parameters, the function will compute a “local maximum response” of all neighbors, and only return points that have the maximum value of all of its neighbors. If you call imcorner with a percentile, the function does not do mean response and instead returns all the points above the percentile you specified. (Yet another way is to pass a threshold and “false”, which just does thresholding without taking percentiles into account, but this is hard to calibrate across methods).

Choosing the local maximum response or the percentile method is one of choice. Usually, local maximum response means you don’t have too many feature points that are very close together. This is great for downstream parts of your pipeline, since there are likely to be fewer feature points that look very much alike.

In [16]:
features_1 = imcorner(img1, 0.90, true, method=harris);
In [17]:
summary(features_1)
Out[17]:
"500×375 Array{Bool,2}"
In [18]:
sizeof(features_1)
Out[18]:
187500

One weird quirk of this method is that if you use imcorner and pass arguments, you don’t get back a BitArray, but instead an Array of Bool values. Compared to the $23440$ bytes from the BitArray above, this takes $187500 = 500 \cdot 375$ bytes, which is $187500 * 8 = 1500000$ bits, or 8 times more bits.

However, this also means that our implementation for draw_points won’t work. We specifically specified that the mask argument had to be a BitArray.

In [19]:
draw_points(img1, imcorner(img1, 0.90, true, method=harris))
Out[19]:
MethodError: no method matching draw_points(::Array{ColorTypes.RGB{FixedPointNumbers.Normed{UInt8,8}},2}, ::Array{Bool,2})
Closest candidates are:
  draw_points(::AbstractArray{T,N}, ::BitArray{2}; c) at In[10]:2
  draw_points{T,N}(::AbstractArray{T,N}, ::BitArray{N}) at In[12]:2
  draw_points{T,N}(::AbstractArray{T,N}, ::BitArray{N}, ::ColorTypes.Colorant{T,N}) at In[12]:2

What we can do is generalize our function a little bit more. Rather than forcing mask to be a BitArray, we can say that it has to be a type of AbstractArray{Bool, N}, the parent type of both BitArray and Array{Bool}. (Thanks StackOverflow post).

In [20]:
BitArray <: AbstractArray{Bool}
Out[20]:
true
In [21]:
Array{Bool} <: AbstractArray{Bool}
Out[21]:
true
In [22]:
function draw_points{T,N}(image::AbstractArray{T,N}, mask::AbstractArray{Bool, N}, c::Colorant=colorant"yellow")
    new_image = copy(image);
    new_image[mask] = c;
    return new_image;
end
Out[22]:
draw_points (generic function with 5 methods)

And now we can draw again!

In [23]:
draw_points(img1, imcorner(img1, 0.90, true, method=harris))
Out[23]:

Let’s see how the percentile affects the Harris corners (I really love Interact!!)

In [24]:
@manipulate for percentile=linspace(0.5, 1.0, 50)
    draw_points(img1, imcorner(img1, percentile, true, method=harris))
end
Out[24]:

From the outputs, I think we can safely say that the local-max response is the best option for finding feature points of those built into Images.jl – the points are not densely clustered, and we can match them more easily.

Of course, we can combine the two methods, just for kicks:

In [25]:
function corners_percentile_and_local_maximum(img::AbstractArray, percentile)
    percentile_corners = imcorner(img, percentile, true, method=harris)
    corners = falses(size(percentile_corners))
    maxima = map(CartesianIndex{2}, findlocalmaxima(percentile_corners))
    corners[maxima] = true
    return corners
end
Out[25]:
corners_percentile_and_local_maximum (generic function with 1 method)
In [26]:
@manipulate for percentile=linspace(0.01, 1.0, 50)
    draw_points(img1, corners_percentile_and_local_maximum(img1, percentile), c=colorant"red")
end
Out[26]:

As you can see, the resulting corners are very sparse (perhaps too sparse for our needs), so we’re just going to stick to the plain old Harris corner detector without playing with percentiles.

Calculating descriptors

Now that we have points on the image that we think are interesting, we want to compute a description of each point that will hopefully help us when trying to find similar points in another image. The purpose of a feature descriptor is to store information about the “orientations” of edges and their magnitudes. The idea is that edges have directionality – a horizontal edge that is darker on the top half is different than a horizontal edge that is darker on the bottom half. So if you are eventually looking to match edges (or corners) from two images together, you need to somehow encode the directionality of these edges in what we call feature descriptors.

There are many options for feature descriptors. You might have heard of SIFT feature points – those work well, except for two problems: (1) the SIFT algorithm is patented, so you can’t use it without paying license fees, and (2) SIFT generates a 128-dimensional vector for all the feature points. The good news is that feature descriptors like BRISK, BRIEF, and ORB were created that have similar properties to SIFT while being free and open, and they use fewer bits of information!

The ImageFeatures package provides convenient wrappers around keypoints and features, since this is a pretty common operation in image processing. Of course, this means I’ll have to make a new draw_points() method to see my keypoints. But no biggie, it’s the same code. Keypoints can be constructed from a BitArray or an Array{Bool}, so we are in the clear.

In [27]:
function draw_points(image::AbstractArray, mask::Keypoints; c::Colorant=colorant"yellow")
    new_image = copy(image);
    new_image[mask] = c;
    return new_image;
end
Out[27]:
draw_points (generic function with 6 methods)
In [28]:
draw_points(img1, Keypoints(imcorner(img1, method=harris)))
Out[28]:

The thing to note is that usually the feature descriptors are calculated a the grayscale version of an image. As it turns out, the information stored in Red, Green, and Blue channels are redundant (self-aside: maybe I’ll do an exploratory post on that at some future date), so you really only need to use a single channel that describes all of them to calculate descriptors. Let’s say we want to calculate BRISK features on our image. It computes a vector of binary features (i.e. 0s and 1s) that determine whether a particular orientation is present for that keypoint. There’s a pretty good graphic on the ImageFeatures.jl docs page about how to visualize the regions used by BRISK features – you can find it here.

ImageFeatures provides an interface for creating a descriptor, but first we need to convert our keypoints to the Feature type so we can store the orientations for each feature point.

In [29]:
?Features
Out[29]:
search: Features ImageFeatures feature_transform Feature

features = Features(boolean_img)
features = Features(keypoints)

Returns a Vector{Feature} of features generated from the true values in a boolean image or from a list of keypoints.

In [30]:
brisk_params = BRISK();
features_1 = Features(Keypoints(imcorner(img1, method=harris)));
desc_1, ret_features_1 = create_descriptor(Gray.(img1), features_1, brisk_params);

Matching keypoints

If we calculate the descriptors for both images, we can use the match_keypoints function to match both images together. What it does (as implemented in ImageFeatures), is match feature points computed on two images using a Hamming Distance function on the feature descriptors, since BRISK features output a set of binary vectors for each keypoint. The Hamming Distance is basically “how many bits different” are two vectors. So the vectors [1, 1, 1] and [0, 0, 0] have a Hamming Distance of 3, while the vectors [1, 1, 1] and [1, 1, 0] have a Hamming Distance of 1. What this means is that the smaller the Hamming distance between two keypoint descriptors, the more likely they are to match. To make sure that only one keypoint in each image is matched, for each keypoint in the first image, take the keypoint in the second image with the smallest Hamming distance.

I’m going to make a wrapper get_descriptors function so I don’t need to remember all the arguments to pass.

In [31]:
function get_descriptors(img::AbstractArray)
    brisk_params = BRISK();
    features = Features(Keypoints(imcorner(img, method=harris)));
    desc, ret_features = create_descriptor(Gray.(img), features, brisk_params);
end

function match_points(img1::AbstractArray, img2::AbstractArray, threshold::Float64=0.1)
    desc_1, ret_features_1 = get_descriptors(img1);
    desc_2, ret_features_2 = get_descriptors(img2);
    matches = match_keypoints(Keypoints(ret_features_1), Keypoints(ret_features_2), desc_1, desc_2, threshold);
    return matches;
end
Out[31]:
match_points (generic function with 2 methods)

We can then use these matches to draw a line between the matched points from the two images. I’ll need to install ImageDraw to make that work, because it implements a line function to draw a line between two points on an image. It is not an official package yet, so we just clone it from Github the same way we did with ImageFeatures.

In [32]:
# Pkg.clone("https://github.com/JuliaImages/ImageDraw.jl");
using ImageDraw;
In [33]:
matches = match_points(img1, img2, 0.1);
In [34]:
function draw_matches(img1, img2, matches)
    grid = [img1 img2];
    offset = CartesianIndex(0, size(img1, 2));
    for m in matches
        line!(grid, m[1], m[2] + offset)
    end
    grid
end
Out[34]:
draw_matches (generic function with 1 method)

The end result

In [35]:
draw_matches(img1, img2, matches)
Out[35]:

You can see that we get some pretty decent results! The sky is problematic, but skies are always going to be problematic, so I’m not too worried. We can probably get rid of that at the filtering and stitching stage in the next post.

However, to do the actual stitching we’ll have to use the matched points to compute a transformation from one image to another (known as a homography), while rejecting the bad matches. Then we’ll have to actually do the image stitching portions and correct any errors there.

Join me next time when I explore homographies, image transformations, and stitching. I found this package about projective geometry, so maybe I’ll take a look at that one!



Final thoughts

Extracting image features is a tricky business, and it is often dependent on the domain, type of image, white balance, etc. Before neural networks became popular, these manual orientation and gradient methods were all the rage. In computational photography and graphics, these methods are still going strong because they yield great results. I would be interested, when I’m more familiar with Julia, to dive into using a neural-network-based approach to extracting and matching keypoints between images. Who knows where my next project will take me!

Thank you for reading, as usual! I would love to hear from you if you have any suggestions, comments, or ideas for what I should learn next.

]]>
3551
Teaching a course using Julia http://www.juliabloggers.com/teaching-a-course-using-julia/ Fri, 24 Mar 2017 13:34:22 +0000 http://tpapp.github.io/post/teaching-a-course/ I just finished teaching a graduate course on practical aspects of dynamic optimization to our economics students. This year, for the first time, I taught the course using Julia, and this is a writeup of the experience. This was an intensive, 10-week course, with two classes per week, taught in the computer lab. A course on the theory of dynamic optimization was a prerequisite, this one was all about the actual numerical methods. The students had prior exposure to R and Matlab, and some of them have been using Julia for a while. In some classes, I talked about theory, sometimes I wrote code, ran it, and made improved versions, sometimes we treated the class as a practice session.

I wanted to focus on the actual methods, so I decided to use a subset of the language, “Julia light”, using the following concepts:

  1. scalars, algebra, arrays, indexing
  2. functions (with very basic dispatch, on an argument that contained problem parameters)
  3. control flow: for and if
  4. comprehension, concatenation
  5. structures (only immutable)
  6. docstrings

The purpose of the course was to show that one can easily implement seemingly abstract methods encountered in textbooks, dissect them, look at the caveats, and possibly adapt them to particular problems. Writing what I think of as production code would have involved teaching many new concepts to a class coming with very heterogeneous experience in programming, so I decided to steer clear of the following topics:

  1. modules
  2. macros
  3. the type system
  4. writing efficient code (even though we ended up doing a bit on that, and benchmarking, could not resist)

We used NLsolve, Optim, and Plots extensively, and ForwardDiff under the hood. Parameters was very useful for clean code.

Perspective of the instructor

Even when I wrote something in a suboptimal manner, it turned out to be fast enough. Julia is great in that respect. However, compilation time dominated almost everything that we did, especially for plots.

I was using Jupyter notebooks, inspired by 18.S096. While I am much, much slower writing code in Jupyter compared to Emacs, I think that this turned out to be a benefit: jumping around between windows is very difficult to follow. Interact was just great.

I made a small package for code we wrote in class, and distributed new code via Pkg.update(). This worked well most of the time.

We were using v0.5.0 and later transitioned to v0.5.1, which was seamless.

Since I was not using modules, sometimes the best way to extricate myself from a state was restarting the kernel. This became a running joke among the students (“when in doubt, restart the kernel”). This actually worked very well for the infamous #265.

Jupyter is great for interactive notes in class. I mixed a great deal of marked up text and LaTeX equations into the workbooks. Problem sets were handed in using Jupyter notebooks, and the exam (solving a dynamic programming problem) was also written in and graded as a notebook.

Unicode specials are addictive. Once you learn about α, you never name a variable alpha again.

Perspective of the students

I talked to the class at the end of the course about their experience with Julia, and some of them individually. The biggest issue for them was lack of easily searchable answers to common questions: for R and Matlab, a “how do you …” query turns up 100+ answers because many people have encountered the problem before. This was not the case for Julia. Lack of examples was an especially difficult issue for plots.

Debugging in Jupyter was difficult, since it mostly amounted to debugging by bisection, isolation, and occasionally printing. Students found some of the error messages cryptic (especially when it was about not having a matching method, since we did not really go into the type system).

The most puzzling transient bugs were because of #265 (“but I recompiled that function!”). This was solved by restarting the kernel, so the latter became somewhat of a panacea. Since compilation time dominated everything, this slowed things down considerably.

Takeaway

Would definitely do it again. Even with the issues, Julia was the most comfortable language to teach in.

]]>
By: Julia on Tamás K. Papp's blog

Re-posted from: http://tpapp.github.io/post/teaching-a-course/

I just finished teaching a graduate course on practical aspects of dynamic optimization to our economics students. This year, for the first time, I taught the course using Julia, and this is a writeup of the experience. This was an intensive, 10-week course, with two classes per week, taught in the computer lab. A course on the theory of dynamic optimization was a prerequisite, this one was all about the actual numerical methods. The students had prior exposure to R and Matlab, and some of them have been using Julia for a while. In some classes, I talked about theory, sometimes I wrote code, ran it, and made improved versions, sometimes we treated the class as a practice session.

I wanted to focus on the actual methods, so I decided to use a subset of the language, “Julia light”, using the following concepts:

  1. scalars, algebra, arrays, indexing
  2. functions (with very basic dispatch, on an argument that contained problem parameters)
  3. control flow: for and if
  4. comprehension, concatenation
  5. structures (only immutable)
  6. docstrings

The purpose of the course was to show that one can easily implement seemingly abstract methods encountered in textbooks, dissect them, look at the caveats, and possibly adapt them to particular problems. Writing what I think of as production code would have involved teaching many new concepts to a class coming with very heterogeneous experience in programming, so I decided to steer clear of the following topics:

  1. modules
  2. macros
  3. the type system
  4. writing efficient code (even though we ended up doing a bit on that, and benchmarking, could not resist)

We used NLsolve, Optim, and Plots extensively, and ForwardDiff under the hood. Parameters was very useful for clean code.

Perspective of the instructor

Even when I wrote something in a suboptimal manner, it turned out to be fast enough. Julia is great in that respect. However, compilation time dominated almost everything that we did, especially for plots.

I was using Jupyter notebooks, inspired by 18.S096. While I am much, much slower writing code in Jupyter compared to Emacs, I think that this turned out to be a benefit: jumping around between windows is very difficult to follow. Interact was just great.

I made a small package for code we wrote in class, and distributed new code via Pkg.update(). This worked well most of the time.

We were using v0.5.0 and later transitioned to v0.5.1, which was seamless.

Since I was not using modules, sometimes the best way to extricate myself from a state was restarting the kernel. This became a running joke among the students (“when in doubt, restart the kernel”). This actually worked very well for the infamous #265.

Jupyter is great for interactive notes in class. I mixed a great deal of marked up text and LaTeX equations into the workbooks. Problem sets were handed in using Jupyter notebooks, and the exam (solving a dynamic programming problem) was also written in and graded as a notebook.

Unicode specials are addictive. Once you learn about α, you never name a variable alpha again.

Perspective of the students

I talked to the class at the end of the course about their experience with Julia, and some of them individually. The biggest issue for them was lack of easily searchable answers to common questions: for R and Matlab, a “how do you …” query turns up 100+ answers because many people have encountered the problem before. This was not the case for Julia. Lack of examples was an especially difficult issue for plots.

Debugging in Jupyter was difficult, since it mostly amounted to debugging by bisection, isolation, and occasionally printing. Students found some of the error messages cryptic (especially when it was about not having a matching method, since we did not really go into the type system).

The most puzzling transient bugs were because of #265 (“but I recompiled that function!”). This was solved by restarting the kernel, so the latter became somewhat of a panacea. Since compilation time dominated everything, this slowed things down considerably.

Takeaway

Would definitely do it again. Even with the issues, Julia was the most comfortable language to teach in.

]]>
3578
Technical preview: Native GPU programming with CUDAnative.jl http://www.juliabloggers.com/technical-preview-native-gpu-programming-with-cudanative-jl/ Tue, 14 Mar 2017 00:00:00 +0000 http://julialang.org/blog/2017/03/cudanative After 2 years of slow but steady development, we would like to announce the first preview release of native GPU programming capabilities for Julia. You can now write your CUDA kernels in Julia, albeit with some restrictions, making it possible to use Julia’s high-level language features to write high-performance GPU code.

The programming support we’re demonstrating here today consists of the low-level building blocks, sitting at the same abstraction level of CUDA C. You should be interested if you know (or want to learn) how to program a parallel accelerator like a GPU, while dealing with tricky performance characteristics and communication semantics.

You can easily add GPU support to your Julia installation (see below for detailed instructions) by installing CUDAnative.jl. This package is built on top of experimental interfaces to the Julia compiler, and the purpose-built LLVM.jl and CUDAdrv.jl packages to compile and execute code. All this functionality is brand-new and thoroughly untested, so we need your help and feedback in order to improve and finalize the interfaces before Julia 1.0.

How to get started

CUDAnative.jl is tightly integrated with the Julia compiler and the underlying LLVM framework, which complicates version and platform compatibility. For this preview we only support Julia 0.6 built from source, on Linux or macOS. Luckily, installing Julia from source is well documented in the main repository’s README. Most of the time it boils down to the following commands:

$ git clone https://github.com/JuliaLang/julia.git
$ cd julia
$ git checkout v0.6.0-pre.alpha  # or any later tag
$ make                           # add -jN for N parallel jobs
$ ./julia

From the Julia REPL, installing CUDAnative.jl and its dependencies is just a matter of using the package manager. Do note that you need to be using the NVIDIA binary driver, and have the CUDA toolkit installed.

> Pkg.add("CUDAnative")

# Optional: test the package
> Pkg.test("CUDAnative")

At this point, you can start writing kernels and execute them on the GPU using CUDAnative’s @cuda! Be sure to check out the examples, or continue reading for a more textual introduction.

Hello World Vector addition

A typical small demo of GPU programming capabilities (think of it as the GPU Hello World) is to perform a vector addition. The snippet below does exactly that using Julia and CUDAnative.jl:

using CUDAdrv, CUDAnative

function kernel_vadd(a, b, c)
    # from CUDAnative: (implicit) CuDeviceArray type,
    #                  and thread/block intrinsics
    i = (blockIdx().x-1) * blockDim().x + threadIdx().x
    c[i] = a[i] + b[i]

    return nothing
end

dev = CuDevice(0)
ctx = CuContext(dev)

# generate some data
len = 512
a = rand(Int, len)
b = rand(Int, len)

# allocate & upload on the GPU
d_a = CuArray(a)
d_b = CuArray(b)
d_c = similar(d_a)

# execute and fetch results
@cuda (1,len) kernel_vadd(d_a, d_b, d_c)    # from CUDAnative.jl
c = Array(d_c)

using Base.Test
@test c == a + b

destroy(ctx)

How does it work?

Most of this example does not rely on CUDAnative.jl, but uses functionality from CUDAdrv.jl. This package makes it possible to interact with CUDA hardware through user-friendly wrappers of CUDA’s driver API. For example, it provides an array type CuArray, takes care of memory management, integrates with Julia’s garbage collector, implements @elapsed using GPU events, etc. It is meant to form a strong foundation for all interactions with the CUDA driver, and does not require a bleeding-edge version of Julia. A slightly higher-level alternative is available under CUDArt.jl, building on the CUDA runtime API instead, but hasn’t been integrated with CUDAnative.jl yet.

Meanwhile, CUDAnative.jl takes care of all things related to native GPU programming. The most significant part of that is generating GPU code, and essentially consists of three phases:

  1. interfacing with Julia: repurpose the compiler to emit GPU-compatible LLVM IR (no calls to CPU libraries, simplified exceptions, …)
  2. interfacing with LLVM (using LLVM.jl): optimize the IR, and compile to PTX
  3. interfacing with CUDA (using CUDAdrv.jl): compile PTX to SASS, and upload it to the GPU

All this is hidden behind the call to @cuda, which generates code to compile our kernel upon first use. Every subsequent invocation will re-use that code, convert and upload arguments1, and finally launch the kernel. And much like we’re used to on the CPU, you can introspect this code using runtime reflection:

# CUDAnative.jl provides alternatives to the @code_ macros,
# looking past @cuda and converting argument types
julia> CUDAnative.@code_llvm @cuda (1,len) kernel_vadd(d_a, d_b, d_c)
define void @julia_kernel_vadd_68711 {
    [LLVM IR]
}

# ... but you can also invoke without @cuda
julia> @code_ptx kernel_vadd(d_a, d_b, d_c)
.visible .func julia_kernel_vadd_68729(...) {
    [PTX CODE]
}

# or manually specify types (this is error prone!)
julia> code_sass(kernel_vadd, (CuDeviceArray{Float32,2},CuDeviceArray{Float32,2},CuDeviceArray{Float32,2}))
code for sm_20
        Function : julia_kernel_vadd_68481
[SASS CODE]

Another important part of CUDAnative.jl are the intrinsics: special functions and macros that provide functionality hard or impossible to express using normal functions. For example, the {thread,block,grid}{Idx,Dim} functions provide access to the size and index of each level of work. Local shared memory can be created using the @cuStaticSharedMem and @cuDynamicSharedMem macros, while @cuprintf can be used to display a formatted string from within a kernel function. Many math functions are also available; these should be used instead of similar functions in the standard library.

What is missing?

As I’ve already hinted, we don’t support all features of the Julia language yet. For example, it is currently impossible to call any function from the Julia C runtime library (aka. libjulia.so). This makes dynamic allocations impossible, cripples exceptions, etc. As a result, large parts of the standard library are unavailable for use on the GPU. We will obviously try to improve this in the future, but for now the compiler will error when it encounters unsupported language features:

julia> nope() = println(42)
nope (generic function with 1 method)

julia> @cuda (1,1) nope()
ERROR: error compiling nope: emit_builtin_call for REPL[1]:1 requires the runtime language feature, which is disabled

Another big gap is documentation. Most of CUDAnative.jl mimics or copies CUDA C, while CUDAdrv.jl wraps the CUDA driver API. But we haven’t documented what parts of those APIs are covered, or how the abstractions behave, so you’ll need to refer to the examples and tests in the CUDAnative and CUDAdrv repositories.

Another example: parallel reduction

For a more complex example, let’s have a look at a parallel reduction for Kepler-generation GPUs. This is a typical well-optimized GPU implementation, using fast communication primitives at each level of execution. For example, threads within a warp execute together on a SIMD-like core, and can share data through each other’s registers. At the block level, threads are allocated on the same core but don’t necessarily execute together, which means they need to communicate through core local memory. Another level up, only the GPU’s DRAM memory is a viable communication medium.

The Julia version of this algorithm looks pretty similar to the CUDA original: this is as intended, because CUDAnative.jl is a counterpart to CUDA C. The new version is much more generic though, specializing both on the reduction operator and value type. And just like we’re used to with regular Julia code, the @cuda macro will just-in-time compile and dispatch to the correct specialization based on the argument types.

So how does it perform? Turns out, pretty good! The chart below compares the performance of both the CUDAnative.jl and CUDA C implementations2, using BenchmarkTools.jl to measure the execution time. The small constant overhead (note the logarithmic scale) is due to a deficiency in argument passing, and will be fixed.

Performance comparison of parallel reduction
implementations.

We also aim to be compatible with tools from the CUDA toolkit. For example, you can profile Julia kernels using the NVIDIA Visual Profiler, or use cuda-memcheck to detect out-of-bound accesses3:

$ cuda-memcheck julia examples/oob.jl
========= CUDA-MEMCHECK
========= Invalid __global__ write of size 4
=========     at 0x00000148 in examples/oob.jl:14:julia_memset_66041
=========     by thread (10,0,0) in block (0,0,0)
=========     Address 0x1020b000028 is out of bounds

Full debug information is not available yet, so cuda-gdb and friends will not work very well.

Try it out!

If you have experience with GPUs or CUDA development, or maintain a package which could benefit from GPU acceleration, please have a look or try out CUDAnative.jl! We need all the feedback we can get, in order to prioritize development and finalize the infrastructure before Julia hits 1.0.

I want to help

Even better! There’s many ways to contribute, for example by looking at the issues trackers of the individual packages making up this support:

Each of those packages are also in perpetual need of better API coverage, and documentation to cover and explain what has already been implemented.

Thanks

This work would not have been possible without Viral Shah and Alan Edelman arranging my stay at MIT. I’d like to thank everybody at Julia Central and around, it has been a blast! I’m also grateful to Bjorn De Sutter, and IWT Vlaanderen, for supporting my time at Ghent University.


  1. See the README for a note on how expensive this currently is. 

  2. The measurements include memory transfer time, which is why a CPU implementation was not included (realistically, data would be kept on the GPU as long as possible, making it an unfair comparison). 

  3. Bounds-checked arrays are not supported yet, due to a bug in the NVIDIA PTX compiler

]]>
By: Julia Developers

Re-posted from: http://feedproxy.google.com/~r/JuliaLang/~3/rcBb4UCZ4pI/cudanative

After 2 years of slow but steady development, we would like to announce the first preview
release of native GPU programming capabilities for Julia. You can now write your CUDA
kernels in Julia, albeit with some restrictions, making it possible to use Julia’s
high-level language features to write high-performance GPU code.

The programming support we’re demonstrating here today consists of the low-level building
blocks, sitting at the same abstraction level of CUDA C. You should be interested if you
know (or want to learn) how to program a parallel accelerator like a GPU, while dealing with
tricky performance characteristics and communication semantics.

You can easily add GPU support to your Julia installation (see below for detailed
instructions) by installing CUDAnative.jl. This
package is built on top of experimental interfaces to the Julia compiler, and the
purpose-built LLVM.jl and
CUDAdrv.jl packages to compile and execute code.
All this functionality is brand-new and thoroughly untested, so we need your help and
feedback in order to improve and finalize the interfaces before Julia 1.0.

How to get started

CUDAnative.jl is tightly integrated with the Julia compiler and the underlying LLVM
framework, which complicates version and platform compatibility. For this preview we only
support Julia 0.6 built from source, on Linux or macOS. Luckily, installing Julia from
source is well documented in the main repository’s
README
.
Most of the time it boils down to the following commands:

$ git clone https://github.com/JuliaLang/julia.git
$ cd julia
$ git checkout v0.6.0-pre.alpha  # or any later tag
$ make                           # add -jN for N parallel jobs
$ ./julia

From the Julia REPL, installing CUDAnative.jl and its dependencies is just a matter of using
the package manager. Do note that you need to be using the NVIDIA binary driver, and have
the CUDA toolkit installed.

> Pkg.add("CUDAnative")

# Optional: test the package
> Pkg.test("CUDAnative")

At this point, you can start writing kernels and execute them on the GPU using CUDAnative’s
@cuda! Be sure to check out the
examples, or continue
reading for a more textual introduction.

Hello World Vector addition

A typical small demo of GPU programming capabilities (think of it as the GPU Hello World)
is to perform a vector addition. The snippet below does exactly that using Julia and
CUDAnative.jl:

using CUDAdrv, CUDAnative

function kernel_vadd(a, b, c)
    # from CUDAnative: (implicit) CuDeviceArray type,
    #                  and thread/block intrinsics
    i = (blockIdx().x-1) * blockDim().x + threadIdx().x
    c[i] = a[i] + b[i]

    return nothing
end

dev = CuDevice(0)
ctx = CuContext(dev)

# generate some data
len = 512
a = rand(Int, len)
b = rand(Int, len)

# allocate & upload on the GPU
d_a = CuArray(a)
d_b = CuArray(b)
d_c = similar(d_a)

# execute and fetch results
@cuda (1,len) kernel_vadd(d_a, d_b, d_c)    # from CUDAnative.jl
c = Array(d_c)

using Base.Test
@test c == a + b

destroy(ctx)

How does it work?

Most of this example does not rely on CUDAnative.jl, but uses functionality from CUDAdrv.jl.
This package makes it possible to interact with CUDA hardware through user-friendly wrappers
of CUDA’s driver API. For example, it provides an array type CuArray, takes care of memory
management, integrates with Julia’s garbage collector, implements @elapsed using GPU
events, etc. It is meant to form a strong foundation for all interactions with the CUDA
driver, and does not require a bleeding-edge version of Julia. A slightly higher-level
alternative is available under CUDArt.jl, building
on the CUDA runtime API instead, but hasn’t been integrated with CUDAnative.jl yet.

Meanwhile, CUDAnative.jl takes care of all things related to native GPU programming. The
most significant part of that is generating GPU code, and essentially consists of three
phases:

  1. interfacing with Julia: repurpose the compiler to emit GPU-compatible LLVM IR (no
    calls to CPU libraries, simplified exceptions, …)
  2. interfacing with LLVM (using LLVM.jl): optimize the IR, and compile to PTX
  3. interfacing with CUDA (using CUDAdrv.jl): compile PTX to SASS, and upload it to the
    GPU

All this is hidden behind the call to @cuda, which generates code to compile our kernel
upon first use. Every subsequent invocation will re-use that code, convert and upload
arguments1, and finally launch the kernel. And much like we’re used to on the CPU, you
can introspect this code using runtime reflection:

# CUDAnative.jl provides alternatives to the @code_ macros,
# looking past @cuda and converting argument types
julia> CUDAnative.@code_llvm @cuda (1,len) kernel_vadd(d_a, d_b, d_c)
define void @julia_kernel_vadd_68711 {
    [LLVM IR]
}

# ... but you can also invoke without @cuda
julia> @code_ptx kernel_vadd(d_a, d_b, d_c)
.visible .func julia_kernel_vadd_68729(...) {
    [PTX CODE]
}

# or manually specify types (this is error prone!)
julia> code_sass(kernel_vadd, (CuDeviceArray{Float32,2},CuDeviceArray{Float32,2},CuDeviceArray{Float32,2}))
code for sm_20
        Function : julia_kernel_vadd_68481
[SASS CODE]

Another important part of CUDAnative.jl are the intrinsics: special functions and macros
that provide functionality hard or impossible to express using normal functions. For
example, the {thread,block,grid}{Idx,Dim} functions provide access to the size and index
of each level of work. Local shared memory can be created using the @cuStaticSharedMem and
@cuDynamicSharedMem macros, while @cuprintf can be used to display a formatted string
from within a kernel function. Many math
functions
are also available;
these should be used instead of similar functions in the standard library.

What is missing?

As I’ve already hinted, we don’t support all features of the Julia language yet. For
example, it is currently impossible to call any function from the Julia C runtime library
(aka. libjulia.so). This makes dynamic allocations impossible, cripples exceptions, etc.
As a result, large parts of the standard library are unavailable for use on the GPU. We will
obviously try to improve this in the future, but for now the compiler will error when it
encounters unsupported language features:

julia> nope() = println(42)
nope (generic function with 1 method)

julia> @cuda (1,1) nope()
ERROR: error compiling nope: emit_builtin_call for REPL[1]:1 requires the runtime language feature, which is disabled

Another big gap is documentation. Most of CUDAnative.jl mimics or copies CUDA
C
, while CUDAdrv.jl wraps the CUDA
driver API
. But we haven’t documented what
parts of those APIs are covered, or how the abstractions behave, so you’ll need to refer to
the examples and tests in the CUDAnative and CUDAdrv repositories.

Another example: parallel reduction

For a more complex example, let’s have a look at a parallel
reduction
for Kepler-generation
GPUs
. This
is a typical well-optimized GPU implementation, using fast communication primitives at each
level of execution. For example, threads within a warp execute together on a SIMD-like core,
and can share data through each other’s registers. At the block level, threads are allocated
on the same core but don’t necessarily execute together, which means they need to
communicate through core local memory. Another level up, only the GPU’s DRAM memory is a
viable communication medium.

The Julia version of this algorithm
looks pretty similar to the CUDA original: this is as intended, because CUDAnative.jl is a
counterpart to CUDA C. The new version is much more generic though, specializing both on the
reduction operator and value type. And just like we’re used to with regular Julia code, the
@cuda macro will just-in-time compile and dispatch to the correct specialization based on
the argument types.

So how does it perform? Turns out, pretty good! The chart below compares the performance of
both the CUDAnative.jl and CUDA C implementations2, using BenchmarkTools.jl to measure
the execution time
. The small
constant overhead (note the logarithmic scale) is due to a deficiency in argument passing,
and will be fixed.

Performance comparison of parallel reduction
implementations.

We also aim to be compatible with tools from the CUDA toolkit. For example, you can profile
Julia kernels
using the NVIDIA Visual
Profiler, or use cuda-memcheck to detect out-of-bound accesses3:

$ cuda-memcheck julia examples/oob.jl
========= CUDA-MEMCHECK
========= Invalid __global__ write of size 4
=========     at 0x00000148 in examples/oob.jl:14:julia_memset_66041
=========     by thread (10,0,0) in block (0,0,0)
=========     Address 0x1020b000028 is out of bounds

Full debug information is not
available
yet, so cuda-gdb and
friends will not work very well.

Try it out!

If you have experience with GPUs or CUDA development, or maintain a package which could
benefit from GPU acceleration, please have a look or try out CUDAnative.jl! We need all the
feedback we can get, in order to prioritize development and finalize the infrastructure
before Julia hits 1.0.

I want to help

Even better! There’s many ways to contribute, for example by looking at the issues trackers
of the individual packages making up this support:

Each of those packages are also in perpetual need of better API coverage, and documentation
to cover and explain what has already been implemented.

Thanks

This work would not have been possible without Viral Shah and Alan Edelman arranging my stay
at MIT. I’d like to thank everybody at Julia Central and around, it has been a blast! I’m
also grateful to Bjorn De Sutter, and IWT Vlaanderen, for supporting my time at Ghent
University.


  1. See the README for a note on how expensive this currently is. 

  2. The measurements include memory transfer time, which is why a CPU implementation was not included (realistically, data would be kept on the GPU as long as possible, making it an unfair comparison). 

  3. Bounds-checked arrays are not supported yet, due to a bug in the NVIDIA PTX compiler

]]>
3530
Super fast pattern search: The FFT trick http://www.juliabloggers.com/super-fast-pattern-search-the-fft-trick/ Sun, 12 Mar 2017 00:00:00 +0000 https://juliohm.github.io/science/fft-trick/ In today’s post, I would like to document an old trick in image processing sometimes referred to as “The FFT trick” for exhaustive pattern search.

Although I would like to give credit to the person who introduced the trick in the literature, I don’t have this information with me. Please leave a comment if you have references.

Learning with a sliding window

Filtering an image with a sliding window is a routine operation in various disciplines like image processing, computer vision, geostatistics, and deep learning (e.g. convolutional neural networks), to name a few.

Sliding window on my breakfast.

Depending on the application, this operation serves different purposes. It is used everywhere as illustrated in the table:

Field of study Typical window Purposes
Image processing
  • Sobel kernel: $$\small\begin{bmatrix}1 & 0 & -1 \\ 2 & 0 & -2\\ 1 & 0 & -1\end{bmatrix}$$
  • Gaussian blur: $$\small\frac{1}{16}\begin{bmatrix}1 & 2 & 1 \\ 2 & 4 & 2\\ 1 & 2 & 1\end{bmatrix}$$
  • Edge detection
  • Blurring
  • Denoising
Computer vision Same as in Image processing
  • Image descriptors (e.g. GIST, HoG)
  • Invariance and alignment
Geostatistics A spatial pattern:
  • Pattern search
Deep learning Random weights:
  • Convolutional neural networks

BRAINSTORM: Why sliding windows? Couldn’t we learn more by using a different technique? What does a sliding window misses in terms of learning? :thinking:

Given its importance, a great deal of time was devoted by the scientific community to optimizing software and hardware for filtering images (or arrays) with sliding windows.

This post is about a clever trick that exploits legacy high-performance filtering algorithms to speed up pattern search on very large images.

FFT filtering

Consider a first naive implementation of the filtering operation in Julia that loops over the input image, extracts a subimage, multiplies it with the window entrywise, and adds the results:

function naivefilter(image::AbstractArray, window::AbstractArray)
  nx, ny = size(image)
  wx, wy = size(window)

  [sum(window .* image[i:i+wx-1,j:j+wy-1]) for i=1:nx-wx+1, j=1:ny-wy+1]
end

It works well, but is relatively slow for industry standards. Compare it to the efficient implementation available in Images.jl:

using Images
using BenchmarkTools

image = zeros(100,100)
window = zeros(3,3)

@benchmark naivefilter(image, window)
@benchmark imfilter(image, window)

# BenchmarkTools.Trial: (naivefilter)
#   memory estimate:  7.99 MiB
#   allocs estimate:  192093
#   --------------
#   minimum time:     6.512 ms (0.00% GC)
#   median time:      6.638 ms (0.00% GC)
#   mean time:        7.209 ms (7.67% GC)
#   maximum time:     10.569 ms (15.77% GC)
#   --------------
#   samples:          693
#   evals/sample:     1
#   time tolerance:   5.00%
#   memory tolerance: 1.00%
#
# BenchmarkTools.Trial: (imfilter)
#   memory estimate:  204.03 KiB
#   allocs estimate:  268
#   --------------
#   minimum time:     378.637 μs (0.00% GC)
#   median time:      388.839 μs (0.00% GC)
#   mean time:        406.210 μs (2.51% GC)
#   maximum time:     2.035 ms (78.63% GC)
#   --------------
#   samples:          10000
#   evals/sample:     1
#   time tolerance:   5.00%
#   memory tolerance: 1.00%

From the output, we notice turning into s and MiB turning into KiB. Besides the cache-friendly code that Tim Holy and others have written, I would like to comment on another optimization that Images.jl does that is not available in other image processing packages and programming languages.

Image filtering can alternatively be implemented with Fast Fourier Transforms (FFTs). The details of the algorithm and the theory behind it are beyond the scope of this post. The message that I want to convey is that depending on the window size, particularly for large windows, the FFT algorithm is much faster.

In Images.jl you can call the FFT algorithm explicitly by using:

imfilter(image, window, Algorithm.FFT())

but you don’t need to, Images.jl will call the faster algorithm depending on the window size you pass to it! :raised_hands:

The switch point between algorithms was set by means of benchmarks on a few modern processors. Feel free to submit a pull request if you feel that the switch point isn’t quite right for your hardware.

Exploiting FFT filtering for pattern search

Now that we’ve seen that image filtering is a super efficient operation (specially in Julia), let’s exploit it to find patterns in images.

Given an image and a spatial pattern , an exhaustive pattern search consists of computing a distance (or dissimilarity) between and all subimages of . The subimages for which the distance is smallest are called matches. If the computed distance is zero, the match is called a perfect match.

Now is the trick. Consider the Euclidean distance and relax the notation to mean

Loop over all subimages of and compute

where and have the same size. It turns out that we can implement this operation by rewriting

and replacing the original inefficient loop with two FFT filtering + one reduction:

 = imfilter(A.^2, ones(B))
AB = imfilter(A, B)
 = sumabs2(B)

D = abs( - 2AB + ) # here "abs" takes care of floating point errors

Isn’t that awesome? This trick was crucial in my ImageQuilting.jl package, it allowed me to process large 3D images in minutes instead of hours.

Finally, I would like to add that high-performance GPU implementations of FFT filtering are available in Julia. You can set an option in your package to use CLFFT.jl and have the fastest software in the market. :blush:

]]>
By: Júlio Hoffimann

Re-posted from: https://juliohm.github.io/science/fft-trick/

In today’s post, I would like to document an old trick in image processing sometimes referred to as “The FFT trick” for exhaustive pattern search.

Although I would like to give credit to the person who introduced the trick in the literature, I don’t have this information with me. Please leave a comment if you have references.

Learning with a sliding window

Filtering an image with a sliding window is a routine operation in various disciplines like image processing, computer vision, geostatistics, and deep learning (e.g. convolutional neural networks), to name a few.

Sliding window on my breakfast.

Depending on the application, this operation serves different purposes. It is used everywhere as illustrated in the table:

Field of study Typical window Purposes
Image processing
  • Sobel kernel: $$\small\begin{bmatrix}1 & 0 & -1 \\ 2 & 0 & -2\\ 1 & 0 & -1\end{bmatrix}$$
  • Gaussian blur: $$\small\frac{1}{16}\begin{bmatrix}1 & 2 & 1 \\ 2 & 4 & 2\\ 1 & 2 & 1\end{bmatrix}$$
  • Edge detection
  • Blurring
  • Denoising
Computer vision Same as in Image processing
  • Image descriptors (e.g. GIST, HoG)
  • Invariance and alignment
Geostatistics A spatial pattern:
  • Pattern search
Deep learning Random weights:
  • Convolutional neural networks

BRAINSTORM: Why sliding windows? Couldn’t we learn more by using a different technique? What does a sliding window misses in terms of learning? :thinking:

Given its importance, a great deal of time was devoted by the scientific community to optimizing software and hardware for filtering images (or arrays) with sliding windows.

This post is about a clever trick that exploits legacy high-performance filtering algorithms to speed up pattern search on very large images.

FFT filtering

Consider a first naive implementation of the filtering operation in Julia that loops over the input image, extracts a subimage, multiplies it with the window entrywise, and adds the results:

function naivefilter(image::AbstractArray, window::AbstractArray)
  nx, ny = size(image)
  wx, wy = size(window)

  [sum(window .* image[i:i+wx-1,j:j+wy-1]) for i=1:nx-wx+1, j=1:ny-wy+1]
end

It works well, but is relatively slow for industry standards. Compare it to the efficient implementation available in Images.jl:

using Images
using BenchmarkTools

image = zeros(100,100)
window = zeros(3,3)

@benchmark naivefilter(image, window)
@benchmark imfilter(image, window)

# BenchmarkTools.Trial: (naivefilter)
#   memory estimate:  7.99 MiB
#   allocs estimate:  192093
#   --------------
#   minimum time:     6.512 ms (0.00% GC)
#   median time:      6.638 ms (0.00% GC)
#   mean time:        7.209 ms (7.67% GC)
#   maximum time:     10.569 ms (15.77% GC)
#   --------------
#   samples:          693
#   evals/sample:     1
#   time tolerance:   5.00%
#   memory tolerance: 1.00%
#
# BenchmarkTools.Trial: (imfilter)
#   memory estimate:  204.03 KiB
#   allocs estimate:  268
#   --------------
#   minimum time:     378.637 μs (0.00% GC)
#   median time:      388.839 μs (0.00% GC)
#   mean time:        406.210 μs (2.51% GC)
#   maximum time:     2.035 ms (78.63% GC)
#   --------------
#   samples:          10000
#   evals/sample:     1
#   time tolerance:   5.00%
#   memory tolerance: 1.00%

From the output, we notice turning into s and MiB turning into KiB. Besides the cache-friendly code that Tim Holy and others have written, I would like to comment on another optimization that Images.jl does that is not available in other image processing packages and programming languages.

Image filtering can alternatively be implemented with Fast Fourier Transforms (FFTs). The details of the algorithm and the theory behind it are beyond the scope of this post. The message that I want to convey is that depending on the window size, particularly for large windows, the FFT algorithm is much faster.

In Images.jl you can call the FFT algorithm explicitly by using:

imfilter(image, window, Algorithm.FFT())

but you don’t need to, Images.jl will call the faster algorithm depending on the window size you pass to it! :raised_hands:

The switch point between algorithms was set by means of benchmarks on a few modern processors. Feel free to submit a pull request if you feel that the switch point isn’t quite right for your hardware.

Now that we’ve seen that image filtering is a super efficient operation (specially in Julia), let’s exploit it to find patterns in images.

Given an image and a spatial pattern , an exhaustive pattern search consists of computing a distance (or dissimilarity) between and all subimages of . The subimages for which the distance is smallest are called matches. If the computed distance is zero, the match is called a perfect match.

Now is the trick. Consider the Euclidean distance and relax the notation to mean

Loop over all subimages of and compute

where and have the same size. It turns out that we can implement this operation by rewriting

and replacing the original inefficient loop with two FFT filtering + one reduction:

 = imfilter(A.^2, ones(B))
AB = imfilter(A, B)
 = sumabs2(B)

D = abs( - 2AB + ) # here "abs" takes care of floating point errors

Isn’t that awesome? This trick was crucial in my ImageQuilting.jl package, it allowed me to process large 3D images in minutes instead of hours.

Finally, I would like to add that high-performance GPU implementations of FFT filtering are available in Julia. You can set an option in your package to use CLFFT.jl and have the fastest software in the market. :blush:

]]>
3528
Build & Deploy Machine Learning Apps on Big Data Platforms with Microsoft Linux Data Science Virtual Machine http://www.juliabloggers.com/build-deploy-machine-learning-apps-on-big-data-platforms-with-microsoft-linux-data-science-virtual-machine/ Fri, 10 Mar 2017 00:30:20 +0000 https://blogs.technet.microsoft.com/machinelearning/?p=11235 Read more]]> By: Cortana Intelligence and ML Blog Team

Re-posted from: https://blogs.technet.microsoft.com/machinelearning/2017/03/09/deploy-machine-learning-apps-to-big-data-platforms-with-linux-data-science-virtual-machine/

This post is authored by Gopi Kumar, Principal Program Manager in the Data Group at Microsoft.

This post covers our latest additions to the Microsoft Linux Data Science Virtual Machine (DSVM), a custom VM image on Azure, purpose-built for data science, deep learning and analytics. Offered in both Microsoft Windows and Linux editions, DSVM includes a rich collection of tools, seen in the picture below, and makes you more productive when it comes to building and deploying advanced machine learning and analytics apps.

The central theme of our latest Linux DSVM release is to enable the development and testing of ML apps for deployment to distributed scalable platforms such as Spark, Hadoop and Microsoft R Server, for operating on data at a very large scale. In addition, with this release, DSVM also offers Julia Computing’s JuliaPro on both Linux and Windows editions.


Here’s more on the new DSVM components you can use to build and deploy intelligent apps to big data platforms:

Microsoft R Server 9.0

Version 9.0 of Microsoft R Server (MRS) is a major update to enterprise-scale R from Microsoft, supporting parallel and distributed computation. MRS 9.0 supports analytics execution in the Spark 2.0 context. There’s a new architecture and simplified interface for deploying R models and functions as web services via a new library called mrsdeploy, which makes it easy to consume models from other apps using the open Swagger framework.

Local Spark Standalone Instance

Spark is one of the premier platforms for highly scalable big data analytics and machine learning. Spark 2.0 launched in mid-2016 and brings several improvements such as the revised machine learning library (MLLib), scaling and performance optimization, better ANSI SQL compliance and unified APIs. The Linux DSVM now offers a standalone Spark instance (based on the Apache Spark distribution), PySpark kernel in Jupyter to help you build and test applications on the DSVM and deploy them on large scale clusters like Azure HDInsight Spark or your own on-premises Spark cluster. You can develop your code using either Jupyter notebook or with the included community edition of the Pycharm IDE for Python or RStudio for R.

Single Node Local Hadoop (HDFS and YARN) Instance

To make it easier to develop Hadoop programs and/or use HDFS storage locally for development and testing, a single node Hadoop installation is built into the VM. Also, if you are developing on the Microsoft R Server for execution in Hadoop or Spark remote contexts, you can first test things locally on the Linux DSVM and then deploy the code to a remote scaled out Hadoop or Spark cluster or to Microsoft R Server. These DSVM additions are designed to help you iterate rapidly when developing and testing your apps, before they get deployed into large-scale production big data clusters.

The DSVM is also a great environment for self-learning and running training classes on big data technologies. We provide sample code and notebooks to help you get started quickly on the different data science tools and technologies offered.

DSVM Resources

New to DSVM? Here are resources to get you started:

Linux Edition

Windows Edition

The goal of DSVM is to make data scientists and developers highly productive in their work and provide a broad array of popular tools. We hope you find it useful to have these new big data tools pre-installed with the DSVM.

We always appreciate feedback, so please send in your comments below or share your thoughts with us at the DSVM community forum.

Gopi

]]>
3525
imfilter and arrays http://www.juliabloggers.com/imfilter-and-arrays/ Thu, 09 Mar 2017 00:00:00 +0000 http://learningjulia.com/2017/03/09/imfilter-and-arrays ]]> By: Michele Pratusevich

Re-posted from: http://learningjulia.com/2017/03/09/imfilter-and-arrays.html

Whereas in my last post I manually wrote a blur kernel and code to convolve an image, I didn’t want to do that every time an image convolution came up. So in this post I learned about the imfilter function from the ImageFiltering.jl package. I also learned about the @time macro and had a small aside on array creation.

I’ve included my notebook here. You can see the original on Github if you like.

If you want, you can skip to some headers below:



In this notebook, I will learn how to run some basic edge detection algorithms from Julia. Given an image, return an edge map. To do this, I want to learn how to do the following:

  1. Run a Sobel kernel on an input image
  2. Create an edge map using the high frequency signal from the image
  3. Have a long aside about array notation in Julia

This is a natural follow-up to my blurring computation from a previous exercise, since the Sobel operator is just a different kind of kernel. But it is also a common kernel needed in image manipulation, so I can compare my implemenation timing to the implementation in the ImageFiltering.jl package. After timing the built-in functions, I won’t even try to time my own implemenation…

Setup

First things first, let’s set up for manipulating images.

In [1]:
using Images, FileIO, Colors;

The test image is going to be of our former president, Barack Obama.

In [2]:
img = load("obama.jpg")
Out[2]:

The Sobel kernel should operate on grayscale images, and we can use operator broadcasting to do that:

In [3]:
img_gray = Gray.(img)
Out[3]:

Sobel kernels

The first thing we’ll try doing is manually running a Sobel image kernel on the input image. The Sobel operator is basically an approximation of derivatives in the X and Y directions of the image. The theory is that if there is a high gradient magnitude, there is an edge in that location. The way you compute the Sobel operator is to convolve this kernel:

$$K_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix}$$

in the X direction, and

$$K_y = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 0 & 0 \\ -1 & -2 & -1 \end{bmatrix}$$

in the Y direction. Note how they are just transposes of each other.

Practically, to compute the kernel, we need to iterate over the output image. As we discussed in a previous post, when transforming one image into another, you need to iterate over the output image, and for each pixel, find the pixels from the input image needed to compute that particular pixel. In the case of the Sobel kernel, we need to iterate over the output twice – once for the X direction, which needs 9 pixels for the computation, and once for the Y direction computation, which also needs 9 pixels.

The imfilter function

To apply image kernels, I am going to use the imfilter function from JuliaImages: http://juliaimages.github.io/latest/function_reference.html#ImageFiltering.imfilter. Rather than manually trying to implement out-of-bounds implementations or worrying about applying a dot product / convolution, let’s just use the builtins.

Another awesome feature of the JuliaImages library is the ability to pad the input according to 4 rules:

  1. replicate – repeat the edge value until infinity
  2. circular – image edges “wrap around”
  3. symmetric – reflect relative to the required position
  4. reflect – reflect relative to the edge

Read more here: http://juliaimages.github.io/latest/function_reference.html#Boundaries-and-padding-1. Which you can specify by doing something like:

    imfilter(img, kernel, "replicate")

In my case, I will just use the “replicate” mode.

In [4]:
kernel = [1 0 -1; 2 0 -2;1 0 -1];
sobel_x = imfilter(img_gray, kernel);
grad = imfilter(sobel_x, kernel')
Out[4]:
WARNING: assuming that the origin is at the center of the kernel; to avoid this warning, call `centered(kernel)` or use an OffsetArray
 in depwarn

There are a few things to note about the imfilter function:

  1. It doesn’t do convolution. Instead, it does correlation. The difference is basically that in convolution the kernel is flipped, so if you want to do convolution with imfilter, you should do reflect() around your kernel.
  2. You need to assign a “center” to the kernels. Normally when we think of kernels we think of the center as being the central number in the kernel – in the Sobel kernels above the center is (1, 1). To do this in Julia, the default calling of imfilter will do this, or you can explicitly instantiate the a kernel by calling centered():

     kernel = centered([1 0 -1; 2 0 -2;1 0 -1]);
     imfilter(img_gray, kernel)

    But in case you want a different center, you can use the OffsetArrays package.

Sobel to edges

What you can see from the image after the Sobel kernel is applied in the x and y directions is that there is a lot of noise all over the image – this is because the kernel is only looking at neighborhoods of 3×3. To get around this, we can just take the magnitude of the gradient, or even the 4th power of the gradient.

In [5]:
grad .^ 4
Out[5]:

As you can see, much less noise.

Separable kernels

However, using some interesting properties of the Sobel convolution operation, we can do even better. The Sobel kernel is separable – we can compute the full 3×3 kernel as a multiplication of a 1×3 and 3×1 kernels.

The kernels above ($K_x$ and $K_y$) can each be factored into two 1-dimensional kernels:

$$K_x = \begin{bmatrix} 1 & 0 & -1 \\ 2 & 0 & -2 \\ 1 & 0 & -1 \end{bmatrix} = \begin{bmatrix}1 \\ 2 \\ 1 \end{bmatrix} \cdot \begin{bmatrix}1 & 0 & -1 \end{bmatrix}$$$$K_y = \begin{bmatrix} 1 & 2 & 1 \\ 0 & 0 & 0 \\ -1 & -2 & -1 \end{bmatrix} = \begin{bmatrix}1 \\ 0 \\ -1 \end{bmatrix} \cdot \begin{bmatrix}1 & 2 & 1 \end{bmatrix}$$

So we can pass these 4 smaller kernels into imfilter() to get the same result:

In [6]:
kernel_1 = [1 2 1]';
kernel_2 = [1 0 -1];
grad_sep = imfilter(img_gray, (kernel_1, kernel_2, kernel_2', kernel_1'))
Out[6]:

Note the only difference is that the boundaries of the image are pure black. This is probably because of the kernel size – the kernel that is 3x1 needs only pixels horizontally, not vertically, so the “replication” step of the imfilter code replicates the default value, which is black.

Factoring Kernels

We can do even one better than manually factorizong kernels. We can use a feature built into the imfilter library to automatically factor the kernel itself before calling the imfilter function. You can see a detailed example of how to use it here: http://juliaimages.github.io/latest/imagefiltering.html#Factored-kernels-1.

In [7]:
grad_auto_factoring = imfilter(img_gray, kernelfactors((kernel, kernel')))
Out[7]:

In fact, the imfilter() function implementation automatically tries to factorize the kernels when it is called: https://github.com/JuliaImages/ImageFiltering.jl/blob/master/src/imfilter.jl#L10, so you don’t have to remember to get this functionality for free.

There is one more way to apply Sobel kernels to the image. Of course, imfilter() has a Sobel kernel built in. and it’s even automatically separable. The difference as you can see with the built-in sobel kernel is that it is normalized – the kernel is divided by the sum of the kernel (which in this case is 8), so the sum of all the factors in the kernel is equal to 1. This is a common technique in computer vision and image processing, but for visualizing the results here, we will multiply the output by 64 to get the same output image as above (8 for each kernel, and there are 2 kernels in the sobel kernel).

In [8]:
Kernel.sobel()
Out[8]:
(
[-0.125 -0.25 -0.125; 0.0 0.0 0.0; 0.125 0.25 0.125],

[-0.125 0.0 0.125; -0.25 0.0 0.25; -0.125 0.0 0.125])
In [9]:
grad_builtin_sobel = 64 * imfilter(img_gray, Kernel.sobel())
Out[9]:

The notable lack of OpenCV

When I first started out on this learning, I thought Opencv.jl would have a wrapper around all the functions available in the OpenCV library. However, the Opencv.jl library is not mature, and is only a manual wrapper around functions available in Opencv. The Sobel kernel is not one of the available functions, so I didn’t get a chance to test it! Notably, one of the issues in the Opencv.jl package is to automatically wrap all the Opencv functions: https://github.com/JuliaOpenCV/OpenCV.jl/issues/7. I can’t wait!

Timings

Now let’s compare the timings of all the methods. The Julia documentation talks about timing code regularly with the @time macro, and using that to guide development: http://docs.julialang.org/en/stable/manual/performance-tips/#measure-performance-with-time-and-pay-attention-to-memory-allocation. The most important thing to note about the @time macro is that first time it is called will also time all the compilation time needed for any code that is being timed AND code that actually will be timed. So to get an accurate reading, you should call @time twice. It also tells you how much memory is allocated.

Sometimes the output of @time will say something like 40.71% gc time – this means that 40% of the time is spent garbage-collecting unused variables and freeing memory. So you can either ignore those runs or ignore that amount of time in your final analysis.

We want to time all these runs:

  1. Manually calling imfilter with the full kernel
  2. Manually calling imfilter with the manually-factored kernel
  3. Calling imfilter with an explicit call to kernelfactor
In [10]:
# scenario 1
for i in 1:5
    @time imfilter(img_gray, (kernel, kernel'));
end
Out[10]:
0.023770 seconds (188 allocations: 2.087 MB)
0.020832 seconds (188 allocations: 2.087 MB)
0.018937 seconds (188 allocations: 2.087 MB)
0.017686 seconds (188 allocations: 2.087 MB)
0.016757 seconds (188 allocations: 2.087 MB)
In [11]:
# scenario 2
for i in 1:5
    @time imfilter(img_gray, (kernel_1, kernel_2, kernel_2', kernel_1'));
end
Out[11]:
  0.012856 seconds (1.64 k allocations: 2.164 MB)
  0.012849 seconds (1.64 k allocations: 2.164 MB)
  0.019774 seconds (1.64 k allocations: 2.164 MB, 35.54% gc time)
  0.012376 seconds (1.64 k allocations: 2.164 MB)
  0.012353 seconds (1.64 k allocations: 2.164 MB)
In [12]:
# scenario 3
for i in 1:5
    @time 64 * imfilter(img_gray, Kernel.sobel())
end
Out[12]:
  0.016268 seconds (300 allocations: 6.164 MB)
  0.012701 seconds (300 allocations: 6.164 MB)
  0.016306 seconds (300 allocations: 6.164 MB, 27.14% gc time)
  0.009772 seconds (300 allocations: 6.164 MB)
  0.008432 seconds (304 allocations: 6.164 MB)
  

Based on the timing results, we can see that manually trying to factor yields the WORST results! Julia does a lot of the heavy lifting behind-the-scenes with function inlining and compiling. For the kernel that is built-in, like the Sobel kernel (https://github.com/JuliaImages/ImageFiltering.jl/blob/master/src/kernelfactors.jl#L151), you can actually see that the factors are hand-coded, so it will naturally be faster.

But you never know if this will be true in general! So from what I can tell, the @time macro is critical for development.

Wrapping up with gradients

I originally intended this exploration to be USING the image gradient rather than all about computing it, but instead it turned into an explanation of @time and a divergence into array notation (see below), so next time I will actually use the image gradient to do some fun image manipulation!

An aside on array notation

The major learning from this post was actually about how array notation works in Julia. When I was first trying to get the kernel factoring calls working, I was having a problem with a particular error:

In [13]:
imfilter(img_gray, ([1;2;1]))
Out[13]:
WARNING: assuming that the origin is at the center of the kernel; to avoid this warning, call `centered(kernel)` or use an OffsetArray
 in depwarn(::String, ::Symbol) at ./deprecated.jl:64
 in _kernelshift at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:1048 [inlined]
 in kernelshift at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:1045 [inlined]
 in factorkernel at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:1015 [inlined]
 in imfilter at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:10 [inlined]
 in imfilter(::Array{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},2}, ::Array{Int64,1}) at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:5
 in include_string(::String, ::String) at ./loading.jl:441
 in execute_request(::ZMQ.Socket, ::IJulia.Msg) at /home/mprat/.julia/v0.5/IJulia/src/execute_request.jl:157
 in eventloop(::ZMQ.Socket) at /home/mprat/.julia/v0.5/IJulia/src/eventloop.jl:8
 in (::IJulia.##13#19)() at ./task.jl:360
while loading In[13], in expression starting on line 1
Out[13]:
ArgumentError: ImageFiltering.Pad{1}(:replicate,(1,),(1,)) lacks the proper padding sizes for an array with 2 dimensions

 in padindices(::Array{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},2}, ::ImageFiltering.Pad{1}) at /home/mprat/.julia/v0.5/ImageFiltering/src/border.jl:119
 in padarray at /home/mprat/.julia/v0.5/ImageFiltering/src/border.jl:145 [inlined]
 in imfilter! at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:234 [inlined]
 in imfilter!(::Array{ColorTypes.Gray{Float32},2}, ::Array{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},2}, ::Tuple{OffsetArrays.OffsetArray{Int64,1,Array{Int64,1}}}, ::ImageFiltering.Pad{0}, ::ImageFiltering.Algorithm.FIR) at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:145
 in imfilter!(::Array{ColorTypes.Gray{Float32},2}, ::Array{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},2}, ::Tuple{OffsetArrays.OffsetArray{Int64,1,Array{Int64,1}}}, ::ImageFiltering.Pad{0}) at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:139
 in imfilter(::Array{ColorTypes.Gray{FixedPointNumbers.Normed{UInt8,8}},2}, ::Array{Int64,1}) at /home/mprat/.julia/v0.5/ImageFiltering/src/imfilter.jl:5

Contrast to this:

In [14]:
imfilter(img_gray, ([1 2 1]))
Out[14]:

I couldn’t figure out what was happening. But what I realized was that the actual type that was returned when the kernel array was created was different. Take a look:

In [15]:
[1 2 3]
Out[15]:
1×3 Array{Int64,2}:
 1  2  3
In [16]:
[1; 2; 3;]
Out[16]:
3-element Array{Int64,1}:
 1
 2
 3
In [17]:
[[1 2 3]; [4 5 6]]
Out[17]:
2×3 Array{Int64,2}:
 1  2  3
 4  5  6

My original thought was that ; means “new row” and space means “new column”, but I was wrong. Actually, the distinction is between vertical and horizontal concatenation. So while [1 2 3] gives me a 1x3 array, I expected [1; 2; 3;] to give me a 3x1 array, but it doesn’t – it returns a 3-element array instead. To get a 3x1 array I need to do [1 2 3]':

In [18]:
[1 2 3]'
Out[18]:
3×1 Array{Int64,2}:
 1
 2
 3

I actually thought this was inconsistent and filed an issue on the Julia issue tracker: https://github.com/JuliaLang/julia/issues/20957. But it turns out I was obviously not the first one to notice this or report it. In fact, there is a whole thread on the Julia Discourse forums that I am now following: https://discourse.julialang.org/t/whats-the-meaning-of-the-array-syntax/938, and would be interested in learning how it gets fixed. I will be following this issue closely as the language evolves!

I think consistency would go a long way to making newbies not confused with new syntax. In any case, I learned something new.



Final thoughts

I suspect there will be a lot from the ImageFiltering.jl package in my future… it implements nice things like smart image padding, kernel factorization, and smart filtering. And the package author is active on Github, answering questions and closing issues as needed.

Thank you for reading, as usual! I would love to hear from you if you have any suggestions, comments, or ideas for what I should learn next.

]]>
3535
Flipping coins in continuous time: The Poisson process http://www.juliabloggers.com/flipping-coins-in-continuous-time-the-poisson-process/ Sun, 05 Mar 2017 00:00:00 +0000 https://juliohm.github.io/science/coin-flipping/ In this post, I would like to contrast two ways of thinking about a coin flip process—one of which is very intuitive and one of which can be very useful for certain applications. This insight is inspired on a stochastic modeling class by Ramesh Johari.

Discrete-time coin flip process

Suppose that I ask you to simulate a sequence of i.i.d. (independent and identically distributed) coin flips in a computer where the probability of flipping a coin and observing a head () is and the probability of flipping a coin and observing a tail () is :

, , , , , , ,

There are at least two ways of doing this:

Solution A — popular solution

The most popular solution by far is to go to your favorite programming language (e.g. Julia :blush:) and sample a sequence of :

using Distributions

p = .5  # probability of head
N = 100 # number of coin flips

coin_flips = rand(Bernoulli(p), N)
# 1, 0, 0, 0, 1, 1, ...

Solution B — not so popular solution

Think of a head (or ) as a successful coin flip. For example, you earn $1 whenever the head comes up and nothing otherwise. Keep track of the times when you had a successful outcome

Now is the trick. In a coin flip process, the interarrival times between successes are i.i.d. . Instead of simulating the coin flips directly, simulate the times at which a head will occur:

using Distributions

p = .5 # probability of head
N = 100 # number of coin flips

T = rand(Geometric(p), N)
head_times = cumsum(T+1)

# construct process in a single shot
coin_flips = zeros(N)
head_times = head_times[head_times  N]
coin_flips[head_times] = 1
# 1, 0, 1, 1, 0, 0, ...

Why is solution B relevant?

In other words, why would you care about a solution that is more verbose and somewhat less intuitive?

Imagine that the probability of success is extremely low. Would you like your computer spending hours generating tails (or ) before it can generate a head? For some applications, this is a critical question.

More than that, sometimes Solution A is not available…

Continuous-time coin flip process

The Poisson process is the equivalent of the coin flip process in continuous-time. Imagine that you divide real time (a real number) into small intervals of equal lenght :

If a coin is to be flipped at each of these marks in the limit when , then Solution A is not available! There are uncountably many coin flips in the interval .

However, simulating this process in a computer is possible with Solution B. For a Poisson process with success rate , the interarrival times are i.i.d. .

]]>
By: Júlio Hoffimann

Re-posted from: https://juliohm.github.io/science/coin-flipping/

In this post, I would like to contrast two ways of thinking about a coin flip process—one of which is very intuitive and one of which can be very useful for certain applications. This insight is inspired on a stochastic modeling class by Ramesh Johari.

Discrete-time coin flip process

Suppose that I ask you to simulate a sequence of i.i.d. (independent and identically distributed) coin flips in a computer where the probability of flipping a coin and observing a head () is and the probability of flipping a coin and observing a tail () is :

, , , , , , ,

There are at least two ways of doing this:

The most popular solution by far is to go to your favorite programming language (e.g. Julia :blush:) and sample a sequence of :

using Distributions

p = .5  # probability of head
N = 100 # number of coin flips

coin_flips = rand(Bernoulli(p), N)
# 1, 0, 0, 0, 1, 1, ...

Think of a head (or ) as a successful coin flip. For example, you earn $1 whenever the head comes up and nothing otherwise. Keep track of the times when you had a successful outcome

Now is the trick. In a coin flip process, the interarrival times between successes are i.i.d. . Instead of simulating the coin flips directly, simulate the times at which a head will occur:

using Distributions

p = .5 # probability of head
N = 100 # number of coin flips

T = rand(Geometric(p), N)
head_times = cumsum(T+1)

# construct process in a single shot
coin_flips = zeros(N)
head_times = head_times[head_times  N]
coin_flips[head_times] = 1
# 1, 0, 1, 1, 0, 0, ...

Why is solution B relevant?

In other words, why would you care about a solution that is more verbose and somewhat less intuitive?

Imagine that the probability of success is extremely low. Would you like your computer spending hours generating tails (or ) before it can generate a head? For some applications, this is a critical question.

More than that, sometimes Solution A is not available…

Continuous-time coin flip process

The Poisson process is the equivalent of the coin flip process in continuous-time. Imagine that you divide real time (a real number) into small intervals of equal lenght :

If a coin is to be flipped at each of these marks in the limit when , then Solution A is not available! There are uncountably many coin flips in the interval .

However, simulating this process in a computer is possible with Solution B. For a Poisson process with success rate , the interarrival times are i.i.d. .

]]>
3509
Julia + Weave.jl + hugo test http://www.juliabloggers.com/julia-weave-jl-hugo-test/ Fri, 03 Mar 2017 12:11:53 +0000 http://tpapp.github.io/post/hugo-julia-weave/ ]]> By: Julia on Tamás K. Papp's blog

Re-posted from: http://tpapp.github.io/post/hugo-julia-weave/

Testing the Hugo formatter for Weave.jl.

Testing inline code: 1+1=2.

Testing math:
$$x^2+y^2 = \int_0^1 f(z) dz$$

Testing code:

1+1
2

Testing proper highlighting:

function foo(x, y)
    x+y
end

A plot:

x = 1:10
y = x.^2
scatter(x, y, legend = false)

Caption for this plot

]]>
3580
Julia in Finance Seminar in London on the 16th of March http://www.juliabloggers.com/julia-in-finance-seminar-in-london-on-the-16th-of-march/ Thu, 02 Mar 2017 00:00:00 +0000 http://juliacomputing.com/blog/2017/03/02/Julia-Finance-CQF-London Julia Computing invites to the Julia in Finance Seminar in London on the 16th of March, organised in association with the CQF Institute. This event will introduce you to Julia, the easy-to-learn high-performance mathematical programming language that is taking the finance industry by storm.

The Julia in Finance Seminar takes place on Thursday, March 16th from 6:00 PM to 9 PM followed by refreshments and networking. The venue for this event is the Fitch Learning, The Corn Exchange, 55 Mark Lane, London, EC3R 7NE.

Come find out how quants, traders and data scientists from hedge funds, investment banks, and across financial services industry worldwide are using Julia to gain a mathematical computing advantage over their competitors by processing more data up to 1,000x faster than before. See how Julia enables innovation in the fintech and regtech sectors, helping companies and regulators keep ahead of a fast changing market.

There will be product demos, benchmarks, customer stories and use cases in Finance and Insurance, especially around trading, risk analytics and asset management among others.

Agenda

Topic Speaker Time
Registration & Welcome   6:00 PM
Julia Computing - Company overview, vision and products Dr. Viral Shah, CEO, Julia Computing and Co-Creator of Julia language 6:10 PM
Large Scale Capital Allocation Models with Julia Tim Thornham, Financial Modeling Solutions Director, Aviva 6:30 PM
Demo of JuliaRun - Limitless Scalability Avik Sengupta, VP Engineering, Julia Computing 6:50 PM
Demo of JuliaFin - Time Series Analytics & Financial Contracts Made Easy Simon Byrne, Core Developer, Julia Computing 7:10 PM
Julia powered foreign exchange trading analytics at BestX Pete Eggleston, Co-Founder & Director, and Matt Hardcastle, Senior Architect, BestX Ltd 7:30 PM
Closing Remarks, refreshments and networking   7:50 PM

The event is free to attend in person, and will also be live-streamed worldwide. Please register below to reserve your seat.

About Julia

Julia is the simplest, fastest and most powerful numerical computing language available today. Julia combines the functionality of quantitative environments such as Python and R, with the speed of production programming languages like Java and C++ to solve big data and analytics problems. Julia delivers dramatic improvements in simplicity, speed, capacity, and productivity for data scientists, algorithmic traders, quants, scientists, and engineers who need to solve massive computational problems quickly and accurately.

Julia offers an unbeatable combination of simplicity and productivity with speed that is thousands of times faster than other mathematical, scientific and statistical computing languages.

Partners and users include: Intel, The Federal Reserve Bank of New York, Lincoln Laboratory (MIT), The Moore Foundation and a number of private sector finance and industry leaders, including several of the world’s leading hedge funds, investment banks, asset managers and insurers.

About Julia Computing, Inc.

Julia Computing, Inc. was founded in 2015 to develop products around Julia such as JuliaFin. These products help financial firms leverage the 1,000x improvement in speed and productivity that Julia provides for trading, risk analytics, asset management, macroeconomic modeling and other areas. Products of Julia Computing make Julia easy to develop, easy to deploy and easy to scale.

About CQF Institute

Part of Fitch Learning, the CQF Institute is the awarding body for the Certificate in Quantitative Finance and provides a platform for educating and building the quantitative finance community around the globe. Promoting the highest standard in practical financial engineering, the Institute offers its members exclusive access to educational content featured on the Institute website, keeping its members up to date on the latest quant finance industry practices.

]]>
By: Julia Computing, Inc.

Re-posted from: http://juliacomputing.com/blog/2017/03/02/Julia-Finance-CQF-London.html

Julia Computing invites to the Julia in Finance Seminar in London on the 16th of March, organised in association with the CQF Institute. This event will introduce you to Julia, the easy-to-learn high-performance mathematical programming language that is taking the finance industry by storm.

The Julia in Finance Seminar takes place on Thursday, March 16th from 6:00 PM to 9 PM followed by refreshments and networking. The venue for this event is the Fitch Learning, The Corn Exchange, 55 Mark Lane, London, EC3R 7NE.

Come find out how quants, traders and data scientists from hedge funds, investment banks, and across financial services industry worldwide are using Julia to gain a mathematical computing advantage over their competitors by processing more data up to 1,000x faster than before. See how Julia enables innovation in the fintech and regtech sectors, helping companies and regulators keep ahead of a fast changing market.

There will be product demos, benchmarks, customer stories and use cases in Finance and Insurance, especially around trading, risk analytics and asset management among others.

Agenda

Topic Speaker Time
Registration & Welcome   6:00 PM
Julia Computing – Company overview, vision and products Dr. Viral Shah, CEO, Julia Computing and Co-Creator of Julia language 6:10 PM
Large Scale Capital Allocation Models with Julia Tim Thornham, Financial Modeling Solutions Director, Aviva 6:30 PM
Demo of JuliaRun – Limitless Scalability Avik Sengupta, VP Engineering, Julia Computing 6:50 PM
Demo of JuliaFin – Time Series Analytics & Financial Contracts Made Easy Simon Byrne, Core Developer, Julia Computing 7:10 PM
Julia powered foreign exchange trading analytics at BestX Pete Eggleston, Co-Founder & Director, and Matt Hardcastle, Senior Architect, BestX Ltd 7:30 PM
Closing Remarks, refreshments and networking   7:50 PM

The event is free to attend in person, and will also be live-streamed worldwide. Please register below to reserve your seat.

About Julia

Julia is the simplest, fastest and most powerful numerical computing language available today. Julia combines the functionality of quantitative environments such as Python and R, with the speed of production programming languages like Java and C++ to solve big data and analytics problems. Julia delivers dramatic improvements in simplicity, speed, capacity, and productivity for data scientists, algorithmic traders, quants, scientists, and engineers who need to solve massive computational problems quickly and accurately.

Julia offers an unbeatable combination of simplicity and productivity with speed that is thousands of times faster than other mathematical, scientific and statistical computing languages.

Partners and users include: Intel, The Federal Reserve Bank of New York, Lincoln Laboratory (MIT), The Moore Foundation and a number of private sector finance and industry leaders, including several of the world’s leading hedge funds, investment banks, asset managers and insurers.

About Julia Computing, Inc.

Julia Computing, Inc. was founded in 2015 to develop products around Julia such as JuliaFin. These products help financial firms leverage the 1,000x improvement in speed and productivity that Julia provides for trading, risk analytics, asset management, macroeconomic modeling and other areas. Products of Julia Computing make Julia easy to develop, easy to deploy and easy to scale.

About CQF Institute

Part of Fitch Learning, the CQF Institute is the awarding body for the Certificate in Quantitative Finance and provides a platform for educating and building the quantitative finance community around the globe. Promoting the highest standard in practical financial engineering, the Institute offers its members exclusive access to educational content featured on the Institute website, keeping its members up to date on the latest quant finance industry practices.

]]>
3503
A website with mortality charts built using Julia http://www.juliabloggers.com/a-website-with-mortality-charts-built-using-julia/ Wed, 01 Mar 2017 00:00:00 +0000 http://static-dust.klpn.se/posts/2017-03-01-mcsite.html A website with mortality charts built using Julia
Posted on 2017-03-01 by Karl Pettersson. Tags: epidemiology, julia

Since 2015, I have run a website with cause-specific mortality trends. The idea is to have a static site, which gives fast and easy access to information about international mortality trends, using open data available from WHO (2016), which, for many countries, covers the time period from 1950 up until recent times. The website is inspired by Whitlock (2012), which contains comprehensible charts with mortality trends based on these data, but has been unmaintained since 2013, when its creator died. Other sites with international cause-specific mortality trends I have seen tend to be slower, due to dynamic chart generation, and to cover only shorter time periods.

My implementation of the site generator, which was written in Python and R, had become rather messy, and the chart tools I used (matplotlib and ggplot2) are not really suited to make interactive web charts. I decided to rewrite the routines to generate the charts and the site files in Julia (albeit with the help of some non-Julia tools, as described below). These routines are now available as a GitHub repo, and I use them to generate the site in both English and Swedish versions.

The site is built as follows with the Julia package (see the README in the repo for instructions). The whole process is controlled with a JSON configuration file. YAML, using some non-JSON features, might be less cumbersome, and will perhaps be used once there is full YAML write support implemented in Julia. Julia functions mentioned are in the main Mortchartgen.jl file, if not otherwise stated.

  1. The WHO (2016) data files are downloaded and read into a MySQL database, using the functions in the Download.jl file.
  2. These data files contain cause of death codes from many different versions of the ICD classifications for different time periods and countries, and the codes are also often at a much more detailed level than I use in the charts. Therefore, the data on deaths is grouped using regular expressions defined in the configuration file. To avoid repeating this time-consuming regular expression matching, the resulting DataFrames can be saved in CSV files. There are still some issues with unsupported datatypes in the MySQL.jl package, which mean that grouping cannot be done at the SQL level and that prepared SQL statements cannot be used.
  3. The charts themselves are generated from the DataFrames created in step 2, using the Python Bokeh library, which is well-suited for interactive web visualizations. I call Bokeh directly using PyCall, instead of using the Bokeh.jl package, which is unmaintained. There is a batchplot function to generate all the charts for the site using the settings in the configuration file.
  4. The writeplotsite function generates the charts as well as HTML tables with links to the charts, a documentation file in Markdown format, and navigation menus for a given language, and copies these to a given output location. To generate the site files, except for the charts themselves, templates processed with Mustache.jl are used.
  5. The final generation of the site is done using Hakyll, a static site generator written in Haskell. In the output directory generated in step 4, there will be a Haskell source file, site.hs, which, provided that a Haskell complier and the Hakyll libraries are installed, can be compiled to an executable file. This file can then be run as ./site build to generate the site, which can then be uploaded to a web server. The resulting site is static in the sense that it has no code running on the server-side (but rendering the charts requires JavaScript on the client side).

References

Whitlock, Gary. 2012. “Mortality Trends [archived 21 december 2014].” http://web.archive.org/web/20141221203103/http://www.mortality-trends.org/.

WHO. 2016. “WHO Mortality Database.” http://www.who.int/healthinfo/mortality_data/en/index.html.

]]>
By: Karl Pettersson

Re-posted from: http://static-dust.klpn.se/posts/2017-03-01-mcsite.html

A website with mortality charts built using Julia

Posted on 2017-03-01

by Karl Pettersson.

Tags: epidemiology, julia

Since 2015, I have run a website with cause-specific mortality trends. The idea is to have a static site, which gives fast and easy access to information about international mortality trends, using open data available from WHO (2016), which, for many countries, covers the time period from 1950 up until recent times. The website is inspired by Whitlock (2012), which contains comprehensible charts with mortality trends based on these data, but has been unmaintained since 2013, when its creator died. Other sites with international cause-specific mortality trends I have seen tend to be slower, due to dynamic chart generation, and to cover only shorter time periods.

My implementation of the site generator, which was written in Python and R, had become rather messy, and the chart tools I used (matplotlib and ggplot2) are not really suited to make interactive web charts. I decided to rewrite the routines to generate the charts and the site files in Julia (albeit with the help of some non-Julia tools, as described below). These routines are now available as a GitHub repo, and I use them to generate the site in both English and Swedish versions.

The site is built as follows with the Julia package (see the README in the repo for instructions). The whole process is controlled with a JSON configuration file. YAML, using some non-JSON features, might be less cumbersome, and will perhaps be used once there is full YAML write support implemented in Julia. Julia functions mentioned are in the main Mortchartgen.jl file, if not otherwise stated.

  1. The WHO (2016) data files are downloaded and read into a MySQL database, using the functions in the Download.jl file.
  2. These data files contain cause of death codes from many different versions of the ICD classifications for different time periods and countries, and the codes are also often at a much more detailed level than I use in the charts. Therefore, the data on deaths is grouped using regular expressions defined in the configuration file. To avoid repeating this time-consuming regular expression matching, the resulting DataFrames can be saved in CSV files. There are still some issues with unsupported datatypes in the MySQL.jl package, which mean that grouping cannot be done at the SQL level and that prepared SQL statements cannot be used.
  3. The charts themselves are generated from the DataFrames created in step 2, using the Python Bokeh library, which is well-suited for interactive web visualizations. I call Bokeh directly using PyCall, instead of using the Bokeh.jl package, which is unmaintained. There is a batchplot function to generate all the charts for the site using the settings in the configuration file.
  4. The writeplotsite function generates the charts as well as HTML tables with links to the charts, a documentation file in Markdown format, and navigation menus for a given language, and copies these to a given output location. To generate the site files, except for the charts themselves, templates processed with Mustache.jl are used.
  5. The final generation of the site is done using Hakyll, a static site generator written in Haskell. In the output directory generated in step 4, there will be a Haskell source file, site.hs, which, provided that a Haskell complier and the Hakyll libraries are installed, can be compiled to an executable file. This file can then be run as ./site build to generate the site, which can then be uploaded to a web server. The resulting site is static in the sense that it has no code running on the server-side (but rendering the charts requires JavaScript on the client side).

References

Whitlock, Gary. 2012. “Mortality Trends [archived 21 december 2014].” http://web.archive.org/web/20141221203103/http://www.mortality-trends.org/.

WHO. 2016. “WHO Mortality Database.” http://www.who.int/healthinfo/mortality_data/en/index.html.

]]>
3498
What identifier is the most common in Julia? The answer might surprise you! http://www.juliabloggers.com/what-identifier-is-the-most-common-in-julia-the-answer-might-surprise-you/ Sun, 26 Feb 2017 15:00:00 +0000 http://kristofferc.github.io/post/tokenize/ Tokenize is a Julia package to perform lexical analysis of Julia source code. Lexing is the process of transforming raw source code (represented as normal text) into a sequence of tokens which is a string with an associated meaning. “Meaning” could here be if the string represent an operator, a keyword, a comment etc.

The example below shows lexing (or tokenization) of some simple code.

julia> using Tokenize

julia> collect(tokenize("""
       100
       "this is a string"
       'a'
       type
       foo
       *
       """))
13-element Array{Tokenize.Tokens.Token,1}:
 1,1-1,3          INTEGER        "100"
 1,4-2,0          WHITESPACE     "\n"
 2,1-2,18         STRING         "\"this is a string\""
 2,19-3,0         WHITESPACE     "\n"
 3,1-3,3          CHAR           "'a'"
 3,4-4,0          WHITESPACE     "\n"
 4,1-4,4          KEYWORD        "type"
 4,5-5,0          WHITESPACE     "\n"
 5,1-5,3          IDENTIFIER     "foo"
 5,4-6,0          WHITESPACE     "\n"
 6,1-6,1          OP             "*"
 6,2-7,0          WHITESPACE     "\n"
 7,1-7,0          ENDMARKER      ""

The displayed array containing the tokens has three columns. The first column shows the location where the string of the token starts and ends, which is represented as the line number (row) and at how many characters into the line (columns) the token starts / ends. The second column shows the type (kind) of token and, finally, the right column shows the string the token contains.

One of the different token kinds is the identifier. These are names that refer to different entities in the code. This includes variables, types, functions etc. The name of the identifiers are chosen by the programmer, in contrast to keywords which are chosen by the developers of the language. Some questions I thought interesting are:

  • What is the most common identifier in the Julia Base code (the code making up the standard library). Has it changed from 0.5 to 0.6?
  • How about packages? Is the source code there significantly different from the code in Julia Base in terms of the identifiers used?

The plan is to use Tokenize to lex both Julia Base and a bunch of packages, count the number of occurrences of each identifier and then summarize this as a top 10 list.

A Julia source code identifier counter

First, let’s create a simple counter type to keep track of how many times each identifier occur. This is a just a wrapper around a dictionary with a default value of 0 and a count! method that increments the counter for the supplied key:

immutable Counter{T}
    d::Dict{T, Int}
end
Counter{T}(::Type{T})= Counter(Dict{T, Int}())

Base.getindex{T}(c::Counter{T}, v::T) = get(c.d, v, 0)
getdictionary(c::Counter) = c.d
count!{T}(c::Counter{T}, v::T) = haskey(c.d, v) ? c.d[v] += 1 : c.d[v] = 1

A short example of the Counter type in action is showed below.

julia> c = Counter(String)
Counter{String}(Dict{String,Int64}())

julia> c["foo"]
0

julia> count!(c, "foo"); count!(c, "foo");

julia> c["foo"]
2

Now, we need a function that tokenizes a file and counts the number of identifiers in it. The code for such a function is shown below and a short explanation follows:

function count_tokentypes!(counter, filepath, tokentype)
    f = open(filepath, "r")
    for token in tokenize(f)
        if Tokens.kind(token) == tokentype
            count!(counter, untokenize(token))
        end
    end
    return counter
end

This opens the file at the path filepath, loops over the tokens, and if the kind of token is the tokentype the counter is incremented with the string of the token (extracted with untokenize) as the key. In Tokenize each type of token is represented by an enum, and the one corresponding to identifiers is named Tokens.IDENTIFIER.

As an example, we could run the function on a short file in base (nofloat_hashing.jl):

julia> BASEDIR =  joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia", "base")

julia> filepath = joinpath(BASEDIR, "nofloat_hashing.jl");

julia> c = Counter(String);

julia> count_tokentypes!(c, filepath, Tokens.IDENTIFIER)
Counter{String}(Dict("b"=>2,"x"=>8,"a"=>2,"h"=>8,"UInt32"=>1,"UInt16"=>1,"hx"=>3,"abs"=>1,"Int8"=>1,"Int16"=>1…))

julia> c["h"]
8

We see here that there are 8 occurrences of the identifier h in the file.

The next step is to apply the count_tokentypes function to all the files in the base directory. To that end, we create the applytofolder function:

function applytofolder(path, f)
    for (root, dirs, files) in walkdir(path)
        for file in files
            f(joinpath(root, file))
        end
    end
end

It takes a path to a folder and applies the function f on each file in that path. The walkdir function works recursively so each file will be visited this way.

Finally, we create a Counter and call the previously created count_tokentypes on all files that end with ".jl" using the applytofolder function:

julia> BASEDIR = joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia", "base")

julia> c = Counter(String)

julia> applytofolder(BASEDIR,
                     function(file)
                         if endswith(file, ".jl")
                             count_tokentypes!(c, file, Tokens.IDENTIFIER)
                         end
                     end)

The counter c now contains the count of all identifiers in the base folder:

julia> c["_uv_hook_close"]
12

julia> c["x"]
7643

julia> c["str"]
230

Analysis

We are interested in the most common identifiers so we create a function that extracts the n most common identifiers as two vectors. One with the identifiers and one with the counts:

function getntop(c::Counter, n)
    vec = Tuple{String, Int}[]
    for (k, v) in getdictionary(c)
        push!(vec, (k, v))
    end
    sort!(vec, by = x -> x[2], rev = true)
    vec_trunc = vec[1:n-1]
    identifiers = [v[1] for v in vec_trunc]
    counts      = [v[2] for v in vec_trunc]
    return identifiers, counts
end

To visualize this we use the excellent plotting package UnicodePlots:

julia> using UnicodePlots

julia> identifiers, counts = getntop(c, 10)

julia> barplot(identifiers, counts, title = "Base identifiers")
                   Base identifiers
       ┌────────────────────────────────────────┐
     x │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7643 │
     T │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7202   │
     A │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7001    │
     i │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 5119            │
   Ptr │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4239                │
     s │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4128                 │
     n │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3650                   │
     B │▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3143                     │
    io │▪▪▪▪▪▪▪▪▪▪▪▪ 2714                       │
       └────────────────────────────────────────┘

So there we have it, x is the winner, closely followed by T and A. This is perhaps not very surprising; x is a very common variable name, T is used a lot in parametric functions and A is used a lot in the Linear Algebra code base which is quite large.

Difference vs. 0.6

The plot below shows the same experiment repeated on the 0.6 code base:

               Base identifiers 0.6
       ┌────────────────────────────────────────┐ 
     x │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7718 │ 
     A │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7313   │ 
     T │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 6932    │ 
     i │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 5242            │ 
   Ptr │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4147                 │ 
     s │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4093                 │ 
     n │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3650                   │ 
     B │▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3174                     │ 
    io │▪▪▪▪▪▪▪▪▪▪▪▪▪ 2933                      │ 
       └────────────────────────────────────────┘ 

Most of the counts are relatively similar between 0.5 and 0.6 with the exception that A overtook T for the second place. In fact, the number of T identifers have decreased with almost 300 counts! What could have caused this? The answer is a new syntactic sugar feature available in Julia 0.6 which was implemented by Steven G. Johnson in PR #20414 . This allowed a parametric function with the syntax

foo{T <: Real}(Point{T}) = ...

to instead be written more tersely as

foo(Point{<:Real})...

In PR #20446 Pablo Zubieta went through the Julia code base and updated many of the function signatures to use this new syntax. Since T is a very common name to use for the parameter, the counts of T significantly decreased. And this is how A managed to win over T in 0.6 in the prestigeful “most common identifier”-competition.

Julia packages.

We now perform the same experiment but on the Julia package directory. For me, this includes around 130 packages:

julia> length(readdir(Pkg.dir()))
137

The results are:

                   Package identifiers
        ┌────────────────────────────────────────┐ 
      T │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 15425 │ 
      x │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 15062  │ 
   test │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 13624     │ 
      i │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 9989              │ 
      d │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 9562               │ 
      A │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 8280                 │ 
    RGB │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 8041                  │ 
      a │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7144                    │ 
      n │▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 6470                     │ 
        └────────────────────────────────────────┘ 

When we counted the Julia base folder we excluded all the files used for unit testing. For packages, these files are included and clearly test, used in the @test macro, is unsurprisingly very common. T, x and i are common in packages and Base but for some reason the variable d is more common in packages than in Base.

Conclusion

Doing these type of investigations has perhaps little practical use but it is, at least to me, a lot of fun. Feel free to tweak the code to find the most common string literal (Tokens.STRING) or perhaps most common integer (`Tokens.INTEGER) or anything else you can come up with.

Below is a wordcloud I made with the top 50 identifiers in Julia Base.

]]>
By: Kristoffer Carlsson on Kristoffer Carlsson

Re-posted from: http://kristofferc.github.io/post/tokenize/

Tokenize is a Julia package to perform lexical analysis of Julia source code.
Lexing is the process of transforming raw source code (represented as normal text) into a sequence of tokens which is
a string with an associated meaning. “Meaning” could here be if the string represent an operator, a keyword, a comment etc.

The example below shows lexing (or tokenization) of some simple code.

julia> using Tokenize

julia> collect(tokenize("""
       100
       "this is a string"
       'a'
       type
       foo
       *
       """))
13-element Array{Tokenize.Tokens.Token,1}:
 1,1-1,3          INTEGER        "100"
 1,4-2,0          WHITESPACE     "\n"
 2,1-2,18         STRING         "\"this is a string\""
 2,19-3,0         WHITESPACE     "\n"
 3,1-3,3          CHAR           "'a'"
 3,4-4,0          WHITESPACE     "\n"
 4,1-4,4          KEYWORD        "type"
 4,5-5,0          WHITESPACE     "\n"
 5,1-5,3          IDENTIFIER     "foo"
 5,4-6,0          WHITESPACE     "\n"
 6,1-6,1          OP             "*"
 6,2-7,0          WHITESPACE     "\n"
 7,1-7,0          ENDMARKER      ""

The displayed array containing the tokens has three columns. The first column shows the location where the string of the token starts and ends,
which is represented as the line number (row) and at how many characters into the line (columns) the token starts / ends.
The second column shows the type (kind) of token and, finally, the right column shows the string the token contains.

One of the different token kinds is the identifier. These are names that refer to different entities in the code.
This includes variables, types, functions etc. The name of the identifiers are chosen by the programmer,
in contrast to keywords which are chosen by the developers of the language.
Some questions I thought interesting are:

  • What is the most common identifier in the Julia Base code (the code making up the standard library). Has it changed from 0.5 to 0.6?
  • How about packages? Is the source code there significantly different from the code in Julia Base in terms of the identifiers used?

The plan is to use Tokenize to lex both Julia Base and a bunch of packages, count the number of occurrences of
each identifier and then summarize this as a top 10 list.

A Julia source code identifier counter

First, let’s create a simple counter type to keep track of how many times each identifier occur.
This is a just a wrapper around a dictionary with a default value of 0 and a
count! method that increments the counter for the supplied key:

immutable Counter{T}
    d::Dict{T, Int}
end
Counter{T}(::Type{T})= Counter(Dict{T, Int}())

Base.getindex{T}(c::Counter{T}, v::T) = get(c.d, v, 0)
getdictionary(c::Counter) = c.d
count!{T}(c::Counter{T}, v::T) = haskey(c.d, v) ? c.d[v] += 1 : c.d[v] = 1

A short example of the Counter type in action is showed below.

julia> c = Counter(String)
Counter{String}(Dict{String,Int64}())

julia> c["foo"]
0

julia> count!(c, "foo"); count!(c, "foo");

julia> c["foo"]
2

Now, we need a function that tokenizes a file and counts the number of identifiers in it.
The code for such a function is shown below and a short explanation follows:

function count_tokentypes!(counter, filepath, tokentype)
    f = open(filepath, "r")
    for token in tokenize(f)
        if Tokens.kind(token) == tokentype
            count!(counter, untokenize(token))
        end
    end
    return counter
end

This opens the file at the path filepath, loops over the tokens, and if the kind of token is the tokentype
the counter is incremented with the string of the token (extracted with untokenize) as the key.
In Tokenize each type of token is represented by an enum, and the one corresponding to identifiers is named
Tokens.IDENTIFIER.

As an example, we could run the function on a short file in base (nofloat_hashing.jl):

julia> BASEDIR =  joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia", "base")

julia> filepath = joinpath(BASEDIR, "nofloat_hashing.jl");

julia> c = Counter(String);

julia> count_tokentypes!(c, filepath, Tokens.IDENTIFIER)
Counter{String}(Dict("b"=>2,"x"=>8,"a"=>2,"h"=>8,"UInt32"=>1,"UInt16"=>1,"hx"=>3,"abs"=>1,"Int8"=>1,"Int16"=>1…))

julia> c["h"]
8

We see here that there are 8 occurrences of the identifier h in the file.

The next step is to apply the count_tokentypes function to all the files in the base directory.
To that end, we create the applytofolder function:

function applytofolder(path, f)
    for (root, dirs, files) in walkdir(path)
        for file in files
            f(joinpath(root, file))
        end
    end
end

It takes a path to a folder and applies the function f on each file in that path.
The walkdir function works recursively so each file will be visited this way.

Finally, we create a Counter and call the previously created count_tokentypes on all files
that end with ".jl" using the applytofolder function:

julia> BASEDIR = joinpath(JULIA_HOME, Base.DATAROOTDIR, "julia", "base")

julia> c = Counter(String)

julia> applytofolder(BASEDIR,
                     function(file)
                         if endswith(file, ".jl")
                             count_tokentypes!(c, file, Tokens.IDENTIFIER)
                         end
                     end)

The counter c now contains the count of all identifiers in the base folder:

julia> c["_uv_hook_close"]
12

julia> c["x"]
7643

julia> c["str"]
230

Analysis

We are interested in the most common identifiers so we create a function that
extracts the n most common identifiers as two vectors.
One with the identifiers and one with the counts:

function getntop(c::Counter, n)
    vec = Tuple{String, Int}[]
    for (k, v) in getdictionary(c)
        push!(vec, (k, v))
    end
    sort!(vec, by = x -> x[2], rev = true)
    vec_trunc = vec[1:n-1]
    identifiers = [v[1] for v in vec_trunc]
    counts      = [v[2] for v in vec_trunc]
    return identifiers, counts
end

To visualize this we use the excellent plotting package
UnicodePlots:

julia> using UnicodePlots

julia> identifiers, counts = getntop(c, 10)

julia> barplot(identifiers, counts, title = "Base identifiers")
                   Base identifiers
       ┌────────────────────────────────────────┐
     x │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7643 │
     T │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7202   │
     A │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7001    │
     i │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 5119            │
   Ptr │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4239                │
     s │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4128                 │
     n │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3650                   │
     B │▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3143                     │
    io │▪▪▪▪▪▪▪▪▪▪▪▪ 2714                       │
       └────────────────────────────────────────┘

So there we have it, x is the winner, closely followed by T and A.
This is perhaps not very surprising; x is a very common variable name,
T is used a lot in parametric functions and A is used a lot in the
Linear Algebra code base which is quite large.

Difference vs. 0.6

The plot below shows the same experiment repeated on the 0.6 code base:

               Base identifiers 0.6
       ┌────────────────────────────────────────┐ 
     x │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7718 │ 
     A │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7313   │ 
     T │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 6932    │ 
     i │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 5242            │ 
   Ptr │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4147                 │ 
     s │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 4093                 │ 
     n │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3650                   │ 
     B │▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 3174                     │ 
    io │▪▪▪▪▪▪▪▪▪▪▪▪▪ 2933                      │ 
       └────────────────────────────────────────┘ 

Most of the counts are relatively similar between 0.5 and 0.6 with the exception that A overtook T for the second place.
In fact, the number of T identifers have decreased with almost 300 counts!
What could have caused this?
The answer is a new syntactic sugar feature available in Julia 0.6 which was implemented by Steven G. Johnson in PR #20414 .
This allowed a parametric function with the syntax

foo{T <: Real}(Point{T}) = ...

to instead be written more tersely as

foo(Point{<:Real})...

In PR #20446 Pablo Zubieta went through the Julia code base and updated
many of the function signatures to use this new syntax.
Since T is a very common name to use for the parameter, the counts of T significantly decreased.
And this is how A managed to win over T in 0.6 in the prestigeful “most common identifier”-competition.

Julia packages.

We now perform the same experiment but on the Julia package directory. For me, this includes around 130 packages:

julia> length(readdir(Pkg.dir()))
137

The results are:

                   Package identifiers
        ┌────────────────────────────────────────┐ 
      T │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 15425 │ 
      x │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 15062  │ 
   test │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 13624     │ 
      i │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 9989              │ 
      d │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 9562               │ 
      A │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 8280                 │ 
    RGB │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 8041                  │ 
      a │▪▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 7144                    │ 
      n │▪▪▪▪▪▪▪▪▪▪▪▪▪▪ 6470                     │ 
        └────────────────────────────────────────┘ 

When we counted the Julia base folder we excluded all the files used for unit testing.
For packages, these files are included and clearly test, used in the @test macro, is
unsurprisingly very common. T, x and i are common in packages and Base but for some reason
the variable d is more common in packages than in Base.

Conclusion

Doing these type of investigations has perhaps little practical use but it is, at least to me, a lot of fun.
Feel free to tweak the code to find the most common string literal (Tokens.STRING) or perhaps most common integer (`Tokens.INTEGER)
or anything else you can come up with.

Below is a wordcloud I made with the top 50 identifiers in Julia Base.

]]>
3511
Blurring and manipulation http://www.juliabloggers.com/blurring-and-manipulation/ Fri, 24 Feb 2017 00:00:00 +0000 http://learningjulia.com/2017/02/24/blurring-and-manipulation ]]> By: Michele Pratusevich

Re-posted from: http://learningjulia.com/2017/02/24/blurring-and-manipulation.html

Now that I’ve been able to load an image and play with some colors, I wanted to implement some basic image manipulation functionality: blurring. To handle out-of-bounds errors when back-projecting to the input image, I learned about image overloading, played with the Interpolations.jl package, and generated some really trippy mirror art.

Here is my notebook, annotated with my findings and pretty pictures.

If you want, you can skip to some headers below:



In this notebook I want to play with image manipulation I’m going to implement a blurring function on an input image.

One fundamental concept in computational photography: iterate over the output to generate the appropriate input to use. The reason for this is that when you iterate over the output, you can compute which pixels from the input image(s) are used to compute the output. This ensures that every pixel in the output will be assigned a value.

So, to blur an image:

  1. Generate a kernel
  2. Allocate a new image (blurring can’t be an in-place operation because an input pixel is used multiple times, so can’t be overwritten)
  3. Iterate over the output, compute the input section used for that output, and convolve each input section with the kernel.

I am also curious to see the effects of different methods on the timing of the entire program.

In [1]:
using Images, TestImages, Colors, FileIO;
In [2]:
img = testimage("lighthouse")
Out[2]:

Generating kernels

I want to generate a few custom types (like structs in C++ or enum types in Python 3) to make data manipulation a little bit easier.

I tried to make a type called size, but got this error message (I took a screenshot because I didn’t actually want to overwrite a new method called size). It looks like there is another method defined in the images packages – actually MANY of them! No matter, I will define a new type with a different name!

In [3]:
load("03-overwrite-main.jpg")
Out[3]:
In [4]:
length(methods(size))
Out[4]:
120
In [5]:
type coord
    x::Int
    y::Int
end
In [6]:
methods(coord)
Out[6]:
3 methods for generic function Type:

  • coord(x::Int64, y::Int64) at In[5]:2
  • coord(x, y) at In[5]:2
  • (::Type{T}){T}(arg) at sysimg.jl:53

An interesting feature of Julia types is that they define default function constructors for a type, hence the output of the methods function above.

In the process of creating a function to generate a kernel in 2 dimensions, I want to generate a 1D gaussian. I can do this with an anonymous function. Even cooler, variable names in Julia can be represented as unicode! As a friend says, “it can lead you to do some really awesome things, and some things you probably should not do.” I think using LaTeX symbols in the Julia REPL is quite magical. Certainly easier to represent math!

(To generate unicode as a variable name, type \ followed by the appropriate LaTeX symbol and press tab.)

In [7]:
# interestingly, pi is an internal constant!
# and you can even use LaTeX to represent it
println(pi)
println(π)
println(e)
Out[7]:
π = 3.1415926535897...
π = 3.1415926535897...
e = 2.7182818284590...

The function name is \phi. This is the Gaussian function centered at 0, and you specify an input value x and a standard deviation \sigma.

In [8]:
ϕ(x, σ) = e^(-x^2 / (2 * σ^2)) / (σ * sqrt(2 * π))
Out[8]:
ϕ (generic function with 1 method)

The fact that I can use the Julia dot notation and have it broadcast the function appropriately is magical!

In [9]:
# generate a 1D gaussian kernel
ϕ.(-3:3, 4)
Out[9]:
7-element Array{Float64,1}:
 0.0752844
 0.0880163
 0.096667 
 0.0997356
 0.096667 
 0.0880163
 0.0752844

My coordinate type is initialized as an Int, but I want it to do something even better: I want to, when initializing a coordinate, if it is initialized as a float, have the coordinates rounded and turned into an Int. I can take advantage of the fact that types define constructors by adding a default constructor inside the type definition.

In [10]:
type coord
    x::Int
    y::Int
    coord(x, y) = new(Int(round(x)), Int(round(y)))
end
Out[10]:
WARNING: Method definition (::Type{Main.coord})(Any, Any) in module Main at In[5]:2 overwritten at In[10]:4.
In [11]:
methods(coord)
Out[11]:
3 methods for generic function Type:

  • coord(x::Int64, y::Int64) at In[5]:2
  • coord(x, y) at In[10]:4
  • (::Type{T}){T}(arg) at sysimg.jl:53
In [12]:
# you can see that it works!
coord(2.0, 2.0)
Out[12]:
coord(2,2)
In [13]:
function generate_gaussian_kernel(σ=3.0, kernel_size=coord(3, 3))
    # because of the magic of functions and types, this just works!
    center = coord(kernel_size.x / 2, kernel_size.y / 2);
    kernel = Array{Float64}(kernel_size.x, kernel_size.y);
    for x in 1:kernel_size.x
        for y in 1:kernel_size.y
            distance_to_center = sqrt((center.x - x)^2 + (center.y - y)^2);
            kernel[x, y] = ϕ(distance_to_center, σ);
        end
    end
    # normalize the coefficients at the end
    # so the sum is 1
    kernel /= sum(kernel);
    # and assert it
    @assert sum(kernel) - 1 < 1e-5;
    # explicitly return a kernel
    return kernel;
end
Out[13]:
generate_gaussian_kernel (generic function with 3 methods)
In [14]:
generate_gaussian_kernel(1, coord(3, 3))
Out[14]:
3×3 Array{Float64,2}:
 0.0751136  0.123841  0.0751136
 0.123841   0.20418   0.123841 
 0.0751136  0.123841  0.0751136

Putting it all together: blurring

Now let’s put it all together to generate a blurred version of the image using our kernels!

The first thing you need to do is make sure your input image is floats, so you can do proper arithmetic. After much Googling I found that you need to use the convert function from Images.jl. Alternatively, you can use casting.

In [15]:
[convert(Array{RGB{Float64}}, img) RGB{Float64}.(img)]
Out[15]:
In [16]:
function blur(img; kernel_size=15)
    # explicitly allocate an array of zeroes, to avoid
    # reading random memory, and ensuring the output
    # image starts out black
    img_blurred = zeros(RGB{Float64}, size(img, 1), size(img, 2));
    kernel = generate_gaussian_kernel(1, coord(kernel_size, kernel_size));
    for x in 1:size(img_blurred, 1) - kernel_size
        for y in 1:size(img_blurred, 2) - kernel_size
            # convolve the kernel with the image segment.
            # the .* operator is element-wise multiplication
            # as opposed to the * operator that is
            # matrix multiplication
            img_blurred[x, y] = sum(RGB{Float64}.(
                @view img[x:x+kernel_size-1, y:y+kernel_size-1]) .* kernel)
        end
    end
    return img_blurred;
end
Out[16]:
blur (generic function with 1 method)
In [17]:
img_blurred = blur(img)
Out[17]:

Interpolating nearest neighbors with the Interpolations package

The black pixels on the edges of the image don’t have exactly kernel-size matches in the input image, so for now I am just not not computing those pixels. The way I do this is by modifying the for loop to not iterate over the last pixels. The right way to fix this is to implement an “out of bounds” function to access pixels outside the range of an image.

In fact, the Julia Interpolations.jl package already implemented this!

In [18]:
# Pkg.add("Interpolations");
In [19]:
using Interpolations;

You can set an extrapolation object on an interpolation object.

In [20]:
?extrapolate
Out[20]:
search: extrapolate AbstractExtrapolation

Out[20]:

extrapolate(itp, scheme) adds extrapolation behavior to an interpolation object, according to the provided scheme.

The scheme can take any of these values:

  • Throw – throws a BoundsError for out-of-bounds indices
  • Flat – for constant extrapolation, taking the closest in-bounds value
  • Linear – linear extrapolation (the wrapped interpolation object must support gradient)
  • Reflect – reflecting extrapolation (indices must support mod)
  • Periodic – periodic extrapolation (indices must support mod)

You can also combine schemes in tuples. For example, the scheme (Linear(), Flat()) will use linear extrapolation in the first dimension, and constant in the second.

Finally, you can specify different extrapolation behavior in different direction. ((Linear(),Flat()), Flat()) will extrapolate linearly in the first dimension if the index is too small, but use constant etrapolation if it is too large, and always use constant extrapolation in the second dimension.

extrapolate(itp, fillvalue) creates an extrapolation object that returns the fillvalue any time the indexes in itp[x1,x2,...] are out-of-bounds.

In [21]:
# here I am using Constant interpolation and Constant
# (for some reason called Flat) extrapolation
# a.k.a. Nearest Neighbor interpolation
img_interp = extrapolate(interpolate(img, BSpline(Constant()), OnGrid()), Flat());
In [22]:
function blur_nn(img; kernel_size=15)
    img_blurred = zeros(RGB{Float64}, size(img, 1), size(img, 2));
    kernel = generate_gaussian_kernel(1, coord(kernel_size, kernel_size));
    for x in 1:size(img_blurred, 1)
        for y in 1:size(img_blurred, 2)
            # convolve the kernel with the image segment.
            # the .* operator is element-wise multiplication
            # as opposed to the * operator that is
            # matrix multiplication
            img_blurred[x, y] = sum(RGB{Float64}.(
                img_interp[x:x+kernel_size-1, y:y+kernel_size-1]) .* kernel)
        end
    end
    return img_blurred;
end
Out[22]:
blur_nn (generic function with 1 method)
In [23]:
img_blurred = blur_nn(img)
Out[23]:

Interestingly enough, the last few rows in the blurred image are black. The reason for this that if you look at the last rows in the original image, it is a fully-black row:

In [24]:
img[end, :]
Out[24]:

































































































































































































































































































































































































































































































































































































































































































































































































The previous-to-last row is not black. The original image from the TestImages package just has a row of black pixels for some reason.

In [25]:
img[end - 1, :]
Out[25]:

































































































































































































































































































































































































































































































































































































































































































































































































Creating a custom implementation of mirror interpolation

We can get rid of those artifacts by using a “mirror” interpolation. Every time you try to access a pixel outside the range, you take that index inside the range. I am going to overload the index operation to mirror the index into the image. But, I don’t want to overload the [] operator on every array – instead, I’m going to create a new InterpImage type, and add overload the getindex operator on the InterpImage type.

In [26]:
type InterpImage
    img::Array;
end

And while I’m here, I’ll just make an InterpImage with my original image.

In [27]:
iimg = InterpImage(img);

To overload the getindex function, you need import Base.getindex explicitly. If you don’t, the Julia REPL will tell you to do so.

In [28]:
import Base.getindex;

And as a warning, my mod-multiply-add arithmetic to compute the ranges is a little crazy, but it makes this a one-liner!

In [29]:
function getindex(im::InterpImage, xrange::UnitRange, yrange::UnitRange)
    # we take advantage of all the optimization already done on
    # UnitRanges. instead, we construct a new set of ranges and
    # use a single call to getindex on an image with those ranges
    max_x = size(img, 1);
    max_y = size(img, 2);
    new_xrange = Int.([
            ((floor(abs(x) / max_x) + 1) % 2) * 
            ((abs(x) % (max_x)) + 1) + 
            (floor(abs(x) / max_x) % 2) * 
            (max_x - (abs(x) % (max_x))) for x in xrange]);
    new_yrange = Int.([
            ((floor(abs(x) / max_y) + 1) % 2) *
            ((abs(x) % (max_y)) + 1) + 
            (floor(abs(x) / max_y) % 2) * 
            (max_y - (abs(x) % (max_y))) for x in yrange]);
    return im.img[new_xrange, new_yrange];
end
Out[29]:
getindex (generic function with 320 methods)

And as a result, I can mirror-index into my interpolated image. Even with negative indices! Trippy. I call this one, Lighthouseption.

In [30]:
iimg[-10:500, 200:1300]
Out[30]:

And this one, Lighthouse In The Sky.

In [31]:
iimg[-100:800, 1:800]
Out[31]:

And now we can update our previous blur function to use the mirrored interpolation, if it tries to access a pixel outside the range. The black line at the bottom of the image should go away.

In [32]:
function blur_mirror(img; kernel_size=15)
    img_blurred = zeros(RGB{Float64}, size(img, 1), size(img, 2));
    iimg = InterpImage(img);
    kernel = generate_gaussian_kernel(1, coord(kernel_size, kernel_size));
    for x in 1:size(img_blurred, 1)
        for y in 1:size(img_blurred, 2)
            # convolve the kernel with the image segment.
            # the .* operator is element-wise multiplication
            # as opposed to the * operator that is
            # matrix multiplication
            img_blurred[x, y] = sum(RGB{Float64}.(
                iimg[x:x+kernel_size-1, y:y+kernel_size-1]) .* kernel)
        end
    end
    return img_blurred;
end
Out[32]:
blur_mirror (generic function with 1 method)
In [33]:
img_blurred = blur_mirror(img)
Out[33]:

And so it does!
The problem with this implementation (that I can’t really do anything about, short of writing optimized for loop code), is that because I am not using UnitRange types to access the subsets of the image for my kernel calculations, it is slower than the previous blur implementations. For comparison, all the functions running with kernel size 15:

In [34]:
@elapsed img_blurred = blur(img, kernel_size=15)
Out[34]:
2.238338094
In [35]:
@elapsed img_blurred = blur_nn(img, kernel_size=15)
Out[35]:
3.673340281
In [36]:
@elapsed img_blurred = blur_mirror(img, kernel_size=15)
Out[36]:
13.829673882

Oh well. I don’t know why the mirror code is so much slower than the interpolation code, given that to construct my index range all I am doing is some arithmetic operations, which can’t be that much different than the interpolation code. That exploration is for another time.

An aside on broadcasting linear interpolations

I wanted to broadcast linear interpolations on two input ranges, which didn’t work using the construct above. First I created an interpolate / extrapolate object with linear interpolation in both.

In [37]:
img_interp = extrapolate(interpolate(img, BSpline(Linear()), OnGrid()), Linear());

And Robin showed me how to turn a function call into an index broadcast. I am still wrapping my head around what this means. But basically, the object img_interp, when called as a function with any args..., broadcast those arguments into indexing notation. So when img_interp is called with range arguments like 1:10, it calls each element in the range into index notation. I had to adjust my blurring code to call img_interp as a function, but it works! The right edge of the image looks different than a blur using nearest neighbor interpolation above.

In [38]:
(img_interp::typeof(img_interp))(args...) = img_interp[args...]
In [39]:
img_blurred = Array{RGB{Float64}}(size(img))
kernel_size = 15;
kernel = generate_gaussian_kernel(1, coord(kernel_size, kernel_size));
for x in 1:size(img_blurred, 1)
    for y in 1:size(img_blurred, 2)
        # convolve the kernel with the image segment.
        # the .* operator is element-wise multiplication
        # as opposed to the * operator that is
        # matrix multiplication
        img_blurred[x, y] = sum(RGB{Float64}.(
            img_interp.(x:x+kernel_size-1, (y:y+kernel_size-1))') .* kernel)
    end
end
In [40]:
img_blurred
Out[40]:


Incorrect code

I wanted to share a wrong piece of code and the resulting image with you. I ended up going with a different structure for my final implementation, as you can see from the notebook above, but a question for you: can you find my bug?

Here is the code:

function getindex(im::InterpImage, xrange::UnitRange, yrange::UnitRange)
	new_xrange = [];
	if xrange.stop < size(im.img, 1)
		new_xrange = xrange;
	else
		new_xrange = vcat(
			xrange.start:(xrange.stop - size(im.img, 1)),
			size(im.img, 1):-1:1)
	end

	new_yrange = [];
	if yrange.stop < size(im.img, 2)
		new_yrange = yrange;
	else
		new_yrange = vcat(
			yrange.start:(yrange.stop - size(im.img, 2)),
			size(im.img, 2):-1:1)
	end

	return im.img[new_xrange, new_yrange];
end

And it produces this output:

backwards lighthouse

Maybe THIS one should be Lighthouseception.

Hint: start and stop indices are difficult to get right.

Any comments for me?

]]>
3537
Image tilings and arrays http://www.juliabloggers.com/image-tilings-and-arrays/ Thu, 23 Feb 2017 01:00:04 +0000 http://learningjulia.com/2017/02/23/image-tiling-and-arrays ]]> By: Michele Pratusevich

Re-posted from: http://learningjulia.com/2017/02/23/image-tiling-and-arrays.html

In this notebook I decided to get more familiar with Julia basic functions, arrays, and operations by giving myself a small computational photography challenge! And on a suggestion, I played with the Interact.jl package too.

I’ve included the notebook I used for my image tiling manipulation, ending with playing with the Interact.jl library.

My notebook



In this notebook I want to take an image, break it into regions, and tile it with the maximum color from that region. It is an experiment in computational photography!

In [1]:
using Colors, Images, TestImages;

The one thing I find hard to get used to with Julia is that I don’t need to explicitly call functions on module names, like in Python. Just a small detail, but I’ll get used to it.

In [2]:
img = testimage("lighthouse");

Here’s some documentation about Arrays, which are what images are represented as: http://docs.julialang.org/en/stable/manual/arrays/

In [3]:
# find the size of the image
println(size(img))
Out[3]:
(512,768)
In [4]:
summary(img)
Out[4]:
"512×768 Array{RGB{N0f8},2}"
In [5]:
# create a random dense array the same size as the image
# it gets initialized with random numbers, which makes a really cool pattern!
tiled_img = Array{RGB{N0f8}, 2}(size(img))
Out[5]:

Another way you can initialize an array like another:

In [6]:
tiled_img = similar(img);

Like in IPython, you can add a “?” at the end of a function. In Julia, you put the question mark at the start of a function.

In [7]:
?linspace
Out[7]:
search: 

Iterating and Generating

There are many ways of generating a range of integers (that I will use to index into the image)

In [8]:
linspace(1, size(img, 1), 8)
Out[8]:
8-element LinSpace{Float64}:
 1.0,74.0,147.0,220.0,293.0,366.0,439.0,512.0
In [9]:
Int16.(linspace(1, size(img, 1), 8))
Out[9]:
8-element Array{Int16,1}:
   1
  74
 147
 220
 293
 366
 439
 512
In [10]:
Int16[i for i in linspace(1, size(img, 1), 8)]
Out[10]:
8-element Array{Int16,1}:
   1
  74
 147
 220
 293
 366
 439
 512

You can generate x, y tile indices with an interator-like thing or a generator-like thing.

In [11]:
# or to directly generate ints
1:8:size(img, 1)
Out[11]:
1:8:505
In [12]:
typeof(1:8:size(img, 1))
Out[12]:
StepRange{Int64,Int64}

And the visualizations for strips of images is awesome!

In [13]:
img[1:8:size(img, 1)]
Out[13]:

































































If you use these generators to index in both directions, what it basically does is tile the image and downsample.

In [14]:
stepsize = 16;
img[1:stepsize:size(img, 1), 1:stepsize:size(img, 2)]
Out[14]:
In [15]:
maximum(green.(img[1:10]))
Out[15]:
0.518N0f8
In [16]:
maximum(green.(img[1:16, 1:16]))
Out[16]:
0.533N0f8

And now let’s put it all together to take the max R, G, and B values in a block.

In [17]:
for x in 1:stepsize:size(img, 1)
    for y in 1:stepsize:size(img, 2)
        x_end = min(x + stepsize, size(img, 1));
        y_end = min(y + stepsize, size(img, 2));
        # the @view takes a view into the image rather than making a copy
        imgv = @view img[x:x_end, y:y_end]
        tiled_img[x:x_end, y:y_end] = RGB{N0f8}(
            maximum(red.(imgv)), maximum(green.(imgv)), maximum(blue.(imgv)));
    end
end

And we can concatenate the images to put them size by side.

In [18]:
[img tiled_img]
Out[18]:

I can even put all this into a function to call separately. Small notes about functions:

  • Arguments can either be positional or keyword, but not both. The function below means I have to call stepsize= every time I want to pass stepsize.
  • The last statement is the object returned by the function.
In [19]:
function max_tiling(img; stepsize=16)
    tiled_img = similar(img);
    for x in 1:stepsize:size(img, 1)
        for y in 1:stepsize:size(img, 2)
            x_end = min(x + stepsize, size(img, 1));
            y_end = min(y + stepsize, size(img, 2));
            imgv = @view img[x:x_end, y:y_end];
            tiled_img[x:x_end, y:y_end] = RGB{N0f8}(
                maximum(red.(imgv)), maximum(green.(imgv)), maximum(blue.(imgv)));
        end
    end
    tiled_img
end
Out[19]:
max_tiling (generic function with 1 method)
In [20]:
[img max_tiling(img, stepsize=16)]
Out[20]:
In [21]:
[img max_tiling(img, stepsize=8)]
Out[21]:

I am told that the interact.jl package (https://github.com/JuliaGizmos/Interact.jl) can let me make a slider to change the stepsize!

In [22]:
for x in 1:stepsize:size(img, 1)
    for y in 1:stepsize:size(img, 2)
        x_end = min(x + stepsize, size(img, 1));
        y_end = min(y + stepsize, size(img, 2));
        imgv = @view img[x:x_end, y:y_end]
        tiled_img[x:x_end, y:y_end] = RGB{N0f8}(maximum(red.(imgv)), maximum(green.(imgv)), maximum(blue.(imgv)));
    end
end
In [23]:
# Pkg.add("Interact")
In [24]:
using Interact;
Out[24]:

The widget lets me manipulate the step size and re-generate images directly in the notebook! Highly recommend this package =)

In [25]:
@manipulate for s=[2^x for x in 0:6]
    [img max_tiling(img, stepsize=s)]
end
Out[25]:


The widget in the last output of the notebook did not render properly, so I’m including a screenshot here:

Interact.jl widget

Random memory initialization

In line 4 (In[4]:), an uninitialized array takes random memory and random values. I don’t fully understand where this memory comes from, but I’m pretty sure it has to do with most-recently accessed memory. When I was running on my local machine, line 2 (In[2]:) loaded the original image of the lighthouse we saw in my first post. So when line 4 (creating a random uninitialized array) was executed, I got this pretty lighthouse artifact! You can see it is different image than came up in the second run of the notebook rendered above!

Random memory initialization

Jupyter notebook rendering aside

As another aside, in rendering the Jupyter notebook, the Interact library created widgets that generated custom Javascript during the nbconvert phase. I just removed that entire script section from the generated HTML, which is why I had to include a screenshot of the final widget at the end. I’m sorry for that ugliness – Jupyter and nbconvert are fantastic projects, they just can’t possibly cover all use cases!

Any comments for me?

]]>
3539
Finding ioctls with Clang and Cxx.jl http://www.juliabloggers.com/finding-ioctls-with-clang-and-cxx-jl/ Tue, 21 Feb 2017 00:00:00 +0000 http://juliacomputing.com/blog/2017/02/21/finding-ioctls-with-cxx Among the more popular tools in the linux debugging toolbox is strace, which allows users to easily trace and print out all system calls made by a program (as well as the arguments to these system calls). I recently found myself writing a similar tool to trace all the requests made to a proprietary network driver. I had access to the sources of the userspace-facing API for this driver, but strace proper did not know about it. I thus took this opportunity to write a general tool to extract ioctls from header files. The result is a compelling, but nevertheless compact enough for a blog post, application of Cxx.jl, the Julia-C++ FFI and Clang, LLVM’s C/C++ compiler project. In this blog post, I will walk you through my approach to this problem, highlighting both how to use Cxx.jl, and how to use the Clang C++ API. I will be focusing solely on extracing this data from header files. How to use it to write an strace like tool is a topic for another time.

Aside: About Cxx.jl

If you already know about Cxx.jl, feel free to move on to the next section. Cxx.jl is a julia package (available though the julia package manager), that allows julia to seamlessly interact with C++ code. It does this by taking advantage of Julia’s capabilities for staged code generation to put an actual C++ compiler into julia’s compilation pipeline. This looks roughtly as follows:

1. Julia parses code in julia syntax
2. The Cxx.jl package provides macros that translate from Julia to
   C++ code (either by punning on julia syntax, or by providing a string macro
   that allows the user to write C++ code directly). The package remembers
   what C++ code the user wants to run and leaves behind a generated function
   (basically a way for to ask the compiler to call back into the package
   when it wants to generate code for this particular function).
3. Later when Julia wants to run a function that includes, C++ code, it sees
   the generated function, calls back into Cxx.jl, which then performs
   proper type translation (of any julia values used in the C++ code and
   back to julia from C++ code), and creates a C++ AST which it then passes
   to Clang to compile. Clang compiles this AST and hands back LLVM IR, which
   Cxx.jl can then declare to julia is the implementation of the generated
   function.

Note that we could have generated native code in step 3, instead of LLVM IR, but using LLVM IR, allows cross-language LTO-style optimization.

The easiest way to interact with Cxx.jl package, is through the C++ REPL that comes with the package. After the Cxx package is loaded, this mode is automatically added to the julia REPL and can be accessed by pressing the ‘<’ key:

julia> using Cxx

julia> # Press '<' here

C++ > #include <iostream>
true

C++ > std::cout << "Hello World" << std::endl;
Hello World

The problem

Before getting into the code, let’s first carefully understand the problem at hand. We’re interested in ioctl requests made to a certain driver. ioctl is essentially the catch-all system call for all requests to drivers that don’t fit anywhere else. Such requests generally look like so:

int result = ioctl(fd, REQUEST_CODE, argument);

Where fd is a file descriptor associated with a resource managed by the driver, and argument is generally either an integer or (more commonly) a pointer to a more complicated structure of arguments for this request. REQUEST_CODE is a driver-specific code that specified what kind of request to make. In practice, there are exceptions to these rules, for a variety of reasons, but that’s the general structure. So let’s look at how the possible ioctl requests (I’ll just call them ioctls for short, even though there’s only one ioctl system call) are declared in the kernel headers. To be concrete, I’ll pick out the USB ioctls, but the discussion applies generally. Let’s look at an excerpt from the linked file:

#define USBDEVFS_SETCONFIGURATION  _IOR('U', 5, unsigned int)
#define USBDEVFS_GETDRIVER         _IOW('U', 8, struct usbdevfs_getdriver)
#define USBDEVFS_SUBMITURB32       _IOR('U', 10, struct usbdevfs_urb32)
#define USBDEVFS_DISCARDURB        _IO('U', 11)

Each of these lines defines an ioctl request (what I called request code above). Regular ioctls (ioctls defined like the ones above) have their request code split up as follows:

0xAAAABBCC
  \__/ | |
  Size | Code
    Category

which are the three values encoded by the #define above. The category is a (in theory unique by driver) 8-bit value that identifies to the kernel which driver to route the request to. The code is then used by the driver to identify the requested function. The size portion of the ioctl is ignored by the kernel, but may be used by the driver for backwards compatibility purposes. In the above define, the category is always 'U' (an ASCII-printable value is often chosen, but this is not a requirement), the numerical code follows, and lastly, we have the argument struct, which is used to compute the size.

For our ioctl dumper, we want four pieces of information: 1. The name of the ioctl 2. The category 3. The code 4. The argument struct (for size, as well as to extract field names such that we can print the argument structures in our ioctl dumper).

With a clear understand of what our goal is, let’s get to work!

Playing with the Preprocessor

It is probably possible to accomplish a lot of this using regexes or similar text processing, but there is a few distinct advantages to using a proper C compiler, such as clang for the task: 1. It has a correct preprocessor, so we can see though any defines, as well as making sure to ignore anything not reachable due to ifdef or similar 2. It is easier to use it to automatically extract the fieldnames/offset etc, while seeing through typedefs and anything else that might make it hard for a text processor to understand what’s going on.

So, let’s get us a Clang instance. Setting one up from the C++ API requires a bit of boilerplate, but luckily for us, the Cxx.jl package, comes with the ability to create separate Clang instances from the one it using to process C++ code:

julia> CCompiler = Cxx.new_clang_instance(
    false #= don't julia definitions =#,
    true #= C mode (as opposed to C++) =#)
Cxx.CxxInstance{2}()

Now, let’s use that instance, to load up the header file we discussed above:

julia> Cxx.cxxinclude(CCompiler, "linux/usbdevice_fs.h")
true

To achive our goal, we’ll need to manually work with Clang’s Parser and Preprocessor objects, so let’s extract those for easy reference:

julia> PP = icxx"&$(Cxx.active_instances[2].CI)->getPreprocessor();"
(class clang::Preprocessor *) @0x000055ff269d2380

julia> P  = Cxx.active_instances[2].Parser
(class clang::Parser *) @0x000055ff26338870

Before, going on, let’s see what the preprocessor knows about one of the macros we were looking at above:

C++ > $PP->dumpMacroInfo($PP->getIdentifierInfo("USBDEVFS_SETCONFIGURATION"))
MacroState 0x55ff26a9d3c8 USBDEVFS_SETCONFIGURATION
 DefMacroDirective 0x55ff1acbb980
  MacroInfo 0x55ff1acbb880
    #define <macro> _IOR('U', 5, unsigned int)

Ok, great. We have confirmed that the compiler parsed the header file and that it knows about our macro of interest. Let’s see where we can go from there. Consulting the clang documentation we find out about clang::Preprocessor::getMacroInfo and clang::MacroInfo::tokens, which would give us what we want. Let’s encode this into some julia functions for easy reference:

getIdentifierInfo(PP, name) = icxx"$PP->getIdentifierInfo($name);"
getMacroInfo(PP, II::pcpp"clang::IdentifierInfo") = icxx"$PP->getMacroInfo($II);"
getMacroInfo(PP, name::String) = getMacroInfo(PP, getIdentifierInfo(PP, name))
tokens(MI::pcpp"clang::MacroInfo") = icxx"$MI->tokens();"

We can now do:

julia> tokens(getMacroInfo(PP, "USBDEVFS_SETCONFIGURATION"))
(class llvm::ArrayRef<class clang::Token>) {
 .Data = (const class clang::Token *&) (const class clang::Token *) @0x000055ff269cec60
 .Length = (unsigned long &) 9
}

So we have our array of tokens. Of course, this is not very useful to us in this form, so let’s do two things. First, we’ll teach julia how to properly display Tokens:

# Convert Tokens that are identifiers to strings, we'll use these later
tok_is_identifier(Tok) = icxx"$Tok.is(clang::tok::identifier);"
Base.String(II::pcpp"clang::IdentifierInfo") = unsafe_string(icxx"$II->getName().str();")
function Base.String(Tok::cxxt"clang::Token")
    @assert tok_is_identifier(Tok)
    II = icxx"$Tok.getIdentifierInfo();"
    @assert II != C_NULL
    String(II)
end
getSpelling(PP, Tok) = unsafe_string(icxx"$PP->getSpelling($Tok);")
function Base.show(io::IO, Tok::Union{cxxt"clang::Token",cxxt"clang::Token&"})
    print(io, unsafe_string(icxx"clang::tok::getTokenName($Tok.getKind());"))
    print(io, " '", getSpelling(PP, Tok), "'")
end

Which’ll looks something like this (not I used the pointer from above)[^1]:

[^1] The astute reader may complain that I’m using the global PP instance to print this value. That is a valid complaint, and in the actual code, I made it an IOContext property, but I did not want to complicate this blog post with that discussion.

C++> *(clang::Token*) 0x000055ff269cec60
identifier '_IOR'

Great, we’re on the right track. Let’s also teach julia how to automatically iterate over llvm::ArrayRefs:

# Iteration for ArrayRef
import Base: start, next, length, done
const ArrayRef = cxxt"llvm::ArrayRef<$T>" where T
start(AR::ArrayRef) = 0
function next(AR::cxxt"llvm::ArrayRef<$T>", i) where T
    (icxx"""
        // Force a copy, otherwise we'll retain reference semantics in julia
        // which is not what people expect.
        $T element = ($AR)[$i];
        return element;
    """, i+1)
end
length(AR::ArrayRef) = icxx"$AR.size();"
done(AR::ArrayRef, i) = i >= length(AR)

Even though this may looks a bit complicated, all this is saying is that arrayrefs are indexed from one to AR.size(); and we can use the C++ bracket operator to access elements . With this defined, we get:

julia> collect(tokens(getMacroInfo(PP, "USBDEVFS_SETCONFIGURATION")))
9-element Array{Any,1}:
 identifier '_IOR'
 l_paren '('
 char_constant ''U''
 comma ','
 numeric_constant '5'
 comma ','
 unsigned 'unsigned'
 int 'int'
 r_paren ')'

We’re off to a great start.

Getting all the ioctls

As the previous section may have indicated, defining iteration over an object, is an enourmously powerful way to work with said object. Because everything in julia is generation, enabling iteration over an object, immidiately allows us to use any of the standard iteration tools (e.g. filters, maps, etc) to work with our objects.

With this, in mind, let’s see what we want to do. We know that ioctls are introduced by a macro that expands to _IO(...), _IOR(...), _IOW(...) or _IOWR(...). So let’s define iteration over Clang’s identifier table and write down exactly that:

start(tab::rcpp"clang::IdentifierTable") = icxx"$tab.begin();"
next(tab::rcpp"clang::IdentifierTable", it) = (icxx"$it->second;", icxx"++$it;")
done(tab::rcpp"clang::IdentifierTable", it) = icxx"$it == $tab.end();"
length(tab::rcpp"clang::IdentifierTable") = icxx"$tab.size();"
# Get all identifier that are macros
macros = Iterators.filter(II->icxx"$II->hasMacroDefinition();", icxx"$PP->getIdentifierTable();")
# Expand into tuples of (II, tokens)
IItokens = map(II->(II, collect(tokens(getMacroInfo(PP, II)))), macros)
# Now filter down to the ones we're interested in
ioctl_defs = filter(IItokens) do x
      II, tokens = x
      isempty(tokens) && return false
      tok_is_identifier(tokens[1]) && String(tokens[1]) in ["_IO","_IOR","_IOW","_IOWR"]
  end;

And if all worked well, we end up with:

julia> map(x->(String(x[1]),x[2]), ioctl_defs)
34-element Array{Tuple{String,Array{Any,1}},1}:
 ("USBDEVFS_FREE_STREAMS", Any[identifier '_IOR', l_paren '(', char_constant ''U'', comma ',', numeric_constant '29', comma ',', struct 'struct', identifier 'usbdevfs_streams', r_paren ')'])
 ("USBDEVFS_BULK32", Any[identifier '_IOWR', l_paren '(', char_constant ''U'', comma ',', numeric_constant '2', comma ',', struct 'struct', identifier 'usbdevfs_bulktransfer32', r_paren ')'])
 ("USBDEVFS_DISCARDURB", Any[identifier '_IO', l_paren '(', char_constant ''U'', comma ',', numeric_constant '11', r_paren ')'])

Extracting the fields from the structures

Now, it’s fairly simple to do any any post-processing we want here, and what to do exactly will depend on our intended application, but I do want to highlight how to extract the fields. At first I attempted to simply use the second to last token as the type name, but that doesn’t work very well, because some types are multiple tokens (e.g. unsigned int) and some others are only defined via (sometimes complicated preprocessor rules). Instead, the right way to do this, is to simply feed those tokens back through the parser. We’ll use a couple of definitions

"Given an array of tokens, queue them up for parsing"
function EnterTokenStream(PP::pcpp"clang::Preprocessor", tokens::Vector{cxxt"clang::Token"})
  # Vector memory layout is incompatible, convert to clang::Token**
  toks = typeof(tokens[1].data)[x.data for x in tokens]
  icxx"$PP->EnterTokenStream(llvm::ArrayRef<clang::Token>{
    (clang::Token*)$(pointer(toks)),
    (size_t)$(length(toks))
  },false);"
end
"Advance the parse if it's currently at EOF. This happens in incremental parsing mode and should be called before parsing."
function AdvanceIfEof(P)
  icxx"""
  if ($P->getPreprocessor().isIncrementalProcessingEnabled() &&
    $P->getCurToken().is(clang::tok::eof))
      $P->ConsumeToken();
  """
end
"Parse a type name"
function ParseTypeName(P::pcpp"clang::Parser")
  AdvanceIfEof(P)
  res = icxx"$P->ParseTypeName(nullptr, clang::Declarator::TypeNameContext);"
  !icxx"$res.isUsable();" && error("Parsing failed")
  Cxx.QualType(icxx"clang::Sema::GetTypeFromParser($res.get());")
end
"Parse a constant expression"
function ParseConstantExpression(P::pcpp"clang::Parser")
  AdvanceIfEof(P)
  res = icxx"$P->ParseConstantExpression();"
  !icxx"$res.isUsable();" && error("Parsing failed")
  e = icxx"$res.get();"
  e
end
"Convert a parsed constant literal to a julia Char (asserts on failure)"
function CharFromConstExpr(e)
  Char(icxx"""
    clang::cast<clang::CharacterLiteral>($e)->getValue();
  """)
end
tok_is_comma(Tok) = icxx"$Tok.is(clang::tok::comma);"
tok_is_numeric(Tok) = icxx"$Tok.is(clang::tok::numeric_constant);"

With these definitions:

julia> ioctl_tokens = first(ioctl_defs)[2]
9-element Array{Any,1}:
 identifier '_IOR'
 l_paren '('
 char_constant ''U''
 comma ','
 numeric_constant '29'
 comma ','
 struct 'struct'
 identifier 'usbdevfs_streams'
 r_paren ')'

julia> typename_tokens = Vector{cxxt"clang::Token"}(ioctl_tokens[findlast(tok_is_comma, ioctl_tokens)+1:end-1])
2-element Array{Cxx.CppValue{Cxx.CxxQualType{Cxx.CppBaseType{Symbol("clang::Token")},(false, false, false)},N} where N,1}:
 struct 'struct'
 identifier 'usbdevfs_streams'

julia> EnterTokenStream(PP, typename_tokens); QT = Cxx.desugar(ParseTypeName(P))
Cxx.QualType(Ptr{Void} @0x000055ff24a13960)

C++ > ((clang::RecordType*)&*$QT)->getDecl()->dump()
RecordDecl 0x55951860c240 </usr/lib/gcc/x86_64-linux-gnu/6.2.0/../../../../include/linux/usbdevice_fs.h:153:1, line:157:1> line:153:8 struct usbdevfs_streams definition
|-FieldDecl 0x55951860c300 <line:154:2, col:15> col:15 num_streams 'unsigned int'
|-FieldDecl 0x55951860c358 <line:155:2, col:15> col:15 num_eps 'unsigned int'
`-FieldDecl 0x55951860c418 <line:156:2, col:21> col:16 eps 'unsigned char [0]'

We could process these for exmaple as such. Here I’ll be using a different approach, where instead of using julia to do the iteration, I’ll just write most of the function in C++ and only call back into julia once at the end:

# C structure to julia array of fields
function inspectStruct(CC, S)
    CC = Cxx.instance(CC)
    ASTCtx = icxx"&$(CC.CI)->getASTContext();"
    fields = Any[]
    icxx"""
    auto &ARL = $ASTCtx->getASTRecordLayout($S);
    for (auto field : ($S)->fields()) {
      unsigned i = field->getFieldIndex();
      // Skip these for now
      if (field->isImplicit())
        continue;
      if (field->getType()->isUnionType())
        continue;
      if (field->getType()->isArrayType())
        continue;
      if (field->getType()->isRecordType() ||
          field->getType()->isEnumeralType())
        continue;
      if (field->getType()->isPointerType() &&
          field->getType()->getPointeeOrArrayElementType()->isRecordType())
        continue;
      $:(begin
        QT = Cxx.QualType(icxx"return field->getType();")
        push!(fields, (
          String(icxx"return field;"),
          Cxx.juliatype(QT),
          icxx"return $ASTCtx->toCharUnitsFromBits(ARL.getFieldOffset(i)).getQuantity();"
        ))
      nothing
    end);
    }
    """
    fields
end
julia> inspectStruct(CCompiler, icxx"((clang::RecordType*)&*$QT)->getDecl();")
2-element Array{Any,1}:
 ("num_streams", UInt32, 0)
 ("num_eps", UInt32, 4)

Conclusion

With the above code, we can easily extract and work with the definitions of ioctls in the linux headers. I hope this blog post has given you an idea of both how to use the Clang C++ API to do some C introspection, as well as some idea, of how to use some of the generic programming features in julia. The above is a pretty decent summary some of the first things I do when working with new data sources in julia: 1. Defining printing method for the relevant types 2. Define iteration on any container data structures 3. Use julia’s iteration tools to write whatever query I’m interested in Following this strategy usually gets one pretty far. In this case, it was essentially sufficient to solve our problem and provide a useful list of ioctls and the fields of their arguments to use in our ioctl dumping tool.

]]>
By: Julia Computing, Inc.

Re-posted from: http://juliacomputing.com/blog/2017/02/21/finding-ioctls-with-cxx.html

Among the more popular tools in the linux debugging toolbox is strace, which allows users to easily trace and print out all system calls made by a program (as well as the arguments to these system calls). I recently found myself writing a similar tool to trace all the requests made to a proprietary network driver. I had access to the sources of the userspace-facing API for this driver,
but strace proper did not know about it. I thus took this opportunity to write a general tool to extract ioctls from header files. The result is a compelling, but nevertheless compact enough for a blog post, application of Cxx.jl, the Julia-C++ FFI and Clang, LLVM’s C/C++ compiler project. In this blog post, I will walk you through my approach to this problem, highlighting both how to use Cxx.jl, and how to use the Clang C++ API. I will be focusing solely on extracing this data from header files. How to use it to write an strace like tool is a topic for another time.

Aside: About Cxx.jl

If you already know about Cxx.jl, feel free to move on to the next section. Cxx.jl is a julia package (available though the julia package manager), that allows julia to seamlessly interact with C++ code. It does this by taking advantage of Julia’s capabilities for staged code generation to put an actual C++ compiler into julia’s compilation pipeline. This looks roughtly as follows:

1. Julia parses code in julia syntax
2. The Cxx.jl package provides macros that translate from Julia to
   C++ code (either by punning on julia syntax, or by providing a string macro
   that allows the user to write C++ code directly). The package remembers
   what C++ code the user wants to run and leaves behind a generated function
   (basically a way for to ask the compiler to call back into the package
   when it wants to generate code for this particular function).
3. Later when Julia wants to run a function that includes, C++ code, it sees
   the generated function, calls back into Cxx.jl, which then performs
   proper type translation (of any julia values used in the C++ code and
   back to julia from C++ code), and creates a C++ AST which it then passes
   to Clang to compile. Clang compiles this AST and hands back LLVM IR, which
   Cxx.jl can then declare to julia is the implementation of the generated
   function.

Note that we could have generated native code in step 3, instead of LLVM IR,
but using LLVM IR, allows cross-language LTO-style optimization.

The easiest way to interact with Cxx.jl package, is through the C++ REPL that comes with the package. After the Cxx package is loaded, this mode is automatically added to the julia REPL and can be accessed by pressing the ‘<’ key:

julia> using Cxx

julia> # Press '<' here

C++ > #include <iostream>
true

C++ > std::cout << "Hello World" << std::endl;
Hello World

The problem

Before getting into the code, let’s first carefully understand the problem at hand.
We’re interested in ioctl requests made to a certain driver. ioctl is essentially the catch-all system call for all requests to drivers that don’t fit anywhere else.
Such requests generally look like so:

int result = ioctl(fd, REQUEST_CODE, argument);

Where fd is a file descriptor associated with a resource managed by the driver, and argument is generally either an integer or (more commonly) a pointer to a more complicated structure of arguments for this request. REQUEST_CODE is a driver-specific code that specified what kind of request to make. In practice, there are exceptions to these rules, for a variety of reasons, but that’s the general structure. So let’s look at how the possible ioctl requests (I’ll just call them ioctls for short, even though there’s only one ioctl system call) are declared in the kernel headers. To be concrete,
I’ll pick out the USB ioctls, but the discussion applies generally. Let’s look at an excerpt from the linked file:

#define USBDEVFS_SETCONFIGURATION  _IOR('U', 5, unsigned int)
#define USBDEVFS_GETDRIVER         _IOW('U', 8, struct usbdevfs_getdriver)
#define USBDEVFS_SUBMITURB32       _IOR('U', 10, struct usbdevfs_urb32)
#define USBDEVFS_DISCARDURB        _IO('U', 11)

Each of these lines defines an ioctl request (what I called request code above). Regular ioctls (ioctls defined like the ones above) have their request code split up as follows:

0xAAAABBCC
  \__/ | |
  Size | Code
    Category

which are the three values encoded by the #define above. The category is a (in theory unique by driver) 8-bit value that identifies to the kernel which driver to route the request to. The code is then used by the driver to identify the requested function. The size portion of the ioctl is ignored by the kernel, but may be used by the driver for backwards compatibility purposes. In the above define, the category is always 'U' (an ASCII-printable value is often chosen, but this is not a requirement), the numerical code follows, and lastly, we have the argument struct, which is used to compute the size.

For our ioctl dumper, we want four pieces of information:
1. The name of the ioctl
2. The category
3. The code
4. The argument struct (for size, as well as to extract field names such that we can print the argument structures in our ioctl dumper).

With a clear understand of what our goal is, let’s get to work!

Playing with the Preprocessor

It is probably possible to accomplish a lot of this using regexes or similar text processing, but there is a few distinct advantages to using a proper C compiler, such as clang for the task:
1. It has a correct preprocessor, so we can see though any defines, as well as making sure to ignore anything not reachable due to ifdef or similar
2. It is easier to use it to automatically extract the fieldnames/offset etc, while seeing through typedefs and anything else that might make it hard for a text processor to understand what’s going on.

So, let’s get us a Clang instance. Setting one up from the C++ API requires a bit of
boilerplate, but luckily for us, the Cxx.jl package, comes with the ability to create
separate Clang instances from the one it using to process C++ code:

julia> CCompiler = Cxx.new_clang_instance(
    false #= don't julia definitions =#,
    true #= C mode (as opposed to C++) =#)
Cxx.CxxInstance{2}()

Now, let’s use that instance, to load up the header file we discussed above:

julia> Cxx.cxxinclude(CCompiler, "linux/usbdevice_fs.h")
true

To achive our goal, we’ll need to manually work with Clang’s Parser and
Preprocessor objects, so let’s extract those for easy reference:

julia> PP = icxx"&$(Cxx.active_instances[2].CI)->getPreprocessor();"
(class clang::Preprocessor *) @0x000055ff269d2380

julia> P  = Cxx.active_instances[2].Parser
(class clang::Parser *) @0x000055ff26338870

Before, going on, let’s see what the preprocessor knows about one of the
macros we were looking at above:

C++ > $PP->dumpMacroInfo($PP->getIdentifierInfo("USBDEVFS_SETCONFIGURATION"))
MacroState 0x55ff26a9d3c8 USBDEVFS_SETCONFIGURATION
 DefMacroDirective 0x55ff1acbb980
  MacroInfo 0x55ff1acbb880
    #define <macro> _IOR('U', 5, unsigned int)

Ok, great. We have confirmed that the compiler parsed the header file and that it knows about our macro of interest. Let’s see where we can go from there. Consulting the clang documentation we find out about clang::Preprocessor::getMacroInfo and clang::MacroInfo::tokens, which would give us what we want. Let’s encode this into some julia functions for easy reference:

getIdentifierInfo(PP, name) = icxx"$PP->getIdentifierInfo($name);"
getMacroInfo(PP, II::pcpp"clang::IdentifierInfo") = icxx"$PP->getMacroInfo($II);"
getMacroInfo(PP, name::String) = getMacroInfo(PP, getIdentifierInfo(PP, name))
tokens(MI::pcpp"clang::MacroInfo") = icxx"$MI->tokens();"

We can now do:

julia> tokens(getMacroInfo(PP, "USBDEVFS_SETCONFIGURATION"))
(class llvm::ArrayRef<class clang::Token>) {
 .Data = (const class clang::Token *&) (const class clang::Token *) @0x000055ff269cec60
 .Length = (unsigned long &) 9
}

So we have our array of tokens. Of course, this is not very useful to us in this form, so let’s do two things. First, we’ll teach julia how to properly display Tokens:

# Convert Tokens that are identifiers to strings, we'll use these later
tok_is_identifier(Tok) = icxx"$Tok.is(clang::tok::identifier);"
Base.String(II::pcpp"clang::IdentifierInfo") = unsafe_string(icxx"$II->getName().str();")
function Base.String(Tok::cxxt"clang::Token")
    @assert tok_is_identifier(Tok)
    II = icxx"$Tok.getIdentifierInfo();"
    @assert II != C_NULL
    String(II)
end
getSpelling(PP, Tok) = unsafe_string(icxx"$PP->getSpelling($Tok);")
function Base.show(io::IO, Tok::Union{cxxt"clang::Token",cxxt"clang::Token&"})
    print(io, unsafe_string(icxx"clang::tok::getTokenName($Tok.getKind());"))
    print(io, " '", getSpelling(PP, Tok), "'")
end

Which’ll looks something like this (not I used the pointer from above)[^1]:

[^1] The astute reader may complain that I’m using the global PP instance to print this value. That is a valid complaint, and in the actual code, I made it an IOContext property, but I did not want to complicate this blog post with that discussion.

C++> *(clang::Token*) 0x000055ff269cec60
identifier '_IOR'

Great, we’re on the right track. Let’s also teach julia how to automatically iterate over llvm::ArrayRefs:

# Iteration for ArrayRef
import Base: start, next, length, done
const ArrayRef = cxxt"llvm::ArrayRef<$T>" where T
start(AR::ArrayRef) = 0
function next(AR::cxxt"llvm::ArrayRef<$T>", i) where T
    (icxx"""
        // Force a copy, otherwise we'll retain reference semantics in julia
        // which is not what people expect.
        $T element = ($AR)[$i];
        return element;
    """, i+1)
end
length(AR::ArrayRef) = icxx"$AR.size();"
done(AR::ArrayRef, i) = i >= length(AR)

Even though this may looks a bit complicated, all this is saying is that arrayrefs are indexed from one to AR.size(); and we can use the C++ bracket operator to access elements . With this defined, we get:

julia> collect(tokens(getMacroInfo(PP, "USBDEVFS_SETCONFIGURATION")))
9-element Array{Any,1}:
 identifier '_IOR'
 l_paren '('
 char_constant ''U''
 comma ','
 numeric_constant '5'
 comma ','
 unsigned 'unsigned'
 int 'int'
 r_paren ')'

We’re off to a great start.

Getting all the ioctls

As the previous section may have indicated, defining iteration over an object, is an enourmously powerful way to work with said object. Because everything in julia is generation, enabling iteration over an object, immidiately allows us to use any of the standard iteration tools (e.g. filters, maps, etc) to work with our objects.

With this, in mind, let’s see what we want to do. We know that ioctls are introduced by a macro that expands to _IO(...), _IOR(...), _IOW(...) or _IOWR(...). So let’s
define iteration over Clang’s identifier table and write down exactly that:

start(tab::rcpp"clang::IdentifierTable") = icxx"$tab.begin();"
next(tab::rcpp"clang::IdentifierTable", it) = (icxx"$it->second;", icxx"++$it;")
done(tab::rcpp"clang::IdentifierTable", it) = icxx"$it == $tab.end();"
length(tab::rcpp"clang::IdentifierTable") = icxx"$tab.size();"
# Get all identifier that are macros
macros = Iterators.filter(II->icxx"$II->hasMacroDefinition();", icxx"$PP->getIdentifierTable();")
# Expand into tuples of (II, tokens)
IItokens = map(II->(II, collect(tokens(getMacroInfo(PP, II)))), macros)
# Now filter down to the ones we're interested in
ioctl_defs = filter(IItokens) do x
      II, tokens = x
      isempty(tokens) && return false
      tok_is_identifier(tokens[1]) && String(tokens[1]) in ["_IO","_IOR","_IOW","_IOWR"]
  end;

And if all worked well, we end up with:

julia> map(x->(String(x[1]),x[2]), ioctl_defs)
34-element Array{Tuple{String,Array{Any,1}},1}:
 ("USBDEVFS_FREE_STREAMS", Any[identifier '_IOR', l_paren '(', char_constant ''U'', comma ',', numeric_constant '29', comma ',', struct 'struct', identifier 'usbdevfs_streams', r_paren ')'])
 ("USBDEVFS_BULK32", Any[identifier '_IOWR', l_paren '(', char_constant ''U'', comma ',', numeric_constant '2', comma ',', struct 'struct', identifier 'usbdevfs_bulktransfer32', r_paren ')'])
 ("USBDEVFS_DISCARDURB", Any[identifier '_IO', l_paren '(', char_constant ''U'', comma ',', numeric_constant '11', r_paren ')'])

Extracting the fields from the structures

Now, it’s fairly simple to do any any post-processing we want here, and what to do exactly will depend on our intended application, but I do want to highlight how to extract the fields. At first I attempted to simply use the second to last token as the type name, but that doesn’t work very well, because some types are multiple tokens (e.g. unsigned int) and some others are only defined via (sometimes complicated preprocessor rules). Instead, the right way to do this, is to simply feed those tokens back through the parser. We’ll use a couple of definitions

"Given an array of tokens, queue them up for parsing"
function EnterTokenStream(PP::pcpp"clang::Preprocessor", tokens::Vector{cxxt"clang::Token"})
  # Vector memory layout is incompatible, convert to clang::Token**
  toks = typeof(tokens[1].data)[x.data for x in tokens]
  icxx"$PP->EnterTokenStream(llvm::ArrayRef<clang::Token>{
    (clang::Token*)$(pointer(toks)),
    (size_t)$(length(toks))
  },false);"
end
"Advance the parse if it's currently at EOF. This happens in incremental parsing mode and should be called before parsing."
function AdvanceIfEof(P)
  icxx"""
  if ($P->getPreprocessor().isIncrementalProcessingEnabled() &&
    $P->getCurToken().is(clang::tok::eof))
      $P->ConsumeToken();
  """
end
"Parse a type name"
function ParseTypeName(P::pcpp"clang::Parser")
  AdvanceIfEof(P)
  res = icxx"$P->ParseTypeName(nullptr, clang::Declarator::TypeNameContext);"
  !icxx"$res.isUsable();" && error("Parsing failed")
  Cxx.QualType(icxx"clang::Sema::GetTypeFromParser($res.get());")
end
"Parse a constant expression"
function ParseConstantExpression(P::pcpp"clang::Parser")
  AdvanceIfEof(P)
  res = icxx"$P->ParseConstantExpression();"
  !icxx"$res.isUsable();" && error("Parsing failed")
  e = icxx"$res.get();"
  e
end
"Convert a parsed constant literal to a julia Char (asserts on failure)"
function CharFromConstExpr(e)
  Char(icxx"""
    clang::cast<clang::CharacterLiteral>($e)->getValue();
  """)
end
tok_is_comma(Tok) = icxx"$Tok.is(clang::tok::comma);"
tok_is_numeric(Tok) = icxx"$Tok.is(clang::tok::numeric_constant);"

With these definitions:

julia> ioctl_tokens = first(ioctl_defs)[2]
9-element Array{Any,1}:
 identifier '_IOR'
 l_paren '('
 char_constant ''U''
 comma ','
 numeric_constant '29'
 comma ','
 struct 'struct'
 identifier 'usbdevfs_streams'
 r_paren ')'

julia> typename_tokens = Vector{cxxt"clang::Token"}(ioctl_tokens[findlast(tok_is_comma, ioctl_tokens)+1:end-1])
2-element Array{Cxx.CppValue{Cxx.CxxQualType{Cxx.CppBaseType{Symbol("clang::Token")},(false, false, false)},N} where N,1}:
 struct 'struct'
 identifier 'usbdevfs_streams'

julia> EnterTokenStream(PP, typename_tokens); QT = Cxx.desugar(ParseTypeName(P))
Cxx.QualType(Ptr{Void} @0x000055ff24a13960)

C++ > ((clang::RecordType*)&*$QT)->getDecl()->dump()
RecordDecl 0x55951860c240 </usr/lib/gcc/x86_64-linux-gnu/6.2.0/../../../../include/linux/usbdevice_fs.h:153:1, line:157:1> line:153:8 struct usbdevfs_streams definition
|-FieldDecl 0x55951860c300 <line:154:2, col:15> col:15 num_streams 'unsigned int'
|-FieldDecl 0x55951860c358 <line:155:2, col:15> col:15 num_eps 'unsigned int'
`-FieldDecl 0x55951860c418 <line:156:2, col:21> col:16 eps 'unsigned char [0]'

We could process these for exmaple as such. Here I’ll be using a different approach, where instead of using julia to do the iteration, I’ll just write most of the function in C++ and only call back into julia once at the end:

# C structure to julia array of fields
function inspectStruct(CC, S)
    CC = Cxx.instance(CC)
    ASTCtx = icxx"&$(CC.CI)->getASTContext();"
    fields = Any[]
    icxx"""
    auto &ARL = $ASTCtx->getASTRecordLayout($S);
    for (auto field : ($S)->fields()) {
      unsigned i = field->getFieldIndex();
      // Skip these for now
      if (field->isImplicit())
        continue;
      if (field->getType()->isUnionType())
        continue;
      if (field->getType()->isArrayType())
        continue;
      if (field->getType()->isRecordType() ||
          field->getType()->isEnumeralType())
        continue;
      if (field->getType()->isPointerType() &&
          field->getType()->getPointeeOrArrayElementType()->isRecordType())
        continue;
      $:(begin
        QT = Cxx.QualType(icxx"return field->getType();")
        push!(fields, (
          String(icxx"return field;"),
          Cxx.juliatype(QT),
          icxx"return $ASTCtx->toCharUnitsFromBits(ARL.getFieldOffset(i)).getQuantity();"
        ))
      nothing
    end);
    }
    """
    fields
end
julia> inspectStruct(CCompiler, icxx"((clang::RecordType*)&*$QT)->getDecl();")
2-element Array{Any,1}:
 ("num_streams", UInt32, 0)
 ("num_eps", UInt32, 4)

Conclusion

With the above code, we can easily extract and work with the definitions of ioctls in the linux headers. I hope this blog post has given you an idea of both how to use the Clang C++ API to do some C introspection, as well as some idea, of how to use some of the generic programming features in julia. The above is a pretty decent summary some of the first things I do when working with new data sources in julia:
1. Defining printing method for the relevant types
2. Define iteration on any container data structures
3. Use julia’s iteration tools to write whatever query I’m interested in
Following this strategy usually gets one pretty far. In this case, it was essentially sufficient to solve our problem and provide a useful list of ioctls and the fields of their arguments to use in our ioctl dumping tool.

]]>
3486