Code Refactoring Using Metaprogramming

By: Randy Zwitch

Re-posted from: http://randyzwitch.com/julia-metaprogramming-refactoring/

It’s been nearly a year since I wrote Twitter.jl, back when I seemingly had MUCH more free time. In these past 10 months, I’ve used Julia quite a bit to develop other packages, and I try to use it at work when I know I’m not going to be collaborating with others (since my colleagues don’t know Julia, not because it’s bad for collaboration!).

One of the things that’s obvious from my earlier Julia code is that I didn’t understand how powerful metaprogramming can be, so here’s a simple example where I can replace 50 lines of Julia code with 10.

CTRL-A, CTRL-C, CTRL-P. Repeat.

Admittedly, when I started on the Twitter package, I fully meant to go back and clean up the codebase, but moved onto something more fun instead. The Twitter package started out as a means of learning how to use the Requests.jl library to make API calls, figured out the OAuth syntax I needed (which itself should be factored out of Twitter.jl), then copied-and-pasted the same basic function structure over and over. While fast, what I was left with was this (currently, the help.jl file in the Twitter package):It’s pretty clear that this is the same exact code pattern, right down to the spacing! The way to interpret this code is that for these five Twitter API methods, there are no required inputs. Optionally, there is the ‘options’ keyword that allows for specifying a Dict() of options. For these five functions, there are no options you can pass to the Twitter API, so even this keyword is redundant. These are simple functions so I don’t gain a lot by way of maintainability by using metaprogramming, but at the same time, one of the core tenets of programming is ‘Dont Repeat Yourself’, so let’s clean this up.

For :symbol in symbolslist…

In order to clean this up, we need to take out the unique parts of the function, then pass them as arguments to the @eval macro as follows:
What’s happening in this code is that I define two tuples: one of function names (as symbols, denoted by ‘:’ ) and one of the API endpoints. We can then iterate over the two tuples, substituting the function names and endpoints into the code. When the package is loaded, this code evaluates, defining the five functions for use in the Twitter package.

Wha?

Yeah, so metaprogramming can be simple, can it can also be mind-bending. It’s one thing to not repeat yourself, it’s another to write something so complex that even YOU can’t remember how the code works. But somewhere in between lies a sweet spot where you can re-factor whole swaths of code and streamline your codebase.

Metaprogramming is used throughout the Julia codebase, so if you’re interested in seeing more examples of metaprogramming, check out the Julia source code, the Requests.jl package (where I first saw this) or really anyone who actually knows what they are doing. I’m just a metaprogramming pretender at this point :)

 

To read additional discussion around this specific example, see the Julia-Users discussion at:

https://groups.google.com/forum/#!topic/julia-users/zvJmqB2N0GQ

Fernández-Villaverde to Speak at Becker Friedman Institute

By: Bradley Setzler

Re-posted from: http://juliaeconomics.com/2014/11/17/fernandez-villaverde-to-speak-at-becker-friedman-institute/

To those readers affiliated with the University of Chicago:

The economist Jesús Fernández-Villaverde, University of Pennsylvania, a leader in the field of computational economics and co-author of the paper that motivated my switch to Julia, is speaking at the Computational Economics Colloquium, Becker Friedman Institute, on Thursday at 5pm.

The title of his talk is Massively Parallel Programming for Economists: Challenges and Opportunities. (My tutorial on parallel programming in Julia is available here.)

Hope to see you there.


Bradley J. Setzler

This week in Julia: October 25

By: This week in julia

Re-posted from: http://thisweekinjulia.github.io/julia/2014/10/25/October-25.html

The maintenence release Julia 0.3.2 was posted on Wednesday. A total of 70 commits were meticulously backported from the master development branch, bringing lots of bugfixes, documentation updates, and even a few performance improvements to the 0.3 release.

Major breaking changes on 0.4

The function call operator is now overloadable (#8712)! This has lots of wide-ranging implications… some of which I’m sure haven’t even been realized yet. Here’s what it already means:

  • It makes type constructors first class methods of the call function. You can still create constructor methods with the old function-like T(x,y,z) = ...; syntax, or you can add methods to call directly (e.g., call(::Type{A},x,y,z) = ...;). While I’m not familiar with the internals, it apparently simplifies a lot of the code surrounding type constructor generation. This “big new feature” resulted in a net decrease of 10 lines of code. Impressive!
  • There’s now a fallback method (by default) for all types such that calling T(args...) means convert(T, args...)::T (39dfd92). This means that you can now call Int(2.0) with the leading capital!
  • This is paving the way towards possibly deprecating the lowercase int functions (#1470: int, Int and box). One question is what to do with its elementwise methods (e.g., int([1.0,2.0,3.0]) to convert an Array{Float64} to an Array{Int}), but with one small optimization (2cb98a0), map(Int,[1.0,2.0,3.0]) is now just as fast as the old lowercase method. This is an interesting case; typically map is slower than explicit for loops since it cannot specialize on the function value, but when called with a type it can emit specialized code for that constructor.
  • This same trick is exploited in lots of the map/reduce code, using dummy Functor types to represent specific functions to allow method specialization and inlining. But with call overloading, those ‘dummy’ Functor types just got a lot smarter. Instead of using a secondary helper function to actually evaluate each function, you can overload the call operator for each Functor directly so the type behaves just like its desired function. It makes the Base reduction code a little bit simpler and much nicer to read. (#8790)
  • It also obviates the need for some functions like inf and nan, which are now deprecated. Previously, you would call inf(T) to get the representation of infinity in type T. With this change, however, you can simply type the reverse: T(Inf). (#8776)

I’m sure lots of this is still in flux, but this turned out to be a bigger and more exciting change than I expected. It has also broken lots of packages (although that may just be the result of a few dependencies breaking and cascading through the ecosystem), so in the short term beware.

Other 0.4 project updates

  • Work to allow static compilation and caching of modules (to reduce package load time dramatically) continues. A big backend change was merged (static compile part 3, #8656) in preparation for work on the user interace (as proposed part 4, #8745). There are still some questions about the underlying behaviors, but it’s exciting to see precompilation moving forward.
  • The documentation system also keeps marching along, now in a new PR (#8791).

Package ecosystem

  • A few weeks ago, Tim Holy proposed an alternative to multiple inheritance that has since become known as the “Tim Holy Traits Trick” (THTT) (comment on #2345). In short, it uses functions to dispatch on traits instead of using an inheritance tree to dispatch on ancestors. At its core, though, it allows for multiple interfaces. Mauro Werder took this idea and ran, putting together the Traits.jl package which uses macros to formalize interface definition, implementation, and dispatch. It’s not officially registered, but I think it’s all rather clever and worth a mention.