Differentiable Programming (Keno Fischer Quora Session)

Keno Fischer, Julia Computing Co-Founder and Chief Technology Officer (Tools) participated in a Quora Session March 18-23.

What are the biggest software challenges in machine learning and differentiable programming systems?

Machine learning has revived a significant amount of interest in techniques that were popular in the 70s and 80s but have not received mainstream attention since outside of academia or certain fairly niche use cases. Among these techniques are ideas like polyhedral compilation, optimizing array compilers and automatic differentiation (reverse mode AD in particular) and a few others. Additionally, the hardware folks are reviving ideas that have been out of vogue for a while: systolic arrays, software scheduled architectures (with software-managed hazards), innovation in ISA design and interconnects. Writing good compilers for these architectures is quite hard and not really a solved problem yet.

The reason a lot of these ideas failed last time around was that it turned out there was very little advantage in exploring these exotic ideas, because next year’s general purpose CPU would just be faster than any special purpose hardware you could possibly build. Physical limitations are preventing CPUs from getting faster. I’m writing this answer on a six year old machine and the most powerful single machine I currently have SSH access to is about a decade old – both of these would have been mostly unthinkable even a decade ago. So we need to find innovation elsewhere.

The good news is that (if done properly) all this innovation being driven by machine learning will have significant benefits in other fields. Pervasive availability of production quality automatic differentiation systems (“differentiable programming systems”) will have impact far beyond machine learning. Wherever any sort of optimization process happens, being able to very quickly compute the derivative of your objective function is a crucial prerequisite to getting a good result. This shows up everywhere: finance, astrophysics, medical imaging, personalized medicine, logistics and many others. The same is true for some of the other techniques. Our thesis for building the machine learning stack in Julia is to build it as a set of general infrastructure (AD, compiler support, hardware backends, developer tools, etc.) and have machine learning just fall out as a special case of what’s well supported. That sometimes causes some friction because current-generation machine learning systems tend to not require the full generality that this infrastructure provides, but I think it’s a crucial ingredient for next generation systems [1].

I would be remiss at this point not to point out some of the other work that’s going on in this area. There’s some great work out of Google with Swift for TensorFlow (a very different approach from what we’re doing, but taking differentiable programming the furthest of any of the non-Julia systems I have seen) and MLIR (which despite its name and much to my enthusiastic support is trying to build general purpose next-generation compiler infrastructure), TVM from UW (doing some great work on ML-driven compiler heuristics and search space exploration for generating really high performance kernels on all kinds of architectures). I also liked the goals for Myia when it was announced, but I haven’t heard much recently.

[1] I recently gave a talk on this (https://juliacomputing.com/blog/2019/02/19/growing-a-compiler.html). It’s a good overview of the various things we’re doing at the compiler level to support machine learning and differentiable programming. There’s also a larger point on how all of these things have to work together that I sometimes think gets lost.

What type of support for differentiable programming is available in Julia?

Julia as a language tends to make it very easy to write automatic differentiation packages. If you want forward mode AD, JuliaDiff/ForwardDiff.jl is the way to go. For reverse mode AD, the landscape is a lot larger. The packages that I can think of off the top of my head are:

and probably a couple of others that I’m forgetting. However, most of the cutting edge work is going in Zygote (FluxML/Zygote.jl), which is our next-generation, compiler-based automatic differentiation framework. It’s not quite ready for production use yet, but it’s already quite clear that it will be a significant usability improvement over the previous tracing based AD implementations. In particular:

  1. We’ll be able to differentiate through many more AD-unaware codes (all the tracing based implementation requires code to be written fairly generically, which is generally a good property, but not always easy).

  2. We no longer have the distinction between tracked arrays and arrays of tracked numbers (as well as the associated performance improvements).

  3. The compiler will be able to introspect the AD data structures. This one in particular is quite cool. A lot of the literature on “tape optimizations” comes down to just applying standard compiler techniques jointly on the forward and backward pass.

That said, we’re still actively working on trying to figure out the correct support that the compiler needs to provide to make this possible. We’re generally against privileging any particular approach to solving a problem (so we wouldn’t just want to put AD into the compiler directly). The reason for this is simple. There are always choices to be made (as the long list of reverse mode packages shows), so if what we’ve built doesn’t quite work for your use case, we’d like to figure out why, but we don’t want to prevent you from building a specialized implementation that does what you need in the meantime.

Our approach instead is try to figure out the correct reusable abstractions that allow us to implement the thing we want, but have very clear, simple semantics and hopefully will be useful for use cases we haven’t even thought about yet. For example, right now Zygote is built on top of generated functions, a feature we added to support operations on higher dimensional arrays as well as to be able to embed a C++ compiler in Julia. But by coming up with a generalized abstraction, lots of cool things were built that we didn’t even think of when we implemented generated functions. Tools like Zygote are outgrowing generated functions a bit, so we need to come up with something slightly different (e.g. Very WIP: RFC: Add an alternative implementation of closures by Keno · Pull Request #31253 · JuliaLang/julia may be part of the answer), but we’ve been able to do quite a bit of iteration to make sure we get the interface and features right, just with what we already have. Now it’s just a question of adding a little bit of language support to make it amazing.

Machine Learning (Keno Fischer Quora Session)

Keno Fischer, Julia Computing Co-Founder and Chief Technology Officer (Tools) participated in a Quora Session March 18-23.

What will machine learning look like 15-20 years from now?

Machine learning is a very rapidly moving field, so it’s hard to make predictions about the state of the art 6 months from now, let alone 15 – 20 years. I can however, offer a series of educated guesses based on what I see happening right now.

  1. We are still far away from AGI. Current generation machine learning systems are still very far away from something that could legitimately be called artificial “intelligence”. The systems we have right now are phenomenal at pattern recognition from lots of data (even reinforcement learning systems are mostly about memorizing and recognizing patterns that worked well during training). This is certainly a necessary step, but it is very far away from an intelligent system. In analogy to human cognition, what we have now is analogous to the subconscious processes that allow split second activation of your sympathetic nervous system when your peripheral vision detects a predator approaching or a former significant other turning around a corner – in other words, pattern-based, semi-automatic decisions that our brain does “in hardware”. We don’t currently have anything I can see that would resemble intentional thought and I’m not convinced we’ll get to it from current generation systems.

  2. Traditional programming is not going away. I sometimes hear the claim that most traditional programming will be replaced by machine learning systems. I am highly skeptical of this claim, partly as a corollary to the previous point, but more generally because machine learning isn’t required for the vast majority of tasks. Machine learning excels where dealing with the messiness of the real world, lots of data is available and no reasonably fundamental model is available. If any of these are not true, you generally don’t have to settle for a machine learning model – the traditional alternatives are superior. A fun way to think about it is that humans are the most advanced natural general intelligence around, but yet we invented computers to do certain specialized tasks we’re too slow at. Why should we expect artificial intelligence to be any different?

  3. Machine learning will augment most traditional tasks. That being said, any time one of these traditional systems needs to interact with a human, there is an opportunity for machine-learning based augmentation. For example, a programmer looking at error messages could use a machine learning system that looks at the error messages and suggests a course of action. Computers are a lot more patient than humans. You can’t make a human watch millions of hours of programming sessions to remember the common resolutions to problems, but you can make a machine learning system do the same. In addition, machine learning systems can learn from massive amounts of data very rapidly and on a global scale. If any instance of the machine learning system has ever encountered a particular set of circumstances, it can almost instantly share that knowledge globally, without being limited to the speed of human communication. I don’t think we have yet begun to appreciate the impact of this, but we’ll probably see it in another 5-10 years.

  4. We’ll get better at learning from small or noisy data. At the moment, machine learning systems mostly require large amounts of relatively clean, curated, data sets. There are various promising approaches to relax this requirement in one or the other direction (or combinations of small, well curated data sets to get started with a larger corpus of more noisy data). I expect these to be perfected in the near future.

  5. We’ll see ML systems be combined with traditional approaches. At the moment it seems fairly common to see ML systems that are entirely made of neural networks trying to solve end to end problems. This often works ok, because you can recover many traditional signal processing techniques from these building blocks (e.g. Fourier transforms, edge detection, segmentation, etc.), but the learned versions of these transforms can be significantly more computationally expensive than the underlying approach. I wouldn’t be surprised to see these primitives making a bit of a comeback (as part of a neural network architecture). Similarly, I’m very excited about physics-based ML approaches where you combine a neural network with knowledge of an underlying physical model (e.g. the differential equations governing a certain process) in order to outperform both pure ML approaches and approaches relying solely on the physical process (i.e. simulations).