By: oxinabox.github.io

Re-posted from: https://www.oxinabox.net/2021/02/08/AD-truncation-error.html

Griewank and Walther’s 0th Rule of algorithmic differentiation (AD) states:

Algorithmic differentiation does not incur truncation error.

(2008, “Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation”, Andreas Griewank and Andrea Walther.)

In this blog post I will show you a case that looks like it does in fact incur truncation error.
Though this case will arguably be a misinterpretation of that rule.
This blog post will thus highlight why careful interpretation of the rule is necessary.
Further it will motivate why we need to often add more custom sensitivity rules (custom primitives) to our AD systems, even though you can AD anything with just a few basic rules.

Credit to Mike Innes who pointed this out to me at JuliaCon 2018.
Continue reading →