Author Archives: Picaud Vincent

Julia, custom serialization with JSON.jl

By: Picaud Vincent

Re-posted from: https://pixorblog.wordpress.com/2026/05/05/julia-custom-serialization-with-json-jl/

Introduction

The GitHub:JSON3.jl package has been deprecated. That bothered me a little because I had to migrate a lot of my code to use GitHub:JSON.jl. Luckily, the migration turned out to be easier than I expected.

My use case is a bit special: I have to serialize my structures with type information so that I can retrieve the exact types after deserialization.

I know about GitHub:BSON.jl (see also Wiki:BSON) and Julia:Serialization, but I didn’t want to use them because they produce binary files. I wanted to keep a human‑readable format.

In this note I give a minimal working example that might save you some time.

Code

We’ll need the JSON.jl package. We also use StaticArrays.jl to show how to preserve the right vector type when deserializing an AbstractVector.

using JSON
using StaticArrays 

Let’s imagine we have an abstract type Abstract_Foo and two concrete types: Foo_A and Foo_B.

abstract type Abstract_Foo end

@nonstruct struct Foo_A{V <: AbstractVector}  <: Abstract_Foo
    v::V
    x::Float64
end

@nonstruct struct Foo_B <: Abstract_Foo
    v::AbstractVector
    n::Int
end 

Nothing special here, except the @nonstruct macro. That macro comes from GitHub:StructUtils.jl, a package used by JSON.jl to automate common struct operations (construction, etc.).

Using Doc:@nonstruct in front of a struct definition marks it as “special”. You tell JSON.jl to treat it as a primitive type that should be converted directly using lift() and lower() methods, rather than constructing it from field values. In short, you have to do all the work by hand, but you also get all the freedom to serialize and deserialize the structure however you want.

Serialization

During serialization the lower() method is called. We save the field values but also any type information needed for deserialization. Personally, I store this information in a field called type that holds the type of the structure. The name type isn’t special, you could call it internal_type, but I think it’s good practice to adopt a convention and stick to it.

function StructUtils.lower(to_serialize::Foo_A)

    return (type = string(typeof(to_serialize)),
            v = to_serialize.v,
            x = to_serialize.x)
end

For Foo_B, it’s a bit more complicated because the v field is an AbstractVector type, so we need an extra field to save the type information:

function StructUtils.lower(to_serialize::Foo_B)

    return (type = string(typeof(to_serialize)),
            v_type = string(typeof(to_serialize.v)),
            v = to_serialize.v,
            n = to_serialize.n)
end

Demonstration

Here’s a demonstration of serialization:

a = Foo_A(@SVector(Int[1,2]),1.2)

a_json_str = JSON.json(a, pretty=true)
{
  "type": "Foo_A{SVector{2, Int64}}",
  "v": [
    1,
    2
  ],
  "x": 1.2
}

Now for Foo_B

b = Foo_B(Float16[3,4],34)

b_json_str = JSON.json(b, pretty=true)
{
  "type": "Foo_B",
  "v_type": "Vector{Float16}",
  "v": [
    3.0,
    4.0
  ],
  "n": 34
}

Deserialization

To deserialize you have to define the lift() methods.

First, we intercept all Abstract_Foo occurrences and extract the concrete type. Right now the type is a String, to turn it into a Julia DataType we use Base.eval() and Meta.parse(). Once we have that instantiated type, we continue deserialization with it.

function StructUtils.lift(type::Type{<:Abstract_Foo},
                          to_deserialize)

    actual_type = Base.eval(Main,Meta.parse(to_deserialize.type))
    StructUtils.lift(actual_type,to_deserialize)
end

Now we redefine lift() for the specific concrete types. You have to be careful to define these new methods for all possible specializations, otherwise you’ll get an infinite recursion with the previous function. It would be nice to detect this situation, but how? (feel free to add a comment 🙂 )

For Foo_A:

function StructUtils.lift(type::Type{<:Foo_A{V}},
                          to_deserialize) where {V<:AbstractVector}

    v = StructUtils.lift(V,to_deserialize.v) # deserialize vect.
    x = to_deserialize.x

    type(v,x)
end 

For Foo_B:

function StructUtils.lift(type::Type{<:Foo_B},
                          to_deserialize)

    v_type = Base.eval(Main,Meta.parse(to_deserialize.v_type))
    v = StructUtils.lift(v_type,to_deserialize.v) # deserialize vect.
    n = to_deserialize.n

    type(v,n)
end 

Demonstration

Notice that we don’t need to give the exact type, just Abstract_Foo is enough.

JSON.parse(a_json_str,Abstract_Foo)
Foo_A{SVector{2, Int64}}([1, 2], 1.2)
JSON.parse(b_json_str,Abstract_Foo)
Foo_B(Float16[3.0, 4.0], 34)

Remarks

@kwdef and @nonstruct together

You cannot use @kwdef and @nonstruct together. The following code generates an error:

@nonstruct @kwdef struct Foo_C <: Abstract_Foo
end

The solution is to do the work of @nonstruct by hand. First, look at what this macro does:

@macroexpand @nonstruct  struct Foo_C <: Abstract_Foo
end
quote
    begin
        $(Expr(:meta, :doc))
        struct Foo_C <: Abstract_Foo
        end
    end
    StructUtils.structlike(::StructUtils.StructStyle, ::Type{<:Foo_C}) = false
end

So the fix is simply to replace

@nonstruct @kwdef struct Foo_C <: Abstract_Foo
end

by

@kwdef struct Foo_C <: Abstract_Foo
end

StructUtils.structlike(::StructUtils.StructStyle,
                       ::Type{<:Foo_C}) = false

Writing / reading file

Please follow the JSON.jl official doc, nothing special here:

JSON.json(file, a, pretty=true)      # write file
JSON.parsefile(file, Abstract_Foo)   # read file

Complete code

To make your life easier, here’s the complete code:

using JSON
using StaticArrays

abstract type Abstract_Foo end

@nonstruct struct Foo_A{V <: AbstractVector}  <: Abstract_Foo
    v::V
    x::Float64
end

@nonstruct struct Foo_B <: Abstract_Foo
    v::AbstractVector
    n::Int
end

function StructUtils.lower(to_serialize::Foo_A)

    return (type = string(typeof(to_serialize)),
            v = to_serialize.v,
            x = to_serialize.x)
end

function StructUtils.lower(to_serialize::Foo_B)

    return (type = string(typeof(to_serialize)),
            v_type = string(typeof(to_serialize.v)),
            v = to_serialize.v,
            n = to_serialize.n)
end

a = Foo_A(@SVector(Int[1,2]),1.2)

a_json_str = JSON.json(a, pretty=true)

println(a_json_str)

b = Foo_B(Float16[3,4],34)

b_json_str = JSON.json(b, pretty=true)

println(b_json_str)

function StructUtils.lift(type::Type{<:Abstract_Foo},
                          to_deserialize)

    actual_type = Base.eval(Main,Meta.parse(to_deserialize.type))
    StructUtils.lift(actual_type,to_deserialize)
end

function StructUtils.lift(type::Type{<:Foo_A{V}},
                          to_deserialize) where {V<:AbstractVector}

    v = StructUtils.lift(V,to_deserialize.v) # deserialize vect.
    x = to_deserialize.x

    type(v,x)
end

function StructUtils.lift(type::Type{<:Foo_B},
                          to_deserialize)

    v_type = Base.eval(Main,Meta.parse(to_deserialize.v_type))
    v = StructUtils.lift(v_type,to_deserialize.v) # deserialize vect.
    n = to_deserialize.n

    type(v,n)
end

JSON.parse(a_json_str,Abstract_Foo)

JSON.parse(b_json_str,Abstract_Foo)

Conclusion

There’s nothing more ridiculous than a conclusion, because nothing is ever finished. But I admit it’s still handy to say goodbye 🙂

Generating Julia doc into Org-Mode documents

By: Picaud Vincent

Re-posted from: https://pixorblog.wordpress.com/2018/04/29/generating-julia-doc-into-org-mode-documents/

1 Context

This post presents J4Org.jl a Julia package I have started to develop to include Julia doc into Org-Mode documents. My goal was to be able to code and document Julia packages without leaving Emacs and to reduce as much as possible the burden of documentation.

2 Julia code documentation

Here is a short example. Imagine that your package is:

module Foo

export Point, foo
    
import Base: norm

#+Point L:Point_struct
# This is my Point structure
#
# *Example:*
#
# Creates a point $p$ of coordinates $(x=1,y=2)$.
#
# #+BEGIN_SRC julia :eval never :exports code
# p=Point(1,2)
# #+END_SRC
#
# You can add any valid Org-Mode directive. If you want to use
# in-documentation link, use [[norm_link_example][]]
#
struct Point
    x::Float64
    y::Float64
end

#+Point
# Creates Point at origin $(0,0)$ 
Point() = Point(0,0)

#+Point,Method L:norm_link_example
# A simple function that computes $\sqrt{x^2+y^2}$
#
# *Example:*
#!p=Point(1.0,2.0);
#!norm(p) 
#
# See: [[Point_struct][]]
#
norm(p::Point)::Float64 = sqrt(p.x*p.x+p.y*p.y)

#+Method,Internal
# An internal function
#
# For symbol that are not exported, do not forget the "Foo." prefix:
#!p=Point(1.0,2.0)
#!Foo.foo(2.0,p)
foo(r::Float64,p::Point) = Point(r*p.x,r*p.y)

end

The documentation template is very simple. Before each item you want to document add these comment lines:

#+Tag1,Tag2,... L:an_extra_link_if_required 
#
# Here you can put any Org mode text, for instance $sin(x)$
#
#!sin(5) # julia code to be executed
#
# [[internal_link][]]
struct A_Documented_Struct 
...
end 
  • #+Tag1,Tag2,… is mandatory, “#+” is followed by a list of tags. Later when you want to extract doc you can do filtering according these tags.
  • L:an_extra_link_if_required is not mandatory. It defines a reference if you want to create doc links. The previous statement defines a link target named an_extra_link_if_required.
  • [[internal_link][]] creates a link to a previously defined L:internal_link.
  • !sin(5) will execute Julia code and include the output in the doc.

Also note that you can keep compatibility with docstring as follows:

""" 
    foo() 

foo function...
"""
#+Tag
#
# foo function
foo() = ...

3 Org-Mode side

You need Org-Mode plus ob-julia.el to be installed. For illustration we use this minimal Org-Mode document:

#+PROPERTY: header-args:julia :session *my_session* :exports code :eval no-export
#+OPTIONS: ^:{}
#+TITLE: Getting Started with a minimal example

#+BEGIN_SRC julia :results output none :eval no-export :exports none
using J4Org 
initialize_boxing_module(usedModules=["Foo"]) 
documented_items=create_documented_item_array("Foo.jl")
#+END_SRC

* Example

Prints all documented items, except those tagged with "Internal" 
#+BEGIN_SRC julia :results output drawer :eval no-export :exports results
print_org_doc(documented_items,tag_to_ignore=["Internal"],header_level=0)
#+END_SRC
  • using J4Org uses this package
  • initialize_boxing_module(usedModules=[“Foo”]) defines what are the modules to use when executing Julia code extracted from the doc (the “#!” statements). Here we are documenting the Foo module, hence we must use it.
  • create_documented_item_array(“Foo.jl”) creates the list of documented items from file “Foo.jl”. You can use a list of files or a directory.
  • print_org_doc(documented_items,tag_to_ignore=[“Internal”],header_level=0) prints all documented items, except those tagged with “Internal”.

4 Result after html-export

When html-exported with Org-Mode this will generate this document:

Index: [P] Point [n] norm

  • Point
struct Point

This is my Point structure

Example:

Creates a point p of coordinates (x=1,y=2).

p=Point(1,2)

You can add any valid Org mode directive. If you want to use in-documentation link, use norm(…)

Foo.jl:8, back to index

Point()

Creates Point at origin (0,0)

Foo.jl:27, back to index

  • norm
norm(p::Point)::Float64

A simple function that computes \sqrt{x^2+y^2}

Example:

p=Point(1.0,2.0);
norm(p) 
2.23606797749979

See: struct Point

Foo.jl:31, back to index

5 Further information

You can visit J4Org.jl for further details. You can even install it (Pkg.add("J4Org")). I have used it to document DirectConvolution.jl (this Direct Convolution Package html page for instance).

Disclaimer: this package is still in early development.

Julia with Emacs Org mode

By: Picaud Vincent

Re-posted from: https://pixorblog.wordpress.com/2018/03/07/julia-with-emacs-org-mode/

Introduction

This post details how to use Emacs Org mode to create Julia notebooks and to perform HTML or PDF exports. I tried to get the simplest working solution.

Julia notebook functionality works out of the box thanks to ob-julia.el and this is what I am using instead of Jupiter notebooks. However, the solution to export HTML and PDF is not straightforward.

I wanted to:

  • have nice Julia code snippets with full UTF8 supports,
  • being able to export in both HTML and PDF, including bibliography.

I get a solution which is certainly not perfect, ideas to improve it are welcomed.

There are two points to take care of:

  • LaTeX does not fully support UTF8, nor its listings package.
    • for UTF8 support I had to switch to luatex, biber and minted package
    • I also had to use proper fonts, DejaVu, to support Greek letters and mathematical symbols.
  • to make the bibliography exportable in both HTML and PDF without .org file modification, I had to use a little trick.

The proposed solution uses:

  • ob-julia.el : to support notebook functionality,
  • ox-bibtex.el : used for html-export of the bibliography, requires bibtex2html (Debian package),
  • luatex : under Debian, included in the texlive-latex-base package,
  • biber : under Debian, the biber package,
  • pygments : under Debian, the python-pygments package. Attention with python3-pygments, it does not on my computer. I have not investigated this.

Maybe I have forgotten something, just tell me (I am only using Linux).

There is a GitHub repository to reproduce results of this post. The example.pdf generated file is also present.

Emacs configuration

Getting ob-julia.el and ox-bibtex.el

You can found ob-julia.el and ox-bibtex.el in Org-mode Contributed Packages. Easy download can be performed using:

curl -o emacs_files/ob-julia.el https://code.orgmode.org/bzg/org-mode/raw/master/contrib/lisp/ob-julia.el
curl -o emacs_files/ox-bibtex.el https://code.orgmode.org/bzg/org-mode/raw/master/contrib/lisp/ox-bibtex.el

Minimal init.el file

This is a minimal configuration to reproduce the results. The code with its comments is self-explaining:

;; Use your own packages for classical stuff
(package-initialize)
;; requires Emacs speaks statistics, Org
(require 'ess-site)
(require 'org)

;; removes ugly horizontal lines in html-exported code 
;; (not mandatory)
(setq org-html-keep-old-src t)

;; As ob-julia.el and ox-bibtex are less common, 
;; we use a local repository.
;;
;; Usage: emacs -q --load emacs_files/init.el
;;
;; In a more usual setting one should use:
;; (require 'ob-julia.el)
;; (require 'ox-bibtex)
(load-file "emacs_files/ob-julia.el") ; works with ess-site, our notebook engine
(load-file "emacs_files/ox-bibtex.el"); used for bibliography HTML-export 

;; allows julia src block (requires ob-julia.el)
(setq org-confirm-babel-evaluate nil)

(org-babel-do-load-languages
 'org-babel-load-languages
 '((julia . t)))

;; defines image width in the OrgMode buffer (this is not for html
;; exports, for this you must use #+HTML_ATTR: :width 900px for
;; instance)
;;
;; This is not mandatory, but useful when one uses the gr() Plots.jl
;; backend as it exports wide .png files. CAVEAT: use imagemagick for
;; image resizing.
;;
(setq org-image-actual-width (/ (display-pixel-width) 4))

;; uses the minted package instead of the listings one
(setq org-latex-listings 'minted)

;; defines how to generate the pdf file using lualatex + biber
(setq org-latex-pdf-process
      '("lualatex -shell-escape -interaction nonstopmode -output-directory %o %f"
        "biber %b"
        "lualatex -shell-escape -interaction nonstopmode -output-directory %o %f"
        "lualatex -shell-escape -interaction nonstopmode -output-directory %o %f"))

.org file configuration

For demonstration purpose we define an .org file example. This file is kept very simple to do not distract from the required configuration part.

LaTeX directives

We have the LaTeX configuration part:

# uses minted package instead of listings 
#+LATEX_HEADER: \usepackage{minted}    

# uses fonts to support Greek letters etc...
#+LATEX_HEADER: \usepackage{fontspec}
#+LATEX_HEADER: \setmonofont{DejaVu Sans Mono}[Scale=MatchLowercase]

# defines the \begin{comment} \end{comment} environment, used to avoid
# conflict between bibtex and biblatex
#+LATEX_HEADER: \usepackage{verbatim} 

# uses the biblatex package (and not the old bibtex) 
#+LATEX_HEADER: \usepackage[backend=biber, bibencoding=utf8 ]{biblatex}
# our bibliography file
#+LATEX_HEADER: \addbibresource{my-bib.bib}

We then define our the Julia code highlight style. This style is used by minted for PDF export.

#+BEGIN_EXPORT latex
\definecolor{bg}{rgb}{0.95,0.95,0.95}
\setminted[julia]{
  bgcolor=bg,
  breaklines=true,
  mathescape,
  fontsize=\footnotesize}
#+END_EXPORT

Our notebook

Now this is the beginning of our notebook. One can use Org as usual…

#+TITLE: My title
#+AUTHOR: author

* Very simple demo

#+BEGIN_SRC julia  :eval no-export :session *demo_session* :exports none
using Plots
#+END_SRC 

** UTF8 support + escape math equation
Note that UTF8 is supported (the \alpha variable) :

#+BEGIN_SRC julia :eval no-export :session *demo_session* :exports both :results silent :wrap "SRC julia :eval never"
# Generate a matrix $a_{i,j}=\mathcal{U}([0,1[)$
α=rand(4,5)
#+END_SRC

** Long lines are wrapped

#+BEGIN_SRC julia :eval no-export :session *demo_session* :exports both :results output :wrap "SRC julia :eval never"
function ⊗(a::AbstractArray{T},b::AbstractArray{S}) where {T<:Number,S<:Number} kron(a,b) end;

β=rand(2,5);
γ = α ⊗ β
#+END_SRC

** Plot example

You can easily generate plots, one example from [[http://docs.juliaplots.org/latest/examples/pyplot/][Plots Julia package]],
 is used to generate Figure [[PolarPlot]].

#+BEGIN_SRC julia  :eval no-export :session *demo_session* :exports code :results silent
Θ = linspace(0,1.5π,100)
r = abs(0.1 * randn(100) + sin.(3Θ))
plot(Θ,r,proj=:polar,m=2)
#+END_SRC

#+BEGIN_SRC julia  :eval no-export :session *demo_session* :results graphics :file example.png :exports results
savefig("example.png")
#+END_SRC

#+CAPTION: A polar plot.
#+ATTR_HTML: :width 900px
#+NAME: PolarPlot
#+RESULTS:
[[file:example.png]]

** Org with bibliography

\begin{align}
\label{eq:one_eq}
{\frac {d}{dt}}\iint _{\Sigma (t)}\mathbf {F} (\mathbf {r} ,t)\cdot d\mathbf {A} = & \iint _{\Sigma (t)}\left(\mathbf {F} _{t}(\mathbf {r},t)+\left[\nabla \cdot \mathbf {F} (\mathbf {r} ,t)\right]\mathbf {v}
\right)\cdot d\mathbf {A} - \\
& \oint _{\partial \Sigma (t)}\left[\mathbf{v} \times \mathbf {F} (\mathbf {r} ,t)\right]\cdot d\mathbf {s} \nonumber
\end{align}

Eq. \ref{eq:one_eq} is demonstrated in cite:Flanders1973.

Bibliography

Now we reach a little trick to support both HTML and PDF bibliography exports:

#+BEGIN_EXPORT latex
\printbibliography
#+END_EXPORT

#+BEGIN_EXPORT latex
\begin{comment}
#+END_EXPORT
#+BIBLIOGRAPHY: my-bib plain
#+BEGIN_EXPORT latex
\end{comment}
#+END_EXPORT

Explanation:

To export HTML bibliography, ox-bibtex does the job with only one directive:

#+BIBLIOGRAPHY: my-bib plain

However, for PDF export we do not want to use ox-bibtex, as it does not support UTF8. The solution is to wrap this directive into a comment section in the generated .tex code:

#+BEGIN_EXPORT latex
\begin{comment}
#+END_EXPORT
#+BIBLIOGRAPHY: my-bib plain
#+BEGIN_EXPORT latex
\end{comment}
#+END_EXPORT

Now we must tell LaTeX to use biblatex, this is done thanks to this directive:

#+BEGIN_EXPORT latex
\printbibliography
#+END_EXPORT

Putting everything together you get the proposed solution. This is certainly not the cleanest approach, but I have not found simpler.

The my-bib.bib file

For our example we need a small bibliography my-bib.bib file:

@article{Flanders1973,
  doi = {10.2307/2319163},
  url = {https://doi.org/10.2307/2319163},
  year  = {1973},
  month = {jun},
  publisher = {{JSTOR}},
  volume = {80},
  number = {6},
  pages = {615},
  author = {Harley Flanders},
  title = {Differentiation Under the Integral Sign},
  journal = {The American Mathematical Monthly}
}

Usage

You can visit the GitHub repo to reproduce the results.

Starting Emacs with the local configuration

From project root directory type

emacs -q --load emacs_files/init.el

to start a new Emacs with our local configuration.

Recomputing the notebook

As I potentially have several notebooks to publish I have used the :eval no-export argument. By consequence the notebooks are not evaluated each time you publish but only once. If you want to recompute everything every time, simply remove this option. You can also use the :cache option.

By consequence, before exporting you must begin by a first evaluation of the notebook. Visit the example.org buffer and do M-x org-babel-execute-buffer (or use the C-c C-v b shortcut). Attention, be sure that Plots.jl is installed.

ERROR: MethodError: no method matching start(::…)

In the ∗demo_session∗ Julia session buffer you will certainly see this error:

ERROR: MethodError: no method matching start(::...)

This is not our fault, but a known problem (that would need a fix) julia-print-commands-not-working-in-emacs-org-mode. It does not affect the computed result (but only the output processing). To get the right output (without the error message) one workaround is to restart computation of the source block (C-c C-c).

Exporting

Still from the example.org buffer, you can do:

  • HTML export with: C-c C-e h o
  • PDF export with: C-c C-e l o

This should generate and open fresh hmtl and pdf files.

Note: concerning html files, this is a basic export, you can use your own HTML theme.