Tag Archives: julialang

#MonthOfJulia Day 37: Fourier Techniques

Julia-Logo-Fourier

The Fourier Transform is often applied to signal processing and other analyses. It allows a signal to be transformed between the time domain and the frequency domain. The efficient Fast Fourier Transform (FFT) algorithm is implemented in Julia using the FFTW library.

1D Fourier Transform

Let’s start by looking at the Fourier Transform in one dimension. We’ll create test data in the time domain using a wide rectangle function.

julia> f = [abs(x) <= 1 ? 1 : 0 for x in -5:0.1:5];
julia> length(f)
101

This is what the data look like:
signal-1D-time-domain
We’ll transform the data into the frequency domain using fft().

julia> F = fft(f);
julia> typeof(F)
Array{Complex{Float64},1}
julia> length(F)
101
julia> F = fftshift(F);

The frequency domain data are an array of Complex type with the same length as the time domain data. Since each Complex number consists of two parts (real and imaginary) it seems that we have somehow doubled the information content of our signal. This is not true because half of the frequency domain data are redundant. The fftshift() function conveniently rearranges the data in the frequency domain so that the negative frequencies are on the left.

This is what the resulting amplitude and power spectra look like:
signal-1D-amplitude-spectrum-shifted
signal-1D-power-spectrum-shifted
The analytical Fourier Transform of the rectangle function is the sinc function, which agrees well with numerical data in the plots above.

2D Fourier Transform

Let’s make things a bit more interesting: we’ll look at the analogous two-dimensional problem. But this time we’ll go in the opposite direction, starting with a two-dimensional sinc function and taking its Fourier Transform.

Building the array of sinc data is easy using a list comprehension.

julia> f = [(r = sqrt(x^2 + y^2); sinc(r)) for x in -6:0.125:6, y in -6:0.125:6];
julia> typeof(f)
Array{Float64,2}
julia> size(f)
(97,97)

It doesn’t make sense to think about a two-dimensional function in the time domain. But the Fourier Transform is quite egalitarian: it’s happy to work with a temporal signal or a spatial signal (or a signal in pretty much any other domain). So let’s suppose that our two-dimensional data are in the spatial domain. This is what it looks like:
function-2D-sinc

Generating the Fourier Transform is again a simple matter of applying fft(). No change in syntax: very nice indeed!

julia> F = fft(f);
julia> typeof(F)
Array{Complex{Float64},2}
julia> F = fftshift(F);

The power spectrum demonstrates that the result is the 2D analogue of the rectangle function.
power-2D-sinc

Higher Dimensions and Beyond

It’s just as easy to apply the FFT to higher dimensional data, although in my experience this is rarely required.

Most of the FFTW library’s functionality has been implemented in the Julia interface. For example:

  • it’s possible to generate plans for optimised FFTs using plan_fft();
  • dct() yields the Discrete Cosine Transform;
  • you can exploit conjugate symmetry in real transforms using rfft(); and
  • it’s possible to run over multiple threads using FFTW.set_num_threads().

Watch the video below in which Steve Johnson demonstrates many of the features of FFTs in Julia.

The post #MonthOfJulia Day 37: Fourier Techniques appeared first on Exegetic Analytics.

#MonthOfJulia Day 36: Markdown

Julia-Logo-Markdown

Markdown is a lightweight format specification language developed by John Gruber. Markdown can be converted to HTML, LaTeX or other document formats. You probably knew all that already. The syntax is pretty simple. Check out this useful cheatsheet.

In the latest stable version of Julia support for markdown is provided in the Base package.

julia> using Base.Markdown
julia> import Base.Markdown: MD, Paragraph, Header, Italic, Bold, LineBreak, plain, term, html,
                             Table, Code, LaTeX, writemime

Markdown is stored in objects of type Base.Markdown.MD. As you’ll see below, there are at least two ways to construct markdown objects: either directly from a string (using markdown syntax) or programmatically (using a selection of formatting functions).

julia> d1 = md"foo *italic foo* **bold foo** `code foo`";
julia> d2 = MD(Paragraph(["foo ", Italic("italic foo"), " ", Bold("bold foo"), " ",
               Code("code foo")]));
julia> typeof(d1)
Base.Markdown.MD
julia> d1 == d2
true

You’ll find that Base.Markdown.MD objects are rendered with appropriate formatting in your console.
julia-console-markdown

Functions html() and latex() convert Base.Markdown.MD objects into other formats. Another way of rendering markdown elements is with writemime(), where the output is determined by specifying a MIME type.

julia> html(d1)
"<p>foo <em>italic foo</em> <strong>bold foo</strong> <code>code foo</code></p>n"
julia> latex(d1)
"foo \emph{italic foo} \textbf{bold foo} \texttt{code foo}n"

Markdown has support for section headers, both ordered and unordered lists, tables, code fragments and block quotes.

julia> d3 = md"""# Chapter Title
       ## Section Title
       ### Subsection Title""";
julia> d4 = MD(Header{2}("Section Title"));
julia> d3 |> html
"<h1>Chapter Title</h1>n<h2>Section Title</h2>n<h3>Subsection Title</h3>n"
julia> latex(d4)
"\subsection{Section Title}n"

Most Julia packages come with a README.md markdown file which provides an overview of the package. The readme() function gives you direct access to these files’ contents.

julia> readme("Quandl")
  Quandl.jl
  ============
  (Image: Build Status)
  (Image: Coverage Status)
  (Image: Quandl)

  Documentation is provided by Read the Docs.

  See the Quandl API Help Page for further details about the Quandl API. This package 
  closely follows the nomenclature used by that documentation.

We can also use parse_file() to treat the contents of a file as markdown.

julia> d6 = Markdown.parse_file(joinpath(homedir(), ".julia/v0.4/NaNMath/README.md"));

This is rendered below as LaTeX.

section{NaNMath}
Implementations of basic math functions which return texttt{NaN} instead of throwing a
texttt{DomainError}.
Example:
begin{verbatim}
import NaNMath
NaNMath.log(-100) # NaN
NaNMath.pow(-1.5,2.3) # NaN
end{verbatim}
In addition this package provides functions that aggregate one dimensional arrays and ignore
elements that are NaN. The following functions are implemented:
begin{verbatim}
sum
maximum
minimum
mean
var
std
end{verbatim}
Example:
begin{verbatim}
using NaNMath; nm=NaNMath
nm.sum([1., 2., NaN]) # result: 3.0
end{verbatim}
href{https://travis-ci.org/mlubin/NaNMath.jl}{begin{figure}
centering
includegraphics{https://travis-ci.org/mlubin/NaNMath.jl.svg?branch=master}
caption{Build Status}
end{figure}
}

And here it is as HTML.

NaNMath

Implementations of basic math functions which return NaN instead of throwing a DomainError.
Example:

import NaNMath
NaNMath.log(-100) # NaN
NaNMath.pow(-1.5,2.3) # NaN

In addition this package provides functions that aggregate one dimensional arrays and ignore elements that are NaN. The following functions are implemented:

sum
maximum
minimum
mean
var
std

Example:

using NaNMath; nm=NaNMath
nm.sum([1., 2., NaN]) # result: 3.0

Build Status

What particularly appeals to me about the markdown functionality in Julia is the potential for automated generation of documentation and reports. To see more details of my dalliance with Julia and markdown, visit github.

The post #MonthOfJulia Day 36: Markdown appeared first on Exegetic Analytics.

#MonthOfJulia Day 35: Mapping

Julia-Logo-OpenStreetMap

A lot of my data reflects events happening at different geographic locations (and, incidentally, at different times, but that’s another story). So it’s not surprising that I’m interested in mapping those data. Julia has an OpenStreetMap package which presents an interface to the OpenStreetMap service. The package is well documented and has an extensive range of functionality. As with a number of previous posts in this series, I’m just going to skim the surface of what’s available.

We’ll need to load up the Requests package to retrieve the map data and the OpenStreetMap package to manipulate and process those data.

julia> using Requests
julia> using OpenStreetMap

As far as I can see the OpenStreetMap package doesn’t implement functionality for downloading the map data. So we do this directly through an HTTP request. We’ll specify a map area by giving the latitude and longitude of the bottom-left and top-right corners.

julia> const MAPFILE = "map.osm";
julia> minLon = 30.8821;
julia> maxLon = minLon + 0.05;
julia> minLat = -29.8429;
julia> maxLat = minLat + 0.05;

We then build the query URL using Julia’s convenient string interpolation and execute a GET request against the OpenStreetMap API.

julia> URL = "http://overpass-api.de/api/map?bbox=$(minLon),$(minLat),$(maxLon),$(maxLat)"
"http://overpass-api.de/api/map?bbox=30.8821,-29.8429,30.932100000000002,-29.7929"
julia> osm = get(URL)
Response(200 OK, 10 headers, 1958494 bytes in body)
julia> save(osm, MAPFILE)
"map.osm"

Save the resulting data (it’s just a large blob of XML) to a file. Feel free to open this file in an editor and browse around. Although there is currently no official schema for the OpenStreetMap XML, the documentation gives a solid overview of the format.

$ file map.osm 
map.osm: OpenStreetMap XML data

We process the contents of the XML file using getOSMData().

julia> nodes, highways, buildings, features = getOSMData(MAPFILE);
julia> println("Number of nodes: $(length(nodes))")
Number of nodes: 9360
julia> println("Number of highways: $(length(highways))")
Number of highways: 592
julia> println("Number of buildings: $(length(buildings))")
Number of buildings: 5
julia> println("Number of features: $(length(features))")
Number of features: 12

The call to getOSMData() returns all of the data required to build a map. Amongst these you’ll find a dictionary of features broken down by :class, :detail and :name. It’s always handy to know where the nearest Woolworths is, and this area has two of them.

julia> features
Dict{Int64,OpenStreetMap.Feature} with 12 entries:
  1871785198 => OpenStreetMap.Feature("amenity","pharmacy","Clicks")
  270909308  => OpenStreetMap.Feature("amenity","fuel","BP")
  1932067048 => OpenStreetMap.Feature("shop","supermarket","Spar")
  747740685  => OpenStreetMap.Feature("shop","supermarket","Westville mall")
  3011871215 => OpenStreetMap.Feature("amenity","restaurant","Lupa")
  1871785313 => OpenStreetMap.Feature("shop","clothes","Woolworths")
  1871785167 => OpenStreetMap.Feature("shop","supermarket","Checkers")
  747740690  => OpenStreetMap.Feature("amenity","school","Westville Girl's High")
  1872497461 => OpenStreetMap.Feature("shop","supermarket","Pick n Pay")
  1554106907 => OpenStreetMap.Feature("amenity","pub","Waxy O'Conner's")
  1872497555 => OpenStreetMap.Feature("shop","supermarket","Woolworths")
  1932067047 => OpenStreetMap.Feature("amenity","bank","Standard Bank")
julia> fieldnames(OpenStreetMap.Feature)
3-element Array{Symbol,1}:
 :class 
 :detail
 :name 

There are other dictionarys which list the highways and buildings in the area.

Although we specified the latitudinal and longitudinal extremes of the map originally, we can retrieve these wrapped up in a data structure. Note that these values are given in Latitude-Longitude-Altitude (LLA) coordinates. There’s functionality for transforming to other coordinate systems like East-North-Up (ENU).

julia> bounds = getBounds(parseMapXML(MAPFILE))
Geodesy.Bounds{Geodesy.LLA}(-29.8429,-29.7929,30.8821,30.9321)

We’re ready to take a look at the map using plotMap().

julia> const WIDTH = 800;
julia> plotMap(nodes,
               highways = highways,
               buildings = buildings,
               features = features,
               bounds = bounds,
               width = WIDTH,
               roadways = roads)

And here’s what it looks like. There are ways to further customise the look and feel of the map.

map

Plotting maps is just the beginning. You can use findIntersections() to fing highway intersections; generate a transportation network using createGraph(); and find the shortest and fastest routes between locations using shortestRoute() and fastestRoute(). The package is literally a trove of cool and useful things.

There might be interesting synergies between this package and the GeoInterface, GeoIP, GeoJSON and Geodesy packages. Those will have to wait for another day. But feel free to experiment in the meantime!

The post #MonthOfJulia Day 35: Mapping appeared first on Exegetic Analytics.