Category Archives: Julia

Order Flow Imbalance – A High Frequency Trading Signal

By: Dean Markwick's Blog -- Julia

Re-posted from: https://dm13450.github.io/2022/02/02/Order-Flow-Imbalance.html

I’ll show you how to calculate the ‘order flow imbalance’ and build a
high-frequency trading signal with the results. We’ll see if it is a
profitable strategy and how it might be useful as a market indicator.

A couple of months ago I attended the Oxford Math Finance seminar
where there was a presentation on order book dynamics to predict
short-term price direction. Nicholas Westray presented
Deep Order Flow Imbalance: Extracting Alpha at Multiple Horizons from the Limit Order Book. By
using deep learning they predict future price movements using common
neural network architectures such as the basic multi-layer perceptron
(MLP), Long Term Short Memory network (LSTM) and convolutional neural
networks (CNN). Combining all three networks types lets you extract
the strengths of each network:

  • CNNs: Reduce frequency variations.
    • The variations between each input reduce to some common
      factors.
  • LSTMs: Learn temporal structures
    • How are the different inputs correlated with each other such as
      autocorrelation structure.
  • MPLs: Universal approximators.
    • An MLP can approximate any function.

A good reference for LSTM-CNN combinations is DeepLOB: Deep
Convolution Neural Networks for Limit Order Books

where they use this type of neural network to predict whether the
market is moving up or down.

The Deep Order Flow talk was an excellent overview of deep
learning concepts and described how there is a great overlap between
computer vision and the state of the order book. You can build an
“image” out of the different order book levels and pass this through
the neural networks. My main takeaway from the talk was the concept of
Order Flow Imbalance. This is a transformation that uses the order
book to build a feature to predict future returns.

I’ll show you how to calculate the order flow imbalance and see how
well it predicts future returns.

The Setup

I have a QuestDB database with the best bid and offer price and size
at those levels for BTCUSD from Coinbase over roughly 24 hours. To
read how I collected this data check out my previous post on
streaming data into QuestDB.

Julia easily connects to QuestDB using the LibPQ.jl package. I also
load in the basic data manipulation packages and some statistics
modules to calculate the necessary values.

using LibPQ
using DataFrames, DataFramesMeta
using Plots
using Statistics, StatsBase
using CategoricalArrays
using Dates
using RollingFunctions

conn = LibPQ.Connection("""
             dbname=qdb
             host=127.0.0.1
             password=quest
             port=8812
             user=admin""")

Order flow imbalance is about the changing state of the order book. I
need to pull out the full best bid best offer table. Each row in this
table represents when the best price or size at the best price changed.

bbo = execute(conn, 
    "SELECT *
     FROM coinbase_bbo") |> DataFrame
dropmissing!(bbo);

I add the mid-price in too, as we will need it later.

bbo = @transform(bbo, mid = (:ask .+ :bid) / 2);

It is a big dataframe, but thankfully I’ve got enough RAM.

Calculating Order Flow Imbalance

Order flow imbalance represents the changes in supply and demand. With
each row one of the price or size at the best bid or ask changes which
corresponds to change in the supply or demand, even at a high
frequency level, of Bitcoin.

  • Best bid or size at the best bid increase -> increase in demand.
  • Best bid or size at the best bid decreases -> decrease in demand.
  • Best ask decreases or size at the best ask increases -> increase
    in supply.
  • Best ask increases or size at the best ask decreases ->
    decrease in supply.

Mathematically we summarise these four effects at from time \(n-1\) to
\(n\) as:

\[e_n = I_{\{ P_n^B \geq P^B_{n-1} \}} q_n^B – I_\{ P_n^B \leq
P_{n-1}^B \} q_{n-1}^B – I_\{ P_n^A \leq P_{n-1}^A \}
q_n^A + I_\{ P_n^A \geq P_{n-1}^A \} q_{n-1}^A,\]

where \(P\) is the best price at the bid (\(P^B\)) or ask (\(P^A\)) and
\(q\) is the size at those prices.

Which might be a bit easier to read as Julia code:

e = Array{Float64}(undef, nrow(bbo))
fill!(e, 0)

for n in 2:nrow(bbo)
    
    e[n] = (bbo.bid[n] >= bbo.bid[n-1]) * bbo.bidsize[n] - 
    (bbo.bid[n] <= bbo.bid[n-1]) * bbo.bidsize[n-1] -
    (bbo.ask[n] <= bbo.ask[n-1]) * bbo.asksize[n] + 
    (bbo.ask[n] >= bbo.ask[n-1]) * bbo.asksize[n-1]
    
end

bbo[!, :e] = e;

To produce an Order Flow Imbalance (OFI) value, you need to aggregate
\(e\) over some time-bucket. As this is a high-frequency problem I’m
choosing 1 second. We also add in the open and close price of the
buckets and the return across this bucket.

bbo = @transform(bbo, timestampfloor = floor.(:timestamp, Second(1)))
bbo_g = groupby(bbo, :timestampfloor)
modeldata = @combine(bbo_g, ofi = sum(:e), OpenPrice = first(:mid), ClosePrice = last(:mid), NTicks = length(:e))
modeldata = @transform(modeldata, OpenCloseReturn = 1e4*(log.(:ClosePrice) .- log.(:OpenPrice)))
modeldata = modeldata[2:(end-1), :]
first(modeldata, 5)

5 rows × 6 columns

timestampfloor ofi OpenPrice ClosePrice NTicks OpenCloseReturn
DateTime Float64 Float64 Float64 Int64 Float64
1 2021-07-24T08:50:36 0.0753159 33655.1 33655.1 77 0.0
2 2021-07-24T08:50:37 4.44089e-16 33655.1 33655.1 47 0.0
3 2021-07-24T08:50:38 0.0 33655.1 33655.1 20 0.0
4 2021-07-24T08:50:39 3.05727 33655.1 33655.1 164 0.0
5 2021-07-24T08:50:40 2.40417 33655.1 33657.4 278 0.674467

Now we do the usual train/test split by selecting the first 70% of the
data.

trainInds = collect(1:Int(floor(nrow(modeldata)*0.7)))
trainData = modeldata[trainInds, :]
testData = modeldata[Not(trainInds), :];

We are going to fit a basic linear regression using the OFI value as
the single predictor.

using GLM

ofiModel = lm(@formula(OpenCloseReturn ~ ofi), trainData)
StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}, Matrix{Float64}}

OpenCloseReturn ~ 1 + ofi

Coefficients:
─────────────────────────────────────────────────────────────────────────────
                  Coef.   Std. Error       t  Pr(>|t|)  Lower 95%   Upper 95%
─────────────────────────────────────────────────────────────────────────────
(Intercept)  -0.0181293  0.00231571    -7.83    <1e-14  -0.022668  -0.0135905
ofi           0.15439    0.000695685  221.92    <1e-99   0.153026   0.155753
─────────────────────────────────────────────────────────────────────────────

We see a positive coefficient of 0.15 which is very significant.

r2(ofiModel)
0.3972317963590547

A very high in-sample \(R^2\).

predsTrain = predict(ofiModel, trainData)
predsTest = predict(ofiModel, testData)

(mean(abs.(trainData.OpenCloseReturn .- predsTrain)),
    mean(abs.(testData.OpenCloseReturn .- predsTest)))
(0.3490577385082666, 0.35318460250890665)

Comparable mean absolute error (MAE) across both train and test sets.

sst = sum((testData.OpenCloseReturn .- mean(testData.OpenCloseReturn)) .^2)
ssr = sum((predsTest .- mean(testData.OpenCloseReturn)) .^2)
ssr/sst
0.4104873667550974

An even better \(R^2\) in the test data

extrema.([predsTest, testData.OpenCloseReturn])
2-element Vector{Tuple{Float64, Float64}}:
 (-5.400295917229609, 5.285718311926791)
 (-11.602503514716034, 11.46049286770534)

But doesn’t quite predict the largest or smallest values.

So overall:

  • Comparable R2 and MAE values across the training and test sets.
  • Positive coefficient indicates that values with high positive order flow imbalance will have a large positive return.

But, this all suffers from the cardinal sin of backtesting, we are using information from the future (the sum of the \(e\) values to form the OFI) to predict the past. By the time we know the OFI value, the close value has already happened! We need to be smarter if we want to make trading decisions based on this variable.

So whilst it doesn’t give us an actionable signal, we know that it can explain price moves, we know just have to reformulate our model and make sure there is no information leakage.

Building a Predictive Trading Signal

I now want to see if OFI can be used to predict future price
returns. First up, what do the OFI values look like and what
about if we take a rolling average?

Using the excellent RollingFunctions.jl package we can calculate
the five-minute rolling average and compare it to the raw values.

xticks = collect(minimum(trainData.timestampfloor):Hour(4):maximum(trainData.timestampfloor))
xtickslabels = Dates.format.(xticks, dateformat"HH:MM")

ofiPlot = plot(trainData.timestampfloor, trainData.ofi, label = :none, title="OFI", xticks = (xticks, xtickslabels), fmt=:png)
ofi5minPlot = plot(trainData.timestampfloor, runmean(trainData.ofi, 60*5), title="OFI: 5 Minute Average", label=:none, xticks = (xticks, xtickslabels))
plot(ofiPlot, ofi5minPlot, fmt=:png)

OFI and 5 Minute OFI

It’s very spiky, but taking the rolling average smooths it out. To
scale the OFI values to a known range, I’ll perform the Z-score
transform using the rolling five-minute window of both the mean and
variance. We will also use the close to close returns rather than the
open-close returns of the previous model and make sure it is lagged
correctly to prevent information leakage.

modeldata = @transform(modeldata, ofi_5min_avg = runmean(:ofi, 60*5),
                                  ofi_5min_var = runvar(:ofi, 60*5),
                                  CloseCloseReturn = 1e4*[diff(log.(:ClosePrice)); NaN])

modeldata = @transform(modeldata, ofi_norm = (:ofi .- :ofi_5min_avg) ./ sqrt.(:ofi_5min_var))

modeldata[!, :CloseCloseReturnLag] = [NaN; modeldata.CloseCloseReturn[1:(end-1)]]

modeldata[1:7, [:ofi, :ofi_5min_avg, :ofi_5min_var, :ofi_norm, :OpenPrice, :ClosePrice, :CloseCloseReturn]]

7 rows × 7 columns

ofi ofi_5min_avg ofi_5min_var ofi_norm OpenPrice ClosePrice CloseCloseReturn
Float64 Float64 Float64 Float64 Float64 Float64 Float64
1 0.0753159 0.0753159 0.0 NaN 33655.1 33655.1 0.0
2 4.44089e-16 0.037658 0.00283625 -0.707107 33655.1 33655.1 0.0
3 0.0 0.0251053 0.00189083 -0.57735 33655.1 33655.1 0.0
4 3.05727 0.783146 2.29977 1.49959 33655.1 33655.1 0.674467
5 2.40417 1.10735 2.25037 0.864473 33655.1 33657.4 1.97263
6 2.4536 1.33172 2.10236 0.773732 33657.4 33664.0 0.252492
7 -2.33314 0.808173 3.67071 -1.63959 33664.0 33664.9 -0.531726
xticks = collect(minimum(modeldata.timestampfloor):Hour(4):maximum(modeldata.timestampfloor))
xtickslabels = Dates.format.(xticks, dateformat"HH:MM")

plot(modeldata.timestampfloor, modeldata.ofi_norm, label = "OFI Normalised", xticks = (xticks, xtickslabels), fmt=:png)
plot!(modeldata.timestampfloor, modeldata.ofi_5min_avg, label="OFI 5 minute Average")

A plot of the normalised order flow imbalance with the rolling 5 minute average overlaid.

The OFI values have been compressed from \((-50, 50)\) to \((-10,
10)\). From the average values we can see periods of positive and
negative regimes.

When building the model we split the data into a training and
testing sample, throwing away the early values where the was not
enough values for the rolling statistics to calculate.

We use a basic linear regression with just the normalised OFI value.

trainData = modeldata[(60*5):70000, :]
testData = modeldata[70001:(end-1), :]

ofiModel_predict = lm(@formula(CloseCloseReturn ~ ofi_norm), trainData)
StatsModels.TableRegressionModel{LinearModel{GLM.LmResp{Vector{Float64}}, GLM.DensePredChol{Float64, LinearAlgebra.CholeskyPivoted{Float64, Matrix{Float64}}}}, Matrix{Float64}}

CloseCloseReturn ~ 1 + ofi_norm

Coefficients:
────────────────────────────────────────────────────────────────────────────
                 Coef.  Std. Error      t  Pr(>|t|)    Lower 95%   Upper 95%
────────────────────────────────────────────────────────────────────────────
(Intercept)  0.0020086  0.00297527   0.68    0.4996  -0.00382293  0.00784014
ofi_norm     0.144358   0.00292666  49.33    <1e-99   0.138622    0.150094
────────────────────────────────────────────────────────────────────────────

A similar value in the coefficient compared to our previous model and
it remains statistically significant.

r2(ofiModel_predict)
0.033729601695801414

Unsurprisingly, a massive reduction on in-sample \(R^2\). A value of 3%
is not that bad, in the Deep Order Flow paper they achieve values of
around 1% but over a much larger dataset and across multiple
stocks. My 24 hours of Bitcoin data is much easier to predict.

returnPredictions = predict(ofiModel_predict, testData)

testData[!, :CloseClosePred] = returnPredictions

sst = sum((testData.CloseCloseReturn .- mean(testData.CloseCloseReturn)) .^2)
ssr = sum((testData.CloseClosePred .- mean(testData.CloseCloseReturn)) .^2)
ssr/sst
0.030495583445248473

The out-of-sample \(R^2\) is also around 3%, so not that bad really in
terms of overfitting. It looks like we’ve got a potential model on our
hands.

Does This Signal Make Money?

We can now go through a very basic backtest to see if this signal is
profitable to trade. This will all be done in pure Julia, without any
other packages.

Firstly, what happens if we go long every time the model predicts a
positive return and likewise go short if the model predicts a negative
return. This means simply taking the sign of the model prediction and
multiplying it by the observed returns will give us the returns of the
strategy.

In short, this means if our model were to predict a positive return
for the next second, we would immediately buy at the close and be filled
at the closing price. We would then close out our position after the
second elapsed, again, getting filled at the next close to produce a
return.

xticks = collect(minimum(testData.timestampfloor):Hour(4):maximum(testData.timestampfloor))
xtickslabels = Dates.format.(xticks, dateformat"HH:MM")

plot(testData.timestampfloor, cumsum(sign.(testData.CloseClosePred) .* testData.CloseCloseReturn), 
    label=:none, title = "Cummulative Return", fmt=:png, xticks = (xticks, xtickslabels))

Cumulative return

Up and to the right as we would hope. So following this strategy would
make you money. Theoretically. But is it a good strategy? To measure
this we can calculate the Sharpe ratio, which is measuring the overall
profile of the returns compared to the volatility of the returns.

moneyReturns = sign.(testData.CloseClosePred) .* testData.CloseCloseReturn
mean(moneyReturns) ./ std(moneyReturns)
0.11599938576235787

A Sharpe ratio of 0.12 if we are generous and round up. Anyone with
some experience in these strategies is probably having a good chuckle
right now, this value is terrible. At the very minimum, you would
like a value of 1, i.e. that your average return is greater than the
variance in returns, otherwise you are just looking at noise.

How many times did we correctly guess the direction of the market
though? This is the hit ratio of the strategy.

mean(abs.((sign.(testData.CloseClosePred) .* sign.(testData.CloseCloseReturn))))
0.530163738236414

So 53% of the time I was correct. 3% better than a coin toss, which is
good and shows there is a little bit of information in the OFI values
when predicting.

Does a Threshold Help?

Should we be more selective when we trade? What if we set a threshold and
only trade when our prediction is greater than that value. Plus the
same in the other direction. We can iterate through lots of potential
thresholds and see where the Sharpe ratios end up.

p = plot(ylabel = "Cummulative Returns", legend=:topleft, fmt=:png)
sharpes = []
hitratio = []

for thresh in 0.01:0.01:0.99
  trades = sign.(testData.CloseClosePred) .* (abs.(testData.CloseClosePred) .> thresh)

  newMoneyReturns = trades .* testData.CloseCloseReturn

  sharpe = round(mean(newMoneyReturns) ./ std(newMoneyReturns), digits=2)
  hr = mean(abs.((sign.(trades) .* sign.(testData.CloseCloseReturn))))

  if mod(thresh, 0.2) == 0
    plot!(p, testData.timestampfloor, cumsum(newMoneyReturns), label="$(thresh)", xticks = (xticks, xtickslabels))
  end
  push!(sharpes, sharpe)
  push!(hitratio, hr)
end
p

Equity curves for different thresholds

The equity curves look worse with each higher threshold.

plot(0.01:0.01:0.99, sharpes, label=:none, title = "Sharpe vs Threshold", xlabel = "Threshold", ylabel = "Sharpe Ratio", fmt=:png)

Sharpe Ratio vs Threshold

A brief increase in Sharpe ratio if we set a small threshold, but
overall, steadily decreasing Sharpe ratios once we start trading
less. For such a simple and linear model this isn’t surprising, but
once you start chucking more variables and different modeling
approaches into the mix it can shed some light on what happens around
the different values.

Why you shouldn’t trade this model

So at the first glance, the OFI signal looks like a profitable
strategy. Now I will highlight why it isn’t in practice.

  • Trading costs will eat you alive

I’ve not taken into account any slippage, probability of fill, or
anything that a real-world trading model would need to be
practical. As our analysis around the Sharpe ratio has shown, it wants
to trade as much as possible, which means transaction costs will just
destroy the return profile. With every trade, you will pay the full
bid-ask spread in a round trip to open and then close the trade.

  • The Sharpe ratio is terrible

With a Sharpe ratio < 1 shows that there is not much actual
information in the trading pattern, it is getting lucky vs the actual
volatility in the market. Now, Sharpe ratios can get funky when we are
looking at such high-frequency data, hence why this bullet point is second to the trading costs.

  • It has been trained on a tiny amount of data.

Needless to say, given that we are looking at seconds this dataset
could be much bigger and would give us greater confidence in the
actual results once expanded to a wider time frame of multiple days.

  • I’ve probably missed something that blows this out of the water

Like everything I do, there is a strong possibility I’ve gone wrong
somewhere, forgotten a minus, ordered a time-series wrong, and various other errors.

How this model might be useful

  • An overlay for a market-making algorithm

Making markets is about posting quotes where they will get filled and
collecting the bid-ask spread. Therefore, because our model appears to
be able to predict the direction fairly ok, you could use it to place
a quote where the market will be in one second, rather than where it
is now. This helps put your quote at the top of the queue if the
market does move in that direction. Secondly, if you are traded with
and need to hedge the position, you have an idea of how long to wait
to hedge. If the market is moving in your favour, then you can wait an
extra second to hedge and benefit from the move. Likewise, if this
model is predicting a move against your inventory position, then you
know to start aggressively quoting to minimise that move against.

  • An execution algorithm

If you are potentially trading a large amount of bitcoin, then you
want to split your order up into lots of little orders. Using this
model you then know how aggressive or passive you should trade based on
where the market is predicted to move second by second. If the order
flow imbalance is trending positive, the market is going to go up, so
you want to increase your buying as not to miss out on the move and
again, if the market is predicted to move down, you’ll want to slow
down your buying so that you fully benefit from the lull.

Conclusion

Overall hopefully you now know more about order flow imbalance and how
it can somewhat explain returns. It also has some predictive power and
we use that to try and build a trading strategy around the signal.

We find that the Sharpe ratio of said strategy is poor and that
overall, using it as a trading signal on its own will not have you
retiring to the Bahamas.

This post has been another look at high-frequency finance and the
trials and tribulations around this type of data.

How to Build Your First Web App in Julia with Genie.jl ?‍♂️

By: n Logan Kilpatrick n

Re-posted from: https://www.freecodecamp.org/news/how-to-build-web-apps-in-julia/

How to Build Your First Web App in Julia with Genie.jl ?‍♂️

Julia is a high-level, dynamic, and open-source programming language. It’s designed to be as easy to use as Python while remaining as performant as C or C++.

Many early use cases for Julia were in the scientific domains where massive computational processing was and still is required. But as the language has continued to grow, more and more use cases are gaining steam (hint: web development).

If you are totally new to Julia and want to get a handle on the syntax before you dive into creating your first web application, check out this article on freeCodeCamp.

It goes over the basics, how to install Julia, steps to install packages, and much more!

We will focus this tutorial on all the necessary steps to build your first web application in Julia from the ground up. So let’s begin by checking out the Genie website: https://genieframework.com.

What is Genie.jl? ?

Genie is a modern and highly productive web framework written in Julia. In the project’s own words:

Genie is a full-stack web framework that provides a streamlined and efficient workflow for developing modern web applications. It builds on Julia’s strengths (high-level, high-performance, dynamic, JIT-compiled), exposing a rich API and a powerful toolset for productive web development.

Genie is very similar to the Django Project in that Genie is more than a single framework. Instead, it is an entire ecosystem with extensions and the like.

But why do we need Genie? The simple answer is that as Julia continues to grow in popularity, more and more developers are looking to leverage Julia across their entire stack. Genie provides the ability to deploy websites with Julia code running on the server-side so you can do things like deploy machine learning models as part of your Genie app.

Before we dive into getting started with Genie, you might want to check out a live deployed Genie app to get a sense of what is possible: https://pkgs.genieframework.com.

This project is a community resource where you can query the number of package downloads during a certain time frame for a specific package. Type in “genie” to see the number of daily downloads.

You might also be interested in learning more about other GUI and web development frameworks in Julia. To learn more broadly about the ecosystem, check out this article.

How to Install Genie ⤵️

To get Genie installed, all we need to do is open the Julia REPL and type ] add Genie . This will take care of everything you need. If everything works, you should be able to do:

julia> using Genie

without any issues. You are now all set to begin trying out Genie.

How to Map URLs to Julia Functions ?

A core part of the Genie framework is the idea of a router. Routers take the user action of visiting a specific URL and associate it with a Julia function being called.

Let’s look at a simple example of this. In the REPL, type the following:

julia> using Genie, Genie.Router

julia> route("/hello") do
           "Hello freeCodeCamp"
       end
[GET] /hello => #5 | :get_hello

In this example, we defined the “/hello” URL to return the text “Hello freeCodeCamp”. We can verify that this works by starting the server:

julia> up() # start server
┌ Info: 
└ Web Server starting at http://127.0.0.1:8000 
Genie.AppServer.ServersCollection(Task (runnable) @0x000000011c5c5bb0, nothing)

Now that the server is up and running, we can visit http://127.0.0.1:8000 in our browser. You will notice we get a 404 page, which is expected since the only route we defined was “/hello”. So let’s add that to the URL and see what we get:

Browser window showing nothing but the text "Hello freeCodeCamp"

And there we go! Our first step towards building a fully functional web application is complete. We can also confirm that the page is loading correctly by checking the REPL which shows this:

julia> ┌ Error: GET / 404
└ @ Genie.Router ~/.julia/packages/Genie/UxbVJ/src/Router.jl:163
┌ Error: GET /favicon.ico 404
└ @ Genie.Router ~/.julia/packages/Genie/UxbVJ/src/Router.jl:163
[ Info: GET /hello 200

We see the first attempt where the result was a 404 and on the 2nd attempt where we successfully got the response (the 200 message means everything is okay).

Now that we have a basic example working, let’s now try and build on this with some more depth.

To do this, we will create a new file. I will be using VS Code but you are welcome to use any IDE you find useful. Before we look at the next piece of code, we need to make sure we shut down the server by typing down() into the REPL.

Okay, onto the next example:

using Genie, Genie.Router
using Genie.Renderer, Genie.Renderer.Html, Genie.Renderer.Json

route("/") do
    html("Hey freeCodeCamp")
end

route("/hello.html") do
  html("Hello freeCodeCamp (in html)")
end

route("/hello.json") do
  json("Hi freeCodeCamp (in json)")
end

route("/hello.txt") do
   respond("Hiya freeCodeCamp (in txt format)", :text)
end

# Launch the server on a specific port, 8002
# Run the task asynchronously
up(8002, async = true)

A lot is going on in this example, so let’s walk through what is taking place.

We start by loading in the packages we want. Then, we define 4 different routes. The first one is the index route. So when the user visits http://127.0.0.1:8002 they will see “Hey freeCodeCamp”. The routes after the index highlight that each route can give a custom output. In some cases, it can be HTML, in others, it could be JSON or plain text.

The last line of this example showcases the server launching code. As the comment states, we can set the specific port number and choose if we want the routes to run asynchronously or not. We have now successfully created our first Genie Script!

How to Create a Basic Web Service ?

Now that we have gotten our hands dirty with the basics, we will now begin to get closer to building a fully-fledged web application.

Before we go all the way there, we are going to take the first step which is creating a basic web service. To do so, we will go into the REPL and switch our current directory to one which is easily accessible. I will use my desktop in this tutorial:

shell> cd Desktop
/Users/logankilpatrick/Desktop

To enter shell mode which is shown above, simply type a “;” into the REPL. Now that we have our active directory set to the desktop in my case, we will use the handy generator function to create the service:

julia> Genie.newapp_webservice("freeCodeCampApp")

[ Info: Done! New app created at /Users/logankilpatrick/Desktop/freeCodeCampApp
[ Info: Changing active directory to /Users/logankilpatrick/Desktop/freeCodeCampApp
    /var/folders/tc/519vfm453fj_x5bmd8pwx9480000gn/T/jl_bO1R8h/FreeCodeCampApp/Project.toml
[ Info: Project.toml has been generated
[ Info: Installing app dependencies
...

The newapp_webservice is a very helpful function that automatically creates all the pieces we need for our first web service. Now that we have a project created, we need to open it up in an IDE (in my case, VS Code). You should see the following if you open up the correct folder:

Screen-Shot-2022-01-30-at-7.39.23-PM

There are a lot of files created for us automatically. The main one we will look at is routes.jl which is used to create routes as we did in the section above.

The function we called to generate these folders automatically starts the server, so let’s take a quick look at the existing landing page by visiting http://127.0.0.1:8000:

Screen-Shot-2022-01-30-at-7.51.16-PM

As you might notice, my page looks a little different than yours might because I went in and edited the welcome.html page found in the public folder.

As you can see in routes.jl, when the user visits the main URL /, we route them to the welcome page. We can add in additional routes as we did in the section above and expand this. You are welcome to pause here and play around. We already have a pretty robust website setup.

If you take a peek into some of the other folders like config/env, you will see details around setting the port, host URL, and other relevant parameters. Again, feel free to play around there but we will not go into all the detail of those files in this tutorial.

Before we dive into the next topic, let’s take a look at a few more of the files generated for our basic web service:

  • The public folder has all of the front end files (HTML and CSS)
  • The src folder has the entry point to the web service (in my case freeCodeCampApp.jl)
  • bin contains some additional dependencies we will again ignore
  • Manifest.toml and Project.toml are the key Julia files that allow us to maintain our Julia dependencies. When you created the web service, the script automatically activated your current project environment (which is the app we just created). You can verify this by typing “]” into the REPL which will show the active space in blue:
Screen-Shot-2022-01-30-at-7.59.49-PM

This just means that if we try to add a package, it will add it to the project and manifest file specifically for this project, instead of the globally shared one.

How to Create a Fully Functioning Web App With a Database ?

Now that we have explored the basics, we are going to dive into a full-on web app. Again, Genie provides some nice functions to get us started. Before we create it, we will need to navigate back to the desktop:

shell> pwd
/Users/logankilpatrick/Desktop/freeCodeCampApp

shell> cd ..
/Users/logankilpatrick/Desktop

shell> 

Remember, you can type ; to enter the shell mode and backspace to exit the shell mode. Now, let’s create the app:

julia> Genie.newapp_mvc(Genie.newapp_mvc("freeCodeCampMVC"))
   Resolving package versions...
   ...

You will be prompted to choose a database backend. For this example, we will use SQLite:

Screen-Shot-2022-01-30-at-8.08.31-PM

If you want to use a different database backend, feel free to do so as well. But note that you will need to create the database file automatically. Genie only creates an SQLite file for you.

We now have a MVC app created. But you might be asking yourself, what is an MVC?

The Model-View-Controller paradigm is very common across application development. In the interest of not getting into the weeds on it, I will refer you to this post where you can read about the details. From our perspective as developers, there is not much impact.

Just like we did when we created the last project, we need to open it in the IDE again:

Screen-Shot-2022-02-01-at-6.44.21-AM

Again, we will see much of the same stuff as before with the new addition of the app folder which will contain a lot of critical code. We can see what the new project looks like by typing:

julia> loadapp()

julia> up()

and then navigating too: http://127.0.0.1:8000.

Next up, we will need to connect our database to the web app we created. To do this, head to db/connection.yml and edit the following section:

env: ENV["GENIE_ENV"]

dev:
  adapter: SQLite
  database: db/freeCodeCamp_courses.sqlite

You can leave the rest of the fields blank for now. Then, we need to run:

julia> include(joinpath("config", "initializers", "searchlight.jl"))

which will load the database configuration. Next up, we will continue to configure the database such that we can save data from our app into persistent storage.

We begin this process by creating a new resource:

julia> Genie.newresource("course")

Once we have defined a resource, the next step is to go and edit the database migrations table which can be found at db/migrations/2022020115190055_create_table_courses.jl in my case.

By default, the table is already populated with some placeholder text based on the last few commands we ran. It should look something like this:

Screen-Shot-2022-02-01-at-7.22.35-AM

We will edit the file to match the specific scheme we want. This will be entirely dependent on the application itself. Since I am making courses on this site, I will enter all of the course details as follows:

module CreateTableCourses

import SearchLight.Migrations: create_table, column, columns, pk, add_index, drop_table, add_indices

function up()
  create_table(:courses) do
    [
      pk()
      column(:title, :string, limit = 200)
      column(:authors, :string, limit = 250)
      column(:year, :integer, limit = 4)
      column(:rating, :string, limit = 10)
      column(:categories, :string, limit = 100)
      column(:description, :string, limit = 1_000)
      column(:cost, :float, limit = 1000)
    ]
  end

  add_index(:courses, :title)
  add_index(:courses, :authors)
  add_index(:courses, :categories)
  add_index(:courses, :description)

end

function down()
  drop_table(:courses)
end

end

Again, these are arbitrary and can be whatever you want them to be.

It is worth noting that adding the index is optional. The reason you would add it is that it speeds up the queries, but there are other tradeoffs and you can’t actually load all the columns as indexes. You can read more about some of these tradeoffs here and here.

Now that we have the database table updated, we need to propagate these updates. To do so, we will use SearchLight.jl which functions as our app’s migration system:

julia> using SearchLight

julia> SearchLight.Migration.create_migrations_table()
┌ Info: 2022-02-01 07:37:11 CREATE TABLE `schema_migrations` (
│       `version` varchar(30) NOT NULL DEFAULT '',
│       PRIMARY KEY (`version`)
└     )
[ Info: 2022-02-01 07:37:11 Created table schema_migrations

julia> SearchLight.Migration.status()
[ Info: 2022-02-01 07:37:20 SELECT version FROM schema_migrations ORDER BY version DESC
|   | Module name & status                     |
|   | File name                                |
|---|------------------------------------------|
|   |                 CreateTableCourses: DOWN |
| 1 | 2022020115190055_create_table_courses.jl |

julia> SearchLight.Migration.last_up()
[ Info: 2022-02-01 07:37:29 SELECT version FROM schema_migrations ORDER BY version DESC
[ Info: 2022-02-01 07:37:29 CREATE TABLE courses (id INTEGER PRIMARY KEY , title TEXT  , authors TEXT  , year INTEGER (4) , rating TEXT  , categories TEXT  , description TEXT  , cost FLOAT (1000) )
[ Info: 2022-02-01 07:37:29 CREATE  INDEX courses__idx_title ON courses (title)
[ Info: 2022-02-01 07:37:29 CREATE  INDEX courses__idx_authors ON courses (authors)
[ Info: 2022-02-01 07:37:29 CREATE  INDEX courses__idx_categories ON courses (categories)
[ Info: 2022-02-01 07:37:29 CREATE  INDEX courses__idx_description ON courses (description)
[ Info: 2022-02-01 07:37:29 INSERT INTO schema_migrations VALUES ('2022020115190055')
[ Info: 2022-02-01 07:37:29 Executed migration CreateTableCourses up

We have now successfully completed the migrations. If you were to make a change to the schema, you would need to re-run the commands above for those database changes to take effect.

The last step in this process is to define our model. This will allow us to create objects in Julia code and then save them to the database we just defined. We need to navigate to app/resources/courses/Courses.jl or the equivalent path to make these final updates:

module Courses

import SearchLight: AbstractModel, DbId
import Base: @kwdef

export Course

@kwdef mutable struct Course <: AbstractModel
  id::DbId = DbId()
  title::String = ""
  authors::String = ""
  year::Int = 0
  rating::String = ""
  categories::String = ""
  description::String = ""
  cost::Float64 = 0.0
end

end

Again, this should be the same as the content you previously defined. To make sure this worked, we can do:

julia> using Courses
[ Info: 2022-02-01 07:43:51 Precompiling Courses [top-level]

and then try creating a course via:


julia> c = Course(title = "Web dev with Genie.jl", authors="Logan Kilpatrick")
Course
| KEY                 | VALUE                 |
|---------------------|-----------------------|
| authors::String     | Logan Kilpatrick      |
| categories::String  |                       |
| cost::Float64       | 0.0                   |
| description::String |                       |
| id::DbId            | NULL                  |
| rating::String      |                       |
| title::String       | Web dev with Genie.jl |
| year::Int64         | 0                     |

We have successfully created our first object! But it is not saved to the database right away. We can verify this by doing:

julia> ispersisted(c)
false

so we need to run:

julia> save(c)
[ Info: 2022-02-01 07:47:04 INSERT  INTO courses ("title", "authors", "year", "rating", "categories", "description", "cost") VALUES ('Web dev with Genie.jl', 'Logan Kilpatrick', 0, '', '', '', 0.0) 
[ Info: 2022-02-01 07:47:04 ; SELECT CASE WHEN last_insert_rowid() = 0 THEN -1 ELSE last_insert_rowid() END AS LAST_INSERT_ID
true

and now the course is saved! But to really test this out, we need the user to be able to create a course. Let’s head back to routes.jl and enable that:

using Genie, Genie.Router, Genie.Renderer.Html, Genie.Requests
using Courses

form = """
<form action="/" method="POST" enctype="multipart/form-data">
  <input type="text" name="name" value="" placeholder="What's the course name?" />
  <input type="text" name="author" value="" placeholder="Who is the course author?" />

  <input type="submit" value="Submit" />
</form>
"""

route("/") do
  html(form)
end

route("/", method = POST) do
  c = Course(title=postpayload(:name, "Placeholder"), authors=postpayload(:author, "Placeholder"))
  save(c)
  "Course titled $(c.title) created successfully!"
end

We started by defining a simple HTML form (nothing new or exciting here), then, we made it so the default route / renders the HTML form. Lastly, we create another route for the / URL, but specifically for the POST method. Inside that route, we create a new course by pulling the info we want from the form out of the payload via postpayload.

You can try this by navigating back to: http://127.0.0.1:8000

Screen-Shot-2022-02-01-at-8.11.38-AM

You can try and enter some of the details and then press submit. To make sure the submissions worked, you can do:

julia> all(Course)
[ Info: 2022-02-01 08:10:19 SELECT "courses"."id" AS "courses_id", "courses"."title" AS "courses_title", "courses"."authors" AS "courses_authors", "courses"."year" AS "courses_year", "courses"."rating" AS "courses_rating", "courses"."categories" AS "courses_categories", "courses"."description" AS "courses_description", "courses"."cost" AS "courses_cost" FROM "courses" ORDER BY courses.id ASC
┌ Warning: 2022-02-01 08:10:19 Unsupported SQLite declared type INTEGER (4), falling back to Int64 type
└ @ SQLite ~/.julia/packages/SQLite/aDggE/src/SQLite.jl:416
┌ Warning: 2022-02-01 08:10:19 Unsupported SQLite declared type FLOAT (1000), falling back to Float64 type
└ @ SQLite ~/.julia/packages/SQLite/aDggE/src/SQLite.jl:416
3-element Vector{Course}:
 Course
| KEY                 | VALUE                 |
|---------------------|-----------------------|
| authors::String     | Logan Kilpatrick      |
| categories::String  |                       |
| cost::Float64       | 0.0                   |
| description::String |                       |
| id::DbId            | 1                     |
| rating::String      |                       |
| title::String       | Web dev with Genie.jl |
| year::Int64         | 0                     |

 Course
| KEY                 | VALUE       |
|---------------------|-------------|
| authors::String     | Logan K     |
| categories::String  |             |
| cost::Float64       | 0.0         |
| description::String |             |
| id::DbId            | 2           |
| rating::String      |             |
| title::String       | Test course |
| year::Int64         | 0           |

which should show that the entries were saved in the database.

Wrapping up ?

Wow, that was a lot. We covered a tremendous amount of ground in this single tutorial.

With that said, there is even more to learn about Genie. I highly suggest checking out the docs here, which has lots more tutorials on topics like REST API’s, Authentication, and much more.

Getting help with Genie.jl ?

If you run into issues with this tutorial or when using Genie, please post a question on Stack Overflow with the genie.jl and julia tag or on the Julia Discourse. After that, feel free to tweet the link to the question at me and I will do my best to help: https://twitter.com/OfficialLoganK.