By: Kristoffer Carlsson on Kristoffer Carlsson

Re-posted from: http://kristofferc.github.io/post/julia_issue/

As most of us in the Julia community know, Julia has an issue.
In fact, looking at the Julia repository we see that
there are (at time of writing) 11865 issues, where 1872 are open and 9993 are closed.
An interesting question to ask is:

• How has the ratio between open and closed issues varied over the development of Julia? And how about for pull requests (PRs)?

In this post, the aim is to answer the question above using the data that can be scraped from the GitHub repo.

Getting the data

A large amount of data for a repository hosted on GitHub can be found via the GitHub API.
The GitHub.jl package provides a convenient
interface to communicate with this API from Julia.

Getting the data for the issues (and PRs) is as simple as:

import GitHub
myauth = GitHub.authenticate(ENV["GITHUB_AUTH"])    
repo = GitHub.repo("JuliaLang/julia")
myparams = Dict("state" => "all", "per_page" => 100, "page" => 1);
issues, page_data = GitHub.issues(repo; params = myparams, auth = myauth)

where I have created a “GitHub access token” and saved it in the environment variable GITHUB_AUTH. Note here that the function name GitHub.issues is a bit of a misnomer.
What is returned is actually both issues and PRs.
The variable issues now contains a Vector of all the issues and PRs made to the Julia repo.
Each element in the vector is an Issue, which is a struct containing fields
corresponding to the keys of the returned JSON from the GitHub REST API.
In order to not have to rescrape the data every time Julia is started, it would be nice to
store it to disk. The standard way of storing Julia data is by using the *JLD.jl package.
Unfortunately, JLD.jl has some problems handling Nullable’s (see this issue).
However, there is an unregistered package called JLD2.jl
that does support Nullable’s. The code below uses JLD2 to save the issues to a .jld file
and then reload them again:

using JLD2

function save_issues(issues)
    f = jldopen("issues.jld", "w")
    write(f, "issues", issues)
    close(f)
end

function load_issues()
    f = jldopen("issues.jld", "r")
    issues = read(f, "issues")
    close(f)
    return issues
end

I put up the resulting .jld file here if you don’t feel like doing the scraping yourself.

Digression: a DateVector

In this post we will deal with Vector’s that are naturally indexed by
Date’s instead of standard integers starting at 1. Therefore, using the powerful
AbstractVector interface
we easily create a Vector that can be indexed with single Date’s ranges consisting of Date’s etc.

The implementation below should be fairly self explanatory. We create the DateVector struct
wrapping a Vector and two Date’s (the start and end date) and define a minimum amount of methods needed.
Vector’s with non conventional indices are still quite new in Julia so not everything work
perfectly with them. Here, we just implement the functionality needed
to set and retrieve data using indexing with Dates.

struct DateVector{T} <: AbstractVector{T}
    v::Vector{T}
    startdate::Date
    enddate::Date

    function DateVector(v::Vector{T}, startdate::Date, enddate::Date) where T
        len = (enddate - startdate) ÷ Dates.Day(1) + 1
        if length(v) != len
            throw(ArgumentError("length of vector v $(length(v)) not equal to date range $len"))
        end
        new{T}(v, startdate, enddate)
    end
end

Base.endof(dv::DateVector) = dv.enddate
Base.indices(dv::DateVector) = (dv.startdate:dv.enddate,)
Base.getindex(dv::DateVector, date::Date) = dv.v[(date - dv.startdate) ÷ Dates.Day(1) + 1]
Base.setindex!(dv::DateVector, v, date::Date) = dv.v[(date - dv.startdate) ÷ Dates.Day(1) + 1] = v
Base.checkindex(::Type{Bool}, d::Range{Date}, v::Range{Date}) = length(v) == 0 || (first(v) in d && last(v) in d)
Base.Array(dv::DateVector) = dv.v

The DateVector can now be seen in action by for example indexing into it
with a Range of Date’s:

julia> v = rand(10);

julia> dv = DateVector(v, Date("2015-01-01"), Date("2015-01-10"));

julia> dv[Date("2015-01-02"):Date("2015-01-05")]
4-element Array{Float64,1}:
 0.299136
 0.898991
 0.0626245
 0.585839

julia> v[2:5]
4-element Array{Float64,1}:
 0.299136
 0.898991
 0.0626245
 0.585839

Total number of opened and closed issues / PRs over time

For a given issue we can check the time it was created, what state it is in (open / closed),
what time it was eventually closed and if it is, in fact, a pull request and not an issue:

julia> get(issues[1].created_at)
2017-05-23T18:09:45

julia> get(issues[1].state)
"open"

julia> isnull(issues[1].closed_at)
true

julia> isnull(issues[1].pull_request) # pull_request is null so this is an issue
true

It is now quite easy to write a function that takes an issue and returns two Date-intervals, the first
for when the issue was opened, and the second for when it was closed up until today.
If the issue is still open, we make sure to return an empty interval for the closed period.

function open_closed_range(issue)
    opened = Date(get(issue.created_at))
    if get(issue.state) == "closed"
        closed = Date(get(issue.closed_at))
    else
        closed = Date(now())
    end
    return opened:closed, (closed + Dates.Day(1)):Date(now())
end

As an example, we can test this function on an issue:

julia> open_closed_range(issues[200])
(2017-05-13:1 day:2017-05-14, 2017-05-15:1 day:2017-05-29)

So, here we see that the issue was opened between 2017-05-13 and 2017-05-14 (and then got closed).
Now, we can simply create two DateVector’s, one that will contain the total number of opened
issues and the second the total number of closed issues / PRs for a given date

function count_closed_opened(PRs::Bool, issues)
    min_date = Date(get(minimum(issue.created_at for issue in issues)))
    days_since_min_date = (Date(now()) - min_date) ÷ Dates.Day(1) + 1
    closed_counter = DateVector(zeros(Int, days_since_min_date), min_date, Date(now()))
    open_counter = DateVector(zeros(Int, days_since_min_date), min_date, Date(now()))

    for issue in filter(x -> !isnull(x.pull_request) == PRs, issues)
        open_range, closed_range = open_closed_range(issue)
        closed_counter[closed_range] += 1
        open_counter[open_range] += 1
    end

    return open_counter, closed_counter
end

For a given date, the number of opened and closed issues are now readily available:

julia> opened, closed = count_closed_opened(false, issues);

julia> opened[Date("2016-01-1")]
288

julia> closed[Date("2016-01-1")]
5713

We could plot these now using one of our favorite plotting packages. I am personally a LaTeX fan and one of the popular
plotting packages for LaTeX is pgfplots. PGFPlotsX.jl is a Julia package that makes
it quite easy to interface with pgfplots so this is what I used here. The total number of open and
closed issues for different dates is shown below.

We can see that in the early development of Julia, PRs was not really used.
Also, the number of closed issues and PRs seem to grow at approximately the same rate.
A noticeable difference is in the number of opened issues and PRs. Open PRs accumulate
significantly slower than the number of open issues. This is likely because an open PR
become stale quite quickly while issues can take a long time to fix, or does not really have
a clear actionable purpose.

Let’s plot the ratio between open and closed issues / PRs:

To reduce some of the noise, I started the plot at 2013-06-01. The ratio between open to closed issues seems to slowly increase while for PRs, the number have stabilized at around 0.05.

Conclusions

Using the GitHub API it is quite easy to do some data anlysis on a repo.
It is hard to say how much actual usefulness can be extracted from the data here
but sometimes it is fun to just hack on data.
Possible future work could be to do the same analysis here but for other programming language repos.
Right now, looking at e.g. the Rust repo they have an open to closed PR ratio of 63 / 21200 ≈ 0.003 which is 20 times lower than Julia. Does this mean that the Julia community need to be better at reviewing PRs to make sure they eventually get merged / closed? Or is the barrier to open a PR to Rust higher so that only PRs with high success of merging gets opened?

Cambridge, MA – Julia ranks as one of the top 10 programming languages developed on GitHub as measured by the number of stars and forks.

GitHub users star a repository in order to show appreciation and create a bookmark for easy access. Programmers fork a repository in order to add features or fix bugs and contribute to the project.

Julia ranks #10 in GitHub stars and #8 in forks among programming languages developed on GitHub.

Top GitHub Programming Languages

Ranked by Number of GitHub Stars

About Julia Computing and Julia

Julia Computing (JuliaComputing.com) was founded in 2015 by the co-creators of the open source Julia language to develop products and provide support for businesses and researchers who use Julia.

Julia is the fastest modern high performance open source computing language for data, analytics, algorithmic trading, machine learning and artificial intelligence. Julia combines the functionality and ease of use of Python, R, Matlab, SAS and Stata with the speed of Java and C++. Julia delivers dramatic improvements in simplicity, speed, capacity and productivity. With more than 1 million downloads and +161% annual growth, Julia adoption is growing rapidly in finance, energy, robotics, genomics and many other fields.

1. Julia is lightning fast. Julia provides speed improvements up to
   1,000x for insurance model estimation, 225x for parallel
   supercomputing image analysis and 11x for macroeconomic modeling.

2. Julia is easy to learn. Julia’s flexible syntax is familiar and
   comfortable for users of Python, R and Matlab.

3. Julia integrates well with existing code and platforms. Users of
   Python, R, Matlab and other languages can easily integrate their
   existing code into Julia.

4. Elegant code. Julia was built from the ground up for
   mathematical, scientific and statistical computing, and has advanced
   libraries that make coding simple and fast, and dramatically reduce
   the number of lines of code required – in some cases, by 90%
   or more.

5. Julia solves the two language problem. Because Julia combines
   the ease of use and familiar syntax of Python, R and Matlab with the
   speed of C, C++ or Java, programmers no longer need to estimate
   models in one language and reproduce them in a faster
   production language. This saves time and reduces error and cost.

Julia users, partners and employers looking to hire Julia programmers in 2017 include: Google, Apple, Amazon, Facebook, IBM, Intel, Microsoft, BlackRock, Capital One, PwC, Ford, Oracle, Comcast, DARPA, Moore Foundation, Federal Reserve Bank of New York (FRBNY), UC Berkeley Autonomous Race Car (BARC), Federal Aviation Administration (FAA), MIT Lincoln Labs, Nobel Laureate Thomas J. Sargent, Brazilian National Development Bank (BNDES), Conning, Berkery Noyes, BestX, Path BioAnalytics, Invenia, AOT Energy, AlgoCircle, Trinity Health, Gambit, Augmedics, Tangent Works, Voxel8, Massachusetts General Hospital, NaviHealth, Farmers Insurance, Pilot Flying J, Lawrence Berkeley National Laboratory, National Energy Research Scientific Computing Center (NERSC), Oak Ridge National Laboratory, Los Alamos National Laboratory, Lawrence Livermore National Laboratory, National Renewable Energy Laboratory, MIT, Caltech, Stanford, UC Berkeley, Harvard, Columbia, NYU, Oxford, NUS, UCL, Nantes, Alan Turing Institute, University of Chicago, Cornell, Max Planck Institute, Australian National University, University of Warwick, University of Colorado, Queen Mary University of London, London Institute of Cancer Research, UC Irvine, University of Kaiserslautern, University of Queensland.

Julia is being used to: analyze images of the universe and research dark matter, drive parallel supercomputing, diagnose medical conditions, provide surgeons with real-time imagery using augmented reality, analyze cancer genomes, manage 3D printers, pilot self-driving racecars, build drones, improve air safety, manage the electric grid, provide analytics for foreign exchange trading, energy trading, insurance, regulatory compliance, macroeconomic modeling, sports analytics, manufacturing, and much, much more.

By: Tamás K. Papp

Re-posted from: https://tamaspapp.eu/post/latex-math-increment/

I am working on an appendix for a paper that uses MCMC, and I decided to document some change of varible calculations in the interest of reproducibility (they are quite complex, because of multivariate determinants). But how can I typeset them nicely in $\LaTeX$?

\mathtt{target} += J_f

gives
$$
\mathtt{target} += J_f
$$
which is to be expected, as + is a binary operator and = is a relation, so $\LaTeX$ is not expecting them to show up this way.

We can remedy this as

\mathtt{target} \mathrel{+}= J_f

which shows up as
$$
\mathtt{target} \mathrel{+}= J_f
$$
which is an improvement, but is still not visually appealing.

Making the + a bit smaller with

\mathrel{\raisebox{0.19ex}{$\scriptstyle+$}}=}

yields
$$
\mathtt{target} \mathrel{\raise{0.19ex}{\scriptstyle+}} = J_f
$$
which looks OK enough to preclude further tweaking. Note that MathJax does not support \raisebox, but you can use

\mathrel{\raise{0.19ex}{\scriptstyle+}} = J_f

which renders the as above.