We are pleased to announce that Julia Computing will be offering small grants to fund efforts to heighten diversity and inclusion within the Julia community, using funds graciously provided by the Alfred P. Sloan Foundation. Examples of qualifying projects or activities may include efforts to create teaching materials on or using Julia targeted at late entrants to computing, efforts to improve or translate documentation, and hosting Julia events that will target or benefit underrepresented populations in computing. Each proposal may request up to $4,000 USD, which may be put towards labor hours and resources required to carry out the proposed activity. The deadline to apply is Sunday, November 25, and award announcements will be made within one week.
Details
How this will work
Julia Computing plans to award $15,000 USD in grants for projects that will increase the diversity and inclusiveness of the Julia community. The total amount awarded may be adjusted up or down based on the number and quality of proposals received.
A maximum of $4,000 USD may be requested per project.
Work supported by these diversity and inclusion grants should be undertaken between December 2018 and June 2019.
The deadline to apply is Sunday, November 25.
Recipients will receive payment for the first half of their award at the beginning of the granting period.
Recipients will receive payment for the second half of their award halfway through the project period – either in March 2019 or possibly sooner, for shorter term projects.
Requirements of participation
Each recipient agrees to provide a maximum one page summary of the status of the project before the second half of the award is granted.
Each recipient agrees to write a blog post summarizing the project undertaken and its impact at the end of the project period.
How to apply
Please submit applications via Google form to https://goo.gl/forms/KyXoRIDmF4zStKYC3. There you will be prompted to upload a PDF addressing the following questions:
(1) What is the proposed activity?
(2) How will the proposed activity help make the Julia community more inclusive and ultimately more diverse?
(3) Will you be able to measure the impact of your project? If so, how? If not, why?
(4) What funds/resources does your project require and why? If you were granted 50% of the funds you’ve requested, would you be able to use those funds to do impactful work?
(5) Do you agree to provide a status update halfway through the grant period and a blog post at the end of the grant period?
In addressing the above questions, additional information to consider providing includes whether or not you have experience taking on activities like those you are proposing, additional benefits that your project will bring to the Julia community, and whether your project will focus on attracting or retaining people of diverse backgrounds to the Julia community (or both).
Criteria for evaluation
Proposals will be evaluated based on answers to the questions above. We recognize that efforts to incite systemic change can have long time horizons. Please consider applying if you have an idea for a project/activity that you believe will take an important step towards increasing the Julia community’s inclusiveness, even if you are not sure you will be able to measure the impact of your project on the diversity of our community on the timescale of only a few months.
If you have questions please contact Jane Herriman at [email protected]. We look forward to hearing from you!
Data nowadays are almost everywhere, often stored in as simple as traditional log books, to as complex as multi-connected-databases. Efficient collection of these datasets is crucial for analytics since data processing takes almost 50% of the overall workflow. An example where manual data collection can be automated is in the case of datasets published in the website, where providers are usually government agencies. For example in the Philippines, there is a website dedicated to Open Stat initiated by the Philippine Statistics Authority (PSA). The site hoards public datasets for researchers to use and are well prepared in CSV format, so consumers can simply download the file. Unfortunately, for some agencies this feature is not yet available. That is, users need to either copy-paste the data from the website, or request it to the agency directly (this also takes time). A good example of this is the seismic events of the Philippine Institute of Volcanology and Seismology (PHIVOLCS).
Data encoded in HTML can be parsed and saved into formats that’s workable for doing analyses (e.g. CSV, TSV, etc.). The task of harvesting and parsing data from the web is called web scraping, and PHIVOLCS’ Latest Seismic Events is a good playground for beginners. There are several tutorials available especially for Python (see this) and R (see this), but not much for Julia. Hence, this article is primarily for Julia users. However, this work introduces web tools as well – how to use it for inspecting the components of the website – which can be useful for non-Julia users.
Why Julia?
The creators of the language described it well in their first announcement (I suggest you read the full post): Why we created Julia? Here’s part of it:
We are greedy: we want more.
We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy. We want it interactive and we want it compiled.
(Did we mention it should be as fast as C?)
I used Julia in my master’s thesis for my MCMC simulations and benchmarked it against R. It took seconds in Julia while R took more than an hour (sampling over the posterior distribution). I could have optimized my R code using Rcpp (writting the performance-critical part in C++ to speed up and wrapped/call it in R), but I have no time for that. Hence, Julia solves the two-language problem.
Getting to know HTML
Since the data published in the websites are usually encoded as a table, it is therefore best to understand the structure of the HTML document before performing web scraping. HTML (Hypertext Markup Language) is a standardized system for tagging text files to achieve font, color, graphic, and hyperlink effects on World Wide Web pages [1]. For example, bold text in HTML is enclosed inside the <b> tag, e.g. <b>text</b>, the result is text. A webpage is a HTML document that can be structured in several ways, one possible case is as follows:
Scrapers must be familiar with the hierarchy of the HTML document as this will be the template for the frontend source code of every website. Following the structure of the above figure, data encoded in HTML table are placed inside the td (table data) tag, where td is under tr (table row), tr is under tbody (table body), and so on. td is the lowest level tag (sorting by hierarchy) from the figure above that can contain data. However, td can also take precedence over p (paragraph), a (hyperlink), b (bold), i (italic), span (span), and even div (division). So expect to encounter these under td as well.
As indicated in the figure, each HTML tag can have attributes, such as id and class. To understand how the two differ, consider id="yellow" and id="orange", these are unique identities (ids) of colors. These ids can be grouped into a class, e.g. class="colors". HTML tags are not required to have these attributes but are useful for adding custom styles and behavior when doing web development. This article will not dive into the details of the HTML document, but rather to give the reader a high level understanding. There are many resources available on the web, just google.
Inspecting the Source of the Website
In order to have an idea on the structure of the website, browsers such as Google Chrome and Mozilla Firefox include tools for Web Developers. For purpose of illustration but without loss of generality, this article will only scrape portion (why? read on and see the explanation below) of the September 2018 earthquake events. The web developer tools can be accessed from Tools > Web Developer in Firefox, and can be accessed from View > Developer in Google Chrome. The following video shows how to use the inspector tool of the Mozilla Firefox.
Scraping using Julia
To perform web scraping, Julia offers three libraries for the job, and these are Cascadia.jl, Gumbo.jl and HTTP.jl. HTTP.jl is used to download the frontend source code of the website, which then is parsed by Gumbo.jl into a hierarchical structured object; and Cascadia.jl provides a CSS selector API for easy navigation.
To start with, the following code will download the frontend source code of the PHIVOLCS’ Seismic Events for September 2018.
Extract the HTML source code and parsed it as follows:
Now to extract the header of the HTML table, use the Web Developer Tools for preliminary inspection on the components of the website. As shown in the screenshot below, the header of the table is enclosed inside the p tag of the td. Further, the p tag is of class auto-style33, which can be accessed via CSS selector by simply prefixing it with ., i.e. .auto-style33.
qres contains the HTML tags that matched the CSS selector’s query. The result is further cleaned by removing the tabs, spaces and line breaks via Regular Expressions, and is done as follows:
Having the header names, next is to extract the data from the HTML table. Upon inspection, the tds containing the data next to the header rows seem to have the following classes (see screenshot below): auto-style21 for first column (Date-Time), auto-style81 for second column (Latitude), auto-style80 for third and fourth columns (Longitude and Depth), auto-style74 for fifth column (Magnitude), and auto-style79 for sixth column (Location). Unfortunately, this is not consistent across rows (trs), and is therefore best not to use it with Cascadia.jl. Instead, use Gumbo.jl to navigate down the hierarchy of the Document Object Model of the HTML.
Starting with the table tag which is of class .MsoNormalTable (see screenshot below), the extraction proceeds down to tbody then to tr and finally to td.
The following code describes how parsing is done, read the comments:
Complete Code for PHIVOLCS’ September 2018 (Portion) Seismic Events
The September 2018 Seismic Events are encoded in two separate HTML tables of the same class, named MsoNormalTable. For purpose of simplicity, this article will only scrape the first portion (3rd-indexed, see line 14 below: tbody = html[3][1];) of the table (581 rows). The second portion (4th-indexed, change line 14 below to: tbody = html[4][1];) is left to the reader to try out and scrape it as well.
The following code wraps the parsers into functions, namely htmldoc (downloads and parses the HTML source code of the site), scraper (scrapes the downloaded HTML document), firstcolumn (logic for parsing the first column of the table, used inside scraper function).
Having the data, analyst can now proceed to do exploratory analyses, for example the following is the descriptive statistics of the variables:
describe is clever enough not only to not return mean and median for non-continuous variables, but also determine the min, max and nunique (number of uniques) for these variables (date and location).
End Note
I use Python primarily at work with BeautifulSoup as my go-to library for web scraping. Compared to Cascadia.jl and Gumbo.jl, BeautifulSoup offers comprehensive documentations and other resources that are useful for figuring out bugs, and understand how the module works. Having said, I hope this article somehow contributed to the documentation of the said Julia libraries. Further, I am confident to say that Cascadia.jl and Gumbo.jl are stable enough for the job.
Lastly, as a precaution to beginners, make sure to read the privacy policy (if any) of any website you want to scrape.