Tag Archives: Julia

Julia syntax features

Re-posted from: http://grollchristian.wordpress.com/2014/07/20/julia-syntax-features/

In one of my last posts I already tried to point out some advantages of Julia. Two of the main arguments are quite easily made: Julia is comparatively fast and free and open source. In addition, however, Julia also has a very powerful and expressive syntax compared to other programming languages, but this advantage is maybe less obvious to understand. Hence, I recently gave a short talk where I tried to extend a little bit on this point, while simultaneously also showing some of the convenient publishing feature of the IJulia backend. I thought I’d just share the outcome with you, just in case that anyone else could use the slides to convince some people of Julia’s powerful syntax. In addition to the slides, you can also access the presentation rendered as ijulia notebook here.

Filed under: Julia Tagged: ijulia, Julia, slides

Web scraping with Julia and PhantomJS

By: Alvaro "Blag" Tejada Galindo

Re-posted from: http://blagrants.blogspot.com/2014/07/web-scrapping-with-julia-and-phatomjs.html

As I have been reading some PhantomJS books and I’m always looking to develop something nice using Julia…I thought that integrate them would be an awesome idea -;)

I thought about Twitter and the hashtags…wouldn’t it be nice to write a PhantomJS script to webscrape Twitter and get all the hashtags that I have used?

For this particular script…I’m taking the hashtags from the first 5 Twitter pages linked to my profile…

Hashtags.js
var system = require('system'); var webpage = require('webpage').create(); webpage.viewportSize = { width: 1280, height: 800 }; webpage.scrollPosition = { top: 0, left: 0 }; var userid = system.args[1]; var profileUrl = "http://www.twitter.com/" + userid; webpage.open(profileUrl, function(status) { if (status === 'fail') { console.error('webpage did not open successfully'); phantom.exit(1); } var i = 0, top, queryFn = function() { return document.body.scrollHeight; }; setInterval(function() { top = webpage.evaluate(queryFn); i++; webpage.scrollPosition = { top: top + 1, left: 0 }; if (i >= 5) { var twitter = webpage.evaluate(function () { var twitter = []; forEach = Array.prototype.forEach; var tweets = document.querySelectorAll('[data-query-source="hashtag_click"]'); forEach.call(tweets, function(el) { twitter.push(el.innerText); }); return twitter; }); twitter.forEach(function(t) { console.log(t); }); phantom.exit(); } }, 3000); });

Hashtags.js

var system = require('system');

var webpage = require('webpage').create();
webpage.viewportSize = { width: 1280, height: 800 };
webpage.scrollPosition = { top: 0, left: 0 };

var userid = system.args[1];
var profileUrl = "http://www.twitter.com/" + userid;

webpage.open(profileUrl, function(status) {
 if (status === 'fail') {
  console.error('webpage did not open successfully');
  phantom.exit(1);
 }
 var i = 0,
 top,
 queryFn = function() {
  return document.body.scrollHeight;
 };
 setInterval(function() {
  top = webpage.evaluate(queryFn);
  i++;
   
  webpage.scrollPosition = { top: top + 1, left: 0 };

  if (i >= 5) {
   var twitter = webpage.evaluate(function () {
    var twitter = [];
    forEach = Array.prototype.forEach;
    var tweets = document.querySelectorAll('[data-query-source="hashtag_click"]');
    forEach.call(tweets, function(el) {
     twitter.push(el.innerText);
    });
    return twitter;
   });

   twitter.forEach(function(t) {
    console.log(t);
   });

   phantom.exit();
  }
}, 3000);
});

If we run this…we’re going to have this output…

Now…what I want to do with this information…is to send it to Julia…and get the most used hashtags…so I will summarize them and then get rid of the ones that only appear once…

Let’s see the Julia code…

Twitter_Hashtags.jl
tweets = readall(`phantomjs Hashtags.js Blag`) tweets = split(tweets,"\n") hashtags = Dict() for hash in tweets try hashtags[hash] += 1 catch e hashtags[hash] = 1 end end filter!((k,v)->v>1,hashtags) for (k,v) in hashtags println("$k has been mentioned $v times") end

Twitter_Hashtags.jl

tweets = readall(`phantomjs Hashtags.js Blag`)
tweets = split(tweets,"\n")
hashtags = Dict()
for hash in tweets
 try
  hashtags[hash] += 1
 catch e
  hashtags[hash] = 1
 end
end

filter!((k,v)->v>1,hashtags)

for (k,v) in hashtags
 println("$k has been mentioned $v times")
end

When we run this code…we’re going to have this output…

I still don’t know how to sort Dicts in Julia…so bear with me -:)

Anyway…by looking at the output…we can have my top 3 hashtags -;)

#LeapMotion ==> 14 times
#Flare3D ==> 11 times
#DevHangout ==> 8 times

Hope you like this and see you next time -:)

Greetings,

Blag.
Development Culture.

String Interpolation for Fun and Profit

By: randyzwitch - Articles

Re-posted from: http://randyzwitch.com/string-interpolation-julia/

In a previous post, I showed how I frequently use Julia as a ‘glue’ language to connect multiple systems in a complicated data pipeline. For this blog post, I will show two more examples where I use Julia for general programming, rather than for computationally-intense programs.

String Building: Introduction

The Strings section of the Julia Manual provides a very in-depth treatment of the considerations when using strings within Julia. For the purposes of my examples, there are only three things to know:

Strings are immutable within Julia and 1-indexed
Strings are easily created through the a syntax familiar to most languages:
```
julia> authorname = "randy zwitch"
"randy zwitch"
```
String interpolation is easiest done using dollar-sign notation. Additionally, parenthesis can be used to avoid symbol ambiguity:
```
julia> interpolated = "the author of this blog post is $(authorname)"
"the author of this blog post is randy zwitch"
```

If you are using large volumes of textual data, you’ll want to pay attention to the difference between the various string types that Julia provides (UTF8/16/32, ASCII, Unicode, etc), but for the purposes of this blog post we’ll just be using the ASCIIString type by not explicitly declaring the string type and only using ASCII characters.

Example 1: Repetitive Queries

As part of my data engineering responsibilities at work, I often get requests to pull a sample of every table in a new database in our Hadoop cluster. This type of request is usually from the business owner, who wants to evaluate the data set has been imported correctly, but doesn’t actually want to write any sort of queries. So using the ODBC.jl package, I repeatedly do the same ‘select * from <tablename>’ query and save to individual .tab files:While the query is simple, writing/running this hundreds of times would be a waste of effort. So with a simple loop over the array of tables, I can provide a sample of hundreds of tables in .tab files with five lines of code.

Example 2: Generating Query Code

In another task, I was asked to join a handful of Hive tables, then transpose the table from “long” to “wide”, so that each id value only had one row instead of multiple. This is fairly trivial to do using CASE statements in SQL; the problem arises when you have thousands of potential row values to transpose into columns! Instead of getting carpal tunnel syndrome typing out thousands of CASE statements, I decided to use Julia to generate the SQL code itself:

The example here only repeats the CASE statements five times, which wouldn’t really be that much typing. However, for my actual application, the number of possible values was 2153, leading to a query result which was 8157 columns! Suffice to say, I’d still be writing that code if I decided to do it by hand.

Summary

Like my ‘glue language’ post, I hope this post has shown that Julia can be used for more than grunting about microbenchmark performance. Whereas I used to use Python for doing weird string operations like this, I’m finding that the dollar-sign syntax in Julia feels more comfortable for me than the Python string formatting mini-language (although that’s not particularly difficult either). So if you’ve been hesitant to jump into learning Julia because you think it’s only useful for doing Mandelbrot calculations or complex linear algebra, Julia is just as at-home doing quick general programming tasks as well.

juliabloggers.com

A Julia Language Blog Aggregator

Tag Archives: Julia

Julia syntax features

Web scraping with Julia and PhantomJS

String Interpolation for Fun and Profit

String Building: Introduction

Example 1: Repetitive Queries

Example 2: Generating Query Code

Summary

String Building: Introduction

Example 1: Repetitive Queries

Example 2: Generating Query Code

Summary

Related posts: