Author Archives: Alvaro "Blag" Tejada Galindo

Julia, data analysis’s little sister…meets SAP HANA

By: Alvaro "Blag" Tejada Galindo

Re-posted from: http://blagrants.blogspot.com/2015/10/julia-data-analysiss-little-sistermeets.html

This post was originally posted on Julia, data analysis’s little sister…meets SAP HANA.


Julia is not that young right now…as it first appeared on 2012 -:) It is a high-level dynamic programming language designed to address the requirements of high-performance numerical and scientific computing while also being effective for general purpose programming.
Woaw! That was a big description…so why should we care? Well…maybe because Julia was designed to be the language to rule them all…a language that can be used in any given situation…and without stopping to say if that’s true or not…I must say…Julia is really cool -:)
So…no example or demonstration would be complete if we didn’t hook it up with SAP HANA, right? So…let’s go and do it -;)
First, we need to create a Calculation View and call it “FLIGHTS_BY_CARRIER”. It will be composed of two tables, SCARR and SFLIGHT.
First, we need to create a Join object and link the table by MANDT and CARRID. From here select the following fields as output MANDT, CARRID, CARRNAME, PRICE and CURRENCY.
Then create an Aggregation object selecting the fields CARRNAME, PRICE (As Aggregated Column) and CURRENCY. Filter the CURRENCY field by ‘USD’.
Then create a Projection object and select only PRICE and CARRNAME.
On the Semantics object make sure to select “CROSS CLIENT” as the Default Client.
Now, switch to the SAP HANA Development View and create a new repository. Call it “Flights”.
Create a new “XS Engine” project and call it “Flights” as well. Link it to the “Flights” repository.
Create an empty “.xsapp” file.
Create a file called “.xsaccess” with the following code.
.xsaccess
{
"exposed" : true,
"authentication" : [ { "method" : "Basic" } ]
}

Finally create a file called “flights.xsodata” with the following code

flights.xodata
service {
"Blag/FLIGHTS_BY_CARRIER.calculationview" as "FLIGHTS" keys
generate local "Id";
}

Activate your project and launch it on your browser, you should see something like this…

The SAP HANA part is done…so we can move into the Julia part…

Go into your Julia environment and install the following packages

  • HTTPClient
  • Codecs
  • LightXML

You only need to do Pkg.add(“PackageName”) for each of them.

Then create a file called Julia_HANA_XML.jl on your favorite editor and copy the following code

Julia_HANA_XML.jl
using HTTPClient.HTTPC
using Codecs
using LightXML

credentials=encode(Base64,"SYSTEM:YourPassword")
Auth = bytestring(credentials)
Auth = "Basic " * Auth

flights=HTTPC.get("http://YourServer:8000/Flights/flights.xsodata/FLIGHTS",RequestOptions(headers=[("Authorization",Auth)]))

raw_text = takebuf_string(flights.body)
xdoc = parse_string(raw_text)
xroot = root(xdoc)

entry = get_elements_by_tagname(xroot,"entry")

for flights in entry
print(content(find_element(find_element(find_element(flights,"content"),"properties"),"CARRNAME")),": ",
content(find_element(find_element(find_element(flights,"content"),"properties"),"PRICE")),"n")
end

To run this application, simply go to your Julia environment and type

Include(“Julia_HANA_XML.jl”)

If you are wondering…why didn’t I use JSON instead of XML? Well…there’s an easy explanation for that -:) Somehow…the HTTPClient package have a problem using ?$format=json so I was forced to use XML instead…
Greetings,
Blag.
Development Culture.

Web scraping with Julia and PhantomJS

By: Alvaro "Blag" Tejada Galindo

Re-posted from: http://blagrants.blogspot.com/2014/07/web-scrapping-with-julia-and-phatomjs.html

As I have been reading some PhantomJS books and I’m always looking to develop something nice using Julia…I thought that integrate them would be an awesome idea -;)

I thought about Twitter and the hashtags…wouldn’t it be nice to write a PhantomJS script to webscrape Twitter and get all the hashtags that I have used?

For this particular script…I’m taking the hashtags from the first 5 Twitter pages linked to my profile…

Hashtags.js
var system = require('system');

var webpage = require('webpage').create();
webpage.viewportSize = { width: 1280, height: 800 };
webpage.scrollPosition = { top: 0, left: 0 };

var userid = system.args[1];
var profileUrl = "http://www.twitter.com/" + userid;

webpage.open(profileUrl, function(status) {
if (status === 'fail') {
console.error('webpage did not open successfully');
phantom.exit(1);
}
var i = 0,
top,
queryFn = function() {
return document.body.scrollHeight;
};
setInterval(function() {
top = webpage.evaluate(queryFn);
i++;

webpage.scrollPosition = { top: top + 1, left: 0 };

if (i >= 5) {
var twitter = webpage.evaluate(function () {
var twitter = [];
forEach = Array.prototype.forEach;
var tweets = document.querySelectorAll('[data-query-source="hashtag_click"]');
forEach.call(tweets, function(el) {
twitter.push(el.innerText);
});
return twitter;
});

twitter.forEach(function(t) {
console.log(t);
});

phantom.exit();
}
}, 3000);
});

If we run this…we’re going to have this output…

Now…what I want to do with this information…is to send it to Julia…and get the most used hashtags…so I will summarize them and then get rid of the ones that only appear once…
Let’s see the Julia code…
Twitter_Hashtags.jl
tweets = readall(`phantomjs Hashtags.js Blag`)
tweets = split(tweets,"\n")
hashtags = Dict()
for hash in tweets
try
hashtags[hash] += 1
catch e
hashtags[hash] = 1
end
end

filter!((k,v)->v>1,hashtags)

for (k,v) in hashtags
println("$k has been mentioned $v times")
end

When we run this code…we’re going to have this output…

I still don’t know how to sort Dicts in Julia…so bear with me -:)

Anyway…by looking at the output…we can have my top 3 hashtags -;)

#LeapMotion ==> 14 times
#Flare3D ==> 11 times
#DevHangout ==> 8 times

Hope you like this and see you next time -:)

Greetings,

Blag.
Development Culture.

LED – My first Julia package

By: Alvaro "Blag" Tejada Galindo

Re-posted from: http://blagrants.blogspot.com/2014/07/led-my-first-julia-package.html

So yesterday I was thinking about Julia and how easy people claim package development is…of course…I need to give it a try…

I wanted to start small and simple…so I build something useless mostly for fun and learning…

The LED Package simply writes an LED representation of any given number…

julia> Using LED

julia > ShowLED(12345)

   _  _       _  
| _| _| |_| |_
| |_ _| | _|

As simple as that…and it took me no more than 5 minutes to get it done…

So, I can confirm now that package development in Julia…is a piece of cake -:)

If everything was done nicely…you should be able to do…

julia > Pkg.add(“LED”)

otherwise…please do…

julia > Pkg.clone(“git@github.com:atejada/LED.jl.git”)

Of course…this was just an experiment…so of course I’m planning to put my mind into the work and come up with some nice and useful packages -;)

Greetings,

Blag.
Development Culture.