Friday 13 January 2017

Fast prototyping with Zeppelin, Spark & Scala

Summary


It´s been a while since I wrote anything here (again) but I don´t have much free time nowadays. Currently I´m taking some training on Scala and Spark, which, by the way, brings us here today.

A recipe for quick prototypes for Data Analysis: Scala + Spark + Zeppelin


If you remember well, I wrote some time ago about some personal learning projects I was working into, which basically picked stocks price information from the Web (using Spring Integration) and ran a couple of Spark analysis that were lately displayed in an AngularJS interface.

Nothing complicated at all, but rather verbose and time consuming to set up, specially if you just want to learn the subject.




With Apache Zeppelin, Scala and Spark, a prototype with basically the same functionality, would be reduced to these two code snippets in a Zeppelin notebook:


That reads the data, creates a Dataframe with the price and the moving average and declares a temporal table.
Later on, you can query it with a regular SQL query and its result set will be available as data source for a number of graph types:


This will produce this nice looking graph after just some minutes of coding, showing the index price and our friend, the Simple Moving Average (SMA):



The notebook I´m talking about is hosted in ZeppelinHub and you can access it here.

And that's it! No project setup, no unnecessary pain with Angular JS libraries and no horrible Java Spark API. If you are interested, you can find below the links to the articles explaining the topics seen here today and how were they implemented in the traditional way.

Cheers!

Resources

- Consuming a REST service in Scala
- Calculating Moving Averages with Spark MLLib
- Stock quotations as input for your Spring application
- Calculating and stock indicators with AngularJS and Spark 

3 comments:

  1. Hey Victor,

    I would like to showcase my Github code on blogger as well.
    https://gregperrypage.blogspot.ca/

    Would you direct me to the documentation that allows me to do that?

    I like your blog and I want to simulate the same look and feel, but I would like some direction.

    Greg

    ReplyDelete
    Replies
    1. gtfperry at gmail dot com

      Delete
    2. Hi Greg,
      Thanks for your comment! Well... actually I dont think there is any documentation per se about writing code-related stuff...
      I can share you what I use, however you might find other ways to do that, that are much better:
      - Post my code snippets as Github gists (see https://gist.github.com/discover for some samples)
      - Then you have the option of embedding them in your blog html (see the "embed" combo). Just copy the HTML code and paste it right away.

      Some colleagues of mine use Wordpress and, I have to say, the look and feel of their code blog is really cool...

      Hope it helps!
      Victor

      Delete