Revolutions: data science

data science

February 09, 2017

Job trends for R and Python

When we last looked at job trends from indeed.com, job listings for "R statistics" were on the rise but were still around half the volume of listings for "SAS statistics". Three-and-a-half years later, R has overtaken SAS in job listings for "statistics".

I added Python to the search this time; job listings for "Python statistics" have risen at a similar rate to those for R, but with a somewhat higher volume for R.

Since data science is popular job role these days, let's do the same search for "data scientist":

For "data scientist" jobs, R and Python track very closely, with Python just edging out R in the past few months. This is most likely because R and Python (but unlike SAS) appear together in many data scientist job listings.

You can explore other job titles at indeed.com. (And thanks to reader SK for the suggestion to revisit these searches!)

Posted by David Smith at 14:32 in data science, popularity, python, R | Permalink | Comments (0)

January 31, 2017

Data Science Virtual Machine updated, now includes RStudio, JuliaPro

The Windows edition of the Data Science Virtual Machine (DSVM) was recently updated on the Azure Marketplace. This update upgrades some existing components and adds some new ones as well.

You now have your choice of integrated development environment to use with R. RStudio Desktop is now included in the Data Science Virtual Machine image — no need to install it manually. And R Tools for Visual Studio has been upgraded to Version 0.5.

Microsoft R Server has also been upgraded to version 9.0. This includes the latest R 3.2.2 language engine, the RevoScaleR package for big-data support in R, and also the new MicrosoftML package with several new, high-performance machine learning techniques. If you choose a GPU-enabled NC classs instance for your DSVM, the MicrosoftML package can make use of the GPUs for even more performance.

The DSVM also supports the Julia language with the inclusion of the JuliaPro distribution. This includes the Julia compiler and popular packages, a debugger, and the Juno IDE. If you're new to Julia, the blog post Julia – A Fresh Approach to Numerical Computing provides an introduction.

There are also improved Deep Learning capabilities, with the latest version of the Microsoft Cognitive Toolkit (formerly called CNTK). And if you use a GPU-enabled instance, the Deep Learning Toolkit extension provides GPU-enabled builds of the Cognitive Toolkit, mxNet, and TensorFlow.

If you want to try out the Data Science Virtual Machine, the blog post linked below provides links to the documentation and several tutorials to get you started, along with information about the Linux edition of the DSVM.

Cortana Intelligence and Machine Learning Blog: New Year & New Updates to the Windows Data Science Virtual Machine

Posted by David Smith at 12:15 in data science, Microsoft, python, R | Permalink | Comments (2)

January 26, 2017

Kung Fu R

A great way to hone your skills as a data scientist is to pick a topic you're passionate about, find some data related to it, and analyze the heck out of it. Jim Vallandingham is clearly passionate about old Kung Fu movies — particularly those from the Shaw Brothers Studio — and has used R to analyze data the studio's oeuvre: 260 films over 22 years.

The complete R code behind the analysis is included in the post (and you can also find it as an R Markdown document here). Some interesting notes include:

Use of the tidyjson package to parse the table scraped from the list of Shaw Brothers Martial Arts Films on Letterboxd.
Many applications of the ggplot2 package to visualize directors, actors, film length, Letterboxd watches and likes, and other data about the films.
Using the tidytext package to find the common words used in film titles.
Using the igraph package to create a network digram showing actors who appear in films together, shown below.

You can even iteract with this network, using Jim's Shaw Brothers Actors Network Visualization. The interactive version is based on Javascript and D3, and this tutorial explains how it was made. But for the complete analysis of the Kung Fu movies, follow the link below.

Jim Vallandingham: A Data Driven Exploration of Kung Fu Films

Posted by David Smith at 14:33 in data science, R | Permalink | Comments (0)

January 16, 2017

The fivethirtyeight R package

Andrew Flowers, data journalist and contributor to FiveThirtyEight.com, announced at last weeks' RStudio conference the availability of a new R package containing data and analyses from some of their data journalism features: the fivethirtyeight package. (Andrew's talk isn't yet online, but you can see him discuss several of these stories in his UseR!2016 presentation.) While not an official product of the FiveThirtyEight editorial team, it was developed by Albert Y. Kim, Chester Ismay and Jennifer Chunnunder their guidance. Their motivation for producing the package was to provide a resource for teaching data science:

We are involved in statistics and data science education, in particular at the introductory undergraduate level. As such, we are always looking for data sets that balance being

Rich enough to answer meaningful questions with, real enough to ensure that there is context, and realistic enough to convey to students that data as it exists “in the wild” often needs processing.
Easily and quickly accessible to novices, so that we minimize the prerequisites to research.

The package includes data sets from dozens of data journalism stories, including stories about police killings in the USA, plane crashes, and even references to presidential candidates in hip-hop lyrics. There is also a complete worked analysis of performace of movies satisfying the Bechdel Test, presented as an Rmarkdown vignette.

The package is available for download now on CRAN, and you can find the latest development version on GitHub.

Posted by David Smith at 13:17 in data science, packages, R | Permalink | Comments (2)

January 09, 2017

What can we learn from StackOverflow data?

StackOverflow, the popular Q&A site for programmers, provides useful information to nearly 5 million programmers worldwide with its database of questions and answers — not to mention the additional comments that other programmers provide. (You might be interested in the architecture, based SQL Server 2016, required to deliver the 8.5 billion pages Stack Overflow served last year.) Since its inception, StackOverflow has has a policy of sharing all of this content under a Creative Commons license. This represents a rich trove of unstructured data for analysis, especially given that the database of 13 million questions, 21 million answers and 54 million comments (and growing) is easily accessible via StackExchange Data Explore, Kaggle and Google BigQuery.

Various data scientists have investigated this database, and learned some interesting things about programmers in the process. Here are a few examples, with links to the complete reports.

Sara Robinson analyzed the sentiment of Stack Overflow comments (based on phrases like "thank you" or "stupid") and found that R users seem to be the happiest, while Objective-C users are the angriest.


Tags with happiest comments		Tags with angriest comments

David Robinson analyzed developer job titles over time and found terms that were on the rise (for example "full stack") and terms on the decline (like "webmaster").

David Robinson also analyzed regional differences between programmers and compared the most popular tags used in San Francisco, London, Bangalore and New York. (R is the third most popular language in New York by this measure.)

Max Woolf analyzed the results of the StackOverflow Developer Survey and found this relationship between self-described skill level and salary.

Seen any other interesting analyses of the StackOverflow data, or done one yourself? Let us know in the comments.

Posted by David Smith at 12:38 in data science, R | Permalink | Comments (1)

January 06, 2017

Three reasons to learn R today

If you're just getting started with data science, the Sharp Sight Labs blog argues that R is the best data science language to learn today.

The blog post gives several detailed reasons, but the main arguments are:

R is an extremely popular (arguably the most popular) data progamming language, and ranks highly in several popularity surveys.
Learning R is a great way of learning data science, with many R-based books and resources for probability, frequentist and Bayesian statistics, data visualization, machine learning and more.
Python is another excellent language for data science, but with R it's easier to learn the foundations.

Once you've learned the basics, Sharp Sight also argues that R is also a great data science to master, even though it's an old langauge compared to some of the newer alternatives. Every tool has a shelf life, but R isn't going anywhere and learning R gives you a foundation beyond the language itself.

If you want to get started with R, Sharp Sight labs offers a data science crash course. You might also want to check out the Introduction to R for Data Science course on EdX.

Sharp Sight Labs: Why R is the best data science language to learn today, and Why you should master R (even if it might eventually become obsolete)

Posted by David Smith at 11:33 in data science, popularity, R | Permalink | Comments (0)

December 22, 2016

Take a Test Drive of the Linux Data Science Virtual Machine

If you've been thinking about trying out the Data Science Virtual Machine on Linux, but don't yet have an Azure account, you can now take a free test drive -- no credit card required! Just visit the Linux DSVM Marketplace page and click the blue button:

The Linux Data Science Virtual Machine includes all of the tools a modern data scientist needs, in one easy-to-launch package. With it, you can try exploring data with Apache Drill, train deep neural networks for computer vision with MXNet, develop AI applications with the Cognitive Toolkit, or create statistical models with big data in R with Microsoft R Server 9.0.

For details on how to test drive Linux Data Science Virtual Machine, follow the link below.

Cortana Intelligence and Machine Learning Blog: New Additions to the Data Science Virtual Machine – Test Drive, Community Forums, Deep Learning

Posted by David Smith at 10:46 in big data, data science, Microsoft, R | Permalink | Comments (0)

December 13, 2016

One Page R: A Survival Guide to Data Science with R

If you're looking to get started with data science in R, a great place to start is OnePageR by Graham Williams. (Graham is the creator of Rattle, author of Data Mining with Rattle and R, and Director of Data Science at Microsoft.) This free (CC-licensed) resource is a series of hands-on mini-chapters and associated R code, organized into four main topic areas:

Data Science: introductions to data science, data mining, literate programming, and the R language
Dealing with Data: Reading data files and open access data, basic explorations and visualizations, and two case studies
Building Models: with tutorials for many kinds of models, including association analysis, ensemble models, and multivariate adaptive regression splines
Advanced R and Analytics: with topics including writing functions, parallel processing, and text mining.

Example data files are provided for use in all chapters. There are also two very useful appendix chapters; an R Style Guide, and a guide to creating an R package.

OnePageR is a continual work in progress, and is regularly updated to incorporate advances in R and the R package ecosystem. To download the materials, follow the link below.

Togaware: OnePageR

Posted by David Smith at 13:44 in beginner tips, data science, R | Permalink | Comments (5)

December 02, 2016

Stylometry: Identifying authors of texts using R

Few people expect politicians to write every word they utter themselves; reliance on speechwriters and spokepersons is a long-established political practice. Still, it's interesting to know which statements are truly the politician's own words, and which are driven primarily by advisors or influencers.

Recently, David Robinson established a way of figuring out which tweets from Donald Trump's Twitter account came from him personally, as opposed to from campaign staff, whcih he verified by comparing the sentiment of tweets from Android vs iPhone devices. Now, Ali Arsalan Kazmi has used stylometric analysis to investigate the provenance of speeches by the Prime Minister of Pakistan.

By looking at the aspects of linguistic style (word/sentence length, frequency of word pairings, use of punctuation, etc.) of the speeches of the Prime Minister Nawaz Sharif, Ali found suggestions of at least 2 authors (and possibly more) behind the speeches. This is particularly apparent in this consensus network of appearances of 4-character sequences in speeches, which divides them into two clusters (of possibly differing authorship).

Ali used R and several packages to perform the analysis. These included the openNLP package to extract attributes from the speech data, the stylo package for stylometric analysis, the fpc package for the clustering, and the igraph package to visualize the clusters. The complete R script used for the analysis is available on Github.

For an overview of the analysis, check out this slide presentation by Ali, and for the complete details take a look at the blog post linked below.

A Blog On Data Analytics: How many Authors does the Prime Minister have for his speeches: A Stylometric Analysis

Posted by David Smith at 11:49 in current events, data science, packages, R | Permalink | Comments (0)

December 01, 2016

Using R to Gain Insights into the Emotional Journeys in War and Peace

by Wee Hyong Tok, Senior Data Scientist Manager at Microsoft

How do you read a novel in record time, and gain insights into the emotional journey of main characters, as they go through various trials and tribulations, as an exciting story unfolds from chapter to chapter?

I remembered my experiences when I start reading a novel, and I get intrigued by the story, and simply cannot wait to get to the last chapter. I also recall many conversations with friends on some of the interesting novels that I have read awhile back, and somehow have only vague recollection of what happened in a specific chapter. In this post, I’ll work through how we can use R to analyze the English translation of War and Peace.

War and Peace is a novel by Leo Tolstoy, and captures the salient points about Russian history from the period 1805 to 1812. The novel consists of the stories of five families, and captures the trials and tribulations of various characters (e.g. Natasha and Andre). The novel consists of about 1400 pages, and is one of the longest novels that have been written.

We hypothesize that if we can build a dashboard (shown below), this will allow us to gain insights into the emotional journey undertaken by the characters in War and Peace.

For example, using the dashboard below, and correlating it with the War and Peace story, we can see how Andrey falling in love with Natsha in Book 6 have created a green (positive) highlight in his life. We can also see how Natsha’s family bankruptcy in Book 6 overshadows her budding romance with Andrey. In addition, we can also observe how Pierre emotional journey stays red (negative), especially in Book 11-14, as he almost killed himself, and finally ending up in prison.

Building the War and Peace Visualization

In this post, I’ll show you how we use the d3heatmap package to build the War and Peace visualization. Let’s start with a dataset, where we used Azure Data Lake Analytics, and U-SQL extensions to analyze the English translation of War and Peace.

Continue reading "Using R to Gain Insights into the Emotional Journeys in War and Peace" »

Posted by Guest Blogger at 09:59 in data science, graphics, Microsoft, R | Permalink | Comments (2)

Revolutions

Daily news about using open source R for big data analysis, predictive modeling, data science, and visualization since 2008