Revolutions
type: channel
description: Daily news about using open source R for big data analysis, predictive modeling, data science, and visualization since 2008
language: en-US
date: 2017-06-09T13:59:59-07:00
generator agent: www.typepad.com
items: Because it's Friday: Subway maps to scale
items: Schedule for useR!2017 now available
items: Run massive parallel R jobs cheaply with updated doAzureParallel package
items: How to create dot-density maps in R
items: In case you missed it: May 2017 roundup
items: Powe[R] BI: Free e-book on using R with Power BI
items: Because it's Friday: Disappearing Dots
items: Teach kids about R with Minecraft
items: Python and R top 2017 KDnuggets rankings
items: Watch presentations from R/Finance 2017

Because it's Friday: Subway maps to scale
type: item
description: Many subway maps are masterpieces of information design, but inevitably make compromises in geographic fidelity for clarity. Inspired by a viral post on Reddit, you can now find a collection of subway maps from around the world, morphing from their map format to their real-world layout. Here, for example, is London's: London's tube map — in map form — has always been lovely and clear, and you can easily see how much it deviates from geographic truth (and how much better it is for doing so). Compare it with New York's map: I've always found the NYC subway map one...
encoded: Many subway maps are masterpieces of information design, but inevitably make compromises in geographic fidelity for clarity. Inspired by a viral post on Reddit, you can now find a collection of subway maps from around the world, morphing from their map format to their real-world layout. Here, for example, is London's: London's tube map — in map form — has always been lovely and clear, and you can easily see how much it deviates from geographic truth (and how much better it is for doing so). Compare it with New York's map: I've always found the NYC subway map one of the most confusing to use as a traveler, and the fact that it hews so closely to geography probably has a lot to do with it. That's all from us for this week. Have a great weekend, and we'll be back on Monday!
subject: random
creator: David Smith
date: 2017-06-09T13:59:59-07:00

Schedule for useR!2017 now available
type: item
description: The full schedule of talks for useR!2017, the global R user conference, has now been posted. The conference will feature 16 tutorials, 6 keynotes, 141 full talks, and 86 lightning talks starting on July 5 in Brussels. That's a lot to fir into 4 days, but I'm especially looking forward to the keynote presentations: 20 years of CRAN (Uwe Ligges) Parallel Computation in R: What We Want, and How We (Might) Get It (Norm Matloff) Structural Equation Modeling: models, software and stories (Yves Roseel) Teaching data science to new useRs (Mine Cetinkaya-Rundel) Dose-response analysis: considering dose both as qualitative factor...
encoded: The full schedule of talks for useR!2017, the global R user conference, has now been posted. The conference will feature 16 tutorials, 6 keynotes, 141 full talks, and 86 lightning talks starting on July 5 in Brussels. That's a lot to fir into 4 days, but I'm especially looking forward to the keynote presentations: 20 years of CRAN (Uwe Ligges) Parallel Computation in R: What We Want, and How We (Might) Get It (Norm Matloff) Structural Equation Modeling: models, software and stories (Yves Roseel) Teaching data science to new useRs (Mine Cetinkaya-Rundel) Dose-response analysis: considering dose both as qualitative factor and quantitative covariate using R (Ludwig Hothorn) R tools for the analysis of complex heterogeneous data (Isabella Gollini) I'm also pleased to be attending with several of my Microsoft colleagues. You can find our talks below. Can you keep a secret? (Andrie deVries and Gabor Csardi) [A new package secret allows you to encrypt secrets using public key encryption.] Recommendations for coworker collaborators using R and the Office Graph API (David Smith) [Graph theory, R and Power BI] Deep Learning for Natural Language Processing in R (Angus Taylor) [Using the mxnet package to apply deep learning to text] I hope you can attend too! Registration is still open if you'd like to join in. You can find the complete schedule linked below. Sched: useR!2017
subject: events
subject: R
creator: David Smith
date: 2017-06-09T12:17:33-07:00

Run massive parallel R jobs cheaply with updated doAzureParallel package
type: item
description: At the EARL conference in San Francisco this week, JS Tan from Microsoft gave an update (PDF slides here) on the doAzureParallel package . As we've noted here before, this package allows you to easily distribute parallel R computations to an Azure cluster. The package was recently updated to support using automatically-scaling Azure Batch clusters with low-priority nodes, which can be used at a discount of up to 80% compared to the price of regular high-availability VMs. JS Tan using doAzureParallel #rstats package to run simulation on a cluster of 20 low-priority Azure VMs. Total cost: $0.02 #EARLConf2017 pic.twitter.com/Mpl3IUa9zY —...
encoded: At the EARL conference in San Francisco this week, JS Tan from Microsoft gave an update (PDF slides here) on the doAzureParallel package . As we've noted here before, this package allows you to easily distribute parallel R computations to an Azure cluster. The package was recently updated to support using automatically-scaling Azure Batch clusters with low-priority nodes, which can be used at a discount of up to 80% compared to the price of regular high-availability VMs. JS Tan using doAzureParallel #rstats package to run simulation on a cluster of 20 low-priority Azure VMs. Total cost: $0.02 #EARLConf2017 pic.twitter.com/Mpl3IUa9zY — David Smith (@revodavid) June 7, 2017 Using the doAzureParallel package is simple. First, you need to define the cluster you're going to use as a JSON file. (You can see an example on the right.) Here, you'll specify your Azure credentials, the size of the cluster, and the type of nodes (CPUs and memory) to use in the cluster. You can also specify here R packages (from CRAN and/or Github) to be pre-loaded onto each node, and the maximum number of simultaneous tasks to run on each node (for within-node parallelism). New to this update, the poolSize option allows you to specify the number of dedicated (standard) VM nodes to use, in addition to a number of low-priority nodes to use. Low-priority nodes can be pre-empted by the Azure system at any time, but are much cheaper to use. (Even if a node is pre-empted your parallel computation will be continue; it will just take a little longer with the reduced capacity.) You can even specify a minimum and maximum number of nodes of each class to use, in which case the cluster will automatically scale up and down according to either (your choice) the workload or the time of day (e.g. only expand the low-priority part of the cluster on weekends, when pre-emption is less likely). Once you've defined the parameters of your cluster, all you need to do is declare the cluster as a backend for the foreac…
subject: high-performance computing
subject: Microsoft
subject: packages
subject: R
creator: David Smith
date: 2017-06-08T15:41:11-07:00

How to create dot-density maps in R
type: item
description: Choropleths are a common approach to visualizing data on geographic maps. But choropleths — by design or necessity — aggregate individual data points into a single geographic region (like a country or census tract), which is all shaded a single colour. This can introduce interpretability issues (are we seeing changes in the variable of interest, or just population density?) and can fail to express the richness of the underlying data. For an alternative approach, take a look at the recent Culture of Insight blog post which provides a tutorial on creating dot-density maps in R. The chart below is based...
encoded: Choropleths are a common approach to visualizing data on geographic maps. But choropleths — by design or necessity — aggregate individual data points into a single geographic region (like a country or census tract), which is all shaded a single colour. This can introduce interpretability issues (are we seeing changes in the variable of interest, or just population density?) and can fail to express the richness of the underlying data. For an alternative approach, take a look at the recent Culture of Insight blog post which provides a tutorial on creating dot-density maps in R. The chart below is based on UK Census data. Each point represents 10 London residents, with the colour representing one of five ethnic categories. Now, the UK census only reports ethnic ratios on a borough-by-borough basis, so the approach here is to simulate the individual resident data (which is not available) by randomly distributing points across the borough following the reported distribution. In a way, this is suggesting a level of precision which isn't available in the source data, but it does provide a visualization of London's ethnic diversity that isn't confounded with the underlying population distribution. Follow the link below to the detailed blog post, which includes R code (in both base and ggplot2 graphics) for creating density dot-charts like these. Also be sure to check out the zoomable version of the chart at the top of the page, which used Microsoft's Deep Zoom Composer in conjunction with OpenSeadragon to provide the zooming capability. Culture of Insight: Building Dot Density Maps with UK Census Data in R
subject: graphics
subject: R
creator: David Smith
date: 2017-06-07T13:17:44-07:00

In case you missed it: May 2017 roundup
type: item
description: In case you missed them, here are some articles from May of particular interest to R users. Many interesting presentations recorded at the R/Finance 2017 conference in Chicago are now available to watch. A review of some of the R packages and projects implemented at the 2017 ROpenSci Unconference. An example of applying Bayesian Learning with the "bnlearn" package to challenge stereotypical assumptions Data from the Billboard Hot 100 chart used to find the most popular words in the titles of pop hits. Microsoft R Open 3.4.0 is now available for Windows, Mac and Linux. How to use the "tweenr"...
encoded: In case you missed them, here are some articles from May of particular interest to R users. Many interesting presentations recorded at the R/Finance 2017 conference in Chicago are now available to watch. A review of some of the R packages and projects implemented at the 2017 ROpenSci Unconference. An example of applying Bayesian Learning with the "bnlearn" package to challenge stereotypical assumptions Data from the Billboard Hot 100 chart used to find the most popular words in the titles of pop hits. Microsoft R Open 3.4.0 is now available for Windows, Mac and Linux. How to use the "tweenr" package to create smooth transitions in data animations. A preview of some of the companies and R applications to be presented at the EARL conference in San Francisco. The AzureDSVM package makes it easy to spawn and manage clusters of the Azure Data Science Virtual Machine. An online course on spatial data analysis in R, from the Consumer Data Research Centre in the UK. Video and slides of Lixun Zhang's presentation "R in Financial Services: Challenges and Opportunity" from the New York R Conference. Visual Studio 2017 now features built-in support for both R and Python development. Quantifying the home-field advantage in English Premier League football. Using the new CRAN_package_db function to analyze data about CRAN packages. Stack Overflow Trends tracks the trajectory of questions about R and Python. A recorded webinar on using Microsoft R to predict length of stay in hospitals. The new "Real-Time Scoring" capability in Microsoft R Server creates a service to generate predictions from certain models in milliseconds. "Technical Foundations of Informatics" is an open course guide on data analysis and visualization with R with a modern slant. The Datasaurus Dozen generalizes Anscombe's Quartet with a process to create datasets of any shape with (nearly) the same summary statistics. CareerCast ranks Statistician as the best job to have in 2017. You can now use Microsoft R withi…
subject: R
subject: roundups
creator: David Smith
date: 2017-06-06T12:33:04-07:00

Powe[R] BI: Free e-book on using R with Power BI
type: item
description: A new (and free!) e-book on extending the capabilities of Power BI with R is now available for download, from analytics consultancy BlueGranite. The introduction to the book explains why R and Power BI are a great match together: As a specialized, open source statistical environment, R represents the primary analysis language for a large number of data scientists and statisticians. In recent years, R has also undergone a significant shift in user base by gaining wider adoption in the business world. By extending Power BI with R, Microsoft has opened up numerous opportunities to enhance your Business Intelligence solutions....
encoded: A new (and free!) e-book on extending the capabilities of Power BI with R is now available for download, from analytics consultancy BlueGranite. The introduction to the book explains why R and Power BI are a great match together: As a specialized, open source statistical environment, R represents the primary analysis language for a large number of data scientists and statisticians. In recent years, R has also undergone a significant shift in user base by gaining wider adoption in the business world. By extending Power BI with R, Microsoft has opened up numerous opportunities to enhance your Business Intelligence solutions. In addition to its versatility for data science, R is a great language and ecosystem for work related to both data visualization and data processing. By incorporating R into its products, Microsoft has signaled a strong commitment from Microsoft not only to data science, but the R platform in general. The book provides a step-by-step guide to using R within Power BI, including: How to find, install and use pre-built custom visuals based on R withing Power BI How to create your own R Visuals using the R language, and use them in both Power BI Desktop and the cloud-based Power BI Service How to perform custom data processing with R scripts The e-book is available for download at the link below (free registration required). For more links to Power BI resources, you might also want to check out the blog post, "R with Power BI". BlueGranite: Power[R] BI: Enhance Your Microsoft Power BI Experience (via David Eldersveld)
subject: Microsoft
subject: R
creator: David Smith
date: 2017-06-05T06:51:36-07:00

Because it's Friday: Disappearing Dots
type: item
description: It's been a while since we posted an optical illusion, and this one (via Max Galka) jis ust too good to pass up. Here are the instructions, from the source: First, look at any yellow dot as the figure moves. The yellow dot remains present and stationary. If you concentrate on all three yellow dots, they remain there as well. But now concentrate on the central green dot. You will see one or more of the yellow dots disappearing and then reappearing sporadically. They are not—this is an optical illusion. The dots remain and your brain simply doesn’t register their...
encoded: It's been a while since we posted an optical illusion, and this one (via Max Galka) jis ust too good to pass up. Here are the instructions, from the source: First, look at any yellow dot as the figure moves. The yellow dot remains present and stationary. If you concentrate on all three yellow dots, they remain there as well. But now concentrate on the central green dot. You will see one or more of the yellow dots disappearing and then reappearing sporadically. They are not—this is an optical illusion. The dots remain and your brain simply doesn’t register their presence from time to time. Weird, eh? It's a bit of a puzzle why this one works. Why, for instance, is the rotating grid necessary? The mind works in mysterious ways. That's all for us for this week. We'll be back on Monday, when we'll be checking in from the EARL Conference in San Francisco. See you then, and have a great weekend!
subject: random
creator: David Smith
date: 2017-06-02T11:40:28-07:00

Teach kids about R with Minecraft
type: item
description: As I mentioned earlier this week, I was on a team at the ROpenSci Unconference (with Brooke Anderson, Karl Broman, Gergely Daróczi, and my Microsoft colleagues Mario Inchiosa and Ali Zaidi) to work on a project to interface the R language with Minecraft. The resulting R package, miner, is now available to install from Github. The goal of the package is to introduce budding programmers to the R language via their interest in Minecraft, and to that end there's also a book (R Programming with Minecraft) and associated R package (craft) under development to provide lots of fun examples of...
encoded: As I mentioned earlier this week, I was on a team at the ROpenSci Unconference (with Brooke Anderson, Karl Broman, Gergely Daróczi, and my Microsoft colleagues Mario Inchiosa and Ali Zaidi) to work on a project to interface the R language with Minecraft. The resulting R package, miner, is now available to install from Github. The goal of the package is to introduce budding programmers to the R language via their interest in Minecraft, and to that end there's also a book (R Programming with Minecraft) and associated R package (craft) under development to provide lots of fun examples of manipulating the Minecraft world with R. Create objects in Minecraft with R functions If you're a parent you're probably already aware of the Minecraft phenomenon, but if not: it's kinda like the Lego of the digital generation. Kids (and kids-and-heart) enter a virtual 3-D world composed of cubes representing ground, water, trees and other materials, and use those cubes to build their own structures, which they can then explore with their friends using their in-game avatars. Inspired by the Python-focused book "Learn to program with Minecraft", Karl had the brilliant idea of building a similar interface around R. The miner package provides just a few simple functions to manipulate the game world: find or move a player's position; add or remove blocks in the world; send a message to all players in the world. The functions are deliberately simple, designed to be used to build more complex tasks. For example, you could write a function to detect when a player is standing in a hole. It uses the getPlayerIds and getPlayerPos functions to find the ID and locations of all players in the world, and the getBlocks function to check if the player is surrounded by blocks that are not code 0 (air). The package also provides are also a couple of "listener" functions: you can detect player chat messages, or when players strike a block with their sword. You can then use to write functions to react to …
subject: Microsoft
subject: packages
subject: R
creator: David Smith
date: 2017-06-02T08:19:42-07:00

Python and R top 2017 KDnuggets rankings
type: item
description: The results of KDnuggets' 18th annual poll of data science software usage are in, and for the first time in three years Python has edged out R as the most popular software. While R increased its share of usage from 45.7% in last year's poll to 52.1% this year, Python's usage among data scientists increased even more, from 36.6% of users in 2016 to 52.6% of users this year. There were some interesting moves in the long tail, as well. Several tools entered the KDNuggets chart for the first time, including Keras (9.5% of users), PyCharm (9.0%) and Microsoft R...
encoded: The results of KDnuggets' 18th annual poll of data science software usage are in, and for the first time in three years Python has edged out R as the most popular software. While R increased its share of usage from 45.7% in last year's poll to 52.1% this year, Python's usage among data scientists increased even more, from 36.6% of users in 2016 to 52.6% of users this year. There were some interesting moves in the long tail, as well. Several tools entered the KDNuggets chart for the first time, including Keras (9.5% of users), PyCharm (9.0%) and Microsoft R Server (4.3%). And several returning tools saw big jumps in usage, including Microsoft Cognitive Toolkit (3.4% of users), Tensorflow (20.2%) and Power BI (10.2%). Microsoft SQL Server increased its share to 11.6% (up from 10.8%), whereas SAS (7.1%) and Matlab (7.4%) saw declines. Julia, somewhat surprisingly, remained flat at 1.1%. For the complete results and analysis of the 2017 KDnuggets data science software poll, follow the link below. KDnuggets: New Leader, Trends, and Surprises in Analytics, Data Science, Machine Learning Software Poll
subject: popularity
subject: python
subject: R
creator: David Smith
date: 2017-06-01T14:30:03-07:00

Watch presentations from R/Finance 2017
type: item
description: It was another great year for the R/Finance conference, held earlier this month in Chicago. This is normally a fairly private affair: with attendance capped at around 300 people every year, it's a somewhat exclusive gathering of the best and brightest minds from industry and academia in financial data analysis with R. But for the first time this year (and with thanks to sponsorship from Microsoft), videos of the presentations are available for viewing by everyone. I've included the complete list (copied from the R/Finance website) below, but here are a few of my favourites: No-Bullshit Data Science. Szilard Pafka's...
encoded: It was another great year for the R/Finance conference, held earlier this month in Chicago. This is normally a fairly private affair: with attendance capped at around 300 people every year, it's a somewhat exclusive gathering of the best and brightest minds from industry and academia in financial data analysis with R. But for the first time this year (and with thanks to sponsorship from Microsoft), videos of the presentations are available for viewing by everyone. I've included the complete list (copied from the R/Finance website) below, but here are a few of my favourites: No-Bullshit Data Science. Szilard Pafka's keynote knocks down some of the myths and misconceptions about real-world data science practices. Risk Fast and Slow. A rare treat: in his keynote presentation Dave DeMers offers his perspectives on risk, financial engineering, and major financial events from his days the Prediction Company, Black Mesa and other investment houses. Syberia: A development framework for R (Robert Krzyzanowski). Syberia is a new operationalization framework for R scripts, applicable for any production workflow using R (not just Finance). yuimaGUI: A graphical user interface for the yuima package (Emanuele Guidotti). An impressive front-end to the Yuima project for solving stochastic differential equations. Zero-Revelation RegTech: Detecting Risk through Linguistic Analysis of Corporate Emails and News (Seoyoung Kim). A really interesting approach to looking at sentiment (and length!) of emails (here, the Enron corpus) to predict stock prices. New Tools for Performing Financial Analysis Within the 'Tidy' Ecosystem (Matt Dancho). This lightning talk shows how you can perform piping (%>%) operations with time series data from zoo and xts. The PE package: Modeling private equity in the 21st century (Thomas Harte). An intriguing peek into the mysterious world of private equity finance. Project and conquer (Bryan Lewis). A beautifully elegant introduction to the use of projections …
subject: events
subject: finance
subject: Microsoft
subject: R
creator: David Smith
date: 2017-05-31T14:23:05-07:00