In an earlier post I discussed how to avoid overfitting when using Support Vector Machines. This was achieved using cross validation. In cross validation, prediction accuracy is maximized by varying the cost parameter. Importantly, prediction accuracy is calculated on a different subset of the data from that used for training.

In this blog post I take that concept a step further, by automating the manual search for the optimal cost.

The data set I’ll be using describes different types of glass based upon physical attributes and chemical composition.  You can read more about the data here, but for the purposes of my analysis all you need to know is that the outcome variable is categorical (7 types of glass) and the 4 predictor variables are numeric.

Creating the base support vector machine model

I start, as in my earlier analysis, by splitting the data into a larger 70% training sample and a smaller 30% testing sample. Then I train a support vector machine on the training sample with the following code:

library(flipMultivariates)
svm = SupportVectorMachine(Type ~ RefractiveIndex + Ca + Ba + Fe,
                           subset = training,
                           cost = 1)

This produces output as shown below. There are 2 reasons why we can largely disregard the 64.67% accuracy:

1. We used the training data (and not the independent testing data) to calculate accuracy.
2. We have used a default value for the cost of 1 and not attempted to optimize.

Amending the R code

I am going to amend the code above in order to loop over a range of values of cost. For each value, I will calculate the accuracy on the test sample. The updated code is as follows:

library(flipMultivariates)
library(flipRegression)
costs = c(0.1, 1, 10, 100, 1000, 10000)
i = 1
accuracies = rep(0, length(costs))

for (cost in costs)
{
    svm = SupportVectorMachine(Type ~ RefractiveIndex + Ca + Ba + Fe,
                               subset = training,
                               cost = cost)
    accuracies[i] = attr(ConfusionMatrix(svm, subset = (testing == 1)), "accuracy")
    i = i + 1
}
plot(costs, accuracies, type = "l", log = "x")

The first 5 lines set things up. I load libraries required to run the Support Vector Machine and calculate the accuracy. Next I choose a range of costs, initialize a loop counter i and an empty vector accuracies, where I store the results.

Then I add a loop around the code that created the base model to iterate over costs. The next line calculates and stores the accuracy on the testing sample. Finally I plot the results which tells me that the greatest accuracy appears around 100. This allows us to go back and update costs to a more granular range around this value.

Re-running the code again using the new costs (10, 20, 50, 75, 100, 150, 200, 300, 500, 1000) I get the final chart shown below. This indicates that a cost of 50 gives best performance.

TRY IT OUT
The analysis in this post used R in Displayr. The flipMultivariates package (available on GitHub), which uses the e1071 package, performed the calculations. You can try automatically fitting the Support Vector Machine Cost Parameter yourself using the data in this example. 

This is the second in a series of posts designed to show how the R programming language can be used with cryptocurrency related data sets.  A number of R packages are great for analyzing stocks and bonds and similar financial instruments.  These can also be applied to working with cryptocurrencies.  In this post we will focus on Bitcoin.

Bitcoin has garnered enough attention that it is available through Yahoo’s finance data under the symbol BTCUSD=X.  The quantmod package is comprised of a set of packages and utilities geared towards time series analysis traditionally associated with stocks.  You can load Bitcoin along with other Stock symbols using the loadSymbols function.  In this example we will also load AMD, which makes graphics cards used by cryptocurrency miners.

library(quantmod)
loadSymbols(c(‘BTCUSD=X’,’AMD’))

If you have any issue downloading the data, make sure you update to the latest version of quantmod.  If all goes well, you will have two objects in your global environment named AMD and BTCUSD=X.

ls()
[1] “AMD”      “BTCUSD=X”

You can plot AMD by simply passing it to the plot function.

plot(AMD)

Bitcoin is slightly different simply because the symbol in use includes an equal sign.  To ensure that R evaluates the code properly, the symbol must be surrounded in back ticks.

plot(`BTCUSD=X`)

There is data missing for certain days.  There are other sources for cryptocurrency data which can be substituted if needed.  We will ignore this anomaly for the remainder of this post.  There is data for the last 4 weeks. We can construct a candle chart that focuses on this subset of data.

chartSeries(`BTCUSD=X`, subset=’last 4 weeks’)

This chart can then be modified to include technical analysis – for instance Bollinger Bands.

addBBands()

The capabilities of the quantmod package in an earlier post (see http://www.r-chart.com/2010/06/stock-analysis-using-r.html) where a listing of other functions that can be applied is included.

Inasmuch as cryptocurrencies behave like traditional equities, they lend themselves to similar types of analysis.  The quantmod package is a great place to start when analyzing cryptocurrencies.   

The Ecosystems chapter of my Empirical software engineering book has been added to the draft pdf (download here).

I don’t seem to be able to get away from rewriting everything, despite working on the software engineering material for many years. Fortunately the sparsity of the data keeps me in check, but I keep finding new and interesting data (not a lot, but enough to slow me down).

There is still a lot of work to be done on the ecosystems chapter, not least integrating all the data I have been promised. The basic threads are there, they just need filling out (assuming the promised data sets arrive).

I did not get any time to integrate in the developer and economics data received since those draft chapters were released; there has been some minor reorganization.

As always, if you know of any interesting software engineering data, please tell me.

I’m looking to rerun the workshop on analyzing software engineering data. If anybody has a venue in central London, that holds 30 or so people+projector, and is willing to make it available at no charge for a series of free workshops over several Saturdays, please get in touch.

Projects chapter next.

I’m extremely pleased to announce that the sergeant package is now on CRAN or will be hitting your local CRAN mirror soon.

sergeant provides JDBC, DBI and dplyr/dbplyr interfaces to Apache Drill. I’ve also wrapped a few goodies into the dplyr custom functions that work with Drill and if you have Drill UDFs that don’t work “out of the box” with sergeant‘s dplyr interface, file an issue and I’ll make a special one for it in the package.

I’ve written about drill on the blog before so check out those posts for some history and stay tuned for more examples. The README should get you started using sergeant and/or Drill (if you aren’t running Drill now, take a look and you’ll likely get hooked).

I’d like to take a moment to call out special thanks to Edward Visel for bootstrapping the dbplyr update to sergeant when the dplyr/dbplyr interfaces split. It saved me loads of time and really helped the progress of this package move faster towards a CRAN release.

Last month the French elections viewed through Twitter: a semantic analysis post showed how the two contenders were perceived on Twitter during three key events of the campaign (Macron leaks, presidential debate and election day). The goal of the post is to show how to perform this twitter analysis using R.

Collecting tweets in real time with streamR (Twitter streaming API)

To perform the analysis, I needed an important number of tweets and I wanted to use all of the tweets concerning the election. The Twitter search API is limited since you only have access to a sample of tweets. On the other hand, the streaming API allows you to collect the data in real-time and to collect almost all tweets. Hence, I used the streamR package.

So, I collected tweets on 60 seconds batch and saved them on .json files. The use of batches instead of one large file is to improve RAM consumption (Instead of reading and then subsetting one large file, you can do the subset on each of the batches and then merge them). Here is the code to collect the data with streamR.

###Loading my twitter credentials
load("oauth.Rdata")
##Collecting data
require('streamR')
i=1
while(TRUE)
{
 i=i+1
 filterStream( file=paste0("tweet_macronleaks/tweets_rstats",i,".json"),
 track=c("#MacronLeaks"), timeout=60, oauth=my_oauth,language = 'fr')
}

The code is doing an infinite loop (stopped manually), the filterStream function filters the Twitter stream according to the defined filter. Here, we only take the tweets containing #MacronLeaks which are in French.

Tweets cleaning and pre-processing

Now that the tweets are collected, they need to be cleaned and pre-processed. A raw tweet will contain links, tabulation, @, #, double spaces,  … that will influence the analysis. It will also contain stop words (stop words are very frequent words in the language such as ‘and’, ‘or, ‘with’, …).
In addition to this, some tweets are retweeted (sometimes a lot) and may change the words and text distribution. Enough of the RT are kept to show that some tweets are more popular than others but most of them are removed to avoid them standing too much out of the crowd.

First, the saved tweets need to be read and merged:

require(data.table)
data.tweet=NULL
i=1
while(TRUE)
{
 i=i+1
 print(i)
 print(paste0("tweet_macronleaks/tweets_rstats",i,".json"))

if (is.null(data.tweet))
 data.tweet=data.table(parseTweets(paste0("tweet_macronleaks/tweets_rstats",i,".json")))
 else
 data.tweet=rbind(data.tweet,data.table(parseTweets(paste0("tweet_macronleaks/tweets_rstats",i,".json"))))
}

Then we only keep some of the RT. The retweet count is the indices of a given retweet, hence we only keep log(1+n) of the RT.

data.tweet[,min_RT:=min(retweet_count),by=text]
data.tweet[,max_RT:=max(retweet_count),by=text]
data.tweet=data.tweet[lang=='fr',]
data.tweet=data.tweet[retweet_count<=min_RT+log(max_RT-min_RT+1),]

Then, the text can be cleaned using function from the tm package

###Unaccent and clean the text
Unaccent <- function(x) {
 x=tolower(x)
 x = gsub("@\\w+", "", x) 
 x = gsub("[[:punct:]]", " ", x)
 x = gsub("[ |\t]{2,}", " ", x) 
 x = gsub("^ ", " ", x) 
 x = gsub("http\\w+", " ", x) 
 x=tolower(x)
 x=gsub('_',' ',x,fixed=T)
 x
 
}
require(tm)
###Remove accents
data.tweet$text=Unaccent(iconv(data.tweet$text,from="UTF-8",to="ASCII//TRANSLIT"))
##Remove top words
data.tweet$text=removeWords(data.tweet$text,c('rt','a',stopwords('fr'),'e','co','pr'))
##Remove double whitespaces
data.tweet$text=stripWhitespace(data.tweet$text)

Tokenization and creation of the vocabulary

Now that the tweets have been cleaned, they can be tokenized. During this step, each tweet will be split into tokens of its different words, here each word corresponds to a token.

# Create iterator over tokens
tokens <- space_tokenizer(data.tweet$text)
it = itoken(tokens, progressbar = FALSE)

Now a vocabulary can be created (it is a “summary” of the words distribution) based on the corpus. Then the vocabulary is pruned (very common and rare words are removed).

vocab = create_vocabulary(it)

vocab = prune_vocabulary(vocab,
 term_count_min = 5, 
 doc_proportion_max = 0.4,
 doc_proportion_min = 0.0005)
vectorizer = vocab_vectorizer(vocab, 
 grow_dtm = FALSE, 
 skip_grams_window = 5L)

tcm = create_tcm(it, vectorizer)

Now, we can create the word embedding, in this example, I used a glove embedding to learn vectors representations of the words. The new vector space has around 200 dimensions.

glove = GlobalVectors$new(word_vectors_size = 200, vocabulary = vocab, x_max = 100)
glove$fit(tcm, n_iter = 200)
word_vectors <- glove$get_word_vectors()

How to finish our twitter analysis with Tsne

Now that the words are vectors, we would like to plot them in two dimensions to show the meaning of the words in an appealing (and understandable) way. The number of dimension needs to be reduced to two, to do so, we will use T-sne. T-sne is a non-parametric dimensionality reduction algorithm and tends to perform well on word embedding. R has a package (actually two) to perform Tsne, we will use the most recent one  Rtsne.
To avoid overcrowding in our plot and reduce computing time, only words with more than 50 appearances will be used.

require('Rtsne')
set.seed(123)
word_vectors_sne=word_vectors[which(vocab$vocab$doc_counts>50&!rownames(word_vectors)%in%stopwords('fr')),]
tsne_out=Rtsne(word_vectors_sne,perplexity =2,initial_dims = 200,dims = 2)
DF_proj=data.frame(x=tsne_out$Y[,1],y=tsne_out$Y[,2],word=rownames(word_vectors_sne))

Now that the projection in 2 dimensions has been done, to color the plot we’d like to know which contenders is assigned to each word. To do so, a dictionary is created with the names and pseudo of each of the contenders and the distance from every word to each of these pseudos is computed.
For instance, to assign a candidate to the word ‘democracy’, the minimum distance between ‘democracy’ and ‘mlp’, ‘marine’, ‘fn will be computed. The same thing will be done between ‘democracy’ and ‘macron’, ’em’, ’emmarche’. If the first distance is the smallest then ‘democracy’ will be assigned to Marine Le Pen, otherwise, it will be assigned to Emmanuel Macron.

require(ggplot2)
require(ggrepel)
DF_proj=data.table(DF_proj)
DF_proj$count=vocab$vocab$doc_counts[which(vocab$vocab$doc_counts>500& !(rownames(word_vectors)%in%stopwords('fr')))]
DF_proj=DF_proj[word!='NA']

distance_to_candidat=function(word_vectors,words_list,word_in)
{
 max(sim2(word_vectors[words_list,,drop=F],word_vectors[word_in,,drop=F]))
}
closest_candidat=function(word_vectors,mot_in)
{
 mot_le_pen=c('marine','pen','lepen','fn','mlp')
 mot_macron=c('macron','emmanuel','em','enmarche','emmanuelmacron')
 dist_le_pen=distance_to_candidat(word_vectors,mot_le_pen,mot_in)
 dist_macron=distance_to_candidat(word_vectors,mot_macron,mot_in)
 if (dist_le_pen>dist_macron)
 'Le Pen'
 else
 'Macron'
}
DF_proj[,word:=as.character(word)]
DF_proj=DF_proj[word!=""]
DF_proj[,Candidat:=closest_candidat(word_vectors,word),by=word]

require(plotly)
gg=ggplot(DF_proj,aes(x,y,label=word,color=Candidat))+geom_text(aes(size=sqrt(count+1)))
ggplotly(gg)

You can get our latest news on Twitter:

The post Twitter analysis using R (Semantic analysis of French elections) appeared first on Enhance Data Science.

Ensuring your analytic IP is given the attention it deserves

It is now widely recognised that data is the key to making informed business decisions. As such, models and code are the tools used to extract insight and should be considered very valuable IP for an organisation.

Considering data as a valuable asset, it’s important to store it so it’s easy for others within the organisation to find, reuse and repurpose this code in other projects and areas of the business.

However, there are some key challenges:

Losing code

Even if sharing code is actively encouraged inside an organisation, traditional storage platforms treat analytical code in the same way as any other file. This means that —without prior knowledge of a particular script’s existence— they can be hard to find, and in some cases lost forever in a mass of other files and scripts in the same platform.

Reproducibility

Have you ever been asked to reproduce a piece of analysis from 6 months ago? Or how about two years ago? For many, reproducing an older piece of analysis can be a huge task. Finding the script is one thing, but then knowing which version of a script was used, the data it was run against, the versions of the software that it was originally run in make this a more complex problem that you might originally think.

Wasting time

How many times have you written a script, only to find out a colleague has already written code which does the exact same thing? If this has happened to you, then it has probably happened to your colleagues.

ModSpace offers the solutions to these problems and much more.

Developed by the Mango team with modellers and statisticians in mind, it is a safe place to store analytical code and models. Plus, because ModSpace has the ability to understand analytical code when code is loaded into the system, key information is collected to make it easily discoverable by other users.

By linking ModSpace to multiple repositories around your organisation and integrating it with analysts’ preferred tools —such as R, SAS, Python, MATLAB, NONMEM and many others— you’re providing your teams with an enhanced workflow and analytic hub.

If analytical code and models are a vital part of your business, please join us on 20 July for a FREE demonstration of ModSpace.

Register now.

It’s easy to get stuck in the day-to-day at the office and there’s never time to upskill or even think about career development. However, to really grow and develop your organisation, it’s important to grow and develop your team.

While there are many ways to develop teams, including training and providing time to complete personal (and relevant) projects, conferences provide a range of benefits.

Spark innovation

Some of the best in the business present their projects, ideas and solutions at EARL each year. It’s the perfect opportunity to see what’s trending and what’s really working. Topics at EARL Conferences this year include, best practice SAS to R; Shiny applications; using social media data; web scraping, plus presentations on R in marketing, healthcare, finance, insurance and transport. Take a look at the agenda for London here.

A cross-sector conference like EARL can help your organisation think outside the box because learnings are transferable, regardless of industry.

Imbue knowledge

This brings us to knowledge. Learning from the best in the business will help employees expand their knowledgebase. This can keep them motivated and engaged in what they’re doing; and a wider knowledgebase can also inform their everyday tasks enabling them to advance the way they do their job.

When employees feel like you want to invest in them, they stay engaged and are more likely to remain in the same organisation for longer.

Encourage networking

EARL attracts R users from all levels and industries and not just to speak. The agenda offers plenty of opportunities to network with some of the industry’s most engaged R users. This is beneficial for a number of reasons, including knowledge exchange and sharing your organisation’s values.

Boost inspiration

We often see delegates who have come to an EARL Conference with a specific business challenge in mind. By attending, they get access to the current innovations, knowledge and networking mentioned above, and can return to their team —post-conference— with a renewed vigour to solve those problems using their new-found knowledge.

Making the most out of attending EARL

After all of that, the next step is making sure your organisation makes the most out of attending EARL. We recommend:

Setting goals

Do you have a specific challenge you’re trying to solve in your organisation? Going with a set challenge in mind means your team can plan which sessions to sit in and who they should talk to during the networking sessions.

De-briefing

This is two-fold:

1) Writing a post-conference report will help your team put what they have learnt at EARL into action.

2) Not everyone can attend, so those who do can share their new-found knowledge with their peers who can learn second-hand from their colleague’s just be smarexperience.

Following up

We’re all guilty of going to a conference, coming back inspired and then getting lost in the day-to-day. Assuming you’ve set goals and de-briefed, it should be easy to develop a follow up plan.

You can make the most of inspired team members to put in place new strategies, technologies and innovations through further training, contact follow-ups and new procedure development.

EARL Conference offers a range of discounts, including package deals for organisations looking to send more than 2 delegates.

Buy tickets now or contact the EARL Team