<?xml version="1.0" encoding="UTF-8"?>
		<rss version="2.0" xmlns:a10="http://www.w3.org/2005/Atom">
      <channel xmlns:sxp="blogs.technet.microsoft.com/cloudplatform/rssfeeds/dataplatform" xmlns:sxpMd="http://sxpdata.microsoft.com/metadata">
        <title>Category Name</title>
        <description>Category Name</description>
        <copyright>Copyright (c) 2017 Microsoft</copyright>
        <link>https://blogs.technet.microsoft.com/cloudplatform</link>
        <a10:link href="blogs.technet.microsoft.com/cloudplatform/rssfeeds/dataplatform" rel="self" type="application/rss+xml" />

				
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/because-its-friday-3-d-animation.html</guid>
						<pubDate>Fri, 08 Dec 2017 22:50:26 +0000</pubDate>
						<relativeTime>1 day ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[Because it&#8217;s Friday: 3-D Animation]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/because-its-friday-3-d-animation.html</link>
						<description><![CDATA[<p>We've had 3-D animation for quite a while now, of course, but what happens when a traditional 2-D animator uses a virtual reality system to draw? When famed Disney animator <a href="https://en.wikipedia.org/wiki/Glen_Keane">Glen Keane</a>&nbsp;sketches his most iconic creation &mdash; Ariel from The Little Mermaid &mdash; using Tilt Brush, the result is surprisingly moving.</p>
<p></p>
<p>That's all from us for this week. We'll be back with more on Monday, and in the meantime have a great weekend!</p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/because-its-friday-3-d-animation.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/november-2017-roundup.html</guid>
						<pubDate>Thu, 07 Dec 2017 21:54:02 +0000</pubDate>
						<relativeTime>2 days ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[In case you missed it: November 2017 roundup]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/november-2017-roundup.html</link>
						<description><![CDATA[<p>In case you missed them, here are some articles from November of particular interest to <a href="https://mran.microsoft.com/documents/what-is-r/" target="_self">R</a> users.</p>
<p><a href="http://blog.revolutionanalytics.com/2017/11/r-343-released.html">R 3.4.3 "Kite Eating Tree"</a> has been released.</p>
<p>Several approaches for <a href="http://blog.revolutionanalytics.com/2017/11/how-to-generate-a-secret-santa-list-with-r.html">generating a "Secret Santa" list with R</a>.</p>
<p>The "RevoScaleR" package from Microsoft R Server has now been <a href="http://blog.revolutionanalytics.com/2017/11/revoscalepy.html">ported to Python</a>.</p>
<p>The <a href="http://blog.revolutionanalytics.com/2017/11/rfinance-2018.html">call for papers for the R/Finance 2018 conference</a> in Chicago is now open.</p>
<p><a href="http://blog.revolutionanalytics.com/2017/11/happy-thanksgiving.html">Give thanks</a> to the volunteers behind R.</p>
<p><a href="http://blog.revolutionanalytics.com/2017/11/r-ladies-chicago-part-1.html">Advice for R user groups</a> from the organizer of R-Ladies Chicago.</p>
<p><a href="http://blog.revolutionanalytics.com/2017/11/doazureparallel-containers.html">Use containers to build R clusters</a> for parallel workloads in Azure with the doAzureParallel package.</p>
<p>A collection of R scripts for <a href="http://blog.revolutionanalytics.com/2017/11/charts-in-280-chars.html">interesting visualizations that fit into a 280-character Tweet</a>.</p>
<p><a href="http://blog.revolutionanalytics.com/2017/11/connect-highlights.html">R is featured in a StackOverflow case study</a> at the Microsoft Connect conference.</p>
<p><a href="http://blog.revolutionanalytics.com/2017/11/chicago-water.html">The City of Chicago uses R to forecast water quality</a> and issue beach safety alerts.</p>
<p>A collection of <a href="http://blog.revolutionanalytics.com/2017/11/good-practices-spreadsheets.html">best practices for sharing data in spreadsheets</a>, from a paper by Karl Broman and Kara Woo.</p>
<p>The <a href="http://blog.revolutionanalytics.com/2017/11/an-update-for-mran.html">MRAN website has been updated</a> with faster package search and other improvements.</p>
<p>The <a href="http://blog.revolutionanalytics.com/2017/11/curl-updated.html">curl package has been updated</a> to use the built-in winSSL library on Windows.</p>
<p>Beginner, intermediate and advanced <a href="http://blog.revolutionanalytics.com/2017/11/azure-learning-plans.html">on-line learning plans for developing AI applications on Azure</a>.</p>
<p>A <a href="http://blog.revolutionanalytics.com/2017/11/recap-earl-boston-2017.html">recap of the EARL conference</a> (Effective Applications of the R Language) in Boston.&nbsp;</p>
<p>Giora Simchoni uses R to <a href="http://blog.revolutionanalytics.com/2017/11/slot-machine-edge.html">calculate the expected payout from a slot machine</a>.</p>
<p>An <a href="http://blog.revolutionanalytics.com/2017/11/r-excel-history.html">introductory R tutorial by Jesse Sadler</a> focuses on the analysis of historical documents.</p>
<p>A new RStudio cheat sheet: "<a href="http://blog.revolutionanalytics.com/2017/11/strings-in-r.html">Working with Strings</a>".</p>
<p>An overview of <a href="http://blog.revolutionanalytics.com/2017/11/role-playing-with-probabilities.html">generating distributions in R via simulated gaming dice</a>.</p>
<p>An analysis of StackOverflow survey data <a href="http://blog.revolutionanalytics.com/2017/11/r-is-the-least-disliked-programming-language.html">ranks R and Python among the most-liked and least-disliked languages</a>.</p>
<p>And some general interest stories (not necessarily related to R):</p>
<ul><li><a href="http://blog.revolutionanalytics.com/2017/11/because-its-friday-trombone.html">Siri transcribes a trombone player</a></li>
<li>A collection of short <a href="http://blog.revolutionanalytics.com/2017/11/because-its-friday-chemical-reactions.html">videos of interesting chemical reactions</a></li>
<li>An animation shows the <a href="http://blog.revolutionanalytics.com/2017/11/how-a-drone-impacted-flight-paths.html">impact of a rogue drone on Gatwick airport</a></li>
<li><a href="http://blog.revolutionanalytics.com/2017/11/because-its-friday-fake-celebrities.html">An AI sythesizes novel images</a> of furniture, animals, and celebritiesA</li>
</ul><p>As always, thanks for the comments and please send any suggestions to me at <a href="mailto:davidsmi@microsoft.com">davidsmi@microsoft.com</a>. Don't forget you can follow the blog using an RSS reader, via&nbsp;<a href="http://blogtrottr.com/" target="_self">email using blogtrottr</a>, or by following me on Twitter (I'm <a href="http://twitter.com/revodavid">@revodavid</a>). You can find roundups of previous months&nbsp;<a href="http://blog.revolutionanalytics.com/roundups/">here</a>.</p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/november-2017-roundup.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">https://blogs.msdn.microsoft.com/pythonengineering/2017/12/07/whats-new-for-python-in-visual-studio-15-6-preview-1/</guid>
						<pubDate>Thu, 07 Dec 2017 20:00:32 +0000</pubDate>
						<relativeTime>3 days ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[What’s new for Python in Visual Studio 2017 15.6 Preview 1]]></title>
						<link>https://blogs.msdn.microsoft.com/pythonengineering/2017/12/07/whats-new-for-python-in-visual-studio-15-6-preview-1/</link>
						<description><![CDATA[<p>Today we have released the first preview of our next update to Visual Studio 2017. You will see a notification in Visual Studio within the next few days, or you can download the new installer from <a href="https://visualstudio.com/vs/preview" target="_blank">visualstudio.com</a>.</p>
<p>In this post, we're going to take a look at some of the new features we have added for Python developers. As always, the preview is a way for us to get features into your hands early so you can provide feedback and we can identify issues with a smaller (and hopefully more forgiving!) audience. If you encounter any trouble, please use the Report a Problem tool to let us know.</p>
<h1>Immediate IntelliSense updates with no database</h1>
<p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/NoIntellisenseDB.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/NoIntellisenseDB-300x294.png" alt="Before and after images of the IntelliSense pane of the Python Environments window" width="300" height="294"></a></p>
<p>Remember how every time you installed or updated a package we would make you wait for hours while we "refresh" our "completion DB"? No more! In this update we are fundamentally changing how we handle this for installed Python environments, including virtual environments, so that we can provide IntelliSense immediately without the refresh.</p>
<p>This has been available as an experimental feature for a couple of releases, and we think it's ready to turn on by default. When you open the Python Environments window, you'll see the "IntelliSense" view is disabled and there is no longer a way to refresh the database -- because there is no database!</p>
<p>The new system works by doing lightweight analysis of Python modules as you import them in your code. This includes <tt>.pyd</tt> files, and if you have <tt>.pyi</tt> files alongside your original sources then we will prefer those (see <a href="https://www.python.org/dev/peps/pep-0484/#stub-files" target="_blank">PEP 484</a> for details of <tt>.pyi</tt> files. In essence, these are Python "include" files for editors to obtain information about Python modules, but do not actually have any code in them - just function stubs with type annotations).</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/Editor_pandas.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/Editor_pandas.png" alt="Completions in the editor from the pandas package" width="285" height="218"></a></p>
<p>You should notice some improvements in IntelliSense for packages like <tt>pandas</tt> and <tt>scikit-learn</tt>, though there will likely be some packages that do not work as well as before. We are actively working on improving results for various code constructs, and you will also see better IntelliSense results as packages start including <tt>.pyi</tt> type hint files. We encourage you to post on <a href="https://github.com/Microsoft/PTVS/issues/3046" target="_blank">this github issue</a> to let us know about libraries that still do not work well.</p>
<p>(NOTE: If you install this preview alongside an earlier version of Visual Studio 2017, the preview of this feature will also be enabled in earlier version. You can go back to the old model by disabling the feature in Preview. To do this, open Tools, Options, find the Python/Experimental page, deselect "Use new style IntelliSense" and restart both versions of Visual Studio.)</p>
<h1>conda integration</h1>
<p>If you use Anaconda, you likely already manage your environments and packages using the <tt>conda</tt> tool. This tool installs pre-built packages from the <a href="https://repo.continuum.io/pkgs/free/win-64/" target="_blank">Anaconda repository</a> (warning: <em>long</em> page) and manages compatibility with your environment and the other packages you have installed.</p>
<p>For this preview of Visual Studio, we have added two experimental options to help you work with Anaconda:</p>
<ul><li>Automatically detect when <tt>conda</tt> is a better option for managing packages</li>
<li>Automatically detect any Anaconda environments you have created manually</li>
</ul><p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/Install_Bokeh.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/Install_Bokeh.png" alt="Install package from conda in the Python environments window" width="628" height="216"></a></p>
<p>To enable either or both of these features, open Tools, Options, find the Python/Experimental page, and select the check box. For this preview we are starting with both disabled to avoid causing unexpected trouble, but we intend to turn them on by default in a future release.</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/Options_Conda.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/Options_Conda-300x175.png" alt="Options dialog with experimental conda options highlighted" width="300" height="175"></a></p>
<p>With "Automatically detect Conda environments" enabled, any environments created by the <tt>conda</tt> tool will be detected and listed in the Python Environments window automatically. You can open interactive windows for these environments, assign them in projects or make them your default environment.</p>
<p>With the "Use Conda package manager when available" option enabled, any environments that have <tt>conda</tt> installed will use that for search, install and updating instead of <tt>pip</tt>. Very little will visibly change, but we hope you'll be more successful when adding or removing packages to your environment.</p>
<p>Notice that these two options work independently: you can continue to use <tt>pip</tt> to manage packages if you like, even if you choose to detect environments that were created with <tt>conda</tt>. If you are an Anaconda user, you will likely want to enable both options. However, if you do this and encounter issues, disabling each one in turn and then reporting any differences will help us quickly narrow down the source.</p>
<h1>Other improvements</h1>
<p>We have made a range of other <a href="https://github.com/Microsoft/PTVS/pulls?utf8=%E2%9C%93&amp;q=is%3Apr+base%3Amaster+merged%3A2017-09-20..2017-12-01+-author%3Acsigs+" target="_blank">minor improvements and bug fixes</a> throughout all of our Python language support and there are more to come.</p>
<p>Our "IPython interactive mode" is now using the latest APIs, with improved IntelliSense and the same module and class highlighting you see in the editor.</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/JupyterClient_Version.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/JupyterClient_Version.png" alt="Interactive window showing current jupyter_client version and improved syntax highlighting" width="482" height="194"></a></p>
<p>There are new code snippets for the <a href="https://docs.python.org/3/library/argparse.html" target="_blank"><tt>argparse</tt></a> module. Start typing "arg" in the editor to see what is available.</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/Editor_argparse.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/Editor_argparse.png" alt="In the editor adding an argparse snippet." width="513" height="150"></a></p>
<p>We've also added new color customization options for docstrings and regular expression literals (under Tools, Options, Fonts and Colors). Doc strings have a new default color.</p>
<p><a href="https://msdnshared.blob.core.windows.net/media/2017/12/Editor_RegexLiteral.png"><img src="https://msdnshared.blob.core.windows.net/media/2017/12/Editor_RegexLiteral.png" alt="Doc strings and regex literal strings with customized colors" width="594" height="104"></a></p>
<p>If you encounter any issues, please use the Report a Problem tool to let us know (this can be found under Help, Send Feedback) or continue to use our <a href="https://github.com/Microsoft/PTVS" target="_blank">github</a> page. Follow <a href="https://blogs.msdn.microsoft.com/pythonengineering/">our blog</a> to make sure you hear about our updates first, and thanks for using Visual Studio!</p>]]></description>
						<author>Steve Dower [MSFT]</author>
						<source url="https://blogs.msdn.microsoft.com/pythonengineering/feed/">Python Engineering at Microsoft</source>
						<comments>https://blogs.msdn.microsoft.com/pythonengineering/2017/12/07/whats-new-for-python-in-visual-studio-15-6-preview-1/feed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/bes-reproducible-science.html</guid>
						<pubDate>Wed, 06 Dec 2017 22:26:26 +0000</pubDate>
						<relativeTime>3 days ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[The British Ecological Society&#8217;s Guide to Reproducible Science]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/bes-reproducible-science.html</link>
						<description><![CDATA[<p>The British Ecological Society has published a new volume in their <a href="http://www.britishecologicalsociety.org/publications/guides-to/">Guides to Better Science series</a>: <a href="http://www.britishecologicalsociety.org/wp-content/uploads/2017/12/guide-to-reproducible-code.pdf"><em>A Guide to Reproducible Code in Ecology and Evolution </em></a>(pdf). The introduction, by , describes its scope:</p>
<blockquote>
<p>A Guide to Reproducible Code covers all the basic tools and information you will need to start making your code more reproducible. We focus on R and Python, but many of the tips apply to any programming language. Anna Krystalli introduces some ways to organise files on your computer and to document your workflows. Laura Graham writes about how to make your code more reproducible and readable. Fran&ccedil;ois Michonneau explains how to write reproducible reports. Tamora James breaks down the basics of version control. Finally, Mike Croucher describes how to archive your code. We have also included a selection of helpful tips from other scientists.</p>
</blockquote>
<p>The guide proposes a simple reproducible project workflow, and a guide to organizing projects for reproducibility. The Programming section provides concrete tips and traps to avoid (example: use relative, not absolute pathnames), and the Reproducible Reports section provides a step-by-step guide for generating reports with <a href="http://rmarkdown.rstudio.com/">R Markdown</a>.</p>
<p><a href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c93aa615970b-pi"><img alt="Rmarkdown" border="0" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c93aa615970b-800wi" title="Rmarkdown"></a></p>
<p>While written for an ecology audience (and also including some gorgeous photography of animals), this guide would be useful for anyone in the science looking to implement a reproducible workflow. You can download the guide at the link below.</p>
<p>British Ecological Society:&nbsp;<a href="http://www.britishecologicalsociety.org/wp-content/uploads/2017/12/guide-to-reproducible-code.pdf">A Guide to Reproducible Code in Ecology and Evolution</a>&nbsp;(<a href="https://twitter.com/LauraJaneGraham">via</a> Laura Graham)</p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/bes-reproducible-science.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">https://blogs.technet.microsoft.com/machinelearning/2017/12/06/music-generation-with-azure-machine-learning/</guid>
						<pubDate>Wed, 06 Dec 2017 17:00:31 +0000</pubDate>
						<relativeTime>4 days ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[Music Generation with Azure Machine Learning]]></title>
						<link>https://blogs.technet.microsoft.com/machinelearning/2017/12/06/music-generation-with-azure-machine-learning/</link>
						<description><![CDATA[<p><span><em>This post is authored by Erika Menezes, Software Engineer at Microsoft.<br></em></span></p>
<p><span>Using deep learning to learn feature representations from near-raw input has been shown to outperform traditional task-specific feature engineering in multiple domains in several situations, including in <a href="https://www.nature.com/articles/nature14539">object recognition, speech recognition and text classification</a>. With the recent advancements in neural networks, deep learning has been gaining popularity in computational creativity tasks such as music generation. There has been great progress in this field via projects such as <a href="https://magenta.tensorflow.org/welcome-to-magenta">Magenta</a>, an open-source project focused on creating machine learning projects for art and music, from the Google Brain team, and <a href="http://www.flow-machines.com/">Flow Machines</a>, who have released an entire AI generated pop album. <span>For those of you who are curious about music generation, you can find additional resources <a href="http://www.asimovinstitute.org/analyzing-deep-learning-tools-music/">here</a></span>.<br></span></p>
<p><span>This goal of our work is to provide data scientists who are new to the field of music generation guidance on how to create deep learning models for music generation. As a sample, <a href="https://musicdatasets.blob.core.windows.net/generated-music/sample.mid">here</a> is music that was generated by training an LSTM model.<span><br></span></span></p>
<p><a href="https://musicdatasets.blob.core.windows.net/generated-music/sample.mid"><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera1.png"></a><span><br></span></p>
<p><span>In this post, we show you how to build a deep learning model for simple music generation using the Azure Machine Learning (AML) Workbench for experimentation.<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera2.png"><span><br></span></p>
<p><span>Here are the most important components for a deep learning model for music generation:<br></span></p>
<ul><li><span><strong>Dataset:</strong> The data used for training the model. In this work we will use the <a href="http://www.feelyoursound.com/scale-chords/">scale-chords</a> dataset.<br></span></li>
<li><span><strong>Input Representation:</strong> A meaningful vector representation of music notes. In this work we will use a <a href="http://www.bcp.psych.ualberta.ca/~mike/Pearl_Street/Dictionary/contents/P/pianoroll.html">piano roll</a> representation.<br></span></li>
<li><span><strong>Model Architecture:</strong> The deep learning model architecture for learning the task of predicting some set of musical notes, given an input of preceding musical notes. This work uses a <a href="https://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf">Sequence-to-Sequence</a> model using multi-layered Long Short-Term Memory (LSTM) to achieve this.<br></span></li>
</ul><p><span><span>Dataset</span><span><br></span></span></p>
<p><span>Music is available in a variety of digital audio formats ranging from raw audio (WAV) to more semantic representations such as <a href="https://en.wikipedia.org/wiki/MIDI">MIDI</a> (Musical Instrument Digital Interface), <a href="https://en.wikipedia.org/wiki/ABC_notation">ABC</a>, and sheet music. MIDI data already contains the information needed to feed the Deep Neural Network, we just need to transform it into an appropriate numeric representation to train the model. In next section discusses the details of this transformation.<br></span></p>
<p><span>For this work we will use the scale-chords dataset from <a href="http://www.feelyoursound.com/scale-chords/">here</a>. Download the dataset (free small pack) that contains 156 scale chords files in MIDI format. Let&rsquo;s take a closer look at MIDI.<br></span></p>
<p><span>MIDI<br></span></p>
<p><span>MIDI is a communications protocol for electronic musical instruments. A Python representation of a MIDI file looks something like this:<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera3.png"></p>
<p><span>MIDI represents time in &lsquo;ticks&rsquo; which essentially represent delta times, i.e. each event&rsquo;s tick is relative to the previous one. Each MIDI file&rsquo;s header contains the resolution of that file which gives us the number of ticks per beat. The MIDI file consists of one or more tracks that further consist of event messages such as the following:<br></span></p>
<ul><li><span>SetTempoEvent: Indicates the tempo in 8-bit words.<br></span></li>
<li><span>NoteOnEvent: Indicates that a note has been pressed or turned on.<br></span></li>
<li><span>NoteOffEvent: Indicates that a note has been released or turned off.<br></span></li>
<li><span>EndOfTrackEvent: Indicates that the track has ended.<br></span></li>
</ul><p><span>Music Theory 101<br></span></p>
<ul><li><span>Beat: Basic unit of time in music, a.k.a. quarter note.<br></span></li>
<li><span>Note: Pitch or frequency of the note played. E.g. note 60 in MIDI is C5 on the piano, which is 261.625 Hz.<br></span></li>
<li><span>Tempo: Expressed as Beats per minute (BPM) = Quarter notes per minute (QPM)<br>
Microseconds per quarter note (MPQN) = MICROSECONDS_PER_MINUTE / BPM.<br></span></li>
</ul><p><span><span>Input Representation</span><span><br></span></span></p>
<p><span>Input Representation is a crucial component of any music generation system. To feed the MIDI files to the Neural Network, we transform the MIDI to a piano roll representation. A piano roll is simply a 2D matrix of notes vs. time, where the black squares represent a note being pressed at that point in time. We use the MIDO python library to achieve this. The figure below shows a sample piano roll for notes in one octave.<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera4.png"><span><br></span></p>
<p><span>To transform the MIDI file shown above into a piano roll we need to quantize the MIDI events by time. The important trick to quantize the MIDI events is to understand how to convert MIDI ticks to absolute time. To do this we simply multiply the tempo (beats per minute) by the resolution (ticks per beat) and that gives us the ticks per second.<br></span></p>
<p><span>Model Architecture<br></span></p>
<p><span>Recurrent Neural Networks (RNN) are well suited for sequence prediction tasks as they can memorize long-range dependencies from input sequences using recurrent or looped connections. LSTMs are a special type of RNN that have multiplicative gates that enable them to retain memory for even longer sequences, making them useful for learning the sequential patterns present in musical data.<br></span></p>
<p><span>With this reasoning we decided to use an LSTM Sequence-to-Sequence model as shown in the figure below.<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera5.png"><span><br></span></p>
<p><span>A Sequence-to-Sequence model (Seq2Seq) is made up of an Encoder (encode input) and Decoder (decode output) to convert sequences from one domain, such as sentences in English, to sequences in another domain, such as the same sentences translated into French. This has been commonly used for machine translation or for freeform question answering. As in language translation, in music, the notes played during a given time period depend on several preceding notes, and Sequence-to-Sequence models are able to generate output sequences after seeing the entire input.<br></span></p>
<p><span>In the figure shown above, we train the network to generate some length of music notes given some preceding notes. In order to create the training set we use a sliding window over the piano roll. Consider the case where we have a piano roll of dimensions 12&times;10, where 12 is the number of notes in one octave and 10 is the number of columns in a piano roll, where each column represents some absolute time. Assuming a sliding window of 5, the first 5 columns are fed to the encoder as the input and the next 5 are the target which the model tries to learn. Since we are generating polyphonic music, i.e. multiple notes being on at the same time, this is a multi-label classification problem and hence we use the <a href="https://en.wikipedia.org/wiki/Cross_entropy">binary cross entropy</a> loss.<br></span></p>
<p><span>Once we are done training our network, we can carry out testing that will generate some music for us. In this case, test data is fed to the encoder and the outputs from the decoder represent the music generated by the model!<br></span></p>
<p><span>It should be noted that one limitation of Sequence-to-Sequence models is that they get overwhelmed when given very long inputs, and they need other sources of context such as attention in order to focus on specific parts of the input automatically. We will not be going over this aspect as it&rsquo;s beyond the scope of this post.<span><br></span></span></p>
<p><span>From a higher-level perspective, the flow of data through the system looks like this:<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera6.png"><span><br></span></p>
<p><span>The next section walks you through how to set this up with Azure ML Workbench.<br></span></p>
<p><span>Getting Started with Azure Machine Learning<br></span></p>
<p><a href="https://azure.microsoft.com/en-us/blog/tools-for-the-ai-driven-digital-transformation/"><span>Azure Machine Learning</span></a><span> provides data scientists and ML developers with a toolset for data wrangling and experimentation and it includes the following:<br></span></p>
<ul><li><span>AML Workbench. <span><span>See&nbsp;<a href="https://docs.microsoft.com/en-us/azure/machine-learning/preview/quickstart-installation"></a></span>setup and installation<span>&nbsp;documentation. </span><br></span></span></li>
<li><span>AML Experimentation Service. See <a href="https://docs.microsoft.com/en-us/azure/machine-learning/preview/experimentation-service-configuration">configuration</a> documentation. <span><br></span></span></li>
<li><span>AML Model Management. See <a href="https://docs.microsoft.com/en-us/azure/machine-learning/preview/model-management-overview">manage and deploy</a> documentation. <span><br></span></span></li>
</ul><p><span><span><span>We provisioned Data Science VMs (DSVMs) with GPUs and used the remote Docker execution environment provided by Azure ML Workbench (see&nbsp;<a href="https://docs.microsoft.com/fi-fi/azure/machine-learning/preview/how-to-use-gpu"></a></span>details<span>&nbsp;and more information on&nbsp;<a href="https://docs.microsoft.com/en-us/azure/machine-learning/preview/experiment-execution-configuration"></a></span>execution targets<span>) for training models.</span></span> Azure ML allows you to track your run history and model metrics through the <a href="https://docs.microsoft.com/en-us/azure/machine-learning/preview/reference-logging-api">Azure ML Logging API</a> which helps us compare different experiments and compare results visually.<span><br></span></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera7.png"><span><br></span></p>
<p><span><span>Training a Music Generation Model </span><span><br></span></span></p>
<p><span>The Sequence-to-Sequence model described in the previous section is implemented using Keras. In this section we are going to focus on the training setup.<br></span></p>
<p><span>Using Azure ML Workbench for Training on a Remote VM</span></p>
<p><span>The Azure ML Workbench provides an easy way to scale out to environments such as a Data Science VM with GPUs that enables faster training for deep learning models and isolated, reproducible, and consistent runs for all your experiments. The code for this project can be found <a href="https://github.com/Azure/MachineLearning-MusicGeneration">here</a>.<br></span></p>
<p><span>Step1: Setup remote VM as execution target<br></span></p>
<p><span>az ml computetarget attach &ndash;name &ldquo;my_dsvm&rdquo; &ndash;address &ldquo;my_dsvm_ip_address&rdquo; &ndash;username &ldquo;my_name&rdquo; &ndash;password &ldquo;my_password&rdquo; &ndash;type remotedocker<br></span></p>
<p><span>Step 2: Configure my_dsvm.compute<br></span></p>
<p><span>baseDockerImage: microsoft/mmlspark:plus-gpu-0.7.91<br></span><span>nvidiaDocker: true<br></span></p>
<p><span>Step 3: Configure my_dsvm.runconfig<br></span></p>
<p><span>EnvironmentVariables:<br></span><span>&nbsp;&nbsp;&nbsp;&nbsp; &ldquo;STORAGE_ACCOUNT_NAME&rdquo;:<br><span>&nbsp;&nbsp;&nbsp;&nbsp; </span></span><span>&ldquo;STORAGE_ACCOUNT_KEY&rdquo;:<br></span><span>Framework: Python<br></span><span>PrepareEnvironment: true<br></span></p>
<p><span>We use Azure storage for storing training data, pre-trained models and generated music.&nbsp;The storage account credentials are provided as&nbsp;<em>EnvironmentVariables</em>.<br></span></p>
<p><span>Step 4: Conda_dependencies.yml<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera8.png"><span><br></span></p>
<p><span>Step 5: Prepare the remote machine<br></span></p>
<p><span>az ml experiment &ndash;c prepare m_dsvm<br></span></p>
<p><span>Step 6: Run the experiment<br></span></p>
<p><span>az ml experiment submit -c my_dsvm Musicgeneration/train.py<br></span></p>
<p><span>Evaluation Workflow<br></span></p>
<p><span>Comparing Runs</span></p>
<p><span> </span><span>As part of experimenting with machine learning models, we would like to compare effects of different batch sizes and model hyper-parameters. We can visualize this in the run history for different epoch sizes and compare different runs with custom outputs as shown below.<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera9.png"><span><br></span></p>
<p><span>The next figure below shows how we can compare runs for 10, 50 and 100 epochs and look at the corresponding loss curves.<br></span></p>
<p><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120517_2110_MusicGenera10.png"><span><br></span></p>
<p><span>Scoring = Music Generation!<br></span></p>
<p><span>Now you can generate music by loading the models created in the training step and calling <em>model.predict()</em> to generate some music. The code for this is in MusicGeneration/score.py.<br></span></p>
<p><span>az ml experiment submit -c my_dsvm Musicgeneration/score.py<br></span></p>
<p><span>Summary<br></span></p>
<p><span>In this blog post, we showed you how to build your own deep learning music generation model using Azure Machine Learning. This gives you a framework for agile experimentation with fast iterations and provides an easy path for scaling up and out to remote environments such as Data Science VMs with GPUs.<br></span></p>
<p><span>Once you have an end-to-end deep learning model that can produce music, you can experiment with different sequence lengths and different model architectures and listen to their effects on the music generated. Happy music generation!<br></span></p>
<p><span>Erika</span></p>
<p>&nbsp;</p>
<p><span><strong>Resources<br></strong></span></p>
<ul><li><span>Code for this project can be found <a href="https://github.com/Azure/MachineLearning-MusicGeneration"><span>here</span></a><span>.</span><br></span></li>
<li><span>Azure Machine Learning <a href="https://docs.microsoft.com/en-us/azure/machine-learning/preview/quickstart-installation">documentation</a>.<br></span></li>
<li><span>Dataset: Scale-chords from <a href="http://www.feelyoursound.com/scale-chords/">feelyoursound.com</a>.<br></span></li>
</ul><p><span><strong>Acknowledgements</strong></span></p>
<p><span>Thanks to Wee Hyong Tok and Mathew Salvaris for their guidance and for reviewing this article, and to Matt Winkler, Hai Ning and Serina Kaye for all their help with Azure Machine Learning. </span></p>]]></description>
						<author>Cortana Intelligence and ML Blog Team</author>
						<source url="https://blogs.technet.microsoft.com/machinelearning/feed/">Cortana Intelligence and Machine Learning Blog</source>
						<comments>https://blogs.technet.microsoft.com/machinelearning/2017/12/06/music-generation-with-azure-machine-learning/feed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/on-the-biases-in-data.html</guid>
						<pubDate>Tue, 05 Dec 2017 15:15:53 +0000</pubDate>
						<relativeTime>5 days ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[On the biases in data]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/on-the-biases-in-data.html</link>
						<description><![CDATA[<p>Whether we're developing statistical models, training machine learning recognizers, or developing AI systems, we start with data. And while the suitability of that data set is, lamentably, sometimes measured by its size, it's always important to reflect on where those data come from. Data are not neutral: the data we choose to use has profound impacts on the resulting systems we develop. A recent article in Microsoft's AI Blog discusses the inherent biases found in many data sets:</p>
<blockquote>
<p>&ldquo;The people who are collecting the datasets decide that, &lsquo;Oh this represents what men and women do, or this represents all human actions or human faces.&rsquo; These are types of decisions that are made when we create what are called datasets,&rdquo; she said. &ldquo;What is interesting about training datasets is that they will always bear the marks of history, that history will be human, and it will always have the same kind of frailties and biases that humans have.&rdquo; <br>&mdash; <a href="https://www.microsoft.com/en-us/research/people/kate/">Kate Crawford</a>, Principal Researcher at Microsoft Research and co-founder of AI Now Institute.</p>
<p>&ldquo;When you are constructing or choosing a dataset, you have to ask, &lsquo;Is this dataset representative of the population that I am trying to model?&rsquo;&rdquo;<br>&mdash; <a href="https://www.microsoft.com/en-us/research/people/wallach/">Hanna Wallach</a>, Senior Researcher at&nbsp;Microsoft Research NYC.&nbsp;</p>
</blockquote>
<p>The article discusses the consequences of the data sets that aren't representative of the populations they are set to analyze, and also the consequences of the lack of diversity in the fields of AI research and implementation. Read the complete article at the link below.</p>
<p>Microsoft AI Blog:&nbsp;Debugging data: <a href="https://blogs.microsoft.com/ai/2017/12/04/debugging-data-microsoft-researchers-look-ways-train-ai-systems-reflect-real-world/">Microsoft researchers look at ways to train AI systems to reflect the real world</a></p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/on-the-biases-in-data.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/ml-server-ai-path.html</guid>
						<pubDate>Mon, 04 Dec 2017 20:40:45 +0000</pubDate>
						<relativeTime>6 days ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[AI School: Microsoft R and SQL Server ML Services]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/ml-server-ai-path.html</link>
						<description><![CDATA[<p>If you'd like to learn how you use R to develop AI applications, the Microsoft AI School now features a <a href="https://aischool.microsoft.com/learning-paths/3grXSTv2w8cKMu8ayQ2osq/modules/3UhXWkswNis0GKkaGCisqI">learning path focused on Microsoft R and SQL Server ML Services</a>. This learning path includes eight modules, each comprising detailed tutorials and examples:</p>
<ul><li><a href="https://github.com/Azure/LearnAnalytics-DoingDataAnalysisusingRandMicrosoftROpen">Doing Data Analysis using R</a></li>
<li><a href="https://github.com/Azure/LearnAnalytics-AnalyzingBigDataWithMRS">Analyzing Big Data with Microsoft R Server</a></li>
<li><a href="https://github.com/Azure/LearnAnalytics-SQLServerRServices">Introduction to SQL Server R Services</a></li>
<li><a href="https://github.com/Azure/LearnAnalytics-MicrosoftRforArchitects">Microsoft R for SQL Architects</a></li>
<li><a href="https://github.com/Azure/LearnAnalytics-MicrosoftRforSQLServerProfessional">Microsoft R for T-SQL Professionals</a></li>
<li><a href="https://azure.github.io/LearnAnalytics-mr4ds/">Microsoft R for Data Science</a></li>
<li><a href="https://github.com/Azure/LearnAnalytics-mrs-spark">Microsoft R with Spark on HDInsight</a></li>
<li><a href="https://github.com/Azure/LearnAnalytics-mr4ds-spark">Microsoft R Server for Data Science (with Spark)</a></li>
</ul><p>All of the <a href="https://aischool.microsoft.com/learning-paths">Microsoft AI School</a> learning paths are free to access, and the content is hosted on Github (where feedback is welcome!). You can access this course and many others at the link below.</p>
<p>Microsoft AI School:&nbsp;<a href="https://aischool.microsoft.com/learning-paths/3grXSTv2w8cKMu8ayQ2osq/modules/3UhXWkswNis0GKkaGCisqI">Microsoft R and SQL Server ML Services</a>&nbsp;</p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/ml-server-ai-path.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">https://blogs.technet.microsoft.com/machinelearning/2017/12/01/microsofts-nips-2017-long-beach-ca/</guid>
						<pubDate>Fri, 01 Dec 2017 21:21:30 +0000</pubDate>
						<relativeTime>1 week ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[Microsoft @ NIPS 2017, Long Beach, CA]]></title>
						<link>https://blogs.technet.microsoft.com/machinelearning/2017/12/01/microsofts-nips-2017-long-beach-ca/</link>
						<description><![CDATA[<p><span><em>Re-posted from the Microsoft Research blog.<br></em></span></p>
<p><a href="https://nips.cc/"><img alt="" src="https://msdnshared.blob.core.windows.net/media/2017/12/120117_2116_MicrosoftsN1.png" border="0"></a><span><br></span></p>
<p><span>The <a href="https://nips.cc/">thirty-first annual conference on Neural Information Processing Systems</a> (NIPS) starts on Monday next week, and is being held in Long Beach, CA, from December 4<sup>th</sup> through 9<sup>th</sup>, 2017.<br></span></p>
<p><span>The event, which is completely sold out, is a multi-track machine learning and computational neuroscience conference that includes invited talks, demonstrations, symposia and oral and poster presentations of refereed papers.<br></span></p>
<p><span>Microsoft has always had a strong presence at NIPS and this year is no different. We have <a href="https://www.microsoft.com/en-us/research/event/microsoft-research-nips-2017/">several employees taking part at the event</a>, including as organizing committee members, workshop and symposium organizers, invited speakers and more. Our researchers and engineers have co-authored dozens of accepted papers, contributed to several posters&nbsp;and are also involved in key workshops that are a part of the event.<br></span></p>
<p><span>For complete details, including the list of accepted papers, posters and workshops from our team, as well as links to ML-related job opportunities at Microsoft, be sure to <a href="https://www.microsoft.com/en-us/research/event/microsoft-research-nips-2017/">check out the original post here</a>.<br></span></p>
<p><span>We look forward to meeting several of you at NIPS Long Beach next week!<br></span></p>
<p><span>ML Blog Team<br></span></p>
<p><span><br></span></p>]]></description>
						<author>Cortana Intelligence and ML Blog Team</author>
						<source url="https://blogs.technet.microsoft.com/machinelearning/feed/">Cortana Intelligence and Machine Learning Blog</source>
						<comments>https://blogs.technet.microsoft.com/machinelearning/2017/12/01/microsofts-nips-2017-long-beach-ca/feed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/because-its-friday-1-pixel-moon.html</guid>
						<pubDate>Fri, 01 Dec 2017 21:11:52 +0000</pubDate>
						<relativeTime>1 week ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[Because it&#8217;s Friday: The Whole of the Moon]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/because-its-friday-1-pixel-moon.html</link>
						<description><![CDATA[<p>As we've noted before, the Solar System is a big place. You can <a href="http://blog.revolutionanalytics.com/2015/04/because-its-friday-slow-light.html">watch a voyage from the Sun to Jupiter</a>, and it takes 45 minutes <em>at the speed of light</em>. A scale model of the Solar System, with the Sun the size of a weather balloon, is <a href="http://blog.revolutionanalytics.com/2015/09/a-scale-model-of-the-solar-system.html">3.5 miles across</a> ... and that's not even including Pluto. And this virtual scale model, a browser-based rendition by <a href="http://www.joshworth.com/">John Worth</a> with the <a href="http://joshworth.com/dev/pixelspace/pixelspace_solarsystem.html">moon just one pixel in size</a>, is no less impressive. I can't do it justice here &mdash; the site surely holds the record for the widest horizontal scrollbar on any page on the Web &mdash; so here's a little snippet of the Earth-Moon system:</p>
<p><a href="http://joshworth.com/dev/pixelspace/pixelspace_solarsystem.html"><img alt="One pixel moon" border="0" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c9392796970b-800wi" title="One pixel moon"></a></p>
<p>While you can use the astrological symbols at the top to jump to the planets, try manually scrolling to get the full effect of the vast spaces between them. There are also some useful tidbits to discover along the way...</p>
<p>That's all from us here at the blog for this week. Have a great weekend, and we'll see you back here on Monday. Enjoy!<a href="http://joshworth.com/dev/pixelspace/pixelspace_solarsystem.html"></a></p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/because-its-friday-1-pixel-moon.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/12/sex-marriage-survey.html</guid>
						<pubDate>Fri, 01 Dec 2017 19:23:49 +0000</pubDate>
						<relativeTime>1 week ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[A case study in messy data analysis: the Australian same-sex marriage survey]]></title>
						<link>http://blog.revolutionanalytics.com/2017/12/sex-marriage-survey.html</link>
						<description><![CDATA[<p>Last month the Australian people signaled their approval of legalizing same-sex marriage <a href="http://www.abc.net.au/news/2017-11-15/same-sex-marriage-results-ssm/9145636">by a 62%:38% margin in a national survey</a>. (On a personal note, I was elated and relieved by the result: my husband and I have discussed eventually retiring to Australia, and with this decision our marriage would be recognized there.) While fears of a surprise Brexit-like electoral backlash proved unfounded, researchers including R user Miles McBain <a href="https://medium.com/@miles.mcbain/tidying-the-australian-same-sex-marriage-postal-survey-data-with-r-5d35cea07962">explored the results for correlations to demographic variables</a>. This process wasn't as simple as it might have been though: the Australian Bureau of Statistics <a href="https://marriagesurvey.abs.gov.au/results/downloads.html">released the results</a> as a pair of Excel files that violate just about every <a href="http://blog.revolutionanalytics.com/2017/11/good-practices-spreadsheets.html">good practice for sharing data in spreadsheets</a>:</p>
<p><a href="https://medium.com/@miles.mcbain/tidying-the-australian-same-sex-marriage-postal-survey-data-with-r-5d35cea07962"><img alt="Survey-results" border="0" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01bb09dc35a0970d-800wi" title="Survey-results"></a></p>
<p>Miles shares the R code he used to extract useful data from this spreadsheet as a <a href="https://medium.com/@miles.mcbain/tidying-the-australian-same-sex-marriage-postal-survey-data-with-r-5d35cea07962">blog post that makes a great case study in dealing with messy data</a>&nbsp;using R. The post demonstrates how he used the <code><a href="https://www.rdocumentation.org/packages/readxl/versions/1.0.0/topics/read_excel">read_excel</a></code> function (from <a href="http://readxl.tidyverse.org/">readxl package</a>) to extract specific sub-tables from the spreadsheet by specifying row and column ranges, and then use the <a href="http://dplyr.tidyverse.org/">dplyr package</a> to clean up and merge the data. If you want to explore the data yourself, you can <a href="https://github.com/MilesMcBain/tidy_aus_ssm/">find the R code and the source data in this Github repository</a>.</p>
<p>In a follow-up post, Miles combines the <a href="https://medium.com/@miles.mcbain/combining-australian-census-data-with-the-same-sex-marriage-postal-survey-in-r-39d9b2082249">same-sex marriage survey data with Australian Census data</a> to explore various demographic relationships. Unlike the US Census data (which is easily accessible in R thanks to the <a href="https://mran.microsoft.com/package/tidycensus">tidycensus package</a>), there's no interface package for Australian Census data. (Selected tables are available in the <a href="https://mran.microsoft.com/package/Census2016">Census2016 package</a>, however.) Instead, Miles demonstrates how to use R to download and extract data from the the "<a href="https://datapacks.censusdata.abs.gov.au/datapacks/">Census DataPacks</a>" (CSV data files and Excel data dictionaries) provided by the Australian Bureau of Statistics.&nbsp; Yet more data wrangling allows Miles to create summary charts of the responses, such as this chart of proportion voting No by percent of the district population declaring a religious affiliation, broken down by state. As you may expect, those districts with more religious populations voted No at greater rates.</p>
<p><a href="https://medium.com/@miles.mcbain/combining-australian-census-data-with-the-same-sex-marriage-postal-survey-in-r-39d9b2082249"><img alt="Religious-affliliation" border="0" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c939206b970b-800wi" title="Religious-affliliation"></a></p>
<p>Both of these post provide great examples of working with government data, which is often provided in inconvenient formats with messy structures. Follow the links below for step-by-step guides, including the R code used to extract the data, structure it for analysis, and create useful charts.</p>
<p>Medium (Miles McBain):&nbsp;<a href="https://medium.com/@miles.mcbain/tidying-the-australian-same-sex-marriage-postal-survey-data-with-r-5d35cea07962">Tidying the Australian Same Sex Marriage Postal Survey Data with R</a>;&nbsp;<a href="https://medium.com/@miles.mcbain/combining-australian-census-data-with-the-same-sex-marriage-postal-survey-in-r-39d9b2082249">Combining Australian Census data with the Same Sex Marriage Postal Survey in R</a></p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/12/sex-marriage-survey.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">https://blogs.technet.microsoft.com/dataplatforminsider/2017/11/30/azure-database-migration-service-preview-brings-the-lift-and-shift/</guid>
						<pubDate>Thu, 30 Nov 2017 20:00:15 +0000</pubDate>
						<relativeTime>1 week ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[Azure Database Migration Service Preview brings the “lift and shift”]]></title>
						<link>https://blogs.technet.microsoft.com/dataplatforminsider/2017/11/30/azure-database-migration-service-preview-brings-the-lift-and-shift/</link>
						<description><![CDATA[<p>It&rsquo;s time to get excited&mdash;the <a target="_blank" href="https://azure.microsoft.com/en-us/campaigns/database-migration/" rel="noopener">Azure Database Migration Service Public Preview</a> is here to help you move on-premises SQL Servers to the cloud with near-zero downtime.</p>
<p>There&rsquo;s never been a better time to start migrating your database to the cloud. Getting ahead in business today means moving to the cloud so that your company can grow and succeed.</p>
<p>The Azure Database Migration Service Preview is the quickest and easiest way for businesses to migrate on-premises databases to the cloud. It&rsquo;s a fully managed migration service designed to work together with our time-tested migration engines such as the <a target="_blank" href="https://www.microsoft.com/en-us/download/details.aspx?id=53595" rel="noopener">Data Migration Assistant</a>, the <a target="_blank" href="https://blogs.msdn.microsoft.com/datamigration/tag/dea/" rel="noopener">Database Experimentation Assistant</a>, and <a target="_blank" href="https://docs.microsoft.com/en-us/sql/ssma/sql-server-migration-assistant" rel="noopener">SQL Server Migration Assistant</a>. These tools are tuned to ensure the best migration experience, whether upgrading from legacy versions of Microsoft SQL Server or moving from sources such as Oracle, Sybase, DB2, MySQL, and others. The new service uses a guided, easy-to-implement process to streamline tasks. Regular operations can continue normally during the migration.</p>
<p><a target="_blank" href="https://azure.microsoft.com/en-us/campaigns/database-migration/" rel="noopener">Azure Database Migration Service Public Preview</a> uses this functionality to provide rich orchestration capabilities that enable you to organize your databases into project(s) and perform source assessment, schema and data conversion, and validation activities. These activities can be assigned to one or more compute nodes to meet your budget and timeframe goals. With these capabilities, <a target="_blank" href="https://azure.microsoft.com/en-us/campaigns/database-migration/" rel="noopener">Azure Database Migration Service Public Preview</a> makes it easy to plan migration tasks, run proof-of-concept migrations, author scripting for automation, and ensure that your final production migration to the Microsoft Data Platform is friction-free.</p>
<p>All these capabilities allow you to focus on what&rsquo;s important for your business and accelerate your Data Estate transformation with the cloud. Now is a great time to find out more about how to make your organization&rsquo;s database migration as smooth and seamless as possible. To start, check out the <a target="_blank" href="https://datamigration.microsoft.com/" rel="noopener">Database Migration Guide</a>, which allows you to customize your migration plan based on your organization&rsquo;s source and target data platforms, along with other key criteria. For a broad scope of information about cloud migration in general, visit the <a target="_blank" href="https://azure.microsoft.com/migrate" rel="noopener">Azure Migration Center</a>.</p>
<p>Most importantly, be sure to take advantage of the <a target="_blank" href="https://azure.microsoft.com/en-us/campaigns/database-migration/" rel="noopener">Azure Database Migration Service Public Preview</a> today.</p>]]></description>
						<author>SQL Server Team</author>
						<source url="https://blogs.technet.microsoft.com/dataplatforminsider/feed/">SQL Server Blog</source>
						<comments>https://blogs.technet.microsoft.com/dataplatforminsider/2017/11/30/azure-database-migration-service-preview-brings-the-lift-and-shift/feed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/11/r-343-released.html</guid>
						<pubDate>Thu, 30 Nov 2017 18:18:45 +0000</pubDate>
						<relativeTime>1 week ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[R 3.4.3 released]]></title>
						<link>http://blog.revolutionanalytics.com/2017/11/r-343-released.html</link>
						<description><![CDATA[<p><a href="https://stat.ethz.ch/pipermail/r-announce/2017/000623.html">R 3.4.3 has been released</a>, as announced by the R Core team today. As of this writing, only the source distribution (for those that build R themselves) is available, but binaries for Windows, Mac and Linux should appear on your local CRAN mirror within the next day or so.</p>
<p>This is primarily a bug-fix release. It fixes an issue with incorrect time zones on MacOS High Sierra, and some issues with handling Unicode characters. (Incidentally, representing international and special characters is something that R takes great care in handling properly. It's not an easy task: a 2003 essay by Joel Spolsky describes the <a href="https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/">minefield that is character representation</a>, and not much has changed since then.) You can check out the <a href="https://www.joelonsoftware.com/2003/10/08/the-absolute-minimum-every-software-developer-absolutely-positively-must-know-about-unicode-and-character-sets-no-excuses/">complete list of changes here</a>. Whatever your platform, R 3.4.3 should be backwards-compatible will other R versions in the R 3.4.x series, and so your scripts and packages should continue to function as they did before.</p>
<p>The codename for this release is "Kite-Eating Tree", and as with all R codenames this is a references to a <a href="http://www.gocomics.com/peanuts/1967/02/19"><em>classic Peanuts episode</em></a>. If you're interested in the source of other R release names,&nbsp;Lucy D'Agostino McGowan provides the <a href="http://livefreeordichotomize.com/2017/09/28/r-release-names/">Peanuts references for R release names</a> back to R 2.14.0.</p>
<p><a href="http://livefreeordichotomize.com/2017/09/28/r-release-names/"><img alt="Kite-eating-tree" border="0" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b7c938cb54970b-800wi" title="Kite-eating-tree"></a></p>
<p>r-devel mailing list:&nbsp;<a href="https://stat.ethz.ch/pipermail/r-announce/2017/000623.html">R 3.4.3 is released</a></p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/11/r-343-released.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">http://blog.revolutionanalytics.com/2017/11/how-to-generate-a-secret-santa-list-with-r.html</guid>
						<pubDate>Wed, 29 Nov 2017 22:49:23 +0000</pubDate>
						<relativeTime>1 week ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[How to generate a Secret Santa list with R]]></title>
						<link>http://blog.revolutionanalytics.com/2017/11/how-to-generate-a-secret-santa-list-with-r.html</link>
						<description><![CDATA[<p>Several recent blog posts have explored the <a href="https://en.wikipedia.org/wiki/Secret_Santa">Secret Santa</a> problem and provided solutions in R. This post provides a roundup of various solutions and how they are implemented in R.&nbsp;</p>
<p>If you wanted to set up a "Secret Santa" gift exchange at the office, you could put everyone's name into a hat and have each participant draw a name at random. The problem is that someone might draw their own name, but if that happens you can just reshuffle all the names back into the hat and start the process over. That's essentially what the R code below, from a <a href="http://selbydavid.com/2016/12/07/santa/">blog post by David Selby</a>, does:</p>
<blockquote>
<p dir="ltr" lang="it">&#128584;&#127873; w/ code:<br>"Secret Santa in R" &#9999; <a href="https://twitter.com/TeaStats?ref_src=twsrc%5Etfw">@TeaStats</a><a href="https://t.co/HFrh8w6CWU">https://t.co/HFrh8w6CWU</a> <a href="https://twitter.com/hashtag/rstats?src=hash&amp;ref_src=twsrc%5Etfw">#rstats</a> <a href="https://t.co/cGfNA45CbG">pic.twitter.com/cGfNA45CbG</a></p>
&mdash; Mara Averick (@dataandme) <a href="https://twitter.com/dataandme/status/934119945587298304?ref_src=twsrc%5Etfw">November 24, 2017</a></blockquote>
<p>That's not an entirely satisfying solution (at least to me), with all of the having to check for self-giving and restarting if so. <a href="http://notstatschat.tumblr.com/post/167885514256/secret-santa-collisions">Thomas Lumley calculates</a> that the chance of requiring a do-over is about 63% (for more than 2 participants, anyway), and on average you'd need about 2.7 tries to get a "valid" gift list. (This is an example of the <a href="https://en.wikipedia.org/wiki/Negative_binomial_distribution#Expectation">negative binomial</a> distribution in action: we keep on drawing a set of names from the hat, until we get a set that has no-one giving to themselves. You can generate 100 examples of this playing out in R with <code>rnbinom(100,1, exp(-1))+1</code> &mdash; the <code>+1</code> is for the final successful draw from the hat.)</p>
<p>An easier way might be simply to seat all the participants in random order in a circle and assign them to give a gift to the person on their right. This is easy to do in R, and doing it in code has the benefit of keeping the recipients secret from each other. First, let's select 10 names from the <a href="https://mran.microsoft.com/package/babynames">babynames</a> dataset:</p>

<div>
<pre><span>&gt;</span> library(babynames)
<span>&gt;</span> santas <span>&lt;-</span> sample(babynames<span>$</span>name, <span>10</span>, prob<span>=</span>babynames<span>$</span>prop)&nbsp;</pre>
</div>
<p>(Using <code>prob=babynames$prop</code> selects names according to their prevalence in US births 1880-2015.) Then, it's a simple matter of reordering the names at random, and assigning a gift to the next in line:</p>

<div>
<pre><span>&gt;</span> p <span>&lt;-</span> sample(santas)
<span>&gt;</span> cbind(santa<span>=</span>p, recipient<span>=</span>c(tail(p,<span>-1</span>),p[<span>1</span>]))
      santa       recipient  
 [<span>1</span>,] <span>"Sherman"</span>   <span>"Shayna"</span>   
 [<span>2</span>,] <span>"Shayna"</span>    <span>"Elizabeth"</span>
 [<span>3</span>,] <span>"Elizabeth"</span> <span>"Mary"</span>     
 [<span>4</span>,] <span>"Mary"</span>      <span>"Kathleen"</span> 
 [<span>5</span>,] <span>"Kathleen"</span>  <span>"Russell"</span>  
 [<span>6</span>,] <span>"Russell"</span>   <span>"James"</span>    
 [<span>7</span>,] <span>"James"</span>     <span>"Arlene"</span>   
 [<span>8</span>,] <span>"Arlene"</span>    <span>"Ruth"</span>     
 [<span>9</span>,] <span>"Ruth"</span>      <span>"Darryl"</span>   
[<span>10</span>,] <span>"Darryl"</span>    <span>"Sherman"</span>  
</pre>
</div>
<p>Now, this "sit in a circle" process isn't <em>quite</em> the same as the "keep on drawing names from the hat" process. In the above example, the 10 names form a cycle, and it will never happen that Arlene gives to Ruth and Ruth gives to Arlene. Nonetheless, I think it's the simplest and fairest way of generating a Secret Santa list.</p>
<p>Thinking of the gift-giving relationship as a graph, the process above generates a <a href="https://en.wikipedia.org/wiki/Hamiltonian_path">Hamiltonian path</a> through each of the recipients, and never generates more than one clique. Tristan Mahr explores the <a href="https://tjmahr.github.io/secret-santa-graph-traversal/">graph-theory nature of the Secret Santa problem</a> in a blog post. There, he uses the <a href="https://mran.microsoft.com/package/DiagrammeR">DiagrammeR package</a>&nbsp;to solve variants of the Secret Santa problem by constructing graphs, which can created and visualized quite easily in R.</p>
<p><a href="https://tjmahr.github.io/secret-santa-graph-traversal/"><img alt="Santagraph" src="http://a4.typepad.com/6a0105360ba1c6970c01b7c9388c8c970b-400wi" title="Santagraph"></a></p>
<p>Finally, <a href="http://livefreeordichotomize.com/2017/11/15/secret-sampling/"><span>Sarah Lotspeich and Lucy D'Agostino McGowan</span></a>&nbsp;take the whole process one step further and show how to generate emails each participant using the <a href="https://github.com/ropenscilabs/ponyexpress">ponyexpress package</a>, notifying them of their Secret Santa recipient.</p>]]></description>
						<author>David Smith</author>
						<source url="http://blog.revolutionanalytics.com/atom.xml">Revolutions</source>
						<comments>http://blog.revolutionanalytics.com/2017/11/how-to-generate-a-secret-santa-list-with-r.htmlfeed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">https://blogs.technet.microsoft.com/dataplatforminsider/2017/11/29/sql-server-2017-the-worlds-first-enterprise-class-diskless-database/</guid>
						<pubDate>Wed, 29 Nov 2017 17:00:17 +0000</pubDate>
						<relativeTime>2 weeks ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[SQL Server 2017: The world’s first enterprise-class diskless database]]></title>
						<link>https://blogs.technet.microsoft.com/dataplatforminsider/2017/11/29/sql-server-2017-the-worlds-first-enterprise-class-diskless-database/</link>
						<description><![CDATA[<p><em>This post is authored by Bob Ward, Principal Architect, and Jamie Reding, Senior Program Manager and Performance Architect, Microsoft Database Systems Group.</em></p>
<p>Perhaps you saw the keynote at the recent <a href="https://blogs.technet.microsoft.com/dataplatforminsider/2017/10/30/top-five-reasons-to-join-us-for-the-pass-summit-2017-day-1-keynote/">PASS 2017 Summit</a> where Microsoft demonstrated the performance of the <strong>world&rsquo;s first enterprise-class &ldquo;diskless database&rdquo;</strong>. This demonstration showed how Microsoft, Hewlett Packard Enterprise (HPE), and SUSE Linux Enterprise Server partnered together to deliver &gt; 5x performance on analytic queries directly against storage at up to 50 percent of the cost.</p>
<p>Recently HPE published a new world record <a href="http://h20195.www2.hpe.com/V2/GetDocument.aspx?docname=a00036622enw">1TB TPC-H benchmark result&sup1;</a>&nbsp;using this configuration with their DL380 Gen10 Server showing 1,009,065 QphH at an incredible price/performance of $0.47 USD per QphH. Performance and price are achieved by combining the power of SQL Server 2017, HPE&rsquo;s scalable persistent memory, and SUSE Linux Enterprise 12 SP3 Persistent Memory Support.</p>
<p><img src="https://msdnshared.blob.core.windows.net/media/2017/11/diskless-database.png" alt="" width="600" height="338"></p>
<p>HPE&rsquo;s scalable persistent memory is a new innovation which combines standard memory with the persistence of standard storage. It allows database engines like SQL Server 2017 to retrieve data from its data files in a matter of seconds.</p>
<p>To see this technology in action, <a href="https://youtu.be/QElCYYoVA3Q">check out this video</a>. To learn more about this amazing result and technology, read <a href="https://community.hpe.com/t5/Servers-The-Right-Compute/Diskless-yes-diskless-database-with-HPE-scalable-persistent/ba-p/6989039#.WhR8OEqnGiM">HPE&rsquo;s blog post</a> and <a href="https://www.suse.com/communities/blog/hpe-microsoft-suse-present-industry-leading-performance-enterprise-class-sql-server-diskless-database/">SUSE&rsquo;s blog post</a>. To learn more about SQL Server on SUSE Linux Enterprise Server, check out SUSE&rsquo;s <a href="https://www.suse.com/partners/alliance/microsoft/sql-server-on-linux/">SQL Server on Linux website</a>.</p>
<p>SQL Server 2017 is the world leader in TPC-H performance, price, and value and continues to demonstrate that it is one of the fastest databases on the planet, in your cloud or ours.</p>
<h3>References</h3>
<ul><li><em>&sup1;&nbsp;1TB non-clustered TPC-H result <a href="http://www.tpc.org/3331">http://www.tpc.org/3331</a> as of November 21, 2017.</em></li>
</ul>]]></description>
						<author>SQL Server Team</author>
						<source url="https://blogs.technet.microsoft.com/dataplatforminsider/feed/">SQL Server Blog</source>
						<comments>https://blogs.technet.microsoft.com/dataplatforminsider/2017/11/29/sql-server-2017-the-worlds-first-enterprise-class-diskless-database/feed</comments>
					</item>
        
					<item>
						<guid isPermaLink="true">https://blogs.msdn.microsoft.com/sqlreleaseservices/cumulative-update-2-for-sql-server-2017-rtm/</guid>
						<pubDate>Wed, 29 Nov 2017 01:49:25 +0000</pubDate>
						<relativeTime>2 weeks ago</relativeTime>
						<channelId>DataPlatform</channelId>
						<title><![CDATA[Cumulative Update #2 for SQL Server 2017 RTM]]></title>
						<link>https://blogs.msdn.microsoft.com/sqlreleaseservices/cumulative-update-2-for-sql-server-2017-rtm/</link>
						<description><![CDATA[<p>The 2nd cumulative update release for SQL Server 2017 RTM is now available for download at the Microsoft Downloads site. Please note that registration is no longer required to download Cumulative updates. To learn more about the release or servicing model, please visit:</p>
<ul><li>CU2 KB Article: <a href="https://support.microsoft.com/en-us/help/4052574">https://support.microsoft.com/en-us/help/4052574</a></li>
</ul><p>Starting with SQL Server 2017, we are adopting a new modern servicing model. Please refer to our blog for more details on <a href="https://blogs.msdn.microsoft.com/sqlreleaseservices/announcing-the-modern-servicing-model-for-sql-server/">Modern Servicing Model</a> for SQL Server</p>
<ul><li>Microsoft&reg; SQL Server&reg; 2017 RTM Latest Cumulative Update: <a href="https://www.microsoft.com/download/details.aspx?id=56128">https://www.microsoft.com/download/details.aspx?id=56128</a></li>
<li>Update Center for Microsoft SQL Server: <a href="http://technet.microsoft.com/en-US/sqlserver/ff803383.aspx">http://technet.microsoft.com/en-US/sqlserver/ff803383.aspx</a></li>
</ul>]]></description>
						<author>SQL Server Engineering Team</author>
						<source url="https://blogs.msdn.microsoft.com/sqlreleaseservices/feed/">SQL Server Release Services</source>
						<comments>https://blogs.msdn.microsoft.com/sqlreleaseservices/cumulative-update-2-for-sql-server-2017-rtm/feed</comments>
					</item>
        			</channel>
		</rss>
  