The Internet of Things

The Internet of Things (IoT) starts with your things, the things that matter the most to you and your business. IoT is at an inflection point where the right technologies are coming together, and we are able to connect devices to the cloud and leverage streams of data that were previously out of reach. It's a great time to take a look at game-changing technologies you can use today to make your IoT ideas stand out from the crowd.

Light up the data from your devices with Windows 10 IoT Core and Microsoft Azure. Welcome to the internet of your intelligent things.

From device to cloud

Start at the device end with Windows 10 IoT Core OS. The power of Windows on small devices, such as the Raspberry Pi 3, gives you the opportunity to leverage the Universal Windows Platform.

Now send your data to the cloud. Data ingestion from social media, the web, or an application is suited to Azure Event Hubs. For apps on remote devices, Azure IoT Hub is a highly scalable, publish-subscribe service that can ingest millions of events per second so that you can connect, control, and monitor millions of IoT devices remotely.

The end-to-end IoT picture has many moving parts, so why not start with preconfigured, popular solutions? Azure IoT Suite helps you to get started quickly and the solutions it contains, including remote monitoring and predictive maintenance, can be tailored to your needs.

Real-time data analytics

Feel the demand for real-time answers? Hearing a lot about Hot and Cold paths in IoT scenarios? Look at the capabilities that Azure Stream Analytics, Storm, and Spark on Azure can provide. Businesses are no longer only looking for prescriptive analytics, they want to know what is happening right now, and this can be integrated into your IoT pipeline with limited new coding.

Machine learning

Advanced analytics is the business of deriving actionable insight from data. The most exciting technology in this space is machine learning—the use of advanced statistics to make predictions. Examples in use today cut across all aspects of business, from basket analysis in retail to predictive maintenance in manufacturing and fraud detection in banking.

Introducing the Cortana Intelligence Suite

The Cortana Intelligence Suite provides a number of ways to embed advanced analytics into processes, websites, and applications. These range from “off the shelf” APIs in Project Oxford that add intelligence to text, speech, and images to bringing your own code into your favorite development language to Azure. Azure Machine Learning (ML) sits between these two and allows for the rapid development of APIs based on world-class, built-in algorithms—all from a web browser. It can also be used by more expert data scientists by bringing their own code in R or Python. Two other approaches available are Apache Spark for HDInsight and Microsoft R Server.

Bring your own R

R is a widely-used open source language for statistics , machine learning, and data mining. How you interact with R will depend on what you want to achieve. R algorithms and analysis can be developed in Visual Studio via the R add-in and then used as an algorithm or module in Azure ML. R can also create visualization directly in Power BI.

• R in Visual Studio
• R in Power BI

Bring your own Python

Python has evolved and been extended to give it many of the same statistical and analytical features traditionally associated with R. Jupyter Notebooks in Azure Machine Learning help you develop and debug and modules and algorithms within the browser alongside your experiments.

Visualizations for everyone everywhere

Today’s visualizations need to be easy to consume on a variety of devices and increasingly need to present up-to-the-minute information, as well as historical trends to inform strategy. Users also want to interact with the visualizatons we design, dig into details, and slice the analysis in different ways. Designing so that this can be done, through touch, natural language queries, and voice, is evermore important.

Introducing Power BI

Microsoft’s answer to the visualization needs of modern business is Power BI, an online service that is part of Office 365. This new service can aggregate data from traditional internal sources, as well as newer online services like SalesForce and Google Analytics. Both the Power BI desktop designer and web design experience are user focused. The only deeper technical knowledge required is setting up the gateway to securely connect internal sources to Power BI to use its more advanced features like real-time updates from live feeds and the creation of custom visualizations.

Extending the power of Power BI

Power BI is not only a powerful user tool. In the hands of BI professionals and developers, it becomes more flexible and powerful. Data can be integrated from dozens of cloud services and social media engines, as well as real-time updating from live feeds via Stream Analytics. Data scientists can bring R to Power BI, and developers can use Typescript to create their own visualizations.

Microsoft Power BI Embedded is an Azure service that enables application developers to add interactive Power BI reports into their own applications.

Data at scale

Typically, the term "big data" is misunderstood as simply referring to the volume of data that is being processed. However, variety and velocity are also essential in understanding why new techniques and technologies have had to be developed to process data at scale. Variety recognizes the different types of data we typically store, such as logs, text, images, and video. Velocity is the speed at which new information arrives, and is characterized by news feeds, social media, and data from sensors and devices.

Volume, velocity, and variability

Getting insight from data at scale has its own unique challenges, in addition to those of dealing with more traditional volumes and structures. Special storage and computation is needed, as well as new skills to fully utilize it. However, cloud services are making this easier and more agile, allowing ideas and techniques to be trialed without heavy upfront investment. Microsoft has recently launched Azure Data Lake backed by Hadoop compatible storage (HDFS). Fronting this is Azure Data Factory, which orchestrates loading and processing when projects need to be production-ready.

Hadoop Distributed Filing System (HDFS)

Apache Hadoop is the de facto standard for processing big data, and Azure Data Lake Storage (ADLS) is fully HDFS-compliant, enabling you to quickly lay on the particular Hadoop tooling you need—for example, Hive, Storm, Spark, and Azure Data Lake Analytics. ADLS persists when a cluster is removed, which means that other tools like Power BI can access the results of big data processing without the need to continue to run the cluster.

The many tools of big data

Hive, Storm, and Spark all have their place in bringing analytical techniques to bear on big data in the Hadoop ecosystem, and Azure HD Insight supports all of these running on familiar Linux clusters; or, you can orchestrate your own cluster by creating one based on Cloudera or Horton Works from the Azure VM depot. Microsoft has also launched Azure Data Lake Analytics, which runs a service where you balance cost against runtime using a new query language based on C# SQL, and U-SQL.

Orchestration & management

Yarn exists to manage jobs Hadoop clusters, but requires the cluster to exist for it to work. For this reason, Microsoft introduced Azure Data Factory (ADF) to manage the flow of big data from ingestion to processing and onward storage. ADF is designed for production where data is incrementally submitted and supports extensive monitoring and management of its pipelines.

Data warehousing in Azure

The traditional data warehouse is still an essential part of the data management landscape and combining that with modern cloud services brings the benefits of both technologies. Azure SQL DW does this by separating the compute layer from the storage layer so the data warehouse persists even if the compute is stopped or resized (up or down). To the outside world, Azure SQL DW will appear like SQL Server so your familiar tools will operate as normal.