Microsoft Azure Blog

Creating your first data model in Azure Analysis Services

Josh Caplan — Thu, 08 Dec 2016 09:00:03 Z

Azure Analysis Services is a new preview service in Microsoft Azure where you can host semantic data models. Users in your organization can then connect to your data models using tools like Excel, Power BI and many others to create reports and perform ad-hoc data analysis.

To understand the value of Azure Analysis Services, imagine a scenario where you have data stored in a large database. You want to make that data available to your business users or customers so they can do their own analysis and build their own reports. To do this, one option would be to give those users access to that database. Of course, this option has several drawbacks. The design of that database, including the names of tables and columns may not be easy for a user to understand. They would need to know which tables to query, how those tables should be joined, and other business logic that needs to be applied to get the correct results. They would also need to know a query language like SQL to even get started. Most often this will lead to multiple users reporting the same metrics but with different results.

With Azure Analysis Services, you can encapsulate all the information needed into a semantic model which can be more easily queried by those users in an easy drag-and-drop experience. And you can ensure that all users will see a single version of the truth. Some of the metadata included in the semantic model includes; relationships between tables, friendly table/column names, descriptions, display folders, calculations and row level security.

Once your data is properly modeled for your users to consume, Azure Analysis Services offers additional features to enhance their querying experience. The biggest of which is the option to put the data in an in memory columnar cache which can accelerate queries to sub second performance. This not only improves the query experience but by hitting the cache also reduces the query load on your underlying database.

Ready to give it a try? Follow the steps in the rest of this blog post and you’ll see how easy it is.

Before getting started, you’ll need:

Azure Subscription - Sign up for a free trial.

SQL Server Data Tools - Download the latest version for free.

Power BI Desktop - Download the latest version for free.

Create an Analysis Services server in Azure

1. Go to http://portal.azure.com.

2. In the Menu blade, click New.

3. Expand Intelligence + Analytics, and then click Analysis Services.

4. In the Analysis Services blade, enter the following and then click Create:

Server name: Type a unique name.
Subscription: Select your subscription.
Resource group: Select Create new, and then type a name for your new resource group.
Location: This is the Azure datacenter location that hosts the server. Choose a location nearest you.
Pricing tier: For our simple model, select D1. This is the smallest tier and great for getting started. The larger tiers are differentiated by how much cache and query processing units they have. Cache indicates how much data can be loaded into the cache after it has been compressed. Query processing units, or QPUs, are a sign of how many queries can be supported concurrently. Higher QPUs may mean better performance and allow for a higher concurrency of users.

Now that you’ve created a server, you can build your first model. In the next steps, you’ll use SQL Server Data Tools (SSDT) to create a data model and deploy it to your new server in Azure.

Create a sample data source

Before you can create a data model with SSDT, you’ll need a data source to connect to. Azure Analysis Services supports connecting to many different types of data sources both on-premises and in the cloud. For this post, we’ll use the Adventure Works sample database.

1. In Azure portal, in the Menu blade, click New.

2. Expand Databases, and then click SQL Database.

3. In the SQL Database blade, enter the following and then click Create:

Database name: Type a unique name.
Subscription: Select your subscription.
Resource group: Select the same resource group you created for your Analysis Services server.
Select source: Select Sample (Adventure Works LT).
Server: Choose a location nearest you.
Pricing tier: For your sample database, select B.
Collation: Leave the default, SQL_Latin1_General_CP1_CI_AS.

Now that you’ve created a sample data source, you’ll have some data to connect to when you build your data model.In the next steps, you’ll use SQL Server Data Tools (SSDT) to connect to your new data source, create a data model, and deploy it to your new server in Azure.

Create a data model

To create Analysis Services data models, you’ll use Visual Studio and an extension called SQL Server Data Tools (SSDT).

1. In SSDT, create a new Analysis Services Tabular Project.

If asked to select a workspace type, select Integrated.

2. Click the Import From Data Source icon on the toolbar at the top of the screen.

3. Select Microsoft SQL Azure as your data source type and click Next.

4. Fill in the connection information for the sample SQL Azure database created earlier and click Next.

Server Name: Name of SQL Azure server to connect to.
User Name: Name of the user which will be used to login to the server.
Password: Password for the account.
Database Name: Name of the SQL database to connect to.

Note: If using SQL Azure ensure that you have allowed your IP address access through the firewall. Also, ensure that “Allow access to Azure Services” is set to “on” for the firewall.

5. Select Service Account for the impersonation mode and click Next.

6. Select the tables you wish to import into cache and click Finish:

At this step, you can optionally provide a friendly name for each table. For large tables, which may not fit into cache, you can also specify a filter expression to reduce the number of rows. When complete, click next.
Data will now be read from the database and pulled into a local cache within Visual Studio.
Once loading is complete, you will have your first model created and will be able to see each table and the data within them. You can also switch to a diagram view by clicking the little diagram option at the bottom right of the screen:

The diagram view makes it really easy to see all of the tables and the relationships between them.

Improving the model

Now that your basic model is built, you could start querying it now or you could enhance it further by using more of the available modeling features. Some of these features include:

Create or edit relationships. You can add, remove or change relationships between tables by going to the diagram view and dragging a line between two columns in different tables. Once tables are joined together, they can automatically be queried together when a user selects columns from both tables.
Edit properties for a table or column. You can update multiple properties for tables and columns by clicking on them and updating the values in the properties pane.

Add more business logic to the model by creating calculations and measures.

Deploy

Once your model is complete, you can now deploy it to the Azure AS server which you created in the first step. This can be done with the following steps:

1. Copy your Azure Analysis Services server name for the Azure portal. This can be found at the top of the overview section of your server.

2. In the solution explorer in Visual Studio, right click on the project and click properties.

3. Change the deployment server to the name of your Azure AS server and click OK.

4. Right click the project name again, but this time click Deploy.

Connect

Now that you model has been creating you can connect with it through tools like the Power BI Desktop or Excel.

Power BI Desktop

If you don’t already have the Power BI Desktop, you can download it for free.

1. Open the Power BI Desktop

2. Click Get Data.

3. Select Databases/SQL Server Analysis Services and then click connect.

4. Enter your Azure AS server name and click OK.

5. On the Navigator screen, select your model and click OK.

You will now see your model displayed in the field list on the side. You can drag and drop the different fields on to your page to build out interactive visuals.

Excel

Learn more about connecting through Excel.

Learn more about Azure Analysis Services.

Azure Media Indexer 2: Japanese support, punctuation improvements, no more time limit

Adarsh Solanki — Thu, 08 Dec 2016 08:00:02 Z

On the heels of Microsoft's groundbreaking new developments in speech recognition, we have are continuing along our path: improving the quality of the transcripts generated by Azure Media Indexer and expanding our locale support to eventually accomplish our goal of being able to recognize all human speech on the Azure cloud.

Today we are ready to release the following improvements to Azure Media Indexer 2 Preview:

Japanese language models for public (preview) consumption in Azure Media Indexer 2
Removal of the 10 minute processing limit
Additional quality improvements with respect to punctuation and grammar

This Japanese language works in an identical manner to all other language models, simply provide the proper language code in the configuration file.

The following configuration will allow you to process a file with Japanese speech content (with defaults in all other Options)


{
    'Version': '1.0',
    'Features': [{
        'Options': {
            "Language": "JaJp"
        }        
    }]
}

Still not sure what Azure Media Indexer 2 is? Read the introductory blog post to learn how to extract the speech content from your media files.

To learn more about Azure Media Analytics, check out the introductory blog post.

Have feedback? Share it on our feedback forum.

Query Store ON is the new default for Azure SQL Database

Borko Novakovic — Wed, 07 Dec 2016 12:00:10 Z

We are happy to announce that Query Store is turned ON in all Azure SQL databases (including Elastic Pools) which will bring benefits both to the end users and the entire Azure SQL Database platform.

Why is this important?

Query Store acts as a “flight data recorder” for the database, continuously collecting critical information about the queries. It dramatically reduces resolution time in case of performance incidents, as pre-collected, relevant data is available when you need it, without delays.

You can use Query Store in scenarios when tracking performance and ensuring database performance predictability is critical. The following are some examples where Query Store is going to significantly improve your productivity:

Identifying and fixing application performance regressions (more details in this blog article)
Tuning the most expensive queries considering different consumption metrics (elapsed time, CPU time, used memory, read and write operations, log I/O, etc.)
Keeping performance stability with compatibility level 130 in Azure SQL Database (more details in this blog article)
Assessing impact of any application or configuration change (A/B testing)
Identifying and improving ad-hoc workloads (more details here)

Query Store also provides the foundation for performance monitoring and tuning features, such as the SQL Database Advisor. Query Store powers SQL Database Performance Insights which allows you to monitor and troubleshoot database performance directly from the Azure portal. With Query Store turned ON we ensure that relevant information about your most critical queries is available first time you open the queries chart on SQL Database Performance Insights:

We strongly recommend keeping Query Store ON. Thanks to an optimal default configuration and automatic retention policy, Query Store operates continuously using an insignificant part of the database space with a negligible performance overhead, typically in the range of 1-2%.

The default configuration is automatically applied by Azure SQL Database. If you want to switch to a customized Query Store configuration, use ALTER DATABASE with Query Store options. Also check out Best Practices with the Query Store to learn how to choose optimal parameter values.

Next steps

For more detailed information, check out the online documentation:

Project Bletchley – New Blockchain partners come to Azure Marketplace

Christine Avanessians — Wed, 07 Dec 2016 11:00:09 Z

As promised, we continue to release early and release often in the blockchain space together with our partners. I am pleased to be back to announce more great additions to our blockchain offering on Azure.

In just the last couple of weeks, we have added two new partners and solutions to the growing blockchain ecosystem in the Azure Marketplace.

Parity – Ethcore recently published their high performance, low footprint, reliable Ethereum blockchain client, Parity, on Azure. This offering simplifies setting up a new Parity node in the cloud, with little configuration from the user. In a matter of minutes, a user can have a single node, private Ethereum network up and running for development and test.

Blockstack Core v14 – Blockstack is building a new decentralized web of server-less applications where users can control their own data. Applications run locally and utilize user-specific data stores as their backend to maintain decentralization and control. Users can seamlessly deploy Blockstack Core nodes on Microsoft Azure. Blockstack Core nodes provide the core functionality of the Blockstack stack, processing data from a standard blockchain layer to construct a global view of security and ownership mappings.

In addition to our growing blockchain partner ecosystem, over the last few weeks, we have focused on functionality to improve deployment resiliency for our own consortium network blockchain solution releasing several updates that should improve overall deployment success rates. More new features are already in the works, so stay tuned for updates!

New price-performance choices for Azure SQL Database elastic pools

Morgan Oslake — Wed, 07 Dec 2016 10:00:10 Z

Azure SQL Database elastic pools provide a simple cost effective solution for managing the performance of multiple databases with unpredictable usage patterns. New price-performance choices for elastic pools provide even more cost effectiveness and greater scale than before.

More cost effectiveness
Now available are smaller elastic pool sizes and pools with higher database limits. These new choices lower the starting price for pools, lower the effective cost per database, and reduce price jumps between pool sizes.

Greater scale
Also, now available are larger sizes for Basic, Standard, and Premium pools, and higher eDTU limits per database for Premium pools. These new choices provide more storage and eDTU headroom for greater scale and the most demanding workloads.

Highlights

More pool eDTU sizes	New sizes range from 50 eDTUs for Basic and Standard pools up to 4000 eDTUs for Premium pools with additional sizing choices in between.
More storage for Standard pools	Up to 2.9 TB for 3000 eDTU Standard pools.
Higher database limits per pool	Up to 500 databases for Basic and Standard pools of at least 200 eDTUs. Up to 100 databases for Premium pools of at least 250 eDTUs.
Higher eDTU limits per database for Premium pools	Max eDTUs per database increase to 1750 eDTUs (P11 level) and 4000 eDTUs (P15 level) for the largest Premium pools.

Learn more

To learn more about SQL Database elastic pools and these new choices, please visit the SQL Database elastic pool webpage. And for pricing information, please visit the SQL Database pricing webpage.

Sneak peek: A new Azure Cloud Console

Brendan Burns — Wed, 07 Dec 2016 10:00:09 Z

For a while now, I have been passionate about containers and how they are revolutionizing and truly delivering the promise of cloud native computing. However, even as excited as I am about revolutionizing container compute with Azure, I’m equally passionate about user interface. After all, cloud computing is useless if it can’t be accessed from a useful interface. So, today, I’m excited to show you how we’re bringing these passions together in the new cloud console for the Azure portal.

Traditional cloud user interfaces have been divided into either a web-based graphical interface or a command line terminal interface. Each of these interfaces provide their utility and different users prefer different interfaces for different tasks. However, most Azure users use both interfaces to manage their applications on Azure. Much like developing code before integrated development environments like Visual Studio or Visual Studio Code, switching between these interfaces requires switching between applications, a context switch that slows users and makes it harder to accomplish their goals. In some cases, (for example tablets and other mobile devices) a terminal interface may not even be available and a user may have to switch devices.

To address these needs, we built an integrated workflow enabling users to build their applications on Azure using graphical and command line tools, even on devices where command line tools aren't installed. Today, we're giving you a sneak peek of this new cloud shell experience that we are adding into the Azure portal. As you can see from the video below, the shell is integrated into the portal so users can quickly drop into a command line experience while simultaneously viewing their cloud resources in the graphical web interface.

Using Azure Cloud Console to deploy a VM

Using Azure Cloud Console with GIT

The key features of this experience are:

Automatic authentication to the command line tools from your existing web login
All Azure command line tools, as well as relevant command line utilities pre-installed
Personalized, persistent workspace that preserves your code, configuration and activity across cloud shell sessions.

With a single click, you are dropped into a terminal command line tools pre-configured with your existing Azure credentials. This terminal is a fully featured experience featuring not only the Azure command line tools, but also standard editors and tools you would expect. Further, the cloud shell preserves context for you. When you save files to disk, they are persisted in Azure’s cloud so you can resume where you left off in your next cloud shell session, even if you are on a different device or network.

So how does the cloud console relate to containers? Well, the shell itself is packaged as a container to provide a clean, consistent interface every time you launch a new session. Of course this is based on the container we've already built for the Azure 2.0 command line tool. You can use today on your own machine with:


$ docker run -v ${HOME}:/root -it azuresdk/azure-cli-python:latest

Going forward, we're looking for a few hardy souls who are willing to test and provide feedback on this new console experience as we bring it to general availability over the next few months. If you are interested, please sign up and we'll be in touch!

I'm super excited about how containers are revolutionizing compute on Azure, and especially excited about how we ourselves can use container technology to offer new, integrated interfaces for developing your applications on the Azure cloud.

New features in Azure Application Insights Metrics Explorer

Vitaly Gorbenko — Wed, 07 Dec 2016 09:00:09 Z

Over the past weeks, Application Insights Metrics Explorer introduced several new features that allow more options for visualizing metrics, and simplify chart configuration. This includes new percentage aggregation charts, pinning standard and custom grids to dashboards, ability to control over y-axis boundaries of the charts, and sorting basic and advanced configuration options in chart settings.

Percentage aggregation charts

In many cases the actual value of a segmented metric is not as important as how it compares to the other values in the group. For example, you might want to visualize the percentage of the failed vs. successful requests, where they would sum up to 100%, instead of looking at the raw counts. Or you might want to see the percentage number of the HTTP requests handled by each server instance. To make it possible, we introduced a new percentage aggregation type, allowing users to switch between “Average” and “Average (%)”, “Sum” and “Sum (%)”, etc. Below is a sample chart that is based on the percentage aggregation:

To configure percentage aggregation, you need to turn on grouping, and select percentage aggregation from the Aggregation dropdown selector of chart settings:

Hiding less commonly used advanced settings, unless the user really wants to see them

A common feedback that we caught from new Application Insights users is that there are too many options in the chart settings pane. Some of these options are quite advanced but critical for the experienced users. Mixing basic and advanced options resulted in frustration for the new users who could not identify the essential settings and thus could not configure the charts the way they wanted. What did we do? A new checkbox on top of chart configuration now allows filtering basic and advanced settings. New users are looking at a simpler view. As the user becomes more proficient with metrics explorer, he can add advanced settings by checking the box on top of the chart details dialog:

Pinning Metrics Explorer grids to Azure dashboards

In the past, you could pin metrics explorer charts to dashboards. Now in addition to pinning charts, you can also pin grids. This small but very useful icon lets you customize your dashboards with those metrics that are better represented in a grid format:

Ability to freeze lower and upper boundaries of the y-axis on the charts

Freezing y-axis of the charts becomes important when looking at smaller fluctuations of larger values. For example, when the volume of successful requests drops from 99.99% to 99.5%, it may represent a significant reduction in the quality of service. But from the charting perceptive, noticing a small numeric value fluctuation would be difficult or even impossible. With the new option, you can freeze the lowest boundary of the chart to 99%, which would make this small drop more apparent. Another example can be a fluctuation in the available memory, where the value will technically never reach 0, so fixing the range to a higher value may make the drops in available memory easier to spot. The two charts below duplicates the same metric but with and without fixed y-axis boundary:

To freeze the y-axis boundaries, you need to check advanced settings and specify the desired range under the Y-axis range section of the chart details dialog:

Clustered Columnstore Index in Azure SQL Database

Sunil Agarwal — Tue, 06 Dec 2016 10:00:07 Z

Columnstore index is the preferred technology to run analytics queries in Azure SQL Databases. We recently announced general availability if In-Memory technologies for all Premium databases. Similar to In-Memory OLTP, the columnstore index technology is available in premium databases.

The columnstore technology is available in two flavors; clustered columnstore index (CCI) for DataMart analytics workloads and nonclustered columnstore index (NCCI) to run analytics queries on operational (i.e. OLTP) workload. Please refer to NCCI vs CCI for the differences between these two flavors of columnstore indexes. The columnstore index can speed up the performance of analytics queries up to 100x while significantly reducing the storage footprint. The data compression achieved depends on the schema and the data, but we see around 10x data compression on average when compared to rowstore with no compression. This blog will focus on Analytic workloads using CCI but cover NCCI in a future blog.

Clustered Columnstore index is available in Azure SQL Databases across all premium editions. However, it is not yet available on the Standard and Basic pricing tiers. Using this technology in Azure SQL Databases, you can lower the storage cost and getting a similar or better query performance on lower premium tiers.

The tables below show a typical analytics query with multi-table join running on P1 and P15 both with/without clustered columnstore index and storage savings achieved

Query Performance: Key point to note below is that with clustered columnstore index, the example query runs 5x faster on P1 compared to the same query running on P15 with rowstore with no tuning. This can significantly lower the cost you need to pay to meet your workload requirements.

Pricing Tier	With Rowstore	With Columnstore	Performance Gains
P1	30.6 secs	4.2 secs	14x
P15	19.5 secs	0.319 secs	60x

Storage Size: The storage savings with columnstore compared to PAGE or NONE compressed tables shown below. While the cost of storage is already included with AzureDB, but lower storage can enable you to choose a lower tier. Note, this is generated test data so the compression is lower than what one would get for customer workloads.

Number of Rows	Size Rowstore (MB)	Size columnstore (MB)	Savings
3626191	212 (PAGE compression)	120	1.8x
3626191	756 (NONE compression)	120MB	6.2x

The best part of columnstore index technology is that it does not require any changes to your application. All you need to do is to either create or replace an existing index with columnstore index on your table(s).

How does Columnstore Index work?

As described earlier, the columnstore is just an index that stores data in a table as columns as shown below. The queries can continue to access the table requiring no changes.

Columnstore index delivers significant data compression and query performance due to the following three key factors

Reduced IO and Storage: Since data is stored as individual columns, it compresses really well as all values are drawn from the same domain (i.e. data type) and in many cases, the values repeat or are similar. The compression will depend on the data distribution but typical compression that we have seen is around 10x. This is significant because it enables you reduce the storage as the IO footprint of your database significantly.
Only Referenced columns need to be fetched: Most analytics queries fetch/process only a small set of columns. If you consider a typical Star Schema, the FACT table is the one with most rows and it has large number of columns. With columnstore storage, SQL Server needs to fetch only the referenced columns unlike rowstore where the full row needs to be fetched regardless of number of columns referenced in the query. For example, consider a FACT table with 100 columns and an analytic query accessing this tables references only 5 columns. Now, by fetching only the referenced columns, you can potentially reduce IO by 95% with simplifying assumption that all columns take same storage. Note, this is on top of already 10x data compression provided by columnstore.
Efficient Data Processing: SQL Server has an industry leading query engine for columnstore data to deliver up to 100x speed up in query performance. For details, please refer to Speeding up Analytics Queries.

How do I create clustered columnstore index?

Creating a clustered columnstore index is like creating any other index. For example, I can create a regular rowstore table as follows

CREATE TABLE ACCOUNT (
ACCOUNTKEY	INT NOT NULL,
ACCOUNTDESCRIPTION	NVARCHAR (50),
ACCOUNTTYPE	NVARCHAR (50),
ACCOUNTCODEALTERNATEKEY	INT)


CREATE TABLE ACCOUNT (
    ACCOUNTKEY            INT NOT NULL,
    ACCOUNTDESCRIPTION        NVARCHAR (50),
    ACCOUNTTYPE            NVARCHAR (50),
    ACCOUNTCODEALTERNATEKEY     INT)

Any rows inserted into the table above are stored in rowstore format. Now, if you want to convert this table to store data in 'columnstore', all you need to do is to execute the following SQL statement


CREATE CLUSTERED COLUMNSTORE index ACCOUNT_CI on ACCOUNT

If the rowstore table had a clustered BTREE index, then you can execute the following SQL Statement


CREATE CLUSTERED COLUMNSTORE index ACCOUNT_CI on ACCOUNT WITH (DROP_EXISTING = ON)

When and where should you use clustered columnstore Index?

Clustered Columnstore index primarily targets analytics workloads. The table below shows the common scenarios that have been successfully deployed with this technology.

Columnstore Option	Workload	Compression
CCI (clustered columnstore index)	Traditional DW workload with Star or Snowflake schema: Commonly you enable CCI on the FACT table but keep DIMENSION tables with rowstore with PAGE compression. Additional Considerations: consider CCI for large dimension tables with > 1 million rows Insert mostly workload: Many workloads such as IOT (Internet of things) insert large volume of data with minimal updates/deletes. These workloads can benefit with huge data compression as well as speed up of analytic queries.	10x on average
CCI/NCI (with one or more nonclustered indexes)	Similar to the ones mentioned with CCI but require (a) PK/FK enforcements (b) significant number of queries with equality predicate or short range queries. NCIs speed up the query performance by avoiding full table scans (c) update/delete deletes of rows which can be efficiently located using NCIs.	10X on average + additional storage for NCIs

Resources to get started

For more details, please refer to the following

New Microsoft Azure training and discounted certifications

Microsoft Azure — Tue, 06 Dec 2016 09:00:15 Z

This post is authored by Julia White, Corporate Vice President, Azure + Security Marketing.

As I recently discussed in my Top Cloud Myths of 2016 blog post, the use of cloud technology has become mainstream. Over 90% of the Fortune 500 companies now use at least one of Microsoft’s enterprise-grade services, and more than 60% are using three or more Microsoft cloud technologies. Azure compute usage has more than doubled year-over-year, and across a variety of industries, companies like Rolls Royce, Uber, and Coca-Cola are using Azure to transform their businesses.

With this rapid adoption of Azure, we have consistently heard from developers and IT operators about the need for Azure training and certification to help ensure they have the latest and greatest information. With the rapid innovation cycle of Azure, it’s critical to stay up to date on the latest technology and methodologies.

To make sure you have up-to-date technical skills and best practices using Azure, we are introducing new training resources and discounted access to certification. Announcing today, we have three training offers that combine free access to our library of flexible online courses, discounts on our industry-standard Microsoft Certified Professional exams, and a discount on Linux certification offered through the Linux Foundation. In case you missed the recent news, Microsoft joined the Linux Foundation, further demonstrating our commitment to open source for Azure, and as a company.

We are offering a broad range of learning resources – from a Massively Open Online Course, or MOOC, to full certification offerings. The MOOCs offer online videos, demos, labs, graded assessments, office hours, and more. When you complete a MOOC you get digital certificates for completion, and you get access to reduced-cost Microsoft Certified Professional exams for formal certification.

Check out all the learning options for Azure, and read my colleague Gavriella Schuster’s blog post to dive into more details about our new resources and discounted certifications!

Telemetry Platform Features in Azure Media Services

Dwyane George — Tue, 06 Dec 2016 09:00:07 Z

We are excited to announce new Azure Media Services (AMS) telemetry platform features, generally available through our new Telemetry API. Media Services telemetry allows you to monitor and measure the health of your services through a suite of telemetry data. Telemetry data is written to your Azure Storage account and can be processed and visualized using a wide array of data visualization tools.

Consuming Telemetry Data

This release includes telemetry metrics for Channel, Streaming Endpoint, and Archive entities. Telemetry data is written to an Azure Storage table in the storage account specified when configuring telemetry for your media services account. Telemetry data is stored in aggregate in a table, “TelemetryMetricsYYYYMMDD,” for each day’s data (where “YYYYMMDD” denotes the date timestamp).

Each table entry contains a set of common fields and a record with a set of entity-specific fields. The entry identifying fields include the following:

Property	Value	Example
PartitionKey	{Account ID}_{Entity ID}	e49bef329c29495f9b9570989682069d_64435281c50a4dd8ab7011cb0f4cdf66
RowKey	{Seconds to Midnight}_{Random Value}	01688_00199
Timestamp	The time at which the row entry was created	2016-09-09T22:43:42.241Z
Type	The type of the entity providing telemetry	Channel
Name	The name of the telemetry event	ChannelHeartbeat
ObservedTime	The time at which the event occurred	2016-09-09T22:42:36:924Z
ServiceID	Service ID	f70bd731-691d-41c6-8f2d-671d0bdc9c7e
Entity-specific Properties	{Record as defined by the event}	{Record}

The account ID is included in the partition key to simplify workflows where multiple media services accounts are writing data to the same storage account. The row key start with the number of seconds to midnight to allow top n style data queries within a partition (see the log tail table design pattern for more information). The observed timestamp is an approximate measure provided by the entity reporting telemetry.

Entity-Specific Telemetry

The data in each telemetry row represents an aggregation of telemetry events raised over an aggregation time window, listed below. Each entity pushes telemetry with the following frequencies:

Channels: Every 60 seconds
Streaming Endpoints: Every 30 seconds
Archive: Every 60 seconds

Below is the schema description for channels, streaming endpoints, and archive entities.

Channels

Property	Value	Example
TrackType	Type of track	video
TrackName	Name of the track	video
Bitrate	Expected bitrate of the track	785,000
IncomingBitrate	Incoming bitrate of the track	784,548
OverlapCount	Number of overlapping fragments received in ingest	0
DiscontinuityCount	Number of discontinuities detected in ingest	0
LastTimestamp	Last ingested data timestamp	1800488800
NonincreasingCount	Count of fragments discarded due to non-increasing timestamp	0
UnalignedKeyFrames	Boolean on whether we received fragments where key frames are not aligned	False
UnalignedPresentationTIme	Boolean on whether we received fragments where presentation time is not aligned	False
UnexpectedBitrate	Boolean on whether the IncomingBitrate and Bitrate differ by more than 50% or if IncomingBitrate for an audio or video track is less than 40 kbps if Bitrate is 0	False
Healthy	Boolean on whether OverlapCount, DiscontinuityCount, NonincreasingCount, UnalignedKeyFrames, UnalignedPresentationTime, and UnexpectedBitrate are all zero or false	True
CustomAttributes	Placeholder for custom attributes

Streaming Endpoints

Property	Value	Example
HostName	Hostname of the endpoint	builddemoserver.origin.mediaservices.windows.net
StatusCode	HTTP status code	200
ResultCode	HTTP result code detail	S_OK
RequestCount	Total requests received within the last aggregation window	3
BytesSent	Total bytes sent within the last aggregation window	2,987,358
ServerLatency	Average server latency including storage in milliseconds	130
E2ELatency	Average end-to-end latency in milliseconds	250

Visualizing Telemetry Data

Your Azure Storage account can export data to data visualization tools such as PowerBI, Application Insights, and AMS Live Monitoring Dashboard, among many others. Below is an example of how this data can be imported into and visualized directly with PowerBI.

First, select the telemetry tables for the days you are interested in visualizing. To import your data, select Microsoft Azure Table Storage from the Get Data menu. Enter your storage account credentials and select the tables to import. Next, by selecting Content.ObservedTime as the time axis and Average of Content.Healthy (casted to decimal representation) as the value, plot the health of your channels and archive entities as a line graph. To plot channel health and archive health separately, add a filter on Content.Name. This visualization illustrates entity health where a value of 1 represents perfectly healthy and a value of 0 represents perfectly unhealthy for the given time. Below are examples of the channel and archive health plotted over time.

To visualize the overall health of your services, collapse the time dimension by plotting the data as a pie chart. Selecting Content.Healthy as the legend and Count of Content.Healthy as the value, produces the visualization below.

We’re excited to share these telemetry features and hope they provide useful information about the health of your Azure media services.

Thanks,

Azure Media Services Streaming Team