Microsoft Azure Blog

HDInsight tools for IntelliJ & Eclipse June updates

Jenny Jiang — Mon, 24 Jul 2017 10:00:13 Z

HDInsight tools June release has been released! In this release, you can choose SBT as a build tool in addition to Maven when creating Spark projects in IntelliJ. With improvements to Spark job view and job graph in IntelliJ and Eclipse, more job info and statistics are now provided. You can also view job logs including driver stderr, stdout, and directory info easily in Spark job view.

Summary of key updates

Improved Spark job view and job graph in IntelliJ and Eclipse

In Azure Explorer (View > Tools window > Azure Explorer in IntelliJ), go to HDInsight node, select the Spark cluster, and then click Jobs as shown below.

The left pane of Spark job view shows all the Spark applications that ran in the cluster. Select one Spark job to view more details.

By hovering over the job graph, it will show the job run information. When clicking on job graph, it dives into the stage graph and shows the statistics of the job. You can also open Spark history UI or Yarn UI by clicking the respective link at the top of the Spark job view.

Improved Spark log view in IntelliJ and Eclipse

In the same Spark job view pane, click the Log tab to view the frequently used logs including driver stderr, driver stdout, and directory info, as shown below.

SBT build tool support when creating Spark project in IntelliJ

As shown below, you can now choose SBT as a build tool in addition to Maven when creating a new Spark project in IntelliJ.

Following the wizard to create a Spark project, after it is done, a new build.sbt file is generated which contains the build description for the project. Then you can author, submit, or debug the Spark job following the Spark job submission/debugging instructions.

How to install/update

HDInsight Eclipse plugin: Eclipse will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the Eclipse repository and searching “Azure Toolkit for Java”.

HDInsight IntelliJ plugin: IntelliJ will prompt you for latest update if you have the plugin installed before, or you can get the latest bits by going to the IntelliJ repository and searching “Azure Toolkit for IntelliJ”.

For more information, check out the following:

Learn more about today’s announcements on the Azure blog and Big Data blog, and discover more Azure service updates.

Feedback

We look forward to your comments and feedback. If there is any feature request, customer ask, or suggestion, please send us a note to hdivstool@microsoft.com. For bug submission, please open a new ticket using the template.

More resource policy aliases

Mike Chen — Mon, 24 Jul 2017 09:00:12 Z

Aliases in resource policies enable you to restrict what values or conditions are permitted for a property on a resource. If you are already familiar with p olicy aliases, you know they are a crucial part of managing your Azure environment.

We want to keep adding new policy aliases, so you can more easily govern what gets deployed in your environment. In this blog, I would like to share most recent aliases we have enabled.

First, let’s review how aliases are integrated into user requests. Each policy alias maps to paths in different API versions for a given resource type. During policy evaluation, when the policy engine retrieves the value of a particular field, it looks at the API version of the request and gets the path for that version. The diagram below shows how policy alias works during policy evaluation time.

Custom Image for virtual machines

For security reasons, lots of customers want to make sure only custom images from the central IT team are deployed in their environment. The IT team approves a set of managed images, and puts them in a resource group. To ensure VMs are created from these images, you implement a resource policy. For implementation, you can either specify the resource group which contains the images or explicitly specify the images.

We added the Microsoft.Compute/imageId alias to enable this scenario. You can use it for virtual machines or virtual machine scale sets by modifying the type condition.

The examples below show what the policies look like.

Example1: (use images from certain resource group)


{
    "if": {
        "allOf": [
            {
                "field": "type",
                "equals": "Microsoft.Compute/virtualMachines"
            },
            {
                "not": {
                    "field": "Microsoft.Compute/imageId",
                    "contains": "resourceGroups/testImage"
                }
            }
        ]
    },
    "then": {
        "effect": "deny"
    }
}

Example2: (use specific images)


 {

                    "field": "Microsoft.Compute/imageId",

                    "in": ["<imageId1>","<imageId2>"]

                }

Microsoft.Compute/imageId is the new alias we enabled. You can also use it for virtual machines scale set by modifying the type condition.

Platform Images

We introduced a set of aliases that can be used across resource types. These cross resource type aliases enable you to restrict platform images for virtual machines, virtual machine scale sets, and managed disks. For example, the alias Microsoft.Compute/imagePublisher doesn’t have a resource type name, and can work across different resource types. The linked example shows how to use these aliases.

Use Managed Disk

With the release of managed disk, lots of customers want to require that only managed disks are deployed for VMs. With resource policy, you can now restrict your VM and scale set to use only managed disks. The policy requires that fields related to managed disks are present in user request. Those fields are shown in the linked example. By looking for these fields, you can determine whether managed disks are used with the VM or scale set.

VM Extension Types

Organizations may want to forbid usage of certain type of extensions. For example, a VM extension may not be compatible with certain custom VM images. Or, for security reasons, you don’t want users to reset password for a VM. The example below shows how to block a specific VM extension. It uses publisher and type to determine which extension to block.


{
    "if": {
        "allOf": [
            {
                "field": "type",
                "equals": "Microsoft.Compute/virtualMachines/extensions"
            },
            {
                "field": "Microsoft.Compute/virtualMachines/extensions/publisher",
                "equals": "Microsoft.Compute"
            },
            {
                "field": "Microsoft.Compute/virtualMachines/extensions/type",
                "equals": "VMAccessAgent"

      }
        ]
    },
    "then": {
        "effect": "deny"
    }
}

Azure Hybrid Use Benefit

When you have proper on-premise license, you can save the license fee on your virtual machines. When you don’t have the license, you should forbid the option. The following policy forbids usage of Azure Hybrid Use Benefit (AHUB).


{
    "if": {
        "allOf": [
            {
                "field": "type",
                "in":[ "Microsoft.Compute/virtualMachines","Microsoft.Compute/VirtualMachineScaleSets"]
            },
            {
                "field": "Microsoft.Compute/licenseType",
                "exists": true
            }
        ]
    },
    "then": {
        "effect": "deny"
    }
}

Summary

To do a quick recap, this blog explains how policy alias works and what you can govern through resource policies. Try them and let us know what new things you want to govern!

Go serverless with R Scripts on Azure Function

Thomas Delteil — Mon, 24 Jul 2017 08:00:12 Z

Serverless is all the rage, now you can get in on the action using R! Azure Function supports a variety of languages (C#, F#, js, batch, PowerShell, Python, php and the list is growing). However, R is not natively supported. In the following blog we describe how you can run R scripts on Azure Function using the R site extension.

Azure Functions can be used in several scenarios because of the broad choice of triggers offered:

Timer trigger, executes a Function on a schedule.
Http trigger, execute a Function after an HTTP call.
Azure Queue Storage, Service Bus, Blob Storage, triggers the function when a new object or message is received.

Why would you want to run R scripts on Azure Function?

A typical use-case would be replacing your R jobs currently scheduled with cron for example. Using Azure Function you can set up a timer trigger that triggers your R script on a periodic basis. You get a fully managed solution where you can get alerted on errors and access to the logs or edit the scripts directly from the browser. If you choose the consumption plan, then it is very cost-effective, only paying per use and the underlying storage. (There is a free grant for the 1st million calls on the consumption plan).

The following tutorial will walk you through the steps to create a twitter bot posting a ggplot of the temperature forecast for the next 5 days using only R and Azure Functions:

Running R scripts on Azure Function Tutorial

Go ahead and try it now, it is simpler than you think! Give us some feedback and let us know what you are using it for.

Bing Map v8 available in Azure IoT Suite Remote Monitoring preconfigured solution

Sam George — Thu, 20 Jul 2017 11:00:19 Z

Azure IoT Suite is designed to get customers and partners started quickly to connect and manage their devices in a simple and reliable manner, realizing business value from an IoT solution. Bing Map (v7) has been integrated as part of Azure IoT Suite Remote Monitoring preconfigured solution to visualize device locations and status in the preconfigured solution dashboard. We are pleased to announce that an advanced version of Bing Map (v8) has been effective since July 1st, 2017 for Azure IoT Suite Remote Monitoring preconfigured solution. It provides customers and partners:

High performance: Data renders 10x faster.
Extended culture support for 95 new languages: Uses the Bing Maps REST services to perform geocode and route requests on up to 117 languages.
New features based on developer feedback

To leverage the new advanced map functionality, you can do one of the following:

Deploy a new Remote Monitoring Solution: All solutions deployed from http://azureiotsuite.com will leverage the new V8 control.
Update to the latest code: Get all new features from the master branch, such as Device Management and Bing Map v8.
Update only the map control: Get only the changes for the map control by referencing the single commit.

In all scenarios, the existing Bing Map API Key under your account will be kept valid for you, no specific action is expected from customers due to the migration.

Learn how to re-deploy over an existing Remote Monitoring preconfigured solution.

Introducing the Azure Analysis Services web designer

Bret Grinslade — Thu, 20 Jul 2017 10:00:19 Z

Today we are releasing a preview of the Azure Analysis Services web designer. This new browser-based experience will allow developers to start creating and managing Azure Analysis Services (AAS) semantic models quickly and easily. While SQL Server Data Tools (SSDT) and SQL Server Management Studio (SSMS) are still the primary tools for development, this new experience is intended to make simple changes fast and easy. It is great for getting started on a new model or to do things such as adding a new measure to a development or production AAS model.

This initial release includes three major capabilities, model creation, model editing, and doing queries. The model can be created directly from Azure SQL Database and SQL Data Warehouse, or imported from Power BI Desktop PBIX files. When creating from a database, you can chose which tables to include and the tool will create a Direct Query model to your data source. Then you can view the model metadata, edit the Tabular Model Scripting Language (TMSL) JSON, and add measures. There are shortcuts to open a model in Power BI Desktop, Excel, or even open a Visual Studio project which is created from the model on the fly. You can also create simple queries against the model to see the data or test out a new measure.

Navigate to your Azure Analysis Services server in the Azure Portal and click the “Open” link from the Overview blade.

Once the web designer opens, it will take you directly to your server where you can examine existing models or create new ones.

Now that you have an AAS server, you can add a model directly from a database or you can import a Power BI Desktop PBIX file. Note, for PBIX import, only Azure SQL Database, Azure SQL Data warehouse, Oracle, and Teradata are supported at this time. Also, Direct Query models are not yet supported for import. We will be adding new connection types for import every month, so let us know if your desired data source is not yet supported.

Now that you have created a new model from a database or PBIX file, you can see it on the list of models in the AAS server and can edit the model or browse the model. To edit the model, click the pencil icon next to the model name.

When editing, you have access to the full TMSL and can make changes directly to the metadata of the model. Keep in mind that if you save changes, it is updating the model in the cloud. It is a best practice to make changes to a development server and propagate changes to production with a tool such as BISM Normalizer. For prototyping and development, you can make changes quickly and easily this way.

If you click Query, you can view data through a simple view or you can use the “Open in” button to open in Power BI Desktop or Excel. We think of the simple web view as a quick way to check your model or new measures you have created without having to use SSMS.

Finally you will notice that you can press the blue “Measures” link on the field list to open the simple measure editor. This is a great place to see all of your measures in one place, and also to quickly add measures to test in the UX.

That is a brief introduction to the new Azure Analysis Services web designer. We hope you find the tool useful and we welcome your feedback in the Azure Analysis Services feedback forum. We think this new experience will help you get started quickly and give you new options to make changes right in the web experience. We will be adding additional functionality each month, so let us know what you would like to see next!

Learn more about Azure Analysis Services.

Instant File Recovery from Azure VM backups is now generally available

Kartik P V R — Thu, 20 Jul 2017 09:00:19 Z

Today, we are excited to announce that Instant recovery of files and folders from Azure VM backups by Azure Backup is now generally available (GA). This adds to the repertoire of cloud-first features we have been delivering from Azure Backup. We earlier announced that File-folder recovery from Azure Windows VM backups and Linux VM backups were available in preview. We received great feedback from preview and we have enhanced the feature in terms of customer experience, security, and performance.

Value Proposition Recap

To recap the value proposition, with this file recovery feature, now you can securely

Recover files instantly – Now instantly recover files from the cloud backups of Azure VMs without any additional infrastructure. Whether it’s accidental file deletion or simply validating the backup, instant restore drastically reduces the time to recover your data.
Open application files without restoring them - Our iSCSI-based approach allows you to open/mount application files directly from cloud recovery points to application instances. You need not restore the entire VM and thus save on time taken for recovery and consumption of bandwidth. For e.g. in case of backup of a Azure Linux VM running MongoDB, you can mount BSON data dumps from the cloud recovery point and quickly validate the backup or retrieve individual items such as tables without having to download the entire data dump.

Introducing Enterprise Smart Contracts

Marley Gray — Thu, 20 Jul 2017 08:00:18 Z

Introduction

Enterprise Smart Contracts - Full Whitepaper

It has been a little over a year since we announced Project Bletchley, and since that time we have been working directly with our partners and customers to understand how the cloud can help developers build a new generation of modern applications with blockchain as a core data layer.

Our initial goal for “Blockchain as a Service” was to make it easier for developers to set up a lab environment, get their hands dirty, and build something useful for the enterprise using a blockchain. Microsoft Azure makes it ridiculously easy to spin up the blockchain of your choice, including leading platforms such as Ethereum, Quorum (EEA), Hyperledger Fabric, R3 Corda and Chain Core – we’ve made great progress against those initial goals.

Easing the deployment of this “plumbing” exposed perhaps an even larger challenge: how to build applications for these new environments. Our customers and partners often say to us, “Ok, you’ve made it easy for me to stand up these blockchain networks, but what do I do now?”

This echoes the mid-to-late 1990s when just getting connected to the internet was made much easier with TCP/IP and a web browser included in your operating system for “surfing” the web. The early “surfing” experience, however, was limited to a small set of static web pages and basic email. There was a long way to go from Windows 95 to the .com craze, as the early software development platforms and tools were simply not designed for the web. The result: it took a lot of learning and iterative innovation before you could stand up a viable e-commerce site. Blockchain platforms are in a similar position today: Azure makes it easier to get connected to the “network,” but you can’t do much with it yet.

So, we took a step back to consider what customers truly wanted to build. We learned from early adopters across industries that the business processes most interesting for blockchain were shared. Processes that cross both organizational and trust boundaries. Managing interactions with semi-trusted parties means clearly articulating each parties’ obligations, rules on how to meet them and penalties for failing to meet them. In short, a contract.

The introduction of Smart Contracts in 2015 was largely responsible for the explosion of interest in the enterprise for blockchains. However, the first versions of “Public Smart Contracts” were not designed for enterprise use and cannot deliver against the requirements of the enterprise.

Enterprise Smart Contracts are “what’s next” in the blockchain revolution.

Enterprise Smart Contracts decompose the “Public Smart Contract” approach, reflecting on both “contract” and technology evolution to provide a model for delivering on the promise of blockchain in the enterprise. A critical first step was to introduce separation of concerns in implementation, which modularizes data, logic, contract participants and external dependencies. What Enterprise Smart Contracts deliver are a set of components that can be combined to create contract templates that when executed, provide the privacy, scale, performance and management capabilities expected in the enterprise.

Enterprise Smart Contract Components

What exactly is an Enterprise Smart Contract? Let’s look at the major components:

Schema – the data elements required for the execution and fulfillment of contract obligations between counterparties and the cryptographic proofs needed to maintain the integrity and trust across counterparties and observers such as auditors or regulators.
Logic – business rules defined in the schema and agreed to by the counterparties and observers. Cryptographic proofs required for the execution, versioning and integrity of both the code and its results are persisted to the blockchain as defined in the schema.
Counterparties – identities of participants (people, organizations and things) agreeing to the terms and execution of the contract, authenticated through cryptographic primitives such as digital signatures.
External Sources – external input of data or triggers required to fulfill the execution requirements of the contract. These external sources and conditions for interaction are agreed to by the counterparties and observers. As with the others, cryptographic proof is required to prove authenticity and establish trust in the external sources.
Ledger – the immutable instance of a contract on a distributed ledger (blockchain) containing the data items in the schema to record all contract activities and proofs. This can be either a public “distributed trustless database” or a “shared, permissioned, semi-trusted, discretionarily private database.”
Contract Binding - A Enterprise Smart Contract Binding is the composition of these parts creating a unique instance of an Enterprise Smart Contract. It is created when a contract begins negotiation between counterparties and becomes versioned and locked when each counterparty signs the contract. Once signed and locked the Enterprise Smart Contract begins the execution of the terms and conditions that lead to fulfillment.

Implementing Enterprise Smart Contracts

Now that we have defined what makes up an Enterprise Smart Contract, it’s time to implement them. The cloud is the perfect companion to distributed ledger systems implementing Enterprise Smart Contracts. Since blockchains are comprised of distributed nodes that maintain the database, a globally-distributed, highly-available public cloud provides a great companion platform for services supporting these networks.

“Public Smart Contracts” execute or run every single transaction on every single computer on the network. This is by design and necessary for trustless networks. However, this need not be true for the enterprise. One of the benefits of Enterprise Smart Contracts is the ability to execute terms and conditions “off-chain,” allowing for vastly more flexible performance, scalability, privacy, versioning and an enterprise friendly architecture and development environment. The cloud can provide a shared logic execution platform for Enterprise Smart Contracts at massive scale, while sharing the true cost of running them only between counterparties. I call it “splitting the check.” When two or more people share a contract (dinner) with each other, the bill arrives, you simply split the check. Actually, we can be more discerning than that, but you get the picture. Sharing cloud resources with your counterparties makes “splitting the check” possible while not getting mired in whose datacenter the logic for your contract will execute.

Depending on the underling blockchain, transactions may or may not be private. Meaning, unless your blockchain supports an authorization framework that only allows certain identities to read certain data properties or encrypts data such that only certain identities can decrypt the data, your data is in the clear. If you use encryption to encrypt your data AND execute your Smart Contract logic on the blockchain, then every node on that network needs to be able to decrypt that data in order to be able to compute against it. Some platforms like Quorum and Hyperledger Fabric support these using various ways.

However, executing your logic in an Enterprise Smart Contract means that you can keep not only your data private between counterparties, but your logic as well. The resulting outputs of your logic can be encrypted before posting to the ledger creating a more discrete and flexible privacy model.

This also creates a multi-trust model where Enterprise Smart Contracts can implement different privacy measures based on the blockchain underneath, utilizing privacy features that may be available for certain blockchains. Without Enterprise Smart Contracts, you are limited to the trust model of the blockchain platform only.

Azure – The Enabler

The Microsoft Azure platform provides the building blocks needed to deliver the core technical capabilities for implementing Enterprise Smart Contracts. However, composing them together is a complex undertaking. Last year, when the concept of Cryptlets was introduced, the benefits of separating the logic from the data while using the same cryptographic properties of blockchains was the primary focus. However, it is not enough to just share the same cryptographic primitives as blockchains. You need a platform and a framework that provides the infrastructure for creating Enterprise Smart Contracts. This is not something each counterparty in the network should have to build and maintain. A shared platform and framework for building Enterprise Smart Contracts or PaaS performs the difficult bits like:

Key Management
Secure integration of enterprise identity to secrets/private keys to allow “on-behalf of” transactions
Cryptographic proof generation across cipher suites (ECC, RSA, etc.) and shared infrastructure for expensive operations like Zero Knowledge Proofs, Ring Signatures and Threshold or Homomorphic encryption
Abstracted integration into any blockchain or distributed ledger so logic is portable across platforms
Enable blockchain to blockchain interoperability scenarios like an Enterprise Smart Contract using multiple different blockchains at the same time like recording a transaction proof on a public network for a private transaction. Also, the potential to create Enterprise Smart Contracts that can perform asset transfers from one network to another.
Open APIs for abstracting underlying blockchains for a common way of invoking Enterprise Smart Contracts as well as surfacing blockchain events and data to external systems.
Common enterprise development environments like .NET and Java (CLR, JVM) and newer functional languages and DSLs.
Extensible Data Services plugins to digest and make sense out of blockchain transactions with data lakes, warehouses, BI/Analytics and AI/Machine Learning.
Enterprise Operations and Management tools

Using a combination of enabling technologies in the Azure cloud like Key Vault and Azure Active Directory along with a code attestation engine, which we will detail later, the foundations for this development framework are in place.

Enterprise Smart Contracts - Framework

The Enterprise Smart Contract Framework provides the infrastructure and tools to build on this platform allowing you to harness existing enterprise investments in infrastructure and development skills. This framework is comprised of four major components.

Secrets, Control and Configuration – authorizes access to secrets stored in Azure Key Vault for specific identities (people, cryptlets, IoT devices, etc.) authenticated via Azure Active Directory. Manages the bindings for Enterprise Smart Contracts.
Runtime Environment Services – provides attested execution for Cryptlets and abstracts underlying identity mapping and key management. This allows for developers to write Cryptlets in the language of their choice focusing on Enterprise Smart Contract logic and not the underlying “infrastructure.” The services dynamically create attested execution environments for Cryptlets to run in, securely provision secrets to these environments, and generates cryptographic proofs automatically.
Transaction Builder and Router – marshals and formats Cryptlet Messages into blockchain specific formats and routes these transactions to the appropriate blockchain with reliable delivery.
API – exposes a secure, authenticated message-based API for sending messages to and receiving messages from Enterprise Smart Contracts at massive scale.

Summary

Enterprise Smart Contracts and Microsoft Azure provide the infrastructure required to usher in “real world” enterprise applications to be built on blockchain networks. Cryptlets and the framework provide architects and developers a middle tier platform where existing skills, tools and components can be integrated.

The framework allows enterprises to build blockchain based applications with diverse capabilities:

Publishing of attested data to and from external sources.
Integration point for existing enterprise systems like ERP and CRM.
Logic and behavior for all types of contracts, financial products: derivatives, bonds, insurance policies.
Logic for license contracts: government issued business licenses, certifications, etc.
IoT Integration through Azure IoT suite
Cross blockchain coordination: asset or token transfer between different networks and platforms, e.g. Hyperledger Fabric to Enterprise Ethereum, record private proofs on a public blockchain.
Token or Coin "minter" using off-chain integration, proofs and advanced coinbase behavior.
Personal bot agent for "on behalf of" use cases.
Distributed transaction support between blockchains and other enterprise systems, ledger resource compensation, two-phase commit, etc.

Enterprise Smart Contracts enable these capabilities by providing a secure, confidential, distributed, multi-party application platform for running shared business logic, with a cryptographic proof system that natively integrates with multiple blockchains. Enterprise Smart Contracts and Azure provide this platform that will allow the distribution of costs, risk, identity and more for building next generation distributed applications. These capabilities help unlock the power of blockchain based applications, while taking advantage of the flexibility and power of the cloud, in a way that works in modern enterprise environment.

You can learn more about our Enterprise Smart Contract framework in our technical whitepaper.

How Azure Security Center helps protect your servers with Web Application Firewall

Hayden Hainsworth — Wed, 19 Jul 2017 10:00:06 Z

Our adversaries have many tools available on the Internet for use in mounting cyberattacks. Many of these tools enable them to gain access and control of enterprise IT resources. In the meantime, security professionals are not always aware of the vulnerabilities built into the IT resources they are tasked to defend. Azure Security Center (ASC) can help bridge this gap.

This blog post is for IT and security professionals interested in using Azure Security Center (ASC) to detect and protect Azure-based resources from SQL injection attacks among others. The goal of this post is to 1) explain how this well-known code injection occurs and 2) illustrate how ASC detects and resolves this attack to secure your IT resources.

Tools make SQL injection easy

Servers and applications are easy targets for cybercriminals. One well-known method for attacking data-driven applications is via SQL injection. SQL injection is an attack technique where malicious code is injected for execution which leads to un-intended database access. A popular tool attackers can use for malicious injection is sqlmap. By using sqlmap, it is easy to discover vulnerable SQL databases and expose their contents. An attacker only needs to provide the appropriate request headers to authenticate and discover the databases, their tables, and even dump the users and hashed passwords. Once the attacker has this data, their next step is to use brute force analysis on the exposed hashes, another built-in feature of the sqlmap tool to obtain the plaintext user credentials as depicted below.

Identifying risk with Azure Security Center

Azure Security Center (ASC), available on every subscription tier of Azure including free and trial subscriptions, can help identify connected IT assets with an HTTP endpoint. Additionally, ASC can automate the deployment of a Web Application Firewall (WAF) resource to help protect non-compliant resources, while pointing out detected malicious SQL injection attempts. The list of detections points to unprotected web servers where security remediation is needed. ASC scans virtual machines across an Azure subscription and makes recommendations to add Web Application Firewalls where applicable to at-risk resources.

ASC then offers guidance through the process of deploying and configuring a Web Application Firewall for partner or first party solutions.

Further guidance on tunneling IP traffic through the Web Application Firewall is also provided. This process provides an added layer of protection to the vulnerable web application.

Azure Security Center provides you with visibility, now that it’s been added to your resources, on the protections and detections including the Web Application Firewall.

Remedial actions with prevention

Configuring the WAF into prevention mode will prevent the sqlmap tool from accessing databases and tables it shouldn’t have access to. Thus, sqlmap can be prevented from even enumerating the type of database running on the backend, let alone traversing the databases for content. In prevention mode, the WAF prevents suspicious activity. ASC detects this and reports on the activity as it is blocked!

Conclusion

Attack tools such as sqlmap are cheap and easily available. A “defense in depth” approach is critical to ensure applications are not vulnerable to SQL injection. Having visibility and control to detect and protect your resources against these attacks is crucial. ASC enables IT and security professionals to scan cloud-based resources for at-risk endpoints. Following recommendations by ASC, detection and protection can be achieved, helping organizations to meet their standards of information security compliance. While ASC is available on all subscription tiers of Azure, those on the Standard tiers can access a deeper level of insights, actions, and threat protection. If your enterprise requires a deep and granular level of cloud security, activate a free trial of Azure Standard to see how ASC can help your business.

This blog post compliments a deeper dive, step-by-step playbook. To learn more please read ASC Playbook: Protect Servers with Web App Firewall.

Have questions? Email us at AskCESec@microsoft.com.

- Hayden @hhainsworth

Accelerate websites with Azure CDN using Dynamic Site Acceleration

Richard Li — Wed, 19 Jul 2017 10:00:06 Z

Users expect fast, reliable, and personalized web experiences independent of their browser, location, device, or network. However, the very innovations, such as rich interactive content and applications, that make these experiences so engaging also slow page downloads and put the quality of the consumer experience at risk. Standard CDN capability includes the ability to cache files closer to the end users to speed up delivery of static files. However, this does not help to improve the loading speed of dynamic web applications and APIs, which return unique data to each user along with request that cannot be cached. With Azure CDN Dynamic Site Acceleration (DSA), the performance of web pages with dynamic content is significantly improved.

DSA includes a variety of techniques that benefit the latency and performance of dynamic content.

Route / network optimizations: The fastest and most reliable path to your origin to retrieve and deliver dynamic content is constantly evaluated. Content is delivered using optimized network protocols and by using the optimal network delivery path to ensure Internet congestion points and unnecessarily long routes are avoided.

TCP optimization: Internet communications over TCP are not optimized for performance. With DSA, a number of TCP optimizations are applied to improve the performance of the delivery, such as keeping connections open longer and reusing them for additional requests. These optimizations accelerate connection setup, increase the packet transmission rate, and reduce packet loss.

Resource prefetching (Akamai only): Based on user behavior and access patterns, the CDN learns which assets are required by the application and preemptively fetches content from origin and brings it closer to the user so it can be delivered faster.

Adaptive Image Compression (Akamai only): Continuously monitor internet conditions to identify slower network connections, to automaticaly compress and resize JPEG images to deliver the most optimal image based on connection speed.

To learn how to use this feature and for more details, please visit the documentation page.

DSA is available today as an optimization on top of an Azure CDN from Akamai. This feature will be coming to Verizon standard and premium profiles in the near future. Note that there are additional fees associated with the DSA optimization type, unlike the media and large file optimization types which are available without additional cost.

Build custom video AI workflows with Video Indexer and Logic Apps

Anika Zaman — Wed, 19 Jul 2017 09:00:05 Z

With the Video Indexer connector for Logic Apps, you can now set up custom workflows connecting your most used apps with Video Indexer to further automate the process of extracting deep insights from your videos.

In this blog, I will walk you through an example of using the Video Indexer connector in a Logic App to set up a workflow where whenever a new video file is created in a specific folder of your OneDrive, the video is automatically uploaded and indexed. Once completed, the insights of the newly indexed video will be stored as a JSON file in the designated folder of your OneDrive.

Limitations to note

Currently, there is a 50 MB file size limit for OneDrive and other storage connectors to trigger. The Video Indexer connector allows you to upload a video via file content from a storage connector or a shared access signature URL. Although there is currently no way to access a URL to the video from storage connectors, we are in the process of adding in this feature on OneDrive, OneDrive for Business, and Azure Storage. Once implemented, we will be able to work with videos larger than 50 MB. However, we have to work with the limit for now.

Setting up the Logic App

To begin, log into your Azure Portal and create a new Logic App. You can follow the tutorial to learn how to create and deploy a new Logic App.

Once you have created the Logic App, go to the Logic Apps Designer and select Blank Logic App.

The first thing we will need is a “Trigger” that will fire off an event when a new file has been created in your OneDrive folder for videos.

In the search bar for connectors and triggers, search for “OneDrive”. You will see options for OneDrive (consumer) and OneDrive for Business. You can do either based on the account that you have or want because they have similar steps. In this tutorial, I am using OneDrive.

Click on the OneDrive connector. This will show you all of the triggers available for OneDrive.

Select “When a file is created”. This will fire the trigger in the Logic App each time a new file is dropped into the designated OneDrive folder. Once complete you will be prompted to sign into your OneDrive account.

After you have signed in, you will see the trigger and its different fields. For the Folder field, click on the folder icon and navigate to your folder for videos. I have selected a folder for videos on my OneDrive called “Video”. You can choose any folder that is appropriate or create a new folder specific to your own workflow. It is important to note that this folder should only have video or audio files. Any other files will result in an upload error in the Video Indexer connector.

You can also set how often you want the trigger to check whether a file has been created in the specified folder. Under the “How often do you want to check for items?” section, I have set Frequency to Minute and Interval to 3 to have my trigger check every 3 minutes.

Next, you will need to set an action that uploads the video that has been created in your OneDrive folder to your Video Indexer portal. Click Next Step and select Add an action.

Search for “Video Indexer” and select the Video Indexer connector. You should see the different actions listed out. We currently do not have any triggers for the Video Indexer, however, triggers will come later when support for WebHooks will be added to the Video Indexer API.

You should see two options for uploading a video onto your Video Indexer portal. One is called Upload video and index and will allow you to upload a video using file content data. The other option is called Upload video and index (using a URL) and will allow you to upload a video using an URL. Both options will automatically index the videos upon upload.

In this tutorial we will be uploading the video using file content data, so select Upload video and index.

You should be prompted to create a connection using your Video Indexer API Key. Enter in a name for the connection as well as your API Key. You can follow the tutorial to learn how to subscribe to the Video Indexer API and how to access your API key.

Upon creating the connection, it should open the Upload video and index action. If you click on any of the fields, you should see response elements from the OneDrive trigger. For the File Content field, select the File content response element. For Video Name, you can select the File name response element or type any name that you want. Set your privacy as you want. Here, I have set privacy to “Private”.

Upon clicking Show advanced options, you will see many more fields that you can fill out to provide more information on your video. I will be leaving them blank here because they are not required fields.

The Upload video and index action returns the id of the video upon upload, however, that does not mean that the indexing has been completed. For this, you need to add in a check that will only let the Logic App move forward if the video has been fully processed. Select New Step and then More. You can then select Add a do until.

You should now see an Until loop. Within the Until loop, select Add an action. Search for the Video Indexer connector again and select the action Get processing state. For the Video Id field, select the Video Id response element from Upload video and index.

For the “Choose a value” field in the Until loop, select the State response element from Get processing state. For the field that says “Choose a value”, type in “Processed”. The State being “Processed” is an indication that the indexing of the video is complete.

Within the Until loop and after the Get processing state action, select Add an action. Search for and select the Delay action (it is a part of the Schedule connector). You will need to set the count and unit fields to essentially determine how often to check if the State is “Processed”. Here, I have set Count to “3” and Unit to “Minute” to check every 3 minutes.

The next step is to attain the insights of the newly processed video. Outside of the Until loop, select New Step and search for the Video Indexer Connector. Select the Get video breakdown action. For the Video Id field, select the Video Id response element from Upload video and index. This action will give you all of the insights of the video.

Now that you have the insights of the video through Get video breakdown, you will now create a file with the new insights and store it in an appropriate folder of your OneDrive.

Select Add an action and search for the OneDrive connector. Select the action Create file. For the Folder path field, click on the folder icon and navigate to the appropriate folder for storing the insights of your video. I chose my folder called “Insights”.

For the File name field, type or select a name for the new text file. Here, I selected the Name response element from the Get video breakdown and typed in “Insights” after. For the File content field, select Summarized Insights or whichever specific response element from Get video breakdown that you want to store. Learn more about the response elements.

Save your logic app, and you are done! You should now test the logic app.

Testing the Logic App

Start by selecting Run. Then, to trigger the logic app, upload a video file onto the OneDrive video folder that you specified in the trigger. As mentioned before, there is a 50 MB file size limit for the OneDrive trigger, so you will need to select your video file appropriately.

You should be able to look at the run details of your Logic App under the Run History section of the Overview page of your logic app.

You are now ready to test out many different combinations of workflows using the Video Indexer connector to find what works best to make your processes automated and more efficient.

You can create custom workflows that integrate live and on demand workflows in Media Services with Video Indexer using samples from the Media Services GitHub site and the Video Indexer connector as long as your video files are within the 50 MB limit for now. You can also create a Logic App to push the insights from Video Indexer into systems like Cosmos DB and use Azure Search to query across the metadata or to join the insights to your own custom metadata.

Microsoft Azure Blog

HDInsight tools for IntelliJ & Eclipse June updates

Summary of key updates

Improved Spark job view and job graph in IntelliJ and Eclipse

Improved Spark log view in IntelliJ and Eclipse

SBT build tool support when creating Spark project in IntelliJ

How to install/update

Feedback

More resource policy aliases

Custom Image for virtual machines

Platform Images

Use Managed Disk

VM Extension Types

Azure Hybrid Use Benefit

Summary

Go serverless with R Scripts on Azure Function

Bing Map v8 available in Azure IoT Suite Remote Monitoring preconfigured solution

Introducing the Azure Analysis Services web designer

Instant File Recovery from Azure VM backups is now generally available

Value Proposition Recap

Related links and additional content

Introducing Enterprise Smart Contracts

Introduction

Enterprise Smart Contracts - Full Whitepaper

Enterprise Smart Contract Components

Implementing Enterprise Smart Contracts

Azure – The Enabler

Enterprise Smart Contracts - Framework

Summary

How Azure Security Center helps protect your servers with Web Application Firewall

Tools make SQL injection easy

Identifying risk with Azure Security Center

Remedial actions with prevention

Conclusion

Accelerate websites with Azure CDN using Dynamic Site Acceleration

Build custom video AI workflows with Video Indexer and Logic Apps

Limitations to note

Setting up the Logic App

Testing the Logic App