AWS Blog

Amazon Aurora already allows you to make your choice of five DB instance classes ranging from the db.r3.large (2 vCPUs and 15 GiB of RAM) up to the db.r3.8xlarge (32 vCPUs and 244 GiB of RAM). These instances support a very wide range of production-scale applications and use cases.

Today we giving you a sixth choice, the new db.t2.medium DB instance class with 2 vCPUs and 4 GiB of RAM. Steady-state, this instance class has access to 40% of the performance of a single core, and can burst to full-core performance when processing CPU-intensive queries and other database tasks. Like the similarly-named EC2 instances, this new instance class starts out with a full allocation of CPU Credits, which are spent when the instance is active and accumulate when it is not (my post, New Low Cost EC2 Instances with Burstable Performance, contains a full explanation).

The db.t2.medium should be a great fit for many of your development and test scenarios, and you should also consider them for some of your less-demanding production workloads. You can monitor the CPUCreditUsage and CPUCreditBalance metrics to track the usage and accumulation of credits over tiem.

Now Available
You can create Amazon Aurora database instances that make use of this new DB instance class today. It is available in all regions where Amazon Aurora is currently available. Prices start at $0.082 per hour.

— Jeff;

My colleague Philip “Fitz” Fitzsimons wrote the guest post below to bring you up to date on the popular Well-Architected Framework.

— Jeff;

Working backward is a fundamental part of our innovation process. We start with the customer and what they want, and let that define and guide our efforts. Having released the Well-Architected Framework in 2015 with the blog post “Are You Well-Architected?” we listened to feedback from our customers to see how we could improve the framework.

We made a number of improvements, firstly we updated the Framework based on the findings from thousands of reviews carried out by AWS Solution Architects. We’ve also responded to customers asking for more guidance on how to operate in the cloud by adding a fifth pillar to the Framework – capturing best practice for Operational Excellence. The full set of pillars are:

• Security – The ability to protect information, systems, and assets while delivering business value through risk assessments and mitigation strategies.
• Reliability – The ability of a system to recover from infrastructure or service failures, dynamically acquire computing resources to meet demand, and mitigate disruptions such as misconfigurations or transient network issues.
• Performance Efficiency – The ability to use computing resources efficiently to meet system requirements, and to maintain that efficiency as demand changes and technologies evolve.
• Cost Optimization – The ability to avoid or eliminate unneeded cost or suboptimal resources.
• Operational Excellence – The ability to run and monitor systems to deliver business value and to continually improve supporting processes and procedures.

The AWS Well-Architected Framework whitepaper positions our perspectives on how to think about architecture in the cloud. It outlines a set of best practices for the pillars and offers a set of open-ended questions that help you to understand where you are with respect to the best practices.

Further to this, customers asked us to be more prescriptive in our advice on how to architect on the cloud, so we created a series of whitepapers that include prescriptive advice on how to build cloud native architectures in the context of each pillar. These whitepapers can all be found at the new home for Well-Architected, where you can always find our most current thinking.

Additionally, to help you adopt Well-Architected, we have launched free online training. The AWS Well-Architected training course  is designed to provide a deep dive into the AWS Well-Architected Framework and its five pillars.

We believe that by working backwards we can build better offerings for our customers. We offer the best practices identified in Well-Architected in the spirit of helping you play it forward for your business.

— Philip Fitzsimons, Leader, AWS Well-Architected Team

AWS Mobile Hub allows you to build, test, and monitor mobile applications that make use of one or more AWS services. You can use its single, integrated console to add user authentication, data storage, backend logic, push notifications, content delivery, and analytics features to your apps, with plenty of opportunities for customization.

We are seeing an increasing number of developers take on the challenges of enterprise mobility. They want to build B2C (business to consumer) and B2E (business to employee) applications that are as powerful, flexible, and secure as their existing desktop app. They want to use their existing corporate user directories, and they want to tap in to existing CRM, sales management, accounting, and customer service tools.

Today we are extending Mobile Hub with support for new SaaS integration features that will be of special interest to developers who are building these types of apps. These new features are a followup to our recent launch of support for the use of AWS Lambda functions and Amazon API Gateway endpoints to add Enhanced Cloud Logic to your applications for building custom connectors. Both features work together with recently launched enterprise user management capabilities to authenticate app users.

Enterprise User Management
We recently added new features that make it easier for you to add user authentication for your B2C and B2E apps.

On the B2C side you can create your own fully managed directory in Amazon Cognito and then add sign-up and sign-in features, including multi-factor authentication (MFA). On the B2E side, you can allow users to sign in using their exist corporate credentials. This feature makes use of SAML (Security Assertion Markup Language) and works with multiple directories including Active Directory. To learn more, read Introducing Mobile Hub User Authentication Using SAML Federation or Email and Password Sign-in.

When you make use of either of these sign-in features, AWS Identity and Access Management (IAM) roles are automatically assigned to each user. You can use these credentials to gain access to AWS resources and the new Mobile Hub connectors described below.

SaaS Integration
Many B2C and B2E apps need to access data that is stored within enterprise SaaS applications. Our new Mobile Hub connectors allow you to build apps that access commonly used objects (Accounts, Contacts, Leads, and so forth) across Salesforce, Microsoft Dynamics, Marketo, HubSpot, Zendesk, and QuickBooks. You can also run queries that access and retrieve custom objects from these applications.

The connectors take the form of REST microservices that are implemented as AWS Lambda functions and accessed through Amazon API Gateway, making them easy  to configure and run. To get started, you simply click on the desired connector and enter your credentials:

Calls to the connectors are routed through API Gateway to the underlying SaaS application. This model allows you to audit and meter calls, throttle requests to the SaaS application, and cache responses. The connectors normalize the SaaS applications and provide you with uniform, consistent objects within a category (CRM, marketing automation, customer support, and so forth). We’ve provided Swagger 2.0 definitions for all of the REST APIs and reference mobile apps for each SaaS application.

Behind the scenes, creating a connector starts a process that uses a AWS CloudFormation template that sets up the IAM roles, API definitions, and Lambda functions. Here’s the API definition for the Salesforce connector:

You can also build your own custom connectors using Mobile Hub’s Cloud Logic features and gain all of the benefits that I just listed. Your connector can make use of fine-grained access control; for example, you can restrict operations like PUT, POST, and DELETE on specific resource paths to a limited set of app users.

To learn more and to see the full list of connectors, visit the AWS Mobile Hub Developer Guide.

Available Now
These new features are available now and you can start using them today!

— Jeff;

In today’s guest post, my colleague Colm MacCárthaigh discusses the new CloudWatch Percentile Statistics and the implications and benefits load balancing. He also introduces a new request tracing feature that will make it easier for you to track the progress of individual requests as they arrive at a load balancer and make their way to the application.

— Jeff;

Application Performance Percentile Metrics
AWS Elastic Load Balancers have long included Amazon CloudWatch metrics that provide visibility into the health and performance of the instances and software running behind the load balancer. For instance, ELBs measure the response time of every request sent to the application being load balanced. Many customers use the built-in average and maximum latency metrics to monitor the performance of their application. It’s also common to use these metrics to trigger Auto Scaling to add or remove instances or containers as load fluctuates.

Based on the new CloudWatch Percentile support, ELB Classic and Application Load Balancers (ALBs) now provide more fine-grained data for application response time. These measurements allow you to get a better sense of performance across a range, for example if the 10th percentile value is 2 milliseconds, then 10% of requests took 2 milliseconds or less, if the 99.9th percentile value is 50 ms, that means that 99.9% of requests took 50 ms or less and so on.

For a real-world example of how useful percentiles are, suppose an application serves the majority of requests from a fast cache in one or two milliseconds, but when the cache is empty the requests are served much more slowly, perhaps in the 100 ms to 200 ms range. The traditional maximum metric will reflect only the slowest case, around 200 ms, and the average will reflect a blend of the fast and slow responses without giving much information about how many requests are fast, how many are slow, and what the spread within each group is. Percentile metrics illuminate the application’s performance across the range of requests and give a more comprehensive picture.

Percentile metrics can be used to set more meaningful and more aggressive performance targets for your application. For example, by using the 99th percentile as an Auto Scaling trigger or CloudWatch alarm, you can set a scaling target that will provide enough resources so that no more than 1% of the requests take 2 ms or longer to process.

Integrated Request Tracing
While percentiles provide greater visibility into your application with a “big picture” summary of performance across all requests, AWS Application Load Balancer’s new request tracing feature allows you to dive deep into application performance and health at the nitty-gritty level of individual requests.

ALBs now look for a new custom X-Amzn-Trace-ID HTTP header on all requests to an ALB. If this header is absent on an incoming request, then an ALB will generate one that looks like this:

X-Amzn-Trace-Id: Root=1-58211399-36d228ad5d99923122bbe354 

and pass it on to the application. Because the header was absent on the incoming request, this request is considered a “root” request and the new header that ALB sends will include a randomly-generated Root attribute.

If the header is present on an incoming request, then its contents will be preserved, so the header can also be used to pass and preserve tracing and diagnostic state, such as ancestry and timing data, between requests. For example, a request that is generated like this:

$ curl -H \
  "X-Amzn-Trace-Id: Root=1-58211399-36d228ad5d99923122bbe354; TimeSoFar=112ms; CalledFrom=Foo" \
  request-tracing-demo-1290185324.elb.amazonaws.com 

will cause ALB to generate a slightly different header:

X-Amzn-Trace-Id: Self=1-582113d1-1e48b74b3603af8479078ed6;\
  Root=1-58211399-36d228ad5d99923122bbe354;\
  TotalTimeSoFar=112ms;CalledFrom=Foo 

If, as in this example, the incoming header included its own Root attribute, then ALB will preserve it and add a new randomly-generated “Self” attribute instead. The TotalTimeSoFar and CalledFrom attributes in this example are arbitrary and optional, and are actually meaningless to the Load Balancer itself, but are preserved as-is to allow applications and tracing frameworks to pass data between requests.

For applications that don’t set the header on incoming requests, ALB’s request tracing acts as a simple unique identifier for every request. For applications that pass the header between requests, tracing allows you to easily identify the entry point into the overall system, to find related requests with a Root attribute, and it enables applications to create a trail of “breadcrumbs” that spans requests.

The contents of the X-Amzn-Trace-ID header are logged to Application Load Balancer access logs, which contain error indicators and performance measurements for each request, so the unique identifiers can be used to pinpoint issues with individual requests.

http 2016-11-08T00:04:13.109451Z app/request-tracing-demo/f2c895fd3ea253c5 \
  72.21.217.129:20555 172.31.51.164:80 0.001 0.001 0.000 200 200 276 550 \
  "GET http://request-tracing-demo-1290185324.elb.amazonaws.com:80/ HTTP/1.1" \
  "curl/7.24.0 (x86_64-redhat-linux-gnu) libcurl/7.24.0 NSS/3.16.1 \
  Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2" - - \
  arn:aws:elasticloadbalancing:us-east-1:99999999999:targetgroup/echoheader/21a6f1501d2df426 \
  "Root=1-5821167d-1d84f3d73c47ec4e58577259"

Percentile statistics are available today for all new Application Load Balancers and can be accessed from the CloudWatch Console, AWS SDKs, and API. Support for existing Application Load Balancers and all Classic Load Balancers will be available in the coming weeks. Both new features are included at no extra cost.

— Colm MacCárthaigh, AWS Elastic Load Balancer

Back in 2006, we launched S3 with a revolutionary pay-as-you-go pricing model, with an initial price of 15 cents per GB per month. Over the intervening decade, we reduced the price per GB by 80%, launched S3 in every AWS Region, and enhanced the original one-size-fits-all model with user-driven features such as web site hosting, VPC integration, and IPv6 support, while adding new storage options including S3 Infrequent Access.

Because many AWS customers archive important data for legal, compliance, or other reasons and reference it only infrequently, we launched Glacier in 2012, and then gave you the ability to transition data between S3, S3 Infrequent Access, and Glacier by using lifecycle rules.

Today I have two big pieces of news for you: we are reducing the prices for S3 Standard Storage and for Glacier storage. We are also introducing additional retrieval options for Glacier.

S3 & Glacier Price Reduction
As long-time AWS customers already know, we work relentlessly to reduce our own costs, and to pass the resulting savings along in the form of a steady stream of AWS Price Reductions.

We are reducing the per-GB price for S3 Standard Storage in most AWS regions, effective December 1, 2016. The bill for your December usage will automatically reflect the new, lower prices. Here are the new prices for Standard Storage:

As you can see from the table above, we are also simplifying the pricing model by consolidating six pricing tiers into three new tiers.

We are also reducing the price of Glacier storage in most AWS Regions. For example, you can now store 1 GB for 1 month in the US East (Northern Virginia), US West (Oregon), or EU (Ireland) Regions for just $0.004 (less than half a cent) per month, a 43% decrease. For reference purposes, this amount of storage cost $0.010 when we launched Glacier in 2012, and $0.007 after our last Glacier price reduction (a 30% decrease).

The lower pricing is a direct result of the scale that comes about when our customers trust us with trillions of objects, but it is just one of the benefits. Based on the feedback that I get when we add new features, the real value of a cloud storage platform is the rapid, steady evolution. Our customers often tell me that they love the fact that we anticipate their needs and respond with new features accordingly.

New Glacier Retrieval Options
Many AWS customers use Amazon Glacier as the archival component of their tiered storage architecture. Glacier allows them to meet compliance requirements (either organizational or regulatory) while allowing them to use any desired amount of cloud-based compute power to process and extract value from the data.

Today we are enhancing Glacier with two new retrieval options for your Glacier data. You can now pay a little bit more to expedite your data retrieval. Alternatively, you can indicate that speed is not of the essence and pay a lower price for retrieval.

We launched Glacier with a pricing model for data retrieval that was based on the amount of data that you had stored in Glacier and the rate at which you retrieved it. While this was an accurate reflection of our own costs to provide the service, it was somewhat difficult to explain. Today we are replacing the rate-based retrieval fees with simpler per-GB pricing.

Our customers in the Media and Entertainment industry archive their TV footage to Glacier. When an emergent situation calls for them to retrieve a specific piece of footage, minutes count and they want fast, cost-effective access to the footage. Healthcare customers are looking for rapid, “while you wait” access to archived medical imagery and genome data; photo archives and companies selling satellite data turn out to have similar requirements. On the other hand, some customers have the ability to plan their retrievals ahead of time, and are perfectly happy to get their data in 5 to 12 hours.

Taking all of this in to account, you can now select one of the following options for retrieving your data from Glacier (The original rate-based retrieval model is no longer applicable):

Standard retrieval is the new name for what Glacier already provides, and is the default for all API-driven retrieval requests. You get your data back in a matter of hours (typically 3 to 5), and pay $0.01 per GB along with $0.05 for every 1,000 requests.

Expedited retrieval addresses the need for “while you wait access.” You can get your data back quickly, with retrieval typically taking 1 to 5 minutes.  If you store (or plan to store) more than 100 TB of data in Glacier and need to make infrequent, yet urgent requests for subsets of your data, this is a great model for you (if you have less data, S3’s Infrequent Access storage class can be a better value). Retrievals cost $0.03 per GB and $0.01 per request.

Retrieval generally takes between 1 and 5 minutes, depending on overall demand. If you need to get your data back in this time frame even in rare situations where demand is exceptionally high, you can provision retrieval capacity. Once you have done this, all Expedited retrievals will automatically be served via your Provisioned capacity. Each unit of Provisioned capacity costs $100 per month and ensures that you can perform at least 3 Expedited Retrievals every 5 minutes, with up to 150 MB/second of retrieval throughput.

Bulk retrieval is a great fit for planned or non-urgent use cases, with retrieval typically taking 5 to 12 hours at a cost of $0.0025 per GB (75% less than for Standard Retrieval) along with $0.025 for every 1,000 requests. Bulk retrievals are perfect when you need to retrieve large amounts of data within a day, and are willing to wait a few extra hours in exchange for a very significant discount.

If you do not specify a retrieval option when you call InitiateJob to retrieve an archive, a Standard Retrieval will be initiated. Your existing jobs will continue to work as expected, and will be charged at the new rate.

To learn more, read about Data Retrieval in the Glacier FAQ.

As always, I am thrilled to be able to share this news with you, and I hope that you are equally excited!

— Jeff;

I would like to show you how several different AWS services can be used together to address a challenge faced by many of our customers. Along the way I will introduce you to a new AWS CloudTrail feature that launches today and show you how you can use it in conjunction with CloudWatch Events.

The Challenge
Our customers store many different types of mission-critical data in Amazon Simple Storage Service (S3) and want to be able to track object-level activity on their data. While some of this activity is captured and stored in the S3 access logs, the level of detail is limited and log delivery can take several hours. Customers, particularly in financial services and other regulated industries, are asking for additional detail, delivered on a more timely basis. For example, they would like to be able to know when a particular IAM user accesses sensitive information stored in a specific part of an S3 bucket.

In order to meet the needs of these customers, we are now giving CloudTrail the power to capture object-level API activity on S3 objects, which we call Data events (the original CloudTrail events are now called Management events). Data events include “read” operations such as GET, HEAD, and Get Object ACL as well as “write” operations such as PUT and POST. The level of detail captured for these operations is intended to provide support for many types of security, auditing, governance, and compliance use cases. For example, it can be used to scan newly uploaded data for Personally Identifiable Information (PII), audit attempts to access data in a protected bucket, or to verify that the desired access policies are in effect.

Processing Object-Level API Activity
Putting this all together, we can easily set up a Lambda function that will take a custom action whenever an S3 operation takes place on any object within a selected bucket or a selected folder within a bucket.

Before starting on this post, I created a new CloudTrail trail called jbarr-s3-trail:

I want to use this trail to log object-level activity on one of my S3 buckets (jbarr-s3-trail-demo). In order to do this I need to add an event selector to the trail. The selector is specific to S3, and allows me to focus on logging the events that are of interest to me. Event selectors are a new CloudTrail feature and are being introduced as part of today’s launch, in case you were wondering.

I indicate that I want to log both read and write events, and specify the bucket of interest. I can limit the events to part of the bucket by specifying a prefix, and I can also specify multiple buckets. I can also control the logging of Management events:

CloudTrail supports up to 5 event selectors per trail. Each event selector can specify up to 50 S3 buckets and optional bucket prefixes.

I set this up, opened my bucket in the S3 Console, uploaded a file, and took a look at one of the entries in the trail. Here’s what it looked like:

{
  "eventVersion": "1.05",
  "userIdentity": {
    "type": "Root",
    "principalId": "99999999999",
    "arn": "arn:aws:iam::99999999999:root",
    "accountId": "99999999999",
    "username": "jbarr",
    "sessionContext": {
      "attributes": {
        "creationDate": "2016-11-15T17:55:17Z",
        "mfaAuthenticated": "false"
      }
    }
  },
  "eventTime": "2016-11-15T23:02:12Z",
  "eventSource": "s3.amazonaws.com",
  "eventName": "PutObject",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "72.21.196.67",
  "userAgent": "[S3Console/0.4]",
  "requestParameters": {
    "X-Amz-Date": "20161115T230211Z",
    "bucketName": "jbarr-s3-trail-demo",
    "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
    "storageClass": "STANDARD",
    "cannedAcl": "private",
    "X-Amz-SignedHeaders": "Content-Type;Host;x-amz-acl;x-amz-storage-class",
    "X-Amz-Expires": "300",
    "key": "ie_sb_device_4.png"
  }

Then I create a simple Lambda function:

Next, I create a CloudWatch Events rule that matches the function name of interest (PutObject) and invokes my Lambda function (S3Watcher):

Now I upload some files to my bucket and check to see that my Lambda function has been invoked as expected:

I can also find the CloudWatch entry that contains the output from my Lambda function:

Pricing and Availability
Data events are recorded only for the S3 buckets that you specify, and are charged at the rate of $0.10 per 100,000 events. This feature is available in all commercial AWS Regions.

— Jeff;

Back in 2011 I introduced you to Custom Metrics for CloudWatch and showed you how to publish them from your applications and scripts. At that time, the first ten custom metrics were free of charge and additional metrics were $0.50 per metric per month, regardless of the number of metrics that you published.

Today, I am happy to announce a price change and a quantity discount for CloudWatch metrics. Based on the number of metrics that you publish every month, you can realize savings of up to 96%. Here is the new pricing for the US East (Northern Virginia) Region (the first ten metrics are still free of charge):

If you have EC2 Detailed Monitoring enabled you will also see a price reduction with per-month charges reduced from $3.50 per instance per month to $2.10 or lower based on the volume tier. The new prices will take effect on December 1, 2016 with no effort on your part. At that time, the updated prices will be published on the CloudWatch Pricing page.

By the way, if you are using CloudWatch Metrics, be sure to take advantage of other recently announced features such as Extended Metrics Retention, the CloudWatch Plugin for Collectd, CloudWatch Dashboards, and the new Metrics-to-Logs navigation feature.

— Jeff;

We launched WorkSpaces in late 2013 (Amazon WorkSpaces – Desktop Computing in the Cloud) and have been adding new features at a rapid clip. Here are some highlights from 2016:

• November 2016 – WorkSpaces adds GPU-Powered Graphics Bundles.
• October 2016 – WorkSpaces becomes available in the EU (Frankfurt) Region.
• August 2016 – WorkSpaces offers hourly pricing for all WorkSpaces bundles & AWS Marketplace for Desktop Apps in the Asia Pacific (Singapore) Region.
• July 2016 – WorkSpaces allows you to bring your own Windows 10 desktop licenses.
• June 2016 – WorkSpaces now come with larger root volumes.
• May 2016 – WorkSpaces support tagging.
• April 2016 – AWS Marketplace for Desktop Apps in the EU (Ireland) Region.
• February 2016 – WorkSpaces Application Manager now available in the Asia Pacific (Sydney) and Asia Pacific (Singapore) Regions.
• January 2016 – Support for audio-in, high-DPI devices, and saved registrations.

Today we are adding to this list with the addition of Amazon WorkSpaces Web Access. You can now access your WorkSpace from recent versions of Chrome or Firefox running on Windows, Mac OS X, or Linux. You can now be productive on heavily restricted networks and in situations where installing a WorkSpaces client is not an option. You don’t have to download or install anything, and you can use this from a public computer without leaving any private or cached data behind.

To use Amazon WorkSpaces Web Access, simply visit the registration page using a supported browser and enter the registration code for your WorkSpace:

Then log in with your user name and password:

And here you go (yes, this is IE and Firefox running on WorkSpaces, displayed in Chrome):

This feature is available for all new WorkSpaces that are running the Value, Standard, or Performance bundles or their Plus counterparts. You can access it at no additional charge after your administrator enables it:

Existing WorkSpaces must be rebuilt and custom images must be refreshed in order to take advantage of Web Access.

— Jeff;

I am thrilled by all of the excitement that I see around AWS Lambda and serverless application development. I have shared many serverless success stories, tools, and open source projects in the AWS Week in Review over the last year or two.

Today I would like to tell you about two important additions to Lambda: environment variables and the new Serverless Application Model.

Environment Variables
Every developer likes to build code that can be used in more than one environment. In order to do this in a clean and reusable fashion, the code should be able to accept configuration values at run time. The configuration values customize the environment for the code: table names, device names, file paths, and so forth. For example, many projects have distinct configurations for their development, test, and production environments.

You can now supply environment variables to your Lambda functions. This allows you to effect configuration changes without modifying or redeploying your code, and should make your serverless application development even more efficient. Each environment variable is a key/value pair. The keys and the values are encrypted using AWS Key Management Service (KMS) and decrypted on an as-needed basis. There’s no per-function limit on the number of environment variables, but the total size can be no more than 4 kb.

When you create a new version of a Lambda function, you also set the environment variables for that version of the function. You can modify the values for the latest version of the function, but not for older versions. Here’s how I would create a simple Python function, set some environment variables, and then reference them from my code (note that I had to import the os library):

There’s no charge for this feature if you use the default service key provided by Lambda (the usual per-request KMS charges apply if you choose to use your own key).

To learn more and to get some ideas for other ways to make use of this new feature, read Simplify Serverless Applications With Lambda Environment Variables on the AWS Compute Blog.

AWS Serverless Application Model
Lambda functions, Amazon API Gateway resources, and Amazon DynamoDB tables are often used together to build serverless applications. The new AWS Serverless Application Model (AWS SAM) allows you describe all of these components using a simplified syntax that is natively supported by AWS CloudFormation. In order to use this syntax, your CloudFormation template must include a Transform section (this is a new aspect of CloudFormation) that looks like this:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

The remainder of the template is used to specify the Lambda functions, API Gateway endpoints & resources, and DynamoDB tables. Each function declaration specifies a handler, a runtime, and a URI to a ZIP file that contains the code for the function.

APIs can be declared implicitly by defining events, or explicitly, by providing a Swagger file.

DynamoDB tables are declared using a simplified syntax that requires just a table name, a primary key (name and type), and the provisioned throughput. The full range of options is also available for you to use if necessary.

You can now generate AWS SAM files and deployment packages for your Lamba functions using a new Export operation in the Lambda Console. Simply click on the Actions menu and select Export function:

Then click on Download AWS SAM file or Download deployment package:

Here is the AWS SAM file for my function:

AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Description: A starter AWS Lambda function.
Resources:
  ShowEnv:
    Type: 'AWS::Serverless::Function'
    Properties:
      Handler: lambda_function.lambda_handler
      Runtime: python2.7
      CodeUri: .
      Description: A starter AWS Lambda function.
      MemorySize: 128
      Timeout: 3
      Role: 'arn:aws:iam::99999999999:role/LambdaGeneralRole'

The deployment package is a ZIP file with the code for my function inside. I would simply upload the file to S3 and update the CodeUri in the SAM file in order to use it as part of my serverless application. You can do this manually or you can use a pair of new CLI commands (aws cloudformation package and aws cloudformation deploy) to automate it. To learn more about this option, read the section on Deploying a Serverless app in the new Introducing Simplified Serverless Application Management and Deployment post.

You can also export Lambda function blueprints. Simply click on the download link in the corner:

And click on Download blueprint:

The ZIP file contains the AWS SAM file and the code:

To learn more and to see this new specification in action, read Introducing Simplified Serverless Application Management and Deployment on the AWS Compute Blog.

— Jeff;

The Amazon EMR team is cranking out new features at an impressive pace (guess they have lots of worker nodes)! So far this quarter they have added all of these features:

• September – Data Encryption for Apache Spark, Tez, and Hadoop MapReduce.
• September – Open-sourced EMR-DynamoDB Connector for Apache Hive.
• November – Stream Processing at Scale with Apache Flink.
• November – Fine-grained Access Control Using Cluster Tags.

Today we are adding to this list with the addition of automatic scaling for EMR clusters. You can now use scale out and scale in policies to adjust the number of core and task nodes in your clusters in response to changing workloads and to optimize your resource usage:

Scale out Policies add additional capacity and allow you to tackle bigger problems. Applications like Apache Spark and Apache Hive will automatically take advantage of the increased processing power as it comes online.

Scale in Policies remove capacity, either at the end of an instance billing hour or as tasks complete. If a node is removed while it is running a YARN container, YARN will rerun that container on another node (read Configure Cluster Scale-Down Behavior for more info).

Using Auto Scaling
In order to make use of Auto Scaling, an IAM role that give Auto Scaling permission to launch and terminate EC2 instances must be associated with your cluster. If you create a cluster from the EMR Console, it will create the EMR_AutoScaling_DefaultRole for you. You can use it as-is or customize it as needed. If you create a cluster programmatically or via the command-line, you will need to create it yourself. You can also create the default roles from the command line like this:

$ aws emr create-default-roles

From the console, you can edit the Auto Scaling policies by clicking on the Advanced Options when you create your cluster:

Simply click on the pencil icon to begin editing your policy. Here’s my Scale out policy:

Because this policy is driven by YARNMemoryAvailablePercentage, it will be activated under low-memory conditions when I am running a YARN-based framework such as Spark, Tez, or Hadoop MapReduce. I can choose many other metrics as well; here are some of my options:

And here’s my Scale in policy:

I can choose from the same set of metrics, and I can set a Cooldown period for each policy. This value sets the minimum amount of time between scaling activities, and allows the metrics to stabilize as the changes take effect.

Default policies (driven by YARNMemoryAvailablePercentage and ContainerPendingRatio) are also available in the console.

Available Now
To learn more about Auto Scaling, read about Scaling Cluster Resources in the EMR Management Guide.

This feature is available now and you start using it today. Simply select emr-5.1.0 from the Release menu to get started!

— Jeff;