Posted by Laurence Moroney, Developer Advocate

In this week’s Coffee with a Googler, we’re joined by Heidi Dohse from the Google Cloud team to talk about how she saved her own life through technology. At Google she works on the Cloud Platform team that supports our genomics work, and has a passion for the future of the delivery of health care.

When she was a child, Heidi Dohse had an erratic heartbeat, but, without knowing anything about it, she just ignored it. As a teen she became a competitive windsurfer and skier, and as part of a surgery on her knee between seasons, she had an EKG and discovered that her heart was beating irregularly at 270bpm.

She had an experimental surgery and was immediately given a pacemaker, and became a young heart patient, expecting not to live much longer. Years later, Heidi is now on her 7th pacemaker, but it doesn’t stop her from staying active as an athlete. She’s a competitive cyclist, often racing hundreds of miles, and climbing tens of thousands of feet as she races.

At the core of all this is her use of technology. She has carefully managed her health by collecting and analyzing the data from her pacemaker. The data goes beyond just heartbeat, and includes things such as the gyroscope, oxygen utilization, muscle stimulation and electrical impulses, she can pro-actively manage her health.

It’s the future of health care -- instead of seeing a doctor for an EKG every few months, with this technology and other wearables, one can constantly check their health data, and know ahead of time if there would be any health issues. One can proactively ensure their health, and pre-empt any health issues.

Learn more in the video.

By Jason Polites, Cloud Platform Team

Many mobile apps today suffer from “app-nesia” — the affliction that causes an app to forget who you are. Have you ever re-installed an app only to discover you have to re-create all your carefully crafted preferences? This is typically because the user’s app data lives only on the device.

By connecting your apps to a backend platform, you can solve this issue, but it can be challenging. Whether it’s building basic plumbing, or just trying to load and save data in a network & battery-efficient way, spending time dealing with the backend can take precious time away from building an awesome app. So, we’re introducing two new features to help make your life easier.

Google Cloud Save
Google Cloud Save allows you to easily load and save user data to the cloud without needing to code up the backend. This is handy for situations where you want to save user state and have that state synchronized to multiple devices, or survive an app reinstall.

We handle all the backend logic as well as the synchronization services on the client. The synchronization services work in the background, providing offline support for the data, and minimizing impact on the battery. All you need to do is tell us when and what to save, and you do this with just 4 simple methods:

• .save(client, List<Entity>)
• .delete(client, Query)
• .query(client, Query)
• .requestSync(client)

All data is written locally first, then automatically synchronized in the background. The save, delete and query methods provide your basic CRUD operations while the requestSync method allows you to force a synchronization at any time. On the backend the data is stored in Google Cloud Datastore which means you can access the raw data directly from a Google App Engine or Google Compute Engine instance using the existing Datastore API. Changes on the server will even be automatically synced back to client devices. Importantly, this per-user data belongs to you, the developer, and stored in your own Google Cloud Datastore database. Google Cloud Save is currently in private beta and will be available for general use soon. If you’re interested in participating in the private beta, you can sign up here!

Cloud Tools for Android Studio
To simplify the process of adding an App Engine backend to your app, Android Studio now provides three App Engine backend module templates which you can add to your app:

• App Engine Java Servlet Module - Minimal Backend
• App Engine Java Endpoints Module - Basic Endpoint scaffolding 
• App Engine with Google Cloud Messaging - Push notification wireup 

When you choose one of these template types your project is updated with a new Gradle module containing your new App Engine backend. All of the required dependencies/permissions will be automatically set up for you. Built-in rich editing support for Google Cloud Endpoints
Once you have added the backend module to your Android application, you can use Google Cloud Endpoints to streamline the communication between your backend and your Android app. Cloud Endpoints automatically generates strongly-typed, mobile optimized client libraries from simple Java server-side API annotations, automates Java object marshalling to and from JSON, and provides built-in OAuth 2.0 support. On deployment, this annotated Endpoints API definition class generates a RESTful API. You can explore this generated API (and even make calls to it) by navigating to Endpoints API explorer as shown in the image below:

To simplify calling this generated API from your Android app, Android Studio will automatically set up your project to include all compile dependencies and permissions required to consume Cloud Endpoints, and will re-generate strongly-typed client libraries if your backend changes. This means that you can start calling the client libraries from your Android app immediately after defining the server-side Endpoints API. The underlying work-horses: Gradle, and Gradle plug-in for App Engine Under the hood, Gradle is used to build both your app and your App Engine backend. In fact, when you add an App Engine backend to your Android app, the open-source App Engine plug-in for Gradle is automatically downloaded by Android Studio, and common App Engine tasks become available as Gradle targets. This allows you to use the same build system across your IDE, command-line or continuous integration environments. Checkout more details on the new Cloud Endpoints features in Android Studio on the Android Developer Blog.

Posted by Louis Gray, Googler

By Frances Perry, Google Cloud Platform Team

In today's world, information is being generated at an incredible rate. However, unlocking insights from large datasets can be cumbersome and costly, even for experts.

It doesn’t have to be that way. Yesterday, at Google I/O, you got a sneak peek of Google Cloud Dataflow, the latest step in our effort to make data and analytics accessible to everyone. You can use Cloud Dataflow:

• for data integration and preparation (e.g. in preparation for interactive SQL in BigQuery)
• to examine a real-time stream of events for significant patterns and activities
• to implement advanced, multi-step processing pipelines to extract deep insight from datasets of any size

In these cases and many others, you use Cloud Dataflow’s data-centric model to easily express your data processing pipeline, monitor its execution, and get actionable insights from your data, free from the burden of deploying clusters, tuning configuration parameters, and optimizing resource usage. Just focus on your application, and leave the management, tuning, sweat and tears to Cloud Dataflow.

Cloud Dataflow is based on a highly efficient and popular model used internally at Google, which evolved from MapReduce and successor technologies like Flume and MillWheel. The underlying service is language-agnostic. Our first SDK is for Java, and allows you to write your entire pipeline in a single program using intuitive Cloud Dataflow constructs to express application semantics.

Cloud Dataflow represents all datasets, irrespective of size, uniformly via PCollections (“parallel collections”). A PCollection might be an in-memory collection, read from files on Cloud Storage, queried from a BigQuery table, read as a stream from a Pub/Sub topic, or calculated on demand by your custom code.

Because PCollections can be arbitrarily large, Cloud Dataflow includes a rich library of PTransforms (“parallel transforms”), which you can customize with your own application logic. For example, ParDo (“parallel do”) runs your code over each element in a PCollection independently (like both the Map and Reduce functions in MapReduce or WHERE in SQL), and GroupByKey takes a PCollection of key-value pairs and groups together all pairs with the same key (like the Shuffle step of MapReduce or GROUP BY and JOIN in SQL). In addition, anyone can define new custom transformations by composing other transformations -- this extensibility lets you write reusable building blocks which can be shared across programs. Cloud Dataflow provides a starter set of these composed transforms out of the box, including Count, Top, and Mean.

Writing in this modular, high-level style naturally leads to pipelines that make multiple logical passes over the same data. Cloud Dataflow automatically optimizes your data-centric pipeline code by collapsing multiple logical passes into a single execution pass. However, this doesn't turn the system into a black box: as you can see below, Cloud Dataflow’s monitoring UI uses the building block concept to show you the pipeline as you wrote it, not as the system chooses to execute it.


The same Cloud Dataflow pipeline may run in different ways, depending on the data sources. As you start designing or debugging, you can run against data local to your development environment. When you’re ready to scale up to real data, that same pipeline can run in parallel batch mode against data in Cloud Storage or in distributed real-time processing mode against data coming in via a Pub/Sub topic. This flexibility makes it trivial to transition between different stages in the application development lifecycle: to develop and test applications, to adapt an existing batch pipeline to track time-sensitive trends, or to fix a bug in a real-time pipeline and backfill the historical results.

When you use Cloud Dataflow, you can focus solely on your application logic and let us handle everything else. You should not have to choose between scalability, ease of management and a simple coding model. With Cloud Dataflow, you can have it all.

If you’d like to be notified of future updates about Cloud Dataflow, please join our Google Group.

Posted by Louis Gray, Googler

By Brad Abrams, Google Cloud Platform Team

Whether it’s the next viral game, social sharing app or hit SaaS application, the velocity of your innovation is driven by the productivity of your dev team. This week at Google I/O we talked about several new tools that enable developers to understand, diagnose and improve their systems in production.

Cloud Debugger
Today the state of the art of debugging for cloud applications isn’t much more than writing out diagnostic messages and spelunking the logs for them. When the right data is not being written to the logs, developers have to make a code change and redeploy the application to production. That is the last thing you want to do when investigating an issue in production. Traditional debuggers aren’t well suited for cloud-based services for two reasons. First, it is difficult to know which process to attach to. Second, stopping a process in production makes it hard to reproduce an issue and gives your end-users a bad experience.

The Cloud Debugger completely changes this model. It allows developers to start where they know best - in the code. By simply setting a watchpoint on a line of code, the next time a request on any of your servers hits that line of code, you get a snapshot of all the local variables, parameters, instance variables and a full stack trace. This works no matter how many instances you are running in production. There is zero setup up time and no complex configuration to enable. The debugger is ideal for use in production. There is no overhead for enabling the debugger on a project and when a watchpoint is hit very little noticeable performance impact is seen by your users.

 

Cloud Trace
Performance is an important feature of your service which directly correlates with end user satisfaction and retention. No one intends to build a slow service, but it can be extremely difficult to isolate the root cause of sluggishness when it happens. Especially when the issue hits only a fraction of your users.

Cloud Trace helps you visualize and understand the time spent by your application for request processing. This enables you to quickly identify and fix performance bottlenecks. You can even compare performance from release to release with a detailed report. You can leave Cloud Trace enabled in production because it has very little performance overhead.

In this screenshot, you can see we have investigated a particularly slow trace and we see a detailed breakdown of where the time is being spent. It looks like the problem could be these numerous sequential calls to Datastore, so maybe we should consider batching them.



So we go update our service to batch the Datastore calls, and deploy the updated service. Now we can use Cloud Trace to verify the fix.

  As a developer, you can easily produce a report that shows the performance change in your service from one release to another. In the following report, the blue graph shows the performance without datastore batching and the orange graph shows the performance after releasing the change to use datastore batching. The X-axis of the graph represents the time taken (logarithmic scale) to service requests, and the left shift of the orange graph shows the significant performance gain due to Datastore batching.

 

Cloud Monitoring, Powered by Stackdriver

Cloud Monitoring provides rich dashboards and alerting capabilities that help developers find and fix performance problems quickly.

With minimal configuration and no separate infrastructure to maintain, Cloud Monitoring provides you with deep visibility into your Cloud Platform services. For example, you can use Cloud Monitoring dashboards to diagnose cases where your customers are reporting slow response times or errors accessing your applications:



Likewise, you can create alerting policies so that you are notified when key metrics, such as latency or error rates, pass a given threshold in the future:



You can configure alerts for any metric in the system, including those related to the performance of Cloud SQL databases, App Engine modules and versions, Pub/Sub topics and subscriptions, and Compute Engine VMs. With Compute Engine VMs, you can create alerts for both core system metrics (CPU, memory, etc.) and application services running in the VMs (Apache, Cassandra, MongoDB, etc.).

You can also create dashboards that make it easier to correlate metrics across services. For example, it takes a few clicks to create a dashboard that tracks key metrics for an App Engine module that connects to a set of Redis VMs running on Compute Engine:

Finally, you can create endpoint checks to monitor availability and response times for your end-user facing services. Endpoint checks are performed by probes in Oregon, Texas, Virginia, Amsterdam, and Singapore, enabling monitoring of latency from each of these five regions.

SSH to your VM instantly

Sometimes it is inevitable to connect directly to a VM to debug or fix a production issue. We know this can be a bit of a pain, especially when you are on the road, so now you can do that from just about anywhere. With our new browser based SSH client you can quickly and securely connect to any of your VMs from the Console. No need to install any SDK or tools. The best part is, this works from any desktop device with most major web browsers.



Ready for a Spin? All of these features are just about ready for your applications. Stay tuned to this blog, we will post updates as they are more widely available.

Posted by Louis Gray, Googler

By Greg DeMichillie, Google Cloud Platform team

Today at Google I/O, we are introducing new services that help developers build and optimize data pipelines, create mobile applications, and debug, trace, and monitor their cloud applications in production.

Introducing Google Cloud Dataflow
A decade ago, Google invented MapReduce to process massive datasets using distributed computing. Since then, more devices and information require more capable analytics pipelines — though they are difficult to create and maintain.

Today at Google I/O, we are demonstrating Google Cloud Dataflow for the first time. Cloud Dataflow is a fully managed service for creating data pipelines that ingest, transform and analyze data in both batch and streaming modes. Cloud Dataflow is a successor to MapReduce, and is based on our internal technologies like Flume and MillWheel.

Cloud Dataflow makes it easy for you to get actionable insights from your data while lowering operational costs without the hassles of deploying, maintaining or scaling infrastructure. You can use Cloud Dataflow for use cases like ETL, batch data processing and streaming analytics, and it will automatically optimize, deploy and manage the code and resources required.

Debug, trace and monitor your application in production
We are also introducing several new Cloud Platform tools that let developers understand, diagnose and improve systems in production.

Google Cloud Monitoring is designed to help you find and fix unusual behavior across your application stack. Based on technology from our recent acquisition of Stackdriver, Cloud Monitoring provides rich metrics, dashboards and alerting for Cloud Platform, as well as more than a dozen popular open source apps, including Apache, Nginx, MongoDB, MySQL, Tomcat, IIS, Redis, Elasticsearch and more. For example, you can use Cloud Monitoring to identify and troubleshoot cases where users are experiencing increased error rates connecting from an App Engine module or slow query times from a Cassandra database with minimal configuration.

We know that it can be difficult to isolate the root cause of performance bottlenecks. Cloud Trace helps you visualize and understand time spent by your application for request processing. In addition, you can compare performance between various releases of your application using latency distributions.

Finally, we’re introducing Cloud Debugger, a new tool to help you debug your applications in production with effectively no performance overhead. Cloud Debugger gives you a full stack trace and snapshots of all local variables for any watchpoint that you set in your code while your application continues to run undisturbed in production. This brings modern debugging to cloud-based applications.

New features for mobile development
With rapid autoscaling, caching and other mobile friendly capabilities, many apps like Snapchat or Rising Star have built and run on Cloud Platform. We’re adding new features that make building a mobile app using Cloud Platform even better.

Today, we’re demonstrating a new version of Google Cloud Save, which gives you a simple API for saving, retrieving, and synchronizing user data to the cloud and across devices without needing to code up the backend. Data is stored in Google Cloud Datastore, making the data accessible from Google App Engine or Google Compute Engine using the existing Datastore API. Google Cloud Save is currently in private beta and will be available for general use soon.

We’ve also added tooling to Android Studio, which simplifies the process of adding an App Engine backend to your mobile app. In particular, Android Studio now has three built-in App Engine backend module templates, including Java Servlet, Java Endpoints and an App Engine backend with Google Cloud Messaging. Since this functionality is powered by the open-source App Engine plug-in for Gradle, you can use the same build configuration for both your app and your backend across IDE, CLI and Continuous Integration environments.

We’ll be doing more detailed follow-up posts about these announcements in the coming days, so stay tuned.

Greg DeMichillie has spent his entire career working on developer platforms for web, mobile, and the cloud. He started as a software engineer before making the jump to Product Management. When not coding, he's an avid photographer and gadget geek.

Posted by Louis Gray, Googler

Apache, Nginx, MongoDB, MySQL, Tomcat, IIS, Redis, Elasticsearch and Cassandra are trademarks of their respective owners.

By Felipe Hoffa, Cloud Platform team

Cross-posted from the Google Cloud Platform Blog
Editor's note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.

Last Tuesday we announced an exciting set of changes to Google BigQuery making your experience easier, faster and more powerful. In addition to new features and improvements like table wildcard functions, views, and parallel exports, BigQuery now features increased streaming capacity, lower pricing, and more.


1000x increase in streaming capacity

Last September we announced the ability to stream data into BigQuery for instant analysis, with an ingestion limit of 100 rows per second. While developers have enjoyed and exploited this capability, they've asked for more capacity. You now can stream up to 100,000 rows per second, per table into BigQuery - 1,000 times more than before.

For a great demonstration of the power of streaming data into BigQuery, check out the live demo from the keynote at Cloud Platform Live.

Users often partition their big tables into smaller units for data lifecycle and optimization purposes. For example, instead of having yearly tables, they could be split into monthly or even daily sets. BigQuery now offers table wildcard functions to help easily query tables that match common parameters.

The downside of partitioning tables is writing queries that need to access multiple tables. This would be easier if there was a way to tell BigQuery "process all the tables between March 3rd and March 25th" or "read every table which names start with an 'a'". You can do this with this release.

TABLE_DATE_RANGE() queries all tables that overlap with a time range (based on the table names), while TABLE_QUERY() accepts regular expressions to select the tables to analyze.

For more information, see the documentation and syntax for table wildcard functions.

Improved SQL support and table views

BigQuery has adopted SQL as its query language because it's one of the most well known, simple and powerful ways to analyze data. Nevertheless BigQuery used to impose some restrictions on traditional SQL-92, like having to write multiple sub-queries instead of simpler multi-joins. Not anymore, now BigQuery supports multi-join and CROSS JOIN, and improves its SQL capabilities with more flexible alias support, fewer ORDER BY restrictions, more window functions, smarter PARTITION BY, and more.

A notable new feature is the ability to save queries as views, and use them as building blocks for more complex queries. To define a view, you can use the browser tool to save a query, the API, or the newest version of the BigQuery command-line tool (by downloading the Google Cloud SDK).

User-defined metadata

Now you can annotate each dataset, table, and field with descriptions that are displayed within BigQuery. This way people you share your datasets with will have an easier time identifying them.

JSON parsing functions

BigQuery is optimized for structured data: before loading data into BigQuery, you should first define a table with the right columns. This is not always easy, as JSON schemas might be flexible and in constant flux. BigQuery now lets you store JSON encoded objects into string fields, and you can use the JSON_EXTRACT and JSON_EXTRACT_SCALAR functions to easily parse them later using JSONPath-like expressions.

For example:

SELECT json_extract_scalar(
   "{'book': { 
       'category':'fiction', 
       'title':'Harry Potter'}}", 
   "$.book.category");

Fast parallel exports

BigQuery is a great place to store all your data and have it ready for instant analysis using SQL queries. But sometimes SQL is not enough, and you might want to analyze your data with external tools. That's why we developed the new fast parallel exports: With this feature, you can define how many workers will be consuming the data, and BigQuery exports the data to multiple files optimized for the available number of workers.

Check the exporting data documentation, or stay tuned for the upcoming Hadoop connector to BigQuery documentation.

Massive price reductions

At Cloud Platform live, we announced a massive price reduction: Storage costs are going down 68%, from 8 cents per gigabyte per month to only 2.6, while querying costs are going down 85%, from 3.5 cents per gigabyte to only 0.5. Previously announced streaming costs are now reduced by 90%. And finally, we announced the ability to purchase reserved processing capacity, for even cheaper prices and the ability to precisely predict costs. And you always have the option to burst using on-demand capacity.

I want to take this space to celebrate the latest open source community contributions to the BigQuery ecosystem. R has its own connector to BigQuery (and a tutorial), as Python pandas too (check out the video we made with Pearson). Ruby developers are now able to use BigQuery with an ActiveRecord connector, and send all their logs with fluentd. Thanks all, and keep surprising us!

Felipe Hoffa is part of the Cloud Platform Team. He'd love to see the world's data accessible for everyone in BigQuery.

Posted by Louis Gray, Googler

By Navneet Joneja, Cloud Platform Team

Cross-posted from the Google Cloud Platform blog

Editor’s note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.

For many developers, building a cloud-native application begins with a fundamental decision. Are you going to build it on Infrastructure as a Service (IaaS), or Platform as a Service (PaaS)? Will you build large pieces of plumbing yourself so that you have complete flexibility and control, or will you cede control over the environment to get high productivity?

You shouldn’t have to choose between the openness, flexibility and control of IaaS, and the productivity and auto-management of PaaS. Describing solutions declaratively and taking advantage of intelligent management systems that understand and manage deployments leads to higher availability and quality of service. This frees engineers up to focus on writing code and significantly reduces the need to carry a pager.

Today, we’re introducing Managed Virtual Machines and Deployment Manager. These are our first steps towards enabling developers to have the best of both worlds. Managed Virtual Machines

At Google Cloud Platform Live we took the first step towards ending the PaaS/IaaS dichotomy by introducing Managed Virtual Machines. With Managed VMs, you can build your application (or components of it) using virtual machines running in Google Compute Engine while benefiting from the auto-management and services that Google App Engine provides. This allows you to easily use technology that isn’t built into one of our managed runtimes, whether that is a different programming language, native code, or direct access to the file system or network stack. Further, if you find you need to ssh into a VM in order to debug a particularly thorny issue, it’s easy to “break glass” and do just that.

Moving from an App Engine runtime to a managed VM can be as easy as adding one line to your app.yaml file:

vm:true

At Cloud Platform Live, we also demonstrated how the next stage in the evolution of Managed VMs will allow you to bring your own runtime to App Engine, so you won’t be limited to the runtimes we support out of the box.

Managed Virtual machines will soon launch in Limited Preview, and you can request access here starting today.

Introducing Google Cloud Deployment Manager

A key part of deploying software at scale is ensuring that configuration happens automatically from a single source of truth. This is because accumulated manual configuration often results in “snowflakes” - components that are unique and almost impossible to replicate - which in turn makes services harder to maintain, scale and troubleshoot.

These best practices are baked into the App Engine and Managed VM toolchains. Now, we’d like to make it easy for developers who are using unmanaged VMs to also take advantage of declarative configuration and foundational management capabilities like health-checking and auto-scaling. So, we’re launching Google Cloud Deployment Manager - a new service that allows you to create declarative deployments of Cloud Platform resources that can then be created, actively health monitored, and auto-scaled as needed.

Deployment Manager gives you a simple YAML syntax to create parameterizable templates that describe your Cloud Platform projects, including:

• The attributes of any Compute Engine virtual machines (e.g. instance type, network settings, persistent disk, VM metadata).
• Health checks and auto-scaling
• Startup scripts that can be used to launch applications or other configuration management software (like Puppet, Chef or SaltStack)

Templates can be re-used across multiple deployments. Over time, we expect to extend Deployment Manager to cover additional Cloud Platform resources.

Deployment manager enables you to think in terms of logical infrastructure, where you describe your service declaratively and let Google’s management systems deploy and manage their health on your behalf. Please see the Deployment Manager documentation to learn more and to sign up for the Limited Preview.

We believe that combining flexibility and openness with the ease and productivity of auto-management and a simple tool-chain is the foundation of the next-generation cloud. Managed VMs and Deployment Manager are the first steps we’re taking towards delivering that vision.

Update on operating systems

We introduced support for SuSE and Red Hat Enterprise Linux on Compute Engine in December. Today, SuSE is Generally Available, and we announced Open Preview for Red Hat Enterprise Linux last week. We’re also announcing the limited preview for Windows Server 2008 R2, and you can sign up for access now. Windows Server will be offered at $0.02 per hour for the f1-micro and g1-small instances and $0.04 per core per hour for all other instances (Windows prices are in addition to normal VM charges).

Simple, lower prices

As we mentioned on Tuesday, we think pricing should be simpler and more closely track cost reductions as a result of Moore’s law. So we’re making several changes to our pricing effective April 1, 2014.

First, we’ve cut virtual machine prices by up to 53%:

• We’ve dropped prices by 32% across the board.
• We’re introducing sustained-use discounts, which lower your effective price as your usage goes up. Discounts start when you use a VM for over 25% of the month and increase with usage. When you use a VM for an entire month, this amounts to an additional 30% discount.

What’s more, you don’t need to sign up for anything, make any financial commitments or pay any upfront fees. We automatically give you the best price for every VM you run, and you still only pay for the minutes that you use.

Here’s what that looks like for our standard 1-core (n1-standard-1) instance: Finally, we’ve drastically simplified pricing for App Engine, and lowered pricing for instance-hours by 37.5%, dedicated memcache by 50% and Datastore writes by 30%. In addition, many services, including SNI SSL and PageSpeed are now offered to all applications at no extra cost.

We hope you find these new capabilities useful, and look forward to hearing from you! If you haven’t yet done so, you can sign up for Google Cloud Platform here.

Navneet Joneja, Senior Product Manager

Posted by Louis Gray, Googler

By Cody Bratt, Google Cloud Platform team

Cross-posted from the Google Cloud Platform blog

Editor's note: This post is a follow-up to the announcements we made on March 25th at Google Cloud Platform Live.

Yesterday, we unveiled a new set of developer experiences for Google Cloud Platform that are inspired by the work we've done inside of Google to improve our developers productivity. We want to walk you through these experiences in more detail and how we think they can help you focus on developing your applications and growing your business.

Isolating production problems faster

Understanding how applications are running in production can be challenging and sometimes it's unavoidable that errors will make it into production. We've started adding new capabilities to Cloud Platform this week that make it easier to isolate and fix production problems in your application running on Google App Engine.

We are adding a 'Diff' link to the Releases page (shown below) which will bring you to a rollup of all the commits that happened between deployments and the source code changes they introduced. Invaluable when you are trying to isolate a production issue.

You can see here where Brad introduced an error into production.

But, looking at source changes can still be like looking for a needle in a haystack. This is why we're working to combine data from your application running in production and link it with your source code.
In the new and improved logs viewer, we aggregate logs from all your instances in a single view in near real time. Of course, with high traffic levels, just browsing is unlikely to be helpful. You can filter based on a number of different criteria including regular expressions and the time when you saw an issue. We've also improved the overall usability by letting you scroll continuously rather than viewing 20 logs per page at a time.

Ordinarily, debugging investigations from the logs viewer would require you to find the code, track down the right file and line and ensure it's the same version that was deployed in production. This can cause you to completely lose context. In order to address this, we've added links from stack traces to the associated code causing them for push-to-deploy users. In one click you are brought to the source viewer at the revision causing the problem with the associated line highlighted. Shortening your time to fix

You may have noticed the 'Edit' buttons in the source viewer. We're introducing this because we think that finding the error in production is only part of the effort required when things go awry. We continue to ask ourselves how we can make it even faster for you to fix problems. One of the ways we do this inside Google today is to make the fix immediately from the browser.

So, we're bringing this to you by making it possible to edit a file in place directly from the source viewer in the Console. You can simply click the 'Edit' button, make your changes to that file, and click the 'Commit' button. No local setup required. We also know that sometimes fixes aren't that simple and we're investigating ways we can make it easy to seamlessly open your files in a desktop editor or IDE directly from the Console.

Fix simple problems directly in the browser

Trigger your push-to-deploy setup instantly

Since we've integrated this with your push-to-deploy configuration, the commit triggers any build and test steps you have to ensure your code is fully vetted before it reached production. Further, since we've built this on top of Git, each change is fully attributed and traceable in your Git repository. This is not SSHing into a machine and hacking on code in production.

An integrated ecosystem of tools you know

Speaking of using other editors, we believe that you are most productive when you have access to all the tools you know and love. That's why we're committed to creating a well integrated set of experiences across those tools, rather than than asking you to switch. With the Git push-to-deploy feature we launched last year, we wanted to make it easy for you to deploy your application utilizing standard Git commands while giving you free private hosted Git repositories. In addition, we understand that many of you host your source code on GitHub and in fact so does the Google Cloud Platform team. As you can see from the new 'Releases' page we showed this week, we're introducing the ability to connect your GitHub repository to push-to-deploy. We will register a post-commit hook with GitHub to give you all the deployment benefits without moving your source code. Just push to the master branch of your GitHub repository and your code will be deployed!

Push-to-deploy now supports Java builds

Next, we're excited to introduce new release types for you to use with push-to-deploy. As you can see above, this project is set up to trigger a build, run all your unit tests, and if they all pass, deploy. Taking a peek under the covers, we utilize a standard Google Compute Engine virtual machine that you own running in your project to perform this build and test. In this case, Google has automatically provisioned the machine with Maven and Jenkins and everything you need to build and run your tests. Your build and tests can be as complex as they need to be and they can use all the resources available on that machine they need to run. What's more is that all the builds will be done on a clean machine ensuring reliable, repeatable builds on every release. We're starting with Maven-based Java builds, but working to release support for other languages, test frameworks and build systems in the future.

The release history showing build, test and deployment status

Tying this together, we've simplified getting started on your projects by introducing the new 'gcloud' command in the Google Cloud SDK. The Google Cloud SDK contains all the tools and libraries you need to create and manage resources on the Cloud Platform. We recently posted some Cloud SDK tips and tricks. Now, using the 'gcloud init' command, we make setting up a local copy of your project fast by finding your project's associated Git repository and downloading the source code, so all you have to do is change directories and open your favorite editor.
Once you are initialized, all you need to do is start writing code. After you're done, running 'git push' will push your changes back to the remote repository and trigger your push-to-deploy flow. Then, all your changes will also be available to view on the 'Source' page in the Console.

We'll be rolling out these features to you in stages over the coming weeks, but if you are interested in being a trusted tester for any of them, please contact us here. We're very excited to hear how you're using these tools and what features we can build on top of them to make your developer experience even more seamless and fast. We're just getting started.

Cody Bratt is a product manager on the Google Cloud Platform team

Posted by Louis Gray, Googler

By Urs Hölzle, Senior Vice President

Editors note: Tune in to Google Cloud Platform Live for more information about our announcements. And join us during our 27-city Google Cloud Platform Roadshow which kicks off in Paris on April 7.

Today, at Google Cloud Platform Live we're introducing the next set of improvements to Cloud Platform: lower and simpler pricing, cloud-based DevOps tooling, Managed Virtual Machines (VM) for App Engine, real-time Big Data analytics with Google BigQuery, and more.

Industry-leading, simplified pricing

The original promise of cloud computing was simple: virtualize hardware, pay only for what you use, with no upfront capital expenditures and lower prices than on-premise solutions. But pricing hasn't followed Moore's Law: over the past five years, hardware costs improved by 20-30% annually but public cloud prices fell at just 8% per year.

We think cloud pricing should track Moore's Law, so we're simplifying and reducing prices for our various on-demand, pay-as-you-go services by 30-85%:

• Compute Engine reduced by 32% across all sizes, regions, and classes.
• App Engine pricing simplified, with significant reductions in database operations and front-end compute instances.
• Cloud Storage is now priced at a consistent 2.6 cents per GB. That's roughly 68% less for most customers.
• Google BigQuery on-demand prices reduced by 85%.

Sustained-Use discounts

In addition to lower on-demand prices, you'll save even more money with Sustained-Use Discounts for steady-state workloads. Discounts start automatically when you use a VM for over 25% of the month. When you use a VM for an entire month, you save an additional 30% over the new on-demand prices, for a total reduction of 53% over our original prices. Sustained-Use Discounts automatically reward users who run VMs for over 25% of any calendar month

With our new pricing and sustained use discounts, you get the best performance at the lowest price in the industry. No upfront payments, no lock-in, and no need to predict future use.

Making developers more productive in the cloud

We're also introducing features that make development more productive:

• Build, test, and release in the cloud, with minimal setup or changes to your workflow. Simply commit a change with git and we'll run a clean build and all unit tests.
• Aggregated logs across all your instances, with filtering and search tools.
• Detailed stack traces for bugs, with one-click access to the exact version of the code that caused the issue. You can even make small code changes right in the browser.

We're working on even more features to ensure that our platform is the most productive place for developers. Stay tuned.

Introducing Managed Virtual Machines

You shouldn't have to choose between the flexibility of VMs and the auto-management and scaling provided by App Engine. Managed VMs let you run any binary inside a VM and turn it into a part of your App Engine app with just a few lines of code. App Engine will automatically manage these VMs for you.

Expanded Compute Engine operating system support

We now support Windows Server 2008 R2 on Compute Engine in limited preview and Red Hat Enterprise Linux and SUSE Linux Enterprise Server are now available to everyone.

Real-Time Big Data

BigQuery lets you run interactive SQL queries against datasets of any size in seconds using a fully managed service, with no setup and no configuration. Starting today, with BigQuery Streaming, you can ingest 100,000 records per second per table with near-instant updates, so you can analyze massive data streams in real time. Yet, BigQuery is very affordable: on-demand queries now only cost $5 per TB and 5 GB/sec reserved query capacity starts at $20,000/month, 75% lower than other providers.

Conclusion

This is an exciting time to be a developer and build apps for a global audience. Today we've focused a lot on productivity, making it easier to build and test in the cloud, using the tools you're already familiar with. Managed VMs give you the freedom to combine flexible VMs with the auto-management of App Engine. BigQuery allows big data analysis to just work, at any scale.

And on top of all of that, we're making it more affordable than it's ever been before, reintroducing Moore's Law to the cloud: the cost of virtualized hardware should fall in line with the cost of the underlying real hardware. And you automatically get discounts for sustained use with no long-term contracts, no lock-in, and no upfront costs, so you get the best price and the best performance without needing a PhD in Finance.

We've made a lot of progress this first quarter and you'll hear even more at Google I/O in June.

Posted by Louis Gray, Googler

By Dan Ciruli, Google Cloud Platform Team

We strive to make our APIs accessible to anyone on any platform: ReST, HTTP and JSON mean that from nearly any language on nearly any hardware, you can call any of our APIs. However, to be truly useful on many platforms, it helps to have a client library -- one that packs a lot of functionality like handling auth, streaming media uploads and downloads, and gives you native language idioms.

Today, we are announcing General Availability of the Google APIs Client Library for .NET.

This library is an open-source effort, hosted at NuGet, that lets developers building on the Microsoft® .NET Framework to integrate their desktop or Windows Phone applications with Google’s services. It handles OAuth 2.0 integration, streaming uploads and downloads of media, and batching requests. For more than fifty Google APIs, it is the easiest way to get access for any Windows developer. Whether you are plugging Google Calendar into your .NET Framework-based application, translating text in a Windows Phone app or writing a PowerShell script to start Google Compute Engine instances, the Google APIs Client Library for .NET can save you tons of time.

Want to try it out? Visit the Getting Started tutorial. Want to hear more about about using Google’s services from .NET? Follow the Google APIs Client Library for .NET blog here.

Dan Ciruli is a Product Manager in the Cloud Platform Team intent on making developers' lives easier.

Posted by Louis Gray, Googler

By Greg DeMichillie, Google Cloud Platform team

Cross-posted from the Google Cloud Platform blog

Google’s global Cloud Platform Developer Roadshow is coming to a city near you. As many of you know, on March 25, we will be making major product announcements at Google Cloud Platform Live. The Roadshow will kick off immediately following this event and will visit 26 cities around the world. If you’d like to attend, register here.

In the roadshow, we will be talking about new approaches to computing that enable you to move beyond traditional divisions of PaaS and IaaS. We will also show how we are creating a developer experience that enables you to work more efficiently as you build, test, and deploy your code.

This is a great opportunity to see behind the scenes of the world's biggest cloud and engage with the international Google Cloud Platform team.

The Roadshow will be visiting Europe, Asia, and North America. We hope you can join us.

Greg DeMichillie has spent his entire career working on developer platforms for web, mobile, and the cloud. He started as a software engineer before making the jump to Product Management. When not coding, he's an avid photographer and gadget geek.

Posted by Louis Gray, Googler

By Joe Faith, Product Manager

Cross-posted with the Official Google Enterprise Blog

You want your applications to be fast, even with millions of users. Anytime your user tries to retrieve information from the app or update settings, it should happen instantly. For the best performance, you need faster, larger databases - especially if you have a growing user base to serve.

Google App Engine is designed to scale. And now Google Cloud SQL—a MySQL database that lives in Google’s cloud—has new features to meet the demand for faster access to more data. With today’s updates, you can now work with bigger, faster MySQL databases in the cloud:

• More Storage: We’re increasing the available storage on Cloud SQL to 100GB – ten times more than what used to be available.
  
• Faster Reads: We’re increasing the maximum size of instances to 16GB RAM, a 4 times increase in the amount of data you can cache.
  
• Faster Writes: We’re adding functionality for optional asynchronous replication, which gives the write performance of a non-replicated database, but the availability of a replicated one.
  
• EU datacenter availability: Now you can choose to store your data and run your Cloud SQL database instance in either our US or EU data centers.
  
• Integration with Google Apps Script: We’re making it quick and easy for businesses using Google Apps to use Cloud SQL. Publish and share data with Google Sheets, add data to Google Sites pages or create simple Google Forms without worrying about hosting or configuring servers. 

Introducing a new trial offer 
Many of you have requested a trial offer to test out Cloud SQL. Today, we’re introducing a 6- month trial offer at no charge, effective until June 1, 2013. This will include one Cloud SQL instance with 0.5 GB of storage. Sign up now and get started on Cloud SQL at no cost.


Joe Faith is a Product Manager on the Google Cloud Team. In a previous life he was a researcher in machine learning, bioinformatics, and information visualization, and was founder of charity fundraising site Fundraising Skills.

Posted by Scott Knaster, Editor

By Aleem Mawani, Co-Founder of Streak

Cross-posted with the Google App Engine Blog

This guest post was written by Aleem Mawani, Co-Founder of Streak, a startup alum of Y Combinator, a Silicon Valley incubator. Streak is a CRM tool built into Gmail. In this post, Aleem shares his experience building and scaling their product using Google Cloud Platform.

Everyone relies on email to get work done – yet most people use separate applications from their email to help them with various business processes. Streak fixes this problem by letting you do sales, hiring, fundraising, bug tracking, product development, deal flow, project management and almost any other business process right inside Gmail. In this post, I want to illustrate how we have used Google Cloud Platform to build Streak quickly, scalably and with the ability to deeply analyze our data.

• App Engine to serve our app.

• App Engine Datastore to persist user data.

• Memcache to make operations fast.

• BigQuery to analyze our logs and power dashboards.

• App Engine Search API to let users sift through their data.

• Prediction API to machine learn over user data.

• Google Translate API to translate our app to over 40 languages.

Our core learning is that you should use the best tool for the job. No one technology will be able to solve all your data storage and access needs. Instead, for each type of functionality, you should use a different service. In our case, we aggressively mirror our data in all the services mentioned above. For example, although the source of truth for our user data is in the App Engine Datastore, we mirror that data in the App Engine Search API so that we can provide full text search, Gmail style, to our users. We also mirror that same data in BigQuery so that we can power internal dashboards.

System Architecture

App Engine - We use App Engine for Java primarily to serve our application to the browser and mobile clients in addition to serving our API. App Engine is the source of truth for all our data, so we aggressively cache using Memcache. We also use Objectify to simplify access to the Datastore, which I highly recommend.

Google Cloud Storage - We mirror all of our Datastore data as well as all our log data in Cloud Storage, which acts as a conduit to other Google cloud services. It lets us archive the data as well as push it to BigQuery and the Prediction API.

BigQuery - Pushing the data into BigQuery allows us to run non-realtime queries that can help generate useful business metrics and slice user data to better understand how our product is getting used. Not only can we run complex queries over our Datastore data but also over all of our log data. This is incredibly powerful for analyzing the request patterns to App Engine. We can answer questions like:

• Which requests cost us the most money?
• What is the average response time for every URL on our site over the last 3 days?

BigQuery helps us monitor error rates in our application. We process all of our log data with debug statements, as well as something called an “error type” for any request that fails. If it’s a known error, we'll log something sensible, and we log the exception type if we haven’t seen it before. This is beneficial because we built a dashboard that queries BigQuery for the most recent errors in the last hour grouped by error type. Whenever we do a release, we can monitor error rates in the application really easily.

A Streak dashboard powered by BigQuery showing current usage statistics

In order to move the data into Cloud Storage from the Datastore and LogService, we developed an open source library called Mache. It’s a drop-in library that can be configured to automatically push data into BigQuery via Cloud Storage. The data can come from the Datastore or from LogService and is very configurable - feel free to contribute and give us feedback on it!

Google Cloud Platform also makes our application better for our users. We take advantage of the App Engine Search API and again mirror our data there. Users can then query their Streak data using the familiar Gmail full text search syntax, for example, “before:yesterday name:Foo”. Since we also push our data to the Prediction API, we can help users throughout our app by making smart suggestions. In Streak, we train models based on which emails users have categorized into different projects. Then, when users get a new email, we can suggest the most likely box that the email belongs to.

One issue that arises is how to keep all these mirrored data sets in sync. It works differently for each service based on the architecture of the service. Here’s a simple breakdown:



Having these technologies easily available to us has been a huge help for Streak. It makes our products better and helps us understand our users. Streak’s user base grew 30% every week for 4 consecutive months after launch, and we couldn’t have scaled this easily without Google Cloud Platform. To read more details on why Cloud Platform makes sense for our business, check out our case study and our post on the Google Enterprise blog.


Aleem Mawani is the co-founder of Streak.com, a CRM tool built into Gmail. Previously, Aleem worked on Google Drive and various ads products at Google. He has a degree from the University of Waterloo in Software engineering and an MBA from Harvard University.

Posted by Scott Knaster, Editor

By Ryan Boyd, Developer Advocate for Cloud Data Services

Businesses and developers are using BigQuery to solve a wide variety of use cases – from optimizing advertising campaigns, to spotting inventory shortfalls, to understanding customer behavior. Accommodating these varied use cases requires BigQuery to be flexible, both for the developers integrating applications with the API and for the analysts running ad-hoc queries. Today we’ve made it more flexible by adding batch queries and a connector for Microsoft Excel.

Batch priority queries

BigQuery was designed for ad-hoc, iterative analytics on millions-to-billions of rows of data. When you’re diving into your data to gain insights, you want your queries to run in seconds rather than waiting minutes or hours. Sometimes our customers don’t need these fast responses when they’re running nightly jobs to update dashboards or reports, but want to use the same BigQuery technology and underlying datasets for these queries. We’ve now added batch pricing to accommodate these developers, allowing them to run their queries at a significantly lower cost.

Here’s how to set the priority to ‘batch’ when submitting a new query via the Google APIs Client Library for Java:

    Job job = new Job();
    JobConfiguration config = new JobConfiguration();
    JobConfigurationQuery queryConfig = new JobConfigurationQuery();
    config.setQuery(queryConfig);

    job.setConfiguration(config);
    queryConfig.setQuery(querySql);
    queryConfig.setPriority("BATCH");

    com.google.api.services.bigquery.Bigquery.Jobs.Insert insert =
     bigquery.jobs().insert(projectId, job);

Batch queries will execute between 30 minutes and 3 hours after they are submitted. See more information in our Developers Guide.

BigQuery Connector for Excel

Spreadsheets are a popular tool for analysts, executives and and developers to explore data. Last year we launched the ability for users of Google Spreadsheets to execute BigQuery queries using the Google Apps Script integration. Today, we’re launching the BigQuery Connector for Excel, which allows Microsoft Excel users to do the same with the ‘External Data’ functionality built into the product. Once the BigQuery results are in Excel, you can easily make pivot tables, create charts and integrate it with data from other sources. If you’re interested, you can try it right now!

Let us know what you think of these new features and what else you’d like to see in the roadmap by reaching out on Google+. We’ll also be holding office hours this Friday at 10 AM PDT on Google Developers Live to talk about these new features and answer any questions you have about BigQuery.

Microsoft and Excel are registered trademarks of Microsoft Corporation


Ryan Boyd is a Developer Advocate, focused on cloud data services. He's been at Google for 6 years and previously helped build out the Google Apps ISV ecosystem. He published his first book "Getting Started with OAuth 2.0" with O'Reilly.

Posted by Scott Knaster, Editor