Hey, it's HighScalability time:

(Earth sized solar flare, some more flair)

 

• Google I/O to world: Just try to keep up with us. You can't. But go ahead and try. Nah na na na nah...


• 17 billion: Google Cloud Messaging messages per day with 60ms latency; 1B page views: 500px; 121 billion:  edge graph using Titan; 4 billion hours: hours watched on Netflix per quarter; 4.5 trillion: BigTable transactions per month


• Quotable Quotes:
  • to3m: As with any time you make plans for the future, sometimes you get it wrong. Ars longa vita brevis, and all that.
  • Callaghan’s law: a given row can’t be modified more than once per RTT
  • Josh Haberman: I had an epiphany one day when I realized that the kernel is nothing but a library with an expensive calling convention.
  • fread2281: Insane speed calls for insane measures.
  • Luke Gorrie: hardware really wants to run fast and you only need to avoid getting in the way --  not too hard if you write the whole stack to match your application, but very hard if you depend on abstractions and misunderstand what's going on.
  • Francis Stephens: This exposes an important, and to me non-obvious, property of concurrency. That it's not the locking that's really hard, it's how to be sure that every piece of related data is included in the lock (or STM).
  • @jamesurquhart: "Complexity is a characteristic of the system, not of the parts in it." -Dekker
  • Colin Scott: out of all the datacenter links types, the average downtime was 0.3 days. This translates to roughly three and a half 9’s of reliability, an order of magnitude greater than WAN links.
  • @adocortes: GPU vs CPU 40x faster for image processing in clusters


• Really fast growth really does happen says someone somewhere: Dots game from Betaworks hits 100 million game plays in first 2 weeks. 


• If you love something you should set it free or lose everything. Fred Wilson observes: This is a classic case of the innovator's dilemma. RIM felt that letting BBM out in the open would make it easier for Blackberry users to leave. So they kept it proprietary. For way too long. Now they no longer have a dominant smartphone franchise or a dominant mobile messenger franchise.


• When Big Data ecosystems start merging it's not the end of the world, but building a different world: Amex to tap big data (TripAdvisor) to expose fake reviews.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Imagine that the government decided an intake of 2.500 calories per day should be the maximum for each person, regardless of age, gender, health, metabolism, dietary habits, etc. And imagine that the government also measured and enforced this every day, claiming it is “for your own health”, and handing out daily fines for each person who went over target. How would you feel about this practice?

Now imagine that the government decided that a speed of 130 km/hours should be the maximum for each driver, regardless of age, health, mental condition, road condition, traffic condition, weather condition, or the condition of their cars. And imagine that the government measured and enforced this, claiming it is “for your own safety”, and handing out fines to anyone who went over this “target”. How would you feel about that? Oh, wait… this is an actual practice in many countries!

I drove 14 hours from Bologna to Brussels yesterday, with a proper break every 2 hours, good nutrition, a healthy mind, a well-serviced car, and an excellent track record as a driver. During that trip I saw people not using their indicator lights when switching lanes, people overtaking others on the emergency lane, people using their mobile phones, and people driving vehicles that barely deserved the name “car”. And among those many thousands of drivers, I’m sure there were also some with mental problems, physical problems, mechanical problems, etc. However, the one who got picked out by the government was me. I got flashed twice because I drove “too fast”.

For every complex goal, there is a metric that is clear simple and wrong.

In organizations we see this all the time. Managers have a goal, such as faster time-to-market or higher productivity. But productivity is a very complex thing. It depends on motivation, creativity, innovation, collaboration, etc. And managers can’t measure all that stuff easily. So they reduce the metric to the simplest possible thing that can be measured with a computer: the number of hours people are physically at the office. And then they turn it into a target: the computer requires at least 8! Or else…

Measuring something meaningful is hard, so let’s measure something that is meaningless but easy.

Measuring real safety on the streets for everyone is next to impossible, so the government reduces it to the simplest possible metric that can be delegated to computers: speed.

I fear the day when governments find a way to have computers measure our daily intake of calories.

You’ve read the introduction books, you’ve keyed in the hello world examples, now it’s time to dive deep into solutions from experienced developers working in interesting domains.

Deep Dive

Like many of you I’m sure I’ve collected a bookcase full of books introducing an array of languages and technologies. But like you I crave material that is based on actual experience of developing real-world systems in interesting domains. The excellent Beautiful Architecture and Beautiful Code books present chapters from experienced developers who talk about real-world applications they’ve worked on. F# Deep Dives goes deep on simple functional solutions with F# applied to interesting domains like finance, machine learning and games.

F#

F# is a practical, functional-first language that lets you write simple code to solve complex problems. By focusing on F# we have been able to draw from a wealth of experienced developers working on practical problems in interesting domains. F# is also a very expressive language which means that each chapter can focus on showing solutions without having to sacrifice content because of excessive amounts of boilerplate. The solutions should be easy to follow for developers with or without F# experience and interesting whether you intend to solve problems with F# or another functional-first language like Scala or Clojure.

Deal of the day

If all that sounds interesting I think you’ll enjoy F# Deep Dives:

 

Work on the book started in December 2012, it’s currently in MEAP with Manning and today (17th May 2013) F# Deep Dives is Manning’s deal of the day - use code: dotd0517au

30 Days of Getting Results is a hard-core time management course.   

It’s a 30 Day Sprint with a lesson each day, but you can go at your own pace.   For example, I every now and then I scan through it in about 20 minutes to remind myself of the best time management skills to work on.

Some of you have let me know that you can’t get to the site.   I’m not sure why.  

Regardless, I have a free PDF version of 30 Days of Getting Results available.

It’s powerful stuff.   If you want to master time management, productivity, and work-life balance, this short-course will help you do that.

Time management and extreme productivity are a few of the things that I regularly mentor individuals, teams, and leaders on.

It’s 129 pages, and very easy to flip through.

Each lessons includes an exercise to make it real and drive it home.

If you download and go through it, please rate it on Good Reads.

Enjoy.

Posted by Greg Hartrell, Angana Ghosh, Francesco Nerieri, Francis Ma, and the Google Play services team

Some of the most exciting Android announcements at Google I/O this year are part of our latest Google Play services release, version 3.1.

The new version brings you Google Play games services, part of a new cloud-integrated platform for social gaming based on Google+ identity. Also included are location-based services that make it easier to build efficient location-aware apps. For apps using the popular Google Cloud Messaging platform, you can now take advantage of XMPP messaging and easier setup. Finally, Cross-Platform Single Sign On for Google+ Sign-In is now available to your apps.

You can get started using these APIs and services right away—Google Play services 3.1 is already rolling out to Android devices across the world, with support reaching all the way back to Froyo.

Google Play games services

Games are always popular with Android developers, and the announcement of Google Play game services raised the volume even more.

Google Play games services lets you make your games more social, with achievements, leaderboards, and multiplayer, and they help you extend your user’s games across multiple devices by storing game saves and settings in the cloud.

Several great Android games are already using these new game services, including World of Goo, Super Stickman Golf 2, Beach Buggy Blitz, Kingdom Rush, Eternity Warriors 2, and Osmos.

You can take advantage of the new services right away using the games services SDK included in Google Play services. For all the details, check out the Google Play games services documentation.

Location APIs

If you build location-aware Android apps, you’ll want to check out the new location APIs. They make it easy to build accurate, fast, and efficient apps, with new contextual features.

The Fused Location Provider intelligently manages the underlying location technology and gives you the best location according to your needs. We’ve simplified the location APIs and completely rewritten our location algorithm to make location more accurate, flexible and use less battery.

Using the new geofencing API, your app can set up geographic boundaries around specific locations and then receive notifications when the user enters or leaves those areas.

With apps becoming increasingly contextual, understanding what the user is doing is critical to surfacing the right content. A new activity recognition API makes it easy to check the the user’s current activity — still, walking, cycling, and in-vehicle — with very efficient use of the battery. We use low-power sensors and machine-learning classifiers to recognize the activity, giving you both both high accuracy and low battery usage.

To learn more, head over to our training classes at Making Your App Location Aware or dive directly into the reference docs.

Google Cloud Messaging

We’ve added APIs to make it easier to set up GCM in your apps, and in the service itself we’ve added new messaging capabilities for your apps to use.

A new registration API lets your app register with the service using a single method call and begin receiving messages as soon as the call returns.

Sign Up for Early Access

If you’d like to try out CCS messaging or the User Notifications API, please sign up for early access.

In the GCM service itself we’ve added support for messaging over XMPP with the new GCM Cloud Connection Server (CCS). Your servers now have a persistent connection over which to send large numbers of messages, very quickly, and with no overhead. New APIs in Google Play services let apps send messages back upstream to third-party servers using CCS, without needing to manage network connections. This helps keep battery and data usage to a minimum.

Also new in the GCM service is a User Notifications API. This new API lets you synchronize notifications across a user’s multiple devices — when the user dismisses a notification on one device, the notification disappears automatically from all the other devices. To get started with GCM, head over to the developer documentation.

Google+ Cross-Platform Single Sign On

Many people use apps on multiple devices throughout the day, switching between their laptops, tablets, and mobile devices. After signing-in to an app on one device, it’s natural that when they pick up a different device and use the same app, they would expect to be signed in there as well.

To help you provide this kind of seamless transition between platforms and stay connected with users across devices, we’re adding Cross-Platform Single Sign On to our Google+ Sign-In capabilities.

If your app is already using Google+ Sign-In, you’ve already got support for Cross-Platform Single Sign On. This feature will be enabled automatically over the coming days.

Cross-Platform Single Sign On gives you a great way to build longer-running, cross-platform user experiences, and it dovetails perfectly with the new Google Play games services for bridging game state across devices using the cloud.

To learn more about Google+ Sign-In, check out http://developers.google.com/+.

More About Google Play Services

Google Play Services is our platform for offering you better integration with Google products, and providing new capabilities to use within your apps. To learn more about Google Play services and the APIs available to you through it, visit the Google Services area of the Android Developers site.

Join the discussion on

+Android Developers

Quite a few interesting sessions today:

Google Knowledge Graph

With the Knowledge Graph, Google is trying to give a new dimension to its flagship product: Search.
Users should be able to have a conversation with search, and search should anticipate on a user’s interests by showing related topics.
So search needs to understand the context of searches, in other words: it needs to understand the world. This is where Knowledge Tree comes in handy.
Knowledge Tree is a hugh graph with semantic data about real world entities and their relations. This data has been gathered from Wikipedia, Google+ and many other sources.
The complete tree is viewable and downloadable at freebase.com.

Freebase Api

If you don’t feel like writing your query logic by yourslef, Google also offers a (free) Rest api:
Freebase Api
The following services are offered, many of them have their own JQuery plugin as well:

• autocomplete
• semantic tagging
• entity collections
• geosearch collections
• topical weblinks

Dart: new features

The Dart programming language has been around a few years already. It is Google’s scalable and typesafe alternative for Javascript. It can run in its own VM, in a special version of Chrome, or it can be compiled into Javascript.

During the last year quite a couple of new features have been added. Here are some highlights:

Mixins
Metadata
Lazy loading of libraries
Futures
streams

Also nice: you can try out Dart in realtime on try.dartlang.org

Dart’s performance has been enhanced: native Dart now runs many times faster than Javascript, thanks to, amongst others, utilization of SimD instructions by the Dart VM.

“Chrome DevTools Revolution”

The DevTools in Chrome Canary have been updated with quite a few cool features. It should even be possible now to use Chrome DevTools as your only IDE: inline editing of Javascript and styling was already possible, but now it is also possible to save the edited code to disk (using Ctrl-S).

If you are using an IDE and have edited a source file, Chrome can now detect file system changes and refresh automatically.

Sass support has been added to Chrome DevTools. So you can trace a styling all the way to the Sass reference that defines it, inside Chrome Devtools.
And also here it is possible to automatically reload the page when a sass file has been changed.

A lot of support was added for profiling Javascript:

• Timeline: a view to quickly see which method takes the most cpu cycles
• FlameChart: a view with the same purpose but a different layout
• Canvast profiling: a view showing which paint actions take the most time
• Object allocation tracking: a memory profiler for javascript
• Layout trashing details: a tool that can show inefficiencies or redundant Dom tree building
• and more ..

Portable Native Client

This is Google’s solution to let you run C++ code inside your browser. Actually that was already possible with Native Client, but Portable Native Client makes it easier to create C++ apps that will run on both Arm and x86 architectures.

Obviously Google needed to do a lot to prevent this from being a security risk. So the C++ code needs to be compiled with a custom compiler, which will create a .pexe file; this file runs inside a ‘native client process’ with ‘software fault isolation’. What this means is that Chrome will scan the code before running it, to check for illegal instructions and for illegal access of certain memory regions.
Apart from that, native client code can only be used in applications inside Chrome Webstore.
This all should make Native Clients safe to use.

The advantage of Native Clients is obviously their performance: they run at 80% or 90% of the speed of a fully native application.

But the question is of course: why go through so much trouble, why not just download a fully native application to your cellphone or laptop?
I guess the answer is that such a thing is not possible on Chromebooks. Google is trying hard to make Chrome the universal platform to run all your applications. And with Native Client, you can now get enough performance to run games as well.

In my time playing around with neo4j I’ve run into a problem a few times where I executed a query using the web console (usually accessible @ http://localhost:7474/webadmin/#/console/) and have got absolutely no response.

I noticed a similar thing today when Rickard and I were having a look at why a Lucene index query wasn’t behaving as we expected.

I setup some data in a neo4j database using neography with the following code:

require 'neography'
 
@neo = Neography::Rest.new
 
@neo.create_node_index("Id_Index", "exact", "lucene")
 
node1 = @neo.create_node("Hour" => 1, "name" => "Max")
node2 = @neo.create_node("Hour" => 2, "name" => "Mark")
node3 = @neo.create_node("Hour" => 3, "name" => "Rickard")
 
@neo.add_node_to_index("Id_Index", "Hour", 1, node1)
@neo.add_node_to_index("Id_Index", "Hour", 2, node2) 
@neo.add_node_to_index("Id_Index", "Hour", 3, node3)

I then ran the following query which I was expecting to return all the nodes:

start hour=node:Id_Index("Hour:[00 TO 02] or Hour:[03 TO 05]") RETURN hour

Instead it returned nothing and I couldn’t see anything being logged either.

Rickard pointed out was because the exception is only returned to the API caller and that it would be better to run the query from the Data Browser which is typically accessible from http://localhost:7474/webadmin/#/data/search/

If we run the query from there then we can see what’s going wrong:

BadInputException
 
StackTrace:
org.neo4j.server.rest.repr.RepresentationExceptionHandlingIterable.exceptionOnHasNext(RepresentationExceptionHandlingIterable.java:50)
org.neo4j.helpers.collection.ExceptionHandlingIterable$1.hasNext(ExceptionHandlingIterable.java:60)
org.neo4j.helpers.collection.IteratorWrapper.hasNext(IteratorWrapper.java:42)
org.neo4j.server.rest.repr.ListRepresentation.serialize(ListRepresentation.java:58)
org.neo4j.server.rest.repr.Serializer.serialize(Serializer.java:75)
org.neo4j.server.rest.repr.MappingSerializer.putList(MappingSerializer.java:61)
org.neo4j.server.rest.repr.CypherResultRepresentation.serialize(CypherResultRepresentation.java:57)
org.neo4j.server.rest.repr.MappingRepresentation.serialize(MappingRepresentation.java:42)
org.neo4j.server.rest.repr.OutputFormat.assemble(OutputFormat.java:179)
org.neo4j.server.rest.repr.OutputFormat.formatRepresentation(OutputFormat.java:131)
org.neo4j.server.rest.repr.OutputFormat.response(OutputFormat.java:117)
org.neo4j.server.rest.repr.OutputFormat.ok(OutputFormat.java:55)
org.neo4j.server.rest.web.CypherService.cypher(CypherService.java:94)
java.lang.reflect.Method.invoke(Method.java:597)

There seemed to be some strangeness going on with how Lucene handles the query when a default search field isn’t provided but we noticed that it behaved as expected if we didn’t use an OR since Lucene has an implicit OR between statements anyway.

start hour=node:Id_Index("Hour:[00 TO 02] Hour:[03 TO 05]") RETURN hour

Either way, the lesson for me was if the console isn’t giving a result run the query in the data browser to work out what’s going wrong!

Agile practices draw on the team committing to the assigned work on many levels for example the commitments from the sprint planning or the commitments made during the daily stand-up. Without the team continually committing to the work Agile would just be iterative waterfall. Teams can be defined as a group people that have developed relationships based around a commitment to attain a set of common goals. The commitment of team members to work together is a critical factor in formation of teams and their ultimate success.

Team commitment is impacted by many factors.

• Mission: Knowing that the work supports or fits into the organization’s overall strategy provides a goal for the team to pursue.
• Value: Knowing that the team is valued provides an anchor that supports individuals working together as a group.
• Challenge: Providing a goal and trusting the team to rise to the situation.
• Empowerment: Giving the team the authority to solve the business problem defined in the project.

Knitting mission, value, challenge and empowerment together creates an environment where a group can organize around a mission, develop a solution and commit to achieving that goal because they know that their performance will be valued. The environment that supports commitment is a core component for high performance teams.

By Andrew Jessup, Product Manager

Cross-posted from the Google Cloud Platform Blog

At Google I/O, we announced PHP as the latest supported runtime for Google App Engine in Limited Preview. PHP is one of the world's most popular programming languages, used by developers to power everything from simple web forms to complex enterprise applications.

Now PHP developers can take advantage of the scale, reliability and security features of App Engine. In addition, PHP runs well with other parts of Google Cloud Platform. Let's look at how this works.

Connecting to Google Cloud SQL from App Engine for PHP

Many PHP developers start with MySQL when choosing a database to store critical information, and a wide variety of products and frameworks such as WordPress make extensive use of MySQL’s rich feature set. Google Cloud SQL provides a reliable, managed database service that is MySQL 5.5 compatible and works well with App Engine.

To set up a Cloud SQL database, sign into Google Cloud Console - create a new project, choose Cloud SQL and create a new instance.



After you create the instance, it's automatically associated with your App Engine app.


You will notice Cloud SQL instances don’t need an IP address. Instead they can be accessed via a compound identifier made up of their project name and instance name, such as hello-php-gae:my-cloudsql-instance.

From within PHP, you can access Cloud SQL directly using the standard PHP MySQL libraries - mysql, mysqli or PDO_MySQL. Just specify your Cloud SQL database with its identifier, such as:

<?php

$db = new PDO(
  'mysql:unix_socket=/cloudsql/hello-php-gae:my-cloudsql-instance;dbname=demo_db;charset=utf8',
  'demo_user',
  'demo_password'
);

foreach($db->query('SELECT * FROM users') as $row) {
  echo $row['username'].' '.$row['first_name']; //etc...
}

Methods such as query() work just as you’d expect with any MySQL database. This example uses the popular PDO library, although other libraries such as mysql and mysqli work just as well.

Storing files with PHP and Google Cloud Storage

Reading and writing files is a common task in many PHP projects, whether you are reading stored application state, or generating formatted output (e.g., writing PDF files). The challenge is to find a storage system that is as scalable and secure as Google App Engine itself. Fortunately, we have exactly this in Google Cloud Storage (GCS).

The first step in setting up Google Cloud Storage is to create a bucket:


With the PHP runtime, we’ve implemented native support for GCS. In particular, we’ve made it possible for PHP’s native filesystem functions to read and write to a GCS bucket.

This code writes all prime numbers less than 2000 into a file on GCS:

<?php

$handle = fopen('gs://hello-php-gae-files/prime_numbers.txt','w');

fwrite($handle, "2");
for($i = 3; $i <= 2000; $i = $i + 2) {
  $j = 2;
  while($i % $j != 0) {
    if($j > sqrt($i)) {
      fwrite($handle, ", ".$i);
      break;
    }
    $j++;
  }
}

fclose($handle);

The same fopen() and fwrite() commands are used just as if you were writing to a local file. The difference is we’ve specified a Google Cloud Storage URL instead of a local filepath.

And this code reads the same file back into memory and pulls out the 100th prime number, using file_get_contents():

<?php

$primes = explode(",",
  file_get_contents('gs://hello-php-gae-files/prime_numbers.txt')
);

if(isset($primes[100]))
  echo "The 100th prime number is ".$primes[100];

And more features supported in PHP

Many of our most popular App Engine APIs are now supported in PHP, including our zero-configuration Memcache, Task Queues for asynchronous processing, Users API, Mail API and more. The standard features you’d expect from App Engine, including SSL support, Page Speed Service, versioning and traffic splitting are all available as well.

Open today in Limited Preview

Today we’re making App Engine for PHP available in Limited Preview. Read more about the runtime in our online documentation, download an early developer SDK, and sign up to deploy applications at https://cloud.google.com/appengine/php.


Andrew Jessup is a Product Manager at Google, working on languages and runtimes for Google App Engine.

Posted by Scott Knaster, Editor

To do great things, it helps to study people that do great things and show us better ways to do things.  It helps us build our reference library of what’s possible and it helps inspire us to new levels of success.

Most importantly, it expands our capabilities.

Chalene Johnson is a powerhouse when it comes to personal development.   She continuously pushes herself, while expanding and exploring what’s possible physically, mentally. and emotionally.   She’s a unique blend of entrepreneur, physical fitness expert, choreographer, author, life changer, and motivational speaker … and we can learn a lot from her approach.

I wrote up 27 lessons from Chalene Johson, but my favorite lesson is actually Lesson #7 – Success isn’t magic, it’s a method:

Chalene says, “It’s NOT luck — it’s KNOW HOW. There is a formula for everything.”   You have to study the people that have the results that you want.   Learn from their formula.   Study what made them successful.  If you can find the proven practices and the methods that work, you’ll speed up your success, and you’ll avoid the dead-ends.   Finding a formula helps you establish and practices routines that will help you get better and better over time.

Personally, I’ve found this to be true time and time again.  Whenever I got stuck, it was my strategy or approach.  I just didn’t know the right formula or who to model from.  There’s always a recipe.  One of the most important things I learned on the Microsoft patterns & practices team is that if you look to the right sources, you’ll find the proven practices or the patterns that really work, even if it’s not well-known (in fact, part of our job on the Microsoft patterns & practices team was really to share and scale this knowledge more broadly.)

I’ve shared my personal rapid results formula before in The Way of Success, and it helps elaborate on how to model success in a more effective way.  As Tony Robbins says, success leaves clues.  We just need to be good students of possibility to find them and apply them.

Even if you’re not into working out, I think you'll enjoy lessons from Chalene Johnson on personal development, productivity, motivation, and more.

By Chris Ramsdale, Product Manager

Cross-posted from the Google Cloud Platform Blog

At Google I/O, we announced Google Cloud Datastore, a fully managed solution for storing non-relational data. Based on the popular Google App Engine High Replication Datastore (HRD), Cloud Datastore provides a schemaless, non-relational datastore with the same accessibility of Google Cloud Storage and Google Cloud SQL.

Cloud Datastore builds off the strong growth and performance of HRD, which has over 1PB of data stored, 4.5 trillion transactions per month and a 99.95% uptime. It also comes with the following features:

• Built-in query support: near SQL functionality that allows you to search, sort and filter across multiple indexes that are automatically maintained 
• ACID transactions: data consistency (both Strong and Eventual) that spans multiple replicas and requests 
• Automatic scaling: built on top of Google’s BigTable infrastructure, the Cloud Datastore will automatically scale with your data 
• High availability: by utilizing Google’s underlying Megastore service, the Cloud Datastore ensures that data is replicated across multiple datacenters and is highly available 
• Local development environment: the Cloud Datastore SDK provides a full-featured local environment that allows you to develop, iterate and manage your Cloud Datastore instances efficiently 
• Free to get started: 50k read & write operations, 200 indexes, and 1GB of stored data for free per month  

Getting started with Cloud Datastore 

To get started, head over to the Google Cloud Console and create a new project. After supplying a few pieces of information you will have a Cloud Project that has the Cloud Datastore enabled by default. For this post we’ll use the project ID cloud-demo.

With the project created and the Cloud Datastore enabled, we’ll need to download the Cloud Datastore client library. Once installed, it’s time to start writing some code. For the sake of this post, we’ll focus on accessing the Cloud Datastore from a Python application running on a Compute Engine VM (which is also now in Preview). We’ll assume that you’ve already created a new VM instance.'

import googledatastore as datastore

def main()
  writeEntity()
  readEntity()

Next include writeEntity() and readEntity() functions:

def WriteEntity():
  req = datastore.BlindWriteRequest()
  entity = req.mutation.upsert.add()
  path = entity.key.path_element.add()
  path.kind = 'Greeting'
  path.name = 'foo'
  message = entity.property.add()
  message.name = 'message'
  value = message.value.add()
  value.string_value = 'to the cloud and beyond!'
  try:
    datastore.blind_write(req)
  except datastore.RPCError as e:
    # remember to do something useful with the exception pass

def ReadEntity(): 
  req = datastore.LookupRequest()
  key = req.key.add()
  path = key.path_element.add()
  path.kind = 'Greeting0'
  path.name = 'foo0'
  try:
    resp = datastore.lookup(req)
    return resp
  except datastore.RPCError as e:
    # remember to do something useful with the exception pass

First create a new file called “demo.py”. Inside demo.py, we’ll add code to write and then read an entity from the Cloud Datastore.  Finally we can update main() to print out the property values within the fetched entity:

def main()
  writeEntity();
  resp = readEntity();

  entity = resp.found[0].entity
  for p in entity.property:
    print 'Entity property name: %s', p.name
    v = p.value[0]
    print 'Entity property value: %s', v.string_value

Before we can run this code we need to tell the SDK which Cloud Datastore instance we would like to use. This is done by exporting the following environment variable:

~$ export DATASTORE_DATASET cloud-datastore-demo

Finally we’re able to run the application by simply issuing the following:

~$ python demo.py

Besides the output that we see in console window, we’re also able to monitor our interactions within the Cloud Console. By navigating back to Cloud Console, selecting our cloud-datastore-demo project, and then selecting the Cloud Datastore we’re taken to our instance’s dashboard page that includes number of entities, properties, and property types, as well as index management, ad-hoc query support and breakdown of stored data.

And that’s really just the beginning. To fully harness the features and functionality that the Cloud Datastore offers, be sure to check out the larger Getting Started Guide and the Cloud Datastore documentation.

Cloud Datastore is the latest addition to the Cloud Platform storage family, joining Cloud Storage for storing blob data, Cloud SQL for storing relational data, and Persistent Disk for storing block data. All fully managed so that you can focus on creating amazing solutions and leave the rest to us.

And while this is a Preview Release, the team is off to a great start. As we move the service towards General Availability we’re looking forward to improving JSON support, more deeply integrating with the Cloud Console, streamlining our billing and driving every bit of performance that we can out of the API and underlying service.

Happy coding!


Chris Ramsdale has worked extensively in the mobile space, starting as a Software Engineer at Motorola in 1997, and then joining local start ups as a Tech Lead and Product Manager. Chris is currently a Product Manager for Google Cloud Platform focused on developer tools and platform services like Google App Engine and Google Cloud Datastore.

Posted by Scott Knaster, Editor

Looks like an interesting take on "a completely asynchronous, low-latency transaction management protocol, in line with the fully distributed NoSQL architecture."

Warp: Multi-Key Transactions for Key-Value Stores overview:

Implementing ACID transactions has been a longstanding challenge for NoSQL systems. Because these systems are based on a sharded architecture, transactions necessarily require coordination across multiple servers. Past work in this space has relied either on heavyweight protocols such as Paxos or clock synchronization for this coordination.

This paper presents a novel protocol for coordinating distributed transactions with ACID semantics on top of a sharded data store. Called linear transactions, this protocol achieves scalability by distributing the coordination task to only those servers that hold relevant data for each transaction. It achieves high performance by serializing only those transactions whose concurrent execution could potentially yield a violation of ACID semantics. Finally, it naturally integrates chain-replication and can thus tolerate faults of both clients and servers. We have fully implemented linear transactions in a commercially available data store. Experiments show that the throughput of this system achieves 1-9× more throughput than MongoDB, Cassandra and HyperDex on the Yahoo! Cloud Serving Benchmark, even though none of the latter systems provide transactional guarantees.

Note: Engine Yard friend Daragh Curran, Head of Product Engineering, Intercom has graciously let us post this great piece about code deployment on our blog. Check it out on their own blog here.

Software only becomes valuable when you ship it to customers. Before then it's just a costly accumulation of hard work and assumptions.

Shipping unlocks a feedback loop that confirms or challenges those assumptions. It makes new things possible for your customers, and gives you the opportunity to focus on the next thing.

Shipping brings life to your team, to your product, and to your customers. Shipping is your company's heartbeat.

Shipping will try to kill you

The scramble to get that one last feature done, the late nights, the compromises, the sinking feeling when we realise something major is broken, the post-mortems… It's agony, but if it was easy everyone would do it. Shipping exposes mistakes. We're nervous about it, and our natural reaction is to do it reluctantly and infrequently, which actually carries higher risk, causing more reluctance in the future.

The cost of shipping is approaching zero

Not too long ago, shipping software involved actual ships, disks, and printed manuals. It happened perhaps once a year. Bug fixes weren't automatic over the internet like today. Everything was slower and more controlled. The cost of shipping was massive, the consequence of a mistake was large. Today, the cost of shipping has approached zero. Most people can deploy in seconds or minutes with a single command or button click. With a little thought you can do that without your customers noticing, and with automated monitoring you'll find out immediately if something goes wrong.

Despite the cost of shipping approaching zero, many people still ship software guided by very old habits.

Shipping cadence defines your company

The cadence at which you ship defines your company. A yearly cadence results in a very structured approach to the design->build->test cycle. A few months of building, while the rest is spend fixing. Engineers can join and leave before seeing their hard work end up in the hands of customers. The approach to design becomes one of anticipating all possible needs, rather than focusing and iterating on the important ones.

Obstacles downstream propagate upstream

An obstacle downstream propagates upstream. If you're not allowed to implement new ideas, you stop having them.
- Paul Graham

The right approach to shipping has a positive influence on your company's productivity and your team's happiness & job satisfaction. Shipping infrequently is an obstacle. Ship slow, and you'll introduce challenges that push you to ship even slower. Ship frequently, and see positive effects everywhere in your company. For example, lets examine how behaviour changes along with shipping frequency, while handling a simple request from a customer.

Lets say a customer gets in touch to say "No matter what I do, I cannot save my name correctly, I think it doesn't like hyphens". In a company where you ship continuously, you see this and think Simple — I'll tweak a test and a regex pattern, get a quick code review from my buddy beside me, merge to mainline, and 1 minute later when it's deployed to production, reply to the customer: "Sorry about this, it's fixed now, thanks for letting us know". They'll reply: "Wow, thanks for fixing so quickly". High fives all around!

If we stretch the time to production (TTP) out a little, even to 10 minutes, the behaviour changes. You either do the same, but reply saying it'll be fixed with our next deploy (probably 10 minutes) - or you wait, so that you can communicate with certainty. The waiting is time where you'll shift focus to something else, but have the baggage of having to follow up. Perhaps you'll think, I'll have a quick coffee, then move on to something else afterwards. Even though your deployments are entirely automated, you lose time because of waiting and losing focus.

If TTP is hours, the behaviour changes again. No longer can you say with certainty when the change will be out there, so you're tempted to batch up with other similar small changes. You postpone replying until you get time to do it, sometimes forgetting about it. You're less likely to take prompt action, wow'ing the customer, and you pay some mental cost for having it on a todo list. Since getting to production takes hours now, your team will start restricting to morning only deploys, so miss that slot and it's further delays.

If TTP is days, it exacerbates that further - perhaps you'll reply "Thanks for letting us know. We'll fix this in our next sprint". It gets bundled in with a whole load of other small low, priority items, you spend more time debating estimates, and priorities, than the first guy took to fix it and reply to the customer. Miss the beginning of week deploy window and further slippage. The larger releases bring higher risk, you'll tell your customer it's fixed, only to later require rolling back because of a separate change. Your bug database gets bigger and bigger, with little details that you'll probably never fix.

When TTP is weeks, it exaggerates that even further - perhaps you'll reply "Sorry about this, I'll let the development team know" or something equally lame from your customer’s standpoint. Deep down you realise nothing will be fixed, and the job of talking to customers becomes a cost or hassle, rather than an opportunity to improve your product and nurture happy loyal customers.

Shipping continuously

Better approaches to writing or testing software help us iterate more quickly and confidently, but the benefits are quite local to engineering teams. Continuous shipping on the other hand, touches all parts of your company, as do the benefits, and the behaviours it enables and encourages.

Linkedin's transition to continuous deployment is linked to their recent financial success.

Good products, are a side effect of combining good people with an idea in an environment that helps those people to kick ass. Your attitude to shipping is a big part of that environment you create.

Shipping breathes life into how we think. The feedback loop helps us learn, gain confidence in making quick decisions, and build momentum. Momentum in product improvements excites and engages our customers. Seeing quickly the benefits of our hard work, motivates us to do more. Building a team where people can work hard and move fast attracts others to join you - hiring gets easier.

Shipping continuously isn't an achievement you unlock and then move on. You've got to constantly obsess about it. If you believe in the benefits it brings, you'll be driven to shrink 20 minutes down to 1 minute or less, you'll consider 'ability to ship' as an equal to 'does it scale' when building new systems. And you'll do that because of all the life it breathes into your company and your product.

Shipping is your company's heartbeat.

By Navneet Joneja, Product Manager

Cross-posted from the Google Cloud Platform Blog

Last year we announced Google Compute Engine to enable any business or developer to use Google’s infrastructure for their applications. Now we’re taking the next step: Google Compute Engine is open to everyone in preview, and you can sign up online now.

Over the past year, we’ve launched several features and made significant improvements behind the scenes. We’re now announcing several new capabilities that make it easier and more economical to use Compute Engine for a broader set of applications.

• Sub-Hour Billing: We heard feedback from our early users who wanted more granular billing increments so they could run short-lived workloads. Now all instances are charged for in one-minute increments with a ten-minute minimum, so you don’t pay for compute minutes that you don’t use.
• New shared-core instance types: Compute Engine’s new micro and small instance types are designed as a cost-effective option for running small workloads that don’t need a lot of CPU power, like development and test workloads.
• Larger Persistent Disks: We’re increasing the size of Persistent Disks that can be attached to instances by up to 8,000%. You can now attach up to 10 terabytes of persistent disk to a Compute Engine virtual machine, giving you plenty of persistent storage for a wide variety of applications.
• Advanced Routing Capabilities: Compute Engine now supports software-defined routing capabilities based on our broad SDN innovation. These capabilities are designed to handle your advanced network routing needs like configuring instances to function as gateways, configuring VPN servers and building applications that span your local network and Google’s cloud.
• ISO 27001 Certification: We’ve also completed ISO 27001:2005 certification for Compute Engine, App Engine, and Cloud Storage to demonstrate that these products meet the international standard for managing information security.

To get started, go to the Google Cloud Console, select Compute Engine and click the “New Instance” button.

Fill out the required information and click “Create” on the right hand side. Your new virtual machine will be ready to use in about a minute.

To all of our customers who helped us evolve the product over the past months, thank you; your feedback has helped shape Compute Engine. To those of you who have been eager to try Compute Engine, the wait is over and you can sign up for Compute Engine online today.


Navneet Joneja loves being at the forefront of the next generation of simple and reliable software infrastructure, the foundation on which next-generation technology is being built. When not working, he can usually be found dreaming up new ways to entertain his intensely curious three-year-old.

Posted by Scott Knaster, Editor

Today was the first day of the Google IO conference in San Francisco. I was lucky enough to be able to attend so I can share some (personal) highlights of the day.

In the keynote Google showed some of their latest developments:
It is now possible to upload a batch of pictures to your online storage and let Google decide which are the best ones to keep. For instance when you’ve been on holiday and have taken 600 pictures, too many to show to anybody, Google can propose a selection after doing some intelligent analysis:
It can filter pictures if they are dupicates or if they are unclear. But it also recognizes if pictures contain some of your (Google+) relations, which would make them more interesting. Even more cool: with the help of some machine learning and a lot of training, Google can now estimate if a picture is pretty or not.
Another cool feature: Google will recognize if you have multiple pictures of the same person in the same scene. Based on some intrapolation algorithm, it can generate new pictures in that scene. Some examples were shown and they were actually really good.

Voice recognition was added Chrome (desktop and Chromebook only) so you can give spoken commands to let it do searches. Not really new, but what makes it special is that integration was added with Google Knowledge Tree, a giant graph database which contains semantic relations between entities.
What this means is that Chrome might understand what you mean when you say something like: “Send Misja an e-mail with the shortest route from here to Amsterdam”. This does require that Misja is one of your Google+ contacts. Chrome will infer that ‘here’ refers to your current location.

And there were many more announcements. You can see the full keynote speech here.

In the afternoon there was a choice of about 60 different presentations. Obviously too many to watch or to describe. Here are a few brief highlights:

Google has added App Script support to Google Forms.
App Script support was already present in Gmail and Google Sheets and it enables you to write macro’s or other logic in Javascript. The Javascript can access and manipulate the document you are working on, it can use all Google Api’s and connect to databases.
Now it can also be used in Google Forms. Quite useful if you want to add form validation for instance.
You can try it out here

Google Compute Engine, Google’s cloud service, has gotten some useful pricing options for small businesses. The cheapest one can host your application already for less than 2 cents per hour. There’s not much use for buying your own server anymore if cloud computing becomes so cheap ..

Transparency, when combined with goal-driven behaviors, acts as a powerful tool to motivate and guide development teams. Transparent, goal-driven behaviors are an integral component of all software development techniques, particularly when applying Agile techniques. Public, observable commitments are one form of transparent goal-driven behavior. The process of team members reporting and planning work during the stand-up meeting is an important example of public commitment and transparency. In a well-oiled Agile team, each member of the team accepts work for the day; “taking” the work that is the highest priority or swarming to tasks where help is needed. Everyone knows what everyone else is doing and is free to help guide the process. The act of “taking” the work in public is a tactical form of commitment that brings to bear pressure to perform and support the team. Failure of an individual team member to do what they promised will injure their reputation. Non-Agile project management techniques do not foster the same level of public commitment that the Scrum stand-up technique creates. Techniques that foster stronger commitment to the task at hand will increase the likelihood that a team will deliver what they promise. This will enhance the reputations of the individual, the team and in the long run the whole IT department. Publicly “taking” work during a stand-up combines transparency and commitment, goal-driven behavior, to create motivation.

By Tim Bray, Google Identity Team

During the Android portion of the Google I/O keynote, we showed Cross-Platform Single Sign-On; the effect was that for Wallet and Google+ users, signing in to a Web browser resulted in automatic download of, and sign-in to, an Android app.

To support this, we have introduced general-purpose API tools which allow developers to achieve cross-client authentication and authorization, in particular between Android and Web apps.

Not having to sign in repeatedly feels so natural for users that they don’t even notice it. But as more and more apps deploy this sort of magic, you don’t want to be the hold-out that’s pestering users for passwords on Web sites or, worse, on tiny mobile-device keyboards.

On the Android side, client libraries like PlusClient, GamesClient, and WalletClient have “connect” methods that take care of this as automatically as possible; they check whether any of the accounts on the phone have already been authorized for access to the service in question, conduct sign-in if necessary but avoid it if possible, and when they return to your code, everything’s all set up.

If you’re writing server-side code and using libraries like Google+ Sign-In, once again, all the right things happen automatically; when you start accessing the service, the software imposes the minimum necessary pain on the user, ideally zero, and lets you get to work.

Of course, some people want less automation, and finer control over how things work. If you want to access our services at the HTTP level rather than via a library, or to deal with multiple accounts on an Android device in a customized way, you can do these things and in most cases still deliver the no-sign-in magic.

Of course, this involves working with HTTP message flows, validating tokens, and securing shared secrets. This may sound intimidating but will be straightforward for one well-versed in HTTP-level Web programming. If you’re one of those, check out the low-level protocols and APIs that support this, in “Cross-Client Identity”.

The time is now to start moving your apps towards a sign-in-free future.


Tim says: By day, I help in the struggle against passwords on the Internet.
The rest of my life is fully documented on my blog.

Posted by Scott Knaster, Editor

The last year has been an exciting one for Android developers, with an incredible amount of momentum. In fact, over 48 billion apps have been downloaded from Google Play to date, with over 2.5 billion app downloads in the last month alone.

This week, at Google I/O, our annual developer conference, we’re celebrating this momentum, and putting on stage a number of new features and advancements both for the Android platform and Google Play, to help you design, develop and distribute great apps to your users.

We just wrapped up the keynote, and wanted to share a number of those new features; we’ll be spotlighting some of them throughout the week both here, on Google+, and in 36 Android sessions and sandboxes at the Moscone center in San Francisco (with many of the sessions livestreamed at developer.google.com). Enjoy!

Google Play Services 3.1

Google Play Services is our platform for bringing you easier integration with Google products and new capabilities to use in your apps. Today we announced a new version of Google Play Services that has some great APIs for developers.

• Google Play games services give you great new social features that you can add to your games   achievements, leaderboards, cloud save, and real-time multiplayer
• Location APIs make it easy to add location- and context-awareness to your apps through a fused location provider, geofencing, and activity recognition
• Google Cloud Messaging enhancements let you use bidirectional XMPP messaging between server and devices and dismiss notifications
• Cross-Platform Single Sign On, which lets your users sign in once, for all of their devices using Google+ Sign-In.

Android Studio: A new IDE for Android development

Today we announced a new Integrated Development Environment (IDE) built just for Android, with the needs of Android developers in mind. It’s called Android Studio, it’s free, and it’s available now to try as an early access preview.

To build Android Studio, we worked with with JetBrains, creators of one of the most advanced Java IDEs available today. Based on the powerful, extensible IntelliJ IDEA Community Edition, we've added features and capabilities that are designed specifically for Android development, to simplify and optimize your daily workflow for creating Android apps.

Google Play Developer Console: a better distribution experience

Building awesome Android apps is only part of the story. Today we announced great new features in the Google Play Developer Console that give you more control over how you distribute your app and insight into how your app is doing:

• App translation service: a pilot program that lets you purchase professional translations for your app directly from the Developer Console.
• Revenue graphs: a new tab in the Developer Console gives you a summary of your app global app revenue over time.
• Alpha and beta testing and staged rollouts: you can now distribute your app to controlled alpha and beta test groups, or do staged rollouts to specific percentages of your userbase.
• Optimization tips: design your app for tablets and understand how to expand your app into new language markets.
• Google Analytics: launching later this summer, your Google Analytics usage stats will be viewable right in the Developer Console.
• Referral tracking: also launching later this summer, you’ll get a new report in Google Analytics to show what blogs, campaigns, and ads are driving your installs.

Follow the Android Sessions

Join us for the Android sessions today and through the week by livestream. Visit the I/O Live Stream schedule for details.

By Urs Hölzle, Senior Vice President, Technical Infrastructure, and Google Fellow

Cross-posted from the Google Cloud Platform Blog

LIVE - Watch the stream of the Cloud track kickoff now

Over the last fourteen years we have been developing some of the best infrastructure in the world to power Google’s global-scale services. With Google Cloud Platform, our goal is to open that infrastructure and make it available to any business or developer anywhere. Today, we are introducing improvements to the platform and making Google Compute Engine available for anyone to use.

Google Compute Engine - now available for everyone

Google Compute Engine provides a fast, consistently high-performance environment for running virtual machines. Later today, you’ll be able to go online to cloud.google.com and start using Compute Engine.

In addition, we’re introducing new Compute Engine features:

• Sub-hour billing charges for instances in one-minute increments with a ten-minute minimum, so you don’t pay for compute minutes that you don’t use
• Shared-core instances provide smaller instance shapes for low-intensity workloads
• Advanced Routing features help you create gateways and VPN servers, and enable you to build applications that span your local network and Google’s cloud
• Large persistent disks support up to 10 terabytes per volume, which translates to 10X the industry standard

We’ve also completed ISO 27001:2005 international security certification for Compute Engine, Google App Engine, and Google Cloud Storage.

Google App Engine adds the PHP runtime

App Engine 1.8.0 is now available and includes a Limited Preview of the PHP runtime - your top requested feature. We’re bringing one of the most popular web programming languages to App Engine so that you can run open source apps like WordPress. It also offers deep integration with other parts of Cloud Platform including Google Cloud SQL and Cloud Storage.

We’ve also heard that we need to make building modularized applications on App Engine easier. We are introducing the ability to partition apps into components with separate scaling, deployments, versioning and performance settings.

Introducing Google Cloud Datastore

Google Cloud Datastore is a fully managed and schemaless solution for storing non-relational data. Based on the popular App Engine High Replication Datastore, Cloud Datastore is a standalone service that features automatic scalability and high availability while still providing powerful capabilities such as ACID transactions, SQL-like queries, indexes and more.

Over the last year we have continued our focus on feature enhancement and developer experience across App Engine, Compute Engine, Google BigQuery, Cloud Storage and Cloud SQL. We also introduced Google Cloud Endpoints and Google Cloud Console.

With these improvements, we have seen increased usage with over 3 million applications and over 300,000 unique developers using Cloud Platform in a given month. Our developers inspire us everyday, and we can’t wait to see what you build next.


Urs Hölzle is Senior Vice President of Technical Infrastructure and Google Fellow. As one of Google's first ten employees and its first VP of Engineering, he has shaped much of Google's development processes and infrastructure.

Posted by Scott Knaster, Editor

By Greg Hartrell, Lead Product Manager

We love to talk about games at Google. Especially the old ones, like Pac-man, Pitfall and Frogger. Since those classics, games have changed a lot. They’ve moved from that clunky box in your living room to the screen that you carry with you in your pocket wherever you go. They’re mobile, they’re social, and they’re an important part of Google Play.

Today, we’re launching Google Play game services, a core part of building a gaming platform for the next generation of games. These services help you make your games more social, with achievements, leaderboards, and multiplayer, as well as more powerful, storing game saves and settings in the cloud. They are available on Android, and many on iOS or any other connected device. By building on Google’s strengths in mobile and cloud services, you can focus on what you’re good at as game developers: creating great gaming experiences for your users.

With game services, you can incorporate:

• Achievements that increase engagement and promote different styles of play.
• Social and public leaderboards that seamlessly use Google+ circles to track high scores across friends and across the world.
• Cloud saves that provide a simple and streamlined storage API to store game saves and settings. Now players never have to replay Level 1 again.
• Real-time multiplayer for easy addition of cooperative or competitive game play on Android devices. Using Google+ Circles, a game can have up to 4 simultaneous friends or auto-matched players in a game session together with support for additional players coming soon.

Several great Android games are already using these new game services, including World of Goo, Super Stickman Golf 2, Beach Buggy Blitz, Kingdom Rush, Eternity Warriors 2, and Osmos.

And many more titles launch today as well:



Google Play game services are available today through an SDK for Android, and a native iOS SDK for iPhone and iPad games. Web and other platform developers will also find corresponding REST APIs, with libraries for JavaScript, Java, Python, Go, Dart, PHP, and more.

We’re excited to see what games will do with these new services and experiences, and this is only the beginning. Wait until you get to the boss battle... er.. Check out our developer site to get started: https://developers.google.com/games/.


Greg Hartrell is Lead Product Manager on Google Play game services, devoted to helping developers make incredible games through Google Play. In his spare time, he enjoys jumping from platform to platform, boss battles and matching objects in threes.

Posted by Scott Knaster, Editor