We had a power outage in our data centre yesterday and once it had recovered Jason and I wanted to do a quick check that one of our backend services was still responding in an acceptable amount of time.

Since this particular service only serves HTTP GET requests it was reasonably easy to setup a cURL command to do this:

while true; do curl -k -s -w %{time_total}  https://serviceurl/whatever/something; -o /dev/null; printf "\n"; done > service.txt

The ‘-w’ flag is one I haven’t used before but it allows us to output a bunch of interesting things about the request.

In this case we’re only interested in the end to end time of the request/response but we could be more fine grained if we wanted to be.

We ran that for a little while and then killed it and wrote a little Ruby script to get the times in a format that would make it easy to plug into R:

# output cut for brevity
$ p File.readlines("service.txt").map { |x| x.gsub(/\n/, "").to_f }.sort
[0.042, 0.043, 0.043, 0.045, 0.046...1.083]

We then plotted a Histogram in R:

> times = c(0.042, 0.043, 0.043, 0.045, 0.046, 0.046, 0.046, 0.047, 0.047, 0.047, 0.048, 0.048, 0.048, 0.048, 0.048, 0.048, 0.048, 0.049, 0.05, 0.05, 0.05, 0.051, 0.051, 0.051, 0.051, 0.051, 0.051, 0.051, 0.051, 0.052, 0.052, 0.052, 0.053, 0.053, 0.053, 0.054, 0.054, 0.055, 0.055, 0.056, 0.056, 0.056, 0.056, 0.056, 0.056, 0.056, 0.056, 0.057, 0.057, 0.057, 0.057, 0.058, 0.059, 0.06, 0.06, 0.06, 0.061, 0.061, 0.061, 0.061, 0.061, 0.061, 0.062, 0.062, 0.063, 0.063, 0.064, 0.066, 0.066, 0.067, 0.071, 0.071, 0.072, 0.073, 0.074, 0.079, 0.081, 0.084, 0.085, 0.085, 0.091, 0.108, 0.116, 0.14, 0.182, 0.26, 0.281, 1.083)
> hist(times)

That roughly shows us that the massive majority of requests were being served in under 1/10th a second which was acceptable for this service.

We can also run a couple of other commands on the times to learn more about the distribution:

> summary(times)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
0.04200 0.05100 0.05600 0.07745 0.06325 1.08300

> quantile(times, 0.95)
   95% 
0.1316

From these functions we learn that the max value is an outlier compared to the others and 95% of the requests were served in 0.13 seconds or less which was enough evidence to convince us that the service hadn’t degraded.

I think it's safe to say that more RSS readers are being built this week than at any other point in history. I particularly enjoyed this image from Miguel de Icaza which showed up in my Twitter stream yesterday:

I hate being excluded from what all the cool kids are doing, so I'm going to build an RSS reader which (1) is designed for mobile devices, and (2) has offline support.

An RSS Reader built on SQLite/Zumero, Part 1

The basic design is to use one SQLite file to keep a list of all the feeds, plus one additional dbfile for each feed.

Here's the SQLite script to create the 'all_feeds' database:

.load zumero.dylib

.echo ON

BEGIN TRANSACTION;

CREATE VIRTUAL TABLE feeds USING zumero(
    feedid INTEGER PRIMARY KEY,
    url TEXT NOT NULL UNIQUE
    );

-- configure the permissions on this dbfile to allow 'anyone' to 
-- (1) pull the dbfile, and 
-- (2) add rows to the feeds table  
-- and nothing else.

SELECT zumero_define_acl_table('main');

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    '',
    '*',
    zumero_named_constant('acl_result_deny')
    );

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    zumero_internal_auth_scheme('zumero_users_admin'),
    zumero_named_constant('acl_who_any_authenticated_user'),
    '',
    '*',
    zumero_named_constant('acl_result_allow')
    );

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    '',
    zumero_named_constant('acl_op_pull'),
    zumero_named_constant('acl_result_allow')
    );

INSERT INTO z_acl (scheme,who,tbl,op,result) VALUES (
    '',
    zumero_named_constant('acl_who_anyone'),
    'feeds',
    zumero_named_constant('acl_op_tbl_add_row'),
    zumero_named_constant('acl_result_allow')
    );

INSERT INTO feeds (url) VALUES ('http://feeds.hanselman.com/ScottHanselman');

COMMIT TRANSACTION;

SELECT zumero_sync(
    'main',
    'https://zinst393e9343b87.s.zumero.net',
    'all_feeds',
    zumero_internal_auth_scheme('zumero_users_admin'),
    'admin',
    'SECRETPASSWORD'
    );

Highlights:

• The ".load zumero.dylib" loads the Zumero SQLite extension so that I can use Zumero features in SQLite.

• The main thing going on here is the statement where I CREATE the feeds table. And it's a very simple table. All I need is a URL for the feed.

• Then you see a bunch of stuff to configure permissions. As it says in the comments, I want unauthenticated users to be able to see the feed list or add a feed, but all other modifications are prohibited unless you are, well, me.

• I seed the list with the feed from Hansleman's blog.

• Finally, I sync this SQLite database up to my server. Since I am creating a dbfile and defining its schema, I need to use the admin user/password that was created when I signed up for this Zumero server.

To execute this script, I save it to a file called setup_dbfile_all_feeds.sql and then pipe it into the sqlite3 shell.

eric$ ./sqlite3 ./all_feeds.db < setup_dbfile_all_feeds.sql

Like I said, anyone can add feeds to this list. The server URL is actually the one shown in these examples. I'm going to open my local copy of "all_feeds" and add my own RSS feed URL to the list:

eric$ ./sqlite3 ./all_feeds.db 
SQLite version 3.7.11 2012-03-20 11:35:50
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .load zumero.dylib
sqlite> INSERT INTO feeds (url) VALUES ('http://ericsink.com/rss.xml');
sqlite> SELECT * FROM feeds;
1|http://feeds.hanselman.com/ScottHanselman
2|http://ericsink.com/rss.xml
sqlite> SELECT zumero_sync(
    'main',
    'https://zinst393e9343b87.s.zumero.net',
    'all_feeds'
    );
0;0;3584;0;448;0;1249;1264

Note that I didn't need to pass any authentication credentials to the zumero_sync() function.

So far, my RSS reader doesn't do anything. It's just a list of feeds. Granted, it's a really cool list, since it supports incremental sync, but still. I'm probably going to need my RSS reader to do something with, er, RSS. And for that, I'm going to need more than just SQL statements piped into the sqlite3 shell.

Here's my starting point, in C#:

using System;
using System.Collections.Generic;
using System.ServiceModel.Syndication;
using System.Xml;

using SQLite; // https://github.com/praeclarum/sqlite-net

namespace z
{
    class Program
    {
        // define a little class to represent rows of the feeds table

        public class feed_row
        {
            public string feedid { get; set; }
            public string url { get; set; }
        };

        public static void Main (string[] args)
        {
            // open the local SQLite db
            SQLiteConnection conn = new SQLiteConnection("all_feeds.db");

            // tell SQLite to allow load_extension()
            conn.EnableLoadExtension(1);

            // load the Zumero extension
            // this is the equivalent of 
            // ".load zumero.dylib" in the sqlite3 shell
            conn.ExecuteScalar("SELECT load_extension('zumero.dylib');");

            // iterate over all the rows in the feeds table
            var rows = conn.Query ("SELECT feedid, url FROM feeds;");
            foreach (feed_row q in rows)
            {
                Console.WriteLine(q.url);
                try
                {
                    XmlReader xr = new XmlTextReader(q.url);
                    SyndicationFeed f = SyndicationFeed.Load(xr);

                    // TODO store the items from this feed
                }
                catch (Exception e)
                {
                    // TODO failed trying to retrieve or parse this feed.
                    // TODO log the failure?
                    // TODO delete the feed row?
                    // TODO launch nethack?
                }
            }

            conn.Close();
        }
    }
}

In order to call the SQLite library from C#, I'm using SQLite.cs, by Frank Krueger. That wrapper doesn't have a way to call sqlite3_enable_load_extension(), so I added one. Here's the diff, which I plan to submit as a pull request:

eric$ diff orig_SQLite.cs SQLite.cs 
183a184,192
>         public void EnableLoadExtension(int onoff)
>         {
>             SQLite3.Result r = SQLite3.EnableLoadExtension(Handle, onoff);
>           if (r != SQLite3.Result.OK) {
>               string msg = SQLite3.GetErrmsg (Handle);
>               throw SQLiteException.New (r, msg);
>           }
>         }
> 
2645a2655,2659
>       [DllImport("sqlite3", 
>                 EntryPoint = "sqlite3_enable_load_extension", 
>                 CallingConvention=CallingConvention.Cdecl)]
>       public static extern Result EnableLoadExtension (IntPtr db, int onoff);
> 

I'm using System.ServiceModel.Syndication.SyndicationFeed to do the RSS parsing for me. I don't have any previous experience with that library, but so far, it just works.

The statement I used to compile this program is:

eric$ gmcs -reference:System.ServiceModel.Web.dll \
      -out:z.exe AssemblyInfo.cs SQLite.cs Main.cs
...
Compilation succeeded - 3 warning(s)

Since I'm running on Mac OS X (Lion), I had to bring my own SQLite to the party. On Mac OS, the sqlite3 library and shell are preinstalled with the OS, but they were compiled without support for dynamic extension loading. If you're on a Mac, it's easy to grab and build your own copies:

eric$ curl --silent --remote-name \
      http://www.sqlite.org/sqlite-amalgamation-3071502.zip

eric$ unzip sqlite-amalgamation-3071502.zip 
Archive:  sqlite-amalgamation-3071502.zip
   creating: sqlite-amalgamation-3071502/
  inflating: sqlite-amalgamation-3071502/sqlite3.h  
  inflating: sqlite-amalgamation-3071502/shell.c  
  inflating: sqlite-amalgamation-3071502/sqlite3ext.h  
  inflating: sqlite-amalgamation-3071502/sqlite3.c  

eric$ mv sqlite-amalgamation-3071502/* .

eric$ rmdir sqlite-amalgamation-3071502

eric$ clang -dynamiclib -arch i386 -arch x86_64 \
      -o libsqlite03071502.dylib sqlite3.c

eric$ clang -o sqlite3 -arch i386 -arch x86_64 sqlite3.c shell.c 

eric$ ./sqlite3 --version
3.7.15.2 2013-01-09 11:53:05 c0e09560d26f0a6456be9dd3447f5311eb4f238f

I also created a .config file because SQLite.cs does DllImport("sqlite3", and need to map that to the SQLite library we just built.

<configuration>
    <dllmap dll="sqlite3" target="libsqlite03071502.dylib" />
</configuration>

Running the program yields the expected output:

eric$ mono ./z.exe
http://ericsink.com/rss.xml
http://feeds.hanselman.com/ScottHanselman

So, after all that, this code STILL doesn't really do anything useful. It demonstrates that I can successfully use SQLite and Zumero from C#. And it retrieves and parses each feed. But I have not yet implemented any of the things that are supposed to happen next. Hopefully I'll make more progress in part 2.

 

With the rise of social apps like Facebook, Instagram, YouTube and more, managing user generated content has become a growing challenge and problem to be solved.  Amazon AWS S3, Google Storage, Rackspace Cloud Files, and other similar services have sprung up to help application developers solve a common problem - scalable asset storage.  And of course, they all utilize “the cloud”!

The Problem

Popular social applications, scientific applications, media generating applications and more are able to generate massive amounts of data in a small amount of time.  Here are just a few examples:

• 72 hours of video are uploaded every minute by YouTube users. (source)
• 20 million photos are uploaded to SnapChat every day. (source)
• Pinterest has stored over 8 billion objects and 410 terabytes of data since their launch in 2009. (source)
• Twitter generates roughly 12 terabytes of data per day. (source)

When your application begins to store massive amounts of user generated data, your team will inevitably need to decide where to spend its engineering effort in relation to that data.  If your application is engineered to store assets on your own hardware/infrastructure, your team will spend plenty of time and money related to storing and serving your assets efficiently.  Alternately, an application can easily store assets with a cloud storage provider.  Choosing this route allows application content to scale almost limitlessly while only paying for the resources and space needed to store and serve the assets. In effect, cloud storage frees up your teams engineering time to focus more on creating unique application features, rather than reinventing the file storage wheel when scalability becomes an issue.

When should you consider using cloud based storage for your application?

• When user generated assets are part of your application.Does your application accept uploads from your users?  Does your application generate files serverside?  If your application is going to accept uploads from users or generate content stored on the filesystem, you will likely want to consider using a cloud storage provider sooner rather than later.
• When your application is likely to grow beyond a single server.If your application is small enough to run on a dedicated single server or web host, and you don’t expect it to grow beyond that single server, it doesn’t make sense to use cloud storage for your assets.  Simply store them on the local file system and call it a day.If, however, you expect any growth from your application that would require you to run more than one application server, you will immediately reap the benefits of storing your assets in the cloud.  By storing your assets in the cloud, you can horizontally scale your service to as many front-end application servers as your heart desires without the need to replicate your file system assets to the any new servers.  Because your assets are stored centrally with a cloud service, they will be accessible from a given hostname, no matter how many application servers your application runs.
• When its more cost effective for your team to focus on business critical application features rather than engineering a scalable file storage system.If you are strapped for either time or money, and you expect your application to grow, you can’t go wrong with cloud storage.  Cloud storage gives you the flexibility to get up and running quickly, scale your storage to the growing needs of your application and only pay for the storage and resources you use.  This in turn allows you to focus less on hardware costs, operations and configuration for storing assets and more importantly focus your time on developing your business.

Integration & Access

Most of the leaders in online cloud storage provide API access to their platform allowing developers to integrate web-scale asset storage and file access within their applications.  Below we’ll look at some code examples using an SDK or library to store assets on Amazon S3.  Many libraries and SDKs make setting the storage provider a breeze, allowing you to easily deploy file storage on many of the popular providers.

Ruby & Carrierwave

Code examples below have been adapted from the CarrierWave github repository.

• Install CarrierWave: gem install carrierwave or in your gemfile gem 'carrierwave'
• Install Fog: gem install fog or in your gemfile gem "fog", "~> 1.3.1"
• In an initiailization file add the following:

  CarrierWave.configure do |config|
  config.fog_credentials = {
  :provider               => 'AWS',       # required
  :aws_access_key_id      => 'xxx',       # required
  :aws_secret_access_key  => 'yyy',       # required
  :region                 => 'eu-west-1'  # optional, defaults to 'us-east-1'
  }
  config.fog_directory  = 'name_of_directory'
  config.fog_public     = false
  config.fog_attributes = {'Cache-Control'=>'max-age=315576000'}
  config.asset_host     = 'https://assets.example.com’
  end

• Create your uploader class:

  class AvatarUploader < CarrierWave::Uploader::Base
  storage :fog
  end

• Using your uploader directly:

  uploader = AvatarUploader.new
  uploader.store!(my_file)
  uploader.retrieve_from_store!('my_file.png')

• Using your uploader with ActiveRecord:
• Add a field to your database table and require CarrierWave:

  add_column :users, :avatar, :string in a database migration file
  require 'carrierwave/orm/activerecord' in your model file.

• Mount your uploader to your model:

  class User < ActiveRecord::Base
  mount_uploader :avatar, AvatarUploader
  end

• Work with your model and files:

  u = User.new
  u.avatar = params[:file]
  u.avatar = File.open('somewhere')
  u.save!
  u.avatar.url # => '/url/to/file.png'
  u.avatar.current_path # => 'path/to/file.png'
  u.avatar.identifier # => 'file.png'

Here are some CarrierWave examples for uploading to Amazon S3, Rackspace Cloud Files and Google Storage.  And some gems for using with other ORMs like DataMapper, Mongoid and Sequel.

PHP & AWS SDK

Amazon provides a PHP SDK to work with AWS APIs and services.  For this code example we will be using instructions straight from the SDK repository README and sample code.

• Copy the contents of config-sample.inc.php and add your credentials as instructed in the file.
• Move your file to ~/.aws/sdk/config.inc.php.
• Make sure that getenv('HOME') points to your user directory. If not you'll need to set putenv('HOME=<your-user-directory>')

• // Instantiate the AmazonS3 class
  $s3 = new AmazonS3();
  
  // Create a bucket to upload to
  $bucket = 'YOUR-BUCKET-NAME' . strtolower($s3->key);
  if (!$s3->if_bucket_exists($bucket))
  {
  $response = $s3->create_bucket($bucket,AmazonS3::REGION_US_E1);
  if (!$response->isOK()) die('Could not create `' . $bucket . '`.');
  }
  
  // Download a public object.
  $response = $s3->get_object('aws-sdk-for-php', 'some/path-to-file.ext',array(
  'fileDownload' => './local/path-to-file.ext'
  ));
  
  // Uploading an object.
  $response = $s3->create_object($bucket, 'some/path-to-file.ext', array(
  'fileUpload' => './local/path-to-file.ext'
  ));

Node & Knox
For Node.js I have adapted example code from the Knox Amazon S3 Client on Github.

• // Configure the client
  var client = knox.createClient({
  key: '<api-key-here>'
  , secret: '<secret-here>'
  , bucket: 'BUCKET-NAME'
  });

  // Putting a file on S3
  client.putFile('some/path-to-file.ext', 'bucket/file-name.ext', function(err, res){
  // Logic
  });

  // Getting a file from S3
  client.get('/some/path-to-file.ext’).on('response', function(res){
  console.log(res.statusCode);
  console.log(res.headers);
  res.setEncoding('utf8');
  res.on('data', function(chunk){
  console.log(chunk);
  });
  }).end();

  // Deleting a file on S3
  client.del('/some/path-to-file.ext’).on('response', function(res){
  console.log(res.statusCode);
  console.log(res.headers);
  }).end();

Conclusion
As you can see in the previous code examples, working with the AWS S3 APIs are very straightforward and there are plenty of libraries readily available for most languages.  I definitely recommend taking a hard look into using a cloud storage provider for your next project.  You’ll save time not reinventing file storage solutions, reap the benefits of focusing on developing your application, and have practically unlimited storage scalability as you need it.

Hey, it's HighScalability time: 

• 0: # of Google Readers; 2.5 billion/day: new pieces of content added to Facebook; 2.7 billion/day: likes added to Facebook; 7PB/month: photos added to Facebook
• Quotable Quotes:
  • @cwgem: It seems like cutting down API access is the stock scalability answer these days
  • @abenik: @Prismatic surfaced this article on their architecture for me. How meta.
  • @NewsBlur: The waters are rocky now, but take note that I have some time to get things right. I'm working this week to get things stable, then scale.
  • @Pinboard: Just learned that Google Reader no longer offers direct JSON export. I guess they held the annual "What should we ruin next?" staff retreat
  • @DEVOPS_BORAT: You can not able have unlimit scalability without unlimit outage.
  • Jeff: Amazon RDS Scales Up - Provision 3 TB and 30,000 IOPS Per DB Instance
  • @migueldeicaza: Google recently hired all of the Twitter's scalability team to work on Google IO checkout.
  • @skamille: Interesting to consider the greatly diminished role of networked file systems in modern distributed computing
  • @vambenepe: The server huggers have regrouped. Now they’re VM huggers, ironically. Fighting PaaS with all their might.
  • @jezhumble: If the developers can't self-service everything they need programmatically through an API, it's not a private cloud.
  • @Bremmel: Foursquare users crawl the real world like Google's spiders crawl the web - Dennis Crowley
  • @mollstam: SimCity's API (and I'm guessing region storage) is on Amazon. How can it not be auto-scaling? How can it take three days to add servers?
  • @josephmartz: Scalability gurus: It's about low coupling merging with high cohesion. More encapsulation and f*ck scaling out. I just want one #node.
  • @SQLSniper: great recipe for #sqlserver scaling from @GlennAlanBerry precon :) "scale up is like pets, scale out is like cattle" 
• @NewsBlur's tweet feed is a great blow by blow of the crush that happened when Google Reader became a dead app walking. A signup a second...Now fetching millions of new feeds hourly...Suspended free accounts, premium accounts only....Prices increased...Redis suffered from memory corruption...Moved from one app server to 6...Dropped SES for Mail Gun...Hosting provider died...Bringing up PostgreSQL read slaves...DB server upgrades...Introduction of HAProxy...More app servers.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

This is from an article about the application of Bayesian Statistics to a civil suit in the UK over the source of a building fire.

The idea that you can assign probabilities to events that have already occurred, but where we are ignorant of the result, forms the basis for the Bayesian view of probability. Put very broadly, the 'classical' view of probability is in terms of genuine unpredictability about future events, popularly known as 'chance' or 'aleatory uncertainty'.

The Bayesian interpretation allows probability also to be used to express our uncertainty due to our ignorance, known as 'epistemic uncertainty', and popularly expressed as betting odds. Of course there are all gradations, from pure chance (think radioactive decay) to processes assumed to be pure chance (lottery draws), to future events whose odds depend on a mixture of genuine unpredictability and ignorance of the facts (whether Oscar Pistorius will be convicted of murder), to pure epistemic uncertainty (whether Oscar Pistorius knowingly shot his girlfriend).

When we build probabilistic models of project performance - cost, schedule, and technical - we assume we understand the underlying statistical processes that drive these probabilistic generating functions. These are the aleatory uncertainties in duration, cost, and performance. We define the Probability Density Function in the Monte Carlo Simulator. Then we apply that to the network of work activities (the Integrated Master Schedule), to produce confidence outcomes for completing on or before a planned date and at or below a planned cost. This is all fine and dandy. But we really don't know the underlying drivers that create coupling, correlation, and cross correlations between the work activities, cost, and technical performance. These can be model by discovering the drivers in the network.

For the Epistemic uncertainties we need another modeling tool. The current tools don't actually use Bayesian statistics, rather they use Monte Carlo Simulation and treat the Probability of an Event as an aleatory process integrated with the other PDF's, ranges, and their shapes (Kurtosis and Skew).

We're missing the tools needed to construct a credible epistemic model of how the program works. Using the Integrated Master Schedule (IMS) as the topology for work, the probabilistic behaviours of the work elements at each node - cost, schedule, and technical performance compliance of the products - and the coupling and cohesion of the nodes. With this information - assuming it is credible, which is a HUGE assumption - we could model the behaviour of the program and ask what if questions. 

We use proxies in many situations. My neighbors recently provided a humorous example of a proxy. They decided to build a diorama of log reindeer after they could not induce the local deer to hang out in their front yard and pose. Perhaps because I let the dog out to bark at the real deer. The log deer erected were a proxy for real deer and, in this case, not only will the proxy suffice but they might be better for neighborhood peace.

Naming a proxy for a product owner in an agile project will be far less satisfactory than a herd of log reindeer. A product owner acts as the Voice of the Customer channeling information about how the product will be used to the project team. This information is critical for the team to maximize the value of what they are building. Product owners also provide the team with priorities and decisions based on their knowledge of the business environment. The further the person acting as the product owner is from the financial responsibility (profit and loss) for the product or the day-to-day operations that use the product the lower the fidelity of the information he or she can provide. Poor information or slower decisions will yield lower value to the organization regardless of what is being built. While a proxy product owner might not be a log reindeer, the further they are from the business the more apt you will be to get termites rather than the answers your agile project team needs.

I have a new article up on projectmanagement.com, Managing Programs with Agile and Traditional Projects. You know the problem with a program: you have some agile projects and some not-agile projects, and maybe some projects who suffer from an identity crisis. They might think they are one or the other. You might not agree with their categorization!

Here’s the key: ask for deliverables. You might need to coach the project managers or use your influence. Now, go read the article. Please comment over there. Enjoy!

If you want to know more about how to be an influential agile leader, join Gil and me April 8-9 in Boston. We have created an in-depth, transformative experience for you. If you want to learn how to take your agile leadership to the next level, this event is for you.

Early bird registration ends today, so don’t delay. Register now. Email me with questions.

I am so very excited to unveil our latest endeavor:

What is Zumero?

To describe Zumero, we first describe SQLite.

SQLite is a lightweight (but surprisingly powerful) implementation of a SQL database.

SQLite is the standard database software for iOS, Android, and Windows RT. It is installed on over a billion mobile devices.

But like any other computer, and perhaps more so, a mobile device is not isolated. It needs to share data with a server.

And SQLite has no synchronization capabilities.

Zumero solves that problem.

Zumero is "sync for SQLite".

No, really, what is Zumero?

It's a better database platform for mobile.

Mobile apps need to talk to the cloud about data.

A lot of people are building mobile apps that use the "remote procedure call" model. Every user operation requires a network request to the server.

We think "replicate and sync" is a better model. The app can interact with a local copy of the database instead of going through the network. Synchronization can be handled entirely in the background.

Putting network activity in the background brings two big wins:

• The developer wins because networking issues are isolated in a small part of the app. The real features of the app can be the focus of the developer's attention, and are easier to code.

• The user wins because the app is faster, more responsive, and works offline.

Consider a typical situation where a mobile app responds to a user action by filling a list box with the results of a database query. Neither the developer nor the user wants the network involved between the button tap and the refresh of the display. Mobile networks are just not consistent enough. Even a perfectly-working LTE connection usually has enough latency to make the app feel sluggish.

Oh good grief. Eric, I thought you were a geek? Tell us what Zumero is.

On the client, Zumero is a SQLite extension. The server is built on node.js.

The client tries very hard to be invisible to the developer. We didn't want to introduce a whole new API for everybody to learn, so developing with Zumero isn't much different from SQLite. Porting an existing SQLite app to use Zumero is usually straightforward.

The SQLite extension uses SQLite's "virtual tables". A Zumero table mostly works just like a regular table, except that it can be synced.

The Zumero server manages the central, authoritative copy of the database and handles all the details of synchronization, merging changes, conflict resolution, caching, logging, authentication, and permissions.

Why SQLite?

1. Because it's an incredible piece of software. If you're not familiar with SQLite, give it a serious look. It has come so far that keeping the word "Lite" in the name seems inappropriate. SQLite is fast, robust, and far more capable than most people know.

2. Because it's everywhere. Every smart phone and tablet has SQLite inside. Even Microsoft has given up the notion of shrinking SQL Server and decided to standardize on SQLite as its recommended database for Windows RT.

3. Because it needs a sync feature. The real question here is not "Why SQLite?" but rather, "Why NOT SQLite?" Why should people consider anything except SQLite for dealing with mobile data? The main answer is obvious: SQLite doesn't sync. Zumero was created to fill that need, not by introducing a new mobile database system, but rather, by adding painless synchronization features to the high quality database that is already there.

What platforms do you support?

Right now (March 2013), the Zumero Client SDK includes support for iOS and Android. WinRT is planned. Blackberry is not, but that might change if there is evidence of customer demand.

We will also be supporting two of the cross-platform mobile solutions: Apache Cordova (aka Phonegap) and Xamarin. We've made significant progress on both of these, but neither of them was ready for the initial release.

The client SDK also includes support for (regular) Windows and Mac OS. This may seem odd, since Zumero is all about mobile. But very few so-called mobile apps are accessed exclusively from mobile devices. You may (and probably will) want the ability to also access your data from web apps, desktop applications, or even other servers.

Even if you don't need any of these things, the SQLite command-line shell is a handy way to perform administrative tasks for your Zumero server, such as initial setup of your schema, configuring permissions, or analyzing logs.

How does Zumero store data on the server?

The Zumero server has a plugin architecture designed to support a variety of SQL databases. We currently have two implementations of the database backend: SQLite and PostgreSQL.

At the present time (still March 2013), the Zumero server is available as a cloud-hosted service with the SQLite backend, but it will be made available in other flavors later.

What about server platforms?

From a strictly technical perspective, the Zumero server runs just fine on Linux, Mac, and Windows.

The current cloud-hosted offering is Linux, running in the Amazon cloud.

In the future, we will make the server available in three ways: in the cloud, on-premises, or as a desktop server for development purposes. Not all combinations of platform and deployment are likely to be interesting. Things will probably end up something like this:

Linux Windows Mac Other Cloud Yes Yes
(Azure, later) Where? What? On-premises Yes
(later) Yes
(later) Nobody wants this Maybe Solaris? Development Yes
(later) Yes
(later) Yes
(later) We'll support Commodore 64 right after node.js does Is Zumero open source?

No.

Who are Zumero's competitors?

As always, that depends on how narrowly you define "competitor". :-)

• At one extreme, we are not aware of anybody doing EXACTLY what we are doing.

• At the other extreme, everybody doing anything with data and mobile is a competitor.

If I had to name just one other player, it would be Couchbase Lite. Couchbase founder J. Chris Anderson recently did a great blog entry called Why Mobile Sync?. His tune sounds a lot like the one we're singing about Zumero.

So it wouldn't be outrageous to describe Zumero by saying, "It's like Couchbase Lite. Except it's SQL".

SQL? I thought SQL was dead?

Hardly. The NoSQL offerings are gaining traction, but SQL still has far greater market share, especially in the enterprise.

How does Zumero compare to Parse/Kinvey/Stackmob and their ilk?

We don't really think of Zumero as a "Backend as a Service" (BaaS) company, since we're focused on the database part of the story.

But if you ignore all the other mobile backend stuff (like push notifications), it would be reasonable to describe Zumero by saying, "It's like Parse/Kinvey/Stackmob. Except it's a replicate-and-sync model. And it's SQL."

Is Zumero part of SourceGear?

No. Yes. Sort of.

In terms of market positioning, Zumero is separate. It does not share SourceGear branding. It has no particular connection with SourceGear's products.

But financially, Zumero is currently part of SourceGear. (This approach is similar to what we did with Teamprise, which we sold to Microsoft in 2009.)

Corey and I think of Zumero as our new startup, which is being incubated inside SourceGear.

Mobile data? Really? How is this related to version control?

It's not.

But the genesis of Zumero happened when we did the iPad app for Veracity. We liked the "replicate and sync" model for mobile so much that we decided to take the databasey part of Veracity and productize it as a platform for building all kinds of mobile apps. Along the way, the technology evolved a lot, to the point where it doesn't look much like the original Veracity code, but that's what we started with.

This is actually somewhat of a return to our roots for SourceGear. We spent a lot of our first three years doing mobile development as contract work for Motorola. Of course, today's mobile devices are just a little more powerful than that phone we worked on back in 1997. :-)

What does this mean for SourceGear's other products, Vault and Veracity?

They are continuing to move forward. Vault 7 is underway and aiming for a Q3 release. Veracity 2.5 is done and will be released sometime in the next two weeks.

How can I try Zumero or find out more about it?

http://zumero.com/get-started

 

By Michael Manoochehri, Developer Programs Engineer, Cloud Platform

Google BigQuery is designed to make it easy to analyze large amounts of data quickly. Overwhelmingly, developers have asked us for features to help simplify their work even further. Today we are launching a collection of updates that gives BigQuery a greater range of query and data types, more flexibility with table structure, and better tools for collaborative analysis.

Big JOIN and Big Group Aggregations

Extracting insights from multiple datasets can be challenging and time-consuming. This is especially true when datasets become too large to query with a traditional database system. With traditional databases, SQL functions like joining and grouping are typically used to bring together data for analysis. What happens when your data is too large to fit into a conventional database? Working with multi-terabyte datasets often requires developing complicated MapReduce workflows, investing in expensive infrastructure, and great deal of time. Very often, it's a combination of all three.

In response to developer feedback, we're launching new features that enable analysts and developers to run fast SQL-like join and aggregate queries on datasets without the need for batch-based processing. Our new Big JOIN feature gives users the ability to produce a result set by merging data from two large tables by a common key. Big JOIN simplifies data analysis that would otherwise require a data transformation step, by allowing users to specify JOIN operations using SQL.

Popular web applications produce user activity logs that can grow by billions of rows each week. Dividing users into smaller groups is a key step for analysis. However, each group of users can number in the millions. To handle this for such large volumes, we've enabled Big Group Aggregations, which significantly increases the number of distinct values that can be grouped in a result set.

To use these new features, simply add the EACH modifier to JOIN or GROUP BY clauses.


/* JOIN EACH example
* Selects the top 10 most edited Wikipedia pages
* of words that appear in works of Shakespeare.
*/

SELECT
 TOP(wiki.title, 10), COUNT(*)
FROM
 [publicdata:samples.wikipedia] AS wiki
JOIN EACH
 [publicdata:samples.shakespeare] AS shakespeare
ON
 shakespeare.word = wiki.title;

For more information, including best practices, when using JOIN EACH and GROUP EACH BY, visit the BigQuery query reference.

Native support for TIMESTAMP data type

We are also adding a new TIMESTAMP data type, in response to one of our most frequent requests from developers. This new data type lets you import date and time values in formats familiar to users of databases such as MySQL, while preserving timezone offset information.

Along with the new data type come new functions for converting TIMESTAMP fields into other formats, calculating intervals, and extracting components such as the hour, day of week, and quarter.


/* TIMESTAMP example
* Which hours in the day are the most popular for GitHub actions?
* This query converts github_timeline "created_at" date time   
* strings to BigQuery TIMESTAMP, and extracts the hour from each.
*/

SELECT
 HOUR(TIMESTAMP(created_at)) AS event_create_hour,
 COUNT(*) AS event_count
FROM
 [publicdata:samples.github_timeline]
GROUP BY
 event_create_hour
ORDER BY
 event_count DESC;

Read more about the available TIMESTAMP functions in our query reference guide.

Add columns to existing BigQuery tables

When working with large amounts of fast moving data, it's not uncommon to find out that you need to add additional fields to your tables. In response to developer feedback, we have added the ability to add new columns to existing BigQuery tables.

To take advantage of this feature, simply provide a new schema with additional columns using either the "Tables: update" or "Tables: patch" BigQuery API methods.

For more information on this feature, visit the BigQuery API reference.

BigQuery Web UI: Dataset links and dataset sharing notifications

BigQuery has always provided project owners with very fine-grained control of how their datasets are shared. To make it easier for teams to work on collaborative data analysis, we've added direct links to individual datasets in the BigQuery Web UI. This provides a convenient way for authorized users to quickly access a dataset, and allows for bookmarking and sharing.

In addition, we've also added email notifications to inform users when they've been given dataset access privileges. When a dataset has been shared with another user via the sharing control panel, BigQuery sends a notification email containing a direct link to the dataset.


The BigQuery UI features a collection of public datasets for you to use when trying out these new features. To get started, visit our sign up page and Quick Start guide. You should take a look at our API docs, and ask questions about BigQuery development on Stack Overflow. Finally, don't forget to give us feedback and join the discussion on our Cloud Platform Developers Google+ page.


Michael Manoochehri is a Developer Programs Engineer supporting the Google Cloud Platform. His goal is to help make cloud computing and data analysis universally accessible and useful.

Posted by Scott Knaster, Editor

What 5 measures can you use as personal objectives?

On the modern analyst forum, Andrea asked the question “I have been asked to come up with 5 personal “Management Business Objectives” for myself in my position as Business Systems Analyst. I need to have them by the end of January and I can’t really come up with anything. Any suggestions? They need to be related to my position (within my sphere of influence) and they must be measurable. “

http://www.modernanalyst.com/Community/Forums/tabid/76/forumid/-1/threadid/8326/scope/posts/Default.aspx

Here is my response:

We have stages of organizational maturity which includes what individuals should be doing at each level. This can give you a good idea of areas where you can find metrics.

http://www.seilevel.com/downloader.php?id=16

Here are some examples of metrics that we use for our analysts. In the first year I don’t think there should be targets, there should just be a commitment to measure. Once you have a baseline you can establish an improvement program the following year. You absolutely should not be bonused on any of these, they should be a reference to help you to improve.

1) Perform an ROI analysis before and after every project

2) Measure my end user satisfaction level

3) Measure my end user adoption rate

4) Measure how many requirements defects I created

5) Measure how satisfied the development team was with my documentation

6) Measure velocity per unit of work – we have an estimating template here

http://requirements.seilevel.com/blog/wp-content/uploads/2011/10/Requirements-Estimation-Tool.xls

You can compare yourself to the targets in each of the areas of work.

What 5 measures can you use as personal objectives? is a post from: http://requirements.seilevel.com/blog

In case you forgot the Value in useful form

And then just to make a point

Have you ever ridden a bicycle with a flat tire? A flat seriously affects the amount of effort needed to move the bike. Broken application development and maintenance processes have a similar impact. Process problems can slow delivery by requiring extra testing, extra reviews and added hurdles for implementation to ensure work is done correctly. This added work robs the organization of the added value that could have been delivered if the extra burden could be avoided.

Fixing the problem is not always easy or convenient. Assuming that each new test or review step in the process made sense when it was added, why would it make sense not to do it now? Once, I had to walk my bike all the way to end of the block, just to discover that I did not have a quarter for air. I had to walk the bike back home to get the money and then walk all the way back – highly inconvenient. I added a step to my mental checklist, never go for air without checking that I had a quarter. The process is now one step longer. We evolve processes we use to do work in a very similar manner; one step at a time.

Why don’t we stop work and redesign all processes from scratch? The amount of work required would be daunting. The perception of the change would also be daunting. Since we got into this mess one step at a time, I would suggest that we can get out one step at a time. Incremental change driven by frequent project retrospectives combined with transparent organizational goals is a great mechanism for continuous change. Paraphrasing W. Edwards Deming, we will need constancy of purpose to make continuous process improvement payoff but with that constancy of purpose we won’t need overwhelming changes that could leave the organization feeling like they are riding on a flat tire.

Photo by Clint Gardner @ flickr (CC-BY NC SA 2.0)
I wanted to start this series by focusing on some of the challenges that we face as Product Managers. A key challenge I see repeated in many organizations is the lack of capacity to leap from idea to results, from intention to implementation.
Product Managers focus on generating product ideas, business model, revenue streams, competitor analysis, etc. But that is quite useless unless it leads to coherent action, and ultimately to delivering products to the market. In fact delivering on an idea is - in my mind - the only valid evaluation for a great Product Manager.
Do you have a great idea? Do you know the market very well? Good, but where is the product? No product, no value.
Company strategy is a first step in covering the gap from goal to action: define the goal As I stated in the previous post, the Strategic Product Manager must focus on translating the company strategy into a coherent Product Portfolio. This is the process of transforming business objectives into concrete product goals. However, that is just the first step in covering the gap from ideas to implementation.
A common pitfall is that once the Portfolio is approved the Product Manager retires from the involvement with implementation. Later, when deciding on what projects to start the teams involved will make critical, strategic decisions. If the Product Manager is not part of the decision process, how can she make sure that her Vision for the product is implemented?
Having the best roadmap and then implementing the wrong projects is equivalent to having a bad roadmap.
Now that you have the roadmap, how will you execute on it? Product Management must work with the line organization, with the teams that will develop the software and do what often does not happen: to transform ideas into action! For this to be possible the Strategic Product Manager must be equally capable of transforming the implementation feedback (from the teams) into information that will influence our ideas about strategy (product roadmap and product portfolio).
Ideas are nothing without execution. A key aspect of the Strategic Product Management is to be involved and review the actual projects started and reassess the product strategy, product portfolio and product roadmap based on what is possible to implement once the rubber meets the road and the product development starts.
To effectively implement a product strategy the Strategic Product Manager must be part the decisions on team and resource allocation to certain projects. Reviewing and giving feedback on a second and important portfolio: the Project Portfolio (which projects are started, how much investment they get and when they should stop?).
Castles are built with rocks and mud, not clouds Strategic Product Management is much more than just thinking big thoughts. Strategic Product Management is about creating the necessary relationships and information flows in the organization, that will allow the implementation teams to execute on the strategy for the company and the products. Later in this series we will take a look at some of the practices that help both the Product Manager and the Organization to transform a great Company and Product strategy into a coherent Project Portfolio. These will include organizational as well as project techniques designed to help convert ideas into real value: products in the market.

Just a quick note to say that I've added some new essays to my Software Architecture for Developers book. They are:

• Software architects should be master builders: this talks about the importance of software architects needing deep technology skills and working as a part of the team.
• Technology is not an implementation detail: this is about why you shouldn't leave technology decisions as a after-thought.
• The need for sketches: this explains the importance of sketches in communicating the overall software architecture, both inside and outside of the immediate team.
• Would you code it that way?: this talks about the key question you need to ask yourself whenever you're designing software and visualising that design through sketches.

Leanpub does have the ability to e-mail readers whenever a new version of the book is published, but I try to only do this once or twice a month so as not to spam everybody. There is a version history at the back of the book though, which will tell you when new essays have been added plus it hyperlinks right to them. Hope you enjoy the new content.

I am very excited to announce my latest project – Software Requirements, 3rd Edition with Karl Wiegers by Microsoft Press! We are well into the draft version and it’s due out next summer. I did a more in depth post about the writing process and what it’s about on the Microsoft blog, but here’s a teaser about what’s coming:

• New enhancements for agile projects
• New elicitation and modeling techniques
• How to handle requirements on a handful of specific  project types
• Closer alignment to the IIBA BABOK since it’s really taken off in the last few years

For a couple years now, I’ve been working on the core team for BABOK version 3. It’s been really beneficial and insightful in terms of how the BA profession has evolved. It has been really exciting working with Karl to update this already solid text with new innovations!

More to come over the next couple of months, but to get started, you can pre-order now on Amazon!

Announcing: Software Requirements, 3rd Edition with Karl Wiegers is a post from: http://requirements.seilevel.com/blog

I spoke at AgileIndyConf last week. I had a blast. Met lots of great people from Indianapolis, got to hang with people like Christopher Avery (@christopheraver), Ron Jeffries (@ronjeffries), Chet Hendrickson (@chethendrickson), Angela Harms (@angelaharms), Mike Kelly (@ michael_d_kelly), Joe Astolfi (@joeastolfi), and many more. (If I missed you, please add yourselves in the comments because I’m having what other people call a senior moment. I thought I was too young to have them. I guess not!)

I posted my keynote slides on slideshare. The presentation is Agile Managers: The Essence of Leadership. Agile managers perform the leadership work that managers do in any great organization. They are servant leaders.

Matt Block (@devblock) and all of the AgileIndyConf volunteers did an amazing job organizing the conference. It was the first conference, and it was sold out. Way to go, AgileIndy!

Would you like to continue the learning about leading agile in your organization? Gil Broza and I are leading The Influential Agile Leader in Boston, April 8-9, 2013. The early bird registration expires March 14. Join us.

A colleague asked me the following question:

Assume you were asked to assess a software development team from outside of the organization (that might occur as due diligence or some other context), and you had full access to all internal artifacts of the organization, but you were not allowed to talk directly with anyone from inside. To what degree could you evaluate the quality and effectiveness of the software team just from reviewing just their work, without knowing anything else about them?

This is a wonderful question, and it isn't just theoretical. We do consulting engagements in which we review project artifacts before we talk to team members, and we use those reviews to target the questions we will ask when we do in-person interviews. When we look at "artifacts" we look at code, test cases, documents, drawings, post it notes, emails, wiki pages, graphs, database contents, digital whiteboard photos -- basically any repository for project data.

We look at the following kinds of questions:

What artifacts exist, and what is their scope? Does the project have artifacts that at least attempt to cover all project activities including requirements, design, construction standards, code documentation, general planning, test planning, defect reporting, etc.? If artifacts are not comprehensive, is there any logic behind what is covered and what isn't?

What is the depth of coverage of the artifacts? Do the artifacts try to document every detail, or are they more general? Is the level of detail appropriate to the kind of work the company does?

Are the artifacts substantative? We often see artifacts that are so generic that they are useless to the project. Sometimes we see unmodified boilerplate presented as project documentation. Related: does it appear that the people creating the artifacts understand why they are creating the artifacts, or does it look more like they’re “going through the motions” without understanding why they’re doing what they’re doing.

What is the quality of the work in the artifacts? For example, are requirements statements well formed? Is there evidence that customers have been involved in formulating requirements? Is there evidence that work is getting reviewed? Do the plans look realistic and achievable? Does the design go beyond just drawing boxes and lines and appear to contain some thought?

How long does it take the organzation to produce the artifacts? It isn’t unheard of for organizations to generate artifacts for the first time when they receive our request to show us their work. These organizations know at some level that they should be creating certain artifacts, but they haven’t been.

How recently have the artifacts been updated? This gives one indication of whether the artifacts are actually being used. We assume that if no artifacts have been updated for the past 6 months, they are most likely being ignored (or were never relevant in the first place).

What evidence do we see that the artifacts are being used? I.e., is the team creating “write-only” documentation that isn’t really serving any useful purpose on the project, or are the artifacts being used?

Are the artifacts readily accessible to the project team via a revision control system, wiki pages, or some other means? If team members don't have ready access to materials, that calls into question the degree to which they can actually be using the materials.

We've worked with so many different companies in so many different industries that we no longer have many preconceived notions of what specific artifacts need to look like. We've seen good organizations with minimal documentation, and we've seen bad organizations with extensive documentation. What we are looking for is, Do the artifacts, considered as a set, show us a project that is being run in an organized, deliberate way--paying attention, and learning from its experience? Or do the artifacts show a project that is chaotic, constantly in crisis mode, and mostly working reactively rather than proactively?

When we do assessments with organizations, occasionally we're surprised to find an organization that is more effective than we would have thought based on our document reviews, but that's the exception, and usually we can draw numerous valid conclusions just by doing software archaeology.

A colleague asked me the following question:

Assume you were asked to assess a software development team from outside of the organization (that might occur as due diligence or some other context), and you had full access to all internal artifacts of the organization, but you were not allowed to talk directly with anyone from inside. To what degree could you evaluate the quality and effectiveness of the software team just from reviewing just their work, without knowing anything else about them?

This is a wonderful question, and it isn't just theoretical. We do consulting engagements in which we review project artifacts before we talk to team members, and we use those reviews to target the questions we will ask when we do in-person interviews. When we look at "artifacts" we look at code, test cases, documents, drawings, post it notes, emails, wiki pages, graphs, database contents, digital whiteboard photos -- basically any repository for project data.

We look at the following kinds of questions:

What artifacts exist, and what is their scope? Does the project have artifacts that at least attempt to cover all project activities including requirements, design, construction standards, code documentation, general planning, test planning, defect reporting, etc.? If artifacts are not comprehensive, is there any logic behind what is covered and what isn't?

What is the depth of coverage of the artifacts? Do the artifacts try to document every detail, or are they more general? Is the level of detail appropriate to the kind of work the company does?

Are the artifacts substantative? We often see artifacts that are so generic that they are useless to the project. Sometimes we see unmodified boilerplate presented as project documentation. Related: does it appear that the people creating the artifacts understand why they are creating the artifacts, or does it look more like they’re “going through the motions” without understanding why they’re doing what they’re doing.

What is the quality of the work in the artifacts? For example, are requirements statements well formed? Is there evidence that customers have been involved in formulating requirements? Is there evidence that work is getting reviewed? Do the plans look realistic and achievable? Does the design go beyond just drawing boxes and lines and appear to contain some thought?

How long does it take the organzation to produce the artifacts? It isn’t unheard of for organizations to generate artifacts for the first time when they receive our request to show us their work. These organizations know at some level that they should be creating certain artifacts, but they haven’t been.

How recently have the artifacts been updated? This gives one indication of whether the artifacts are actually being used. We assume that if no artifacts have been updated for the past 6 months, they are most likely being ignored (or were never relevant in the first place).

What evidence do we see that the artifacts are being used? I.e., is the team creating “write-only” documentation that isn’t really serving any useful purpose on the project, or are the artifacts being used?

Are the artifacts readily accessible to the project team via a revision control system, wiki pages, or some other means? If team members don't have ready access to materials, that calls into question the degree to which they can actually be using the materials.

We've worked with so many different companies in so many different industries that we no longer have many preconceived notions of what specific artifacts need to look like. We've seen good organizations with minimal documentation, and we've seen bad organizations with extensive documentation. What we are looking for is, Do the artifacts, considered as a set, show us a project that is being run in an organized, deliberate way--paying attention, and learning from its experience? Or do the artifacts show a project that is chaotic, constantly in crisis mode, and mostly working reactively rather than proactively?

When we do assessments with organizations, occasionally we're surprised to find an organization that is more effective than we would have thought based on our document reviews, but that's the exception, and usually we can draw numerous valid conclusions just by doing software archaeology.

For the last few months I've been programming a system in Go, so I'm always on the lookout for information to feed my confirmation bias. An opportunity popped up when Iron.io wrote about their experience using Go to rewrite IronWorker, their ever busy job execution system, originally coded in Ruby.

The result:

Here is our monthly selection of interesting knowledge material on programming, software testing and project management: Blog: Supporting Fast-Moving Business Requirements Using Approval Branching Blog: Cynical Agile and Scrum Dictionary Blog: Real-time QA: staying confident with code Blog: Ten Things I Hate About Object-Oriented Programming Article: The Estimation Net Article: State of Mobile Testing 2013 Article: Functional-Style Programming in C++ Article: Managing Virtual Software Development Projects Article: Does Scrum Eliminate Project Risk? Tools: Metrics – a Java library giving you insight into production code Tools: Lumbar – a modular JavaScript build tool Video: So You Are Automated… Now Let’s Make it Fast! Video: ...