Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Setting up keys to sign emails in Samsung’s Android email app

Gridshore - Mon, 09/30/2013 - 17:37
Introduction For almost half a year now, I’ve been the proud owner of a Samsung Galaxy SIII Mini (bought it just before the release of the S4, because my phone died and I couldn’t wait for the S4). Since then I’ve got it doing most of what I want it to do, except sign my outgoing emails when I want it to (sign them cryptographically, obviously — I got it to add a text signature within two seconds). The problem here is that setting up the Samsung stock mail app (I don’t use the GMail app) is not immediately obvious. But today I finally got it working, after a long and frustrating day. Read on to find out how…

To sign or not to sign… First of all, let’s take a look at the basic infrastructure for securing your outgoing mail in Samsung’s mail client. This infrastructure is found in the mail application’s settings, which are accessed using the menu key once you start the mail client: OpenSettings.png After you access the settings, find the security options item and tap that:

Screenshot_2013-09-30-17-41-20.png

You should now see a screen like this: Screenshot_2013-09-30-17-41-29.png Hooray, you can manage keys that allow you to sign and/or encrypt your mails!! But this is where things start to get awkward. There are two competing standards out there (both endorsed by the IETF) for signing and encrypting mail. First, there is S/MIME, which uses the same PKI interface also used to secure web traffic and which requires yu to use RSA keypairs and signed certificates. On the other hand there is Pretty Good Privacy (PGP) which uses many types of keypairs, keyservers and a web of trust. So the first question that you run into here is: which do you use? The answer to that is that you use PGP, because S/MIME is not supported by this mail client except for Exchange servers. But you have to dig long and hard on the web to find that out, because there is no official documentation to tell you that. So your next move is going to be to use a tool like GPG to generate your public/private keypair with a passphrase, publish it on a server if you wish and export the public and private keys as .ASC files. After that, you can follow the instructions you find all across the web to place these files in the root of your SD card and import the keys. Which you do by going to Private keys or Public keys in the menu shown above, hitting the menu button and selecting Import keys. And then you will discover that this does not work because no key file is found. You see, for some bizarre reason Samsung chose not to use the onboard key management facilities of Android to manage their keys, instead opting to roll their own. To import the keys into the Samsung mail client, place your key files on your SD card in the directory
/opengpg/export
Yes, that is correct, export. Then, make sure your keyfiles have the correct name. They should be called
<your email address>_<your name as you filled it in in the mail account settings>_0x<the ID of your PGP keypair>_Private_Key.asc

and

<your email address>_<your name as you filled it in in the mail account settings>_0x<the ID of your PGP keypair>_Public_Key.asc

respectively for the private and public keys. If you use other names, the mail app will not find them. You can generate an example if you want: in the mail app, use the Create keys option and export the keys to see what the names look like. You’ll have to get the ID from your GPG tool.

After all that, you should be able to import your keys. Then use the Set default key option to choose a default keypair. You can either select to sign all your mails, or you can use the settings per mail to sign and/or encrypt. Don’t lose your passphrase, you have to fill it in every time you sign a mail!

The post Setting up keys to sign emails in Samsung’s Android email app appeared first on Gridshore.

Categories: Architecture, Programming

Cloning an OpenSuSE 12.3 virtual machine using GRUB2, VirtualBox and cryptfs

Gridshore - Mon, 08/19/2013 - 10:49
Introduction

So I just bought myself a new laptop (an Asus N76VB-T4038H) which I love so far. It’s a great machine and real value for money. Of course it comes preloaded with Windows 8 — which I hate so far, but am holding on to for the occasional multimedia thingie and because I might have to install Microsoft Office at some point. But my main OS is still Linux and in particular OpenSuSE. And given the way I want to use this laptop in the near future, the idea hit me that it would be a good idea to virtualize my Linux machines so that I can clone them and set up new and separate environments when needed. So I installed VirtualBox on my guest OS, downloaded the OpenSuSE 12.3 ISO, set up a new machine to create a golden image, pointed it to the ISO and installed OpenSuSE. Smooth sailing, no problems. Even set up an encrypted home partition without any pain. And then I tried to clone the machine in VirtalBox and start the clone… and found that it couldn’t find my cloned hard drive.

This is actually a reasonably well-known problem in OpenSuSE and some other Linux distros. You’ll find another blog about what is going on here and there are several forum posts. But none of the ones I found cover a solution involving GRUB2, so I thought I’d post my experiences here.

The problem

The basic problem is this: Linux has a single file system tree which spans all of the available hard drives, partitions and so on in one, large virtual structure. To make this happen different parts of the physical storage infrastructure of your machine are mapped to branches of the file system tree (so called mount point). So, for example, the /home directory in your file system may be designated a mount point for the second partition on you primary SCSI drive. Which means that you can type /home and under water the OS will start looking at the partition also known as /dev/sda2. This trick can be applied at any time by the way, using the mount command. This is also what happens, for instance, when you insert a USB drive or DVD: a previously normal directory like /media/USB suddenly and magically becomes the file system location for the USB drive.

Now, recently, Linux has acquired different ways of naming partitions and drives. It all used to be /dev/hdaX and /dev/sdaX, but nowadays several partitions have introduced additional naming schemes using symlinks. For example, on my system, there is a directory /dev/disk which includes several subdirectories containing symlinks to actual device files, all using different naming schemes:

bzt@linux-akf6:/dev/disk> ll
total 0
drwxr-xr-x 2 root root 260 Aug 19 11:27 by-id
drwxr-xr-x 2 root root 140 Aug 19 11:27 by-path
drwxr-xr-x 2 root root 120 Aug 19 11:27 by-uuid
bzt@linux-akf6:/dev/disk>

The by-id directory for instance includes symbolic names for the device files:

bzt@linux-akf6:/dev/disk> ll by-id/
total 0
lrwxrwxrwx 1 root root  9 Aug 19 11:27 ata-VBOX_CD-ROM_VB2-01700376 -> ../../sr0
lrwxrwxrwx 1 root root  9 Aug 19 11:27 ata-VBOX_HARDDISK_VB3ceba069-ef9ac38e -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 19 11:27 ata-VBOX_HARDDISK_VB3ceba069-ef9ac38e-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 19 11:27 ata-VBOX_HARDDISK_VB3ceba069-ef9ac38e-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 19 11:27 ata-VBOX_HARDDISK_VB3ceba069-ef9ac38e-part3 -> ../../sda3
lrwxrwxrwx 1 root root 10 Aug 19 11:27 dm-name-cr_home -> ../../dm-0
lrwxrwxrwx 1 root root 10 Aug 19 11:27 dm-uuid-CRYPT-LUKS1-22b6b0f3ad0c433e855383ca2e64bef1-cr_home -> ../../dm-0
lrwxrwxrwx 1 root root  9 Aug 19 11:27 scsi-SATA_VBOX_HARDDISK_VB3ceba069-ef9ac38e -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 19 11:27 scsi-SATA_VBOX_HARDDISK_VB3ceba069-ef9ac38e-part1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Aug 19 11:27 scsi-SATA_VBOX_HARDDISK_VB3ceba069-ef9ac38e-part2 -> ../../sda2
lrwxrwxrwx 1 root root 10 Aug 19 11:27 scsi-SATA_VBOX_HARDDISK_VB3ceba069-ef9ac38e-part3 -> ../../sda3
bzt@linux-akf6:/dev/disk>

Some of these mount points are fixed, meaning that the operating system automatically remounts them on system reboot. These mount points are recorded in the /etc/fstab file (file system table, a tab-separated table of the information needed to mount the mount points) and in the boot loader (because the boot loader has to know where to start the operating system from). The boot loader files are in /boot/grub2 if you are using GRUB2.

Now, some Linux distributions (including OpenSuSE 12.3) have chosen to generate these configuration files using the new, symbolic names for device files (found in /dev/disk/by-id) rather than the actual device file names (e.g. /dev/sda2). Which is usually no problem. Except when you are on a virtual computer which has been cloned and is using a virtual disk which has also been cloned. Because when you clone a disk, its symbolic name changes. But the configuration files are not magically updated during cloning to reflect this change.

The solution

The solution to this problem is to switch back to the device file names (because those are correct, even after cloning).

To make your Linux virtual machine with GRUB2 clone-ready, perform the following steps:

  1. Take a snapshot of your virtual machine.
  2. Become root on your machine.
  3. Examine the /etc/fstab file. Find all the symbolic names that occur in that file.
  4. Refer to the /dev/disk/by-id directory and examine the symlinks to figure out which device file is equivalent to which symbolic name.
  5. Use vi to edit the /etc/fstab file.
  6. Replace all symbolic names with the correct device file names.
  7. Save the new /etc/fstab file.
  8. Go to the /boot/grub2 directory.
  9. Make the same changes to the device.map and grub.cfg files.
You should now be able to clone the machine and boot the clone and have it find your drives. Encrypted partitions

There is one exception though: if you have encrypted partitions, they still will not work. This is because Linux uses some indirection in the file system table for crypted file systems. To get your encrypted partitions to work, you have to edit the /etc/crypttab file and make the same symbolic-for-device-file name substitutions there.

The post Cloning an OpenSuSE 12.3 virtual machine using GRUB2, VirtualBox and cryptfs appeared first on Gridshore.

Categories: Architecture, Programming

Thought after Test Automation Day 2013

Henry Ford said “Obstacles are those frightful things you see when take your eyes off the goal.” After I’ve been to Test Automation Day last month I’m figuring out why industrializing testing doesn’t work. I try to put it in this negative perspective, because I think it works! But also when is it successful? A lot of times the remark from Ford is indeed the problem. People tend to see obstacles. Obstacles because of the thought that it’s not feasible to change something. They need to change. But that’s not an easy change.

After attending the #TAD2013 as it was on Twitter I saw a huge interest in better testing, faster testing, even cheaper testing by using tools to industrialize. Test automation has long been seen as an interesting option that can enable faster testing. it wasn’t always cheaper, especially the first time, but at least faster. As I see it it’ll enable better testing. “Better?” you may ask. Test automation itself doesn’t enable better testing, but by automating regression tests and simple work the tester can focus on other areas of the quality.

images

And isn’t that the goal? In the end all people involved in a project want to deliver a high quality product, not full of bugs. But they also tend to see the obstacles. I see them less and less. New tools are so well advanced and automation testers are becoming smarter and smarter that they enable us to look beyond the obstacles. I would like to say look over the obstacles.

At the Test Automation Day I learned some new things, but it also proved something I already know; test automation is here to stay. We don’t need to focus on the obstacles, but should focus on the goal.

Categories: Testing & QA

Polyglot Background Jobs

Engine Yard Blog - Tue, 06/25/2013 - 20:05

There's many things we end up needing to perform background jobs for; but the main reason is to provide a snappy, non-blocking user experience.

Whether that task is encoding a video file, batch data import, or (in one case I ran into) jabber instant messaging, we want to offload them from our web servers as quickly as possible.

There are lots of tools to accomplish this across all languages, including Resque, Sidekiq, delayed_job, node-schedule, beanstalkd, Amazon Simple Queue Service (SQS) and then there is my personal favorite: Gearman.

Gearman has client libraries in C, PHP, Ruby, Node.js, Python, Java, Perl, C#/.NET and even includes tools that can be called via shell script, and user-defined functions for both MySQL and PostgreSQL.

Gearman itself is written in C, and is super simple. If you get a chance, I highly recommend checking out the source code. Note: gearman was originally written in Perl and later re-written in C. Be sure not to use the perl version (e.g. dev-perl/Gearman* in Gentoo portage).

The main reason I like gearmand is it's simplicity. Gearman has three parts to it:

  1. GearmanClient submits tasks to the job queue

  2. gearmand is the job queue itself (running as a daemon)

  3. GearmanWorker retrieves the tasks from the job queue and handles them

Gearman Communication Diagram-1

(View Large)

By default, the Gearman queue is stored in memory, however you can also make it persistent and stored in MySQL, PostgreSQL, memcached or SQLite. With memcache, obviously if it's on the same machine as gearmand then you're likely to lose it just as easily as the regular queue. The only difference is that you could re-start gearmand without losing the queue.

However, another potential option is to use the new MySQL 5.6 NoSQL Interface, which supports the memcached protocol. This should be faster than using the Gearman MySQL backend without sacrificing the persistence it brings.

It obviously has the ability to run background jobs being as this is what this post is all about, but it also foreground jobs which allow the GearmanClient and the GearmanWorker to communicate with each other using gearmand as the middle-man.

The best thing about Gearman, is that you can use different languages for different pieces. So you build your website in PHP, but maybe it's not the best option for wrangling text; so you schedule a job with gearmand, and a Python worker picks it up. Or Ruby, or Node.js, or… you get the idea.

What this allows us to do is to pick the correct tool for every task in our stack. Why workaround the pitfalls of our primary language when you can simply pick up a better tool and do things right.

Using Gearman

First we are going to use PHP to schedule a task with the job queue. This uses the pecl/gearman extension.

function createBackgroundJob($task, $data = array()) {
    $client = new \GearmanClient();
    $client->addServer(/* Defaults to 127.0.0.1, 4730 */);
    $handle = $client->doBackground($task, json_encode($data));

    if ($client->returnCode() != GEARMAN_SUCCESS) {
        return false;
    }

    return $handle;
}

In this simple example we create an instance of the \GearmanClient class, tell it to connect to the default server (localhost:4730) and send a background task ($client->doBackground()).

Next we ensure that the task was added successfully, and return the job handle.

We might call it with something like this, passing in the username:

$handle = createBackgroundJob('sendWelcomeEmail', ['username' => 'dshafik']);

We would then want to store the handle so that we can later check the status of the task.

The Worker

Next we'll create a worker, this time in Ruby:

require 'rubygems'
require 'gearman'
require 'json'

servers = ['localhost:4730']
worker = Gearman::Worker.new(servers)

# Add a handler for the "sendWelcomeEmail" task
worker.add_ability('sendWelcomeEmail') do |data,job|
    data = JSON.parse data
    user = User.first(:conditions => [ "username = ?", data["username"] ])
    user.sendWelcomeEmail();
end
loop { worker.work }

Here we use the gearman-ruby gem to create a Gearman::Worker, and then register the task handler.

In this case, we first decode the JSON data passed in from our GearmanClient and then find our user in the database by the username. We then call the sendWelcomeEmail method.

For something that takes more time, you could send back a running status. The job variable is an instance of Gearman::Worker::Job class which allows you to respond using job.report_status(numerator, denominator).

It’s important to note that you can run as many workers for each task as you’d like, Gearman will not hand the same job to multiple workers (however, there is a re-try config option should it fail) and because they are pulling jobs it will not overload the workers, though you may run out of them. The number of workers you run can also act as a way to manage priority — higher priority jobs get more workers — and balance resources.

Checking the Status

Finally, we'll need a way to check the status of the request. For this we’ll use Node.js/Javascript. In our case we are only looking to see if the job has completed as we haven't send any other status.

var http = require('http'), 
    url = require("url"),
    querystring = require("querystring"),
    gearman = require("gearman");

var server = http.createServer(function (request, response) {
    var client = gearman.createClient();
    var query = url.parse(request.url, true).query;

    if (!("handle" in query)) {
        response.writeHead(404, {"Content-Type": "text/plain"});
        response.end("Job not found!\n");
    } else {
        var status = { };
        client.getJobStatus(query.handle, function(s) { 
            if (s) {
                status = s;
            }

            response.writeHead(200, {"Content-Type": "application/json"});
            response.end(JSON.stringify(status));
        });
    }
});

server.listen(8000);

This creates an HTTP server on port 8000 that when passed a handle via GET arguments will return the status.

Using Gearman with Engine Yard Cloud

In order to make Gearman a part of the background job processes on your Engine Yard Cloud account, it is necessary to create a custom chef recipe to compile it yourself (chef recipes can be used to take advantage of software outside of the current stack). For more details on using Chef with Engine Yard Cloud, check out our knowledge base.

As with all background jobs, best practices recommend Gearman be run on an Utility Instance, so that all issues are processed without interfering with the Application Instances themselves.

Can't we all just get along?

So, as you can see, Gearman can act like glue between the various parts of your application. It's super fast, has low resource usage and can be used with almost any language you can think of.

Additionally, it can not only do foreground tasks (with communication), but can also prioritize jobs into high/standard/low priority queues.

You can also easily scale Gearman as the clients and workers both support multiple servers, allowing you to spread your queue, and your workers out over multiple machines.

I highly recommend checking it out at http://gearman.org.

 

The post Polyglot Background Jobs appeared first on Engine Yard Developer Blog.

Categories: Programming

June 21, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 06/21/2013 - 22:38

We've been very busy on a number of exciting projects that we're looking forward to sharing with you all. While I can’t tell you what they are just yet, I can point you to some adorable animal photos.

Have an awesome weekend!

--Tasha Drew, Product Manager

Engineering Updates

PHP customers will be excited to know we’ve released Composer support! Composer is a dependency manager for PHP and allows developers to specify project dependencies in a composer.json file - Composer then handles the rest. Big props to the legendary Ben Chapman for his work getting this ready -- read all about it in our docs!

We’ve released a feature to improve snapshot management for our customers into Early Access. This feature allows you to see snapshots attached to your environments and delete them. You can also set a default policy for how long you want to keep snapshots around via the UI. Our default limit is now 90 days, if you enable the feature, so if you had a stopped environment with some super old snapshots in it, talk to support before you enable it. Hopefully this will make snapshot management significantly more easy and transparent!

Data Data Data

This week Ines and team worked on some deep diving projects dealing with how backups will be performed on new relational clusters going forward. This super forward focused work will eventually begin to reveal itself as we reveal new cluster types, enhancements, and a few new features.

Also, for the Postgres fans out there, Tom Lane’s excellent SF-PUG presentation on The Architecture of the PostgreSQL query planner is up!

Social Calendar (Come say hi!)

Tuesday June 25th, 6:30pm: San Francisco Office: Let’s meet up and talk about LevelDB and Node! Speakers this week include Dominic Tarr, Rod Vagg, Jake Verbaten, Paolo Fragomeni, and Mikeal Rogers.

Tuesday June 25th, 6:30pm: Buffalo Office: Girl Develop It! is teaching an Intro to HTML & CSS course, the third in a series of four.

Wednesday June 26th, 6:30pm: Portland Office: We will be hosting Coder Dojo for students K-12 to learn about software. Parents welcome to attend and participate!

NodeConf (June 27-29, Walker Creek Ranch, CA). We’re sponsoring - if you see Engine Yard t-shirts, come by and say hello! We’re excited to be sponsoring NodeConf for the first time, summer camp style.

Thursday June 27th, 6:30pm: Dublin, Ireland Office: Node.js Dublin will be investigating all things Node. Grab a ticket here!

EuRuKo (June 29-30, Athens, Greece). Stop by the Engine Yard booth! Grab a t-shirt and some swag, learn what’s new with Cloud and meet our awesome community manager, Kelsey Schimmelman.

Lonestar PHP (June 29-30, Dallas, TX). We’re excited to be returning to Lonestar PHP--if you see Davey Shafik walking around, say hi!

Articles of Interest

Google finally admits that those crazy brain teasers do not, in fact, indicate anything  about how good a hire an engineer is going to be. But they’re definitely fun to dream up.

The post June 21, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

How to Troubleshoot PostgreSQL Alerts

Engine Yard Blog - Fri, 06/21/2013 - 17:17

So you have your PostgreSQL application deployed on Engine Yard Cloud and everything is going great. You have enabled a few extensions, have added basic redundancy by spinning a database replica, and are busy developing new features.  One day though, you look at the dashboard and see this message:

What do these alerts mean? Is the database at risk? Should you escalate to support? This post will help you understand PostgreSQL dashboard alerts and correlate them to the health of your database and application.

Monitoring and the checkpoint check

The alerts I showed you popped up in one of our mission-critical applications. This blog discusses the steps I performed to troubleshoot the cause of the problem and the resolution. But first a little bit of background.

We monitor the health of your PostgreSQL database using a combination of our own custom checks and Bucardo’s check_postgres scripts. I’ll wave a big wand here and tell you that either Collectd or Nagios (depends on your stack and features) consume the results of these checks and present them to the Engine Yard dashboard.

The following documentation page provides an explanation of the alerts Engine Yard issues for PostgreSQL. Today I’ll focus only on the alert I received but refer to the documentation if you see something different in your application’s dashboard.

Let’s take a closer look at the message:

POSTGRES_CHECKPOINT CRITICAL: Last checkpoint was 16204 seconds ago

I know from this message that the checkpoint check originated the alert and the severity of the alert is critical. In human talk, the message means that the database has not had a checkpoint for about 4.5 hours!  Here is another example:

POSTGRES_CHECKPOINT WARNING: Last checkpoint was 1265 seconds ago

This message means that the database has not had a checkpoint for about 21 minutes.

We issue a WARNING severity when checkpoint delays range from 20 to 30 minutes. For anything that exceeds 30 minutes, the severity of the alert goes to CRITICAL.

A checkpoint is a point in the transaction log sequence at which all data files have been updated to reflect the information in the log and flushed to disk. If your system crashes recovery will start from the last known checkpoint. So the checkpoint check helps us confirm two things: that your database consistently takes forward the position in which recovery is started, and in the case of replicas that your standby is keeping up with its master (since the activity the replica sees is what the master has sent it).

For more information about checkpoints and replication, please refer to the Postgresql replication write-ahead-log (WAL) documentation.

Back to my App

Now we understand that the alert I received means that there was a problem with the database replica and its ability to checkpoint. The database logs showed nothing out of order, so I logged into the server console and discovered the following:

# psql
psql (9.1.9, server 9.1.3)
Type "help" for help.

The psql prompt showed me that there was a version mismatch between the database server binaries and the running psql process. This typically happens after a stack update (that includes a minor version bump of your database) is applied on a running environment, and the database process is not restarted.  The database server is left in a state where its effectively running two versions at the same time. To ensure that the postgresql process is running the latest version of the database, you MUST always restart the database process after upgrading your environment.

The stack and version update was absolutely necessary as it included critical security patches - See April 4, 3013 - PostgreSQL security update. But when the stack was applied, the person who applied the stack update didn’t restart the postgresql process as outlined in the upgrade instructions. This is typically not a problem, as replication is known to work between patch level versions, but we hit a replication bug in the 9.1.3 to 9.1.9 upgrade which caused replication to break.

Our Solution

So in a nutshell, our database replica became unable to receive WAL archives from its master,  checkpoints started falling behind, and we were alerted. Restarting the database process would have solved the problem but instead we decided to utilize the maintenance window to upgrade the server to PostgreSQL 9.2 and create a new replica.

I performed an in-place upgrade of the database master (something that professional services has a lot of experience with) and within minutes the application was back online running the latest version of PostgreSQL.

But troubleshooting this alert made me aware of the issues with our current upgrade process:

  • Documentation on alerts was lacking. There is no place to quickly look up the alerts we present in the UI and their meaning.
  • Our upgrade message did not remind us to restart the database process (though the release notes did).
  • An unexpected replication bug between patch versions caused my database replica to become stale.

Here is what we we’ll do to make sure you don’t experience the problems I did last week.

New PostgreSQL alert documentation

PostgreSQL alerts will be explained in a new documentation page. We’ll work on documenting MySQL and Riak alerts as well.

Improved stack upgrade messages

We will enhance stack release notes with icons to visually indicate if a process restart is needed when a new version of a database is available.


Ability to lock your database version

Without a doubt, we want customers to keep their database stacks up to date with security releases and patches. But it would be fantastic to be able to lock your entire database version (to the patch level) and still receive stack updates.

We have developed (and are internally testing) a toggle to lock your database version. With this feature, I can schedule a maintenance window (to restart the database process when I’m ready) while continuing to receive stack updates. We are still working on documentation but if this feature is something that interests you, please open a support ticket and let me know. It should be in limited access soon!

Hopefully now you have a little more context and information available to interpret the alerts we display in your environment. Exciting things are happening in the Engine Yard’s Data stack (think new clusters!).  A little hint for the curious, Tasha Drew’s excellent weekly recap of engineering always includes juicy details on what we are up to ;)

The post How to Troubleshoot PostgreSQL Alerts appeared first on Engine Yard Developer Blog.

Categories: Programming

Announcing Composer Support

Engine Yard Blog - Tue, 06/18/2013 - 21:55

We’re pleased to announce Composer support for PHP applications.

This has been one of our most requested features, and should make it even easier for you to manage your apps. If you’re already using Composer, you can dive right in. If not, now is a great time to try it out. We recommend Composer for all PHP apps!

What is Composer?

Composer is a popular dependency manager for PHP. With it, you can specify project dependencies in a composer.json file and Composer will automatically handle the rest. For more information about Composer, take a look at the project website.

Why is It Useful?

Composer allows you to manage third-party dependencies separate from your code, decluttering your repository. What’s more, it makes updating your dependencies a snap. Just run composer update and Composer will fetch the latest compatible versions.

How Can I Use It?

Using Composer with Engine Yard is very simple. We’ll detect the presence of a composer.lock file in your repository, and automatically install your app’s dependencies. To get started with Composer for Engine Yard, take a look at the documentation.

The post Announcing Composer Support appeared first on Engine Yard Developer Blog.

Categories: Programming

June 14, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 06/14/2013 - 17:17

This is the week a big chunk of the San Francisco development team went on a roadtrip to our Portland office to do some intense cross-office feature pollination. Things may have started out with some office rivalry, but developers quickly overcame any differences to work together to build, drink copious amounts of amazing coffee, and figure out the location of some of the awesome restaurants Portland has to offer. Pro-Tip: check out Blue Star donuts #amazing.

--Tasha Drew, Product Manager

Engineering Updates

Customer feedback is important to us and is an important part of how we prioritize work within our product management process. We received a few comments from customers who were frustrated because they couldn’t figure out why they were being charged money when they didn’t have any running instances. The answer was that they still had IP addresses that were detached from instances when the instances were terminated, but not deleted.

Customers can always see IP addresses and manage them in the dashboard by going to Tools -> IP addresses, but we decided to add more messaging to call this out to people.

Going forward, you will see a dashboard notice if you delete an instance and don’t delete the IP address - and you will also receive an email. We will also be sending out emails to any customer who has an account where the only items they’re being billed for are IP addresses and snapshots to let them know.

Hope this helps going forward! Big thanks to one of our newest platform developers, the amazing Daniela, for turning this request around so quickly.

We’re also wrapping up some cool new features around snapshot management which you should be reading about in this space next week!

Data Data Data

Our lead data engineer, Ines, has been busy working on the underlying code for exciting new features that we’ll be rolling out in the next few months. She also handed off a new feature that allows for database version locking to alleviate upgrade pains. The DBA team is actively testing and improving it and we should make it available soon. Watch out for her blogpost next week.

Ines and I were delighted to get to meet up with local Postgres ladies while we were in Portland. Selena Deckelmann has some great thoughts on the intersection of developers and Operations on her blog for those of you who need some fun weekend reading.  Kris Pennella gave me a valuable reminder to take a deep breath when facing stressful situations in her blog, “3 Tips Channeling a Negative into a Positive.”

We also had the pleasure of seeing Basho’s Eric Redmond (author of 7 Databases in 7 weeks and the Little Riak Book). We got a chance to hear some of the features that will come in Riak 1.4 and we are very excited!

Social Calendar (Come say hi!)

Friday June 14 - Saturday June 15: DevOps Days Amsterdam!: Meet the always charming Slava and the ridiculously knowledgable Richard as they hang out and participate in this awesome DevOps conference where we are not only a PaaS -- we are also a cake.

Tuesday June 18, 19:00: Ruby Ireland Meetup at Engine Yard Dublin. We are Going off The Rails this month at Ruby Ireland as we go through some of the options for extending your web apps with mobile apps or through a Javascript framework. Kevin Fagan, Fergal Condron, Simon Rand, Gavin Joyce and Paul Watson will be speaking.

Thursday June 20 - Friday June 21st: Lyon, France, Ruby Lugdunum: Crowd favorite Engine Yard engineer PJ Hagerty will be presenting at Ruby Lugdunum in exotic Lyon, France, on how to grow and nurture your local Ruby group.

Thursday June 20, 18:30: Open Data Ireland #8 at Engine Yard Dublin. General theme for ODI Meetup #8 is 'Open Government Partnership'. This meetup will be facilitated by Denis Parfenov, Tom Stewart and Nuala Haughey. We'll be hosting a brief presentation from OGP representative. The rest of the evening will be dedicated to building topic- specific, multi-stakeholder/multi-disciplinary working groups with a view to taking an active part in co-drafting/crowdsourcing Ireland’s first national Action Plan around OGP principles.

Thursday June 20, 19:00: Engine Yard’s Buffalo Offices: Riak is bustin’ out all over in June, a meetup led by renouned Riakifier Dave Parfitt.

Articles of Interest

Drink coffee: avoid death! The New York Times tells us exactly what we’ve been hoping to hear.

Nobody Understands the GIL: Jesse Storimer explores MRI and analyzes functions for thread safety.

And for our distributed systems fans (that’s everyone, right?) a deep dive into non-blocking transactional atomicity by Peter Bailis.

Call me maybe: Kyle Kingsbury’s summary post on Jepsen looking at how various databases handle network partitions.

The post June 14, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

You Cannot Win Engineering

Engine Yard Blog - Thu, 06/13/2013 - 20:51

For as long as I can remember, I’ve been a fan of Saturday Night Live and improvisational theater. Improv looks chaotic and uncontrolled, but the best practitioners operate under strict rules that govern interactions between players. Some of the most successful entertainers today, people like Stephen Colbert and Tina Fey, directly credit what they have learned in improv with making them better at what they do both on and off screen.

Unlike workplace policies that you are probably used to, the rules of improv aren’t meant to constrain you, but to open you up to the ideas of others. Let’s take a look at some of the rules the Mr. Colbert and Ms. Fey live by and see how they can improve team collaboration.

Agree and Say “Yes”.

Here’s Tina, from her book Bossypants, talking about the rules of engagement:

The first rule of improvisation is AGREE. Always agree and SAY YES. When you’re improvising, this means you are required to agree with whatever your partner has created. So if we’re improvising and I say, “Freeze, I have a gun,” and you say, “That’s not a gun. It’s your finger. You’re pointing your finger at me,” our improvised scene has ground to a halt. But if I say, “Freeze, I have a gun!” and you say, “Yes! The gun I gave you for Christmas! You bastard!” then we have started a scene because we have AGREED that my finger is in fact a Christmas gun.

The same is true of engineering teams. When one of your teammates has an idea, your first response needs to be affirmative. Take any and all ideas from your teammates as positive contributions and you start from a place of being open-minded and welcoming. Nothing kills team morale faster than someone who says “No, that won’t work” in response to any idea that they didn’t come up with.

It’s Not Just “Yes”, it’s “Yes, and…”

Everyone loves games and games are more fun when everyone  plays nicely. Make positive contributions and you will foster a spirit of openness, collaboration and — dare I say — fun.  Make it your habit to answer your teammate’s ideas with “Yes, and…” instead of “No, because”. Always offer your ideas, you just are as entitled to be silly and wrong as everyone else. Ideas seldom spring fully-formed from the head of Zeus and the part you’re holding back out of fear might be the thing that makes it work. “Yes, and…” makes you part of the solution; “No, because” makes you part of the problem.

Your Team is the Most Important Person on Your Team

 Stephen Colbert went back to his alma mater, Northwestern University, to give the commencement address in 2011. He may play a know-it-all blowhard on The Colbert Report, but that’s clearly not the case in real life. Here’s an excerpt from his speech:

…One of the things I was taught early on is that you are not the most important person in the scene. Everybody else is. And if they are the most important people in the scene, you will naturally pay attention to them and serve them. But the good news is you're in the scene too. So hopefully to them you're the most important person, and they will serve you. No one is leading, you're all following the follower, serving the servant.

You cannot win improv.

And life is an improvisation. You have no idea what's going to happen next and you are mostly just making things up as you go along.

And like improv, you cannot win your life.

The software corollary to this is: “You cannot win engineering”.

Think about the implications of this for a moment. If everyone on your team acts as if their teammates are more important than they are, you create an environment of support, giving, and progress that is mutually enriching and productive. You’ll know you have succeeded when no one on your team remembers where a great idea came from. More importantly, no one will care.

When one of your teammates asks you a question, don’t tell them to Google it (which is a bit of a jerk response in any case). Act as if their problems are more important than yours, serve the team by serving them. When you are stuck on a problem, they will treat you the same way.

None of these rules for improvisation will make you funnier or get you a slot on Weekend Update, but applying them to your co-workers will almost certainly make your team awesome. Everyone wins.

The post You Cannot Win Engineering appeared first on Engine Yard Developer Blog.

Categories: Programming

June 7, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 06/07/2013 - 23:41

Things are pretty busy right now as we ship a bunch of customer enhancements on Engine Yard Cloud and continue with our planned infrastructure abstractions and cluster model improvements. Exciting things to come! In the meantime, here’s what’s available as of this week.

--Tasha Drew, Product Manager

Engineering Updates

Now in GA: Application takeover preferences. Based on your application's customizations, you might not want to use the default application takeover behavior we've developed to automatically promote your application slaves when the app master goes away or becomes totally unresponsive for some reason.

Engine Yard Cloud now provides two automated options for replacing capacity in an application takeover situation. We also provide alternatives if you need to handle part or all of an app takeover yourself.

We now have Provisioned IOPs and EBS Optimized instances available for customers to use in Early Access! To enable them for your environment from your cloud dashboard, click the Tools menu -> Early Access, and then enable “EBS Optimized Instances” and “Provisioned IOPs.”

Keep in mind that they work best in tandem, and they will only be an option on instances booted after you enable the feature.

Data Data Data

Databases love I/O and provisioned IOPs and EBS optimized instances are very well suited for applications where the database can use more performance (think backups and snapshots too).  

You can enhance the performance of your application by having a volume with provisioned IOPs on the database master. If your application has been already deployed you can add new replicas to the environment (that have this performance boost) and have them promoted to master.
As usual don’t hesitate to ask us if PIOPs or EBS optimized instances can give your database a boost.

Social Calendar (Come say hi!)

Tuesday, June 11th: Our Buffalo office will be hosting the WNY Ruby Meetup Group. Mark Josef will be providing us with some code katas.

Wednesday, June 12th: Our PDX office will be hosting the weekly CoderDojo K-12 night, ably assisted by one of the San Francisco sprint teams, who will be on site for an off site (as it were).

Wednesday, June 12th: Girl Develop It will be doing a Code and Coffee night in our Buffalo office. The participants be focusing on honing their skills and working in groups. Swing by for the whole thing or just for a part of it.

Friday, June 14th: DevOps Day Amsterdam will be happening! Be sure to meet our own Slava and Richard and let them tell you about how Engine Yard can make your lives easier.

Articles of Interest

Mozilla's John O’Dunn discusses how to use release engineering as a force-multiplier!

David Padilla explains why hash lookups are so fast in Ruby on the Engine Yard blog.

The post June 7, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

Speaking at Conferences: How to write a talk and get it accepted

Engine Yard Blog - Fri, 06/07/2013 - 21:38

At php[tek] 2013, Engine Yard sponsored the Mentorship Summit, a special forum to discuss the value of mentoring to create more connections and advancement opportunities for developers. A common theme that came out of the summit was that speaking at conferences is a great way to further oneself both personally and professionally. During the discussion, inevitably someone said they've submitted numerous times but had never been accepted to speak, then someone else said they don't know how to write a good proposal, and many discussions were had about what it means to write a good talk proposal.

I've had this conversation many times over the past few years with those I mentor, but this year something was different: I was a member of the selection committee for Distill, Engine Yard’s first developer conference.

Being on the other side for once has changed how I think about what is important when submitting for a talk and I thought it might be helpful to share how my perspective has evolved.

Questions and Answers

A proposal typically consists of a title, a short abstract, and sometimes a larger body of text to give the reviewers more detail about what to expect in the talk.

The first two items are really important because in addition to helping your talk stand out to the reviewer, they are also how an attendee will choose your talk over others on the schedule.

It's important to understand that there is an implicit question being asked in the title of your talk. For example, the two talks I presented at php[tek] were:

  1. "PHP 5.5: The New Bits"

  2. "MySQL High Availability, Disaster Recovery and Load Balancing"

The first one has an implicit question along the lines of: What's new in PHP 5.5? and the second: How do I make my database more robust and more scalable?  How do I make it easier to recover from failures?

Your abstract will then further inform the attendee (and the reviewer) of that question, and more importantly, create an expectation, or even a promise of what the talk will cover.

Most negative feedback I hear about talks isn't that the speaker was terrible or didn't know what they were talking about; it’s simply that the talk wasn't what the attendee wanted. Communicating the intent of your talk clearly is essential. If an attendee is disappointed, it’s often either because he misinterpreted the intention of the talk, or the speaker failed to properly set his expectations.

For example, if my MySQL talk focused mainly on using memcache (as a means to reduce load on the DB and therefore reducing the need to scale) it would have been a terrible talk — though a worthwhile topic, it's not addressing the question posed by my title and therefore not meeting the promise of the talk.

However, a talk is not always a matter of simply answering a question. Sometimes you are teaching people how to better ask their question. If the talk had been titled "High performance websites with MySQL", then memcache would certainly be answering some of the implicit question by teaching them how to ask a better question: "How do I make my website faster when using MySQL?"

Is Your Question Relevant?

One of the more important things to remember is to ask the right question. What do people care about? Look at schedules from previous years for the conference (if they exist) to get an idea of what experience level the conference targets, and observe the associated community to see what people are interested in — what are emerging trends and topics that people are fascinated by?

Are you the right person for the job?

A very important thing to realize about your proposal is that who you are matters. To be more specific: why are you qualified to speak on a specific subject, or why do your opinions about it matter?

When selecting talks for Distill, I ran into mostly speakers I was unfamiliar with. So I Googled them.

If you want to establish yourself as a possible speaker, you need to be part of the larger conversation on the subject on which you want to speak. This can take many forms:

  • Blog posts

  • Tweets with people about the subject (yes, that's right, your tweets can matter!)

  • Any other media (podcasts, books, magazine articles)

  • Code contributions

The last one takes the longest to verify and is a last resort — if the reviewer hasn't been grabbed by your title/abstract then they may or may not even bother.

If you are contributing code to projects but doing nothing else, then you should start blogging about what you're contributing.

Submit Early, Submit Often

This one seems like it will be quite controversial to communities other than the PHP community which has made this the norm: submit many proposals.

When I submit to a conference, I will propose no fewer than four proposals. This not only increases your chances (by the numbers), but when selecting talks, if you have absolutely no other information on a speaker, this will at least give the reviewer a better idea of what you do, and a little more about what experience you may have.

Speaking Experience

The ability to communicate as a speaker is also something to worry about when selecting talks. Having a history of speaking, be it at other conferences, user groups, or even via webcasts/podcasts, is a big plus. Be sure to share all the media from your talks: slides, video and audio recordings. Similarly, podcasts are another way to give great insight into your ability to communicate verbally.

However, even writing can give insight into how you communicate.

Hard skills vs Soft skills Talks

There are two types of talks: so-called “hard talks” that teach technical skills, and “soft talks” which teach personal skills (e.g. team working, project management, how to get hired... how to write talk proposals).

When setting up a conference schedule it's very important to get the balance of hard skills vs soft skills talks right. Too few hard skills talks, and it's hard to justify the expense to your employer.

But quite often, the soft skills talks are the ones that get the most people talking, and when you are an experienced developer are often what you need to advance your career.

What this means is that you are far more likely to get a hard skills talk accepted than a soft skills talk — just because of the ratios, and because the few soft skills talks that are selected have to really stand out, typically well established speakers are chosen.

The "Distillation" Process: How We Selected Talks for Distill

For Distill, our review process was what I called "Iron Chef Style". We rated each talk as follows:

  • Content (15 points)

  • Fit (10 points)

  • Speaker (5 points).

The content is the topic covered, the fit is how well the talk fits into the overarching theme of the conference, and the speaker is not who they were, but what made them the right person to talk on the subject.

Once they were all rated by each member of the committee, we tallied the numbers and sorted by the totals.

At this point, we asked if anybody had any specific talks they felt strongly about, and also used the standard deviation to see which talks have the biggest difference between the different reviewers to start discussion on those.

This gave us our top 32. We then looked at any speaker who had multiple talks in the top 32, and made a decision on which one of them we preferred and removed the duplicates (there were only 3).

Finally, as we further refined our theme, we narrowed our list down to the final 15 — who were selected as speakers for Distill.

I was surprised at just how much of an extremely lengthy and difficult task this was. To anyone who has ever done this with my talks… I promise that my bribes will be much bigger in the future, you deserve it!

distill-blogbutton

The post Speaking at Conferences: How to write a talk and get it accepted appeared first on Engine Yard Developer Blog.

Categories: Programming

Hash lookup in Ruby, why is it so fast?

Engine Yard Blog - Tue, 06/04/2013 - 18:00

Note: Our friend and CEO at Crowd Interactive, David Padilla, wrote this great piece about hash lookup in Ruby. Be sure to check out MagmaConf!

Have you ever noticed that in Ruby looking for a specific key-value pair in a Hash is very fast?

Allow me to explain the logic behind hashes in Ruby with a language that you probably understand: Ruby.

Let's imagine for a second that we want to emulate the functionality in hashes because, for some strange reason, they have not been implemented yet.

If we want to have some sort of key-value structure, we'll have to implement it ourselves, so let's get to work.

First, we'll need a Struc to represent a HashEntry, or the key-value objects that we will add to our hashes.

HashEntry = Struct.new(:key, :value)

Now, we'll need a class that represents our Hashes or (to avoid conflicts with the original Hash class) HashTable.

class HashTable
  attr_accessor :bins

  def initialize
    self.bins = []
  end
end

We add the `bins` attribute to the class. `bins` will be an array where we'll store our HashEntry elements.

Now, let's write a method to add a HashEntry to a HashTable. To follow convention, we'll use the traditional `<<` as method name.

class HashTable
  attr_accessor :bins

  def initialize
    self.bins = []
  end

  def <<(entry)
    self.bins << entry
  end
end

Great, now we can add HashEntry elements to our HashTable like so:

entry = HashEntry.new :foo, :bar
table = HashTable.new
table << entry

What if we want to look for an entry by key? Let's write the `[]` method on the HashTable class to handle that.

def [](key)
  self.bins.detect { |entry| entry.key == key }
end

What we're doing here is simply going element by element comparing the given key until we find what we're looking for. Efficient? Let's figure it out.

Benchmarking

We'll use Ruby's benchmarking tools to figure out how much time we're spending looking for elements on our hash tables.

require 'benchmark'

#
# HashTable instance
#
table = HashTable.new

#
# CREATE 1,000,000 entries and add them to the table
#

(1..1000000).each do |i|
  entry = HashEntry.new i.to_s, "bar#{i}"

  table << entry
end

#
# Look for an element at the beginning, middle and end of the HashTable.
# Benchmark it
#
%w(100000 500000 900000).each do |key|
  time = Benchmark.realtime do
    table[key]
  end

  puts "Finding #{key} took #{time * 1000} ms"
end

When we run this benchmark, we get the following results:

Finding 100000 took 33.641 ms
Finding 500000 took 192.678 ms
Finding 900000 took 345.329 ms

What we see here is that lookup times increase depending on the amount of entries and its position within the array or `bins`. This is obviously very inefficient and unacceptable for real life scenarios.

Now, let's see how Ruby tackles this problem internally.

Bins

Instead of using a single array to store all its entries, hashes in Ruby use an array of arrays or "bins".

First, it calculates a unique integer value for each entry. For this example we will use `Object#hash`. Then, Ruby divides this hash integer by the total number of bins and obtains the remainder or modulus. This modulus will be used as the bin index for that specific entry.

When you lookup for a key, you calculate its bin index again using the same algorithm and you look for the corresponding object directly on that bin.

Let's add an attribute on the HashTable class that will determine how many bins each HashTable will have, and we'll initialize it with 500.

class HashTable
  # ...

  attr_accessor :bin_count

  def initialize
    self.bin_count = 500
    self.bins = []
  end

  # ...
end

Now, let's write a method that calculates the bin for a specific entry depending on the number of bins.

class HashTable
  # ...

  def bin_for(key)
    key.hash % self.bin_count
  end

  # ...
end

When storing the HashEntry in the HashTable, we won't just store it on an array, we'll store it on an array that corresponds to the `bins` index depending on what the `bin_for` method returns:

class HashTable
  # ...

  def <<(entry)
    index = bin_for(entry.key)
    self.bins[index] ||= []
    self.bins[index] << entry
  end

  # ...
end

And last, whenever we want to retrieve a HashEntry, we'll recalculate the bin index again using the `bin_for` method and once we have that, we'll know exactly where to look for our entry.

def [](key)
  index = bin_for(key)
  self.bins[index].detect do |entry|
    entry.key == key
  end
end

When we run the same benchmark that we used earlier, we can see times improve dramatically:

Finding 100000 took 0.025 ms
Finding 500000 took 0.094 ms
Finding 900000 took 0.112 ms

Not only did times improve, but we got rid of the variance that we used to have depending on the position of the element in the bin pool.

There's still room for improvement here. Let's add more bins and see what happens.

class HashTable
  # ...

  def initialize
    self.bin_count = 300000
    self.bins = []
  end

  # ...
end

When we run the benchmark we get:

Finding 100000 took 0.014 ms
Finding 500000 took 0.016 ms
Finding 900000 took 0.005 ms

Even more improvement. This mean that the more bins, the less time spent looking for a specific key in a bin.

How many bins does Ruby actually use?

Ruby manages the size of the bins dynamically. It starts with 11 and as soon as one of the bins has 5 or more elements, the bin size is increased and all hash elements are reallocated to their new corresponding bin.

At some point you pay an exponentially increased time penalty while Ruby resizes the bin pool, but if you think about it, its worth the time since this will keep lookup times and memory usage as low as possible.

Further reading

If you want to learn where this algorithm came from and a little more about Ruby internals, I really recommend that you read Pat Shaughnessy's Ruby Under a Microscope book. Pat explains how the Ruby VM works in a way that anyone can understand. No C knowledge required, I really enjoyed reading it.

You can find the working example I used for this example on this gist.

You could also read some of the Rubinius source code. Take a look at their implementation of the Hash class, you'll probably understand a little more of the logic they used after you've read this post and Pat's book.

Thanks for reading.

The post Hash lookup in Ruby, why is it so fast? appeared first on Engine Yard Developer Blog.

Categories: Programming

The Toyota Way: The need for doing it right the first time

After WWII Toyota started developing its Toyota Production System (TPS); which was identified as ‘Lean’ in the 1990s. Taiichi Ohno, Shigeo Shingo and Eiji Toyoda developed the system between 1948 and 1975. In the myth surrounding the system it was not inspired by the American automotive industry, but from a visit to American supermarkets, Ohno saw the supermarket as model for what he was trying to accomplish in the factor and perfect the Just-in-Time (JIT) production system. While accomplishing this low inventory levels were a key outcome of the TPS, and an important element of the philosophy behind its system is to work intelligently and eliminate waste so that only minimal inventory is needed.

As TPS and Lean have their own principles as outlined by Toyota:

  • Long-term Philosophy
  • Right process will produce the right results
  • Value to organization by developing people
  • Solving root problems drives organizational learning

As these principles were summed up and published by Toyota in 2001, by naming it “The Toyota Way 2001”. It consists the above named principles in two key areas: Continuous Improvement, and Respect for People.

the-toyota-way

The principles for a continuous improvement include establishing a long-term vision, working on challenges, continual innovation, and going to the source of the issue or problem. The principles relating to respect for people include ways of building respect and teamwork. When looking at the ALM all these principles come together in the ‘first time right’ approach already mentioned. And from Toyota’s view they were outlined as followed:

  • The right process will produce the right results
    • Create continuous process flow to bring problems to the surface.
    • Use the ‘pull’ system to avoid overproduction (kanban)
    • Level out the workload (heijunka).
    • Build a culture of stopping to fix problems, to get quality right from the first (jidoka)
  • Continuously solving root problems drives organizational learning
    • Go and see for yourself to thoroughly understand the situation (Genchi Genbutsu);
    • Make decisions slowly by consensus, thoroughly considering all options (nemawashi); implement decisions rapidly;
    • Become a learning organization through relentless reflection (hansei) and continuous improvement (kaizen).
Let’s do it right now!

As the economy is changing and IT is more common sense throughout ore everyday life the need for good quality software products has never been this high. Software issues create bigger and bigger issues in our lives. Think about trains that cannot ride due to software issues, bank clients that have no access to their bank accounts, and people oversleeping because their alarm app didn’t work on their iPhone. As Capers Jones [Jones, 2011] states in his 2011 study that “software is blamed for more major business problems than any other man-made product” and that “poor quality has become one of the most expensive topics in human history”. The improvement of software quality is a key topic for all industries.

 Right the first time vs jidoka

In both TPS and Lean autonomation or jidoka are used. Autonomation can be described as ‘intelligent autonomation’, it means that when an abnormal situation arises the ‘machine’ stops and fix the abnormality. Autonomation prevents the production of defective products, eliminates overproduction, and focuses attention on understanding the problem and ensuring that it never recurs; a quality control process that applies the following four principles:

  • Detect the abnormality.
  • Stop.
  • Fix or correct the immediate condition.
  • Investigate the root cause and install a countermeasure.
Find defects as early as possible

In other words autonomation helps to get quality right the first time perfectly. With IT projects being different from the Toyota car production line, ‘perfectly’ may be a bit too much, but the process around quality assurance should be the same:

  • Find the defect.
  • Stop.
  • Fix or correct the error.
  • Investigate the root cause and take countermeasures.

The defect should be found as early as possible to be fixed as early as possible. And as with Lean and TPS the reason behind this is to make it possible to address the identification and correction of defects immediately in the process.

Categories: Testing & QA

May 31, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 05/31/2013 - 23:47

Spent an awesome week at DevOps Day Berlin and visiting customers in Berlin with application engineer extraordinaire Kevin Holler and our fearless sales leader Bridget Gleason. So fantastic to see the exchange of best practices and ideas around DevOps and get a chance to check out the Berlin tech community!

We’re planning to be at DevOps Day Amsterdam in a few weeks too, so hope to see you there!

--Tasha Drew, Product Manager

Engineering Updates

The EngineYard gem 2.1.0 has been released with a bunch of enhanced capabilities, including new asset strategies. We received a lot of customer requests around these areas and hope that this will make you super happy. Read all about it on the change log and the readme on github.

Ruby 2.0 is now in GA on Engine Yard Cloud! Read all about it.

We have released our new hardened Gentoo stack in GA (Gentoo 12.11), with all the latest and greatest packages and features that the new Gentoo has to offer. It only works in new environments, so if you are going to spin up a new environment, check it out!

We've updated the worker counts for Passenger/Unicorn/etc.. and exposed some easy ways for customers to alter the worker counts, memory thresholds, and other cool stuff. Read all about how to do it here! This allows customers to avoid writing custom chef to alter some of this stuff. As always, if you need any help, open a support ticket!

Workers are the processes that allow your application to respond to incoming web requests. Regardless of the application server stack you select, it provides one or more workers per instance to run your application.  Learn all about how to tune your workers per server and use an excellent new calculator to figure out your ideal set up in our new documentation!

Social Calendar (Come say hi!)

Postgres Ireland: June 4, Engine Yard Dublin. Come and meet some Postgres backend developers based in Ireland, and discuss the recent landmark 9.2 release. We encourage participation from people of all levels of experience with Postgres - from people who aren't current users, and would like to know more, right up to experienced users with large installations. Greg Stark, a long-time Postgres contributor and committer, will speak.

Magma Rails: June 5-7, Manzanillo, Mexico. MagmaRails is now MagmaConf!, one of the most important web development conferences in Mexico; with Ruby, Ruby on Rails, Frontend and more web technologies sessions for developers and engineers that build cutting-edge web applications.

GoRuco: New York’s premier community-organized Ruby conference. Stop by the Engine Yard table and say hello!

Articles of Interest

Our inaugural developer’s conference, Distill, on Treasure Island in the San Francisco Bay, still has early bird tickets left. Hope to see you there!

Northwestern University’s engineering students are given dance lessons to help them have more balanced brains.

Fan favorite and lead AP support manager Chris Rigor shares his talking points from his presentation at Ruby Kaigi this week, in which he created a debian wheezy vagrant box with chef recipes to set up a rails app!

And this was my favorite slide of the week, where Meagan Fisher demystifies design process at BTConf.

The post May 31, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

We Love Our Sponsors!

Engine Yard Blog - Fri, 05/31/2013 - 22:15

Over the years, Engine Yard has been a proud sponsor of hundreds of incredible open source conferences. When it came time to seek sponsors for Distill, our inaugural developer conference, we were thrilled and humbled by the amount of love and support we received from companies who also put community first. We are proud to announce our first round of sponsors for this year--without them, Distill would not be possible! AppFirst will be making sure that all talks can be seen on demand after the conference by sponsoring our videos! Get the AppFirst DevOps dashboard and free offering for full stack insights across all of your Engine Yard environments. GitHub, helping us all build software better, together, will be sponsoring the lightning talks portion of our schedule. These rapid, info-packed talks are always a conference highlight and are sure to be meaningful, memorable and entertaining. Kony helps you take your business mobile. They’re also going to be mobilizing attendees by sponsoring the transportation from the conference hotel to Treasure Island and back again. MongoLab takes the sweetest sponsorship--make sure to thank them for the delicious Smitten ice cream you’ll be enjoying at Distill! They provide a fully-managed MongoDB Database-as-a-Service platform automating the operational aspects of running MongoDB in the cloud.

New Relic will be sponsoring our kick-off party, Moonshine. New Relic makes awesome app-monitoring software. Check it out.

SendGrid will be keeping you alert throughout Distill (especially the morning following Moonshine) with excellent coffee from Ritual Coffee Roasters! SendGrid are the masters of simple, reliable email delivery. Zendesk makes customer support painless. Think of them as you jot down words of wisdom in your custom Field Notes notebook! If you haven’t bought your ticket to Distill yet, do it quick! The early bird discount ends on June 1. At that time, the price will go up from $400 to $500. distill-blogbutton

The post We Love Our Sponsors! appeared first on Engine Yard Developer Blog.

Categories: Programming

PaaS Pricing and Transparency – A Message from the CEO

Engine Yard Blog - Wed, 05/29/2013 - 19:12

At Engine Yard, we work hard to cultivate close relationships with you, our valued customers.  Every part of our organization, from sales and support to product development and engineering, is keenly focused on our clients' needs.  We believe that is the only way we can continue to deliver increasing value so you can spend your time and attention on important innovation.  So far this year, in direct response to your input, we’ve rolled out a variety of new features and capabilities that we hope make your jobs easier. (see our This Week at Engine Yard blog post series and our 2013 announcements).  Most recently, we announced significant price reductions for Engine Yard Cloud that lower the cost to get started (just 5¢ an hour) and make it easier to scale up (with additional reductions on our largest instance sizes).

As we go forward, we’ll continue to invest our resources based on your feedback.  We look to you to give us advice, help us set priorities and share your best-practice experiences. For example, from your feedback we’ve seen how important transparency is.  Today our service is tightly coupled to the specific infrastructure instance size – and both PaaS and IaaS are combined in a single price.  While this does give you full visibility into exactly what kind of infrastructure your app is running (unlike other PaaS offerings), we’re considering going one step further and breaking out the pricing of the infrastructure from the price of our platform.  As part of this approach, we are looking at tiering our platform services to more closely match the varying requirements you have.  Our platform services for small, simple environments would cost less than our services for large complex environments that utilize the most advanced features.  We’d love your feedback on these approaches – and would greatly appreciate your comments and thoughts.  Please post them below!

The post PaaS Pricing and Transparency – A Message from the CEO appeared first on Engine Yard Developer Blog.

Categories: Programming

When will you start with test automation?

I just came back from vacation and when I started again I noticed a slight change in resource requests I now see coming by; as almost all requests are with a statement around test automation. In the last two days I had two separate sessions around test automation tools. Has test automation all of a sudden become more important? Did people follow up on my last post, where I state that tools are a prerequisite in testing today, or actually: yesterday!

If you missed the latest cycle in new tools for test automation you’re either an ostrich with your head in the ground (“sorry vacation was in Southern Africa”), or you just simply still afraid. Afraid of change that test automation would cannibalize your manual test execution.

Test automation is not anymore that it takes over test execution in a very complex and unmanageable way. No it offers higher efficiency on test design, test execution, but also more options to test certain non-functional parts of applications – that could not be done without those tools – like security and performance, and virtual environments to do end-2-end tests without test environments that are down all the time. Tools are now also offering more support for testing mobile solutions. Tools are everywhere!

automated-testing

Test automation offers us testers the opportunity to do more, faster, les risky, and cheaper. I set these words specifically in that order. Test automation is often seen as a way to do test cheaper. You can, but you also can do more, for instance:

  • Let the tool do the checks and you explore the application further;
  • Setup a virtual test environment that doesn’t go down after 1 hour of use and test more in the same time;
  • Create and execute more test cases by generating and executing them automatically.
  • Get higher quality by really doing a thorough regression test, instead of a simple check, to find integration errors.

There are enough reasons to work on test automation and I don’t see why not. I think now it is even time for the next step in test automation. What that is time will tell, but I look forward to hearing that at the Test Automation Day in June. Where Bryan Bakker will tell more on this next step in his presentation “Design for Testability – the next step in Test Automation”. After the congress I’ll post my ideas here.

Categories: Testing & QA

RailsInstaller: An Unexpected Journey

Engine Yard Blog - Tue, 05/28/2013 - 21:39

In the course of every man’s life, there comes a time when he must rise up and own the moment. He must buckle down and become the leader he was meant to be. He must bear a great burden and push through to become a stronger person. This is not that journey…or is it?

hobbit (1)

The past defines us

A long time ago (2008) I was a lowly college student. I had started a course that would change the way I look at the programming world. This course introduced me to Ruby on Rails, though on a Windows machine it was anything but pleasant.

At that time, setting up an environment for building Rails applications was not a simple task. There were many different components that may or may not be easy to find and installing them was not trivial. While I was able to get things figured out, I dreamt of the day when somebody would make something to ease my installation woes.

Enter the wizard

It is now 2011 and Wayne E. Seguin has been conjuring up something that would shock the Rails community. This “magic” would be the saving grace of so many a developer on Windows. One package that would finally put an end to the madness and make installation easy. And like that, RailsInstaller was born.

Along with the installer, a website was created as a place to download RailsInstaller and get more information about Ruby on Rails development. This also included a tutorial screencast, done by Wayne, that deployed a simple application to Engine Yard Cloud

My journey begins

In June 2011, I was hired by Engine Yard to be what is known as a PANDA. Among my other many duties, I pretty much took on the role of helping Windows users who were trying to deploy applications on our platform. It was at this time that I really became familiar with RailsInstaller as well as the people using it. I helped a few users and also worked on a couple of the updated releases but did not really get too much into it until about a year later.

Into darkness

After working as a PANDA for about six months, I transitioned into the role of Application Support Engineer. Although I didn’t typically work with Windows users since most of our customers are using Mac or Linux, I did keep RailsInstaller in the back of my mind. I released an updated version of the video for Windows and also did the one for the brand new OSX version of the installer, built by Michal Papis.

One of the other things I did was submit a talk proposal to Aloha Ruby Conf, titled “Rails Development on Windows. Seriously” which focused on using RailsInstaller to get started. The talk was accepted and at this point, I knew that I’d need to spend much more time living in the Windows world as a Rails developer.

I spent the next six months before the conference, trying to immerse myself in the Windows world. At this time, I was also following the RailsInstaller Google Group and the questions/issues/discussions happening there. I came to the realization that while RailsInstaller will get a development environment set up quickly, there are many issues that can crop up afterwards. I also saw the same questions popping up repeatedly, even though they had just been answered in a previous thread.

All of this time spent working in an operating system that seemed counter-productive, working my daily support duties, and answering questions on the Google Group had me pretty burnt out. After the conference was over, I disappeared from the RailsInstaller world.

An interesting proposition

At the beginning of 2013, all of the Support Engineers went over employee development plans with their respective managers. One of the things in the plan is a week-long sabbatical to work on something outside of support tickets. I talked with my manager for a while and he brought up the idea of doing something with RailsInstaller. At this time, there wasn’t really anybody guiding the project and there hadn’t been an update to the Windows version in months. We picked a week in April for me to take over as the project maintainer and re-focus the vision.

The return

During the week of April 15th, I stepped into my new role. I focused on learning how both versions worked and the different parts that are needed to build each one. One of the other things I wanted to accomplish was to change the focus of the project.

Originally, RailsInstaller had been pegged as a way for new developers to get started with building Ruby on Rails applications. While it is good to bring in new people and teach them development skills, there are also many things to learn outside of just Ruby on Rails. I really wanted it to be more focused on getting from a fresh, new computer to a full-blown development environment as fast as possible.

So today, we’re re-launching RailsInstaller and we’re focused on building applications and making money instead of messing with setting up everything manually. Please check out the updated site at railsinstaller.org as well as the organization on Github. If you need help with the installer or want to contribute, feel free to reach out in the Google Group, IRC, or hit me up on Twitter.

I really appreciate everyone who has contributed to the project as well as everyone who has used the installer. Looking forward to what comes next and the evolution of RailsInstaller. Thanks!

The post RailsInstaller: An Unexpected Journey appeared first on Engine Yard Developer Blog.

Categories: Programming

May 24, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 05/24/2013 - 21:58

Just finished speaking at the delightful Cloud East Conference in Cambridge, England! Glad to say that my velociraptor slide went over well. Bridget and I have been visiting customers in London and Brighton, and are going to Berlin on Monday for DevOps Day! If you’re there, be sure to say hi! The intrepid Kevin Holler and inimitable Slava will also be in attendance.

--Tasha Drew, Product Manager

Engineering Updates

We have a couple fun and useful items in our early access phase! If you go to the dashboard, and from the menu select Tools >> Early Access, you will now find Application Takeover Preference and Ruby 2.0.

Application takeover preference allows you to easily set how you want our automated application takeovers to occur, in case our default behavior isn’t working for you.  Once enabled, you can go to “Edit Environment,” and you will see a new dropdown:

So if you’ve been having any trouble with your specific customizations not booting properly when we are automatically handling application takeover, you can select to instead boot from a new volume; have a takeover occur, but without booting a new application slave to replace the one being promoted; or disable the feature entirely.

As with all our early access features, we would be delighted if you shared your thoughts on these as you use them. Need something additional? Anything not working correctly? Documentation confusing? Love it and want to let us know? Tell us any and everything at our Early Access Forum!

And, also in early access, we have big news for our Rubyists: Engine Yard now has Ruby 2.0! We’ve had Rails 4 for a while now if you want to try them together. The full featured Ruby 2.0 integration in the UI, etc., is still getting some work done, so for now the installation instructions have some unique steps (i.e. not through the dashboard). See our docs for all the details!

Data Data Data

As we continue to enhance our new cluster model, we are in the planning phase for upcoming new data stacks! One of our top priorities is our mySQL database stack and how to make it more awesome as it takes advantage of the new data model. We are in the early stages of re-productizing it, so if you have any wish lists of things you’d like to see from our mySQL offering, please let us know in the feature request forums!

Social Calendar (Come say hi!)

Monday May 27th - Tuesday May 28th: Engine Yard will be participating as a sponsor of DevOps Day Berlin! Kevin Holler will be delivering a 5 minute talk about Engine Yard Cloud, and Tasha Drew and Bridget Gleason will cheer him on from the audience.

Wednesday May 29th: Our Portland office will be hosting the weekly Coder Dojo PDX!

Thursday May 30th: Engine Yard's Dublin office is host to another addition of Node.js Dublin, featuring Dominykas Blyžė, Daniel McKay, and Isaac Schlueter.  Pizza and beer will be served!

Saturday June 1st: Engine Yard Dublin will host an IXDA Prototyping workshop. In this workshop you will learn how to create wireframes and interactive prototypes using the popular tool Axure RP.

Articles of Interest 

Cern launched a public appeal to find the world’s oldest webpage, and this is what they found.

What would the Games of Thrones characters look like if the show was staged in the 1990’s? Now we know.

The post May 24, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

Announcing: Moonshine, the Distill Kick-off Party, Sponsored by New Relic

Engine Yard Blog - Thu, 05/23/2013 - 21:38

By now you’ve heard that Engine Yard is proudly presenting our inaugural developer conference, Distill, on August 8-9. In addition to a lovely Treasure Island venue and a stellar lineup of speakers, we will also be throwing a kickoff party the likes of which you’ve never seen.

In keeping with the theme of distillation, which during the daytime events means learning, best practices and collaboration, we’ve dubbed the sure-to-be-awesome kick-off soiree “Moonshine”  as a nod to the tasty beverages we’ll be providing for you all. World-class DJs will kick out the jams as you enjoy our distillation-themed smorgasboard of hors d'oeuvres and the company of your fellow conference-goers. Throughout the evening, we’ll also be doing a few surprise giveaways (and if you know Engine Yard, you know that we don’t disappoint!)

Moonshine will take place at the Old Mint in San Francisco. We want to thank our partner New Relic for sponsoring. If you haven’t purchased your ticket for Distill yet, hurry and grab your First Batch (early bird discounted) ticket! Tickets will be increasing in cost from $400 to $500 on June 1.

Categories: Programming