Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

You Cannot Win Engineering

Engine Yard Blog - Thu, 06/13/2013 - 20:51

For as long as I can remember, I’ve been a fan of Saturday Night Live and improvisational theater. Improv looks chaotic and uncontrolled, but the best practitioners operate under strict rules that govern interactions between players. Some of the most successful entertainers today, people like Stephen Colbert and Tina Fey, directly credit what they have learned in improv with making them better at what they do both on and off screen.

Unlike workplace policies that you are probably used to, the rules of improv aren’t meant to constrain you, but to open you up to the ideas of others. Let’s take a look at some of the rules the Mr. Colbert and Ms. Fey live by and see how they can improve team collaboration.

Agree and Say “Yes”.

Here’s Tina, from her book Bossypants, talking about the rules of engagement:

The first rule of improvisation is AGREE. Always agree and SAY YES. When you’re improvising, this means you are required to agree with whatever your partner has created. So if we’re improvising and I say, “Freeze, I have a gun,” and you say, “That’s not a gun. It’s your finger. You’re pointing your finger at me,” our improvised scene has ground to a halt. But if I say, “Freeze, I have a gun!” and you say, “Yes! The gun I gave you for Christmas! You bastard!” then we have started a scene because we have AGREED that my finger is in fact a Christmas gun.

The same is true of engineering teams. When one of your teammates has an idea, your first response needs to be affirmative. Take any and all ideas from your teammates as positive contributions and you start from a place of being open-minded and welcoming. Nothing kills team morale faster than someone who says “No, that won’t work” in response to any idea that they didn’t come up with.

It’s Not Just “Yes”, it’s “Yes, and…”

Everyone loves games and games are more fun when everyone  plays nicely. Make positive contributions and you will foster a spirit of openness, collaboration and — dare I say — fun.  Make it your habit to answer your teammate’s ideas with “Yes, and…” instead of “No, because”. Always offer your ideas, you just are as entitled to be silly and wrong as everyone else. Ideas seldom spring fully-formed from the head of Zeus and the part you’re holding back out of fear might be the thing that makes it work. “Yes, and…” makes you part of the solution; “No, because” makes you part of the problem.

Your Team is the Most Important Person on Your Team

 Stephen Colbert went back to his alma mater, Northwestern University, to give the commencement address in 2011. He may play a know-it-all blowhard on The Colbert Report, but that’s clearly not the case in real life. Here’s an excerpt from his speech:

…One of the things I was taught early on is that you are not the most important person in the scene. Everybody else is. And if they are the most important people in the scene, you will naturally pay attention to them and serve them. But the good news is you're in the scene too. So hopefully to them you're the most important person, and they will serve you. No one is leading, you're all following the follower, serving the servant.

You cannot win improv.

And life is an improvisation. You have no idea what's going to happen next and you are mostly just making things up as you go along.

And like improv, you cannot win your life.

The software corollary to this is: “You cannot win engineering”.

Think about the implications of this for a moment. If everyone on your team acts as if their teammates are more important than they are, you create an environment of support, giving, and progress that is mutually enriching and productive. You’ll know you have succeeded when no one on your team remembers where a great idea came from. More importantly, no one will care.

When one of your teammates asks you a question, don’t tell them to Google it (which is a bit of a jerk response in any case). Act as if their problems are more important than yours, serve the team by serving them. When you are stuck on a problem, they will treat you the same way.

None of these rules for improvisation will make you funnier or get you a slot on Weekend Update, but applying them to your co-workers will almost certainly make your team awesome. Everyone wins.

The post You Cannot Win Engineering appeared first on Engine Yard Developer Blog.

Categories: Programming

June 7, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 06/07/2013 - 23:41

Things are pretty busy right now as we ship a bunch of customer enhancements on Engine Yard Cloud and continue with our planned infrastructure abstractions and cluster model improvements. Exciting things to come! In the meantime, here’s what’s available as of this week.

--Tasha Drew, Product Manager

Engineering Updates

Now in GA: Application takeover preferences. Based on your application's customizations, you might not want to use the default application takeover behavior we've developed to automatically promote your application slaves when the app master goes away or becomes totally unresponsive for some reason.

Engine Yard Cloud now provides two automated options for replacing capacity in an application takeover situation. We also provide alternatives if you need to handle part or all of an app takeover yourself.

We now have Provisioned IOPs and EBS Optimized instances available for customers to use in Early Access! To enable them for your environment from your cloud dashboard, click the Tools menu -> Early Access, and then enable “EBS Optimized Instances” and “Provisioned IOPs.”

Keep in mind that they work best in tandem, and they will only be an option on instances booted after you enable the feature.

Data Data Data

Databases love I/O and provisioned IOPs and EBS optimized instances are very well suited for applications where the database can use more performance (think backups and snapshots too).  

You can enhance the performance of your application by having a volume with provisioned IOPs on the database master. If your application has been already deployed you can add new replicas to the environment (that have this performance boost) and have them promoted to master.
As usual don’t hesitate to ask us if PIOPs or EBS optimized instances can give your database a boost.

Social Calendar (Come say hi!)

Tuesday, June 11th: Our Buffalo office will be hosting the WNY Ruby Meetup Group. Mark Josef will be providing us with some code katas.

Wednesday, June 12th: Our PDX office will be hosting the weekly CoderDojo K-12 night, ably assisted by one of the San Francisco sprint teams, who will be on site for an off site (as it were).

Wednesday, June 12th: Girl Develop It will be doing a Code and Coffee night in our Buffalo office. The participants be focusing on honing their skills and working in groups. Swing by for the whole thing or just for a part of it.

Friday, June 14th: DevOps Day Amsterdam will be happening! Be sure to meet our own Slava and Richard and let them tell you about how Engine Yard can make your lives easier.

Articles of Interest

Mozilla's John O’Dunn discusses how to use release engineering as a force-multiplier!

David Padilla explains why hash lookups are so fast in Ruby on the Engine Yard blog.

The post June 7, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

Speaking at Conferences: How to write a talk and get it accepted

Engine Yard Blog - Fri, 06/07/2013 - 21:38

At php[tek] 2013, Engine Yard sponsored the Mentorship Summit, a special forum to discuss the value of mentoring to create more connections and advancement opportunities for developers. A common theme that came out of the summit was that speaking at conferences is a great way to further oneself both personally and professionally. During the discussion, inevitably someone said they've submitted numerous times but had never been accepted to speak, then someone else said they don't know how to write a good proposal, and many discussions were had about what it means to write a good talk proposal.

I've had this conversation many times over the past few years with those I mentor, but this year something was different: I was a member of the selection committee for Distill, Engine Yard’s first developer conference.

Being on the other side for once has changed how I think about what is important when submitting for a talk and I thought it might be helpful to share how my perspective has evolved.

Questions and Answers

A proposal typically consists of a title, a short abstract, and sometimes a larger body of text to give the reviewers more detail about what to expect in the talk.

The first two items are really important because in addition to helping your talk stand out to the reviewer, they are also how an attendee will choose your talk over others on the schedule.

It's important to understand that there is an implicit question being asked in the title of your talk. For example, the two talks I presented at php[tek] were:

  1. "PHP 5.5: The New Bits"

  2. "MySQL High Availability, Disaster Recovery and Load Balancing"

The first one has an implicit question along the lines of: What's new in PHP 5.5? and the second: How do I make my database more robust and more scalable?  How do I make it easier to recover from failures?

Your abstract will then further inform the attendee (and the reviewer) of that question, and more importantly, create an expectation, or even a promise of what the talk will cover.

Most negative feedback I hear about talks isn't that the speaker was terrible or didn't know what they were talking about; it’s simply that the talk wasn't what the attendee wanted. Communicating the intent of your talk clearly is essential. If an attendee is disappointed, it’s often either because he misinterpreted the intention of the talk, or the speaker failed to properly set his expectations.

For example, if my MySQL talk focused mainly on using memcache (as a means to reduce load on the DB and therefore reducing the need to scale) it would have been a terrible talk — though a worthwhile topic, it's not addressing the question posed by my title and therefore not meeting the promise of the talk.

However, a talk is not always a matter of simply answering a question. Sometimes you are teaching people how to better ask their question. If the talk had been titled "High performance websites with MySQL", then memcache would certainly be answering some of the implicit question by teaching them how to ask a better question: "How do I make my website faster when using MySQL?"

Is Your Question Relevant?

One of the more important things to remember is to ask the right question. What do people care about? Look at schedules from previous years for the conference (if they exist) to get an idea of what experience level the conference targets, and observe the associated community to see what people are interested in — what are emerging trends and topics that people are fascinated by?

Are you the right person for the job?

A very important thing to realize about your proposal is that who you are matters. To be more specific: why are you qualified to speak on a specific subject, or why do your opinions about it matter?

When selecting talks for Distill, I ran into mostly speakers I was unfamiliar with. So I Googled them.

If you want to establish yourself as a possible speaker, you need to be part of the larger conversation on the subject on which you want to speak. This can take many forms:

  • Blog posts

  • Tweets with people about the subject (yes, that's right, your tweets can matter!)

  • Any other media (podcasts, books, magazine articles)

  • Code contributions

The last one takes the longest to verify and is a last resort — if the reviewer hasn't been grabbed by your title/abstract then they may or may not even bother.

If you are contributing code to projects but doing nothing else, then you should start blogging about what you're contributing.

Submit Early, Submit Often

This one seems like it will be quite controversial to communities other than the PHP community which has made this the norm: submit many proposals.

When I submit to a conference, I will propose no fewer than four proposals. This not only increases your chances (by the numbers), but when selecting talks, if you have absolutely no other information on a speaker, this will at least give the reviewer a better idea of what you do, and a little more about what experience you may have.

Speaking Experience

The ability to communicate as a speaker is also something to worry about when selecting talks. Having a history of speaking, be it at other conferences, user groups, or even via webcasts/podcasts, is a big plus. Be sure to share all the media from your talks: slides, video and audio recordings. Similarly, podcasts are another way to give great insight into your ability to communicate verbally.

However, even writing can give insight into how you communicate.

Hard skills vs Soft skills Talks

There are two types of talks: so-called “hard talks” that teach technical skills, and “soft talks” which teach personal skills (e.g. team working, project management, how to get hired... how to write talk proposals).

When setting up a conference schedule it's very important to get the balance of hard skills vs soft skills talks right. Too few hard skills talks, and it's hard to justify the expense to your employer.

But quite often, the soft skills talks are the ones that get the most people talking, and when you are an experienced developer are often what you need to advance your career.

What this means is that you are far more likely to get a hard skills talk accepted than a soft skills talk — just because of the ratios, and because the few soft skills talks that are selected have to really stand out, typically well established speakers are chosen.

The "Distillation" Process: How We Selected Talks for Distill

For Distill, our review process was what I called "Iron Chef Style". We rated each talk as follows:

  • Content (15 points)

  • Fit (10 points)

  • Speaker (5 points).

The content is the topic covered, the fit is how well the talk fits into the overarching theme of the conference, and the speaker is not who they were, but what made them the right person to talk on the subject.

Once they were all rated by each member of the committee, we tallied the numbers and sorted by the totals.

At this point, we asked if anybody had any specific talks they felt strongly about, and also used the standard deviation to see which talks have the biggest difference between the different reviewers to start discussion on those.

This gave us our top 32. We then looked at any speaker who had multiple talks in the top 32, and made a decision on which one of them we preferred and removed the duplicates (there were only 3).

Finally, as we further refined our theme, we narrowed our list down to the final 15 — who were selected as speakers for Distill.

I was surprised at just how much of an extremely lengthy and difficult task this was. To anyone who has ever done this with my talks… I promise that my bribes will be much bigger in the future, you deserve it!


The post Speaking at Conferences: How to write a talk and get it accepted appeared first on Engine Yard Developer Blog.

Categories: Programming

Hash lookup in Ruby, why is it so fast?

Engine Yard Blog - Tue, 06/04/2013 - 18:00

Note: Our friend and CEO at Crowd Interactive, David Padilla, wrote this great piece about hash lookup in Ruby. Be sure to check out MagmaConf!

Have you ever noticed that in Ruby looking for a specific key-value pair in a Hash is very fast?

Allow me to explain the logic behind hashes in Ruby with a language that you probably understand: Ruby.

Let's imagine for a second that we want to emulate the functionality in hashes because, for some strange reason, they have not been implemented yet.

If we want to have some sort of key-value structure, we'll have to implement it ourselves, so let's get to work.

First, we'll need a Struc to represent a HashEntry, or the key-value objects that we will add to our hashes.

HashEntry =, :value)

Now, we'll need a class that represents our Hashes or (to avoid conflicts with the original Hash class) HashTable.

class HashTable
  attr_accessor :bins

  def initialize
    self.bins = []

We add the `bins` attribute to the class. `bins` will be an array where we'll store our HashEntry elements.

Now, let's write a method to add a HashEntry to a HashTable. To follow convention, we'll use the traditional `<<` as method name.

class HashTable
  attr_accessor :bins

  def initialize
    self.bins = []

  def <<(entry)
    self.bins << entry

Great, now we can add HashEntry elements to our HashTable like so:

entry = :foo, :bar
table =
table << entry

What if we want to look for an entry by key? Let's write the `[]` method on the HashTable class to handle that.

def [](key)
  self.bins.detect { |entry| entry.key == key }

What we're doing here is simply going element by element comparing the given key until we find what we're looking for. Efficient? Let's figure it out.


We'll use Ruby's benchmarking tools to figure out how much time we're spending looking for elements on our hash tables.

require 'benchmark'

# HashTable instance
table =

# CREATE 1,000,000 entries and add them to the table

(1..1000000).each do |i|
  entry = i.to_s, "bar#{i}"

  table << entry

# Look for an element at the beginning, middle and end of the HashTable.
# Benchmark it
%w(100000 500000 900000).each do |key|
  time = Benchmark.realtime do

  puts "Finding #{key} took #{time * 1000} ms"

When we run this benchmark, we get the following results:

Finding 100000 took 33.641 ms
Finding 500000 took 192.678 ms
Finding 900000 took 345.329 ms

What we see here is that lookup times increase depending on the amount of entries and its position within the array or `bins`. This is obviously very inefficient and unacceptable for real life scenarios.

Now, let's see how Ruby tackles this problem internally.


Instead of using a single array to store all its entries, hashes in Ruby use an array of arrays or "bins".

First, it calculates a unique integer value for each entry. For this example we will use `Object#hash`. Then, Ruby divides this hash integer by the total number of bins and obtains the remainder or modulus. This modulus will be used as the bin index for that specific entry.

When you lookup for a key, you calculate its bin index again using the same algorithm and you look for the corresponding object directly on that bin.

Let's add an attribute on the HashTable class that will determine how many bins each HashTable will have, and we'll initialize it with 500.

class HashTable
  # ...

  attr_accessor :bin_count

  def initialize
    self.bin_count = 500
    self.bins = []

  # ...

Now, let's write a method that calculates the bin for a specific entry depending on the number of bins.

class HashTable
  # ...

  def bin_for(key)
    key.hash % self.bin_count

  # ...

When storing the HashEntry in the HashTable, we won't just store it on an array, we'll store it on an array that corresponds to the `bins` index depending on what the `bin_for` method returns:

class HashTable
  # ...

  def <<(entry)
    index = bin_for(entry.key)
    self.bins[index] ||= []
    self.bins[index] << entry

  # ...

And last, whenever we want to retrieve a HashEntry, we'll recalculate the bin index again using the `bin_for` method and once we have that, we'll know exactly where to look for our entry.

def [](key)
  index = bin_for(key)
  self.bins[index].detect do |entry|
    entry.key == key

When we run the same benchmark that we used earlier, we can see times improve dramatically:

Finding 100000 took 0.025 ms
Finding 500000 took 0.094 ms
Finding 900000 took 0.112 ms

Not only did times improve, but we got rid of the variance that we used to have depending on the position of the element in the bin pool.

There's still room for improvement here. Let's add more bins and see what happens.

class HashTable
  # ...

  def initialize
    self.bin_count = 300000
    self.bins = []

  # ...

When we run the benchmark we get:

Finding 100000 took 0.014 ms
Finding 500000 took 0.016 ms
Finding 900000 took 0.005 ms

Even more improvement. This mean that the more bins, the less time spent looking for a specific key in a bin.

How many bins does Ruby actually use?

Ruby manages the size of the bins dynamically. It starts with 11 and as soon as one of the bins has 5 or more elements, the bin size is increased and all hash elements are reallocated to their new corresponding bin.

At some point you pay an exponentially increased time penalty while Ruby resizes the bin pool, but if you think about it, its worth the time since this will keep lookup times and memory usage as low as possible.

Further reading

If you want to learn where this algorithm came from and a little more about Ruby internals, I really recommend that you read Pat Shaughnessy's Ruby Under a Microscope book. Pat explains how the Ruby VM works in a way that anyone can understand. No C knowledge required, I really enjoyed reading it.

You can find the working example I used for this example on this gist.

You could also read some of the Rubinius source code. Take a look at their implementation of the Hash class, you'll probably understand a little more of the logic they used after you've read this post and Pat's book.

Thanks for reading.

The post Hash lookup in Ruby, why is it so fast? appeared first on Engine Yard Developer Blog.

Categories: Programming

The Toyota Way: The need for doing it right the first time

After WWII Toyota started developing its Toyota Production System (TPS); which was identified as ‘Lean’ in the 1990s. Taiichi Ohno, Shigeo Shingo and Eiji Toyoda developed the system between 1948 and 1975. In the myth surrounding the system it was not inspired by the American automotive industry, but from a visit to American supermarkets, Ohno saw the supermarket as model for what he was trying to accomplish in the factor and perfect the Just-in-Time (JIT) production system. While accomplishing this low inventory levels were a key outcome of the TPS, and an important element of the philosophy behind its system is to work intelligently and eliminate waste so that only minimal inventory is needed.

As TPS and Lean have their own principles as outlined by Toyota:

  • Long-term Philosophy
  • Right process will produce the right results
  • Value to organization by developing people
  • Solving root problems drives organizational learning

As these principles were summed up and published by Toyota in 2001, by naming it “The Toyota Way 2001”. It consists the above named principles in two key areas: Continuous Improvement, and Respect for People.


The principles for a continuous improvement include establishing a long-term vision, working on challenges, continual innovation, and going to the source of the issue or problem. The principles relating to respect for people include ways of building respect and teamwork. When looking at the ALM all these principles come together in the ‘first time right’ approach already mentioned. And from Toyota’s view they were outlined as followed:

  • The right process will produce the right results
    • Create continuous process flow to bring problems to the surface.
    • Use the ‘pull’ system to avoid overproduction (kanban)
    • Level out the workload (heijunka).
    • Build a culture of stopping to fix problems, to get quality right from the first (jidoka)
  • Continuously solving root problems drives organizational learning
    • Go and see for yourself to thoroughly understand the situation (Genchi Genbutsu);
    • Make decisions slowly by consensus, thoroughly considering all options (nemawashi); implement decisions rapidly;
    • Become a learning organization through relentless reflection (hansei) and continuous improvement (kaizen).
Let’s do it right now!

As the economy is changing and IT is more common sense throughout ore everyday life the need for good quality software products has never been this high. Software issues create bigger and bigger issues in our lives. Think about trains that cannot ride due to software issues, bank clients that have no access to their bank accounts, and people oversleeping because their alarm app didn’t work on their iPhone. As Capers Jones [Jones, 2011] states in his 2011 study that “software is blamed for more major business problems than any other man-made product” and that “poor quality has become one of the most expensive topics in human history”. The improvement of software quality is a key topic for all industries.

 Right the first time vs jidoka

In both TPS and Lean autonomation or jidoka are used. Autonomation can be described as ‘intelligent autonomation’, it means that when an abnormal situation arises the ‘machine’ stops and fix the abnormality. Autonomation prevents the production of defective products, eliminates overproduction, and focuses attention on understanding the problem and ensuring that it never recurs; a quality control process that applies the following four principles:

  • Detect the abnormality.
  • Stop.
  • Fix or correct the immediate condition.
  • Investigate the root cause and install a countermeasure.
Find defects as early as possible

In other words autonomation helps to get quality right the first time perfectly. With IT projects being different from the Toyota car production line, ‘perfectly’ may be a bit too much, but the process around quality assurance should be the same:

  • Find the defect.
  • Stop.
  • Fix or correct the error.
  • Investigate the root cause and take countermeasures.

The defect should be found as early as possible to be fixed as early as possible. And as with Lean and TPS the reason behind this is to make it possible to address the identification and correction of defects immediately in the process.

Categories: Testing & QA

May 31, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 05/31/2013 - 23:47

Spent an awesome week at DevOps Day Berlin and visiting customers in Berlin with application engineer extraordinaire Kevin Holler and our fearless sales leader Bridget Gleason. So fantastic to see the exchange of best practices and ideas around DevOps and get a chance to check out the Berlin tech community!

We’re planning to be at DevOps Day Amsterdam in a few weeks too, so hope to see you there!

--Tasha Drew, Product Manager

Engineering Updates

The EngineYard gem 2.1.0 has been released with a bunch of enhanced capabilities, including new asset strategies. We received a lot of customer requests around these areas and hope that this will make you super happy. Read all about it on the change log and the readme on github.

Ruby 2.0 is now in GA on Engine Yard Cloud! Read all about it.

We have released our new hardened Gentoo stack in GA (Gentoo 12.11), with all the latest and greatest packages and features that the new Gentoo has to offer. It only works in new environments, so if you are going to spin up a new environment, check it out!

We've updated the worker counts for Passenger/Unicorn/etc.. and exposed some easy ways for customers to alter the worker counts, memory thresholds, and other cool stuff. Read all about how to do it here! This allows customers to avoid writing custom chef to alter some of this stuff. As always, if you need any help, open a support ticket!

Workers are the processes that allow your application to respond to incoming web requests. Regardless of the application server stack you select, it provides one or more workers per instance to run your application.  Learn all about how to tune your workers per server and use an excellent new calculator to figure out your ideal set up in our new documentation!

Social Calendar (Come say hi!)

Postgres Ireland: June 4, Engine Yard Dublin. Come and meet some Postgres backend developers based in Ireland, and discuss the recent landmark 9.2 release. We encourage participation from people of all levels of experience with Postgres - from people who aren't current users, and would like to know more, right up to experienced users with large installations. Greg Stark, a long-time Postgres contributor and committer, will speak.

Magma Rails: June 5-7, Manzanillo, Mexico. MagmaRails is now MagmaConf!, one of the most important web development conferences in Mexico; with Ruby, Ruby on Rails, Frontend and more web technologies sessions for developers and engineers that build cutting-edge web applications.

GoRuco: New York’s premier community-organized Ruby conference. Stop by the Engine Yard table and say hello!

Articles of Interest

Our inaugural developer’s conference, Distill, on Treasure Island in the San Francisco Bay, still has early bird tickets left. Hope to see you there!

Northwestern University’s engineering students are given dance lessons to help them have more balanced brains.

Fan favorite and lead AP support manager Chris Rigor shares his talking points from his presentation at Ruby Kaigi this week, in which he created a debian wheezy vagrant box with chef recipes to set up a rails app!

And this was my favorite slide of the week, where Meagan Fisher demystifies design process at BTConf.

The post May 31, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

We Love Our Sponsors!

Engine Yard Blog - Fri, 05/31/2013 - 22:15

Over the years, Engine Yard has been a proud sponsor of hundreds of incredible open source conferences. When it came time to seek sponsors for Distill, our inaugural developer conference, we were thrilled and humbled by the amount of love and support we received from companies who also put community first. We are proud to announce our first round of sponsors for this year--without them, Distill would not be possible! AppFirst will be making sure that all talks can be seen on demand after the conference by sponsoring our videos! Get the AppFirst DevOps dashboard and free offering for full stack insights across all of your Engine Yard environments. GitHub, helping us all build software better, together, will be sponsoring the lightning talks portion of our schedule. These rapid, info-packed talks are always a conference highlight and are sure to be meaningful, memorable and entertaining. Kony helps you take your business mobile. They’re also going to be mobilizing attendees by sponsoring the transportation from the conference hotel to Treasure Island and back again. MongoLab takes the sweetest sponsorship--make sure to thank them for the delicious Smitten ice cream you’ll be enjoying at Distill! They provide a fully-managed MongoDB Database-as-a-Service platform automating the operational aspects of running MongoDB in the cloud.

New Relic will be sponsoring our kick-off party, Moonshine. New Relic makes awesome app-monitoring software. Check it out.

SendGrid will be keeping you alert throughout Distill (especially the morning following Moonshine) with excellent coffee from Ritual Coffee Roasters! SendGrid are the masters of simple, reliable email delivery. Zendesk makes customer support painless. Think of them as you jot down words of wisdom in your custom Field Notes notebook! If you haven’t bought your ticket to Distill yet, do it quick! The early bird discount ends on June 1. At that time, the price will go up from $400 to $500. distill-blogbutton

The post We Love Our Sponsors! appeared first on Engine Yard Developer Blog.

Categories: Programming

PaaS Pricing and Transparency – A Message from the CEO

Engine Yard Blog - Wed, 05/29/2013 - 19:12

At Engine Yard, we work hard to cultivate close relationships with you, our valued customers.  Every part of our organization, from sales and support to product development and engineering, is keenly focused on our clients' needs.  We believe that is the only way we can continue to deliver increasing value so you can spend your time and attention on important innovation.  So far this year, in direct response to your input, we’ve rolled out a variety of new features and capabilities that we hope make your jobs easier. (see our This Week at Engine Yard blog post series and our 2013 announcements).  Most recently, we announced significant price reductions for Engine Yard Cloud that lower the cost to get started (just 5¢ an hour) and make it easier to scale up (with additional reductions on our largest instance sizes).

As we go forward, we’ll continue to invest our resources based on your feedback.  We look to you to give us advice, help us set priorities and share your best-practice experiences. For example, from your feedback we’ve seen how important transparency is.  Today our service is tightly coupled to the specific infrastructure instance size – and both PaaS and IaaS are combined in a single price.  While this does give you full visibility into exactly what kind of infrastructure your app is running (unlike other PaaS offerings), we’re considering going one step further and breaking out the pricing of the infrastructure from the price of our platform.  As part of this approach, we are looking at tiering our platform services to more closely match the varying requirements you have.  Our platform services for small, simple environments would cost less than our services for large complex environments that utilize the most advanced features.  We’d love your feedback on these approaches – and would greatly appreciate your comments and thoughts.  Please post them below!

The post PaaS Pricing and Transparency – A Message from the CEO appeared first on Engine Yard Developer Blog.

Categories: Programming

When will you start with test automation?

I just came back from vacation and when I started again I noticed a slight change in resource requests I now see coming by; as almost all requests are with a statement around test automation. In the last two days I had two separate sessions around test automation tools. Has test automation all of a sudden become more important? Did people follow up on my last post, where I state that tools are a prerequisite in testing today, or actually: yesterday!

If you missed the latest cycle in new tools for test automation you’re either an ostrich with your head in the ground (“sorry vacation was in Southern Africa”), or you just simply still afraid. Afraid of change that test automation would cannibalize your manual test execution.

Test automation is not anymore that it takes over test execution in a very complex and unmanageable way. No it offers higher efficiency on test design, test execution, but also more options to test certain non-functional parts of applications – that could not be done without those tools – like security and performance, and virtual environments to do end-2-end tests without test environments that are down all the time. Tools are now also offering more support for testing mobile solutions. Tools are everywhere!


Test automation offers us testers the opportunity to do more, faster, les risky, and cheaper. I set these words specifically in that order. Test automation is often seen as a way to do test cheaper. You can, but you also can do more, for instance:

  • Let the tool do the checks and you explore the application further;
  • Setup a virtual test environment that doesn’t go down after 1 hour of use and test more in the same time;
  • Create and execute more test cases by generating and executing them automatically.
  • Get higher quality by really doing a thorough regression test, instead of a simple check, to find integration errors.

There are enough reasons to work on test automation and I don’t see why not. I think now it is even time for the next step in test automation. What that is time will tell, but I look forward to hearing that at the Test Automation Day in June. Where Bryan Bakker will tell more on this next step in his presentation “Design for Testability – the next step in Test Automation”. After the congress I’ll post my ideas here.

Categories: Testing & QA

RailsInstaller: An Unexpected Journey

Engine Yard Blog - Tue, 05/28/2013 - 21:39

In the course of every man’s life, there comes a time when he must rise up and own the moment. He must buckle down and become the leader he was meant to be. He must bear a great burden and push through to become a stronger person. This is not that journey…or is it?

hobbit (1)

The past defines us

A long time ago (2008) I was a lowly college student. I had started a course that would change the way I look at the programming world. This course introduced me to Ruby on Rails, though on a Windows machine it was anything but pleasant.

At that time, setting up an environment for building Rails applications was not a simple task. There were many different components that may or may not be easy to find and installing them was not trivial. While I was able to get things figured out, I dreamt of the day when somebody would make something to ease my installation woes.

Enter the wizard

It is now 2011 and Wayne E. Seguin has been conjuring up something that would shock the Rails community. This “magic” would be the saving grace of so many a developer on Windows. One package that would finally put an end to the madness and make installation easy. And like that, RailsInstaller was born.

Along with the installer, a website was created as a place to download RailsInstaller and get more information about Ruby on Rails development. This also included a tutorial screencast, done by Wayne, that deployed a simple application to Engine Yard Cloud

My journey begins

In June 2011, I was hired by Engine Yard to be what is known as a PANDA. Among my other many duties, I pretty much took on the role of helping Windows users who were trying to deploy applications on our platform. It was at this time that I really became familiar with RailsInstaller as well as the people using it. I helped a few users and also worked on a couple of the updated releases but did not really get too much into it until about a year later.

Into darkness

After working as a PANDA for about six months, I transitioned into the role of Application Support Engineer. Although I didn’t typically work with Windows users since most of our customers are using Mac or Linux, I did keep RailsInstaller in the back of my mind. I released an updated version of the video for Windows and also did the one for the brand new OSX version of the installer, built by Michal Papis.

One of the other things I did was submit a talk proposal to Aloha Ruby Conf, titled “Rails Development on Windows. Seriously” which focused on using RailsInstaller to get started. The talk was accepted and at this point, I knew that I’d need to spend much more time living in the Windows world as a Rails developer.

I spent the next six months before the conference, trying to immerse myself in the Windows world. At this time, I was also following the RailsInstaller Google Group and the questions/issues/discussions happening there. I came to the realization that while RailsInstaller will get a development environment set up quickly, there are many issues that can crop up afterwards. I also saw the same questions popping up repeatedly, even though they had just been answered in a previous thread.

All of this time spent working in an operating system that seemed counter-productive, working my daily support duties, and answering questions on the Google Group had me pretty burnt out. After the conference was over, I disappeared from the RailsInstaller world.

An interesting proposition

At the beginning of 2013, all of the Support Engineers went over employee development plans with their respective managers. One of the things in the plan is a week-long sabbatical to work on something outside of support tickets. I talked with my manager for a while and he brought up the idea of doing something with RailsInstaller. At this time, there wasn’t really anybody guiding the project and there hadn’t been an update to the Windows version in months. We picked a week in April for me to take over as the project maintainer and re-focus the vision.

The return

During the week of April 15th, I stepped into my new role. I focused on learning how both versions worked and the different parts that are needed to build each one. One of the other things I wanted to accomplish was to change the focus of the project.

Originally, RailsInstaller had been pegged as a way for new developers to get started with building Ruby on Rails applications. While it is good to bring in new people and teach them development skills, there are also many things to learn outside of just Ruby on Rails. I really wanted it to be more focused on getting from a fresh, new computer to a full-blown development environment as fast as possible.

So today, we’re re-launching RailsInstaller and we’re focused on building applications and making money instead of messing with setting up everything manually. Please check out the updated site at as well as the organization on Github. If you need help with the installer or want to contribute, feel free to reach out in the Google Group, IRC, or hit me up on Twitter.

I really appreciate everyone who has contributed to the project as well as everyone who has used the installer. Looking forward to what comes next and the evolution of RailsInstaller. Thanks!

The post RailsInstaller: An Unexpected Journey appeared first on Engine Yard Developer Blog.

Categories: Programming

May 24, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 05/24/2013 - 21:58

Just finished speaking at the delightful Cloud East Conference in Cambridge, England! Glad to say that my velociraptor slide went over well. Bridget and I have been visiting customers in London and Brighton, and are going to Berlin on Monday for DevOps Day! If you’re there, be sure to say hi! The intrepid Kevin Holler and inimitable Slava will also be in attendance.

--Tasha Drew, Product Manager

Engineering Updates

We have a couple fun and useful items in our early access phase! If you go to the dashboard, and from the menu select Tools >> Early Access, you will now find Application Takeover Preference and Ruby 2.0.

Application takeover preference allows you to easily set how you want our automated application takeovers to occur, in case our default behavior isn’t working for you.  Once enabled, you can go to “Edit Environment,” and you will see a new dropdown:

So if you’ve been having any trouble with your specific customizations not booting properly when we are automatically handling application takeover, you can select to instead boot from a new volume; have a takeover occur, but without booting a new application slave to replace the one being promoted; or disable the feature entirely.

As with all our early access features, we would be delighted if you shared your thoughts on these as you use them. Need something additional? Anything not working correctly? Documentation confusing? Love it and want to let us know? Tell us any and everything at our Early Access Forum!

And, also in early access, we have big news for our Rubyists: Engine Yard now has Ruby 2.0! We’ve had Rails 4 for a while now if you want to try them together. The full featured Ruby 2.0 integration in the UI, etc., is still getting some work done, so for now the installation instructions have some unique steps (i.e. not through the dashboard). See our docs for all the details!

Data Data Data

As we continue to enhance our new cluster model, we are in the planning phase for upcoming new data stacks! One of our top priorities is our mySQL database stack and how to make it more awesome as it takes advantage of the new data model. We are in the early stages of re-productizing it, so if you have any wish lists of things you’d like to see from our mySQL offering, please let us know in the feature request forums!

Social Calendar (Come say hi!)

Monday May 27th - Tuesday May 28th: Engine Yard will be participating as a sponsor of DevOps Day Berlin! Kevin Holler will be delivering a 5 minute talk about Engine Yard Cloud, and Tasha Drew and Bridget Gleason will cheer him on from the audience.

Wednesday May 29th: Our Portland office will be hosting the weekly Coder Dojo PDX!

Thursday May 30th: Engine Yard's Dublin office is host to another addition of Node.js Dublin, featuring Dominykas Blyžė, Daniel McKay, and Isaac Schlueter.  Pizza and beer will be served!

Saturday June 1st: Engine Yard Dublin will host an IXDA Prototyping workshop. In this workshop you will learn how to create wireframes and interactive prototypes using the popular tool Axure RP.

Articles of Interest 

Cern launched a public appeal to find the world’s oldest webpage, and this is what they found.

What would the Games of Thrones characters look like if the show was staged in the 1990’s? Now we know.

The post May 24, 2013: This Week at Engine Yard appeared first on Engine Yard Developer Blog.

Categories: Programming

Announcing: Moonshine, the Distill Kick-off Party, Sponsored by New Relic

Engine Yard Blog - Thu, 05/23/2013 - 21:38

By now you’ve heard that Engine Yard is proudly presenting our inaugural developer conference, Distill, on August 8-9. In addition to a lovely Treasure Island venue and a stellar lineup of speakers, we will also be throwing a kickoff party the likes of which you’ve never seen.

In keeping with the theme of distillation, which during the daytime events means learning, best practices and collaboration, we’ve dubbed the sure-to-be-awesome kick-off soiree “Moonshine”  as a nod to the tasty beverages we’ll be providing for you all. World-class DJs will kick out the jams as you enjoy our distillation-themed smorgasboard of hors d'oeuvres and the company of your fellow conference-goers. Throughout the evening, we’ll also be doing a few surprise giveaways (and if you know Engine Yard, you know that we don’t disappoint!)

Moonshine will take place at the Old Mint in San Francisco. We want to thank our partner New Relic for sponsoring. If you haven’t purchased your ticket for Distill yet, hurry and grab your First Batch (early bird discounted) ticket! Tickets will be increasing in cost from $400 to $500 on June 1.

Categories: Programming

A Conversation About Testing in PHP

Engine Yard Blog - Wed, 05/22/2013 - 21:45

We are proud to sponsor Chris Hartjes and Ed Finkler's Development Hell podcast series where they record their freewheeling, uncensored discussions on programming the web, so future generations can learn from their failures.

Read on to get the low down on different testing tools and their relative merits--check it out as Ed and Chris weep for the future, come to some interesting conclusions and get their hands dirty so you don't have to.

To hear more from Chris and Ed tune in to their podcast, /dev/hell

Ed and Chris had a little chat about testing in PHP.

Chris: Okay, so today's topic is PHP testing

Ed: Word up

Chris: Now, Ed, I know that for the most part you are not a big fan of the mainstream PHP testing tools

Ed: Yes, that's true

Chris: So what is it that you don't like about them

Ed: I guess realistically my complaints are aimed at PHPUnit . It's very powerful and very complete from what I can tell, but I think it's difficult to pick up and I think that difficulty makes people less likely to use it. Because it's by far the best known testing tool, I think that tends to limit the use of unit testing, period, in PHP. That's not necessarily PHPUnit's fault per se. I just think it's the situation we're in. I think the documentation, the setup, and just obtaining PHPUnit is a challenge, particularly when compared to unit testing options I've seen in other languages. Python, for example, has a simple but effective unit testing library built into the core.

Chris: So, when you say "difficult to pick up", is it because tests look like this?

class Labels
    public $db;

     * @param GrumpyDb $db
    public function __construct($db)
        $this->db = $db;

     * Turns label values like codingStandardsSuck into
    public function screamingSnakeLabels()
        $results = $db->query("SELECT name FROM labels");
        $labels = array();
        foreach ($results as $result) {
            $labels[] = $this->_camelToScreamingSnake($result);
        return $labels;

     * Method that takes a camelCase string into SCREAMING_SNAKE_CASE
     * @param string $value
    protected function _camelToScreamingSnake($value)
        $result = preg_replace_callback(
            function ($match) {
                return "_" . strtolower($match[0]);
        return strtoupper($result);

class DevhellTest extends PHPUnit_Framework_TestCase
    public function testShowEdHow()
        $db = $this->getMockBuilder('Foo')
            ->will($this->returnValue(array('devHell', 'camelCase'));
        $label = new Label($db);
        $expectedResults = array('DEV_HELL', 'CAMEL_CASE');
        $testResults = $label->screamingSnakeLabels();
            "Labels were not correctly coverted to screaming snake case"

Chris: Maybe it's because I've worked with it a lot, all I see is some boilerplate and then a few statements that seem pretty intuitive to me.

Ed: I think boilerplate is part of the issue. I think that's intimidating. Tools can mitigate that to some extent, but I don't think it eliminates the problem entirely. I just don't think writing a simple test should be anything more than a couple lines of code. Then you can build upon that iteratively as you need. I think that approach of starting simply and building up your set of tests really helps you understand what's going on, and I think it makes testing a lot more accessible to people who haven't done it before. A lot of testing framework docs I see throw a ton of nomenclature out at the reader. I think if you don't already understand that nomenclature, you won't understand what's up.

Chris: So when you say 'nomenclature', you're talking about things like what exactly? Assertions and mocks?

Ed: Knowing how to mock that stuff up is pretty complex. In my experience the majority of people who work with PHP don't have a lot of formal training and even if they do, it often doesn't cover testing concepts. Like, what's a "unit?" What's an assertion? What's a mock or a stub?

Chris: I weep for the future, Ed. A unit is a small amount of code that you're trying to test

In PHP, that's usually one object, An assertion is simply a statement that "I am saying that the following is true", whatever that assertion happens to be. I do agree that there is lots of confusion about what a mock or a stub is so in my book I devote a chapter to explaining those things.

Ed: So I know what that stuff is (although I get confused about the diff between a mock and stub). But the real problem is that in order to write tests, you have to already know how to program, and that in itself is super-intimidating for people. PHP has a very shallow learning curve: the time between learning and becoming productive in some way is very short. That's certainly one of the reasons PHP is so popular. We need, I think, to mirror that in how we present testing, and make it easy to get into. It shouldn't be something that is terribly complex to set-up and do.

Chris: In that light, I understand the motivation to develop your own testing tools, but I still think PHPUnit is the way to go. So many people use it and there are so many resources available to learn it, that picking it up isn't as difficult as I think you're making it out to be. Alternately, I think the Behavior-Driven Development (BDD) model that Behat offers is appealing, and easier to pick up than the xUnit style. Behat combined with Mink is a solid alternative to PHPUnit.

Ed: If you are doing acceptance testing (meaning that you only care that the application as a whole is working) I don't
think you can go wrong with being able to write tests that look like this:

    Scenario: Main page loads
    Given I am on "/index.php"
    Then I should see "Lies I Told My Kids"

    Scenario: Empty form fields trigger errors
    Given I am on "/index.php"
    When I press "submitButton"
    Then I should see "You submitted an invalid e-mail address"

    Scenario: Missing description triggers errors
    Given I am on "/index.php"
    When I fill in "email" with ""
    And I press "submitButton"
    Then I should see "You submitted a blank description"

Chris: The Behat and Mink combo can let you create some very interesting acceptance tests, and it even provides you with tools that will tell you when you when you will have to write your own helpers to supplement what they can provide you. It took me a few days to figure out Behat's own way of doing things but once I did I was able to create some very interesting tests, even ones where JavaScript (long the bane of automated acceptance testing) was being used.

If your mind doesn't align well with unit testing, then something like Behat is definitely the way to go. There's something neat about watching PHP run Behat which in turn opens up a browser and starts acting like a user and hopefully using your application correctly.

Ed: Ultimately, though, a lot of the problem with testing in PHP is that PHP's insane flexibility makes it super easy to write code that you cannot test. That and PHP is almost always working in concert with other systems, like a web server, so it can be tough to know what you can easily test inside the CLI and what you'd need to use a different approach.

To write testable code, you really have to be thinking about testing when you write your code. It takes a bit of time to get used to that, but I think it's very doable. In much the same way, it's taken us a long time to make security a first-order concern in PHP development, but I think we've done a decent job of that. We need to do that for testing as well.

Chris: If only, Ed. If only.

Categories: Programming

May 17, 2013: This Week at Engine Yard

Engine Yard Blog - Fri, 05/17/2013 - 19:36

I spent this week with the team of engineers who made Riak on Engine Yard Cloud possible, attending RICON East: all Distributed Systems, all the time. Later in the week we took advantage of being in New York City to visit local customers and discuss the various features we’re working on and field any technical, product, and data questions.

Both our engineering and product teams love incorporating customer feedback into our direction. Speaking of which -- if you’re in San Francisco, I’m organizing customer UX feedback sessions! Hit me up :)

--Tasha Drew, Product Manager

Engineering Updates

PHP is now GA on Engine Yard Cloud! Per Product Manager Noah Slater: “PHP has been an important part of Engine Yard’s growing family since the acquisition of Orchestra in 2011. And now, PHP on Engine Yard Cloud represents the culmination of our efforts to deliver the industry’s best Platform as a Service for PHP developers. The result of this work is a unified service offering for PHP, Node.js, and Ruby applications.” Read all about the GA launch announced by Davey Shafik at php[tek] in Chicago this week!

Data Data Data

Riak and Clusters are live! See our blog post for more info -

A cluster is a new way to organize and manage instances that share a specific function.  Clusters take much of the functionality that was once placed at the environment level, and moves it down to the cluster level. One environment can have many clusters, and each cluster can run different cookbooks and be in different regions.

We drove the cluster model hand in hand with our productization of Riak on Cloud because the distributed model of Riak paired perfectly with where we wanted to drive the future of our platform. We can now take this underlying work and begin to re-productize other offerings to take advantage of its flexibility in many ways.

Social Calendar (Come say hi!)

Tuesday May 20th: Engine Yard Dublin hosts the PHP meetup where Eugene Kenny, discusses his "Developer Toolbox", and then Matthew Weier O'Phinney of Zend Framework & Nate Abele of Lithium go head to head on the subject of Frameworks.

Wednesday May 21st: Engine Yard’s San Francisco HQ will be hosting the monthly Riak meetup! Lead data engineer and fan favorite Ines Sombra will be presenting about Riak on Engine Yard Cloud, followed by Basho’s Mark Phillips discussing Riak CS.

Wednesday May 21st: Our PDX office will be hosting Coder Dojo for students K-12 to learn about software! Grab a ticket and bring your parents for some software fun.

Thursday May 22nd: Engine Yard Dublin plays host to Open Data Ireland, “Give us our health data!”

Friday May 23rd: In which I talk about myself in the 3rd person? Tasha Drew will be speaking at Cloud East in Cambridge, UK, about deployments in the cloud, including various strategies we at Engine Yard see for environments of different sizes -- and concluding with sharing our own deployment strategy.

Articles of Interest 

Lightweight screenshot and annotation tool has gained some fans in our office!

Engine Yard friend Daragh Curran, Head of Product Engineering, Intercom shared an awesome blog here. “Shipping brings life to your team, to your product, and to your customers. Shipping is your company’s heartbeat.”

Categories: Programming

Shipping is your company’s heartbeat

Engine Yard Blog - Thu, 05/16/2013 - 12:38

Note: Engine Yard friend Daragh Curran, Head of Product Engineering, Intercom has graciously let us post this great piece about code deployment on our blog. Check it out on their own blog here.

Software only becomes valuable when you ship it to customers. Before then it's just a costly accumulation of hard work and assumptions.

Shipping unlocks a feedback loop that confirms or challenges those assumptions. It makes new things possible for your customers, and gives you the opportunity to focus on the next thing.

Shipping brings life to your team, to your product, and to your customers. Shipping is your company's heartbeat.

Shipping will try to kill you

The scramble to get that one last feature done, the late nights, the compromises, the sinking feeling when we realise something major is broken, the post-mortems… It's agony, but if it was easy everyone would do it. Shipping exposes mistakes. We're nervous about it, and our natural reaction is to do it reluctantly and infrequently, which actually carries higher risk, causing more reluctance in the future.

The cost of shipping is approaching zero

Not too long ago, shipping software involved actual ships, disks, and printed manuals. It happened perhaps once a year. Bug fixes weren't automatic over the internet like today. Everything was slower and more controlled. The cost of shipping was massive, the consequence of a mistake was large. Today, the cost of shipping has approached zero. Most people can deploy in seconds or minutes with a single command or button click. With a little thought you can do that without your customers noticing, and with automated monitoring you'll find out immediately if something goes wrong.

Despite the cost of shipping approaching zero, many people still ship software guided by very old habits.

Shipping cadence defines your company

The cadence at which you ship defines your company. A yearly cadence results in a very structured approach to the design->build->test cycle. A few months of building, while the rest is spend fixing. Engineers can join and leave before seeing their hard work end up in the hands of customers. The approach to design becomes one of anticipating all possible needs, rather than focusing and iterating on the important ones.

Obstacles downstream propagate upstream

An obstacle downstream propagates upstream. If you're not allowed to implement new ideas, you stop having them.
- Paul Graham

The right approach to shipping has a positive influence on your company's productivity and your team's happiness & job satisfaction. Shipping infrequently is an obstacle. Ship slow, and you'll introduce challenges that push you to ship even slower. Ship frequently, and see positive effects everywhere in your company. For example, lets examine how behaviour changes along with shipping frequency, while handling a simple request from a customer.

Time to production behavior

Lets say a customer gets in touch to say "No matter what I do, I cannot save my name correctly, I think it doesn't like hyphens". In a company where you ship continuously, you see this and think Simple — I'll tweak a test and a regex pattern, get a quick code review from my buddy beside me, merge to mainline, and 1 minute later when it's deployed to production, reply to the customer: "Sorry about this, it's fixed now, thanks for letting us know". They'll reply: "Wow, thanks for fixing so quickly". High fives all around!

If we stretch the time to production (TTP) out a little, even to 10 minutes, the behaviour changes. You either do the same, but reply saying it'll be fixed with our next deploy (probably 10 minutes) - or you wait, so that you can communicate with certainty. The waiting is time where you'll shift focus to something else, but have the baggage of having to follow up. Perhaps you'll think, I'll have a quick coffee, then move on to something else afterwards. Even though your deployments are entirely automated, you lose time because of waiting and losing focus.

Customer support shipping

If TTP is hours, the behaviour changes again. No longer can you say with certainty when the change will be out there, so you're tempted to batch up with other similar small changes. You postpone replying until you get time to do it, sometimes forgetting about it. You're less likely to take prompt action, wow'ing the customer, and you pay some mental cost for having it on a todo list. Since getting to production takes hours now, your team will start restricting to morning only deploys, so miss that slot and it's further delays.

If TTP is days, it exacerbates that further - perhaps you'll reply "Thanks for letting us know. We'll fix this in our next sprint". It gets bundled in with a whole load of other small low, priority items, you spend more time debating estimates, and priorities, than the first guy took to fix it and reply to the customer. Miss the beginning of week deploy window and further slippage. The larger releases bring higher risk, you'll tell your customer it's fixed, only to later require rolling back because of a separate change. Your bug database gets bigger and bigger, with little details that you'll probably never fix.

When TTP is weeks, it exaggerates that even further - perhaps you'll reply "Sorry about this, I'll let the development team know" or something equally lame from your customer’s standpoint. Deep down you realise nothing will be fixed, and the job of talking to customers becomes a cost or hassle, rather than an opportunity to improve your product and nurture happy loyal customers.

Shipping continuously

Better approaches to writing or testing software help us iterate more quickly and confidently, but the benefits are quite local to engineering teams. Continuous shipping on the other hand, touches all parts of your company, as do the benefits, and the behaviours it enables and encourages.

Linkedin's transition to continuous deployment is linked to their recent financial success.

Good products, are a side effect of combining good people with an idea in an environment that helps those people to kick ass. Your attitude to shipping is a big part of that environment you create.

Shipping breathes life into how we think. The feedback loop helps us learn, gain confidence in making quick decisions, and build momentum. Momentum in product improvements excites and engages our customers. Seeing quickly the benefits of our hard work, motivates us to do more. Building a team where people can work hard and move fast attracts others to join you - hiring gets easier.


Shipping continuously isn't an achievement you unlock and then move on. You've got to constantly obsess about it. If you believe in the benefits it brings, you'll be driven to shrink 20 minutes down to 1 minute or less, you'll consider 'ability to ship' as an equal to 'does it scale' when building new systems. And you'll do that because of all the life it breathes into your company and your product.

Shipping is your company's heartbeat.

Categories: Programming

Riak is GA on Engine Yard Cloud

Engine Yard Blog - Tue, 05/14/2013 - 20:56

Hello from NYC! We stopped by RICON East to share great news. We are thrilled to announce the General Availability of Riak on Engine Yard Cloud.

Riak is our first highly available, non-relational database and the first component of our stack to use a new cluster provisioning model. Riak exemplifies the future of Engine Yard and you should totally check it out! Here’s why.

Highlights of Using Riak on Engine Yard Cloud

Riak’s use case primarily fits applications with loosely structured data where even seconds of downtime are unacceptable. Riak has a key/value data model and is completely data agnostic, meaning you can store anything you want in a value (media, json, xml, text, etc.).

Riak is masterless. You can send writes to any node in the cluster and data will be appropriately stored, even in the case of individual node failures. Riak also supports tunable consistency, allowing you to make the datastore more strict on certain types of data and more responsive on others.

Painless Installation, Management, and Support

We have invested in simplifying Riak's installation and configuration to make the learning curve less steep. In one easy step you can define the flavor and size of your cluster, the location of your data (EBS, ephemeral, etc.), optimize your cluster by selecting desired backends, and even enable full text search.

Once your cluster is up and running you can painlessly grow it if you need to add capacity. Removing nodes is also a trivial operation. If for any reason you want to archive your entire cluster, you can easily do this, too.

Riak clusters come with the fantastic support you have come to expect from Engine Yard. As partners of Riak's makers, Basho, we can quickly escalate tickets on your behalf when they require extra engineering insight.

A Whole New (Clustered) World

The cluster model used by Riak evolves the deployment topology of  environments. Environments become more flexible with the ability to specify zero to many clusters per environment, and have all clusters properly deployed and balanced within availability zones in your region. We are also working on the ability to have clusters within a single environment provisioned in a different region.

As of today, clusters are exposed to all customers.

We will be migrating individual stack components to our new cluster model. All supported databases will be re-done and acquire the provisioning features you see in Riak. We are very excited about what we'll be releasing over the next few months.

Introducing Cluster Behaviors

The cluster provisioning model also allow us to express cluster-specific behaviors and act upon them in a scheduled way (or on demand). For example: all Riak clusters have access to rolling backups as their first supported behavior.

With rolling backups we can archive the entire contents of a cluster one node at a time without compromising its overall performance and ability to respond to requests.  We will be introducing new behaviors (like rolling snapshots) very soon.

Things You Must Know

To prepare for the migration of legacy components to clusters we have decided to change the way environments update. We have pushed stack responsibilities down to the cluster level. This means that clusters are now responsible for managing their stacks and updates which gives us greater granularity and flexibility (it’s a great thing, we promise!).

An important thing to note is that environment-wide custom Chef runs will no longer be applied to cluster instances. Clusters are isolated from system-wide versions of Chef as they carry their own stack and updates.

What Comes Next?

Here are a few things we have in store as we continue to evolve Riak and clusters:

We want to make Riak’s management tasks more intuitive than ever, so we will roll out enhancements to the environment page and overall cluster user experience.  We are also working towards improve cluster monitoring and alerting.

Enhancements to instance booting times are in the pipeline. You will be able to go from zero to a fully running cluster faster than ever!

Where Can I Learn More?

Our documentation has been updated and it’s a great place to get started. We will be leveraging Basho’s excellent Riak documentation, too.

If you are in San Francisco we will be giving a tour of Riak on Engine Yard on May 22nd. Come ask questions! We’ll hand out a few gifts to the best ones

Still Have Not Tried Riak?

Riak is available on all trial accounts. Simply sign up, boot up a cluster, and you’ll be able to experiment with it.

Also feel free to open a Support ticket if you are wondering if Riak is a good fit for your application.  We love hearing from our customers and want your feedback.

Categories: Programming

Announcing PHP on Engine Yard Cloud

Engine Yard Blog - Tue, 05/14/2013 - 15:45

We’re excited to announce the general availability of PHP on Engine Yard Cloud.

PHP has been an important part of Engine Yard’s growing family since the acquisition of Orchestra in 2011. And now, PHP on Engine Yard Cloud represents the culmination of our efforts to deliver the industry’s best Platform as a Service for PHP developers. The result of this work is a unified service offering for PHP, Node.js, and Ruby applications.

With PHP on Engine Yard Cloud, users get a proven, robust platform on which they can both horizontally and vertically scale applications – including content, media, e-commerce, and more. As a highly configurable PaaS, Engine Yard Cloud gives PHP developers – from enterprises to digital agencies to SMBs – a wider range of instance sizes, a fully curated PHP stack, and advanced automation and orchestration features such as database replication and failover.

Whether deploying a simple Wordpress blog or an advanced MySQL-backed web application, developers get a range of control over configuration, deployment and management of their application environments, including full root access on virtual servers and the flexibility of using custom Chef recipes to control and automate entire environments, regardless of size.

Get Started With Our Lowest Entry-Level Cost Ever

We recently announced several big price reductions including a new entry level price that gives you a dedicated EC2 small instance for $0.05 per hour. That's an average of $36.50 per month — almost 50 percent less than the original price! This means you can immediately start using Engine Yard Cloud to deploy your PHP applications at an entry level cost so low, it's less than the cost of a basic application on Orchestra.

What’s more, if you haven't already made use of the free trial, you can login to Engine Yard Cloud with your existing login and claim your free 500 hours to get started!

Want to try it out? Head over to our documentation and give things a whirl.

What Does This Mean for Orchestra Customers?

We plan to retire Orchestra later this year, as we have already communicated to our Orchestra customers. In fact, we are already working with some customers to help them migrate to Cloud. And if you haven't already migrated, there are several reasons why you might want to try PHP on Engine Yard Cloud right away.

Some of the benefits of PHP on Engine Yard Cloud:

  • Choose the dedicated instance sizes you need
  • Run your database in your environment. No more third party providers required!
  • More control over your deployments
  • SSH access. Logs. Debugging.
  • Automated backups and snapshots of your environment
  • Stop and start environments

If you haven't migrated yet, and you can open a support ticket and we will work with you on the migration. Or you can read more about our plans in the unification FAQ.


We know we couldn’t have gotten this far without the support from this community, so we’d like to say a big “THANK YOU” to everyone involved. The whole Orchestra team is now working on Engine Yard Cloud. And we hope you’re as excited as we are about the expanded PHP service with more deployment choices, increased flexibility, better management, and — as always — the industry’s best support included.

Please note: GA features will go live at 1 pm PST today.

Categories: Programming

Mobile Application Privacy: 10 Tips to Protect Your Brand, Reputation and Customer Data

Engine Yard Blog - Mon, 05/13/2013 - 17:23

Note: Engine Yard friend Himanshu Dwivedi, CEO/Founder, Data Theorem has graciously let us post this great piece about mobile security on our blog.

Mobile privacy is one of those hot topics lately. Every week there seems to be another article about a mobile app or platform having a privacy issue. Indeed, Path was recently hit with a $800,000 fine by the FTC for breaches in privacy. The focus on privacy extends across application platforms to even BlackBerry which have been well regarded for their security measures. The tricky part of providing privacy for your customers on your mobile application is that there are many kinds of data stored as well as places this data can be stored.  Here are the top 10 tips to ensure privacy for your customers as you develop your mobile application.

Please Don't:

1) Use UDID and equivalents

This feature has been deprecated in iOS 6.0 but still largely pulled. This is an advertiser's dream but it tracks without the users knowledge which raises concerns with privacy advocates.

Recommendation: Generate an app-specific random session ID that can't be cross-linked by other applications.

2) Be careless with geo-location

Treat geo-location tags and data as sensitive. This means that you don't want to send information over the network in clear-text. Another thing to watch for is storing this information client side where there's a cross-link with identifiers, making it vulnerable to hackers.

Recommendation: If you're going to use geo-location, be sure to store it server side but remove it from these systems over time. Long-term storage can imply that you are tracking your customers over a long period of time including where they have been which is a big no-no from a privacy perspective.

3) Do not access contact lists without permission

While Contact lists are a great way to get more users, especially with a "cold start" applications, unfortunately privacy advocates don't like it. The case of Path demonstrates that storing contact list server side and cross-linking is a bad idea and one that may get you slapped with a  fine.

Recommendation: Accessing contact list is one thing but strong server-side is a a big no-no. Do not access a user’s contact list without permission, even if it means for a better user experience.

Watch Out for…

4) The Copy/Paste function

In iOS you want to be aware of the UI paste board as information stored in the clipboard (aka UI Pasteboard) is accessible to all other applications on the device. This is important because applications have been written specifically to monitor the clipboard.

Recommendation: Even though it's a UX trade-off, disable the UI Pasteboard especially for sensitive fields: username. passwords, phone numbers, addresses.

5) Cache.db

The cache.db file was introduced between iOS 4.x and 5. The information stored on cache. db database file is not encrypted and accessible to anyone who has access to the device. There isn't a great deal of documentation on this so often developers don't realize what information the cache.db is collected.

Recommendation:Turn off caching using NSURLCache settings.

6) Auto-Correct

The iOS caches each non-numeric keyboard tap of the user for the auto-correction feature to work, which cannot be disabled by any application. Since almost every non-numeric  is stored, it's important to disable auto-complete for confidential fields such as address, mother's maiden name, city of birth, etc.

Recommendation: Set atuocorrectionType prperty to UITExtAutocorrection NO.

7) Backgrounded Screen

A backgrounded screen is when a screenshot is taken with a state change, such as an incoming call. All applications on the device have access to the screen shot which may include confidential data. Luckily this is an easy fix.

Recommendations: Detect state decision of the application specially applicationDidEnterBackground. When the app is backgrounded, enable a splash screenshot that consists only of nonsensitive data such as the organization's logo.


8) Keychain

There are keychain dumper tools to dump everything out of the keychain. If you want data that is secure at all costs you shouldn't put it in the keychain.

Recommendation: Many items in the KeyChain are accessible, but just not of the box.

9) Encryption

If the key to decrypt data is stored client side on the device, it only slows attackers with physical but does not protect the data 100%. For offline mode, decryption keys are often stored on the device itself.

Recommendation: To truly encrypt data, ensure the decryption key (private key) is stored server side only

10) SD Cards

External storage (e.g. SD cards) have no file permission protection which means the data is accessible to all applications (copy, tamper, etc).

Recommendation: Do not store any private/confidential data on external SD cards. If data needs to be shared between device apps, store the data server-side  and allow access to the storage using client-side authentication/authorization tokens. Be aware that the Sandbox model does not apply on the SD card either the Android or Windows phone.

With these tips, your application will definitely be up-to-date with the latest in mobile security and privacy.

With the widespread adoption of mobile applications, ensuring the security and privacy of your application and customer data is paramount. By applying these tips, you will help ensure your application is up-to-date with the latest in mobile security and privacy.

Categories: Programming

May 10, 2013: This Week at Engine Yard

Engine Yard Blog - Sat, 05/11/2013 - 01:53

I’m heading to Ricon East with our lead data engineer, Ines, and dapper platform engineers Josh and Thom! Come say hi!

For our php friends, my counterpart, Josh Hamilton, will be at php[tek] with Davey and PJ, who would also enjoy a friendly “wassup!”

--Tasha Drew, Product Manager

Engineering Updates

Application takeover preferences are now in Early Access. For customers who need a non-standard application takeover scenario, you can now select between boot-options, or disable entirely within the UI.

We have made great progress towards availability of provisioned IOPs on volumes for legacy instances (Riak clusters have had this feature for a while).  We are making the feature available to customers in Limited Access this week. We do have some more work to do  on improving the UX and providing documentation before making it more widely available -- please open a ticket with support if you are interested in checking it out before its Early Access release.

Data Data Data

We continue to enhance the behavior of new Clusters. Rolling backups will be the way to permanently archive data stored in a Riak cluster.

Rolling backups extract data one node at a time while your cluster continues to server requests. You will be able to manage the extracted backup files from the UI and even see which nodes they came from!

Social Calendar (Come say hi!)

Monday May 13 - Tuesday May 14th: Engine Yard is sponsoring the lightning talks at Ricon East 2013! We will also have a bunch of people in attendance. Come say hi!

Tuesday May 14th - Friday May 17th: php[tek]!!! Davey Shafik will be giving a talk, and we will have a product manager and engineers on hand to join in the festivities.

Tuesday May 14th, San Francisco Office:Product Lover’s: PM Fast-Track: What do Product Managers really do?

Tuesday May 14th, Buffalo Office: WNY Ruby: May we all enjoy our Rubies!

Tuesday May 14th, Dublin, Ireland Office:Crafthouse #003: looking at the various ways in which we learn web design and investigate ways to improve upon them.

Wednesday May 15th, PDX Office: Coder Dojo PDX, K-12 Coder Night

Wednesday May 15th, Buffalo Office: Girl Develop It Meetup: Code & Coffee Night!

Thursday May 16th, Buffalo Office: Database Seminar, May is for MongoDB

Thursday May 16th - Friday May 17th:NodePDX

Thursday May 16th: Dublin, Ireland Office:UXPA Ireland: My favorite UX tool!

Articles of Interest

Did you miss Chef Conf? Check out all the videos here!

The East Coast of the USA is about to overrun with billions of cicadas. Humans will be outnumbered approximately 600:1.

Davey Shafik does a dive into authentication choices he investigated as he built out the Distill website.

Categories: Programming

Authentication: Not necessarily a social activity

Engine Yard Blog - Thu, 05/09/2013 - 21:18

For Distill, Engine Yard's developer conference, we chose to use social authentication to reduce the barrier to entry for our call for papers. We supported Twitter, Facebook and Github.

While developing the site, my concern was that the registration flow be simple, and that it actually work. Once we launched the site, I realized that I had trouble remembering which provider I had signed up with.

Maybe that's just me (I am terribly forgetful!), but I imagine at least a few other people had this issue. Sure, on the backend we can link multiple accounts, but that means users went through the registration process multiple times. This is not optimal.

For those interested in the numbers, here is how the providers stacked up:

  • Github: 59%
  • Twitter: 38%
  • Facebook: 3%

Why did we make this choice? Probably the same reason everyone else does:

Users don't want, or need another fricken login to remember, just use Facebook/Twitter/Google+/LinkedIn/Github/Yahoo!/...

This is the primary argument for using social auth. And lets be honest, who wants to be in charge of Yet Another Login System?

But is social auth the best option? Lets explore that.

In favor of social auth for our users, we have:

  • One less password to remember
  • Possible to revoke access
  • Automagic integration with my online social presence (that I can control… if I know how!)
  • Users are often always logged into their social sites, so they don't even see a login screen — a few redirects and it can be avoided entirely.

In favor of our bespoke system... it's the same thing we've been doing for years.

What does Social Auth mean to our users?

Let's break down what all of these things really mean.

One Less Password

Is this a good thing? Just like using the same password for everything, using the same social account for everything is not necessarily a good thing.

Sure, we can use an 80 character, expanded charset password on that social account, but "guessing" your username and password isn't necessarily the only way in. Server security breaches, man-in-the-middle attacks (negated by proper application of SSL), fishing or social attacks are still out there! Just like with a bespoke system.

I personally use 1Password as a tool to maintain a list of the literally hundreds of accounts I have, and their respective login credentials. I generate a random password for every site I sign up for, and never even think about remembering it.

I have my password database stored in Dropbox (which, yes, means I'm trusting Dropbox's security) so it's available on all my devices, and it can even function standalone with a built-in web interface!

There are plenty of other free tools even that will allow you to do the same thing (e.g. KeePass Password Safe).

For me the one-less-password argument holds little water.

Possible to revoke access

This one is an argument I rarely hear, but is quite important. Most websites (though it's changing, as people get on the free-data bandwagon) do not allow you to delete your account. And there's a good reason for this: Your data is valuable to the website, even if you're not using it! (Remember: if something is free, you [and your data] are what brings value to the business.)

With social auth, you can not only revoke permissions entirely (denying access to your social data that they haven't yet collected), you can revoke permissions partially (assuming the social site implements that).

I think this is quite important.

Automatic Integration with Social Presence

Social integration is arguable the main reason for even using social authentication (other than the lazy factor), and it can definitely bring value to our experience.

The point of being social, is sharing things with people, and good online experiences are things we want to share. Making that quick and easy is beneficial to both the user, and the business — it's word of mouth advertising, and is priceless.

However, a lot of users don't want to be social. Either with your site in particular ("I don't know this site, why would I want it to see my stuff?") or in general ("I'm a grumpy curmudgeon" or "I don't want the government to spy on me!").

The general answer to this problem, is to make social auth optional. Users can sign up for a bespoke account, or signup via social auth. Or Both.

Unfortunately, I find that when multiple signup options are available — be that bespoke + one social auth, bespoke + multiple social auths, or just multiple social auths — I forget which one I've used. Did I sign in with Github? Or did I create a new account?

1Password does help me in some regard here, because it would have my bespoke credentials available, but with multiple social auth options? It's no help.

Automatic Login

Automatic login is arguably a good thing — it can allow anyone with access to the computer to not only access the social platform (because you're already logged in), but who knows what else.

Think about the people who leave their Facebook logged in at the Apple Store, apparently nobody has yet realized you can look at the list of Apps they have authenticated with and then can simply visit that site and choose to login with Facebook. Suddenly you now have access to their Pandora, Instagram, Klout, and 100 other apps you've authorized!

Now, it is possible as a developer to require login, at least with Facebook auth — but we usually want it to be easy for our users and don't bother with it!

What about a middle ground?

Is there a middle ground? We've already discussed the pitfalls of providing bespoke + social auth(s), so what else can we do?

I think the best middle ground, is to provide bespoke authentication, and then behind that, allow for social connection expressly for the purpose of being social. That is, make it optional.

We can get some of the benefits of social authentication, by allowing users who have connected their social accounts to use them for password resets, rather than email. Simply ask them to re-verify their social account and once they have, you can direct them straight to the password reset form — no emails getting lost, no tokens, simple.

One final option, is that you can use social authentication without getting access to more than the users basic information — especially, you can do it without requesting write permissions. Then, later, you can ask for write permissions should the user wish to utilize that aspect of your site.

Being Socially Responsible — A Social Contract

I have, over the course of thinking about these things, decided how I'm going to interact with my users socially.

  1. Social integration is always optional.
  2. Permissions are granted only as-needed. Ask for the minimum permissions to do the requested action, and no more. When more permissions are needed, ask again.
  3. Always give the user the final say on what gets posted — I will always allow my users to edit the message, and never insert automated text on the end — except the URL to thing they are sharing.
  4. Never automatically post anything. This is really part of the previous one, but it's better to be explicit.

This is my social contract, it will be publicly posted on my site, and presented when users look at the option to connect their social accounts. I think this is the responsible way to interact with my users, and allow them to interact with folks who will hopefully become my users.

What about you? What's your social contract look like?

We recently announced the Distill speaker lineup and first batch ticket sales.  To learn more, visit

Categories: Programming