Each week in our "Who's @ Google I/O" blog series, we'll highlight the latest from a featured track at I/O. This week, the spotlight is on Enterprise -- a major theme of this year's event.

In 2009, we saw an increasing number of large companies moving to the cloud (and "Going Google"), choosing the web as their platform of choice. At I/O, we'll share our enterprise and commercial developer offerings and focus on how to build business apps in the cloud.

We'll be adding new Enterprise sessions over the next couple of months, but here's a preview of some of the sessions you'll see at I/O:

Customizing Google Apps & integrating with customer environments
Hear real-life examples of customizing Google Apps to meet customer requirements from several panelists, including two of our Sandbox participants -- Iein Valdez of Appirio and Michael Cohn of CloudSherpas. Explore integration issues and deployment best practices with the people who have done it.

Run corporate applications on Google App Engine? Yes we do.
Our CIO, Ben Fried, describes how Google IT and other companies use the latest Google App Engine enhancements to respond more quickly to business needs while reducing operational burden to near zero.

It’s 2010: How is your move to the cloud doing?
Come discover the latest innovations from Google enabling IT and ISV developers to build on Google's cloud-based storage and computing offerings. This talk will give a complete overview of Google's commercial developer products and provide insights and best practices so enterprise developers can take more advantage of the cloud.

Launch your app inside of Google Apps with gadgets
Gadgets represent a valuable opportunity to get in front of the many Google Apps users who use Gmail, Google Calendar, and Google Sites throughout the day. This session will talk about how you can write gadgets as natural extensions of your existing products and take advantage of the unique opportunities available to gadgets in Google Apps.

Making Freemium work - converting free users to paying customers
Don Dodge will moderate a panel of prominent venture capital leaders (Brad Feld, Dave McClure, Jeff Clavier, Matt Holleran) to help you understand how to build free apps that can be upgraded to paid & how to build products that can be profitable.

You can find the current list of Enterprise sessions here.

To learn more about and register for Google I/O, visit code.google.com/io. We add new sessions and content to the I/O website each week so follow @googleio on Twitter to keep up with changes!

By Joyce Sohn, Google Developer Team

Morgan Tocker has an awesome article and comment thread in the MySQL Performance Blog about When should you store serialized objects in the database? Before the NoSQL age is was very common to simulate schemalessness by storing blobs in MySQL. Sharding was implemented by running multiple MySQL instances and spreading writes across them. While not ideal for the purpose, developers felt comfortable with MySQL. They knew how to install it, back it up, replicate it, in short:  they knew how to make it work. Yet they also needed to store objects without the penalty of joins. Searches and aggregate queries were handled by indexes kept in separate tables, this offloaded the fast path to objects.

This all made perfect sense. Usually we just want stuff to work and going with what you know is often the best path to that goal. And what we have known is MySQL. All the different pros and cons of this approach are covered wonderfully in the post.

But the world has changed.

Structural typing in Scala is the way to describe types by their structure, not by their name as with other typing. Structural typing reduces coupling and the need for inheritance. In Java you would mainly use interfaces instead of structural typing. I go with the same examples as in “Scala Goodness: Compound types” so it’s easier [...]

You may have seen this on Matt Deacon's blog already ... I'm running a session for the IASA UK chapter on the 9th of March in London.

Where do you start?

One of the hardest things about software development is being asked to come up with a design when all you're given is a set of requirements and a blank sheet of paper. Many software teams will dive straight into the code and while this can initially be very productive, the slippery slope of constant refactoring is awaiting those teams that haven't quite found a design that works. Often, a little forethought is all that's needed to get the development process heading in the right direction. So where do you start?

This session will answer this question, presenting some simple techniques for tackling software architecture while dispelling the myths about the need for complex tools and big design up front.

I'm planning the content for this session at the moment and it's going to be a mix of presentation, discussion and probably some group work to get you all drawing a few boxes and lines. We're going to cover a very very quick overview of what software architecture is all about before moving on to see what drives and influences it. Once this is done, we'll look at how to actually design software from a blank sheet of paper, focussing on some techniques to help you determine what components you need and how best to organise them. A nice side effect of the techniques we'll use to define architecture is that they can also be used to share architecture, so we'll wrap up by looking at how to document software architectures in a simple yet effective way.

You can find more information about this event and register here. I'm really looking forward to it and hope to see you there.

Many software testers I know are relatively young, just finished their study and haven’t much experience with complex business processes for example in the banking or pension business. And specially these people have to test, for instance the finance and insurance application of large banks and insurance companies. It looks like they are selected especially [...]

One of the things which Nat Pryce and Steve Freeman suggest in their book Growing Object Oriented Software guided by tests is the idea of wrapping any third party libraries that we use in our own code.

We came across a situation where we did this and then later on I made the mistake of not following this advice.

To start with my colleague David had created a DSL which kept all the calls to Selenium nicely wrapped inside one class.

The problem we were experiencing was that we hadn't evolved the DSL with the webpage evolution to the point where we weren't taking into account that some fields on the page weren't visible until ones before them had been filled in.

We needed to change the DSL slightly and it seemed like an interesting opportunity to try and convert them to use Webdriver as it seems more suited for heavy filling in of forms which is our use case.

My first thought was that it should just be possible to change those Selenium calls to call the equivalent Webdriver methods instead but having done that we realised that the way the two tools interact with the page is slightly different so the direct replacement approach wasn't really working.

We decided to adopt a different approach whereby we would just try and change individual tests to use Webdriver instead and leave all the other tests as they are using Selenium.

I thought about creating another version of the DSL to encapsulate the Webdriver interaction with the page but decided against the idea as it didn't seem like it would add much value and the only way I thought of at the time was to create a clone of the original DSL.

We managed to get one of our tests working more effectively using Webdriver having sorted out the problems with the different interactions between fields but unfortunately the current C# API doesn't seem that stable and seems to fail somewhat randomly for reasons we haven't been able to work out yet.

As a result we now want to convert those tests I'd rewritten to take advantage of the new way they're written but to use Selenium instead!

Sadly the approach I took has made this really difficult and it's now a very frustrating journey to get the tests back into shape.

It's quite frustrating to make this type of mistake especially when I read about a solution so recently.

In hindsight I think a better approach may have been to pull our an interface to represent our DSL – currently it's a series of method calls on a few classes – and then create Webdriver and Selenium specific versions of that.

A few of things stand out for me from this experience:

• I talked myself out of wrapping Webdriver because I saw the main reason for doing so being to shield us in case we chose to change the library and I didn't anticipate having to do that. As it is I was wrong but I didn't totally appreciate how we can benefit from defining an API that defines how we want to interact with the web page rather than how the library wants us to.
• We need to evolve our DSLs with the application and not be afraid to change them if the application changes.
• It probably wasn't a good idea to try and fix the DSL and change the underlying library at the same time. Small steps!

I read an article today about how Chrome has increased a lot in market share in the past year, mostly at the expense of Firefox.  I wanted to see how Chrome usage on this blog has progressed in the past year so i got some stats from Google Analytics and created this little graph:

(note: these numbers are obviously only from visits to this blog and are thus not representative of the entire internet (yet) etc…)

I started at the month before the first Chrome users were reported to give you an idea of Firefox’s score at that time.  Naturally, the people that come to this blog are into technology so it’s no wonder that Firefox is the dominant browser instead of IE.  Before Chrome entered the picture, nearly 70% of all visits to this blog in August 2008 were from Firefox users.  In its first month, Chrome immediately got about 8%, which i thought was pretty impressive.  Firefox’s numbers dropped below 50% for the first time less than a year later.  That’s a pretty big impact in a relatively short time.   And as you can see, Chrome usage has been growing a lot in the past 6 months ago, at the expense of both Firefox and IE.  I wouldn’t be surprised if Chrome catches up to Firefox in another 6 months.

I’m not fanatic about any browser (as long as i don’t have to use IE, right?)… i use Firefox for OS X at home, but i use Chrome at work.  I’m not really into the whole plugin/extension thing (except for Adblock obviously) so i’m not really ‘tied’ to a specific browser.  Now, some people are pretty fanatic about their browser so what i’m wondering is: what would it take for you to switch to another browser (no matter which one you use now)?  Are you hooked on extensions that you can’t go without in another browser?  What was the reason you picked your current browser and why are you sticking with it? Or is it all pretty much the same for you?  Do you frequently use multiple browsers?

In some weird way, i find that kinda stuff interesting so please do share

This year, we're introducing I/O BootCamp, a new event happening the day before Google I/O. BootCamp is an opportunity for attendees to get a crash course in our major development platforms and tools before they head into Google I/O. BootCamp will feature introductory "101" content, hands-on lab sessions, and community-led discussions.

BootCamp is only available to those who are registered to attend Google I/O. Since space is limited, we ask that interested Google I/O attendees please register at our BootCamp site.

To register for Google I/O, please visit code.google.com/io.

By Stephanie Liu, Google Developer Team

Managing software packages on Unix has always been, to put it politely, a giant pain, and most Linux distributions are built around the different ways we’ve all been trying to alleviate that pain. In this post, I’ll walk you through Homebrew, a fantastic new option for package management made simple.

Pre-Homebrew, there were various attempts to create effective package managers for OS X. The two most popular efforts were Fink and MacPorts, but they each had their frustrations. In both cases, creating packages or portfiles was still complex and difficult.

Max Howell’s done a great job with Homebrew; it’s easy to edit, and creating new packages is a breeze. Let’s dig in!

What Does It Do?

The pitch is simple: Homebrew alleviates the drudgery and repetition of downloading and installing Unix software packages on OS X. If you’re sick of ./configure && make && make install, Homebrew can help.

Why Homebrew?

As previously mentioned, OS X already has two package managers: Fink and MacPorts. If one of those is working for you, great. But if you’ve been frustrated by them in the past, I strongly suggest you give Homebrew a try. It’s easy to create and edit formulae, and even to edit Homebrew itself, since the core is just a few hundred lines of Ruby code.

It doesn’t impose external structure on you: the default is to install it to /usr/local, but you can install it anywhere. Inside your Homebrew directory, software is installed in subdirectories inside Homebrew’s cellar, like Cellar/git/1.6.5.4/. After installation, Homebrew symlinks the software into the regular Unix directories. If you want to hand-install a package or version that isn’t officially part of Homebrew yet, it can happily coexist in the same location.

That’s usually not necessary, though, since formulae can install directly from version control. If a package has a public git, svn, cvs, or mercurial repository, you can install the latest development version as often as you’d like with a simple brew install.

Installing packages is faster, too, because Homebrew also works hard to avoid package duplication. No more installing yet another version of Perl as a package dependency when you already have a working install of Perl built into OS X. Best of all, Homebrew has a basic philosophy that you shouldn’t have to use sudo to install or manage software on your computer.

Sounds Pretty Great… How Do I Get It?

The first (and only) dependency that Homebrew has is the OS X Developer Tools, which are on the OS X installer disc, and available from Apple as a free download.

Unless you have a reason not to, the easiest place to install Homebrew is in /usr/local. You can do that in just a few steps on the command line:

# Take ownership of /usr/local so you don't have to sudo
sudo chown -R `whoami` /usr/local
# Fix the permissions on your mysql installation, if you have one
sudo chown -R mysql:mysql /usr/local/mysql
# Download and install Homebrew from github
curl -L http://github.com/mxcl/homebrew/tarball/master | tar xz --strip 1 -C /usr/local

Once you’ve done that, you’re good to go! Assuming /usr/local/bin is in your PATH, feel free to try it out:

brew install wget
brew info git

The Homebrew wiki also has more about integrating with RubyGems, CPAN, and Python’s EasyInstall.

Keeping your copy of Homebrew up to date is easy, too:

brew install git
brew update

Once you have git installed, you can just run brew update any time you want to pull down the latest formulae.

Contributing

Creating a new formula is almost that easy. If Homebrew didn’t have a formula for wget, you could create one like this:

brew create http://ftp.gnu.org/gnu/wget/wget-1.12.tar.bz2

After you save your formula, you can test it out with brew install -vd wget, to enable verbose logging and debug mode. If you need help getting your formula working, there’s more documentation on the Homebrew wiki. You can also learn by example from already existing formula, like git or flac.

You can check out lots of example formulae, as well as the internals of Homebrew, by running brew edit. The code is pretty straightforward. If you have questions, or are interested in future plans, the contributors to Homebrew tend to hang out in the #machomebrew channel on Freenode.

Once you have a working new formula, it’s easy to create your own fork of Homebrew on GitHub to push your new formula to, by using the github gem:

git add .
git commit -m "Added a formula for wget"
gem install json github
github fork
git push <your github username> mastergitx

After pushing your change to GitHub, go to the Homebrew issue tracker and create a ticket with the subject “New formula: “. Assuming everything checks out, your formula will be added to the main Homebrew repository and available for everyone else to use.

Wrapping Up

Homebrew is a compelling alternative to MacPorts and Fink. The Homebrew core and all the formulae are written in Ruby, so it’s easy to add new packages or even new features. If you’re looking for more control over the Unix software you have installed on your Mac, or you’ve been frustrated by other package managers in the past, check it out. I think you’ll be happily surprised.

There is one of those semi-heated discussions on an agile forum around CMMI, Agile and the confusion between them. Here's a summary from CMMI DEV V1.2. Note that the software development activities live in Engineering.

An Update

I'm presenting to a graduate class at Carnegie Mellon West Monday and Tuesday of this week. As well I'm in a loop discussion on an agile forum about CMMI and agile.

It's breathtaking how many people confuse CMMI with a software development method. Much in the same way there is confusion between PMBOK and a project management method.

The chart above - if you in fact take CMMI as a framework for software based product development maturity assessment - shows where "development" activities live and where other activities live. CMMI says you need all these process areas to increase the probability of success.

So agile software development provides methods to fulfill some of these process areas. Specifically the one in the Engineering Process Group. But there are process area where agile has little our nothing to say.

So Agile is not project management in the sense used by those defining the processes of project management. If you redefine the process needed to increase the probability of success of the software project, then maybe you can call agile project management.

BigDataMatters is focused on the issues faced when processing and managing large amounts of data. In light of this, it would be a crime not to blog about the security of this data. Over the next few weeks, I will write a series of posts focused on identity management in the enterprise. Before you read any more, how is your identity secured?

Read more on BigDataMatters.com

 

In an earlier blog (Nov 2009: Closing the Gap between Business and IT) I described the gap between business and IT and suggested a way forward: we must speak the same language. That language must be more than just a spoken language; some simple drawings or models are often useful. However, beware of business models inspired by software models, which assume an underlying abstract computational machine. We must work together pursuing common goals and results – without resorting to passing documents from one side to the other. And we must deliver high-quality results on regular intervals.

Forrester says we should have a “fusion” of the business and IT, going beyond mere alignment. The idea is sound, but it is better to get them to play on the same team together and win.

In a previous column (May 2009: Scaling Agile Teams), I discussed the structure of a soccer team. A soccer team has specific positions with specialized skills: goalkeeper, defender, forward. Despite the specialization, anyone can kick the ball, including the goalkeeper. If we look at the business-IT “team” we also have many kinds of specialists. From the business we have people with knowledge of business processes and resources of various kinds (human, machines, etc.). From IT we have people with knowledge of how to write code, people with knowledge of how to test software, and people who have knowledge of how to understand needs and devise solutions. Despite this specialization, all of the participants need to contribute to achieve a common goal in order for everyone to be successful.

On a soccer team everyone must also understand the basic rules of the game in order to contribute to winning. On the business-IT team everyone needs a common understanding of how they will work together to build software for the benefit of the business. For business people, this means understanding how software can be built in many small steps based on a long-term roadmap, and how their participation is essential. They need to understand how to participate in the development effort by communicating the essence of their business processes and articulating their needs for improvements in ways that preserve the ability to devise creative solutions. For the IT people this means understanding the business processes and the desired outcomes that the business is looking to achieve, and it means being able to devise creative solutions that create business value. Everyone must know how to play their part and work together for the team to win.

Perhaps “fusion” is not quite the right word – since business and IT will continue to exist as separate units, and they will always have some things that they do alone. But they must be able to join together on a team consisting of both business and IT people, working toward a single goal: deliver creative solutions that create business value.

The team will need to agree on a way of working, and on the practices they think they need to solve the problem at hand. These practices may involve proven techniques such as developing iteratively, formulating requirements and tests as use cases and test cases, planning product releases based on scenarios, measuring progress through successfully passed tests, and so on.

These practices will involve both sides in meaningful collaborative work that, if correctly performed, leads to excellent result. The business and IT will win together. This would be smart!

Of course, this would only be a start. In my next blog I will talk more about some of the other things that are needed.

One of the more interesting higher order functions that I've come across while playing with F# is the partition function which is similar to the filter function except it returns the values which meet the predicate passed in as well as the ones which don't.

I came across an interesting problem recently where we needed to do exactly this and had ended up taking a more imperative for each style approach to solve the problem because this function doesn't exist in C# as far as I know.

In F# the function makes use of a tuple to do this so if we want to create the function in C# then we need to define a tuple object first.

public class Tuple<TFirst, TSecond>
{
	private readonly TFirst first;
	private readonly TSecond second;
 
	public Tuple(TFirst first, TSecond second)
	{
		this.first = first;
		this.second = second;
	}
 
	public TFirst First
	{
		get { return first; }
	}
 
	public TSecond Second
	{
		get { return second; }
	}
}

public static class IEnumerableExtensions
{
	public static Tuple<IEnumerable<T>, IEnumerable<T>> Partition<T>(this IEnumerable<T> enumerableOf, Func<T, bool> predicate)
	{
		var positives = enumerableOf.Where(predicate);
		var negatives = enumerableOf.Where(e => !predicate(e));
		return new Tuple<IEnumerable<T>, IEnumerable<T>>(positives, negatives);
 
	}
}

I'm not sure of the best way to write this function – at the moment we end up creating two iterators to cover the two different filters that we're running over the collection which seems a bit strange.

In F# 'partition' is on List so the whole collection would be evaluated whereas in this case we're still only evaluating each item as it's needed so maybe there isn't a way to do it without using two iterators.

If we wanted to use this function to get the evens and odds from a collection we could write the following code:

var evensAndOdds = Enumerable.Range(1, 10).Partition(x => x % 2 == 0);
 
var evens = evensAndOdds.First;
var odds = evensAndOdds.Second;

The other thing that's nice about F# is that we can assign the result of the expression to two separate values in one go and I don't know of a way to do that in C#.

let evens, odds = [1..10] |> List.partition (fun x -> x % 2 = 0)

We don't need to have the intermediate variable 'evensAndOdds' which doesn't really add much to the code.

I'd be interested in knowing if there's a better way to do this than what I'm trying out.

AntiPattern Name: Groundhog Day
General Form:
Important architectural decisions that were once made get lost, forgotten or are not communicated effectively.  
Symptoms and Consequences:

• People forget or don’t know a decision was made.
• The same decision is made more than once, possibly differently.
• New people joining the project don’t understand why a decision was made.

Refactored Solution:

• Capture important decisions in the “Architectural Decisions” work product.
• Ensure a process is in place for making and ratifying decisions (maybe a Design Authority responsibility).
• Ensure decisions get known about by all the right people.

This is an excerpt from my article Building Super Scalable Systems: Blade Runner Meets Autonomic Computing in the Ambient Cloud.

If datacenters are the new castles, then what will be the new gunpowder? As soon as gunpowder came on the scene, castles, which are defensive structures, quickly became the future's cold, drafty hotels. Gunpowder fueled cannon balls make short work of castle walls.

There's a long history of "gunpowder" type inventions in the tech industry. PCs took out the timeshare model. The cloud is taking out the PC model. There must be something that will take out the cloud.

Right now it's hard to believe the cloud will one day be no more. They seem so much the future, but something will transcend the cloud.

I just returned from Tokyo, where I keynoted at JaSST, the Japan Symposium on Software Testing. 10 years ago, when they started the conference, maybe it was just about testing, but now it’s evolved to be about quality in the organization.

Some highlights from my trip:

• Everyone (and everything) I met appeared quite orderly. Everything had a place and everything was in its place. I saw this at the lost-luggage counter, in the hotel, and at the conference.
• I was pleasantly surprised that the subway ticket machines had an “English” button so I could buy my ticket and know what I was doing. The maps were in English as well as Japanese, so I could know in advance what my trip would be and which stop to get off at. I had a little trouble with which track, but that’s probably because I was jet-lagged.
• I was pleasantly surprised to see evidence that the simultaneous interpretation for my keynote worked fairly well. I could tell because people laughed when they were supposed to :-)
• For the tutorial, I did not allow enough time for the consecutive interpretation or for the questions about agile, so I needed another 20 minutes, which I did not have :-(
• I was a little concerned that when the panel prepared for the questions, I thought we might be boring. Nope, we were thought-provoking and funny.
• My Japanese hosts were amazingly solicitous and helpful for my entire experience: to/from the airport, to/from the conference, to/from sessions at the conference

I had a blast. I hope I have an opportunity to return to Japan. Now, all I have to do is get enough sleep so I’m awake during the day…

Tweet This Post

Let’s talk about programming magic. It seems hip to use the most powerful language or framework and power often goes hand in hand with magic. Many kinds of magic are available, especially in languages which are renown for their magic, like Ruby. But there is a lot of magic even in Java: Class loading Garbage collection Terracotta (which [...]

You didn’t have the time or resources to travel last year and regret that you have missed some conferences? Now you can find a lot of complete conference sessions recording on the Web. My title has obviously a little bit of marketing twist, but I share with you a fair and diversified selection of excellent conferences presentations videos. Here is my list “in no particular order” as they say on TV.

What they Don’t Teach You About Software at School: Be Smart!

This is a keynote given by Ivar Jacobson at Jazoon. One of the most popular buzzwords in software development is agile. Today everyone wants to be agile. That is good! However, being agile is not enough. You also need to be smart.

http://jazoon.com/

Challenges and Opportunities for Python

In this PyCon 2009 talk Ted Leung discusses some of the challenges and opportunities that he sees for Python.

http://us.pycon.org/

Just For Fun: Rediscovering Coding as a Hobby

In this RubyConf talk, Adam Keys talks about getting back when coding was more fun and less serious.

http://rubyconf.org/

Nano-Incremental Development, a.k.a. Elephant Carpaccio

During this Agile Roots workshop, Alistair Cockburn made people think about cutting features requests in small pieces.

http://www.agileroots.com/

Design Fundamentals for Developers

At the Microsoft’s Mix conference, Robby Ingebretsen presented the fundamentals of interface design for developers.

http://live.visitmix.com/

Reading the Flex Source Code

In this talk at 360Flex, Jonathan Branam gave an introduction to the Flex source code, explaining the class hierarchy, compositional classes and the importance of interfaces

http://www.360flex.com/

Patterns for Lovers of JavaScript

In this talk at JSConf 2009, Petter Higgins shows that Dojo teaches fundamentally sounds techniques for high performance JavaScript applications across the board. You will learn how these techniques provide a stable, professional-grade foundation for creating highly maintainable, scalable projects of any size

http://jsconf.us/

Kanban, Flow & Cadence

During this Lean Software & Systems Conference session, Karl Scotland introduced the three lean concepts of Kanban, Flow and Cadence, which combine to generate a more pipeline-based approach to software development, as opposed to the typical timebox-based approaches used by more traditional Agile methods.

http://www.leanssc.org/conferences/

GWT Can Do What?!?! A Preview of Google Web Toolkit 2.0

In this talk at the Google I/O 2009 conference, Bruce Johnson presents the new version of GWT.
GWT 2.0 contains huge improvements, including dynamic script loading, a new catalog of compiler optimizations, and a new approach to hosted mode debugging that promises to revolutionize your productivity.

http://code.google.com/events/io/2010/sessions.html

C++, Java and .NET: Lessons Learned from the Internet Age

Java’s appearance at the dawn of the Internet Age helped to propel it to near-instant prominence, and lodged cross-platform virtual machines and garbage-collection firmly into our mainstream consciousness. In Java’s wake, .NET introduced the concept of the “cross-language” virtual machine, and helped to foster a new discussion on the benefits of functional programming. Did Java and C# have an evolutionary advantage over C++, or were they simply “Cool” (the original code name for C# / .NET)?

http://www.oredev.org/

You Want More?

If you want to search for more videos, SoftDevTube.com has currently catalogued and classified more than 1600 software development videos, screencasts and tutorials. To prepare your conference schedule for this year or find more conferences archives, go to SoftDevConferences.com. If you want to read some in-depth articles on software development topics, visit the Methods & Tools magazine web site and download past PDF issues.

Over the past two years, Carl and I have been working on-again off-again on the problem of dependency resolution. Fabien Franzen implemented the first partial solution for Merb, with thor merb:gem:install.

When we started working on Rails, we knew we wanted to finally crack the nut, making it possible for Rails itself to have some of its own dependencies, and solving some persistent, long-term problems with gem plugins.

Earlier this year, we released bundler, our first real attempt to solve this problem. Among other things, it shipped with the first true dependency resolver for Rubygems. It also modeled the entire dependency process on top of Rubygems, allowing us to support git repositories in the same dependency resolution process as Rubygems themselves.

Over the next few months, we refined the bundler quite a bit. We’re proud of the fact that individuals, web shops, and deployment companies have adopted bundler, and Gemfile has become a standard way of expressing gem dependencies for an app.

During this time, we’ve also received quite a bit of feedback. As we approach the release of Bundler 1.0, which we hope to ship along with Rails 3.0 final, we took the opportunity to take a look at all the feedback people have sent so far.

Having done so, we’re proud to announce Bundler 0.9 with radically improved workflows that fit our needs and the needs of those who have contributed feedback and patches.

The Cache

Before Bundler 0.9, Bundler installed gems and git repositories to a local application cache. When we looked at the deployment workflow, we found that while people liked the ability to cache their .gem files in their application, they had a lot of trouble at deployment time.

Essentially, people (and we!) wanted the ability to expand the gems to a central cache that they could reuse across deployments. People tried various symlinking approaches, but it became clear to us Bundler 1.0 needed to support this workflow natively.

On a related note, people who developed a number of applications on a single machine wanted to be able to reuse a single system cache across their application, and not need to connect to remotes so often when they already had all the gems they needed on their systems.

Finally, Rails itself grated against the default Bundler workflow. Most people installing a new Rails app run gem install rails, then rails my_app, and then want to go into their application and start working. But with Bundler 0.5 to 0.8, they needed to hit the remotes again even though, by definition, the system gem cache already had all the gems they needed.

Runtime

The fully-packaged application works fantastically for deployment, enabling a repeatable, reliable development to staging to production workflow. However, the need to explicitly bundle the application after added a new dependency in the very early stages of an application’s life cycle feels more like a compile step than a lean mean agile machine.

In Bundler 0.9, we’re adding the ability to run your application against gems already on your system, with the same reliability that you’ve grown to love from our dependency resolver.

Once you’ve gotten your application working, you can lock down the dependencies, so coworkers or production will use exactly the same environment that worked for you.

This allows you to use your common system gem repository across multiple apps, with a fully resolved virtual environment for each application based on the app’s dependencies specified in the Gemfile.

Locking

A lot of people (including me) love the idea of storing the .gem files in the application to create a single deployable unit with no network dependencies.

Storing the .gem files in the repository also provides a record of the fully resolved dependencies for your application, so that a new released gem between testing and deployment cannot change the environment. In Bundler 0.9, we provide a mechanism for you to save that record without having to also store the .gem files.

This won’t give you a single, dependency-less deployment package, but it will save you from unexpected environment changes.

New Commands

When we started Bundler, we had just one command: gem bundle. Over the following months, the command took on a life of its own, with a slew of flags representing both options and verbs. Bundler 1.0 will have a small, streamlined list of commands, each with its own set of flags:

bundle pack

Take all .gem files needed for the application and place them into the local application. By default, bundler places them in vendor/gems.

bundle lock

Resolve all dependencies in the Gemfile, and store a record of that process. In the future, use that record, preventing any changes in the environment or in the remote sources from changing the gems used by the application.

bundle install

Install the bundle specified by the Gemfile to the cache. By default, Bundler installs the gems (and git repositories) into system gems. You can override this to install into another location, or into your local application, just like in Bundler 0.8 and before.

After cloning a new repository for the first time, you will probably want to run this command.

If you packed gems, Bundler will use the gems from your application. If you packed and locked gems, Bundler will install the gems immediately without updating the remote sources.

bundle check

Check to see whether your cache has all the required dependencies. If it does not, Bundler will print out the list of missing gems.

bundle exec …

Run the specified command using the local environment. For instance, if you have Rails 2.3 and 3.0 installed, if you run bundle exec rails . in a Rails 2.3 application, Bundler will run the rails command using Rails 2.3 (and other gems in the current environment).

Environments

In Bundler 0.9, as in Bundler 0.8 and before, the Gemfile specifies an app-specific environment. New in Bundler 0.9, you may store the .gem files and unpacked gems in a central location, as well as in the application itself.

Because Bundler ships with a dependency resolver, you do not need to specify named environments and switch between them. Instead, the environments are virtual, based on the gems you specify as application dependencies in the Gemfile, and you still get all the benefits of a shared, system-wide cache.

addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2010%2F02%2F01%2Fbundler-0-9-heading-toward-1-0%2F'; addthis_title = 'Bundler+0.9%3A+Heading+Toward+1.0'; addthis_pub = '';

As you may have read, Rails adds XSS protection by default in Rails 3. This means that you no longer have to manually escape user input with the h helper, because Rails will automatically escape it for you.

However, it’s not as simple as all that. Consider the following:

Hello <strong>friends</strong>!
 
<%= tag(:p, some_text) %>
<%= some_text %>

In the above example, we have a few different scenarios involving HTML tags. First off, Rails should not escape the strong tag surrounding “friends”, because it is unambiguously not user input. Second, Rails should escape some_text in the <p> tag, but not the <p> tag itself. Finally, the some_text in the final tag should be escaped.

If some_text is <script>evil_js</script>, the above should output:

Hello <strong>friends</strong>!
 
<p>&lt;script&gt;evil_js&lt;/script&gt;</p>
&lt;script&gt;evil_js&lt;/script&gt;

In order to make this happen, we have introduced a new pervasive concept called html_safe into Rails applications. If a String is html_safe (which Rails determines by calling html_safe? on the String), ERB may insert it unaltered into the output. If it is not safe, ERB must first escape it before inserting it into the output.

def tag(name, options = nil, open = false, escape = true)
  "<#{name}#{tag_options(options, escape) if options}#{open ? ">" : " />"}".html_safe
end

Here, Rails creates the tag, telling tag_options to escape the contents, and then marks the entire body as safe. As a result, the <p> and </p> will emerge unaltered, while Rails will escape the user-supplied content.

The first implementation of this, in Koz’s rails-xss plugin, accomplished the above requirements by adding a new flag to all Strings. Rails, or Rails applications, could mark any String as safe, and Rails overrode + and << to mark the resulting String appropriately based on the input Strings.

However, during my last performance pass of Rails, I noticed that overriding every String concatenation resulted in quite a bit of performance overhead. Worse, the performance overhead was linear with the number of <%= %> in a template, so larger templates didn’t absorb the cost (as they would if the problem was once-per-template).

Thinking about the problem more, I realized (and confirmed with Koz, Jeremy, and Evan Phoenix of Rubinius), that we could implement roughly the same feature-set in a more performant way with a smaller API impact on Ruby. Because the problem itself is reasonably complex, I won’t go into a lot of detail about the old implementation, but will explain how you should use the XSS protection with the new implementation. If you already used Koz’s plugin or are working with the prereleases of Rails, you’ll notice that today’s commit changes very little.

SafeBuffer

In Rails 3, the ERB buffer is an instance of ActiveSupport::SafeBuffer. SafeBuffer inherits from String, overriding +, concat and << so that:

• If the other String is safe (another SafeBuffer), the buffer concatenates it directly
• If the other String is unsafe (a plain String), the buffer escapes it first, then concatenates it

Calling html_safe on a plain String returns a SafeBuffer wrapper. Because SafeBuffer inherits from String, Ruby creates this wrapper extremely efficiently (just sharing the internal char * storage).

As a result of this implementation, I was starting to see a lot of the following idiom in the codebase:

buffer << other_string.html_safe

Here, Rails is creating a new SafeBuffer for the other_string, then passing it to the << method of the original SafeBuffer, which then checks to see if it is safe. For cases like this, I created a new safe_concat method on the buffer which uses the original, native concat method, skipping both the need to create a new SafeBuffer and the need to check it.

Similarly, concat and safe_concat in ActionView proxy to the concat and safe_concat on the buffer itself, so you can use safe_concat in a helper if you have some HTML you want to concatenate to the buffer with no checks and without escaping.

ERB uses safe_concat internally on the parts of the template outside of <% %> tags, which means that with the changes I pushed today, the XSS protection code adds no performance impact to those cases (basically, all of the plain text in your templates).

Finally, ERB can now detect the raw helper at compile time, so if you do something like <%= raw some_stuff %>, ERB will use safe_concat internally, skipping the runtime creation of a SafeBuffer and checks for html_safety.

Summary

In summary, the XSS protection has the following characteristics:

• If a plain String is passed into a <%= %>, Rails always escapes it
• If a SafeBuffer is passed into a <%= %>, Rails does not escape it. To get a SafeBuffer from a String, call html_safe on it. The XSS system has a very small performance impact on this case, limited to a guard calling the html_safe? method
• If you use the raw helper in a <%= %>, Rails detects it at compile-time of the template, resulting in zero performance impact from the XSS system on that concatenation
• Rails does not escape any part of a template that is not in an ERB tag. Because Rails handles this at template compile-time, this results in zero performance impact from the XSS system on these concatenations

In comparison, the initial implementation of XSS impacted each concatenation or + of String, had impact even if the app used the raw helper, and even on plain Strings in templates.

That said, I want to extend personal thanks to Koz for getting the first draft out the door. It worked, demonstrated the concept, and let the community test it out. All in all, an excellent first pass.

addthis_url = 'http%3A%2F%2Fyehudakatz.com%2F2010%2F02%2F01%2Fsafebuffers-and-rails-3-0%2F'; addthis_title = 'SafeBuffers+and+Rails+3.0'; addthis_pub = '';