When it comes to requirements “communication” is the name of the game. An analyst could crank out top-notch technical documents or business cases, but if their intended audience isn’t able to consume it then ultimately the work was for naught. It’s to this end that we use things like visual models to help get the point across that is separate from a system shall statement or a user story; so that our consumers can truly consume and then deliver.

However, there are small bits of information that can often go unknown, or unconsidered, because it doesn’t necessarily have a place in a model, or is not important enough to truly highlight, but rather mention in a model. Additionally, there are little tricks that can be used if you’re utilizing models in elicitation session in order to discover missing information.

Below is an example of a typical dimension of information being added to a model through a callout:

 

As you can see the information for the pre-population of billing information into shipping information is additional information that would otherwise make the model too complex/distract from the business process that is attempting to be communicated. This is just a typical callout, however. There are a few ways we can add additional dimensions to our models through text, color, and shapes.

 

Here we have an L2 system flow from that L1 we had earlier. It’s a typical process for submitting an order. However, the analyst doesn’t know what the name of the Order System is. The analyst could simply make a note of it in their issues list, and bring it up when they have time to comb through the list, but it may work better to get it finished in elicitation. Showing this model to a stakeholder it’s very easy to notice the yellow highlighting and the callout text accompanying it, and the question can be addressed right then and there.

If you stick the standard red, green, yellow associations that people typically have (i.e. yellow meaning caution/observance, red meaning stop or bad, and green meaning good or go) those colors can be used to communicated dimensions of questions you have on a model. In the above example of the Order System swimlane is colored yellow, as it is not necessarily critical to name, but needs to be observed and considered. Juxtapose that with the red step of 5.3 where the current state/future state consideration is so pressing that it must utilize red. This is useful, but if abused you can end up distancing yourself from who you are communicating with, so it should be used cautiously.

When you’re thinking outside the box there can be many other ways to call attention to specific details of a model in order to address them in requirements or use them in elicitation. Most recently, a colleague was working on a project that involved a large mix of features being requested by several companies that were merging with a customer. In order to demonstrate the various capabilities being requested/needed for each acquisition she utilized what has been dubbed “skittles” within her flows and UI maps.

 

Here we can see an example of a possible acquisition flow where each company’s capabilities are marked on the process steps through use of the skittles. These chiclets are good for specifying certain detail as well as tacking on another dimension of traceability such that each step must be analyzed for the capabilities of each company.

Communication extra dimensions of information can be accomplished in a lot of different ways, and is certainly not restricted to the methods I’ve listed here. In fact, any method of communication that ends up working best for everyone involved in a project, even if that means breaking from the status quo, is the best way. At the end of the day everyone just needs to understand each other, and small things can make all the difference between “huh?” and “oh!”

Adding Dimensions to your Models: Nuanced Communication for Business Analysts is a post from: http://requirements.seilevel.com/blog

Drum roll please… I am very pleased to announce that Hiring Geeks That Fit is also available at the Pragmatic Bookshelf!

I self-published the book at leanpub, and announced it over on Hiring Technical People, and have been writing about it in my newsletter for a few months. And, not everyone knew about it. I asked the Prags if they would carry it, so that people familiar with my books could buy it, and they said yes. I am thrilled.

Here’s what the Prags say on their site:

This book was written and produced entirely by the author. We are proud to be distributing it.

Thank you. I’m very happy you folks are carrying it.

If you have to hire anyone for your technical team—including managers—check out this book. You need it. I suggest you start somewhere and iterate.

If you bought my original hiring book, this book is better. It’s slimmed down. The stories are better. I’ve added information on how to use Twitter and LinkedIn and how to use a multitude of online recruiting options, which the original book did not have.

If you can’t hire people inside of a couple of months, you need this book. You have options to speed things along. And, start reading Hiring Technical People, because that’s where I blog about those issues.

I have some quick-start hiring guides planned for this year, too. Sign up for the Pragmatic Manager newsletter to learn about them.

Teamwork is not just for sports and development team. Last year I was reminded just much teamwork can help raise your game. There are all sorts of sayings about teams and teamwork.

• Teachers call it copying..we call it teamwork!
• Teamwork divides the task and multiplies the success.
• True…there is no “i” in team.
• There are three types of workers: those who get things done, those who watch things get done, and those who wonder how so much got done.

All true and the list can go on and on but the best teams make team members better than they were before being on the team.

Here’s a classic scenario

You work on a project that stores transactional data in a database.  The application gets deployed to production and early on the performance is great, selecting data from the database is snappy and insert latency goes unnoticed.  Over a time period of days/weeks/months the database starts to get bigger and queries slow down.

There are various approaches that can help you make your application and database run faster.

A Database Administrator (DBA) will take a look and see that the database is tuned. They offer suggestions to add certain indexes, move logging to separate disk partitions, adjust database engine parameters and verify that the database is healthy.  You can also add Provisioned iOPS on EBS volumes or obtain faster (not just separate) disk partitions.  This will buy you more time and may resolve this issues to a degree.

At a certain point you realize the data in the database is the bottleneck.  

In many application databases some of the data is historical information that becomes less important after a certain amount of time.  If you figure out a way to get rid of this data your queries will run faster, backups run quicker and use less disk space.  You can delete it but then the data is gone forever.  You could run a slew of DELETE statements causing a ton of logging and consuming database engine resources.  So how do we get rid of old data efficiently but not lose the data forever?

Table Partitioning

Table partitioning is a good solution to this very problem. You take one massive table and split it into many smaller tables - these smaller tables are called partitions or child tables.  Operations such as backups, SELECTs, and DELETEs can be performed against individual partitions or against all of the partitions.  Partitions can also be dropped or exported as a single transaction requiring minimal logging.

Terminology

Let’s start with some terminology that you will see in the remainder of this blog.

Master Table

Also referred to as a Master Partition Table, this table is the template child tables are created from.  This is a normal table, but it doesn’t contain any data and requires a trigger (more on this later).  There is a one-to-many relationship between a master table and child tables, that is to say that there is one master table and many child tables.

Child Table

These tables inherit their structure (in other words, their Data Definition Language or DDL for short) from the master table and belong to a single master table.  The child tables contain all of the data.  These tables are also referred to as Table Partitions.

Partition Function

A partition function is a Stored Procedure that determines which child table should accept a new record. The master table has a trigger which calls a partition function. There are two typical methodologies for routing records to child tables:

1. By Date Values - An example of this is purchase order date.  As purchase orders are added to the master table, this function is called by the trigger.  If you create partitions by day, each child partition will represent all the purchase orders entered for a particular day. This method is covered by this article.

2. By Fixed Values - An example of this is by geographic location such as states. In this case, you can have 50 child tables, one for each US state.  As INSERTs are fired against the master the partition function sorts each new row into one of the child tables.  This methodology isn’t covered by this article because it doesn’t help us remove older data.

Let's Try Configuring Table Partitions!

The example solution demonstrates the following:

• Automatically creating database table partitions based on date

• Schedule the export of older table partitions to compressed flat files in the OS

• Drop the old table partition without impacting performance

• Reload older partitions so they are available to the master partition

Take the time to let that last piece of the solution sink in.  Most of the documentation on partitioning I’ve read through simply uses partitioning to keep the database lean and mean.  If you needed older data, you’d have to keep an old database backup.  I’ll show you how you can keep the database lean and mean through partitioning but also have the data available to you when you need it without db backups.

Conventions

Commands run in shell as the root user will be prefixed by:

root#

Commands run in a shell as a non-root user, eg. postgres will be prefixed by:

postgres$

Commands run within the PostgreSQL database system will look as follows:

my_database>

What you’ll need

The examples below use PostgreSQL 9.2 on Engine Yard. You will also need git for installing plsh.

Summary of Steps

Here’s a summary of what we are going to do:

1. Create a master table

2. Create a trigger function

3. Create a table trigger

4. Create partition maintenance function

5. Schedule the partition maintenance

6. Reload old partitions as needed

Create a Master Table

For this example we’ll be creating a table to store basic performance data (cpu, memory, disk) about a group of servers (server_id) every minute (time).

CREATE TABLE myschema.server_master
(
id BIGSERIAL NOT NULL,
server_id BIGINT,
cpu REAL,
memory BIGINT,
disk TEXT,
"time" BIGINT,
PRIMARY KEY (id)
);

Notice that in the code above the column name time is in quotes. This is necessary because time is a keyword in PostgreSQL. For more information on Date/Time keywords and functions visit the PostgreSQL Manual.

Create Trigger Function

The trigger function below does the following

• Creates child partition child tables with dynamically generated “CREATE TABLE” statements if the child table does not  already exist.

• Partitions (child tables) are determined by the values in the “time” column, creating one partition per calendar day.

• Time is stored in epoch format which is an integer representation of the number of seconds since 1970-01-01 00:00:00+00.  More information on Epoch can be found at http://en.wikipedia.org/wiki/Unix_time

• Each day has 86400 seconds, midnight for a particular day is an epoch date divisible by 86400 without a remainder.

• The name of each child table will be in the format of myschema.server_YYYY-MM-DD.

CREATE OR REPLACE FUNCTION
  myschema.server_partition_function()
RETURNS TRIGGER AS 
$BODY$
DECLARE
  _new_time int;
  _tablename text;
  _startdate text;
  _enddate text;
  _result record;
BEGIN
  --Takes the current inbound "time" value and determines when midnight is for the given date
  _new_time := ((NEW."time"/86400)::int)*86400;
  _startdate := to_char(to_timestamp(_new_time), 'YYYY-MM-DD');
  _tablename := 'server_'||_startdate;

  -- Check if the partition needed for the current record exists
  PERFORM 1
  FROM   pg_catalog.pg_class c
  JOIN   pg_catalog.pg_namespace n ON n.oid = c.relnamespace
  WHERE  c.relkind = 'r'
  AND    c.relname = _tablename
  AND    n.nspname = 'myschema';

  -- If the partition needed does not yet exist, then we create it:
  -- Note that || is string concatenation (joining two strings to make one)
  IF NOT FOUND THEN
    _enddate:=_startdate::timestamp + INTERVAL '1 day';
    EXECUTE 'CREATE TABLE myschema.' || quote_ident(_tablename) || ' (
      CHECK ( "time" >= EXTRACT(EPOCH FROM DATE ' || quote_literal(_startdate) || ')
        AND "time" < EXTRACT(EPOCH FROM DATE ' || quote_literal(_enddate) || ')
      )
    ) INHERITS (myschema.server_master)';

  -- Table permissions are not inherited from the parent.
  -- If permissions change on the master be sure to change them on the child also.
  EXECUTE 'ALTER TABLE myschema.' || quote_ident(_tablename) || ' OWNER TO postgres';
  EXECUTE 'GRANT ALL ON TABLE myschema.' || quote_ident(_tablename) || ' TO my_role';

  -- Indexes are defined per child, so we assign a default index that uses the partition columns
  EXECUTE 'CREATE INDEX ' || quote_ident(_tablename||'_indx1') || ' ON myschema.' || quote_ident(_tablename) || ' (time, id)';
END IF;

-- Insert the current record into the correct partition, which we are sure will now exist.
EXECUTE 'INSERT INTO myschema.' || quote_ident(_tablename) || ' VALUES ($1.*)' USING NEW;
RETURN NULL;
END;
$BODY$
LANGUAGE plpgsql;

Create a Table Trigger

Now that the Partition Function has been created an Insert Trigger needs to be added to the Master Table which will call the partition function when new records are inserted.

CREATE TRIGGER server_master_trigger
BEFORE INSERT ON myschema.server_master
FOR EACH ROW EXECUTE PROCEDURE myschema.server_partition_function();

At this point you can start inserting rows against the Master Table and see the rows being inserted into the correct child table.

Create Partition Maintenance Function

Now let’s put the master table on a diet.  The function below was built generically to handle the partition maintenance, which is why you won’t see any direct syntax for server.

How it works

• All of the child tables for a particular master table are scanned looking for any partitions where the name of the partition corresponds to a date older than 15 days ago.
• Each “too old” partition is exported/dumped to the local file system by calling the db function myschema.export_partition(text, text).  More on this is in the next section.
• If and only if the export to the local filesystem was successful the child table is dropped.
• This function assumes that the folder /db/partition_dump exists on the local db server.  More on this in the next section.  If you are wondering where the partitions are exported to this is where you should look!

CREATE OR REPLACE FUNCTION
  myschema.partition_maintenance(in_tablename_prefix text, in_master_tablename text, in_asof date)
RETURNS text AS
$BODY$
DECLARE
  _result record;
  _current_time_without_special_characters text;
  _out_filename text;
  _return_message text;
  return_message text;
BEGIN
   -- Get the current date in YYYYMMDD_HHMMSS.ssssss format
   _current_time_without_special_characters := 
        REPLACE(REPLACE(REPLACE(NOW()::TIMESTAMP WITHOUT TIME ZONE::TEXT, '-', ''), ':', ''), ' ', '_');

   -- Initialize the return_message to empty to indicate no errors hit
   _return_message := '';

   --Validate input to function
   IF in_tablename_prefix IS NULL THEN
     RETURN 'Child table name prefix must be provided'::text;
   ELSIF in_master_tablename IS NULL THEN
     RETURN 'Master table name must be provided'::text;
   ELSIF in_asof IS NULL THEN
     RETURN 'You must provide the as-of date, NOW() is the typical value';
   END IF;

   FOR _result IN SELECT * FROM pg_tables WHERE schemaname='myschema' LOOP

   IF POSITION(in_tablename_prefix in _result.tablename) > 0 AND char_length(substring(_result.tablename from '[0-9-]*$')) <> 0 AND (in_asof - interval '15 days') > to_timestamp(substring(_result.tablename from '[0-9-]*$'),'YYYY-MM-DD') THEN

      _out_filename := '/db/partition_dump/' || _result.tablename || '_' || _current_time_without_special_characters || '.sql.gz';
      BEGIN
        -- Call function export_partition(child_table text) to export the file
        PERFORM myschema.export_partition(_result.tablename::text, _out_filename::text);
        -- If the export was successful drop the child partition
        EXECUTE 'DROP TABLE myschema.' || quote_ident(_result.tablename);
        _return_message := return_message || 'Dumped table: ' || _result.tablename::text || ', ';
        RAISE NOTICE 'Dumped table %', _result.tablename::text;
      EXCEPTION WHEN OTHERS THEN
        _return_message := return_message || 'ERROR dumping table: ' || _result.tablename::text || ', ';
        RAISE NOTICE 'ERROR DUMPING %', _result.tablename::text;
      END;
     END IF;
   END LOOP;

   RETURN _return_message || 'Done'::text;
 END;
$BODY$
LANGUAGE plpgsql VOLATILE COST 100;

ALTER FUNCTION myschema.partition_maintenance(text, text, date) OWNER TO postgres;

GRANT EXECUTE ON FUNCTION myschema.partition_maintenance(text, text, date) TO postgres;
GRANT EXECUTE ON FUNCTION myschema.partition_maintenance(text, text, date) TO my_role;

The function below is again generic and allows you to pass in the table name of the file you would like to export to the operating system and the name of the compressed file that will contain the exported table.

-- Helper Function for partition maintenance
CREATE OR REPLACE FUNCTION myschema.export_partition(text, text) RETURNS text AS
$BASH$
#!/bin/bash
tablename=${1}
filename=${2}
# NOTE: pg_dump must be available in the path.
pg_dump -U postgres -t myschema."${tablename}" my_database| gzip -c > ${filename} ;
$BASH$
LANGUAGE plsh;

ALTER FUNCTION myschema.export_partition(text, text) OWNER TO postgres;

GRANT EXECUTE ON FUNCTION myschema.export_partition(text, text) TO postgres;
GRANT EXECUTE ON FUNCTION myschema.export_partition(text, text) TO my_role;

Note that the code above uses the plsh language extension which is explained below. Also note that on our systems bash is located at /bin/bash, this may vary.

That Was Fun, Where are we?

Almost there.  So far we've made all the necessary changes within the database to accommodate table partitions:

• Created a new master table
• Created the trigger and trigger function for the master table
• Created partition maintenance functions to export older partitions to the os and drop the old partitions

You could stop here if you'd like and proceed to the section "Now Let's See it in Action" but sure to continue below to configure automated maintenance.

What we have left to do for automated maintenance is:

• Install the plsh extension
• Setup the os to store partition dumps
• Create a cron job to automate the calling of the maintenance partition function

Configure PostgreSQL and OS

Enabling PLSH in PostgreSQL

The PLSH extension is needed for PostgreSQL to execute shell commands. This is used by myschema.export_partition(text,text) to dynamically create a shell string to execute pg_dump. Starting as root, execute the following commands,

root# cd /usr/local/src # Build the extension .so files for postgresql
root# curl -L href="https://github.com/petere/plsh/archive/9a429a4bb9ed98e80d12a931f90458a712d0adbd.tar.gz">https://github.com/petere/plsh/archive/9a429a4bb9ed98e80d12a931f90458a712d0adbd.tar.gz -o plsh.tar.gz
root# tar zxf plsh.tar.gz
root# cd plsh-*/
root# make all install # Note that the postgres header files must be available
root# su - postgres # Or whatever account postgresql is running under
postgres$ psql my_database  # Substitute the name of your database with the partitioned tables
my_database> CREATE EXTENSION plsh; # NOTE: This must be done once for each database

Create the directory,

root# mkdir -p /db/partition_dump

Ensure that the postgres user owns the directory, and your deployment user’s group has permissions as well so that it can read the files. The default deploy user is ‘deploy’ on Engine Yard Cloud.

root# chown postgres:deploy /db/partition_dump

Even further information on PL/SH can be found in the plsh project’s README.

Schedule Partition Maintenance

The commands below will schedule the partition_maintenance job to run at midnight every day

root# su - postgres                    ## This is the OS user that will run the cron job
postgres$ mkdir -p $HOME/bin/pg_jobs   ## This is the folder that will contain the script below
postgres$ cat > $HOME/bin/pg_jobs/myschema_partition_maintenance
#!/bin/bash
# NOTE: psql must be available in the path.
psql -U postgres glimpse <<SQL
 SELECT myschema.partition_maintenance('server'::text, 'server_master'::text, now()::date );
SQL
## Now press the <ctrl+d> keys to terminate the “cat” commands input mode
postgres$ exit                        ## Exit the postgres os user and execute as root:
root# chmod +x /home/postgres/bin/pg_jobs/myschema_partition_maintenance # Make script executable< root# crontab -u postgres -e          ## Add the line:
0 0 * * * /home/postgres/bin/pg_jobs/myschema_partition_maintenance

View the cron jobs for the postgres user to ensure the crontab line is correct:

root# crontab -u postgres -l

0 0 * * * /home/postgres/bin/pg_jobs/myschema_partition_maintenance

Make sure your /db/partition_dump folder is backed up if you are not using an Engine Yard Cloud instance.  If you ever need the data again you’ll need these files to restore the old partitions.  This may be as simple as rsyncing (copying) these files to another server just to be sure. We find that sending these to S3 works well for archival purposes.

Now your master tables are scheduled for partition maintenance and you can rest knowing that you’ve create something special; a nimble database that will keep itself on a weight loss program!

Reload Old Partitions

If you have separation anxiety from your old data or maybe a dull compliance request landed on your desk then you can reload the old partitions from the file system.

To reload a partition first we navigate to /db/partition_dump on the local db server and identify the file and then as the postgres user we import the file back into the database.

postgres$ cd /db/partition_dump
postgres$ ls # find the filename of the partition dump you need
postgres$ psql my_database < name_of_partition_dump_file

After the partition file is loaded it will be queryable from the master table.  Be aware that when the next time the partition maintenance job runs the newly imported partition will be exported again.

Now Let's See it in Action Create Child Tables

Let's insert two rows of data to see creation of new child partitions in action.  Open a psql session and execute the following:

postgres$ psql my_database

my_database> INSERT INTO myschema.server_master (server_id, cpu, memory, disk, time) VALUES (123, 20.14, 4086000, '{sda1:510000}', 1359457620);   --Will create "myschema"."servers_2013-01-29"

my_database> INSERT INTO myschema.server_master (server_id, cpu, memory, disk, time) VALUES (123, 50.25, 4086000, '{sda1:500000}', 1359547500);   --Will create "myschema"."servers_2013-01-30"

So what happened?  Assuming this is the first time you've run this two new child tables were created, see the comments inline with the sql statement on the child tables that were created.  The first insert can be seen by selecting against either the parent or child:

SELECT * FROM myschema.server_master;        --Both records seen

SELECT * FROM myschema."server_2013-01-29";  --Only the first insert is seen

Note the use of double quotes around the child partition table name, they aren't there because the table is inherited, they are there because of hyphen used between the year-month-day.

Perform Partition Maintenance

The two rows we inserted are more than 15 days old.  Manually running the partition maintenance job (same job as would be run by cron) will export these two partitions to the os and drop the partitions.

postgres$ /home/postgres/bin/pg_jobs/myschema_partition_maintenance

When the job is done you can see the two exported files:

postgres$ cd /db/partition_dump

postgres$ ls -alh
…
-rw------- 1 postgres postgres 1.0K Feb 16 00:00 servers_2013-01-29_20130216_000000.000000.sql.gz
-rw------- 1 postgres postgres 1.0K Feb 16 00:00 servers_2013-01-30_20130216_000000.000000.sql.gz

Selecting against the master table should yield 0 rows, the two child tables will also no longer exist.

Reload Exported Partitions

If you want to reload the first child partition from the exported file, gunzip it then reload it using psql:

postgres$ cd /db/partition_dump

postgres$ gunzip servers_2013-01-29_20130216_000000.000000.sql.gz

postgres$ psql my_database < servers_2013-01-29_20130216_000000.000000.sql

Selecting against the master table will yield 1 row, the first child table will now exist as well.

Notes

Our database files reside on a partition mounted at /db which is separate from our root (‘/’) partition.

For more information on PostgreSQL extensions visit the extensions documentation.

The database engine doesn't return the number of rows affected correctly (always 0 rows affected) when performing INSERTs and UPDATEs against the master table.  If you use Ruby, be sure to adjust your code for the fact that the pg gem won't have the correct value when reporting cmd_tuples. If you are using an ORM then hopefully they adjust for this accordingly.

Make sure you are backing up the exported partition files in /db/partition_dump, these files lie outside of the database backup path.

The database user that is performing the INSERT against the master table needs to also have DDL permissions to create the child tables.

There is a small performance impact when performing an INSERT against a master table since the trigger function will be executed.

Ensure that you are running the absolute latest point release for your version of PostgreSQL, this ensures you are running the most stable and secure version available.

This solution works for my situation, your requirements may vary so feel free to modify, extend, mutilate, laugh hysterically or copy this for your own use.

Next Steps

One of the original assumptions was the creation of partitions for each 24 hour period, but this can be any interval of time (1 hour, 1 day, 1 week, every 435 seconds) with a few modifications.  Part II of this blog post will discuss the necessary changes needed to the partition_maintenance function and table trigger.  I'll also explore how to create a second "archive" database that you can use to automatically load old partition data, keeping the primary database lean and mean for everyday use.

Wired has a wonderful interview with Clayton Christensen, author of the tech ninja's bible, Innovator's Dilemma. Innovation is the name of the game in Silicon Valley and if you want to understand the rules of the game this article is a quick and clear way of learning. Everything is simply explained with compelling examples by the man himself.

Just as every empire has fallen, every organization is open to disruption. It's the human condition to become comfortable and discount potential dangers. It takes a great deal of mindfulness to outwit and outlast the human condition. If you want to be the disruptor and avoid being the disruptee, this is good stuff.

He also talks about his new book, The Capitalist's Dilemma, which addresses this puzzle: if corporations are doing so well why are individuals doing so bad?

If someone can help you see a deep meaningful pattern in life then they haven't brought you a fish, they've taught you how to fish. That's what Christensen does. Here's a gloss of his world view changing points:

On Tuesday I did a lightning talk at the MVP Summit in Bellevue on connecting to all things, from statistics to statistical programming languages with Intellisense and LINQ using F# Type Providers, a feature shipped with Visual Studio 2012:

 

Connecting to an Excel file:

Querying users stored on SQL Azure:

query {
  for u in db.Users do
  where (u.UserId <> userId)
  select u
}

Querying genres via the Netflix API:

query {
    for g in ctx.Genres do
    where (g.Name = genre)
    for t in g.Titles do
    where (t.ReleaseYear ?<= 1959)
    where (t.ReleaseYear ?>= 1934)
    sortByNullable t.AverageRating
    skip nToSkip
    take nToTake
    select t
}

Running B-Movie Sample on Azure: b-movies.azurewebsites.net

Connecting to the World Bank:

type WorldBank = WorldBankDataProvider<"World Development Indicators">
let data = WorldBank.GetDataContext()

let countries () = 
 [| data.Countries.India        
    data.Countries.``United Kingdom``
    data.Countries.``United States`` |]

Rendering charts asynchronously using JavaScript High Charts using TypeScript definitions:

let render () = async {
 let opts = h.HighchartsOptions() 
 opts.chart <- h.HighchartsChartOptions(renderTo = "chart", ``type`` = "line")
 opts.title <- h.HighchartsTitleOptions(text = "School enrollment")

Running in the browser using FunScript (F# to JavaScript compiler):

Querying Hadoop Hive in the browser:

query {
 for i in hive.iris do
 groupBy i.``class`` into g
 select (g.Key, Seq.length g) 
}

 

"Be thankful for problems. If they were less difficult, someone with less ability might have your job." James Lovell - Apollo13 

I was sitting in a software requirements review meeting with a couple of project stakeholders last week, and one of them said something along the lines of “You’ve done a really great job with this.” My first reaction was to think “Are you kidding? There are tons of stuff we still don’t understand about this process.” Of course, I filtered that, and what came out of my mouth was “Thanks to your help.” But then I got to thinking about the conversation, and I realized that discovering just how much about the process we didn’t know was, in fact, exactly what needed to be done.

In the past, I’ve worked with a few folks who worked very hard to appear knowledgeable in every meeting and conversation. It’s a great gig, if you can work it. I have a Facebook connection who lives in Saudi Arabia. We get into conversations about soccer with a lot of help from Google translate. He thought my Arabic was pretty good. I confessed immediately to my cheat. I know I would get in way over my head if I pretended to be fluent in a language I don’t actually know a word of.

On a new project, you don’t even have Google translate to help you when you’re sitting in a meeting and the acronyms and unfamiliar terms start flying around. So I start by asking questions, lots of questions. You know what? It often turns out that I’m not the only person in the room with questions. I’m not the only person in the room who doesn’t know what that acronym means. I’m not the only person who doesn’t know that the order processing system does this and the manufacturing system does that. Asking those questions not only gives the person who has the knowledge the pleasure of educating the rest of us, but it gives the other team members the freedom to start asking questions too.

Apparently there’s a name for this. It’s what Professor Dan Berry calls “the importance of ignorance in software engineering.” When I heard him speak on this, a few weeks ago, I had an “ah-ha!” moment. There’s a name for it! Not knowing is a good thing! The obsessive-compulsive part of my brain that wants to be on top of everything all the time can just shut up and sit down. From now on, I’m going to focus on being a really good ignoramus.

The Importance of Being – Ignorant? is a post from: http://requirements.seilevel.com/blog

 

There is a natural tendency to bridle at rules that we did not help craft. Not all rules are unwarranted. A quick test is that if you would not want to clean up the mess then breaking the rule is a bad idea. If breaking a rule will cause a messy problem then MAYBE there is a good reason for the rule. If you still want to break the rules grab a plunger because if you clogged it, it is your responsibility and you own the problem!

 

My sister, Erica, is not a programmer. Normally she’s not a tester, either. But recently she paired with me, playing a tester role, and spotted bugs while I wrote in Perl. In the process, it became clear to me that testers do not need to become programmers in order to help programmers write programs in real-time.

The Context

While working on the report for the Rapid Testing Intensive, recently, I needed a usable archive of the materials. That meant taking all of the pages, comments, and attachments out of my Confluence site and putting them in a form easier to shuffle, subdivide, organize, refer to, and re-distribute. It would be great if that were a feature of Confluence, but the closest I can get to that is either manually downloading each item or downloading an entire archive and dealing with a big abstract blob of XML and cryptically named files with no extensions.

(Note to Atlassian: Please enhance Confluence to include a archivist-friendly (as opposed to system administrator-friendly) archive function that separates pages, attachments, and comments into discrete viewable units with reasonable names.)

The Deflection

While Erica catalogued the names of all the attachments representing student work and the person or persons who created them, I was supposed to write a program to extract the corresponding material from the archive. Instead, I procrastinated. I think I checked email, but I admit it’s possible I was playing Ghost Recon or watching episode 13 of Arang and the Magistrate on Hulu. So, when she was ready with the spreadsheet, I hadn’t even started my program.

To cover my laziness, I thought I’d invite her to sit with me while I wrote it… you know, as if I had been waiting for her on purpose to show her the power of code or whatever. I expected her to decline, since like many computer power users, she has no interest in programming, and no knowledge of it.

The Surprising Outcome

She did not decline. She sat with me and we wrote the program together. She found six or seven important bugs while I typed, and many other little ones. The programming was more fun and social for me. I was more energized and focused. We followed up by writing a second, bigger program together. She told me she wants to do more of this kind of work. We both want to do more.

A Question

How does someone who knows nothing about Perl programming, and isn’t even a tester, productively find bugs almost immediately by looking at Perl code?

That’s kind of a misleading question, because that’s not what really happened. She didn’t just look at my code. She looked at my code in the context of me talking to her about what I was trying to do as I was trying to do it. The process unfolded bit by bit, and she followed the logic as it evolved. It doesn’t take any specific personal quality on the part of the “coding companion,” just general ones like alertness, curiosity, and basic symbolic intelligence. It doesn’t take any particular knowledge, although it can help a lot.

Perhaps this would not work well for all kinds of coding. We weren’t working on something that required heaps of fiddly design, or hours of doodling in a notebook to conceive of some obscure algorithm.

My Claim

A completely non-technical but eager and curious companion can help me write code in real-time by virtue of three things:

1. The dynamic and interactive legibility of the coding process. I narrate what I’m doing as it comes together step-by-step. The companion doesn’t eat the whole elephant in a bite; and the companion encounters the software mediated by my continuous process of interpretation. I tell him what and why and how. I do this repeatedly, and answer his questions along the way. This makes the process accessible (or in the parlance I like to use “legible” because that word specifically means the accessibility of information). The legibility is not that of a static piece of code, sitting there, but rather a property of something that is changing within a social environment. It’s the same experience as watching a teacher fill a whiteboard with equations. If you came at the end of the class, it would look bewildering, but if you watched it in process, it looks sensible.
2. The conceptual simplicity of many bugs. Some bugs are truly devious and subtle, but many have a simple essence or an easily recognized form. As I fix my own bugs and narrate that process, my coding companion begins to pick up on regularities and consistency relationships that must be preserved. The companion programs himself to find bugs, as I go.
3. The sensemaking faculties of a programmer seeking to resolve the confusion of a companion. When my dogs bark, I want to know why they are barking. I don’t know if there’s a good reason or a bad reason, but I want to resolve the mystery. In the course of doing that, I may learn something important (like “the UPS guy is here”). Similarly, when my coding companion says “I don’t understand why you put the dollar sign there and there, but not over there” my mind is directed to that situation and I need to make sense of it. It may be a bug or not a bug, but that process helps me be clear about what I’m doing, no matter what.

And Therefore…

A tester of any kind can contribute early in a development process, and become better able to test, by pairing with a programmer regardless of his own ability to code.

When you have determined what your values and purpose are as a team or organization, you have to put your money where your mouth is. Turn your values and purpose into action! Create your own story!

Are you aiming for honesty, excellence, and service? Get the team in a car, drive to a client where something went wrong with your product, and sing a song about how sorry you are while handing over a big bouquet of flowers. Should your work be focused on creativity, discipline, and orderliness? Get your team to make a perfectly executed work of art out of sticky notes, carefully measured out and modeled on the computer. They key is not only to promise to keep these values in mind, but also to do something to prove that these values matter.

At the end of the day, you just ask yourself, ‘How did our vision and values influence decisions I made today?’ If they did not, then they are pretty much BS.

- Peter Senge, The Fifth Discipline

My Values and Purpose

I am preparing the launch of a business called Happy Melly, together with Maarten Volders and Vasco Duarte. As our core values we have defined freedom, transparency, and holism (originally described as system thinking). And our purpose is the pursuit of happiness in work. Do I have a story to express those values? I certainly do!

One month ago I wrote the post Why I Won’t Take Your Call. In that post I explained in an honest and transparent way that I prefer emails over phone calls because I don’t aim for local optimization (of conversations). Instead I take a more holistic long-term view of the job of being a writer, and daily phone calls simply don’t fit the picture. I claim the freedom to place such constraints on my communication with people, not because I’m an autist (though maybe I am), but because I claim a right to happiness as a worker.

I didn’t realize it at the time of writing, but this story perfectly illustrates the values and purpose of our Happy Melly business, at least from my point of view. (And I’m sure Vasco and Maarten can produce similar examples, without the autism bit.)

What are your values?

What’s your story?

This text is part of Value Story, a Management 3.0 Workout article. Read more here.

Subscribe to my mailing list and be the first to receive future articles!

Posted by Fred Chung, Android Developer Relations team

Android 4.2, Jelly Bean, introduced quite a few new features, and under the covers it also added a number of security enhancements to ensure a more secure environment for users and developers.

This post highlights a few of the security enhancements in Android 4.2 that are especially important for developers to be aware of and understand. Regardless whether you are targeting your app to devices running Jelly Bean or to earlier versions of Android, it's a good idea to validate these areas in order to make your app more secure and robust.

Content Provider default access has changed

Content providers are a facility to enable data sharing amongst app and system components. Access to content providers should always be based on the principle of least privilege — that is, only grant the minimal possible access for another component to carry out the necessary tasks. You can control access to your content providers through a combination of the exported attribute in the provider declaration and app-specific permissions for reading/writing data in the provider.

In the example below, the provider ReadOnlyDataContentProvider sets the exported attribute to "true", explicitly declaring that it is readable by any external app that has acquired the READ_DATA permission, and that no other components can write to it.

<provider android:name=”com.example.ReadOnlyDataContentProvider”
    android:authorities=”com.example”
    android:exported=”true”
    android:readPermission=”com.example.permission.READ_DATA” />

Since the exported attribute is an optional field, potential ambiguity arises when the field is not explicitly declared in the manifest, and that is where the behavior has changed in Android 4.2.

Prior to Jelly Bean, the default behavior of the exported field was that, if omitted, the content provider was assumed to be "exported" and accessible from other apps (subject to permissions). For example, the content provider below would be readable and writable by other apps (subject to permissions) when running on Android 4.1 or earlier. This default behavior is undesirable for sensitive data sources.

<provider android:name=”com.example.ReadOnlyDataContentProvider”
    android:authorities=”com.example” />

Starting in Android 4.2, the default behavior for the same provider is now “not exported”, which prevents the possibility of inadvertent data sharing when the attribute is not declared. If either the minSdkVersion or targetSdkVersion of your app is set to 17 or higher, the content provider will no longer be accessible by other apps by default.

While this change helps to avoid inadvertent data sharing, it remains the best practice to always explicitly declare the exported attribute, as well as declaring proper permissions, to avoid confusion. In addition, we strongly encourage you to make use of Android Lint, which among other things will flag any exported content providers (implicit or explicit) that aren't protected by any permissions.

New implementation of SecureRandom

Android 4.2 includes a new default implementation of SecureRandom based on OpenSSL. In the older Bouncy Castle-based implementation, given a known seed, SecureRandom could technically (albeit incorrectly) be treated as a source of deterministic data. With the new OpenSSL-based implementation, this is no longer possible.

In general, the switch to the new SecureRandom implementation should be transparent to apps. However, if your app is relying on SecureRandom to generate deterministic data, such as keys for encrypting data, you may need to modify this area of your app. For example, if you have been using SecureRandom to retrieve keys for encrypting/decrypting content, you will need to find another means of doing that.

A recommended approach is to generate a truly random AES key upon first launch and store that key in internal storage. For more information, see the post "Using Cryptography to Store Credentials Safely".

JavascriptInterface methods in WebViews must now be annotated

Javascript hosted in a WebView can directly invoke methods in an app through a JavaScript interface. In Android 4.1 and earlier, you could enable this by passing an object to the addJavascriptInterface() method and ensuring that the object methods intended to be accessible from JavaScript were public.

On the one hand, this was a flexible mechanism; on the other hand, any untrusted content hosted in a WebView could potentially use reflection to figure out the public methods within the JavascriptInterface object and could then make use of them.

Beginning in Android 4.2, you will now have to explicitly annotate public methods with @JavascriptInterface in order to make them accessible from hosted JavaScript. Note that this also only takes effect only if you have set your app's minSdkVersion or targetSdkVersion to 17 or higher.

// Annotation is needed for SDK version 17 or above.
@JavascriptInterface
public void doSomething(String input) {
   . . .
}

Secure USB debugging

Android 4.2.2 introduces a new way of protecting your apps and data on compatible devices — secure USB debugging. When enabled on a device, secure debugging ensures that only host computers authorized by the user can access the internals of a USB-connected device using the ADB tool included in the Android SDK.

Secure debugging is an extension of the ADB protocol that requires hosts to authenticate before accessing any ADB services or commands. At first launch, ADB generates an RSA key pair to uniquely identifies the host. Then, when you connect a device that requires secure debugging, the system displays an authorization dialog such as the one shown below.

The user can allow USB debugging for the host for a single session or can give automatic access for all future sessions. Once a host is authorized, you can execute ADB commands for the device in the normal way. Until the device is authorized, it remains in "offline" state, as listed in the adb devices command.

For developers, the change to USB debugging should be largely transparent. If you've updated your SDK environment to include ADB version 1.0.31 (available with SDK Platform-tools r16.0.1 and higher), all you need to do is connect and authorize your device(s). If your development device appears in "offline" state, you may need to update ADB. To so so, download the latest Platform Tools release through the SDK Manager.

Secure USB debugging is enabled in the Android 4.2.2 update that is now rolling out to Nexus devices across the world. We expect many more devices to enable secure debugging in the months ahead.

More information about security best practices

For a full list of security best practices for Android apps, make sure to take a look at the Security Tips document.

Join the discussion on

+Android Developers

Note: This article was written by Andreas Garnæs of Subsis, one of our awesome partners.

The default ActiveRecord cache key implementation only takes the model's attributes into account, not what the current time is. If your model changes with the passing of time -- e.g. a deal that expires -- using the default `#cache_key` implementation wlll not do. This blog post explains how to implement `#cache_key` correctly to handle models that will change state automatically at a future time. Let's say you have a Rails application that uses fragment caching and the rendering of one of your models is dependent on the current time, in addition to the stored attribute values in the database. As an example let’s say we are selling deals, and our deal expires after 24 hours. The rendering of a deal is expensive so we want use fragment caching:

   - cache ["v1", @deal] do
     = render @deal

Without changing `Deal#cache_key`, this partial will not be updated even when the deal expires. We can solve this by defining a custom `#cache_key` method in the Deal model. The default `#cache_key` method in Rails includes the model name and the `#updated_at` timestamp, so the only thing we need to add is a time-dependent component of the cache key:

  class Deal < ActiveRecord::Base
     def cache_key
      "#{super}-#{expired?}"
    end

    def expired?
        Time.now < expires_at
    end
  end

We can now render the partial correctly with the passing of time. Simple. Now consider the case of rendering a collection of such deals. We might have a category page, showing a list of all deals in the category:

  %h1= @category.title

  - @category.deals.each do |deal|
    = render deal

Again we can start by doing simple fragment caching:

  %h1= @category.title

  - @category.deals.each do |deal|
    - cache ["v1", "deal_in_category", deal] do
      = render deal

This works, but every time we render a category, we have to stitch the deal fragments together and there could be a lot of them. Let's try to add fragment caching to the rendering of the category as well:

  - cache ["v1", @category] do
    %h1= @category.title

    - @category.deals.each do |deal|
      - cache ["v1", "deal_in_category", deal] do
        = render deal

Our first problem is that the view is not refreshed when a deal gets updated. To get automatic invalidation of the fragment cache for categories when a deal is updated, we let the Deal model touch its category’s `updated_at` timestamp:

  class Deal < ActiveRecord::Base
    belongs_to :category, touch: true

    # cache key code...
  end

When we update a deal, the category page is now updated accordingly, but when a deal expires, the view is stale -- we haven't solved the time dependent part of the cache scheme. One solution would be to inflate all the deals and mangle the cache keys into a single composite key, which is appended to the category’s key. But to be useful, cache keys should be fast to calculate and the that scheme is slow and memory consuming. There is a faster solution:

  class Category < ActiveRecord::Base
    has_many :deals

    def cache_key
      expiration_sum = deals.where("deals.expires_at < ?", Time.now).sum(:expires_at)

      "#{super}-#{expiration_sum}"
    end
  end

Basically, we fetch all the deal-timestamps that the category depends on, discard all timestamps in the past, and sum the remaining timestamps to form the second half of the cache key for the category. That makes the category’s cache key dependent on all deals and the current time. This solution only requires 2 database queries (one for the deal timestamps and one for the category’s `updated_at` timestamp). All calculations are done in the database, which is very efficient. More complex scenarios may require you to do some calculations in Ruby, however.

Note, that it’s sound to sum the timestamps as we are guaranteed that the set of timestamps is constant for unchanged database values for the category and its associated deals. Otherwise, the first half of the cache key would have changed. With the above, we are able to cache the category-partial and render it in no time. The above observations can be generalized to other kinds of time dependencies: use `touch: true` for `has_many`-associations, and override the `cache_key` method to include time dependencies.

This is a guest post by LogicMonitor's Director of Tech Ops, Jesse Aukeman, about the different ways they're monitoring the success or failure of Puppet runs.

If you are like us, you are running some type of linux configuration management tool. The value of centralized configuration and deployment is well known and hard to overstate. Puppet is our tool of choice. It is powerful and works well for us, except when things don't go as planned. Failures of puppet can be innocuous and cosmetic, or they can cause production issues, for example when crucial updates do not get properly propagated.

Why?

In the most innocuous cases, the puppet agent craps out (we run puppet agent via cron). As nice as puppet is, we still need to goose it from time to time to get past some sort of network or host resource issue. A more dangerous case is when an administrator temporarily disables puppet runs on a host in order to perform some test or administrative task and then forgets to reenable it. In either case it’s easy to see how a host may stop receiving new puppet updates. The danger here is that this may not be noticed until that crucial update doesn't get pushed, production is impacted, and it’s the client who notices.

How to implement monitoring?

Who's Hiring?

• OLO's food ordering platform powers some of the largest restaurant chains and feeds millions of consumers. We're looking for Senior C# Software Engineers and DevOps Engineers to help us scale our system. Apply here.
• The AWS Relational Database Service (RDS) automates management of relational databases in the cloud. We have a wide variety of customers and are part of many mission-critical applications, like the ones built by the 2012 Obama re-election campaign. If you're interested in joining a fast-growing service and team, please send your resume to rds-jobs@amazon.com.
• Hiring! Director of Site Operations at Zoosk.  We’re looking for an innovator. Someone who wants to take site operations along with a smart team of Sys Admins to the next level. This is a very hands-on leadership role in a high-availability production environment. Full details here. 
• Teradata Aster is looking for Distributed Systems, Analytic Applications,  and Performance Architects. As a member of the Architecture Group you will help define the technical roadmap for the product.
• The New York Times is seeking a developer focused on infrastructure to join its newsroom development team. Read the full description here and send resumes to chadas@nytimes.com.
• New Relic is looking for a Java Scalability Engineer in Portland, OR. Ready to scale a web service with more incoming bits/second than Twitter?  http://newrelic.com/about/jobs
• Aerospike is Hiring! You dream in C - and like it? Then join us as a Senior Distributed Systems Engineer or Client / Application Engineer. People covent your bag of tricks for troubleshooting systems and network issues? Join our Operations and QA team. See if these positions are a fit for you! 

Fun and Informative Events

• It's back! Join the MySQL Community at the annual Percona Live MySQL Conference and Expo in Santa Clara, April 22-25. This year's conference features an outstanding lineup of 92 speakers delivering 112 breakout sessions over three days! 

Cool Products and Services

• aiCache creates a better user experience by increasing the speed scale and stability of your web-site. Test aiCache acceleration for free. No sign-up required. http://aicache.com/deploy
• New Benchmark shows Aerospike nearly 10x Faster than the Competition. Thumbtack Technology YCSB Benchmark shows Aerospike nearly 20x faster than Cassandra, Couchbase and Mongodb. Read it now!
• ScaleOut Software. In-Memory Data Grids for the Enterprise. Download a Free Trial.
• LogicMonitor - Hosted monitoring of your entire technology stack. Dashboards, trending graphs, alerting. Try it free and be up and running in just 15 minutes.
• AppDynamics is the very first free product designed for troubleshooting Java performance while getting full visibility in production environments. Visit http://www.appdynamics.com/free.
• ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.
• www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

There are five core processes for project success. Each must be in place. Each must be applied. The outcomes of each must be produced for the success of the project.

1. Identify Needed Capabailities - Define the set of capabilities needed to achieve the program objectives or the particular end state for a specific scenario. Using a Concept of Operations, define the details of who, where, and how it is to be accomplished, employed and executed.
2. Establish the Requirements Baseline - Define the technical and operational requirements that must be in place for the system capabilities to be fulfilled. First, define these requirements in terms that are isolated from any implementation technical products. Only then bind the requirements with technology. 
3. Establish the Performance Measurement Baseline (PMB) - Build a time–phased network of schedule activities describing the work to be performed, the budgeted cost for this work, the organizational elements that produce the deliverables, and the performance measures showing this work is proceeding according to plan.
4. Execute the PMB - Execute work packages, while assuring all performance assessment are 0%/100% complete before proceeding. No rework, no forward transfer of activities to the future. Assure every requirement is traceable to work and all work is traceable to requirements, and all requirements traceable to the needed capabilities.
5. Perform Continuous Risk Management - Perform the 6 process areas of Continuous Risk Management for each Performance-Based Project Management(sm) process area to Identify, Analyze, Plan, Track, Control, and Communicate programmatic and technical risk.

These are the foundation for increasing the Probability of Project Success (PoPS).

Related articles Requirements Elicitation Principles of Project Dynamics It Can't Be Any Clearer Than This Agile Project Management

At the end of the first quarter of 2012 there was a slight fluttering through the wind chimes of business journals,

technology news sites, Google plus, twitter, and Facebook: Red Hat, the producers of the Red Hat Enterprise Linux (RHEL), and sponsors of the Fedora Project, had become the first billion-dollar open source software company. Yes, that’s “Billion”, with a ‘B’. Yes, that’s “open source”, as in publicly available and changeable code. Perhaps this deserves a finer point to be put upon it: A company, based on a product where all code was available to the public, was able to monetize their product and reach a critical mass of a billion dollars.

A typical profit-centric analysis could say that this should not be possible. Never mind the business plan that Red Hat uses to leverage support licenses for their product, the simple fact that software available to everyone could generate such staggering revenues has spun the software world on its head. No longer is the cheeky alternative operating system of Linux confined to hobbyist systems and mailing list discussions. This is a new era of software development, one that starts with the user, and ends with the user, but it goes well beyond the operating system you use. The open source software world is filled with projects for everything you can imagine, and all the code is available for free.

So, what’s a Business Analyst to do in this brave new world?

As Business Analysts everything is supposed to be tied to money. Would adding this new feature increase the value of our product? Will that translate into revenue dollars? Would support costs decrease? These are all questions that the Business Analyst thinks over whenever they consider the requirements, and eventual solutions, to the problems a business will face and the software it creates to solve those problems. But when you remove the money from the equation, where does your guidance come? Even though the dollars are removed from the equation, the same problems still stand: users need to get things done, programs need to accomplish goals, and deadlines need to be met.

This gives Business Analysts who may be operating in an open source environment just the kind of wiggle room they always wish they’d had in a business environment: creative freedom. When you remove the money from the equation, you’re only left with 1: What can you accomplish with what you have? And more importantly 2: What exactly do you want this thing to do?

Rather than assigning dollar signs to goals and tying success to them open source projects are most concerned with completing their product concept. This simple statement of what the product should be would be the basis of future conversations in determining scope and work to be done. It’s these conversations, once documented in requirements that push the development throughout the project, as it is the very idea that is the lifeblood of an open source project.

Every open source software project starts as an idea. It’s this spark of imagination that can do anything from creating something openly available that may not have been available to the public before freely, such as a CD burning program, or attempt a revolution of the desktop, like with Linux operating systems. However, these dreams still can be assigned goals, and these goals can be analyzed, quantified, and accomplished. These goals can be spelled out by Business Analysts.

On a typical project I would use a Business Objectives Model to trace the goals of a project to dollar values. As we’ve already discussed, this isn’t all that valuable in an open source environment; essentially your objectives are going to come from your stakeholders minds. Who are the stakeholders? Well they’re the people with the ideas, and they are the direct users themselves. You can sit down with the steering committee for an open source project and chart out their goals and aspirations like you would with a typical revenue-generating project. Those needs will come out into business-objective-like statements. Those statements can be used to drive capability and feature discussions, and continue on into written requirements; an actual plan for the project, mapped out, written down, consumable, and testable. Open source projects are only limited by the volunteers that they have, and the requirements can be scoped according to the availability of development resources for a project. It’s that simple.

This level of planning does not necessarily take place in a typical Open Source project, at least in the sense that it is used today in the business world. Though, it would be a small matter to implement a requirement methodology on an open source project. Many open source projects go into planning with an engineering-driven attitude which, while functional, can leave a lot to be desired in the way of ensuring completeness of vision. Often what a project looks to accomplish can snuff out the small force of volunteers available to them, and a lack of documentation of vision can make it difficult for interested volunteers to know how and where they should participate to make the project successful. A clearly defined initial scope tailored for a minimum viable product and the resources available could substantially increase the success of an open source effort.

Past the planning effort and defining of scope the requirements created to help drive the vision of the project can then be used after development has begun to create test plans beyond normal programmatic unit testing, and ensure that the original objectives of the project are being met. Clean test plans can be produced from the documentation that the project creates up front in the planning stages, just like in a business environment.

Is it likely that the open source movement would adopt requirements methodologies into their normal planning and procedures? Perhaps not, given that business analysis is still a fledgling field, and many engineers are still unconvinced. However, while it is easy to dismiss those who point to scope and objectives as merely pounding sand, it’s almost always the Business Analysts who can tell you exactly why a project failed.

Business Analysis for Open Source Projects is a post from: http://requirements.seilevel.com/blog

You might have met a manager on a bad manager day. Equally as frustrating is when you work for a manager who has rules about problem solving.

I once worked for a manager who proudly said to me, “Don’t bring me a problem without bringing me a solution.” I blinked once and said, “Why would I bring you a problem I could solve?”

He stopped, and said, “Ooh.” Some of you will recognize that as the programmer’s refrain. “Oooh,” is what you say when you realize the computer has done something you told it to do, but is not what you meant it to do.

“Don’t bring me a problem without bringing me a solution” is an example of management incongruence. Not because a manager means to be. But because a manager might not know better. My manager wanted to challenge me. Believe me, I was challenged! I wasn’t being lazy. I wasn’t being stupid. I was stuck. I needed help. I didn’t know where to go for help.

Even in agile teams, the manager might be the right person to go to. The manager might not be. The manager might not have the answer. But the manager might be the right person to free the impediment, to know who has the answer, or to help with problem-solving.

This is why when managers have rules about problem solving, they make life difficult for everyone else. Managers don’t have to be perfect. They have to work work hard at staying congruent, which is different than being perfect. Much different than being perfect.

This is a picture of what I mean by congruence. When the manager takes him or herself, the other person, and the context into account, the manager is congruent. When the manager stops taking the other person into account, the manager blames the other person. When you bump into manager rules such as “Don’t bring me a problem without a solution,” your manager is blaming you for not having a solution.

When the manager stops taking him/herself into account, the manager placates. Managers who say, “Yes,” to all work and never say No and don’t manage the project portfolio placate the rest of the organization.

Managers who ignore both themselves and the other person are super-reasonable. Remember Ever Have a Bad Manager Day? I was being super-reasonable, ignoring me and the other person and the fact that we were human. Hah! That didn’t last long. There are are other incongruent stances, but those are the big three.

Does this mean managers can’t be human? Oh, no, they sure can be, and are! And, they need to watch out for these rules that make them less effective. Incongruent stances do not help managers manage. Incongruent stances and rules make it more difficult for managers to do a great job.

If you would like to read more about bumping into manager rules, take a look at my next myth, Management Myth 14: I Must Always Have a Solution to the Problem. Let me know if you like my suggestions.

Here is our monthly selection of interesting knowledge material on programming, software testing and project management: Blog: A Scrum Product Owner Checklist Blog: How to create a User Story Map Blog: Effective Steps to reduce technical debt: An agile approach Blog: Top 5 TDD Mistakes Article: Scaling the Sprint Review: The Fair Article: Case Study of Customer Input For a Successful Product (PDF) Article: Choose the best PaaS cloud for your needs Article: SQL Injection through HTTP Headers Article: Using Commercial Scrum Tools for Free Tools: TestMob, a distributed Javascript unit test system Tools: Cmockery, unit tests for C applications Video: Software ...

Not all views offer the same unobstructed view of reality. Sometimes things, important things, inescapable things can get in the way. It is not always possible to change our perspective due to location, barriers or time therefore we need to use our imagination. Sherlock and Mycroft Holmes used deductive reasoning to take a few facts and paint an accurate picture of a real life that they had not yet seen or experienced. While none of us are one of the Holmes brothers, we need to realize that there are times where making a decision based on imperfect information is necessary. The grate on the bridge is a barrier, it blocks some of what we can see in the distance; we can fill in the blanks but only if we use a small portion of our imagination.