Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Next Book For Re-read Saturday

Peabody Library

Peabody Library – So many books, so little time.

We live and work in a dynamic era. In the software development field we are experiencing changes across the board in computing power, management styles, frameworks and techniques. Movements such as Agile and lean are just the tip of the iceberg. In order to build a base of knowledge and grow, IT practitioners need to read, listen, collaborate and experiment. While blogs, podcasts and conferences are great tools to explore the cutting edge, books are an important tool for building or expanding a base of personal knowledge.

I introduced the Re-Read SaturdayĀ feature on the Software Process and Measurement blog to help expose both my readers and myself to at least a few of the most important books. We have now re-read Coveyā€™s Seven Habits of Highly Effective People and we finished Kotterā€™s Leading Change last week. I choose the first two books, and it is your turn to choose the next book. Over the last twelve weeks I asked you to send me the two books that you felt were most influential to your career. A few observations:

  1. The list has 30 entries.
  2. There was NO runaway leader on the list.
  3. The first five on the list each got two mentions.

Since there was no clear winner, I have created a poll. The poll will allow each person to vote for three books. Pick the top three books that have had major impact on your career, OR perhaps the books you always wanted to read. The book that is on the top of the list on February 14 will be the next to be featured on Re-Read Saturday.

Take Our Poll (function(d,c,j){if(!d.getElementById(j)){var pd=d.createElement(c),s;pd.id=j;pd.src='https://s1.wp.com/wp-content/mu-plugins/shortcodes/js/polldaddy-shortcode.js';s=d.getElementsByTagName(c)[0];s.parentNode.insertBefore(pd,s);} else if(typeof jQuery !=='undefined')jQuery(d.body).trigger('pd-script-load');}(document,'script','pd-polldaddy-loader'));
Categories: Process Management

Setting up an OpenVPN server inside an AWS VPC

Agile Testing - Grig Gheorghiu - Sat, 01/31/2015 - 23:56
My goal in this post is to show how to set up an OpenVPN server within an AWS VPC so that clients can connect to EC2 instances via a VPN connection. It's fairly well documented how to configure a site-to-site VPN with an AWS VPC gateway, but articles talking about client-based VPN connections into a VPC are harder to find.

== OpenVPN server setup ==

Let's start with setting up the OpenVPN server. I launched a new Ubuntu 14.04 instance in our VPC and I downloaded the latest openvpn source code via:

wget http://swupdate.openvpn.org/community/releases/openvpn-2.3.6.tar.gz

In order for the 'configure' step to succeed, I also had to install the following Ubuntu packages:

apt-get install build-essential openssl libssl-dev lzop liblzo2-dev libpam-dev

I then ran the usual commands:

./configure; make; sudo make install

At this point I proceeded to set up my own Certificate Authority (CA), per the OpenVPN HOWTO guide.  As it turned out, I needed the easy-rsa helper scripts on the server running openvpn. I got them from github:

git clone https://github.com/OpenVPN/easy-rsa.git

To generate the master CA certificate & key, I did the following:

cd ~/easy-rsa/easyrsa3
cp vars.example vars

- edited vars file and set these variables with the proper values for my organization:
set_var EASYRSA_REQ_COUNTRY
set_var EASYRSA_REQ_PROVINCE
set_var EASYRSA_REQ_CITY
set_var EASYRSA_REQ_ORG
set_var EASYRSA_REQ_EMAIL
set_var EASYRSA_REQ_OU

./easyrsa build-ca
(this will use the info specified in the vars file above)

To generate the OpenVPN server certificate and key, I ran:

./easyrsa build-server-full server
(I was prompted for a password for the server key)

To generate an OpenVPN client certificate and key for user myuser, I ran:

./easyrsa  build-client-full myuser
(I was prompted for a password for the client key)

The next step was to generate the Diffie Hellman (DH) parameters for the server by running:

./easyrsa gen-dh

I was ready at this point to configure the OpenVPN server.

I created a directory called /etc/openvpn and copied the pki directory under ~/easy-rsa/easyrsa3 to /etc/openvpn. I also copied the sample server configuration file ~/openvpn-2.3.6/sample/sample-config-files/server.conf to /etc/openvpn.

I edited /etc/openvpn/server.conf and specified the following:

ca /etc/openvpn/pki/ca.crt
cert /etc/openvpn/pki/issued/server.crt
key /etc/openvpn/pki/private/server.key  # This file should be kept secret
dh /etc/openvpn/pki/dh.pem

server 10.9.0.0 255.255.255.0
ifconfig-pool-persist /etc/openvpn/ipp.txt

push "route 172.30.0.0 255.255.0.0"

The first block specifies the location of the CA certificate, the server key and certificate, and the DH certificate.

The 'server' parameter specifies a new subnet from which both the OpenVPN server and the OpenVPN clients connecting to the server will get their IP addresses. I set it to 10.9.0.0/24. The client IP allocations will be saved in the ipp.txt file, as specified in the ifconfig-pool-persist parameter.

One of the most important options, which I missed when I initially configured the server, is the 'push route' one. This makes the specified subnet (i.e. the instances in the VPC that you want to get to via the OpenVPN server) available to the clients connecting to the OpenVPN server without the need to create static routes on the clients. In my case, all the EC2 instances in the VPC are on the 172.30.0.0/16 subnet, so that's what I specified above.

Two more very important steps are needed on the OpenVPN server. It took me quite a while to find them so I hope you will be spared the pain.

The first step was to turn on IP forwarding on the server:

- uncomment the following line in /etc/sysctl.conf:
net.ipv4.ip_forward=1

- run
sysctl -p

The final step in the configuration of the OpenVPN server was to make it do NAT via itpables masquerading (thanks to rbgeek's blog post for these last two critical steps):

- run
iptables -t nat -A POSTROUTING -s 10.9.0.0/24 -o eth0 -j MASQUERADE

- also add the above line to /etc/rc.local so it gets run on reboot

Now all that's needed on the server is to actually run openvpn. You can run it in the foreground for troubleshooting purposes via:

openvpn /etc/openvpn/server.conf

Once everything works, run it in daemon mode via:

openvpn --daemon --config /etc/openvpn/server.conf

You will be prompted for the server key password when you start up openvpn. Haven't looked yet on how to run the server in a fully automated way.

Almost forgot to specify that you need to allow incoming traffic to UDP port 1194 in the AWS security group where your OpenVPN server belongs. Also allow traffic from that security group to the security groups of the EC2 instances that you actually want to reach over the OpenVPN tunnel.

== OpenVPN client setup ==

This is on a Mac OSX Mavericks client, but I'm sure it's similar for other clients.

Install tuntap
- download tuntap_20150118.tar.gz from http://tuntaposx.sourceforge.net
- untar and install tuntap_20150118.pkg

Install lzo
wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.06.tar.gz
tar xvfz lzo-2.06.tar.gz
cd lzo-2.06
./configure; make; sudo make install

Install openvpn
- download openvpn-2.3.6.tar.gz from http://openvpn.net/index.php/open-source/downloads.html
tar xvf openvpn-2.3.6.tar
cd openvpn-2.3.6
./configure; make; sudo make install

At this point ā€˜openvpn --helpā€™ should work.
The next step for the client setup is to copy the CA certificate ca.crt, and the client key and certificate (myuser.key and myuser.crt) from the OpenVPN server to the local client. I created an openvpn directory under my home directory on my Mac and dropped ca.crt in ~/openvpn/pki, myuser.key in ~/openvpn/pki/private and myuser.crt in ~/openvpn/pki/issued. I also copied the sample file ~/openvpn-2.3.6/sample/sample-config-files/client.conf to ~/openvpn and specified the following parameters in that file:
remote EXTERNAL_IP_ADDRESS_OF_OPENVPN_SERVER 1194
ca /Users/myuseropenvpn/pki/ca.crtcert /Users/myuser/openvpn/pki/issued/myuser.crtkey /Users/myuser/openvpn/pki/private/myuser.key
Then I started up the OpenVPN client via:
sudo openvpn ~/openvpn/client.conf(at this point I was prompted for the password for myuser.key)
To verify that the OpenVPN tunnel is up and running, I ping-ed the internal IP address of the OpenVPN server (in my case it was 10.9.0.1 on the internal subnet I specified in server.conf), as well as the internal IPs of various EC2 instances behind the OpenVPN server. Finally, I ssh-ed into those internal IP addresses and declared victory.
That's about it. Hope this helps!


























Late Start = Late Finish

Herding Cats - Glen Alleman - Sat, 01/31/2015 - 18:31

In a CrossTalk article Risk Management for Dummies, Tom DeMarco speaks about late software projects and the approaches for the solution. The notion ofĀ early start as the solution forĀ late finish needs to address the core question, but fails to address the root cause ofĀ lateness - no schedule margin.

How soon should we have started to finishĀ on or beforeĀ the needed finish time?

Screen Shot 2015-01-29 at 7.36.47 PM

The answer to this is simple:

  • If we have a schedule for the project, which contains the needed work efforts, in the planned sequence with an estimated duration for that effort - the Most Likely duration - then we can model the probability of a completion date with Monte Carlo Simulation.
  • The question isĀ what is the range of possible durations an activity can have, before we actually start performing that work?

Good question. And of course like all questions about some future activity we'll need toĀ Estimate those values. We can't actually manage - in any credible manner - without estimating. Anyone suggesting otherwise has likely only encountered trivial projects where theĀ Value at RiskĀ was low enough that no one cared is you were late or over budget.

An Example of Managing In The Presence of Uncertainty

From the Forward of Technical Risk Management, Jack V. Michaels, by Norman Augustine author ofĀ Augustine'sĀ Laws.

Columbus proposed a voyage to Ferdinand and Isabella in 1486. The monarchs promptly set up a special commission ofĀ learned men to study the proposal, which they found vague and arcane. After four years of unsatisfactory discussions with the master mariner, the commission's finally rejected the proposal.

To their credit they did not abandon Columbus and soonĀ recalled Columbus not to plan again, but to name his price for carrying out the voyage. The price was enormous. Columbus insisted on being knighted, appointed a grand admiral and viceroy, and given the unwillingness to compromise, the king and queen dismissed Columbus. Eventually, they relented and accepted all of Columbus's demand.

I (Norm Augustine) put forth this example because all parties involved could have benefited from risk management. Columbus could have presented his proposal containing each element of risk, but consider the payoff, well worth the investment.

Ferdinand and Isabella could have assessed the "price of doing nothing." The commission of experts could have have a framework in which they could evaluate theĀ technicalĀ feasibility of the proposal. If the parties had studied the technical risk management, perhaps we'd be celebrating 1486, not 1492

For an simple schedule, like the Wright Brothers meeting their contractual data to the U.S. Army of August 31 1908 for the delivery of the Wright Flyer, they needed a confidence level of 80% for theĀ on or beforeĀ date. The schedule margin of 10 days provided theĀ protection for that date.

Screen Shot 2015-01-31 at 8.22.36 AM

What Does This Mean?

The first meaning, missing from the CrossTalk article is.

Schedules without margin are late on day one

It meansĀ Estimating is Risk Management. No estimates, no risk management. No risk management, lower probability of project success. For schedule risk we needĀ irreducible andĀ reducibleĀ uncertainties that drive the schedule. The for theĀ reducible uncertainties, we need activities to reduce the risk from that uncertainty. ForĀ irreducibleĀ uncertainties, we need margin to protect each major deliverable.

Risk Management is How Adults Manage Projects - Tim Lister

Remember that next time you hearĀ we can't estimate, we don't want to estimate, estimates are a waste.

Here's How to Manage in the Presence of Uncertainty


Related articles Qualitative Risk Management and Quantitative Risk Management We Suck At Estimating Building the Perfect Schedule Estimating is Risk Management Probability and Statistics of Project Work
Categories: Project Management

Python/matpotlib: Plotting occurrences of the main characters in How I Met Your Mother

Mark Needham - Fri, 01/30/2015 - 22:29

Normally when I’m playing around with data sets in R I get out ggplot2 to plot some charts to get a feel for the data but having spent quite a bit of time with Python and How I met your mother transcripts I haven’t created a single plot. I thought I’d better change change that.

After a bit of searching around it seems that matplotlib is the go to library for this job and I thought an interesting thing to plot would be how often each of the main characters appear in each episode across the show.

I’ve already got all the sentences from each episode as well as the list of episodes pulled out into CSV files so we can start from there.

This is a sample of the sentences file:

$ head -n 10 data/import/sentences.csv
SentenceId,EpisodeId,Season,Episode,Sentence
1,1,1,1,Pilot
2,1,1,1,Scene One
3,1,1,1,[Title: The Year 2030]
4,1,1,1,"Narrator: Kids, I'm going to tell you an incredible story. The story of how I met your mother"
5,1,1,1,Son: Are we being punished for something?
6,1,1,1,Narrator: No
7,1,1,1,"Daughter: Yeah, is this going to take a while?"
8,1,1,1,"Narrator: Yes. (Kids are annoyed) Twenty-five years ago, before I was dad, I had this whole other life."
9,1,1,1,"(Music Plays, Title ""How I Met Your Mother"" appears)"

My first step was to transform the CSV file into an array of words grouped by episode. I created a dictionary, iterated over the CSV file and then used nltk’s word tokeniser to pull out words from sentences:

import csv
from collections import defaultdict
 
episodes = defaultdict(list)
with open("data/import/sentences.csv", "r") as sentencesfile:
    reader = csv.reader(sentencesfile, delimiter = ",")
    reader.next()
    for row in reader:
        episodes[row[1]].append([ word for word in nltk.word_tokenize(row[4].lower())] )

Let’s have a quick look what’s in our dictionary:

>>> episodes.keys()[:10]
['165', '133', '132', '131', '130', '137', '136', '135', '134', '139']

We’ve got some episode numbers as we’d expect. Now let’s have a look at some of the words for one of the episodes:

>>> episodes["165"][5]
['\xe2\x99\xaa', 'how', 'i', 'met', 'your', 'mother', '8x05', '\xe2\x99\xaa']

So we’ve got an list of lists of words for each episode but gensim (which I wanted to play around with) requires a single array of words per document.

I transformed the data into the appropriate format and fed it into a gensim Dictionary:

from gensim import corpora
texts = []
for id, episode in episodes.iteritems():
    texts.append([item for sublist in episode for item in sublist])
dictionary = corpora.Dictionary(texts)

If we peek into ‘texts’ we can see that the list has been flattened:

>>> texts[0][10:20]
['a', 'bit', 'of', 'a', 'dog', ',', 'and', 'even', 'though', 'he']

We’ll now convert our dictionary of words into a sparse vector which contains pairs of word ids and the number of time they occur:

corpus = [dictionary.doc2bow(text) for text in texts]

Let’s try and find out how many times the word ‘ted’ occurs in our corpus. First we need to find out the word id for ‘ted':

>>> dictionary.token2id["ted"]
551

I don’t know how to look up the word id directly from the corpus but you can get back an individual document (episode) and its words quite easily:

>>> corpus[0][:5]
[(0, 8), (1, 1), (2, 2), (3, 13), (4, 20)]

We can then convert that into a dictionary and look up our word:

>>> dict(corpus[0]).get(551)
16

So ‘ted’ occurs 16 times in the first episode. If we generify that code we end up with the following:

words = ["ted", "robin", "barney", "lily", "marshall"]
words_dict = dict()
for word in words:
    word_id = dictionary.token2id[word]
    counts = []
    for episode in corpus:
        count = dict(episode).get(word_id) or 0
        counts.append(count)
    words_dict[word] = counts

There’s quite a lot of counts in there so let’s just preview the first 5 episodes:

>>> for word, counts in words_dict.iteritems():
       print word, counts[:5]
 
lily [3, 20, 47, 26, 41]
marshall [8, 25, 63, 27, 34]
barney [9, 94, 58, 92, 102]
ted [16, 46, 66, 32, 44]
robin [18, 43, 25, 24, 34]

Now it’s time to bring out matplotlib and make this visual! I initially put all the characters on one chart but it looks very messy and there’s a lot of overlap so I decided on separate charts.

The only thing I had to do to achieve this was call plt.figure() at the beginning of the loop to create a new plot:

import matplotlib
matplotlib.use('TkAgg')
import matplotlib.pyplot as plt
pylab.show()
 
for word, counts in words_dict.iteritems():
    plt.figure()
    plt.plot(counts)
    plt.legend([word], loc='upper left')
    plt.ylabel('occurrences')
    plt.xlabel('episode')
    plt.xlim(0, 208)
    plt.savefig('images/%s.png' % (word), dpi=200)

This generates plots like this:

2015 01 30 21 15 03

This is good but I thought it’d be interesting to put in the season demarcations to see if that could give any more insight. We can call the function plt.axvline and pass in the appropriate episode number to achieve this effect but I needed to know the episode ID for the last episode in each season which required a bit of code:

import pandas as pd
df = pd.read_csv('data/import/episodes.csv', index_col=False, header=0)
last_episode_in_season = list(df.groupby("Season").max()["NumberOverall"])
 
>>> last_episode_in_season
[22, 44, 64, 88, 112, 136, 160, 184, 208]

Now let’s plug that into matplotlib:

for word, counts in words_dict.iteritems():
    plt.figure()
    plt.plot(counts)
    for episode in last_episode_in_season:
        plt.axvline(x=episode, color = "red")
    plt.legend([word], loc='upper left')
    plt.ylabel('occurrences')
    plt.xlabel('episode')
    plt.xlim(0, 208)
    plt.savefig('images/%s.png' % (word), dpi=200)

2015 01 30 21 10 52

The last thing I wanted to do is get all the plots on the same scale for which I needed to get the maximum number of occurrences of any character in any episode. It was easier than I expected:

>>> y_max = max([max(count) for count in words_dict.values()])
>>> y_max
260

And now let’s plot again:

for word, counts in words_dict.iteritems():
    plt.figure()
    plt.plot(counts)
    for episode in last_episode_in_season:
        plt.axvline(x=episode, color = "red")
    plt.legend([word], loc='upper left')
    plt.ylabel('occurrences')
    plt.xlabel('episode')
    plt.xlim(0, 208)
    plt.ylim(0, y_max)
    plt.savefig('images/%s.png' % (word), dpi=200)


Our charts are now easy to compare:

2015 01 30 21 23 48

2015 01 30 21 24 03

For some reason there’s a big spike of the word ‘ted’ in the middle of the 7th season – I’m clearly not a big enough fan to know why that is but it’s a spike of 30% over the next highest value.

The data isn’t perfect – some of the episodes list the speaker of the sentence and some don’t so it may be that the spikes indicate that rather than anything else.

I find it’s always nice to do a bit of visual exploration of the data anyway and now I know it’s possible to do so pretty easily in Python land.

Categories: Programming

Episode 218: Udi Dahan on CQRS (Command Query Responsibility Segregation)

Guest Udi Dahan talks with host Robert Blumen about the CQRS (command query responsibility segregation) architectural pattern. The discussion begins with a review of the command pattern. Then a high-level overview of CQRS, which consists of a separation of a command processing subsystem that updates a write model from one or more distinct and separate, […]
Categories: Programming

Interview with The Entrepreneur's Library on Getting Results the Agile Way

What is Agile Results all about?   What are the most important keys to using Agile Results to master productivity, time management, and work-life balance?

If you've ever wondered what Getting Results the Agile Way is all about, or want to know how to make the most of the book, this is it.  I answer these questions and more in my interview with The Entrepreneur's Library on Getting Results the Agile Way:

Interview with The Entrepreneur's Library on Agile Results

In this interview, Wade Danielson, the creator of The Entrepreneurā€™s Library, asks me the following questions:

  1. What was the inspiration behind writing this book?
  2. What makes this book different from others regarding this same topic?
  3. If the reader could only take one concept/principle/action item out of the entire book, what would you want that to be?
  4. Do you have a favorite quote from your book?
  5. If there was only one book you recommend to our listeners based on the way it has impacted your life, what would that be?

Wade also gives me a chance to give a walkthrough of the book, Getting Results the Agile Way, where I explain how to make the most of the book, and what each section is really about.

Itā€™s a unique chance to get the philosophy behind Agile Results and why itā€™s really a personal results system for work and life.   Itā€™s not a system that you break yourself against.  Instead, itā€™s a simple system for meaningful results that supports you and the way you work.  It helps you optimize your productivity by focusing on the wins that matter, playing to your strengths, and using your best energy for your best results.

An amazing thing happens when you become more focused and productive ā€¦

You get more out of life.

And you can get more done in a day than other people get done all week.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For January 30th, 2015

Hey, it's HighScalability time:


It's a strange world...exotic, gigantic molecules Fit Inside Each Other like Russian nesting dolls
  • 1.39 billion: Facebook Monthly Active Users; $18 billion profit: Apple in 3 months; 200 million: Kik users; 11.2 billion: age of the oldest known solar system; 3 billion: videos viewed per day on Facebook
  • Quotable Quotes:
    • @kevinroose: This dude wins SF bingo. RT @caro: An Uber driver is Airbnb'ing the trunk of his Tesla for $85/night.
    • @BenedictEvans: Only 16% of Facebook DAUs aren't using it on mobile
    • @rezendi: Yo's Law: "in the 21st century tech industry, satire and reality are not merely indistinguishable but actually interchangeable."
    • Brent Ozar: I recommend that people back up data, not servers.
    • @AnnaPawlicka: "Shared State is the Root of All Evil"
    • Peter Lawrey: micro-day - about 1/12 of a second. micro-century - 51.3 minutes. femto-parsec - about 30 metres.
    • TapirLiu: OH: docker is like a condom to protect your computer from Node.
    • @DigitCurator: "The Next Decade In Storage": Resistive RAM promises better scaling, efficiency, and 1000x endurance of flash memory 
    • @BenedictEvans: At the end of 2014 Apple had ~650-675m live iOS devices. With zero unit sales growth, 700-720m by end 2015. Consumer PCs in use - 7-800m
    • @MailChimp: We sent 14.1 billion emails in December, including 741 million on Cyber Monday.
    • @mjpt777:  That's in the past. We can now do 20 million per second :-) per stream.
    • @bradwilson: Conclusions: 1. Ethernet over power does not perform as well as WiFi (??) 2. Ethernet over power hates being shared among multiple PCs
    • @mjpt777: Specialized Evolution of the General-Purpose CPU  - note that performance per watt is approx doubling per generation. 
    • @nighitingale: "The Earth is 4.6 billion years old. Scaling to 46 years, humans have been here 4 hours, the industrial..."
    • Joseph Campbell: The hero’s journey always begins with the call. One way or another, a guide must come to say, “Look, you’re in Sleepy Land. Wake. Come on a trip."
    • Frank Herbert: the most persistent principles of the universe were accident and error.

  • Will Facebook ever figure out this mobile thing? Not long ago that was the big question. We have an answer. In the fourth quarter, the percentage of its advertising revenue from mobile devices increased to 69%, up from 66% in the third quarter and 53% a year earlier. Mobile daily active users were 745 million on average for December 2014, an increase of 34 percent year-over-year.

  • The power of smart: Facebook’s Powerful Ad Tools Grew Its Revenue 25X Faster Than User Count. Facebook might be running out of people, but they aren't running out of ways of monetizing those people. Math grows faster than users.

  • The Cathedral of Computation by Ian Bogost. Agree in part. There does seem to be an uncritical acceptance of algorithms, as if because they enliven machines they are some how pure and objective, when the opposite is the case. Algorithms are made for human purposes by teams of humans and show the biases and hubris of their makers. And like all creatures, algorithms should be subject to skepticism, law, and review.

  • We have many long running debates in tech. Server side vs client side rendering is just one of them. A thoughtful analysis: Tradeoffs in server side and client side rendering by Malte Ubl.  Bret Slatkin boldly claims: Experimentally verified: "Why client-side templating is wrong". He concludes: I hope never to render anything server-side ever again. I feel more comfortable in making that choice than ever thanks to all this data. I see rare occasions when server-side rendering could make sense for performance, but I don't expect to encounter many of those situations in the future.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

We Suck At Estimating

Herding Cats - Glen Alleman - Fri, 01/30/2015 - 17:40

SadeeyoreWhen I hear "we suck at estimating," or "we can't make good estimates, because people take them as commitments," I wonder what would be the answer if those statements were restated as...Ā 

"We suck at designing good database schemas." "I can never figure out how to configure the IIS VPN tunneling server." "This Sharepoint Server UI in SP 2012 is so much different than 2007, I can't possibly figure out how to help my customer."

Or, this 401(k) calculation for estimating my retirement is just beyond my ability to comprehend." Or, I can't possibly figure out how much money I need for my summer vacation with my family on a European trip, with relatives I haven't even met, and hotels and rental cars, train tickets, and meals, and other expenses, and airline reservations, and all the unknown and may be unknowable stuff. I'm just going to sit here and mope. Yep, that's me, Eeyore, woo is me. Ā 

What is it in some software domains that allows well educated, experienced developers to simply give up, walk way, allow themselves the be subjected to Dilbert Bosses? When they'd never talk that way about something they are skilled, experienced, and capable of doing?

It's a repeating theme in the #NoEstimates domain. You'd never hear a Flight Avionics engineer at a major space and defense company here in Denver say "we suck at estimating, we suck at testing, we suck at systems integration." Those engineers are asked all the time to "invent new physics to solve a problem." Or other colleagues literally inventing solutions of complex oil & gas software based process control algorithms. Or other colleagues at a major consulting firms deploying ERP and encountering legacy systems they've never seen or integrated before, saying "we can't figure this out, it's too hard and our managers will take our estimates as commitments, so we're simply not going to estimate our work."

Is it missing the engineering discipline taught in college. Where a probability and statistics class is mandatory for all 4 years to be called an engineer. Or even in the business and finance discipline estimating, using complex approaches - time series analysis, principle component analysis, bayesian statistics - is mandatory to be called a finance major. Or in the sciences in general. I say this from direct experience with our children and their significant others who are in finance, bio-sciences, and mechanical engineering all writing software for money as part of their actual profession.

And this whole notion that somehow writing software is not engineering is simply BS. It is engineering in many business domains. So maybe it's a domain issue. Maybe non-engineering cultures can't figure this out.Ā 

Or is it maturity? Many appear to be very mature developers? Or is it culture, not geographic culture, but lack of having experienced a development culture where business and engineering work together to determine risks, corrective actions, increasing maturity of the deliverables development processes that allow for growth in both technical and business capabilities - all using agile by the way.

Everything is Uncertain

Business operates on risk taking. Risk taking is about making decisions in the presence of uncertainty. Uncertainty is about probable outcomes - positive or negative - in the future. To make decisions about uncertain outcomes in the future, we need to estimate those outcomes, the cost to achieve them, and the possible benefits from that cost.

No credible business person fails to understand this.

If they do, they won't be in business for long. It's taught in school - business school. Same for engineering and computer science school.

What is it that brings smart, capable, intelligent people to say "we suck," when the facilities (classes, books, articles, tools, external advisors) for learning how to estimate for the required reasons of making decisions about other peoples money for future outcomes, is the very basis of all business - is at hand, with simple learning and practice. These facilities are also the basis of all learning. "We suck" says "we're not capable of learning."

Are those of the #NoEstimate ilk saying "I can't learn?" or are they saying "I don't want to learn?" Because those providing them the money have a vested interest in knowing how much money is needed to produce that treasuredĀ Value they claim to be producing.Ā 

Just like learning a programming language. Moving from SP 2007 to SP 2012 (that actually sucks), dealing with cyber security. We donā€™t say ā€œwe suckā€ and give up. What is actually going on here? Is some portion of the development community become Eeyore?

Related articles Your Project Needs a Budget and Other Things Building the Perfect Schedule Don't Manage By Quoting Dilbert Qualitative Risk Management and Quantitative Risk Management Estimating is Risk Management
Categories: Project Management

Run your iOS app without overwriting the App Store version

Xebia Blog - Fri, 01/30/2015 - 13:59

Sometimes when you're developing a new version of your iOS app, you'd like to run it on your iPhone or iPad and still be able to run the current version that is released on the App Store. Normally when you run your app from Xcode on your device, it will overwrite any existing version. If you then want to switch back to the version from the App Store, you'll have to delete the development version and download it again from the App Store.

In this blog post I'll describe how you can run a test version of your app next to your production version. This method also works when you have embedded extensions (like the Today Widget or WatchKit app) in your app or when you want to beta test your app with Apple's TestFlight.

There are two different ways to accomplish this. The first method is to create a new target within your project that runs the same app but with a different name and identifier. With iOS 8 it is now possible to embed extensions in your app. Since these extensions are embedded in the app target, this approach doesn't work when you have extensions and therefore I'll only describe the second method which is based on User-Defined Settings in the build configuration.

Before going into detail, here is a quick explanation of how this works. Xcode already creates two build configurations for us: Release and Debug. By adding some User-Defined Settings to the build configurations we can run the Debug configuration with a different Bundle identifier than the Release configuration. This essentially creates a separate app on your device, keeping the one from the App Store (which used the Release configuration) intact.

To beta distribution of the app built with debug configuration easier we'll create multiple schemes.

The basics

Follow these steps exactly to run a test version on your device next to your App Store version. These steps are based on Xcode 6.1.1 with an Apple Developer Account with admin privileges.

Click on your project in the Project Navigator and make sure the main target is selected. Under both the General and Info tabs you will find the Bundle identifier. If you change this name and run your app, it will create a new app next to the old one. Add -test to your current Bundle identifier so you get something like com.example.myapp-test. Xcode will probably tell you that you don't have a provisioning profile. Let Xcode fix this issue for you and it will create a new Development profile for you named something like iOSTeam Provisioning Profile: com.example.myapp-test.

So we're already able to run a test version of our app next to the App Store version. However, manually changing the Bundle identifier all the time isn't very practical. The Bundle identifier is part of the Info.plist and it's not derived from a Build configuration property like for example the Bundle name (which uses ${PRODUCT_NAME} from the build configuration by default). Therefore we can't simply specify a different Bundle identifier for a different Build configuration using an existing property. But, we can use a User-Defined Setting for this.

Go to the Build Settings tab of your project and click the + button to add a User-Defined Setting. Name the new setting BUNDLE_ID and give it the value of the test Bundle identifier you created earlier (com.example.myapp-test). Then click the small triangle in front of the setting to expand the setting. Remove the -test part from the Release configuration. You should end up with something like this:

Screen Shot 2015-01-30 at 11.29.43

Now go to the Info tab and change the value of the Bundle identifier to ${BUNDLE_ID}. In the General tab you will still see the correct Bundle Identifier, but if you click on it you see the text is slightly grayed out, which means it's taken from a Build configuration setting.

Screen Shot 2015-01-30 at 11.35.52

To test if this works, you can change edit the current Scheme and change the Build Configuration of the Run action from Debug to Release. Close the Scheme window and go to the General tab again to see that the Bundle Identifier has changed to the Release BUNDLE_ID. (If you still had the General tab open and don't see the change then switch to another tab and back; the panel will reload the identifier). Make sure to change the Build configuration back to Debug in your scheme afterwards.

When you now Archive an app before you release it to the App Store, it will use the correct identifier from the Release configuration and when you run the app from Xcode on your device, it will use the identifier for testing. That way it no longer overwrites your App Store version on your device.

App name

Both our App Store app and test app still have the same name. This makes it hard to know which one is which. To solve this, find the Product Name in the Build Settings and change the name for the Debug configuration to something else, like MyApp Test.

Beta distribution

What if you want to distribute a version of the app for Beta testing (through TestFlight or something similar) to other people that also shouldn't overwrite their Apple Store version? Our Archive action is using the Release build configuration. We could change that manually to Debug to have the test Bundle identifier but then we would be getting all of the Debug settings in our app, which is not something we want. We need to create another Build configuration.

Go to the project settings of your project (so not the target settings). Click the + button under Configurations and duplicate the Release configuration. Call the new configuration AdHoc. (You might already have such a Build configuration for other reasons, in that case you can skip this step and use that one for the next steps.)

Now go to the Build Settings of your main target and change the AdHoc value of the User-Defined Setting BUNDLE_ID to the same as the Debug value. Do the same for the Product name is you changed that in the previous step.

We could already make a Beta test Archive now by manually changing the configuration of the Archive action to Debug. But it makes it easier if we create a new scheme to do this. So go to the Manage Schemes and click to + button at the bottom left to create a new scheme. Make sure that your main target is selected as Target and add " Test" to the name so you end up with something like "MyApp Test". Also check the Shared checkbox if you are sharing your schemes on your version control system.

Double click the new scheme to edit it and change the build configuration of the Archive action to AdHoc. Now you can Archive with the MyApp Test scheme selected to create a test version of your app that you can distribute to your testers without it overwriting their App Store version.

To avoid confusion about which build configuration is used by which scheme action, you should also change the configuration of the Profile action to AdHoc. And in your normal non-test scheme, you can change the build configuration of all actions to Release. This allows you to run the app in both modes of your device, which sometimes might be necessary, for example when you need to test push notifications that only work for the normal Bundle identifier.

Extensions

As mentioned in the intro, the main reason to use multiple schemes with different build configurations and User-Defined settings as opposed to creating multiple targets with different Bundle identifiers is because of Extensions, like the Today extension or a WatchKit extension. An extension can only be part of a single target.

Extensions make things even more complex since their Bundle identifier needs to be prefixed with the parent app's bundle identifier. And since we just made that one dynamic, we need to make the Bundle identifier of our extensions dynamic as well.

If you don't already have an existing extension, create a new one now. I've tested the approach described below with Today extensions and WatchKit extensions but it should work with any other extension as well.

The steps for getting a dynamic Bundle identifier for the extensions is very similar as the once for the main target so I won't go into too much detail here.

First open the Info tab of the new target that was created for the extension, e.g. MyAppToday. You'll see here that the Bundle display name is not derived from the PRODUCT_NAME. This is probably because the product name (which is derived from the target name) is something not very user friendly like MyAppToday and it is assumed that you will change it. In case of the Today extension, running a test version of the app next to the App Store version will also create 2 Today extensions in the Today view of the phone. To be able to differentiate between the two we'll also make the Bundle display name dynamic.

Change the value of it to ${BUNDLE_DISPLAY_NAME} and then add a User-Defined Setting for BUNDLE_DISPLAY_NAME with different names for Debug/AdHoc and Release.

You might have noticed that the Bundle identifier of the extension is already party dynamic, something like com.example.myapp.$(PRODUCT_NAME:rfc1034identifier). Change this to ${PARENT_BUNDLE_ID}.$(PRODUCT_NAME:rfc1034identifier) and add a User-Defined Setting for PARENT_BUNDLE_ID to your Build Settings. The values of the PARENT_BUNDLE_ID should be exactly the same as the ones you used in your main target, e.g. com.example.myapp for Release and com.example.myapp-test for Debug and AdHoc.

That's it, you can now Run and Archive your app with extensions who's Bundle identifier are prefixed with the parent's Bundle identifier.

App Groups entitlements

You might have an extension that shares UserDefaults data or Core Data stores with the main app. In that case you need to have matching App Groups entitlements in both your main app and extensions. Since we have dynamic Bundle identifiers that use different provisioning profiles, we also have to make our App Groups dynamic.

If you don't have App Groups entitlements (or other entitlements) yet, go to the Capabilities tab of your main target and switch on App Groups. Add an app group in the form off group.[bundle identifier], e.g. group.com.example.myapp. This will generate an entitlements file for your project (MyApp.entitlements) and set the Code Signing Entitlements of your Build Settings to something like MyApp/MyApp.entitlements. Locate the entitlements file in Finder and duplicate it. Change the name of the copy by replacing " Copy" with "Test" (MyAppTest.entitlements). Drag the copy into your project. You should now have two entitlement files in your project. Open the Test entitlements file in Xcode's Property List editor and add "-test" to the value of Item 0 under com.apple.security.application-groups to match it with the Bundle identifier we used for testing, e.g. com.example.myapp-test. Now go back to the Build Settings and change the Debug and AdHoc values of Code Signing Entitlements to match with the file name of the Test entitlements.

Repeat all these steps for the Extension target. Xcode will also generate the entitlements file in the extension folder. You should end up with two entitlements files for your main target and two entitlements files for every extension.

The Application Groups for testing need to be added to the provisioning profiles which Xcode will handle automatically for you. It might warn/ask you for this while building.

Conclusion

It might be quite some work to follow all these steps and to get everything to work, but if you use your normal iPhone for development and still want to use or show a stable version of your app at the same time, it's definitely worth doing. And your Beta testers will thank your for it as well.

R: ggplot2 ā€“ Each group consist of only one observation. Do you need to adjust the group aesthetic?

Mark Needham - Fri, 01/30/2015 - 01:27

I’ve been playing around with some weather data over the last couple of days which I aggregated down to the average temperature per month over the last 4 years and stored in a CSV file.

This is what the file looks like:

$ cat /tmp/averageTemperatureByMonth.csv
"month","aveTemperature"
"January",6.02684563758389
"February",5.89380530973451
"March",7.54838709677419
"April",10.875
"May",13.3064516129032
"June",15.9666666666667
"July",18.8387096774194
"August",18.3709677419355
"September",16.2583333333333
"October",13.4596774193548
"November",9.19166666666667
"December",7.01612903225806

I wanted to create a simple line chart which would show the months of the year in ascending order with the appropriate temperature.

My first attempt was the following:

df = read.csv("/tmp/averageTemperatureByMonth.csv")
df$month = factor(df$month, month.name)
 
ggplot(aes(x = month, y = aveTemperature), data = df) + 
  geom_line( ) + 
  ggtitle("Temperature by month")

which resulted in the following error:

geom_path: Each group consist of only one observation. Do you need to adjust the group aesthetic?

My understanding is that the points don’t get joined up by default because the variable on the x axis is not a continuous one but rather a factor variable.

One way to work around this problem is to make it numeric, like so:

ggplot(aes(x = as.numeric(month), y = aveTemperature), data = df) + 
  geom_line( ) + 
  ggtitle("Temperature by month")

which results in the following chart:

2015 01 30 00 25 18

This isn’t bad but it’d be much nicer if we could have the month names along the bottom instead.

It turns out we can but we need to specify a group that each point belongs to. ggplot will then connects points which belong to the same group.

In this case we don’t really have one so we’ll define a dummy one instead:

ggplot(aes(x = month, y = aveTemperature, group=1), data = df) + 
  geom_line( ) + 
  ggtitle("Temperature by month")

And now we get the visualisation we want:

2015 01 29 23 28 23

Categories: Programming

3 Pā€™s of Agile Centers of Excellence

photo

An Agile center of excellence (ACoE) provides support and energy to an Agile transformation within an organization. It supports through leadership, evangelization, best practices, research, support and/or training for agile and lean ideas. ACoEā€™s support can be categorized in three inter-related areas. These areas, the three ā€œPā€™s,ā€ are people, process and project.

People are the heart and soul of any development process. As we have noted, Agile has an enormous focus on people (remember the Agile value of valuing people over process). The ACoE provides support to people though bringing new ideas into the organization, by providing coaching, developing coaches and acting as change agents.

Agile is a set of processes, or sets of steps taken to achieve a specific end. A recipe is a process, as is a daily stand-up meeting or checking code in and out of configuration management tool. The ACoE supports Agile processes by capturing process, identifying and fostering the use of relevant metrics (collection and reporting are typically PMO functions ā€“ to be discussed in the near future), facilitating communities of practices and providing tools.

Projects are the currency of most IT organizations. At its simplest a project is an enterprise with a start and end that is organized to deliver a result. ACoEs support the performance of Agile teams at a project-level as coaches. Coaches are folks who deliver help to teams, stakeholders and other leaders within an organization so they learn how to be Agile. At the project-level, coaches help teams use and tweak processes to meet the teamā€™s needs, provide training and support for tools and processes and help the team learn how to ask the hard questions about how the team is using Agile.

The primary goal of the ACoE is to provide practitioners with the tools, techniques and capabilities they need to be Agile. By helping teams perform, the ACoE also helps sell and maintain the Agile transformation. Both of these goals begin as an organization starts a transformation to Agile and continue to be important as teams evolve and continuously improve. The ACoE delivers value by addressing the three Ps. For example, through the role of coaching and by facilitating communities of practice, the ACoE helps to promote an environment where there is consistency of practice and where innovation can happen. While the combination of innovation and consistency might sound contradictory, coaches often act as an Agile Johnny Appleseed. ACoE coaches see how teams work, the changes that have made to the processes and why those changes were made. The ACoE can then help to spread ideas that prove to be valuable through coaching, referrals or discussion in communities of practice.


Categories: Process Management

Another elasticsearch blog post, now about Shield

Gridshore - Thu, 01/29/2015 - 21:13

<p>I just wrote another piece of text on my other blog. This time I wrote about the recently release elasticsearch plugin called Shield. If you want to learn more about securing your elasticsearch cluster, please head over to my other blog and start reading</p>

http://amsterdam.luminis.eu/2015/01/29/elasticsearch-shield-first-steps-using-java/

The post Another elasticsearch blog post, now about Shield appeared first on Gridshore.

Categories: Architecture, Programming

Setting boundaries in Agile Requirements Management

Software Requirements Blog - Seilevel.com - Thu, 01/29/2015 - 19:05
I was once part of a teamĀ atĀ a consumer-facing technology company, who described themselves as incredibly fast-paced, operating within a rapidly changingĀ environment. This team was smaller (4 developers in a 10-person business unit), so I expected the ability to get things done quickly to be one of our strengths. Upon joining, I was surprised to hear […]
Categories: Requirements

Estimating is Risk Management

Herding Cats - Glen Alleman - Thu, 01/29/2015 - 18:06

Risk Management is about many things - but it is first and foremost about estimating future outcomes.Ā 

Suppose one of you wants to build a tower. Will he not first sit down and estimate the cost to see if he has enough money to complete it? For if he lays the foundation and is not able to finish it, everyone who sees it will ridicule him saying, "This fellow began to build and was not able to finish." - Luke 14:28-30

To manage the risk ofĀ not enough money, not enough time, the probability of unacceptable outcomes we need to estimate. It's as simple as that and as complex as that.

Risk management is essential for development and production of products and services because key information about cost schedule, and technical performance are uncertain and many times unknown until later in the project.

Proceeding to spend other peoples money in the presence of these uncertainties is not only bad management, it's bad economics - the microeconomics of decision making requires estimating the impacts of any decision - theĀ opportunity costĀ of that decision.

Risk management is concerned with the outcome of future events, whose exact outcome is unknown, and with how to deal with these uncertainties as a range of possible outcomes. -Ā Effective Risk Management: Some Keys to Success, 2nd Edition, Edmund H. Conrow.

So when you hereĀ we can make decisions about the future without estimating ask yourself, did that speaker ever read about risk management. And as Tim Lister says

Risk Management is How Adults Manage Projects

Related articles Probability and Statistics of Project Work Qualitative Risk Management and Quantitative Risk Management Decision Making in the Presence of Uncertainty All The World's A Random Process Your Project Needs a Budget and Other Things
Categories: Project Management

Some Personal Questions About Programming

Making the Complex Simple - John Sonmez - Thu, 01/29/2015 - 16:00

In this video, I answer some questions from Pluralsight about my career and my views on programming.

The post Some Personal Questions About Programming appeared first on Simple Programmer.

Categories: Programming

What Development & Test Managers do in Agile Organizations

Is there room for functional managers, such as development and test managers, in agile organizations? Maybe. It depends on whether they take the role of an agile manager.

If you have organized as a feature teams-based organization, the functional managers (development, test, analysis, whatever) can take these responsibilities:

  • Develop trusting relationships with the people on the project team, and in their function.
  • Provide coaching and mentoring opportunities for people.
  • Provide communities of practice for the people.
  • Remove obstacles for the people and team.
  • Start and nurture the hiring effort whenever you need to hire new people.
  • Help people with career development.
  • Provide feedback to people, and help people learn how to provide feedback (meta-feedback).
  • Provide coaching and meta-coaching when people want it.
  • Help the organization understand its capacity and make decisions about the project portfolio.
  • Help influence the rest of the organization with the agile culture.

Functional managers are champions for the team, and shepherds for the process. They are servant leaders.

Here’s what functional managers do not do:

  • Have status conversations. If the team is agile, the team understands its status. If you need help seeing their board, that’s a problem the team needs to solve. If they need help seeing their status, they need to change their boardĀ or their process for updating each other.
  • Move people on or off teams, once you or the team establishes itself.
  • Ask people to do something the team has not committed to, or that the product owner has not added to the kanban board. That’s right. “Your” team doesn’t work for you; the team works for the product owner.
  • Micromanage any part of the project work. Or, manage any part of the project work.

What does this mean? It means that the team members are leaders. Agile pushes responsibility into the teams, and away from traditional management. Agile requires leadership at all levels.

Agile challenges managers to recreate their jobs. An agile transformation requires managers work in an agile way, and work differently than before.

If you want to learn more about the role of leaders and managers in agile, join Gil Broza and me at The Influential Agile Leader, either in San Francisco or London this year. We still have an early bird price until mid-February. Don’t miss this opportunity to help your agile transition and your career.

Categories: Project Management

Live Webinar with JetBrains: Software Architecture as Code

Coding the Architecture - Simon Brown - Thu, 01/29/2015 - 14:17

I'm doing a live and free webinar with Trisha Gee and the other fine people over at JetBrains on February 12th at 15:00 GMT. The topic is "software architecture as code" and I'll be talking about/showing how you can create a software architecture model in code, rather than drawing static diagrams in tools such as Microsoft Visio.

Over the past few years, I've been distilling software architecture down to its essence, helping organisations adopt a lightweight style of software architecture that complements agile approaches. This includes doing "just enough" up front design to understand the significant structural elements of the software, some lightweight sketches to communicate that vision to the team, identifying the highest priority risks and mitigating them with concrete experiments. Software architecture is inherently about technical leadership, stacking the odds of success in your favour and ensuring that everybody is heading in the same direction.

But it's 2015 and, with so much technology at our disposal, we're still manually drawing software architecture diagrams in tools like Microsoft Visio. Furthermore, these diagrams often don't reflect the implementation in code, and vice versa. This session will look at why this happens and how to resolve the conflict between software architecture and code through the use of architecturally-evident coding styles and the representation of software architecture models as code.

Please sign-up here if you'd like to join us.

Categories: Architecture

Join Me on My Private Virtual Street Team in February!

NOOP.NL - Jurgen Appelo - Thu, 01/29/2015 - 09:26
Michal Moitk

Do you want to join me on my Virtual Street Team in February?

As my Virtual Street Team member you get:

Exclusive access to me via a private channel
Several exclusive on-line chats with me
A free Kindle or ePub copy of the #Workout book
Free signed copies of the book and my previous two books

The post Join Me on My Private Virtual Street Team in February! appeared first on NOOP.NL.

Categories: Project Management

Instagram Strategy to Radically Reduce Traffic: Kill all the spambots!

RIP to my fallen robot followers on Instagram, if there's a heaven for robot instagram users, you guys are in there

— alldaychubbyboy (@Allday)

How do you scale to handle increased user traffic? Have less traffic. No, this is not a koan. The best way to deal with traffic is not to have it. 

In a two day span Instagram disappeared 18.9 million users or more than 29 percent of their "followers." Justin Bieber lost 3.5 million followers (15 percent), Kim Kardashian lost 1.3 million followers (5.5 percent), Rihanna lost 1.2 million followers.

Instagram explains this dramatic reckoning was achieved by "removing deactivated spam accounts and accounts that violated its community guidelines." 

In an age when high user counts and tantalizing engagement metrics are more valuable than bitcoins, this can't have been an easy decision, but it was made after being bought by Facebook.

Why? Gabe Madway, an Instagram spokesman, tells us why: We totally get that it’s uncomfortable for people. The overall goal is we want it to be perceived that the people following you are real.

Uncomfortable is an understatement. A BuzzFeed article nicely captured some of the anger, here's just one example (could be NSFW):

Categories: Architecture

Building the Perfect Schedule

Herding Cats - Glen Alleman - Wed, 01/28/2015 - 16:30

Plans are critical, a schedule to implement that plan comes next. With the Plan, the sequence of the work is needed to assure the value to the customer is delivered in the proper order. That order is an order because the business (or the mission) usually can't acceptĀ all the features and functions at once. So the Plan is the top level sequence, and the schedule is the next level down.

Building the perfect schedule (v6) from Glen Alleman Related articles Your Project Needs a Budget and Other Things Qualitative Risk Management and Quantitative Risk Management
Categories: Project Management