Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Programmers and Politics

Making the Complex Simple - John Sonmez - Mon, 07/13/2015 - 16:00

I have to admit it, after all this time, I still can’t understand why so many otherwise intelligent people make themselves look like absolute idiots by arguing on the internet. This week, I’m looking at my Facebook feed—perhaps I shouldn’t, because I can’t help but see a stream of idiots arguing with other idiots. Not […]

The post Programmers and Politics appeared first on Simple Programmer.

Categories: Programming

SPaMCAST 350 - Arlene Minkiewicz, Technical Debt

Software Process and Measurement Cast - Sun, 07/12/2015 - 22:00

Software Process and Measurement Cast 350 features our interview with Arlene Minkiewicz. Arlene and I talked technical debt.  Our discussion included the definition of technical debt, a set of philosophies for technical debt and perhaps a few solutions. Regardless of philosophy or approach, a little technical debt goes a long way!

Arlene F. Minkiewicz is the Chief Scientist at PRICE Systems, LLC with over 30 years of experience at PRICE building cost models.  She leads the cost research activity for TruePlanning, the suite of cost estimating products that PRICE provides.  She is a software measurement expert dedicated to finding creative solutions focused on helping make software development professionals successful.  She is widely published and speaks frequently on software related topics.  She has a BS in Electrical Engineering from Lehigh University and an MS in Computer Science from Drexel University.

Email: Arlene.Minkiewicz@PRICESystems.com
Twiitter: @arlenemink
Website:  www.pricesystems.com

Call to Action!

I have a challenge for the Software Process and Measurement Cast listeners for the next few weeks. I would like you to find one person that you think would like the podcast and introduce them to the cast. This might mean sending them the URL or teaching them how to download podcasts. If you like the podcast and think it is valuable they will be thankful to you for introducing them to the Software Process and Measurement Cast. Thank you in advance!

Re-Read Saturday News

We have just begun the Re-Read Saturday of The Mythical Man-Month (buy it here to support the podcast)We are off to rousing start beginning with the Tar Pit. Get a copy now and start reading!

The Re-Read Saturday and other great articles can be found on the Software Process and Measurement Blog.

Remember: We just completed the Re-Read Saturday of Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement which began on February 21nd. What did you think?  Did the re-read cause you to read The Goal for a refresher? Visit the Software Process and Measurement Blog and review the whole re-read.

Note: If you don’t have a copy of the book, buy one. If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

Upcoming Events

Software Quality and Test Management 

September 13 – 18, 2015

San Diego, California

http://qualitymanagementconference.com/

I will be speaking on the impact of cognitive biases on teams!  Let me know if you are attending!

 

More on other great conferences soon!

Next SPaMCast

The next Software Process and Measurement Cast will feature our essay on distributed Agile. Distributed Agile is not just Scrum and other Agile techniques over a distance.  As distribution and cultural diversity increase what worked for a co-located team will often fall short.  We will identify solutions next week!

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

Categories: Process Management

SPaMCAST 350 – Arlene Minkiewicz, Technical Debt

 www.spamcast.net

http://www.spamcast.net

Listen Now

Subscribe on iTunes

Software Process and Measurement Cast 350 features our interview with Arlene Minkiewicz. Arlene and I talked technical debt.  Our discussion included the definition of technical debt, a set of philosophies for technical debt and perhaps a few solutions. Regardless of philosophy or approach, a little technical debt goes a long way!

Arlene F. Minkiewicz is the Chief Scientist at PRICE Systems, LLC with over 30 years of experience at PRICE building cost models.  She leads the cost research activity for TruePlanning, the suite of cost estimating products that PRICE provides.  She is a software measurement expert dedicated to finding creative solutions focused on helping make software development professionals successful.  She is widely published and speaks frequently on software related topics.  She has a BS in Electrical Engineering from Lehigh University and an MS in Computer Science from Drexel University.

Email: Arlene.Minkiewicz@PRICESystems.com
Twiitter: @arlenemink
Website:  www.pricesystems.com

Call to Action!

I have a challenge for the Software Process and Measurement Cast listeners for the next few weeks. I would like you to find one person that you think would like the podcast and introduce them to the cast. This might mean sending them the URL or teaching them how to download podcasts. If you like the podcast and think it is valuable they will be thankful to you for introducing them to the Software Process and Measurement Cast. Thank you in advance!

Re-Read Saturday News

We have just begun the Re-Read Saturday of The Mythical Man-Month (buy it here to support the blog and podcast). We are off to rousing start beginning with the Tar Pit. Get a copy now and start reading!

The Re-Read Saturday and other great articles can be found on the Software Process and Measurement Blog.

Remember: We just completed the Re-Read Saturday of Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement which began on February 21nd. What did you think?  Did the re-read cause you to read The Goal for a refresher? Visit the Software Process and Measurement Blog and review the whole re-read.

Note: If you don’t have a copy of the book, buy one. If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

Upcoming Events

Software Quality and Test Management 

September 13 – 18, 2015

San Diego, California

http://qualitymanagementconference.com/

I will be speaking on the impact of cognitive biases on teams!  Let me know if you are attending!

 

More on other great conferences soon!

Next SPaMCast

The next Software Process and Measurement Cast will feature our essay on distributed Agile. Distributed Agile is not just Scrum and other Agile techniques over a distance.  As distribution and cultural diversity increase what worked for a co-located team will often fall short.  We will identify solutions next week!

 

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.


Categories: Process Management

R: I write more in the last week of the month, or do I?

Mark Needham - Sun, 07/12/2015 - 10:53

I’ve been writing on this blog for almost 7 years and have always believed that I write more frequently towards the end of a month. Now that I’ve got all the data I thought it’d be interesting to test that belief.

I started with a data frame containing each post and its publication date and added an extra column which works out how many weeks from the end of the month that post was written:

> df %>% sample_n(5)
                                                               title                date
946  Python: Equivalent to flatMap for flattening an array of arrays 2015-03-23 00:45:00
175                                         Ruby: Hash default value 2010-10-16 14:02:37
375               Java/Scala: Runtime.exec hanging/in 'pipe_w' state 2011-11-20 20:20:08
1319                            Coding Dojo #18: Groovy Bowling Game 2009-06-26 08:15:23
381                   Continuous Delivery: Removing manual scenarios 2011-12-05 23:13:34
 
calculate_start_of_week = function(week, year) {
  date <- ymd(paste(year, 1, 1, sep="-"))
  week(date) = week
  return(date)
}
 
tidy_df  = df %>% 
  mutate(year = year(date), 
         week = week(date),
         week_in_month = ceiling(day(date) / 7),
         max_week = max(week_in_month), 
         weeks_from_end = max_week - week_in_month,
         start_of_week = calculate_start_of_week(week, year))
 
> tidy_df %>% select(date, weeks_from_end, start_of_week) %>% sample_n(5)
 
                    date weeks_from_end start_of_week
1023 2008-08-08 21:16:02              3    2008-08-05
800  2014-01-31 06:51:06              0    2014-01-29
859  2014-08-14 10:24:52              3    2014-08-13
107  2010-07-10 22:49:52              3    2010-07-09
386  2011-12-20 23:57:51              2    2011-12-17

Next I want to get a count of how many posts were published in a given week. The following code does that transformation for us:

weeks_from_end_counts =  tidy_df %>%
  group_by(start_of_week, weeks_from_end) %>%
  summarise(count = n())
 
> weeks_from_end_counts
Source: local data frame [540 x 4]
Groups: start_of_week, weeks_from_end
 
   start_of_week weeks_from_end year count
1     2006-08-27              0 2006     1
2     2006-08-27              4 2006     3
3     2006-09-03              4 2006     1
4     2008-02-05              3 2008     2
5     2008-02-12              3 2008     2
6     2008-07-15              2 2008     1
7     2008-07-22              1 2008     1
8     2008-08-05              3 2008     8
9     2008-08-12              2 2008     5
10    2008-08-12              3 2008     9
..           ...            ...  ...   ...

We group by both ‘start_of_week’ and ‘weeks_from_end’ because we could have posts published in the same week but different month and we want to capture that difference. Now we can run a correlation on the data frame to see if there’s any relationship between ‘count’ and ‘weeks_from_end':

> cor(weeks_from_end_counts %>% ungroup() %>% select(weeks_from_end, count))
               weeks_from_end       count
weeks_from_end     1.00000000 -0.08253569
count             -0.08253569  1.00000000

This suggests there’s a slight negative correlation between the two variables i.e. ‘count’ decreases as ‘weeks_from_end’ increases. Let’s plug the data frame into a linear model to see how good ‘weeks_from_end’ is as a predictor of ‘count':

> fit = lm(count ~ weeks_from_end, weeks_from_end_counts)
 
> summary(fit)
 
Call:
lm(formula = count ~ weeks_from_end, data = weeks_from_end_counts)
 
Residuals:
    Min      1Q  Median      3Q     Max 
-2.0000 -1.5758 -0.5758  1.1060  8.0000 
 
Coefficients:
               Estimate Std. Error t value Pr(>|t|)    
(Intercept)     3.00000    0.13764  21.795   <2e-16 ***
weeks_from_end -0.10605    0.05521  -1.921   0.0553 .  
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
 
Residual standard error: 1.698 on 538 degrees of freedom
Multiple R-squared:  0.006812,	Adjusted R-squared:  0.004966 
F-statistic:  3.69 on 1 and 538 DF,  p-value: 0.05527

We see a similar result here. The effect of ‘weeks_from_end’ is worth 0.1 posts per week with a p value of 0.0553 so it’s on the border line of being significant.

We also have a very low ‘R squared’ value which suggests the ‘weeks_from_end’ isn’t explaining much of the variation in the data which makes sense given that we didn’t see much of a correlation.

If we charged on and wanted to predict the number of posts likely to be published in a given week we could run the predict function like this:

> predict(fit, data.frame(weeks_from_end=c(1,2,3,4,5)))
       1        2        3        4        5 
2.893952 2.787905 2.681859 2.575812 2.469766

Obviously it’s a bit flawed since we could plug in any numeric value we want, even ones that don’t make any sense, and it’d still come back with a prediction:

> predict(fit, data.frame(weeks_from_end=c(30 ,-10)))
        1         2 
-0.181394  4.060462

I think we’d probably protect against that with a function wrapping our call to predict that doesn’t allow ‘weeks_from_end’ to be greater than 5 or less than 0.

So far it looks like my belief is incorrect! I’m a bit dubious about my calculation of ‘weeks_from_end’ though – it’s not completely capturing what I want since in some months the last week only contains a couple of days.

Next I’m going to explore whether it makes any difference if I calculate that value by counting the number of days back from the last day of the month rather than using week number.

Categories: Programming

R: Filling in missing dates with 0s

Mark Needham - Sun, 07/12/2015 - 09:30

I wanted to plot a chart showing the number of blog posts published by month and started with the following code which makes use of zoo’s ‘as.yearmon’ function to add the appropriate column and grouping:

> library(zoo)
> library(dplyr)
> df %>% sample_n(5)
                                                  title                date
888        R: Converting a named vector to a data frame 2014-10-31 23:47:26
144  Rails: Populating a dropdown list using 'form_for' 2010-08-31 01:22:14
615                    Onboarding: Sketch the landscape 2013-02-15 07:36:06
28                        Javascript: The 'new' keyword 2010-03-06 15:16:02
1290                Coding Dojo #16: Reading SUnit code 2009-05-28 23:23:19
 
> posts_by_date  = df %>% mutate(year_mon = as.Date(as.yearmon(date))) %>% count(year_mon)
> posts_by_date %>% head(5)
 
    year_mon  n
1 2006-08-01  1
2 2006-09-01  4
3 2008-02-01  4
4 2008-07-01  2
5 2008-08-01 38

I then plugged the new data frame into ggplot to get the chart:

> ggplot(aes(x = year_mon, y = n), data = posts_by_date) + geom_line()

2015 07 12 09 07 47

The problem with this chart is that it’s showing there being 4 posts per month for all the dates between September 2006 and February 2008 even though I didn’t write anything! It’s doing the same thing between February 2008 and July 2008 too.

We can fix that by filling in the gaps with 0s.

First we’ll create a vector containing every month in the data range contained by our data frame:

> all_dates = seq(as.Date(as.yearmon(min(df$date))), as.Date(as.yearmon(max(df$date))), by="month")
 
> all_dates
  [1] "2006-08-01" "2006-09-01" "2006-10-01" "2006-11-01" "2006-12-01" "2007-01-01" "2007-02-01" "2007-03-01"
  [9] "2007-04-01" "2007-05-01" "2007-06-01" "2007-07-01" "2007-08-01" "2007-09-01" "2007-10-01" "2007-11-01"
 [17] "2007-12-01" "2008-01-01" "2008-02-01" "2008-03-01" "2008-04-01" "2008-05-01" "2008-06-01" "2008-07-01"
 [25] "2008-08-01" "2008-09-01" "2008-10-01" "2008-11-01" "2008-12-01" "2009-01-01" "2009-02-01" "2009-03-01"
 [33] "2009-04-01" "2009-05-01" "2009-06-01" "2009-07-01" "2009-08-01" "2009-09-01" "2009-10-01" "2009-11-01"
 [41] "2009-12-01" "2010-01-01" "2010-02-01" "2010-03-01" "2010-04-01" "2010-05-01" "2010-06-01" "2010-07-01"
 [49] "2010-08-01" "2010-09-01" "2010-10-01" "2010-11-01" "2010-12-01" "2011-01-01" "2011-02-01" "2011-03-01"
 [57] "2011-04-01" "2011-05-01" "2011-06-01" "2011-07-01" "2011-08-01" "2011-09-01" "2011-10-01" "2011-11-01"
 [65] "2011-12-01" "2012-01-01" "2012-02-01" "2012-03-01" "2012-04-01" "2012-05-01" "2012-06-01" "2012-07-01"
 [73] "2012-08-01" "2012-09-01" "2012-10-01" "2012-11-01" "2012-12-01" "2013-01-01" "2013-02-01" "2013-03-01"
 [81] "2013-04-01" "2013-05-01" "2013-06-01" "2013-07-01" "2013-08-01" "2013-09-01" "2013-10-01" "2013-11-01"
 [89] "2013-12-01" "2014-01-01" "2014-02-01" "2014-03-01" "2014-04-01" "2014-05-01" "2014-06-01" "2014-07-01"
 [97] "2014-08-01" "2014-09-01" "2014-10-01" "2014-11-01" "2014-12-01" "2015-01-01" "2015-02-01" "2015-03-01"
[105] "2015-04-01" "2015-05-01" "2015-06-01" "2015-07-01"

Now we need to create a data frame containing those dates and merge it with the original:

posts_by_date_clean = merge(data.frame(date = all_dates),
                            posts_by_date,
                            by.x='date',
                            by.y='year_mon',
                            all.x=T,
                            all.y=T)
 
> posts_by_date_clean %>% head()
        date  n
1 2006-08-01  1
2 2006-09-01  4
3 2006-10-01 NA
4 2006-11-01 NA
5 2006-12-01 NA
6 2007-01-01 NA

We’ve still got some ‘NA’ values in there which won’t plot so well. Let’s set those to 0 and then try and plot our chart again:

> posts_by_date_clean$n[is.na(posts_by_date_clean$n)] = 0
> ggplot(aes(x = date, y = n), data = posts_by_date_clean) + geom_line()
2015 07 12 09 17 10

Much better!

Categories: Programming

Re-Read Saturday: Mythical Man-month Part 2

The Mythical Man-Month

The Mythical Man-Month

This week we will break the rule of covering two essays per week (maybe it was a poor plan) and re-read the essay The Mythical Man-Month. The Mythical Man-Month is the essay that provides the book with its title, and more importantly establishes Brooks Law (we will explore this law in a few paragraphs). In my opinion, if Fred Brooks stopped with this essay his ideas would still would have shaped how work is (or at least should be) approached in IT organizations. The ideas in this essay are easy to trace in contemporary project management and in its influence in lean and Agile principles.

In the essay, The Mythical Man-Month, Brooks begins by postulating that much of the problems experienced in large software-centric projects are caused by a lack of calendar time. When calendar time is constrained, leaders and teams often make a number of mistakes. Those mistakes include compressing testing, not planning for things to go awry or adding people or teams to a project in an often vain attempt to make a date. Often these techniques generate the opposite impact, they make the project later or reduce quality. In the essay, Brooks identifies problems that the lack of calendar time can generate in projects. He explores four of those five in detail (progress tracking he tackled in a later essay). The four addressed in this essay are:

  1. Optimism generates behavior that reduces the effectiveness of estimation (this is true for all types of estimates). One of the core assumptions that optimism generates is that everything will go well. In my interview on the Software Process and Measurement Cast 84, Dr. Ricardo Valardi indicated that his research strongly supports this thesis. Software developers are trained problem solvers that haven’t met a problem they can’t surmount, therefore they fail to anticipate that problems will occur. And when problems do occur they will require more effort than planned.
    Brooks points out that software development bears similarities to other creative endeavors. The developer conceives of the solution to any tasks in a perfect state. As the task is implemented all of the constraints that did not exist in the perfect state come home to roost, thereby slowing progress. When the solution is exposed to interaction with others (such as stakeholders) even more flaws are exposed, requiring additional effort and calendar time to correct. Even from this small piece of the larger essay we can see the rationale for many of the principles in the Agile Manifesto. For example, the principle calling for teams to work with business on a day-to-day basis is a direct reflection of the need to move interaction from the end of the process to the beginning.
    Brooks completes his discussion of the impact of optimism by pointing out that while the probability of a problem with any individual task might be low, large projects are composed of many tasks that happen one after each other. Therefore the overall probability of a problem is always significant.
  2. Effort is often confused with progress, which stems from using a man-month as a unit of project size (it is amusing to reflect on how archaic the term man-month has become in 2015). Even today I am often ask how large a project is only to be told how many hours or sprints are estimated. Conflating effort and size leads to the assumption that people and time are interchangeable. While this might be true for independent tasks that don’t require interaction with others, like mowing the lawn, it is untrue for creative tasks that require interaction between people or teams. It is also untrue for projects or tasks that can’t be pulled apart and done in parallel. Brooks uses the now classic analogy that nine women can’t have a baby in a month to drive home the point.
    All projects have a proper staffing level above which the project will take longer and below which adding people will increase throughput (remember the concept of bottlenecks from the re-read of The Goal). Staffing equilibrium is recognized in Agile and is reflected in the size of Scrum teams (5 -9 people) and Dunbar’s number for projects or programs. The staffing equilibrium is influenced by a wide range of factors that Brooks didn’t explore; we will discuss this in a future essay. Brooks’ rationale for why more people will require more time is relatively straight-forward. More people require more overhead, more communication and interaction to get work done. Unstated but inferred is that if you think you staffed the work correctly to begin with then adding extra people is a bad idea.
  3. Gutless estimating leads teams and projects to deliver before they are ready. Brooks suggests that because we are uncertain about any estimate we are often affected by the urgency of others. The lack of a track record using measured and demonstrable factors such as velocity, productivity, functional size or even technical makes it is difficult to defend any estimate without reverting to pure opinion. Acceding to the urgency of others or any other factor that demands the completion of work before it is ready will have negative consequences. Those consequences are often identified as technical debt that reduces that value of the functionality delivered by the software .
  4. When schedule slippage is recognized the natural tendency of most managers is to add people in order to attempt to recover. This almost always makes things worse. This observation lead to Brooks’ Law. Brooks’ Law states that adding manpower to a late project only makes it later. For some reason EVERYONE admits to understanding Brooks’ Law, however very few seem to be able to live within the Law’s constraints. Adding people to any project requires re-planning, training, coordination and more testing (more integration testing if nothing else), which means that any possible gain is usually lost. This often induces the impression that even more people are needed to get back on track. Brooks called this a regenerative cycle, adding people seems to generate the need to add even more people. The argument that is often made is that automation and newer tools have invalidated Brooks law. The argument misses the point, every added person is an additional communication and integration node in a network that needs to work together to deliver software. More nodes require more human communication, which tools and other automation might help, but they don’t replace human interaction. The slope of impact might (and I underscore might) be less steep, but it has not been erased. If you doubt the veracity of the comment, go find a late project and offer people directly involved with the work to inject new people into the mix and see what the reaction is.

The messages in the essay are timeless and have influenced nearly every theorist, methodologist or coach whether they are working with classic, lean or Agile techniques. Now we have to work on getting all practitioners to follow through on using the ideas on projects rather than assuming what didn’t work before will somehow work this time.


Categories: Process Management

Skilled for Life

A while back, a colleague challenged me to find something simple and sticky for the big idea behind Sources of Insight.  After trying several phrases, here’s the one that stuck:

Skilled for Life

He liked it because it had punch.  It also had a play on words, and you could read it two different ways.

I like it because it captured the big idea behind Sources of Insight.   The whole purpose behind the site is to help as many people improve the quality of their life as possible.

With skill.

I’ve found that skills can make or break somebody’s chance for success.   And, I don’t just mean from a career perspective.   To be effective in all areas of our life, we need skills across several domains:

  • Mind
  • Body
  • Emotions
  • Career
  • Finance
  • Relationships
  • Fun

Skilled for Life is meant to be a very simple phrase, with a very intentional outcome:

Equip you with the skills you need to survive and thrive in today’s world.

It’s all about personal empowerment.

Not everybody gets the right mentors, or the right training, or the right breaks.   So Sources of Insight is designed from the ground up to be your personal success library that helps you make your own breaks, create your opportunities, and own your destiny.

How?

By sharing the world’s best insight and action for work and life.  By providing you with very real skills for mastering emotional intelligence, intellectual horsepower, creative brilliance, interpersonal relationships, career growth, health, and happiness (yeah, happiness is a skill you can learn).  And by providing you with principles, patterns, and practices for a smarter, more creative, and more capable you.

To give you one simple example of how happiness is a skill, let me tell you about the three paths of happiness according to Dr. Martin Seligman:

  1. The Pleasant Life
  2. The Good Life
  3. The Meaningful Life

You can think of them like this:  The Pleasant Life is all about pleasures, here and now.  The Good Life is about spending more time in your values.  The Meaningful Life is about fulfillment by helping the greater good, using your unique skills.   It’s giving our best where we have our best to give, and moving up Maslow’s stack.

When you know the three paths of happiness, you can more effectively build your happiness muscles.  For example, you can Discover Your Values, so that you can spend more time in them, and live life on your terms.

That’s just one example of how you can improve your self-efficacy with skill.

There is a vast success library of everything from inspirational quotes to inspirational heroes, as well as principles, patterns, and practices for skills to pay the bills and lead a better life.  Sources of Insight is a dojo of personal development, and your jump start for realizing your potential.

I invite you to check out the following page on Sources of Insight, where I share what Skilled for Life is all about:

Skilled for Life

Skills empower you.

Categories: Architecture, Programming

R: Date for given week/year

Mark Needham - Fri, 07/10/2015 - 23:01

As I mentioned in my last couple of blog posts I’ve been looking at the data behind this blog and I wanted to plot a chart showing the number of posts per week since the blog started.

I started out with a data frame with posts and publication date:

> library(dplyr)
> df = read.csv("posts.csv")
> df$date = ymd_hms(df$date)
 
> df %>% sample_n(10)
                                                                                title                date
538                                    Nygard Big Data Model: The Investigation Stage 2012-10-10 00:00:36
341                                                            The read-only database 2011-08-29 23:32:26
1112                                  CSS in Internet Explorer - Some lessons learned 2008-10-31 15:24:51
143                                                       Coding: Mutating parameters 2010-08-26 07:47:23
433  Scala: Counting number of inversions (via merge sort) for an unsorted collection 2012-03-20 06:53:18
618                                    neo4j/cypher: SQL style GROUP BY functionality 2013-02-17 21:05:27
1111                                 Testing Hibernate mappings: Setting up test data 2008-10-30 13:24:14
462                                       neo4j: What question do you want to answer? 2012-05-05 13:20:41
1399                                       Book Club: Design Sense (Michael Feathers) 2009-09-29 14:42:29
494                                    Bash Shell: Reusing parts of previous commands 2012-07-05 23:42:35

The first step was to add a couple of columns representing the week and year for the publication date. The ‘lubridate’ library came in handy here:

byWeek = df %>% 
  mutate(year = year(date), week = week(date)) %>% 
  group_by(week, year) %>% summarise(n = n()) %>% 
  ungroup() %>% arrange(desc(n))
 
> byWeek
Source: local data frame [352 x 3]
 
   week year  n
1    33 2008 14
2    35 2008 11
3    53 2012 11
4     9 2013 10
5    12 2013  9
6    21 2009  9
7    22 2009  9
8    38 2013  9
9    40 2008  9
10   48 2012  9
..  ...  ... ..

The next step is to calculate the start date of each of those weeks so that we can plot the counts on a continuous date scale. I spent a while searching how to do this before realising that the ‘week’ function I used before can set the week for a given data as well. Let’s get to work:

calculate_start_of_week = function(week, year) {
  date <- ymd(paste(year, 1, 1, sep="-"))
  week(date) = week
  return(date)
}
 
> calculate_start_of_week(c(1,2,3), c(2015,2014,2013))
[1] "2015-01-01 UTC" "2014-01-08 UTC" "2013-01-15 UTC"

And now let’s transform our data frame and plot the counts:

ggplot(aes(x=start_of_week, y=n, group=1), 
       data = byWeek %>% mutate(start_of_week = calculate_start_of_week(week, year))) + 
  geom_line()

2015 07 10 22 43 54

It’s a bit erratic as you can see. Some of this can be explained by the fact that I do in fact post in an erratic way while some of it is explained by the fact that some weeks only have a few days if they start on the 29th onwards.

Categories: Programming

Applying the Right Ideas to the Wrong Problem

Herding Cats - Glen Alleman - Fri, 07/10/2015 - 20:58

Sistine1In the project management business we have numerous methods, process, tools, and techniques to plan and execute project. When we search for tools, processes, and practices. 

Project success starts with a simple principle. We have to Know What Done looks like before we start.

In  Michelangelo's painting to the left, the two fingers are not touching. In the paradigm of a deity this may be sufficient to complete the job. In our our mortal world this is a nice example of almost done

It's more than common that we are stuck at 90% complete for a long time after the planned completion date. There are several independent variables here that are the sources of this problem. 

  • The estimated and then planned completion date and the associated budget are incorrect - because they are not credible. So arriving late and over budget is the outcome, since the confident in the actual data and budget was low on day one. We're steering to the wring target.
  • The execution of the credible Performance Measurement Baseline is flawed. We can't stay on schedule, budget, or technical performance targets. Why lots of the reasons, but the plan is good but we can't execute.
  • The plans and execution churn too much. The target is moving.

So What's a Project Manager to Do?

Here are six steps to creating a credible picture of what done looks like and execution to that understanding.

  1. Define the work in terms of products or services that must be delivered to implement the needed capabilities from the project.
  2. Build a strategy showing how each of the deliverables will be produced and how these deliverables will increase in their maturity as time passes.
  3. Identify the reducible risks for each of the deliverables and the work needed to reduce this risk and the measures of this risk reduction to assure they are handled.
  4. Build plan of the order of the work that provides the increasing maturity according to the strategy.
  5. Identify the irreducible risks and margins needed to protect the project from the unfavorable variances that arise. This means schedule margin, cost margin, and technical margin.
  6. Baseline this information and apply change controls so variances can be used to produce corrective actions.

These six steps can be applied to any project management or product development approach from agile to formal DOD acquisition. 

The key here is to connect the Programmatic performance (cost and schedule) with the Technical performance of the project, measure the variances of actuals to plan and take corrective actions to get back on plan or better yet stay on plan. 

 

Categories: Project Management

Cast Remote Display API: Processing

Google Code Blog - Fri, 07/10/2015 - 18:08

Posted by Leon Nicholls, Developer Programs Engineer

Remote Display on Google Cast allows your app to display both on your mobile and Cast device at the same time. Processing is a programming language that allows artists and hobbyists to create advanced graphics and interactive exhibitions. By putting these two things together we were able to quickly create stunning visual art and display it on the big screen just by bringing our phone to the party or gallery. This article describes how we added support for the Google Cast Remote Display APIs to Processing for Android and how you can too.

An example app from the popular Processing toxiclibs library on Cast. Download the code and run it on your own Chromecast! A little background

Processing has its own IDE and has many contributed libraries that hide the technical details of various input, output and rendering technologies. Users of Processing with just basic programming skills can create complicated graphical scenes and visualizations.

To write a program in the Processing IDE you create a “sketch” which involves adding code to life-cycle callbacks that initialize and draw the scene. You can run the sketch as a Java program on your desktop. You can also enable support for Processing for Android and then run the same sketch as an app on your Android mobile device. It also supports touch events and sensor data to interact with the generated apps.

Instead of just viewing the graphics on the small screen of the Android device, we can do better by projecting the graphics on a TV screen. Google Cast Remote Display APIs makes it easy to bring graphically intensive apps to Google Cast receivers by using the GPUs, CPUs and sensors available on the mobile devices you already have.

How we did it

Adding support for Remote Display involved modifying the Processing for Android Mode source code. To compile the Android Mode you first need to compile the source code of the Processing IDE. We started with the source code of the current stable release version 2.2.1 of the Processing IDE and compiled it using its Ant build script (detailed instructions are included along with the code download). We then downloaded the Android SDK and source code for the Android Mode 0232. After some minor changes to its build config to support the latest Android SDK version, we used Ant to build the Android Mode zip file. The zip file was unzipped into the Processing IDE modes directory.

We then used the IDE to open one of the Processing example sketches and exported it as an Android project. In the generated project we replaced the processing-core.jar library with the source code for Android Mode. We also added a Gradle build config to the project and then imported the project into Android Studio.

The main Activity for a Processing app is a descendent of the Android Mode PApplet class. The PApplet class uses a GLSurfaceView for rendering 2D and 3D graphics. We needed to change the code to use that same GLSurfaceView for the Remote Display API.

It is a requirement in the Google Cast Design Checklist for the Cast button to be visible on all screens. We changed PApplet to be an ActionBarActivity so that we can show the Cast button in the action bar. The Cast button was added by using a MediaRouteActionProvider. To only list Google Cast devices that support Remote Display, we used a MediaRouteSelector with an App ID we obtained from the Google Cast SDK Developer Console for a Remote Display Receiver.

Next, we created a class called PresentationService that extends CastRemoteDisplayLocalService. The service allows the app to keep the remote display running even when it goes into the background. The service requires a CastPresentation instance for displaying its content. The CastPresentation instance uses the GLSurfaceView from the PApplet class for its content view. However, setting the CastPresentation content view requires some changes to PApplet so that the GLSurfaceView isn’t initialized in its onCreate, but waits until the service onRemoteDisplaySessionStarted callback is invoked.

When the user selects a Cast device in the Cast button menu and the MediaRouter onRouteSelected event is called, the service is started with CastRemoteDisplayLocalService.startService. When the user disconnects from a Cast device using the Cast button, MediaRouter onRouteUnselected event is called and the service is stopped by using CastRemoteDisplayLocalService.stopService.

For the mobile display, we display an image bitmap and forward the PApplet touch events to the existing surfaceTouchEvent method. When you run the Android app, you can use touch gestures on the display of the mobile device to control the interaction on the TV. Take a look at this video of some of the Processing apps running on a Chromecast.

Most of the new code is contained in the PresentationService and RemoteDisplayHelper classes. Your mobile device needs to have at least Android KitKat and Google Play services version 7.5.71.

You can too

Now you can try the Remote Display APIs in your Processing apps. Instead of changing the generated code every time you export your Android Mode project, we recommend that you use our project as a base and simply copy your generated Android code and libraries to our project. Then simply modify the project build file and update the manifest to start the app with your sketch’s main Activity.

To see a more detailed description on how to use the Remote Display APIs, read our developer documentation. We are eager to see what Processing artists can do with this code in their projects.

Categories: Programming

Stuff The Internet Says On Scalability For July 10th, 2015

Hey, it's HighScalability time:


Spying on an ant holding its favorite microchip. (@SciencePorn)

 

  • 1,425%: malware attack ROI; 33333/sec: BP oil well datapoints generated; 8 million: mumified dogs; 5 billion: Apple map requests per week; 10 billion: parameter neural net in your basement; 1 trillion: records in Yahoo's Sherpa.
  • Quotable Quotes:
    • @warriors: "It's ironic but what the unexpected thing was that everything went exactly as we hoped. That never happens." 
    • @georgeblazer: At scale, architecture dominates material. Alan Kay #gophercon
    • Nassim Nicholas Taleb: How unpredictability rises faster and faster with dimensionality: add one variable to 100 and error doubles.
    • Elon Musk~ one of the biggest challenges in the CRS-7 event is matching events to the exact time. When you are talking a matter of milliseconds it's hard to match the ground track video to the sensors
    • @MeltingIce: NYSE halted, United grounds flights due to computer glitch, WSJ website is down. This is how it starts.
    • The Shut-In Economy: They end up asking each other which apps they work for: Postmates. Seamless. EAT24. GrubHub. Safeway.com
    • @aphyr: At a Devops Chicago meetup I asked how many people had experienced partitions in their datacenters, and over half the room raised hands.
    • @mjpt777: Simplifying code rocks. @toddlmontgomery and I are seeing new throughput highs on Aeron after simplifying and cutting indirection.
    • aphyr: Real software is fuzzier: our processes are usually not realtime, which means the network effectively extends within the node. Garbage collection, in particular, is a notorious cause of “network” partitions, because it delays messages.
    • @fgens: 2 key msgs at #AWSSummit : "Developers, we love you (IaaS is so yesterday)!" and "Go 'server-less' -- deploy code as Lambda microservices"
    • @BenedictEvans: @Jim_Edwards but the boom is caused by having 3bn people online and a generational change in the tech platform from PC to mobile.
    • @Obdurodon: "Our distributed file system doesn't work quite right and performs like crap." "OK, we'll just call it an object store then."
    • @viktorklang: Suspected main causes for slow programs: A) Doing unnecessary work B) Waiting for unnecessary work to be done
    • @jasongorman: There are only so many times one can re-learn the same patterns for remote procedure calls before one gets mightily sick of it
    • @BenedictEvans: Devices in use, end of 2014:  ~1.5bn PCs 7-800m consumer PCs 1.2-1.3bn closed Android 4-500m open Android 650-675m iOS 80m Macs, ~75m Linux.
    • vardump: When performance matters, don't chase pointers (references) and don't ever do random memory accesses if you can avoid it. Ensure variables needed about same time are near each other, so that a single cache line fill will get all. Except when other threads/cores frequently update a variable, try to keep those separately to reduce false sharing.
    • @JZdziarski: Quantum entanglement dictates if one programmer is writing productive good code, another somewhere is inexplicably porting to JavaScript.
    • There are even more quotes for your perusal, please click through to the full article.

  • Love this! jamiesonbecker: AMAZON DEPRECATES EC2 November 3, 2017, SEATTLE At the AWS Re:Invent in Las Vegas today, Amazon Web Services today announced the deprecation of Elastic Compute Cloud as it shifts toward lighter-weight, more horizontally scalable services. Amazon announced that it was giving customers the opportunity to migrate toward what it claims are lower cost "containers" and "Lambda processes".

  • Horace Dediu with an interesting view of Humanism++. This time it's preferring humans over algorithms instead of humans over faith. Curate, don't automate...or sermonize. Also interesting was the discussion on functional vs divisional organizations. Apple is the largest functional org outside of the army. Functional orgs prevent cross divisional fights for resources and power.

  • OK, this is funny: Using locks to solve your concurrency problems.

  • Do we have a precedence for the rise of walled gardens? 1492: The Year the World Began: shift of initiative—the upset in the normal state of the world—started in 1492, when the resources of the Americas began to be accessible to Westerners while remaining beyond the reach of other rival or potentially rival civilizations.

  • Here's how the new StackExchange blog was rebuilt. Pretty radical. They got rid of WordPress for a static blog built on Jekyll. While there was some contention with the move on HackerNewsjonhmchan set the record straight: Performance also wasn't the only plus here. Closing major security holes, making more of our content and technology more open, and moving to a platform that our devs liked working in are just some of the other wins. It's too early to say definitively now, but we think the change is probably a good one.

  • Should Uber be the new McDonalds? While I viscerally agree with the following statement, the idea that every place in the world must have the same laws is equally obviously wrong. Vive la difference! @paulg: Uber is so obviously a good thing that you can measure how corrupt cities are by how hard they try to suppress it.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

The App Developer Business Kit: Now available in 10 languages

Android Developers Blog - Fri, 07/10/2015 - 07:05

Posted by Sean Meng, a Product Marketing Manager on the AdMob team

Today we’re excited to launch The App Developer Business Kit in 10 more languages. The website includes tips for new app developers on building, promoting and monetizing your app. Check out the Business Kit in your language:

To help you make decisions about growing your app business in other regions, we’ve added 6 new market reports providing great insights about app users in Italy, Spain, Germany, Brazil, France, and Russia. Did you know that Brazilian smartphone users engage with ads more frequently than users in the US and Japan? Or that while nearly 2/3rds of French users exclusively download free apps, only 31% of Brazilian smartphone users do? Check out statistics like these about exciting regions around the world here.

Stay connected on all things mobile apps by following us on Google+ and Twitter.

Categories: Programming

Scaling Agile: Scrum of Scrums – Anti-patterns Part 2

10358869_10203282385195091_1761744082221812386_n

Different sort of pyramid syndrome

A Scrum of Scrums (SoS) is a mechanism to coordinate a group of teams so that they act as a team of teams. Powerful tools often have side effects that, if not countered, can do more harm than good. There are several “anti-patterns” that organizations adopt that negatively impact the value a SoS can deliver. In Scaling Agile: Scrum of Scrums: Anti-patterns Part 1 we explored The Soapbox, The Blame Game and Surrogate Project Manager, which three typical anti-patterns. Two other common anti-patterns are the Three Bears and Pyramid syndromes.

The Three Bears Syndrome. In the children’s version of the classic fairy tale, Goldilocks broke into the three bear’s house and sampled three bowls of porridge. One bowl was too hot, one bowl was too cold and one was just right. Many organizations determine what the right cadence is for SoS meetings for the organization regardless of context. The problem is that context is really important. Much like the story of The Three Bears, having too many meetings steals time from getting work done, while too few SoS meetings can keep work from being accomplished by delaying issue resolution and decisions. When the number of meetings is just right, work flows through the process with minimal delay caused by the need to wait for coordination or decisions. The most common version of this anti-pattern is the required single daily SoS. Many times organizations rigorously require SoS meetings on a daily basis because they believe that what is good for the daily stand-up/Scrum meeting is good for the SoS. Daily sounds like a good cadence, however some projects, for example projects with a handful of teams whose work is only loosely coupled, might not need a daily SoS. Alternately, projects with a large number of very tightly coupled teams late in the development cycle might need multiple SoS meetings on a daily basis. Another variation of this anti-pattern is seen in organizations that reduce the SoS cadence (usually coupled with lengthening the duration the meetings when they do occur) for projects with distributed teams. In real life, this is the opposite of what is needed, the complexity of projects with distributed teams typically demands more coordination, not less.

One possible solution is to empower the SoS to regulate itself.

  • Allow the SoS to self-regulate its own cadence based on the need of the project. A coach should facilitate the SoS in negotiating a minimum floor for when they will meet. The minimum should then be reviewed periodically so that people attending the SoS can change the meeting cadence as the need for decisions and project risk wax and wane.

Pyramid Syndrome. One of the more exciting features of the Scrum of Scrums technique is the ability to scale up the meetings up like a pyramid or a hierarchy. For example, a REALLY big project might have 10 to 20 teams. 20 teams with 7 members (Scrum teams are typically 5 – 9 people) which would equate to approximately 140 team members. 140 overall team members is still below the common interpretation of Dunbar’s Number. SoS meetings should be limited to the same size as a typical Scrum team (5-9) with smaller groups typically being better to ensure quick coordination. A project with 20 teams using SoS meetings of 7 people would require 3 SoS meetings for the teams and a 4th with a representative from each of the team-level SoS meetings. In some cases the ability to create layers of SoS meetings allows organizations to believe they can create megaprojects with hundreds of team members. Megaprojects and programs leveraging normal SoS techniques would need many layers of SoS meetings. Each meeting takes time and requires shuttling information between teams (with potential fidelity loss). A few years ago I observed an organization in which the some SoS attendees lost several hours a day to SoS meetings. Projects requiring SoS meeting of three or more levels are too large. One possible solution is simply to split the project or program into smaller chunks.

    • Split projects or programs up so that the number of team members involved stays below the approximate limit suggested by Dunbar’s number (150 people).

The Three Bears and Pyramid Syndromes are two additional anti-patterns that can plague Scrum of Scrums. None of the five anti-patterns we have explored are insurmountable. The solution for problems with how SoS meetings are work generally first requires diagnosing the problem and then coaching to help replace bad behaviors with good behaviors.


Categories: Process Management

M Developer Preview Gets Its First Update

Android Developers Blog - Thu, 07/09/2015 - 19:31

By Jamal Eason, Product Manager, Android

Earlier this summer at Google I/O, we launched the M Developer Preview. The developer preview is an early access opportunity to test and optimize your apps for the next release of Android. Today we are releasing an update to the M Developer Preview that includes fixes and updates based on your feedback.

What’s New

The Developer Preview 2 update includes the up to date M release platform code, and near-final APIs for you to validate your app. To provide more testing support, we have refined the Nexus system images and emulator system images with the Android platform updates. In addition to platform updates, the system images also include Google Play services 7.6.

How to Get the Update

If you are already running the M developer preview launched at Google I/O (Build #MPZ44Q) on a supported Nexus device (e.g. Nexus 5, Nexus 6, Nexus 9, or Nexus Player), the update can be delivered to your device via an over-the-air update. We expect all devices currently on the developer preview to receive the update over the next few days. We also posted a new version of the preview system image on the developer preview website. (To view the preview website in a language other than English, select the appropriate language from the language selector at the bottom of the page).

For those developers using the emulator, you can update your M preview system images via the SDK Manager in Android Studio.

What are the Major Changes?

We have addressed many issues brought up during the first phase of the developer preview. Check out the release notes for a detailed list of changes in this update. Some of the highlights to the update include:

  • Android Platform Changes:
    • Modifications to platform permissions including external storage, Wi-Fi & Bluetooth location, and changes to contacts/identity permissions. Device connections through the USB port are now set to charge-only mode by default. To access the device, users must explicitly grant permission.
  • API Changes:
    • Updated Bluetooth Stylus APIs with updated callback events. View.onContextClickListener and GestureDetector.OnContextClickListener to listen for stylus button presses and to perform secondary actions.
    • Updated Media API with new callback InputDevice.hasMicrophone() method for determining if a device microphone exists.
  • Fixes for developer-reported issues:
    • TextInputLayout doesn't set hint for embedded EditText. (fixed issue)
    • Camera Permission issue with Legacy Apps (fixed issue)
Next Steps

With the final M release still on schedule for this fall, the platform features and API are near final. However, there is still time to report critical issues as you continue to test and validate your apps on the M Developer Preview. You can also visit our M Developer Preview community to share ideas and information.

Thanks again for your support. We look forward to seeing your apps that are ready to go for the M release this fall.

Join the discussion on

+Android Developers
Categories: Programming

Strategy is Not the Same as Operational Effectiveness

Herding Cats - Glen Alleman - Thu, 07/09/2015 - 16:57

It is common to confuse strategy with operational effectiveness. Strategy for Information Technology (IT) projects contains three major themes. These  form the foundation of the IT Strategy as well as the tactical processes that will be deployed in support of these strategies.

  • Business improvements are enabled by Information Technology that is integrated not disintegrated. This integration must be horizontal versus vertical. Horizontal systems remove islands of information and build bridges between the business units. In this integrated system, multiple data sources are made transparent to the end users as well as the applications that utilize them.
  • Information Technology requirements are always growing, changing, and being extended. The Information Technology is no longer static, but dynamic, adapting to the changing business requirements.
  • Information Technology is about abstractions. If hardware, software and data were the only foundations of a system, then Information Technology would not be able to keep pace with the business requirements. Instead, business processes, objects and services are the foundation of the system, which then drives the business process in their adaptation of the changing market forces.

What Is Strategy?

Strategy is creating fit among a company’s activities. The success of a strategy depends on doing many things well – not just a few. The things that are done well must operate within a close nit system. If there is no fit among the activities, there is no distinctive strategy and little to sustain the strategic deployment process. Management then reverts to the simpler task of overseeing independent functions. When this occurs operational effectiveness determines the relative performance of the organization. [1]

Improving operational effectiveness is a necessary part of management, but it is not strategy. In confusing the two, managers will be unintentionally backed into a way of thinking about the business environment that drives the business processes (IT) away from the strategic support and toward the tactical improvement of operational effectiveness.

Managers must be able to clearly distinguish operational effectiveness from strategy. Both are essential, but the two agendas are different. The operational effectiveness agenda involves continual improvement business processes that have no trade–offs associated with them. The operational effectiveness agenda is the proper place for constant change, flexibility, and relentless efforts to achieve best practices. In contrast, the strategic agenda is the place for making clear tradeoffs and tightening the fit between the participating business components. Strategy involves the continual search for ways to reinforce and extend the company’s position in the market place.

The concept of fit among functional units is one of the oldest ideas in strategy. Gradually however, it has been supplanted with new concepts of core competencies, critical resources and key success factors. In fact fit is far more critical to the success of the IT systems than realized. [3] Strategic fit among the various systems components and the business processes they support is fundamental not only to competitive advantage but also to the sustainability of that advantage.

Fit among a company’s activities creates pressures and incentives to improve operational effectiveness. Fit means that poor performance in one activity will degrade performance in others, so that weaknesses are exposed drawing management’s attention. Conversely, with increasing fit, improvements of one activity will pay dividends in other areas.

The challenge now is to create fit among the IT components and their matching business components.

Building A Strategy

To define our Vision, Strategic Objectives, Performance Goals, Critical Success Factors in achieving those, and the measures of effectiveness and performance in pursuit of those strategic goals and objectives, we need a method that collects all of these in a single place. 

If we are going to make tradeoffs in pursuit of strategy, we need to know what those tradeoffs are, how much will be the opportunity cost for each trade and how each trade impacts our strategic decision making.

To dive into the details, to make those opportunity cost tradeoffs about future outcomes in the presence of uncertainty we must of course ESTIMATEThere can be no execution of the strategy without make estimates of the benefits of the outcomes of the project that delivers the capabilities that implement the strategy.

The Balanced Scorecard presentation below shows how to build the strategy. Page 49 - 52 shows how to connect the dots between strategy and project execution, where the work is done, at or below the planned cost, on or before the needed time, and with the planned effectiveness and performance of the delivered capabilities. Showing up late, over budget, and with missing capabilities will not enable the strategy to fulfill it's mission and vision. It's a closed loop system - all parts must work in combination for success.

 

[1] “What is Strategy,” M. E. Porter, Harvard Business Review, Volume 74, Number 6, pp. 61–78.

Jack Welch Speaks: Wisdom from the World’s Greatest Business Leader, J. Welch and J. C. Lowe, John Wiley & Sons, 1998.

Control Your Destiny or Someone Else Will: Lessons in Mastering Change–From the Principles Jack Welch Used to Revolutionize GE, N. M. Tichy and S. Sherman, Harpers Business, 1994.

Related articles Information Technology Estimating Quality Capabilities Based Planning What Happened to our Basic Math Skills? Systems Thinking, System Engineering, and Systems Management Making Decisions In The Presence of Uncertainty Everything I Learned About PM Came From a Elementary School Teacher
Categories: Project Management

Are Estimates Really The Smell of Dysfunction?

Herding Cats - Glen Alleman - Thu, 07/09/2015 - 16:48

The #NoEstimates advocates have asked us to see estimates as a smell: an indication of possible decision making dysfunction. It might be useful to explore what's causing the smell.

In the normal business process world, when we encounter a dysfunction, Root Cause Analysis is an approach to discover the cause and effects of the dysfunction.

Since the Primary Effect is described as dysfunction but not stating what this dysfunction is, let's apply RCA in the form of the Apollo Method to the statement of the #NoEstimates advocates. 

But first some background on RCA and the Apollo Method.

What are some dysfunctions of software development management caused by the need to estimate?:
  • Inability to estimate when the project will complete, how much it will cost when complete, and what capabilities will be delivered at completion and along the way that meet the business plan.
  • Misuse of estimates.
  • Abuse of estimates.
  • Cost of estimates not appropriate for the Value at Risk.

It's been suggested that asking 5 Whys is a place to start. It is well understood that simply asking may be necessary but far from sufficient to discover the Root Cause of any dysfunction. Source of the problem with the 5 Whys approach starts with our natural story telling approach to problem solving.

Finding the source of any dysfunction is straightforward:

  1. For each Primary Effect - in this case Management Dysfunction - ask why
    • Just asking why is necessary but not sufficient
    • The answer needs to have a connect to the Primary Effect - the dysfunction
    • Estimates are hard - without a traceable connection to the dysfunction, suggesting  stopping estimates will remove the dysfunction is nonsense. Stopping them has no good reason, other that estimates are hard.
    • This is like asking your young child to eat their peas. I don't like themWhyBecause they taste bad, Why, because I don't like them.
  2. Look for Causes in actions and conditions
    • Find a condition from the Event. What has to be in place for the Event to take place?
    • Find an action for the Event. What was actually DONE to cause the primary event?
    • The only value of knowing if a cause is an action or a condition is that it tells us which 
  3. Connect all the causes with a caused by association
    • Connect all these in some form to see the causal linkage between the Action and Conditions and the Primary Event.
  4. Support each cause with evidence 
    • Here's the hard part and where the #NoEstimates advocate fall flat on their face. 
    • Show evidence that the existence of estimating and resulting estimates (good or bad) are the CAUSE of the dysfunction. 
    • Then and here's the really hard part in the RCA paradigm - show that the correction action will actually remove the Primary Effect.
    • Show that by not estimating the Management Dysfunction is removed.

None  is this is in palce for the #NoEstimates conjecture that estimates are the smell of dysfunction

So unless we have some understanding of the Dysfunction, conjecturing that estimates are the smell and the Not Estimating will remove the dysfunction has little chance of actual success.

No Estimates is a solution looking for a problem to solve. And to date that problem has not been identified, and most importantly the conjecture that Not Estimating fixes the problem has no tangible evidence to confirm it will fix the problem.

Related articles The Dysfunctional Approach to Using "5 Whys" Who's Budget is it Anyway? Mr. Franklin's Advice
Categories: Project Management

How Would You Start an Online Business?

Making the Complex Simple - John Sonmez - Thu, 07/09/2015 - 15:00

In this episode, I talk about how I would start an online business. Full transcript: John:               Hey, John Sonmez from simpleprogrammer.com. I’ve got a question here about starting an online business. This question is from Stig and Stig says, “Hello John, a new subscriber here. I study computer science with a hefty amount of focus […]

The post How Would You Start an Online Business? appeared first on Simple Programmer.

Categories: Programming

R: dplyr – Error: cannot modify grouping variable

Mark Needham - Thu, 07/09/2015 - 06:55

I’ve been doing some exploration of the posts made on this blog and I thought I’d start with answering a simple question – on which dates did I write the most posts?

I started with a data frame containing each post and the date it was published:

> library(dplyr)
> df %>% sample_n(5)
                                                title                date
1148 Taiichi Ohno's Workplace Management: Book Review 2008-12-08 14:14:48
158     Rails: Faking a delete method with 'form_for' 2010-09-20 18:52:15
331           Retrospectives: The 4 L's Retrospective 2011-07-25 21:00:30
1035       msbuild - Use OutputPath instead of OutDir 2008-08-14 18:54:03
1181                The danger of commenting out code 2009-01-17 06:02:33

To find the most popular days for blog posts we can write the following aggregation function:

> df %>% mutate(day = as.Date(date)) %>% count(day) %>% arrange(desc(n))
 
Source: local data frame [1,140 x 2]
 
          day n
1  2012-12-31 6
2  2014-05-31 6
3  2008-08-08 5
4  2013-01-27 5
5  2009-08-24 4
6  2012-06-24 4
7  2012-09-30 4
8  2012-10-27 4
9  2012-11-24 4
10 2013-02-28 4

So we can see a couple of days with 6 posts, a couple with 5 posts, a few more with 4 posts and then presumably loads of days with 1 post.

I thought it’d be cool if we could blog a histogram which had on the x axis the number of posts and on the y axis how many days that number of posts occurred e.g. for an x value of 6 (posts) we’d have a y value of 2 (occurrences).

My initial attempt was this:

> df %>% mutate(day = as.Date(date)) %>% count(day) %>% count(n)
Error: cannot modify grouping variable

Unfortunately that isn’t allowed. I tried ungrouping and then counting again:

 df %>% mutate(day = as.Date(date)) %>% count(day) %>% ungroup() %>% count(n)
Error: cannot modify grouping variable

Still no luck. I did a bit of googlign around and came across a post which suggested using a combination of group_by + mutate or group_by + summarize.

I tried the mutate approach first:

> df %>% mutate(day = as.Date(date)) %>% 
+     group_by(day) %>% mutate(n = n()) %>% ungroup() %>% sample_n(5)
                                                        title                Source: local data frame [5 x 4]
 
                                    title                date        day n
1 QCon London 2009: DDD & BDD - Dan North 2009-03-13 15:28:04 2009-03-13 2
2        Onboarding: Sketch the landscape 2013-02-15 07:36:06 2013-02-15 1
3                           Ego Depletion 2013-06-04 23:16:29 2013-06-04 1
4                 Clean Code: Book Review 2008-09-15 09:52:33 2008-09-15 1
5            Dreyfus Model: More thoughts 2009-08-10 10:36:51 2009-08-10 1

That keeps around the ‘title’ which is a bit annoying. We can get rid of it using a distinct on ‘day’ if we want and if we also implement the second part of the function we end up with the following:

> df %>% mutate(day = as.Date(date)) %>% 
    group_by(day) %>% mutate(n = n()) %>% distinct(day) %>% ungroup() %>% 
    group_by(n) %>%
    mutate(c = n()) %>%
    distinct(n)  
 
Source: local data frame [6 x 5]
Groups: n
 
                                                title                date        day n   c
1       Functional C#: Writing a 'partition' function 2010-02-01 23:34:02 2010-02-01 1 852
2                            Willed vs Forced designs 2010-02-08 22:48:05 2010-02-08 2 235
3                            TDD: Testing collections 2010-07-28 06:05:25 2010-07-28 3  41
4  Creating a Samba share between Ubuntu and Mac OS X 2012-06-24 00:40:35 2012-06-24 4   8
5            Gamification and Software: Some thoughts 2012-12-31 10:57:19 2012-12-31 6   2
6 Python/numpy: Selecting specific column in 2D array 2013-01-27 02:10:10 2013-01-27 5   2

Annoyingly we’ve still got the ‘title’, ‘date’ and ‘day’ columns hanging around which we’d need to get rid of with a call to ‘select’. The code also feels quite icky, especially the use of distinct in a couple of places.

In fact we can simplify the code if we use summarize instead of mutate:

> df %>% mutate(day = as.Date(date)) %>% 
    group_by(day) %>% summarize(n = n()) %>% ungroup() %>% 
    group_by(n) %>% summarize(c = n())
 
 
Source: local data frame [6 x 2]
 
  n   c
1 1 852
2 2 235
3 3  41
4 4   8
5 5   2
6 6   2

And we’ve got also rid of the extra columns in the bargain which is great! And now we can plot our histogram:

> library(ggplot2)
> post_frequencies = df %>% mutate(day = as.Date(date)) %>% 
    group_by(day) %>% summarize(n = n()) %>% ungroup() %>% 
    group_by(n) %>% summarize(c = n())
> ggplot(aes(x = n, y = c), data = post_frequencies) + geom_bar(stat = "identity")

2015 07 09 06 44 47

In this case we don’t actually need to do the second grouping to create the bar chart since ggplot will do it for us if we feed it the following data:

. ggplot(aes(x = n), 
         data = df %>% mutate(day = as.Date(date)) %>% group_by(day) %>% summarize(n = n()) %>% ungroup()) +
    geom_bar(binwidth = 1) +
    scale_x_continuous(limits=c(1, 6))
2015 07 09 06 55 12

Still, it’s good to know how!

Categories: Programming

Zephyr Mobile App Press Release

Making the Complex Simple - John Sonmez - Wed, 07/08/2015 - 21:59

I received an email from Zephyr about their new mobile app and I want to share the news to all of you. I’m posting this because this can benefit the software development and QA community. Here are some of the reasons why the app can make testing easy. One, you’ll be able to easily execute […]

The post Zephyr Mobile App Press Release appeared first on Simple Programmer.

Categories: Programming

RebornDB: the Next Generation Distributed Key-Value Store

There are many key-value stores in the world and they are widely used in many systems. E.g, we can use a Memcached to store a MySQL query result for later same query, use MongoDB to store documents for better searching, etc.

For different scenarios, we should choose different key-value store. There is no silver-bullet key-value store for all solutions. But if you just want a simple key-value store, easy to use, very fast, supporting many powerful data structures, redis may be a good choice for your start.  

Redis is advanced key-value cache and store, under BSD license. It is very fast, has many data types(String, Hash, List, Set, Sorted Set …), uses RDB or AOF persistence and replication to guarantee data security, and supplies many language client libraries.

Most of all, market chooses Redis. There are many companies using Redis and it has proved its worth.

Although redis is great, it still has some disadvantages, and the biggest one is memory limitation.  Redis keeps all data in memory, which limits the whole dataset size and lets us save more data impossibly.

The official redis cluster solves this by splitting data into many redis servers, but it has not been proven in many practical environments yet. At the same time, it need us to change our client libraries to support “MOVED” redirection and other special commands, this is unacceptable in running production too. So redis cluster is not a good solution now.

QDB

We like redis, and want to go beyond its limitation, so we building a service named QDB, which is compatible with redis, saves data in disk to exceed memory limitation and keeps hot data in memory for performance.

Introduction

QDB is a redis like, fast key-value store.It has below good features:

Categories: Architecture