Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Merry Christmas, 2014

Merry Christmas 2014

Merry Christmas 2014

Eric Sevareid, a CBS news journalist from 1939 to 1977, said, ‚ÄúChristmas is a necessity. There has to be at least one day of the year to remind us that we’re here for something else besides ourselves.‚ÄĚ Whether in your day job you gather requirements, write or test code, facilitate teams or lead change Christmas represents a time to reflect on the larger world, reflect and make it better for someone other than yourself. NPR‚Äôs Morning Edition on December 24 included a segment discussing research that suggests that generosity is a hardwired human attribute.¬† The research by Dr.¬†Lara Aknin¬†is an Assistant Professor of Psychology at Simon Fraser University suggests that people feel better about themselves and the world around them when they are generous to others. Christmas is a day of celebration but also a reminder that to feel good about ourselves we need to give of ourselves to others.

Regardless of whether you celebrate Christmas, take the few minutes you would spend reading this blog and reach out to a family member you have lost track of, a co-worker that might be alone on this day or to a friend you have only talked to only on Facebook and say hello. Whether, like Fred in the Dicken’s Christmas Carol, invite your Uncle Scrooge to dinner or just spend a few minutes on the phone reaching out to another fellow human.

Categories: Process Management

Blame Yourself First

Making the Complex Simple - John Sonmez - Thu, 12/25/2014 - 16:00

In this video, I talk about why it is important to take responsibility for things that happen to you instead of blaming others and circumstances. When you choose to blame others and “bad luck” or make excuses for yourself, you give your power away to make a positive change in your situation.

The post Blame Yourself First appeared first on Simple Programmer.

Categories: Programming

Five Problems That Impact Testing Effectiveness and Efficiency: #3 Organizational Management

Expectations and environment affect behavior.

Expectations and environment affect behavior.

When I was a manager of QA (testing), my manager more than once shared the expectation that testing should have found all of the defects in a piece of software before it was put into production. I actually think that expectation found its way into my annual objectives just prior to my changing jobs. Not only is it not possible to find all defects, the responsibility for defect removal must be shared by developers and testers alike. Organizational management can impact test efficiency and effectiveness in many ways, however three are fairly common.

  1. Unrealistic expectations ‚Äď One of the principles of testing is that exhaustive testing is impossible. ¬†The over-emphasis of removing all defects or perhaps slightly less dangerous – all significant defects – takes the focus away reducing risks. Testers and test managers must use testing and other techniques (reviews, for example) to focus the available time and effort they have on what is important and risky. Developing an understanding of potential impact and possibility of problems (risk) is needed to target testing resources. That is impossible when trying to find all possible problems.
  2. Undervaluing /Underfunding testing ‚Äď One the indicators of how an organization values test (and testers) is funding. Funding includes people, tools and other resources. Organizations can have enough testers and but not the proper logistics to test effectively or efficiently.¬† Another common indication that testing is considered a commodity or not a core function is when an organization outsources testing.
  3. Failure to pursue root causes of defects ‚Äď Testing shows that defects exist (another principle of testing). Testing does not create defects; said another way defects come from somewhere else. The only way to stem the tide of defects is to change how the software is developed by improving the development process (inclusive of people, process and tools) by finding the root cause of defects and then improving the process. If process improvement does not include the development process the rate of defects production will not change. Ignoring the lessons from defects found in testing dooms the organization to repeating the mistakes that led to those defects.

Organizational management can have a direct impact on the effectiveness and efficiency of testing.  The actions that are taken are very rarely because they want buggy software, but rather because they are unsure of the role, scope and goal of testing in the development and maintenance of software.  As a general rule, the role of testing whether through reviews requirements, design and code or by executing cases is to find defects. Software is complex enough that exhaustive testing is either not possible or not cost effective, therefore the goal of testing is to reduce the risk to the users of the software (and to the organization) that defects that will be delivered. The organization has the responsibility to provide the resources and tools needed to meet the goals and strategy of the testing that is needed to meet the organization’s goals.

Categories: Process Management

Quote of the Day - Do It Right the First Time (Update)

Herding Cats - Glen Alleman - Wed, 12/24/2014 - 19:12

Screen Shot 2014-12-23 at 1.17.48 PM

This quote has two interpretations:

  • New ideas from beginners provide many more opportunities for good results than from experts.
  • Experts understand the consequences of untested, unproven, ill conceived¬†ideas on the probability of success for mission critical, high¬†value at risk projects.

I want to thank Tom Peters @tom_peters for stimulating this clarity.

In parallel with this notion of novices know better and experts have limited possibilities is the conjecture that: 

Do it Right the First Time is the dumbest statement ever uttered by a human being

In the absence of a domain and context in that domain, that statement is of little value - a nice platitude with no actionable outcomes.

Here's an example from personal experience. 

Mars Science Laboratory represents the first use of a "soft landing" technique called the Sky Crane maneuver. The sheer mass of Mars Science Laboratory prevented engineers from using the familiar airbags to deliver their rover safely to the martian surface. As rovers become more capable and carry more instruments, they become larger. So, in order to accommodate this advanced mission, engineers designed a sky-crane method that will lower the rover to the surface.

This mission was built and tested by the same people who built and flew most of the other Mars missions, right here in town. The decent shield, engines and other processes had to work the first time as a system and could not be tested as a system, since the final testing could only take place on Mars.

"Courisity relies on untried Sky Crane for Mars descent" 

So in this case (domain) it has to work the first time, or the mission is lost.

The consequences failing for the first time to reach your audience and gets a round of applause at a conference on a new topic may mean not being invited back.

So when we read all those posts about agreeing with the do it right is the worst things that ever happened, ask if those posters have ever worked on a mission critical must work project where billions are at risk? Maybe lives aren't a risk on Mars lander, but Nuke Power start ups, weapons plants cleanups with loose plutonium laying on the floor, off shore unattended gas platforms, ABM intercept missiles, and the like are actually in the domain of do it right the first time.

Those who use the terms fail often may not have experienced these high value at risk conditions, where failure is actually not an option. Lovell's comment to his ground partner during their joint talk at a PMI conference was Gene says Failure was not an option. I'd have to say Success was our only option.

Gene Kranz Glen Kevin and Lovell

Succeeding on the first try is mandated in many domains. Doesn't mean you haven't prepared in depth for that first try. But it is naive at best to think that failure is to be tolerated in the absence of a domain and context that assesses the consequences of that failure.

Those who suggest  fail often and you'll learn more in the absence of a domain, context, and Value at Risk are just repeating a platitude in the absence of witnessing the BBR aftermath of failure. 

Our past experiences informs our current world view, mine is from mission critical - literally  do or die - experiences. So I'm jaded on making over generalized statements in the absence of domain and context.

A good example of this do it right the first time can be found in the history of RFETS, where the consequences of failure were very high. One of the brilliant strategies of the first CEO was to hire Navy ship and submarine Captains who understood those consequences and behaved accordingly when faced with a high value at risk situation.

Making the impossible possible from Glen Alleman

So when you hear I've tried this, maybe you should try it as well. Or just try it, when you've failed you'll be better for it. Ask if that speaker has any experience outside his own anecdotal domain to make a credible assessment of the applicability of his suggestion?


Related articles What is Governance? Complex Project Management Is Programming the Same as Software Engineering? The Myth and Half-Truths of "Myths and Half-Truths"
Categories: Project Management

Santa Lands on a Virgin Atlantic Plane with 4D Technology

Microsoft and Virgin help land Santa on top of a plane at 30,000 feet.  If you‚Äôve been wondering where Santa‚Äôs been, he landed on top of a Virgin Atlantic plane and did a photo shoot with the passengers.

Microsoft teamed up with Richard Branson and Virgin Atlantic to bring the magic of Christmas to life.  In the world‚Äôs first 4D experience in flight, Santa Claus appears to land on top of a Virgin Atlantic plane at 30,000 feet. 

How’s that for some fancy flying with modern technology?!

Each passenger was also given a Windows tablet so they could track Father Christmas and chat with him during the flight.

Here’s the video of Santa landing on top of the plane and visiting with the passengers:

Video: Santa Lands on Top of a Virgin Atlantic Plane at 30,000 Feet

Here are a few scenes that show Santa in action …

Here’s one of Santa’s reindeer peering down into the cabin from on top of the plane:


Here’s Santa peering down into the cabin from above the plane before he goes inside:


Santa sees somebody he recognizes:


Santa boards the plane and walks the cabin:


The kids are excited to see Santa:




Adults are happy, too:



Santa has time for some photo shoots:



Santa leaves to get back to his sleigh on top of the plane:


Virgin Atlantic and Microsoft wish everybody a very, merry Christmas:


Here’s Richard Branson’s post on the story:

Santa Lands on Virgin Atlantic Plane at 30,000 Feet

Merry Christmas to all and to all a good night.


You Might Also Like

10 Big Ideas from a Christmas Carol

25 Holiday Classic Movies and Lessons Learned

Microsoft Cloud Case Studies at a Glance

Categories: Architecture, Programming

Five Problems That Impact Testing Effectiveness and Efficiency: #2 Involvement


Riding a horse requires the involvement of the horse!

Riding a horse requires the involvement of the horse!

Effective testing scenarios require a tester to work with users, product owners, business analysts, developers and others to determine whether a deliverable is what it is supposed to be, whether it meets the definition of done and standards of quality. Testing is a collaborative enterprise. Testing is less effective when the right people are not involved in testing. In order to meet the basic level of collaborative involvement for effective testing, testers need interaction with the broad stakeholder community in three broad categories of activity. They are:

  1. Developing requirements ‚Äď Requirements, whether written as user stories, use cases or paragraphs of text, are a critical input into the testing process. Requirements become the basis¬†of¬†comparison to determine if what is being developed is what the the business wants. Business and technical stakeholders provide the input that becomes the requirements. Testing provides feedback to challenge the assumptions of the stakeholders and development teams as the requirements are formed, groomed and transformed into functional code.
  2. Defining realistic acceptance tests and criteria РA well-formed user story includes not only the story itself, but a set of acceptance criteria. Acceptance criteria provide the development team both with a deeper understanding of how to interpret the story and with a tool to understand when development is complete. Business and technical stakeholders provide the diversity of knowledge to develop relevant acceptance tests and criteria. When testing is a silo’ed task the development team will have to make assumptions that will need to be validated later in the process (later is never better than sooner when it comes to validation). Agile roles such as the product owner and techniques like the Three Amigos are tools to codify involvement.
  3. Participating in reviews, discussions, demonstrations and acceptance testing ‚Äď I have written a few lines of code, managed a few projects and teams and tested a few projects in my career. In most of those cases a business facing product owner and/or user(s) where involved in validating and verifying the work as it progressed toward completion. I can‚Äôt count the number of defects or poor assumptions that were caught and fixed along the way. In those scenarios where real stakeholders did not participate in reviewing, refining and testing the product, the quality was generally lower. Colleagues over the years have expressed similar observations.

When you look in a cube and see a developer or tester reading a user story or use case while typing out a test or perhaps even executing a specific test, it might seem that testing is a solitary event. But you would probably be wrong. What you would have missed is the interaction between developers, testers, product owners that preceded those events, in which knowledge and perspectives were shared. You would have also missed the involvement of users, product owners, developers and stakeholders to review the results so that what gets delivered is not only technically correct, but also has business value. Without involvement, testing provides far less value than possible.

Categories: Process Management

Speaking computers for more fun !

I didn't try it on mono, but it should also work with some tweaking, see details here

Xmas is a good time to surprise kids, and what's more fun than a talking computer ?!

Hello world !

Nothing's easier, and this kind of Hello World will appeal them to programming in a flash :

#r "System.Speech"
open System.Speech.Synthesis

let synt = new SpeechSynthesizer()
let say s = synt.Speak(s: string)

say "Hello world !" 

Of course, if you're french like me, it'll say this with an awful french accent - something like hélo ouorld !

But you can select a different voice if available by providing hints:

open System.Globalization
let english = CultureInfo.GetCultureInfo("en-US")

synt.SelectVoiceByHints(VoiceGender.NotSet, VoiceAge.NotSet, 1, english)
say "Hello world !"

Far better !

Can you beat it ?

Now, a talking fizz buzz, up to 100 ! Can you beat it ?

[1 .. 100]
|> (fun n -> 
    match n%3, n%5 with
    | 0, 0 -> "FizzBuzz"
    | 0, _ -> "Fizz"
    | _, 0 -> "Buzz"
    | _ -> string n )
|> List.iter say
Even harder !

Now with a recognizer, we can wait for voice user input.

The problem with the Grammar API is that it's totally mutable and not really DSL oriented. Let's correct that :

open System.Speech.Recognition

type Grammar =
    | Phrase of text:string * result: string
    | Lst of Grammar list
    | Alt of Grammar list
    | Repeat of min: int * max: int * Grammar

let rec build = function
    | Phrase (text, result) -> 
        // Just build the a single phrase

    | Lst grammars -> 
        // Append parts of grammars one after the other
        let builder = GrammarBuilder()
        |> build
        |> List.iter builder.Append

    | Alt alternatives -> 
        // Create alternatives
        let choices =
            |> build 
            |> List.toArray

    | Repeat(min, max, grammar) -> 
        // Repeat a part of the grammar
        GrammarBuilder(build grammar, min, max)

This is not a full DSL for speach recognition, you can look at all the GrammarBuilder methods to add more possibilities.. Even here, I'll use only Phrase and Alt.

Now, we need a recognizer and wire the grammar with functions that will be called when a part of the grammar is recognized or rejected. It is mandatory to set grammar's culture to the recognizer's culture. There's usually a single recognizer installed by default on your system and it uses installed system's culture. In my case, it'll be french.

let recog = new SpeechRecognizer()

let recognize grammar recognized rejected  = 
    let builder = build grammar
    builder.Culture <- recog.RecognizerInfo.Culture
    printfn "%A" recog.RecognizerInfo.Culture
    recog.LoadGrammar(Grammar builder)
    recog.SpeechRecognized |> Event.add (fun e -> recognized e.Result.Text (string e.Result.Semantics.Value))
    recog.SpeechRecognitionRejected |> Event.add (fun e -> rejected ())

We can then use this to create a little Christmass quizz thanks to the FSharp.Data FreeBase Type Provider !

We'll use free base to find a list of Actors who plaid Santa in movies.

For this, install the FSharp.Data NuGet:

nuget install FSharp.Data -o packages -x

The dll should be in .\packages\FSharp.Data\lib\net40\FSharp.Data.dll

#r @"packages\FSharp.Data\lib\net40\FSharp.Data.dll"
open FSharp.Data

let fb =FreebaseData.GetDataContext()

Let's build the grammar

let santaActorsFilms =
    fb.``Arts and Entertainment``
      .``Film characters``
      .``Santa Claus``
      .``Portrayed in films``
    |> (fun c -> c.Actor.Name, c.Film.Name)
    |> Seq.toList

let santaActorsGrammar =
    |> (fun (actor,film) -> Phrase(actor, film))
    |> Alt

Here is the function to call when an actor is recognized.

I tried to pass a discriminated union as a value, but even if the API uses an object, the documentation states that it has to be a bool, an int or a string. I used only strings here.

let recognized text value =
    say (sprintf "True ! %s was Santa in %s" text value)

Here is the function when the speech could not be matched with the grammar.

It is also possible to get the audio of the text in this case. I decided to ignore it due to time constraints.

let rejected () = say "No, Not a Santa !"

Now, let's run it !!

recognize santaActorsGrammar recognized rejected

At this point the speech recognition configuration should appear if it's the first time you use it.

Once done you should be able to try the quizz !

If your OS culture is not english, don't hesitate to use a local accent for actor's name !


I hope you had fun with this API, and that you'll want to tweak it for your own demo !

The full code - using FSharp.Formatting - is on my gist

Happy Christmass !

namespace System namespace System.Speech namespace System.Speech.Synthesis val synt : SpeechSynthesizer

Full name: XMas fun.synt Multiple items
type SpeechSynthesizer =
  new : unit -> SpeechSynthesizer
  member AddLexicon : uri:Uri * mediaType:string -> unit
  member Dispose : unit -> unit
  member GetCurrentlySpokenPrompt : unit -> Prompt
  member GetInstalledVoices : unit -> ReadOnlyCollection<InstalledVoice> + 1 overload
  member Pause : unit -> unit
  member Rate : int with get, set
  member RemoveLexicon : uri:Uri -> unit
  member Resume : unit -> unit
  member SelectVoice : name:string -> unit

Full name: System.Speech.Synthesis.SpeechSynthesizer

SpeechSynthesizer() : unit val say : s:string -> unit

Full name: XMas fun.say val s : string SpeechSynthesizer.Speak(promptBuilder: PromptBuilder) : unit
SpeechSynthesizer.Speak(prompt: Prompt) : unit
SpeechSynthesizer.Speak(textToSpeak: string) : unit Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

type string = System.String

Full name: Microsoft.FSharp.Core.string namespace System.Globalization val english : CultureInfo

Full name: XMas fun.english Multiple items
type CultureInfo =
  new : name:string -> CultureInfo + 3 overloads
  member Calendar : Calendar
  member ClearCachedData : unit -> unit
  member Clone : unit -> obj
  member CompareInfo : CompareInfo
  member CultureTypes : CultureTypes
  member DateTimeFormat : DateTimeFormatInfo with get, set
  member DisplayName : string
  member EnglishName : string
  member Equals : value:obj -> bool

Full name: System.Globalization.CultureInfo

CultureInfo(name: string) : unit
CultureInfo(culture: int) : unit
CultureInfo(name: string, useUserOverride: bool) : unit
CultureInfo(culture: int, useUserOverride: bool) : unit CultureInfo.GetCultureInfo(name: string) : CultureInfo
CultureInfo.GetCultureInfo(culture: int) : CultureInfo
CultureInfo.GetCultureInfo(name: string, altName: string) : CultureInfo SpeechSynthesizer.SelectVoiceByHints(gender: VoiceGender) : unit
SpeechSynthesizer.SelectVoiceByHints(gender: VoiceGender, age: VoiceAge) : unit
SpeechSynthesizer.SelectVoiceByHints(gender: VoiceGender, age: VoiceAge, voiceAlternate: int) : unit
SpeechSynthesizer.SelectVoiceByHints(gender: VoiceGender, age: VoiceAge, voiceAlternate: int, culture: CultureInfo) : unit type VoiceGender =
  | NotSet = 0
  | Male = 1
  | Female = 2
  | Neutral = 3

Full name: System.Speech.Synthesis.VoiceGender field VoiceGender.NotSet = 0 type VoiceAge =
  | NotSet = 0
  | Child = 10
  | Teen = 15
  | Adult = 30
  | Senior = 65

Full name: System.Speech.Synthesis.VoiceAge field VoiceAge.NotSet = 0 Multiple items
module List

from Microsoft.FSharp.Collections

type List<'T> =
  | ( [] )
  | ( :: ) of Head: 'T * Tail: 'T list
  interface IEnumerable
  interface IEnumerable<'T>
  member Head : 'T
  member IsEmpty : bool
  member Item : index:int -> 'T with get
  member Length : int
  member Tail : 'T list
  static member Cons : head:'T * tail:'T list -> 'T list
  static member Empty : 'T list

Full name: Microsoft.FSharp.Collections.List<_> val map : mapping:('T -> 'U) -> list:'T list -> 'U list

Full name: val n : int val iter : action:('T -> unit) -> list:'T list -> unit

Full name: Microsoft.FSharp.Collections.List.iter namespace System.Speech.Recognition type Grammar =
  | Phrase of text: string * result: string
  | Lst of Grammar list
  | Alt of Grammar list
  | Repeat of min: int * max: int * Grammar

Full name: XMas fun.Grammar union case Grammar.Phrase: text: string * result: string -> Grammar union case Grammar.Lst: Grammar list -> Grammar type 'T list = List<'T>

Full name: Microsoft.FSharp.Collections.list<_> union case Grammar.Alt: Grammar list -> Grammar union case Grammar.Repeat: min: int * max: int * Grammar -> Grammar val min : e1:'T -> e2:'T -> 'T (requires comparison)

Full name: Microsoft.FSharp.Core.Operators.min Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name:

type int = int32

Full name:

type int<'Measure> = int

Full name:<_> val max : e1:'T -> e2:'T -> 'T (requires comparison)

Full name: Microsoft.FSharp.Core.Operators.max val build : _arg1:Grammar -> GrammarBuilder

Full name: XMas val text : string val result : string Multiple items
type GrammarBuilder =
  new : unit -> GrammarBuilder + 7 overloads
  member Append : phrase:string -> unit + 7 overloads
  member AppendDictation : unit -> unit + 1 overload
  member AppendRuleReference : path:string -> unit + 1 overload
  member AppendWildcard : unit -> unit
  member Culture : CultureInfo with get, set
  member DebugShowPhrases : string
  static member Add : phrase:string * builder:GrammarBuilder -> GrammarBuilder + 4 overloads

Full name: System.Speech.Recognition.GrammarBuilder

GrammarBuilder() : unit
GrammarBuilder(phrase: string) : unit
GrammarBuilder(alternateChoices: Choices) : unit
GrammarBuilder(key: SemanticResultKey) : unit
GrammarBuilder(value: SemanticResultValue) : unit
GrammarBuilder(phrase: string, subsetMatchingCriteria: SubsetMatchingMode) : unit
GrammarBuilder(phrase: string, minRepeat: int, maxRepeat: int) : unit
GrammarBuilder(builder: GrammarBuilder, minRepeat: int, maxRepeat: int) : unit Multiple items
type SemanticResultValue =
  new : value:obj -> SemanticResultValue + 2 overloads
  member ToGrammarBuilder : unit -> GrammarBuilder

Full name: System.Speech.Recognition.SemanticResultValue

SemanticResultValue(value: obj) : unit
SemanticResultValue(phrase: string, value: obj) : unit
SemanticResultValue(builder: GrammarBuilder, value: obj) : unit val grammars : Grammar list val builder : GrammarBuilder GrammarBuilder.Append(value: SemanticResultValue) : unit
GrammarBuilder.Append(key: SemanticResultKey) : unit
GrammarBuilder.Append(alternateChoices: Choices) : unit
GrammarBuilder.Append(builder: GrammarBuilder) : unit
GrammarBuilder.Append(phrase: string) : unit
GrammarBuilder.Append(phrase: string, subsetMatchingCriteria: SubsetMatchingMode) : unit
GrammarBuilder.Append(builder: GrammarBuilder, minRepeat: int, maxRepeat: int) : unit
GrammarBuilder.Append(phrase: string, minRepeat: int, maxRepeat: int) : unit val alternatives : Grammar list val choices : GrammarBuilder [] val toArray : list:'T list -> 'T []

Full name: Microsoft.FSharp.Collections.List.toArray Multiple items
type Choices =
  new : unit -> Choices + 2 overloads
  member Add : params phrases:string[] -> unit + 1 overload
  member ToGrammarBuilder : unit -> GrammarBuilder

Full name: System.Speech.Recognition.Choices

Choices() : unit
Choices(params phrases: string []) : unit
Choices(params alternateChoices: GrammarBuilder []) : unit val min : int val max : int val grammar : Grammar val recog : SpeechRecognizer

Full name: XMas fun.recog Multiple items
type SpeechRecognizer =
  new : unit -> SpeechRecognizer
  member AudioFormat : SpeechAudioFormatInfo
  member AudioLevel : int
  member AudioPosition : TimeSpan
  member AudioState : AudioState
  member Dispose : unit -> unit
  member EmulateRecognize : inputText:string -> RecognitionResult + 2 overloads
  member EmulateRecognizeAsync : inputText:string -> unit + 2 overloads
  member Enabled : bool with get, set
  member Grammars : ReadOnlyCollection<Grammar>

Full name: System.Speech.Recognition.SpeechRecognizer

SpeechRecognizer() : unit val recognize : grammar:Grammar -> recognized:(string -> string -> unit) -> rejected:(unit -> unit) -> bool

Full name: XMas fun.recognize val recognized : (string -> string -> unit) val rejected : (unit -> unit) property GrammarBuilder.Culture: CultureInfo property SpeechRecognizer.RecognizerInfo: RecognizerInfo property RecognizerInfo.Culture: CultureInfo val printfn : format:Printf.TextWriterFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.printfn SpeechRecognizer.LoadGrammar(grammar: Grammar) : unit event SpeechRecognizer.SpeechRecognized: IEvent<System.EventHandler<SpeechRecognizedEventArgs>,SpeechRecognizedEventArgs> Multiple items
module Event

from Microsoft.FSharp.Control

type Event<'T> =
  new : unit -> Event<'T>
  member Trigger : arg:'T -> unit
  member Publish : IEvent<'T>

Full name: Microsoft.FSharp.Control.Event<_>

type Event<'Delegate,'Args (requires delegate and 'Delegate :> Delegate)> =
  new : unit -> Event<'Delegate,'Args>
  member Trigger : sender:obj * args:'Args -> unit
  member Publish : IEvent<'Delegate,'Args>

Full name: Microsoft.FSharp.Control.Event<_,_>

new : unit -> Event<'T>

new : unit -> Event<'Delegate,'Args> val add : callback:('T -> unit) -> sourceEvent:IEvent<'Del,'T> -> unit (requires delegate and 'Del :> System.Delegate)

Full name: Microsoft.FSharp.Control.Event.add val e : SpeechRecognizedEventArgs property RecognitionEventArgs.Result: RecognitionResult property RecognizedPhrase.Text: string property RecognizedPhrase.Semantics: SemanticValue property SemanticValue.Value: obj event SpeechRecognizer.SpeechRecognitionRejected: IEvent<System.EventHandler<SpeechRecognitionRejectedEventArgs>,SpeechRecognitionRejectedEventArgs> val e : SpeechRecognitionRejectedEventArgs property SpeechRecognizer.Enabled: bool namespace FSharp namespace FSharp.Data val fb : FreebaseData.ServiceTypes.FreebaseService

Full name: XMas fun.fb type FreebaseData =
  static member GetDataContext : unit -> FreebaseService
  nested type ServiceTypes

Full name: FSharp.Data.FreebaseData

<summary>Typed representation of Freebase data. See for terms and conditions.</summary>
FreebaseData.GetDataContext() : FreebaseData.ServiceTypes.FreebaseService val santaActorsFilms : (string * string) list

Full name: XMas fun.santaActorsFilms property FreebaseData.ServiceTypes.Film.Film.Film_characterDataIndividualsAZ.S: FreebaseData.ServiceTypes.Film.Film.Film_characterDataIndividualsAZ.Film_characterDataIndividualsIndexedS

<summary>An indexing of specific named individuals of type &apos;Film character&apos; in the web data store</summary>
module Seq

from Microsoft.FSharp.Collections val map : mapping:('T -> 'U) -> source:seq<'T> -> seq<'U>

Full name: val c : FreebaseData.ServiceTypes.Film.Film.PerformanceData property FreebaseData.ServiceTypes.Film.Film.PerformanceData.Actor: FreebaseData.ServiceTypes.Film.Film.ActorData

property Runtime.Freebase.IFreebaseObject.Name: string property FreebaseData.ServiceTypes.Film.Film.PerformanceData.Film: FreebaseData.ServiceTypes.Film.Film.FilmData

val toList : source:seq<'T> -> 'T list

Full name: Microsoft.FSharp.Collections.Seq.toList val santaActorsGrammar : Grammar

Full name: XMas fun.santaActorsGrammar val actor : string val film : string val recognized : text:string -> value:string -> unit

Full name: XMas fun.recognized val value : string val sprintf : format:Printf.StringFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.sprintf val rejected : unit -> unit

Full name: XMas fun.rejected
Categories: Architecture, Requirements

Making a performant watch face

Android Developers Blog - Tue, 12/23/2014 - 19:42

Posted by Hoi Lam, Developer Advocate, Android Wear

What’s a better holiday gift than great performance? You’ve got a great watch face idea -- now, you want to make sure the face you’re presenting to the world is one of care and attention to detail.

At the core of the watch face's process is an onDraw method for canvas operations. This allows maximum flexibility for your design, but also comes with a few performance caveats. In this blog post, we will mainly focus on performance using the real life journey of how we optimised the Santa Tracker watch face, more than doubling the number of fps (from 18 fps to 42 fps) and making the animation sub-pixel smooth.

Starting point - 18 fps

Our Santa watch face contains a number of overlapping bitmaps that are used to achieve our final image. Here's a list of them from bottom to top:

  1. Background (static)
  2. Clouds which move to the middle
  3. Tick marks (static)
  4. Santa figure and sledge (static)
  5. Santa’s hands - hours and minutes
  6. Santa’s head (static)

The journey begins with these images...

Large images kill performance (+14 fps)

Image size is critical to performance in a Wear application, especially if the images will be scaled and rotated. Wasted pixel space (like Santa’s arm here) is a common asset mistake:

Before: 584 x 584 = 341,056 pixelsAfter: 48*226 = 10,848 (97% reduction)

It's tempting to use bitmaps from the original mock up that have the exact location of watch arms and components in absolute space. Sadly, this creates problems, like in Santa's arm here. While the arm is in the correct position, even transparent pixels increase the size of the image, which can cause performance problems due to memory fetch. You'll want to work with your design team to extract padding and rotational information from the images, and rely on the system to apply the transformations on our behalf.

Since the original image covers the entire screen, even though the bitmap is mostly transparent, the system still needs to check every pixel to see if they have been impacted. Cutting down the area results in significant gains in performance. After correcting both of the arms, the Santa watch face frame rate increased by 10 fps to 28 fps (fps up 56%). We saved another 4 fps (fps up 22%) by cropping Santa’s face and figure layer. 14 fps gained, not bad!

Combine Bitmaps (+7 fps)

Although it would be ideal to have the watch tick marks on top of our clouds, it actually does not make much difference visually as the clouds themselves are transparent. Therefore there is an opportunity to combine the background with the ticks.


When we combined these two views together, it meant that the watch needed to spend less time doing alpha blending operations between them, saving precious CPU time. So, consider collapsing alpha blended resources wherever we can in order to increase performance. By combining two full screen bitmaps, we were able to gain another 7 fps (fps up 39%).

Anti-alias vs FilterBitmap flags - what should you use? (+2 fps)

Android Wear watches come in all shapes and sizes. As a result, it is sometimes necessary to resize a bitmap before drawing on the screen. However, it is not always clear what options developers should select to make sure that the bitmap comes out smoothly. With canvas.drawBitmap, developers need to feed in a Paint object. There are two important options to set - they are anti-alias and FilterBitmap. Here’s our advice:

  • Anti-alias does not do anything for bitmaps with transparent edges. We often switch on the anti-alias option by default as developers when we are creating a Paint object. However, this option only really makes sense for vector objects. For bitmaps, this is used to blend the rectangular edges if it is rotated or skewed and it has no impact if the edge pixels are transparent (as we would imagine most watch face arms would be). The hand on the left below has anti-alias switched on, the one on the right has it switched off. So turn off anti-aliasing for bitmaps to gain performance back. For our watch face, we gained another 2 fps (fps up 11%) by switching this option off.
  • Switch on FilterBitmap for all bitmap objects which are on top of other objects - this option smooths the edges when drawBitmap is called. This should not be confused with the filter option on Bitmap.createScaledBitmap for resizing bitmaps. We need both to be turned on. The bitmaps below are the magnified view of Santa‚Äôs hand. The one on the left has FilterBitmap switched off and the one on the right has FilterBitmap switched on.
Eliminate expensive calls in the onDraw loop (+3 fps)

onDraw is the most critical function call in watch faces. It's called for every drawable frame, and the actual painting process cannot move forward until it's finished. As such, our onDraw method should be as light and as performant as possible. Here's some common problems that developers run into that can be avoided:

  1. Do move heavy and common code to a precompute function - e.g. if we commonly grab R.array.cloudDegrees, try doing that in onCreate, and just referencing it in the onDraw loop.
  2. Don’t repeat the same image transform in onDraw - it’s common to resize bitmaps at runtime to fit the screen size but this is not available in onCreate. To avoid resizing the bitmap over and over again in onDraw, override onSurfaceChanged where width and height information are available and resize images there.
  3. Don't allocate objects in onDraw - this leads to high memory churn which will force garbage collection events to kick off, killing frame rates.
  4. Do analyze the CPU performance by using a tool such as the Android Device Monitor. It’s important that the onDraw execution time is short and occurs in a regular period.

Following these simple rules will improve rendering performance drastically.

In the first version, the Santa onDraw routine has a rogue line:

int[] cloudDegrees = 

This loads the int array on every call from resources which is expensive. By eliminating this, we gained another 3 fps (fps up 17%).

Sub-pixel smooth animation (-2 fps)

For those keeping count, we should be 44 fps, so why is the end product 42 fps? The reason is a limitation with canvas.drawBitmap. Although this command takes left and top positioning settings as a float, the API actually only deals with integers if it is purely translational for backwards compatibility reasons. As a result, the cloud can only move in increments of a whole pixel resulting in janky animations. In order to be sub-pixel smooth, we actually need to draw and then rotate rather than having pre-rotate clouds which moves towards Santa. This additional rotation costs us 2 fps. However, the effect is worthwhile as the animation is now sub-pixel smooth.

Before - fast but janky and wobbly

for (int i = 0; i < mCloudBitmaps.length; i++) {
    float r = centerX - (timeElapsed / mCloudSpeeds[i]) % centerX;
    float x = centerX + 
        -1 * (r * (float) Math.cos(Math.toRadians(cloudDegrees[i] + 90)));
    float y = centerY - 
        r * (float) Math.sin(Math.toRadians(cloudDegrees[i] + 90));
    mCloudFilterPaints[i].setAlpha((int) (r/centerX * 255));
    Bitmap cloud = mCloudBitmaps[i];
        x - cloud.getWidth() / 2,
        y - cloud.getHeight() / 2,

After - slightly slower but sub-pixel smooth

for (int i = 0; i < mCloudBitmaps.length; i++) {;
    canvas.rotate(mCloudDegrees[i], centerX, centerY);
    float r = centerX - (timeElapsed / (mCloudSpeeds[i])) % centerX;
    mCloudFilterPaints[i].setAlpha((int) (r / centerX * 255));
    canvas.drawBitmap(mCloudBitmaps[i], centerX, centerY - r,

Before: Integer translation values create janky, wobbly animation. After: smooth sailing!

Quality on every wrist

The watch face is the most prominent UI element in Android Wear. As craftspeople, it is our responsibility to make it shine. Let’s put quality on every wrist!

Join the discussion on

+Android Developers
Categories: Programming

Sponsored Post: MemSQL, Campanja, Hypertable, Sprout Social, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • DevOps Engineer for Wikia. Wikia is the go-to place for fan content that is created entirely by fans! As a Quantcast Top 20 site with over 120 million monthly uniques we are tackling very interesting problems at a scale you won't find at many other places. We embrace a DevOps culture and are looking to expand our team with people that are excited about working with just about every piece of our stack. You'll also partner with our platform team as they break down the monolith and move towards service oriented architecture. Please apply here.

  • Engineer Manager - Platform. At Wikia we're tackling interesting problems at a scale you won't find at many other places. We're a Quantcast Top 20 site with over 120 million monthly uniques. 100% of the content on our 400,000+ communities is user generated. That combination of scale and UGC creates some pretty compelling challenges and on top of that we're working on moving away from a monolithic architecture and actively working on finding the best technologies to best suit each individual piece of our platform. We're currently in search of an experienced Engineer Manager to help drive this process. Please apply here.

  • Campanja is an Internet advertising optimization company born in the cloud and today we are one of the nordics bigger AWS consumers, the time has come for us to the embrace the next generation of cloud infrastructure. We believe in immutable infrastructure, container technology and micro services, we hope to use PaaS when we can get away with it but consume at the IaaS layer when we have to. Please apply here.

  • Performance and Scale EngineerSprout Social, will be like a physical trainer for the Sprout social media management platform: you will evaluate and make improvements to keep our large, diverse tech stack happy, healthy, and, most importantly, fast. You'll work up and down our back-end stack - from our RESTful API through to our myriad data systems and into the Java services and Hadoop clusters that feed them - searching for SPOFs, performance issues, and places where we can shore things up. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Sign Up for New Aerospike Training Courses.  Aerospike now offers two certified training courses; Aerospike for Developers and Aerospike for Administrators & Operators, to help you get the most out of your deployment.  Find a training course near you.
Cool Products and Services
  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here:

  • Aerospike Hits 1M writes per second with 6x Fewer Servers than Cassandra. A new Google Compute Engine benchmark demonstrates how the Aerospike database hit 1 million writes per second with just 50 nodes - compared to Cassandra's 300 nodes. Read the benchmark:

  • Hypertable Inc. Announces New UpTime Support Subscription Packages. The developer of Hypertable, an open-source, high-performance, massively scalable database, announces three new UpTime support subscription packages – Premium 24/7, Enterprise 24/7 and Basic. 24/7/365 support packages start at just $1995 per month for a ten node cluster -- $49.95 per machine, per month thereafter. For more information visit us on the Web at Connect with Hypertable: @hypertable--Blog.

  • FoundationDB launches SQL Layer. SQL Layer is an ANSI SQL engine that stores its data in the FoundationDB Key-Value Store, inheriting its exceptional properties like automatic fault tolerance and scalability. It is best suited for operational (OLTP) applications with high concurrency. Users of the Key Value store will have free access to SQL Layer. SQL Layer is also open source, you can get started with it on GitHub as well.

  • Diagnose server issues from a single tab. The Scalyr log management tool replaces all your monitoring and analysis services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. It's a universal tool for visibility into your production systems. Log aggregation, server metrics, monitoring, alerting, dashboards, and more. Not just “hosted grep” or “hosted graphs,” but enterprise-grade functionality with sane pricing and insane performance. Trusted by in-the-know companies like Codecademy – try it free!

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

What is Governance?

Herding Cats - Glen Alleman - Tue, 12/23/2014 - 18:13

Magna-carta-014In the business management business and the management of spending other people's money to produce a product or provide a service, there is a distinct bright line between organizations that have governance or organizations that have no governance. 

Governance and the ideas of governance have been around for a long time. The signing of the Magna Carta 800 years ago set the stage for governance in political systems, which flowed to business systems, and down to the management of projects in both business and government.

Business Governance is ...

... the set of decisions that defines expectations, grants power, or verifies performance. It consists either of a separate process or of a specific part of management or leadership processes.

There are well developed business governance frameworks, starting with SOX. But I work primarily in the Program Governance  domain, guided by ITIL, CMMI, CMS (Center for Medicaid Services), and other internal paradigms.

Program and Project Governance is ...

... the framework which ensures the project has been correctly conceived and is being executed in accordance with best project management practice within the wider framework of the firms organizational governance process.

Effective project governance ensures projects deliver the value expected of them.  An appropriate governance framework ensures that all expenditure is appropriate for the risks being handled. 

Project governance is not about micro‚Äďmanagement, it is about setting the terms of reference and operating framework, defining the boundaries and ensuring that planning and execution are carried out in a way which ensures that the project delivers benefits.

The difference between business governance and project governance is ...

  • Business Governance is the organization control of the business's ability to deliver value.
  • Project Governance is the oversight of the execution of the project's that produce that value.

Screen Shot 2014-12-23 at 9.11.12 AM

So What Does This Mean and Why Should We Care?

Let's start with the question what's the value at risk? † The literature, blog-sphere, and other outlets are full of examples of IT project failures. From the poor statistics of Standish, to the Root Cause Analysis reports at RAND and IDA, to research on processes and practices searching for solutions to the cost, schedule, and technical shortfalls - the common missing element is deciding what governance processes to apply in what manner to protect the Value At Risk.

When we hear of some new, untested, idea of how to improve the performance of a project, we need to start with:

  • Where have you seen this work in improving the probability of success?
  • What's the¬†Value at Risk for the projects you want to apply this idea to?
  • What Governance processes are in place, that would be violated if we applied you clever idea?
  • What basic principles of business management would be violated in the presence of this idea?

These questions are governance questions? 

If you have no governance, then the answers to the questions are unimportant, proceed to spend your customer's money. 

When we encunter Dilbert-Isk situations in management, it is likley we are missing the Governance of Other Peoples Money factor, that is often used as the excuse for not doing good management. Here's a small sample of current conditions, and desired conditions from a recent engagement.

Screen Shot 2014-12-23 at 8.21.51 AM

A Test of Any New Idea

When we hear about the newest idea on managing other peoples money, ask how is this idea going to address the root cause of the current undesirable condition? 

Then ask how would we see tangible evidence that this new idea is applicable in out domain, for our "value at risk," within our governance model?

No answers? Just I'm not going to tell you how, I'm just going to tell you my experiences. 


† "Using Value-at-Risk for IS/IT Project and Portfolio Appraisal and Risk Management," Stefan Koch Department of Information Business, Vienna University of Economics and BA, Austria

† Portfolio Optimization, using Value at Risk, 

Categories: Project Management

Isn’t Santa really just a Product Manager?

Software Requirements Blog - - Tue, 12/23/2014 - 17:30
I used to do some fun holiday jingles requirements-style. I fell out of habit with it a few years ago, but luckily they are timeless and can just be reused! That said, I was trying to get in the holiday spirit to try again this year, and I found myself going down a different path. […]
Categories: Requirements

Five Problems That Impact Testing Effectiveness and Efficiency: #1 Planning


Forewarned is forearmed!

Forewarned is forearmed!

Testing is an important set of tools for influencing the quality of the functionality that gets delivered. The term testing can cover a wide range of activities ranging from reviews to user acceptance testing with a nearly infinite number of stops in between. Some organizations expend up to 60 ‚Äď 70% of project effort just on testing (this is abnormally high), often with the intention of ‚Äútesting in‚ÄĚ quality. When the effectiveness and efficiency of testing has been derailed there are five typical root causes.

  • Planning
  • Involvement
  • Organizational Management
  • Capability
  • Process

Planning is the first of the root causes. Poor planning will yield poor results. Test planning defines the approach a project will use to verify and validate the deliverable they are creating. There are several inputs that impact and shape a test plan. The inputs are:

  1. Risk ‚Äď Risk is a reflection of possibilities that might impact the project if they occur. Some projects will inherently have a bigger impact on the organization if things go wrong. Testing can be tuned to act as insurance that risks do not become issues. Agile projects use a wide range of techniques to incorporate risk into how testing is approached. Planning techniques include release planning, sprint planning, tools like the definition of done, techniques such as test first development (including test driven development) or classic test planning documentation.
  2. Test Goals ‚Äď Test goals reflect an organization‚Äôs business strategy and needs. Test goals can be as simple as ensuring software is defect free (this is not only simple, but a simplistic) to improving the product‚Äôs delivered quality (as compared to a standard or a baseline).
  3. Test Policy ‚Äď A policy defines a course of action that supports a long-term purpose. A test policy describes the type of behavior without defining the ends, means or approach. Good policies are aligned with the business strategy and the test goals. Test policies must be agreed upon (at least passively) by all of the stakeholders. ¬†Policies that the involved parties don’t agree with will potentially generate poor behavior as stakeholders struggle against what they perceive as artificial constraints. For example a policy that requires all applications to be stress tested even if they are standalone, one person applications will not be perceived as fitting the environment. ¬†The policy will be ignored when¬†practitioners don’t think it applies which opens the door for failure if something is not stress tested that should be.
  4. Test Strategy ‚Äď At an organizational level, a test strategy represents a broad outline that will orperationalize the test policy.At a project level, a test strategy will define the how the project will conform to the test policy and meet the test goals. The strategy operationalizes the testing goals and policies based on the business and project risks.

The typical image painted when discussing test planning is an omnibus document that defines how a project will approach testing, sometimes in mind-numbing detail. While that level of detail might be important in some scenarios, in most Agile projects the adoption of standard techniques provides the policy, strategy and guidance to ensure a good test planning. Agile techniques that improve planning include:

  1. Well-formed user stories (including acceptance criteria),
  2. Test first development (such as test driven development or acceptance test driven development), and
  3. A solid definition of done.

Planning is requirement for any activity development activity, testing included.  Good planning is a requirement for good testing.

Categories: Process Management

The Best Books I Read in 2014

I wrote a roundup of the best books I read in 2014.  

I read a lot of books, and not all of them are worth sharing.  I‚Äôm a believer that leaders are readers, and as Bill Gates says, ‚Äúreading is how I learn best.‚ÄĚ

Here are a few of the books that made my list:

  • All in Startup: Launching a New Idea When Everything is On the Line, by Diana Kander
  • Blind Ambition: How to Envision Your Limitless Potential and Achieve the Success You Want, by Patricia Walsh
  • Eat for Health, by Dr. Joel Fuhrman, M.D.
  • How to Fail at Almost Everything and Still Win Big: Kind of the Story of My Life, by Scott Adams
  • Leading Digital: Turning Technology into Business Transformation, by George Westerman, Didier Bonnet, and Andrew McAfee
  • The Undefeated Mind: On the Science of Constructing an Indestructible Self, by Alex Lickerman
  • Use Your Brain to Change Your Age: Secrets to Look, Feel, and Think Younger Every Day, by Daniel G. Amen, M.D

For the full list and some quick perspective on each book, you can read my post on the best books I read in 2014.

Categories: Architecture, Programming

Scalability as a Service

This is a guest post by Thierry Schellenbach, CEO and author of the open source Stream-Framework, which enables you to build scalable newsfeeds using Cassandra or Redis.

We first wrote about our newsfeed architecture on High Scalability in October 2013. Since then our open source Stream-Framework grew to be the most used package for building scalable newsfeeds. We’re very grateful to the High Scalability community for all the support.

In this article I want to highlight the current trend in our industry of moving  towards externally hosted components. We’re going to compare the hosted solutions for search, newsfeeds and realtime functionality to their open source alternative. This move towards hosted components means you can add scalable components to your app at a fraction of the effort it took just a few years ago.

1.) Search servers
Categories: Architecture

Agile Analysis, Self-Selecting Teams, TDD & BDD in Methods & Tools Winter 2014 issue

From the Editor of Methods & Tools - Mon, 12/22/2014 - 15:22
Methods & Tools ‚Äď the free e-magazine for software developers, testers and project managers ‚Äď has just published its Winter 2014 issue that discusses Agile Analysis, Self-Selecting Teams,Collaborative Development of Domain-specific Languages, TDD with Mock Objects, BDDfire. Methods & Tools Winter 2014 contains the following articles: * Analysis on Analysts in Agile * Self-Selecting Teams – Why You Should Try Self-selection * Collaborative Development of Domain-specific Languages, Models and Generators * TDD with Mock Objects: Design Principles and Emergent Properties * BDDfire: Instant Ruby-Cucumber Framework 55 pages of software development knowledge that you can freely download from ...

It’s beginning to smell like the holiday season!

Software Requirements Blog - - Mon, 12/22/2014 - 15:00
It‚Äôs about that time of year now when people are thinking of what to get others as gifts during the holiday season. Each year my family does a Yankee Swap (which may also be known as a White Elephant), which means that I need to get a gift which has one requirement: $20 or less. […]
Categories: Requirements

Is It Possible The Sony Hack Was Just A Publicity Stunt?

Making the Complex Simple - John Sonmez - Mon, 12/22/2014 - 14:00

What if Sony never was really hacked? Or worse yet, if Sony hacked themselves? I know this sounds a bit crazy–and I admit, I AM a bit crazy–but, indulge me for a few minutes of your time and perhaps I’ll convince you that it just might be possible that the whole thing was staged for a very specific purpose… Why ... Read More

The post Is It Possible The Sony Hack Was Just A Publicity Stunt? appeared first on Simple Programmer.

Categories: Programming

R: Vectorising all the things

Mark Needham - Mon, 12/22/2014 - 12:46

After my last post about finding the distance a date/time is from the weekend Hadley Wickham suggested I could improve the function by vectorising it…

@markhneedham vectorise with pmin(pmax(dateToLookup – before, 0), pmax(after – dateToLookup, 0)) / dhours(1)

— Hadley Wickham (@hadleywickham) December 14, 2014

…so I thought I’d try and vectorise some of the other functions I’ve written recently and show the two versions.

I found the following articles useful for explaining vectorisation and why you might want to do it:

Let’s get started.

Distance from the weekend

We want to find out how many hours away from the weekend i.e. nearest Saturday/Sunday a particular date/time is. We’ll be using the following libraries and set of date/times:

options("scipen"=100, "digits"=4)
times = ymd_hms("2002-01-01 17:00:00") + c(0:99) * hours(1)
data = data.frame(time = times)
> data %>% head()
1 2002-01-01 17:00:00
2 2002-01-01 18:00:00
3 2002-01-01 19:00:00
4 2002-01-01 20:00:00
5 2002-01-01 21:00:00
6 2002-01-01 22:00:00

Let’s have a look at the non vectorised version first:

distanceFromWeekend = function(dateToLookup) {
  before = floor_date(dateToLookup, "week") + hours(23) + minutes(59) + seconds(59)
  after  = ceiling_date(dateToLookup, "week") - days(1)
  timeToBefore = dateToLookup - before
  timeToAfter = after - dateToLookup
  if(timeToBefore < 0 || timeToAfter < 0) {
  } else {
    if(timeToBefore < timeToAfter) {
      timeToBefore / dhours(1)
    } else {
      timeToAfter / dhours(1)

Now let’s run it against our data frame:

> system.time(
    data %>% mutate(ind = row_number()) %>% group_by(ind) %>% mutate(dist = distanceFromWeekend(time))  
   user  system elapsed 
  1.837   0.020   1.884

And now for Hadley’s vectorised version:

distanceFromWeekendVectorised = function(dateToLookup) {
  before = floor_date(dateToLookup, "week") + hours(23) + minutes(59) + seconds(59)
  after  = ceiling_date(dateToLookup, "week") - days(1)
  pmin(pmax(dateToLookup - before, 0), pmax(after - dateToLookup, 0)) / dhours(1)
> system.time(data %>% mutate(dist = distanceFromWeekendVectorised(time)))
   user  system elapsed 
  0.020   0.001   0.023
Extracting start date

My next example was from cleaning up Google Trends data and extracting the start date from a cell inside a CSV file.

We’ll use this data frame:

googleTrends = read.csv("/Users/markneedham/Downloads/report.csv", row.names=NULL)
names(googleTrends) = c("week", "score")
> googleTrends %>% head(10)
                        week score
1  Worldwide; 2004 - present      
2         Interest over time      
3                       Week neo4j
4    2004-01-04 - 2004-01-10     0
5    2004-01-11 - 2004-01-17     0
6    2004-01-18 - 2004-01-24     0
7    2004-01-25 - 2004-01-31     0
8    2004-02-01 - 2004-02-07     0
9    2004-02-08 - 2004-02-14     0
10   2004-02-15 - 2004-02-21     0

The non vectorised version looked like this:

> system.time(
    googleTrends %>% 
      mutate(ind = row_number()) %>% 
      group_by(ind) %>%
      mutate(dates = strsplit(week, " - "),
             start = dates[[1]][1] %>% strptime("%Y-%m-%d") %>% as.character())
   user  system elapsed 
  0.215   0.000   0.214

In this case it’s actually not possible to vectorise the code using the strsplit so we need to use something else. Antonios showed me how to do so using substr:

> system.time(googleTrends %>% mutate(start = substr(week, 1, 10) %>% ymd()))
   user  system elapsed 
  0.018   0.000   0.017
Calculating haversine distance

I wanted to work out the great circular distance from a collection of venues to a centre point in London. I started out with this data frame:

centre = c(-0.129581, 51.516578)
venues = read.csv("/tmp/venues.csv")
> venues %>% head()
                       venue   lat      lon
1              Skills Matter 51.52 -0.09911
2                   Skinkers 51.50 -0.08387
3          Theodore Bullfrog 51.51 -0.12375
4 The Skills Matter eXchange 51.52 -0.09923
5               The Guardian 51.53 -0.12234
6            White Bear Yard 51.52 -0.10980

My non vectorised version looked like this:

> system.time(venues %>% 
    mutate(distanceFromCentre = by(venues, 1:nrow(venues), function(row) { distHaversine(c(row$lon, row$lat), centre)  }))
   user  system elapsed 
  0.034   0.000   0.033

It’s pretty quick but we can do better – the distHaversine function allows us to calculate multiple distances if the first argument ot it is a matrix of lon/lat values rather than a vector:

> system.time(
    venues %>% mutate(distanceFromCentre = distHaversine(cbind(venues$lon, venues$lat), centre))
   user  system elapsed 
  0.001   0.000   0.001
One I can’t figure out…

And finally I have a function which I can’t figure out how to vectorise but maybe someone with more R skillz than me can?

I have a data frame containing the cumulative member counts of various NoSQL London groups:

cumulativeMeetupMembers = read.csv("/tmp/cumulativeMeetupMembers.csv")
> cumulativeMeetupMembers %>% sample_n(10)
                      dayMonthYear    n
4734            Hadoop Users Group UK   2013-10-26 1144
4668            Hadoop Users Group UK   2013-08-03  979
4936            Hadoop Users Group UK   2014-07-31 1644
5150                      Hive London   2012-10-15  109
8020        Neo4j - London User Group   2014-03-15  826
7666        Neo4j - London User Group   2012-08-06   78
1030                  Big Data London   2013-03-01 1416
6500        London MongoDB User Group   2013-09-21  952
8290 Oracle Big Data 4 the Enterprise   2012-06-04   61
2584              Data Science London   2012-03-20  285

And I want to find out the number of members for a group on a specific date. e.g. given the following data…

> cumulativeMeetupMembers %>% head(10)
                                 dayMonthYear  n
1  Big Data / Data Science / Data Analytics Jobs   2013-01-29  1
2  Big Data / Data Science / Data Analytics Jobs   2013-02-06 15
3  Big Data / Data Science / Data Analytics Jobs   2013-02-07 28
4  Big Data / Data Science / Data Analytics Jobs   2013-02-10 31
5  Big Data / Data Science / Data Analytics Jobs   2013-02-18 33
6  Big Data / Data Science / Data Analytics Jobs   2013-03-27 38
7  Big Data / Data Science / Data Analytics Jobs   2013-04-16 41
8  Big Data / Data Science / Data Analytics Jobs   2013-07-17 53
9  Big Data / Data Science / Data Analytics Jobs   2013-08-28 58
10 Big Data / Data Science / Data Analytics Jobs   2013-11-11 63

…the number of members for the ‘Big Data / Data Science / Data Analytics Jobs’ group on the 10th November 2013 should be 58.

I created this data frame of groups and random dates:

dates = ymd("2014-09-01") + c(0:9) * weeks(1)
groups = cumulativeMeetupMembers %>% distinct( %>% select(
groupsOnDate = merge(dates, groups)
names(groupsOnDate) = c('date', 'name')
> groupsOnDate %>% sample_n(10)
          date                                            name
156 2014-10-06                                 GridGain London
153 2014-09-15                                 GridGain London
70  2014-11-03                                Couchbase London
185 2014-09-29                           Hadoop Users Group UK
105 2014-09-29                             Data Science London
137 2014-10-13            Equal Experts Technical Meetup Group
360 2014-11-03                        Scale Warriors of London
82  2014-09-08 Data Science & Business Analytics London Meetup
233 2014-09-15                 London ElasticSearch User Group
84  2014-09-22 Data Science & Business Analytics London Meetup

The non vectorised version looks like this:

memberCount = function(meetupMembers) {
  function(groupName, date) {
    (meetupMembers %>% 
       filter( == groupName & dayMonthYear < date) %>% do(tail(., 1)))$n    
findMemberCount = memberCount(cumulativeMeetupMembers)
> system.time(groupsOnDate %>% mutate(groupMembers = by(groupsOnDate, 1:nrow(groupsOnDate), function(row) { 
          findMemberCount(row$name, as.character(row$date))
        }) %>% 
        cbind() %>% 
        as.vector() ))
   user  system elapsed 
  2.259   0.005   2.269

The output looks like this:

          date                                     name groupMembers
116 2014-10-06                      DeNormalised London          157
322 2014-09-08                 OpenCredo Tech Workshops            7
71  2014-09-01                  Data Enthusiasts London             
233 2014-09-15          London ElasticSearch User Group          614
171 2014-09-01 HPC & GPU Supercomputing Group of London           80
109 2014-10-27                      Data Science London         3632
20  2014-11-03            Big Data Developers in London          708
42  2014-09-08              Big Data Week London Meetup           96
127 2014-10-13          Enterprise Search London Meetup          575
409 2014-10-27                            Women in Data          548

I’ve tried many different approaches but haven’t been able to come up with a version that lets me pass in all the rows to memberCount and calculate the count for each row in one go.

Any ideas/advice/hints welcome!

Categories: Programming

SPaMCAST 321 -11 Reasons For Agile Success, Communication, and Cloud Development


Listen to the podcast

SPaMCAST 321 features our essay on the reasons for success with Agile.  I asked friends and colleagues what they think are the top reasons an organization succeeds with Agile.  The answers were not always what I expected. We review the top 11 factors leading to success with Agile. Listen and share your feedback.

This episode also includes the next installment of Jo Ann Sweeney’s new column Explaining Change.  Jo Ann discusses whether communication always adds value to a project. Visit Jo Ann’s website at and let her know what you think of her new column.

The third segment of this podcast is a new installment of the Software Sensei, where Kim Pries shines light on the area of cloud development. Development for cloud computing is red hot. Understand the nuances that developing for the cloud to enhance your effectiveness!

Call to action!

We are in the middle of a re-read of John Kotter’s classic Leading Change on the Software Process and Measurement Blog.  Are you participating in the re-read? Please feel free to jump in and add your thoughts and comments!

After we finish the current re-read will need to decide which book will be next.  We are building a list of the books that have had the most influence on readers of the blog and listeners to the podcast.  Can you answer the question?

What are the two books that have most influenced you career (business, technical or philosophical)?  Send the titles to

First, we will compile a list and publish it on the blog.¬† Second, we will use the list to drive future ¬†‚ÄúRe-read‚ÄĚ Saturdays. Re-read Saturday is an exciting new feature that began on the Software Process and Measurement blog on November 8th. ¬†Feel free to choose you platform; send an email, leave a message on the blog, Facebook or just tweet the list (use hashtag #SPaMCAST)!


The next Software Process and Measurement Cast will feature our interview with Clareice and Clyneice Chaney. Clareice and Clyneice provide insights and practical advice into how Agile and contracting can work together.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques¬†co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: ‚ÄúThis book will prove that software projects should not be a tedious process, neither for you or your team.‚ÄĚ Support SPaMCAST by buying the book¬†here.

Available in English and Chinese.

Categories: Process Management

SPaMCAST 321 -11 Reasons For Agile Success, Communication, and Cloud Development

Software Process and Measurement Cast - Sun, 12/21/2014 - 23:00

SPaMCAST 321 features our essay on the reasons for success with Agile.  I asked friends and colleagues what they think are the top reasons an organization succeeds with Agile.  The answers were not always what I expected. We review the top 11 factors leading to success with Agile. Listen and share your feedback.

This episode also includes the next installment of Jo Ann Sweeney’s new column Explaining Change.  Jo Ann discusses whether communication always adds value to a project. Visit Jo Ann’s website at and let her know what you think of her new column.

The third segment of this podcast is a new installment of the Software Sensei, where Kim Pries shines light on the area of cloud development. Development for cloud computing is red hot. Understand the nuances that developing for the cloud to enhance your effectiveness!

Call to action!

We are in the middle of a re-read of John Kotter’s classic Leading Change on the Software Process and Measurement Blog.  Are you participating in the re-read? Please feel free to jump in and add your thoughts and comments!

After we finish the current re-read will need to decide which book will be next.  We are building a list of the books that have had the most influence on readers of the blog and listeners to the podcast.  Can you answer the question?

What are the two books that have most influenced you career (business, technical or philosophical)?  Send the titles to

First, we will compile a list and publish it on the blog.  Second, we will use the list to drive future  “Re-read” Saturdays. Re-read Saturday is an exciting new feature that began on the Software Process and Measurement blog on November 8th.  Feel free to choose you platform; send an email, leave a message on the blog, Facebook or just tweet the list (use hashtag #SPaMCAST)!


The next Software Process and Measurement Cast will feature our interview with Clareice and Clyneice Chaney. Clareice and Clyneice provide insights and practical advice into how Agile and contracting can work together.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

Categories: Process Management