Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Connecting the Dots Between Principles and Practices of Project Success

Herding Cats - Glen Alleman - Tue, 04/22/2014 - 00:57

When we hear about all the methods of managing projects, the PMI Body of Knowledge, PRINCE2, home grown and commercial  solutions - always look at them in the light of these 5 Immutable Principles and the 5 Practices then implement the principles.

Principles and Practices of Performance-Based Project Management® from Glen Alleman
Categories: Project Management

Reviews and Inspections: The Basics

 With Formality Come Rules

Reviews and Inspections: With Formality Come Rules

Reviews and inspections are an integral part of building most everything. You find reviews and inspections in manufacturing, in construction and even in publishing. Software development and maintenance is no different. Reviews and inspections can be powerful tools to remove defects before they can impact production and to share knowledge with the team and stakeholders. They are part of a class of verification and validation techniques called static techniques. These techniques are considered static because the system or application being built is not executed. Instead of executing the code, the code or other written deliverables are examined either by people (generally called reviews and inspections) or mechanically by a tool (known as static analysis).  Reviews and inspections can be applied to any product generated as part of the development process, while static analysis can only be applied to code-based products.

Reviews and inspections come in various levels of formality to meet different situations and needs. Informal reviews typically do not follow a detailed written process and results are generally not documented for later review and analysis. They are often used to ensure knowledge sharing and training at a one-on-one level. One classic method used for informal reviews is called the desk check.  In a desk check, a team member emails a deliverable or code to another team member who reviews it and gives them feedback. Another form of informal review is pair programing.

Walkthroughs are a step up the formality ladder.  Walkthroughs are group sessions in which the author takes the group he or she is presenting to through the deliverable.  The attendees of walkthroughs generally include  team members and technical specialists. Walkthroughs can be very informal (an impromptu gathering) or the degree of formality can be increased by requiring meeting preparation and collection of issues and defects. Walkthroughs are used to discover defects, make decisions and to distribute information.

Technical reviews leverage a defined process for defect detection, and include participation by peers, technical experts and often management personnel. They are more formal than the typical walkthrough and are much more formal than desk checks. A trained moderator, who is not the author, generally leads a technical review (to enforce independence) comparing the deliverable to organizational standards. In addition to defect discovery, decision making and information distribution, technical reviews are often used as a formal approval mechanism. For example, I recently observed an organization where all projects go through an architectural review. Technical reviews are a type of technical review usually based on defined organizational standards. The architecture review in the example was based on the organization’s published standard architecture.

Inspections are the most formal of the review and inspection techniques. The most well-known inspection technique is based on the process defined by Michael Fagan of Fagan Reviews.  The inspection process includes highly defined roles such as moderator, author, scribe and reviewer. All inspection processes typically include required pre-work, logging of defects, collection and publication of metrics, formal follow-up procedures and, in many cases, the use of statistical process control techniques. The goal of inspections is to find and remove defects.

Reviews and inspections are highly effective and powerful tools for finding and removing defects from software and other deliverables. Review and inspections are used in all software development and maintenance methods. The type of review and degree of formality is usually a function of the type of project. For example, inspections are almost always used on mission critical applications, such as medical devices and weapons systems, regardless of whether they are using Agile or plan-based techniques. Reviews and inspections remove defects and share knowledge so teams can maximize the value they deliver.


Categories: Process Management

The Code Is NOT Greener On The Other Side Of The Cubicle

Making the Complex Simple - John Sonmez - Mon, 04/21/2014 - 17:40

One of the worst traps we can fall into as software developers is discontentment. It’s really easy to become discontented with our current situation and to want to seek greener pastures elsewhere. I’m not saying that there aren’t necessarily better situations you could seek out, but finding a better job may not be your problem. […]

The post The Code Is NOT Greener On The Other Side Of The Cubicle appeared first on Simple Programmer.

Categories: Programming

This is why Microsoft won. And why they lost.

My favorite kind of histories are those told from an insider's perspective. The story of Richard the Lionheart is full of great battles and dynastic intrigue. The story of one of his soldiers, not so much. Yet the soldiers' story, as someone who has experienced the real consequences of decisions made and actions taken, is more revealing.

We get such a history in Chat Wars, a wonderful article written by David Auerbach, who in 1998 worked at Microsoft on MSN Messenger Service, Microsoft’s instant messaging app (for a related story see The Rise and Fall of AIM, the Breakthrough AOL Never Wanted).

It's as if Herodotus visited Microsoft and wrote down his experiences. It has that same sort of conversational tone, insightful on-the-ground observations, and facts no outsider might ever believe.

Much of the article is a play-by-play account of the cat and mouse game David plays changing Messenger to track AOL's Instant Messenger protocol changes. AOL repeatedly tried to make it so Messenger could not interoperate with AIM and each time Messenger countered with changes of their own. AOL finally won the game with a radical and unexpected play. A great read for programmers. 

For a general audience David's explanation of how and why Microsoft came to dominance and why they lost that dominance is most revealing. It stares directly into the heart of the entropy that brings everything down in the end.

Why Microsoft Won 
Categories: Architecture

Really, Can You Be Agile and Not Disciplined?

NOOP.NL - Jurgen Appelo - Mon, 04/21/2014 - 16:00
Agility versus Discipline

Last week, it happened again. On Facebook I posted a screenshot of the huge checklist that I use as a quality assurance gate for my book chapters. Some people commented that this looked quite disciplined.

Frankly speaking, I don’t see what’s so special about my discipline. If you want to be agile you have to be disciplined!

I don’t see how it can be any different.

The post Really, Can You Be Agile and Not Disciplined? appeared first on NOOP.NL.

Categories: Project Management

Design Your Agile Project, Part 4

If you are thinking of agile as part of a program, each team has to have its own approach to agile. Why? Because each team has its own risks and problems. You don’t need to standardize agile for anyone. If you treat people as if they are adults and explain the principles that you want (working software all the time and expose the interdependencies), provide training or whatever other resources they need, you are likely to get what you want.

That’s why I wrote Design Your Agile Project, Part 1 for teams who are collocated, and can use any approach to agile. Design Your Agile Project, Part 2 is for teams who have many challenges, but are collocated, and can work through those challenges. Design Your Agile Project, Part 3 is for teams who are geographically distributed and have to think about what to do.

In the program, what you need is for each team to deliver, all the time. The program wants as close to continuous delivery as possible. You can reduce the interdependencies—or plan for them. You can certainly expose them.

Feedback is Necessary

Did you see Jason Yip’s tweet (@jchyip) the other day, where he quoted this, “”Large organizations…lose their resilience simply because the feedback mechanisms…have too many layers of delay and distortion.”

This is why you cannot standardize on anything for any team in a program. Why? A program needs resilience. It needs to be able to release early and often. Just because it’s a program does not mean it does not need to be able to obtain feedback.

Each team must know the principles:

  1. You need to see all the work in progress.
  2. You want to flow work through the entire team.
  3. You want to see working software, sooner, rather than later.

Teams use agile and lean principles. Management treats the people on the teams as if they are adults. The teams look at their own issues and design their own approach to agile/lean, always keeping in mind the idea that they need to deliver early and often.

Now, let’s talk about what happens when you want to meld these teams into a program.

Each Team is the Heart of the Program

You might think that things roll down from the program manager or the core team. Not in my world. This is where each team’s autonomy, each team’s ability to make its own decisions about how it implements its approach to agile or lean is key.

The teams are the heart of the program. All of the teams: the core team, the technical teams, the teams that the core team represents.

This is a change from traditional process thinking. It’s a change from traditional program management thinking. This kind of thinking requires that the teams know how to do agile already, and are new to the program, not to agile. This kind of thinking also means that the program manager is a servant leader.

In a program, you will have interdependencies. You want to minimize them. How do you do that? By reducing batch size, the size of each feature. By coordinating the features with a program roadmap. And, by looking at the value of each feature, not its cost (estimate).

That means the teams need to become good at delivering, not estimating. It also means the product owners need to become very good at creating ranked backlogs for the teams, and changing them. It means that the program needs a roadmap that can and does change.

Remember, agile is about the ability to change, because the teams get to done at regular intervals, whether those intervals are iterations or because the teams work in flow.

What If the Teams are New to Agile?

What if you want to have a program with teams that are new to agile or lean? You can do anything on your program. You need to assess your risks. Let’s look at the Cynefin framework to see where your risks might place you, if your teams are new to agile.

CynefinIf your teams are new to agile, and they are all in one physical location, and they know the domain of the product under development, i.e. you are only changing how they are developing, maybe you are just in the Complex part of the framework. You are changing the culture of the program.

However, if you have new-to-agile teams, who don’t know the product domain, who are geographically distributed or dispersed from one another, and you want to transition to agile, do you see that you might be in the Chaotic part of the framework? That you have no way of knowing the risks?

That is much more difficult.

Let me provide you an example. Imagine that you are working with developers in Europe who have a 15-person development team. They have only programmers on their team. They have never worked with testers. Why? They have never seen the need to do so. They have been successful, according to them, up until now.

You are in New York, where the product management is. You know the product domain. The developers? Well, maybe not so much.

Several years ago, I consulted to a company that had this precise organization. They were going to “revolutionize aerospace.” Only the product managers understood the aerospace information. The developers had no domain expertise and they were several timezones away. The developers had worked together before, but had never worked with testers. They had never seen the need. They had never worked on a regulated product before.

When I suggested they had “unknowable unknowns,” and that their risks were higher than they thought they had, the senior management laughed at me. I told them yes, agile was fine, but I thought they should use one- or two-week iterations with kanban boards to expose their bottlenecks.

They ignored my advice. The company went with four-week iterations, spent a pile of money, had no product to show after 18 months. Oh, the developers bandied words such as “refactoring” and “TDD” and “BDD.” What they did was BDUF (Big Design Up Front) because they had no idea what they were doing. The company went under.

What do you do when not everyone knows what “agile” is, when you create a new product, when you are introducing that much cultural change?

Don’t panic! You work in ways to reduce your risk.

Stay tuned for Part 5.

Categories: Project Management

SPaMCAST 286 – Brian Wernham, Agile Project Management for Government

www.spamcast.net

http://www.spamcast.net

Listen to the Software Process and Measurement Cast 286. SPaMCAST 286 features our interview with Brian Wernham, author of Agile Project Management for Government. Agile software development in government does not have to be an oxymoron.

Brian Wernham has more than 30 years of experience in adaptive change program leadership.  He is an independent consultant and works in both the public and private sector.  He has extensive international experience in the USA, UK, Canada, Hong Kong, Germany and offshore development in Bangalore.

By the time that the term ‘Agile leadership’ was first coined, Brian had already been successfully leading iterative, adaptive projects for over 10 years on both sides of the Atlantic.  He works as a hands-on program director and has real-world implementation expertise together with a comprehensive understanding of the related international research.  He has consulted for major strategic international organizations such as Deloitte, PwC, Gartner Group, the National Audit Office in London and Seer Technologies in North Carolina. His comprehensive public sector experience includes the Department for International Development (DFID), the World Bank, the United Nations (Geneva), and local government authorities.

Brian is a Fellow of the Association for Project Management, a Fellow of the BCS and has a MBA from Henley Management College.  He applies adaptive planning approaches as an offshore Yachtmaster and as a keen off-piste skier.  He is currently consulting for the UK Government in London.

Read Brian’s blog, visit his website and of course buy the book!

Get in touch with us anytime or leave a comment here on the blog. Help support the SPaMCAST by reviewing and rating it on iTunes. It helps people find the cast. Like us on Facebook while you’re at it.

Next week we will feature our essay on Scrumban. Scrumban is the combination of agile (Scrum) and lean (Kanban) concepts that can be used to manage projects.

Upcoming Events

StarEast

I will be speaking at the StarEast Conference May 4th – 9th in Orlando, Florida.  I will be presenting a talk titled, The Impact of Cognitive Biases on Test and Project Teams. Follow the link for more information on StarEast.

An ITMPI Webinar!

On June 3 I will be presenting the webinar titled “Rescuing a Troubled Project With Agile.” The webinar will demonstrate how Agile can be used to rescue troubled projects.  Your will learn how to recognize that a project is in trouble and how the discipline, focus, and transparency of Agile can promote recovery. Register now!

Upcoming DCG Webinars:

May 22 11:30 EDT – Agile User Stories
June 19 11:30 EDT – How To Split User Stories
July 24 11:30 EDT – The Impact of Cognitive Bias On Teams

Check these out at http://www.davidconsultinggroup.com

I look forward to seeing or hearing all SPaMCAST readers and listeners at all of these great events!

The Software Process and Measurement Cast has a sponsor.

As many you know I do at least one webinar for the IT Metrics and Productivity Institute (ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI’s mission is to pull together the expertise and educational efforts of the world’s leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.


Categories: Process Management

SPaMCAST 286 – Brian Wernham, Agile Project Management for Government

Software Process and Measurement Cast - Sun, 04/20/2014 - 22:00

Listen to the Software Process and Measurement Cast 286. SPaMCAST 286 features our interview with Brian Wernham, author of Agile Project Management for Government. Agile government does not have to be an oxymoron.

Brian Wernham has more than 30 years of experience in adaptive change program leadership.  He is an independent consultant and works in both the public and private sector.  He has extensive international experience in the USA, UK, Canada, Hong Kong, Germany and offshore development in Bangalore.

By the time that the term ‘Agile leadership’ was first coined, Brian had already been successfully leading iterative, adaptive projects for over 10 years on both sides of the Atlantic.  He works as a hands-on program director and has real-world implementation expertise together with a comprehensive understanding of the related international research.  He has consulted for major strategic international organizations such as Deloitte, PwC, Gartner Group, the National Audit Office in London and Seer Technologies in North Carolina. His comprehensive public sector experience includes the Department for International Development (DFID), the World Bank, the United Nations (Geneva), and local government authorities.

Brian is a Fellow of the Association for Project Management, a Fellow of the BCS and has a MBA from Henley Management College.  He applies adaptive planning approaches as an offshore Yachtmaster and as a keen off-piste skier.  He is currently consulting for the UK Government in London.

Read Brian’s blog, visit his website and of course buy the book!

Get in touch with us anytime or leave a comment here on the blog. Help support the SPaMCAST by reviewing and rating it on iTunes. It helps people find the cast. Like us on Facebook while you’re at it.

Next week we will feature our essay on Scrumban. Scrumban is the combination of agile (Scrum) and lean (Kanban) concepts that can be used to manage projects.

Upcoming Events

StarEast

I will be speaking at the StarEast Conference May 4th – 9th in Orlando, Florida.  I will be presenting a talk titled, The Impact of Cognitive Biases on Test and Project Teams. Follow the link for more information on StarEast.

An ITMPI Webinar!

On June 3 I will be presenting the webinar titled “Rescuing a Troubled Project With Agile.” The webinar will demonstrate how Agile can be used to rescue troubled projects.  Your will learn how to recognize that a project is in trouble and how the discipline, focus, and transparency of Agile can promote recovery. Register now!

Upcoming DCG Webinars:

May 22 11:30 EDT – Agile User Stories
June 19 11:30 EDT – How To Split User Stories
July 24 11:30 EDT - The Impact of Cognitive Bias On Teams

Check these out at www.davidconsultinggroup.com

I look forward to seeing or hearing all SPaMCAST readers and listeners at all of these great events! 

 The Software Process and Measurement Cast has a sponsor.

As many you know I do at least one webinar for the IT Metrics and Productivity Institute (ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI's mission is to pull together the expertise and educational efforts of the world's leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI.

 Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: "This book will prove that software projects should not be a tedious process, neither for you or your team." Support SPaMCAST by buying the book here.

Available in English and Chinese. 

Categories: Process Management

We Can Know the Business Value of What We Build?

Herding Cats - Glen Alleman - Sun, 04/20/2014 - 01:58

In a recent post titled #NoEstimates - Really? there was an interesting comment.

Clearly, the business value of any feature or project can not be known with much certainty in advance of it being implemented. Still, for the purpose of keeping the analysis simple for now, let’s table this issue for a bit.

This is not the case in a governance based organization or a Capabilities Based Planning organization, where the "valuation" of the resulting product, service, or purchased or built product is part of the planning process.

It's a "build to valuation"

Knowing - to some probabilistic level of confidence - what business value or mission fulfillment the project or product will produce is the core of any decision making process. Knowing the cost of the value is about making decisions, analysis of alterntaives, or assessing the trade space

With the "estimated" value and the "estimated" cost for that value, a decision can be made in what is called "analysis of alternatives" in our software intensive domain.

Only by having both estimates - value and  cost of acheiving  that value, their most likely numbers and the probabilistic range of those numbers (measured usually in dollars), can we make that "analysis of alternatives," or "trade space" needed to Govern both the business and the project and products that enable the business to meet its goals.

&

So there are popular myths around the estimating of cost and value discussion, and a few that are just flat out bogus:

  • We can't know the value of our outcomes, until they are done.
    • Start with a Balanced Scorecard strategy approach, where you define the needed value of any work before starting, determine the estimated cost of achieving that value for the busines or mission and do the math of ROI = (Value - Cost) / Cost.
  • No requirement can be confirmed to be correct until the software is complete and put to work.
    • Read any - my favorite The Requirements Engineering Handbook - requirements management book to see if this makes any sense at all.
    • This assumes - the myth that is - that those developing the software really don't know what done looks like in any units of measure meanigful to the decision makers. Requirements definition is the role of Systems Engineering. So if the developers can't know, maybe they need to the Systems Engineers. Don't have Systems Engineers? Now that's a different problem.
    • All the Capabilities Based Planned and Requirements Management processes are based on assessing the value of those requirements before the product or service arrives.
    • Assessing that value is part of all Systems Engineering process using Measures of Effectiveness and Measures of Performance.
    • During the project's execution, Technical Performance Measures and their supporting Quantifiable Backup Data are used to assue that what is being built is meeting the needs of the customer.
  • Making Estimates is a waste of time when we don;t know what the requirements are.
    • This is true.
    • But if you don't know what the requirements are, someone has to pay to find out. This someone is usually the customer. This is the basis of Agile, and it works.
      • But someone has to pay. 
    • Even in the realm where the requirements are vague, there is hopefully some  notion of what Capabilities are needed from the project or product.
    • What are you going to do with the results of the project or the product when it arrives?
    • How much are you willing to pay for those capabilities?
    • Don't have the answers to some level of confidence? - Just set fire to the money now and save all the effort.
  Related articles The "Real" Root Cause of IT Project Failure Resources for Moving Beyond the "Estimating Fallacy" 5 Questions That Need Answers for Project Success Why Johnny Can't Estimate or the Dystopia of Poor Estimating Practices
Categories: Project Management

Teams and Overly Self-interested Behavior

Overly Self-interested Behavior Is Bad For Teams!

Overly Self-interested Behavior Is Bad For Teams!

Hand Drawn Chart Saturday

Teams are an important concept in most IT organizations, regardless of their development philosophy. Philosophies like Agile may put more emphasis on teams, but even in organizations that do not embrace Agile philosophies, teams are important. Dan Ariely in his Ted Talk, “What Makes Us Feel Good About Our Work” suggested that overly self-interested, cynical behavior can negatively impact organizations by reducing their ability to communicate and innovate. The same problem can occur, albeit on a small scale, at the team level. In a recent presentation, a fellow Agile coach described a team engaged in overly self-interested behavior. He described a scenario in an organization that cuts the bottom 10% of all groups annually and stated vision that IT should maximize the value it deliverers to its customers. After losing a popular team member the previous year, the team had decided to make sure that his replacement was given the worst assignments in order to ensure he stayed on the low end of the performance scale in the coming year. Their goal was to ensure that the core team stayed intact during the next review cycle. In their mind, keeping the core team together ensured that they would deliver more value to their internal customers. The behavior of the team attempted to circumvent the idea that adding new and more highly qualified personnel would lead to improved performance. Viewed from the point of view of organizational policy, the whole team was acting in an overly self-interested behavior manner, but from the point of view of core team they are acting rationally and within their interpretation of the rules as seen through IT’s vision of value delivery. The team did not believe that their behavior was at odds with the behavior the organization wanted to incent.

What we perceive as overly self-interested at a team level is based on collective cognitive biases. Biases are powerful psychological filters that affect how both individuals and teams perceive the world around themselves and then guide how they behave. Biases reflect shortcuts in how we interpret and react to stimulus. In many cases, these reactions are valuable, however they can also cause problems (as many shortcuts occasionally can). Understanding how biases impact how individuals and teams perceive the world around them can help teams make better decisions and therefore deliver value more effectively. In the example, the core team had decided to protect itself by defining the new person as an outsider, which allowed them to hold them at arm’s length. In-group favoritism is a typical team level bias that causes teams to favor those inside the team’s boundaries over those perceived to be on the outside. This type of bias negatively affect how outsiders are perceived to perform. Negative perceptions of outsiders will effect whether we listen to their ideas and whether we trust them to deliver value.

Teams need to break overly self-interested patterns of behavior by striving ensure that they are pursuing a goal that is greater than team’s own self-interest. At a project level, product owners need to clearly identify and communicate the overall value proposition, so the team has something greater than themselves to pursue. When behavior goes off track coaching can help to identify issues that the team can’t see and diagnosis themselves. However, in many cases overly self-interested behavior at the team level can be a reflection of poor philosophies at the organization level. Organizational philosophies, like decimation or objectives that foster individual competition rarely support intergroup communication and innovation.


Categories: Process Management

Neo4j: Cypher – Creating a time tree down to the day

Mark Needham - Sat, 04/19/2014 - 22:15

Michael recently wrote a blog post showing how to create a time tree representing time down to the second using Neo4j’s Cypher query language, something I built on top of for a side project I’m working on.

The domain I want to model is RSVPs to meetup invites – I want to understand how much in advance people respond and how likely they are to drop out at a later stage.

For this problem I only need to measure time down to the day so my task is a bit easier than Michael’s.

After a bit of fiddling around with leap years I believe the following query will create a time tree representing all the days from 2011 – 2014, which covers the time the London Neo4j meetup has been running:

WITH range(2011, 2014) AS years, range(1,12) as months
FOREACH(year IN years | 
  MERGE (y:Year {year: year})
  FOREACH(month IN months | 
    CREATE (m:Month {month: month})
    MERGE (y)-[:HAS_MONTH]->(m)
    FOREACH(day IN (CASE 
                      WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) 
                      WHEN month = 2 THEN 
                        CASE
                          WHEN year % 4 <> 0 THEN range(1,28)
                          WHEN year % 100 <> 0 THEN range(1,29)
                          WHEN year % 400 <> 0 THEN range(1,29)
                          ELSE range(1,28)
                        END
                      ELSE range(1,30)
                    END) |      
      CREATE (d:Day {day: day})
      MERGE (m)-[:HAS_DAY]->(d))))

The next step is to link adjacent days together so that we can easily traverse between adjacent days without needing to go back up and down the tree. For example we should have something like this:

(jan31)-[:NEXT]->(feb1)-[:NEXT]->(feb2)

We can build this by first collecting all the ‘day’ nodes in date order like so:

MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day)
WITH year,month,day
ORDER BY year.year, month.month, day.day
WITH collect(day) as days
RETURN days

And then iterating over adjacent nodes to create the ‘NEXT’ relationship:

MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day)
WITH year,month,day
ORDER BY year.year, month.month, day.day
WITH collect(day) as days
FOREACH(i in RANGE(0, length(days)-2) | 
    FOREACH(day1 in [days[i]] | 
        FOREACH(day2 in [days[i+1]] | 
            CREATE UNIQUE (day1)-[:NEXT]->(day2))))

Now if we want to find the previous 5 days from the 1st February 2014 we could write the following query:

MATCH (y:Year {year: 2014})-[:HAS_MONTH]->(m:Month {month: 2})-[:HAS_DAY]->(:Day {day: 1})<-[:NEXT*0..5]-(day)
RETURN y,m,day
2014 04 19 22 14 04

If we want to we can create the time tree and then connect the day nodes all in one query by using ‘WITH *’ like so:

WITH range(2011, 2014) AS years, range(1,12) as months
FOREACH(year IN years | 
  MERGE (y:Year {year: year})
  FOREACH(month IN months | 
    CREATE (m:Month {month: month})
    MERGE (y)-[:HAS_MONTH]->(m)
    FOREACH(day IN (CASE 
                      WHEN month IN [1,3,5,7,8,10,12] THEN range(1,31) 
                      WHEN month = 2 THEN 
                        CASE
                          WHEN year % 4 <> 0 THEN range(1,28)
                          WHEN year % 100 <> 0 THEN range(1,29)
                          WHEN year % 400 <> 0 THEN range(1,29)
                          ELSE range(1,28)
                        END
                      ELSE range(1,30)
                    END) |      
      CREATE (d:Day {day: day})
      MERGE (m)-[:HAS_DAY]->(d))))
 
WITH *
 
MATCH (year:Year)-[:HAS_MONTH]->(month)-[:HAS_DAY]->(day)
WITH year,month,day
ORDER BY year.year, month.month, day.day
WITH collect(day) as days
FOREACH(i in RANGE(0, length(days)-2) | 
    FOREACH(day1 in [days[i]] | 
        FOREACH(day2 in [days[i+1]] | 
            CREATE UNIQUE (day1)-[:NEXT]->(day2))))

Now I need to connect the RSVP events to the tree!

Categories: Programming

Neo4j 2.0.1: Cypher – Concatenating an empty collection / Type mismatch: expected Integer, Collection<Integer> or Collection<Collection<Integer>> but was Collection<Any>

Mark Needham - Sat, 04/19/2014 - 20:51

Last weekend I was playing around with some collections using Neo4j’s Cypher query language and I wanted to concatenate two collections.

This was easy enough when both collections contained values…

$ RETURN [1,2,3,4] + [5,6,7];
==> +---------------------+
==> | [1,2,3,4] + [5,6,7] |
==> +---------------------+
==> | [1,2,3,4,5,6,7]     |
==> +---------------------+
==> 1 row

…but I ended up with the following exception when I tried to concatenate with an empty collection:

$ RETURN [1,2,3,4] + [];
==> SyntaxException: Type mismatch: expected Integer, Collection<Integer> or Collection<Collection<Integer>> but was Collection<Any> (line 1, column 20)
==> "RETURN [1,2,3,4] + []"
==>                     ^

I figured there was probably some strange type coercion going on for the empty collection and came up with the following work around using the RANGE function:

$ RETURN [1,2,3,4] + RANGE(0,-1);
==> +-------------------------+
==> | [1,2,3,4] + RANGE(0,-1) |
==> +-------------------------+
==> | [1,2,3,4]               |
==> +-------------------------+
==> 1 row

While writing this up I decided to check if it behaved the same way in the recently released 2.0.2 and was pleasantly surprised to see that the work around is no longer necessary:

$ RETURN [1,2,3,4] + [];
==> +----------------+
==> | [1,2,3,4] + [] |
==> +----------------+
==> | [1,2,3,4]      |
==> +----------------+
==> 1 row

So if you’re seeing the same issue get yourself upgraded!

Categories: Programming

10 Big Ideas from Mindset: The New Psychology of Success

My parents taught me early on to focus on growth over greatness.

The idea was that while natural ability can take you only so far, it’s things like curiosity, challenges, continuous learning, the power of persistence, taking risks, etc. that would take you further.

They also taught me that if I worried about whether I was naturally good, that I would give up on things where I didn’t start off so great.

It was great advice, even if it wasn’t scientific.

But there is science.

In fact, there’s a lot of science about how choosing a growth mindset over a fixed mindset help people to become the best in their field.  A growth mindset is what actually creates better parents, teachers, coaches, and CEOs.   A growth mindset creates better students, better artists, and even better geniuses.

Why?

Because people with a growth mindset embrace the challenges, struggles, criticisms, and setbacks as a source of growth.

And that’s how they rise above any limitation of “natural” ability.

Teaching, learning, and continuous growth takes them further than relying on talent or fear of taking risks where they might look bad or might not start off so great.

Carol S. Dweck, Ph.D.  wrote an outstanding book on how our mindsets shape us and how they can limit or enable us to realize our potential.

I wrote up my take aways using a “10 Big Ideas from …” style:

10 Big Ideas from Mindset: The New Psychology of Success

I think you'll enjoy the insights and I think you’ll appreciate how you can apply them to work and life.

Categories: Architecture, Programming

Get Up And Code 50: Fitness Life Change With Miguel Carrasco

Making the Complex Simple - John Sonmez - Sat, 04/19/2014 - 15:00

I finally got a chance to interview Miguel Carrasco. I’ve been watching as he has successfully launched a new business around his passion for fitness. Miguel has an awesome perspective on life and is helping many people achieve their fitness goals. Best part, Miguel just recently left his life as a developer to pursue this […]

The post Get Up And Code 50: Fitness Life Change With Miguel Carrasco appeared first on Simple Programmer.

Categories: Programming

Neo4j: Cypher – Creating relationships between a collection of nodes / Invalid input ‘[‘:

Mark Needham - Sat, 04/19/2014 - 07:33

When working with graphs we’ll frequently find ourselves wanting to create relationships between collections of nodes.

A common example of this would be creating a linked list of days so that we can quickly traverse across a time tree. Let’s say we start with just 3 days:

MERGE (day1:Day {day:1 })
MERGE (day2:Day {day:2 })
MERGE (day3:Day {day:3 })
RETURN day1, day2, day3

And we want to create a ‘NEXT’ relationship between adjacent days:

(day1)-[:NEXT]->(day2)-[:NEXT]->(day3)

The most obvious way to do this would be to collect the days into an ordered collection and iterate over them using FOREACH, creating a relationship between adjacent nodes:

MATCH (day:Day)
WITH day
ORDER BY day.day
WITH COLLECT(day) AS days
FOREACH(i in RANGE(0, length(days)-2) | 
  CREATE UNIQUE (days[i])-[:NEXT]->(days[i+1]))

Unfortunately this isn’t valid syntax:

Invalid input '[': expected an identifier character, node labels, a property map, whitespace, ')' or a relationship pattern (line 6, column 32)
"            CREATE UNIQUE (days[i])-[:NEXT]->(days[i+1]))"
                                ^

It doesn’t seem to like us using array indices where we specify the node identifier.

However, we can work around that by putting days[i] and days[i+1] into single item arrays and using nested FOREACH loops on those, something Michael Hunger showed me last year and I forgot all about!

MATCH (day:Day)
WITH day
ORDER BY day.day
WITH COLLECT(day) AS days
FOREACH(i in RANGE(0, length(days)-2) | 
  FOREACH(day1 in [days[i]] | 
    FOREACH(day2 in [days[i+1]] | 
      CREATE UNIQUE (day1)-[:NEXT]->(day2))))

Now if we do a query to get back all the days we’ll see they’re connected:

2014 04 19 07 32 37
Categories: Programming

Brainstorming . . . Well Maybe Not

Crowds are susceptible to groupthink.

Crowds are susceptible to groupthink.

Brainstorming has been a staple in the business world since its creation in the mid-1940s.  Other forces have combined to reinforce the technique such as the mania over crowd sourcing and consensus management styles.  The problem is that the data shows that the technique is not as effective as we all believe, and in some cases can actually be unproductive – such as when groupthink occurs.  There are better ways to generate innovative ideas.

Brainstorming is a process that through free association generates ideas to find a solution or conclusion for a specific problem.  There are a core set of tenants that define brainstorming.  It is generally a group activity that includes a focus on generating as many ideas as possible (quantity), which includes welcoming unusual ideas, combining and improving ideas and the avoidance of criticism.  Great stuff!   However, research[1] has recently questioned whether the process is as effective and efficient as the popularity of the method would suggest.  The research suggests that the reliance on groups and a lack of debate and criticism causes the technique to be less effective at generating creative ideas than other techniques.

Why do you and I care about this topic?  Developing new ideas and innovative solutions is an integral part of software development, enhancement and maintenance.  Using the most effective techniques, or at least knowing what the most effective techniques, is of more just of academic interest.

Groupthink happens when the desire for harmony in a group overrides a realistic appraisal of alternatives.  The after effects of a groupthink gone wrong might include the passive aggressive comment: “I knew this wouldn’t work but did not say anything.”  “Idea gravity wells” and the forbearance of criticism help to create the problem.

Brainstorming and other group techniques for generating ideas leverage the diversity of thought a well-defined group[2] can create. Groups that allow a single powerful individual to dominate can create an idea gravity well that curtails innovation or green-field thinking as new ideas can’t escape the status quo. The problem is that conclusions that have been influenced by groupthink may not hold up in the bright light of the marketplace.

The second criticism is the lack of immediate feedback.  Forbearing debating and criticizing ideas, like a critique in art school, avoids exploring the depths of an idea and deciding quickly what is relevant.  Critiquing ideas makes sure the good parts are honed and built upon and the bad stuff either refined or sloughed off.   Debate, discussion and criticism create a platform to provide immediate feedback, allowing a group to reshape the idea being examined rather than building on a poor base.

I have been experimenting with and honing a process that attacks two of the major criticisms of brainstorming.  The process begins by making sure you have the right team (see the well-formed team footnote).  I strongly suggest using video if you have remote participants.  Make sure you have someone that will act as the scribe and someone that will also act as the moderator for the session.  Second, provide the “team” with the topic being explored and all background information before the planned session.   Each team member is responsible for reviewing the data and developing a list of their best ideas . . . individually.  Generating the initial set of ideas helps to avoid the possibility of groupthink or idea gravity well occurring directly out of the box.  Note: The list of ideas from each person is the price of admission; lack of preparation should bar a potential participant from the session.

The ideation session begins with a round-robin presentation of ideas from each participant with discussion, debate, criticism and refinement (no fisticuffs or off-topic criticisms).  The round-robin presentation generally continues until everyone has put at least one idea on the table or until the initial set of ideas are used up.  I allow and encourage participants to switch to more of free-for-all for laying out and discussing ideas as soon as everyone has completed laying out at least one idea.  Get one idea on the table from each person makes sure all of the participants understand that they have permission to participate.

I allow the process to continue until team members’ lose the will to live, run out of ideas or reach a consensus on an idea or idea set.  I believe that the facilitator or moderator should push a team at least slightly past what is comfortable in order to generate potentially extraordinary results.  Pushing past what is comfortable is one of the reasons I ask interviewees on my podcast for two ideas to solve the problem presented at the end of the interview rather than just one, which would be far more comfortable (note off the record: many interviewees have indicated that one response to the “two things” question would be far easier).

My wife suggests that after the session is complete, give everyone twenty four hours to chew on the topic and then reconvene to make sure that reflection has not provided tweaks to make the solution or solution set better.  Remember that not everyone thinks at the same rate as everyone else.

Why do we think brainstorming works?  We all have experience with the process.  It is comfortable, it is safe and in many cases it generates good ideas . . . just not the best ideas possible.  To brainstorm or not to brainstorm, that is the question, perhaps not exactly Shakespeare, but relevant nevertheless.  As we gain a better understanding of what works and why, when it comes to generating innovative ideas isn’t it time to put aside preconceived notions based on hearsay?  The data from academia suggests that what we know is based on just that — hearsay.  Don’t let data get in the way, you might say; we believe what we believe.  This week I announced to a group that brainstorming was not as effective as other techniques in terms of generating innovative ideas.  You would have thought that I had insulted their children.  Everyone has a story about successes using brainstorming.  Those successes may well have helped us get out of tight situations, but because of those successes it is hard to acknowledge that there are better ways to generate solutions and new ideas.  This is true whether we are discussing idea generation, software development or Marigolds and Moonflowers (something absolutely unrelated). Brainstorming may still have a place at the innovation table, but it should no longer sit at the head of the table.  There are better ways to generate creative solutions and now is not the time to leave creative ideas on the table . . . maybe it is never time to leave creative ideas on the table!

[1] There are several sources of research on brainstorming including:
http://www.cpsb.com/resources/downloads/public/302-Brainstorm.pdf
http://www.cbsnews.com/8301-505125_162-30640906/the-case-against-brainstorming/

http://www.tcp-uk.co.uk/creative_articles_nemeth.htm

[2] Well-formed means that only the minimum number of people should participate and that those that participate include a range of experience and backgrounds.


Categories: Process Management

Why Projects Fail, No Matter the Domain

Herding Cats - Glen Alleman - Fri, 04/18/2014 - 23:07

There are endless reasons for why projects fail. Some are correct, some are bogus, some are down right nonsense. Into this list I'd like at add my opinion, informed by hands on experience on software intensives programs in defense, energy, bio-pharma, commercial products, and enterprise IT.

It starts and ends, with the

Failure to know what DONE looks like in units of measure meaningful to the decision makers

The starting point for the description of DONE is the clear and concise statement of the Needed Capabilities for the resulting project that produce value for the stakeholders. These capabilities are stated in Measures of Effectiveness.

These capabilities state what the results from the project will do for the business or mission when they are available. The planned availability date is part of their capability. Only then come the requirements, then planning for the delivery of the technical and operational components that enable these requirements. Then several more enablers of projects success are needed.

All these activities, outcomes, processes, methods are encapsulated in the paradigm of Systems Engineering. The picture below are the notional elements of Systems Engineering. This picture is from the FAA Systems Engineering Handbook. This handbook is for the National Airspace System (NAS), a software intensive program to upgrade the exiting software, hardware, and processes. Other agencies and business have similar pictures.

SE Elements

  • Synthesis - Transforms requirements into physical solutions.
  • Functional Analyses - describes the functional characteristics that are used to derive the products or services of the system resulting from the project
  • Requirements Management - identifies and manages the requirements that describe the desires capabilities of the project
  • Interface Management - identifies and manages the interactions between components of the system or interactions with external systems
  • Integrated Technical Planning - Plans the projects efforts and products
  • Speciality Engineering - Analyzes the system, requirements, functions, solutions, and/or interfaces using specialized skills and tools. Assists in the derivation of requirements, synthesis of solutions, selection of alternatives, and validation and verification of requirements.
  • Integrity of Analyses - Ensures that the analyses provide the required level of fidelity and accuracy.
  • Lifecycle Engineering - Identifies and manages requirements for system lifecycle attributes, including real estate management, deployment and transition, integrated logistics support, sustainment / technology evolution, and disposal.
  • Risk Management - Identifies, analyzes, and manages the uncertainties of achieving program requirements by developing strategies to reduce the severity or likelihood of those uncertainties.
  • Trade Studies - assists decision making by analyuzing selecting the best-balanced solutions to match requirements that enable the capabilities
  • Configuration Management - Establishes and maintains consistency and manages change in the system performance, functional, and physical attributes.
  • Verification and Validation - Determines if system requirements are correct. Determines that the solution meets the validated requirements.

What Does All This Mean?

When we hear our customers don't know what they want. Or, our customers don't know the target budget for the work they are asking us to do for them. What they are saying is I don't know what DONE looks like in an tangible way that can be used to measure the performance of the project.

So one of two things has to happen when this is the case...

  1. The customer has to spend money to find out what they actually want. This is done all the time on our larger enterprise, defense, and space programs. It's paying to explore. It's establishing the credibility of the capabilities by spending money to do so
  2. Prepare for project failure.

So when we here that famous - infamous actually - phrase let's explore. Ask a serious question, and demand a serious answer ...

Who's pay for us to explore to discover what we should have already know? 

Don't get an answer? Run away from that project, it's going to be a failed project.

 

Categories: Project Management

Unicode

Eric.Weblog() - Eric Sink - Fri, 04/18/2014 - 19:00

(This entry is part of a series. The audience: SQL Server developers. The topic: SQLite on mobile devices.)

Well, Actually

First, go read this blog entry by Miguel de Icaza. Right now. I'll wait.

Welcome back. Now let me apologize. I don't want to be a pedantic jerk who quibbles about minor details. But the topic here is Unicode, so there really is no other way.

All Unicode, all the time

The relevant difference with SQLite is easy to describe:

  • In the world of Windows and SQL Server, you have all kinds of possible code pages.

  • In SQLite, everything is Unicode.

But if you don't have much context on these issues, I haven't really told you much. Let's go further.

But let's not go too far

I don't want to rewrite articles that have already been written quite well. So you should probably also go read this blog entry by Joel Spolsky. I'll be here when you get back.

OK, now let's get started

SQL Server has two basic ways you store text:

  • You can use the char/varchar types, which can be used with one of several collations, each of which implies a specific code page, which implies a specific character encoding.

  • Or you can use the nchar/nvarchar types (note the extra 'n'), which are Unicode.

SQLite has no such distinction. All text in SQLite is Unicode.

What the heck is Unicode again?

It's a character set: a collection of characters, each with a number that can be used to refer to it.

More specifically, it's the only character set which is [trying to be] complete. If you choose any character set or encoding which is not Unicode, there will be characters you cannot use.

And what's an encoding?

Saying that SQLite uses Unicode doesn't tell you how the text is actually represented. Unicode is not an encoding. It is more abstract than that. There are lots of different ways of representing Unicode as bytes.

Microsoft's sad history with Unicode

In the Windows and SQL Server world, there is a long history of encoding Unicode in 16-bit-ish ways. Originally, this was UCS-2, a simple encoding which represents each character as a 16-bit number. But then the Unicode consortium realized that 16 bits are not enough, so they expanded the space to 32 bits. This left Microsoft in an awkward spot. UCS-2 is a fundamentally defective encoding of Unicode, since there are many characters in Unicode that simply cannot be represented.

If the goal of Unicode is to be complete, it is reasonable to say that, well actually, UCS-2 is not Unicode.

The conceptual replacement for UCS-2 is to use 32 bits for every character. This encoding is called UCS-4 or UTF-32. But now the wasted space for storing a simple English string is getting out of hand. Switching the complete works of Arthur Conan Doyle from ASCII (which is also an encoding) to UCS-4 would take four times as much space.

Gradually, the world seems to be adopting UTF-8 as the most popular Unicode encoding. This is a variable width encoding. Sometimes a single character is represented with just one byte. Sometimes it needs more. That's very unfortunate, but the only fixed width alternative is UCS-4, which is also very unfortunate. Choose which problem you prefer, but keep in mind that almost everybody has chosen to accept the problems of UTF-8.

But Microsoft has so much history with UCS-2 that transitioning everything to UTF-8 would be really hard. So they have been moving from UCS-2 to UTF-16, which is basically a variable width encoding built around a 16-bit unit instead of an 8-bit unit. UTF-16 is approximately the worst correct way of representing Unicode, unless you have invested billions of dollars in the fundamentally broken UCS-2, in which case UTF-16 is a pretty awesome way out of the mess you ended up in.

Just remember that if you're going out tonight to a club for pedantic nerds and you want to impress someone, you've got to keep the terminology straight:

  • Unicode is an abstraction, not an encoding, not a code page, not a data format, and not a font.

  • Saying your text is Unicode says nothing about how it is represented. It might be UTF-8. It might UTF-16. It might be code point numbers handwritten on Post-It notes stuck on the wall. All of these are valid representations of Unicode.

  • If you ever say anything to suggest that you think Unicode is 16 bits per character, you will be identified as clueless.

  • If you say that your text is stored in Unicode, you are not entirely incorrect, but people will wonder about whether you really know the difference between Unicode and the encodings of same.

SQLite

SQLite always uses Unicode to represent text.

(Hopefully you are now screaming at me saying, "Yeah, but which encoding?!?")

The best answer to this question is: SQLite uses UTF-8. Forget about everything else.

A more correct answer to this question is: SQLite uses UTF-8 but also supports UTF-16.

Either way, there is no distinction between char and nchar. There is no way to save storage space in a column by realizing that you only use lower case english characters so it's safe to use char instead of nchar. There are no code pages. There is no way to move your Shift JIS data into SQLite without converting it to Unicode (or storing it as blobs, I suppose).

Summary

Microsoft has done a lot of things right, but its history with Unicode is very unfortunate. And it's not entirely their fault. They adopted Unicode early and it changed underneath them.

With respect to its emphasis on UTF-8, SQLite is far more typical of most non-Microsoft software today.

 

Stuff The Internet Says On Scalability For April 18th, 2014

Hey, it's HighScalability time:


Scaling to the top of "Bun Mountain" in Hong Kong
  • 44 trillion gigabytes: size of the digital universe by 2020; 6 Times: we have six times more "stuff" than the generation before us.
  • Quotable Quotes:
    • Facebook: Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB.
    • @windley: The problem with the Internet of Things is right now it’s more like the CompuServe of Things
    • Chip Overclock: If you want to eventually generate revenue, you must first optimize for developer productivity; everything else is negotiable.
    • @igrigorik: if you are gzipping your images.. you're doing it wrong: http://bit.ly/1pb8JhK  - check your server configs! and your CDN... :)
    • Seth Lloyd: The arrow of time is an arrow of increasing correlations.
    • @kitmacgillivray: When will Google enable / require all android apps to have full deep search integration so all content is visible to the engine?
    • Neal Ford: Yesterday's best practices become tomorrow's anti-patterns.
    • Rüdiger Möller: just made a quick sum up of concurrency related issues we had in a 7-15 developer team soft realtime application (clustered inmemory data grid + GUI front end). 95% of threads created are not to improve throughput but to avoid blocking (e.g. IO, don't block messaging receiver thread, avoid blocking the event thread in a GUI app, ..).
    • Ansible: When done correctly, automation tools are giving them time back -- and helping out of this problem of needing to wear many hats.

  • Amazon and Google are in an epic battle to dominate the cloud—and Amazon may already have won: If Amazon’s entire public cloud were a single computer, it would have five times more capacity than those of its next biggest 14 competitors—including Google—combined. Every day, one-third of people who use the internet visit a site or use a service running on Amazon’s cloud.

  • What books would you select to help sustain or rebuild civilization? Here's Neal Stephenson’s list. He was too shy to do so, but I would certainly add his books to my list. 

  • 12-Step Program for Scaling Web Applications on PostgreSQL. Great write up of lessons learned by wanelo.com that they used to sustain 10s of thousand of concurrent users at 3K req/sec. Highly detailed. 1) Add more cache, 2) Optimize SQL, 3) Upgrade Hardware and RAM, 4) Scale reads by replication, 5) Use more appropriate tools, 6) Move write heave table out, 7) Tune Postures and your File System, 8) Buffer and serialize frequent updates, 9) Optimize DB Schema, 10) Shard busy tables vertically, 11) Wrap busy tables with services, 12) Shard services backend horizontally.

  • Is this a soap opera? It turns out Google and not Facebook is buying Titan Aerospace, makers of high flying solar powered drones. Google's fiber network would make a great backbone network for a drone and loon powered wireless network, completely routing around the telcos.

  • Building Carousel, Part I: How we made our networked mobile app feel fast and local. Dropbox changes to an eventualy consistent / optimistic replication syncing model to make their app "feel fast, responsive, and local, even though the data on which users operate is ultimately backed by the Dropbox servers."  Lesson: don’t block the user! Instead of requiring changes to be propagated to the server synchronously. Local and remote photos are treated as equivalent objects. Actions take effect immediately locally then work there way out globally. Changes are used to stay consistent. A fast hash is used to tell what photos have not been backed up to dropbox. Remote operations happen asynchronously.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

EVM World Coming Soon

Herding Cats - Glen Alleman - Fri, 04/18/2014 - 15:04

Speaking on Techncial Performance Measures and conducting workshop on same topic with Mr. Kranz and Tom Coonce of IDA.

Categories: Project Management