Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Work-Life Fusion, not Work-Life Balance

NOOP.NL - Jurgen Appelo - Mon, 07/14/2014 - 14:44
Work-Life Fusion

Sometimes people ask me, “What do you mean with work-life fusion? How is that different from work-life balance?”

Well…

Work-life balance means using Twitter for your work-life and Facebook for your private life. Instead, I have friends, family, and business contacts everywhere! My 13th happy anniverselfie on Facebook was liked by 86 people, half of them were friends and family, and the other half were business contacts. I call that work-life fusion.

The post Work-Life Fusion, not Work-Life Balance appeared first on NOOP.NL.

Categories: Project Management

The Failure of Open Loop Thinking

Herding Cats - Glen Alleman - Mon, 07/14/2014 - 14:12

Screen Shot 2014-07-13 at 9.18.12 PMWhen there are charts showing an Ideal line or a chart of samples of past performance - say software delivered - in the absence of a baseline for what the performance of the work effort or duration should have been, was planned to be, or even better could have, this is called Open Loop control.

The issue of forecasting the Should, Will, Must cost problem has been around for a long time. This work continues in DOD, NASA, Heavy Construction, BioPharma, and other high risk, software intensive domains.

When we see graphs where the baseline to which the delays or cost overages are compared and those baselines are labeled Ideal, (like the chart below), it's a prime example of How to LIe With Statistics, Darrell Huff, 1954. This can be over looked in an un-refereed opinion paper in a IEEE magazine, or a self-published presentation, but a bit of homework will reveal that charts like the one below are simply bad statistics.

Screen Shot 2014-07-13 at 9.26.15 PM

This chart is now being used as the basis of several #NoEstimates presentations, which further propagates the misunderstandings of how to do statistics properly.

Todd does have other papers that are useful Context Adaptive Agility is one example from his site. But this often used and misused chart is not an example of how to properly identify problems with estimates,

Here's some core issues:

  • If we want to determine something about a statistical process, we of course need to collect data about that process. This data is empirical - much misused term itself - to show what happened over time. A time series of samples.
  • To computer a trend, we can of course draw a line through population of data, like above.
  • Then we can compare this data with some reference data to determine the variances between the reference data and the data under measurement.

Here's where the process goes in the ditch - literally.

  • The reference data has no basis of reference. It's just labeled ideal. Meaning a number that was established with no basis of estimate. Just this is what was estimated, now let's compare actuals to it and if actuals matched the estimate' let's call it ideal.
  • Was that ideal credible? Was it properly constructed? What's the confidence level of that estimate? What's the allowable variance of that estimate that can still be considered OK (within the upper and lower limites of OK)? Questions and their answers are there. It's just a line.

We can use the ne plus ultra put-down of theoretical physicist Wolfgang Pauli's "This isn't right. It's not even wrong."  As well the projects were self-selected, and like the Standish Report, self-selected statistics can be found in the How to Lie book

It's time to look at these sort of conjectures in the proper light. They are Bad Statisics, and we can't draw any conclusion from any of the data, since the baseline to which the sampled values are compared Aren't right. They're not even wrong."  We have no way of knowing why the sampled data has a variance from the ideal the bogus ideal

  • Was the original estimate simple naĂŻve?
  • Was the project poorly managed?
  • Did the project change direction and the ideal estimate never updated?
  • Were the requirements, productivity, risks, funding stability, and all the other project variables held constant, while assessing the completion date? if not the fundamental principles of experiment desgin was violated. These principles are taught in every design of experiments class in every university on the planet. Statistics for Experimenters is still on my shelf. George Box as one of the authors, whose often misused and hugely misunderstood statement all models are wrong, some are useful.

So time to stop using these charts and start looking for the Root Causes for the estimating problem.

  • No reference classes
  • No past performance
  • No parametric models
  • No skills or experience constructing credible estimates
  • No experience with estimating tools, processes, databases (and there are many for both commerical and government software intensive programs).
  • Political pressure to come up with the right number
  • Misunderstanding of the purpose of estimating - provide information to make decisions.

A colleague (former NASA cost director) has three reasons for cost, schedule, and technical shortfalls

  1. They didn't know
  2. They couldn't know
  3. They didn't want to know

Only the 2nd is a credible reason for project shortfalls in performance.

Without a credible, calibrated, statistically sound baseline, the measurements and the decisions based on those measurements are Open Loop.

You're driving your car with no feedback other than knowing you ran off the road after you ran off the road, or you arrived at your destination after you arrived at your destination.

Just like this post Control Systems - Their Misuse and Abuse

 

Related articles How to "Lie" with Statistics How to Fib With Statistics Control Systems - Their Misuse and Abuse Seven Immutable Activities of Project Success
Categories: Project Management

Software Development is Like Play Writing

Herding Cats - Glen Alleman - Sun, 07/13/2014 - 23:23

WaitingForGodotI stole this idea of this blog from Stephen Wilson's post of a similar name. And like all good borrows I've added, subtracted, and made some changes, because everything is a remix.

I don't know Stephen, but his post is provacutative. I'm assigned to a client outside my normal Defense Department and NASA comfort zone. The client needs a Release Management System integrated with a Change Control Board. Both are the basis of our defense and space software world. This client is trying to use agile, but has little in the way of the discipline needed to actually make it work.

The SWDev is like play writing is a beautiful concept that can be applied to the choas of the new client and also connected back to our process driven space and defense, which by the way makes heavy use of agile, but without all the drama of the it's all about me developer community.

Let's start here:

In both software and play writing, structure is almost entirely arbitrary. Because neither obey the laws of physics, the structure of software and plays comes from the act of composition. A good software engineer will know their composition from end to end. But another programmer can always come along and edit the work, inserting their own code as they see fit. It is received wisdom in programming that most bugs arise from imprudent changes made to old code.

It turns out of course neither of those statements is correct in the sense we may think. There is the act of composition, but that composition needs a framework in which to be developed. Otherwise we wouldn't know what we're watching or know what we're developing until it is over. And neither is actually how plays are written or software is written. It may be an act of composition, but it is not an act of spontaneous creation.

Let's start with play writing. It may be that the act of writing a play where the structure is entirely arbitrary is possible, but it's unlikely that would be a play you'd pay money to see. A Harold Pinter play may be unstructured Waiting for Gadot may be unstructured, but that's not really how plays are written. They follow a structured approach - there is a law of physics for play writing.

  • You need characters - a protagonist and the supporting cast
  • You need a setting - where are we and what's going on that supports the story
  • You need some sort of imbalance between the characters, the setting
  • You need some way to restore the imbalance or leaving it hanging
  • You need to engage your audience in some way that will resonant with the personal feelings

That's about the story of the play. To actually write a play, here's a well traveled path to success. These guidelines are for the outcome of the writing effort starting in the beginning.

  • A rough title is needed to anchor the play in the readers mind.
  • An action statement, describing what  the characters are going to do in the play, as a group, as individual. How are they going to change and who changes
  • The form of the play describing the organization of the characters, the situation, the environment. How does the play's action relate to the emotional qualities of the characters and most importantly to the audience.
  • The circumstances of the time and place of the action and other important conditions.
  • The subject as an informational platform.
  • The characters of course, describing what are the forces driving them and their relationships with each other, the circumstance, and the environment.
  • The conflict. It's boring watching a play that is boring. What are the obstacles that have to be overcome by the characters?
  • The meaning for the play. What is the point of view? What are the key thoughts for the play as a whole and the characters in the play?
  • The style of the dialogue, the composition and manner of this dialogue?
  • And finally a schedule for writing the play. When will it be done?

When we talk about writing software there is a similar story line

  • Who are the protagonist? - they're the end users of the software.
  • What is the setting? - there is a stated need for something new.
  • What is the imbalance? - something is not getting done and it needs to be improved.
  • How do we restore the balance? - by closing the gaps between the imbalance and the balance with a software solution.

The story line is the basis of Capabilities Based Planning. With the capabilities, the requirements can be elicited. From those requirements, decisions can be made for what order to deliver them to produce the best value for the business or the mission. 

This process is about decision making. And decision making uses information about the future. This future information comes many times from estimates about the future. 

 

 

 

Related articles Agile Requires Discipline, In Fact Successful Projects Require Discipline People, Process, or Tools
Categories: Project Management

SPaMCAST 298 – Brian Federici, Continuous Integration, A Practitioners

Listen to SPaMCAST 298

SPaMCAST 298 features our interview with Brian Federici.  Brian discussed continuous integration and nearly continuous delivery from a practitioner’s point of view. Continuous integration and delivery are at the heart of fast, better and higher customer satisfaction promised by Agile methods however, in many environments, continuous integration and nearly continuous delivery are still a merely interesting theory.  Brian discussed how these concepts affect how work is delivered.

Brian has been programming professionally since 2003, focusing primarily on C# and Web technologies.  He’s a big proponent of test-driven and behavior-driven development.  He has contributed to the NuGet open source project as well as created some of his own.  He is also an active member of Agile Philly, Philly .NET, Philly GameWorks, and Philly Application Lifecycle Management groups.  In his free time, he enjoys video games, drumming, tennis, and dodge ball. You can interact with Brian in many different ways!

Personal:
http://brian-federici.com
http://twitter.com/brianfederici
http://www.extra-life.org

Code:
https://bitbucket.org/mrcode
https://gist.github.com/benerdin

Groups:
http://www.agilephilly.com/
http://phillydotnet.org/
http://phillygameworks.org/
http://www.meetup.com/Philly-Application-Lifecycle-Management-User-Group/

 

Next week we will feature our essay on Systems Thinking.  Many process improvement programs falter when, despite best efforts because they don’t improve the overall performance of IT. The impact of fixing individual processes can easily get lost in the weeds, the impact overtaken by the inertia of the overall systems. Systems Thinking is a way to view the world, including organizations, from a broad perspective that includes structures, patterns, and events, rather simply being influence by a single event.

 

Upcoming Events

Upcoming DCG Webinars:

July 24 11:30 EDT – The Impact of Cognitive Bias On Teams
Check these out at www.davidconsultinggroup.com

I will be attending Agile 2014 in Orlando, July 28 through August 1, 2014.  It would be great to get together with SPaMCAST listeners, let me know if you are attending.

I will be presenting at the International Conference on Software Quality and Test Management in San Diego, CA on October 1.  I have a great discount code!!!! Contact me if you are interested!

I will be presenting at the North East Quality Council 60th Conference October 21st and 22nd in Springfield, MA.

More on all of these great events in the near future! I look forward to seeing all SPaMCAST readers and listeners that attend these great events!

The Software Process and Measurement Cast has a sponsor.

As many you know I do at least one webinar for the IT Metrics and Productivity Institute(ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI’s mission is to pull together the expertise and educational efforts of the world’s leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

 

 


Categories: Process Management

SPaMCAST 298 – Brian Federici, Continuous Integration, A Practitioners View

Software Process and Measurement Cast - Sun, 07/13/2014 - 22:00

Listen to SPaMCAST 298

SPaMCAST 298 features our interview with Brian Federici.  Brian discussed continuous integration and nearly continuous delivery from a practitioner’s point of view. Continuous integration and delivery are at the heart of fast, better and higher customer satisfaction promised by Agile methods however, in many environments, continuous integration and nearly continuous delivery are still a merely interesting theory.  Brian discussed how these concepts affect how work is delivered.

Brian has been programming professionally since 2003, focusing primarily on C# and Web technologies.  He's a big proponent of test-driven and behavior-driven development.  He has contributed to the NuGet open source project as well as created some of his own.  He is also an active member of Agile Philly, Philly .NET, Philly GameWorks, and Philly Application Lifecycle Management groups.  In his free time, he enjoys video games, drumming, tennis, and dodge ball. You can interact with Brian in many different ways!  

Personal:
http://brian-federici.com
http://twitter.com/brianfederici
http://www.extra-life.org

Code:
https://bitbucket.org/mrcode
https://gist.github.com/benerdin

Groups:
http://www.agilephilly.com/
http://phillydotnet.org/
http://phillygameworks.org/
http://www.meetup.com/Philly-Application-Lifecycle-Management-User-Group/

 

Next week we will feature our essay on Systems Thinking.  Many process improvement programs falter when, despite best efforts because they don't improve the overall performance of IT. The impact of fixing individual processes can easily get lost in the weeds, the impact overtaken by the inertia of the overall systems. Systems Thinking is a way to view the world, including organizations, from a broad perspective that includes structures, patterns, and events, rather simply being influence by a single event. 

Upcoming Events

Upcoming DCG Webinars:

July 24 11:30 EDT – The Impact of Cognitive Bias On Teams
Check these out at www.davidconsultinggroup.com

I will be attending Agile 2014 in Orlando, July 28 through August 1, 2014.  It would be great to get together with SPaMCAST listeners, let me know if you are attending.

I will be presenting at the International Conference on Software Quality and Test Management in San Diego, CA on October 1.  I have a great discount code!!!! Contact me if you are interested!

I will be presenting at the North East Quality Council 60th Conference October 21st and 22nd in Springfield, MA.

More on all of these great events in the near future! I look forward to seeing all SPaMCAST readers and listeners that attend these great events!

The Software Process and Measurement Cast has a sponsor.

As many you know I do at least one webinar for the IT Metrics and Productivity Institute(ITMPI) every year. The ITMPI provides a great service to the IT profession. ITMPI’s mission is to pull together the expertise and educational efforts of the world’s leading IT thought leaders and to create a single online destination where IT practitioners and executives can meet all of their educational and professional development needs. The ITMPI offers a premium membership that gives members unlimited free access to 400 PDU accredited webinar recordings, and waives the PDU processing fees on all live and recorded webinars. The Software Process and Measurement Cast some support if you sign up here. All the revenue our sponsorship generates goes for bandwidth, hosting and new cool equipment to create more and better content for you. Support the SPaMCAST and learn from the ITMPI.

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.

 

Categories: Process Management

Better than something or someone or just good

Gridshore - Sun, 07/13/2014 - 15:56

Recently I saw some tweets about the possibility to finally have a website to express your hate for Eclipse. Since I am a strong believer in another tool than eclipse and I really don’t want to work with eclipse, at first I thought this was funny. Than I was checking some news posts and found a post on apple insider where they show a Samsung advertorial that bashes the iPhone battery life. Again, funny if you like samsung or you think android is better than the iPhone. Maybe even funny if you still love the iPhone.

These tweets and sources of information made me think about marketing of products, opinions of people and sometimes company strategies. As a company, do you want to prove you are better than someone else, or do you want to prove that you are really good. Personally I do not really care if I am better than someone else, I only care if I am good or maybe even superb.

Does it really work to be better than … ? Was Argentina better than the Dutch just a week a go? If not, like so many say, did it help the Dutch? Better than is an interesting concept in soccer. If your soccer teams never wins the championship, you as a fan still think they are the best. I get the same with my local soccer team, every parent thinks his kid is better than the other kids. So better than is incredibly subjective. If you say you create better software, does it sell better? What is good software? If you are into software quality you probably have a few tools to calculate the quality of software. Are you a better programmer if the lines of code you write have shorter length? Are you a better software developer if you use another tool than eclipse? I don’t really care. Use eclipse if you want to, use something else if you can afford it.

I like to make fun of people that use eclipse just like I make fun of people using linux or windows. But in the end, I don’t really care. I know people using windows or linux, working on eclipse can make better software than I can. I do not want to work for a company that tries to sell projects only by stating that they are better than someone else. I want my company to tell customers about what we are good at.

So if you are in the hating or Better than business put a hate comment on this blog, if not have a good day.

The post Better than something or someone or just good appeared first on Gridshore.

Categories: Architecture, Programming

WorldCup 2014 Retrospective: The Magic about Mindset and Leadership

Xebia Blog - Sun, 07/13/2014 - 13:56

This weekend preparing this bjohan-cruijff-hollande-1974logpost, I ran into a brilliant quote from Johan Cruijff. At a conference a few years ago for the Dutch local government, he told a great story about a talented blind golfer, Ronald Boef he played golf with.  Despite his handicap, Ronald Boef played his best golf in difficult mental circumstances like playing balls over a big pond or consistent putting. The conclusion of Johan Cruijff: "Ronald doesn’t “see" the problems, he is only focussing on the next target. He thinks from a positive mindset".   I couldn’t agree more.  In my opinion, this is one of the fundamentals behind eXtreme Manufacturing (XM) and the reason why the Dutch team didn’t made it through the WorldCup finals.

Like many consultants, topsport is an inspiring source for me.  Almost every day I show or tell stories from great sport coaches like Marc Lammers or Johan Cruijjff.   Like every major sports event, also this WorldCup in Brasil contained some interesting lessons for me I wanted to share with you.

The Big Question: You can have the best individual team members but still not be able to perform.  Why?

 3rd Place Playoff - 2014 FIFA World Cup Brazil

Top Team Ingredient #1: Mindset

The defeat of Spain against the Netherlands, the glorious win of Germany over Brazil showed having fun, faith and determination pay off and a lack of these ingredients will bring you in a lot of trouble.   Until the penalty series of the semi-finals, the right side of this recipe also worked for the Dutch squad. Now, penalty series are for no one a fun exercise, which only leaves faith and determination.   Unlike the previous penalty series against Costa Rica, the Dutch team had no faith in their keeper as a penalty-killer which directly effected the teams determination. They became more hesitant and aware of what could happen when missing a penalty.  Yes, Ronald Boef probably would have taken the penalties better than the Dutch team did against Argentina.. ;-)

Top Team Ingredient #2: Leadership

Like Johan Cruijjf stated in the same video, the leader on the pitch should be 100% concentrated on every detail and also (in my words) be the natural leader of the team, coaching them in keeping the spirit up and giving them enough room “to grow".  Despite his great qualities as a football-player, as a captain Robin van Persie was obviously not the natural leader of the team. Arjan Robben was. The natural leadership of Arjan Robben in combination with his determination was an important reason why The Netherlands were able to regain their motivation and pull off a highly respected 3rd place in this WorldCup.

In my opinion, a high performing team should always have a natural leader.  The options:

  1. A formal leader with natural leadership qualities is the perfect combination.
  2. A formal leader without natural leadership qualities but able to delegate this to another team member is also okay.
  3. A formal leader without natural leadership qualities and ignoring don’t having this competence, is bad news for the team, the team’s environment but above all, for the formal leader himself.

For the new coach van the Dutch team, Guus Hiddink, it will be a challenge convincing Robin van Persie to step back as the 1st captain after nominating Arjan Robben.  Robin van Persie should keep one thing in mind here:  no one is doubting his qualities as a top world class striker.  As a natural leader however, he is not that world class.  Trying to be one is effecting his performance as a world class striker and that would in the end be a disappointment for his supporters but above all, for Robin van Persie himself.

What does this imply for Leadership within organizations?

Leadership, especially natural leadership, is crucial for having highly motivated and productive teams.  The team stays motivated and focussed on their goal.

How ever, a lot of employees are still instrumentally “nominated” to become a coach or manager without having any leadership skills.  In my opinion, natural leadership is something you can’t gain by nomination or just by learning it.  You can improve it, but there should be some basis of natural leadership.  Ignoring this can be even counter-productive: conflicts will arise, the spirit and productivity will go down.

The Agile Team Charter Is a Team Tool

In the age of empires, kings and queens granted charters to companies.

In the age of empires, kings and queens granted charters to companies.

Project managers and developers have long been at odds with each other over deliverables like the project charter. The project management view is that the information is needed, albeit by parties outside the project team, and the project teams see their effort being siphoned off.  Generally, the charter has been considered a control document, therefore the bastion of stakeholders and project managers.  Agile techniques and the institution of the Agile team charter has changed that, however all of the parties that consumed the information from the classic charter still have information needs. There are three audience for the information in the classic charter.  Agile while still generating and sharing information provides it via different channels.

Agile teams are the primary consumer of the Agile team charter. The charter is a tool to help the team identify the project goal in their own words and then to concentrate the team’s attention specifically on that goal. Attention is an asset that is never overabundant, but is critical for any team if they are going to deliver the stories to which they have committed.  Along with direction, the charter also serves as a tool to guide the team’s behavior. The team identifies norms that establish an environment that is conductive to performance.

In the past, charters have been developed or framed by stakeholders or sponsors. Because of their authorship, it often takes on the feeling of a contract that can constrain and bind. Agile projects, and by extension Agile team charters, are flexible and dynamic to enhance the team’s ability to respond to the users and product owner needs, as they are discovered.

In many circumstances what stakeholders are looking for in the charter is a communication channel, or at the very least, a place at the table to influence and guide the project. In the past they have either written the charter or have signed off on the contents in the charter. Agile projects respect and embrace this need by providing techniques that generate involvement. Inclusion of the product owner as a direct part of the team, and the participation by a wide range of stakeholders and users in demonstrations where the team seeks approval and feedback are examples of Agile team’s recognition of the business’ place at the table. Instead of asking stakeholders to specifically define what is going to be delivered in the charter up front, Agile uses inclusion to dynamically define the projects outcome.  This ensures that as need change so do the goals of the project.

The classic project charter provides executives with an understanding of the important projects within their organizations. The need for information about strategic projects, whether Agile or any other another method, is no different. Charters are often used in organizations to authorize or approve a project; when an executive signs off on the charter it signals the beginning of a project. Authorization and notification is taken care of with one signature.  The Agile team charter is not the right tool for authorization because the charter now guides the team rather than authorizing the team. Agile organizations separate the team charter from authorization and typically develop a simple authorization form that is separate from the charter.

In the age of empires, kings and queens granted charters to companies. These charters identified the organization’s rights and obligations. Typically the charter established the governance structure for the endeavor. Today, many project charters follow in the same proud tradition. However when practicing Agile, much of the structure of classic project charters is overhead or shifted to other, more interactive techniques.  In today’s environment rarely do we need to approach projects as if we were chartering the East India Trading Company. Agile team charters work hand-in-hand with other Agile techniques like including product owner as team members and demonstrations that seek input to share information with a wider community.

 


Categories: Process Management

Quote of the Day

Herding Cats - Glen Alleman - Fri, 07/11/2014 - 23:44

Gentlemen, we have run out of money; now we have to think - Winston Churchill

The role of estimating in project and product development is many fold...

  • For product, the cost of development must recouped   over the life cycle of the product. Knowing the sunk cost of the product provides decision making information to the business if the target margin will be achieved and on what day.
  • For projects, the cost of development is part of the ROI equation. ROI = (Value - Cost) / Cost
  • For day to day business operations cash flow is the actual cost of producing outcomes. Budget is not the same as cost. We have define a budget for our work, but some forecast of the cost of that work, gathered from current operations or past performance, let's us know if we have sufficient budget.
  • For products when marginal cost exceeds marginal profit, we're going go out of business if we don't do something about controlling the cost. But our cost forecast and revenue forecast are the steering points to provide feedback for making choices.
  • For projects, the marginal cost and the marginal benefits obey the same rules of microeconomics.

In both cases the future cost and future monetized value are probabilistic numbers.

  • This project or product will cost $257,000 or less with an 80% confidence
  • This project or product will complete on or before May 2015 with a 75% confidence

With both these numbers and their Probability Distribution Function, decisions can be made about options - choices that can be made to influence the probability of project or product success.

Without this information, the microeconomics of writing software for money is not possible and the foundation of business processes abandoned.

In order to make these estimates of cost, schedule, and the technical performance of the project or product, some  model is needed and the underlying uncertainty of the elements of the model. These uncertainties come in two forms

  • Reducible (epistemic uncertainty) - money can be spent to reduce this uncertainty. Testing, prototypes, incremental development. 
  • Irreducible (aleatory uncertainty) - this is the normal variance in the process or technical components. The Deming uncertainty. The only action to reduce this uncertainty is margin, Cost margin, schedule, and technical margin. The cost margin is then part of the total project or product budget and the schedule margin part of the total period of performance for the project or the planned release date for the product.

To suggest decisions can be made without knowing this future information violates the principles of microeconomics of business 

Related articles Both Aleatory and Epistemic Uncertainty Create Risk Uncertainty is the Source of Risk Elements of Project Success More Uncertainty and Risk
Categories: Project Management

Agile Charters: More Barnacles!

The bigger the ships, the more barnacles.

The bigger the ships, the more barnacles.

As we noted in Agile Charter Barnacles, the basic Agile team charter is brief and to the point. However there is a natural tendency to add components back from classic charters.  The charter is a tool for the team, therefore the team needs to find the components that add value to how they work.  When these components add value to the team(s) doing the work adding these sections makes sense.  A number of readers of the Software Process and Measurement blog have asked about whether adding specific charter sections make sense in an Agile project or whether there are other techniques to address their needs.

  1. Roles:  Classic charters typically include a section that describes the roles that team members will play on the project team.  Agile teams are self-organizing, and roles change to ensure that the team can deliver the work they commit.  In large Agile programs, identifying areas of focus for teams is often valuable to help direct communication. Projects that are using classic project management methods with a smattering of Agile techniques will need to document roles.  However, at the team level, documenting regimented roles does not make sense for most Agile projects.
  2. Resources: Resources represent the assets that a team can draw from to deliver value and function effectively.  Resources can range from physical space to knowledge capital.  In general I only recommend identifying and documenting resources when they are outside of the norm and could be easily overlooked.  For example, I was once asked if was important to identify and document that the project team was going to use a new team room.  Since the team would tend not to forget where they worked, I suggested that they probably did not need to document the room as a resource.  While the statement sounds cynical, the discussion then turned to a discussion about who the target audience for the charter was, the team or others outside of the team.  An Agile team charter is a team-level tool, not an external communication vehicle.
  3. Context: The context for the project can be helpful for the team(s) involved in developing and delivering a solution.  In a very real sense the product owner is physical instantiation of context.  In small projects, the access of the product owner will generally suffice as a means to share context. In larger projects or programs with broad or complicated business objectives, capturing context is important.  I generally recommend in cases where capturing the narrative context is important that a separate document (this is generally not flip chart territory) is generated and that the product owner spearheads its creation.
  4. Milestones and Delivery Cadence: The Agile release plan documents the strategy that the team intends to use for delivering the project.  Release plans will identify whether functionality will be delivered to production on continuous basis, at the end of every sprint or after a number of sprints as a release. Identify the strategy the team intends to follow in the charter if it outside of the normal pattern the team follows. The release plan itself should be documented and maintained separately from the Agile team charter.
  5. Assumptions:  Assumptions are defined as things that accepted as true or certain to happen without proof.  Teams should always reflect on what they accept or believe is true that is not supported by evidence.  Assumptions that are outside of the norm need to be treated as potential blockers, captured and pursued by scrum master/coach to verify or treated as risks (documented as user stories and included in the backlog).

To add components to the Agile team charter or not to add components to the Agile team charter, that is the question. Since the Agile team charter is a tool that helps focus and guide the project teams — then the answer is add a component only if it adds value to the team.


Categories: Process Management

Stuff The Internet Says On Scalability For July 11th, 2014

Hey, it's HighScalability time:


Yesterday in history: Nikola Tesla's Birthday, born in 1856. The greatest geek who ever lived?
  • 10Gbps: New world record broadband speed of 10 Gbps over copper.
  • Quotable Quotes:
    • @BenedictEvans: There were 40m internet users when Netscape IPOed. The time's not far off when a startup with 40m users will be too small to get funded.
    • Scott Aaronson: In any case, the question I asked myself about CLEVER/PageRank was not the one that, maybe in retrospect, I should have asked: namely, “how can I leverage the fact that I know the importance of this idea before most people do, in order to make millions of dollars?”
    • chub79: µservices aren't technological as much as they are cultural.
    • @Elmood: I thought of a new term when talking about code: "It's made from unmaintainium."
    • @lxt: Amazing how quickly a bunch of nines go up in smoke.
    • @martinrue: Knock knock. Race condition. Who's there?

  • The Master Switch: The Rise and Fall of Information Empires by Tim Wu: History shows a typical progression of information technologies: from somebody’s hobby to somebody’s industry; from jury-rigged contraption to slick production marvel; from a freely accessible channel to one strictly controlled by a single corporation or cartel—from open to closed system. History also shows that whatever has been closed too long is ripe for ingenuity’s assault: in time a closed industry can be opened anew, giving way to all sorts of technical possibilities and expressive uses for the medium before the effort to close the system likewise begins again.

  • Tim Freeman indulges a well developed Technothantos Complex and comes up with a great big list of outage postmortems. You'll find the usual, outages from configuration issues, failover failures, quorumnesia, protocol flapping, bugs in not your stuff that causes bugs in your stuff, power outages, capacity problems, JPOBs (just plain old bugs), DDOS attacks, and good old operator error. 

  • Pinterest describes PinLater, An asynchronous job execution system. PinLater executes hundreds of different job types at a processing rate of over 100,000 per second. So you may say yet another async job system, but it's clear keeping such a critical part of their infrastructure in house makes sense. The article is a good explanation of a fairly standard approach. It used Thrift for the API, it's written in Java, Twitter’s Finagle is used for the RPC framework. MySQL is "used for relatively low throughput use cases and those that schedule jobs over long periods and thus can benefit from storing jobs on disk rather than purely in memory." Redis is "used for high throughput job queues that are normally drained in real time." Horizontal scaling is via sharding. 

  • In science class we did this one day, but I just couldn't do it. Dissecting Message Queues. Tyler Treat looks at both brokerless and brokered queues by looking a throughput benchmarks, latency benchmarks, and through qualitative analysis. No winner was declared, but if you are making a choice in this area it's well worth reading. 

  • 40 Million hits a day on WordPress using a $10 VPS. Sure, it's a static site, but still a good example of what can be done these days. Stack: Nginx + PHP-FPM (aka LEMP Stack) + Microcaching. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Why Little's Law Works...Always

Xebia Blog - Fri, 07/11/2014 - 12:00

On the internet there is much information on Little's Law. It is described an explained in many places [Vac]. Recently, the conditions under which it is true got  attention [Ram11]. As will be explained in this blog the conditions under which the law is true are very mild. It will be shown that for teams working on backlog items virtually there are no conditions.

Why does it work? Under what conditions does it hold?

Airplane Folding

In the previous post (Applying Little's Law in Agile Games) I described how to calculate the quantities in Little's Law. As an example this was applied to the Agile game of folding airplanes consisting of 3 rounds of folding.

airplane-1

Let's look in more detail in which round an airplane was picked up and in which round it was completed. This is depicted in the following figure.

The horizontal axis shows the number of rounds. The vertical axis describes each airplane to fold. The picture is then interpreted as follows. Airplane no. 2 is picked up in round 1 and competed in the same round. It has a waiting time of 1 round. This is indicated at the right of the lowest shaded rectangle.
Airplane no. 8 was picked up in round 1 and finished in round 3. A waiting time of 3 rounds. Airplane no 12 (top most shaded area) was picked up in round 3 and unfinished. Up to round 3 a waiting time of 1 round.

The number 3, 5, and 10 denote the number of completed airplanes at the end of round 1, 2, and 3 respectively.

Waiting Times

The waiting times are determined by counting the number of 'cells' in a row.

The pictures show that we have 12 airplanes (12 'rows'), 3 completed in the first round, 2 more completed in the second round and 5 additionally  folded airplanes in the third and last round giving a total of 10 finished paper airplanes.

All twelve airplanes have waiting times of 1, 1, 1, 2, 2, 3, 3, 3, 3, 3, 3, and 1 respectively.

Work in Progress

In the figure below the number of airplanes picked up by the team in each round are indicated with red numbers above the round. new doc 6_2
In round 1 the team has taken up the folding of 11 airplanes (of which 3 are completed). In round 2 the team was folding 8 airplanes (of which 2 were competed) and in round 3 the team was folding 7 airplanes (of which it completed 5).

Work in progress is determined by counting the number of 'cells' in a column.

Little's Law.....Again

Now that we have determined the waiting times and amount of work in progress, let's calculate the average waiting time and average work in progress.

Average Waiting Time. This quantity we get by adding all waiting times and dividing by the number of items. This gives 26/12.

Average Work in Progress. This quantity is equal to (11+8+7)/3 = 26/3.

Average input rate. This is equal to 12 (the height of the third column) divided by 3 which gives 4.

Again we find that: Average Waiting Time = Average Work in Progress / Average input rate.

Why It Works

new doc 6_3 Little's Law works....always....because the average waiting times is got by adding the lengths of all the rows dividing by the number of rows, so it is proportional to the size of the shaded area in the picture to the right.

The average work in progress is got by adding the heights of the columns in the shaded area which is also proportional to the size of the shaded area.

Both the waiting time and work in progress relate to the size of the shaded area: one by adding the heights and the other by adding the rows. The proportionality corresponds to the average input rate.

Conditions

What assumptions did we make? None...well this is not exactly true. The only assumptions we make in this calculation:

  • We count discrete items
  • There are a finite number of rounds (or sprints)
  • Items enter and possibly leave the system.

That's it. It doesn't need to be stable, ageing (items having increasingly larger waiting times) is not a problem, prioritisation/scheduling of items (also known as queueing discipline), etc. Only the above assumptions need to be true.

Note: Especially the second condition is important, i.e. Little's Law is measured over a finite time interval. For infinite time interval additional conditions need to be fulfilled.

Note: When applying this to agile teams we always consider finite time intervals, e.g. 6 months, 1 year, 8 sprints, etc.

Conclusion

Little's Law is true because the average waiting time is proportional to the size of the shaded area (see figure) and the average work in progress is also proportional to the size of the same shaded area.

Only 3 basic conditions need to be met for Little's Law to be true.

References

[Vac] Little’s Law Part I, Dan Vacanti, http://corporatekanban.com/littles-law-part-i/

[Ram11] Little’s Law – It’s not about the numbers, Agile Ramblings, http://agileramblings.com/2012/12/11/littles-law-its-not-about-the-numbers/

Building Successful Global App Businesses

Android Developers Blog - Fri, 07/11/2014 - 06:34

By: Purnima Kochikar, Director, Google Play Apps & Games

With over 1 billion active Android users, an increasing number of developers like you are building successful global businesses on Google Play. Since the last Google I/O, we’ve also paid out more than $5 billion to developers.

This week at Google I/O, we announced new ways to help you build a successful business. These solutions work together at scale to help you find more users, understand and engage them, and effectively convert your active users into buyers.

Build an engaging app

Last year, Google Play became an even better place to try new ideas. Since May 2013, Google Play offers Alpha and Beta Testing so that you can engage users early to get feedback on your new app. Feedback provided by users is private, allowing you to fix issues before publicly launching the app, and without impacting your public ratings and reviews. Over 80,000 apps on Google Play are actively using beta testing. You can also ensure new versions get a positive response by updating through staged rollouts.

Scale operations

As your app business grows, you dedicate more time to release management. Today we announced the Google Play Developer Publishing API to help you scale your release operations. The new API will let you upload APKs, manage your in-app products and localized store listings. You will be able to integrate publishing operations with your release processes and toolchain through a RESTful API. With the Google Play Developer Publishing API you’ll spend less time managing your releases and more time managing your business. This API is currently in closed beta and we look forward to making it available to all developers.

Actionable insights

The Google Play Developer Console now offers more actionable insights into your app’s performance by sending you email notifications for Alerts and providing Optimization Tips. We’re also offering new revenue metrics including number of buyers and average revenue per paying user. You’ll also be able to export user reviews for further analysis. Click on Announcements in the Developer Console for a list of new features.

For game developers, we recently launched enhanced Play Games statistics on the Google Play Developer Console. You get a daily dashboard that visualizes player and engagement statistics for signed in users, including daily active users, retention analysis, and achievement and leaderboard performance.

Enhance discovery and engagement

With AdWords, we're building a robust platform to help you promote your app and drive re-engagement. This week we are launching Installed App Category Targeting, a new way to promote your app to new users. It helps you reach potential customers across the AdMob network who have already installed apps from related categories on Google Play and other app stores. For example, an action-oriented game developer may wish to reach users who have previously installed apps from the category Action & Adventure Games.

Ads can also remind users about the apps they already have. Through Google mobile display and search ads deep linking, you can re-engage users who have already installed your Android app by taking them directly to specific pages in the app. Let’s say someone has the Hotel Tonight app installed on their phone. If they search Google for “hotels in San Francisco," they'll see an ad that will open Hotel Tonight app and take them directly to a list of San Francisco hotels.

This deep-linking is also available through search for all apps that implement app indexing. If a user with the Walmart Android app searches for “Chromecast where to buy”, they’ll go directly to the Chromecast page in the Walmart app. The new App Indexing API is now open to all Android developers, globally. Get started now.

New services for game developers

For game developers using Play Games, we announced a new Game Profile that is automatically customized based on the gameplay and achievements earned in those games. Since its launch last year, users have loved saving their game progress in the cloud. We’re now evolving this feature to Saved Games, where users can save up to 3 “bookmarks” of their progress in the Play Games app, complete with images and descriptions. Finally, we announced a new service called Quests — it you run online, time-based goals in your game; for example, players can collect bunch of in-game items on a specific day, and the quests services coordinates with your game to know who completed the goal. These APIs run events for your players, and reward them, without the need to update your game.

New monetization tools

Today, we announced that users who have set up Direct Carrier Billing on their smartphone can also make purchases on Google Play from their tablet, charging to the same mobile phone bill. In addition to our recent launch of payments through PayPal, these new user payment options expand monetization opportunities for your apps.

As announced earlier this year, Google Analytics is now directly available in the AdMob interface, giving you powerful segmentation tools to determine the best monetization strategy for each user. For example, you might want to display in-app purchase ads to users most interested in buying, while showing regular ads to those less likely to buy right now. Once you’ve segmented your audience in this way, you can use AdMob to build interstitial ads that promote in-app purchase items to users at a point in your app that’s useful to them. This creates a more customized experience for users, can help prolong engagement and grow in-app purchase revenue. Learn more.

Join us

If you're at Google I/O 2014, please join us at our breakout sessions today and tomorrow, where we'll be talking about these features in much more detail. (Add us to your calendar!) And if you can't make I/O, you can always join us on the livestream or watch the videos online later.

.footer__link, .footer__text{ color: #8b8b8b !important; display: block; font-size: 12px; line-height: 1.33333; margin-bottom: 10px; text-decoration: none; } .hoverable .footer__link:hover { text-decoration: underline; }

Google I/O 2014 I/O Livestreams I/O Bytes Videos +Android Developers

L Developer Preview Material Design Android Wear Android TV Android Auto

Get it on Google Play
Categories: Programming

Building an Agile Charter

Flip charts!

Flip charts!

Building an Agile team charter is generally one the first events that marks the beginning of a new endeavor.  The charter helps a team clearly capture the project’s goals and definition of success.   For most projects, project initiation is a time full of possibilities, a time before all the mundane issues and the day-to-day begin. The process of framing the charter must acknowledge the excitement of the team without letting that excitement lead them astray.  A standard, moderated process process to create a charter provides focus and direction for a team raring to deliver value.

There are two primary prerequisites to starting an Agile Charter.

  • A product owner or stakeholder has conceived of a project and gotten approval to at least create a charter.
  • A project team has been assigned to the work (product owner, scrum master/coach, team).

Without a team or a project, creating a charter does not make sense.

A simple process for building a charter is shown below:

  1. Before convening the entire team, the product owner and scrum master should decide which components will be included in the Agile team charter (see Agile Team Charter and Agile Charter Barnacles). The scrum master/coach will moderate Agile team charter session and should do some work in preparation.
  2. For each section that will be included, prep the flip charts.  For example, label a single flip chart for each component and for the elevator speech put the outline down for the team.
  3. Create a set of framing questions for each component.  These questions can be used to facilitate the discussion.
  4. Convene the meeting to build the Agile team charter.  This session should typically be scheduled for half of a day. All team members should attend in person. When that is not possible, then ensure good tele- or videoconference packages are used. Include lunch if at all possible.  It is imperative that all team members, including the product owner, participate.  The Agile team charter critical for focusing a team
  5. Before beginning work on the charter, review with the team why they there, any ground rules (e.g. no smart phones) and the components that are being proposed as part of the charter.  Tailor the list of components based on feedback from the team. . I almost always suggest discussing risk in the session held to build the charter however never include the risks in the charter (add immediately to the backlog).
  6. Pass out markers, sticky notes, voting flags or any other items that the team will use during the session.  No one should have to spend time looking for office supplies during the session.  Generally when there are remote participants the moderator or someone helping the moderator will act as an intermediary to scribe their comments.
  7. Iteratively complete the components in the Agile team charter.  Have team members scribe their own comments on the flip charts.  One mark of a good session is multiple handwriting styles on each flip chart because is a reminder of the whole teams engagement creating the charter.
  8. Tape the completed components to the wall in the team room in a prominent place so that every team member can review the charter as needed.

The scrum master/coach is in the room to help the team complete the charter, not to complete it for them.  The moderator should manage the flow of discussion and the clock.  Defining the charter is time boxed so that is completed in one session.

The process of building a typical Agile team charter occurs once as a project begins.  This is no different than in a classic project.  However the in an Agile project the Agile team charter is referenced on a daily basis.  I recommend reviewing the charter at least once every ninety days or if any significant changes happen on the team.

An Agile team charter provides direction for the whole team. A standard process harnesses the energy teams have to build a charter and begin the project moving in same direction.


Categories: Process Management

Bootstrapping and monitoring multiple processes in Docker using monit

Xebia Blog - Thu, 07/10/2014 - 23:01

If you have every tried to start a docker container and keep it running, you must have encountered the problem that this is no easy task. Most stuff I like to start in container are things like http servers, application servers and various other middleware components which tend to have start scripts that daemonize the program. Starting a single process is a pain, starting multiple processes becomes nasty. My advise is to use monit to start all but the most simple Docker application containers! When I found monit while delving through the inner works Cloud Foundry, I was ecstatic about it! It was so elegant, small, fast, with a beautiful DSL that I thought it was the hottest thing since sliced bread! I was determined to blog it off the roof tops. Until.... I discovered that the first release dated from somewhere in 2002. So it was not hot and new; Clearly I had been lying under a UNIX rock for quite a while. This time, the time was right to write about it! Most of the middleware components I want to start in a docker container, have a habit to start the process, daemonize it and exit immediately, with the docker container on its tail. My first attempt to circumvent this while starting a tomcat server in Docker looked something like this:

/bin/bash -c "service tomcat7 start;while service tomcat7 status;do sleep 1;done

Quite horrific. Imaging the ugliness when you have to start multiple processes. A better solution is needed:  With the zabbix docker container  the problem was solved using simplevisor. As you can read in this post that was not a pleasant experience either. As I knew little about simplevisor and could not solve the problem, I put in an issue and  resorted to a plain installation. But a voice in my head started nagging: "Why don't you fix it and send a pull request?"  (actually, it was the voice of my colleague Arjan Molenaar). Then, I remembered from my earlier explorations to the inner workings of Cloud Foundry, a tool that would be really suitable for the job: monit. Why? It will:

  1. Give you a beautiful,readable specification file stating which processes to start
  2. Make sure that your processes will keep on running
  3. Deliver you a clean and crisp monitoring application
  4. Reduce all your Docker starts to a single command!

In the case of the Zabbix server there were seven processes to start: the zabbix server, agent, java agent, apache, mysql and sshd. In monit this looks as follows:

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
        start program = "/sbin/service mysqld start"
        stop program = "/sbin/service mysqld stop"

check process zabbix-server with pidfile /var/run/zabbix/zabbix_server.pid
        start program = "/sbin/service zabbix-server start"
        stop program = "/sbin/service zabbix-server stop"
        depends on mysqld

check process zabbix-agent with pidfile /var/run/zabbix/zabbix_agentd.pid
        start program = "/sbin/service zabbix-agent start"
        stop program = "/sbin/service zabbix-agent stop"

check process zabbix-java-gateway with pidfile /var/run/zabbix/zabbix_java.pid
        start program = "/sbin/service zabbix-java-gateway start"
        stop program = "/sbin/service zabbix-java-gateway stop"

check process httpd with pidfile /var/run/httpd/httpd.pid
        start program = "/sbin/service httpd start"
        stop program = "/sbin/service httpd stop"
        depends on zabbix-server

check process sshd with pidfile /var/run/sshd.pid
        start program = "/sbin/service sshd start"
        stop program = "/sbin/service sshd stop"

Normally when you start monit it will start as a daemon. But fortunately, you can prevent this with the following configuration.

set init

Your Dockerfile CMD can now always look the same:

    monit -d 10 -Ic /etc/monitrc

Finally, by adding the following statement to the configuration you get an application to view the status of your container processes,

set httpd
     port 2812
     allow myuser:mypassword

After starting the container, surf to port 2812 and you will get a beautiful page showing the state of your processes and the ability to stop and restart them.

monit overview monit process control

Just delve into the documentation of monit and you will find much more features that will allow you to monitor network ports and files, start corrective actions and send out alerts.

Monit is true to its UNIX heritage: it is elegant and promotes an autonomous monitoring system. Monit is cool!

New Cross-Platform Tools for Game Developers

Android Developers Blog - Thu, 07/10/2014 - 19:25

By Ben Frenkel, Google Play Games team

There was a lot of excitement at Google I/O around Google Play Games, and today we’re delighted to share that the following tools are now available:

  • Updated Play Games cross-platform C++ SDK
  • Updated Play Games SDK for iOS
  • New game services alerts in the Developer Console

Here's a quick look at the cool new stuff for developers.

Updated Play Games C++ SDK

We've updated the Google Play Games C++ SDK with more cross-platform support for the new services and experiences we announced at I/O. Learn more»

The new C++ SDK now supports all of the following:

Cocos2D-x, a popular game engine, is an early adopter of the Play Games C++ SDK and is bringing the power of Play Games to their developers. Additionally, the Cocos2D-x team created Wagon War, a prototype game showcasing the capabilities of the Cocos2D-x engine with Play Games C++ SDK integration.

Wagon War is also a powerful reference for developers — it gives you immediately usable code samples to accelerate your C++ implementations. You can browse or download the game sources on the Wagon War page on GitHub.

Updated Play Games iOS SDK

The Play Games iOS SDK is now updated with support for Quests and Saved Games, enabling iOS developers to integrate the latest services and experiences with the Objective-C based tool-chains they are already familiar with. Learn more»

The new Play Games SDK for iOS now supports all of the following:

  • Quests and Events. Learn more»
  • Saved Games. Learn more»
  • Game Profile and related Player XP APIs — the SDK now also provides the UI for Game Profile and access to Player XP data for players.
New types of games services alerts

Last, you can now see new types of games services alerts in the Developer Console to learn about issues that might be affecting your users' gameplay experiences. For example, if your app implements Game Gifts, you'll now see an alert when players are unable to send a gift; if your app implements Multiplayer, you'll now see an alert when players are unable to join a match. Learn more»


Join the discussion on
+Android Developers


Categories: Programming

Design Sprints for Developers

Google Code Blog - Thu, 07/10/2014 - 19:04
Author PictureBy Nadya Direkova, Staff Designer and Design Evangelist at Google[x]

At Google and throughout the industry, we all agree that two things matter: design and speed. But how can we do great design quickly? For our teams, one of our most important tools is the design sprint.

While a typical product design process takes months or years, a design sprint compresses this into a week or less. The design sprint combines key design and research methods and focuses on a single challenge or multiple challenges in parallel. It brings all the stakeholders—designers, developers, product managers, and other decision makers—into one place to work together on a short deadline. It often leads to insights and solutions more quickly than anyone thought possible. At Google, we've been using design sprints for over four years, from external projects like Ads, Glass and Project Loon to our internal tools.

One team has even run a huge sprint with 175 participants in 23 teams. How did that feel? As Cordell Ratzlaff, User Experience Director for Ads & Commerce, says: “When you participate in a sprint, you either win or you learn.” Our latest Google Design Minutes short tells this story:

Design sprints at scale: Cordell Ratzlaff and team on the importance of constraints

We’re really excited about sharing our design sprint methods more broadly. Design sprints were an important theme in the “Design, Develop, Distribute” message at Google I/O 2014, where developers got a chance to learn about and experience short sprints in person.

The design sprint: from Google Ventures to Google[x]; Daniel Burka, Jake Knapp, Nadya Direkova share insights with developers at Google I/O 2014

However, this was just a first glimpse; over the summer, we’ll be hosting design sprints for select developers in the Bay Area, helping developers design for platforms like Glass and Android Wear or build with the material design approach. To get updates when these limited-seating events become available, sign up here.

No matter what your challenge and design process, design sprints can help you reduce the time it takes to create great ideas. So make great things, and make them quickly!

Categories: Programming

Neo4j: LOAD CSV – Processing hidden arrays in your CSV documents

Mark Needham - Thu, 07/10/2014 - 15:54

I was recently asked how to process an ‘array’ of values inside a column in a CSV file using Neo4j’s LOAD CSV tool and although I initially thought this wouldn’t be possible as every cell is treated as a String, Michael showed me a way of working around this which I thought was pretty neat.

Let’s say we have a CSV file representing people and their friends. It might look like this:

name,friends
"Mark","Michael,Peter"
"Michael","Peter,Kenny"
"Kenny","Anders,Michael"

And what we want is to have the following nodes:

  • Mark
  • Michael
  • Peter
  • Kenny
  • Anders

And the following friends relationships:

  • Mark -> Michael
  • Mark -> Peter
  • Michael -> Peter
  • Michael -> Kenny
  • Kenny -> Anders
  • Kenny -> Michael

We’ll start by loading the CSV file and returning each row:

$ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row RETURN row;
+------------------------------------------------+
| row                                            |
+------------------------------------------------+
| {name -> "Mark", friends -> "Michael,Peter"}   |
| {name -> "Michael", friends -> "Peter,Kenny"}  |
| {name -> "Kenny", friends -> "Anders,Michael"} |
+------------------------------------------------+
3 rows

As expected the ‘friends’ column is being treated as a String which means we can use the split function to get an array of people that we want to be friends with:

$ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row RETURN row, split(row.friends, ",") AS friends;
+-----------------------------------------------------------------------+
| row                                            | friends              |
+-----------------------------------------------------------------------+
| {name -> "Mark", friends -> "Michael,Peter"}   | ["Michael","Peter"]  |
| {name -> "Michael", friends -> "Peter,Kenny"}  | ["Peter","Kenny"]    |
| {name -> "Kenny", friends -> "Anders,Michael"} | ["Anders","Michael"] |
+-----------------------------------------------------------------------+
3 rows

Now that we’ve got them as an array we can use UNWIND to get pairs of friends that we want to create:

$ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row 
  WITH row, split(row.friends, ",") AS friends 
  UNWIND friends AS friend 
  RETURN row.name, friend;
+-----------------------+
| row.name  | friend    |
+-----------------------+
| "Mark"    | "Michael" |
| "Mark"    | "Peter"   |
| "Michael" | "Peter"   |
| "Michael" | "Kenny"   |
| "Kenny"   | "Anders"  |
| "Kenny"   | "Michael" |
+-----------------------+
6 rows

And now we’ll introduce some MERGE statements to create the appropriate nodes and relationships:

$ load csv with headers from "file:/Users/markneedham/Desktop/friends.csv" AS row 
  WITH row, split(row.friends, ",") AS friends 
  UNWIND friends AS friend  
  MERGE (p1:Person {name: row.name}) 
  MERGE (p2:Person {name: friend}) 
  MERGE (p1)-[:FRIENDS_WITH]->(p2);
+-------------------+
| No data returned. |
+-------------------+
Nodes created: 5
Relationships created: 6
Properties set: 5
Labels added: 5
373 ms

And now if we query the database to get back all the nodes + relationships…

$ match (p1:Person)-[r]->(p2) RETURN p1,r, p2;
+------------------------------------------------------------------------+
| p1                      | r                  | p2                      |
+------------------------------------------------------------------------+
| Node[0]{name:"Mark"}    | :FRIENDS_WITH[0]{} | Node[1]{name:"Michael"} |
| Node[0]{name:"Mark"}    | :FRIENDS_WITH[1]{} | Node[2]{name:"Peter"}   |
| Node[1]{name:"Michael"} | :FRIENDS_WITH[2]{} | Node[2]{name:"Peter"}   |
| Node[1]{name:"Michael"} | :FRIENDS_WITH[3]{} | Node[3]{name:"Kenny"}   |
| Node[3]{name:"Kenny"}   | :FRIENDS_WITH[4]{} | Node[4]{name:"Anders"}  |
| Node[3]{name:"Kenny"}   | :FRIENDS_WITH[5]{} | Node[1]{name:"Michael"} |
+------------------------------------------------------------------------+
6 rows

…you’ll see that we have everything.

If instead of a comma separated list of people we have a literal array in the cell…

name,friends
"Mark", "[Michael,Peter]"
"Michael", "[Peter,Kenny]"
"Kenny", "[Anders,Michael]"

…we’d need to tweak the part of the query which extracts our friends to strip off the first and last characters:

$ load csv with headers from "file:/Users/markneedham/Desktop/friendsa.csv" AS row 
  RETURN row, split(substring(row.friends, 1, length(row.friends) -2), ",") AS friends;
+-------------------------------------------------------------------------+
| row                                              | friends              |
+-------------------------------------------------------------------------+
| {name -> "Mark", friends -> "[Michael,Peter]"}   | ["Michael","Peter"]  |
| {name -> "Michael", friends -> "[Peter,Kenny]"}  | ["Peter","Kenny"]    |
| {name -> "Kenny", friends -> "[Anders,Michael]"} | ["Anders","Michael"] |
+-------------------------------------------------------------------------+
3 rows

And then if we put the whole query together we end up with this:

$ load csv with headers from "file:/Users/markneedham/Desktop/friendsa.csv" AS row 
  WITH row, split(substring(row.friends, 1, length(row.friends) -2), ",") AS friends 
  UNWIND friends AS friend  
  MERGE (p1:Person {name: row.name}) 
  MERGE (p2:Person {name: friend}) 
  MERGE (p1)-[:FRIENDS_WITH]->(p2);;
+-------------------+
| No data returned. |
+-------------------+
Nodes created: 5
Relationships created: 6
Properties set: 5
Labels added: 5
Categories: Programming

Fitting In With Corporate Culture

Making the Complex Simple - John Sonmez - Thu, 07/10/2014 - 15:00

In this video I answer a question about fitting into corporate culture when you come from a different background.

The post Fitting In With Corporate Culture appeared first on Simple Programmer.

Categories: Programming

Registration Opens for More Workshops

NOOP.NL - Jurgen Appelo - Thu, 07/10/2014 - 09:51
open-registration

I’m very happy to say that the Management 3.0 Workout sessions so far have been well received, and they’re getting better every time! I’m almost ready for the summer break, but registration has opened for the remaining workshops this year. Please remember, I will personally visit each city only once!
It was a very joyful and insightful workshop

The post Registration Opens for More Workshops appeared first on NOOP.NL.

Categories: Project Management