Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

The Use, Misuse, and Abuse of Complexity and Complex

Herding Cats - Glen Alleman - Wed, 03/04/2015 - 16:54

Our world is complex and becoming more complex all the time. We are connected and in turn driven by a complex web of interacting technology and processes. These interacting technologies and processes are implemented by information and communication technologies that are themselves complex. It is difficult to apply such a broad topic as complexity to the equally broad topic of developing software systems or even broader topic of engineered systems.

Measuring complexity in engineered systems is a highly varying concept.1

The result of these complex systems many times creates complexity. But care is needed is tossing around words like complex and complexity. If these systems are in fact Engineered, rather than simply left to emerge on their own, we can apply some principles to control the unwanted complexity of these complex systems.

First some definitions. These are not the touchy feely definitions found in places like Cynefin where the units of measure of complex, complexity, and chaos, are no where to be found. Cynefin was developed in the context of management and organizational strategy by David Snowden. We're interested in the system complexity of things, the people who build them, and the environments where they are deployed. But this also means measuring complex and the complexity in units meaningful to the decision makers. These units must somehow be connected to the cost, schedule, and probability of success for those paying for the work.

Complexity has turned out to be very difficult to define. The dozens of definitions that have been offered all fall short in one respect or another, classifying something as complex which we intuitively would see as simple, or denying an obviously complex phenomenon the label of complexity. Moreover, these definitions are either only applicable to a very restricted domain, such as computer algorithms or genomes, or so vague as to be almost meaningless. (From Principia Cybernetica) 

Some more background about complex systems and their complexity4

  • A System is a set of interacting components - whether human-made, naturally-occurring, or a combination of both.
  • By "interact", it means the exchange of physical force, energy, mass flow, or information, such that one component can change the state of another component. Or for software systems, the exchange of information, state knowledge, or impact an outcome of other component.
  • The technologies or natures of these systems may be mechanical, chemical, electronic, biological, informational (software), or combinations of these or others.
  • The behavior of a system can include "emergent" aspects that are not a characteristic of any individual component, but arise from their combination.
  • Emergent properties can be valuable (e.g., delivery of new services) or undesired (e.g., dangerous or unstable).
  • The behavior of a system is often not easily predicted from the behavior of its individual components, and may also be more complex.
  • The complexity of human-engineered systems is growing, in response to demands for increased sophistication, in response to market or government competition, and enabled by technologies.
  • It has become relatively easy to construct systems which cannot be so readily understood. 

What is complexity then?

The words chaos and complexity have been used to mean disorder and complications for centuries. Only in the last thirty years have they been used to refer to mathematical and scientific bodies of knowledge.6

Often something is called complex when we can't fully understand its structure or behavior. It is uncertain, unpredictable, complicated, or just difficult to understand.  Complexity is often described as the inability of a human mind to grasp the whole of a complex problem and predict the outcome as Subjective Complexity.5

The complex and complexity I'm speaking about are for Engineered Systems, products and services used by organizations, but engineered for their use.  Their use can be considered complex, even create complexity and many times emergent. But the Cynefin approach to complex is ethereal, without the principled basis found in engineering and more importantly Systems Engineering.3 This appears to be why agilest toss around the terms found in Cynefin, since engineering of the software is not a core principle of agile development, rather emergent design and architecture is the basis of the Agile Manifesto.

Here's an example of the engineering side of complex systems, from Dr. Sheard's presentation "Twelve Roles and Three Types of Systems Engineering," NASA Goddard Space Flight Center, Systems Engineering Seminar, Complexity and Systems Engineering, April 5, 2011,

Screen Shot 2015-03-01 at 9.02.58 AM

Before applying these definitions to problems for developing software, there is more work to do.

In complex systems there are entities that participate in the system2

  • The technical system being designed and built.
  • The socio-technical systems that are building the systems - the project team or production team.
  • The technological Environment into which the system will be inserted when the system is complete and deployed. The socio-political system related to the technological environment. This is generally the interaction of the system stakeholders with the resulting system.
  • The subjective human experience when thinking about, designing, or using the system, called Cognition.

Cynefin does not make these distinctions, instead separates the system into Complex, Complicated, Chaotic, and Obvious, without distinguishing to which engineered portion of the system these are applicable.

So when we hear about Complex Adaptive Systems in the absence of a domain and the mathematics of such a system, care is needed. It is likely no actionable information will be available in units of measure meaningful to the decision makers to help them make decisions. Just a set words.

References

1 "Complexity Types: from Science to Systems Engineering," Sarah Sheard, ; Mostashari, Ali, Proceedings of the 21th Annual International Symposium of the International Council of Systems Engineering

2 "Systems Engineering in Complexity Context," Sarah Sheard, Proceedings of the 23nd Annual International Symposium of the International Council of Systems Engineering

3 Systems Engineering Principles and Practices, 2nd Edition, Alexander Kossiakoff William N. Sweet Samuel J. Seymour Steven M. Biemer, John Wiley & Sons.

4 The Challenge of Complex Systems, INCOSE Crossroads of America Chapter.

5 “On Systems architects and systems architecting: some thoughts on explaining the art and science of system architecting” H. G. Sillitto, Proceedings of INCOSE IS 2009, Singapore, 20-23 July. 

6 Practical Applications of Complexity Theory for Systems Engineers, Sarah Sheard, Proceedings of the 15th Annual International Symposium of the International Council on Systems Engineering

Related articles Risk Management is How Adults Manage Projects Start with Problem, Than Suggest The Solution I Think You'll Find It's a Bit More Complicated Than That
Categories: Project Management

Failure is not an Option

Herding Cats - Glen Alleman - Wed, 03/04/2015 - 04:30

LastTitan_vandenberg_fThere is a popular noton in the agile world, and some business guru's that failure is encouraged as part of the learning process. What is not stated is when and where this failure can take place.

The picture to the left is the flight of the last Titan IV launch vehicle. I was outside the SCIF, but got to see everything up the 2nd stage separation.

The Martin Company’s launch vehicle built a five-decade legacy goes back to the earliest rockets designed and built in the United States. The Intercontinental Ballistic Missile (ICBM) program; Project Gemini, NASA’s 2nd human spaceflight program; Mars Viking landers; Voyager deep space probes; communications and reconnaissance satellites—all of these programs and more relied on the Titan for a safe and dependable launch. 

The final version flew when the program was retired after delivering National Reconnaissance Office payload to orbit on October 19, 2005. A total 368 Titans were flown, with capabilities ranging from Earth reconnaissance and military and civil communications to human and robotic exploration.

In this domain, failure is not an option. Many would correctly say, failures were found before use. And that is correct, Design, Development, Test, and Evaluation (DDT&E) is the basis of must work when commanded to do so when commanded to do so.

In domains without the needed capability that must perform on demand - fail fast and fail often may be applicable. 

Choose domain before suggesting a process idea is applicable

Related articles Software Engineering is a Verb Self-Organization Quote of the Day
Categories: Project Management

The House of Lean, or Is That The House of Quality?

House of Lean

Lean is the systematic removal of waste within a process, known as muda in Japanese. Much of our understanding of lean as a process improvement tool is a reflection of the Toyota Production System (TPS). In the parlance of TPS, the container for lean ideas and concepts is the House of Quality. Larman and Leffingwell and others have modified the metaphor to the House of Quality that Toyota leveraged to the House of Lean (HoL) to focus on the flow of work. The focus on flow makes lean concepts an important component for scaling Agile in frameworks like SAFe. Even without the rebranding, the core of lean provides a focus how work is done, which improves the flow or smoothness of work. The focus on flow reduces variance (know as muri in Japanese). Lean identifies variance from by comparing the outcome of work to development standards to expose existing problems so waste can be reduced. I am assuming that once waste is exposed that you do something about it. The concept of the House of Lean or the House of Quality has many variants. All that I have studied are valuable, however we will use the Larman/Leffingwell version of the HoL as this version seems to have found fertile ground in the software development field. The House of Lean we will explore consists of six components. They are:

  1. A ceiling or roof, which represents the goal. The goal is typically defined as delivering the maximum value in the shortest sustainable lead-time while providing the highest value and quality to customers, people and society.
  2. Pillar one (typically shown on the left side of the hours or lean) represents respect for people. Work of any sort is built around people. People need to be empowered to assess and evolve how they work within the standards of the organization. Agile reflects this respect for people in the principles of self-organization.
  3. Pillar two (typically shown on the right side of the hours or lean) represents continuous improvement. Continuous improvement, often termed Kiazen or good change, is the relentless removal of inefficiencies. Inefficiencies (waste) detract or keep an organization from attaining the goal.
  4. Between the two pillars are housed:
    1. Delivery practices that reflect the techniques used to deploy lean, such as great engineers, cadence, cross-functional teams, team rooms and process visualization. In the SAFe version of the HoL, the 14 Lean Principles often subsume a discussion of delivery practices. The inclusion in the HoL of specific types of lean and Agile delivery practices helps practitionersto clearly see the linkage between theory in the 14 Lean Principles and the two pillars of lean and the practice of developing software.
    2. 14 Lean Principles (See my interview with Don Reinertsen on SPaMCAST and my review of his book, The Principles of Product Development Flow: Second Generation Lean Product Development). The 14 Lean Principles espoused by Reinertsen are a mechanism to remove waste from the flow of work. In the original TPS, versions of the HoQ this was reflect by an element called Reduction of Mudas (reduction of wastes). Reinertsen provides a set of principles that are more easily translated to software development and maintenance.
  5. A base which represents lean/Agile leadership. Many of the representations of the HoL/HoQ call the base management support. Leadership is far stronger than support. Leadership reflects a management that is trained in lean and Agile AND believes in lean and Agile. Belief is reflected in decisions that are philosophically in sync with the 12 Agile Principles and the 14 Principles of Product Development.

The House of Lean is a convenient container to hold the concepts and ideas that began as the Toyota Production System and have evolved as tools to be less manufacturing-oriented. The evolution of the HoL to include concepts and techniques familiar to Agile practitioners have not only helped to reduce muda and muri, but also is a useful tool to help reduce overhead when scaling Agile using frameworks like SAFe.


Categories: Process Management

Off-Time: It’s OK to Do Nothing

NOOP.NL - Jurgen Appelo - Tue, 03/03/2015 - 18:37

On my final trip last year, I had been looking forward to run in Rio de Janeiro, along Ipanema Beach and Copacabana. But British Airways lost my luggage. I had no running gear and no time to purchase alternative shoes and clothes. I felt a bit sad and disappointed.

The post Off-Time: It’s OK to Do Nothing appeared first on NOOP.NL.

Categories: Project Management

Sponsored Post: Apple, InMemory.Net, Sentient, Couchbase, VividCortex, Internap, Transversal, MemSQL, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Apple is hiring a Software Engineer for Maps Services. The Maps Team is looking for a developer to support and grow some of the core backend services that support Apple Map's Front End Services. Ideal candidate would have experience with system architecture, as well as the design, implementation, and testing of individual components but also be comfortable with multiple scripting languages. Please apply here.

  • Sentient Technologies is hiring several Senior Distributed Systems Engineers and a Senior Distributed Systems QA Engineer. Sentient Technologies, is a privately held company seeking to solve the world’s most complex problems through massively scaled artificial intelligence running on one of the largest distributed compute resources in the world. Help us expand our existing million+ distributed cores to many, many more. Please apply here.

  • Linux Web Server Systems EngineerTransversal. We are seeking an experienced and motivated Linux System Engineer to join our Engineering team. This new role is to design, test, install, and provide ongoing daily support of our information technology systems infrastructure. As an experienced Engineer you will have comprehensive capabilities for understanding hardware/software configurations that comprise system, security, and library management, backup/recovery, operating computer systems in different operating environments, sizing, performance tuning, hardware/software troubleshooting and resource allocation. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Rise of the Multi-Model Database. FoundationDB Webinar: March 10th at 1pm EST. Do you want a SQL, JSON, Graph, Time Series, or Key Value database? Or maybe it’s all of them? Not all NoSQL Databases are not created equal. The latest development in this space is the Multi Model Database. Please join FoundationDB for an interactive webinar as we discuss the Rise of the Multi Model Database and what to consider when choosing the right tool for the job.
Cool Products and Services
  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • Top Enterprise Use Cases for NoSQL. Discover how the largest enterprises in the world are leveraging NoSQL in mission-critical applications with real-world success stories. Get the Guide.
    http://info.couchbase.com/HS_SO_Top_10_Enterprise_NoSQL_Use_Cases.html

  • VividCortex Developer edition delivers a groundbreaking performance management solution to startups, open-source projects, nonprofits, and other organizations free of charge. It integrates high-resolution metrics on queries, metrics, processes, databases, and the OS and hardware to deliver an unprecedented level of visibility into production database activity.

  • SQL for Big Data: Price-performance Advantages of Bare Metal. When building your big data infrastructure, price-performance is a critical factor to evaluate. Data-intensive workloads with the capacity to rapidly scale to hundreds of servers can escalate costs beyond your expectations. The inevitable growth of the Internet of Things (IoT) and fast big data will only lead to larger datasets, and a high-performance infrastructure and database platform will be essential to extracting business value while keeping costs under control. Read more.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • Aerospike demonstrates RAM-like performance with Google Compute Engine Local SSDs. After scaling to 1 M Writes/Second with 6x fewer servers than Cassandra on Google Compute Engine, we certified Google’s new Local SSDs using the Aerospike Certification Tool for SSDs (ACT) and found RAM-like performance and 15x storage cost savings. Read more.

  • Diagnose server issues from a single tab. The Scalyr log management tool replaces all your monitoring and analysis services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. It's a universal tool for visibility into your production systems. Log aggregation, server metrics, monitoring, alerting, dashboards, and more. Not just “hosted grep” or “hosted graphs,” but enterprise-grade functionality with sane pricing and insane performance. Trusted by in-the-know companies like Codecademy – try it free! (See how Scalyr is different if you're looking for a Splunk alternative.)

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

The Cost Estimating Problem

Herding Cats - Glen Alleman - Tue, 03/03/2015 - 17:32

Screen Shot 2015-03-02 at 12.32.11 PM

In probability theory, de Finetti's theorem† explains why exchangeable observations are conditionally independent given some latent variable to which an epistemic probability distribution would then be assigned. It is named in honor of Bruno de Finetti.

It states that an exchangeable sequence of Bernoulli random variables is a "mixture" of independent and identically distributed (i.i.d.) Bernoulli random variables – while the individual variables of the exchangeable sequence are not themselves i.i.d., only exchangeable, there is an underlying family of i.i.d. random variables.

Thus, while observations need not be i.i.d. for a sequence to be exchangeable, there are underlying, generally unobservable, quantities which are i.i.d. – exchangeable sequences are (not necessarily i.i.d.) mixtures of i.i.d. sequences.

All of this actually has importance. When we start to assess risks using probabilistic process based on statistical processes, we need to be very careful to understand the underlying mathematics.

There are four approach to saying what we mean when we say “probability”

  1. Logical – weak implications
  2. Propensity – physical properties
  3. Frequency – attributed to sequences of observations
  4. Subjective – personal opinion

† Finetti’s Theorem is at the heart of estimating random variables. Cost is a random variable, like schedule durations and the technical outcomes from the effort based on cost and schedule. In statistical assessment of cost and schedule, Frequentist (counting) statistics is one approach. The second is Bayesian inference (used in most science). The exchangeability of the random variables is critical to building times series of sampled data from the project to forecast future performance.

The Bigger Problem in Estimating

Screen Shot 2015-03-02 at 12.38.40 PM

Exchangability and de Finetti's Theorem 

Finite Exchangeable Sequences

Categories: Project Management

101 Proven Practices for Focus

“Lack of direction, not lack of time, is the problem. We all have twenty-four hour days.” -- Zig Ziglar

Here is my collection of 101 Proven Practices for Focus.   It still needs work to improve it, but I wanted to shared it, as is, because focus is one of the most important skills we can develop for work and life.

Focus is the backbone of personal effectiveness, personal development, productivity, time management, leadership skills, and just about anything that matters.   Focus is a key ingredient to helping us achieve the things we set out to do, and to learn the things we need to learn.

Without focus, we can’t achieve great results.

I have a very healthy respect for the power of focus to amplify impact, to create amazing breakthroughs, and to make things happen.

The Power of Focus

Long ago one of my most impactful mentors said that focus is what separates the best from the rest.  In all of his experience, what exceptional people had, that others did not, was focus.

Here are a few relevant definitions of focus:
A main purpose or interest.
A center of interest or activity.
Close or narrow attention; concentration.

I think of focus simply as  the skill or ability to direct and hold our attention.

Focus is a Skill

Too many people think of focus as something either you are good at, or you are not.  It’s just like delayed gratification.

Focus is a skill you can build.

Focus is actually a skill and you can develop it.   In fact, you can develop it quite a bit.  For example, I helped a colleague get themselves off of their ADD medication by learning some new ways to retrain their brain.   It turned out that the medication only helped so much, the side effects sucked, and in the end, what they really needed was coping mechanisms for their mind, to better direct and hold their attention.

Here’s the surprise, though.  You can actually learn how to direct your attention very quickly.  Simply ask new questions.  You can direct your attention by asking questions.   If you want to change your focus, change the question.

101 Proven Practices at a Glance

Here is a list of the 101 Proven Practices for Focus:

  1. Align  your focus and your values
  2. Ask new questions to change your focus
  3. Ask yourself, “What are you rushing through for?”
  4. Beware of random, intermittent rewards
  5. Bite off what you can chew
  6. Breathe
  7. Capture all of your ideas in one place
  8. Capture all of your To-Dos all in one place
  9. Carry the good forward
  10. Change your environment
  11. Change your physiology
  12. Choose one project or one thing to focus on
  13. Choose to do it
  14. Clear away all distractions
  15. Clear away external distractions
  16. Clear away internal distractions
  17. Close your distractions
  18. Consolidate and batch your tasks
  19. Create routines to help you focus
  20. Decide to finish it
  21. Delay gratification
  22. Develop a routine
  23. Develop an effective startup routine
  24. Develop an effective shutdown routine
  25. Develop effective email routines
  26. Develop effective renewal activities
  27. Develop effective social media routines
  28. Direct your attention with skill
  29. Do less, focus more
  30. Do now what you could put off until later
  31. Do things you enjoy focusing on
  32. Do worst things first
  33. Don’t chase every interesting idea
  34. Edit later
  35. Exercise your body
  36. Exercise your mind
  37. Expand your attention span
  38. Find a way to refocus
  39. Find the best time to do your routine tasks
  40. Find your flow
  41. Finish what you started
  42. Focus on what you control
  43. Force yourself to focus
  44. Get clear on what you want
  45. Give it the time and attention it deserves
  46. Have a time and place for things
  47. Hold a clear picture in your mind of what you want to accomplish
  48. Keep it simple
  49. Keep your energy up
  50. Know the tests for success
  51. Know what’s on your plate
  52. Know your limits
  53. Know your personal patterns
  54. Know your priorities
  55. Learn to say no – to yourself and others
  56. Limit your starts and stops
  57. Limit your task switching
  58. Link it to good feelings
  59. Make it easy to pick back up where you left off
  60. Make it relentless
  61. Make it work, then make it right
  62. Master your mindset
  63. Multi-Task with skill
  64. Music everywhere
  65. Narrow your focus
  66. Pair up
  67. Pick up where you left off
  68. Practice meditation
  69. Put the focus on something bigger than yourself
  70. Rate your focus each day
  71. Reduce friction
  72. Reduce open work
  73. Reward yourself along the way
  74. See it, do it
  75. Set a time frame for focus 
  76. Set goals
  77. Set goals with hard deadlines
  78. Set mini-goals
  79. Set quantity limits
  80. Set time limits
  81. Shelve things you aren’t actively working on
  82. Single Task
  83. Spend your attention with skill
  84. Start with WHY
  85. Stop starting new projects
  86. Take breaks
  87. Take care of the basics
  88. Use lists to avoid getting overwhelmed or overloaded
  89. Use metaphors
  90. Use Sprints to scope your focus
  91. Use the Rule of Three
  92. Use verbal cues
  93. Use visual cues
  94. Visualize your performance
  95. Wake up at the same time each day
  96. Wiggle your toes – it’s a fast way to bring yourself back to the present
  97. Write down your goals
  98. Write down your steps
  99. Write down your tasks
  100. Write down your thoughts
  101. Work when you are most comfortable

When you go through the 101 Proven Practices for Focus, don’t expect it to be perfect.  It’s a work in progress.   Some of the practices for focus need to be fleshed out better.   There is also some duplication and overlap, as I re-organize the list and find better ways to group and label ideas.

In the future, I’m going to revamp this collection to have some more precision, better naming, and some links to relevant quotes, and some science where possible.   There is a lot more relevant science that explains why some of these techniques work, and why some work so well.

What’s important is that you find the practices that resonate for you, and the things that you can actually practice.

Getting Started

You might find that from all the practices, only one or two really resonate, or help you change your game.   And, that’s great.   The idea of having a large list to select from is that it’s more to choose from.  The bigger your toolbox, the more you can choose the right tool for the job.  If you only have a hammer, then everything looks like a nail.

If you don’t consider yourself an expert in focus, that’s fine.  Everybody has to start somewhere.  In fact, you might even use one of the practices to help you get better:  Rate your focus each day.

Simply rate yourself, on a scale of 1-10, where 10 is awesome and 1 means you’re a squirrel with a sugar high, dazed and confused, and chasing all the shiny objects that come into site.   And then see if your focus improves over the course of a week.

If you adopt just one practice, try either Align  your focus and your values or Ask new questions to change your focus.  

Feel Free to Share It With Friends

At the bottom of the 101 Proven Practices for Focus, you’ll find the standard sharing buttons for social media to make it easier to share.

Share it with friends, family, your world, the world.

The ability to focus is really a challenge for a lot of people.   The answer to improve your attention and focus is through proven practices, techniques, and skill building.  Too many people hope the answer lies in a pill, but pills don’t teach you skills.

Even if you struggle a bit in the beginning, remind yourself that growth feels awkward.   You' will get better with practice.  Practice deliberately.  In fact, the side benefit of focusing on improving your focus, is, well, you guessed it 
 you’ll improve your focus.

What we focus on expands, and the more we focus our attention, and apply deliberate practice, the deeper our ability to focus will grow.

Grow your focus with skill.

You Might Also Like

The Great Inspirational Quotes Revamped

The Great Happiness Quotes Collection Revamped

The Great Leadership Quotes Collection Revamped

The Great Love Quotes Collection Revamped

The Great Motivational Quotes Revamped

The Great Personal Development Quotes Collection Revamped

The Great Positive Thinking Quotes Collection

The Great Productivity Quotes Collection Revamped

Categories: Architecture, Programming

A product manager's perfection....

Xebia Blog - Tue, 03/03/2015 - 15:59

is achieved not there are no more features to add, but when there are no more features to take away. -- Antoine de Saint Exupéry

Not only was Antoine a brilliant writer, philosopher and pilot (well arguably since he crashed in the Mediterranean) but most of all he had a sharp mind about engineering, and I frequent quote him when I train product owners, product managers or in general product companies, about what makes a good product great. I also tell them their most important word in their vocabulary is "no". But the question then becomes, what is the criteria to say "yes"?

Typically we will look at the value of a feature and use different ways to prioritise and rank different features, break them down to their minimal proposition and get the team going. But what if you already have a product? and it’s rate of development is slowing. Features have been stacked on each other for years or even decades, and it’s become more and more difficult for the teams to wade through the proverbial swamp the code has become?

Too many features

Too many features

Turns out there are a number of criteria that you can follow:

1.) Working software, means it’s actually being used.

Though it may sound obvious, it’s not that easy to figure out. I was once part of a team that had to rebuild a rather large piece (read huge) of software for an air traffic control system. The managers ensured us that every functionality was a must keep, but the cost would have been prohibitory high.

One of the functions of the system was a record and replay mode for legal purposes. It basically registers all events throughout the system to serve as evidence that picture compilers would be accountable, or at least verifiable. One of our engineers had the bright insight that we could catalogue this data anonymously to figure out which functions were used and which were not.

Turned out the Standish Group was pretty right in their claims that 80% of the software is never used. Carving that out was met with fierce resistance, but it was easier to convince management (and users) with data, than with gut.

Another upside? we also knew what functions they were using a lot, and figured out how to improve those substantially.

2.) The cost of platforms

Yippee we got it running on a gazillion platforms! and boy do we have a reach, the marketing guys are going frenzy. Even if is the right choice at the time, you need to revisit this assumption all the time, and be prepared to clean up! This is often looked upon as a disinvestment: “we spent so much money on finally getting Blackberry working” or “It’s so cost effective that we can also offer it on platform XYZ”.

In the web world it’s often the number of browsers we support, but for larger software systems it is more often operating systems, database versions or even hardware. For one customer we would refurbish hardware systems, simply because it was cheaper than moving on to a more modern machine.

Key take away: If the platform is deprecated, remove it entirely from the codebase, it will bog the team down and you need their speed to respond to an ever increasing pace of new platforms.

3.) Old strategies

Every market and every product company pivots at least every few years (or dies). Focus shifts from consumer groups, to type of clients, type of delivery, shift to service or something else which is novel, hip and most of all profitable. Code bases tend to have a certain inertia. The larger the product, the bigger the inertia, and before you know it there a are tons of features in their that are far less valuable in the new situation. Cutting away perfectly good features is always painful but at some point you end up with the toolbars of Microsoft Word. Nice features, but complete overkill for the average user.

4.) The cause and effect trap

When people are faced with an issue they tend to focus on fixing the issue as it manifests itself. It's hard for our brain to think in problems, it tries to think in solutions. There is an excellent blog post here that provides a powerful method to overcome this phenomena by asking five times "why".

  • "We need the system to automatically export account details at the end of the day."
  • "Why?"
  • "So we can enter the records into the finance system"
  • "So it sounds like the real problem is getting the data into the finance system, not exporting it. Exporting just complicates the issue. Let's implement a data feed that automatically feeds the data to the finance system"

The hard job is to continuously keep evaluating your features, and remove those who are no longer valuable. It may seem like your throwing away good code, but ultimately it is not the largest product that survives, but the one that is able to adapt fast enough to the changing market. (Freely after Darwin)

 

The Virtue of Purgatory in Software Development

From the Editor of Methods & Tools - Tue, 03/03/2015 - 14:51
Having some decade of experience in software development behind me, I had the time to accumulate a lot of mistakes. One of the recurring patterns in these failures was the ambition to solve code issues too quickly. This was especially the case when the problem was related to code that I wrote, which made me feel responsible for the situation. Naturally, I had also often think that my code couldn’t be bad and somebody must have changed it after I deliver it, but this is another story ;O) When you detect ...

Four Tips for Managing Performance in Agile Teams

I’ve been talking with clients recently about their managers’ and HR’s transition to agile. I hear this common question: “How do we manage performance of the people on our agile teams?”

  1. Reframe “manage performance” to “career development.” People on agile teams don’t need a manager to manage their performance. If they are retrospecting at reasonable intervals, they will inspect-and-adapt to work better together. Well, they will if managers don’t interfere with their work by creating experts or moving people off project teams.
  2. The manager creates a trusting relationship with each person on the team. That means having a one-on-one weekly or bi-weekly with each person. At the one-on-one, the manager provides tips for feedback and offers coaching.  (If the person needs it or wants it from the manager.) The person might want to know where else he or she can receive coaching. The manager removes obstacles if the person has them. They discuss career development.
  3. When managers discuss career development, each person needs to see an accurate view of the value they bring to the organization. That means each person has to know how to give and receive feedback. They each have to know how to ask for and accept coaching. The manager provides meta-feedback and meta-coaching.
  4. If you, as a manager, meet with each person at least once every two weeks, no problem is a problem for too long. The people in the team have another person to discuss issues with. The manager sees the system and can change it to help the people on the team.

Now, what does this mean for raises?

I like to separate the raise from the feedback. People need feedback all the time, not just once a year. That’s why I like weekly or biweekly one-on-ones. Feedback isn’t just from the manager to the employee; it’s two-way feedback. If people have trouble working in the current environment, the managers might have a better chance to change it than an employee who is not a manager.

What about merit raises? This is tricky. So many managers and HR people continue to think one person is a star. No, on well-functioning agile teams, the team is the star—not individuals. You have options:

  • Make sure you pay each person at parity. This might not be trivial. You need expertise criteria for each job level.
  • When it comes to merit raises, provide a pot of money for the team and ask them to distribute it.
  • Distribute the merit money to each person equally. Explain that you are doing this, so people provide feedback to each other.
  • Here’s something radical: When people think they are ready for a raise or another level, have a discussion with the team. Let the team vote on it.

Managers have to not get in the way when it comes to “performance management.” The people on the team are adult humans. They somehow muddle through the rest of their lives, successfully providing and receiving feedback. They know the worth of things outside work. It’s just inside work that we keep salary secret.

It might not fit for you to have open-book salaries. On the other hand, how much do your managers and HR do that interferes with a team? You have to be careful about this.

If you reward individuals and ask people to work together as a team, how long do you think they will work together as a team? I don’t know the answer to that question.

Long ago, my managers asked me to be a “team player.”  One guy got a huge raise—and I didn’t, although I had saved his tush several times—I stopped working as a “team” member. I got my big raise the following year. (Year!) This incongruent approach is why people leave organizations—when the stated way “we work here” is not congruent with the stated goals: agile and self-organizing teams.

What do you want? Managers and HR to manage people? Or, to lead people using servant leadership, and let the teams solve their problems and manage their own performance?

If teams don’t know how to improve, that’s one thing. But, I bet your teams do know how to improve. You don’t have to manage their performance. You need to create an environment in which people can do their best work—that’s the manager’s job and the secret to “managing performance.”

Related posts:

 

Categories: Project Management

Quote of the Day

Herding Cats - Glen Alleman - Mon, 03/02/2015 - 20:09

GalileoChristina_edGalileo Galilei, Letter to the Grand Duchess Christina of Tuscany (1615)

.... Considering the force exerted by logical deductions, they may ascertain that it is not in the power of the professors of demonstrative sciences to change their opinions at will and apply themselves first to one side and then to the other.

There is a great difference between commanding a mathematician or a philosopher and influencing a lawyer or a merchant, for demonstrated conclusions about things in nature or in the heavens cannot be changed with the same facility as opinions about what is or is not lawful in a contract, bargain, or bill of exchange.

If those suggesting we abandon the principles of Microeconomics of Software Development (decision making in the presence of scarcity, abundance, and economic value)†, requiring that decisions made today with their impacts on future outcomes, do so without probabilistic knowledge of those impacts can be done in the absence of estimating those impacts - think again.

It Just Ain't So

† Software Project Effort Estimation: Foundations and Best Practice Guidelines for Success, May 7, 2014 by Adam Trendowicz and Ross Jeffery

Categories: Project Management

Change Your Life With this Free Blogging Course

Making the Complex Simple - John Sonmez - Mon, 03/02/2015 - 17:00

Around the mid December 2014, I decided to launch a completely free blogging course that would be delivered via email over 3 weeks. I had no idea how popular and successful that blogging course would turn out to be. (Here’s a good book to check out) At the time of writing this post, almost 3,000 […]

The post Change Your Life With this Free Blogging Course appeared first on Simple Programmer.

Categories: Programming

Tutum, first impressions

Xebia Blog - Mon, 03/02/2015 - 16:40

Tutum is a platform to build, run and manage your docker containers. After shortly playing with it some time ago, I decided to take a bit more serious look at it this time. This article describes first impressions of using this platform, more specifically looking at it from a continuous delivery perspective.

The web interface

First thing to notice is the clean and simple web interface. Basically there are two main sections, which are services and nodes. The services view lists the services or containers you have deployed with status information and two buttons, one to stop (or start) and one to terminate the container, which means to throw it away.

You can drill down to a specific service, which provides you with more detailed information per service. The detail page provides you information about the containers, a slider to scale up and down, endpoints, logging, some metrics for monitoring and more .

Screen Shot 2015-02-23 at 22.49.33

The second view is a list of nodes. The list contains the VM's on which containers can be deployed. Again with two simple buttons to start/stop and to terminate the node. For each node it displays useful information about the current status, where it runs, and how many containers are deployed on it.

The node page also allows you to drill down to get more information on a specific node.  The screenshot below shows some metrics in fancy graphs for a node, which can potentially be used to impress your boss.

Screen Shot 2015-02-23 at 23.07.30

 

Creating a new node

You’ll need a node to deploy containers on it. In the node view you see two big green buttons. One states: “Launch new node cluster”. This will bring up a form with currently four popular providers Amazon, Digital Ocean, Microsoft Azure and Softlayer. If you have linked your account(s) in the settings you can select that provider from a dropdown box. It only takes a few clicks to get a node up and running. In fact you create a node cluster, which allows you to easily scale up or down by adding or removing nodes from the cluster.

You also have an option to ‘Bring you own node’. This allows you to add your own Ubuntu Linux systems as nodes to Tutum. You need to install an agent onto your system and open up a firewall port to make your node available to Tutum. Again very easy and straight forward.

Creating a new service

Once you have created a node, you maybe want to do something with it. Tumtum provides jumpstart images with popular types of services for storage, cacheing, queueing and more, providing for example MongoDB, Elasticsearch or Tomcat. Using a wizard it takes only four steps to get a particular service up and running.

Besides the jumpstart images that Tutum provides, you can also search public repositories for your image of choice. Eventually you would like to have your own images running your homegrown software. You can upload your image to a Tutum private registry. You can either pull it from Docker Hub or upload your local images directly to Tutum.

Automating

We all know real (wo)men (and automated processes) don’t use GUI’s. Tutum provides a nice and extensive command line interface for both Linux and Mac. I installed it using brew on my MBP and seconds later I was logged in and doing all kinds of cool stuff with the command line.

Screen Shot 2015-02-24 at 22.23.30

The cli is actually doing rest calls, so you can skip the cli all together and talk HTTP directly to a REST API, or if it pleases you, you can use the python API to create scripts that are actually maintainable. You can pretty much automate all management of your nodes, containers, and services using the API, which is a must have in this era of continuous everything.

A simple deployment example

So let's say we've build a new version of our software on our build server. Now we want to get this software deployed to do some integration testing, or if you feeling lucky just drop it straight into production.

build the docker image

tutum build -t test/myimage .

upload the image to Tutum registry

tutum image push <image_id>

create the service

tutum service create <image_id>

run it on a node

tutum service run -p <port> -n <name> <image_id>

That's it. Of course there are lots of options to play with, for example deployment strategy, set memory, auto starting etc. But the above steps are enough to get your image build, deployed and run. Most time I had to spend was waiting while uploading my image using the flaky-but-expensive hotel wifi.

Conclusion for now

Tutum is clean, simple and just works. I’m impressed with ease and speed you can get your containers up and running. It takes only minutes to get from zero to running using the jumpstart services, or even your own containers. Although they still call it beta, everything I did just worked, and without the need to read through lots of complex documentation. The web interface is self explanatory and the REST API or cli provides everything you need to integrate Tutum in your build pipeline, so you can get your new features in production with automation speed.

I'm wondering how challenging managing would be at a scale of hundreds of nodes and even more containers, when using the web interface. You'd need a meta-overview or aggregate view or something. But then again, you have a very nice API to

Python: scikit-learn – Training a classifier with non numeric features

Mark Needham - Mon, 03/02/2015 - 08:48

Following on from my previous posts on training a classifier to pick out the speaker in sentences of HIMYM transcripts the next thing to do was train a random forest of decision trees to see how that fared.

I’ve used scikit-learn for this before so I decided to use that. However, before building a random forest I wanted to check that I could build an equivalent decision tree.

I initially thought that scikit-learn’s DecisionTree classifier would take in data in the same format as nltk’s so I started out with the following code:

import json
import nltk
import collections
 
from himymutil.ml import pos_features
from sklearn import tree
from sklearn.cross_validation import train_test_split
 
with open("data/import/trained_sentences.json", "r") as json_file:
    json_data = json.load(json_file)
 
tagged_sents = []
for sentence in json_data:
    tagged_sents.append([(word["word"], word["speaker"]) for word in sentence["words"]])
 
featuresets = []
for tagged_sent in tagged_sents:
    untagged_sent = nltk.tag.untag(tagged_sent)
    sentence_pos = nltk.pos_tag(untagged_sent)
    for i, (word, tag) in enumerate(tagged_sent):
        featuresets.append((pos_features(untagged_sent, sentence_pos, i), tag) )
 
clf = tree.DecisionTreeClassifier()
 
train_data, test_data = train_test_split(featuresets, test_size=0.20, train_size=0.80)
 
>>> train_data[1]
({'word': u'your', 'word-pos': 'PRP$', 'next-word-pos': 'NN', 'prev-word-pos': 'VB', 'prev-word': u'throw', 'next-word': u'body'}, False)
 
>>> clf.fit([item[0] for item in train_data], [item[1] for item in train_data])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/tree/tree.py", line 137, in fit
    X, = check_arrays(X, dtype=DTYPE, sparse_format="dense")
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/sklearn/utils/validation.py", line 281, in check_arrays
    array = np.asarray(array, dtype=dtype)
  File "/Users/markneedham/projects/neo4j-himym/himym/lib/python2.7/site-packages/numpy/core/numeric.py", line 460, in asarray
    return array(a, dtype, copy=False, order=order)
TypeError: float() argument must be a string or a number

In fact, the classifier can only deal with numeric features so we need to translate our features into that format using DictVectorizer.

from sklearn.feature_extraction import DictVectorizer
 
vec = DictVectorizer()
X = vec.fit_transform([item[0] for item in featuresets]).toarray()
 
>>> len(X)
13016
 
>>> len(X[0])
7302
 
>>> vec.get_feature_names()[10:15]
['next-word-pos=EX', 'next-word-pos=IN', 'next-word-pos=JJ', 'next-word-pos=JJR', 'next-word-pos=JJS']

We end up with one feature for every key/value combination that exists in featuresets.

I was initially confused about how to split up training and test data sets but it’s actually fairly easy – train_test_split allows us to pass in multiple lists which it splits along the same seam:

vec = DictVectorizer()
X = vec.fit_transform([item[0] for item in featuresets]).toarray()
Y = [item[1] for item in featuresets]
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.20, train_size=0.80)

Next we want to train the classifier which is a couple of lines of code:

clf = tree.DecisionTreeClassifier()
clf = clf.fit(X_train, Y_train)

I wrote the following function to assess the classifier:

import collections
import nltk
 
def assess(text, predictions_actual):
    refsets = collections.defaultdict(set)
    testsets = collections.defaultdict(set)
    for i, (prediction, actual) in enumerate(predictions_actual):
        refsets[actual].add(i)
        testsets[prediction].add(i)
    speaker_precision = nltk.metrics.precision(refsets[True], testsets[True])
    speaker_recall = nltk.metrics.recall(refsets[True], testsets[True])
    non_speaker_precision = nltk.metrics.precision(refsets[False], testsets[False])
    non_speaker_recall = nltk.metrics.recall(refsets[False], testsets[False])
    return [text, speaker_precision, speaker_recall, non_speaker_precision, non_speaker_recall]

We can call it like so:

predictions = clf.predict(X_test)
assessment = assess("Decision Tree", zip(predictions, Y_test))
 
>>> assessment
['Decision Tree', 0.9459459459459459, 0.9210526315789473, 0.9970134395221503, 0.9980069755854509]

Those values are in the same ball park as we’ve seen with the nltk classifier so I’m happy it’s all wired up correctly.

The last thing I wanted to do was visualise the decision tree that had been created and the easiest way to do that is export the classifier to DOT format and then use graphviz to create an image:

with open("/tmp/decisionTree.dot", 'w') as file:
    tree.export_graphviz(clf, out_file = file, feature_names = vec.get_feature_names())
dot -Tpng /tmp/decisionTree.dot -o /tmp/decisionTree.png


The decision tree is quite a few levels deep so here’s part of it:

DecisionTreeSection

The full script is on github if you want to play around with it.

Categories: Programming

A Mathematical Interlude

Herding Cats - Glen Alleman - Mon, 03/02/2015 - 06:50

The epitaph on Diophantus reads tomb:

This tomb holds Diophantus. Ah, what a marvel! And the tomb tells scientifically the measure of his life. God vouchsafed that he should be a boy for the sixth part part of his life; when a twelfth was added., his cheeks acquired a beard; he kindled for him the light of marriage after the seventh, and in the fifth year of his marriage He granted him a son. Alas! late-begotten and miserable child, when he had reached the measure of half his father's life, the chill grave took him. After consoling his grief by this science of numbers for four years, he reached the end of his life. Greek Mathematical Works II: Aristarchus to Pappus of Alexandria, Loeb Classical Library, Translated by Ivor Thomas, Harvard University Press, 1941.2

How old was Diophantus when he died and how old was his son? Let's assume the phrase half his father's life to mean half the total life, not half the life at the time of the sons death. 

Assume Diophantus lived to x years.

The least common denominator of the denominators is 84. Multiplying out all the terms with this common denominator gives

Grouping multipliers of x on one side and constants on the other gives.

Diophantus was X = 84 years old. He was a boy for 14 years and grew a beard after 7 more (21). Twelve year later he marries at age 33 and had a son 5 years after that (38). The son died at the age of 42, when Diophantus was 80. Diophantus dies at the age of 84.

Related articles Software Engineering is a Verb Quote of the Day Self-Organization
Categories: Project Management

The Great Positive Thinking Quotes Collection

I revamped my positive thinking quotes collection on Sources of Insight to help you amp up your ability to generate more positive thoughts. 

It’s a powerful one.

Why positive thinking?

Maybe Zig Ziglar said it best:

“Positive thinking will let you do everything better than negative thinking will.”

Positive Thinking Can Help You Defeat Learned Helplessness

Actually, there’s a more important reason for positive thinking: 

It’s how you avoid learned helplessness.

Learned helplessness is where you give up, because you don’t think you have any control over the situation, or what happens in your life.  You explain negative situations as permanent, personal, and pervasive, instead of temporary, situational, and specific.

That’s a big deal.

If you fall into the learned helplessness trap, you spiral down.  You stop taking action.  After all, why take action, if it won’t matter.  And, this can lead to depression.

But that’s a tale of woe for others, not you.   Because you know how to defeat learned helplessness and how to build the skill of learned optimism.

You can do it by reducing negative thinking, and by practicing positive thinking.   And what better way to improve your positive thinking, than through positive thinking quotes.

Keep a Few Favorite Positive Thinking Quotes Handy

Always keep a few positive thinking quotes at your fingertips so that they are there when you need them.

Here is a quick taste of a few of my favorites from the positive thinking quotes collection:

“A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort.” –  Herm Albright

"Attitudes are contagious.  Are yours worth catching?" — Dennis and Wendy Mannering

"Be enthusiastic.  Remember the placebo effect – 30% of medicine is showbiz." — Ronald Spark

"I will not let anyone walk through my mind with their dirty feet." — Mahatma Gandhi

"If the sky falls, hold up your hands." – Author Unknown

Think Deeper About Positivity By Using Categories for Positive Thinking

But this positive thinking quotes collection is so much more.  I’ve organized the positive thinking quotes into a set of categories to chunk it up, and to make it more insightful:

Actions
Adaptability and Flexibility
Anger and Frustration
Appreciation and Gratitude
Attitude, Disposition, and Character
Defeat, Setbacks, and Failures
Expectations
Focus and Perspective
Hope and Fear
Humor
Letting Things Go and Forgiveness
Love and Liking
Opportunity and Possibility
Positive Thinking (General)

The distinctions you can add to your mental repertoire, the more powerful of a positive thinker you will be.

You can think of each positive thinking quote as a distinction that can add more depth.

Draw from Wisdom of the Ages and Modern Sages on the Art and Science of Positive Thinking

I've included positive thinking quotes from a wide range of people including  Anne Frank, Epictetus, Johann Wolfgang von Goethe, Napoleon, Oscar Wilde, Ralph Waldo Emerson, Robert Frost, Voltaire, Winston Churchill, and many, many more.

You might even find useful positive thinking mantras from the people that you work with, or the people right around you. 

For example, here are a few positive thinking thoughts from Satya Nadella:

“The future we're going to invent together, express ourselves in the most creative ways.”

“I want to work in a place where everybody gets more meaning out of their work on an everyday basis.

I want each of us to give ourselves permission to be able to move things forward.  Each of us sometimes overestimate the power others have to do things vs. our own ability to make things happen.

Challenge Yourself to New Levels of Positive Thinking through Positive Quotes

As you explore the positive thinking quotes collection, try to find the quotes that challenge you the most, that really make you think, and give you a new way to generate more positive thoughts in your worst situations. 

In the words of Friedrich Nietzsche, "That which does not kill us makes us stronger."

You can use your daily trials and tribulations in the workplace as your personal dojo to practice and build your positive thinking skills.

The more positivity you can bring to the table, the more you’ll empower yourself in ways you never thought possible.

As you get tested by your worst scenarios, it’s good to keep in mind, the words of F. Scott Fitzgerald:

"The test of a first-rate intelligence is the ability to hold two opposed ideas in the mind at the same time, and still retain the ability to function.  One should, for example, be able to see that things are hopeless and yet be determined to make them otherwise."

I’ll also point out that as you grow your toolbox of positive thinking quotes and you build your positive thinking skills, you need to also focus on taking positive action.

Don’t just imagine a better garden, get out and actually weed it.

Don’t Just Imagine Everything Going Well – Imagine How You’ll Deal with the Challenges

Here’s another important tip about positivity and positive thinking 


If you use visualization as part of your approach to getting results, it’s important to include dealing with setbacks and challenges.   It’s actually more effective to imagine the most likely challenges coming up, and walking through how you’ll deal with them, if they occur.   This is way more effective than just picturing the perfect plan where everything goes without a hitch.

The reality is things happen, stuff comes up, and setbacks occur.

But your ability to mentally prepare for the setbacks, and have a plan of action, will make you much more effective in dealing with the challenge that actually do occur.  This will help you respond vs. react in more situations, and to stay in a better place mentally while you evaluate options, and decide a course of action.  (Winging it under stress doesn’t work very well because we shut down our prefrontal cortex – the smart part of our brain – when we go into flight-or-fight mode.)

If I missed any of your favorite positive thinking quotes in my positive thinking quotes collection, please let me know.

In closing, please keep handy one of the most powerful positive thinking quotes of all time:

“May the force be with you.”

Always.

You Might Also Like

The Great Inspirational Quotes Revamped

The Great Happiness Quotes Collection Revamped

The Great Leadership Quotes Collection Revamped

The Great Love Quotes Collection Revamped

The Great Motivational Quotes Revamped

The Great Personal Development Quotes Collection Revamped

The Great Productivity Quotes Collection Revamped

Categories: Architecture, Programming

SPaMCAST 331 – Coaching Not Managing, Microservices, Channels

www.spamcast.net

http://www.spamcast.net

Listen Now

Subscribe on iTunes

This week’s Software Process and Measurement Cast is a magazine feature with three columns. This week we have columns from Gene Hughson – Form Follows Function, completing a three-column arc on microservices. In Jo Ann Sweeney’s new Explaining Change column, Jo Ann tackles the concept of communication channels. The SPaMCAST essay this week is on Agile Coaching. Coaches help teams and projects deliver the most value, however many times organizations eschew coaches or conflate management and coaching.  This week we will have an external coach versus management death match!

Contest

We are having a contest! Anthony has offered a copy of his great new book to a randomly selected SPaMCAST listener, ANYWHERE IN THE WORLD.  Enter between February 22th and March 7th.  The winner will be announced on March 8th.  If you want a copy of Agile Project Management you have two options: send your name and email address to spamcastinfor@gmail.com (I will act as the broker and notify the winner at which point we can deal with other types of addresses), OR you can buy a copy.  Remember buying a copy through the Software Process and Measurement Cast helps support the podcast.

Dead Tree Version or Kindle Version

Call to action!

Can you tell a friend about the podcast? This week Julie Davis introduced two of her co-workers to the podcast and then emailed us at spamcastinfo@gmail.com.  Welcome and Joe and Cindy! Pictures of you and your friends listening to the podcast would be great. If your friends don’t know how to subscribe or listen to podcast, show them how you listen to the Software Process and Measurement Cast and subscribe them!  Remember to send us the name of you person you subscribed (and a picture) and I will give both you and the horde you have converted to listeners a call out on the show.

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

Dead Tree Version or Kindle Version 

Upcoming Events

CMMI Institute Conference EMEA 2015
March 26 -27 London, UK
I will be presenting “Agile Risk Management.”
http://cmmi.unicom.co.uk/

International Conference on Software Quality and Test Management
Washington D.C. May 31 – June 5, 2015
Wednesday June 3, 2015

http://qualitymanagementconference.com/

I will be presenting a new and improved version of “The Impact of Cognitive Biases on Test and Project Teams.”

Next SPaMCast

In the next Software Process and Measurement Cast we will feature our interview with Shirly Ronen-Harel. We began by talking about the book she co-authored (or is co-authoring) The Coaching Booster, which is 80% complete on LeanPub. We branched out into other topics including coaching, lean, Agile and using lean and Agile in startups. This was an incredibly content-rich podcast.  Have your notepad ready when you listen because Shirly provides ideas and advice that can change how you work!

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

Available in English and Chinese.


Categories: Process Management

SPaMCAST 331 - Coaching Not Managing, Microservices, Channels

Software Process and Measurement Cast - Sun, 03/01/2015 - 23:00

This week’s Software Process and Measurement Cast is a magazine feature with three columns. This week we have columns from Gene Hughson – Form Follows Function, completing a three-column arc on microservices. In Jo Ann Sweeney’s new Explaining Change column, Jo Ann tackles the concept of communication channels. The SPaMCAST essay this week is on Agile Coaching. Coaches help teams and projects deliver the most value, however many times organizations eschew coaches or conflate management and coaching.  This week we will have an external coach versus management death match!

Contest

We are having a contest! Anthony has offered a copy of his great new book to a randomly selected SPaMCAST listener, ANYWHERE IN THE WORLD.  Enter between February 22th and March 7th.  The winner will be announced on March 8th.  If you want a copy of Agile Project Management you have two options: send your name and email address to spamcastinfor@gmail.com (I will act as the broker and notify the winner at which point we can deal with other types of addresses), OR you can buy a copy.  Remember buying a copy through the Software Process and Measurement Cast helps support the podcast.

Dead Tree Version or Kindle Version

Call to action!

Can you tell a friend about the podcast? This week Julie Davis introduced two of her co-workers to the podcast and then emailed us at spamcastinfo@gmail.com.  Welcome and Joe and Cindy! Pictures of you and your friends listening to the podcast would be great. If your friends don’t know how to subscribe or listen to podcast, show them how you listen to the Software Process and Measurement Cast and subscribe them!  Remember to send us the name of you person you subscribed (and a picture) and I will give both you and the horde you have converted to listeners a call out on the show.

 

Re-Read Saturday News

The Re-Read Saturday focus on Eliyahu M. Goldratt and Jeff Cox’s The Goal: A Process of Ongoing Improvement began on February 21nd. The Goal has been hugely influential because it introduced the Theory of Constraints, which is central to lean thinking. The book is written as a business novel. Visit the Software Process and Measurement Blog and catch up on the re-read.

Note: If you don’t have a copy of the book, buy one.  If you use the link below it will support the Software Process and Measurement blog and podcast.

 Dead Tree Version or Kindle Version 

Upcoming Events

CMMI Institute Conference EMEA 2015
March 26 -27 London, UK
I will be presenting “Agile Risk Management.”
http://cmmi.unicom.co.uk/

International Conference on Software Quality and Test Management
Washington D.C. May 31 - June 5, 2015
Wednesday June 3, 2015
http://qualitymanagementconference.com/

I will be presenting a new and improved version of “The Impact of Cognitive Biases on Test and Project Teams.”

Next SPaMCast

In the next Software Process and Measurement Cast we will feature our interview with Shirly Ronen-Harel. We began by talking about the book she co-authored (or is co-authoring) The Coaching Booster, which is 80% complete on LeanPub. We branched out into other topics including coaching, lean, Agile and using lean and Agile in startups. This was an incredibly content-rich podcast.  Have your notepad ready when you listen because Shirly provides ideas and advice that can change how you work!

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: “This book will prove that software projects should not be a tedious process, neither for you or your team.” Support SPaMCAST by buying the book here.

 

Available in English and Chinese.

Categories: Process Management

Python: Detecting the speaker in HIMYM using Parts of Speech (POS) tagging

Mark Needham - Sun, 03/01/2015 - 03:36

Over the last couple of weeks I’ve been experimenting with different classifiers to detect speakers in HIMYM transcripts and in all my attempts so far the only features I’ve used have been words.

This led to classifiers that were overfitted to the training data so I wanted to generalise them by introducing parts of speech of the words in sentences which are more generic.

First I changed the function which generates the features for each word to also contain the parts of speech of the previous and next words as well as the word itself:

def pos_features(sentence, sentence_pos, i):
    features = {}
 
    features["word"] = sentence[i]
    features["word-pos"] = sentence_pos[i][1]
 
    if i == 0:
        features["prev-word"] = "<START>"
        features["prev-word-pos"] = "<START>"
    else:
        features["prev-word"] = sentence[i-1]
        features["prev-word-pos"] = sentence_pos[i-1][1]
 
    if i == len(sentence) - 1:
        features["next-word"] = "<END>"
        features["next-word-pos"] = "<END>"
    else:
        features["next-word"] = sentence[i+1]
        features["next-word-pos"] = sentence_pos[i+1][1]
 
    return features

Next we need to tweak our calling code to calculate the parts of speech tags for each sentence and pass it in:

featuresets = []
for tagged_sent in tagged_sents:
    untagged_sent = nltk.tag.untag(tagged_sent)
    sentence_pos = nltk.pos_tag(untagged_sent)
    for i, (word, tag) in enumerate(tagged_sent):
        featuresets.append((pos_features(untagged_sent, sentence_pos, i), tag) )

I’m using nltk to do this and although it’s slower than some alternatives, the data set is small enough that it’s not an issue.

Now it’s time to train a Decision Tree with the new features. I created three variants – one with both words and POS; one with only words; one with only POS.

I took a deep copy of the training/test data sets and then removed the appropriate keys:

def get_rid_of(entry, *keys):
    for key in keys:
        del entry[key]
 
import copy
 
# Word based classifier
tmp_train_data = copy.deepcopy(train_data)
for entry, tag in tmp_train_data:
    get_rid_of(entry, 'prev-word-pos', 'word-pos', 'next-word-pos')
 
tmp_test_data = copy.deepcopy(test_data)
for entry, tag in tmp_test_data:
    get_rid_of(entry, 'prev-word-pos', 'word-pos', 'next-word-pos')
 
c = nltk.DecisionTreeClassifier.train(tmp_train_data)
c.classify(tmp_test_data)
 
# POS based classifier
tmp_train_data = copy.deepcopy(train_data)
for entry, tag in tmp_train_data:
    get_rid_of(entry, 'prev-word', 'word', 'next-word')
 
tmp_test_data = copy.deepcopy(test_data)
for entry, tag in tmp_test_data:
    get_rid_of(entry, 'prev-word', 'word', 'next-word')
 
c = nltk.DecisionTreeClassifier.train(tmp_train_data)
c.classify(tmp_test_data)

The full code is on my github but these were the results I saw:

$ time python scripts/detect_speaker.py
Classifier              speaker precision    speaker recall    non-speaker precision    non-speaker recall
--------------------  -------------------  ----------------  -----------------------  --------------------
Decision Tree All In             0.911765          0.939394                 0.997602              0.996407
Decision Tree Words              0.911765          0.939394                 0.997602              0.996407
Decision Tree POS                0.90099           0.919192                 0.996804              0.996008

There’s still not much in it – the POS one has slightly more false positives and false positives when classifying speakers but on other runs it performed better.

If we take a look at the decision tree that’s been built for the POS one we can see that it’s all about POS now as you’d expect:

>>> print(c.pseudocode(depth=2))
if next-word-pos == '$': return False
if next-word-pos == "''": return False
if next-word-pos == ',': return False
if next-word-pos == '-NONE-': return False
if next-word-pos == '.': return False
if next-word-pos == ':':
  if prev-word-pos == ',': return False
  if prev-word-pos == '.': return False
  if prev-word-pos == ':': return False
  if prev-word-pos == '<START>': return True
  if prev-word-pos == 'CC': return False
  if prev-word-pos == 'CD': return False
  if prev-word-pos == 'DT': return False
  if prev-word-pos == 'IN': return False
  if prev-word-pos == 'JJ': return False
  if prev-word-pos == 'JJS': return False
  if prev-word-pos == 'MD': return False
  if prev-word-pos == 'NN': return False
  if prev-word-pos == 'NNP': return False
  if prev-word-pos == 'NNS': return False
  if prev-word-pos == 'POS': return False
  if prev-word-pos == 'PRP': return False
  if prev-word-pos == 'PRP$': return False
  if prev-word-pos == 'RB': return False
  if prev-word-pos == 'RP': return False
  if prev-word-pos == 'TO': return False
  if prev-word-pos == 'VB': return False
  if prev-word-pos == 'VBD': return False
  if prev-word-pos == 'VBG': return False
  if prev-word-pos == 'VBN': return True
  if prev-word-pos == 'VBP': return False
  if prev-word-pos == 'VBZ': return False
if next-word-pos == '<END>': return False
if next-word-pos == 'CC': return False
if next-word-pos == 'CD':
  if word-pos == '$': return False
  if word-pos == ',': return False
  if word-pos == ':': return True
  if word-pos == 'CD': return True
  if word-pos == 'DT': return False
  if word-pos == 'IN': return False
  if word-pos == 'JJ': return False
  if word-pos == 'JJR': return False
  if word-pos == 'JJS': return False
  if word-pos == 'NN': return False
  if word-pos == 'NNP': return False
  if word-pos == 'PRP$': return False
  if word-pos == 'RB': return False
  if word-pos == 'VB': return False
  if word-pos == 'VBD': return False
  if word-pos == 'VBG': return False
  if word-pos == 'VBN': return False
  if word-pos == 'VBP': return False
  if word-pos == 'VBZ': return False
  if word-pos == 'WDT': return False
  if word-pos == '``': return False
if next-word-pos == 'DT': return False
if next-word-pos == 'EX': return False
if next-word-pos == 'IN': return False
if next-word-pos == 'JJ': return False
if next-word-pos == 'JJR': return False
if next-word-pos == 'JJS': return False
if next-word-pos == 'MD': return False
if next-word-pos == 'NN': return False
if next-word-pos == 'NNP': return False
if next-word-pos == 'NNPS': return False
if next-word-pos == 'NNS': return False
if next-word-pos == 'PDT': return False
if next-word-pos == 'POS': return False
if next-word-pos == 'PRP': return False
if next-word-pos == 'PRP$': return False
if next-word-pos == 'RB': return False
if next-word-pos == 'RBR': return False
if next-word-pos == 'RBS': return False
if next-word-pos == 'RP': return False
if next-word-pos == 'TO': return False
if next-word-pos == 'UH': return False
if next-word-pos == 'VB': return False
if next-word-pos == 'VBD': return False
if next-word-pos == 'VBG': return False
if next-word-pos == 'VBN': return False
if next-word-pos == 'VBP': return False
if next-word-pos == 'VBZ': return False
if next-word-pos == 'WDT': return False
if next-word-pos == 'WP': return False
if next-word-pos == 'WRB': return False
if next-word-pos == '``': return False

I like that it’s identified the ‘:‘ pattern:

if next-word-pos == ':':
  ...
  if prev-word-pos == '<START>': return True

Next I need to drill into the types of sentence structures that it’s failing on and work out some features that can handle those. I still need to see how well a random forest of decision trees would as well.

Categories: Programming

Re-Read Saturday: The Goal: A Process of Ongoing Improvement. Part 2

IMG_1249

Chapters 1 through 3 actively present the reader with a burning platform. The plant and division are failing. Alex Rogo has actively pursued increased efficiency and automation to generate cost reductions, however performance is falling even further behind and fear has become central feature in the corporate culture. If the book stopped here it would be brief tragedy, however Chapter 4 begins the path towards the redemption of Alex Rogo and the ideas that are the bedrock of lean.

New Characters

  • Jonah – advisor
  • Lou – plant’s chief accountant

Chapter 4:

In Chapter 3, Alex was at a company meeting that communicated the depths of the problems the division was having and to search for answers. The meeting was not holding Alex’s attention. He found a cigar in his jacket and flashed back to a chance meeting in an airport lounge. While smoking a cigar, Alex recognizes and strikes up a conversation with a professor from grad school. The discussion turns to the problems at the plant, and even though they have pursued changes that have yielded great efficiencies the problems still exist and perhaps are getting worse. Alex proudly shows Jonah a chart that “proves” that a 36% improvement in efficiency from using robots and automation. Jonah asks one very simple question: was profit up 36% too? Alex struggles and answers that, “it is not that simple.” In fact the number of people in the plant did not go down, inventory did not go down and not one additional widget had been shipped. This interaction foreshadows one of the key ideas that The Goal presents to the reader. We are measuring the wrong thing! When we measure the wrong thing we send the wrong message and we get the wrong results. Chapter 4 closes with Johan and Rogo talking about the real meaning of productivity. Productivity is defined as accomplishing something in terms of a goal. Without knowing the goal, measuring productivity and efficiency is meaningless.

Chapter 5:

In Chapter 5 we snap back to the all-day meeting to discuss the division’s performance (Chapter 3). Alex continues to ruminate on Jonah’s comments. Alex leaves the meeting under the pretext of a problem back at the plant. As he drives back to the plant he begins to reflect on the “the goal” in the definition of productivity identified in Chapter 4.  As Alex drives back he decides that he will not have time to think due to the day-to-day demands (also known as the tyranny of the urgent but not important – See Habit 3 of Stephen Covey) therefore heads to a favorite pizzeria for pizza and beer. Goldratt and Cox use Alex’s inner dialog show why most of current internal goals and measures Alex is being asked to pursue miss the point. The bottom-line goal is that the plant’s goal is to make money. If it does not make money, the rest does not matter. While The Goal is set in a manufacturing plant, the point is that unless any group or department does not materially impact the real goal of an organization it should not exist.

Chapter 6:

Chapter 6 begins with a search for the overall measures that contribute (or predict) whether the plant is meeting the goal of profitability. One the first questions Alex poses to himself is whether he can assume that making people work and making money are the same thing. This sounds like a funny question, however I often see managers and leaders mistake being busy with delivering value. Alex and Lou brainstorm a set of 3 metrics that impact the goal. They are: 1. net profit, 2. ROI, 3. cash flow. In this conversation Alex tells Lou the truth about the state of the division and the potential closure of the plant. The 3 metrics sound right, however Alex does not see the immediate connection between the measures and day-to-day operations. The chapter ends with Alex asking the 3rd Shift Supervisor how is activities impact net profit, ROI and cash flow. He simply gets the deer-in-the-headlight look.

Chapters 4 – 6 shift the focus from steps in the process to the process as a whole. Organizations have an ultimate goal. In this case the ultimate goal of the plant is make money. The goal is not quality, efficiency or even employment because in the long-run if the plant doesn’t deliver product that can be sold it won’t exist. Whether an organization is for-profit or non-profit, if they don’t attain their ultimate goal they won’t exist.


Categories: Process Management