Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

Stuff The Internet Says On Scalability For January 20th, 2017

Hey, it's HighScalability time:

 

Absolutely. Do we agree that the cerebellum is amazingly beautiful? (@PeppeGanga)
If you like this sort of Stuff then please support me on Patreon.
  • 900 GB: data stolen in Cellebrite hack; 99.24%: users identified by cross-browser fingerprinting; 72%: intend to migrate to a hybrid cloud; 90%: Google & Facebook ad traffic is useless; 5.2 terabytes per second: data from Australian Square Kilometre Array Pathfinder; 10 billion: searches on DuckDuckGo in 2016; $330m: Amazon's loss on Alexa; 

  • Quotable Quotes:
    • @brucel: Breaking: Programmer accused of writing unreadable code refuses to comment.
    • @asymco: Remember Android first? App Annie believes the Apple’s App Store produced about twice as much revenue as Google Play
    • @bridgetkromhout: Describing your old-timer ranting as "greybeard" just makes me want to fight you with sed & awk at twenty paces. Be there tomorrow at dawn.
    • @StevenShorrock: Root Cause Analysis is: * Acceptable for simple systems * Inappropriate for complicated systems * Ludicrous for complex systems
    • @swardley: Five years ago Amazon was worth about half of Walmart, today Walmart is worth about half of Amazon.
    • Eric Raymond: In practice, I found Rust painful to the point of unusability. The learning curve was far worse than I expected; it took me those four days of struggling with inadequate documentation to write 67 lines of wrapper code for the server.
    • @swardley: past history shows many major players won't announce they're getting into the battle until some time after war has ended
    • @benthompson: Apple wasn't billed as phone maker / Amazon wasn't billed as infrastructure provider / FB wasn't billed as portal / Snapchat wasn't billed as TV
    • Jessitron: the biggest consideration in choosing whether to use libraries or services for distribution of effort / modularization is that choice of who decides when it deploys. Who controls which code is in production at a given time.
    • Hi Ben: The disruption of TV will follow a similar path: a different category will provide better live sports, better story-telling, or better escapism. Said category will steal attention, and when TV no longer commands enough attention of enough people, the entire edifice will collapse. Suddenly.
    • @leonidasfromxiv: I also don't understand why people compare Go with Rust. If you need a GC-less programming language: Rust; if you need a board game: Go.
    • Carlo Rovelli: The world isn’t just a mass of colliding atoms; it is also a web of correlations between sets of atoms, a network of reciprocal physical information between physical systems.
    • Chris Dixon: In the beginning, hardware-focused companies make gadgets with ever increasing laundry lists of features. Then a company with strong software expertise (often a new market entrant) comes along that replaces these feature-packed gadgets with full-fledged computers. 
    • Animats: The real question is "what do we do with a lot of CPUs without shared memory?" Such hardware has been built many times - Thinking Machines, Ncube, the PS2's Cell - and has not been too useful for general purpose computing.
    • @taavet: Very unfortunate that incumbents see tech only as a way to cut costs. Versus seeing tech to offer much better products.
    • NelsonMinar: This is what security looks like when your threat model is well funded government agencies.
    • Don Norman: The solution requires a different approach to the design of automation: collaboration. Instead of automating what can be automated, leaving the rest to the driver, we must develop collaborative systems so that the driver is continually involved in giving high-level guidance, thereby always staying active, always being in the loop. 
    • Thomas Frey: It took 50 years for the world to install the first million industrial robots. The next million will take only eight. Will this cause more jobs or few jobs in the future? I'm not convinced we know the answer.
    • @jtauber: "Every shot in Piper is composed of millions of grains of sand, each one of them around 5000 polygons."
    • rackforms: my point is the current situation, basically 2 companies controlling so much traffic, seems, well, bad for small business in this country. I value what they bring to the table and fully understand why they're so popular. But is things keep on this way where does that lead the guys like me? Is this just the way it has to be? Is the dream of the open Internet already dead?
    • @sheeshee: I think I know why it's called "DevOps" - "DevOops" was too obvious... ;)
    • greenspot: The open solution to a faster mobile web would have been so easy: Just penalize large and slow web pages without defining a dedicated mobile specification. That's it. This wasn't done in the past, slow pages outperformed fast ones on the SERPs because of some weird Google voodoo ranking, heck sometimes even desktop sites outperformed responsive ones on smartphones. If they had just tweaked these odd ranking rules in way that speed and size got more impact on the overall ranking there wouldn't have been any reason for AMP—the market would have regulated itself.
    • Juergen Schmidhuber: General purpose quantum computation won’t work (my prediction of 15 years ago is still standing). Related: The universe is deterministic, and the most efficient program that computes its entire history is short and fast, which means there is little room for true randomness, which is very expensive to compute. What looks random must be pseudorandom, like the decimal expansion of Pi, which is computable by a short program. Many physicists disagree, but Einstein was right: no dice. There is no physical evidence to the contrary

  • RethinkDB is shutting down and here's the post-portem. Lessons: the database market is like Mad Max fighting in the Thunderdome; it's better to optimize for useless microbenchmarks than it is to be good; optimism isn't a strategy.

  • Apple isn't alone in using custom hardware to thwart nation state level attackers. Google Infrastructure Security Design Overview. Good overview at Google reveals its servers all contain custom security silicon. Google designs "custom chips, including a hardware security chip that is currently being deployed on both servers and peripherals. These chips allow us to securely identify and authenticate legitimate Google devices at the hardware level."  Google encrypts data before it is written to disk, to make it harder for malicious disk firmware to access data. Google uses automated and manual code review techniques. Google uses automated software and code reviews to detect bugs in software its developers write. Google scans user-installed apps, downloads, browser extensions, and content browsed from the web for suitability on corp clients. Google uses a custom version of the KVMhypervisor. Good discussion on HackerNews, where a lot of the comments are on how Google needs this level sophistication to evade the prying eyes of governments.

  • What happens when you embed machine learning into a DBMS in order to continuously optimise its runtime performance? You get Self-driving database management systems. Humans suck at tuning databases so this is just one more job AIs will eventually toss into the dust bin of history. TensorFlow was integrated inside Peleton training two RNNs on 52 million queries from one month of traffic for a popular site. Does it help?: early results are promising: (1) RNNs accurately predict the expected arrival rate of queries. (2) hardware-accelerated training has a minor impact on the DBMS’s CPU and memory resources, and (3) the system deploys actions without slowing down the application. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Emotional Intelligence Is Not Always The Right Tool

30065778560_cd98edf55f_k

Emotional intelligence often touted as a tool that can be used to make every outcome better. However, the more academic literature suggests economic intelligence is not a panacea.  There are numerous papers that identify scenarios in which emotional Intelligence has not discernible impact on business outputs and might actually get in the way. Several are described below:

Emotional Intelligence in Non-Emotional Environments

Dana Joseph of the University of Central Florida and Daniel Newman of the University of Illinois comprehensively analyzed studies that showed a link between Emotional Intelligence and job performance. The study of studies found that the level found emotional intelligence was not consistently related to job performance.  However when the data was broken down into jobs that require an extensive understanding of emotions and those that did not the analysis told a different story. When the studies for jobs requiring extensive attention to emotions were analyzed separately, higher levels of emotional intelligence translated into better performance. Examples of jobs requiring significant levels of emotional intelligence include salespeople, call center representatives and other customer-facing roles. While not included in the studies we would expect project managers and Scrum masters in this group of roles. Alternately, when personnel in the study with higher levels of emotional intelligence in jobs that involved fewer emotional demands were studied they were found to have lower performance.  The authors suggest that spending time reading emotions of others when you should be doing something more introspective, such as writing code, is counterproductive.

Emotional Intelligence as a Tool for Manipulation

One often quoted study supporting this potential problem was published by University of Cambridge professor Jochen Menges.  Dr. Menges found that when a leader gave an inspiring speech filled with emotion, the audience was less likely to scrutinize the message and remembered less of the content. People that hone their emotional skills are often better at manipulating others. While leaders often manipulate emotions to generate a good output for an organization, examples abound of those leveraging their understanding the emotions of others to motivate them to act against their own best interests.

There are a number of other areas of concern noted in a recent article on the Harvard Business review blog.

  1. The potential impact of emotional intelligence on creativity and innovation. The article points out that attributes typically associated with creativity and innovation are generally at odds with those associated with emotional intelligence.  Emotionally intelligent people are more apt to stay within bounds of common process and go along with others because those actions are less likely to cause emotional pain.  Whereas innovation and creativity often require breaking pushing boundaries and breaking rules. Steve Jobs was not an emotionally intelligent person, he was an innovator.
  2. People with high emotional intelligence tend to have difficulty giving or taking negative or difficult feedback.  It is emotionally painful to deliver negative feedback and people with high emotional intelligence feel that pain, and as a general rule people avoid pain whenever possible.
  3. Similarly, people with high emotional intelligence tend to not to make unpopular decisions for much the same reason as they avoid delivering negative feedback. While you might put an emotionally intelligent person in charge of communication a corporate downsizing they probably wouldn’t the best person to plan and execute the downsizing because it would be painful. 
  4. People with high levels of emotional intelligence tend to avoid taking risks. People with high emotional intelligence avoid taking risks because they want to avoid negative emotional feedback if the risk comes to fruition.

Emotional intelligence can be used to manipulate individuals or teams.  The natural tendency is to consider manipulation as a negative; however, when an emotionally intelligent leader helps position a team to avoid a negative emotional problem they are acting in the best interest in a team. Manipulation, in this case, would be good (and would probably be called leadership).  For example, I often ask team members to remember and share positive events prior to team meetings. The goal is to increase sharing, cause bonding, increase trust, and most importantly put the team in a positive mood before the meeting. In this case, manipulation is a good thing. If I manipulated them to act against their own greater good or that of the team, in the long run, I think I would be judged less positively.  

 


Categories: Process Management

App Security Improvements: Looking back at 2016

Android Developers Blog - Thu, 01/19/2017 - 23:46
Posted by Rahul Mishra, Android Security Program Manager
In April 2016, the Android Security team described how the Google Play App Security Improvement (ASI) program has helped developers fix security issues in 100,000 applications. Since then, we have detected and notified developers of 11 new security issues and provided developers with resources and guidance to update their apps. Because of this, over 90,000 developers have updated over 275,000 apps!
ASI now notifies developers of 26 potential security issues. To make this process more transparent, we introduced a new page where developers can find information about all these security issues in one place. This page includes links to help center articles containing instructions and additional support contacts. Developers can use this page as a resource to learn about new issues and keep track of all past issues.

Make sure to check out our new Security for Android Developers page, which highlights the latest security posts, security best practices documents and security checklist. These resources are all aimed at improving your understanding of general security concepts and giving you examples that can help you address app-specific issues.

How you can help:
For feedback or questions, please reach out to us through the Google PlayDeveloper Help Center.
To report potential security issues in apps, email us at security+asi@android.com.
Categories: Programming

Android Developer Story: Wallapop improves user conversions with store listing experiments on Google Play

Android Developers Blog - Thu, 01/19/2017 - 17:42
Posted by Lily Sheringham, Developer Marketing, Google Play
Wallapop is a mobile app developer based in Barcelona, Spain. The app provides a platform to users for selling and buying things to others nearby in a virtual flea market by using geolocalization. Wallapop now has over 70% of their user base on Android.

Watch Agus Gomez, Co-Founder & CEO, and Marta Gui, Growth Hacking Manager, explain how using store listing experiments has increased their conversion rate by 17%, and has allowed them to optimize organic installs.


Learn more about store listing experiments. Get the Playbook for Developers app to stay up-to-date with more features and best practices that will help you grow a successful business on Google Play.


How useful did you find this blogpost? ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ                                                                               
Categories: Programming

Quote of the Month January 2017

From the Editor of Methods & Tools - Thu, 01/19/2017 - 11:37
Although many organizations are certified at SEI CMM Level 5, when a team hits a production snag, those certificates don‚Äôt carry much weight. People need IT heroes in the team to pull things through. Dependencies on folks who are technically competent, can think on their feet, and can manage time pressures increase that much more. […]

Tips from developers Peak and Soundcloud on how to grow your startup on Google Play

Android Developers Blog - Thu, 01/19/2017 - 10:02

Posted by Francesca Di Felice, Developer Marketing at Google Play
At Playtime 2016, Google Play's series of developer events, we met with top app and game developers from around the world to share learnings on how to build successful businesses on Google Play. Several startups, including game developer Peaklabs and audio platform SoundCloud, presented on stage their own best practices for growth, which you might find helpful.

Testing for growth, by Peak

Hear from Kevin Shanahan, Product Manager from Peak, a brain training app, on how to grow sustainably.



  • Test lots of ideas: You can't be sure of what will work and what won't, so you need to test lots of ideas. Peak ran four different tests to try to increase conversions to Pro (their subscriber offering):
  1. Made the ability to replay games a Pro feature
  2. Reduced price of Pro by 25% in top 2 markets
  3. Bundled add-on modules from partners into Pro
  4. Showed a preview of Pro-only content
          One of these tests resulted in a 50% increase in conversions.

  • Get the basics right: Start with a great product and have a data-informed culture. Don't only test app features, experimenting your store listing using store listing experiments is also important.
  • Build a robust A/B testing process: Having a well-defined A/B testing process and a system for tracking your experiments is key to testing quickly and effectively.

Improving user retention, by SoundCloud

Andy Carvell, former Product Manager at SoundCloud, an online audio distribution platform that enables its users to upload, record, promote, and share their originally-created sounds, explains how they focus on retention to improve growth.

 
  • Design your retention strategy: Apps with poor retention grow slowly. To increase your retention you should:
    • Convert new users to repeat visitors by providing a strong onboarding experience for new users and taking a high-touch approach during the first days and weeks.
    • Increase visit frequency within this group by providing frequent, timely, and relevant messaging about content or activity on the platform.
    • Target returning users who were not seen over the last period, who are 'at risk of churn' users, by giving them reasons to come back for another session before losing them.
    • Re-activate lapsed (long-term churned) users with campaigns to remind them about your app and offer an incentive to return.
  • Build 'growth machines': Create repeatable processes that testing has proven to positively impact retention, retaining users, and preventing churn.
  • Use activity notifications in a personalised and effective way: At SoundCloud there are plenty of things that happen when users are not in the app that might be relevant to them, for example new content releases or social interactions. They tested 5 new notification types, always keeping a control group to better keep track of the impact, and managed to increase retention in a 5%. Watch the video above for more of Andy's tips on making better use of notifications.

Other speakers, such as Silicon Valley VC Greylock, have also shared their tips for startup growth. Watch more sessions from this year's Playtime events to learn best practices from other apps and game partners, and the Google Play team. Get the Playbook for Developers app to stay up to date with news and tips to help you grow a successful business on Google Play.

How useful did you find this blogpost? ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ                                                                               
Categories: Programming

Project vs product teams

Actively Lazy - Wed, 01/18/2017 - 22:45

One of the hardest things for companies trying to be agile is how to structure teams. Back in the bad-old days, teams would form around a project. Then six months later, everyone would dissipate and go onto new teams. By the time a team has formed and become effective it is ripped apart again. You get no sense of ownership, no continuity.

children-laptop

Nowadays everyone knows that projects are bad, you need scrum teams instead. So a scrum team is formed with a product owner to prioritise the work. But what often happens is that what gets prioritised onto the backlog is a project in bite-size pieces. For example, I saw one team that ran out of work to do. The backlog was empty because, except for bugs, none of the outstanding projects had been signed off. There’s that word again. Project.

Behind the scenes a scrum team often becomes a slightly better way of delivering projects. You get the benefits of team consistency and continuity and the added benefit that the business can carry on thinking of the work in terms of projects. The downside of this approach is the scrum team can lack clear focus: there’s no overarching goal for the team. From sprint to sprint the focus might change as the relative importance of different projects changes. This makes it hard for the team to feel committed to a big idea, to some greater purpose. It ends up an endless procession through the backlog.

Why does this happen? I think it comes down to money. Somebody, somewhere is watching the money. Somebody wants to know “if I spend ¬£x here, how much am I going to make back and by when?” The idea of the project is very easy to fit into this model. The team costs ¬£x per day. The project is estimated to take n days. It’s expected to deliver ¬£y profit. From this we can calculate the expected return on our investment. The trouble is, most of these numbers are entirely made up. If not fundamentally unknowable.

Let’s start with the obvious one: how long is the project going to take? Really, we still actually ask this question? Have we learnt nothing from agile? It seems not: many, many people still think about the world in terms of delivery dates and certainties. When will we learn that the best way is always to deliver a little, inspect the results; then decide whether to keep on the same path or deliver something different. You can’t have an end date with this approach – it’s not even meaningful. Keep on delivering one thing until there’s something better you could be doing, then go do that. Rinse, repeat.

What about the other question: how much profit will this project make? Well, let’s assume for now that the entire project, as originally conceived, will actually be delivered (as if this ever actually happens in software). Can you tell how much money it’s made you? Really? Independent of every other change that the organisation has made at the same time? From software to operations to marketing?

Now sometimes you can come up with a good estimate of expected returns, but often it’s just a pipe dream. But, if you’re vigorously disagreeing with me: I assume you’re religiously tracking actual costs and feed that back into future project planning? I have seen very, very few companies actually do this. If you’re not actually measuring how much you made from a project, how do you know your original estimates were any good?

So we have two made up numbers, both almost certainly unachievable in practice – but we use this to dictate the team’s priority order. I once saw a project signed off and jump to the top of the priority order because it predicted something like a 10% uplift in revenue for the company. This was a very large number for a single project and clearly ridiculous to everyone involved, but it was signed off and duly implemented. Revenue projections later that year were re-estimated downwards and downwards due to difficult market conditions. And some blatant over-estimation. And yet, this non-science is what passes for return on investment planning in all-too-many organisations.

What’s the alternative? The best teams I’ve seen have been structured around products. Give the team complete ownership of one or more products. Any and all changes to those products go via the product team. A product owner guides product direction. As an area expert they are entrusted to decide what are the most important things to work on. They can discuss long term directions with the team and have a consistent, coherent vision for where the product will evolve towards. While, inevitably, some changes are large and sufficiently inter-dependent that they become a project (if one part is delivered then it all must be); the team understands the business benefit of the¬†solution and can evolve the implementation to meet the underlying business need, instead of trying to satisfy some arbitrary internal project deadline. This gives teams the complete freedom to inspect and adapt each iteration. With an understanding of the business priorities for their products they can make sensible trade-offs as each iteration surfaces more information.

What about the money? It’s hard, but let’s be honest about it: return on investment is not clear with the project model of software delivery, so accept that it isn’t clear. The hard thing is working out which products are making you money and which could make¬†more money if more was invested. The trouble is I’ve worked in teams where, honestly, the product was so profitable with so little scope for uplift that the most cost-effective thing to do would have been to fire the dev team and just keep milking the cash cow.

So how can we decide where to spend our money? I think the empirical model of agile could fit here perfectly well. Let’s assume for a minute that the amount of money you have for the delivery team as a whole is fixed – your only choice is where to put it. How much to spend on product A vs how much on product B. Can you estimate how much money each product is making for the business? How is it changing over time?

If one product is making more profit each month – if it’s a growing product – then invest more resources there, to accelerate the growth. If a product is slowing down, with smaller increases in profit each month, or even with profit decreasing – then stop spending so much money on it. This naturally means that your money goes where it seems to be delivering the biggest return. Put your money where it seems to be delivering results.

The hardest thing with this is that it takes time to get the feedback: changing resource allocation could take months to show up on the bottom line. But at least we’re being honest about the impact our decisions have.¬†Instead of trying to micro-manage delivery via projects, manage where resources are put and let the product owner manage the priority order.


Categories: Programming, Testing & QA

Welcoming Fabric to Google

Google Code Blog - Wed, 01/18/2017 - 22:26
Originally posted on the Firebase Blog

Posted by Francis Ma, Firebase Product Manager

Almost eight months ago, we launchedthe expansion of Firebase to help developers build high-quality apps, grow their user base, and earn more money across iOS, Android and the Web. We've already seen great adoption of the platform, which brings together the best of Google's core businesses from Cloud to mobile advertising.

Our ultimate goal with Firebase is to free developers from so much of the complexity associated with modern software development, giving them back more time and energy to focus on innovation.

As we work towards that goal, we've continued to improve Firebase, working closely with our user community. We recently introducedmajor enhancements to many core features, including Firebase Analytics, Test Lab and Cloud Messaging, as well as added support for game developers with a C++ SDK and Unity plug-in.


We're deeply committed to Firebase and are doubling down on our investment to solve developer challenges.
Fabric and Firebase Joining Forces

Today, we're excited to announce that we've signed an agreement to acquire Fabric to continue the great work that Twitter put into the platform. Fabric will join Google's Developer Product Group, working with the Firebase team. Our missions align closely: help developers build better apps and grow their business.
As a popular, trusted tool over many years, we expect that Crashlytics will become the main crash reporting offering for Firebase and will augment the work that we have already done in this area. While Fabric was built on the foundation of Crashlytics, the Fabric team leveraged its success to launch a broad set of important tools, including Answers and Fastlane. We'll share further details in the coming weeks after we close the deal, as we work closely together with the Fabric team to determine the most efficient ways to further combine our strengths. During the transition period, Digits, the SMS authentication services, will be maintained by Twitter.


The integration of Fabric is part of our larger, long-term effort of delivering a comprehensive suite of features for iOS, Android and mobile Web app development.

This is a great moment for the industry and a unique opportunity to bring the best of Firebase with the best of Fabric. We're committed to making mobile app development seamless, so that developers can focus more of their time on building creative experiences.
Categories: Programming

Get the guide to finding success in new markets on Google Play

Android Developers Blog - Wed, 01/18/2017 - 20:48
Posted by Lily Sheringham, Developer Marketing at Google Play


With just a few clicks, you can publish an app to Google Play and access a global audience of more than 1 billion 30 days active users. Finding success in global markets means considering how each market differs, planning for high quality localization, and tailoring your activity to the local audience. The new Going Global Playbook provides best practices and tips, with advice from developers who've successfully gone global.

This guide includes advice to help you plan your approach to going global, prepare your app for new markets, take your app to market, and also include data and insights for key countries and other useful resources.

This ebook joins others that we've recently published including The Building for Billions Playbook and The News Publisher Playbook. All of our ebooks are promoted in the Playbook for Developers app, which is where you can stay up to date with all the news and best practices you need to find success on Google Play.

How useful did you find this blogpost?

‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ                                                        
Categories: Programming

We’re Listening

Recently, the IEEE Computer Society’s popular SE Radio podcast included a sponsored advertising campaign that sparked a negative reaction among many of those involved with producing SE Radio, as well as among a number of listeners. In response to that reaction, the Computer Society has reviewed the advertisement and removed it from the podcast. In […]
Categories: Programming

We’re Listening

Recently, the IEEE Computer Society’s popular SE Radio podcast included a sponsored advertising campaign that sparked a negative reaction among many of those involved with producing SE Radio, as well as among a number of listeners. In response to that reaction, the Computer Society has reviewed the advertisement and removed it from the podcast. In […]
Categories: Programming

Emotional Intelligence, Useful?

Lots of screens!

So many things to learn and so little time to do it!

I was asked why emotional intelligence was important and whether emotional intelligence can be learned.

With a little probing on the second part of the question, it was suggested that there was a school of thought that emotional intelligence is an inherent human attribute; you have it or you don‚Äôt. The ‚Äúeither you are or aren‚Äôt‚ÄĚ argument is similar those with a fixed mindset make about most capabilities. ¬†The concept of a fixed in the book Mindset written by Carol Dweck. In the book, Dweck argues that mindsets are not fixed and we have identified several attributes that comprise emotional intelligence that can be improved in the essay, A Few Steps To Improving Your Emotional Intelligence. I do not accept that emotional intelligence is a fixed human ability. Simply put, your emotional intelligence quotient is not fixed and can be learned.

The second question suggests that emotional intelligence, while interesting, is not useful in the day-to-day operation of an Agile team (or by extension an organization). Fortunately, you do not have to look far to find applications of emotional intelligence in many day-to-day scenarios, ranging from team meetings to sales. As a leader or coach, it is easy to identify scenarios when emotional intelligence can be useful.  Three examples are:   

Diffusing Problem Situations

Problems happen and most involve people.  The large problems, for example, an irate client or someone that is acting counter to the team needs, are often easily spotted once they have happened. However, they are often the result of an accumulation of little issues; minor abrasions can add up. Examples of minor issues might include an occasional bit of underperformance, a bad mood, some failing to wish you happy birthday, talking over you on occasion.  The list can go on. Emotional intelligence helps not only to recognize the problem but more importantly when combined with listening to those within the boundary of the problem helps everyone to unburden.  Understanding and listening are input into empathy which is needed to come up with a fitting solution. Emotional intelligence is a tool to defuse problem situations.

Curiosity

Two of the five competencies of emotional intelligence are awareness and self-awareness of emotions in yourself and others and secondly, the ability to construct relationships. These two competencies are typically reflected as a curiosity. In my re-read of Carol Dweck’s Mindset, one of the attributes of the growth mindset is insatiable need to learn and experience challenges, a related form of curiosity. Emotional intelligence is linked to the growth mindset through curiosity (at the very least).  Curiosity and the desire to learn are important capabilities in stable cross-functional teams.  Agile teams with emotionally intelligent members will be able to stretch to meet their customer needs because leveraging their curiosity they can identify what the skill they need to know, then learn new skills and discover new solutions. Curiosity may have killed the cat, but emotional intelligence fosters curiosity and learning will make the Agile team.

Repeat Clients

I have many friends that are fellow consultants, both independent and part of a larger organization. ¬†All of them are intelligent, all of them have great pedigrees and many of them are successful as independents. When consultants gather, the one conversation all consultants have is getting and keeping clients. ¬†I have observed that the consultants that have repeating/recurring clients have significantly more emotional intelligence than those that are great at getting clients but less so at generating repeat business. ¬†Emotional intelligence is a tool to build meaningful relationships that make it easier get repeat business. ¬†The essay, Emotional Intelligence: A Few Basics referenced Daniel Kahneman‚Äôs statement ¬†‚ÄĚthat people would rather do business with a person they like and trust rather than someone they don‚Äôt.‚ÄĚ ¬†Emotional intelligence makes it easier to build solid relationships that translate into repeat clients. Without emotional intelligence, it is difficult to generate the empathy needed to invest time into growing relationships based on anything other than sales volume.

Emotional intelligence is useful for identifying and defusing problems, generating relationships and to facilitate repeat sales. You are not born with all of the emotional intelligence that you can or will ever need.  Emotional intelligence is a reflection of a set of capabilities that can be improved.  We should invest the time, effort and money needed to get increase our emotional intelligence capability.    


Categories: Process Management

Emotional Intelligence, Useful?

Lots of screens!

So many things to learn and so little time to do it!

I was asked why emotional intelligence was important and whether emotional intelligence can be learned.

With a little probing on the second part of the question, it was suggested that there was a school of thought that emotional intelligence is an inherent human attribute; you have it or you don‚Äôt. The ‚Äúeither you are or aren‚Äôt‚ÄĚ argument is similar those with a fixed mindset make about most capabilities. ¬†The concept of a fixed in the book Mindset written by Carol Dweck. In the book, Dweck argues that mindsets are not fixed and we have identified several attributes that comprise emotional intelligence that can be improved in the essay, A Few Steps To Improving Your Emotional Intelligence. I do not accept that emotional intelligence is a fixed human ability. Simply put, your emotional intelligence quotient is not fixed and can be learned.

The second question suggests that emotional intelligence, while interesting, is not useful in the day-to-day operation of an Agile team (or by extension an organization). Fortunately, you do not have to look far to find applications of emotional intelligence in many day-to-day scenarios, ranging from team meetings to sales. As a leader or coach, it is easy to identify scenarios when emotional intelligence can be useful.  Three examples are:   

Diffusing Problem Situations

Problems happen and most involve people.  The large problems, for example, an irate client or someone that is acting counter to the team needs, are often easily spotted once they have happened. However, they are often the result of an accumulation of little issues; minor abrasions can add up. Examples of minor issues might include an occasional bit of underperformance, a bad mood, some failing to wish you happy birthday, talking over you on occasion.  The list can go on. Emotional intelligence helps not only to recognize the problem but more importantly when combined with listening to those within the boundary of the problem helps everyone to unburden.  Understanding and listening are input into empathy which is needed to come up with a fitting solution. Emotional intelligence is a tool to defuse problem situations.

Curiosity

Two of the five competencies of emotional intelligence are awareness and self-awareness of emotions in yourself and others and secondly, the ability to construct relationships. These two competencies are typically reflected as a curiosity. In my re-read of Carol Dweck’s Mindset, one of the attributes of the growth mindset is insatiable need to learn and experience challenges, a related form of curiosity. Emotional intelligence is linked to the growth mindset through curiosity (at the very least).  Curiosity and the desire to learn are important capabilities in stable cross-functional teams.  Agile teams with emotionally intelligent members will be able to stretch to meet their customer needs because leveraging their curiosity they can identify what the skill they need to know, then learn new skills and discover new solutions. Curiosity may have killed the cat, but emotional intelligence fosters curiosity and learning will make the Agile team.

Repeat Clients

I have many friends that are fellow consultants, both independent and part of a larger organization. ¬†All of them are intelligent, all of them have great pedigrees and many of them are successful as independents. When consultants gather, the one conversation all consultants have is getting and keeping clients. ¬†I have observed that the consultants that have repeating/recurring clients have significantly more emotional intelligence than those that are great at getting clients but less so at generating repeat business. ¬†Emotional intelligence is a tool to build meaningful relationships that make it easier get repeat business. ¬†The essay, Emotional Intelligence: A Few Basics referenced Daniel Kahneman‚Äôs statement ¬†‚ÄĚthat people would rather do business with a person they like and trust rather than someone they don‚Äôt.‚ÄĚ ¬†Emotional intelligence makes it easier to build solid relationships that translate into repeat clients. Without emotional intelligence, it is difficult to generate the empathy needed to invest time into growing relationships based on anything other than sales volume.

Emotional intelligence is useful for identifying and defusing problems, generating relationships and to facilitate repeat sales. You are not born with all of the emotional intelligence that you can or will ever need.  Emotional intelligence is a reflection of a set of capabilities that can be improved.  We should invest the time, effort and money needed to get increase our emotional intelligence capability.    


Categories: Process Management

Silence speaks louder than words when finding malware

Google Code Blog - Tue, 01/17/2017 - 23:06
Originally posted on Android Developer Blog

Posted by Megan Ruthven, Software Engineer
In Android Security, we're constantly working to better understand how to make Android devices operate more smoothly and securely. One security solution included on all devices with Google Play is Verify apps. Verify apps checks if there are Potentially Harmful Apps (PHAs) on your device. If a PHA is found, Verify apps warns the user and enables them to uninstall the app.

But, sometimes devices stop checking up with Verify apps. This may happen for a non-security related reason, like buying a new phone, or, it could mean something more concerning is going on. When a device stops checking up with Verify apps, it is considered Dead or Insecure (DOI). An app with a high enough percentage of DOI devices downloading it, is considered a DOI app. We use the DOI metric, along with the other security systems to help determine if an app is a PHA to protect Android users. Additionally, when we discover vulnerabilities, we patch Android devices with our security update system. This blog post explores the Android Security team's research to identify the security-related reasons that devices stop working and prevent it from happening in the future.
Flagging DOI Apps
To understand this problem more deeply, the Android Security team correlates app install attempts and DOI devices to find apps that harm the device in order to protect our users.
With these factors in mind, we then focus on 'retention'. A device is considered retained if it continues to perform periodic Verify apps security check ups after an app download. If it doesn't, it's considered potentially dead or insecure (DOI). An app's retention rate is the percentage of all retained devices that downloaded the app in one day. Because retention is a strong indicator of device health, we work to maximize the ecosystem's retention rate. Therefore, we use an app DOI scorer, which assumes that all apps should have a similar device retention rate. If an app's retention rate is a couple of standard deviations lower than average, the DOI scorer flags it. A common way to calculate the number of standard deviations from the average is called a Z-score. The equation for the Z-score is below.
  • N = Number of devices that downloaded the app.
  • x = Number of retained devices that downloaded the app.
  • p = Probability of a device downloading any app will be retained.

In this context, we call the Z-score of an app's retention rate a DOI score. The DOI score indicates an app has a statistically significant lower retention rate if the Z-score is much less than -3.7. This means that if the null hypothesis is true, there is much less than a 0.01% chance the magnitude of the Z-score being as high. In this case, the null hypothesis means the app accidentally correlated with lower retention rate independent of what the app does.
This allows for percolation of extreme apps (with low retention rate and high number of downloads) to the top of the DOI list. From there, we combine the DOI score with other information to determine whether to classify the app as a PHA. We then use Verify apps to remove existing installs of the app and prevent future installs of the app.
Difference between a regular and DOI app download on the same device.
Results in the wild
Among others, the DOI score flagged many apps in three well known malware families‚ÄĒ Hummingbad, Ghost Push, and Gooligan. Although they behave differently, the DOI scorer flagged over 25,000 apps in these three families of malware because they can degrade the Android experience to such an extent that a non-negligible amount of users factory reset or abandon their devices. This approach provides us with another perspective to discover PHAs and block them before they gain popularity. Without the DOI scorer, many of these apps would have escaped the extra scrutiny of a manual review.
The DOI scorer and all of Android's anti-malware work is one of multiple layers protecting users and developers on Android. For an overview of Android's security and transparency efforts, check out our page.
Categories: Programming

Silence speaks louder than words when finding malware

Android Developers Blog - Tue, 01/17/2017 - 22:59
Posted by Megan Ruthven, Software Engineer
In Android Security, we're constantly working to better understand how to make Android devices operate more smoothly and securely. One security solution included on all devices with Google Play is Verify apps. Verify apps checks if there are Potentially Harmful Apps (PHAs) on your device. If a PHA is found, Verify apps warns the user and enables them to uninstall the app.
But, sometimes devices stop checking up with Verify apps. This may happen for a non-security related reason, like buying a new phone, or, it could mean something more concerning is going on. When a device stops checking up with Verify apps, it is considered Dead or Insecure (DOI). An app with a high enough percentage of DOI devices downloading it, is considered a DOI app. We use the DOI metric, along with the other security systems to help determine if an app is a PHA to protect Android users. Additionally, when we discover vulnerabilities, we patch Android devices with our security update system.

This blog post explores the Android Security team's research to identify the security-related reasons that devices stop working and prevent it from happening in the future.
Flagging DOI Apps

To understand this problem more deeply, the Android Security team correlates app install attempts and DOI devices to find apps that harm the device in order to protect our users.
With these factors in mind, we then focus on 'retention'. A device is considered retained if it continues to perform periodic Verify apps security check ups after an app download. If it doesn't, it's considered potentially dead or insecure (DOI). An app's retention rate is the percentage of all retained devices that downloaded the app in one day. Because retention is a strong indicator of device health, we work to maximize the ecosystem's retention rate.

Therefore, we use an app DOI scorer, which assumes that all apps should have a similar device retention rate. If an app's retention rate is a couple of standard deviations lower than average, the DOI scorer flags it. A common way to calculate the number of standard deviations from the average is called a Z-score. The equation for the Z-score is below.

  • N = Number of devices that downloaded the app.
  • x = Number of retained devices that downloaded the app.
  • p = Probability of a device downloading any app will be retained.

In this context, we call the Z-score of an app's retention rate a DOI score. The DOI score indicates an app has a statistically significant lower retention rate if the Z-score is much less than -3.7. This means that if the null hypothesis is true, there is much less than a 0.01% chance the magnitude of the Z-score being as high. In this case, the null hypothesis means the app accidentally correlated with lower retention rate independent of what the app does.
This allows for percolation of extreme apps (with low retention rate and high number of downloads) to the top of the DOI list. From there, we combine the DOI score with other information to determine whether to classify the app as a PHA. We then use Verify apps to remove existing installs of the app and prevent future installs of the app.

Difference between a regular and DOI app download on the same device.


Results in the wild
Among others, the DOI score flagged many apps in three well known malware families‚ÄĒ Hummingbad, Ghost Push, and Gooligan. Although they behave differently, the DOI scorer flagged over 25,000 apps in these three families of malware because they can degrade the Android experience to such an extent that a non-negligible amount of users factory reset or abandon their devices. This approach provides us with another perspective to discover PHAs and block them before they gain popularity. Without the DOI scorer, many of these apps would have escaped the extra scrutiny of a manual review.
The DOI scorer and all of Android's anti-malware work is one of multiple layers protecting users and developers on Android. For an overview of Android's security and transparency efforts, check out our page.


Categories: Programming

Sponsored Post: Contentful, Stream, Loupe, New York Times, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?
  • Contentful is looking for a JavaScript BackEnd Engineer to join our team in their mission of getting new users - professional developers - started on our platform within the shortest time possible. We are a fun and diverse family of over 100 people from 35 nations with offices in Berlin and San Francisco, backed by top VCs (Benchmark, Trinity, Balderton, Point Nine), growing at an amazing pace. We are working on a content management developer platform that enables web and mobile developers to manage, integrate, and deliver digital content to any kind of device or service that can connect to an API. See job description.

  • The New York Times is looking for a Software Engineer for its Delivery/Site Reliability Engineering team. You will also be a part of a team responsible for building the tools that ensure that the various systems at The New York Times continue to operate in a reliable and efficient manner. Some of the tech we use: Go, Ruby, Bash, AWS, GCP, Terraform, Packer, Docker, Kubernetes, Vault, Consul, Jenkins, Drone. Please send resumes to: technicaljobs@nytimes.com
Fun and Informative Events
  • Your event here!
Cool Products and Services
  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

Wouldn't it be nice if everyone knew a little queuing theory?

After many days of rain one lane of this two lane road collapsed into the canyon. It's been out for a month and it will be many more months before it will be fixed. Thanks to Google maps way too many drivers take this once sleepy local road. 

How do you think drivers go through this chokepoint? 

 

 

One hundred experience points to you if you answered one at a time.

One at a time! Through a half-duplex pipe following a first in first out discipline takes forever!

Yes, there is a stop sign. And people default to this mode because it appeals to our innate sense of fairness. What could be fairer than alternating one at a time?

The problem is it's stupid.

While waiting, stewing, growing angrier, I often think if people just knew a little queueing theory we could all be on our way a lot faster.

We can't make the pipe full duplex, so that's out. Let's assume there's no priority involved, vehicles are roughly the same size and take roughly the same time to transit the network. Then what do you do?

Why can't people figure out its faster to drive through in batches? If we went in groups of say, three, the throughput would be much higher. And when one side's queue depth grows larger because people are driving to or from work that side's batch size should increase. 

Since this condition will last a long time we have a possibility to learn because the same people take this road all the time. So what happens if you try to change the culture by showing people what a batch is by driving right behind someone as they take their turn?

You got it. Honking. There's a simple heuristic, a deeply held ethic against line cutting, so people honk, flip you off, and generally make heir displeasure known.

It's your classic battle of reason versus norms. The smart thing is the thing we can't do by our very natures. So we all just keep doing the dumb thing.

 

Categories: Architecture

My Most Popular Posts of 2016

Mike Cohn's Blog - Tue, 01/17/2017 - 16:00

Because I wrote a lot last year--25 blog posts and 50 weekly email tips--I wanted to start something new this year. So here’s a list of the most popular blog posts here during 2016. I hope it helps you catch up on any you missed during the year.

Using our own little algorithm that is a combination of page views, comments and time spent on the pages, here are my top 10 blog posts from 2016, counting down from number 10:

10) Applying Agile Beyond Software Development

Agile can be applied well beyond software development. It’s been used for construction, planning weddings, marketing and more. These are my thoughts on how agile could have saved a hotel chain from an expensive mistake.

9) What Are Story Points?

Story points are perhaps the most misunderstood topic in agile. Story points are not based on just one factor--such as complexity, as is often mistakenly claimed. Instead, story points are based on a combination of factors.

8) Advice on How to Split Reporting User Stories

Splitting stories has long been one of the biggest challenges facing agile teams. Here are some examples of splitting some reporting stories to demonstrate ways of splitting stories.

7) Does a Scrum Team Need a Retrospective Every Sprint?

Conventional wisdom says that a team should do a retrospective every sprint. But if your sprints are one week, can you do them every few sprints? That would still be more often than a team doing four-week sprints.

6) How to Prevent Estimate Inflation

A rose by any other name may smell as sweet, but a five-point story better not go by any other names. Or numbers. Here’s how to maintain consistency across estimates.

5) Summarizing the Results of a Sprint

Although you may wish it weren’t the case, some Scrum Masters need to document how a sprint went. Here’s advice on how to do that in a lightweight, agile manner.

4) The Dangers of a Definition of Ready

After seeing the value of a Definition of Done, some teams introduce a Definition of Ready. For many teams, this is a big mistake and a first step towards a waterfall process.

3) Don’t Estimate the Sprint Backlog Using Task Points

Some teams like story points so much, they invent task points and use those for sprint planning. Bad idea. Here’s why.

2) Sprint Planning for Agile Teams That Have Lots of Interruptions

Most of the Scrum literature describes a situation in which a team is allowed to work without interruption. But that’s not realistic. Here’s how an interrupt-driven team can plan its sprints.

1) A Simple Way to Run a Sprint Retrospective

There are many ways you can run a sprint retrospective. Here’s the simplest way and still my favorite.

What Do You Think?

Please let me know what you think. Is this list missing any of your favorites?

An Inferno on the Head of a Pin

Coding Horror - Jeff Atwood - Tue, 01/17/2017 - 12:37

Today's processors contain billions of heat-generating transistors in an ever shrinking space. The power budget might go from:

  • 1000 watts on a specialized server
  • 100 watts on desktops
  • 30 watts on laptops
  • 5 watts on tablets
  • 1 or 2 watts on a phone
  • 100 milliwatts on an embedded system

That's three four orders of magnitude. Modern CPU design is the delicate art of placing an inferno on the head of a pin.

Look at the original 1993 Pentium compared to the 20th anniversary Pentium:

Intel Pentium 66 1993
Pentium
66 Mhz
16kb L1
3.2 million transistors
Intel Pentium G3258 20th Anniversary Edition 2014
Pentium G3258
3.2 Ghz × 2 cores
128kb L1, 512kb L2, 3MB L3
1.4 billion transistors

I remember cooling the early CPUs with simple heatsinks; no fan. Those days are long gone.

A roomy desktop computer affords cooling opportunities (and thus a watt budget) that a laptop or tablet could only dream of. How often will you be at peak load? For most computers, the answer is "rarely". The smaller the space, the higher the required performance, the more … challenging your situation gets.

Sometimes, I build servers.

Inspired by Google and their use of cheap, commodity x86 hardware to scale on top of the open source Linux OS, I also built our own servers. When I get stressed out, when I feel the world weighing heavy on my shoulders and I don't know where to turn … I build servers. It's therapeutic.

Servers are one of those situations where you may be at full CPU load more often than not. I prefer to build 1U servers which is the smallest rack mountable unit, at 1.75" total height.

You get plenty of cores on a die these days, so I build single CPU servers. One reason is price; the other reason is that clock speed declines proportionally to the number of cores on a die (this is for the Broadwell Xeon V4 series):

coresGHz E5-163043.7$406 E5-165063.6$617 E5-168083.4$1723 E5-2680122.4$1745 E5-2690142.6$2090 E5-2697182.3$2702

Yes, there are server CPUs with even more cores, but if you have to ask how much they cost, you definitely can't afford them … and they're clocked even slower. What we do is serviced better by a smaller number of super fast cores than a larger number of slow cores, anyway.

With that in mind, consider these two Intel Xeon server CPUs:

As you can see from the official Intel product pages for each processor, they both have a TDP heat budget of 140 watts. I'm scanning the specs, thinking maybe this is an OK tradeoff.

Unfortunately, here's what I actually measured with my trusty Kill-a-Watt for each server build as I performed my standard stability testing, with completely identical parts except for the CPU:

  • E5-1630: 40w idle, 170w mprime
  • E5-1650: 55w idle, 250w mprime

I am here to tell you that Intel's TDP figure of 140 watts for the 6 core version of this CPU is a terrible, scurrilous lie!

This caused a bit of a problem for me as our standard 1U server build now overheats, alarms, and throttles with the 6 core CPU — whereas the 4 core CPU was just fine. Hey Intel! From my home in California, I stab at thee!

But, you know..

Better Heatsink

The 1.75" maximum height of the 1U server form factor doesn't leave a lot of room for creative cooling of a CPU. But you can switch from an Aluminum cooler to a Copper one.

Copper is significantly more expensive, plus heavier and harder to work with, so it's generally easier to throw an ever-larger mass of aluminum at the cooling problem when you can. But when space is a constraint, as it is with a 1U server, copper dissipates more heat in the same form factor.

The famous "Ninja" CPU cooler came in identical copper and aluminum versions so we can compare apples to apples:

  • Aluminum Ninja — 24C rise over ambient
  • Copper Ninja — 17C rise over ambient

You can scale the load and the resulting watts of heat by spinning up MPrime threads for the exact number of cores you want to "activate", so that's how I tested:

  • Aluminum heatsink — stable at 170w (mprime threads=4), but heat warnings with 190w (mprime threads=5)
  • Copper heatsink — stable at 190w (mprime threads=5) but heat warnings with 230w (mprime threads=6)

Each run has to be overnight to be considered successful. This helped, noticeably. But we need more.

Better Thermal Interface

When it comes to server builds, I stick with the pre-applied grey thermal interface pad that comes on the heatsinks. But out of boredom and a desire to experiment, I …

  • Removed the copper heatsink.
  • Used isopropyl alcohol to clean both CPU and heatsink.
  • Applied fancy "Ceramique" thermal compound I have on hand, using an X shape pattern.

I wasn't expecting any change at all, but to my surprise with the new TIM applied it took 5x longer to reach throttle temps with mprime threads=6. Before, it would thermally throttle within a minute of launching the test, and after it took ~10 minutes to reach that same throttle temp. The difference was noticeable.

That's a surprisingly good outcome, and it tells us the default grey goop that comes pre-installed on heatsinks is ... not great. Per this 2011 test, the difference between worst and best thermal compounds is 4.3°C.

But as Dan once bravely noted while testing Vegemite as a thermal interface material:

If your PC's so marginal that a CPU running three or four degrees Celsius warmer will crash it [or, for modern CPUs, cause the processor to auto-throttle itself and substantially reduce system performance], the solution is not to try to edge away from the precipice with better thermal compound. It's to make a big change to the cooling system, or just lower the darn clock speed.

An improved thermal interface just gets you there faster (or slower); it doesn't address the underlying problem. So we're not done here.

Ducted Airflow

Most, but not all, of the SuperMicro cases I've used have included a basic fan duct / shroud that lays across the central fans and the system. Given that the case fans are pretty much directly in front of the CPU anyway, I've included the shroud in the builds out of a sense of completeness more than any conviction that it was doing anything for the cooling performance.

This particular server case, though, did not include a fan duct. I didn't think much about it at the time, but considering the heat stress this 6-core CPU and its 250 watt heat generation was putting on our 1U build, I decided I should build a quick duct out of card stock and test it out.

(I know, I know, it's a super janky duct! But I was prototyping!)

Sure enough, this duct, combined with the previous heatsink and TIM changes, enabled the server to remain stable overnight with a full MPrime run of 12 threads.

I think we've certainly demonstrated the surprising (to me, at least) value of fan shrouds. But before we get too excited, let's consider one last thing.

Define "CPU Load"

Sometimes you get so involved with solving the problem at hand that you forget to consider whether you are, in fact, solving the right problem.

In these tests, we defined 100% CPU load using MPrime. Some people claim MPrime is more of a power virus than a real load test, because it exerts so much heat pressure on the CPUs. I initially dismissed these claims since I've used MPrime (and its Windows cousin, Prime95) for almost 20 years to test CPU stability, and it's never let me down.

But I did more research and I found that MPrime, since 2011, uses AVX2 instructions extensively on newer Intel CPUs:

The newer versions of Prime load in a way that they are only safe to run at near stock settings. The server processors actually downclock when AVX2 is detected to retain their TDP rating. On the desktop we're free to play and the thing most people don't know is how much current these routines can generate. It can be lethal for a CPU to see that level of current for prolonged periods.

That's why most stress test programs alternate between different data pattern types. Depending on how effective the rotation is, and how well that pattern causes issues for the system timing margin, it will, or will not, catch potential for instability. So it's wise not to hang one's hat on a single test type.

This explains why I saw such a large discrepancy between other CPU load programs like BurnP6 and MPrime.

MPrime does an amazing job of generating the type of CPU load that causes maximum heat pressure. But unless your servers regularly chew through zillions of especially power-hungry AVX2 instructions this may be completely unrepresentative of any real world load your server would actually see.

Your Own Personal Inferno

Was this overkill? Probably. Even with the aluminum heatsink, no change to thermal interface material, and zero ducting, we'd probably see no throttling under normal use in our server rack. But I wanted to be sure. Completely sure.

Is this extreme? Putting 140 TDP of CPU heat in a 1U server? Not really. Nick at Stack Overflow told me they just put two 22 core, 145W TDP Xeon 2699v4 CPUs and four 300W TDP GPUs in a single Dell C4130 1U server. I'd sure hate to be in the room when those fans spin up. I'm also a little afraid to find out what happens if you run MPrime plus full GPU load on that box.

Servers are an admittedly rare example of big CPU performance heat and size tradeoffs, one of the few left. It is fun to play at the extremes, but the SoC inside your phone makes the same tradeoffs on a smaller scale. Tiny infernos in our pockets, each and every one.

[advertisement] At Stack Overflow, we put developers first. We already help you find answers to your tough coding questions; now let us help you find your next job.
Categories: Programming

Programmatically promote your package quality with Release Views in VSTS

Xebia Blog - Mon, 01/16/2017 - 22:04
A short while ago, Package Management  in VSTS became publicly available. Agreed, the features in the initial version are not overwhelming, but the potential is very high. If you are not familiar yet, with Package Management in VSTS , I highly recommend you take a look at the documentation on the Visual Studio site, to