Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

AMP Compression Update

Google Code Blog - Tue, 06/06/2017 - 18:00
Posted by Zachary Nado, Software Engineer

Recently we announcedthe addition of Brotli compression to the Google AMP Cache. All AMP documents served from the Google AMP Cache can now be served with Brotli, which will save a considerable amount of bandwidth for our users and further our goal of improving the mobile experience.

Brotliis a newer, more efficient compression algorithm created by Jyrki Alakuijala and Zolt√°n Szabadka with the Google Research Europe Compression Team. Launched in 2015, it has already been used to enable considerable savings in other areas of Google. While it is a generic compression algorithm, it has particularly impressive performance when applied to web documents; we have seen an average decrease in document size of around 10% when using Brotli instead of gzip, which has amounted to hundreds of gigabytes of bandwidth saved per day across the Google AMP Cache.

With smaller document sizes, pages load faster while also saving bandwidth which can amount to noticeable savings for users on limited data plans. The Google AMP Cache is just the beginning though, as engineering teams are working on Brotli support in many other products which can enable bandwidth savings throughout Google.

Categories: Programming

SE-Radio Episode 293: Yakov Fain on Angular

Yakov Fain talks with SE Radio‚Äôs Matthew Farwell about the Angular web development framework. The show covers the philosophy behind Angular; who would want to use the platform; how an Angular application is composed, including how to handle form submission and validation; why Typescript was chosen for Angular; how Angular uses reactive programming (RxJS, in […]
Categories: Programming

#Noestimates Has Come to This

Herding Cats - Glen Alleman - Tue, 06/06/2017 - 02:47

Screen Shot 2017-06-02 at 9.36.50 AM

This is a chart from an agile conference where #Noestimates was a topic. For the most part, the briefing follows Tony Magennis' approach from his book. So the first part is following standard estimating principles. What's fascinating is the evolution from the original post going on 4 years ago ... 

No Estimates

... which states clearly and concisely that decisions can be made (in the presence of uncertainty - which is ALWAYS present on software development projects) - WITHOUT estimating the impact of those decisions.

To this Manifest style statement that is restating in different words the principles used in all estimating processes in any domain as well as the estimating principles stated in the references found in Chapter 5 of the Bibliography for Agile at Scale.

So let's look at this manifesto:

  1. Probabilistic over Deterministic - there is NO deterministic process in software development. There's not even a deterministic process in the production of Toyota Camry's. It's always probabilistic. By probability, it means there is uncertainty. Actually, these processes are both probabilistic processes and statistical processes which produce Uncertainty in two forms: Aleatory and Epistemic. Aleatory uncertainty produces an irreducible risk to the project. Only margin can be with aleatory uncertainty. Epistemic uncertainty produces reducible risk, which can be dealt with directly with experiments, tests, redundancy, and other intervention processing. So not credible management system assume deterministic process. If it does it's doomed to fail on day one. 
  2. Delivery over Development Time - to Deliver you need to Develop. The granularity of the delivery is the flexible parameter. It's a strawman to use waterfall anymore. Even in the US DOD Waterfall is not allowed. So drop the Waterfall Strawman. Our performance management system requires an assessment of Physical Percent Complete every month in the Integrated Performance Management Report (IPMR). To have this be credible, we status the work every Thursday with Physical Percent Complete assessment. This manifesto statement is a false equivalency. †
  3. MVP Scope over Full Scope - the full scope of any project is NOT possible in any real sense up front on any Software Intensive System of Systems (our domain). It's rarely possible to have that much insight into any development project. From large construction to software development. This is another false equivalency. Rolling waves, planning packages, emerging processes are all standard project management processes.
  4. Data over Intuition - this is an example of not attending the High School statistics class. Past performance, reference classes, parametric models, Monte Carlo simulation, method of moments, and other estimating processes are standard practice. In our domain, best engineering estimates are the LEAST desirable form of estimation, and in some acquisition domains not allowed. Another false equivalency.

There are lots of issues with estimating, even in sophisticated estimating environments. I work several programs with state-of-the-art estimating processes and tools and we still have issues. Does that mean we don't need estimating? 

Hardley, and here's why. 

No Principle for NEThese are better questions to ask, but these questions do not lead to the answer of #NoEstimates either. So let's answer these questions with the material found in existing guidance...

Screen Shot 2017-06-02 at 9.38.44 AM

  1.  In what context would estimates bring value, and what are we willing to do about it? This is one of those inverted logic question used by #Noestimates advocates - like so senator when did you stop beating your wife?  The real question is when are estimates NOT providing value? The simple answer is when we are working a de minimus project. That is when there is no downside to being wrong about the cost and schedule. If you accept the There is NO Principle ... quote above about making decisions in the presence of uncertainty, then an estimate of some kind is needed to decide.
  2. How much time do we want to invest in this? This is a simple question and answer. What's the Value at Risk? Low risk - low value, quick estimate. High Risk and/or High Value, you'd better have a higher confidence of that cost and schedule, along with increased probability that you'll be delivering the right value at the needed time for the needed cost top those paying for that value. By the way, YOU CANNOT KNOW THE VALUE OF ANYTHING  WITHOUT KNOWING THE COST TO ACHIEVE  THAT VALUE. This is basic Microeconomics of decision making.
  3. What can you do to maximize value and reduce risk in planning and delivery? You can start with Tim Lister's quote. Risk Management is How Adults Manage Projects. Risk management means managing in the presence of Aleatory and Epistemic uncertainty. This management REQUIRES you make estimates. No Estimates? No Risk Management. No Risk Management? No Adult Management. Be an adult, manage risk with estimates of aleatory and epistemic uncertainty, estimates of the impact of those risks if they were to turn into issues, the estimates of the cost to mitigate those risks, estimates of the residual uncertainty, and estimates of the impacts of that residual certainty.

† False equivalence is a logical fallacy in which two opposing arguments appear to be logically equivalent when in fact they are not. This fallacy is categorized as a fallacy of inconsistency.

 

Related articles What's the Smell of Dysfunction? Humpty Dumpty and #NoEstimates Herding Cats: Book of the Month Herding Cats: Risk Management for Agile Software Development Projects Information Technology Estimating Quality Herding Cats: Five Immutable Principles of Project Success and Estimating Why Guessing is not Estimating and Estimating is not Guessing Capabilities Based Planning Who's Budget is it Anyway?
Categories: Project Management

Google Play’s policy on incentivized ratings, reviews, and installs

Android Developers Blog - Mon, 06/05/2017 - 18:00

Posted by Kazushi Nagayama, Ninja Spamologist and Bryan Woodward, Policy Specialist

Ensuring Google Play remains trusted and secure is one of our top priorities. We've recently announced improvements in fighting spam installs as well as fake ratings & reviews. In order to underscore these announcements and provide more clarity, we have now updated our Developer Program Policies on incentivized ratings, reviews, and installs:


Developers must not attempt to manipulate the placement of any apps in the Store. This includes, but is not limited to, inflating product ratings, reviews, or install counts by illegitimate means, such as fraudulent or incentivized installs, reviews and ratings.


Defining an incentivized action

We deem an action to be incentivized if a user is offered money, goods, or the equivalent in exchange for the action ‚Äď be it a rating, review or install. Incentivized ratings and reviews have always been against our policies and we will continue to take action against them in order to protect the integrity of our store. Installs done with the intent to manipulate the placement of an app in Google Play will be detected and filtered.

Incentivized installs as user acquisition

We've observed instances where incentivized installs are utilized solely to manipulate the placement of apps in Google Play; these instances are a policy violation. However, we also recognize that incentivized installs can be a legitimate user acquisition channel for some developers. In order to recognize these two distinct use cases, we are taking the following approach:

  • Whilst we won't automatically remove apps from the store just because they utilize incentivized installs as one of their user acquisition channels, we will monitor for, and take action against behaviour that compromises the integrity of the store.
  • To address those whose intent we perceive is to manipulate the placements of their apps, we will monitor and filter incentivized installs in our systems, including removal from the top charts. If warranted, identified apps also may be removed from the Store.

Through this approach, we hope to further ensure that the top charts and other discovery mechanisms on Google Play reflect the reality of the popularity of an app.

As a general rule, we advise against utilizing incentivized actions. Incentivized users are a very different user base than users found through other acquisition channels. In an internal analysis, the Google Research team found that incentivized users generally have lower retention rates and make fewer in-app purchases than users found through paid or organic acquisition channels.

For more information on the Google Play policies, please visit the developer policy center. For tips and best practices to find success on Google Play, visit the Android Developers website.

How useful did you find this blogpost?

‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ .post-content img { margin: 0; padding: 0; }
Categories: Programming

Google Play’s policy on incentivized ratings, reviews, and installs

Android Developers Blog - Mon, 06/05/2017 - 18:00

Posted by Kazushi Nagayama, Ninja Spamologist and Bryan Woodward, Policy Specialist

Ensuring Google Play remains trusted and secure is one of our top priorities. We've recently announced improvements in fighting spam installs as well as fake ratings & reviews. In order to underscore these announcements and provide more clarity, we have now updated our Developer Program Policies on incentivized ratings, reviews, and installs:


Developers must not attempt to manipulate the placement of any apps in the Store. This includes, but is not limited to, inflating product ratings, reviews, or install counts by illegitimate means, such as fraudulent or incentivized installs, reviews and ratings.


Defining an incentivized action

We deem an action to be incentivized if a user is offered money, goods, or the equivalent in exchange for the action ‚Äď be it a rating, review or install. Incentivized ratings and reviews have always been against our policies and we will continue to take action against them in order to protect the integrity of our store. Installs done with the intent to manipulate the placement of an app in Google Play will be detected and filtered.

Incentivized installs as user acquisition

We've observed instances where incentivized installs are utilized solely to manipulate the placement of apps in Google Play; these instances are a policy violation. However, we also recognize that incentivized installs can be a legitimate user acquisition channel for some developers. In order to recognize these two distinct use cases, we are taking the following approach:

  • Whilst we won't automatically remove apps from the store just because they utilize incentivized installs as one of their user acquisition channels, we will monitor for, and take action against behaviour that compromises the integrity of the store.
  • To address those whose intent we perceive is to manipulate the placements of their apps, we will monitor and filter incentivized installs in our systems, including removal from the top charts. If warranted, identified apps also may be removed from the Store.

Through this approach, we hope to further ensure that the top charts and other discovery mechanisms on Google Play reflect the reality of the popularity of an app.

As a general rule, we advise against utilizing incentivized actions. Incentivized users are a very different user base than users found through other acquisition channels. In an internal analysis, the Google Research team found that incentivized users generally have lower retention rates and make fewer in-app purchases than users found through paid or organic acquisition channels.

For more information on the Google Play policies, please visit the developer policy center. For tips and best practices to find success on Google Play, visit the Android Developers website.

How useful did you find this blogpost?

‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ ‚ėÖ .post-content img { margin: 0; padding: 0; }
Categories: Programming

Quote of the Day

Herding Cats - Glen Alleman - Mon, 06/05/2017 - 15:46

lux in tenebris lucet et tenebrae eam non comprehenderunt

Categories: Project Management

What‚Äôs the ‚ÄúRight‚ÄĚ Term for an Agile Transition or Transformation?

As I finish the agile project management book (Create Your Successful Agile Project), I am working on what stays in this book and what I use in the agile management book (next up after the PO book).

That leads me to this question: What do you call a transition to agile approaches? Is there a correct term?

Here’s what I’m thinking right now and I’m totally open to changing my thinking:

Transition to agile:¬†to me, it means a transition from planning for predictability to planning for predictable deliveries. In other words, it’s the delivery of value again and again that allows us to replan again and again. This requires a change in thinking. Maybe I should call this a transition to agile thinking? (Am I the only one who thinks this way?)

Agile transformation: We transform the organization to working and thinking in an agile way. We transform the culture. For me, this is¬†such a big idea, I don’t see it happening. I don’t even think I¬†know what it totally means. On the other hand, agile approaches are a cultural shift from how people thought and how they organized to new¬†ways of thinking and organizing.

Agile adoption: To me, this is about adopting an agile mindset and values. On the other hand, I’ve seen too much bounce-back when people (especially managers) talk about agile adoption. They tend to think of agile approaches as yet another project management life cycle.

Do you use another term? Do you find its baggage acceptable? Please let me know what you think the “right” term is for an agile transition or transformation. Thank you!

Categories: Project Management

What‚Äôs the ‚ÄúRight‚ÄĚ Term for an Agile Transition or Transformation?

As I finish the agile project management book (Create Your Successful Agile Project), I am working on what stays in this book and what I use in the agile management book (next up after the PO book).

That leads me to this question: What do you call a transition to agile approaches? Is there a correct term?

Here’s what I’m thinking right now and I’m totally open to changing my thinking:

Transition to agile:¬†to me, it means a transition from planning for predictability to planning for predictable deliveries. In other words, it’s the delivery of value again and again that allows us to replan again and again. This requires a change in thinking. Maybe I should call this a transition to agile thinking? (Am I the only one who thinks this way?)

Agile transformation: We transform the organization to working and thinking in an agile way. We transform the culture. For me, this is¬†such a big idea, I don’t see it happening. I don’t even think I¬†know what it totally means. On the other hand, agile approaches are a cultural shift from how people thought and how they organized to new¬†ways of thinking and organizing.

Agile adoption: To me, this is about adopting an agile mindset and values. On the other hand, I’ve seen too much bounce-back when people (especially managers) talk about agile adoption. They tend to think of agile approaches as yet another project management life cycle.

Do you use another term? Do you find its baggage acceptable? Please let me know what you think the “right” term is for an agile transition or transformation. Thank you!

Categories: Project Management

SPaMCAST 445 ‚Äď Capers Jones, Selecting Software Metrics

SPaMCAST Logo

http://www.spamcast.net

Listen Now
Subscribe on iTunes
Check out the podcast on Google Play Music

SPaMCAST 445 features the return of a favorite, Capers Jones. ¬†It is always fun to talk with someone with their own page in Wikepedia. ¬†Capers and I talked about his new book, A Guide to Selecting Software Measures and Metrics. Capers is passionate about software quality and measurement. Capers said, ‚ÄúHigh-quality software is not expensive. High-quality software is faster and cheaper to build and maintain than low-quality software, from initial development all the way through total cost of ownership.‚ÄĚ from The Economics of Software Quality. ¬†As usual, Capers was engaging, educational and controversial. ¬†Spending time with Capers is always a learning experience!

Capers biography is long and storied.  Let it be said that Capers is a serial author, public speaker, pundit, guru and deep thinker.  Check out his Wikipedia page or Linkedin.

Capers can be contacted at capers.jones3@gmail.com.

Capers first appeared on SPaMCAST 3 and  last appeared on SPaMCAST 53

Re-Read Saturday News

This week we tackle Chapter 7 of Holacracy: The New Management System for a Rapidly Changing World by Brian J. Robertson published by Henry Holt and Company in 2015.  Chapter 7 shows how to generate alignment between roles, circles, and the overall organization.  Lots of inspect and adapt talk this week.

Catch up on the first four entries in the re-read

Week 1:  Logistics and Introduction

Week 2: Evolving Organization

Week 3: Distribution Authority

Week 4: Organizational Structure

Week 5: Governance

Week 6: Operations

Week 7: Facilitating Governance

Week 8: Strategy and Dynamic Control

Visit the Software Process and Measurement Cast blog to participate in this and previous re-reads.

 

A Call To Action

If you got a new idea this week while listening to the podcast, please give the SPaMCAST a short, honest review in iTunes, Stitcher or wherever you are listening.  If you leave a review please send a copy to spamcastinfo@gmail.com.  Reviews help guide people to the cast!

Next SPaMCAST

SPaMCAST 446 will feature our essay on questions.  Questions are a coach and facilitator’s secret power!   Do you have a favorite go to question you like to ask?  Care to share?

We will also have columns from Gene Hughson and Jon Quigley (and maybe more)!

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: ‚ÄúThis book will prove that software projects should not be a tedious process, for you or your team.‚ÄĚ Support SPaMCAST by buying the book here. Available in English and Chinese.

 


Categories: Process Management

SPaMCAST 445 - Capers Jones, Selecting Software Metrics

Software Process and Measurement Cast - Sun, 06/04/2017 - 22:00

SPaMCAST 445 features the return of a favorite, Capers Jones. ¬†It is always fun to talk with someone with their own page in Wikepedia. ¬†Capers and I talked about his new book, A Guide to Selecting Software Measures and Metrics. Capers is passionate about software quality and measurement. Capers said, ‚ÄúHigh-quality software is not expensive. High-quality software is faster and cheaper to build and maintain than low-quality software, from initial development all the way through total cost of ownership.‚ÄĚ ¬†Jones, Caper, Bonsignour, Olivier, and Jitendra Subramanyam, Jitendra, The Economics of Software Quality. ¬†As usual, Capers was engaging, educational and controversial. ¬†Spending time with Capers is always a learning experience!

Capers biography is long and storied.  Let it be said that Capers is a serial author, public speaker, pundit, guru and deep thinker.  Check out his Wikipedia page or Linkedin.

Capers can be contacted at capers.jones3@gmail.com.

Capers first appeared on SPaMCAST 3 and  last appeared on SPaMCAST 53

Re-Read Saturday News

This week we tackle Chapter 7 of Holacracy: The New Management System for a Rapidly Changing World by Brian J. Robertson published by Henry Holt and Company in 2015.  Chapter 7 shows how to generate alignment between roles, circles, and the overall organization.  Lots of inspect and adapt talk this week.

Catch up on the first four entries in the re-read

Week 1:  Logistics and Introduction

Week 2: Evolving Organization

Week 3: Distribution Authority

Week 4: Organizational Structure

Week 5: Governance

Week 6: Operations

Week 7: Facilitating Governance

Week 8: Strategy and Dynamic Control

Visit the Software Process and Measurement Cast blog to participate in this and previous re-reads.

 

A Call To Action

If you got a new idea this week while listening to the podcast, please give the SPaMCAST a short, honest review in iTunes, Stitcher or wherever you are listening.  If you leave a review please send a copy to spamcastinfo@gmail.com.  Reviews help guide people to the cast!

 

Next SPaMCAST

 SPaMCAST 446 will feature our essay on questions.  Questions are a coach and facilitator’s secret power!   Do you have a favorite go to question you like to ask?  Care to share?

We will also have columns from Gene Hughson and Jon Quigley (and maybe more)!

 

Shameless Ad for my book!

Mastering Software Project Management: Best Practices, Tools and Techniques co-authored by Murali Chematuri and myself and published by J. Ross Publishing. We have received unsolicited reviews like the following: ‚ÄúThis book will prove that software projects should not be a tedious process, for you or your team.‚ÄĚ Support SPaMCAST by buying the book here. Available in English and Chinese.

Categories: Process Management

Five Immutable Principles of Project Success

Herding Cats - Glen Alleman - Sun, 06/04/2017 - 20:52

No matter the size of the project. No matter the project domain. No matter the engineering processes used to produce the outcomes from the project. There are Five Immutable Principles of project success. These principles are Immutable because they are universally applicable. Without some assessment of what Done looks like before the project starts, you won't be able to recognize Done when it arrives. Without these measures, Done will usually mean 

Without some description of what Done looks like before the project starts, you won't be able to recognize Done when it arrives. Without these measures, Done will usually mean we ran out of time and money. 

Plans are strategies for success. For the project to be successful, we need a Plan. Strategies are hypotheses. For a hypothesis to correctly represent the outcome it needs tests. The test of the hypothesis for projects are the Measures of Effectiveness, Measures of Performance, Key Performance Parameters, and techTechnicalformance Measures used to assess progress to plan for the project.

We need resources - time, money, facilities, talent, support, guidance. How much? Well, that's a good question. Estimating how much is the starting point. Measuring the effectiveness of these resources is needed to take corrective actions to assure we have enough to finish the project on time. Yes, deadlines are critical to project success. Deadline for getting the tomatoes in. Too late, there won't be time for fruit to set. Show up late to the launch site for our Mars mission and we have to wait till the next time Mars comes around - at our own cost. Show up over budget for the ERP, the ROI we told the Board of Director is now wrong.

Risk Management is How Adults Manage Projects - Tim Lister. No risk management plan with mitigations, the risks are still there, and we're not managing other people's money like adults. Since all risks come from uncertainty, we have to estimate those uncertainties, their impacts, their residual risks after mitigation. Estimating is part of risk management. Risk management is managing as an adult. No Estimating? No Adult Management.

Physical Percent Complete is the only measure of progress to plan. This is true for planting tomatoes, writing software with Scrum, flying to Mars.

So when you hear - well that process you're talking about is applicable in your domain but not mine. And that process is based on one of more the Five Immutable Principles, ask if that person suggesting that unsubstainanted claim has any understanding that Principles are needed before any Processes and Practices can be effective. No? Low Probability of Project Success ahead.

Five immutable principles from Glen Alleman Related articles Deadlines Always Matter Humpty Dumpty and #NoEstimates
Categories: Project Management

Kaggle: House Prices: Advanced Regression Techniques ‚Äď Trying to fill in missing values

Mark Needham - Sun, 06/04/2017 - 10:22

I’ve been playing around with the data in Kaggle’s House Prices: Advanced Regression Techniques and while replicating Poonam Ligade’s exploratory analysis I wanted to see if I could create a model to fill in some of the missing values.

Poonam wrote the following code to identify which columns in the dataset had the most missing values:

import pandas as pd
train = pd.read_csv('train.csv')
null_columns=train.columns[train.isnull().any()]

>>> print(train[null_columns].isnull().sum())
LotFrontage      259
Alley           1369
MasVnrType         8
MasVnrArea         8
BsmtQual          37
BsmtCond          37
BsmtExposure      38
BsmtFinType1      37
BsmtFinType2      38
Electrical         1
FireplaceQu      690
GarageType        81
GarageYrBlt       81
GarageFinish      81
GarageQual        81
GarageCond        81
PoolQC          1453
Fence           1179
MiscFeature     1406
dtype: int64

The one that I’m most interested in is LotFrontage, which describes ‘Linear feet of street connected to property’. There are a few other columns related to lots so I thought I might be able to use them to fill in the missing LotFrontage values.

We can write the following code to find a selection of the rows missing a LotFrontage value:

cols = [col for col in train.columns if col.startswith("Lot")]
missing_frontage = train[cols][train["LotFrontage"].isnull()]

>>> print(missing_frontage.head())
    LotFrontage  LotArea LotShape LotConfig
7           NaN    10382      IR1    Corner
12          NaN    12968      IR2    Inside
14          NaN    10920      IR1    Corner
16          NaN    11241      IR1   CulDSac
24          NaN     8246      IR1    Inside

I want to use scikit-learn‘s linear regression model which only works with numeric values so we need to convert our categorical variables into numeric equivalents. We can use pandas get_dummies function for this.

Let’s try it out on the LotShape column:

sub_train = train[train.LotFrontage.notnull()]
dummies = pd.get_dummies(sub_train[cols].LotShape)

>>> print(dummies.head())
   IR1  IR2  IR3  Reg
0    0    0    0    1
1    0    0    0    1
2    1    0    0    0
3    1    0    0    0
4    1    0    0    0

Cool, that looks good. We can do the same with LotConfig and then we need to add these new columns onto the original DataFrame. We can use pandas concat function to do this.

import numpy as np

data = pd.concat([
        sub_train[cols],
        pd.get_dummies(sub_train[cols].LotShape),
        pd.get_dummies(sub_train[cols].LotConfig)
    ], axis=1).select_dtypes(include=[np.number])

>>> print(data.head())
   LotFrontage  LotArea  IR1  IR2  IR3  Reg  Corner  CulDSac  FR2  FR3  Inside
0         65.0     8450    0    0    0    1       0        0    0    0       1
1         80.0     9600    0    0    0    1       0        0    1    0       0
2         68.0    11250    1    0    0    0       0        0    0    0       1
3         60.0     9550    1    0    0    0       1        0    0    0       0
4         84.0    14260    1    0    0    0       0        0    1    0       0

We can now split data into train and test sets and create a model.

from sklearn import linear_model
from sklearn.model_selection import train_test_split

X = data.drop(["LotFrontage"], axis=1)
y = data.LotFrontage

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42, test_size=.33)

lr = linear_model.LinearRegression()

model = lr.fit(X_train, y_train)

Now it’s time to give it a try on the test set:

>>> print("R^2 is: \n", model.score(X_test, y_test))
R^2 is: 
 -0.84137438493

Hmm that didn’t work too well – an R^2 score of less than 0 suggests that we’d be better off just predicting the average LotFrontage regardless of any of the other features. We can confirm that with the following code:

from sklearn.metrics import r2_score

>>> print(r2_score(y_test, np.repeat(y_test.mean(), len(y_test))))
0.0

whereas if we had all of the values correct we’d get a score of 1:

>>> print(r2_score(y_test, y_test))
1.0

In summary, not a very successful experiment. Poonam derives a value for LotFrontage based on the square root of LotArea so perhaps that’s the best we can do here.

The post Kaggle: House Prices: Advanced Regression Techniques – Trying to fill in missing values appeared first on Mark Needham.

Categories: Programming

Holacracy: Re-read Week 8, Chapter 7: Strategy and Dynamic Control

Book Cover

Holacracy

This week we tackle Chapter 7 of Holacracy: The New Management System for a Rapidly Changing World by Brian J. Robertson published by Henry Holt and Company in 2015.  Chapter 7 shows how to generate alignment between roles, circles, and the overall organization.  Lots of inspect and adapt talk this week.

One of the major attributes of Holacracy is the requirement of individuals to manage and prioritize their own tasks and responsibilities. Without a mechanism to generate alignment between the team and the organization anarchy can occur.  This is often a criticism of team-based Agile frameworks and methods.  In Holacracy, tactical meetings help alignment, but another key element is strategy.

One important component of a strategy that provides alignment is a focus on the future. A strategy provides a link between the organization‚Äôs goals and the team’s behavior. The individual and the circles they inhabit become the operational element to interpret the strategy. ¬†¬†The ability of individuals to manage their own tasks and responsibilities is how strategies are translated and pursued

Robertson uses a quote from Nassim Nicholas Taleb to reinforce that a strategy is not a prediction. Taleb said,

We cannot truly plan because we do not understand the future ‚Äď but this is not necessarily bad news. We could plan while bearing in mind such limitations. It just takes guts.¬†

Holacracy provides the structure to use strategy as a planning tool.

The flexibility provided through roles and tactical meetings provides an ability to sense and respond to what is happening in the present moment. ¬†Strategies act as an anchoring bias that guides rather than locks tactical activities in stone. ¬†Agile uses the phrase ‘inspect and adapt’ to define this type of behavior. ¬†Adaptive behavior is often quashed in standard organizations by pushing decisions about changing and adapting into committee decision-making that is more interested in maintaining an equilibrium. ¬†A great book title that says a lot about this type of behavior is You Can’t Teach a Kid to Ride a Bike at a Seminar (a great book on sales); Sandler (author) suggests that you learn by experience. ¬†¬†Robertson culminates on alignment by stating that we need to relentlessly face reality and adapt continuously.

Holacracy’s more dynamic control process of governance and tactical meeting processes focuses on quickly reaching a workable decision and then letting reality inform the next step rather than agonizing about what might happen to create the theoretical best decision that will not quite get it right anyway.

In Holacracy, a strategy acts a form of heuristic (rule of thumb) that allows roles and circles to dynamic create strategic alignment. Strategies provide a tool to inform decisions when we encounter them without trying to predict the future exactly. ¬†For example, I have a strategy to keep from getting lost when I am running in cities that I unfamiliar. ¬†I take only right-hand turns and I count the number of turns. ¬†The strategy is easy to remember and aids in making decisions. ¬†When I come to the fork in the road, I go right and increment the register by one (I coded once upon a time). ¬†Roberston suggests that strategies follow the form of ‚Äúemphasize X, even over Y‚ÄĚ. In my running case, emphasize not being lost, over exploration and distance. ¬†When I become more familiar with an area (the context changes), the strategy can be changed.

Unlike other major components of Holacracy, Robertson does not provide a single best way to generate strategies suggesting that the process is driven by the lead link (represents the larger organization in the circle) or like other major components, the use of structured meetings to shape the strategies.  The meeting engages the whole team and is Robertson’s preferred approach.

The strategy meeting:

  1. ¬†¬†¬†¬†¬†¬†Check In Round ‚Äď generate presence.
  2. ¬†¬†¬†¬†¬†¬†Orientation ‚Äď Grounds the circle in why we are here and the accountabilities of the circle to the larger organization. (This reminds me of the front end of the Program Increment Meeting ¬†in SAFe)
  3. ¬†¬†¬†¬†¬†¬†Retrospective ‚Äď This step leverages a form of affinity diagraming to capture how the team arrived in its current state. As in all forms of affinity diagraming the team works silently until after grouping and then the facilitator gathers comments and tensions group by group.
  4. ¬†¬†¬†¬†¬†¬†Strategy Generation ‚Äď Based on the tensions generated in step 3, each person individually reflects and captures draft strategies (what should we emphasize on a day-to-day basis to address the tension. Once all the written the draft strategy are posted. They are discussed collectively. ¬†When discussed the lead link uses the ideas and discussion to frame a few (one or two) strategies. ¬†These are processed by the team using the integrative decision-making process identified in the previous chapter.
  5. ¬†¬†¬†¬†¬†¬†Unpack the Strategy ‚Äď When the strategy or strategies are established each member of the circle need to reflect on what the roles they perform could do to enact the strategy. ¬†These are captured and enacted or routed to the appropriate governance or tactical meeting.
  6. ¬†¬†¬†¬†¬†¬†Closing Round ‚Äď Final reflections.

The approach of integrating this form of strategy into the flow of governance and tactical meetings provides a platform for an organization to truly inspect and adapt.  Robertson states that this approach makes an organization evolutionary rather than just adaptive.  Left to my own devices, I would state that an organization can dynamically change to need the needs of real life.  That might be exactly what Robertson means by the term evolutionary.

Remember to buy a copy of Holacracy (use the link to help support and defray the costs of the Software Process and Measurement Cast blog and podcast).

Previous Entries in the re-read:

Week 1:  Logistics and Introduction

Week 2: Evolving Organization

Week 3: Distribution Authority

Week 4: Organization Structure

Week 5: Governance

Week 6: Operations

Week 7: Facilitating Governance

 

 


Categories: Process Management

2017 Android Security Rewards

Android Developers Blog - Fri, 06/02/2017 - 18:50
Posted by Mayank Jain and Scott Roberts of the Android Security team

Two years ago, we launched the Android Security Rewards program. In its second year, we've seen great progress. We received over 450 qualifying vulnerability reports from researchers and the average pay per researcher jumped by 52.3%. On top of that, the total Android Security Rewards payout doubled to $1.1 million dollars. Since it launched, we've rewarded researchers over $1.5 million dollars.

Here are some of the highlights from the Android Security Rewards program's second year:

  • There were no payouts for the top reward for a complete remote exploit chain leading to TrustZone or Verified Boot compromise, our highest award amount possible.
  • We paid 115 individuals with an average of $2,150 per reward and $10,209 per researcher.
  • We paid our top research team, C0RE Team, over $300,000 for 118 vulnerability reports.
  • We paid 31 researchers $10,000 or more.

Thank you to all the amazing researchers who submitted complete vulnerability reports to us last year.

Improvements to Android Security Rewards program

We're constantly working to improve the Android Security Rewards program and today we're making a few changes to all vulnerability reports filed after June 1, 2017.

Because every Android release includes more security protections and no researcher has claimed the top reward for an exploit chains in 2 years, we're excited to increase our top-line payouts for these exploits.

  • Rewards for a remote exploit chain or exploit leading to TrustZone or Verified Boot compromise increase from $50,000 to $200,000.
  • Rewards for a remote kernel exploit increase from $30,000 to $150,000.

In addition to rewarding for vulnerabilities, we continue to work with the broad and diverse Android ecosystem to protect users from issues reported through our program. We collaborate with manufacturers to ensure that these issues are fixed on their devices through monthly security updates. Over 100 device models have a majority of their deployed devices running a security update from the last 90 days. This table shows the models with a majority of deployed devices running a security update from the last two months:

Manufacturer Device BlackBerry PRIV Fujitsu F-01J General Mobile GM5 Plus d, GM5 Plus, General Mobile 4G Dual, General Mobile 4G Gionee A1 Google Pixel XL, Pixel, Nexus 6P, Nexus 6, Nexus 5X, Nexus 9 LGE LG G6, V20, Stylo 2 V, GPAD 7.0 LTE Motorola Moto Z, Moto Z Droid Oppo CPH1613, CPH1605 Samsung Galaxy S8+, Galaxy S8, Galaxy S7, Galaxy S7 Edge, Galaxy S7 Active, Galaxy S6 Active, Galaxy S5 Dual SIM, Galaxy C9 Pro, Galaxy C7, Galaxy J7, Galaxy On7 Pro, Galaxy J2, Galaxy A8, Galaxy Tab S2 9.7 Sharp Android One S1, 507SH Sony Xperia XA1, Xperia X Vivo Vivo 1609, Vivo 1601, Vivo Y55 Source: Google May 29th, 2017.

Thank you to everyone who helped make Android safer and stronger in the past year. Together, we made a huge investment in security research that helps Android users everywhere. If you want to get involved to make next year even better, check out our detailed Program Rules. For tips on how to submit complete reports, see Bug Hunter University.

table { border-collapse: collapse; border-spacing: 0; width: 100%; border: 1px solid #ddd; } th { border: 1px solid #ddd; text-align: left; padding: 8px; background-color: #d3d3d3; text-align: center; vertical-align: middle; } td { border: 1px solid #ddd; text-align: left; padding: 8px; vertical-align: middle; min-width: 100px; } .note { font-size: 10px; padding-top: 0; margin; 0; text-align: center; }
Categories: Programming

Gone Fishin'

Well, not exactly Fishin', but I'll be on a month long vacation starting today. I won't be posting (much) new content, so we'll all have a break. Disappointing, I know. Please use this time for quiet contemplation and other inappropriate activities. Au revoir!

Categories: Architecture

Hacker, Hack Thyself

Coding Horror - Jeff Atwood - Fri, 06/02/2017 - 09:11

We've read so many sad stories about communities that were fatally compromised or destroyed due to security exploits. We took that lesson to heart when we founded the Discourse project; we endeavor to build open source software that is secure and safe for communities by default, even if there are thousands, or millions, of them out there.

However, we also value portability, the ability to get your data into and out of Discourse at will. This is why Discourse, unlike other forum software, defaults to a Creative Commons license. As a basic user on any Discourse you can easily export and download all your posts right from your user page.

Discourse Download All Posts

As a site owner, you can easily back up and restore your entire site database from the admin panel, right in your web browser. Automated weekly backups are set up for you out of the box, too. I'm not the world's foremost expert on backups for nothing, man!

Discourse database backup download

Over the years, we've learned that balancing security and data portability can be tricky. You bet your sweet ASCII a full database download is what hackers start working toward the minute they gain any kind of foothold in your system. It's the ultimate prize.

To mitigate this threat, we've slowly tightened restrictions around Discourse backups in various ways:

  • Administrators have a minimum password length of 15 characters.

  • Both backup creation and backup download administrator actions are formally logged.

  • Backup download tokens are single use and emailed to the address of the administrator, to confirm that user has full control over the email address.

The name of the security game is defense in depth, so all these hardening steps help … but we still need to assume that Internet Bad Guys will somehow get a copy of your database. And then what? Well, what's in the database?

  • Identity cookies

    Cookies are, of course, how the browser can tell who you are. Cookies are usually stored as hashes, rather than the actual cookie value, so having the hash doesn't let you impersonate the target user. Furthermore, most modern web frameworks rapidly cycle cookies, so they are only valid for a brief 10 to 15 minute window anyway.

  • Email addresses

    Although users have reason to be concerned about their emails being exposed, very few people treat their email address as anything particularly precious these days.

  • All posts and topic content

    Let's assume for the sake of argument that this is a fully public site and nobody was posting anything particularly sensitive there. So we're not worried, at least for now, about trade secrets or other privileged information being revealed, since they were all public posts anyway. If we were, that's a whole other blog post I can write at a later date.

  • Password hashes

    What's left is the password hashes. And that's … a serious problem indeed.

Now that the attacker has your database, they can crack your password hashes with large scale offline attacks, using the full resources of any cloud they can afford. And once they've cracked a particular password hash, they can log in as that user … forever. Or at least until that user changes their password.

‚ö†ÔłŹ That's why, if you know (or even suspect!) your database was exposed, the very first thing you should do is reset everyone's password.

Discourse database password hashes

But what if you don't know? Should you preemptively reset everyone's password every 30 days, like the world's worst bigco IT departments? That's downright user hostile, and leads to serious pathologies of its own. The reality is that you probably won't know when your database has been exposed, at least not until it's too late to do anything about it. So it's crucial to slow the attackers down, to give yourself time to deal with it and respond.

Thus, the only real protection you can offer your users is just how resistant to attack your stored password hashes are. There are two factors that go into password hash strength:

  1. The hashing algorithm. As slow as possible, and ideally designed to be especially slow on GPUs for reasons that will become painfully obvious about 5 paragraphs from now.

  2. The work factor or number of iterations. Set this as high as possible, without opening yourself up to a possible denial of service attack.

I've seen guidance that said you should set the overall work factor high enough that hashing a password takes at least 8ms on the target platform. It turns out Sam Saffron, one of my Discourse co-founders, made a good call back in 2013 when he selected the NIST recommendation of PBKDF2-HMAC-SHA256 and 64k iterations. We measured, and that indeed takes roughly 8ms using our existing Ruby login code on our current (fairly high end, Skylake 4.0 Ghz) servers.

But that was 4 years ago. Exactly how secure are our password hashes in the database today? Or 4 years from now, or 10 years from now? We're building open source software for the long haul, and we need to be sure we are making reasonable decisions that protect everyone. So in the spirit of designing for evil, it's time to put on our Darth Helmet and play the bad guy – let's crack our own hashes!

We're gonna use the biggest, baddest single GPU out there at the moment, the GTX 1080 Ti. As a point of reference, for PBKDF2-HMAC-SHA256 the 1080 achieves 1180 kH/s, whereas the 1080 Ti achieves 1640 kH/s. In a single video card generation the attack hash rate has increased nearly 40 percent. Ponder that.

First, a tiny hello world test to see if things are working. I downloaded hashcat. I logged into our demo at try.discourse.org and created a new account with the password 0234567890; I checked the database, and this generated the following values in the hash and salt database columns for that new user:

hash
93LlpbKZKficWfV9jjQNOSp39MT0pDPtYx7/gBLl5jw=
salt
ZWVhZWQ4YjZmODU4Mzc0M2E2ZDRlNjBkNjY3YzE2ODA=

Hashcat requires the following input file format: one line per hash, with the hash type, number of iterations, salt and hash (base64 encoded) separated by colons:

type   iter  salt                                         hash  
sha256:64000:ZWVhZWQ4YjZmODU4Mzc0M2E2ZDRlNjBkNjY3YzE2ODA=:93LlpbKZKficWfV9jjQNOSp39MT0pDPtYx7/gBLl5jw=  

Let's hashcat it up and see if it works:

./h64 -a 3 -m 10900 .\one-hash.txt 0234567?d?d?d

Note that this is an intentionally tiny amount of work, it's only guessing three digits. And sure enough, we cracked it fast! See the password there on the end? We got it.

sha256:64000:ZWVhZWQ4YjZmODU4Mzc0M2E2ZDRlNjBkNjY3YzE2ODA=:93LlpbKZKficWfV9jjQNOSp39MT0pDPtYx7/gBLl5jw=:0234567890

Now that we know it works, let's get down to business. But we'll start easy. How long does it take to brute force attack the easiest possible Discourse password, 8 numbers – that's "only" 108 combinations, a little over one hundred million.

Hash.Type........: PBKDF2-HMAC-SHA256  
Time.Estimated...: Fri Jun 02 00:15:37 2017 (1 hour, 0 mins)  
Guess.Mask.......: ?d?d?d?d?d?d?d?d [8]  

Even with a top of the line GPU that's … OK, I guess. Remember this is just one hash we're testing against, so you'd need one hour per row (user) in the table. And I have more bad news for you: Discourse hasn't allowed 8 character passwords for quite some time now. How long does it take if we try longer numeric passwords?

?d?d?d?d?d?d?d?d?d [9]
Fri Jun 02 10:34:42 2017 (11 hours, 18 mins)

?d?d?d?d?d?d?d?d?d?d [10]
Tue Jun 06 17:25:19 2017 (4 days, 18 hours)

?d?d?d?d?d?d?d?d?d?d?d [11]
Mon Jul 17 23:26:06 2017 (46 days, 0 hours)

?d?d?d?d?d?d?d?d?d?d?d?d [12]
Tue Jul 31 23:58:30 2018 (1 year, 60 days)  

But all digit passwords are easy mode, for babies! How about some real passwords that use at least lowercase letters, or lowercase + uppercase + digits?

Guess.Mask.......: ?l?l?l?l?l?l?l?l [8]  
Time.Estimated...: Mon Sep 04 10:06:00 2017 (94 days, 10 hours)

Guess.Mask.......: ?1?1?1?1?1?1?1?1 [8] (-1 = ?l?u?d)  
Time.Estimated...: Sun Aug 02 09:29:48 2020 (3 years, 61 days)  

A brute force try-every-single-letter-and-number attack is not looking so hot for us at this point, even with a high end GPU. But what if we divided the number by eightby putting eight video cards in a single machine? That's well within the reach of a small business budget or a wealthy individual. Unfortunately, dividing 38 months by 8 isn't such a dramatic reduction in the time to attack. Instead, let's talk about nation state attacks where they have the budget to throw thousands of these GPUs at the problem (1.1 days), maybe even tens of thousands (2.7 hours), then … yes. Even allowing for 10 character password minimums, you are in serious trouble at that point.

If we want Discourse to be nation state attack resistant, clearly we'll need to do better. Hashcat has a handy benchmark mode, and here's a sorted list of the strongest (slowest) hashes that Hashcat knows about benchmarked on a rig with 8 Nvidia GTX 1080 GPUs. Of the things I recognize on that list, bcrypt, scrypt and PBKDF2-HMAC-SHA512 stand out.

My quick hashcat results gave me some confidence that we weren't doing anything terribly wrong with the Discourse password hashes stored in the database. But I wanted to be completely sure, so I hired someone with a background in security and penetration testing to, under a signed NDA, try cracking the password hashes of two live and very popular Discourse sites we currently host.

I was provided two sets of password hashes from two different Discourse communities, containing 5,909 and 6,088 hashes respectively. Both used the PBKDF2-HMAC-SHA256 algorithm with a work factor of 64k. Using hashcat, my Nvidia GTX 1080 Ti GPU generated these hashes at a rate of ~27,000/sec.

Common to all discourse communities are various password requirements:

  • All users must have a minimum password length of 10 characters.
  • All administrators must have a minimum password length of 15 characters.
  • Users cannot use any password matching a blacklist of the 10,000 most commonly used passwords.
  • Users can choose to create a username and password or use various third party authentication mechanisms (Google, Facebook, Twitter, etc). If this option is selected, a secure random 32 character password is autogenerated. It is not possible to know whether any given password is human entered, or autogenerated.

Using common password lists and masks, I cracked 39 of the 11,997 hashes in about three weeks, 25 from the ‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą community and 14 from the ‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą‚Ėą community.

This is a security researcher who commonly runs these kinds of audits, so all of the attacks used wordlists, along with known effective patterns and masks derived from the researcher's previous password cracking experience, instead of raw brute force. That recovered the following passwords (and one duplicate):

007007bond
123password
1qaz2wsx3e
A3eilm2s2y
Alexander12
alexander18
belladonna2
Charlie123
Chocolate1
christopher8
Elizabeth1
Enterprise01
Freedom123
greengrass123
hellothere01
I123456789
Iamawesome
khristopher
l1ghthouse
l3tm3innow
Neversaynever
password1235
pittsburgh1
Playstation2
Playstation3
Qwerty1234
Qwertyuiop1
qwertyuiop1234567890
Spartan117
springfield0
Starcraft2
strawberry1
Summertime
Testing123
testing1234
thecakeisalie02
Thirteen13
Welcome123

If we multiply this effort by 8, and double the amount of time allowed, it's conceivable that a very motivated attacker, or one with a sophisticated set of wordlists and masks, could eventually recover 39 × 16 = 624 passwords, or about five percent of the total users. That's reasonable, but higher than I would like. We absolutely plan to add a hash type table in future versions of Discourse, so we can switch to an even more secure (read: much slower) password hashing scheme in the next year or two.

bcrypt $2*$, Blowfish (Unix)  
  20273 H/s

scrypt  
  886.5 kH/s

PBKDF2-HMAC-SHA512  
  542.6 kH/s 

PBKDF2-HMAC-SHA256  
 1646.7 kH/s 

After this exercise, I now have a much deeper understanding of our worst case security scenario, a database compromise combined with a professional offline password hashing attack. I can also more confidently recommend and stand behind our engineering work in making Discourse secure for everyone. So if, like me, you're not entirely sure you are doing things securely, it's time to put those assumptions to the test. Don't wait around for hackers to attack you — hacker, hack thyself!

[advertisement] At Stack Overflow, we put developers first. We already help you find answers to your tough coding questions; now let us help you find your next job.
Categories: Programming

Getting Stuff Done: Scrumban and Beginning With A Backlog

Longer races usually use "bins" to group runners, like classes of service.

Longer races usually use “bins” to group runners, like classes of service.

Without some sort of structure, projects, daily to-dos, ideas and just flat stuff can quickly overwhelm anyone. Many, if not most, of us have spent time taking time management classes of all types in an attempt to find the secret sauce for managing the chaos that is the 21st century. My wife is a sort of adherent of GTD¬ģ. Once upon a time I took classes for the Franklin Covey Planner, and I dutifully carried it everywhere. In recent years I have used Scrum and Kanban to manage projects. Many of the lessons in Agile and lean project management coupled with time management concepts are a useful synthesis: a personal Scrumban (Kanban-y Scrumban) approach. The approach begins with deciding on a set of classes of service and then developing an initial backlog.

In a typical implementation of Kanban, classes of service allow teams to break backlog items into different groups either based on risk or the cost of delay. In our personal Scrumban, we combine the concept of cost of delay with different focus areas. Unlike a typical work environment where a person and team would focus on one thing at a time, a personal process for handling the overwhelming list of projects and tasks that occur in everyday life needs to acknowledge life is more than a project or a sprint.

I have developed an approach that recognizes five classes of work ranging from association work items to work items associated with my work (I am process consultant and manager at the David Consulting Group). Each class of service has a higher or lower priority based on the day of the week and time of day.

My Classes of Service

My Classes of Service

For example, daily items like running or editing a podcast segment typically have higher priority between the time I get up and beginning the work day. In a similar manner house/personal and podcast/blog entries priorities are driven by day of week and/or time of day. Alternately work and association items are driven by cost of delay. The backlog items in each class of service vie for the slices of attention available 24 hours a day and seven days a week.

Backlog items are captured in a wide variety of manners. For example, project work items can be captured using standard techniques for building an initial backlog (observation, showing, asking and/or synthesis). Backlogs for most projects can be developed using these techniques. Many smaller items or grand concepts will be discovered while encountering day-to-day trails and just generally living life. These need to be captured and logged (a habit that has been drilled into me by my wife), where they can be broken down and prioritized at leisure rather than being forgotten. Just as in a typical backlog, items that that are higher priority (by class of service) are broken down into next steps that are small enough to be completed in one to two days.

Using Scrumban as an approach to bring order out of chaos can be combined with other time management techniques. Real life is more complicated than a single project. For example, real life might be a project at work, prepping the yard for winter on the weekend, training for a half marathon and writing a book before sunrise. Each type of work is its own class of service that needs to be addressed separately to focus on what is important, when it is important.


Categories: Process Management

SSL termination and http caching with HAProxy, Varnish and Apache

Agile Testing - Grig Gheorghiu - Thu, 06/01/2017 - 23:53

A common requirement when setting up a development or staging server is to try to mimic production as much as possible. One scenario I've implemented a few times is to use Varnish in front of a web site but also use SSL. Since Varnish can't handle encrypted traffic, SSL needs to be terminated before it hits Varnish. One fairly easy way to do it is using HAProxy to terminate both HTTP and HTTPS traffic, then forwarding the unencrypted traffic to Varnish, which then forwards non-cached traffic to Apache or nginx. Here are the steps to achieve this on an Ubuntu 16.04 box.

1) Install HAProxy and Varnish

# apt-get install haproxy varnish

2) Get SSL certificates from Let’s Encrypt

# wget https://dl.eff.org/certbot-auto
# chmod +x certbot-auto
# ./certbot-auto -a webroot --webroot-path=/var/www/mysite.com -d mysite.com certonly

3) Generate combined chain + key PEM file to be used by HAProxy

# cat /etc/letsencrypt/live/mysite.com/fullchain.pem /etc/letsencrypt/live/mysite.com/privkey.pem > /etc/ssl/private/mysite.com.pem

4) Configure HAProxy

Edit haproxy.cfg and add frontend sections for ports 80 and 443 + backend section pointing to varnish on port 8888

# cat /etc/haproxy/haproxy.cfg
global
        log /dev/log    local0
        log /dev/log    local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
        user haproxy
        group haproxy
        daemon

        # Default SSL material locations
        ca-base /etc/ssl/certs
        crt-base /etc/ssl/private

        # Default ciphers to use on SSL-enabled listening sockets.
        # For more information, see ciphers(1SSL). This list is from:
        #  https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/
        ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS
        ssl-default-bind-options no-sslv3
        tune.ssl.default-dh-param 2048

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        timeout connect 5000
        timeout client  50000
        timeout server  50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

frontend www-http
   bind 172.31.8.204:80
   http-request set-header "SSL-OFFLOADED" "1"
   reqadd X-Forwarded-Proto:\ http
   default_backend varnish-backend

frontend www-https
   bind 172.31.8.204:443 ssl crt mysite.com.pem
   http-request set-header "SSL-OFFLOADED" "1"
   reqadd X-Forwarded-Proto:\ https
   default_backend varnish-backend

backend varnish-backend
   redirect scheme https if !{ ssl_fc }
   server varnish 172.31.8.204:8888 check

Enable UDP in rsyslog for haproxy logging by uncommenting 2 lines in /etc/rsyslog.conf:

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

Restart rsyslog and haproxy

# service rsyslog restart
# service haproxy restart

5) Configure varnish to listen on port 8888

Ubuntu 16.04 is using systemd for service management. You need to edit 2 files to configure the port varnish will listen on:

/lib/systemd/system/varnish.service
/etc/default/varnish

In both, set the port after the -a flag to 8888, then stop the varnish service, reload the systemctl daemon and restart the varnish service:

# systemctl stop varnish.service
# systemctl daemon-reload
# systemctl start varnish.service

By default, Varnish will send non-cached traffic to port 8080 on localhost.

6) Configure Apache or nginx to listen on 8080

For Apache, change port 80 to 8080 in all virtual hosts, and also change 80 to 8080 in /etc/apache2/ports.conf.




Empowering a new generation of localization professionals

Google Code Blog - Wed, 05/31/2017 - 18:58
Posted by The Google Localization Team

When her grandmother turned 80, Christina Hayek ‚ÄĒ Arabic Language Manager at Google ‚ÄĒ and her sisters wanted to give their beloved sitto a gift that would bring her closer to them. Chadia lives in Lebanon but her children and grandchildren are spread across the world. To bridge this geographical gap, Christina and her siblings gave their grandmother an Android smartphone. Much to Chadia‚Äôs surprise, she was able to use her phone in Arabic straight out of the box.

This isn‚Äôt magic‚ÄĒit‚Äôs the work of a dedicated localization team at Google. Spread over more than 30 countries, our team makes sure that all Google products are fun and easy to use in more than 70 languages. Localization goes beyond translation. While references to baseball and donuts work well in the US, these are not necessarily popular concepts in other cultures. Therefore we change these, for example, to football in Italy and croissant in France. Our mission is to create a diverse user experience that fits every language and every culture. We do this through a network of passionate translators and reviewers who localize Google products to make sure they sound natural to people everywhere.

With more and more people from around the world coming online every day, the localization industry keeps growing‚ÄĒand so does the demand for great translators, reviewers, and localization professionals. So, as part of Google‚Äôs mission to build products for everyone and make the web globally accessible, no matter where users are, we‚Äôre launching a massive open online course (MOOC) called Localization Essentials. In the words of Peter Lubbers, Google's Head of Developer Training:

"The language industry is one of the fastest growing in the world today, and as a former Internationalization Product Manager (and Dutch translator), I am absolutely thrilled that we've added Localization Essentials to our Google/Udacity training course catalog. The course is now available‚ÄĒfree of charge‚ÄĒto students all over the world. This was a huge cross-functional effort; a large team of localization experts across Google came together and rallied to create this course. It was great to see how everybody poured their heart and soul into this effort and it really shows in the course quality."

Localization Essentials was developed in collaboration with Udacity, and is free to access. It covers all localization basics needed to develop global products. This is how Bert Vander Meeren, Director of Localization at Google, described the collaboration:

‚ÄúToday, localization is becoming more and more important because the internet user base is growing rapidly, especially in non-English speaking countries. At the same time, education opportunities in the field are limited. This is an issue for our team and any business in need of large numbers of localization resources. So we decided to take the lead and address the issue, because who knows localization better than dedicated localization professionals with years of experience? Udacity already helped us develop and host several successful courses for Android developers, so this partnership was more than logical. This course is a great opportunity for anyone who wants to get knowledge and new skills in a still lesser-known field that‚Äôs important to develop products for a truly global audience. Whether you are a student, a professional, or an entrepreneur, you will learn a lot and expand your horizons.‚ÄĚ

By sharing our knowledge we hope that more culturally relevant products will become available to users everywhere, to provide opportunities to them that they didn’t have before.

We’re looking forward to seeing how sharing this localization knowledge will impact users from all over the world.

Categories: Programming

Defining ‚ÄúScaling‚ÄĚ Agile, Part 3: Creating Agile Product Development Capabilities

In the “Scaling” Agile: Part 1, I wrote about cross-functional collaborative teams. The cross-functional collaborative feature team is the basis for “scaling” agile. In “Scaling” Agile, Part 2, I wrote about programs. I realized I need another part before my original part 3 :-). This new part 3 is about creating agile product development capabilities.

In Agile and Lean Program Management, I wrote about continuous planning. (I should have called it “continual” but I didn’t. Oh well. In the second edition I can do that :-).)

Here’s a problem I see a lot in organizations who want to “scale” agile. They plan and replan less often than they should. (I am working on a product owner book to help with that problem. It’s next in the queue.)

This lack of replanning is a legacy from the more traditional yearly planning.

When organizations planned the project portfolio on a yearly basis,¬†people didn’t perceive a need to plan the products more often. However, with agile approaches, it’s ¬†possible and helpful to plan the project portfolio on a quarterly basis, if not more often. And, in order to replan more often, the teams need to deliver finished features more often.

In Manage Your Project Portfolio, I recommend people start with quarterly planning and move to more frequent planning as the teams get better at delivering value in the form of completed features.

It’s the same idea for product management and product ownership. Product management and product ownership are two different albeit related activities.

Product management manages the product over its lifetime. Product managers meet with customers. (So should real UX people so they can create and run experiments, but that’s a different post.) Product owners are part of an agile team and work with that team on a continuous basis.

The continuous basis part is part of what the PO needs to do: create and refine (small) stories, accept stories, and replan the roadmap and the next backlog. I wrote about rolling wave replanning in Consider Rolling Wave Roadmap and Backlog Planning.

Agile approaches allow us to replan. This is particularly helpful when we want to replan the project portfolio and product roadmaps.

This graph to the left is what we¬†hope are the product’s capabilities over time. Every project helps us create more capabilities.

Agile approaches allow us to release value on a regular basis, at least as often as every month in my world. Hopefully, every week or two. If you use continuous delivery, you can release value multiple times a day.

The more often you release, the more often the people with strategy responsibilities can manage the project portfolio. Have we done enough for this project for now? If so, feed the team another product. (In my world, there are often too many products for the number of teams. We flow work through a team, so the team can move to another product.)

The more often the product value team (I’ve been calling this the product owner value team) can replan in rolling waves and help the team(s) deliver value more often, the more often the managers can change the project portfolio, the mix of projects the organization can deliver for the number of teams.

Agile product development capabilities require:

  • Cross-functional collaborative feature teams that deliver value often. The more often a team can deliver, the easier everything else is. (This is Part 1 of this series.)
  • Product owners and their colleagues to replan¬†often. Again, the more the team(s) can deliver value, the easier it is to replan.
  • Managers can then make small safe-to-fail bets on the project portfolio. If they realize the will “lose” a bet, replan the project portfolio. Choose another mix of projects to see different value from products.

That’s what I mean about agile product development capabilities. If the product development organization can become good at delivering value, the product value team people can replan what they have in the roadmap and backlog. The project portfolio people can replan the project portfolio. The product development effort in the large becomes agile.

If you have questions, please ask in the comments. I’m just starting to explain this in writing. I’m sure I am not as clear as I want to be.

Categories: Project Management