Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Playing around with Yo

Xebia Blog - Fri, 08/01/2014 - 13:09

Yo has been quite a bit in the news lately. Mainly because it got a lot of investment, which surprised and shocked some people because it all seems too simple. Yo only allows you to send a Yo notification to other users. However, it has a lot of potential to become big while staying that simple.

Screenshot 2014-08-01 14.23.48

After reading Why A Stupid App Like Yo May Have Billion-Dollar Platform Potential a few days ago I felt it was time to play around a bit with Yo and it's API.

I came up with 3 very simple use cases that should be simple to solve with Yo:

  • Get a Yo when it's time to have lunch when I'm at work
  • Get a Yo when I forgot to check out from my parking App
  • Get a Yo when a new blog post is published here on the Xebia Blog

Time to register a couple of Yo developer usernames. The Yo API is, just like Yo itself, very simple. You register a username from which you want to receive notifications at http://yoapi.justyo.co after which you'll receive an API token for that username. Once you have that, people can subscribe to that username with the Yo app. Then you can send a Yo to either all subscribers or to a single subscribe with a simple POST request and the token. All this is explained at https://medium.com/@YoAppStatus/e7f2f0ec5c3c in more detail.

Let's start with our lunch notifications. I created the TIME2LUNCH username for this. I want my notifications at 12:00pm, since that's the time I go to lunch. So all I need is some service that sends the POST request every day at 12:00pm (for now I don't care about getting one in the weekend as well and I only care about my own time zone, Central European Time). Using NodeJS, it's just a single line of code to do such a request:

require('request')
  .post('http://api.justyo.co/yoall/', 
    {
      form: {api_token: 'my_secret_api_token'}
    }
  );

Now we need to have a scheduler that executes it every day at 12:00pm. Luckily Heroku has a scheduler that can do exactly that:

Screenshot 2014-07-31 17.43.51

So after deploying our javascript file with a single line of code and creating our scheduled job we will receive our daily Yo from TIME2LUNCH. Not bad for a first attempt.

Usually my co-workers will remind me when it's time to go to lunch so let's do something that's actually a bit less useless.

To park my car near the office I always use the Parkmobile service. At the end of the day I have to check out to avoid paying too much. Unfortunately it's an easy thing to forget. Parkmobile knows this and can send sms alerts at a specific time or after parking a certain amount of hours. Unfortunately they charge € 0.25 per sms. They also send e-mails, but they're easier to miss. It would be nice to get a Yo instead, for free of course.

What we need is to send the Yo POST request each time we receive the Parkmobile e-mails. Sounds like we might be able to use IFTTT (if this then that) to accomplish this. When browsing for channels and recipes on IFTTT I saw that they already support Yo as a channel. I thought I was gonna be done fast. Unfortunately it's only possible to use Yo as a trigger (if Yo then that) and not as an action (if this then Yo). So we need another solution here. I couldn't find a way to send a cURL request directly from IFTTT, but when Googling for a solution I found a webhook project: https://github.com/captn3m0/ifttt-webhook. The ifttt-webhook works by acting as a WordPress site, which is something that can act as an action of IFTTT. It then allows us to send a POST request to a specific URL. Not exactly the POST requests that are accepted by the Yo API though. But we already made some NodeJS code to send a Yo request so I'm sure we can add some code to accept a request from the ifttt-webhook and then pass it on to something that Yo does understand.

If we follow the instructions on the Github page and set our username to our Yo username and use our API token as password, then the webhook will send a POST request with a JSON body that looks something like this:

{ user: 'MYUSERNAME', pass: 'ab1234-1234-abcd-1234-abcd1234abcd', title: '' }

We can handle that in NodeJS like this:

var express = require('express');
var bodyParser = require('body-parser')
var app = express();
var request = require('request');

app.use(bodyParser.json());
app.post('/api/yo', function (req, res) {
  var user = req.body.user;
  var apiToken = req.body.pass;
  request.post('http://api.justyo.co/yo/',
    {
      form: {
        api_token: apiToken,
        username: user
      }
    });
});

var port = Number(process.env.PORT || 5000);
app.listen(port, function() {
  console.log('Listening on ' + port);
});

This is just a simple express web server that listens for POST calls on /api/yo and then uses the user and pass fields from the body to send a POST request to the Yo API.

This is deployed at http://youser.herokuapp.com/ so everyone can use it as a IFTTT to Yo action.

We can now create our IFTTT recipe. Creating the this step is easy. I receive the e-mails from Parkmobile in my Gmail and their e-mail address is norepy@parkmobile.com. So the rule becomes to trigger each time when I receive an email from them. Then in the that step I activate the WordPress channel with the Yo username and api token and in the body I set the http://youser.herokuapp.com/api/yo URL.

Here is the recipe:

Screenshot 2014-07-31 18.32.12

The last use case I had is to send a Yo each time a new blog post was posted on this blog. For that I registered the XEBIABLOG username (so make sure to subscribe to that in your Yo app if you want to get Yo'd as well for each new blog post).

Since this blog has an RSS feed, I figured I could poll that once in a while to check for new posts. We also already used the Heroku scheduler, so we might as well use that again. I found a little node library called feed-read that makes reading RSS feeds easy. So here is our little app that runs every hour:

var feed = require("feed-read");
var request = require('request');
var ONE_HOUR = 60 * 60 * 1000;

feed("http://blog.xebia.com/feed/", function(err, articles) {
  if (err) throw err;

  var lastArticle = articles[0];
  if ((new Date() - lastArticle.published) < ONE_HOUR) {
    console.log('Sending Yo for ' + lastArticle.title);
    request.post('http://api.justyo.co/yoall/',
      {
        form: {
          api_token: 'my_secret_token'
        }
      });
  }
});

We now have completed our 3 little use cases. Not the most useful things but nice non nonetheless. When looking back on them, we can imagine a couple of improvements. For example for the TIME2LUNCH it would be possible to make a little service where people could register and set their timezone at which they want to receive their notification. We could create a little database that store Yo usernames and the zone. But at this moment it's not possible to verify that USERX is really USERX. Yo doesn't support third party authentication like Facebook and Twitter have with OAuth. Perhaps that's something they will add in the future to make platform more useable for user specific notifications.

Paper: ZooKeeper: Wait-free coordination for Internet-scale systems

Do you really need to roll your own? ZooKeeper: Wait-free coordination for Internet-scale systems: In this paper, we describe ZooKeeper, a service for coordinating processes of distributed applications. Since ZooKeeper is part of critical infrastructure, ZooKeeper aims to provide a simple and high performance kernel for building more complex coordination primitives at the client. It incorporates elements from group messaging, shared registers, and distributed lock services in a replicated, centralized service. The interface exposed by Zoo-Keeper has the wait-free aspects of shared registers with an event-driven mechanism similar to cache invalidations of distributed file systems to provide a simple, yet powerful coordination service.

 

The ZooKeeper interface enables a high-performance service implementation. In addition to the wait-free property, ZooKeeper provides a per client guarantee of FIFO execution of requests and linearizability for all requests that change the ZooKeeper state. These design decisions enable the implementation of a high performance processing pipeline with read requests being satisfied byvlocal servers. We show for the target workloads, 2:1 to 100:1 read to write ratio, that ZooKeeper can handle tens to hundreds of thousands of transactions per second. This performance allows ZooKeeper to be used extensively by client applications.

ZooKeeper achieves throughput values of hundreds of thousands of operations per second for read-dominant workloads by using fast reads with watches, both of which served by local replicas. Although our consistency guarantees for reads and watches appear to be weak, we have shown with our use cases that this combination allows us to implement efficient and sophisticated coordination protocols at the client even though reads are not precedence-ordered and the implementation of data objects is wait-free. The wait-free property has proved to be essential for high performance.

Although we have described only a few applications, there are many others using ZooKeeper. We believe such a success is due to its simple interface and the powerful abstractions that one can implement through this interface. Further, because of the high-throughput of ZooKeeper, applications can make extensive use of it, not only course-grained locking.

Related Articles
Categories: Architecture

Preventing the Dogpile Effect - Problem and Solution

This is a guest repost Przemek Sobstel, who believes that dogpile effect issue is not covered enough, especially in the PHP world. Orignal article: Preventing dogpile effect.

The Dogpile effect occurs when cache expires and websites are hit by numerous requests the same time. From my own experiences working on big-traffic websites this is what I consider best the best solution. It was used sucessfully in the wild and it worked. Many people mention storing two redundant values FRESH + STALE, but for big traffic websites it was killing our network. We thought it worth sharing our solution and starting a discussion for sharing experiences.

Preventing Dogpiles
Categories: Architecture

100+ Motivational Quotes to Inspire Your Greatness

I updated my Motivational Quotes page.

I’ve got more than 100 motivational quotes on the page to help you find your inner-fire.

It’s not your ordinary motivational quotes list.  

It’s deep and it draws from several masters of inspiration including Bruce Lee, Jim Rohn, and Zig Ziglar.

Here is a sampling of some of my personal favorite motivational quotes ..

“If you always put limit on everything you do, physical or anything else. It will spread into your work and into your life. There are no limits. There are only plateaus, and you must not stay there, you must go beyond them.” – Bruce Lee

“Knowing is not enough; we must apply. Willing is not enough; we must do.” - Johann Wolfgang von Goethe

“Kites rise highest against the wind; not with it.” – Winston Churchill

“To hell with circumstances; I create opportunities.” – Bruce Lee

“Our greatest glory is not in never falling but in rising every time we fall.” – Confucius

“There is no such thing as failure. There are only results.” – Tony Robbins

“When it’s time to die, let us not discover that we have never lived.” -Henry David Thoreau

“People who say it cannot be done should not interrupt those who are doing it.” – Anonymous

“Motivation alone is not enough. If you have an idiot and you motivate him, now you have a motivated idiot.” – Jim Rohn

“If you love life, don’t waste time, for time is what life is made up of.” – Bruce Lee

For more quotes, check out my motivational quotes page.

It’s a living page and at some point I’ll do a complete revamp.

I think in the future I’ll organize it by sub-categories within motivation rather than by people.I think at the time it made sense to have words of wisdom by various folks, but now I think grouping motivational quotes by sub-categories would work much better, especially when there is such a large quantity of quotes.

Categories: Architecture, Programming

Using C3js with AngularJS

Gridshore - Tue, 07/29/2014 - 17:17

C3js is a graph javascript library on top of D3js. For a project we needed graphs and we ended up using c3js. How this happened and some of the first steps we took is written down by Roberto van der Linden in his blogpost: Creating charts with C3.js

Using C3js without other javascript libraries is of course perfectly doable. Still I think using it in combination with AngularJS is interesting to evaluate. In this blogpost I am going to document some steps I took to go from the basic c3 sample to a more advanced AngularJS powered sample.

Setup project – most basic sample

The easiest setup is by cloning by github repository: c3-angular-sample. If you are more the follow a long kind of person, than these are the steps.

We start by downloading C3js, d3 and AngularJS and create the first html page to get us started.

Next we create the index1.html that loads the stylesheet as well as the required javascript files. Two things are important to notice. One, the div with id chart. This is used to position the chart. Two, the script block that creates the chart object with data.

<!doctype html>
<html>
<head>
	<meta charset="utf-8">

	<link href="css/c3-0.2.4.css" rel="stylesheet" type="text/css">
</head>
<body>
	<h1>Sample 1: Most basic C3JS sample</h1>
	<div id="chart"></div>

	<!-- Load the javascript libraries -->
	<script src="js/d3/d3-3.4.11.min.js" charset="utf-8"></script>
	<script src="js/c3/c3-0.2.4.min.js"></script>

	<!-- Initialize and draw the chart -->
	<script type="text/javascript">
		var chart = c3.generate({
		    bindto: '#chart',
		    data: {
		      columns: [
		        ['data1', 30, 200, 100, 400, 150, 250],
		        ['data2', 50, 20, 10, 40, 15, 25]
		      ]
		    }
		});
	</script>
</body>
</html>

Next we introduce AngularJS to create the chart object.

Use AngularJS to create the chart

The first step is to remove the script block and add more javascript libraries to load. We load the AngularJS library and our own js file called graph-app-2.js. In the html tag we initialise the AngularJS app using ng-app=”graphApp”. In the body tag we initialise the Graph controller and call the function when the Angular app is initialised.

<html ng-app="graphApp">
<body ng-controller="GraphCtrl" ng-init="showGraph()">

Now let us have a look at some AngularJS javascript code. Notice that their is hardly a difference between the JavaScript code in both samples.

var graphApp = angular.module('graphApp', []);
graphApp.controller('GraphCtrl', function ($scope) {
	$scope.chart = null;
	
	$scope.showGraph = function() {
		$scope.chart = c3.generate({
			    bindto: '#chart',
			    data: {
			      columns: [
			        ['data1', 30, 200, 100, 400, 150, 250],
			        ['data2', 50, 20, 10, 40, 15, 25]
			      ]
			    }
			});		
	}
});

Still everything is really static, it makes use of mostly defaults. Let us move on and make it possible for the user to make some changes to the data and the type of chart.

Give the power to the user to change the data

We have two input boxes that accept common separated data and two drop downs to change the type of the chart. Fun to try out all the different types available. Below first he piece of html that contains the form to accept the user input.

	<form novalidate>
		<p>Enter in format: val1,val2,val3,etc</p>
		<input ng-model="config.data1" type="text" size="100"/>
		<select ng-model="config.type1" ng-options="typeOption for typeOption in typeOptions"></select>
		<p>Enter in format: val1,val2,val3,etc</p>
		<input ng-model="config.data2" type="text" size="100"/>
		<select ng-model="config.type2" ng-options="typeOption for typeOption in typeOptions"></select>
	</form>
	<button ng-click="showGraph()">Show graph</button>

This should be easy to understand. There are a few fields taken from the model as well as a button that generates the graph. Below is the complete javascript code for the controller.

var graphApp = angular.module('graphApp', []);

graphApp.controller('GraphCtrl', function ($scope) {
	$scope.chart = null;
	$scope.config={};
	$scope.config.data1="30, 200, 100, 200, 150, 250";
	$scope.config.data2="70, 30, 10, 240, 150, 125";

	$scope.typeOptions=["line","bar","spline","step","area","area-step","area-spline"];

	$scope.config.type1=$scope.typeOptions[0];
	$scope.config.type2=$scope.typeOptions[1];


	$scope.showGraph = function() {
		var config = {};
		config.bindto = '#chart';
		config.data = {};
		config.data.json = {};
		config.data.json.data1 = $scope.config.data1.split(",");
		config.data.json.data2 = $scope.config.data2.split(",");
		config.axis = {"y":{"label":{"text":"Number of items","position":"outer-middle"}}};
		config.data.types={"data1":$scope.config.type1,"data2":$scope.config.type2};
		$scope.chart = c3.generate(config);		
	}
});

In the first part we initialise all the variables in the $scope that are used on the form. In the showGraph function we have added a few things compared to the previous sample. We do not use the column data provider anymore, now we use the json provider. therefore we create arrays out of the comma separated numbers string. We also add a label to the y-axis and we set the types of charts using the same names as in the son object with the data. I think this is a good time to show a screen dump, stil it is much nicer to open the index3.html file yourself and play around with it.

Screen Shot 2014 07 29 at 17 52 26

Now we want to do more with angular, introduce a service that could obtain data from the server, but also make the data time based data.

Introducing time and data generation

The html is very basic, it only contains to buttons. One to start generating data and one to stop generating data. Let us focus on the javascript we have created. When creating the application we now tell it to look for another module called graphApp.services. Than we create the service using the factory and register it with the application. That way we can inject the service later on into the controller.

var graphApp = angular.module('graphApp', ['graphApp.services']);
var services = angular.module('graphApp.services', []);
services.factory('dataService', [function() {
	function DataService() {
		var data = [];
		var numDataPoints = 60;
		var maxNumber = 200;

		this.loadData = function(callback) {
			if (data.length > numDataPoints) {
				data.shift();
			}
			data.push({"x":new Date(),"data1":randomNumber(),"data2":randomNumber()});
			callback(data);
		};

		function randomNumber() {
			return Math.floor((Math.random() * maxNumber) + 1);
		}
	}
	return new DataService();
}]);

This service has one exposed method, loadData(callback). The result is that you get back a collection of objects with 3 fields: x, data1 and data2. The number of points is maximised to 60 and they have a random value between 0 and 200.

Next is the controller. I start with the controller, the parameters that are initialised as well as the method to draw the graph.

graphApp.controller('GraphCtrl', ['$scope','$timeout','dataService',function ($scope,$timeout,dataService) {
	$scope.chart = null;
	$scope.config={};

	$scope.config.data=[]

	$scope.config.type1="spline";
	$scope.config.type2="spline";
	$scope.config.keys={"x":"x","value":["data1","data2"]};

	$scope.keepLoading = true;

	$scope.showGraph = function() {
		var config = {};
		config.bindto = '#chart';
		config.data = {};
		config.data.keys = $scope.config.keys;
		config.data.json = $scope.config.data;
		config.axis = {};
		config.axis.x = {"type":"timeseries","tick":{"format":"%S"}};
		config.axis.y = {"label":{"text":"Number of items","position":"outer-middle"}};
		config.data.types={"data1":$scope.config.type1,"data2":$scope.config.type2};
		$scope.chart = c3.generate(config);		
	}

Take note of the $scope.config.keys which is used later on in the config.data.keys object. With this we move from an array of data per line to a an object with all the datapoints. The x is coming from the x property, the value properties are coming from data1 and data2. Also take note of the config.axis.x property. Here we specify that we a re now dealing with timeseries data and we format the ticks to show only the seconds. This is logical since we configured the have a maximum of 60 points and we create a new point every second, which we will see later on. The next code block shows the other three functions in the controller.

	$scope.startLoading = function() {
		$scope.keepLoading = true;
		$scope.loadNewData();
	}

	$scope.stopLoading = function() {
		$scope.keepLoading = false;
	}

	$scope.loadNewData = function() {
		dataService.loadData(function(newData) {
			var data = {};
			data.keys = $scope.config.keys;
			data.json = newData;
			$scope.chart.load(data);
			$timeout(function(){
				if ($scope.keepLoading) {
					$scope.loadNewData()				
				}
			},1000);			
		});
	}

In the loadNewData method we use the $timeout function of AngularJS. After a second we call the loadNewData again if we did not set the keepLoading property explicitly to false. Note that adding a point to the graph and calling the function recursively is done in the callback of the methode as send to the dataService.

Now run the sample for longer than 60 seconds and see what happens. Below is a screen dump, but is becomes more funny when you run the sample yourself.

Screen Shot 2014 07 29 at 18 16 01

That is about it, now I am going to integrate this with my elasticsearch-ui plugin

The post Using C3js with AngularJS appeared first on Gridshore.

Categories: Architecture, Programming

No Slack = No Innovation

"To accomplish great things we must dream as well as act." -- Anatole France

Innovation is the way to leap frog and create new ways to do things better, faster, and cheaper.

But it takes slack.

The problem is when you squeeze the goose, to get the golden egg, you lose the slack that creates the eggs in the first place.

In the book The Future of Management, Gary Hamel shares how when there is a lack of slack, there is no innovation.

The Most Important Source of Productivity is Creativity

Creativity unleashes productivity.  And it takes time to unleash creativity.  But the big bold bet is that the time you give to creativity and innovation, pays you back with new opportunities and new ways to do things better, faster, or cheaper.

Via The Future of Management:

“In the pursuit of efficiency, companies have wrung a lot of slack out of their operations.  That's a good thing.  No one can argue with the goal of cutting inventory levels, reducing working capital, and slashing over-head.  The problem, though, is that if you wring all the slack out of a company, you'll wring out all of the innovation as well.  Innovation takes time -- time to dream, time to reflect, time to learn, time to invent, and time to experiment.  And it takes uninterrupted time -- time when you can put your feet up and stare off into space.  As Pekka Himanen put it in his affectionate tribute to hackers, '... the information economy's most important source of productivity is creativity, and it is not possible to create interesting things in a constant hurry or in a regulated way from nine to five.'”

There is No “Thinking Time”

Without think time, creativity lives in a cave.

Via The Future of Management:

“While the folks in R&D and new product development are given time to innovate, most employees don't enjoy this luxury.  Every day brings a barrage of e-mails, voice mails, and back-to-back meetings.  In this world, where the need to be 'responsive' fragments human attention into a thousand tiny shards, there is no 'thinking time.'  And therein lies the problem.  However creative your colleagues may be, if they don't have the right to occasionally abandon their posts and work on something that's not mission critical, most of their creativity will remain dormant.”

Are People Encouraged to Quietly Dream Up the Future?

If you want more innovation, make space for it.

Via The Future of Management:

“OK, you already know that -- but how is that knowledge reflected in your company's management processes?  How hard is it for a frontline employee to get permission to spend 20 percent of her time working on a project that has nothing to do with her day job, nor your company's 'core businesses'?  And how often does this happen?  Does your company track the number of hours employees spend working on ideas that are incidental to their core responsibilities? Is 'slack' institutionalized in the same way that cost efficiency is?  Probably not.  There are plenty of incentives in your company for people to stay busy.  ('Maybe if I look like I'm working flat out, they won't send my job offshore.')  But where are the incentives that encourage people to spend time quietly dreaming up the future?”

Are you slacking your way to a better future?

You Might Also Like

Innovation Quotes

The Drag of Old Mental Models on Innovation and Change

The New Competitive Landscape

The New Realities that Call for New Organizational and Management Capabilities

Who’s Managing Your Company

Categories: Architecture, Programming

3 Challenges to Help You Set New Benchmarks in Innovation

If you want to change your game, you need to know what the key challenges are.

Innovation is a game that you can play much better, if you know where and how to debottleneck it.

In the book The Future of Management, Gary Hamel shares 3 challenges that he believes can help you unleash your organization’s capacity for innovation.

  1. How can you enroll every individual within your company in the work of innovation, and equip each one with creativity-boosting tools?
  2. How can you ensure that top management's hallowed beliefs don't straightjacket innovation, and that heretical ideas are given the chance to prove their worth?
  3. How can you create the time and space for grassroots innovation in an organization that is running flat to deliver today's results?

According to Hamel, "Make progress on these challenges and your company will set new benchmarks in innovation."

If I think back through the various teams I’ve been on at Microsoft, one team that I was on was especially good at helping innovation flourish, and we were constantly pushing the envelope to “be what’s next.”   Our innovation flourished the most when we directly addressed the challenges above.  People were challenged to share and test their ideas more freely and innovation was baked into how we planned our portfolio, programs, and projects.

Innovation was a first-class citizen – by design.

You Might Also Like

High-Leverage Strategies for Innovation

Innovation Life Cycle

Lessons Learned from the Most Successful Innovators

The Drag of Old Mental Models on Innovation and Change

The New Realities that Call for New Organizational and Management Capabilities

Categories: Architecture, Programming

The Great Microservices vs Monolithic Apps Twitter Melee

Once upon a time a great Twitter melee was fought for the coveted title of Consensus Best Way to Structure Systems. The competition was between Microservices and Monolithic Apps. 

Flying the the logo of Microservices, from a distant cloud covered land, is the Kingdom of Netflix, whose champion was Sir Adrian Cockcroft (who has pledged fealty to another). And for the Kingdom of ThoughtWorks we have Sir Sam Newman as champion.

Flying the logo of the Monolithic App is champion Sir John Allspaw, from the fair Kingdom of Etsy.

Knights from the Kingdom of Digital Ocean and several independent realms filled out the list.

To the winner goes a great prize: developer mindshare and the favor of that most fickle of ladies, Lady Luck.

May the best paradigm win.

The opening blow was wielded by the highly ranked Sir Cockcroft, a veteran of many tournaments:

Categories: Architecture

Fearless Speaking

“Do one thing every day that scares you.” ― Eleanor Roosevelt

I did a deep dive book review.

This time, I reviewed Fearless Speaking.

The book is more than meets the eye.

It’s actually a wealth of personal development skills at your fingertips and it’s a powerful way to grow your personal leadership skills.

In fact, there are almost fifty exercises throughout the book.

Here’s an example of one of the techniques …

Spotlight Technique #1

When you’re overly nervous and anxious as a public speaker, you place yourself in a ‘third degree’ spotlight.  That’s the name for the harsh bright light police detectives use in days gone by to ‘sweat’ a suspect and elicit a confession.  An interrogation room was always otherwise dimly lit, so the source of light trained on the person (who was usually forced to sit in a hard straight backed chair) was unrelenting.

This spotlight is always harsh, hot, and uncomfortable – and the truth is, you voluntarily train it on yourself by believing your audience is unforgiving.  The larger the audience, the more likely you believe that to be true.

So here’s a technique to get out from under this hot spotlight that you’re imagining so vividly turn it around! Visualize swiveling the spotlight so it’s aimed at your audience instead of you.  After all, aren’t you supposed to illuminate your listeners? You don’t want to leave them in the dark, do you?

There’s no doubt that it’s cooler and much more comfortable when you’re out under that harsh light.  The added benefit is that now the light is shining on your listeners – without question the most important people in the room or auditorium!

I like that there are so many exercises and techniques to choose from.   Many of them don’t fit my style, but there were several that exposed me to new ways of thinking and new ideas to try.

And what’s especially great is knowing that these exercise come from professional actors and speakers – it’s like an insider’s guide at your fingertips.

My book review on Fearless Speaking includes a list of all the exercises, the chapters at a glance, key features from the book, and a few of my favorite highlights from the book (sort of like a movie trailer for the book.)

You Might Also Like

7 Habits of Highly Effective People at a Glance

347 Personal Effectiveness Articles to Help You Change Your Game

Effectiveness Blog Post Roundup

Categories: Architecture, Programming

Transform the input before indexing in elasticsearch

Gridshore - Sat, 07/26/2014 - 07:51

Sometimes you are indexing data and want to have as little to do in the input, or maybe even no influence on the input. Still you need to make changes, you want other content, or other fields. Maybe even remove fields. In elasticsearch 1.3 a new feature is introduces called Transform. In this blogpost I am going to show some of the aspects of this new feature.

Insert the document with the problem

The input we get is coming from a system that puts the string null in a field if it is empty. We do not want null as a string in elasticsearch index. Therefore we want to remove this field completely when indexing a document like that. We start with the example and the proof that you can search on the field.

PUT /transform/simple/1
{
  "title":"This is a document with text",
  "description":"null"
}

Now search for the word null in the description.

For completeness I’ll show you the response as well.

Response:
{
   "took": 1,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 0.30685282,
      "hits": [
         {
            "_index": "transform",
            "_type": "simple",
            "_id": "1",
            "_score": 0.30685282,
            "_source": {
               "title": "This is a document with text",
               "description": "null"
            }
         }
      ]
   }
}
Change mapping to contain transform

Next we are going to use the transform functionality to remove the field if it contains the string null. To do that we need to remove the index and create a mapping containing the transform functionality. We use the groovy language for the script. Beware that the script is only validated when the first document is inserted.

PUT /transform
{
  "mappings": {
    "simple": {
      "transform": {
        "lang":"groovy",
        "script":"if (ctx._source['description']?.equals('null')) ctx._source['description'] = null"
      },
      "properties": {
        "title": {
          "type": "string"
        },
        "description": {
          "type": "string"
        }
      }
    }
  }
}

When we insert the same document as before and execute the same query we do not get hits. The description field is no longer indexed. An important aspect is that the actual _source is not changed. When requesting the _source of the document you still get back the original document.

GET transform/simple/1/_source
Response:
{
   "title": "This is a document with text",
   "description": "null"
}
Add a field to the mapping

To add a bit more complexity, we add a field called nullField which will contain the name of the field that was null. Not very useful but it suits to show the possibilities.

PUT /transform
{
  "mappings": {
    "simple": {
      "transform": {
        "lang":"groovy",
        "script":"if (ctx._source['description']?.equals('null')) {ctx._source['description'] = null;ctx._source['nullField'] = 'description';}"
      },
      "properties": {
        "title": {
          "type": "string"
        },
        "description": {
          "type": "string"
        },
        "nullField": {
          "type": "string"
        }
      }
    }
  }
}

Notice that we script has changed, not only do we remove the description field, now we also add a new field called nullField. Check that the _source is still not changed. Now we do a search and only return the fields description and nullField. Before scrolling to the response think about the response that you would expect.

GET /transform/_search
{
  "query": {
    "match_all": {}
  },
  "fields": ["nullField","description"]
}

Did you really think about it? Try it out and notice that the nullField is not returned. That is because we did not store it in the index and it is not obtained from the source. So if we really need this value, we can store the nullField in the index and we are fine.

PUT /transform
{
  "mappings": {
    "simple": {
      "transform": {
        "lang":"groovy",
        "script":"if (ctx._source['description']?.equals('null')) {ctx._source['description'] = null;ctx._source['nullField'] = 'description';}"
      },
      "properties": {
        "title": {
          "type": "string"
        },
        "description": {
          "type": "string"
        },
        "nullField": {
          "type": "string",
          "store": "yes"
        }
      }
    }
  }
}

Than with the match all query for two fields we get the following response.

GET /transform/_search
{
  "query": {
    "match_all": {}
  },
  "fields": ["nullField","description"]
}
Response:
{
   "took": 2,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
         {
            "_index": "transform",
            "_type": "simple",
            "_id": "1",
            "_score": 1,
            "fields": {
               "description": [
                  "null"
               ],
               "nullField": [
                  "description"
               ]
            }
         }
      ]
   }
}

Yes, now we do have the new field. That is it, but wait there is more you need to know. There is a way to check what is actually passed to the index for a certain document.

GET transform/simple/1?pretty&_source_transform
Result:
{
   "_index": "transform",
   "_type": "simple",
   "_id": "1",
   "_version": 1,
   "found": true,
   "_source": {
      "description": null,
      "nullField": "description",
      "title": "This is a document with text"
   }
}

Notice the null description and the nullField in the _source.

Final remark

You cannot update the transform part, think about what would happen to your index when some documents did pass the transform version 1 and others version 2.

I would be gentile with this feature, try to solve it before sending it to elasticsearch, but maybe you just have the usecase for this feature, now you know it exists.

In my next blogpost I dive a little bit deeper into the scripting module.

The post Transform the input before indexing in elasticsearch appeared first on Gridshore.

Categories: Architecture, Programming

Marketing scrum vs IT scrum - a report published and presented at agile 2014

Xebia Blog - Fri, 07/25/2014 - 17:49

As we know, Scrum is the perfect framework for IT / software development projects to learn, adapt to change and deliver great software of value, faster.

But is Scrum also usable outside of software development? Can we apply similar or maybe even the same principals in other departments in the enterprise?

Yes, we can! And yes there are differences but there are also a lot of similarities.

We (Remco en Me)  successfully implemented Scrum in the marketing departments of two large companies: The ANWB and ING Bank. Both companies are now using Scrum for the development of new campaigns, their full commercial expressions and even at the product development level. They wanted a faster time to market, more ownership, and greater innovation. How did we approach and realized a transition with those goals in the marketing environment? And what are the results?

So when we are not delivering software but other things, how does Scrum change? Well, a great deal actually. The people working in these other departments are, in general, quite different to those in Software Development (and yes more than you would expect). This means coaches or change agents need to take another approach.

Since the people are different, it is possible to go faster or ‘deeper’ in certain areas. Entrepreneurial skills or ambitions are more present in marketing. This gives a sense of ‘act first apologize later’, taking ownership, a higher drive to succeed, and upfront and willing behavior. Scrumming here means thinking more about business goals and KPIs (how to go from department to scrumteam goals for example). After that the fun begins…

I will be speaking about this topic at agile 2014. A great honor offcourse to be standing there. I will also attende the conference and therefor try to post some updates here.

To read more about this topic you can read my publication about marketing scrum. It has the extensive research paper I publisched about this story. Please feel free to give me comments and questions either about agile 2014 or the paper.

 

Enjoy reading the paper:

Marketing scrum vs IT scrum – two marketing case studies who now ‘act first and apologize later'

 

Stuff The Internet Says On Scalability For July 25th, 2014

Hey, it's HighScalability time:


It's systems all the way down. Bugs That Call Us Home.
  • 1 million users in just 4 days: Yo;  30 billion: Pinterest Pins
  • Quotable Quotes:
    • @GlennF: Amazon still dreams it is a startup, like a dog dreaming of chasing rabbits, twitching its legs while asleep.
    • @mfdii: Nobody knows how git works. We all just type in commands like monkeys trying to write Shakespeare. #devopsdays 
    • Benedict Evans: When you pull these strands together, smartphones don't just increase the size of the internet by 2x or 3x, but more like 5x or 10x. It's not just how many devices, but how different those devices are, that has the multiplier effect.
    • @Aaronontheweb: @codinghorror I broke this rule for myself last week. Spent 3 days fixing a problem that we finally solved by a $0.06/hour AWS bill increase
    • Physicist George Ellis: Barring something very unforeseen – the possible tests of the very large and the very small are coming towards the limits of whatever will be possible.
    • The Master Switch: Once the industry had concluded that its profits could be maximized if more people listened to fewer stations, the government, acting as if the business of America were only business, did the industry’s bidding, showing only the most feeble awareness of its consequences for the American ideal of free expression.

  • Ex-Googlers try to recreate Spanner with CockroachDB (awesome name!), which is A Scalable, Geo-Replicated, Transactional Datastore. The design is here and looks good. There's an article on Wired. Good discussion on HackerNews. A globally distributed transactional database it's not, yet, but it's early days yet. After all, they can only work in the dark.

  • Useful post on Handling 1 Billion requests a week with Symfony2. Symfony2 provides good performance and a nice development environment. HAProxy distributes to application servers. Varnish in every application’s server to keep high availability – without having a single point of failure (SPOF). Redis and MySQL for storing data. MySQL is mostly used as a third-tier cache layer (Varnish > Redis > MySQL) for non-expiring resources. 

  • The truest form of the Interest Graph on the net? Details on how Pinterest scales their data infrastructure to create a personalized discovery engine. 20 terabytes of new data each day. 10 petabytes of data in S3. 100 regular Mapreduce users run over 2,000 jobs each day through Qubole. 6 standing Hadoop clusters comprised of over 3,000 nodes. 

  • Just like Captain Kirk. Shifts In Algorithm Design: Now today, in the 21st century, we have a better way to attack problems. We change the problem, often to one that is more tractable and useful. In many situations solving the exact problem is not really what a practitioner needs. If computing X exactly requires too much time, then it is useless to compute it. A perfect example is the weather: computing tomorrow’s weather in a week’s time is clearly not very useful. The brilliance of the current approach is that we can change the problem. 

  • Wet Computing Could Put a Terabyte in a Tablespoon: Researchers from the University of Michigan and New York University demonstrated how plastic nanoparticles, deposited in a liquid, can form a one-bit cluster—the essential building block for information storage. It's called "wet computing," and the technique mimics other biological processes found in nature, like DNA in living cells.

  • Daniel Eloff: The world is not just going massively multicore, it's going heterogeneous core. The one core fits all model of programming is going away. Big performance and efficiency gains can be had from splitting your application among different types of specialized processors. Programmable hardware with FPGAs seems like a natural extension of this trend.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

The Drag of Old Mental Models on Innovation and Change

“Don’t worry about people stealing your ideas. If your ideas are any good, you’ll have to ram them down people’s throats.” — Howard Aiken

It's not a lack of risk taking that holds innovation and change back. 

Even big companies take big risks all the time.

The real barrier to innovation and change is the drag of old mental models.

People end up emotionally invested in their ideas, or they are limited by their beliefs or their world views.  They can't see what's possible with the lens they look through, or fear and doubt hold them back.  In some cases, it's even learned helplessness.

In the book The Future of Management, Gary Hamel shares some great insight into what holds people and companies back from innovation and change.

Yesterday’s Heresies are Tomorrow’s Dogmas

Yesterday's ideas that were profoundly at odds with what is generally accepted, eventually become the norm, and then eventually become a belief system that is tough to change.

Via The Future of Management:

“Innovators are, by nature, contrarians.  Trouble is, yesterday's heresies often become tomorrow's dogmas, and when they do, innovation stalls and the growth curve flattens out.”

Deeply Held Beliefs are the Real Barrier to Strategic Innovation

Success turns beliefs into barriers by cementing ideas that become inflexible to change.

Via The Future of Management:

“... the real barrier to strategic innovation is more than denial -- it's a matrix of deeply held beliefs about the inherent superiority of a business model, beliefs that have been validated by millions of customers; beliefs that have been enshrined in physical infrastructure and operating handbooks; beliefs that have hardened into religious convictions; beliefs that are held so strongly, that nonconforming ideas seldom get considered, and when they do, rarely get more than grudging support.”

It's Not a Lack of Risk Taking that Holds Innovation Back

Big companies take big risks every day.  But the risks are scoped and constrained by old beliefs and the way things have always been done.

Via The Future of Management:

“Contrary to popular mythology, the thing that most impedes innovation in large companies is not a lack of risk taking.  Big companies take big, and often imprudent, risks every day.  The real brake on innovation is the drag of old mental models.  Long-serving executives often have a big chunk of their emotional capital invested in the existing strategy.  This is particularly true for company founders.  While many start out as contrarians, success often turns them into cardinals who feel compelled to defend the one true faith.  It's hard for founders to credit ideas that threaten the foundations of the business models they invented.  Understanding this, employees lower down self-edit their ideas, knowing that anything too far adrift from conventional thinking won't win support from the top.  As a result, the scope of innovation narrows, the risk of getting blindsided goes up, and the company's young contrarians start looking for opportunities elsewhere.”

Legacy Beliefs are a Much Bigger Liability When It Comes to Innovation

When you want to change the world, sometimes it takes a new view, and existing world views get in the way.

Via The Future of Management:

“When it comes to innovation, a company's legacy beliefs are a much bigger liability than its legacy costs.  Yet in my experience, few companies have a systematic process for challenging deeply held strategic assumptions.  Few have taken bold steps to open up their strategy process to contrarian points of view.  Few explicitly encourage disruptive innovation.  Worse, it's usually senior executives, with their doctrinaire views, who get to decide which ideas go forward and which get spiked.  This must change.”

What you see, or can’t see, changes everything.

You Might Also Like

The New Competitive Landscape

The New Realities that Call for New Organizational and Management Capabilities

Who’s Managing Your Company

Categories: Architecture, Programming

Playing with two most interesting new features of elasticsearch 1.3.0

Gridshore - Fri, 07/25/2014 - 11:47

Just a few days a go elasticsearch released version 1.3.0 of their flagship product. The first one is the most waited for feature called the Top hits aggregation. Basically this is what is called grouping. You want to group certain items based on one characteristic, but within this group you want to have the best matching result(s) based on score. Another very important feature is the new support for scripts. Better security options when using scripts using sandboxed script languages.

In this blogpost I am going to explain and show the top_hits feature as well as the new scripting support.


Top hits

I am going to show a very simple example of top hits using my music index. This index contains all the songs I have in my itunes library. First step is to find songs by genre, the following query gives (the default) 10 hits based on the match_all query and the terms aggregation as requested.

GET /mymusic/_search
{
  "aggs": {
    "byGenre": {
      "terms": {
        "field": "genre",
        "size": 10
      }
    }
  }
}

The response is of the format:

{
	"hits": {},
	"aggregations": {
		"byGenre": {
			"buckets": [
				{"key":"rock","doc_count":1910},
				...
			]
		}
    }
}

Now we add the query to the request, songs containing the word love in the title.

GET /mymusic/_search
{
  "query": {
    "match": {
      "name": "love"
    }
  }, 
  "aggs": {
    "byGenre": {
      "terms": {
        "field": "genre",
        "size": 10
      }
    }
  }
}

Now we have less hits, still a number of buckets and the amount of songs that match our query within that bucket. The biggest change is the score in the returned hits. In te previous query the score was always 1, now the score is different due to the query we execute. The highest score now is the song Love by The Mission. The genre for this song is Rock and the song is from the year 1990. Time to introduce the top hits aggregation. With this query we can return the top song containing the word love in the title per genre

GET /mymusic/_search
{
  "query": {
    "match": {
      "name": "love"
    }
  },
  "aggs": {
    "byGenre": {
      "terms": {
        "field": "genre",
        "size": 5
      },
      "aggs": {
        "topFoundHits": {
          "top_hits": {
            "size": 1
          }
        }
      }
    }
  }
}

Again we get hits, but they are not different from the query before. The interesting part is in the aggs part. Here we add a sub aggregation to the byGenre aggregation. This aggregation is called topFoundHits of type top_hits. We only return the best hit per genre. The next code block shows the part of the response with the top hits, I did remove the content of the _source field in the top_hits to keep the response shorter.

{
   "took": 4,
   "timed_out": false,
   "_shards": {
      "total": 3,
      "successful": 3,
      "failed": 0
   },
   "hits": {
      "total": 141,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "byGenre": {
         "buckets": [
            {
               "key": "rock",
               "doc_count": 52,
               "topFoundHits": {
                  "hits": {
                     "total": 52,
                     "max_score": 4.715253,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "4147",
                           "_score": 4.715253,
                           "_source": {
                              "name": "Love",
                           }
                        }
                     ]
                  }
               }
            },
            {
               "key": "pop",
               "doc_count": 39,
               "topFoundHits": {
                  "hits": {
                     "total": 39,
                     "max_score": 3.3341873,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "11381",
                           "_score": 3.3341873,
                           "_source": {
                              "name": "Love To Love You",
                           }
                        }
                     ]
                  }
               }
            },
            {
               "key": "alternative",
               "doc_count": 12,
               "topFoundHits": {
                  "hits": {
                     "total": 12,
                     "max_score": 4.1945505,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "7889",
                           "_score": 4.1945505,
                           "_source": {
                              "name": "Love Love Love",
                           }
                        }
                     ]
                  }
               }
            },
            {
               "key": "b",
               "doc_count": 9,
               "topFoundHits": {
                  "hits": {
                     "total": 9,
                     "max_score": 3.0271564,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "2549",
                           "_score": 3.0271564,
                           "_source": {
                              "name": "First Love",
                           }
                        }
                     ]
                  }
               }
            },
            {
               "key": "r",
               "doc_count": 7,
               "topFoundHits": {
                  "hits": {
                     "total": 7,
                     "max_score": 3.0271564,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "2549",
                           "_score": 3.0271564,
                           "_source": {
                              "name": "First Love",
                           }
                        }
                     ]
                  }
               }
            }
         ]
      }
   }
}

Did you note a problem with my analyser for genre? Hint R&B!

More information on the top_hits aggregation can be found here:

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-metrics-top-hits-aggregation.html

Scripting

Elasticsearch has support for scripts for a long time. The default scripting language was and is mvel up to version 1.3. It will change to groovy in 1.4. Mvel is not a well known scripting language. The biggest advantage is that mvel is very powerful. The disadvantage is that it is to powerful. Mvel does not come with a sandbox principle. Therefore is is possible to write some very nasty scripts even when only doing a query. This was very well shown by a colleague of mine (Byron Voorbach) who created a query to read private keys on developer machines who did not safeguard their elasticsearch instance. Therefore dynamic scripting was switched off in version 1.2 by default.

This came with a very big disadvantage, now it was not possible anymore to use the function_score query without resorting to stored scripts on the server. In version 1.3 of elasticsearch a much better way is introduced. Now you use sandboxed scripting languages like groovy to keep using the flexible approach. Groovy can be configured to include object creation and method calls that are allowed. More information about this is provided in the elasticsearch documentation about scripting.

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-scripting.html

Next is an example query that is querying my music index. This index contains all the songs from my music library. It queries all he songs after the year 1999 and calculates the score based on the year. So the newest songs get the highest score. And yes I know a sort by year desc would have given the same result.

GET mymusic/_search
{
  "query": {
    "function_score": {
      "query": {
        "range": {
          "year": {
            "gte": 2000
          }
        }
      },
      "functions": [
        {
          "script_score": {
            "lang": "groovy", 
            "script": "_score * doc['year'].value"
          }
        }
      ]
    }
  }
}

The score now becomes high, since we do a range query we get back only scores of one. Using the function_score as the multiplication of the year with the score, the end score is the year. I added the year as the only field to return, some of the results than are:

{
   "took": 4,
   "timed_out": false,
   "_shards": {
      "total": 3,
      "successful": 3,
      "failed": 0
   },
   "hits": {
      "total": 2895,
      "max_score": 2014,
      "hits": [
         {
            "_index": "mymusic",
            "_type": "itunes",
            "_id": "12965",
            "_score": 2014,
            "fields": {
               "year": [
                  "2014"
               ]
            }
         },
         {
            "_index": "mymusic",
            "_type": "itunes",
            "_id": "12975",
            "_score": 2014,
            "fields": {
               "year": [
                  "2014"
               ]
            }
         }
      ]
   }
}

Next up is the last sample, a combination of top_hits and scripting.

Top hits with scripting

We start with the sample from top_hits using my music index. Now we want to sort the buckets on the score of the best matching document in the bucket. The default is the number of documents in the bucket. As mentioned in the documentation you need a trick to do this.

The top_hits aggregator isn’t a metric aggregator and therefor can’t be used in the order option of the terms aggregator.

GET /mymusic/_search?search_type=count
{
  "query": {
    "match": {
      "name": "love"
    }
  },
  "aggs": {
    "byGenre": {
      "terms": {
        "field": "genre",
        "size": 5,
        "order": {
          "best_hit":"desc"
        }
      },
      "aggs": {
        "topFoundHits": {
          "top_hits": {
            "size": 1
          }
        },
        "best_hit": {
          "max": {
            "lang": "groovy", 
            "script": "doc.score"
          }
        }
      }
    }
  }
}

The results of this query again with most of the _source taken out is following. Compare it to the query in the top_hits section. Notice the different genres that we get back now. Also check the scores.

{
   "took": 4,
   "timed_out": false,
   "_shards": {
      "total": 3,
      "successful": 3,
      "failed": 0
   },
   "hits": {
      "total": 141,
      "max_score": 0,
      "hits": []
   },
   "aggregations": {
      "byGenre": {
         "buckets": [
            {
               "key": "rock",
               "doc_count": 37,
               "topFoundHits": {
                  "hits": {
                     "total": 37,
                     "max_score": 4.715253,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "4147",
                           "_score": 4.715253,
                           "_source": {
                              "name": "Love",
                           }
                        }
                     ]
                  }
               },
               "best_hit": {
                  "value": 4.715252876281738
               }
            },
            {
               "key": "alternative",
               "doc_count": 12,
               "topFoundHits": {
                  "hits": {
                     "total": 12,
                     "max_score": 4.1945505,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "7889",
                           "_score": 4.1945505,
                           "_source": {
                              "name": "Love Love Love",
                           }
                        }
                     ]
                  }
               },
               "best_hit": {
                  "value": 4.194550514221191
               }
            },
            {
               "key": "punk",
               "doc_count": 3,
               "topFoundHits": {
                  "hits": {
                     "total": 3,
                     "max_score": 4.1945505,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "7889",
                           "_score": 4.1945505,
                           "_source": {
                              "name": "Love Love Love",
                           }
                        }
                     ]
                  }
               },
               "best_hit": {
                  "value": 4.194550514221191
               }
            },
            {
               "key": "pop",
               "doc_count": 24,
               "topFoundHits": {
                  "hits": {
                     "total": 24,
                     "max_score": 3.3341873,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "11381",
                           "_score": 3.3341873,
                           "_source": {
                              "name": "Love To Love You",
                           }
                        }
                     ]
                  }
               },
               "best_hit": {
                  "value": 3.3341872692108154
               }
            },
            {
               "key": "b",
               "doc_count": 7,
               "topFoundHits": {
                  "hits": {
                     "total": 7,
                     "max_score": 3.0271564,
                     "hits": [
                        {
                           "_index": "mymusic",
                           "_type": "itunes",
                           "_id": "2549",
                           "_score": 3.0271564,
                           "_source": {
                              "name": "First Love",
                           }
                        }
                     ]
                  }
               },
               "best_hit": {
                  "value": 3.027156352996826
               }
            }
         ]
      }
   }
}

This is just a first introduction into the top_hits and scripting. Stay tuned for more blogs around these topics.

The post Playing with two most interesting new features of elasticsearch 1.3.0 appeared first on Gridshore.

Categories: Architecture, Programming

Review The Twitter Story by Nick Bilton

Gridshore - Thu, 07/24/2014 - 23:35


A lot of people dream of creating the new Facebook, Twitter or Instagram. Nobody knows when they will create it, most of the ideas just came to be big. Personally I do not believe I will ever create such a product. To busy with to much different things, usually based on great ideas of others. One thing I do like is reading about the success stories of others. I read the book about starbucks, microsoft, apple and a few others.

Recently I started reading the Twitter Story. It reeds like an exciting story, nevertheless it is telling a story based on interviews and facts behind one of the most exciting companies on the internet of the last century.

I do not think it is a coincidence that a lot of what I read in the Lean Startup but also the starbucks story is coming back in the twitter story. One thing that struck me in this book is what business does to friendship. It is also showing that people with great ideas are usually not the people that make this ideas into a profitable company.

When starting a company based on your terrific idea, read this book and learn from it. It might make your life a lot better.

The post Review The Twitter Story by Nick Bilton appeared first on Gridshore.

Categories: Architecture, Programming

The New Realities that Call for New Organizational and Management Capabilities

“The only people who can change the world are people who want to. And not everybody does.” -- Hugh MacLeod

Is it just me or is the world changing faster than ever?

I hear from everybody around me (inside and outside of Microsoft) how radically their worlds are changing under their feet, business models are flipped on their heads, and the game of generating new business value for customers is at an all-time competitive high.

Great.

Challenge is where growth and greatness come from.  It’s always a chance to test what we’re capable of and respond to whatever gets thrown our way.  But first, it helps to put a finger on what exactly these changes are that are disrupting our world, and what to focus on to survive and thrive.

In the book The Future of Management, Gary Hamel shares some great insight into the key challenges that companies are facing that create even more demand for management innovation.

The New Realities We’re Facing that Call for Management Innovation

I think Hamel describes our new world pretty well …

Via The Future of Management:

  • “As the pace of change accelerates more, more and more companies are finding themselves on the wrong side of the change curve.  Recent research by L.G. Thomas and Richard D'Aveni suggests that industry leadership is changing hands more frequently, and competitive advantage is eroding more rapidly, than ever before.  Today, it's not just the occasional company that gets caught out by the future, but entire industries -- be it traditional airlines, old-line department stores, network television broadcasters, the big drug companies, America's carmakers, or the newspaper and music industries.”
  • “Deregulation, along with the de-scaling effects of new technology, are dramatically reducing the barriers to entry across a wide range of industries, from publishing to telecommunications to banking to airlines.  As a result, long-standing oligopolies are fracturing and competitive 'anarchy' is on the rise.”
  • “Increasingly, companies are finding themselves enmeshed in 'value webs' and 'ecosystems' over which they have only partial control.  As a result, competitive outcomes are becoming less the product of market power, and more the product of artful negotiation.  De-verticalization, disintermediation, and outsource-industry consortia, are leaving firms with less and less control over their own destinies.”
  • “The digitization of anything not nailed down threatens companies that make their living out of creating and selling intellectual property.  Drug companies, film studios, publishers, and fashion designers are all struggling to adapt to a world where information and ideas 'want to be free.'”
  • The internet is rapidly shifting bargaining power from producers to consumers.  In the past, customer 'loyalty' was often an artifact of high search costs and limited information, and companies frequently profited from customer ignorance.  Today, customers are in control as never before -- and in a world of near-perfect information, there is less and less room for mediocre products and services.”
  • Strategy cycles are shrinking.  Thanks to plentiful capital, the power of outsourcing, and the global reach of the Web, it's possible to ramp up a new business faster than ever before.  But the more rapidly a business grows, the sooner it fulfills the promise of its original business model, peaks, and enters its dotage.  Today, the parabola of success is often a short, sharp spike.”
  • “Plummeting communication costs and globalization are opening up industries to a horde of new ultra-low-cost competitors.  These new entrants are eager to exploit the legacy costs of the old guard.  While some veterans will join the 'race to the bottom' and move their core activities to the world's lowest-cost locations, many others will find it difficult to reconfigure their global operations.  As Indian companies suck in service jobs and China steadily expands its share of global manufacturing, companies everywhere will struggle to maintain their margins.”
Strategically Adaptive and Operationally Efficient

So how do you respond to the challenges.   Hamel says it takes becoming strategically adaptable and operationally efficient.   What a powerful combo.

Via The Future of Management:

“These new realities call for new organizational and managerial capabilities.  To thrive in an increasingly disruptive world, companies must become as strategically adaptable as they are operationally efficient.  To safeguard their margins, they must become gushers of rule-breaking innovation.  And if they're going to out-invent and outthink a growing mob of upstarts, they must learn how to inspire their employees to give the very best of themselves every day.  These are the challenges that must be addressed by 21st-century management innovations.”

There are plenty of challenges.  It’s time to get your greatness on. 

If there ever was a chance to put to the test what you’re capable of, now is the time.

No matter what, as long as you live and learn, you’ll grow from the process.

You Might Also Like

Simplicity is the Ultimate Enabler

The New Competitive Landscape

Who’s Managing Your Company

Categories: Architecture, Programming

Troubleshooting haproxy 502 errors related to malformed/large HTTP headers

Agile Testing - Grig Gheorghiu - Wed, 07/23/2014 - 00:02
We had a situation recently where our web application started to behave strangely. First nginx (which sits in front of the application) started to error out with messages of this type:

upstream sent too big header while reading response header from upstream

A quick Google search revealed that a fix for this is to bump up proxy_buffer_size in nginx.conf, for both http and https traffic, along these lines:

proxy_buffer_size   256k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;

Now nginx was happy when hit directly. However, haproxy was still erroring out with a 502 'bad gateway' return code, followed by PH. Here is a snippet from the haproxy log file:

Jul 22 21:27:13 127.0.0.1 haproxy[14317]: 172.16.38.57:53408 [22/Jul/2014:21:27:12.776] www-frontend www-backend/www2:80 1/0/1/-1/898 502 8396 - - PH-- 0/0/0/0/0 0/0 "GET /someurl HTTP/1.1"

Another Google search revealed that PH means that haproxy rejected the header from the backend because it was malformed.

At this point, an investigation into the web app did discover a loop in the code that kept adding elements to a cookie included in the response header.

Anyway, I leave this here in the hope that somebody will stumble on it and benefit from it.

Sponsored Post: Apple, Asana, FoundationDB, Cie Games, BattleCry, Surge, Dreambox, Chartbeat, Monitis, Netflix, Salesforce, Cloudant, CopperEgg, Logentries, Gengo, Couchbase, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngin

Who's Hiring?
  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
    • Software Developer in Test. The iOS Systems team is looking for a Quality Assurance engineer. In this role you will be expected to work hand-in-hand with the software engineering team to find and diagnose software defects. The ideal candidate will also seek out ways to further automate all aspects of our existing process. This is a highly technical role and requires in-depth knowledge of both white-box and black-box testing methodologies. Please apply here
    • Senior Software Engineer -iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of a fast-paced environment make your heart leap? Do you want your technical abilities to be challenged every day, and for your work to make a difference in the lives of millions of people? If so, the iOS Systems Carrier Services team is looking for a talented software engineer who is not afraid to share knowledge, think outside the box, and question assumptions. Please apply here.
    • Software Engineering Manager, IS&T WWDR Dev Systems. The WWDR development team is seeking a hands-on engineering manager with a passion for building large-scale, high-performance applications. The successful candidate will be collaborating with Worldwide Developer Relations (WWDR) and various engineering teams throughout Apple. You will lead a team of talented engineers to define and build large-scale web services and applications. Please apply here.
    • C++ Senior Developer and Architect- Maps. The Maps Team is looking for a senior developer and architect to support and grow some of the core backend services that support Apple Map's Front End Services. Ideal candidate would have experience with system architecture, as well as the design, implementation, and testing of individual components but also be comfortable with multiple scripting languages. Please apply here.

  • Cie Games, small indie developer and publisher in LA, is looking for rock star Senior Game Java programmers to join our team! We need devs with extensive experience building scalable server-side code for games or commercial-quality applications that are rich in functionality. We offer competitive comp, great benefits, interesting projects, and exceptional growth opportunities. Check us out at http://www.ciegames.com/careers.

  • BattleCry, the newest ZeniMax studio in Austin, is seeking a qualified Front End Web Engineer to help create and maintain our web presence for AAA online games. This includes the game accounts web site, enhancing the studio website, our web and mobile- based storefront, and front end for support tools. http://jobs.zenimax.com/requisitions/view/540

  • FoundationDB is seeking outstanding developers to join our growing team and help us build the next generation of transactional database technology. You will work with a team of exceptional engineers with backgrounds from top CS programs and successful startups. We don’t just write software. We build our own simulations, test tools, and even languages to write better software. We are well-funded, offer competitive salaries and option grants. Interested? You can learn more here.

  • Asana. As an infrastructure engineer you will be designing software to process, query, search, analyze, and store data for applications that are continually growing in scale. You will work with a world-class team of engineers on deploying and operating existing systems, and building new ones for problems that are unique to our problem space. Please apply here.

  • Operations Engineer - AWS Cloud. Want to grow and extend a cutting-edge cloud deployment? Take charge of an innovative 24x7 web service infrastructure on the AWS Cloud? Join DreamBox Learning’s creative team of engineers, designers, and educators. Help us radically change education in an environment that values collaboration, innovation, integrity and fun. Please apply here. http://www.dreambox.com/careers

  • Chartbeat measures and monetizes attention on the web. Our traffic numbers are growing, and so is our list of product and feature ideas. That means we need you, and all your unparalleled backend engineer knowledge to help up us scale, extend, and evolve our infrastructure to handle it all. If you've these chops: www.chartbeat.com/jobs/be, come join the team!

  • The Salesforce.com Core Application Performance team is seeking talented and experienced software engineers to focus on system reliability and performance, developing solutions for our multi-tenant, on-demand cloud computing system. Ideal candidate is an experienced Java developer, likes solving real-world performance and scalability challenges and building new monitoring and analysis solutions to make our site more reliable, scalable and responsive. Please apply here.

  • Sr. Software Engineer - Distributed Systems. Membership platform is at the heart of Netflix product, supporting functions like customer identity, personalized profiles, experimentation, and more. Are you someone who loves to dig into data structure optimization, parallel execution, smart throttling and graceful degradation, SYN and accept queue configuration, and the like? Is the availability vs consistency tradeoff in a distributed system too obvious to you? Do you have an opinion about asynchronous execution and distributed co-ordination? Come join us

  • Human Translation Platform Gengo Seeks Sr. DevOps Engineer. Build an infrastructure capable of handling billions of translation jobs, worked on by tens of thousands of qualified translators. If you love playing with Amazon’s AWS, understand the challenges behind release-engineering, and get a kick out of analyzing log data for performance bottlenecks, please apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • OmniTI has a reputation for scalable web applications and architectures, but we still lean on our friends and peers to see how things can be done better. Surge started as the brainchild of our employees wanting to bring the best and brightest in Web Operations to our own backyard. Now in its fifth year, Surge has become the conference on scalability and performance. Early Bird rate in effect until 7/24!
Cool Products and Services
  • A third party vendor benchmarked the performance of three databases: Couchbase Server, MongoDB and DataStax Enterprise. The databases were benchmarked with two different workloads (read-intensive / balanced) via YCSB on dedicated servers. Read.

  • Now track your log activities with Log Monitor and be on the safe side! Monitor any type of log file and proactively define potential issues that could hurt your business' performance. Detect your log changes for: Error messages, Server connection failures, DNS errors, Potential malicious activity, and much more. Improve your systems and behaviour with Log Monitor.

  • The NoSQL "Family Tree" from Cloudant explains the NoSQL product landscape using an infographic. The highlights: NoSQL arose from "Big Data" (before it was called "Big Data"); NoSQL is not "One Size Fits All"; Vendor-driven versus Community-driven NoSQL.  Create a free Cloudant account and start the NoSQL goodness

  • Finally, log management and analytics can be easy, accessible across your team, and provide deep insights into data that matters across the business - from development, to operations, to business analytics. Create your free Logentries account here.

  • CopperEgg. Simple, Affordable Cloud Monitoring. CopperEgg gives you instant visibility into all of your cloud-hosted servers and applications. Cloud monitoring has never been so easy: lightweight, elastic monitoring; root cause analysis; data visualization; smart alerts. Get Started Now.

  • Whitepaper Clarifies ACID Support in Aerospike. In our latest whitepaper, author and Aerospike VP of Engineering & Operations, Srini Srinivasan, defines ACID support in Aerospike, and explains how Aerospike maintains high consistency by using techniques to reduce the possibility of partitions.  Read the whitepaper: http://www.aerospike.com/docs/architecture/assets/AerospikeACIDSupport.pdf.

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Who's Managing Your Company?

One of the best books I’m reading lately is The Future of Management, by Gary Hamel.

It’s all about how management innovation is the best competitive advantage, whether you look through the history of great businesses or the history of great militaries.  Hamel makes a great case that strategic innovation, product or service innovation, and operational innovation are fleeting advantages, but management innovation leads to competitive advantage for the long haul.

In The Future of Management, Hamel poses a powerful question …

“Who is managing your company?”

Via The Future of Management:

“Who's managing your company?  You might be tempted to answer, 'the CEO,' or 'the executive team,' or 'all of us in middle management.'  And you'd be right, but that wouldn't be the whole truth.  To a large extent, your company is being managed right now by a small coterie of long-departed theorists and practitioners who invented the rules and conventions of 'modern' management back in the early years of the 20th century.  They are the poltergeists who inhabit the musty machinery of management.  It is their edicts, echoing across the decades, that invisibly shape the way your company allocates resources, sets budgets, distributes power, rewards people, and makes decisions.”

That’s why it’s easy for CEOs to hop around companies …

Via The Future of Management:

“So pervasive is the influence of these patriarchs that the technology of management varies only slightly from firm to firm.  Most companies have a roughly similar management hierarchy (a cascade of EVPs, SVPs, and VPs).  They have analogous control systems, HR practices, and planning rituals, and rely on comparable reporting structures and review systems.  That's why it's so easy for a CEO to jump from one company to another -- the levers and dials of management are more or less the same in every corporate cockpit.”

What really struck me here is how much management approach has been handed down through the ages, and accepted as status quo.

It’s some great good for thought, especially given that management innovation is THE most powerful form of competitive advantage from an innovation standpoint (which Hamel really builds a strong case here throughout the entirety of the book.)

You Might Also Like

The New Competitive Landscape

Principles and Values Define a Culture

The Enterprise of the Future

Cognizant on The Next Generation Enterprise

Satya Nadella on the Future is Software

Categories: Architecture, Programming

StackOverflow Update: 560M Pageviews a Month, 25 Servers, and It's All About Performance

The folks at Stack Overflow remain incredibly open about what they are doing and why. So it’s time for another update. What has Stack Overflow been up to?

The network of sites that make up StackExchange, which includes StackOverflow, is now ranked 54th for traffic in the world; they have 110 sites and are growing at a rate of 3 or 4 a month; 4 million users; 40 million answers; and 560 million pageviews a month.

This is with just 25 servers. For everything. That’s high availability, load balancing, caching, databases, searching, and utility functions. All with a relative handful of employees. Now that’s quality engineering.

This update is based on The architecture of StackOverflow (video) by Marco Cecconi and What it takes to run Stack Overflow (post) by Nick Craver. In addition, I’ve merged in comments from various sources. No doubt some of the details are out of date as I meant to write this article long ago, but it should still be representative. 

Stack Overflow still uses Microsoft products. Microsoft infrastructure works and is cheap enough, so there’s no compelling reason to change. Yet SO is pragmatic. They use Linux where it makes sense. There’s no purity push to make everything Linux or keep everything Microsoft. That wouldn’t be efficient. 

Stack Overflow still uses a scale-up strategy. No clouds in site. With their SQL Servers loaded with 384 GB of RAM and 2TB of SSD, AWS would cost a fortune. The cloud would also slow them down, making it harder to optimize and troubleshoot system issues. Plus, SO doesn’t need a horizontal scaling strategy. Large peak loads, where scaling out makes sense, hasn’t  been a problem because they’ve been quite successful at sizing their system correctly.

So it appears Jeff Atwood’s quote: "Hardware is Cheap, Programmers are Expensive", still seems to be living lore at the company.

Marco Ceccon in his talk says when talking about architecture you need to answer this question first: what kind of problem is being solved?

First the easy part. What does StackExchange do? It takes topics, creates communities around them, and creates awesome question and answer sites. 

The second part relates to scale. As we’ll see next StackExchange is growing quite fast and handles a lot of traffic. How does it do that? Let’s take a look and see….

Stats
Categories: Architecture