Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

How I Explained My Job to My Grandmother

Well, she wasn’t my grandmother, but you get the idea.

I was trying to explain to somebody that’s in a very different job, what my job is all about.

Here’s what I said …

As far as my day job, I do complex, complicated things. 

I'm in the business of business transformation

I help large Enterprises get ahead in the world through technology and innovation.

I help Enterprises change their capabilities -- their business capabilities, technology capabilities, and people capabilities. 

It’s all about capabilities.

This involves figuring out their current state, their desired future state, the gaps between, the ROI of addressing the gaps, and then a Roadmap for making it happen.  

The interesting thing I've learned though is how much business transformation applies to personal transformation

It's all about figuring out your unique service and contribution to the world -- your unique value -- and then optimizing your strengths to realize your potential and do what you do best in a way that's valued -- where you can both generate value, as well as capture the value -- and lift the world to a better place.

Interestingly, she said she got it, it made sense, and it sounds inspiring.

What a relief.

Categories: Architecture, Programming

Introduction to Agile Presentation

I gave an Introduction to Agile talk recently:

Introduction to Agile Presentation (Slideshow)

I kept it focused on three simple things:

  1. What is Agile and the Agile Mindset (the Values and Principles)
  2. A rapid tour of the big 3 (Extreme Programming, Scrum, and Lean)
  3. Build a shared vocabulary and simple mental models so teams could hit the ground running and work more effectively with each other.

The big take away that I wanted the audience to have was that it’s a journey, but a very powerful one.

It’s a very healthy way to create an organization that embraces agility, empowers people, and ship stuff that customers care about.

In fact, the most powerful aspect of going Agile is that you create a learning organization.

The system and ecosystem you are in can quickly improve if you simply embrace change and focus on learning as a way of driving both continues improvement as well as growing capability.

So many things get a lot better over time, if they get a little better every day.

This was actually my first real talk on Agile and Agile development.  I’ve done lots of talks on Getting Results the Agile Way, and lots of other topics from security to performance to application architecture to team development and the Cloud.  But this was the first time a group asked me to share what I learned from Agile development in patterns & practices.

It was actually fun.

As part of the talk, I shared some of my favorite take aways and insights from the Agile World.

I’ll be sure to share some of these insights in future posts.

For now, if there is one thing to take away, it’s a reminder from David Anderson (Agile Management):

“Don’t do Agile.  Embrace agility.”

Way to be.

I shared my slides on SlideShare at Introduction to Agile Presentation (Slides) to help you learn the language, draw the visuals, and spread the word.

I’ll try to share more of my slides in the future, now that SlideShare seems to be a bit more robust.

You Might Also Like

Don’t Push Agile, Pull It

Extreme Programing at a Glance (Visual)

Scrum at a Glance (Visual)

Waterfall to Agile

What is Agile?

Categories: Architecture, Programming

Deploying a Node.js app to Docker on CoreOS using Deis

Xebia Blog - Wed, 06/18/2014 - 17:00

The world of on-premise private PaaSes is changing rapidly. A few years ago, we were building on on-premise private PaaSes based upon the existing infrastructure and using Puppet as an automation tool to quickly provision new application servers.  We provided a self-service portal where development teams could get any type of server in any type of environment running within minutes.  We created a virtual server for each application to keep it manageable, which of course is quite resource intensive.

Since June 9th, Docker has been declared production ready, so this opens  the option of provisioning light weight containers to the teams instead of full virtual machine. This will increase the speed of provisioning even further while reducing the cost of creating a platform and minimising resource consumption.

To illustrate how easy life is becoming, we are going to deploy an original CloudFoundry node.js application to Docker on a CoreOS cluster. This hands-on experiment is based on MacOS, Vagrant and VirtualBox.

Step 1. Installing  etcdctl and fleetctl

Before you start, you need to install etcdctl and fleetctl on your host.  etcd is a nifty distributed key-value store while fleet manages the deployment of  (Docker) services to a CoreOS cluster.
$ brew install go etcdctl
$ git clone https://github.com/coreos/fleet.git
$ cd fleet && ./build && mv bin/fleetctl /usr/local/bin

 

Step 2. Install the Deis Command Line Client

To control the PaaS you need to install the Deis command line client:

$ brew install python
$ sudo pip install deis

Step 3. Build the platform

Deis  provides all the facilities for building, deployment and managing applications.

$ git clone https://github.com/deis/deis.git
$ cd deis
$ vagrant up

$ ssh-add ~/.vagrant.d/insecure_private_key
$ export DOCKER_HOST=tcp://172.17.8.100:4243
$ export FLEETCTL_TUNNEL=172.17.8.100
$ make pull

Step 4. Start the platform

Now all is set to start the platform:

$ make run

After this run has completed, you can see that the 7 components in the Deis Architecture have been started using the list-units command: The builder, the cache, the controller, the database, the logger, the registry and the router. This architecture looks quite similar to the architecture of CloudFoundry.

$ fleetctl list-units

UNIT STATE LOAD ACTIVE SUB DESC MACHINE deis-builder.service launched loaded active running deis-builder 79874bde.../172.17.8.100 deis-cache.service launched loaded active running deis-cache 79874bde.../172.17.8.100 deis-controller.service launched loaded active running deis-controller 79874bde.../172.17.8.100 deis-database.service launched loaded active running deis-database 79874bde.../172.17.8.100 deis-logger.service launched loaded active running deis-logger 79874bde.../172.17.8.100 deis-registry.service launched loaded active running deis-registry 79874bde.../172.17.8.100 deis-router.1.service launched loaded active running deis-router 79874bde.../172.17.8.100

Alternatively, you can inspect the result by looking inside the virtual machine:

$ vagrant ssh -c "docker ps"

Now we have our platform running, we can start using it!

Step 5. Register a new user to Deis and add the public ssh key

$ deis register 172.17.8.100:8000 \
     --username=mark \
     --password=goddesses \
     --email=mark.van.holsteijn@..com
$ deis keys:add ~/.ssh/id_rsa.pub

Step 6. Create a Cluster

Create a application cluster under the domain 'dev.172.17.8.100.xip.io'.  The --hosts specifies all hosts in the cluster: the only available host  at this moment in the cluster is 172.17.8.100.

$ deis clusters:create dev  dev.172.17.8.100.xip.io \
        --hosts=172.17.8.100 \
        --auth=~/.vagrant.d/insecure_private_key

Step 7. Get the app

We created a simple but effective  node.js application that show you what happens when you scale or push a new version of the application.

$ git clone git@github.com:mvanholsteijn/sample_nodejs_cf.git
$ deis apps:create appmon --cluster=dev
$ deis config:set RELEASE=deis-v1
$ git push deis master

Step 8. open your application

Voila! Your application is running. Now click on start monitoring.

$ deis apps:open 

you should see something like this:

app-mon-1

Step 9. scaling your application

To see scaling in action,  type the following command:

$ deis ps:scale web=4

It will start 3 new containers which will show up in the list.

app-mon-4

 

Step 10. upgrading your application

Now make a change to the application, for instance change the message to 'Greetings from Deis release' and push your change:

$ git commit -a -m "Message changed"
$ git  push deis master

After a while you will see the following on your monitor!

app-mon-8

 

Step 11. Looking on CoreOS

You can use  fleetctl again to look at the new services that have been added to the platform!

 

$ fleetctl list-units

UNIT STATE LOAD ACTIVE SUB DESC MACHINE app-mon_v7.web.1-announce.service launched loaded active running app-mon_v7.web.1 announce 79874bde.../172.17.8.100 app-mon_v7.web.1-log.service launched loaded active running app-mon_v7.web.1 log 79874bde.../172.17.8.100 app-mon_v7.web.1.service launched loaded active running app-mon_v7.web.1 79874bde.../172.17.8.100 app-mon_v7.web.2-announce.service launched loaded active running app-mon_v7.web.2 announce 79874bde.../172.17.8.100 app-mon_v7.web.2-log.service launched loaded active running app-mon_v7.web.2 log 79874bde.../172.17.8.100 app-mon_v7.web.2.service launched loaded active running app-mon_v7.web.2 79874bde.../172.17.8.100 app-mon_v7.web.3-announce.service launched loaded active running app-mon_v7.web.3 announce 79874bde.../172.17.8.100 app-mon_v7.web.3-log.service launched loaded active running app-mon_v7.web.3 log 79874bde.../172.17.8.100 app-mon_v7.web.3.service launched loaded active running app-mon_v7.web.3 79874bde.../172.17.8.100 app-mon_v7.web.4-announce.service launched loaded active running app-mon_v7.web.4 announce 79874bde.../172.17.8.100 app-mon_v7.web.4-log.service launched loaded active running app-mon_v7.web.4 log 79874bde.../172.17.8.100 app-mon_v7.web.4.service launched loaded active running app-mon_v7.web.4 79874bde.../172.17.8.100 deis-builder.service launched loaded active running deis-builder 79874bde.../172.17.8.100 deis-cache.service launched loaded active running deis-cache 79874bde.../172.17.8.100 deis-controller.service launched loaded active running deis-controller 79874bde.../172.17.8.100 deis-database.service launched loaded active running deis-database 79874bde.../172.17.8.100 deis-logger.service launched loaded active running deis-logger 79874bde.../172.17.8.100 deis-registry.service launched loaded active running deis-registry 79874bde.../172.17.8.100 deis-router.1.service launched loaded active running deis-router 79874bde.../172.17.8.100

 

Conclusion

Deis is a very simple and easy to use way to create a PaaS using Docker and CoreOS. The node.js application we created, could be deployed using Deis without a single modification. We will be diving into Deis and CoreOS in more details in following posts!

One Change at a Time

Xebia Blog - Tue, 06/17/2014 - 08:22

One of the twelve Agile principles states "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behaviour accordingly" [1]. Many Agile teams have retrospectives every two weeks that result in concrete actions that are executed in the next time period (sprint). There are many types of tuning and adjustments that a team can do. Examples are actions that improve the flow of work, automation of tasks, team coorporation.

Is it good habit for retrospectives to focus on the same type of improvement or should the team alter the type of improvements that will be done? In this blog I will look into the effect of multiple consecutive actions that affect the flow of work.

The simulation is inspired by the GetKanban game [2].

An Experiment

Ideally one would have a set-up for an experiment in which two exactly equivalent teams are compared. One team that would perform consecutive actions to improve the flow of work. The other team would just make one adjustment to improve the flow, and focus subsequent improvements on other areas than flow. At the same time the flow is measured and verified after a certain period of time whether the first or the second team has achieved better results in terms of flow.

Such an experiment is in practise (very) difficult to perform. In this blog I will study the same experiment by making use of simulations.

Simulation

For the purpose of simulation I consider a team consisting of three specialists: 1 designer, 1 developer, and 1 tester. The team uses a kanban process to achieve flow. See the picture below for the begin situation.


foto

In the simulation, at the beginning of each working day it is determined how much work will be completed by the team. During the simulation the average cycle time is measured. The initial Work in Progress (WiP) limits are set to 3 for each column and is indicated by the red '3's in the picture.

The average amount of work done by the 'team' and the average effort of one work item are such that on average it takes one card about 5,5 days to complete.

At the end of each work day, cards are pulled into the next columns (if allowed by the WiP limits). The policy is to always pull in as much work as allowed so the columns are maximally filled. Furthermore, the backlog is assumed to always have enough user stories ready to be pulled into the 'design' column. This very much resembles developing a new product when the backlog is filled with more than enough stories.

The system starts with a clean board and all column empty. After letting the system run for 75 simulated work days, we will trigger a policy change. Particularly the WiP limit for the 'design' is increased from '3' to '5'. After this policy change the system runs for another 100 work days.

From the chart showing the average cycle time we will be able to study the effect of WiP limit changing adjustments.

Note:

The simulation assumes a simple uniform distribution for the amount of work done by the team and the effort assigned to a work item. I assume this is OK for the purpose of this blog. A consequence of this, is that the result probably can't be scaled. For instance, the situation in which a column in the picture above is a single scrum team is not applicable since a more complex probability distribution should be used instead of the uniform distribution.

Results

The picture below shows the result of running the experiment.

 

retro_length

 

After the start it takes the system little over 40 work days to reach the stable state of an average cycle time of about 24(*) days. This is the cycle time one would expect. Remember, the 'ready' column has a size of '3' and the other columns get work done. So, one would expect a cycle time of around 4 times 5,5 which equals 22 days which is close to 24.

At day 75 the WiP limit is changed. As can be inferred from the picture, the cycle time starts to rise only at day 100 (takes about 1 cycle time (24 days) to respond). The new stable state is reached at day 145 with an average cycle time of around 30(**) days. It takes 70 days(!) to reach the new equilibrium.

The chart shows the following interesting features:

  1. It takes roughly 2 times the (new) average cycle time to reach the equilibrium state,
  2. The response time (when one begins to notice an effect of the policy change) is about the length of the average cycle time.

(*) One can calculate (using transition matrices) the theoretical average cycle time for this system to be 24 days.

(**) Similar, the theoretical average cycle time of the system after policy change is 31 days.

 

Conclusion

In this blog we have seen that when a team makes adjustments that affect the flow, the system needs time to get to its new stable state. Until this state has been reached any new tuning of the flow is questionable. Simulations show that the time it takes to reach the new stable state is about 2 times the average cycle time.

For scrum teams that have 2-week sprints, the system may need about 2 months before new tuning of flow is effective. Meanwhile, the team can very well focus on other improvements, e.g. retrospectives that focus on the team aspect or collaboration with the team's environment.

Moreover, don't expect to see any changes in measurements of e.g. cycle time within the time period of the average cycle time after making a flow affecting change.

To summarise, after making flow affecting changes (e.g. increasing or decreasing WiP limits):

  • Let the system run for at least the duration of the average cycle time so it has time to respond to the change,
  • After it responds, notice the effect of the change,
  • If the effect is positive, let the system run for another duration of the average cycle time, to get to the new stable state,
  • If the effect is negative, do something else, e.g. go back to the old state, and remember that the system needs to respond to this as well!
References

[1] Agile manifesto, http://agilemanifesto.org/principles.html

[2] GetKanban, http://getkanban.com

Migrating to XtraDB Cluster in EC2 at PagerDuty

This is a guest post by Doug Barth, a software generalist who has currently found himself doing operations work at PagerDuty. Prior to joining PagerDuty, he worked for Signal in Chicago and Orbitz, an online travel company.

A little over six months ago, PagerDuty switched its production MySQL database to XtraDB Cluster running inside EC2. Here's the story of why and how we made the change.

How the Database Looked Before

PagerDuty's MySQL database was a fairly typical deployment. We had:

  • A pair of Percona Servers writing data to a DRBD-backed volume.

  • EBS disks for both the primary and secondary servers backing the DRBD volume.

  • Two synchronous copies of the production database. (In the event of a failure of the primary, we wanted to be able to quickly switch to the secondary server without losing any data.)

  • A number of asynchronous replication slaves for disaster recovery, backups and accidental modification recovery.

Problems With the Old Setup
Categories: Architecture

Combining Salt with Docker

Xebia Blog - Sat, 06/14/2014 - 10:59

You could use Salt to build and run Docker containers but that is not how I use it here. This blogpost is about Docker containers that run Salt minions, which is just an experiment. The use case? Suppose you have several containers that run a particular piece of middleware, and this piece of middleware needs a security update, i.e. an OpenSSL hotfix. It is necessary to perform the update immediately.

 

The Dockerfile

In order to build a container you have to write down the container description in a file called Dockerfile. Here is the Dockerfile:

#-------
# Standard heading stuff

FROM centos
MAINTAINER No Reply noreply@xebia.com

# Do Salt install stuff and squeeze in a master.conf snippet that tells the minion
# to contact the master specified.

RUN rpm -Uvh http://ftp.linux.ncsu.edu/pub/epel/6/i386/epel-release-6-8.noarch.rpm
RUN yum install -y salt-minion --enablerepo=epel-testing
RUN [ ! -d /etc/salt/minion.d ] && mkdir /etc/salt/minion.d
ADD ./master.conf /etc/salt/minion.d/master.conf

# Run the Salt Minion and do not detach from the terminal.
# This is important because the Docker container will exit whenever
# the CMD process exits.

CMD /usr/bin/salt-minion
#-------

 

Build the image

Time to run the Dockerfile through docker. The command is:

$ docker build --rm=true -t salt-minion .

provided that you run this command in the directory where file Dockerfile and master.conf resides. Docker creates an image with tag ‘salt-minion’ and throws away all intermediate images after a successful build.

 

Run a container

The command is:

$ docker run -d salt-minion

and Docker returns:

aab154310ba6452ba2c686d15b1e3ca5fd85124d38c7935f1200d33b3a3e7ced

The Salt minion on the container is started and searches for a Salt master to connect to, defined by the configuration setting “master” in file /etc/salt/minion.d/master.conf. You might want to run the Salt master in “auto_accept” mode so that minion keys are accepted automatically. Docker assigns a container id to the running container. That is the magic key that docker reports as a result of the run command.

The following command shows the running container:

$ docker ps
CONTAINER ID        IMAGE                COMMAND                CREATED             STATUS              NAMES
273a6b77a8fa        salt-minion:latest   /bin/sh -c /etc/rc.l   3 seconds ago       Up 3 seconds        distracted_lumiere

 

Apply the hot fix
There you are: the Salt minion is controlled by your Salt master. Provided that you have a state module that contains the OpenSSL hot fix, you can now easily update all docker nodes to include the hotfix:

salt \* state.sls openssl-hotfix

That is all there is to it.

Setting up the hostname in Ubuntu

Agile Testing - Grig Gheorghiu - Fri, 06/13/2014 - 23:04
Most people recommend setting up the hostname on a Linux box so that:

1) running 'hostname' returns the short name (i.e. myhost)
2) running 'hostname -f' returns the FQDN (i.e. myhost.prod.example.com)
3) running 'hostname -d' returns the domain name (i.e prod.example.com)

After experimenting a bit and also finding this helpful Server Fault post, here's what we did to achieve this (we did it via Chef recipes, but it amounts to the same thing):

  • make sure we have the short name in /etc/hostname:
myhost

(also run 'hostname myhost' at the command line)
  • make sure we have the FQDN as the first entry associated with the IP of the server in /etc/hosts:
10.0.1.10 myhost.prod.example.com myhost myhost.prod
  • make sure we have the domain name set up as the search domain in /etc/resolv.conf:
search prod.example.com

Reboot the box when you're done to make sure all of this survives reboots.


Simplicity is the Ultimate Enabler

“Everything should be made as simple as possible, but not simpler.” – Albert Einstein

Simplicity is among the ultimate of pursuits.  It’s one of your most efficient and effective tools in your toolbox.  I used simplicity as the basis for my personal results system, Agile Results, and it’s served me well for more than a decade.

And yet, simplicity still isn’t treated as a first-class citizen.

It’s almost always considered as an afterthought.  And, by then, it’s too little, too late.

In the book, Simple Architectures for Complex Enterprises (Developer Best Practices), Roger Sessions shares his insights on how simplicity is the ultimate enabler to solving the myriad of problems that complexity creates.

Complex Problems Do Not Require Complex Solutions

Simplicity is the only thing that actually works.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“So yes, the problems are complex.  But complex problems do not ipso facto require complex solutions.  Au contraire!  The basic premise of this book is that simple solutions are the only solutions to complex problems that work.  The complex solutions are simply too complex.”

Simplicity is the Antidote to Complexity

It sounds obvious but it’s true.  You can’t solve a problem with the same complexity that got you there in the first place.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The antidote to complexity is simplicity.  Replace complexity with simplicity and the battle is three-quarters over.  Of course, replacing complexity with simplicity is not necessarily simple.” 

Focus on Simplicity as a Core Value

If you want to achieve simplicity, you first have to explicitly focus on it as a core value.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The first thing you need to do to achieve simplicity is focus on simplicity as a core value.  We all discuss the importance of agility, security, performance, and reliability of IT systems as if they are the most important of all requirements.  We need to hold simplicity to as high a standard as we hold these other features.  We need to understand what makes architectures simple with as much critical reasoning as we use to understand what makes architectures secure, fast, or reliable.  In fact, I argue that simplicity is not merely the equal of these other characteristics; it is superior to all of them.  It is, in many ways, the ultimate enabler.”

A Security Example

Complex systems work against security.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Take security for example.  Simple systems that lack security can be made secure.  Complex systems that appear to be secure usually aren't.  And complex systems that aren't secure are virtually impossible to make either simple or secure.”

An Agility Example

Complexity works against agility, and agility is the key to lasting solutions.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Consider agility.  Simple systems, with their well-defined and minimal interactions, can be put together in new ways that were never considered when these systems were first created.  Complex systems can never used in an agile wayThey are simply too complex.  And, of course, retrospectively making them simple is almost impossible.”

Nobody Ever Considers Simplicity as a Critical Feature

And that’s the problem.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“Yet, despite the importance of simplicity as a core system requirement, simplicity is almost never considered in architectural planning, development, or reviews.  I recently finished a number of speaking engagements.  I spoke to more than 100 enterprise architects, CIOs, and CTOs spanning many organizations and countries.  In each presentation, I asked if anybody in the audience had ever considered simplicity as a critical architectural feature for any projects on which they had participated. Not one person had. Ever.”

The Quest for Simplicity is Never Over

Simplicity is a quest.  And the quest is never over.  Simplicity is a ongoing pursuit and it’s a dynamic one.  It’s not a one time event, and it’s not static.

Via Simple Architectures for Complex Enterprises (Developer Best Practices):

“The quest for simplicity is never over.  Even systems that are designed from the beginning with simplicity in mind (rare systems, indeed!) will find themselves under a never-ending attack. A quick tweak for performance here, a quick tweak for interoperability there, and before you know it, a system that was beautifully simple two years ago has deteriorated into a mass of incomprehensibility.”

Simplicity is your ultimate sword for hacking your way through complexity … in work … in life … in systems … and ecosystems.

Wield it wisely.

You Might Also Like

10 Ways to Make Information More Useful

Reduce Complexity, Cost, and Time

Simple Enterprise Strategy

Categories: Architecture, Programming

Sponsored Post: Apple, Netflix, Salesforce, Blizzard Entertainment, Cloudant, CopperEgg, Logentries, Wargaming.net, PagerDuty, HelloSign, CrowdStrike, Gengo, ScaleOut Software, Couchbase, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics

Who's Hiring?

  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here.
    • Enterprise Software Engineer. Apple's Emerging Technology Services group provides a Java based SOA platform for various applications to interact with each other. The platform is designed to handle millions of messages a day with very low latency. We have an immediate opening for a talented Software Engineer in a highly visible team who is passionate about exploring emerging technologies to create elegant scalable solutions. Please apply here
    • Mobile Services Software Engineer. The Emerging Technologies/Mobile Services team is looking for a proactive and hardworking software engineer to join our team. The team is responsible for a variety of high quality and high performing mobile services and applications for internal use. Please apply here
    • Sr. Software Engineer-iOS Systems. Do you love building highly scalable, distributed web applications? Does the idea of performance tuning Java applications make your heart leap? If so, iOS Systems is looking for a highly motivated, detail-oriented, energetic individual with excellent written and oral skills who is not afraid to think outside the box and question assumptions. Please apply here
    • Senior Software Engineering Manager. As a Senior Software Engineering Manager on our team, you will be managing teams of very dedicated and talented engineering team. You will be responsible for managing the development of mobile point of sale system on iPod touch hardware. Please apply here.
    • Sr Software Engineer - Messaging Services. An exciting opportunity for a Software Engineer to join Apple's Messaging Services team. We build the cloud systems that power some of the busiest applications in the world, including iMessage, FaceTime and Apple Push Notifications. We handle hundreds of millions of active users using some of the most desirable devices on the planet and several Billion iMesssages/day, 40 billion push notifications/day, 16+ trillion push notifications sent to date. Please apply here.
    • Senior Software Engineer. Join Apple's Internet Applications Team, within the Information Systems and Technology group, as a Senior Software Engineer. Be involved in challenging and fast paced projects supporting Apple's business by delivering Java based IS Systems. Please apply here.
    • Sr Software Engineer. Join Apple's Internet Applications Team, within the Information Systems and Technology group, as a Senior Software Engineer. Be involved in challenging and fast paced projects supporting Apple's business by delivering Java based IS Systems. Please apply here.
    • Senior Payment Engineer. As a Software Engineer on our team, you will be responsible for working with cross-functional teams and developing Java server-based solutions to address business and technological needs. You will be helping design and build next generation retail solutions. You will be reviewing design and code developed by others on the team.You will build services and integrate with both internal as well as external services in a SOA environment. You will design and develop frameworks to be used by a large community of developers within the organization. Please apply here.

  • The Salesforce.com Core Application Performance team is seeking talented and experienced software engineers to focus on system reliability and performance, developing solutions for our multi-tenant, on-demand cloud computing system. Ideal candidate is an experienced Java developer, likes solving real-world performance and scalability challenges and building new monitoring and analysis solutions to make our site more reliable, scalable and responsive. Please apply here.

  • Sr. Software Engineer - Distributed Systems. Membership platform is at the heart of Netflix product, supporting functions like customer identity, personalized profiles, experimentation, and more. Are you someone who loves to dig into data structure optimization, parallel execution, smart throttling and graceful degradation, SYN and accept queue configuration, and the like? Is the availability vs consistency tradeoff in a distributed system too obvious to you? Do you have an opinion about asynchronous execution and distributed co-ordination? Come join us

  • Java Software Engineers of all levels, your time is now. Blizzard Entertainment is leveling up its Battle.net team, and we want to hear from experienced and enthusiastic engineers who want to join them on their quest to produce the most epic customer-facing site experiences possible. As a Battle.net engineer, you'll be responsible for creating new (and improving existing) applications in a high-load, high-availability environment. Please apply here.

  • Engine Programmer - C/C++. Wargaming|BigWorld is seeking Engine Programmers to join our team in Sydney, Australia. We offer a relocation package, Australian working visa & great salary + bonus. Your primary responsibility will be to work on our PC engine. Please apply here

  • Senior Engineer wanted for large scale, security oriented distributed systems application that offers career growth and independent work environment. Use your talents for good instead of getting people to click ads at CrowdStrike. Please apply here.

  • Ops Engineer - Are you passionate about scaling and automating cloud-based websites? Love Puppet and deployment scripts? Want to take advantage of both your sys-admin and DevOps skills? Join HelloSign as our second Ops Engineer and help us scale as we grow! Apply at http://www.hellosign.com/info/jobs

  • Human Translation Platform Gengo Seeks Sr. DevOps Engineer. Build an infrastructure capable of handling billions of translation jobs, worked on by tens of thousands of qualified translators. If you love playing with Amazon’s AWS, understand the challenges behind release-engineering, and get a kick out of analyzing log data for performance bottlenecks, please apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events

  • The Biggest MongoDB Event Ever Is On. Will You Be There? Join us in New York City June 23-25 for MongoDB World! The conference lineup includes Amazon CTO Werner Vogels and Cloudera Co-Founder Mike Olson for keynote addresses.  You’ll walk away with everything you need to know to build and manage modern applications. Register before April 4 to take advantage of super early bird pricing.

  • Upcoming Webinar: Practical Guide to SQL - NoSQL Migration. Avoid common pitfalls of NoSQL deployment with the best practices in this May 8 webinar with Anton Yazovskiy of Thumbtack Technology. He will review key questions to ask before migration, and differences in data modeling and architectural approaches. Finally, he will walk you through a typical application based on RDBMS and will migrate it to NoSQL step by step. Register for the webinar.
Cool Products and Services
  • The NoSQL "Family Tree" from Cloudant explains the NoSQL product landscape using an infographic. The highlights: NoSQL arose from "Big Data" (before it was called "Big Data"); NoSQL is not "One Size Fits All"; Vendor-driven versus Community-driven NoSQL.  Create a free Cloudant account and start the NoSQL goodness

  • Finally, log management and analytics can be easy, accessible across your team, and provide deep insights into data that matters across the business - from development, to operations, to business analytics. Create your free Logentries account here.

  • CopperEgg. Simple, Affordable Cloud Monitoring. CopperEgg gives you instant visibility into all of your cloud-hosted servers and applications. Cloud monitoring has never been so easy: lightweight, elastic monitoring; root cause analysis; data visualization; smart alerts. Get Started Now.

  • PagerDuty helps operations and DevOps engineers resolve problems as quickly as possible. By aggregating errors from all your IT monitoring tools, and allowing easy on-call scheduling that ensures the right alerts reach the right people, PagerDuty increases uptime and reduces on-call burnout—so that you only wake up when you have to. Thousands of companies rely on PagerDuty, including Netflix, Etsy, Heroku, and Github.

  • Aerospike Releases Client SDK for Node.js 0.10.x. This client makes it easy to build applications in Node.js that need to store and retrieve data from a high-performance Aerospike cluster. This version exposes Key-Value Store functionality - which is the core of Aerospike's In-Memory NoSQL Database. Platforms supported: CentOS 6, RHEL 6, Debian 6, Debian7, Mac OS X, Ubuntu 12.04. Write your first app: https://github.com/aerospike/aerospike-client-nodejs.

  • consistent: to be, or not to be. That’s the question. Is data in MongoDB consistent? It depends. It’s a trade-off between consistency and performance. However, does performance have to be sacrificed to maintain consistency? more.

  • Do Continuous MapReduce on Live Data? ScaleOut Software's hServer was built to let you hold your daily business data in-memory, update it as it changes, and concurrently run continuous MapReduce tasks on it to analyze it in real-time. We call this "stateful" analysis. To learn more check out hServer.

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Is there a future for Map/Reduce?

8w9jj

Google’s Jeffrey Dean and Sanjay Ghemawat filed the patent request and published the map/reduce paper  10 year ago (2004). According to WikiPedia Doug Cutting and Mike Cafarella created Hadoop, with its own implementation of Map/Reduce,  one year later at Yahoo – both these implementations were done for the same purpose – batch indexing of the web.

Back than, the web began its “web 2.0″ transition, pages became more dynamic , people began to create more content – so an efficient way to reprocess and build the web index was needed and map/reduce was it. Web Indexing was a great fit for map/reduce since the initial processing of each source (web page) is completely independent from any other – i.e.  a very convenient map phase and you need  to combine the results to build the reverse index. That said, even the core google algorithm –  the famous pagerank is iterative (so less appropriate for map/reduce), not to mention that  as the internet got bigger and the updates became more and more frequent map/reduce wasn’t enough. Again Google (who seem to be consistently few years ahead of the industry) began coming up with alternatives like Google Percolator  or  Google Dremel (both papers were published in 2010, Percolator was introduced at that year, and dremel has been used in Google since 2006).

So now, it is 2014, and it is time for the rest of us to catch up with Google and get over Map/Reduce and  for multiple reasons:

  • end-users’ expectations (who hear “big data” but interpret that as  “fast data”)
  • iterative problems like graph algorithms which are inefficient as you need to load and reload the data each iteration
  • continuous ingestion of data (increments coming on as small batches or streams of events) – where joining to existing data can be expensive
  • real-time problems – both queries and processing

In my opinion, Map/Reduce is an idea whose time has come and gone – it won’t die in a day or a year, there is still a lot of working systems that use it and the alternatives are still maturing. I do think, however, that if you need to write or implement something new that would build on map/reduce – you should use other option or at the very least carefully consider them.

So how is this change going to happen ?  Luckily, Hadoop has recently adopted YARN (you can see my presentation on it here), which opens up the possibilities to go beyond map/reduce without changing everything … even though in effect,  a lot  will change. Note that some of the new options do have migration paths and also we still retain the  access to all that “big data” we have in Hadoopm as well as the extended reuse of some of the ecosystem.

The first type of effort to replace map/reduce is to actually subsume it by offering more  flexible batch. After all saying Map/reduce is not relevant, deosn’t mean that batch processing is not relevant. It does mean that there’s a need to more complex processes. There are two main candidates here  Tez and Spark where Tez offers a nice migration path as it is replacing map/reduce as the execution engine for both Pig and Hive and Spark has a compelling offer by combining Batch and Stream processing (more on this later) in a single engine.

The second type of effort or processing capability that will help kill map/reduce is MPP databases on Hadoop. Like the “flexible batch” approach mentioned above, this is replacing a functionality that map/reduce was used for – unleashing the data already processed and stored in Hadoop.  The idea here is twofold

  • To provide fast query capabilities* – by using specialized columnar data format and database engines deployed as daemons on the cluster
  • To provide rich query capabilities – by supporting more and more of the SQL standard and enriching it with analytics capabilities (e.g. via MADlib)

Efforts in this arena include Impala from Cloudera, Hawq from Pivotal (which is essentially greenplum over HDFS), startups like Hadapt or even Actian trying to leverage their ParAccel acquisition with the recently announced Actian Vector . Hive is somewhere in the middle relying on Tez on one hand and using vectorization and columnar format (Orc)  on the other

The Third type of processing that will help dethrone Map/Reduce is Stream processing. Unlike the two previous types of effort this is covering a ground the map/reduce can’t cover, even inefficiently. Stream processing is about  handling continuous flow of new data (e.g. events) and processing them  (enriching, aggregating, etc.)  them in seconds or less.  The two major contenders in the Hadoop arena seem to be Spark Streaming and Storm though, of course, there are several other commercial and open source platforms that handle this type of processing as well.

In summary – Map/Reduce is great. It has served us (as an industry) for a decade but it is now time to move on and bring the richer processing capabilities we have elsewhere to solve our big data problems as well.

Last note  – I focused on Hadoop in this post even thought there are several other platforms and tools around. I think that regardless if Hadoop is the best platform it is the one becoming the de-facto standard for big data (remember betamax vs VHS?)

One really, really last note – if you read up to here, and you are a developer living in Israel, and you happen to be looking for a job –  I am looking for another developer to join my Technology Research team @ Amdocs. If you’re interested drop me a note: arnon.rotemgaloz at amdocs dot com or via my twitter/linkedin profiles

*esp. in regard to analytical queries – operational SQL on hadoop with efforts like  Phoenix ,IBM’s BigSQL or Splice Machine are also happening but that’s another story

illustration idea found in  James Mickens’s talk in Monitorama 2014 –  (which is, by the way, a really funny presentation – go watch it) -ohh yeah… and pulp fiction :)

Categories: Architecture

An architecturally-evident coding style

Coding the Architecture - Simon Brown - Sun, 06/01/2014 - 12:51

Okay, this is the separate blog post that I referred to in Software architecture vs code. What exactly do we mean by an "architecturally-evident coding style"? I built a simple content aggregator for the local tech community here in Jersey called techtribes.je, which is basically made up of a web server, a couple of databases and a standalone Java application that is responsible for actually aggegrating the content displayed on the website. You can read a little more about the software architecture at techtribes.je - containers. The following diagram is a zoom-in of the standalone content updater application, showing how it's been decomposed.

techtribes.je content updater - component diagram

This diagram says that the content updater application is made up of a number of core components (which are shown on a separate diagram for brevity) and an additional four components - a scheduled content updater, a Twitter connector, a GitHub connector and a news feed connector. This diagram shows a really nice, simple architecture view of how my standalone content updater application has been decomposed into a small number of components. "Component" is a hugely overloaded term in the software development industry, but essentially all I'm referring to is a collection of related behaviour sitting behind a nice clean interface.

Back to the "architecturally-evident coding style" and the basic premise is that the code should reflect the architecture. In other words, if I look at the code, I should be able to clearly identify each of the components that I've shown on the diagram. Since the code for techtribes.je is open source and on GitHub, you can go and take a look for yourself as to whether this is the case. And it is ... there's a je.techtribes.component package that contains sub-packages for each of the components shown on the diagram. From a technical perspective, each of these are simply Spring Beans with a public interface and a package-protected implementation. That's it; the code reflects the architecture as illustrated on the diagram.

So what about those core components then? Well, here's a diagram showing those.

techtribes.je core components

Again, this diagram shows a nice simple decomposition of the core of my techtribes.je system into coarse-grained components. And again, browsing the source code will reveal the same one-to-one mapping between boxes on the diagram and packages in the code. This requires conscious effort to do but I like the simple and explicit nature of the relationship between the architecture and the code.

When architecture and code don't match

The interesting part of this story is that while I'd always viewed my system as a collection of "components", the code didn't actually look like that. To take an example, there's a tweet component on the core components diagram, which basically provides CRUD access to tweets in a MongoDB database. The diagram suggests that it's a single black box component, but my initial implementation was very different. The following diagram illustrates why.

techtribes.je tweet component

My initial implementation of the tweet component looked like the picture on the left - I'd taken a "package by layer" approach and broken my tweet component down into a separate service and data access object. This is your stereotypical layered architecture that many (most?) books and tutorials present as a way to build (e.g.) web applications. It's also pretty much how I've built most software in the past too and I'm sure you've seen the same, especially in systems that use a dependency injection framework where we create a bunch of things in layers and wire them all together. Layered architectures have a number of benefits but they aren't a silver bullet.

This is a great example of where the code doesn't quite reflect the architecture - the tweet component is a single box on an architecture diagram but implemented as a collection of classes across a layered architecture when you look at the code. Imagine having a large, complex codebase where the architecture diagrams tell a different story from the code. The easy way to fix this is to simply redraw the core components diagram to show that it's really a layered architecture made up of services collaborating with data access objects. The result is a much more complex diagram but it also feels like that diagram is starting to show too much detail.

The other option is to change the code to match my architectural vision. And that's what I did. I reorganised the code to be packaged by component rather than packaged by layer. In essence, I merged the services and data access objects together into a single package so that I was left with a public interface and a package protected implementation. Here's the tweet component on GitHub.

But what about...

Again, there's a clean simple mapping from the diagram into the code and the code cleanly reflects the architecture. It does raise a number of interesting questions though.

  • Why aren't you using a layered architecture?
  • Where did the TweetDao interface go?
  • How do you mock out your DAO implementation to do unit testing?
  • What happens if I want to call the DAO directly?
  • What happens if you want to change the way that you store tweets?
Layers are now an implementation detail

This is still a layered architecture, it's just that the layers are now a component implementation detail rather than being first-class architectural building blocks. And that's nice, because I can think about my components as being my architecturally significant structural elements and it's these building blocks that are defined in my dependency injection framework. Something I often see in layered architectures is code bypassing a services layer to directly access a DAO or repository. These sort of shortcuts are exactly why layered architectures often become corrupted and turn into big balls of mud. In my codebase, if any consumer wants access to tweets, they are forced to use the tweet component in its entirety because the DAO is an internal implementation detail. And because I have layers inside my component, I can still switch out my tweet data storage from MongoDB to something else. That change is still isolated.

Component testing vs unit testing

Ah, unit testing. Bundling up my tweet service and DAO into a single component makes the resulting tweet component harder to unit test because everything is package protected. Sure, it's not impossible to provide a mock implementation of the MongoDBTweetDao but I need to jump through some hoops. The other approach is to simply not do unit testing and instead test my tweet component through its public interface. DHH recently published a blog post called Test-induced design damage and I agree with the overall message; perhaps we are breaking up our systems unnecessarily just in order to unit test them. There's very little to be gained from unit testing the various sub-parts of my tweet component in isolation, so in this case I've opted to do automated component testing instead where I test the component as a black-box through its component interface. MongoDB is lightweight and fast, with the resulting component tests running acceptably quick for me, even on my ageing MacBook Air. I'm not saying that you should never unit test code in isolation, and indeed there are some situations where component testing isn't feasible. For example, if you're using asynchronous and/or third party services, you probably do want to ability to provide a mock implementation for unit testing. The point is that we shouldn't blindly create designs where everything can be mocked out and unit tested in isolation.

Food for thought

The purpose of this blog post was to provide some more detail around how to ensure that code reflects architecture and to illustrate an approach to do this. I like the structure imposed by forcing my codebase to reflect the architecture. It requires some discipline and thinking about how to neatly carve-up the responsibilities across the codebase, but I think the effort is rewarded. It's also a nice stepping stone towards micro-services. My techtribes.je system is constructed from a number of in-process components that I treat as my architectural building blocks. The thinking behind creating a micro-services architecture is essentially the same, albeit the components (services) are running out-of-process. This isn't a silver bullet by any means, but I hope it's provided some food for thought around designing software and structuring a codebase with an architecturally-evident coding style.

Categories: Architecture

Software architecture vs code

Coding the Architecture - Simon Brown - Thu, 05/29/2014 - 13:01

I presented two talks last week with the title "Software architecture vs code" - first as the opening keynote for the inaugural Software Design and Development conference and also the next day as a regular conference session at GOTO Chicago. Videos from both should be available at some point and the slides are available now. The talk itself seems to polarise people, with responses ranging from Without a doubt, Simon delivered one of the best keynotes I have seen. I got a lot from it, with plenty 'food for thought' moments. through to "hmmm, meh".

Separating software architecture from code

The basic premise of the talk is that the architecture and code of a software system never quite match up. The traditional way to communicate the architecture of a software system is with diagrams based upon a number of views ... a logical view, a functional view, a module view, a physical view, etc, etc. Philippe Kruchten's 4+1 model is an example often cited as a starting point for such approaches. I've followed these approaches in the past myself and, although I can get my head around them, I don't find them an optimal way to describe a software system. The "why?" has taken me a while to figure out, but the thing I dislike is the way in which you get an artificial separation between the architecture-related views (logical, module, functional, etc) and the code-related views (implementation, design, etc). I don't like treating the architecture and the code as two separate things, but this seems to be the starting point for many of the ways in which software systems are communicated/documented. If you want a good example of this, take a look at the first chapter of "Software Architecture in Practice" where it describes the relationship between modules, components, and component instances. It makes my head hurt.

The model-code gap

This difference between the architecture and code views is also exaggerated by what George Fairbanks calls the "model-code gap" in his book titled "Just Enough Software Architecture" (highly recommended reading, by the way). George basically says that your architecture models will include abstract concepts (e.g. components, services, modules, etc) but the code usually doesn't reflect this. This matches my own experience of helping people communicate their software systems ... people will usually draw components or services, but the actual implementation is a bunch of classes sitting inside a traditional layered architecture. Actually, if I'm being honest, this matches my own experience of building software myself because I've done the same thing! :-)

The intersection of software architecture and code

My approach to all of this is to ensure that the architecture and code views of a software system are one and the same thing, albeit from different levels of abstraction. In other words, my primary focus when describing a software system is the static structure, which ranges from code (classes) right up through components and containers. I model this with my C4 approach, which recognises that software developers are the primary stakeholders in software architecture. Other views of the software system (deployment, infrastructure, etc) slot into place really easily when you understand the static structure.

To put this all very simply, your code should reflect the architecture diagrams that you draw. If your diagrams include abstract concepts such as components, your code should reflect this. If the diagrams and code don't line up, you have to question the value of the diagrams because they're creating a fantasy and there's little point in referring to them.

Challenging the traditional layered approach

This deserves a separate blog post, but something I also mentioned during the talk was that teams should challenge the traditional layered architecture and the way that we structure our codebase. One way to achieve a nice mapping between architecture and code is to ensure that your code reflects the abstract concepts shown on your architecture diagrams, which can be achieved by writing components rather than classes in layers. Another side-effect of changing the organisation of the code is less test-induced design damage. The key question to ask here is whether layers are architecturally significant building blocks or merely an implementation detail, which should be wrapped up inside of (e.g.) components. As I said, this needs a separate blog post.

Thoughts?

As I said, the slides are here. Aligning the architecture and the code raises a whole bunch of interesting questions but provides some enormous benefits for a software development team. A clean mapping between diagrams and code makes a software system easy to explain, the impact of change becomes easier to understand and architectural refactorings can seem much less daunting if you know what you have and where you want to get to. I'm interested in your thoughts on things like the following:

  • Aligning the architecture and code - is this something you do. If so, how? If not, why not?
  • Is your codebase more than just a bunch of classes in layers? Do you follow what George Fairbanks calls an "architecturally-evident coding style"? If yes, how? If not, why not?
  • If you have any architecture documentation (e.g. diagrams), is it useful? If not, why not?

Convincing people to structure the code underlying their monolithic systems as a bunch of collaborating components seems to be a hard pill to swallow, yet micro-service architectures are going to push people to reconsider how they structure a software system, so I think this discussion is worth having. Thoughts?

Categories: Architecture

Hadoop YARN overview

I did a short overview of Hadoop YARN to our big data development team. The presentation covers the motivation for YARN, how it works and its major weaknesses

You can watch/download on slideshare

Categories: Architecture

Using Dropwizard in combination with Elasticsearch

Gridshore - Thu, 05/15/2014 - 21:09

Dropwizard logo

How often do you start creating a new application? How often have you thought about configuring an application. Where to locate a config file, how to load the file, what format to use? Another thing you regularly do is adding timers to track execution time, management tools to do thread analysis etc. From a more functional perspective you want a rich client side application using angularjs. So you need a REST backend to deliver json documents. Does this sound like something you need regularly? Than this blog post is for you. If you never need this, please keep on reading, you might like it.

In this blog post I will create an application that show you all the available indexes in your elasticsearch cluster. Not very sexy, but I am going to use: AngularJS, Dropwizard and elasticsearch. That should be enough to get a lot of you interested.


What is Dropwizard

Dropwizard is a framework that combines a lot of other frameworks that have become the de facto standard in their own domain. We have jersey for REST interface, jetty for light weight container, jackson for json parsing, free marker for front-end templates, Metric for the metrics, slf4j for logging. Dropwizard has some utilities to combine these frameworks and enable you as a developer to be very productive in constructing your application. It provides building blocks like lifecycle management, Resources, Views, loading of bundles, configuration and initialization.

Time to jump in and start creating an application.

Structure of the application

The application is setup as a maven project. To start of we only need one dependency:

<dependency>
    <groupId>io.dropwizard</groupId>
    <artifactId>dropwizard-core</artifactId>
    <version>${dropwizard.version}</version>
</dependency>

If you want to follow along, you can check my github repository:


https://github.com/jettro/dropwizard-elastic

Configure your application

Every application needs configuration. In our case we need to configure how to connect to elasticsearch. In drop wizard you extend the Configuration class and create a pojo. Using jackson and hibernate validator annotations we configure validation and serialization. In our case the configuration object looks like this:

public class DWESConfiguration extends Configuration {
    @NotEmpty
    private String elasticsearchHost = "localhost:9200";

    @NotEmpty
    private String clusterName = "elasticsearch";

    @JsonProperty
    public String getElasticsearchHost() {
        return elasticsearchHost;
    }

    @JsonProperty
    public void setElasticsearchHost(String elasticsearchHost) {
        this.elasticsearchHost = elasticsearchHost;
    }

    @JsonProperty
    public String getClusterName() {
        return clusterName;
    }

    @JsonProperty
    public void setClusterName(String clusterName) {
        this.clusterName = clusterName;
    }
}

Then you need to create a yml file containing the properties in the configuration as well as some nice values. In my case it looks like this:

elasticsearchHost: localhost:9300
clusterName: jc-play

How often did you start in your project to create the configuration mechanism? Usually I start with maven and quickly move to tomcat. Not this time. We did do maven, now we did configuration. Next up is the runner for the application.

Add the runner

This is the class we run to start the application. Internally jetty is started. We extend the Application class and use the configuration class as a generic. This is the class that initializes the complete application. Used bundles are initialized, classes are created and passed to other classes.

public class DWESApplication extends Application<DWESConfiguration> {
    private static final Logger logger = LoggerFactory.getLogger(DWESApplication.class);

    public static void main(String[] args) throws Exception {
        new DWESApplication().run(args);
    }

    @Override
    public String getName() {
        return "dropwizard-elastic";
    }

    @Override
    public void initialize(Bootstrap<DWESConfiguration> dwesConfigurationBootstrap) {
    }

    @Override
    public void run(DWESConfiguration config, Environment environment) throws Exception {
        logger.info("Running the application");
    }
}

When starting this application, we have no succes. A big error because we did not register any resources.

ERROR [2014-05-14 16:58:34,174] com.sun.jersey.server.impl.application.RootResourceUriRules: 
	The ResourceConfig instance does not contain any root resource classes.
Nothing happens, we just need a resource.

Before we can return something, we need to have something to return. We create a pogo called Index that contains one property called name. For now we just return this object as a json object. The following code shows the IndexResource that handles the requests that are related to the indexes.

@Path("/indexes")
@Produces(MediaType.APPLICATION_JSON)
public class IndexResource {

    @GET
    @Timed
    public Index showIndexes() {
        Index index = new Index();
        index.setName("A Dummy Index");

        return index;
    }
}

The @GET, @PATH and @Produces annotations are from the jersey rest library. @Timed is from the metrics library. Before starting the application we need to register our index resource with jersey.

    @Override
    public void run(DWESConfiguration config, Environment environment) throws Exception {
        logger.info("Running the application");
        final IndexResource indexResource = new IndexResource();
        environment.jersey().register(indexResource);
    }

Now we can start the application using the following runner from intellij. Later on we will create the executable jar.

Running the app from intelij

Run the application again, this time it works. You can browse to http://localhost:8080/index and see our dummy index as a nice json document. There is something in the logs though. I love this message, this is what you get when running the application without health checks.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!    THIS APPLICATION HAS NO HEALTHCHECKS. THIS MEANS YOU WILL NEVER KNOW      !
!     IF IT DIES IN PRODUCTION, WHICH MEANS YOU WILL NEVER KNOW IF YOU'RE      !
!    LETTING YOUR USERS DOWN. YOU SHOULD ADD A HEALTHCHECK FOR EACH OF YOUR    !
!         APPLICATION'S DEPENDENCIES WHICH FULLY (BUT LIGHTLY) TESTS IT.       !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Creating a health check

We add a health check, since we are creating an application interacting with elasticsearch, we create a health check for elasticsearch. Don’t think to much about how we connect to elasticsearch yet. We get there later on.

public class ESHealthCheck extends HealthCheck {

    private ESClientManager clientManager;

    public ESHealthCheck(ESClientManager clientManager) {
        this.clientManager = clientManager;
    }

    @Override
    protected Result check() throws Exception {
        ClusterHealthResponse clusterIndexHealths = clientManager.obtainClient().admin().cluster().health(new ClusterHealthRequest())
                .actionGet();
        switch (clusterIndexHealths.getStatus()) {
            case GREEN:
                return HealthCheck.Result.healthy();
            case YELLOW:
                return HealthCheck.Result.unhealthy("Cluster state is yellow, maybe replication not done? New Nodes?");
            case RED:
            default:
                return HealthCheck.Result.unhealthy("Something is very wrong with the cluster", clusterIndexHealths);

        }
    }
}

Just like with the resource handler, we need to register the health check. Together with the standard http port for normal users, another port is exposed for administration. Here you can find the metrics reports like Metrics, Ping, Threads, Healthcheck.

    @Override
    public void run(DWESConfiguration config, Environment environment) throws Exception {
        Client client = ESClientFactorybean.obtainClient(config.getElasticsearchHost(), config.getClusterName());

        logger.info("Running the application");
        final IndexResource indexResource = new IndexResource(client);
        environment.jersey().register(indexResource);

        final ESHealthCheck esHealthCheck = new ESHealthCheck(client);
        environment.healthChecks().register("elasticsearch", esHealthCheck);
    }

You as a reader now have an assignment to start the application and check the admin pages yourself: http://localhost:8081. We are going to connect to elasticsearch in the mean time.

Connecting to elasticsearch

We connect to elasticsearch using the transport client. This is taken care of by the ESClientManager. We make use of the drop wizard managed classes. The lifecycle of these classes is managed by drop wizard. From the configuration object we take the host(s) and the cluster name. Now we can obtain a client in the start method and pass this client to the classes that need it. The first class that needs it is the health check, but we already had a look at that one. Using the ESClientManager other classes have access to the client. The managed interface mandates the start as well as the stop method.

    @Override
    public void start() throws Exception {
        Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", clusterName).build();

        logger.debug("Settings used for connection to elasticsearch : {}", settings.toDelimitedString('#'));

        TransportAddress[] addresses = getTransportAddresses(host);

        logger.debug("Hosts used for transport client : {}", (Object) addresses);

        this.client = new TransportClient(settings).addTransportAddresses(addresses);
    }

    @Override
    public void stop() throws Exception {
        this.client.close();
    }

We need to register our managed class with the lifecycle of the environment in the runner class.

    @Override
    public void run(DWESConfiguration config, Environment environment) throws Exception {
        ESClientManager esClientManager = new ESClientManager(config.getElasticsearchHost(), config.getClusterName());
        environment.lifecycle().manage(esClientManager);
    }	

Next we want to change the IndexResource to use the elasticsearch client to list all indexes.

    public List<Index> showIndexes() {
        IndicesStatusResponse indices = clientManager.obtainClient().admin().indices().prepareStatus().get();

        List<Index> result = new ArrayList<>();
        for (String key : indices.getIndices().keySet()) {
            Index index = new Index();
            index.setName(key);
            result.add(index);
        }
        return result;
    }

Now we can browse to http://localhost:8080/indexes and we get back a nice json object. In my case I got this:

[
	{"name":"logstash-tomcat-2014.05.02"},
	{"name":"mymusicnested"},
	{"name":"kibana-int"},
	{"name":"playwithip"},
	{"name":"logstash-tomcat-2014.05.08"},
	{"name":"mymusic"}
]
Creating a better view

Having this REST based interface with json documents is nice, but not if you are human like me (well kind of). So let us add some AngularJS magic to create a slightly better view. The following page can of course also be created with easier view technologies, but I want to demonstrate what you can do with dropwizard.

First we make it possible to use free marker as a template. To make this work we need to additional dependencies: dropwizard-views and dropwizard-views-freemarker. The first step is a view class that knows the free marker template to load and provide the fields that you template can read. In our case we want to expose the cluster name.

public class HomeView extends View {
    private final String clusterName;

    protected HomeView(String clusterName) {
        super("home.ftl");
        this.clusterName = clusterName;
    }

    public String getClusterName() {
        return clusterName;
    }
}

Than we have to create the free marker template. This looks like the following code block

<#-- @ftlvariable name="" type="nl.gridshore.dwes.HomeView" -->
<html ng-app="myApp">
<head>
    <title>DWAS</title>
</head>
<body ng-controller="IndexCtrl">
<p>Underneath a list of indexes in the cluster <strong>${clusterName?html}</strong></p>

<div ng-init="initIndexes()">
    <ul>
        <li ng-repeat="index in indexes">{{index.name}}</li>
    </ul>
</div>

<script src="/assets/js/angular-1.2.16.min.js"></script>
<script src="/assets/js/app.js"></script>
</body>
</html>

By default you put these template in the resources folder using the same sub folders as your view class has for the package. If you look closely you see some angularjs code, more on this later on. First we need to map a url to the view. This is done with a resource class. The following code block shows the HomeResource class that maps the “/” to the HomeView.

@Path("/")
@Produces(MediaType.TEXT_HTML)
public class HomeResource {
    private String clusterName;

    public HomeResource(String clusterName) {
        this.clusterName = clusterName;
    }

    @GET
    public HomeView goHome() {
        return new HomeView(clusterName);
    }
}

Notice we now configure it to return text/html. The goHome method is annotated with GET, so each GET request to the PATH “/” is mapped to the HomeView class. Now we need to tell jersey about this mapping. That is done in the runner class.

final HomeResource homeResource = new HomeResource(config.getClusterName());
environment.jersey().register(homeResource);
Using assets

The final part I want to show is how to use the assets bundle from drop zone to map a folder “/assets” to a part of the url. To use this bundle you have to add the following dependency in maven: dropwizard-assets. Than we can easily map the assets folder in our resources folder to the web assets folder

    @Override
    public void initialize(Bootstrap<DWESConfiguration> dwesConfigurationBootstrap) {
        dwesConfigurationBootstrap.addBundle(new ViewBundle());
        dwesConfigurationBootstrap.addBundle(new AssetsBundle("/assets/", "/assets/"));
    }

That is it, now you can load the angular javascript file. My very basic sample has one angular controller. This controller uses the $http service to call our /indexes url. The result is used to show the indexes in a list view.

myApp.controller('IndexCtrl', function ($scope, $http) {
    $scope.indexes = [];

    $scope.initIndexes = function () {
        $http.get('/indexes').success(function (data) {
            $scope.indexes = data;
        });
    };
});

And the result

the very basic screen showing the indexes

Concluding

This was my first go at using drop wizard, I must admit I like what I have seen so far. Not sure if I would create a big application with it, on the other hand it is really structured. Before moving on I would need to reed a bit more about the library and check all of its options. There is a lot more possible than what I have showed you in here.

References

The post Using Dropwizard in combination with Elasticsearch appeared first on Gridshore.

Categories: Architecture, Programming

Yet More Change for the Capitals

DevHawk - Harry Pierson - Sat, 04/26/2014 - 21:13

Six years ago, I was pretty excited about the future for the Washington Capitals. They had just lost their first round match up with the Flyers – which was a bummer – but they had made the playoffs for the first time in 3 seasons. I wrote at the time:

Furthermore, even though they lost, these playoffs are a promise of future success. I tell my kids all the time that the only way to get good at something is to work hard while you’re bad at it. Playoff hockey is no different. Most of the Caps had little or no playoff experience going into this series and it really showed thru the first three games. But they kept at it and played much better over the last four games of the series. They went 2-2 in those games, but the two losses went to overtime. A little more luck (or better officiating) and the Caps are headed to Pittsburgh instead of the golf course.

What a difference six seasons makes. Sure, they won the President’s Trophy in 2010. But the promise of future playoff success has been broken, badly. The Caps have been on a pretty steep decline after getting beat by the eighth seed Canadians in the first round of the playoffs in 2010. Since then, they’ve switched systems three times and head coaches twice. This year, they missed the playoffs entirely even with Alex Ovechkin racking up a league-leading 51 goals.

Today, the word came down that both the coach and general manager have been let go. As a Caps fan, I’m really torn about this. I mean, I totally agree that the coach and GM had to go – frankly, I was surprised it didn’t happen 7-10 days earlier. But now what do you do? The draft is two months and one day away, free agency starts two days after that. The search for a GM is going to have to be fast. Then the GM will have to make some really important decisions about players at the draft, free agency and compliance buyouts with limited knowledge of the players in our system. Plus, he’ll need to hire a new head coach – preferably before the draft as well.

The one positive note is that the salary cap for the Capitals looks pretty good for next year. The Capitals currently have the second largest amount of cap space / open roster slot in the league. (The Islanders are first with $14.5 million / open roster slot. The Caps have just over $7 million / open roster slot.) They have only a handful of unrestricted free agents to resign – with arguably only one “must sign” (Mikhail Grabovski) in the bunch. Of course, this could also be a bug rather than a feature – having that many players under contract may make it harder for the new GM to shape the team in his image.

Who every the Capitals hire to be GM and coach, I’m not expecting a promising start. It feels like the next season is already a wash, and we’re not even finished with the first round of this year’s playoffs yet.

I guess it could be worse.

I could be a Toronto Leafs fan.

Categories: Architecture, Programming

Brokered WinRT Components Step Three

DevHawk - Harry Pierson - Fri, 04/25/2014 - 16:45

So far, we’ve created two projects, written all of about two lines of code and we have both our brokered component and its proxy/stub ready to go. Now it’s time to build the Windows Runtime app that uses the component. So far, things have been pretty easy – the only really tricky and/or manual step so far has been registering the proxy/stub, and that’s only tricky if you don’t want to run VS as admin. Unfortunately, tying this all together in the app requires a few more manual steps.

But before we get to the manual steps, let’s create the WinRT client app. Again, we’re going to create a new project but this time we’re going to select “Blank App (Windows)” from the Visual C# -> Store Apps -> Windows App node of the Add New Project dialog. Note, I’m not using “Blank App (Universal)” or “Blank App (Windows Phone)” because the brokered WinRT component feature is not support on Windows Phone. Call the client app project whatever you like, I’m calling mine “HelloWorldBRT.Client”.

Before we start writing code, we need to reference the brokered component. We can’t reference the brokered component directly or it will load in the sandboxed app process. Instead, the app need to reference a reference assembly version of the .winmd that gets generated automatically by the proxy/stub project. Remember in the last step when I said Kieran Mockford is an MSBuild wizard? The proxy/stub template project includes a custom target that automatically publishes the reference assembly winmd file used by the client app. When he showed me that, I was stunned – as I said, the man is a wizard. This means all you need to do is right click on the References node of the WinRT Client app project and select Add Reference. In the Reference Manager dialog, add a reference to the proxy/stub project you created in step two.

Now I can add the following code to the top of my App.OnLaunched function. Since this is a simple Hello World walkthru, I’m not going to bother to build any UI. I’m just going to inspect variables in the debugger. Believe me, the less UI I write, the better for everyone involved. Note, I’ve also added the P/Invoke signatures for GetCurrentProcess/ThreadID and to the client app like I did in the brokered component in step one. This way, I can get the process and thread IDs for both the app and broker process and compare them.

var pid = GetCurrentProcessId();
var tid = GetCurrentThreadId();

var c = new HelloWorldBRT.Class();
var bpid = c.CurrentProcessId;
var btid = c.CurrentThreadId;

At this point the app will compile, but if I run it the app will throw a TypeLoadException when it tries to create an instance of HelloWorldBRT.Class. The type can’t be loaded because the we’re using the reference assembly .winmd published by the proxy/stub project – it has no implementation details, so it can’t load. In order to be able to load the type, we need to declare the HelloWorldBRT.Class as a brokered component in the app’s pacakge.appxmanifest file. For non-brokered components, Visual Studio does this for you automatically. For brokered components we have to do it manually unfortunately. Every activatable class (i.e. class you can construct via “new”) needs to be registered in the appx manifest this way.

To register HelloWorldBRT.Class, right click the Package.appxmanifest file in the client project, select “Open With” from the context menu and then select “XML (Text) editor” from the Open With dialog. Then you need to insert inProcessServer extension that includes an ActivatableClass element for each class you can activate (aka has a public constructor). Each ActivatableClass element contains an ActivatableClassAttribute element that contains a pointer to the folder where the brokered component is installed. Here’s what I added to Package.appxmainfest of my HelloWorldBRT.Client app.

<Extensions>
  <Extension Category="windows.activatableClass.inProcessServer">
    <InProcessServer>
      <Path>clrhost.dll</Path>
      <ActivatableClass ActivatableClassId="HelloWorldBRT.Class" 
                        ThreadingModel="both">
        <ActivatableClassAttribute 
             Name="DesktopApplicationPath" 
             Type="string" 
             Value="D:\dev\HelloWorldBRT\Debug\HelloWorldBRT.PS"/>
      </ActivatableClass>
    </InProcessServer>
  </Extension>
</Extensions>

The key thing here is the addition of the DesktopApplicationPath ActivatableClassAttribute. This tells the WinRT activation logic that HelloWorldBRT.Class is a brokered component and where the managed .winmd file with the implementation details is located on the device. Note, you can use multiple brokered components in your side loaded app, but they all have the same DesktopApplicationPath.

Speaking of DesktopApplicationPath, the path I’m using here is path the final location of the proxy/stub components generated by the compiler. Frankly, this isn’t an good choice to use in a production deployment. But for the purposes of this walk thru, it’ll be fine.

ClientWatchWindow

Now when we run the app, we can load a HelloWorldBRT.Class instance and access the properties. re definitely seeing a different app process IDs when comparing the result of calling GetCurrentProcessId directly in App.OnLoaded vs. the result of calling GetCurrentProcessId in the brokered component. Of course, each run of the app will have different ID values, but this proves that we are loading our brokered component into a different process from where our app code is running.

Now you’re ready to go build your own brokered components! Here’s hoping you’ll find more interesting uses for them than comparing the process IDs of the app and broker processes in the debugger! :)

Categories: Architecture, Programming

Brokered WinRT Components Step Two

DevHawk - Harry Pierson - Fri, 04/25/2014 - 16:43

Now that we have built the brokered component , we have to build a proxy/stub for it. Proxies and stubs are how WinRT method calls are marshalled across process boundaries. If you want to know more – or you have insomnia – feel free to read all the gory details up on MSDN.

Proxies and stubs look like they might be scary, but they’re actually trivial (at least in the brokered component scenario) because 100% of the code is generated for you. It couldn’t be much easier.

Right click the solution node and select Add -> New Project. Alternatively, you can select File -> New -> Project in the Visual Studio main menu, but if you do that make sure you change the default solution from “Create new Solution” to “Add to Solution”. Regardless of how you launch the new project wizard, search for “broker” again, but this time select the “Brokered Windows Runtime ProxyStub” template. Give the project a name – I chose “HelloWorldBRT.PS”.

ProxyStubAddReferenceOnce you’ve created the proxy/stub project, you need to set a reference to the brokered component you created in step 1. Since proxies and stubs are native, this is a VC++ project. Adding a reference in a VC++ is not as straightforward as it is in C# projects. Right click the proxy/stub project, select “Properties” and then select Common Properties -> References from the tree on the left. Press the “Add New Reference…” button to bring up the same Add Reference dialog you’ve seen in managed code projects. Select the brokered component project and press OK.

Remember when I said that 100% of the code for the proxy/stub is generated? I wasn’t kidding – creating the template and setting referencing the brokered component project is literally all you need to do. Want proof? Go ahead and build now. If you watch the output windows, you’ll see a bunch of output go by referencing IDL files and MIDLRT among other stuff. This proxy/stub template has some custom MSBuild tasks that generates the proxy/stub code using winmdidl and midlrt. The process is similar to what is described here. BTW, if you get a chance, check out the proxy/stub project file – it is a work of art. Major props to Kieran Mockford for his msbuild wizardry.

ProxyStubRegisterOutputUnfortunately, it’s not enough just to build the proxy/stub – you also have to register it. The brokered component proxy/stub needs to be registered globally on the machine, which means you have to be running as an admin to do it. VS can register the proxy/stub for you automatically, but that means you have to run VS as an administrator. That always makes me nervous, but if you’re OK with running as admin you can enable proxy/stub registration by right clicking the proxy/stub project file, selecting Properties, navigating to Configuration properties -> Linker -> General in the tree of the project properties page, and then changing Register Output to “Yes”.

If you don’t like running VS as admin, you can manually register the proxy/stub by running “regsvr32 <proxystub dll>” from an elevated command prompt. Note, you do have to re-register every time the public surface area of your brokered component changes so letting VS handle registration admin is definitely the easier route to go.

In the third and final step, we’ll build a client app that accesses our brokered component.

Categories: Architecture, Programming

Brokered WinRT Components Step One

DevHawk - Harry Pierson - Fri, 04/25/2014 - 16:41

In this step, we’ll build the brokered component itself. Frankly, the only thing that makes a brokered component different than a normal WinRT component is some small tweaks to the project file to enable access to the full .NET Runtime and Base Class Library. The brokered component whitepaper describes the these tweaks in detail, but the new brokered component template takes care of these small tweaks for you.

BRT_NewProjectStart by selecting File -> New -> Project in Visual Studio. With the sheer number of templates to choose from these days, I find it’s easier to just search for the one I want. Type “broker” in the search box in the upper left, you’ll end up with two choices – the brokered WinRT component and the brokered WinRT proxy/stub. For now, choose the brokered component. We’ll be adding a brokered proxy/stub in step two. Name the project whatever you want. I named mine “HelloWorldBRT”.

This is probably the easiest step of the three as there’s nothing really special you have to do – just write managed code like you always do. In my keynote demo, this is where I wrote the code that wrapped the existing ADO.NET based data access library. For the purposes of this walkthrough, let’s do something simpler. We’ll use P/Invoke to retrieve the current process and thread IDs. These Win32 APIs are supported for developing WinRT apps and will make it obvious that the component is running in a separate process than the app. Here’s the simple code to retrieve those IDs (hat tip to pinvoke.net for the interop signatures):

public sealed class Class
{
    [DllImport("kernel32.dll")]
    static extern uint GetCurrentThreadId();

    [DllImport("kernel32.dll")]
    static extern uint GetCurrentProcessId();

    public uint CurrentThreadId
    {
        get { return GetCurrentThreadId(); }
    }

    public uint CurrentProcessId
    {
        get { return GetCurrentProcessId(); }
    }
}

That’s it! I didn’t even bother to change the class name for this simple sample.

Now, to be clear, there’s no reason why this code needs to run in a broker process. As I pointed out, the Win32 functions I’m wrapping here are supported for use in Windows Store apps. For this walkthrough, I’m trying to keep the code simple in order to focus on the specifics of building brokered components. If you want to see an example that actually leverages the fact that it’s running outside of the App Container, check out the NorthwindRT sample.

In the next step, we’ll add the proxy/stub that enables this component to communicate across a process boundary.

Categories: Architecture, Programming

Brokered WinRT Components Step-by-Step

DevHawk - Harry Pierson - Fri, 04/25/2014 - 16:40

Based on the feedback I’ve gotten since my keynote appearance @ Build – both in person and via email & twitter – there are a lot of folks who are excited about the Brokered WinRT Component feature. However, I’ve been advising folks to hold off a bit until the new VS templates were ready. Frankly, the developer experience for this feature is a bit rough and the VS template makes the experience much better. Well, hold off no longer! My old team has published the Brokered WinRT Component Project Templates up on the Visual Studio Gallery!

Now that the template is available, I’ve written a step-by-step guide demonstrating how to build a “Hello World” style brokered component. Hopefully, this will help folks in the community take advantage of this cool new feature in Windows 8.1 Update.

To keep it readable, I’ve broken it into three separate posts:

Note, this walkthrough assumes you’re running Windows 8.1 Update, Visual Studio 2013 with Update 2 RC (or later) and the Brokered WinRT Component Project Templates installed.

I hope this series helps you take advantage of brokered WinRT components. If you have any further questions, feel free to drop me an email or hit me up on Twitter.

Categories: Architecture, Programming

New book review ‘SOA Made Simple’ coming soon.

Ben Wilcock - Tue, 02/05/2013 - 14:24

My review copy has arrived and I’ll be reviewing it just as soon as I can, but in the meantime if you’d like more information about this new book go to http://bit.ly/wpNF1J


Categories: Architecture, Programming