Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Mutation Testing: How Good are your Unit Tests?

Xebia Blog - Thu, 11/06/2014 - 20:56

You write unit tests for every piece of code you deliver. Your test coverage is close to 100%. So when the point comes when you have to make some small changes to the existing code, you feel safe and confident that your test suite will protect you against possible mistakes.
You make your changes, and all your tests still pass. You should be fairly confident now that you can commit your new code without breaking anything, right?

Well, maybe not. Maybe your unit tests were fooling you. Sure they covered every line of your code, but they could have performed the wrong assertions.
In this post I will introduce mutation testing. Mutation testing can help you find omissions in your unit tests.

Let's begin with a small example:

package com.xebia;

public class NameParser {
  public Person findPersonWithLastName(String[] names, String lastNameToFind) {
    Person result = null;
    for(int i=0; i <= names.length; i++) { // bug 1
      String[] parts = names[i].split(" ");
      String firstName = parts[0];
      String lastName = parts[1];
      if(lastName.equals(lastNameToFind)) {
        result = new Person(firstName, lastName);
        break;
      }
    }
    return result;
  }
}

NameParser takes a list of strings which consist of a first name and a last name, searches for the entry with a given last name, instantiates a Person object out of it and returns it.
Here is the Person class:

package com.xebia;

public class Person {
  private final String firstName;
  private final String lastName;

  public Person(String firstName, String lastName) {
    this.firstName = firstName;
    this.lastName = lastName;
  }

  public String getFirstName() {
    return firstName;
  }

  public String getLastName() {
    return firstName; // bug 2
  }
}

You can see that there are two bugs in the code. The first one is in the loop in NameParser, which loops past the end of the names array. The second on is in Person, which mistakenly returns firstName in its getLastName method.

NameParser has a unit test:

package com.xebia;

import org.junit.Before;
import org.junit.Test;
import static org.junit.Assert.assertEquals;

public class NameParserTest {
  private NameParser nameParser;
  private String[] names;

  @Before
  public void setUp() {
    nameParser = new NameParser();
    names = new String[]{"Mike Jones", "John Doe"};
  }

  @Test
  public void shouldFindPersonByLastName() {
    Person person = nameParser.findPersonWithLastName(names, "Doe");
    String firstName = person.getFirstName();
    String lastName = person.getLastName();
    assertEquals("John", firstName);
  }
}

The unit tests covers the Person and NameParser code for 100% and succeeds!
It doesn't find the bug in Person.getLastName because it simply forgets to do an assertion on it. And it doesn't find the bug in the loop in NameParser because it doesn't test the case where the names list does not contain the given last name; so the loop always terminates before it has a chance to throw an IndexOutOfBoundsException.

Especially the last case is easy to overlook, so it would be nice if there existed a tool which could detect these kinds of problems.
And there is one: actually there are a couple. For this post I have chosen PIT, down at the end are links to some alternatives.

But first: what is mutation testing?

A mutation test will make a small change to your code and then run the unit test(s). Such a change is called a 'mutant'. If a change can be made and the unit tests still succeed, it will generate a warning saying that the mutant 'survived'.
The test framework will try to apply a number of predefined mutants at every point in your code where they are applicable. The higher the percentage of the mutants that get killed by your unit tests, the better the quality of your test suite.
Examples of mutants are: negating a condition in an If statement, changing a conditional boundary in a For loop, or throwing an exception at the end of a method.

Putting NameParser's testcase to the test with PIT

PIT stands for Parallel Isolated Test, which is what the project originally was meant for. But it turned out to be a much more interesting goal to do mutation testing, which required much of the same plumbing.

PIT integrates with JUnit or TestNG and can be configured with Maven, Gradle and others. Or it can be used directly as a plugin in Eclipse or IntelliJ.
I'm choosing for the last option: the IntelliJ plugin. The setup is easy: just install PITest from the plugin manager and you are ready to go. Once you're done, you'll find a new launch configuration option in the 'edit configurations' menu called PIT.

PIT launch configuration

You have to specify the classes where PIT will makes its mutations under 'Target classes'.
When we run the mutation test, PIT creates a Html report with the results for every class.
Here are the results for the NameParser class:

NameParser mutation testing results

As you can read under 'Mutations', PIT has been able to apply five code mutations to the NameParser class. Four of them resulted in a failing NameParserTest, which is exactly what we'd like to see.
But one of them did not: when the condition boundary in line 6, the loop constraint, was changed, NameParserTest still succeeded!
PIT changes loop constraints with a predefined algorithm; in this case, when the loop constraint was i <= names.length, it changed the '<=' into a '<'. Actually this accidentally corrected the bug in NameParser, and of course that didn't break the unit test.
So PIT found an omission in our unit test here, and it turned out that this omission even left a bug undetected!

Note that this last point doesn't always need to be the case. It could be that for the correct behavior of your class, there is some room for some conditions to change.
In the case of NameParser for instance, it could have been a requirement that the names list always contained an entry with the last name that was to be found. In that case the behavior for a missing last name would be unspecified and an IndexOutOfBoundsException would have been as good a result as anything else.
So PIT can only find strong indications of omissions in your unit tests, but they don't necessarily have to be.

And here are the results for the Person class:

Person mutation test results

PIT was able to do two mutations in the Person class; one in every getter method. Both times it replaced the return value with null. And as expected, the mutation in the getLastName method went undetected by our unit test.
So PIT found the second omission as well.

Conclusion

In this case, mutation testing would have helped us a lot. But there can still be cases where possible bugs can go unnoticed. In our code for example, there is no test in NameParser test that verifies the behavior when an entry in the names list does not contain both a first name and a last name. But PIT didn't find this omission.
Still it might make good sense to integrate mutation testing in your build process. PIT can be configured to break your Maven build if too many warnings are found.
And there's a lot more that can be configured as well, but for that I recommend to bring a visit to the website of PIT at www.pitest.org.

Alternatives

PIT is not the only mutation testing framework for Java, but it is the most popular and the one most actively maintained. Others are µJava and Jumble.
Although most mutation testing frameworks are written for Java, probably because it's so easy to dynamically change its bytecode, mutation testing is possible in some other languages as well: notable are grunt-mutation-testing for Javascript and Mutator, a commercial framework which is available for a couple of languages.

How To Improve the IT-Business Relationship

It’s possible to change IT from a poorly respected cost center to a high-functioning business partner.

Driving business transformation is a people, process, and technology thing.

Some people think they can change their business without IT.   The challenge is that technology is the enabler of significant business change in today’s digital landscape.  Cloud, Mobile, Social, and Big Data all bring significant capabilities to the table, and IT can hold the keys.

But the business doesn’t want to hear that.

Business Leaders Want to Hear About the WHY and WHAT of the Business

Business leaders don’t want to hear about the HOW of technology.

Business leaders want to hear about the impact on their business.   They want to hear about how predictive analytics can help them build a better pipeline, or target more relevant offers.   Business leaders want to hear about how they can create cross-sell/upsell opportunities in real-time.   And, business leaders want to hear about the business benefits and the KPI that will be impacted by choosing a particular strategy.

The reality is that the new Digital Masters of the emerging Digital Economy bring their IT with them, and in many cases, their IT even helps lead the business into the new Digital Frontier.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are digital masters that created their digital visions and are driving business change.

How IT Can Change Its Game

While it takes work on both sides, IT can change it’s game by creating transparency around performance, roles, and value.  This includes helping employees think and talk differently about what they do.   IT can show very clearly how it delivers value for the money.  And IT can change the way IT and business leaders make investment decisions and assess the returns.

IT Needs to Speak Business

The CIO and everybody in IT, needs to speak the language of business.

Via Leading Digital:

“Poor relations between IT and business leaders can have many causes.  Sometimes it's the personality of the IT leader.  A common complaint among senior executives is that their CIO seems to speak a different language from the business.  Another is that the CIO doesn't seem to understand what's really important.  For example, a chemical company CIO we interviewed described how he communicates regularly with business executives about the innovative possibilities of digital technologies.  Yet none of his business executive peers (whom we interviewed separately) seemed to find the discussions credible.”

IT Needs to Deliver Better, Faster, and More Reliably than Outsourcing

It’s a competitive world and IT needs to continuously find ways to deliver solutions in a way that makes business sense.

Via Leading Digital:

“Sometimes the issue arises from IT's delivery capability.  According to Bud Mathaisel, who has served as CIO in several large companies, 'It starts with competence in delivering services reliably, economically, and at very high quality.  It is the absolute essential to be even invited into meaningful dialog about how you then build on that competence to do something very interesting with it.'  Unfortunately, some IT units today do not have this competence.  One business executive we interviewed said, 'IT is a mess.  It's costs are not acceptable.  It proposes things in nine or ten months, where external firms could do them in three to nine weeks.  We started offshoring our IT, and now our IT guys are trying to change.' A legacy of poor communication, byzantine decision processes, and broken commitments is no foundation on which to build a strong IT-business relationship.”

IT Needs a Good Digital Platform to Be High-Performing IT

In order to bet on IT, it needs to be high-performing.  And in order for IT to be high-performing, it needs to have a good digital platform.

Via Leading Digital:

“However, the fault doesn't always rest only with IT leaders.  In many cases, business executive share some of the blame ... high-performing IT requires a good digital platform, and good platforms require discipline.  If your approach to working with IT can be characterized by impatience, unreasonable expectations, or insisting on doing things your way, then you'll need to think about how to change your side of the relationship.”

CIOs Can Lead Digital Business Transformation

Key business transformation takes technology.  CIOs can help lead the business transformation, whether it's through shared goals with the business, creating a new governance mechanism, or creating a new shared digital unit.

Via Leading Digital:

“Regardless of the case, if your IT-business relationships are poor, it's essential to fix the problem.  A bank executive stated, 'IT has been brought closer to business during the last five years.  It is very important to success because man of the important transformations in our business are enabled by technology.'  With strong relationships, IT executives can help business executives meet their goals, and business executives listen when IT people suggest innovations.  Executives on both sides are willing to be flexible in creating new governance mechanisms or shared digital units.  At Codelco, Asian Paints, and P&G, the CIO even leads digital transformation for the company.”

CIOs Can Help Drive the Bus with the Executive Team

CIOs can help drive the bus, but it takes more than senior sponsorship.

Via Leading Digital:

“So, how can you start to improve your IT-business relationship?  Angela Ahrendts, CEO of Burberry, told her CIO he needed to help drive the bus with the executive team.  However, leadership changes or top-down mandates are only the start of the change.  Few CIOs can change the business by themselves, and not all business executives will climb on the bus with the CIO, even if the CEO demands it.”

Fix How You Communicate to Fix the IT-Business Relationship

Start by fixing how you communicate between the business and IT.

Via Leading Digital:

“Fixing the IT-business relationship can take time, as people learn how to trust each other and redefine the way they work together.  As with any struggling relationship, the best starting point is to fix the way you communicate.  Does IT really cost too much, or are costs reasonable, given what IT has to do? Is the IT unit really too bureaucratic, or do all of those procedures actually serve a useful purpose?  Are you a good partner to IT or a difficult one?  How can IT make it easier for you to get what you need, while still making sure things are done correctly?  What investments can help IT improve its technology, internal processes, cost-effectiveness, quality, or speed?”

Change IT from a Poorly Respected Cost Center to a High-Functioning Business Partner

It’s possible to change IT from a low performing cost center to a high-performing business partner.  Companies do it all the time, and MIT has the research.

Via Leading Digital:

“MIT research into IT turnarounds has identified a series of steps that can change IT from a poorly respected cost center to a high-functioning business partner.  The key change mechanism is transparency--around performance, roles, and value.  The first step is to help IT employees think, and talk, differently about what they do.  The second step proceeds to showing very clearly how well (or how poorly) IT delivers value for money--the right services at the right quality and right price, and where problems still exist.  And then the third step moves to changing the way IT and business leaders make investment decisions and assess the returns that projects deliver.  Through transparency around roles, performance, and investments, both sides can make smoother decisions and work together to identify and deliver innovation.”

If you’re part of a business that wants to change the world, start by reimagining how IT can help you achieve the art of the possible.

You Might Also Like

Building Better Business Cases for Digital Initiatives

Drive Business Transformation by Reenvisioning Your Customer Experience

Drive Business Transformation by Reenvisioning Your Operations

How Digital is Changing Physical Experiences

The Future of IT Leaders

Categories: Architecture, Programming

Swift Function Currying

Xebia Blog - Thu, 11/06/2014 - 12:55

One of the lesser known features of Swift is Function Currying. When you read the Swift Language Guide you won't find anything about curried functions. Apple only describes it in their Swift Language Reference. And that's a pity, since it's a very powerful and useful feature that deserves more attention. This post will cover the basics and some scenarios in which it might be useful to use curried functions.

I assume you're already somewhat familiar with function currying since it exists in many other languages. If not, there are many articles on the Internet that explain what it is and how it works. In short: you have a function that receives one or more parameters. You then apply one or more known parameters to that function without already executing it. After that you get a function reference to a new function that will call the original function with the applied parameters.

One situation in which I find it useful to use curried functions is with completion handlers. Imagine you have a function that makes a http request and looks something like this:

func doGET(url: String, completionHandler: ([String]?, NSError?) -> ()) {
    // do a GET HTTP request and call the completion handler when receiving the response
}

This is a pretty common pattern that you see with most networking libraries as well. We can call it with some url and do a bunch of things in the completion handler:

doGET("http://someurl.com/items?all=true", completionHandler: { results, error in
    self.results = results
    self.resultLabel.text = "Got all items"
    self.tableView.reloadData()
})

The completion handler can become a lot more complex and you might want to reuse it in different places. Therefore you can extract that logic into a separate function. Luckily with Swift, functions are just closures so we can immediately pass a completion handler function to the doGET function:

func completionHandler(results: [String]?, error: NSError?) {
    self.results = results
    self.resultLabel.text = "Got all items"
    self.tableView.reloadData()
}

func getAll() {
    doGET("http://someurl.com/items?all=true", completionHandler)
}

func search(search: String) {
    doGET("http://someurl.com/items?q=" + search, completionHandler)
}

This works well, as long as the completion handler should always do exactly the same thing. But in reality, that's usually not the case. In the example above, the resultLabel will always display "Got all items". Lets change that into "Got searched items" for the search request:

func search(search: String) {
    doGET("http://someurl.com/items?q=" + search, {results, error in
        self.completionHandler(results, error: error)
        self.resultLabel.text = "Got searched items"
    })
}

This will work, but it doesn't look very nice. What we actually want is to have this dynamic behaviour in the completionHandler function. We can change the completionHandler in such a way that it accepts the text for the resultLabel as a parameter and then returns the actual completion handler as a closure.

func completionHandler(text: String) -> ([String]?, NSError?) -> () {
    return {results, error in
        self.results = results
        self.resultLabel.text = text
        self.tableView.reloadData()
    }
}

func getAll() {
    doGET("http://someurl.com/items?all=true", completionHandler("Got all items"))
}

func search(search: String) {
    doGET("http://someurl.com/items?q=" + search, completionHandler("Got searched items"))
}

And as it turns out, this is exactly what we can also do using currying. We just need to add the parameters of the actual completion handler as a second parameters group to our function:

func completionHandler(text: String)(results: [String]?, error: NSError?) {
    self.results = results
    self.resultLabel.text = text
    self.tableView.reloadData()
}

Calling this with the first text parameter will not yet execute the function. Instead it returns a new function with the [String]?, NSError? as parameters. Once that function is called the completionHandler function is finally executed.

You can create as many levels of this currying as you want. And you can also leave the last parameter group empty just to get a reference to the fully applied function. Let's look at another example. We have a simple function that sets the text of the resultLabel:

func setResultLabelText(text: String) {
    resultLabel.text = text
}

And for some reason, we need to call this method asynchronously. We can do that using the Grand Central Dispatch functions:

dispatch_async(dispatch_get_main_queue(), {
    self.setResultLabelText("Some text")
})

Since the dispatch_async function only accepts a closure without any parameters, we need to create an inner closure here. If the setResultLabelText was a curried function, we could fully apply it with the parameter and get a reference to a function without parameters:

func setResultLabelText(text: String)() { // now curried
    resultLabel.text = text
}

dispatch_async(dispatch_get_main_queue(), setResultLabelText("Some text"))

But you might not always have control over such functions, for example when you're using third party libraries. In that case you cannot change the original function into a curried function. Or you might not want to change it since you already using it at many other places and you don't want to break anything. In that case we can achieve something similar by creating a function that creates the curried function for us:

// defined in global scope
func curry<T>(f: (T) -> (), arg: T)() {
    f(arg)
}

We can now use it as following:

func setResultLabelText(text: String) {
    resultLabel.text = text
}

dispatch_async(dispatch_get_main_queue(), curry(setResultLabelText, "Some text"))

Probably in this example it might be just as easy to go with the inner closure, but being able to pass around partial applied functions is very powerful and used in many programming languages already.

Unfortunately the last example also show a great drawback about the way currying is implemented in Swift: you cannot simply curry normal functions. It would be great to be able to curry any function that takes multiple parameters instead of having to explicitly create curried functions. Another drawback is that you can only curry in the defined order of parameters. That doesn't allow you to do reverse currying (e.g. apply only the last parameter) or even apply just any parameter you want, regardless of its position. Hopefully the Swift language will evolve in this and get more powerful currying features.

A Few Really Good Tech Podcasts for Your Listening Pleasure

Podcasts are back. Since I listen to a bunch of podcasts every week I was quite surprised to learn about their resurrection. But facial hair is back too, so I will revel in my short lived trendiness.

Why are podcasts back? One suggested reason is a little sad. People commute more than 25 hours a day and since cars are now faithful little bluetooth slaves, broadcasting podcasts over luxury car speakers is as easy as a smart technology and a fast cell network can make it. Podcasts are now a viable P2P radio replacement.

That’s the demand, means, motive, and opportunity side of things. What about supply?

Historically podcasts start in fire, the passion quickly burning to ash as podcasters learn making podcasts is a lot of hard work...and poorly unremunerated work at that. So podcasts have a high mortality rate.

What’s changed? Money. Strangely, people will exchange effort for money, so if podcasts can make money they will have the fuel they need to burn bright through the night.

Money from Subscriptions is New
Categories: Architecture

Improve small job completion times by 47% by running full clones.

The idea is most jobs are small. Researchers found 82% of jobs on Facebook's cluster were less than 10 tasks. Clusters have a median utilization of under 20%. And since small jobs are particularly sensitive to stragglers the audacious solution is to proactively launch clones of a job as they are submitted and pick the result from the earliest clone. The result is an average completion time of all the small jobs improved by 47% using cloning, at the cost of just 3% extra resources.

For more details take a look at the very interesting Why Let Resources Idle? Aggressive Cloning of Jobs with Dolly.

Related Articles
Categories: Architecture

Drive Business Transformation by Reenvisioning Your Operations

When you create your digital vision, you have a few places to start.

One place to start is by reenvisioning your customer experience.   Another place to start is by reenvisioning your operations.   And, a third place to start is by renvisioning your business model.

In this post, let’s take a look at reenvisioning your operations.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are digital masters that created their digital visions and are driving business change.

Start with Reenvisioning Operations When Financial Performance is Tied to Your Supply Chain

If your financial performance is closely connected to the performance of your core operations and supply chain, then reenvisioning your operations can be a great place to start.

Via Leading Digital:

“Organizations whose fortunes are closely tied to the performance of their core operations and supply chains often start with reenvisioning their operations.”

Increase Process Visibility, Decision Making Speed, and Collaboration

There are many great business reasons to focus on improving your operations.   A few of the best include increasing process visibility, increasing speed of decision making, and improving collaboration across the board.

Via Leading Digital:

“The business drivers of operational visions include efficiency and the need to integrate disparate operations.  Executives may want to increase process visibility and decision making speed or to collaborate across silos.”

Proctor & Gamble Reenvisions Operational Excellence

Proctor and Gamble changed their game by focusing on operational excellence.  The key was to be able to manage the business in real time so they could keep up with their ever-changing world.

Via Leading Digital:

“For instance, in 2011, Proctor & Gamble put operational excellence at the center of its digital vision: 'Digitizing P&G will enable us to manage the business in real time and on a very demand-driven basis.  We'll be able to collaborate more effectively and efficiently, inside and outside the company.'  Other companies in industries from banking to manufacturing, have transformed themselves through similar operationally focused visions.”

Operational Visions are Key to Businesses that Sell to Other Businesses

If your business is a provider of products or services to other businesses, then your operational vision is especially important as it can have a ripple effect on what your customers do.

Via Leading Digital:

“Operational visions are especially useful for businesses that sell largely to other businesses.  When Codelco first launched its Codelco Digital initiative, the aim was to improve mining operations radically through automation and data integration.  As we described in chapter 3, Codelco continued to extend this vision to include new mining automation and integration operations-control capability.  Now, executives are envisioning radical new ways to redefine the mining process and possibly the industry itself.”

Operational Visions Can Change the Industry

When you change your operations, you can change the industry.

Via Leading Digital:

“The operational visions of some companies go beyond an internal perspective to consider how the company might change operations in its industry or even with its customers.“

Changes to Operations Can Enable Customers to Change Their Own Operations

When you improve your operations,  you can help others move up the solution stack.

Via Leading Digital:

“For example, aircraft manufacturer Boeing envisions how changes to its products may enable customers to change their own operations.  'Boeing believes the future of the aviation industry lie in 'the digital airline,' the company explained on its website. 'To succeed in the marketplace, airlines and their engineering and IT teams must take advantage of the increasing amount of data coming off of airplanes, using advanced analytics and airplane technology to take operational efficiency to the next level.' “

Get Information to the People Who Need it Most, When They Need It Most

One of the best things you can do when you improve operations is to put the information in the hands of the people that need it most, when they need it most, where they need it most.

Via Leading Digital:

“The manufacturer goes on to paint a clear picture of what a digital airline means in practice: 'The key to to the digital airline is delivering secure, detailed operational and maintenance information to the people who need it most, when they need it most.  That means that engineering will share data with IT, but also with the finance, accounting, operational and executive functions.' “

Better Operations Enables New Product Designs and Services

When you improve operations, you enable and empower business breakthroughs in all parts of the business.

Via Leading Digital:

“The vision will improve operations at Boeing's customers, but will also help Boeing's operations as the information from airplanes should help the company identify new ways to improve its product designs and services.  The day may also lead to new business models as Boeing uses the information to provide new services to customers.”

When you create your digital vision, while there are lots of places you could start, the key is to take an end-to-end view.

If your financial performance is tied to your core operations and your supply chain, and/or you are a provider of products and services to others, then consider starting your business transformation by reenvisioning your operations.

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

The Future of Jobs

Management Innovation is at the Top of the Innovation Stack

Reenvision Your Customer Experience

Categories: Architecture, Programming

iOS localization tricks for Storyboard and NIB files

Xebia Blog - Fri, 10/31/2014 - 23:36

Localization in iOS from Interface Builder designed UI has never been without any problems. The right way of doing localization is by having multiple Strings files. Duplicating Nib or Storyboard files and then changing the language is not an acceptable method. Luckily Xcode 5 has improved this for Storyboards by introducing Base Localization, but I've personally come across several situations where this didn't work at all or when it seemed buggy. Also Nib (Xib) files without ViewController don't support it.

In this post I'll show a couple of tricks that can help with the Localization of Storyboard and Nib files.

Localized subclasses

When you use this method, you create specialized subclasses of view classes that handle the localization in the awakeFromNib() method. This method is called for each view that is loaded from a Storyboard or Nib and all properties that you've set in Interface Builder will be set already.

For UILabels, this means getting the text property, localizing it and setting the text property again.

Using Swift, you can create a single file (e.g. LocalizationView.swift) in your project and put all your subclasses there. Then add the following code for the UILabel subclass:

class LocalizedLabel : UILabel {
    override func awakeFromNib() {
        if let text = text {
            self.text = NSLocalizedString(text, comment: "")
        }
    }
}

Now you can drag a label onto your Storyboard and fill in the text in your base language as you would normally. Then change the Class to LocalizedLabel and it will get the actual label from you Localizable.strings file.

Screen Shot 2014-10-31 at 22.45.46

Screen Shot 2014-10-31 at 22.46.56

No need to make any outlets or write any code to change it!

You can do something similar for UIButtons, even though they don't have a single property for the text on a button.

class LocalizedButton : UIButton {
    override func awakeFromNib() {
        for state in [UIControlState.Normal, UIControlState.Highlighted, UIControlState.Selected, UIControlState.Disabled] {
            if let title = titleForState(state) {
                setTitle(NSLocalizedString(title, comment: ""), forState: state)
            }
        }
    }
}

This will even allow you to set different labels for the different states like Normal and Highlighted.

User Defined Runtime Attributes

Another way is to use the User Defined Runtime Attributes. This method requires slightly more work, but has two small advantages:

  1. You don't need to use subclasses. This is nice when you already use another custom subclass for your labels, buttons and other view classes.
  2. Your keys in the Strings file and texts that show up in the Storyboard don't need to be the same. This works well when you use localization keys such as myCoolTableViewController.header.subtitle. It doesn't look very nice to see those everywhere in your Interface Builder labels and buttons.

So how does this work? Instead of creating a subclass, you instead add a computed property to an existing view class. For UILabels you use the following code:

extension UILabel {

    var localizedText: String {
        set (key) {
            text = NSLocalizedString(key, comment: "")
        }
        get {
            return text!
        }
    }

}

Now you can add a User Defined Runtime Attribute with the key localizedText to your UILabel and have the Localization key as its value.

Screen Shot 2014-10-31 at 23.05.18

Screen Shot 2014-10-31 at 23.06.16

Also here if you want to make this work for buttons, it becomes slightly more complicated. You will have to add a property for each state that needs a label.

extension UIButton {
    var localizedTitleForNormal: String {
        set (key) {
            setTitle(NSLocalizedString(key, comment: ""), forState: .Normal)
        }
        get {
            return titleForState(.Normal)!
        }
    }

    var localizedTitleForHighlighted: String {
        set (key) {
            setTitle(NSLocalizedString(key, comment: ""), forState: .Highlighted)
        }
        get {
            return titleForState(.Highlighted)!
        }
    }
}
Conclusion

Always try and pick the best solution for your problem. Use Storyboard Base Localization if that works well for you. If it doesn't, use the approach with subclasses if you don't need to use another subclass and if you don't care about using your base location strings as localization keys. Else, use the last approach with User Defined Runtime Attributes.

Stuff The Internet Says On Scalability For October 31st, 2014

Hey, it's HighScalability time:


A CT scanner without its clothes on. Sexy.
  • 255Tbps: all of the internet’s traffic on a single fiber; 864 million: daily Facebook users
  • Quotable Quotes:
    • @chr1sa: "No dominant platform-level software has emerged in the last 10 years in closed-source, proprietary form”
    • @joegravett: Homophobes boycotting Apple because of Tim Cook's brave announcement are going to lose it when they hear about Turing.
    • @CloudOfCaroline: #RICON MySQL revolutionized Scale-out. Why? Because it couldn't Scale-up. Turned a flaw into a benefit - @martenmickos
    • chris dixon: We asked for flying cars and all we got was the entire planet communicating instantly via pocket supercomputers
    • @nitsanw: "In the majority of cases, performance will be programmer bound" - Barker's Law
    • @postwait: @coda @antirez the only thing here worth repeating: we should instead be working with entire distributions (instead of mean or q(0.99))
    • Steve Johnson: inventions didn't come about in a flash of light — the veritable Eureka! moment — but were rather the result of years' worth of innovations happening across vast networks of creative minds.

  • On how Google is its own VC. cromwellian: The ads division is mostly firewalled off from the daily concerns of people developing products at Google. They supply cash to the treasury, people think up cool ideas and try to implement them. It works just like startups, where you don't always know what your business model is going to be. Gmail started as a 20% project, not as a grand plan to create an ad channel. Lots of projects and products at Google have no business model, no revenue model, the company does throw money at projects and "figure it out later" how it'll make money. People like their apps more than the web. Mobile ads are making a lot of money.  

  • Hey mobile, what's for dinner? "The world," says chef Benedict Evans, who has prepared for your pleasure a fine gourmet tasting menu: Presentation: mobile is eating the world. Smart phones are now as powerful as Thor and Hercules combined. Soon everyone will have a smart phone. And when tech is fully adopted, it disappears. 

  • How much bigger is Amazon’s cloud vs. Microsoft and Google?: Amazon’s cloud revenue at more than $4.7 billion this year. TBR pegs Microsoft’s public cloud IaaS revenue at $156 million and Google’s at $66 million. If those estimates are correct than Amazon’s cloud revenue is 30 times bigger than Microsoft’s.

  • Great discussion on the Accidental Tech Podcast (at about 25 minutes in) on how the the time of open APIs has ended. People who made Twitter clients weren't competing with Twitter, they were helping Twitter become who they are today. For Apple, developers add value to their hardware and since Apple makes money off the hardware this is good for Apple, because without apps Apple hardware is way less valuable. With their new developer focus Twitter and developer interests are still not aligned as Twitter is still worried about clients competing with them. Twitter doesn't want to become an infrastructure company because there's no money in it. In the not so distant past services were expected to have an open API, in essence services were acting as free infrastructure, just hoping they would become popular enough that those dependent on the service could be monetized. New services these days generally don't have full open APIs because it's hard to justify as a business case. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Azure: Announcing New Real-time Data Streaming and Data Factory Services

ScottGu's Blog - Scott Guthrie - Fri, 10/31/2014 - 07:39

The last three weeks have been busy ones for Azure.  Two weeks ago we announced a partnership with Docker to enable great container-based development experiences on Linux, Windows Server and Microsoft Azure.

Last week we held our Cloud Day event and announced our new G-Series of Virtual Machines as well as Premium Storage offering.  The G-Series VMs provide the largest VM sizes available in the public cloud today (nearly 2x more memory than the largest AWS offering, and 4x more memory than the largest Google offering).  The new Premium Storage offering (which will work with both our D-series and G-series of VMs) will support up to 32TB of storage per VM, >50,000 IOPS of disk IO per VM, and enable sub-1ms read latency.  Combined they provide an enormous amount of power that enables you to run even bigger and better solutions in the cloud.

Earlier this week, we officially opened our new Azure Australia regions – which are our 18th and 19th Azure regions open for business around the world.  Then at TechEd Europe we announced another round of new features – including the launch of the new Azure MarketPlace, a bunch of great network improvements, our new Batch computing service, general availability of our Azure Automation service and more.

Today, I’m excited to blog about even more new services we have released this week in the Azure Data space.  These include:

  • Event Hubs: is a scalable service for ingesting and storing data from websites, client apps, and IoT sensors.
  • Stream Analytics: is a cost-effective event processing engine that helps uncover real-time insights from event streams.
  • Data Factory: enables better information production by orchestrating and managing diverse data and data movement.

Azure Event Hub is now available in general availability, and the new Azure Stream Analytics and Data Factory services are now in public preview. Event Hubs: Log Millions of events per second in near real time

The Azure Event Hub service is a highly scalable telemetry ingestion service that can log millions of events per second in near real time.  You can use the Event Hub service to collect data/events from any IoT device, from any app (web, mobile, or a backend service), or via feeds like social networks.  We are using it internally within Microsoft to monitor some of our largest online systems.

Once you collect events with Event Hub you can then analyze the data using any real-time analytics system (like Apache Storm or our new Azure Stream Analytics service) and store/transform it into any data storage system (including HDInsight and Hadoop based solutions).

Event Hub is delivered as a managed service on Azure (meaning we run, scale and patch it for you and provide an enterprise SLA).  It delivers:

  • Ability to log millions of events per second in near real time
  • Elastic scaling support with the ability to scale-up/down with no interruption
  • Support for multiple protocols including support for HTTP and AMQP based events
  • Flexible authorization and throttling device policies
  • Time-based event buffering with event order preservation

The pricing model for Event Hubs is very flexible – for just $11/month you can provision a basic Event Hub with guaranteed performance capacity to capture 1 MB/sec of events sent to your Event Hub.  You can then provision as many additional capacity units as you need if your event traffic goes higher. 

Getting Started with Capturing Events

You can create a new Event Hub using the Azure Portal or via the command-line.  Choose New->App Service->Service Bus->Event Hub in the portal to do so:

image

Once created, events can be sent to an Event Hub with either a strongly-typed API (e.g. .NET or Java client library) or by just sending a raw HTTP or AMQP message to the service.  Below is a simple example of how easy it is to log an IoT event to an Event Hub using just a standard HTTP post request.  Notice the Authorization header in the HTTP post – you can use this to optionally enable flexible authentication/authorization for your devices:

POST https://your-namespace.servicebus.windows.net/your-event-hub/messages?timeout=60&api-version=2014-01 HTTP/1.1<?xml:namespace prefix = "o" />

Authorization: SharedAccessSignature sr=your-namespace.servicebus.windows.net&sig=tYu8qdH563Pc96Lky0SFs5PhbGnljF7mLYQwCZmk9M0%3d&se=1403736877&skn=RootManageSharedAccessKey

ContentType: application/atom+xml;type=entry;charset=utf-8

Host: your-namespace.servicebus.windows.net

Content-Length: 42

Expect: 100-continue

 

{ "DeviceId":"dev-01", "Temperature":"37.0" }

Your Event Hub can collect up to millions of messages per second like this, each storing whatever data schema you want within them, and the Event Hubs service will store them in-order for you to later read/consume.

Downstream Event Processing

Once you collect events, you no doubt want to do something with them.  Event Hubs includes an intelligent processing agent that allows for automatic partition management and load distribution across readers.  You can implement any logic you want within readers, and the data sent to the readers is delivered in the order it was sent to the Event Hub.

In addition to supporting the ability for you to write custom Event Readers, we also have two easy ways to work with pre-built stream processing systems: including our new Azure Stream Analytics Service and Apache Storm.  Our new Azure Stream Analytics service supports doing stream processing directly from Event Hubs, and Microsoft has created an Event Hubs Storm Spout for use with Apache Storm clusters.

The below diagram helps express some of the many rich ways you can use Event Hubs to collect and then hand-off events/data for processing:

image

Event Hubs provides a super flexible and cost effective building-block that you can use to collect and process any events or data you can stream to the cloud.  It is very cost effective, and provides the scalability you need to meet any needs.

Learning More about Event Hubs

For more information about Azure Event Hubs, please review the following resources:

Stream Analytics: Distributed Stream Processing Service for Azure

I’m excited to announce the preview our new Azure Stream Analytics service – a fully managed real-time distributed stream computation service that provides low latency, scalable processing of streaming data in the cloud with an enterprise grade SLA. The new Azure Stream Analytics service easily scales from small projects with just a few KB/sec of throughput to a gigabyte/sec or more of streamed data messages/events.  

Our Stream Analytics pricing model enable you to run low throughput streaming workloads continuously at low cost, and enables you to only have to scale up as your business needs increase.  We do this while maintaining built in guarantees of event delivery, and state management for fast recovery which enables mission critical business continuity.

Dramatically Simpler Developer Experience for Stream Processing Data

Stream Analytics supports a SQL-like language that dramatically lowers the bar of the developer expertise required to create a scalable stream processing solution. A developer can simply write a few lines of SQL to do common operations including basic filtering, temporal analysis operations, joining multiple live streams of data with other static data sources, and detecting stream patterns (or lack thereof).

This dramatically reduces the complexity and time it takes to develop, maintain and apply time-sensitive computations on real-time streams of data. Most other streaming solutions available today require you to write complex custom code, but with Azure Stream Analytics you can write simple, declarative and familiar SQL.

Fully Managed Service that is Easy to Setup

With Stream Analytics you can dramatically accelerate how quickly you can derive valuable real time insights and analytics on data from devices, sensors, infrastructure, or applications. With a few clicks in the Azure Portal, you can create a streaming pipeline, configure its inputs and outputs, and provide SQL-like queries to describe the desired stream transformations/analysis you wish to do on the data. Once running, you are able to monitor the scale/speed of your overall streaming pipeline and make adjustments to achieve the desired throughput and latency.

You can create a new Stream Analytics Job in the Azure Portal, by choosing New->Data Services->Stream Analytics:

image

Setup Streaming Data Input

Once created, your first step will be to add a Streaming Data Input.  This allows you to indicate where the data you want to perform stream processing on is coming from.  From within the portal you can choose Inputs->Add An Input to launch a wizard that enables you to specify this:

image

We can use the Azure Event Hub Service to deliver us a stream of data to perform processing on. If you already have an Event Hub created, you can choose it from a list populated in the wizard above.  You will also be asked to specify the format that is being used to serialize incoming event in the Event Hub (e.g. JSON, CSV or Avro formats).

Setup Output Location

The next step to developing our Stream Analytics job is to add a Streaming Output Location.  This will configure where we want the output results of our stream processing pipeline to go.  We can choose to easily output the results to Blob Storage, another Event Hub, or a SQL Database:

image

Note that being able to use another Event Hub as a target provides a powerful way to connect multiple streams into an overall pipeline with multiple steps.

Write Streaming Queries

Now that we have our input and output sources configured, we can now write SQL queries to transform, aggregate and/or correlate the incoming input (or set of inputs in the event of multiple input sources) and output them to our output target.  We can do this within the portal by selecting the QUERY tab at the top.

image

There are a number of interesting queries you can write to processing the incoming stream of data.  For example, in the previous Event Hub section of this blog post I showed how you can use an HTTP POST command to submit JSON based temperature data from an IoT device to an Event Hub with data in JSON format like so:

{ "DeviceId":"dev-01", "Temperature":"37.0" }

When multiple devices are streaming events simultaneously into our Event Hub like this, it would feed into our Stream Analytics job as a stream of continuous data events that look like the sequence below:

Wouldn’t it be interesting to be able to analyze this data using a time-window perspective instead?  For example, it would be useful to calculate in real-time what the average temperature of each device was in the last 5 seconds of multiple readings.

With the Stream Analytics Service we can now dynamically calculate this over our incoming live stream of data just by writing a SQL query like so:

SELECT DateAdd(second,-5,System.TimeStamp) as WinStartTime, system.TimeStamp as WinEndTime, DeviceId, Avg(Temperature) as AvgTemperature, Count(*) as EventCount 
    FROM input
    GROUP BY TumblingWindow(second, 5), DeviceId

Running this query in our Stream Analytics job will aggregate/transform our incoming stream of data events and output data like below into the output source we configured for our job (e,g, a blog storage file or a SQL Database):

The great thing about this approach is that the data is being aggregated/transformed in real time as events are being streamed to us, and it scales to handle literally gigabytes of data event streamed per second.

Scaling your Stream Analytics Job

Once defined, you can easily monitor the activity of your Stream Analytics Jobs in the Azure Portal:

image

You can use the SCALE tab to dynamically increase or decrease scale capacity for your stream processing – allowing you to pay only for the compute capacity you need, and enabling you to handle jobs with gigabytes/sec of streamed data. 

Learning More about Stream Analytics Service

For more information about Stream Analytics, please review the following resources:

Data Factory: Fully managed service to build and manage information production pipelines

Organizations are increasingly looking to fully leverage all of the data available to their business.  As they do so, the data processing landscape is becoming more diverse than ever before – data is being processed across geographic locations, on-premises and cloud, across a wide variety of data types and sources (SQL, NoSQL, Hadoop, etc), and the volume of data needing to be processed is increasing exponentially. Developers today are often left writing large amounts of custom logic to deliver an information production system that can manage and co-ordinate all of this data and processing work.

To help make this process simpler, I’m excited to announce the preview of our new Azure Data Factory service – a fully managed service that makes it easy to compose data storage, processing, and data movement services into streamlined, scalable & reliable data production pipelines. Once a pipeline is deployed, Data Factory enables easy monitoring and management of it, greatly reducing operational costs. 

Easy to Get Started

The Azure Data Factory is a fully managed service. Getting started with Data Factory is simple. With a few clicks in the Azure preview portal, or via our command line operations, a developer can create a new data factory and link it to data and processing resources.  From the new Azure Marketplace in the Azure Preview Portal, choose Data + Analytics –> Data Factory to create a new instance in Azure:

image

Orchestrating Information Production Pipelines across multiple data sources

Data Factory makes it easy to coordinate and manage data sources from a variety of locations – including ones both in the cloud and on-premises.  Support for working with data on-premises inside SQL Server, as well as Azure Blob, Tables, HDInsight Hadoop systems and SQL Databases is included in this week’s preview release. 

Access to on-premises data is supported through a data management gateway that allows for easy configuration and management of secure connections to your on-premises SQL Servers.  Data Factory balances the scale & agility provided by the cloud, Hadoop and non-relational platforms, with the management & monitoring that enterprise systems require to enable information production in a hybrid environment.

Custom Data Processing Activities using Hive, Pig and C#

This week’s preview enables data processing using Hive, Pig and custom C# code activities.  Data Factory activities can be used to clean data, anonymize/mask critical data fields, and transform the data in a wide variety of complex ways.

The Hive and Pig activities can be run on an HDInsight cluster you create, or alternatively you can allow Data Factory to fully manage the Hadoop cluster lifecycle on your behalf.  Simply author your activities, combine them into a pipeline, set an execution schedule and you’re done – no manual Hadoop cluster setup or management required. 

Built-in Information Production Monitoring and Dashboarding

Data Factory also offers an up-to-the moment monitoring dashboard, which means you can deploy your data pipelines and immediately begin to view them as part of your monitoring dashboard.  Once you have created and deployed pipelines to your Data Factory you can quickly assess end-to-end data pipeline health, pinpoint issues, and take corrective action as needed.

Within the Azure Preview Portal, you get a visual layout of all of your pipelines and data inputs and outputs. You can see all the relationships and dependencies of your data pipelines across all of your sources so you always know where data is coming from and where it is going at a glance. We also provide you with a historical accounting of job execution, data production status, and system health in a single monitoring dashboard:

image

Learning More about Stream Analytics Service

For more information about Data Factory, please review the following resources:

Other Great Data Improvements

Today’s releases make it even easier for customers to stream, process and manage the movement of data in the cloud.  Over the last few months we’ve released a bunch of other great data updates as well that make Azure a great platform to perform any data needs.  Since August: 

We released a major update of our SQL Database service, which is a relational database as a service offering.  The new SQL DB editions (Basic/Standard/Premium ) support a 99.99% SLA, larger database sizes, dedicated performance guarantees, point-in-time recovery, new auditing features, and the ability to easily setup active geo-DR support. 

We released a preview of our new DocumentDB service, which is a fully-managed, highly-scalable, NoSQL Document Database service that supports saving and querying JSON based data.  It enables you to linearly scale your document store and scale to any application size.  Microsoft MSN portal recently was rewritten to use it – and stores more than 20TB of data within it.

We released our new Redis Cache service, which is a secure/dedicated Redis cache offering, managed as a service by Microsoft.  Redis is a popular open-source solution that enables high-performance data types, and our Redis Cache service enables you to standup an in-memory cache that can make the performance of any application much faster.

We released major updates to our HDInsight Hadoop service, which is a 100% Apache Hadoop-based service in the cloud. We have also added built-in support for using two popular frameworks in the Hadoop ecosystem: Apache HBase and Apache Storm.

We released a preview of our new Search-As-A-Service offering, which provides a managed search offering based on ElasticSearch that you can easily integrate into any Web or Mobile Application.  It enables you to build search experiences over any data your application uses (including data in SQLDB, DocDB, Hadoop and more).

And we have released a preview of our Machine Learning service, which provides a powerful cloud-based predictive analytics service.  It is designed for both new and experienced data scientists, includes 100s of algorithms from both the open source world and Microsoft Research, and supports writing ML solutions using the popular R open-source language.

You’ll continue to see major data improvements in the months ahead – we have an exciting roadmap of improvements ahead.

Summary

Today’s Microsoft Azure release enables some great new data scenarios, and makes building applications that work with data in the cloud even easier.

If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Microsoft Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu omni

Categories: Architecture, Programming

Driving Business Transformation by Reenvisioning Your Customer Experience

You probably hear a lot about the Mega-Trends (Cloud, Mobile, Social, and Big Data), or the Nexus of Forces (the convergence of social, mobility, cloud and data insights patterns that drive new business scenarios), or the Mega-Trend of Mega-Trends (Internet of Things).

And you are probably hearing a lot about digital transformation and maybe even about the rise of the CDO (Chief Digital Officer.)

All of this digital transformation is about creating business change, driving business outcomes, and driving better business results.

But how do you create your digital vision and strategy?   And, where do you start?

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share some of their lessons learned from companies that are digital masters that created their digital visions and are driving business change.

3 Perspectives of Digital Vision

When it comes to creating your digital vision, you can focus on reenvisioning the customer experience, the operational processes, or your business model.

Via Leading Digital:

“Where should you focus your digital vision? Digital visions usually take one of three perspectives: reenvisioning the customer experience, reenvisioning operational processes, or combining the previous two approaches to reenvision business models.  The approach you take should reflect your organization’s capabilities, your customer’s needs, and the nature of competition in your industry.”

Start with Your Customer Experience

One of the best places to start is with your customer experience.  After all, a business exists to create a customer.  And the success of the business is how well it creates value and serves the needs of the customer.

Via Leading Digital:

“Many organizations start by reenvisioning the way they interact with customers.  They want to make themselves easier to work with, and they want to be smarter in how they sell to (and serve) customers.  Companies start from different places when reenvisioning the customer experience.”

Transform the Relationship

You can use the waves of technologies (Cloud, Mobile, Social, Data Insights, and Internet of Things), to transform how you interact with your customers and how they experience your people, your products, and your services.

Via Leading Digital:

“Some companies aim to transform their relationships with their customers.  Adam Bortman, chief digital officer of Starbucks, shared this vision: 'Digital has to help more partners and help the company be the way we can ... tell our story, build our brand, and have a relationship with our customers.' Burberry's CEO Angela Ahredts focused on multichannel coherence. 'We had a vision, and the vision was to be the first company who was fully digital end-to-end ... A customer will have total access to Burberry across any device, anywhere.'  Mare Menesquen, managing director of strategic marketing at cosmetics gitan L'Oreal, said, 'The digital world multiples the way our brands can create an emotion-filled relationship with their customers.’”

Serve Your Customers in Smarter Ways

You can use technology to personalize the experience for your customers, and create better interactions along the customer experience journey.

Via Leading Digital:

“Other companies envision how they can be smarter in serving (and selling to) their customers through analytics.  Caesars started with a vision of using real-time customer information to deliver a personalized experience to each customer.  The company was able to increase customer satisfaction and profits per customer using traditional technologies.  Then, as new technologies arose, it extended the vision to include a mobile, location-based concierge in the palm of every customer's hand.”

Learn from Customer Behavior

One of the most powerful things you can now do with the combination of Cloud, Mobile, Social, Big Data and Internet of Things is gain better customer insights.  For example, you can learn from the wealth of social media insights, or you can learn through better integration and analytics of your existing customer data.

Via Leading Digital:

“Another approach is to envision how digital tools might help the company to learn from customer behavior.  Commonwealth Bank of Australia sees new technologies as a key way of integrating customer inputs in its co-creation efforts.  According to CIO Ian Narev, 'We are progressively applying new technology to enable customers to play a greater part in product design.  That helps us create more intuitive products and services, readily understandable to our customers and more tailored to their individual needs.”

Change Customers’ Lives

If you focus on high-value activities, you can create breakthroughs in the daily lives of your customers.

Via Leading Digital:

“Finally, some companies are extending their visions beyond influencing customer experience to actually changing customers' lives.  For instance, Novartis CEO Joseph Jimenez wrote of this potential: ‘The technologies we use in our daily lives, such as smart phones and tablet devices, could make a real difference in helping patients to manage their own health.  We are exploring ways to use these tools to improve compliance rates and enable health-care professionals to monitor patient progress remotely.’”

If you want to change the world, one of the best places to start is right from wherever you are.

With a Cloud and a dream, what can you do to change the world?

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

The Future of Jobs

Management Innovation is at the Top of the Innovation Stack

McKinsey on Unleashing the Value of Big Data

Categories: Architecture, Programming

How to Dockerize your Dropwizard Application

Xebia Blog - Wed, 10/29/2014 - 10:47

If you want to deploy your Dropwizard Application on a Docker server, you can Dockerize your Dropwizard Application. Since a Dropwizard Application is already packaged as an executable Java ARchive file, creating a Docker image for such an application should be easy.

 

In this blog, you will learn how to Dockerize a Dropwizard Application using 4 easy steps.

Before you start

  • You are going to use the Dropwizard-example application, which can be found at the Dropwizard GitHub repository.
  • Additionally you need Docker. I used Boot2Docker to run the Dockerized Dropwizard Application on my laptop. If you use boot2Docker, you may need this Boot2Docker workaround to access your Dockerized Dropwizard application.
  • This blog does not describe how to create Dropwizard applications. The Dropwizard getting started guide provides an excellent starting point if you like to know more about building your own Dropwizard applications.

 

Step 1: create a Dockerfile

You can start with creating a Dockerfile. Docker can automatically build images by reading the instructions described in this file. Your Dockerfile could look like this:

FROM dockerfile/java:openjdk-7-jdk

ADD dropwizard-example-1.0.0.jar /data/dropwizard-example-1.0.0.jar

ADD example.keystore /data/example.keystore

ADD example.yml /data/example.yml

RUN java -jar dropwizard-example-1.0.0.jar db migrate /data/example.yml

CMD java -jar dropwizard-example-1.0.0.jar server /data/example.yml

EXPOSE 8080

 

The Dropwizard Application needs a Java Runtime, so you can start from an base image already available at Docker Hub, for example: dockerfile/java:openjdk-7-jdk.

You must add the Dropwizard Application files to the image, using the ADD instruction in your Dockerfile.

Next, simply specify the commands of your Dropwizard Application, which you want to execute during image build and container runtime. In the example above, the db migrate command is executed when the Docker image is build and the server command is executed when you issue a Docker run command to create a running container.

Finally, the EXPOSE instruction tells Docker that your container will listen on the specified port(s) at runtime.

 

Step 2: build the Docker image

Place the Dockerfile and your application files in a directory and execute the Docker build command to build an Docker image.

docker@boot2docker:~$ docker build -t dropwizard/dropwizard-example ~/dropwizard/

 

In the console output you should be able to that the Dropwizard Application db migrate command is executed. If everything is ok, the last line reported informs you that the image is successfully build.

Successfully built dd547483b57b

 

Step 3: run the Docker image

Use the Docker run command to create a container based on the image you have created. If you need to find your image id use the Docker images command to list your images. It should take around 3 seconds to start the Dockerized Dropwizard example application.

Docker run –p 8080:8080 dd547483b57b

Notice that I included the –p option to include a network port binding, which maps 8080 inside the container to port 8080 on the Docker host.  You can verify whether your container is running using the docker ps command.

docker@boot2docker:~$ docker ps

CONTAINER ID        IMAGE                                  COMMAND                CREATED             STATUS              PORTS                    NAMES

3b6fb75adad6        dropwizard/dropwizard-example:latest   "/bin/sh -c 'java -j   3 minutes ago       Up 3 minutes        0.0.0.0:8080->8080/tcp   high_turing

 

  1. Test the application

Now the application is ready for use. You can access the application using your Docker host ip address and the forward port 8080. For example, use the Google Advanced Rest Client App to register “John Doe”.

GoogleRestClient

Sponsored Post: Apple, TokuMX, Hypertable, VSCO, Gannett, Sprout Social, Scalyr, FoundationDB, AiScaler, Aerospike, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Apple has multiple openings. Changing the world is all in a day's work at Apple. Imagine what you could do here. 
    • Site Reliability Engineer. As a member of the Apple Pay SRE team, you’re expected to not just find the issues, but to write code and fix them. You’ll be involved in all phases and layers of the application, and you’ll have a direct impact on the experience of millions of customers. Please apply here.
    • Software Engineering Manager. In this role, you will be communicating extensively with business teams across different organizations, development teams, support teams, infrastructure teams and management. You will also be responsible for working with cross-functional teams to delivery large initiatives. Please apply here
    • Sr. Software Developer. We are looking for a solid senior-level Java/C programmer who will be working on security software development. This software will provide the data protection, integrity, and service authentication services for iOS devices. Please apply here.
    • DevOps Software Engineer - Apple Pay, iOS Systems.  The iOS Systems team is looking for an outstanding DevOps software engineer to help make our rapidly growing platform manageable, scalable, and reliable using state of the art technologies and cutting edge system automation. Come join the team to strategize, architect, and build infrastructure to help our systems perform and scale. Please apply here

  • VSCO. Do you want to: ship the best digital tools and services for modern creatives at VSCO? Build next-generation operations with Ansible, Consul, Docker, and Vagrant? Autoscale AWS infrastructure to multiple Regions? Unify metrics, monitoring, and scaling? Build self-service tools for engineering teams? Let’s talk about working together. vs.co/careers.

  • Gannett Digital is looking for talented Front-end developers with strong Python/Django experience to join their Development & Integrations team. The team focuses on video, user generated content, API integrations and cross-site features for Gannett Digital’s platform that powers sites such as http://www.usatoday.com, http://www.wbir.com or http://www.democratandchronicle.com. Please apply here.

  • Platform Software Engineer, Sprout Social, builds world-class social media management software designed and built for performance, scale, reliability and product agility. We pick the right tool for the job while being pragmatic and scrappy. Services are built in Python and Java using technologies like Cassandra and Hadoop, HBase and Redis, Storm and Finagle. At the moment we’re staring down a rapidly growing 20TB Hadoop cluster and about the same amount stored in MySQL and Cassandra. We have a lot of data and we want people hungry to work at scale. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Sign Up for New Aerospike Training Courses.  Aerospike now offers two certified training courses; Aerospike for Developers and Aerospike for Administrators & Operators, to help you get the most out of your deployment.  Find a training course near you. http://www.aerospike.com/aerospike-training/

  • November TokuMX Meetups for Those Interested in MongoDB. Join us in one of the following cities in November to learn more about TokuMX and hear TokuMX use cases. 11/5 - London;11/11 - San Jose; 11/12 - San Francisco. Not able to get to these cities? Check out our website for other upcoming Tokutek events in your area - www.tokutek.com/events.
Cool Products and Services
  • Hypertable Inc. Announces New UpTime Support Subscription Packages. The developer of Hypertable, an open-source, high-performance, massively scalable database, announces three new UpTime support subscription packages – Premium 24/7, Enterprise 24/7 and Basic. 24/7/365 support packages start at just $1995 per month for a ten node cluster -- $49.95 per machine, per month thereafter. For more information visit us on the Web at http://www.hypertable.com/. Connect with Hypertable: @hypertable--Blog.

  • FoundationDB launches SQL Layer. SQL Layer is an ANSI SQL engine that stores its data in the FoundationDB Key-Value Store, inheriting its exceptional properties like automatic fault tolerance and scalability. It is best suited for operational (OLTP) applications with high concurrency. Users of the Key Value store will have free access to SQL Layer. SQL Layer is also open source, you can get started with it on GitHub as well.

  • Diagnose server issues from a single tab. Scalyr replaces all your monitoring and log management services with one, so you can pinpoint and resolve issues without juggling multiple tools and tabs. Engineers say it's powerful and easy to use. Customer support teams use it to troubleshoot user issues. CTO's consider it a smart alternative to Splunk, with enterprise-grade functionality, sane pricing, and human support. Trusted by in-the-know companies like Codecademy – learn more!

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Azure: New Marketplace, Network Improvements, New Batch Service, Automation Service, more

ScottGu's Blog - Scott Guthrie - Tue, 10/28/2014 - 15:35

Today we released a major set of updates to Microsoft Azure. Today’s updates include:

  • Marketplace: Announcing Azure Marketplace and partnerships with key technology partners
  • Networking: Network Security Groups, Multi-NIC, Forced Tunneling, Source IP Affinity, and much more
  • Batch Computing: Public Preview of the new Azure Batch Computing Service
  • Automation: General Availability of the Azure Automation Service
  • Anti-malware: General Availability of Microsoft Anti-malware for Virtual Machines and Cloud Services
  • Virtual Machines: General Availability of many more VM extensions – PowerShell DSC, Octopus, VS Release Management

All of these improvements are now available to use immediately (note that some features are still in preview).  Below are more details about them: Marketplace: Announcing Azure Marketplace and partnerships with key technology partners

Last week, at our Cloud Day event in San Francisco, I announced a new Azure Marketplace that helps to better connect Azure customers with partners, ISVs and startups.  With just a couple of clicks, you can now quickly discover, purchase, and deploy any number of solutions directly into Azure.

Exploring the Marketplace

You can explore the Azure Marketplace by clicking the Marketplace title that is pinned by default to the home-screen of the Azure Preview Portal:

image

Clicking the Marketplace tile will enable you to explore a large selection of applications, VM images, and services that you can provision into your Azure subscription:

image

Using the marketplace provides a super easy way to take advantage of a rich ecosystem of applications and services integrated to run great with Azure.  Today’s marketplace release includes multi-VM templates to run Hadoop clusters powered by Cloudera or Hortenworks, Linux VMs powered by Unbuntu, CoreOS, Suse, CentOS, Microsoft SharePoint Server Farms, Cassandra Clusters powered by DataStax, and a wide range of security virtual appliances.

You can click any of the items in the gallery to learn more about them and optionally deploy them.  Doing so will walk you though a simple to follow creation wizard that enables you to optionally configure how/where they will run, as well as display any additional pricing required for the apps/services/VM images that you select.

For example, below is all it takes to stand-up an 8-node DataStax Enterprise cluster:

image

Solutions you purchase through the Marketplace will be automatically billed to your Azure subscription (avoiding the need for you to setup a separate payment method).  Virtual Machine images will support the ability to bring your own license or rent the image license by the hour (which is ideal for proof of concept solutions or cases where you need the solution for only a short period of time).  Both Azure Direct customers as well as customers who pay using an Enterprise Agreement can take advantage of the Azure Marketplace starting today.

You can learn more about the Azure Marketplace as well as browse the items within it here.

Networking: Lots and lots of New Features and Improvements

This week’s Azure update includes a ton of new capabilities to the Azure networking stack.  You can use these new networking capabilities immediately in the North Europe region, and they will be supported worldwide in all regions in November 2014.  The new network capabilities include:

Network Security Groups

You can now create Network Security groups to define access control rules for inbound and outbound traffic to a Virtual machine or a group of virtual machines in a subnet. The security groups and the rules can be managed and updated independent of the life cycle of the VM.

Multi-NIC Support

You can now create and manage multiple virtual network interfaces (NICs) on a VM.  Multi-NIC support is a fundamental requirement for a majority of network virtual appliances that can be deployed in Azure. Having this support now enabled within Azure will enable even richer network virtual appliances to be used.

Forced Tunneling

You can now redirect or “force” all Internet-bound traffic that originates in a cloud application back through an on-premises network via a Site-to-Site VPN tunnel for inspection and auditing. This is a critical security capability for enterprise grade applications.

ExpressRoute Enhancements

You can now share a single ExpressRoute connection across multiple Azure subscriptions. Additionally, a single Virtual Network in Azure can now be linked to more than one ExpressRoute circuit, thereby enabling much richer backup and disaster recovery scenarios.

image

New VPN Gateway Sizes

To cater to the growing hybrid connectivity throughput needs and the number of cross premise sites, we are announcing the availability of a higher performance Azure VPN gateway. This will enable a faster ExpressRoute and Site-to-Site VPN gateways with more tunnels.

Operations and audit logs for VNet Gateways and ExpressRoute

You can now view operations logs for Virtual Network Gateways and ExpressRoute circuits. The Azure portal will now show operations logs and information on all API calls you make and important infrastructure changes made such as scheduled updates to gateways.

Advanced Virtual Network Gateway policies

We now enable the ability for you to control encryption for the tunnel between Virtual Networks. You now have a choice between 3DES, AES128, AES256 and Null encryption, and you can also enable Perfect Forward Secrecy (PFS) for IPsec/IKE gateways.

Source IP Affinity

The Azure Load Balancer now supports a new distribution mode called Source IP Affinity (also known as session affinity or client IP affinity). You can now load balance traffic based on a 2-tuple (Source-IP, Destination-IP) or 3-tuple (Source-IP, Destination-IP and Protocol) distribution modes.

Nested policies for Traffic Manager

You can now create nested policies for traffic management. This allows tremendous flexibility in creating powerful load-balancing and failover schemes to support the needs of larger, more complex deployments.

Portal Support for Managing Internal Load Balancer, Reserved and Instance IP addresses for Virtual Machines

It is now possible to use the Azure Preview Portal to manage creating and setting up internal load balancers, as well as reserved and instance IP addresses for virtual machines.

Automation: General Availability of Azure Automation Service

I am excited to announce the General Availability of the Azure Automation service. Azure Automation enables the creation, deployment, monitoring, and maintenance of resources in an Azure environment using a highly scalable and reliable workflow engine. The service can be used to orchestrate time-consuming and frequently repeated operational tasks across Azure and third-party systems while decreasing operating expenses.

Azure Automation allows you to build runbooks (PowerShell Workflows) to describe your administration processes, provides a secure global assets store so you don’t need to hardcode sensitive information within your runbooks, and offers scheduling so that runbooks can be triggered automatically.

Runbooks can automate a wide range of scenarios – from simple day to day manual tasks to complex processes that span multiple Azure services and 3rd party systems. Because Automation is built on PowerShell, you can take advantage of the many existing PowerShell modules, or author your own to integrate with third party systems.

Creating and Editing Runbooks

You can create a runbook from scratch, or start by importing an existing template in the runbook gallery:

image

Editing experience for runbooks can also be performed directly in the administration portal:

image

Pricing

Available as a pay-as-you-go service, Automation is billed based on the number of job run time minutes used in a given Azure subscription.  500 minutes of free job runtime credits are also included each month for Azure customers to use at no charge.

Learn More

To learn more about Azure Automation, check out the following resources:

Batch Service: Preview of Azure Batch - new job scheduling service for parallel and HPC apps

I’m excited to announce the public preview of our new Azure Batch Service. This new platform service provides “job scheduling as a service” with auto-scaling of compute resources, making it easy to run large-scale parallel and high performance computing (HPC) work in Azure. You submit jobs, we start the VMs, run your tasks, handle any failures, and then shut things down as work completes.

Azure Batch is the job scheduling engine that we use internally to manage encoding for Azure Media Services, and for testing Azure itself. With this preview, we are excited to expand our SDK with a new application framework from GreenButton, a company Microsoft acquired earlier in the year. The Azure Batch SDK makes it easy to cloud-enable parallel, cluster, and HPC applications by describing jobs with the required resources, data, and one or more compute tasks.

Azure Batch can be used to run large volumes of similar tasks or applications in parallel, programmatically. A command line program or script takes a set of data files as input, processes the data in a series of tasks, and produces a set of output files. Examples of batch workloads that customers are running today in Azure include calculating risk for banks and insurance companies, designing new consumer and industrial products, sequencing genes and developing new drugs, searching for new energy sources, rendering 3D animations, and transcoding video.

Azure Batch makes it easy for these customers to use hundreds, thousands, tens of thousands of cores, or more on demand. With job scheduling as a service, Azure developers can focus on using batch computing in their applications and delivering services without needing to build and manage a work queue, scaling resources up and down efficiently, dispatching tasks, and handling failures.

image

The scale of Azure helps batch computing customers get their work done faster, experiment with different designs, run larger and more precise models, and test a large number of different scenarios without having to invest in and maintain large clusters.

Learn more about Azure Batch and start using it for your applications today. Virtual Machines: General Availability of Microsoft Anti-Malware for VMs and Cloud Services

I’m excited to announce that the Microsoft Anti-malware security extension for Virtual Machines and Cloud Services is now generally available.  We are releasing it as a free capability that you can use at no additional charge.

The Microsoft Anti-malware security extension can be used to help identify and remove viruses, spyware or other malicious software.  It provides real-time protection from the latest threats and also supports on-demand scheduled scanning.  Enabling it is a good security best practice for applications hosted either on-premises or in the cloud.

Enabling the Anti-Malware Extension

You can select and configure the Microsoft Antimalware security extension for virtual machines using the Azure preview portal, Visual Studio or API’s/PowerShell.  Antimalware events are then logged to the customer configured Azure Storage account via Azure Diagnostics and can be piped to HDInsight or a SIEM tool for further analysis. More information is available in the Microsoft Antimalware Whitepaper.

To enable antimalware feature on existing virtual machine, select the EXTENSIONS tile on a Virtual Machine in the Azure Preview Portal, then click ADD in the command bar and select the Microsoft Antimalware extension. Then, click CREATE and customize any settings:

image Virtual Machines: General Availability of even more VM Extensions

In addition to enabling the Microsoft Anti-Malware extension for Virtual Machines, today’s release also includes support for a whole bunch more new VM extensions that you can enable within your Virtual Machines.  These extensions can be added and configured using the same EXTENSIONS tile on Virtual Machine resources within the Azure Preview Portal (the same screen-shot as in the Anti-malware section above).

The new extensions enabled today include:

PowerShell Desired State Configuration

The PowerShell Desired State Configuration Extension can be used to deploy and configure Azure VMs using Desired State Configuration (DSC) technology. DSC enables you to declaratively specify how you want your software environment to be configured. DSC configuration can also be automated using the Azure PowerShell SDK, and you can push configurations to any Azure VM and have them enacted automatically. For more details, please see this desired state configuration blog post.

image 

Octopus

Octopus simplifies the deployment of ASP.NET web applications, Windows Services and other applications by automatically configuring IIS, installing services and making configuration changes. Octopus integration of Azure was one of the top requested features on Azure UserVoice and with this integration we will simplify the deployment and configuration of octopus on the VM.

image

Visual Studio Release Management

Release Management for Visual Studio is a continuous delivery solution that automates the release process through all of your environments from TFS through to production. Visual Studio Release Management is integrated with TFS and you can configure multi-stage release pipelines to automatically deploy and validate your applications on multiple environments. With the new Visual Studio Release Management extension, VMs can be preconfigured with the necessary components for required for Release Management to operate.

image Summary

Today’s Microsoft Azure release enables a ton of great new scenarios, and makes building applications hosted in the cloud even easier.

If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Microsoft Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu omni

Categories: Architecture, Programming

Sketching API Connections

Coding the Architecture - Simon Brown - Tue, 10/28/2014 - 14:52

Daniel Bryant, Simon and I recently had a discussion about how to represent system communication with external APIs. The requirement for integration with external APIs is now extremely common but it's not immediately obvious how to clearly show them in architectural diagrams.


How to Represent an External System?

The first thing we discussed was what symbol to use for a system supplying an API. Traditionally, UML has used the Actor (stick man) symbol to represent a "user or any other system that interacts with the subject" (UML Superstructure Specification, v2.1.2). Therefore a system providing an API may look like this:


Actor


I've found that this symbol tends to confuse those who aren't well versed in UML as most people assume that the Actor symbol always represents a *person* rather than a system. Sometimes this is stereotyped to make it more obvious e.g.


Actor with stereotype


However the symbol is very powerful and tends to overpower the stereotype. Therefore I prefer to use a stereotyped box for an external system supplying an API. Let's compare two context diagrams using Boxes vs Stick Actors.


Actor

In which diagram is it more obvious what are systems or people?

Note that ArchiMate has a specific symbol for Application Service that can be used to represent an API:


Actor (Application Service notation from the Open Group's ArchiMate 2.1 Specification)


An API or the System that Supplies it?

Whatever symbol we choose, what we've done is to show the *system* rather than the actual API. The API is a definition of a service provided by the system in question. How should we provide more details about the API?

There are a number of ways we could do this but my preference is to give details of the API on the connector (line connecting two elements/boxes). In C4 the guidelines for a container diagram includes listing protocol information on the connector and an API can be viewed as the layer above the protocol. For example:


Actor


Multiple APIs per External System

Many API providers supply multiple services/APIs (I'm not referring to different operations within an API but multiple sets of operations in different APIs, which may even use different underlying protocols.) For example a financial marketplace may have APIs that do the following:

  1. Allow a bulk, batch download of static data (such as details of companies listed on a stock market) via xml over HTTP.
  2. Supply real time, low latency updates of market prices via bespoke messages over UDP.
  3. Allow entry of trades via industry standard FPML over a queuing system.
  4. Supply a bulk, batch download of trades for end-of-day reconciliation via FPML over HTTP.

Two of the services use the same protocol (xml over HTTP) but have very different content and use. One of the APIs is used to constantly supply information after user subscription (market data) and the last service involves the user supplying all the information with no acknowledgment (although it should reconcile at EOD).

There are multiple ways of showing this. We could:

  1. Have a single service element, list the APIs on it and have all components linking to it.
  2. Show each service/API as a separate box and connect the components that use the individual service to the relevant box.
  3. Show a single service element with multiple connections. Each connection is labeled and represents an API.
  4. Use a Port and Connector style notation to represent each API from the service provider. Provide a key for the ports.
  5. Use a UML style 'cup and ball' notation to define interfaces and their usage.

Some examples are below:


A Single Service element and simple description


Actor


In the above diagram the containers are stating what they are using but contain no information about how to use the APIs. We don't know if it is a single API (with different operations) or anything about the mechanisms used to transport the data. This isn't very useful for anyone implementing a solution or resolving operational issues.


Single, Service box with descriptive connectors


Actor

In this diagram there is a single, service box with descriptive connectors. The above diagram shows all the information so is much more useful as a diagnostic or implementation tool. However it does look quite crowded.


Services/APIs shown as separate boxes


Actor


Here the external system has its services/APIs shown as separate boxes. This contains all the information but might be mistaken as defining the internal structure of the external system. We want to show the services it provides but we know nothing about the internal structure.


Using Ports to Represent APIs


Actor with


In the above diagram the services/APIs are shown as 'ports' on the external system and the details have been moved into a separate key/table. This is less likely to be mistaken as showing any internal structure of the external service. (Note that I could have also shown outgoing rPorts from the Brokerage System.)


UML Interfaces


Actor

This final diagram is using a UML style interface provider and requirer. This is a clean diagram but requires the user to be aware of what the cup and ball means (although I could have explained this in the key).


Conclusion

Any of these solutions could be appropriate depending on the complexity of the API set you are trying to represent. I'd suggest starting with a simple representation (i.e. fully labeled connections) and moving to a more complex one if needed BUT remember to use a key to explain any elements you use!

Categories: Architecture

Software architecture vs code (DevDay 2014)

Coding the Architecture - Simon Brown - Mon, 10/27/2014 - 19:10

I had the pleasure of delivering the closing keynote at the DevDay 2014 conference in Krakow, Poland last month. It's a one day event, with a bias towards the .NET platform, and one of my favourite conferences from this year. Beautiful city, fantastic crowd and top-notch hospitality. If you get the chance to attend next year, do it!

If you missed it, you can find videos of the talks to watch online. Here's mine called Software architecture vs code. It covers the conflict between software architecture and code, how we can resolve this, the benefits of doing so, fishing and a call for donations to charity every time you write public class without thinking. Enjoy!

p.s. I've written about some of these same topics on the blog ... for example, Modularity and testability and Software architecture vs code. My Structurizr project is starting to put some of this into practice too.

Categories: Architecture

Microservices in Production - the Good, the Bad, the it Works

This is a guest repost written by Andrew Harmel-Law on his real world experiences with Microservices. The original article can be found here.

It’s reached the point where it’s even a cliche to state “there’s a lot written about Microservices these days.” But despite this, here’s another post on the topic. Why does the internet need another? Please bear with me…

We’re doing Microservices. We’re doing it based on a mash-up of some “Netflix Cloud” (as it seems to becoming known - we just call it “Archaius / Hystrix”), a gloop of Codahale Metrics, a splash of Spring Boot, and a lot of Camel, gluing everything together. We’ve even found time to make a bit of Open Source ourselves - archaius-spring-adapter - and also contribute some stuff back.

Lets be clear; when I say we’re “doing Microservices”, I mean we’ve got some running; today; under load; in our Production environment. And they’re running nicely. We’ve also got a lot more coming down the dev-pipe.

All the time we’ve been crafting these we’ve been doing our homework. We’ve followed the great debate, some contributions of which came from within Capgemini itself, and other less-high-profile contributions from our very own manager. It’s been clear for a while that, while there is a lot of heat and light generated in this debate, there is also a lot of valid inputs that we should be bearing in mind.

Despite this, the Microservices architectural style is still definitely in the honeymoon period, which translates personally into the following: whenever I see a new post on the topic from a Developer I respect my heart sinks a little as I open it and read… Have they discovered the fatal flaw in all of this that everyone else has so far missed? Have they put their finger on the unique aspect that mean 99% of us will never realise the benefits of this new approach and that we’re all off on a wild goose chase? Have they proven that Netflix really are unicorns and that the rest of us are just dreaming?

Despite all this we’re persisting. Despite always questioning every decision we make in this area far more than we normally would, Microservices still feel right to us for a whole host of reasons. In the rest of this post I hope I’ll be able to point out some of the subtleties which might have eluded you as you’ve researched and fiddled, and also, I’ve aimed to highlight some of the old “givens” which might not be “givens” any more.

The Good
Categories: Architecture

How to create Java microservices with Dropwizard

Xebia Blog - Mon, 10/27/2014 - 15:10

On Tuesday October 14th the Amsterdam Middleware Meetup experimented with Dropwizard. The idea was to find out what this technology is about, where it could be useful and what the alternatives are. So below I’ll give you an overview of Dropwizard and compare it to Spring Boot.
The Dropwizard website claims:

Dropwizard pulls together stable, mature libraries from the Java ecosystem into a simple, light-weight package that lets you focus on getting things done.

I’ll discuss each of these claims below.

Stable and mature
Dropwizard uses Jetty, Jersey, Jackson and Metrics as its most important frameworks, but also a host of other stuff like Guava, Liquibase and Joda Time. The latest Dropwizard release is version 0.7.1, released on June 20th 2014. It depends on these versions of some core libraries:
Jetty - 9.2.3.v20140905 - May 2014
Jackson - 2.4.1 - June 2014
Jersey - 2.11 - July 2014

The table shows that stable != out-of-date which is fine of course. The versions of core libraries used are recent though. I guess ‘stable’ means libraries with a long history.

Simple
The components of a Dropwizard application are shown below (taken from the tutorial
http://dropwizard.io/getting-started.html):
Dropview components overview

  1. Application (HelloWorldApplication.java): the applications main method, responsible for startup.
  2. Configuration (HelloWorldConfiguration.java) sets configuration for an environment, this is where you may set hostnames for systems the application depends on or set usernames.
  3. Data object (Saying.java).
  4. Resource (HelloWordResource.java): service implementation entry point
  5. Health Check (TemplateHealthCheck.java): runtime tests that show if the application still works.

Light weight
We did some experiments trying to answer the question whether Dropwizard applications are light weight. The table below summarizes some of the sizes of deployments and tools.
Tomcat size 14 mb
Tomcat lib folder size 7 MB
Jetty size 14,6 MB
Jetty in Dropwizard jar: 5,4 MB
Dropwizard tutorial example 10 mb
Dropwizard extended example 20 MB
Dropwizard Hibernate classes in package: 5 MB

A Tomcat or Jetty installation takes about 14 MB, but if you count only the lib folder the size goes down to about 7 MB. The Jetty folder in Dropwizard however is only 5.5 MB. Apparently Dropwizard managed to strip away some code you don’t really need (or is packaged somewhere else, we didn’t look into that).
Building the tutorial results in a 10 MB jar, so if you would run a webapp in its own Tomcat container, switching to Dropwizard saves quite a bit. On the other hand, deployment size isn’t all that important if we’re still talking < 50 MB.
Compared to your default Weblogic install (513 MB, Weblogic-only on OSX) however, savings are humongous (but this is also true when you compare Weblogic to Tomcat or Jetty).

Productivity
We tried to run the build for the tutorial application (dropwizard-example in the dropwizard project on Github). This works fine and takes about 8 seconds using mocks for external connections. One option to explore would be to run tests against a deployed application. What we’re used to is that deploying an application for test takes lots of time and resources, but starting a Dropwizard app is quite cheap. Therefore it would be possible to run an integration test of services at the end of a build. This would be quite hard to do with e.g. Weblogic or Websphere.

Spring boot
Spring boot is interesting, as well as the discussion around the differences between Spring boot and Dropwizard. See https://groups.google.com/forum/#!topic/dropwizard-user/vH1h2PgC8bU

The official Spring boot website says: Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that can you can "just run". We take an opinionated view of the Spring platform and third-party libraries so you can get started with minimum fuss. Most Spring Boot applications need very little Spring configuration.
It’s good to see a platform change according to new insights, but still, I remember Rod Johnson saying some ten years ago that J2EE was bloated and complex and Spring was the answer. Now it seems we need Spring boot to make Spring simple? Or is it just that we don’t need application servers anymore to divide resources among processes?

Dropwizard and Docker
Finally we experimented with running Dropwizard in a Docker container. This can be done with limited effort because Dropwizard applications have such a small number of dependencies. Thomas Kruitbosch will report on this later.

References
Spring boot: http://projects.spring.io/spring-boot/
Dropwizard: http://dropwizard.io/

100 Top Agile Blogs

Luis Goncalves has put together a list called the 100 Top Agile Blogs:

If you don't know Luis, he lives and breathes driving adoption of Agile practices.

Luis is also an Agile Coach, Co-Author, Speaker, and Blogger.  He is also the co-founder of a MeetUp group called High Performing Teams, and he is a certified Scrum Master and Product Owner.

Here is a preview of the list of top 100 Agile Blogs:

image

 

For the rest of the list, check out 100 Top Agile Blogs.

Lists like these are a great way to discover blogs you may not be aware of.  

While there will be a bunch of blogs you already know, chances are, with that many at a glance, there will be at least a few new ones you can add to your reading list.

Categories: Architecture, Programming

Big changes

Gridshore - Sun, 10/26/2014 - 08:27

The first 10 years of my career I worked as a consultant for Capgemini and Accenture. I learned a lot in that time. One of the things I learned was that I wanted something else. I wanted to do something with more impact, more responsibility and together with people that wanted to do challenging projects. Not pure to get up in your career but more because they like doing cool stuff. Therefore I left Accenture to become part of a company called JTeam. It has been over 6 years that this took place.

I started as Chief Architect at JTeam. The goal was to become a leader to the other architects and create a team together with Bram. That time I was lucky that Allard joined me. We share a lot of ideas, which makes it easier to set goals and accomplish them. I got to learn a few very good people at JTeam, to bad that some of them left, but that is life.

After a few years bigger changes took place. Leonard left for Steven and the shift to a company that needs to grow started. We took over two companies (Funk and Neteffect), we now had all disciplines of software development available. From front-end to operations. As the company grew some things had to change. I got more involved in arranging things like internships, tech events, partnerships and human resource management.

We moved into a bigger building and we had better opportunities. One of the opportunities was a search solution created by Shay Banon. Gone was Steven, together with Shay he founded Elasticsearch. We got acquired by Trifork. In this change we lost most of our search expertise because all of our search people joined the elasticsearch initiative. Someone had to pick up search at Trifork and that was me together with Bram.

For over 2 years I invested a lot of time in learning about mainly elasticsearch. I created a number of workshops/trainings and got involved with multiple customers that needed search. I have given trainings to a number of customers to groups varying between 2 and 15 people. In general they were all really pleased with the trainings I have given.

Having so much focus for a while gave me a lot of time to think, I did not need to think about next steps for the company, I just needed to get more knowledgeable about elasticsearch. In that time I started out on a journey to find out what I want. I talked to my management about it and thought about it myself a lot. Then, right before summer holiday I had a diner with two people I know through the Nljug, Hans and Bert. We had a very nice talk and in the end they gave me an opportunity that I really had to have some good thoughts about. It was really interesting, a challenge, not really a technical challenge, but more an experience that is hard to find. During summer holiday I convinced myself this was a very interesting direction and I took the next step.

I had a lunch meeting with my soon to be business partner Sander. After around 30 minutes it already felt good. I really feel the energy of creating something new, I feel inspired again. This is the feeling I have been missing for a while. In September we were told that Bram was leaving Trifork. Since he is the person that got me into JTeam back in the days it felt weird. I understand his reasons to go out and try to start something new. Bram leaving resulted in a vacancy for a CTO and the management team had decided to approach Allard for this role. This was a surprise to me, but a very nice opportunity for Allard and I know he i going to do a good job. At the end of September Sander and myself presented the draft business plan to the board for Luminis. That afternoon hands were shaken. It was than that I made the last call and decided to resign from my Job at Trifork and take this new opportunity at Luminis.

I feel sad about leaving some people behind. I am going to mis the morning talks in the car with Allard about everything related to the company, I am going to mis doing projects with Roberto (We are a hell of team), I am going to mis Byron for his capabilities (You make me feel proud that I guided your first steps within Trifork), I am going to mis chasing customers with Henk (We did a good job the passed year) and I am going to mis Daphne and the after lunch walks . To all of you and all the others at Trifork, it is a small world …

Luminis logo

Together with Sander, and with the help of all the others at Luminis, we are going to start Luminis Amsterdam. This is going to be a challenge for me, but together with Sander I feel we are going to make it happen. I feel confident that the big changes to come will be good changes.

The post Big changes appeared first on Gridshore.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For October 24th, 2014

Hey, it's HighScalability time:


This is an ultrasound powered brain implant! (65nm GP CMOS technology, high speed, low power (100 µW))
  • 70: percentage of the worlds transactions processed using COBOL.  
  • Quotable Quotes:
    • John Siracusa: Apple has shown that it wants to succeed more than it fears being seen as a follower.
    • @Dries: "99% of Warren Buffett's wealth was built after his 50th birthday."
    • @Pinboard: It is insane to run a bookmarking site on AWS at any kind of scale. Unless you are competing with me, in which case it’s a great idea—do it!
    • @dvellante: I sound like a broken record but AWS has the scale to make infrastructure outsourcing marginal costs track SW curve 
    • @BrentO: LOL RT @SQLPerfTips: "guess which problem you are more likely to have - needing joins, or scaling beyond facebook?"
    • @astorrs: Legacy systems? Yes they're still relevant. ~20x the number of transactions as Google searches @IBM #DOES14 
    • @SoberBuildEng: "It was all the Agile guys' fault at the beginning.Y'know, if the toilet overflowed, it was 'What, are those Agile guys in there?!'" #DOES14
    • @cshl1: #DOES14  @netflix "$1.8M revenue / employee" << folks, this is an amazing number
    • Isaac Asimov: Probably more inhibiting than anything else is a feeling of responsibility. The great ideas of the ages have come from people who weren’t paid to have great ideas, but were paid to be teachers or patent clerks or petty officials, or were not paid at all. The great ideas came as side issues.

  • With Fabric can Twitter mend the broken threads of developer trust? A good start would be removing 3rd party client user limit caps. Not sure a kit of many colors will do it.

  • Not only do I wish I had said this, I wish I had even almost thought it. tjradcliffe: I distinguish between two types of puzzles: human-made (which I call puzzles) and everything else (which I call problems.) In those terms, I hate puzzles and love problems. Puzzles are contrived by humans and are generally as much psychology problems as anything else. They basically require you to think like the human who created them, and they have bizarre and arbitrary constraints that are totally unlike the real world, where, as Feyrabend told us, "Anything goes."

  • David Rosenthal with a great look at Facebook's Warm Storage: 9 [BLOB] types have dropped by 2 orders of magnitude within 8 months...the vast majority of the BLOBs generate I/O rates at least 2 orders of magnitude less than recently generated BLOBs...Within a data center it uses erasure coding...Between data centers it uses XOR coding...When fully deployed, this will save 87PB of storage...heterogeneity as a way of avoiding correlated failures.

  • Gene Tene on is it a CPU bound future: I don't think CPU speed is a problem. The CPUs and main RAM channels are still (by far) the highest performing parts of our systems. For example, yes, you can move ~10-20Gbps over various links today (wired or wifi, "disk" (ssd) or network), but a single Xeon chip today can sustain well over 10x that bandwidth in random access to DRAM. A single chip has more than enough CPU bandwidth to stream through that data, too. E.g. a single current Haswell core can move more than that 10-20Gbps in/out of it's cache levels. and even relatively low end chips (e.g. laptops) will have 4 or more of these cores on a single chip these days. < BTW, a great thread if you are interested in latency issues.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture