Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/categories/7&page=6' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

Dancing with GetKanban (Using POLCA)

Xebia Blog - Mon, 11/02/2015 - 12:59

Very recently POLCA got some attention on twitter. The potential and application of POLCA to knowledge work I explained in my blog 'Squeeze more out of kanban with POLCA!' [Rij11] of 4 years ago.

In this blog the GetKanban [GetKanban] game is played by following the the initial 'standard' rules for handling Work in Progress (WiP) limits and by changing the rules of the game inspired by POLCA (See [POLCA]).

The results show an equal throughput between POLCA and non-overlapping WiP limits, with smaller inventory size in the case of POLCA way of approaching WiP limits.

First a short introduction to the GetKanban game is given and a description of the set-up together with the basic results.

Second a brief introduction to POLCA is given and the change of rules in the game is explained. Thirdly, the set-up of the game using POLCA and the results are discussed.

Finally, a few words are spent on the team's utilization.

Simulation: GetKanban Game

The step-up with standard WiP limits is shown below. The focus is on a basic simulation of the complete 24 project days of only the regular work items. The expedite, fixed delivery date, and intangibles are left out. In addition, the events described on the event cards are ignored. Reason being to get a 'clean' simulation showing the effect of applying the WiP limits in a different manner.

kanban-pan

Other policies taken from the game: a billing cycle of 3 days, replenishment of the 'Ready' column is allowed only at the end of project days 9, 12, 15, 18, and 21.

The result of running the game at the end of day 24 iskanban

The picture shows the state of the board, lead time distribution diagram, control chart, and cumulative flow diagram.

From these it can be inferred that (a) 10 items are in progress, (b) throughput is 25 items in 24 days, (c) median of 9 days for the lead time of items from Ready to Deployed.

kanban-LTDkanban-CTkanban-cfd

Interesting is that the control chart (middle chart) shows the average lead time dropping to 5-6 days in the last three days of the simulation. Since the game starts at day 9, this shows that it takes 12 days before the system settles in a new stable state with 5-6 as an average lead time compared to the 9 days at the beginning.

POLCA: New Rules

In POLCA (see [POLCA]) the essence is to make the WiP limit overlap. The 'O' of POLCA stands for 'Overlapping':

POLCA - Paired Overlapping Loops of Cards with Authorization

One of the characteristics differentiating POLCA from e.g. kanban based systems is that it is a combination of push & pull: 'Push when you know you can pull'.

Setting WiP limits to support pushing work when you know it can subsequently be pulled. The set-up in the game for POLCA is as follows:

polca-pan

For clarity the loops are indicated in the 'expedite' row.

How do the limits work? Two additional rules are introduced:

Rule 1)
In the columns associated with each loop a limit on the number of work items is set. E.g. the columns 'Development - Done' and 'Test' can only accommodate for a maximum of 3 cards. Likewise, the columns underneath the blue and red loops have a limit of 4 and 4 respectively.

Rule 2)
Work can only be pulled if a) the loop has capacity, i.e. has fewer cards than the limit, b) the next, or overlapping loop, has capacity.

These are illustrated with a number of examples that typically occur during the game:

Example 1: Four items in the 'Development - Done' column
No items are allowed to be pulled in 'Ready' because there is no capacity available in the blue loop (Rule 2)

Example2: Two items in 'Test' & two items in 'Analysis - Done'
One item can be pulled in 'Development - In Progress' (Rule 1 and Rule2).

Results for POLCA

The main results for running the game for 24 days with the above rules is shown in the charts below.

Adaptability
The Control Chart shows that it takes roughly 6 days for all existing items to flow out of the system. The effect of the new rules (POLCA style WiP limits) are seen starting from day 15.

Lead Time
On average the charts show a lead time of 3 days, starting from day 18. This is also clearly visible on the lead time distribution chart and on the narrow cumulative flow diagram.

Throughput
The number of items produced by the project is 24 items in 24 days. This is equal enough to the through put as measured using the standard set of rules.

Work In Progress
The total number of items in progress in only 3 to 4 items. This is less than half of the items as seen in the simulation run using the standard set of rules.

polca-new-3 polca-new-1 polca-new-2

Note: The cumulative flow diagram clearly shows the 3-day billing day and replenishment (step-wise increments of black and purple lines).

Learning

As described in my blog 'One change at a time' [Rij15] getting feedback from improvements/changes that effect the flow of the system take some time before the system settles in the new stable state.

With POLCA it is expected that this learning cycle can be shortened. In running the game the control charts show that it takes approximately 12 days before the system reaches the stable state, whereas with the POLCA set of rules this is reached in half the time.

Results for POLCA - Continuous Replenishment

As described above, until now we have adhered to the billing cycle of 3 days, which also allows for replenishment every 3 days.

What happens if replenishment is allowed when possible. The results are shown int he charts below.

polca2-1 polca2-2 polca2-3

The cumulative flow diagram shows the same throughput, namely 24 items over a period of 24 days. Work in progress is larger because it is pulled in earlier instead of at the end of every third day.

What is interesting is that the Control Chart shows a large variation in lead time: from 3 to 6 days. What I noted during playing the game is that at regular times 3 to 4 items are allowed to be pulled into 'Ready'. These would sit for some time in 'Ready' and then suddenly completed all the way to 'Ready for Deployment'. Then another bunch of 3 to 4 items are pulled into 'Ready'.
This behavior is corroborated by the Control Chart (staircase pattern). The larger variation is shown in the Lead Time Distribution Chart.

What is the reason for this? My guess is that the limit of 4 on the red loop is too large. When replenishment was only allowed at days 9, 12, 15, ... this basically meant a lower limit for the red loop.
Tuning the limits is important for establishing a certain cadence. Luckily this behavior can be seen in the Control Chart.

Utilization

In the GetKanban game specialists in the team are represented by colored dices: green for testers, blue for developers, and red for analysts. Effort spent is simulated by throwing the dices. Besides spending the available effort in their own speciality, it can also be spent in other specialities in which case the effort to spend is reduced.

During the game it may happen that utilization is less than 100%:

  1. Not spending effort in the speciality, e.g. assigning developers to do test work.
  2. No work item to spend the effort on because of WiP limits (not allowed to pull work).

The picture below depicts the utilization as happened during the game: on average a utilization of 80%.

polca-utilitzation

Conclusion

In this blog I have shown how POLCA style of setting WiP limits work, how overlapping loops of limits help in pulling work fast through the system, and the positive effect on the team's learning cycle.

In summary, POLCA allows for

  • Shorter lead times
  • Lower work in progress, enabling a faster learning cycle

Tuning of the loop limits seems to be important for establishing a regular cadence. A 'staircase' pattern in the Control Chart is a strong indication that loop limits are not optimal.

References

[GetKanban] GetKanban Game: http://getkanban.com

[Rij11] Blog: Squeeze more out of kanban with POLCA!

[Rij15] Blog: One change at a time

[POLCA] POLCA: http://www.business-improvement.eu/qrm/polca_eng.php

 

The power of map and flatMap of Swift optionals

Xebia Blog - Sun, 11/01/2015 - 01:22

Until recently, I always felt like I was missing something in Swift. Something that makes working with optionals a lot easier. And just a short while ago I found out that the thing I was missing does already exist. I'm talking about the map and flatMap functions of Swift optionals (not the Array map function). Perhaps it's because they're not mentioned in the optionals sections of the Swift guide and because I haven't seen it in any other samples or tutorials. And after asking around I found out some of my fellow Swift programmers also didn't know about it. Since I find it an amazing Swift feature that makes your Swift code often a lot more elegant I'd like to share my experiences with it.

If you didn't know about the map and flatMap functions either you should keep on reading. If you did already know about it, I hope to show some good, real and useful samples of it's usage that perhaps you didn't think about yet.

What do map and flatMap do?

Let me first give you a brief example of what the functions do. If you're already familiar with this, feel free to skip ahead to the examples.

The map function transforms an optional into another type in case it's not nil, and otherwise it just returns nil. It does this by taking a closure as parameter. Here is a very basic example that you can try in a Swift Playground:

var value: Int? = 2
var newValue = value.map { $0 * 2 }
// newValue is now Optional(4)

value = nil
newValue = value.map { $0 * 2 }
// newValue is now nil

At first, this might look odd because we're calling a function on an optional. And don't we always have to unwrap it first? In this case not. That's because the map function is a function of the Optional type and not of the type that is wrapped by the Optional.

The flatMap is pretty much the same as map, except that the return value of the closure in map is not allowed to return nil, while the closure of flatMap can return nil. Let's see another basic example:

var value: Double? = 10
var newValue: Double? = value.flatMap { v in
    if v < 5.0 {
        return nil
    }
    return v / 5.0
}
// newValue is now Optional(2)

newValue = newValue.flatMap { v in
    if v < 5.0 {
        return nil
    }
    return v / 5.0
}
// now it's nil

If we would try to use map instead of flatMap in this case, it would not compile.

When to use it?

In many cases where you use a ternary operator to check if an optional is not nil, and then return some value if it's not nil and otherwise return nil, it's probably better to use one of the map functions. If you recognise the following pattern, you might want to go through your code and make some changes:

var value: Int? = 10
var newValue: Int? = value != nil ? value! + 10 : nil 
// or the other way around:
var otherValue: Int? = value == nil ? nil : value! + 10

The force unwrapping should already indicate that something is not quite right. So instead use the map function shown previously.

To avoid the force unwrapping, you might have used a simple if let or guard statement instead:

func addTen(value: Int?) -> Int? {
  if let value = value {
    return value + 10
  }
  return nil
}

func addTwenty(value: Int?) -> Int? {
  guard let value = value else {
    return nil
  }
  return value + 20
}

This still does exactly the same as the ternary operator and thus is better written with a map function.

Useful real examples of using the map functions

Now let's see some real examples of when you can use the map functions in a smart way that you might not immediately think of. You get the most out of it when you can immediately pass in an existing function that takes the type wrapped by the optional as it's only parameter. In all of the examples below I will first show it without a map function and then again rewritten with a map function.

Date formatting

Without map:

var date: NSDate? = ...
var formatted: String? = date == nil ? nil : NSDateFormatter().stringFromDate(date!)

With map:

var date: NSDate? = ...
var formatted: date.map(NSDateFormatter().stringFromDate)
Segue from cell in UITableView

Without map:

func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {
  if let cell = sender as? UITableViewCell, let indexPath = tableView.indexPathForCell(cell) {
    (segue.destinationViewController as! MyViewController).item = items[indexPath.row]
  }
}

With map:

func prepareForSegue(segue: UIStoryboardSegue, sender: AnyObject?) {
  if let indexPath = (sender as? UITableViewCell).flatMap(tableView.indexPathForCell) {
    (segue.destinationViewController as! MyViewController).item = items[indexPath.row]
  }
}
Values in String literals

Without map:

func ageToString(age: Int?) -> String {
    return age == nil ? "Unknown age" : "She is (age!) years old"
}

With map:

func ageToString(age: Int?) -> String {
    return age.map { "She is ($0) years old" } ?? "Unknown age"
}

(Please note that in the above examples there need to be backslashes before (age!) and ($0) but unfortunately :-( that breaks the formatting of WordPress in this post)

Localized Strings

Without map:

let label = UILabel()
func updateLabel(value: String?) {
  if let value = value {
    label.text = String.localizedStringWithFormat(
      NSLocalizedString("value %@", comment: ""), value)
  } else {
    label.text = nil
  }
}

With map:

let label = UILabel()
func updateLabel(value: String?) {
  label.text = value.map { 
    String.localizedStringWithFormat(NSLocalizedString("value %@", comment: ""), $0) 
  }
}
Enum with rawValue from optional with default

Without map:

enum State: String {
    case Default = ""
    case Cancelled = "CANCELLED"

    static func parseState(state: String?) -> State {
        guard let state = state else {
            return .Default
        }
        return State(rawValue: state) ?? .Default
    }
}

With map:

enum State: String {
    case Default = ""
    case Cancelled = "CANCELLED"

    static func parseState(state: String?) -> State {
        return state.flatMap(State.init) ?? .Default
    }
}
Find item in Array

With Item like:

struct Item {
    let identifier: String
    let value: String
}

let items: [Item]

Without map:

func find(identifier: String) -> Item? {
    if let index = items.indexOf({$0.identifier == identifier}) {
        return items[index]
    }
    return nil
}

With map:

func find(identifier: String) -> Item? {
    return items.indexOf({$0.identifier == identifier}).map({items[$0]})
}
Constructing objects with json like dictionaries

With a struct (or class) like:

struct Person {
    let firstName: String
    let lastName: String

    init?(json: [String: AnyObject]) {
        if let firstName = json["firstName"] as? String, let lastName = json["lastName"] as? String {
            self.firstName = firstName
            self.lastName = lastName
            return
        }
        return nil
    }
}

Without map:

func createPerson(json: [String: AnyObject]) -> Person? {
    if let personJson = json["person"] as? [String: AnyObject] {
        return Person(json: personJson)
    }
    return nil
}

With map:

func createPerson(json: [String: AnyObject]) -> Person? {
    return (json["person"] as? [String: AnyObject]).flatMap(Person.init)
}
Conslusion

The map and flatMap functions can be incredibly powerful and make your code more elegant. Hopefully with these examples you'll be able to spot when situations where it will really benefit your code when you use them.

Please let me know in the comments if you have similar smart examples of map and flatMap usages and I will add them to the list.

Android Resource Configuration override and Large Text Mode

Xebia Blog - Sat, 10/31/2015 - 13:15

In Android, the resource Configuration dictates what resources and assets are selected. The system populates a default configuration to match your device and settings (screen size, device orientation, language). Sometimes, you need to deviate from these defaults. Since API 17 you can use applyOverrideConfiguration(Configuration) to specify an alternative resource config. The normal place to do so is in the attachBaseContext(Context) method of your Activity.

public class MainActivity extends Activity {

@Override
protected void attachBaseContext(Context newBase) {
    super.attachBaseContext(newBase);
    final Configuration override = new Configuration();
    override.locale = new Locale("nl", "NL");
    applyOverrideConfiguration(override);
}

//...

}

Here's what that looks like:

Screenshot Screenshot

Unfortunately, there's a catch.

Android has a "Large Text" setting in its accessibility options (and in some cases a different text size setting in the display options). If you use the overrideConfiguration method to set your own resource configurtation, you will wipe out the Large Text preference, hurting your accessibilty support. This problem is easily overlooked, and luckily, easily fixed.

Screenshot Screenshot

The large fonts setting works by changing the Configuration.fontScale attribute, which is a public float. This works with the scaled density-independent pixels (sp's) that you use to define fontSize attributes. All sp dimensions have this fontScale multiplier applied. My Nexus 5 has two font size settings, normal at 1.0 and large at 1.3. The Nexus 5 emulator image has four, and many Samsung devices have seven different font sizes you can choose from.

When you set the override configuration, the new Configuration object has its fontScale set to 1.0f, thereby breaking the large fonts mode. To fix this problem, you simply have to copy the current fontScale value from the base context. This is best done using the copy constructor, which will also account for any other properties that come with the same issue.

public class MainActivity extends Activity {

@Override
protected void attachBaseContext(Context newBase) {
    super.attachBaseContext(newBase);
    final Configuration override = new Configuration(
            // Copy the original configuration so it isn't lost.
            newBase.getResources().getConfiguration()
        );
    override.locale = new Locale("nl", "NL");
    applyOverrideConfiguration(override);

    // BTW: You can also access the fontScale value using Settings.System:
    // Settings.System.getFloat(getContentResolver(), Settings.System.FONT_SCALE, 1.0f);
}

//...

}

The app now works as intended, with accessibility support intact.

Screenshot

Long story short: when you use applyOverrideConfiguration, always test your app in the Large Fonts accessibility setting. Be sure to copy the original Configuration in your new Configuration constructor, or use the System.Settings.FONT_SCALE property to retrieve the font scale separately.

Stuff The Internet Says On Scalability For October 30th, 2015

Hey, it's HighScalability time:


Movie goers Force Crashed websites with record ticket presales. Yoda commented: Do. Or do not. There is no try.
  • $51.5 billion: Apple quarterly revenue; 1,481: distance in light years of a potential Dyson Sphere; $470 billion: size of insurance industry data play; 31,257: computer related documents in a scanned library; $1.2B: dollars lost to business email scams; 46 billion: pixels in largest astronomical image; 27: seconds of distraction after doing anything interesting in a car; 10 billion: transistor SPARC M7 chip; 10K: cost to get a pound in to low earth orbit; $8.2 billion: Microsoft cloud revenue; 

  • Quotable Quotes:
    • @jasongorman: A $trillion industry has been built on the very lucky fact that Tim Berners-Lee never thought "how do I monetise this?"
    • Cade Metz: Sure, the app [WhatsApp] was simple. But it met a real need. And it could serve as a platform for building all sorts of other simple services in places where wireless bandwidth is limited but people are hungry for the sort of instant communication we take for granted here in the US.
    • Adrian Hanft: Brand experts insist that success comes from promoting your unique attributes, but in practice differentiation is less profitable than consolidation.
    • Jim Butcher: It’s a tradition. Were traditions rational, they’d be procedures.
    • Albert Einstein~ Sometimes I pretend I’m the Mayor of my kitchen and veto fish for dinner. ‘Too fishy’ is what I say!
    • @chumulu: “Any company big enough to have a research lab is too big to listen to it" -- Alan Kay
    • Robin Harris: So maybe AWS has all the growth it can handle right now and doesn’t want more visibility. AWS may be less scalable than we’d like to believe.
    • Michael Nielsen: Every finitely realizable physical system can be simulated efficiently and to an arbitrary degree of approximation by a universal model (quantum) computing machine operating by finite means.
    • Sundar Pichai~ there are now more Google mobile searches than desktop searches worldwide.
    • Joe Salvia~ The major advance in the science of construction over the last few decades has been the perfection of tracking and communication.
    • apy: In other words, as far as I can tell docker is replacing people learning how to use their package manager, not changing how software could or should have been deployed.
    • @joelgrus: "Data science is a god-like power." "Right, have you finished munging those CSVs yet?""No, they have time zone data in them!"
    • @swardley: "things are getting worse. Companies are increasingly financialised and spending less on basic research" @MazzucatoM 
    • Dan Rayburn: The cause of what Akamai is seeing is a result of Apple, Microsoft and Facebook moving a larger percentage of their traffic to their in-house delivery networks.
    • @littleidea: containers will not fix your broken architecture you are welcome
    • spawndog: I've typically found the best gameplay optimization comes from a greater amount of creative freedom like you mention. Lets not do it. Lets do it less frequently. Lets organize the data into something relative to usage pattern like spatial partitions.
    • @awealthofcs: The 1800s: I hope I survive my 3 month voyage to deliver a message to London Now: The streaming on this NFL game in London is a bit spotty
    • @ddwoods2: just having buffers ≠ resilience; resilience = the capacities for changing position/size/kind of buffers, before events eat those buffers
    • unoti: There's a dangerous, contagious illness that developers of every generation get that causes them to worry about architecture and getting "street cred" even more than they worry about solving business problems. I've fallen victim to this myself, because street cred is important to me. But it's a trap.
    • @kelseyhightower: Kubernetes is getting some awesome new features: Auto scaling pods, Jobs API (batch), and a new deployment API for serve side app rollouts.

  • Great story on Optimizing League of Legends. The process: Identification: profile the application and identify the worst performing parts; Comprehension: understand what the code is trying to achieve and why it is slow; Iteration: change the code based on step 2 and then re-profile. Repeat until fast enough. Result: memory savings of 750kb and a function that ran one to two milliseconds faster. 

  • Fantastic article on Medium's architecture: 25 million uniques a month;  service-oriented architecture, running about a dozen production services; GitHub; Amazon’s Virtual Private Cloud; Ansible; mostly Node with some Go; CloudFlare, Fastly, CloudFront with interesting traffic allocations; Nginx and HAProxy; Datadog, PagerDuty, Elasticsearch, Logstash, Kibana; DynamoDB, Redis, Aurora, Neo4J; Protocol Buffers used as contract between layers; and much more.

  • Are notifications the new Web X.0? Notification: the push and the pull: Right now we are witnessing another round of unbundling as the notification screen becomes the primary interface for mobile computing.

  • Algorithm hacking 101. Uber Surge Price? Research Says Walk A Few Blocks, Wait A Few Minutes.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Angular reverse URL construction

Xebia Blog - Thu, 10/29/2015 - 09:02

Let's discuss a problem you probably aren't aware you are having in your web app:

Hardcoded URLs

These hardcoded, static URLs are simple at first, but the code is error prone and increases the app complexity. Instead what you should be doing is creating dynamic URLs by means of reverse URL construction. I have grown up using the Django web framework which extensively uses reverse routing and I love it. Let us look at how this simple system improves your coding and I'm sure you'll love it too.

We'll look at URL construction using an AngularJS example. Even if you don't use Angular, the general idea behind reverse URLs should become apparent.

The stage

Say we have simple web application with a contact page, a list of users and user pages:

 /contact -> contact.html
 /users -> users.html
 /user/foo -> user.html for user "foo"

We will reference these routes directly in our page using the `<a>` tag and occasionally we might even require an URL inside JavaScript. A link to the contact page is easily created using: `<a href="/contact">Contact us</a>. There are however a few drawbacks to this approach.

The first issue is that the URL is now defined in two places, the route configuration and the anchor tag. If you ever decide to change the URL to `contact-us`, all references should be updated as well. As the app grows so do all the duplicate references and you'll end up changing a lot of them. Chances are that you will miss one and cause a bad link.

More problems arise as soon as the URL contains a dynamic element. Lets say the page behind `/users` list all users in our application as follows:

<ul ng-repeat="user in users">
  <li><a ng-href="/user/{{user.name}}"></a></li>
</ul>

Because the URL is constructed using simple interpolation we can't check whether the resulting URL is valid. Lets say the app router matches the URL on the pattern: `^/user/([a-z]+)$`. This would mean the user's name may only consist of letters, for example "foo". If the name contains an integer, say "foo2015", the URL won't be recognized by our router. So we allow bad URLs to be constructed.

So hardcoded URLs aren't great, what is?

Reverse URL resolution

The solution has many names: URL construction, reverse URL resolution and reverse routing. The essence of this method is that every route is identifiable by a unique element. Reversing the URL is done by referencing the route and supplying parameters. The route itself is stored in a single location, the router. This route is used both for routing a request and constructing a URL.

We will add reverse routing to the standard Angular "$routeProvider". A small project angular-reverse-url adds the reverse resolution capability. We can use a filter to build URLs by referencing either the route name or its controller name. In this example we will identify the route using the route name. But bear in mind that the controller approach would work just as well (given a route is uniquely defined by a controller).

Now let's see how reverse URL construction is done. We will use the example setup as follows:

function($routeProvider){
  $routeProvider
    .when('/contact', {
      template: 'contact.html',
      name: 'contact'
    })
    .when('/users', {
      template: '/users.html',
      name: 'user-list'
    })
    .when('/user/:id/:name', {
      template: '/users.html',
      name: 'user-detail'
    })
}]);
Constructing static URLs

Our web app needs needs a navigation bar. Inside our navigation we can link to the contact page using:

<nav>
  <ul>
    <li><a ng-href="{{ 'contact' | reverseUrl }}">Contact us</a></li>
  </ul>
</nav>

The reverseUrl filter simply uses the 'contact' name to construct the URL from the route defined in the $routeProvider.

Parameter based URL generation

Creating plain URLs without any parameters is hardly interesting. The benefits of reverse URL resolution become apparent when we start introducing route arguments. We can list the user pages in our application using:



<li ng-repeat="user in users">
  <a href="{{ 'user-detail' | reverseUrl:{id: user.id, name: user.name} }}">{{user.name}}</a>
</li>


Clean and simple!

There is not duplicate route defined and parameters are conveniently mapped by name. We explicitly map the parameters in this example, but we can also pass an object to the filter directly to greatly simplifying the statement. If you so prefer there is also the option to provide positional arguments:

<a href="{{ 'user-detail' | reverseUrl:[user.id, user.name] }}">{{user.name}}</a>
In code URL resolution

Occasionally the need arises to compose an URL directly in code. As `reverseUrl` is an Angular filter, we can simply call it directly inside our JavaScript:

$filter('reverseUrl')('user-detail', { id: 1, name: 1 })

This works just fine, though I would suggest wrapping the filter in a service if you often construct URLs in code.

Other solutions

The library presented here offers a very simple yet elegant way to do reverse URL resolution. It should be relatively easy to extend with extra functionality such as parameter type checking. There are also quite a lot of other libraries that offer reverse URL construction. We'll look at the new Angular router and the popular ui-router.

Angular >= 1.5

The new Angular 1.5 router offers reverse URL resolution out of the box. If you've made the leap to the new component based routing, reverse resolution should be a breeze. There are some minor syntactical differences:

Configuration

AppController.$routeConfig = [
  { path: '/user/:id/:name', component: 'user-detail' }
];

Template usage

<a ng-link="user-detail({ id: 1, name:'foo' })">Foo's page</a>
UI router

The wildly popular Angular ui-router also offers reverse routing. Chances are if you are using ui-router, you are using this reverse resolution of URLs already. If you are using ui-router and are still hardcoding URLs, switch today! Setup is easy:

Configuration

$stateProvider
 .state('user-detail', {
   URL: "/user/:id/:name",
   controller: "UserDetailCtrl",
   templateUrl: "views/user.html"
 });

Template usage

<a ui-sref="user-detail({ id: 1, name: 'foo' })">Foo's page</a>
Stop hard coding URLs

With so many options, there really is no good reason to hard code URLs in your application. From now on, all URLs you create in any sizable project should be using a reverse URL resolution system.

Reverse URL construction is really simple, it keeps your routes in one place and standardizes URL generation in your app.

If you want to take a closer look at the Angular example code, see the full example: reverse-routing-Angular-example

Five Lessons from Ten Years of IT Failures

IEEE Spectrum has a wonderful article series on Lessons From a Decade of IT Failures. It’s not your typical series in that there are very cool interactive graphs and charts based on data collected from past project failures. They are really fun to play with and I can only imagine how much work it took to put them together.

The overall takeaway of the series is:

Even given the limitations of the data, the lessons we draw from them indicate that IT project failures and operational issues are occurring more regularly and with bigger consequences. This isn’t surprising as IT in all its various forms now permeates every aspect of global society. It is easy to forget that Facebook launched in 2004, YouTube in 2005, Apple’s iPhone in 2007, or that there has been three new versions of Microsoft Windows released since 2005. IT systems are definitely getting more complex and larger (in terms of data captured, stored and manipulated), which means not only are they increasing difficult and costly to develop, but they’re also harder to maintain.

Here are the specific lessons:

Categories: Architecture

Sponsored Post: Digit, iStreamPlanet, Instrumental, Redis Labs, Jut.io, SignalFx, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Digit Game Studios, Irish’s largest game development studio, is looking for game server engineers to work on existing and new mobile 3D MMO games. Our most recent project in development is based on an iconic AAA-IP and therefore we expect very high DAU & CCU numbers. If you are passionate about games and if you are experienced in creating low-latency architectures and/or highly scalable but consistent solutions then talk to us and apply here.

  • As a Networking & Systems Software Engineer at iStreamPlanet you’ll be driving the design and implementation of a high-throughput video distribution system. Our cloud-based approach to video streaming requires terabytes of high-definition video routed throughout the world. You will work in a highly-collaborative, agile environment that thrives on success and eats big challenges for lunch. Please apply here.

  • As a Scalable Storage Software Engineer at iStreamPlanet you’ll be driving the design and implementation of numerous storage systems including software services, analytics and video archival. Our cloud-based approach to world-wide video streaming requires performant, scalable, and reliable storage and processing of data. You will work on small, collaborative teams to solve big problems, where you can see the impact of your work on the business. Please apply here.

  • At Scalyr, we're analyzing multi-gigabyte server logs in a fraction of a second. That requires serious innovation in every part of the technology stack, from frontend to backend. Help us push the envelope on low-latency browser applications, high-speed data processing, and reliable distributed systems. Help extract meaningful data from live servers and present it to users in meaningful ways. At Scalyr, you’ll learn new things, and invent a few of your own. Learn more and apply.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Your event could be here. How cool is that?
Cool Products and Services
  • Instrumental is a hosted real-time application monitoring platform. In the words of one of our customers: "Instrumental is the first place we look when an issue occurs. Graphite was always the last place we looked." - Dan M

  • Real-time correlation across your logs, metrics and events.  Jut.io just released its operations data hub into beta and we are already streaming in billions of log, metric and event data points each day. Using our streaming analytics platform, you can get real-time monitoring of your application performance, deep troubleshooting, and even product analytics. We allow you to easily aggregate logs and metrics by micro-service, calculate percentiles and moving window averages, forecast anomalies, and create interactive views for your whole organization. Try it for free, at any scale.

  • Turn chaotic logs and metrics into actionable data. Scalyr replaces all your tools for monitoring and analyzing logs and system metrics. Imagine being able to pinpoint and resolve operations issues without juggling multiple tools and tabs. Get visibility into your production systems: log aggregation, server metrics, monitoring, intelligent alerting, dashboards, and more. Trusted by companies like Codecademy and InsideSales. Learn more and get started with an easy 2-minute setup. Or see how Scalyr is different if you're looking for a Splunk alternative or Sumo Logic alternative.

  • SignalFx: just launched an advanced monitoring platform for modern applications that's already processing 10s of billions of data points per day. SignalFx lets you create custom analytics pipelines on metrics data collected from thousands or more sources to create meaningful aggregations--such as percentiles, moving averages and growth rates--within seconds of receiving data. Start a free 30-day trial!

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex goes beyond monitoring and measures the system's work on your servers, providing unparalleled insight and query-level analysis. This unique approach ultimately enables your team to work more effectively, ship more often, and delight more customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

What ideas in IT must die?

Are there ideas in IT that must die for progress to be made?

Max Planck wryly observed that scientific progress is often less meritocracy and more Lord of the Flies:

A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it.

Playing off this insight is a thought provoking book collection of responses to a question posed on the Edge: This Idea Must Die: Scientific Theories That Are Blocking Progress. From the book blurb some of the ideas that should transition into the postmortem are: Jared Diamond explores the diverse ways that new ideas emerge; Nassim Nicholas Taleb takes down the standard deviation; Richard Thaler and novelist Ian McEwan reveal the usefulness of "bad" ideas; Steven Pinker dismantles the working theory of human behavior.

Let’s get edgy: Are there ideas that should die in IT?

What ideas do you think should pass into the great version control system called history? What ideas if garbage collected would allow us to transmigrate into a bright shiny new future? Be as deep and bizarre as you want. This is the time for it.

I have two: Winner Takes All and The Homogeneity Principle.

Winner Takes All
Categories: Architecture

Learning about test automation with Lego

Xebia Blog - Mon, 10/26/2015 - 15:44

“Hold on, did you say that I can learn about test automation by playing with Lego? Shut up and take my money!” Yes, I am indeed saying that you can. It will cost you a couple hundred Euro’s, because Lego isn’t cheap, especially the Mindstorm EV3 Lego. It turns out that Lego robots eat at a lot of AA batteries, so buy a couple of packs of these as well. On the software side you need to have a computer with a Java development environment and an IDE of your choice (the free edition of IntelliJ IDEA will do). 

“Okay, hold on a second. Why do you need Java? I thought Lego had its own programming language?”. Yes, that’s true. Orginally, Lego provides you with their own visual programming language. I mean, the audience for the EV3 is actually kids, but it will be our little secret. Because Lego is awesome, even for adults. Some hero made a Java library that can communicate with the EV3 hardware, LeJos, so you can do more awesome stuff with it. Another hero dedicated a whole website to his Mindstorm projects, including instructions on how to build them.

Starting the project

So, on a sunny innovation day in August at Xebia, Erik Zeedijk and I started our own Lego project. The goal was to make something cool and relevant for Testworks Conf. We decided to go for The Ultimate Machine, also known as The Most Useless Machine.  It took us about three hours to assemble the Lego. If you’re not familiar with the Useless Machine, check this video below. 

Somehow, we had to combine Lego with test automation. We decided to use the Cucumber framework and write acceptance tests in it. That way, we could also use that to figure out what options we wanted to give the machine (sort of a requirements phase...what did I just say!?). The Ultimate Machine can do more than just turn off the switch, as you could tell if you watched the above video. It can detect when a hand is hovering above the switch and that can trigger all kinds of actions: driving away to trick the human, hiding the switch to trick the human, etc. With Acceptance Test Driven Development, we could write out all these actions in tests and use those tests to drive our coding. In that sense, we were also using Test Driven Development. In the picture below is an example of a Cucumber feature file that we used.

feature_picture

The idea sounded really simple, but executing it was a bit harder. We made a conceptual mistake at first. To run our tests, we first coded them in a way that still required a human (someone who turned the switch on). Also, the tests were testing the Lego hardware too (the sensors) and not our own code. The Lego hardware has quite some bugs in it, we noticed. Some of the sensors aren’t really accurate in the values they return. After some frustration and thinking we found a way to solve our problem. In the end, the solution is pretty elegant and in retrospect I face-palm because of my own inability to see it earlier.

We had to mock the Lego hardware (the infrared sensor and the motors), because it was unreliable and we wanted to test our own code. We also had to mock the human out of the tests. This meant that we didn’t even need the Lego robot anymore to run our tests. We decided to use Mockito for our mock setup. In the end, the setup looked like this. 

robot setup

The LeJos Java library uses a couple of concepts that are important to grasp. An arbitrator decides which behavior should run. All the behaviors are put in a behaviorList Inside each behavior is a boolean wantControl that becomes 'true' when certain conditions arise. See the picture below for an example 'wantControl' in the DriveBehavior class. 

wantControl

Then the behavior starts to run and when it is finished it returns 'idle = true'. The arbitrator then picks a new behavior to run. Because some behaviors had the same conditions for 'wantControl' we had to think of a way to prevent the same behavior from triggering all the time. In each behavior we put a boolean chanceOfBehaviorHappening and we assigned a chance to it. After a bit of tweaking we had the robot running the way we liked it.

The tests were reliable after this refactoring and super fast. The test code was neatly separated from the code that implemented the robot’s behaviour. In addition, you could start the real Lego robot and play with it. This is a picture of our finished robot. 

Lego Ultimate Machine

We didn’t implement all the behaviors we identified on purpose, because our goal was to get attendants of TestWorks Conf to code for our robot. This little project has taught both Erik and me more about writing good Cucumber feature files, TDD and programming. We are both not really Java experts, so this project was a great way of learning for both of us. I certainly improved my understanding of Object Oriented programming. But even if you are a seasoned programmer, this project could be nice to increase your understanding of Cucumber, TDD or ATDD. So, convince your boss to shell out a couple hundred to buy this robot for you and start learning and have fun.

FYI: I will take the robot with me to the Agile Testing Days, so if you are reading this and going there too, look me up and have a go at coding.

Model updates in Presentation Controls

Xebia Blog - Mon, 10/26/2015 - 11:15

In this post I'll explain how to deal with updates when you're using Presentation Controls in iOS. It's a continuation of my previous post in which I described how you can use Presentation Controls instead of MVVM or in combination with MVVM.

The previous post didn't deal with any updates. But most often the things displayed on screen can change. This can happen because new data is fetched from a server, through user interaction or maybe automatically over time. To make that work, we need to inform our Presentation Controls of any updates of our model objects.

Let's use the Trip from the previous post again:

struct Trip {

    let departure: NSDate
    let arrival: NSDate
    let duration: NSTimeInterval

    var actualDeparture: NSDate
    var delay: NSTimeInterval {
        return self.actualDeparture.timeIntervalSinceDate(self.departure)
    }
    var delayed: Bool {
        return delay > 0
    }

    init(departure: NSDate, arrival: NSDate, actualDeparture: NSDate? = nil) {
        self.departure = departure
        self.arrival = arrival
        self.actualDeparture = actualDeparture ?? departure

        // calculations
        duration = self.arrival.timeIntervalSinceDate(self.departure)
    }
}

Instead of calculating and setting the delay and delayed properties in the init we changed them into computed properties. That's because we'll change the value of the actualDeparture property in the next examples and want to display the new value of the delay property as well.

So how do we get notified of changes within Trip? A nice approach to do that is through binding. You could use ReactiveCocoa to do that but to keep things simple in this post I'll use a class Dynamic that was introduced in a post about Bindings, Generics, Swift and MVVM by Srdan Rasic (many things in my post are inspired by the things he writes so make sure to read his great post). The Dynamic looks as follows:

class Dynamic<T> {
  typealias Listener = T -> Void
  var listener: Listener?

  func bind(listener: Listener?) {
    self.listener = listener
  }

  func bindAndFire(listener: Listener?) {
    self.listener = listener
    listener?(value)
  }

  var value: T {
    didSet {
      listener?(value)
    }
  }

  init(_ v: T) {
    value = v
  }
}

This allows us to register a listener which is informed of any change of the value. A quick example of its usage:

let delay = Dynamic("+5 minutes")
delay.bindAndFire {
    print("Delay: $$$0)")
}

delay.value = "+6 minutes" // will print 'Delay: +6 minutes'

Our Presentation Control was using a TripViewViewModel class to get all the values that it had to display in our view. These properties were all simple constants with types such as String and Bool that would never change. We can replace the properties that can change with a Dynamic property.

In reality we would probably make all properties dynamic and fetch a new Trip from our server and use that to set all the values of all Dynamic properties, but in our example we'll only change the actualDeparture of the Trip and create dynamic properties for the delay and delayed properties. This will allow you to see exactly what is happening later on.

Our new TripViewViewModel now looks like this:

class TripViewViewModel {

    let date: String
    let departure: String
    let arrival: String
    let duration: String

    private static let durationShortFormatter: NSDateComponentsFormatter = {
        let durationFormatter = NSDateComponentsFormatter()
        durationFormatter.allowedUnits = [.Hour, .Minute]
        durationFormatter.unitsStyle = .Short
        return durationFormatter
    }()

    private static let durationFullFormatter: NSDateComponentsFormatter = {
        let durationFormatter = NSDateComponentsFormatter()
        durationFormatter.allowedUnits = [.Hour, .Minute]
        durationFormatter.unitsStyle = .Full
        return durationFormatter
    }()

    let delay: Dynamic<String?>
    let delayed: Dynamic<Bool>

    var trip: Trip

    init(_ trip: Trip) {
        self.trip = trip

        date = NSDateFormatter.localizedStringFromDate(trip.departure, dateStyle: .ShortStyle, timeStyle: .NoStyle)
        departure = NSDateFormatter.localizedStringFromDate(trip.departure, dateStyle: .NoStyle, timeStyle: .ShortStyle)
        arrival = NSDateFormatter.localizedStringFromDate(trip.arrival, dateStyle: .NoStyle, timeStyle: .ShortStyle)

        duration = TripViewViewModel.durationShortFormatter.stringFromTimeInterval(trip.duration)!

        delay = Dynamic(trip.delayString)
        delayed = Dynamic(trip.delayed)
    }

    func changeActualDeparture(delta: NSTimeInterval) {
        trip.actualDeparture = NSDate(timeInterval: delta, sinceDate: trip.actualDeparture)

        self.delay.value = trip.delayString
        self.delayed.value = trip.delayed
    }

}

extension Trip {

    private var delayString: String? {
        return delayed ? String.localizedStringWithFormat(NSLocalizedString("%@ delay", comment: "Show the delay"), TripViewViewModel.durationFullFormatter.stringFromTimeInterval(delay)!) : nil
    }
}

Using the changeActualDeparture method we can increase or decrease the time of trip.actualDeparture. Since the delay and delayed properties on trip are now computed properties their returned values will be updated as well. We use them to set new values on the Dynamic delay and delayed properties of our TripViewViewModel. Also the logic to format the delay String has moved into an extension on Trip to avoid duplication of code.

All we have to do now to get this working again is to create bindings in the TripPresentationControl:

class TripPresentationControl: NSObject {

    @IBOutlet weak var dateLabel: UILabel!
    @IBOutlet weak var departureTimeLabel: UILabel!
    @IBOutlet weak var arrivalTimeLabel: UILabel!
    @IBOutlet weak var durationLabel: UILabel!
    @IBOutlet weak var delayLabel: UILabel!

    var tripModel: TripViewViewModel! {
        didSet {
            dateLabel.text = tripModel.date
            departureTimeLabel.text = tripModel.departure
            arrivalTimeLabel.text  = tripModel.arrival
            durationLabel.text = tripModel.arrival

            tripModel.delay.bindAndFire { [unowned self] in
                self.delayLabel.text = $0
            }

            tripModel.delayed.bindAndFire { [unowned self] delayed in
                self.delayLabel.hidden = !delayed
                self.departureTimeLabel.textColor = delayed ? .redColor() : UIColor(red: 0, green: 0, blue: 0.4, alpha: 1.0)
            }
        }
    }
}

Even though everything compiles again, we're not done yet. We still need a way to change the delay. We'll do that through some simple user interaction and add two buttons to our view. One to increase the delay with one minute and one to decrease it. Handling of the button taps goes into the normal view controller since we don't want to make our Presentation Control responsible for user interaction. Our final view controller now looks like as follows:

class ViewController: UIViewController {

    @IBOutlet var tripPresentationControl: TripPresentationControl!

    let tripModel = TripViewViewModel(Trip(departure: NSDate(timeIntervalSince1970: 1444396193), arrival: NSDate(timeIntervalSince1970: 1444397193), actualDeparture: NSDate(timeIntervalSince1970: 1444396493)))

    override func viewDidLoad() {
        super.viewDidLoad()

        tripPresentationControl.tripModel = tripModel
    }

    @IBAction func increaseDelay(sender: AnyObject) {
        tripModel.changeActualDeparture(60)
    }

    @IBAction func decreaseDelay(sender: AnyObject) {
        tripModel.changeActualDeparture(-60)
    }
}

We now have an elegant way of updating the view when we tap the button. Our view controller communicates a change logical change of the model to the TripViewViewModel which in turn notifies the TripPresentationControl about a change of data, which in turn updates the UI. This way the Presentation Control doesn't need to know anything about user interaction and our view controller doesn't need to know about which UI components it needs to change after user interaction.

And the result:
http://blog.xebia.com/wp-content/uploads/2015/10/viewcontrol.mp4

Hopefully this post will give you a better understanding about how to use Presentation Controls and MVVM. As I mentioned in my previous post, I recommend you to read Introduction to MVVM by Ash Furrow and From MVC to MVVM in Swift by Srdan Rasic as well as his follow up post mentioned at the beginning of this post.

And of course make sure to join the do {iOS} Conference in Amsterdam the 9th of November, 2015. Here Natasha "the Robot" Murashev will be giving a talk about Protocol-oriented MVVM.

Keeping an eye on your Amazon EC2 firewall rules

Xebia Blog - Sun, 10/25/2015 - 14:56

Amazon AWS makes it really easy for anybody to create and update firewall rules that provide access to the virtual machines inside AWS. Within seconds you can add your own IP address so you can work from home or the office. However, it is also very easy to forget to remove them once your are finished. The utility aws-sg-revoker , will help you maintain your firewall rules.

aws-sg-revoker inspects all your inbound access permission and compares them with the public IP addresses of the machines in your AWS account. For grants to IP addresses not found in your account, it will generate a aws CLI revoke command. But do not be afraid: it only generates, it does not execute it directly. You may want to investigate before removal. Follow the following 4 steps to safeguard your account!

step 1. Investigate

First run the following command to generate a list of all the IP address ranges that are referenced but not in your account.

aws-sg-revoker -l

1.2.3.4 x.y.z.
4.5.6.7. hostname.com.
5.5.123.4 a.b.c.
8.9.10.11/16
....

You may find that you have to install jq and the aws CLI :-)

step 2. Exclude known addresses

Exclude the ip addresses that are ok. These addresses are added as regular expressions.

aws-sg-revoker -l -w 1\.2\.\3\.4 -w 8\.9\.10\.11/16
step 3. generate revoke commands

Once you are happy, you can generate the revoke commands:

aws-sg-revoker -w 1\.2\.\3\.4 -w 4\.5\.6\.7 -w 8\.9\.10\.11/16

aws ec2 revoke-security-group-ingress --group-id sg-aaaaaaaa --port 22-22 --protocol tcp --cidr 5.5.123.4/32 # revoke from sg blablbsdf
aws ec2 revoke-security-group-ingress --group-id sg-aaaaaaaa --port 9200-9200 --protocol tcp --cidr 5.5.123.4/32 # revoke from sg blablbsdf
aws ec2 revoke-security-group-ingress --group-id sg-aaaaaaaa --port 9080-9080 --protocol tcp --cidr 5.5.123.4/32 # revoke from sg blablbsdf
aws ec2 revoke-security-group-ingress --group-id sg-bbbbbbbb --protocol -1 --cidr 5.5.123.4/32 # revoke from sg sg-1
aws ec2 revoke-security-group-ingress --group-id sg-bbbbbbbb --protocol -1 -cidr 5.5.123.4/32 # revoke from sg sg-3
step 4. Execute!

If the revokes look ok, you can execute them by piping them to a shell:

aws-sg-revoker -w 1\.2\.\3\.4 -w 8\.9\.10\.11/16 | tee revoked.log | bash
conclusion

This utility makes it easy to for you to regularly inspect and maintain your firewall rules and keep your AWS resources safe!

Agile, but still really not Agile? What Pipeline Automation can do for you. Part 2.

Xebia Blog - Thu, 10/22/2015 - 13:51

Organizations adopting Agile and teams delivering on a feature-by-feature basis producing business value at the end of every sprint. Quite possibly this is also the case in your organization. But do these features actually reach your customer at the same pace and generate business value straight away? And while we are at it: are you able to actually use feedback from your customer and apply it for use in the very next sprint?

Possibly your answer is “No”, which I see very often. Many companies have adopted the Agile way of working in their lines of business, but for some reason ‘old problems’ just do not seem to go away...

Hence the question:

“Do you fully capitalize on the benefits provided by working in an Agile manner?”

Straight forward Software Delivery Pipeline Automation might help you with that.

In this post I hope to inspire you to think about how Software Development Pipeline automation can help your company to move forward and take the next steps towards becoming a truly Agile company. Not just a company adopting Agile principles, but a company that is really positioned to respond to the ever changing environment that is our marketplace today. To explain this, I take the Agile Manifesto as a starting point and work from there.

In my previous post, I addressed Agile Principles 1 to 4, please read below where I'll explain about how automation can help you for Agile Principles 5 to 8.

 

Agile Principle 5: Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

This is an important aspect in Agile. People get motivated by acknowledged empowerment, responsibility, ownership and trusted support when performing a job. This is one of the reasons Agile teams often feel so vibrant and dynamic. Still, in many organizations development-teams work Agile but “subsequent teams” do not. Resulting in mini-waterfalls slowing down your delivery cycle as a whole.trust-in-hands

“Environment and the support needed” means that the Agile team should work in a creative and innovative environment where team-members can quickly test new features. Where the team can experiment, systems “just work” and “waiting” is not required. The team should be enabled, so to speak .. in terms of automation and in terms of innovation. This means that a build should not take hours, a deployment should not take days and the delivery of new infrastructure should not take weeks.

Applying rigorous automation will help you to achieve the fifth objective of the Agile manifesto. There is a bit of a chicken and egg situation here, but I feel it is safe to say that a sloppy, broken, quirky development environment will not help in raising the bar in terms of motivating individuals. Hence "give them the environment and support they need, and trust them to get the job done".

 

Agile Principle 6: The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

When working Agile, individuals and interactions are valued over the use of processes and tools. When starting a new project, teams should not be hindered with ticketing systems, extensive documentation to explain themselves and long service times. These type of “services” often exist on boundaries of business-units of bringing different ‘disciplines’ to the solution.500px-People_together.svg

Although working Agile, many companies still have these boundaries in place. An important aspect of Continuous Delivery is executing work in Product teams dedicated to delivery and/or maintenance of an end-product. These product teams have all required disciplines working together in one and the same team. Operating in this manner alleviates the need for slow tooling & ticketing systems and inspires people to work together and get the job done.

Organizing people as a team working on a product instead of individuals performing a task, which in itself has no meaning, will help you to achieve the sixth objective of the Agile Manifesto. There is not a lot automation can do for you here.

 

Agile Principle 7: Working software is the primary measure of progress.

Agile aims towards delivering working software at the end of each sprint. For the customer that is basically what counts: working software, which can actually be used. Working software means software without defects. There is no point in delivering broken software at the end of every sprint.Working-Software

When sending a continuous flow of new functions to the customer, each function should adhere to the required quality level straight away. In terms of quality, new functions might need to be ‘reliable’, ‘secure’, maintainable’, ‘fast’, etc, which all are fully testable properties. Testing these type of properties should be integral part of team activities. One of the principles related to Continuous Delivery addresses this topic through Test automation. Without it, it is not possible to deliver working production-ready software by the end of each sprint.

Proper implementation of test-disciplines, fostering a culture of delivering high quality software, testing every single feature, adhering to testing disciplines and applying matching & automated test tooling addresses topics related to the seventh object of the Agile Manifesto. Supply a test for every function you add to the product and automate this test.

 

Agile Principle 8: Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

As software complexity grows exponentially, it will become more difficult overtime to maintain a constant pace of delivering new features, when assembling, deploying, testing or provision manually. Humans are simply not made to perform multiple tasks fast, repetitively and consistently over a longer period of time, that is what machines are for!

The eighth Agile principle typically comes down to a concept called ‘flow’. You might have an Agile team in place for creating new softflow-clipart-9183-flow-designware, but what about the flow in the rest of your organization? Should the team wait for requirements to drip through, should it wait for the testers to manually test software, or is it the Operations team that needs to free resources in order to deploy software? To address this, from idea to product, handover moments should be minimized as much as possible and where possible principles of automation should be applied. This brings us back to build automation, test automation, deployment automation and the automation of infrastructure.

 

Stay tuned for the next post, where I'll address the final four Agile principles of the bunch.

 

Michiel Sens.

AzureCon Keynote Announcements: India Regions, GPU Support, IoT Suite, Container Service, and Security Center

ScottGu's Blog - Scott Guthrie - Thu, 10/01/2015 - 06:43

Yesterday we held our AzureCon event and were fortunate to have tens of thousands of developers around the world participate.  During the event we announced several great new enhancements to Microsoft Azure including:

  • General Availability of 3 new Azure regions in India
  • Announcing new N-series of Virtual Machines with GPU capabilities
  • Announcing Azure IoT Suite available to purchase
  • Announcing Azure Container Service
  • Announcing Azure Security Center

We were also fortunate to be joined on stage by several great Azure customers who talked about their experiences using Azure including: Jet.com, Nascar, Alaska Airlines, Walmart, and ThyssenKrupp. Watching the Videos

All of the talks presented at AzureCon (including the 60 breakout talks) are now available to watch online.  You can browse and watch all of the sessions here.

image

My keynote to kick off the event was an hour long and provided an end-to-end look at Azure and some of the big new announcements of the day.  You can watch it here.

Below are some more details of some of the highlights:

Announcing General Availability of 3 new Azure regions in India

Yesterday we announced the general availability of our new India regions: Mumbai (West), Chennai (South) and Pune (Central).  They are now available for you to deploy solutions into.

This brings our worldwide presence of Azure regions up to 24 regions, more than AWS and Google combined. Over 125 customers and partners have been participating in the private preview of our new India regions.   We are seeing tremendous interest from industry sectors like Public Sector, Banking Financial Services, Insurance and Healthcare whose cloud adoption has been restricted by data residency requirements.  You can all now deploy your solutions too. Announcing N-series of Virtual Machines with GPU Support

This week we announced our new N-series family of Azure Virtual Machines that enable GPU capabilities.  Featuring NVidia’s best of breed Tesla GPUs, these Virtual Machines will help you run a variety of workloads ranging from remote visualization to machine learning to analytics.

The N-series VMs feature NVidia’s flagship GPU, the K80 which is well supported by NVidia’s CUDA development community. N-series will also have VM configurations featuring the latest M60 which was recently announced by NVidia. With support for M60, Azure becomes the first hyperscale cloud provider to bring the capabilities of NVidia’s Quadro High End Graphics Support to the cloud. In addition, N-series combines GPU capabilities with the superfast RDMA interconnect so you can run multi-machine, multi-GPU workloads such as Deep Learning and Skype Translator Training.

Announcing Azure Security Center

This week we announced the new Azure Security Center—a new Azure service that gives you visibility and control of the security of your Azure resources, and helps you stay ahead of threats and attacks.  Azure is the first cloud platform to provide unified security management with capabilities that help you prevent, detect, and respond to threats.

image

The Azure Security Center provides a unified view of your security state, so your team and/or your organization’s security specialists can get the information they need to evaluate risk across the workloads they run in the cloud.  Based on customizable policy, the service can provide recommendations. For example, the policy might be that all web applications should be protected by a web application firewall. If so, the Azure Security Center will automatically detect when web apps you host in Azure don’t have a web application firewall configured, and provide a quick and direct workflow to get a firewall from one of our partners deployed and configured:

image

Of course, even with the best possible protection in place, attackers will still try to compromise systems. To address this problem and adopt an “assume breach” mindset, the Azure Security Center uses advanced analytics, including machine learning, along with Microsoft’s global threat intelligence network to look for and alert on attacks. Signals are automatically collected from your Azure resources, the network, and integrated security partner solutions and analyzed to identify cyber-attacks that might otherwise go undetected. Should an incident occur, security alerts offer insights into the attack and suggest ways to remediate and recover quickly. Security data and alerts can also be piped to existing Security Information and Events Management (SIEM) systems your organization has already purchased and is using on-premises.

image

No other cloud vendor provides the depth and breadth of these capabilities, and they are going to enable you to build even more secure applications in the cloud.

Announcing Azure IoT Suite Available to Purchase

The Internet of Things (IoT) provides tremendous new opportunities for organizations to improve operations, become more efficient at what they do, and create new revenue streams.  We have had a huge interest in our Azure IoT Suite which until this week has been in public preview.  Our customers like Rockwell Automation and ThyssenKrupp Elevators are already connecting data and devices to solve business problems and improve their operations. Many more businesses are poised to benefit from IoT by connecting their devices to collect and analyze untapped data with remote monitoring or predictive maintenance solutions.

In working with customers, we have seen that getting started on IoT projects can be a daunting task starting with connecting existing devices, determining the right technology partner to work with and scaling an IoT project from proof of concept to broad deployment. Capturing and analyzing untapped data is complex, particularly when a business tries to integrate this new data with existing data and systems they already have. 

The Microsoft Azure IoT Suite helps address many of these challenges.  The Microsoft Azure IoT Suite helps you connect and integrate with devices more easily, and to capture and analyze untapped device data by using our preconfigured solutions, which are engineered to help you move quickly from proof of concept and testing to broader deployment. Today we support remote monitoring, and soon we will be delivering support for predictive maintenance and asset management solutions.

These solutions reliably capture data in the cloud and analyze the data both in real-time and in batch processing. Once your devices are connected, Azure IoT Suite provides real time information in an intuitive format that helps you take action from insights. Our advanced analytics then enables you to easily process data—even when it comes from a variety of sources, including devices, line of business assets, sensors and other systems and provide rich built-in dashboards and analytics tools for access to the data and insights you need. User permissions can be set to control reporting and share information with the right people in your organization.

Below is an example of the types of built-in dashboard views that you can leverage without having to write any code:

image

To support adoption of the Azure IoT Suite, we are also announcing the new Microsoft Azure Certified for IoT program, an ecosystem of partners whose offerings have been tested and certified to help businesses with their IoT device and platform needs. The first set of partners include Beaglebone, Freescale, Intel, Raspberry Pi, Resin.io, Seeed and Texas Instruments. These partners, along with experienced global solution providers are helping businesses harness the power of the Internet of Things today.  

You can learn more about our approach and the Azure IoT Suite at www.InternetofYourThings.com and partners can learn more at www.azure.com/iotdev. Announcing Azure IoT Hub

This week we also announced the public preview of our new Azure IoT Hub service which is a fully managed service that enables reliable and secure bi-directional communications between millions of IoT devices and an application back end. Azure IoT Hub offers reliable device-to-cloud and cloud-to-device hyper-scale messaging, enables secure communications using per-device security credentials and access control, and includes device libraries for the most popular languages and platforms.

Providing secure, scalable bi-directional communication from the heterogeneous devices to the cloud is a cornerstone of any IoT solution which Azure IoT hub addresses in the following way:

  • Per-device authentication and secure connectivity: Each device uses its own security key to connect to IoT Hub. The application back end is then able to individually whitelist and blacklist each device, enabling complete control over device access.
  • Extensive set of device libraries: Azure IoT device SDKs are available and supported for a variety of languages and platforms such as C, C#, Java, and JavaScript.
  • IoT protocols and extensibility: Azure IoT Hub provides native support of the HTTP 1.1 and AMQP 1.0 protocols for device connectivity. Azure IoT Hub can also be extended via the Azure IoT protocol gateway open source framework to provide support for MQTT v3.1.1.
  • Scale: Azure IoT Hub scales to millions of simultaneously connected devices, and millions of events per seconds.

Getting started with Azure IoT Hub is easy. Simply navigate to the Azure Preview portal, and use the Internet of Things->Azure IoT Hub. Choose the name, pricing tier, number of units and location and select Create to provision and deploy your IoT Hub:

image

Once the IoT hub is created, you can navigate to Settings and create new shared access policies and modify other messaging settings for granular control.

The bi-directional communication enabled with an IoT Hub provides powerful capabilities in a real world IoT solution such as the control of individual device security credentials and access through the use of a device identity registry.  Once a device identity is in the registry, the device can connect, send device-to-cloud messages to the hub, and receive cloud-to-device messages from backend applications with just a few lines of code in a secure way.

Learn more about Azure IoT Hub and get started with your own real world IoT solutions. Announcing the new Azure Container Service

’We’ve been working with Docker to integrate Docker containers with both Azure and Windows Server for some time. This week we announced the new Azure Container Service which leverages the popular Apache Mesos project to deliver a customer proven orchestration solution for applications delivered as Docker containers.

image[24]

The Azure Container Service enables users to easily create and manage a Docker enabled Apache Mesos cluster. The container management software running on these clusters is open source, and in addition to the application portability offered by tooling such as Docker and Docker Compose, you will be able to leverage portable container orchestration and management tooling such as Marathon, Chronos and Docker Swarm.

When utilizing the Azure Container Service, you will be able to take advantage of the tight integration with Azure infrastructure management features such as tagging of resources, Role Based Access Control (RBAC), Virtual Machine Scale Sets (VMSS) and the fully integrated user experience in the Azure portal. By coupling the enterprise class Azure cloud with key open source build, deploy and orchestration software, we maximize customer choice when it comes to containerize workloads.

The service will be available for preview by the end of the year. Learn More

Watch the AzureCon sessions online to learn more about all of the above announcements – plus a lot more that was covered during the day.  We are looking forward to seeing what you build with what you learn!

Hope this helps,

Scott omni

Categories: Architecture, Programming

Announcing General Availability of HDInsight on Linux + new Data Lake Services and Language

ScottGu's Blog - Scott Guthrie - Mon, 09/28/2015 - 21:54

Today, I’m happy to announce several key additions to our big data services in Azure, including the General Availability of HDInsight on Linux, as well as the introduction of our new Azure Data Lake and Language services. General Availability of HDInsight on Linux

Today we are announcing general availability of our HDInsight service on Ubuntu Linux.  HDInsight enables you to easily run managed Hadoop clusters in the cloud.  With today’s release we now allow you to configure these clusters to run using both a Windows Server Operating System as well as an Ubuntu based Linux Operating System.

HDInsight on Linux enables even broader support for Hadoop ecosystem partners to run in HDInsight providing you even greater choice of preferred tools and applications for running Hadoop workloads. Both Linux and Windows clusters in HDInsight are built on the same standard Hadoop distribution and offer the same set of rich capabilities.

Today’s new release also enables additional capabilities, such as, cluster scaling, virtual network integration and script action support. Furthermore, in addition to Hadoop cluster type, you can now create HBase and Storm clusters on Linux for your NoSQL and real time processing needs such as building an IoT application.

Create a cluster

HDInsight clusters running using Linux can now be easily created from the Azure Management portal under the Data + Analytics section.  Simply select Ubuntu from the cluster operating system drop-down, as well as optionally choose the cluster type you wish to create (we support base Hadoop as well as clusters pre-configured for workloads like Storm, Spark, HBase, etc).

image

All HDInsight Linux clusters can be managed by Apache Ambari. Ambari provides the ability to customize configuration settings of your Hadoop cluster while giving you a unified view of the performance and state of your cluster and providing monitoring and alerting within the HDInsight cluster.

image

Installing additional applications and Hadoop components

Similar to HDInsight Windows clusters, you can now customize your Linux cluster by installing additional applications or Hadoop components that are not part of default HDInsight deployment. This can be accomplished using Bash scripts with script action capability.  As an example, you can now install Hue on an HDInsight Linux cluster and easily use it with your workloads:

image

Develop using Familiar Tools

All HDInsight Linux clusters come with SSH connectivity enabled by default. You can connect to the cluster via a SSH client of your choice. Moreover, SSH tunneling can be leveraged to remotely access all of the Hadoop web applications from the browser.

image New Azure Data Lake Services and Language

We continue to see customers enabling amazing scenarios with big data in Azure including analyzing social graphs to increase charitable giving, analyzing radiation exposure and using the signals from thousands of devices to simulate ways for utility customers to optimize their monthly bills. These and other use cases are resulting in even more data being collected in Azure. In order to be able to dive deep into all of this data, and process it in different ways, you can now use our Azure Data Lake capabilities – which are 3 services that make big data easy.

The first service in the family is available today: Azure HDInsight, our managed Hadoop service that lets you focus on finding insights, and not spend your time having to manage clusters. HDInsight lets you deploy Hadoop, Spark, Storm and HBase clusters, running on Linux or Windows, managed, monitored and supported by Microsoft with a 99.9% SLA.

The other two services, Azure Data Lake Store and Azure Data Lake Analytics introduced below, are available in private preview today and will be available broadly for public usage shortly. Azure Data Lake Store

Azure Data Lake Store is a hyper-scale HDFS repository designed specifically for big data analytics workloads in the cloud. Azure Data Lake Store solves the big data challenges of volume, variety, and velocity by enabling you to store data of any type, at any size, and process it at any scale. Azure Data Lake Store can support near real-time scenarios such as the Internet of Things (IoT) as well as throughput-intensive analytics on huge data volumes. The Azure Data Lake Store also supports a variety of computation workloads by removing many of the restrictions constraining traditional analytics infrastructure like the pre-definition of schema and the creation of multiple data silos. Once located in the Azure Data Lake Store, Hadoop-based engines such as Azure HDInsight can easily mine the data to discover new insights.

Some of the key capabilities of Azure Data Lake Store include:

  • Any Data: A distributed file store that allows you to store data in its native format, Azure Data Lake Store eliminates the need to transform or pre-define schema in order to store data.
  • Any Size: With no fixed limits to file or account sizes, Azure Data Lake Store enables you to store kilobytes to exabytes with immediate read/write access.
  • At Any Scale: You can scale throughput to meet the demands of your analytic systems including the high throughput needed to analyze exabytes of data. In addition, it is built to handle high volumes of small writes at low latency making it optimal for near real-time scenarios like website analytics, and Internet of Things (IoT).
  • HDFS Compatible: It works out-of-the-box with the Hadoop ecosystem including other Azure Data Lake services such as HDInsight.
  • Fully Integrated with Azure Active Directory: Azure Data Lake Store is integrated with Azure Active Directory for identity and access management over all of your data.
Azure Data Lake Analytics with U-SQL

The new Azure Data Lake Analytics service makes it much easier to create and manage big data jobs. Built on YARN and years of experience running analytics pipelines for Office 365, XBox Live, Windows and Bing, the Azure Data Lake Analytics service is the most productive way to get insights from big data. You can get started in the Azure management portal, querying across data in blobs, Azure Data Lake Store, and Azure SQL DB. By simply moving a slider, you can scale up as much computing power as you’d like to run your data transformation jobs.

image

Today we are introducing a new U-SQL offering in the analytics service, an evolution of the familiar syntax of SQL.  U-SQL allows you to write declarative big data jobs, as well as easily include your own user code as part of those jobs. Inside Microsoft, developers have been using this combination in order to be productive operating on massive data sets of many exabytes of scale, processing mission critical data pipelines. In addition to providing an easy to use experience in the Azure management portal, we are delivering a rich set of tools in Visual Studio for debugging and optimizing your U-SQL jobs. This lets you play back and analyze your big data jobs, understanding bottlenecks and opportunities to improve both performance and efficiency, so that you can pay only for the resources you need and continually tune your operations.

image Learn More

For more information and to get started, check out the following links:

Hope this helps,

Scott

omni
Categories: Architecture, Programming

Online AzureCon Conference this Tuesday

ScottGu's Blog - Scott Guthrie - Mon, 09/28/2015 - 04:35

This Tuesday, Sept 29th, we are hosting our online AzureCon event – which is a free online event with 60 technical sessions on Azure presented by both the Azure engineering team as well as MVPs and customers who use Azure today and will share their best practices.

I’ll be kicking off the event with a keynote at 9am PDT.  Watch it to learn the latest on Azure, and hear about a lot of exciting new announcements.  We’ll then have some fantastic sessions that you can watch throughout the day to learn even more.

image

Hope to see you there!

Scott

omni
Categories: Architecture, Programming

Better Density and Lower Prices for Azure’s SQL Elastic Database Pools

ScottGu's Blog - Scott Guthrie - Wed, 09/23/2015 - 21:41

A few weeks ago, we announced the preview availability of the new Basic and Premium Elastic Database Pools Tiers with our Azure SQL Database service.  Elastic Database Pools enable you to run multiple, isolated and independent databases that can be auto-scaled automatically across a private pool of resources dedicated to just you and your apps.  This provides a great way for software-as-a-service (SaaS) developers to better isolate their individual customers in an economical way.

Today, we are announcing some nice changes to the pricing structure of Elastic Database Pools as well as changes to the density of elastic databases within a pool.  These changes make it even more attractive to use Elastic Database Pools to build your applications.

Specifically, we are making the following changes:

  • Finalizing the eDTU price – With Elastic Database Pools you purchase units of capacity that we can call eDTUs – which you can then use to run multiple databases within a pool.  We have decided to not increase the price of eDTUs as we go from preview->GA.  This means that you’ll be able to pay a much lower price (about 50% less) for eDTUs than many developers expected.
  • Eliminating the per-database fee – In additional to lower eDTU prices, we are also eliminating the fee per database that we have had with the preview. This means you no longer need to pay a per-database charge to use an Elastic Database Pool, and makes the pricing much more attractive for scenarios where you want to have lots of small databases.
  • Pool density – We are announcing increased density limits that enable you to run many more databases per Elastic Database pool. See the chart below under “Maximum databases per pool” for specifics. This change will take effect at the time of general availability, but you can design your apps around these numbers.  The increase pool density limits will make Elastic Database Pools event more attractive.

image

 

Below are the updated parameters for each of the Elastic Database Pool options with these new changes:

image

For more information about Azure SQL Database Elastic Database Pools and Management tools go the technical overview here.

Hope this helps,

Scott omni

Categories: Architecture, Programming

Announcing the Biggest VM Sizes Available in the Cloud: New Azure GS-VM Series

ScottGu's Blog - Scott Guthrie - Wed, 09/02/2015 - 18:51

Today, we’re announcing the release of the new Azure GS-series of Virtual Machine sizes, which enable Azure Premium Storage to be used with Azure G-series VM sizes. These VM sizes are now available to use in both our US and Europe regions.

Earlier this year we released the G-series of Azure Virtual Machines – which provide the largest VM size provided by any public cloud provider.  They provide up to 32-cores of CPU, 448 GB of memory and 6.59 TB of local SSD-based storage.  Today’s release of the GS-series of Azure Virtual Machines enables you to now use these large VMs with Azure Premium Storage – and enables you to perform up to 2,000 MB/sec of storage throughput , more than double any other public cloud provider.  Using the G5/GS5 VM size now also offers more than 20 gbps of network bandwidth, also more than double the network throughout provided by any other public cloud provider.

These new VM offerings provide an ideal solution to your most demanding cloud based workloads, and are great for relational databases like SQL Server, MySQL, PostGres and other large data warehouse solutions. You can also use the GS-series to significantly scale-up the performance of enterprise applications like Dynamics AX.

The G and GS-series of VM sizes are available to use now in our West US, East US-2, and West Europe Azure regions.  You’ll see us continue to expand availability around the world in more regions in the coming months. GS Series Size Details

The below table provides more details on the exact capabilities of the new GS-series of VM sizes:

Size

Cores

Memory

Max Disk IOPS

Max Disk Bandwidth

(MB per second)

Standard_GS1

2

28

5,000

125

Standard_GS2

4

56

10,000

250

Standard_GS3

8

112

20,000

500

Standard_GS4

16

224

40,000

1,000

Standard_GS5

32

448

80,000

2,000

Creating a GS-Series Virtual Machine

Creating a new GS series VM is very easy.  Simply navigate to the Azure Preview Portal, select New(+) and choose your favorite OS or VM image type:

image

Click the Create button, and then click the pricing tier option and select “View All” to see the full list of VM sizes. Make sure your region is West US, East US 2, or West Europe to select the G-series or the GS-Series:

image

When choosing a GS-series VM size, the portal will create a storage account using Premium Azure Storage. You can select an existing Premium Storage account, as well, to use for the OS disk of the VM:

image

Hitting Create will launch and provision the VM. Learn More

If you would like more information on the GS-Series VM sizes as well as other Azure VM Sizes then please visit the following page for additional details: Virtual Machine Sizes for Azure.

For more information on Premium Storage, please see: Premium Storage overview. Also, refer to Using Linux VMs with Premium Storage for more details on Linux deployments on Premium Storage.

Hope this helps,

Scott

omni
Categories: Architecture, Programming

Announcing Great New SQL Database Capabilities in Azure

ScottGu's Blog - Scott Guthrie - Thu, 08/27/2015 - 17:13

Today we are making available several new SQL Database capabilities in Azure that enable you to build even better cloud applications.  In particular:

  • We are introducing two new pricing tiers for our  Elastic Database Pool capability.  Elastic Database Pools enable you to run multiple, isolated and independent databases on a private pool of resources dedicated to just you and your apps.  This provides a great way for software-as-a-service (SaaS) developers to better isolate their individual customers in an economical way.
  • We are also introducing new higher-end scale options for SQL Databases that enable you to run even larger databases with significantly more compute + storage + networking resources.

Both of these additions are available to start using immediately.  Elastic Database Pools

If you are a SaaS developer with tens, hundreds, or even thousands of databases, an elastic database pool dramatically simplifies the process of creating, maintaining, and managing performance across these databases within a budget that you control. 

image

A common SaaS application pattern (especially for B2B SaaS apps) is for the SaaS app to use a different database to store data for each customer.  This has the benefit of isolating the data for each customer separately (and enables each customer’s data to be encrypted separately, backed-up separately, etc).  While this pattern is great from an isolation and security perspective, each database can end up having varying and unpredictable resource consumption (CPU/IO/Memory patterns), and because the peaks and valleys for each customer might be difficult to predict, it is hard to know how much resources to provision.  Developers were previously faced with two options: either over-provision database resources based on peak usage--and overpay. Or under-provision to save cost--at the expense of performance and customer satisfaction during peaks.

Microsoft created elastic database pools specifically to help developers solve this problem.  With Elastic Database Pools you can allocate a shared pool of database resources (CPU/IO/Memory), and then create and run multiple isolated databases on top of this pool.  You can set minimum and maximum performance SLA limits of your choosing for each database you add into the pool (ensuring that none of the databases unfairly impacts other databases in your pool).  Our management APIs also make it much easier to script and manage these multiple databases together, as well as optionally execute queries that span across them (useful for a variety operations).  And best of all when you add multiple databases to an Elastic Database Pool, you are able to average out the typical utilization load (because each of your customers tend to have different peaks and valleys) and end up requiring far fewer database resources (and spend less money as a result) than you would if you ran each database separately.

The below chart shows a typical example of what we see when SaaS developers take advantage of the Elastic Pool capability.  Each individual database they have has different peaks and valleys in terms of utilization.  As you combine multiple of these databases into an Elastic Pool the peaks and valleys tend to normalize out (since they often happen at different times) to require much less overall resources that you would need if each database was resourced separately:

databases sharing eDTUs

Because Elastic Database Pools are built using our SQL Database service, you also get to take advantage of all of the underlying database as a service capabilities that are built into it: 99.99% SLA, multiple-high availability replica support built-in with no extra charges, no down-time during patching, geo-replication, point-in-time recovery, TDE encryption of data, row-level security, full-text search, and much more.  The end result is a really nice database platform that provides a lot of flexibility, as well as the ability to save money.

New Basic and Premium Tiers for Elastic Database Pools

Earlier this year at the //Build conference we announced our new Elastic Database Pool support in Azure and entered public preview with the Standard Tier edition of it.  The Standard Tier allows individual databases within the elastic pool to burst up to 100 eDTUs (a DTU represents a combination of Compute + IO + Storage performance) for performance. 

Today we are adding additional Basic and Premium Elastic Database Pools to the preview to enable a wider range of performance and cost options.

  • Basic Elastic Database Pools are great for light-usage SaaS scenarios.  Basic Elastic Database Pools allows individual databases performance bursts up to 5 eDTUs.
  • Premium Elastic Database Pools are designed for databases that require the highest performance per database. Premium Elastic Database Pools allows individual database performance bursts up to 1,000 eDTUs.

Collectively we think these three Elastic Database Pool pricing tier options provide a tremendous amount of flexibility and optionality for SaaS developers to take advantage of, and are designed to enable a wide variety of different scenarios. Easily Migrate Databases Between Pricing Tiers

One of the cool capabilities we support is the ability to easily migrate an individual database between different Elastic Database Pools (including ones with different pricing tiers).  For example, if you were a SaaS developer you could start a customer out with a trial edition of your application – and choose to run the database that backs it within a Basic Elastic Database Pool to run it super cost effectively.  As the customer’s usage grows you could then auto-migrate them to a Standard database pool without customer downtime.  If the customer grows up to require a tremendous amount of resources you could then migrate them to a Premium Database Pool or run their database as a standalone SQL Database with a huge amount of resource capacity.

This provides a tremendous amount of flexibility and capability, and enables you to build even better applications. Managing Elastic Database Pools

One of the the other nice things about Elastic Database Pools is that the service provides the management capabilities to easily manage large collections of databases without you having to worry about the infrastructure that runs it.   

You can create and mange Elastic Database Pools using our Azure Management Portal or via our Command-line tools or REST Management APIs.  With today’s update we are also adding support so that you can use T-SQL to add/remove new databases to/from an elastic pool.  Today’s update also adds T-SQL support for measuring resource utilization of databases within an elastic pool – making it even easier to monitor and track utilization by database.

image Elastic Database Pool Tier Capabilities

During the preview, we have been and will continue to tune a number of parameters that control the density of Elastic Database Pools as we progress through the preview.

In particular, the current limits for the number of databases per pool and the number of pool eDTUs is something we plan to steadily increase as we march towards the general availability release.  Our plan is to provide the highest possible density per pool, largest pool sizes, and the best Elastic Database Pool economics while at the same time keeping our 99.99 availability SLA.

Below are the current performance parameters for each of the Elastic Database Pool Tier options in preview today:

 

Basic Elastic

Standard Elastic

Premium Elastic

Elastic Database Pool

eDTU range per pool (preview limits)

100-1200 eDTUs

100-1200 eDTUs

125-1500 eDTUs

Storage range per pool

10-120 GB

100-1200 GB

63-750 GB

Maximum database per pool (preview limits)

200

200

50

Estimated monthly pool and add-on  eDTU costs (preview prices)

Starting at $0.2/hr (~$149/pool/mo).

Each additional eDTU $.002/hr (~$1.49/mo)

Starting at $0.3/hr (~$223/pool mo). 

Each additional eDTU $0.003/hr (~$2.23/mo)

Starting at $0.937/hr (`$697/pool/mo).

Each additional eDTU $0.0075/hr (~$5.58/mo)

Storage per eDTU

0.1 GB per eDTU

1 GB per eDTU

.5 GB per eDTU

Elastic Databases

eDTU max per database (preview limits)

0-5

0-100

0-1000

Storage max per DB

2 GB

250 GB

500 GB

Per DB cost (preview prices)

$0.0003/hr (~$0.22/mo)

$0.0017/hr (~$1.26/mo)

$0.0084/hr (~$6.25/mo)

We’ll continue to iterate on the above parameters and increase the maximum number of databases per pool as we progress through the preview, and would love your feedback as we do so.

New Higher-Scale SQL Database Performance Tiers

In addition to the enhancements for Elastic Database Pools, we are also today releasing new SQL Database Premium performance tier options for standalone databases. 

Today we are adding a new P4 (500 DTU) and a P11 (1750 DTU) level which provide even higher performance database options for SQL Databases that want to scale-up. The new P11 edition also now supports databases up to 1TB in size.

Developers can now choose from 10 different SQL Database Performance levels.  You can easily scale-up/scale-down as needed at any point without database downtime or interruption.  Each database performance tier supports a 99.99% SLA, multiple-high availability replica support built-in with no extra charges (meaning you don’t need to buy multiple instances to get an SLA – this is built-into each database), no down-time during patching, point-in-time recovery options (restore without needing a backup), TDE encryption of data, row-level security, and full-text search.

image

Learn More

You can learn more about SQL Databases by visiting the http://azure.microsoft.com web-site.  Check out the SQL Database product page to learn more about the capabilities SQL Databases provide, as well as read the technical documentation to learn more how to build great applications using it.

Summary

Today’s database updates enable developers to build even better cloud applications, and to use data to make them even richer more intelligent.  We are really looking forward to seeing the solutions you build.

Hope this helps,

Scott

omni
Categories: Architecture, Programming

Announcing Windows Server 2016 Containers Preview

ScottGu's Blog - Scott Guthrie - Wed, 08/19/2015 - 17:01

At DockerCon this year, Mark Russinovich, CTO of Microsoft Azure, demonstrated the first ever application built using code running in both a Windows Server Container and a Linux container connected together. This demo helped demonstrate Microsoft's vision that in partnership with Docker, we can help bring the Windows and Linux ecosystems together by enabling developers to build container-based distributed applications using the tools and platforms of their choice.

Today we are excited to release the first preview of Windows Server Containers as part of our Windows Server 2016 Technical Preview 3 release. We’re also announcing great updates from our close collaboration with Docker, including enabling support for the Windows platform in the Docker Engine and a preview of the Docker Engine for Windows. Our Visual Studio Tools for Docker, which we previewed earlier this year, have also been updated to support Windows Server Containers, providing you a seamless end-to-end experience straight from Visual Studio to develop and deploy code to both Windows Server and Linux containers. Last but not least, we’ve made it easy to get started with Windows Server Containers in Azure via a dedicated virtual machine image. Windows Server Containers

Windows Server Containers create a highly agile Windows Server environment, enabling you to accelerate the DevOps process to efficiently build and deploy modern applications. With today’s preview release, millions of Windows developers will be able to experience the benefits of containers for the first time using the languages of their choice – whether .NET, ASP.NET, PowerShell or Python, Ruby on Rails, Java and many others.

Today’s announcement delivers on the promise we made in partnership with Docker, the fast-growing open platform for distributed applications, to offer container and DevOps benefits to Linux and Windows Server users alike. Windows Server Containers are now part of the Docker open source project, and Microsoft is a founding member of the Open Container Initiative. Windows Server Containers can be deployed and managed either using the Docker client or PowerShell. Getting Started using Visual Studio

The preview of our Visual Studio Tools for Docker, which enables developers to build and publish ASP.NET 5 Web Apps or console applications directly to a Docker container, has been updated to include support for today’s preview of Windows Server Containers. The extension automates creating and configuring your container host in Azure, building a container image which includes your application, and publishing it directly to your container host. You can download and install this extension, and read more about it, at the Visual Studio Gallery here: http://aka.ms/vslovesdocker.

Once installed, developers can right-click on their projects within Visual Studio and select “Publish”:

image

Doing so will display a Publish dialog which will now include the ability to deploy to a Docker Container (on either a Windows Server or Linux machine):

image

You can choose to deploy to any existing Docker host you already have running:

image

Or use the dialog to create a new Virtual Machine running either Window Server or Linux with containers enabled.  The below screen-shot shows how easy it is to create a new VM hosted on Azure that runs today’s Windows Server 2016 TP3 preview that supports Containers – you can do all of this (and deploy your apps to it) easily without ever having to leave the Visual Studio IDE:

image Getting Started Using Azure

In June of last year, at the first DockerCon, we enabled a streamlined Azure experience for creating and managing Docker hosts in the cloud. Up until now these hosts have only run on Linux. With the new preview of Windows Server 2016 supporting Windows Server Containers, we have enabled a parallel experience for Windows users.

Directly from the Azure Marketplace, users can now deploy a Windows Server 2016 virtual machine pre-configured with the container feature enabled and Docker Engine installed. Our quick start guide has all of the details including screen shots and a walkthrough video so take a look here https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/azure_setup.

image

Once your container host is up and running, the quick start guide includes step by step guides for creating and managing containers using both Docker and PowerShell. Getting Started Locally Using Hyper-V

Creating a virtual machine on your local machine using Hyper-V to act as your container host is now really easy. We’ve published some PowerShell scripts to GitHub that automate nearly the whole process so that you can get started experimenting with Windows Server Containers as quickly as possible. The quick start guide has all of the details at https://msdn.microsoft.com/en-us/virtualization/windowscontainers/quick_start/container_setup.

Once your container host is up and running the quick start guide includes step by step guides for creating and managing containers using both Docker and PowerShell.

image Additional Information and Resources

A great list of resources including links to past presentations on containers, blogs and samples can be found in the community section of our documentation. We have also setup a dedicated Windows containers forum where you can provide feedback, ask questions and report bugs. If you want to learn more about the technology behind containers I would highly recommend reading Mark Russinovich’s blog on “Containers: Docker, Windows and Trends” that was published earlier this week. Summary

At the //Build conference earlier this year we talked about our plan to make containers a fundamental part of our application platform, and today’s releases are a set of significant steps in making this a reality.’ The decision we made to embrace Docker and the Docker ecosystem to enable this in both Azure and Windows Server has generated a lot of positive feedback and we are just getting started.

While there is still more work to be done, now users in the Window Server ecosystem can begin experiencing the world of containers. I highly recommend you download the Visual Studio Tools for Docker, create a Windows Container host in Azure or locally, and try out our PowerShell and Docker support. Most importantly, we look forward to hearing feedback on your experience.

Hope this helps,

Scott omni

Categories: Architecture, Programming

Spark, Parquet and S3 – It’s complicated.

(A version of this post was originally posted in AppsFlyer’s blog. Also special thanks to Morri Feldman and Michael Spector from AppsFlyer data team that did most of the work solving the problems discussed in this article)

TL;DR; The combination of Spark, Parquet and S3 (& Mesos) is a powerful, flexible and cost effective analytics platform (and, incidentally, an alternative to Hadoop). However making all these technologies gel and play nicely together is not a simple task. This post describes the challenges we (AppsFlyer) faced when building our analytics platform on these technologies, and the steps we  took to mitigate them and make it all work.

Spark is shaping up as the leading alternative to Map/Reduce for several reasons including the wide adoption by the different Hadoop distributions, combining both batch and streaming on a single platform and a growing library of machine-learning integration (both in terms of included algorithms and the integration with machine learning languages namely R and Python). At AppsFlyer, we’ve been using Spark for a while now as the main framework for ETL (Extract, Transform & Load) and analytics. A recent example is the new version of our retention report  that we recently released, which utilized Spark  to crunch several data streams (> 1TB a day) with ETL (mainly data cleansing) and analytics (a stepping stone towards full click-fraud detection) to produce the report.
One of the main changes we introduced in this report is the move from building onSequence files to using Parquet files. Parquet is a columnar data format, which is probably the best option today for storing long term big data for analytics purposes (unless you are heavily invested in Hive, where Orc is the more suitable format). The advantages of Parquet  vs. Sequence files are performance and compression without losing the benefit of wide support by big-data tools  (Spark, Hive, Drill, Tajo, Presto etc.).

One relatively unique aspect of our infrastructure for big data is that we do not use Hadoop (perhaps that’s a topic for a separate post). We are using Mesos as a resource manager instead of YARN and we use Amazon S3 instead of HDFS as a distributed storage solution. HDFS has several advantages over S3, however, the cost/benefit for maintaining long running HDFS clusters on AWS  vs. using S3 are overwhelming in favor of S3.

That said, the combination of Spark, Parquet and S3 posed several challenges for us and this post will list the major ones and the solutions we came up with to cope with them.

Parquet & Spark

Parquet and Spark seem to have been in a  love-hate relationship for a while now. On the one hand, the Spark documentation touts  Parquet as one of the best formats for analytics of big data (it is) and on the other hand the support for Parquet in Spark is incomplete and annoying to use. Things are surely moving in the right direction but there are still a few quirks and pitfalls to watch out for.

To start on a positive note,Spark and Parquet integration has come a  long way in the past few months. Previously, one had to jump through hoops just to be able to convert existing data to Parquet. The introduction of DataFrames to Spark made this process much, much simpler. When the input format is supported by the DataFrame API e.g. the input is JSON  (built-in) or Avro (which isn’t built in Spark yet, but you can use a library to read it) converting to Parquet is just a matter of reading the input format on one side and persisting it as Parquet on the other. Consider for example the following snippet in Scala:

View the code on Gist.

Even when you are handling a format where the schema isn’t part of the data, the conversion process is quite simple as Spark lets you specify the schema programmatically. The Spark documentation is pretty straightforward and contains examples in Scala, Java and Python. Furthermore, it isn’t too complicated to define schemas in other languages. For instance, here (AppsFlyer), we use Clojure as our main development language so we developed a couple of helper functions to do just that. The sample code below provides the details:

The first thing is to extract the data from whatever structure we have and specify the schema we like. The code below takes an event-record and extracts various data points from it into a vector of the form [:column_name value optional_data_type]. Note that the data type is optional since it is defaults to string if not specified.

View the code on Gist.

The next step is to use the above mentioned structure to both extract the schema and convert to DataFrame Rows:

View the code on Gist.

Finally we apply these functions over an RDD , convert it to a data frame and save as parquet:

View the code on Gist.

As mentioned above, things are on the up and up for Parquet and Spark but the road is not clear yet. Some of the problems we encountered include:

  • a critical bug in 1.4 release where a race condition when writing parquet files caused massive data loss on jobs (This bug is fixed in 1.4.1 – so if you are using Spark 1.4 and parquet upgrade yesterday!)
  • Filter pushdown optimization, which is turned off  by default since Spark still uses Parquet 1.6.0rc3  -even though 1.6.0 has been out for awhile (it seems Spark 1.5 will use parquet 1.7.0 so the problem will be solved)
  • Parquet is not “natively” supported in Spark, instead, Spark relies on Hadoop support for the parquet format – this is not a problem in itself, but for us it caused major performance issues when we tried to use Spark and Parquet with S3 – more on that in the next section

Parquet, Spark & S3

Amazon S3 (Simple Storage Services) is an object storage solution that is relatively cheap to use. It does have a few disadvantages vs. a “real” file system; the major one is eventual consistency i.e. changes made by one process are not immediately visible to other applications. (If you are using Amazon’s EMR you can use EMRFS “consistent view” to overcome this.) However, if you understand this limitation, S3 is still a viable input and output source, at least for batch jobs.

As mentioned above, Spark doesn’t have a native S3 implementation and relies on Hadoop classes to abstract the data access to Parquet. Hadoop provides 3 file system clients to S3:

  • S3 block file system (URI schema of the form “s3://..”) which doesn’t seem to work with Spark which only work on EMR (Edited: 12/8/2015 thanks to Ewan Leith)
  • S3 Native Filesystem (“s3n://..” URIs) – download  Spark distribution that supports Hadoop 2.* and up if you want to use this (tl;dr – you don’t)
  • S3a – a replacement for S3n that removes some of the limitations and problems of S3n. Download “Spark with Hadoop 2.6 and up” support to use this one (tl;dr – you want this but it needs some work before it is usable)

 

When we used Spark 1.3 we encountered many problems when we tried to use S3, so we started out using s3n – which worked for the most part, i.e. we got jobs running and completing but a lot of them failed with various read timeout and host unknown exceptions. Looking at the tasks within the jobs the picture was even grimmer with high percentages of failures that pushed us to increase timeouts and retries to ridiculous levels. When we moved to Spark 1.4.1, we took another stab at trying s3a. This time around we got it to work. The first thing we had to do was to set both spark.executor.extraClassPath and spark.executor.extraDriverPath to point at the aws-java-sdk and the hadoop-aws jars since apparently both are missing from the “Spark with Hadoop 2.6” build. Naturally we used the 2.6 version of these files but then we were hit by this little problem. Hadoop 2.6 AWS implementation has a bug which causes it to split S3 files  in unexpected ways (e.g. a 400 files jobs ran with 18 million tasks) luckily using  Hadoop AWS jar to version 2.7.0 instead of the 2.6 one solved this problem –  So,with all that set  s3a prefixes works without hitches (and provides better performance than s3n).

Finding the right S3 Hadoop library contributes to the stability of our jobs but  regardless of S3 library (s3n or s3a) the performance of Spark jobs that use Parquet files was  still abysmal. When looking at the Spark UI, the actual work of handling the data seemed quite reasonable but Spark spent a huge amount of time before actually starting the work and after the job was “completed” before it actually terminated. We like to call this phenomena the “Parquet Tax.”

Obviously we couldn’t live with the “Parquet Tax” so we delved into the log files of our jobs and discovered several issues. This first one has to do with startup times of Parquet jobs. The people that built Spark understood that schema can evolve over time and provides a nice feature for DataFrames called “schema merging.” If you look at schema in a big data lake/reservoir (or whatever it is called today) you can definitely expect the schema to evolve over time. However if you look at a directory that is the result of a single job there is no difference in the schema… It turns out that when Spark initializes a job, it reads the footers of all the Parquet files to perform the schema merging. All this work is done from the driver before any tasks are allocated to the executor and can take long minutes, even hours (e.g. we have jobs that look back at half a year of install data). It isn’t documented but looking at the Spark code  you can override this behavior by specifying mergeSchema as false :

In Scala:

View the code on Gist.

and in Clojure:

View the code on Gist.

Note that this doesn’t work in Spark 1.3. In Spark 1.4 it works as expected and in Spark 1.4.1 it causes Spark only to look at _common_metadata file which is not the end of the world since it is a small file and there’s only one of these per directory. However, this brings us to another aspect of the “Parquet Tax” – the “end of job” delays.

Turning off schema merging and controlling the schema used by Spark helped cut down the job start up times but, as mentioned we still suffered from long delays at the end of jobs. We already knew of one Hadoop<->S3 related problem when using text files. Hadoop  being immutable first writes files to a temp directory and then copies them over. With S3 that’s not a problem but the copy operation is very very expensive. With text files, DataBricks created DirectOutputCommitter  (probably for their Spark SaaS offering). Replacing the output committer for text files is fairly easy – you just need to set “spark.hadoop.mapred.output.committer.class” on the Spark configuration e.g.:

View the code on Gist.

A similar solution exists for Parquet and unlike the solution for text files it is even part of the Spark distribution. However, to make things complicated you have to configure it on Hadoop configuration and not on the Spark configuration. To get the Hadoop configuration you first need to create a Spark context from the Spark configuration, call hadoopConfiguration on it and then set “spark.sql.parquet.output.committer.class” as in:

View the code on Gist.

Using the DirectParquetOutputCommitter provided a significant reduction in the “Parquet Tax” but we still found that some jobs were taking a very long time to complete. Again the problem was the file system assumptions Spark and Hadoop hold which were  the culprits. Remember the “_common_metadata” Spark looks at the onset of a job – well, Spark spends a lot of time at the end of the job creating both this file and an additional MetaData file with additional info from the files that are in the directory. Again this is all done from one place (the driver) rather than being handled by the executors. When the job results in small files (even when there are couple of thousands of those) the process takes reasonable time. However, when the job results in larger files (e.g. when we ingest a full day of application launches) this takes upward of an hour. As with mergeSchema the solution is to manage metadata manually so we set “parquet.enable.summary-metadata” to false (again on the Hadoop configuration and generate the _common_metadata file ourselves (for the large jobs)

To sum up, Parquet and especially Spark are works in progress – making cutting edge technologies work for you can be a challenge and require a lot of digging. The documentation is far from perfect at times but luckily all the relevant technologies are open source (even the Amazon SDK), so you can always dive into the bug reports, code etc. to understand how things actually work and find the solutions you need. Also, from time to time you can find articles and blog posts that explain how to overcome the common issues in technologies you are using. I hope this post clears off some of the complications  of integrating Spark, Parquet and S3, which are, at the end of the day, all great technologies with a lot of potential.

Image by xkcd. Licensed under creative-commons 2.5

Categories: Architecture