Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Three Fast Data Application Patterns

This is guest post by John Piekos, VP Engineering at VoltDB. I understand this is a little PRish, but I think the ideas are solid.

The focus of many developers and architects in the past few years has been on Big Data, specifically mining historical intelligence from the Data Lake (usually a Hadoop stack containing terabytes to petabytes of data).

Now, product architects are asking how they can use this business intelligence for competitive advantage. As a result, application developers have come to see the value of using and acting in real-time on streams of fast data; using OLAP reporting wisdom, they can realize the benefits of both fast data and Big Data. As a result, a new set of application patterns have emerged. The applications are designed to capture value from fast-moving streaming data, before it reaches Hadoop.

At VoltDB we call this new breed of applications “fast data” applications. The goal of these fast data applications is to do more than just push data into Hadoop asap, but also to capture real-time value from the data the moment the data arrives.  

Because traditional databases historically haven’t been fast enough, developers have been forced to go to great effort to build fast data applications - they build complex multi-tier systems often involving a handful of tools typically utilizing a dozen or more servers.  However, a new class of database technology, especially NewSQL offerings, has changed this equation.

If you have a relational database that is fast enough, highly available, and able to scale horizontally, the ability to build fast data applications becomes less esoteric and much more manageable. Three new real-time application patterns have emerged as the necessary dataflows to implement real-time applications. These patterns, enabled by new, fast database technology, are:

Categories: Architecture

Stuff The Internet Says On Scalability For April 10th, 2015

Hey, it's HighScalability time:


Beautiful, isn't it? It's the cerebral cortex of a rat that is organized like a mini-Internet.
  • $47 million: value of Cannabis per square km; $3.7 trillion: worldwide IT spending in 2014;  $41B: spend on spectrum; 48,000 square km: How Much Land Would it Take to Power the US via Solar; 2,000: Hadoop clusters in the world; 650 pounds: projected size of ET
  • Quotable Quotes:
    • John Hugg: The number one rule of 21st century data management: If a problem can be solved with an instance of MySQL, it’s going to be.
    • @sarahnovotny: "there is no compression algorithm for experience" - great quote from Andy Jassy at #AWSSummit
    • Steve Martin: I did stand-up comedy for eighteen years. Ten of those years were spent learning, four years were spent refining, and four were spent in wild success.
    • Yossi Vardi: Revenues kill the dream.
    • @AWSSummits: AdRoll's retargeting and real-time betting operates at 6 billion impressions/day at 100ms latency on #AWS #AWSSummit 
    • @AWS_Partners: Nike is operating 70+ services as production loads in #aws today #AWSSummit 
    • @bernardgolden: S3 usage up 102% YOY, ec2 93%: #AWSSummit
    • @bernardgolden: AWS growing over 40% yoy. Next earnings announcement s/b v interesting. #awssummit 
    • @AlexBalk: Here is my Apple Watch review: Your life is largely meaningless. No gadget can obscure its emptiness. You are dying every day.
    • Jonas: Google: all apps become search. Facebook: all apps become feeds. 
    • @jon_moore: most scalable/fast/reliable systems follow these principles: elastic; responsive; resilient; message-driven. #phillyete
    • mrmondo: NVMe [Non-Volatile Memory Express] is one of the most important changes to storage over the past decade.
    • Peter Thiel: Often the smarter people are more prone to trendy, fashionable thinking because they can pick up on things, they can pick up on cues more easily, and so they’re even more trapped by it than people of average ability
    • @nickstenning: The women and men who wrote the nearly bug-free code that controlled a $4Bn space shuttle and the lives of astronauts worked 8am to 5pm.

  • Have you been let down by miracle materials like carbon nanotubes, buckyballs, and graphene? MOFs  (metal–organic frameworks) are here and they are real. This Nature podcast and article tells you all about them (about 13 minutes in). MOFs are scaffolds made of metal containing nodes linked by carbon-based struts. They are pieces that you can plug together and build up into big networks which have spaces in-between. It's those spaces that make MOFs useful. You can trap things in those holes and do things to the molecules when they are trapped. You can store gasses like methane and hydrogen. You can separate mixture of things by varying the pore sizes. Carbon capture is one big use. They also can be used as chemical sensors, maybe in some future version of your watch. Also perhaps write-once-read-many times memory.

  • Is Amazon recreating the Sun ecosystem in the cloud? We now have the Amazon Elastic File System so everything is remote mounted. WorkSpaces feels like diskless workstations. Storage is over on some NAS. The database is somewhere on the network. And so on. Let's hope NFS lock contention failures and network UI jitter don't also make a comeback. OK, I don't remember having anything like Amazon Machine Learning

  • Etsy is giving Facebook's HipHop Virtual Machine (HHVM) for PHP a try. Why? Their API and web code was diverging under parallel development pressures. And they were developing many small API endpoints that used many small requests instead of larger requests that do more work per request. And instead of sharing state in an inherently shared nothing architecture they went with the strategy of just making things faster. This is where HHMV comes in.

  • OK, that's impressive. Migrating from Heroku to AWS (using Docker). It took two engineers about one month. Performance increased 2x and average API response time dropped from around 220ms to under 100ms, and our background task execution times dropped in half as well. Half the number of servers were needed.

  • I was excited to see AWS is opening up Lambda. It's close to some ideas I've been talking about for a while (Building Super Scalable Systems, What Google App Engine Price Changes Say About The Future Of Web Architecture). When it first came out I rehabed my atrophied node.js skills and gave it a shot. Played around a bit, got some code working, but the problem was Lambda only exposed a few integration points and none of those were anything I cared about. Now, they've made Lambda much more general and in the process much more useful. Worth another look. I also suspect their NFS product was necessary to generalize Lambda. Code could be instantly available on every machine via a mount point. Just like back in the day.

  • How Early Adopters Are Using Unikernels - With and Without Containers: The creator of MirageOS, Anil Madhavapeddy’s group is working on a new tool stack called Jitsu (Just-in-Time Summoning of Unikernels), which can start a unikernel in ~20ms in response to a network request. < Also, Towards Heroku for Unikernels: Part 2 - Self Scaling Systems.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Boost Your Confidence with the Right Words

Confidence is one of those things that makes all the difference when it comes to realizing your potential.

So many people have great ideas and great ambitions, but lack the confidence to follow through when it counts.

They hold themselves back.

Their amazing and bold ideas turn into lackluster ideas, as fear starts talking (if they talk at all.)

A while back, a team in HR interviewed me on confidence, because enough people pointed back to me as somebody they saw as confident.

What HR wanted to know is, where does my confidence come from?

For me, it mostly came from a relentless focus on making impact.

I put my focus on doing great things, creating raving fan customers, and taking on big challenges.

Where you put your focus, instantly changes your confidence.

If you’re too worried about how you look or how you sound, then you aren’t putting enough focus on the amazing thing you are trying to do.

So it wasn’t confidence per se.  It was more like putting my focus on the right things.

But there was more to it.  I was confident because of a few basic beliefs:

  1. I believed I’ll figure it out
  2. I believed that if I screw up, I’ll learn faster
  3. I believed that I don’t need to be the answer, but that I can always find the answer

Another thing that helped is that one of our leaders was relentless about “exposing our thinking.”   He wanted us to always detach ourselves from our ideas.   He wanted us to present ideas without being attached to them so they could be criticized and evaluated in more objective ways.

This sounds simple, but you’d be surprised how difficult it can be to detach yourself from ideas.   But the beauty is that when you are able to do this, your focus changes from defending your ideas, to really helping to create better ideas.   And this little shift takes you from fear or lack of confidence, to purposeful exploration, with full confidence.

Anyway, I think we become the thoughts that we think, so I think it’s really important to fill our head with the right words on confidence.

To that end, here is roundup of some of the greatest confidence quotes of all time:

Confidence Quotes

See if you can find at least three that you can use to help add more juice to your day.

If there is one that I find myself referring to all the time, to remind myself to get up to bat, it’s this one:

“A ship is safe in harbor, but that’s not what ships are for.” – William Shedd

I learned it long ago, and it’s served me well, ever since.

While that one reminds me to do what I do best, it’s really this one that inspires me to expand what I’m capable of:

“Life shrinks or expands in proportion to one’s courage.” — Anaïs Nin

I hope you find the right words that give your confidence just the boost it needs, when you need it most.

JD

Categories: Architecture, Programming

The Black Magic of Systematically Reducing Linux OS Jitter

How do you systematically identify the sources of jitter and remove them one by one in a low-latency trading system? That was the question asked on the mechanical-sympathy email list. 

Gil Tene, Vice President of Technology and CTO, Co-Founder, Azul Systems gave the sort of answer that comes from the accumulated wisdom born from lots of real-life experience. It's an answer that needed sharing. And here it is:

Finding the cause of hiccups/jitters in a a Linux system is black magic. You often look at the spikes and imagine "what could be causing this". 

Based on empirical evidence (across many tens of sites thus far) and note-comparing with others, I use a list of "usual suspects" that I blame whenever they are not set to my liking and system-level hiccups are detected. Getting these settings right from the start often saves a bunch of playing around (and no, there is no "priority" to this - you should set them all right before looking for more advice...).

My current starting point for Linux systems that are interested in avoiding many-msec hiccup levels is:

Categories: Architecture

Rolling upgrade of Docker applications using CoreOS and Consul

Xebia Blog - Mon, 04/06/2015 - 21:17

In the previous blog post, we showed you how to setup a High Available Docker Container Application platform using CoreOS and Consul. In this short blog post, we will show you how easy it is to perform a rolling upgrade of a deployed application.

Architecture

In this blog post we will use the same architecture but for the deployment we used Vagrant instead of Amazon AWS which is a little bit snappier to use  :-).

coreos-vagrant-cluster

In the Vagrant architecture we have replaced the AWS Elastic Load Balancer with our own consul-http-router Load Balancer and moved it inside the cluster. This is a HA proxy which dynamicly routes traffic to any of the http routers in the cluster.

Getting Started

In order to get your own container platform as a service running on vagrant, clone the repository and start your cluster.


git clone https://github.com/mvanholsteijn/coreos-container-platform-as-a-service.git
cd coreos-container-platform-as-a-service/vagrant
vagrant up
...
vagrant up
Bringing machine 'core-01' up with 'virtualbox' provider...
Bringing machine 'core-02' up with 'virtualbox' provider...
Bringing machine 'core-03' up with 'virtualbox' provider...
...
Checkout the cluster

After the cluster has started, you can use the following command to checkout whether your cluster is fully operational. You should see 6 units running on each machine.

for node in 1 2 3 ; do
  vagrant ssh -c "systemctl | grep consul" core-0$node
done
...
consul-http-router-lb.service loaded active running consul-http-router-lb
consul-http-router.service loaded active running consul-http-router
consul-server-announcer.service loaded active running Consul Server Announcer
consul-server-registrator.service loaded active running Registrator
consul-server.service loaded active running Consul Server Agent

 

Deploying the application

Once the cluster is running, you can deploy or paas-monitor application. This happens in two stages. First you submit the template.

fleetctl submit paas-monitor\@.service

Then, you load and start the new instances.

fleetctl load paas-monitor\@{1..6}.service
fleetctl start paas-monitor\@{1..6}.service
fleetctl list-units | grep paas-monitor

You can now see the paas-monitor in operation on your machine by opening http://paas-monitor.127.0.0.1.xip.io:8080 and clicking on start.  You should see something like shown in the table below. Leave this running while we are going to update the application!

host release message # of calls avg response time last response time 47ea72be3817:1337 v1 Hello World from v1 53 8 11 cc4227a493d7:1337 v1 Hello World from v1 59 11 8 04a58012910c:1337 v1 Hello World from v1 54 7 6 090caf269f6a:1337 v1 Hello World from v1 58 7 7 096d01a63714:1337 v1 Hello World from v1 53 7 9 d43c0744622b:1337 v1 Hello World from v1 55 7 6 Updating the application

Now we are going to update your application. Normally, we would expect that you specify a higher version of the Docker image into unit file. But instead of changing the version of the image to be executed change the value of the environment variable RELEASE in the unit template file  paas-monitor\@.service.

sed -i -e 's/--env RELEASE=[^ ]*/--env RELEASE=v2/' paas-monitor\@.service

Now we have changed the unit template file, you should destroy the old unit file and submit the new one.

fleetctl destroy paas-monitor\@.service
fleetctl submit paas-monitor\@.service

Now you have two options, a slow one and a fast one.

Slow Option

The slow option is to  iterate over the running instances, stop them one by one, destroy them and start a new instance based on the newly submitted template. Because this is boring, repetitive work we have created a small script that does this :-)


./rolling-upgrade.sh paas-monitor\@.service

Fast Option

The fast option is to start 6 new ones and stop all 6 old ones after the new ones are running.

fleetctl load paas-monitor\@1{1..6}.service
fleetctl start paas-monitor\@1{1..6}.service
fleetctl list-units | grep 'paas-monitor\@1[1-6].service' | grep running
fleetctl stop paas-monitor@{1..6}.service
fleetctl destroy paas-monitor@{1..6}.service

When you watch your monitor, you should see the new instance appear one by one.

host release message # of calls avg response time last response time 47ea72be3817:1337 v1 Hello World from v1 53 8 11 cc4227a493d7:1337 v1 Hello World from v1 59 11 8 04a58012910c:1337 v1 Hello World from v1 54 7 6 090caf269f6a:1337 v1 Hello World from v1 58 7 7 096d01a63714:1337 v1 Hello World from v1 53 7 9 d43c0744622b:1337 v1 Hello World from v1 55 7 6 fee39f857479:1337 v2 Hello World from v2 18 7 9 99c1a5aa3b8b:1337 v2 Hello World from v2 2 7 9 Conclusion

CoreOS and Consul provides all the basic functionality to manage your Docker containers and perform rolling upgrade of your application.

 

What do we know about how Meerkat Works?

“The future is live. The future is real-time. The future is now.” I wrote that in 2010 about live video innovator Justin.tv (which pivoted into Twitch.tv). Five years later it appears the future is now once again.

 Meerkat has burst on the scene with a viral vengeance, so I became curious. Meerkat is throwing around a lot of live video. It must be chewing up cash at an early funding round crushing rate. How does it work?

Unfortunately, after digging deep, I found few specific details on their backend architecture. What do we know?

  • The cash burning surmise seems to be correct. Meerkat secured another $12 million in funding. The streams will continue to flow. Bandwidth is cheaper than it used to be, but it is still expensive. Aether Wu did made an estimate over on Quora: So let's consider a scale of 1m users online simultaneously. Every 20 minutes, it costs 100k gigabyte, which means $4k per hour/$96k per day/ $2.9m per month. So if we scale the business to 10 times bigger, it is about $1m per day?

  • Meerkat was three years in development by an Israeli based team of up to 11 developers.

  • Meerkat can handle thousands of live streams while maintaining good video quality. Perhaps implementation details were discussed in a Meerkat session, but, well, you know. Ben Rubin, the thoughtful and nearly ubiquitous founder of Meerkat, wrote on ProductHunt that “I'm in love with HLS. For Meerkat use case HLS is better despite the 10-15 sec delay as it giving advantages in stable, crystal clear quality. We can use it to shift stream to audio only when connection is low, or do all sort of tricks.” HLS is HTTP Live Streaming.

That’s about it. While the backend architecture remains a mystery, what I did find is still very interesting. It’s the story of how a team creatively hunts and sifts through a space until they create/find that perfect blending of features that makes people fall in love. Twitter did that. SnapChat did that. Now Meerkat has done it. How did they do it?

Stats
Categories: Architecture

Increasing the tap area of UIButtons made with PaintCode

Xebia Blog - Sun, 04/05/2015 - 21:04

Whether you're a developer or not, every iPhone user has experienced the case where he or she tries to tap a button (with an image and nothing happens. Most likely because the user missed the button and pressed next to it. And that's usually not the fault of the user, but the fault of the developer/designer because the button is too small. The best solution is to have bigger icons (we're only talking about image buttons, no text only buttons) so it's easier for the user to tap. But sometimes you (or your designer) just wants to use a small icon, because it simply looks better. What do you do then?

For buttons with normal images this is very easy. Just make the button bigger. However, for buttons that draw themselves using PaintCode, this is slightly harder. In this blogpost I'll explain why and show two different ways to tackle this problem.

I will not go into the basics of PaintCode. If you're new to PaintCode have a look at their website or go to Freek Wielstra's blogpost Working With PaintCode And Interface Builder In XCode. I will be using the examples of his post as basis for the examples here so it's good to read his post first (though not strictly necessary).

To get a better understanding of how a UIButton with an image differs from a UIButton drawn with PaintCode, I will first show what a non-PaintCode button would look like. Below we have a PNG image of 25 by 19 pixels (for non-Retina). Apple recommends buttons to be 44 by 44 points so we increase the size of the button. The button has a gray background and we can see that the image stays the original size and is nicely in the center while the touch target becomes big enough for the user to easily tap it.

uibutton-content

The reason it behaves like this is because of the Control settings in the Attribute Inspector. We could let the content align different or even make it stretch. But stretching this would be a bad idea since it's not a vector image and it would look pixelated.

That's why we love PaintCode graphics. They can scale to any size so we can use the same graphic in different places and at different screen scales (non-retina, retina @2x and iPhone 6 plus which uses @3x). But what if we don't want the graphic to scale of to the entire size of the button like above?

A bad solution would be not to use a frame in PaintCode. Yes, the image will stay the size you gave it when you put it in a button, but it will be left-top aligned. Also you won't be able to use it anywhere else.

If you are really sure that you will never use the image anywhere else than on your button you can group your vector graphic and tell it to have flexible margins within the frame and have a fixed width and height. Using the email icon from Freek Wielstra this will look something like this:

Screen Shot 2015-04-05 at 20.39.36

You can verify that this works by changing the size of the frame. The email icon should always stay in the center both horizontal and vertical and it should stay the same size. The same will happen if you use this in a custom UIButton.

Now let's have a look at a better solution that will allow us to reuse the graphic at different sizes and have some padding around it in a button. It's easier than you might expect and it follows the actual rules of a UIButton. The UIButton class has a property named contentEdgeInsets with the following description: "The inset or outset margins for the rectangle surrounding all of the button’s content." That sounds exactly like what we need. Our PaintCode graphics are the content of the buttons right? Unfortunately it's not treated as such since we are doing the drawing ourselves which does not adhere to that property. However we can very easily make it adhere to it:

@IBDesignable
class EmailButton: UIButton {

    override func drawRect(rect: CGRect) {
        StyleKit.drawEmail(frame: UIEdgeInsetsInsetRect(rect, contentEdgeInsets), color: tintColor!)
    }
    
}

And that's really all there is to it. Just make sure to put a frame around your graphics in PaintCode and make it stretch to all sides of the frame. Now you can specify the content insets of your buttons in Interface Builder and you icon will have padding around it.

Screen Shot 2015-04-05 at 20.59.42

Working with PaintCode and Interface Builder in XCode

Xebia Blog - Sat, 04/04/2015 - 08:00

Every self-respecting iOS developer should know about PaintCode by now, an OSX app for drawing graphics that don't save as images, but as lengths of code that draw graphics. The benefits of this are vastly reduced app installation size - no need to include three resolutions of the same image for every image - and seamlessly scalable graphics.

One thing that I personally struggled with for a while now was how to use them effectively in combination with Interface Builder, the UI development tool for iOS and OSX apps. In this blog I will explain an effective and simple method to draw PaintCode graphics in a way where you can see what you're doing in Interface Builder, using the relatively new @IBDesignable annotation. I will also go into setting colors, and finally about how to deal with views that depend on dynamic runtime data to draw themselves.

First, let's create a simple graphic in PaintCode. Let's draw a nice envelope or email icon. Add a frame around it, and set the constraints to variable so it will adjust to the size of the frame (drag the frame corners around within PaintCode to confirm). This gives us a nice, scalable icon. Save the file, then export it to your xcode project in the language of your choice.

Email icon in PaintCode

Squiggly lines are the best

 

Now, the easiest and most straightforward way to get a PaintCode image into your Interface Builder is to simply create a UIView subclass, and call the relevant StyleKit method in its drawRect method. You can do this too in either Swift or Objective-C; this example will be in Swift, but if you're stuck with the slow Swift compiler in XCode 6.x, you might prefer Objective-C for this kind of simple and straightforward code.

To use it in Interface Builder, simply create a storyboard or xib, drag an UIView onto it, and change its Custom Class to your UIView subclass. Make sure to have your constraints set properly, and run - you should be able to see your custom icon in your working app.

class EmailIcon: UIView {
  override func drawRect(rect: CGRect) {
    StyleKit.drawEmail(frame: rect)
  }
}
Screenshot 2015-04-03 21.36.42

Ghost views, the horror of Interface Builder

 

As you probably noticed though, it's far from practical as it is in Interface Builder right now - all you see (or don't see really) is an empty UIView. I guess you could give it a background color in IB and unset that again in your code, but that's far from practical. Instead, we'll just slap an @IBDesignable annotation onto the UIView subclass. Going back to IB, it should start to compile code, and a moment later, your PaintCode image shows up, resizable and all.

@IBDesignable
class EmailIcon: UIView {
  override func drawRect(rect: CGRect) {
    StyleKit.drawEmail(frame: rect)
  }
}
@IBDesignable view in Interface Builder

like magic

 

Adding color

We can configure our graphic in PaintCode to have a custom color. Just go back to PaintCode and change the Color to 'Parameter' - see image.

Set color to Parameter

Export the file again, and change the call to the StyleKit method to include the color. It's easiest in this case to just pass the UIView's own tintColor property. Going back to Interface Builder, we can now set the default tintColor property, and it should update in the IB preview instantly.

As an alternative, you can add an @IBInspectable color property to your UIView, but I would only recommend this for more complicated UIViews - if they include multiple StyleKit graphics, for example.

@IBDesignable
class EmailIcon: UIView {
  override func drawRect(rect: CGRect) {
    StyleKit.drawEmail(frame: rect, color: tintColor)
  }
}

let's make it Xebia purple, to please the boss

 

Dealing with dynamic properties

One case I had to deal with is an UIView that can draw a selection of StyleKit images, depending on the value of a dynamic model value. I considered a few options to deal with that:

  1. Set a default value, to be overridden at runtime. This is a bit dangerous though; forget to reset or unset this property at runtime and your user will be stuck with some default placeholder, or worse, flat-out wrong information.
  2. Make the property @IBInspectable. This only works with relatively simple values though (strings, numbers, colors), and it has the same problem as the above.

What I needed was a way to set a property, but only from within Interface Builder. Luckily, that exists, as I found out later. In UIView, there is a method called prepareForInterfaceBuilder, which does exactly what it says on the tin - override the method to set properties only relevant in Interface Builder. So in our case:

@IBDesignable
class EmailIcon: UIView {
  // none set by default
  var iconType: String? = nil

  override func drawRect(rect: CGRect) {
    if iconType == "email" {
      StyleKit.drawEmail(frame: rect, color: tintColor)
    }

    if iconType = "email-read" {
      StyleKit.drawEmailRead(frame: rect, color: tintColor)
    }

    // if none of the above, draw nothing
  }

  // draw the 'email' icon by default in Interface Builder
  override func prepareForInterfaceBuilder() {
    iconType = "email"
  }
}

And that's all there is to it. You can use this method to create all of your graphics, keep them dynamically sized and colored, and to use the full power of Interface Builder and PaintCode combined.

Stuff The Internet Says On Scalability For April 3rd, 2015

Hey, it's HighScalability time:


Luscious SpaceX photos have been launched under Creative Commons.
  • 1,000: age of superbug treatment; 18 million: number of laws in the US
  • Quotable Quotes:
    • @greenberg: Only in the Bay Area would you find a greeting card for closing a funding round.
    • @RichardWarburto: "Do Not Learn Frameworks. Learn the Architecture"
    • Alex Dzyoba: Know your data and develop a simple algorithm for it.
    • @BenedictEvans: Akamai: 17% of US mobile connections are >4 Mbps. Most of the rest of the developed world is over 50%
    • Linus: Linux is just a hobby, won’t be big and professional like GNU
    • jhugg: This just lines up with what we've seen in the KV space over the last 5 years. Mutating data and key-lookup are all well and good, but without a powerful query language and real index support, it's much less interesting.
    • Facebook: Whatever the scale of your engineering organization, developer efficiency is the key thing that your infrastructure teams should be striving for. This is why at Facebook we have some of our top engineers working on developer infrastructure.
    • mysticreddit: Micro-optimization is a complete waste of time when you haven't spent time focusing on the meta & macro optimization
    • @adriancolyer: If you think cross-partition transactions can't scale, it's well worth taking a look at the RAMP model: 
    • @jasongorman: Microservices are a great solution to a problem you probably don't have
    • @dbrady: If 1 service dies and your whole system breaks, you don't have SOA. You have a monolith whose brain has been chopped up and stuck in jars.

  • Fascinating realization. We live in a world in which every tech interaction is subject to a man-in-the-middle attack. Future Crimes: All of this is possible because the screens on our phones show us not reality but a technological approximation of it. Because of this, not only can the caller ID and operating system on a mobile device be hacked, but so too can its other features, including its GPS modules. That’s right, even your location can be spoofed.

  • That's every interaction. Pin-pointing China's attack against GitHub: The way the attack worked is that some man-in-the-middle device intercepted web requests coming into China from elsewhere in the world, and then replaced the content with JavaScript code that would attack GitHub. 

  • Messaging and mobile platforms: If you take all of this together, it looks like Facebook is trying not to compete with other messaging apps but to relocate itself within the landscape of both messaging and the broader smartphone interaction model. 

  • Martin Thompson: Love the point that the compiler can only solve problems in the 1-10% problem space. The 90% problem space is our data access which is all about data structures and algorithms. The summary is he shows how instruction processing can be dwarfed by cache misses. This resonates for me with what I've seen in the field with customers in the high-performance space. Obvious caveat is applications where time is dominated by IO.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Animate constraints with a simple UIView extension in Swift

Xebia Blog - Wed, 04/01/2015 - 22:01

iOS Developers that are getting started with Auto Layout for the first time often run in trouble once they try to animate their views. The solution is quite simple, but in this post I'll create a simple Swift extension that makes it even easier.

When you don't use Auto Layout, you need to make the changes to your UIView frames inside an animations block. Those changes will then be animated. So without knowing any better, you would do the same for your Auto Layout constraints, which means changing the constraint constants inside the animations block. This doesn't work however, and a quick google search on "how to animate constraints" shows that your need only need to call view.layoutIfNeeded within the animations block after you've made the changes to the constraints.

A simple example of this:

heightConstraint.constant = 100
UIView.animateWithDuration(0.5, animations: { [weak self] in
  self?.view.layoutIfNeeded() ?? ()
})

it's necessary to call it this wat since updating constants of constraints doesn't immediately change the frames of the views affected by those constraints. That's why we need to call layoutIfNeeded, to update all the frames of our views.

Since we just want to be always safe with memory management we'll use a weak reference of self within the closure, even though we know the closure is not retained after the animation runs and the view will not be deallocated before the animation finishes (it's good practice to use weak whenever you can to avoid memory related problems).

The above sample is not a lot of code, but each time you want to animate something with constraints you need to write the same thing. Wouldn't it be much easier to write the following instead?

heightConstraint.constant = 100
view.animateConstraintWithDuration()

We can achieve that by adding an extension to UIView:

extension UIView {
  func animateConstraintWithDuration(duration: NSTimeInterval = 0.5) {
    UIView.animateWithDuration(duration, animations: { [weak self] in
      self?.layoutIfNeeded() ?? ()
    })
  }
}

Default values for the method parameters makes this even nicer. Above we used 0.5 as default duration but of course you can fill in duration you want when calling the method. Also we can add parameters with default values for the other parameters that animateWithDuration has.

The final extension
extension UIView {
  func animateConstraintWithDuration(duration: NSTimeInterval = 0.5, delay: NSTimeInterval = 0.0, options: UIViewAnimationOptions = nil, completion: ((Bool) -> Void)? = nil) {
    UIView.animateWithDuration(duration, delay:delay, options:options, animations: { [weak self] in
      self?.layoutIfNeeded() ?? ()
    }, completion: completion)
  }
}

There we have it. An extremely simple extension method that not only saves us a little bit of code each time we need to animate constraints but is also makes the code a lot cleaner since the name of the method clearly indicates what it does, as apposed to the layoutIfNeeded method within an animations block.

6 Ways to Defeat the Coming Robot Army Swarms


Phalanx Weapon System by EumenesOfCardia

For every new weapon there’s an often unexpected move made to counter it. You develop a rock, I develop a shield. You develop a castle, I develop a cannon. You develop a knight in shining armor, I develop a long bow. You develop nukes and I go MAD. You develop a hacker army, I corrupt technology.

You develop a swarming robot army, what do I do?

To answer that question Paul Scharre over on the War on the Rocks blog has written a mesmerizing 6 part series of articles on robotic swarm warfare (1, 2, 3, 4, 5, Counter-Swarm: A Guide to Defeating Robotic Swarms).

But first, let’s set up the fear part of the article.

War Has Gone Open-Source
Categories: Architecture

Sponsored Post: MongoDB, Aerospike, Nervana, SignalFx, InMemory.Net, Couchbase, VividCortex, Transversal, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • At Scalyr, we're analyzing multi-gigabyte server logs in a fraction of a second. That requires serious innovation in every part of the technology stack, from frontend to backend. Help us push the envelope on low-latency browser applications, high-speed data processing, and reliable distributed systems. Help extract meaningful data from live servers and present it to users in meaningful ways. At Scalyr, you’ll learn new things, and invent a few of your own. Learn more and apply.

  • Nervana Systems is hiring several engineers for cloud positions. Nervana is a startup based in Mountain View and San Diego working on building a highly scalable deep learning platform on CPUs, GPUs and custom hardware. Deep Learning is an AI/ML technique breaking all the records by a wide-margin in state of the art benchmarks across domains such as image & video analysis, speech recognition and natural language processing. Please apply here and mention “highscalability.com” in your message.

  • Linux Web Server Systems EngineerTransversal. We are seeking an experienced and motivated Linux System Engineer to join our Engineering team. This new role is to design, test, install, and provide ongoing daily support of our information technology systems infrastructure. As an experienced Engineer you will have comprehensive capabilities for understanding hardware/software configurations that comprise system, security, and library management, backup/recovery, operating computer systems in different operating environments, sizing, performance tuning, hardware/software troubleshooting and resource allocation. Apply here.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • MongoDB World brings together over 2,000 developers, sysadmins, and DBAs in New York City on June 1-2 to get inspired, share ideas and get the latest insights on using MongoDB. Organizations like Salesforce, Bosch, the Knot, Chico’s, and more are taking advantage of MongoDB for a variety of ground-breaking use cases. Find out more at http://mongodbworld.com/ but hurry! Super Early Bird pricing ends on April 3.
Cool Products and Services
  • Looking for a scalable NoSQL database alternative? Aerospike is validating the future of ACID compliant NoSQL with our open source Key-Value Store database for real-time transactions. Download our free Community Edition or check out the Trade-In program to get started. Learn more.

  • SignalFx: just launched an advanced monitoring platform for modern applications that's already processing 10s of billions of data points per day. SignalFx lets you create custom analytics pipelines on metrics data collected from thousands or more sources to create meaningful aggregations--such as percentiles, moving averages and growth rates--within seconds of receiving data. Start a free 30-day trial!

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • Top Enterprise Use Cases for NoSQL. Discover how the largest enterprises in the world are leveraging NoSQL in mission-critical applications with real-world success stories. Get the Guide.
    http://info.couchbase.com/HS_SO_Top_10_Enterprise_NoSQL_Use_Cases.html

  • VividCortex goes beyond monitoring and measures the system's work on your MySQL and PostgreSQL servers, providing unparalleled insight and query-level analysis. This unique approach ultimately enables your team to work more effectively, ship more often, and delight more customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Trends for 2015

It’s time to blow your mind with the amazing things going on around the world.

Each year, I try to create a bird’s-eye view of key trends.   It’s a mash up of all the changes in technology, consumerism, and ideas that seem to be taking off.

Here are my trends for 2015:

Trends for 2015:  The Year of Digital Transformation

I call it the year of Digital Transformation because of the encroachment of technology into all things business, work, and life.

Technology is everywhere (and it’s on more wrists than I’ve ever seen before.)

What’s different now is how the combination of Cloud + Mobile + Social + Big Data + Internet-of-Things are really changing how business gets done.  Businesses around the world are using Cloud, Mobile, Social, Big Data, and Internet-of-Things to leap frog ahead of their competitors and to differentiate in new and exciting ways.

Businesses around the world are using technology to gains insights and to shape their customer experience, transform their workforce, and streamline their back-office operations.

It’s a new race for leadership positions in the Digital Economy.   With infinite compute and capacity, along with new ways to connect around the world, business leaders and IT leaders are re-imagining the art of the possible for their businesses.

While Cloud, Mobile, Social, Big Data, and Internet-of-Things might be the backbone of the changes all around us, it’s business model innovation that is bringing these changes to the market and helping them take hold.

Here is a preview of 10 key trends from Trends for 2015:  The Year of Digital Transformation:

  1. The Age of Instant Gratification.
  2. It’s a Mobile Everything World.
  3. Businesses Look to the Cloud.
  4. Internet of Things (IoT)
  5. Design is everywhere.
  6. Economy Re-Imagined (“The Circular Economy”)
  7. Culture of Health.
  8. Money is Reimagined (“The Future of Payments and Currency”)
  9. Digital Personal Assistants are Everywhere.
  10. Renegades and Rebels Rule the World

As a tip, when you read the post, try to scan it first, all the way down, so you can see the full collection of ideas.  Then circle back and slow down to really absorb the full insight.   You’ll find that you’ll start to see more patterns across the trends, and you’ll start to connect more dots.

I designed the post to make it easy to scan, as well as to read it end-to-end in depth.  I think it’s more valuable to be able to quickly take the balcony view, before diving in.   This way, you get more of a full picture view of what’s happening around the world.  Even if you don’t master all the trends, a little bit of awareness can actually go a long way. 

In fact, you might surprise yourself as some of the trends pop into your mind, while you’re working on something completely different.   By having the trends at my fingertips, I’m finding myself seeing new patterns in business, along with new ways that technology can enhance our work and life.

Trends actually become a vocabulary for generating and shaping new ideas.   There are so many ways to arrange and re-arrange the constellation of ideas.  You’ll find that I paid a lot of attention to the naming of each trend.  I tried to share what was already pervasive and sticky, or if it was complicated, I tried to turn it into something more memorable.

Use the trends as fodder and insight as you pave your way through your own Digital Transformation.

Categories: Architecture, Programming

Components vs classes

Coding the Architecture - Simon Brown - Tue, 03/31/2015 - 10:40

When discussing my C4 model for describing software architecture, I often get asked what the difference is between components and classes. In a nutshell, I like to think of a component as being a grouping of related functionality behind a nice clean interface. Of course, you could say the same about services, microservices or classes. So, let me show an example.

The Spring PetClinic application is a sample codebase used to illustrate how to use the Spring framework for building web applications in Java. If you download a copy of the GitHub repository and open it in your IDE of choice, you'll see the code looks like this.

Spring Petclinic code

Let's visualise this by drawing a class diagram of the code.

Spring Petclinic

This diagram shows all of the classes/interfaces and all of the relationships between them. The properties and methods are hidden from view because that adds too much noise into the picture. This isn't a complex codebase by any stretch of the imagination, but the diagram is showing too much detail. Let's remove those classes which aren't relevant to having an "architecture" discussion about the system. In other words, let's only try to show those classes that have some structural significance. In concrete terms, this means excluding the model (domain) and util classes.

Spring Petclinic

This diagram is much better, but in order to show the true picture of the dependencies, we've needed to show the interface and implementation classes for the service and repositories. Now that we have a much simpler diagram with which to reason about the software architecture, perhaps we can now show the methods.

Spring Petclinic

Then again, perhaps not! And this is a shame. Although I like the simpler diagram we saw before, it doesn't really tell me anything about the responsibilities of the classes. And having the interfaces and classes shown separately on the diagram seems a little like a workaround. Instead, let's treat the ClinicService and each of the *Repository things as a "component" by collapsing the interface and implementation classes. This is exactly what I'm trying to achieve with Structurizr.

Spring Petclinic

As this nicely illustrates, for me anyway, a component is simply a collection of implementation classes behind an interface. And rather than diagramming the component internals, I want to be able to zoom out slightly to see these components and how they are related to one another. This still leaves me with the ability to zoom in to see the internals of a component if I need to, but I don't want that view by default. It's too detailed and, besides, I can find that in the code myself if I know where to look. Thankfully the diagram you see above does have a relationship with the code. Try double-clicking on a component in the live version of the Spring PetClinic diagram. The diagram reflects the code.

Categories: Architecture

How We Scale VividCortex's Backend Systems

This is guest post by Baron Schwartz, Founder & CEO of VividCortex, the first unified suite of performance management tools specifically designed for today's large-scale, polyglot persistence tier.

VividCortex is a cloud-hosted SaaS platform for database performance management. Our customers install agents that measure the work their servers perform (queries, processes, etc) and generate metrics and events from that at high frequency. The agents send the resulting data to our APIs, where we host our analysis backend. The backend system is a collection of databases, internal services (quasi-microservices), and web-facing APIs. These APIs also power our AngularJS frontend application.

We deal with a lot of data. We ingest metrics and events at high speed. We also perform analytics that touch large amounts of data interactively. We are not unique and I don't want to imply we are somehow impressive in the scheme of things. We don't yet operate at "web scale." Nevertheless, our workload has some relatively unusual characteristics, and we've been able to scale as far as we have, while remaining pretty efficient in terms of cost and infrastructure. And my career in consulting has taught me that building systems like this is usually a challenge for a company (as it has been for us). Our story might be useful to others. For that reason I will go into unnecessary detail on specific parts of our workload and the challenges it brings.

What We Do
Categories: Architecture

Internet of things

Xebia Blog - Sat, 03/28/2015 - 11:26

Do you remember the time that you were not connected to the internet? When you had to call your friend instead of sending a message via Whatsapp to make an appointment. The times you had to apply for a job by writing a letter with  a pen instead of sending your Linkedin page? We all have gone through a major revolution in which we've all got a digital identity. The same revolution has started for things, right now!

Internet of Things

Potential

Things we use all day, like your watch or thermostat, get suddenly connected to the internet. These things are equipped with electronics, sensors and connectivity. Whether you like it or not, you will not escape this movement. According to Gartner there will be nearly 26 billion devices on the Internet of Things by 2020.
The potential of the Internet of Things isn’t just linking millions of devices. It is more like the potential of Google or Amazon had when building massive IT infrastructure. The IoT is about transforming business models and enabling companies to sell products in entirely new and better ways.

Definition

The definition of IoT is fairly simple. The internet of things involves `every thing that is connected to the internet`. You should exclude mobiles and computers, because they've already become mainstream.

In fact, the Internet of things is a name for a set of trends. First of all we have the trend that objects, we use all day,  are suddenly connected to the internet. For example bathroom scales that compares your weight with others.
Another trend is that devices that were already shipped with CPUs have become much more powerful when they’re connected to a network. Functionality of the hardware depends on local computing as well as powerful cloud computing. Fitness tracker, can carry out some functionality, like basic data cleaning and tabulation, on local hardware, but it depends on sophisticated machine-learning algorithms on an ever improving cloud service to offer prescription and insight.
Besides that we have the makers movement. Creating hardware isn’t for the big companies like Sony or Samsung anymore, just like creating software in the 80’s and 90’s was a privilege for IBM and Microsoft.

What’s new?

Devices that are connected to the internet aren’t new. For many years, we've connected devices to the internet, although not at that size. I think we have to compare it concepts like 'the cloud'. Before I heard the name 'the cloud' I was already using services that run on the internet, like Hotmail. But one swallow doesn't make a summer. When suddenly lots of services are running on the web, we speak of a trend and the market will give it a name to it. The same is happening with the internet of things. Suddenly all kinds of 'normal' devices are connected to the internet and we have a movement towards that direction that is unstoppable. And the possibilities it will bring are endless.

Splitting up in pieces

We can divide the internet of things into five smart categories. First of all we have the smart wearable. It is designed for a variety of purposes as well as for wear on a variety of parts of the body. Think of a bicycle helmet that detects a crash. Next to it we have the smart home. It’s goal is to make the experience of living at home more convenient and pleasant. A thermostat that learns what temperatures users like and builds a context-aware personalized schedule. A third and fourth category are the smart city and smart environment. It focuses on finding sustainable solutions to the growing problems. The last category is very interesting. It is the smart enterprise. One can think of real-time shipment tracking or smart metering solution that manages energy consumption at the individual appliance and machine level.

Changes of business models

There are plenty of new business models due to the internet of things. An example is a system that provides a sensor-embedded trash, so it’s capable of real-time context analysis and alerting the authorities when it is full and needs to be emptied. Another example is the insurance world, which will adopt the internet of things. There's already behaviour-based car insurance pricing available. Customers who agree to install a monitoring device on their car's diagnostic port, get a break on their insurance rates and pricing that varies according to usage and driving habits.

Big data

Other disciplines in software development also have interest in the new movement, like big data. Internet of things will not generate big data, but obesitas data. Did you know that within every flight, there are  3 terabytes of data generated with all the sensors in an airplane? Think about how much data will be generated, that has to be analyzed, when all buildings in a town are equipped with sensors to measure the quality of the air?

Conclusion

The intersection between software and the physical world becomes smaller every day and that is the result of the unstoppable movement of internet of things. It is still at the beginning. Business models are changing and there are lots of opportunities. You just have to find them...

7 Habits of Highly Motivated People

Motivation is a powerful skill.

It can lift you up from the worst of places, and inspire you to new heights.

After all, nothing is worse than slogging your way through your days, or working your way through a bunch of mundane tasks.

But, like I said, motivation is a skill.

You need to learn it.   For many people it does not come naturally.   And chances are, many of us have had bad models, bad advice, and worse, bad habits for a lifetime.

One of the most important insights I found was said by Stephen Covey long ago – satisfied needs don’t motivate.

It’s why we need to stay hungry.

Here’s how you stay hungry -- find a problem you hate, and focus on creating a solution you love.

But how you light your fire from the inside out in a sustainable way?

That’s where the 7 habits of highly motivated people comes in.

I wanted to put together a very simple set of habits and practices that actually work for building your motivational muscle and finding your inner mojo.

Here are the 7 habits of highly motivated people at a glance:

  1. Find Your WHY
  2. Change Your Beliefs About What’s Possible
  3. Change Your Beliefs That Limit You
  4. Spend More Time In Your Values
  5. Surround Yourself With Catalysts
  6. Build Better Feedback Loops
  7. “Pull” Yourself with Compelling Goals

There is a lot of science behind the habits.  If you’re that motivated, you can research it through bunches of books, bunches of sites, and brilliant TED talks.

But, I’d much rather that you spent the time simply adopting and applying the habits, so you can set your motivation on fire.

It’s time to do more of what you were born to do.

It’s time to live and breathe the things that you want to live and breathe.

It’s time to rise again from whatever ashes might have burned you down, and let your phoenix fly.

If you aren’t sure where to get started, first read 7 habits of highly motivated people, and then adopt habit #1:

Find Your WHY.

You’ll be glad you did.

I can see your pilot light is on already.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For March 27th, 2015

Hey, it's HighScalability time:


@scienceporn: That Hubble Telescope picture explained in depth. I have never had anything blow my mind so hard.

  • hundreds of billions: files in Dropbox; $2 billion: amount Facebook saved building their own servers, racks, cooling, storage, flat fabric, etc.
  • Quotable Quotes:
    • Buckminster Fuller: I was born in the era of the specialist. I set about to be purposely comprehensive. I made up my mind that you don't find out something just to entertain yourself. You find out things in order to be able to turn everything not just into a philosophical statement, but actual tools to reorganize the environment of man by which greater numbers of men can prosper. That's been my main undertaking.
    • @mjpt777: PCI-e based SSDs are getting so fast. Writing at 1GB/s for well less than $1000 is so cool.
    • @DrQz: All meaning has a pattern, but not all patterns have a meaning.
    • Stu: “Exactly once” has *always* meant “at least once but dupe-detected”. Mainly because we couldn’t convince customers to send idempotent and communitative state changes.
    • @solarce: When some companies have trouble scaling their database they use Support or Consultants. Apple buys a database vendor. 
    • @nehanarkhede: Looks like Netflix will soon surpass LinkedIn's Kafka deployment of 800B events a day. Impressive.
    • @ESPNFantasy: More than 11.57 million brackets entered. Just 14 got the entire Sweet 16 correct.
    • @BenedictEvans: A cool new messaging app getting to 1m users is the new normal. Keeping them, and getting to 100m, is the question.
    • @jbogard: tough building software systems these days when your only two choices are big monoliths and microservices
    • @nvidia: "It isn't about one GPU anymore, it's about 32 GPUs" Andrew Ng quotes Jen-Hsun Huang. GPU scaling is important #GTC15

  • FoundationDB, a High Scalability advertiser and article contributer, has been acquired. Apple scooped them up. Though saving between 5% to 10% less hardware than Cassandra seems unlikely. And immediately taking their software off GitHub is a concerning trend. It adds uncertainty to the entire product selection dance. Something to think about.

  • In the future when an AI tries to recreate a virtual you from your vast data footprint, the loss of FriendFeed will create a big hole in your virtual personality. I think FF catches a side of people that isn't made manifest in other mediums. Perhaps 50 years from now people will look back on our poor data hygiene with horror and disbelief. How barbaric they were in the past, people will say. 

  • When the nanobots turn the world to goo this 3D printer can recreate it again. New 3-D printer that grows objects from goo. Instead of a world marked by an endless battle of good vs evil we'll have a ceaseless cycle of destruction and rebirth through goo. That's unexpected. A modern mythology in the making.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

An interview with O'Reilly

Coding the Architecture - Simon Brown - Thu, 03/26/2015 - 15:02

While at the O'Reilly Software Architecture conference in Boston last week, I was interviewed by O'Reilly about a number of things, including the software architecture role and the tension between software architecture and code.

This interview originally appeared in Signals from the O’Reilly Software Architecture Conference 2015 that looks at some of the key insights from the event. The slides from my talk titled Software architecture vs code are available to view online/download.

Categories: Architecture

The Convergence that Changes Everything: Data Gravity + Containers + Microservices

What will change everything this time?

  • Data Gravity (move compute to data) + 
  • Containers (lightweight computation) +
  • Microservices (composable compute modules)

That’s according to John Willis and Damon Edwards on a highly recommended recent DevOps Cafe Episode.

The high concept goes something like:

  • Containers are the killer app for microservices.

  • Data Gravity is the killer app for containers.

The logic goes something like:

Categories: Architecture