Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Released Today: Visual Studio 2015, ASP.NET 4.6, ASP.NET 5 & EF 7 Previews

ScottGu's Blog - Scott Guthrie - Mon, 07/20/2015 - 16:14

Today is a big day with major release announcements for Visual Studio 2015, Visual Studio 2013 Update 5, and .NET Framework 4.6. All these releases have been covered in great detail on Soma’s Blog, Visual Studio Blog, and .NET Blog

Join us online for the Visual Studio 2015 Release Event, where you can see Soma, Brian Harry, Scott Hanselman, and many other demo new Visual Studio 2015 features and technologies. This year, in a new segment called “In The Code”, we share how a team of Microsoft engineers created a real app in 3 days. There will be opportunities along the way to interact in live Q&A with the team on subjects such as Agile development, web and cloud development, cross-platform mobile dev and much more. 

In this post I’d like to specifically talk about some of the ground we have covered in ASP.NET and Entity Framework.  In this release of Visual Studio, we are releasing ASP.NET 4.6, updating our Visual Studio Web Development Tools, and updating the latest beta release of our new ASP.NET 5 framework.  Below are details on just a few of the great updates available today: ASP.NET Tooling Improvements

Today’s VS 2015 release delivers some great updates for web development.  Here are just a few of the updates we are shipping in this release: JSON Editor

JSON has become a first class experience in Visual Studio 2015 and we are now giving you a great editor to allow you to maintain your JSON content.  With support for JSON Schema validation, intellisense, and support for SchemaStore.org writing and producing JSON content has never been as easy.  We’ve also added intellisense support for bower.json and package.json files for bower and npm package manager use.

image HTML Editor Updates

Our HTML editor received a lot of attention in this update.  We wanted to deliver an editor that kept up with HTML 5 standards and provided rich support for popular new frameworks and libraries.  We previously shipped the bootstrap responsive web framework with our ASP.NET templates, and we are now providing intellisense for their classes with an indicator icon to show that they are bootstrap CSS classes.

image

 

This helps you keep clear the classes that you wrote in your project, like the page-inner class above, and the bootstrap classes marked with the B icon.

We are also keeping up with support for the emerging web components standard with the import link for the web components that markup imports.

 image

We are also providing intellisense for AngularJS directives and attributes with an appropriate Angular icon so you know you’re triggering AngularJS functionality

 image JavaScript Editor Improvements

With the VS 2015 release we are introducing support for AngularJS structures including controllers, services, factories, directives and animations.  There is also support for the new EcmaScript 6 features such as classes, arrow functions, and template strings. We are also bringing a navigation bar to the editor to help you navigate between the major elements of your JavaScript.  With JSDoc support to deliver intellisense, JavaScript development gets easier.

 image ReactJS Editor Support

We spent some time with the folks at Facebook to make sure that we delivered first class capabilities for developers using their ReactJS framework.  With appropriate syntax highlighting and intellisense for React methods, developers should be very comfortable building React applications with the new Visual Studio:

 image Support for JavaScript package managers like Grunt and Gulp and Task Runners

JavaScript and modern web development techniques are the new recommended way to build client-side code for your web application.  We support these tools and programming techniques with our new Task Runner Explorer that executes grunt and gulp task runners.  You can open this tool window with the Ctrl+Alt+Backspace hotkey combination.

 image

Execute any of the tasks defined in your gruntfile.js or gulpfile.js by right-clicking on the task name in the left panel and choosing “Run” from the context menu that appears.  You can even use this context menu to attach grunt or gulp tasks to project build events in Visual Studio like “After Build” as shown in the figure above.  Every time the .NET objects in your web project are completed compiling, the ‘build’ task will be executed from the gruntfile.js

Combined with the intellisense support for JavaScript and JSON editors, we think that developers wanting to use grunt and gulp tasks will really enjoy this new Visual Studio experience.  You can add grunt and gulp tasks with the newly integrated npm package manager capabilities.  When you create a package.json file in your web project, we will install and upgrade local copies of all packages referenced.  Not only do we deliver syntax highlighting and intellisense for package.json terms, we also provide package name and version lookup against the npmjs.org gallery.

 image

The bower package manager is also supported with great intellisense, syntax highlighting and the same package name and version support in the bower.json file that we provide for package.json.

 image

These improvements in managing and writing JavaScript configuration files and executing grunt or gulp tasks brings a new level of functionality to Visual Studio 2015 that we think web developers will really enjoy.

ASP.NET 4.6 Runtime Improvements

Today’s release also includes a bunch of enhancements to ASP.NET from a runtime perspective. HTTP/2 Support

Starting with ASP.NET 4.6 we are introducing support for the HTTP/2 standard.  This new version of the HTTP protocol delivers a true multiplexing of requests and responses between browser and web server.  This exciting update is as easy as enabling SSL in your web projects to immediately improve your ASP.NET application responsiveness.

 image

With SSL enabled (which is a requirement of the HTTP/2 protocol), IISExpress on Windows 10 will begin interacting with the browser using the updated protocol.  The difference between the protocols is clear.  Consider the network performance presented by Microsoft Edge when requesting the same website without SSL (and receiving HTTP/1.x) and with SSL to activate the HTTP/2 protocol:

image

image

Both samples are showing the default ASP.NET project template’s home page.  In both scenarios the HTML for the page is retrieved in line 1.  In HTTP/1.x on the left, the first six elements are requested and we see grey bars to indicate waiting to request the last two elements.  In HTTP/2 on the right, all eight page elements are loaded concurrently, with no waiting. Support for the .NET Compiler Platform

We now support the new .NET compilers provided in the .NET Compiler Platform (codenamed Roslyn).  These compilers allow you to access the new language features of Visual Basic and C# throughout your Web Forms markup and MVC view pages.  Our markup can look much simpler and readable with new language features like string interpolation:

Instead of building a link in Web Forms like this:

  <a href="/Products/<%: model.Id %>/<%: model.Name %>"><%: model.Name %></a>

We can deliver a more readable piece of markup like this:

  <a href="<%: $"/Products/{model.Id}/{model.Name}" %>"><%: model.Name %></a>

We’ve also bundled the Microsoft.CodeDom.Providers.DotNetCompilerPlatform NuGet package to enable your Web Forms assets to compile significantly faster without requiring any changes to your code or project. Async Model Binding for Web Forms

Model binding was introduced for Web Forms applications in ASP.NET 4, and we introduced async methods in .NET 4.5  We heard your requests to be able to execute your model binding methods on a Web Form asynchronously with the new language features.  Our team has made this as easy as adding an async=”true” attribute to the @Page directive and return a Task from your model binding methods:

    public async Task<IEnumerable<Product>> myGrid_GetData()

    {

      var repo = new Repository();

      return await repo.GetAll();

    }

We have a blog post demonstrating with more information and tips about this feature on our MSDN Web Development blog. ASP.NET 5

I introduced ASP.NET 5 back in February and shared in detail what this release would bring. I’ll reiterate just a few high level points here, check out my post Introducing ASP.NET 5 for a more complete run down. 

ASP.NET 5 works with .NET Core as well as the full .NET Framework to give you greater flexibility when hosting your web apps. With ASP.NET MVC 6 we are merging the complimentary features and functionality from MVC, Web API, and Web Pages. With ASP.NET 5 we are also introducing a new HTTP request pipeline based on our learnings from Katana which enables you to add only the components you need with an opt-in strategy. Additionally, included in this release are multiple development features for improved productivity and to enable you to build better web applications. ASP.NET 5 is also open source. You can find us on GitHub, view and download the code, submit changes, and track when changes are made.   

The ASP.NET 5 Beta 5 runtime packages are in preview and not recommended for use in production, so please continue using ASP.NET 4.6 for building production grade apps. For details on the latest ASP.NET 5 beta enhancements added and issues fixed, check out the published release notes for ASP.NET 5 beta 5 on GitHub. To get started with ASP.NET 5 get the docs and tutorials on the ASP.NET site

To learn more and keep an eye on all updates to ASP.NET, checkout the Webdev blog and read along with the tutorials and documentation at www.asp.net/vnext Entity Framework

With today’s release, we not only have an update to Entity Framework 6 that primarily includes bug fixes and community contributions, but we also released a preview version of Entity Framework 7, keep reading for details: Entity Framework 6.x

Visual Studio 2015 includes Entity Framework 6.1.3. EF 6.1.3 primarily focuses on bug fixes and community contributions; you can see a list of the changes included in EF 6.1.3 in this EF 6.1.3 announcement blog post. The Entity Framework 6.1.3 runtime is included in a number of places in this release. In EF 6.1.3 when you can create a new model using the Entity Framework Tools in a project that does not already have the EF runtime installed, the runtime is automatically installed for you. Additionally, the runtime is pre-installed in new ASP.NET projects, depending on the project template you select.

image 

To learn more and keep an eye on all updates to Entity Framework, checkout the ADO.NET blog.   Entity Framework 7

Entity Framework 7 is in preview and not yet ready for production yet. This new version of Entity Framework enables new platforms and new data stores. Universal Windows Platform, ASP.NET 5, and traditional desktop applications can now use EF7. EF7 can also be used in .NET applications that run on Mac and Linux. Visual Studio 2015 includes an early preview of the EF7 runtime that is installed in new ASP.NET 5 projects. 

image

For more information on EF7, check out the GitHub page for what is EF7 all about.

image Summary

Today’s Visual Studio release is a big one that we are proud to share with you all. Thank you for your continued support by providing feedback on the interim releases (CTPs, Preview, RC).  We are really looking forward to seeing what you build with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me @scottgu omni

Categories: Architecture, Programming

How To Get Smarter By Making Distinctions

"Whatever you do in life, surround yourself with smart people who'll argue with you." -- John Wooden

There’s a very simple way to get smarter.

You can get smarter by creating categories.

Not only will you get smarter, but you’ll also be more mindful, and you’ll expand your vocabulary, which will improve your ability to think more deeply about a given topic or domain.

In my post, The More Distinctions You Make, the Smarter You Get, I walk through the ins and outs of creating categories to increase your intelligence, and I use the example of “fat.”   I attempt to show how “Fat is bad” isn’t very insightful, and how by breaking “fat” down into categories, you can dive deeper and reveal new insight to drive better decisions and better outcomes.

I’m this post, I’m going to walk this through with an example, using “security” as the topic.

The first time I heard the word “security”, it didn’t mean much to me, beyond “protect.”

The next thing somebody taught me, was how I had to focus on CIA:  Confidentiality, Integrity, and Availability.

That was a simple way to break security down into meaningful parts.

And then along came Defense in Depth.   A colleague explained that Defense in Depth meant thinking about security in terms of multiple layers:  Network, Host, Application, and Data.

But then another colleague said, the real key to thinking about security and Defense in Depth, was to think about it in terms of people, process, and technology.

As much as I enjoyed these thought exercises, I didn’t find them actionable enough to actually improve software or application security.  And my job was to help Enterprise developers build better Line-Of-Business applications that were scalable and secure.

So our team went to the drawing board to map out actionable categories to take application security much deeper.

Right off the bat, just focusing on “application” security vs. “network” security or “host” security, helped us to get more specific and make security more tangible and more actionable from an Line-of-Business application perspective.

Security Categories

Here are the original security categories that we used to map out application security and make it more actionable:

  1. Input and Data Validation
  2. Authentication
  3. Authorization
  4. Configuration Management
  5. Sensitive Data
  6. Session Management
  7. Cryptography
  8. Exception Management
  9. Auditing and Logging

Each of these buckets helped us create actionable principles, patterns, and practices for improving security.

Security Categories Explained

Here is a brief description of each application security category:

Input and Data Validation
How do you know that the input your application receives is valid and safe? Input validation refers to how your application filters, scrubs, or rejects input before additional processing. Consider constraining input through entry points and encoding output through exit points. Do you trust data from sources such as databases and file shares?

Authentication
Who are you? Authentication is the process where an entity proves the identity of another entity, typically through credentials, such as a user name and password.

Authorization
What can you do? Authorization is how your application provides access controls for resources and operations.

Configuration Management
Who does your application run as? Which databases does it connect to? How is your application administered? How are these settings secured? Configuration management refers to how your application handles these operational issues.

Sensitive Data
How does your application handle sensitive data? Sensitive data refers to how your application handles any data that must be protected either in memory, over the network, or in persistent stores.

Session Management
How does your application handle and protect user sessions? A session refers to a series of related interactions between a user and your Web application.

Cryptography
How are you keeping secrets (confidentiality)? How are you tamper-proofing your data or libraries (integrity)? How are you providing seeds for random values that must be cryptographically strong? Cryptography refers to how your application enforces confidentiality and integrity.

Exception Management
When a method call in your application fails, what does your application do? How much do you reveal? Do you return friendly error information to end users? Do you pass valuable exception information back to the caller? Does your application fail gracefully?

Auditing and Logging
Who did what and when? Auditing and logging refer to how your application records security-related events.

As you can see, just by calling out these different categories, you suddenly have a way to dive much deeper and explore application security in depth.

The Power of a Security Category

Let’s use a quick example.  Let’s take Input Validation.

Input Validation is a powerful security category, given how many software security flaws and how many vulnerabilities and how many attacks all stem from a lack of input validation, including Buffer Overflows.

But here’s the interesting thing.   After quite a bit of research and testing, we found a powerful security pattern that could help more applications stand up to more security attacks.  It boiled down to the following principle:

Validate for length, range, format, and type.

That’s a pithy, but powerful piece of insight when it comes to implementing software security.

And, when you can’t validate the input, make it safe by sanitizing the output.  And along these lines, keep user input out of the control path, where possible.

All of these insights flow from just focusing on Input Validation as a security category.

Threats, Attacks, Vulnerabilities, and Countermeasures

Another distinction our team made was to think in terms of threats, attacks, vulnerabilities, and countermeasures.  We knew that threats could be intentional and malicious (as in the case of attacks), but they could also be accidental and unintended.

We wanted to identify vulnerabilities as weaknesses that could be addressed in some way.

We wanted to identify countermeasures as the actions to take to help mitigate risks, reduce the attack surface, and address vulnerabilities.

Just by chunking up the application security landscape into threats, attacks, vulnerabilities, and countermeasures, we empowered more people to think more deeply about the application security space.

Security Vulnerabilities Organized by Security Categories

Using the security categories above, we could easily focus on finding security vulnerabilities and group them by the relevant security category.

Here are some examples:

Input/Data Validation

  • Using non-validated input in the Hypertext Markup Language (HTML) output stream
  • Using non-validated input used to generate SQL queries
    Relying on client-side validation
  • Using input file names, URLs, or user names for security decisions
  • Using application-only filters for malicious input
  • Looking for known bad patterns of input
  • Trusting data read from databases, file shares, and other network resources
  • Failing to validate input from all sources including cookies, query string parameters, HTTP headers, databases, and network resources

Authentication

  • Using weak passwords
  • Storing clear text credentials in configuration files
  • Passing clear text credentials over the network
  • Permitting over-privileged accounts
  • Permitting prolonged session lifetime
  • Mixing personalization with authentication

Authorization

  • Relying on a single gatekeeper
  • Failing to lock down system resources against application identities
  • Failing to limit database access to specified stored procedures
  • Using inadequate separation of privileges

Configuration Management

  • Using insecure administration interfaces
  • Using insecure configuration stores
  • Storing clear text configuration data
  • Having too many administrators
  • Using over-privileged process accounts and service accounts

Sensitive Data

  • Storing secrets when you do not need to
  • Storing secrets in code
  • Storing secrets in clear text
  • Passing sensitive data in clear text over networks

Session Management

  • Passing session identifiers over unencrypted channels
  • Permitting prolonged session lifetime
  • Having insecure session state stores
  • Placing session identifiers in query strings

Cryptography

  • Using custom cryptography
  • Using the wrong algorithm or a key size that is too small
  • Failing to secure encryption keys
  • Using the same key for a prolonged period of time
  • Distributing keys in an insecure manner

Exception Management

  • Failing to use structured exception handling
  • Revealing too much information to the client

Auditing and Logging

  • Failing to audit failed logons
  • Failing to secure audit files
  • Failing to audit across application tiers
Threats and Attacks Organized by Security Categories

Again, using our security categories, we could then group threats and attacks by relevant security categories.

Here are some examples of security threats and attacks organized by security categories:

Input/Data Validation

  • Buffer overflows
  • Cross-site scripting
  • SQL injection
  • Canonicalization attacks
  • Query string manipulation
  • Form field manipulation
  • Cookie manipulation
  • HTTP header manipulation

Authentication

  • Network eavesdropping
  • Brute force attacks
  • Dictionary attacks
  • Cookie replay attacks
  • Credential theft

Authorization

  • Elevation of privilege
  • Disclosure of confidential data
  • Data tampering
  • Luring attacks

Configuration Management

  • Unauthorized access to administration interfaces
  • Unauthorized access to configuration stores
  • Retrieval of clear text configuration secrets
  • Lack of individual accountability

Sensitive Data

  • Accessing sensitive data in storage
  • Accessing sensitive data in memory (including process dumps)
  • Network eavesdropping
  • Information disclosure

Session Management

  • Session hijacking
  • Session replay
  • Man-in-the-middle attacks

Cryptography

  • Loss of decryption keys
  • Encryption cracking

Exception Management

  • Revealing sensitive system or application details
  • Denial of service attacks

Auditing and Logging

  • User denies performing an operation
  • Attacker exploits an application without trace
  • Attacker covers his tracks
Countermeasures Organized by Security Categories

Now here is where the rubber really meets the road.  We could group security countermeasures by security categories to make them more actionable.

Here are example security countermeasures organized by security categories:

Input/Data Validation

  • Do not trust input
  • Validate input: length, range, format, and type
  • Constrain, reject, and sanitize input
  • Encode output

Authentication

  • Use strong password policies
  • Do not store credentials
  • Use authentication mechanisms that do not require clear text credentials to be passed over the network
  • Encrypt communication channels to secure authentication tokens
  • Use HTTPS only with forms authentication cookies
  • Separate anonymous from authenticated pages

Authorization

  • Use least privilege accounts
  • Consider granularity of access
  • Enforce separation of privileges
  • Use multiple gatekeepers
  • Secure system resources against system identities

Configuration Management

  • Use least privileged service accounts
  • Do not store credentials in clear text
  • Use strong authentication and authorization on administrative interfaces
  • Do not use the Local Security Authority (LSA)
  • Avoid storing sensitive information in the Web space
  • Use only local administration

Sensitive Data

  • Do not store secrets in software
  • Encrypt sensitive data over the network
  • Secure the channel

Session Management

  • Partition site by anonymous, identified, and authenticated users
  • Reduce session timeouts
  • Avoid storing sensitive data in session stores
  • Secure the channel to the session store
  • Authenticate and authorize access to the session store

Cryptography

  • Do not develop and use proprietary algorithms (XOR is not encryption. Use platform-provided cryptography)
  • Use the RNGCryptoServiceProvider method to generate random numbers
  • Avoid key management. Use the Windows Data Protection API (DPAPI) where appropriate
  • Periodically change your keys

Exception Management

  • Use structured exception handling (by using try/catch blocks)
  • Catch and wrap exceptions only if the operation adds value/information
  • Do not reveal sensitive system or application information
  • Do not log private data such as passwords

Auditing and Logging

  • Identify malicious behavior
  • Know your baseline (know what good traffic looks like)
  • Use application instrumentation to expose behavior that can be monitored

As you can see, the security countermeasures can easily be reviewed, updated, and moved forward, because the actionable principles are well organized by the security categories.

There are many ways to use creating categories as a way to get smarter and get better results.

In the future, I’ll walk through how we created an Agile Security approach, using categories.

Meanwhile, check out my post on The More Distinctions You Make, the Smarter You Get to gain some additional insights into how to use empathy and creating categories to dive deeper, learn faster, and get smarter on any topic you want to take on.

Categories: Architecture, Programming

We Help Our Customers Transform

"Innovation—the heart of the knowledge economy—is fundamentally social." -- Malcolm Gladwell

I’m a big believer in having clarity around what you help your customers do.

I was listening to Satya Nadella’s keynote at the Microsoft Worldwide Partner Conference, and I like how he put it so simply, that we help our customers transform.

Here’s what Satya had to say about how we help our customers transform their business:

“These may seem like technical attributes, but they are key to how we drive business success for our customers, business transformation for our customers, because all of what we do, collectively, is centered on this core goal of ours, which is to help our customers transform.

When you think about any customer of ours, they're being transformed through the power of digital technology, and in particular software.

There isn't a company out there that isn't a software company.

And our goal is to help them differentiate using digital technology.

We want to democratize the use of digital technology to drive core differentiation.

It's no longer just about helping them operate their business.

It is about them excelling at their business using software, using digital technology.

It is about our collective ability to drive agility for our customers.

Because if there is one truth that we are all faced with, and our customers are faced with, it's that things are changing rapidly, and they need to be able to adjust to that.

And so everything we do has to support that goal.

How do they move faster, how do they interpret data quicker, how are they taking advantage of that to take intelligent action.

And of course, cost.

But we'll keep coming back to this theme of business transformation throughout this keynote and throughout WPC, because that's where I want us to center in on.

What's the value we are adding to the core of our customer and their ability to compete, their ability to create innovation.

And anchored on that goal is our technical ambition, is our product ambition.”

Transformation is the name of the game.

You Might Also Like

Satya Nadella is All About Customer Focus

SatyaSatya Nadella on a Mobile-First, Cloud-First World

Satya Nadella on Empower Every Person on the Planet

Satya Nadella on Everyone Has To Be a Leader

Satya Nadella on How the Key To Longevity is To Be a Learning Organization

Satya Nadella on Live and Work a Meaningful Life

Sayta Nadelle on The Future of Software

Categories: Architecture, Programming

Satya Nadella on a Mobile-First, Cloud-First World

You hear Mobile-First, Cloud-First all the time.

But do you ever hear it really explained?

I was listening to Satya Nadella’s keynote at the Microsoft Worldwide Partner Conference, and I like how he walked through how he thinks about a Mobile-First, Cloud-First world.

Here’s what Satya had to say:

“There are a couple of attributes.

When we talk about Mobile-First, we are talking about the mobility of the experience.

What do we mean by that?

As we look out, the computing that we are going to interface with, in our lives, at home and at work, is going to be ubiquitous.

We are going to have sensors that recognize us.

We are going to have computers that we are going to wear on us.

We are going to have computers that we touch, computers that we talk to, the computers that we interact with as holograms.

There is going to be computing everywhere.

But what we need across all of this computing, is our experiences, our applications, our data.

And what enables that is in fact the cloud acting as a control plane that allows us to have that capability to move from device to device, on any given day, at any given meeting.

So that core attribute of thinking of mobility, not by being bound to a particular device, but it's about human mobility, is very core to our vision.

Second, when we think about our cloud, we think distributed computing will remain distributed.

In fact, we think of our servers as the edge of our cloud.

And this is important, because there are going to be many legitimate reasons where people will want digital sovereignty, people will want data residency, there is going to be regulation that we can't anticipate today.

And so we have to think about a distributed cloud infrastructure.

We are definitely going to be one of the key hyper-scale providers.

But we are also going to think about how do we get computing infrastructure, the core compute, storage, network, to be distributed throughout the world.

These may seem like technical attributes, but they are key to how we drive business success for our customers, business transformation for our customers, because all of what we do, collectively, is centered on this core goal of ours, which is to help our customers transform.”

That’s a lot of insight, and very well framed for creating our future and empowering the world.

You Might Also Like

Microsoft Explained: Making Sense of the Microsoft Platform Story

Satya Nadella is All About Customer Focus

Satya Nadella on Empower Every Person on the Planet

Satya Nadella on Everyone Has To Be a Leader

Satya Nadella on How the Key To Longevity is To Be a Learning Organization

Satya Nadella on Live and Work a Meaningful Life

Sayta Nadelle on The Future of Software

Categories: Architecture, Programming

Empower Every Person on the Planet to Achieve More

It’s great to get back to the basics, and purpose is always a powerful starting point.

I was listening to Satya Nadella’s keynote at the Microsoft Worldwide Partner Conference, and I like how he walked through the Microsoft mission in a mobile-first, cloud-first world.

Here’s what Satya had to say:

“Our mission:  Empowering every person and every business on the planet to achieve more.

(We find that by going back into our history and re-discovering that core sense of purpose, that soul ... a PC in every home, democratizing client/server computing.)

We move forward to a Mobile-First, Cloud-First world.

We care about empowerment.

There is no other ecosystem that is primarily, and solely, built to help customers achieve greatness.

We are focused on helping our customers achieve greatness through digital technology.

We care about both individuals and organizations.  That intersection of people and organizations is the cornerstone of what we represent as excellence.

We are a global company.  We want to make sure that the power of technology reaches every country, every vertical, every organization, irrespective of size.

There will be many goals.

What remains constant is this sense of purpose, the reason why this ecosystem exists.

This is a mission that we go and exercise in a Mobile-First, Cloud-First world.”

If I think back to why I originally joined Microsoft, it was to empower every person on the planet to achieve more.

And the cloud is one powerful enabler.

You Might Also Like

Satya Nadella is All About Customer Focus

Satya Nadella on Everyone Has To Be a Leader

Satya Nadella on How the Key To Longevity is To Be a Learning Organization

Satya Nadella on Live and Work a Meaningful Life

Sayta Nadelle on The Future of Software

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For July 17th, 2015

Hey, it's HighScalability time:


In case you were wondering, the world is weird. Large Hadron Collider discovers new pentaquark particle.

 

  • 3x: Uber bigger than taxi market; 250x: traffic in HotSchedules' DDoS attack; 92%: Apple’s share of the smartphone profit pie; 7: Airbnb rejections
  • Quotable Quotes:
    • Netflix: A slow or unhealthy server is worse than a down server 
    • @inconshreveable: ngrok production servers, max GC pause: Go1.4 (top) vs Go1.5. Holy 85% reduction! /cc Go team
    • Nic Fleming: The fungal internet exemplifies one of the great lessons of ecology: seemingly separate organisms are often connected, and may depend on each other.
    • @IBMResearch: With 20+ billion transistors on new chip, that's a 50% scaling improvement over today’s tech #ibmresearch #7nm 

  • Apple and Google Race to See Who Can Kill the App First. Honest question, how are people supposed to make money in this new world? Apps are quickly becoming just an identity that ties together 10 or so components that appear integrated as part of the OS, but don't look like your app at all. Reminds me of laminar flow. We are seeing a rebirth of CORBA, COM and OLE 2, this time the container is an app bound by deep linking and some ad hoc ways to push messages around. Show developers the money.

  • The dark side of Google 10x: One former exec told Business Insider that the gospel of 10x, which is promoted by top execs including CEO Larry Page, has two sides. “It’s enormously energizing on one side, but on the other it can be totally paralyzing,”

  • Wait, are we going all RAM or all flash? So confusing. MIT Develops Cheaper Supercomputer Clusters By Nixing Costly RAM In Favor Of Flash: researchers presented evidence at the International Symposium on Computer Architecture that if servers executing a distributed computation go to disk for data even just 5 percent of the time, performance takes a hit to where it's comparable with flash memory anyway. 40 servers with 10 terabytes of RAM wouldn't chew through a 10.5TB computation any better than 20 servers with 20TB of flash memory. What's involved here is moving a little computational power off of the servers and onto the chips that control the flash drives.

  • Is disruption merely a Silicon Valley fantasy? Corporate America Hasn’t Been Disrupted: the advantage enjoyed by incumbents, always substantial, has been growing in recent years...more Americans worked for big companies...Large companies are becoming more dominant in part by buying up their rivals...Consolidation could explain at least part of the rising failure rate among startups...The startup rate has declined in every major industry, every state and nearly every city, and the failure rate’s rise has been nearly as universal. 

  • What's a unikernel and why should you care? Amir Chaudhry reveals all in his Unikernels talk given at PolyConf 15. And here's the supporting blog post. Why are we still applications on top of operating systems? Most applications are single purpose so why all the complexity? Why are we building software for the cloud the same way we build it for desktops? We can do better with Unikerels where every application is a single purpose VM with a single address space.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

This. Just. This.

In response to an honest comment about some of Instagram's rather "ordinary engineering choices", mikeyk (Co-founder @ Instagram) had what I consider the perfect response:  We (at IG) aren't claiming to be doing revolutionary things on infrastructure--but one thing I found super valuable when scaling Instagram in the early days was having access to stories from other companies on how they've scaled. That's the spirit in which I encourage our engineers to blog about our DB scaling, our search infra, etc--I think the more open we are (as a company, but more broadly as an industry) about technical approaches + solutions, the better off we'll be. This could be the anthem for HS and is a key reason our industry continues to get better. And in case you are interested, here are just a few of those stories from Instagram:

On HackerNews

Categories: Architecture

64 Network DO’s and DON’Ts for Game Engines. Part IIIa: Server-Side

This article originally appeared on ITHare.com. It's one article from an excellent series of articles: Part I. Client Side; Part IIa. Protocols and APIs; Part IIb; Protocols and APIs; Part IIIb. Server-Side (deployment, optimizations, and testing); Part IV. Great TCP-vs-UDP Debate; Part V. UDP; Part VI. TCP.

In Part III of the article, we’ll discuss issues specific to server-side, as well as certain DO’s and DON’Ts related to system testing. Due to the size, part III has been split, and in this part IIIa we’ll concentrate on the issues related to Store-Process-and-Forward architecture.

18. DO consider Event-Driven programming model for Server Side too

As discussed above (see item #1 in Part I), the event-driven programming is a must for the client side; in addition, it also comes handy on the server side. Having multi-threaded logic is still a nightmare for the server-side [NoBugs2010], and keeping logic single-threaded simplifies development a lot. Whether to think that multi-threaded game logic is normal, and single-threaded logic is a big improvement, or to think that single-threaded game logic is normal, and multi-threaded logic is a nightmare – is up to you. What is clear is that if you can keep your game logic single-threaded – you’ll be very happy compared to the multi-threaded alternative.

However, unlike the client-side where performance and scalability rarely pose problems, on the server side where you need to serve hundreds of thousands of players, they become really (or, if your project is successful, “really really”) important. I know two ways of handling performance/scalability for games, while keeping logic single-threaded.

18a. Off-loading
Categories: Architecture

A Very Old Version of the PlentyOfFish Architecture that was Just Sold for $575 Million

PlentyOfFish was acquired by the Match Group for $575 million in cash. And it all goes to Markus Frind. Here's the story of the acquisition

Way back in 2009 I wrote architecture article on PlentyOfFish, which I'll reproduce here for historical perspective. The main theme at that time was how Markus was making great fat stacks of cash from adsense by running this huge site all by himself on a Microsoft stack.

We know the adsense goldmine played out long ago. What else has changed? We don't really know. Sometime ago we stopped getting updates on PlentyOfFish architecture changes, so that's all we have.

I doubt much remains the same however. Now 75 people work at PlentyOfFish, there are 90 million registered users, and a whopping 3.6 million active daily users, so something must be happening.

Anyway, here's the old PlentyOfFish Architecture. It still makes for interesting reading. I'm just wondering, when you get done reading, is being sold for $575 Million the ending you would expect?

PlentyOfFish Architecture
Categories: Architecture

Continuous Delivery of Docker Images

Xebia Blog - Mon, 07/13/2015 - 20:05

Our customer wanted to drastically cut down time to market for the new version of their application. Large quarterly releases should be replaced by small changes that can be rolled out to production multiple times a day. Below we will explain how to use Docker and Ansible to support this strategy, or, in our customer’s words, how to ‘develop software at the speed of thought’.

To facilitate development at the speed of thought we needed the following:

  1. A platform to deploy Docker images to
  2. Set up logging, monitoring and alerting
  3. Application versioning
  4. Zero downtime deployment
  5. Security

We’ll discuss each property below.

Platform
Our Docker images run on an Ubuntu host because we needed a supported Linux version that is well known. In our case we install the OS using an image and run all other software in a container. Each Docker container hosts exactly one process so it is easy to see what a container is supposed to do. Examples of containers include:

  • Java VMs to run our Scala services
  • HA Proxy
  • Syslog-ng
  • A utility to rotate log files
  • And even an Oracle database (not on acceptance and production because we expected support issues with that setup, but for development it works fine)

Most of the software running in containers is started with a bash script, but recently we started experimenting with Go so a container may need no more than a single executable.

Logging, monitoring and alerting
To save time we decided to offload the development effort of monitoring and alerting to hosted services where possible. This resulted in contracts with Loggly to store application log files, Librato to collect system metrics and OpsGenie to alert Ops based on rules defined in Loggly. Log files are shipped to Loggly using their Syslog-NG plugin. Our application was already relying on statsd so to avoid having to rewrite code, we decided to create a statsd emulator to push metrics to Librato. This may change in the future if we find the time, but for now it works fine. We’re using the Docker stats API to collect information at the container level.

Application versioning
In the Java world the deliverable would be a jar file published to a repository like Artifactory or Nexus. This is still possibile when working with Docker but it makes more sense to use Docker images as deliverables. The images contain everything needed to run the service, including the jar file. Like jar files, Docker images are published, in this case to the Docker registry. We started with Docker hub online but we wanted faster delivery and more control over who can access the images so we introduced our private Docker registry on premise. This works great and we are pushing around 30 to 50 images a day.
The version tag we use for a container is the date and time it was built. When the build starts we tag the sources in Git with a name based on the date and time, e.g. 20150619_1504. Components that pass their test are assembled in a release based on a text file, a composition, that lists all components that should be part of a release. The composition is tagged with a c_ prefix and a date/time stamp and is deployed to the integration test environment. Then a new test is run to find out whether the assembly still works. If so, the composition is labeled with a new rc tag, rc_20150619_1504 in our example. Releases that pass the integration test are deployed to acceptance and eventually production, but not automatically. We decided to make deployment a management decision, executed by a Jenkins job.
This strategy allows us to recreate a version of the software from source, by checking out tags that make up a release and building again, or from the Docker repository by deploying all versions of components as they are listed in the composition file.
Third-party components are tagged using the version number of the supplier.

Zero downtime deployment
To achieve high availability, we chose Ansible to deploy a Docker container based on the composition file mentioned above. Ansible connects to a host and then uses the Docker command to do the following:

  1. Check if the running container version differs from the one we want to deploy
  2. If the version is different, stop the old container and start the new one
  3. If the version is the same, don’t do anything

This saves a lot of time because Ansible will only change containers that need to be changed and leave all others alone.
Using Ansible we can also implement Zero Downtime Deployment:

  1. First shut down the health container on one node
  2. This causes the load balancer to remove the node from the list of active nodes
  3. Update the first node
  4. Restart the health container
  5. Run the update script in parallel on all other nodes.

Security
The problem with the Docker API is that you are either in or out with no levels in between. This means, e.g. that if you add the Docker socket to a container to look at Docker stats you also allow starting and stopping images. Or if you allow access to the Docker executable you also grant access to configuration information like passwords passed to the container at deployment time. To fix this problem we created a Docker wrapper. This wrapper forbids starting privileged containers and hides some information returned by Docker inspect.
One simple security rule is that software that is not installed or is not running, can’t be exploited. Applied to Docker images this means we removed everything we don’t need and made the image as small as possible. Teams extend the base Linux image only by adding the jar file for their application. Recently we started experimenting with Go to run utilities because a Go executable needs no extra software to run. We’re also testing smaller container images.
Finally, remember not to run as root and carefully consider what file systems to share between container and host.

Summary
In summary we found a way to package software in containers, both standard utilities and Scala components, create a tagged and versioned composition that is tested and moves from one environment to the next as a unit. Using Ansible we orchestrated deployment of new releases while keeping always at least one server running.
In the future we plan to work on reducing image size by stripping the base OS and deploying more utilities as Go containers. We will also continue work on our security wrapper and plan to investigate Consul to replace our home made service registry.

This blog was based on a talk by Armin Čoralić at XebiCon 2015. Watch Armin’s presentation here.

Algolia's Fury Road to a Worldwide API

Guest post by Julien Lemoine, co-founder & CTO of Algolia, a developer friendly search as a service API.

The most frequent questions we answer for developers and devops are about our architecture and how we achieve such high availability. Some of them are very skeptical about high availability with bare metal servers, while others are skeptical about how we distribute data worldwide. However, the question I prefer is “How is it possible for a startup to build an infrastructure like this”. It is true that our current architecture is impressive for a young company:

  • Our high-end dedicated machines are hosted in 13 worldwide regions with 25 data-centers

  • our master-master setup replicates our search engine on at least 3 different machines

  • we process over 6 billion queries per month

  • we receive and handle over 20 billion write operations per month

Just like Rome wasn't build in a day, our infrastructure wasn't as well. This series of posts will explore the 15 instrumental steps we took when building our infrastructure. I will even discuss our outages and bugs in order to you to understand how we used them to improve our architecture.

This first part will focus on the first three first steps we took when building the service while in beta from March 2013 to August 2013.

The Cloud versus Bare metal debate
Categories: Architecture

Talking with Tech Leads

Coding the Architecture - Simon Brown - Mon, 07/13/2015 - 16:45

A printed copy of Talking with Tech Leads, by Patrick Kua (a Tech Lead at Thoughtworks), arrived in the post this morning. We often discuss the technical side of software development and rarely the “softer” side. This book is a collection of short interviews with people new to or in technical leadership roles. They share their stories, tips and experiences of making that leap from developer to tech lead. To clarify, by "tech lead", Patrick is referring to somebody in a technical leadership role who still writes code. This is synonymous with what I call the software architecture role, with an expectation that part of the role includes coding.

I actually recommend this book during my software architecture workshops as there are some fascinating “eureka” moments presented in the book, especially related to leadership and dealing with people. One of the messages you'll see repeated again and again is that, as software developers, nobody really prepares you for how to deal with people when you make the jump into your first tech lead role. Looking back at my own career and of people I worked with at the time, I'd say the same was true. Hindsight is a wonderful thing, and I wish I had a book like this earlier in my career.

Talking with Tech Leads

I'd certainly recommend taking a look if you're interested in the softer side of the technical leadership/software architecture role. The book is available on Leanpub and as a printed book via Amazon.com and Amazon.co.uk.

Disclaimer: I'm interviewed in the book, as is Robert Annett and a bunch of other people you may recognise ... I receive no royalties for recommending it though. :-)

Categories: Architecture

Two conference keynotes in October

Coding the Architecture - Simon Brown - Mon, 07/13/2015 - 16:09

I'm delighted to say that I'll be presenting two conference keynotes during October, both about software architecture and a little bit about microservices.

1. Software Architect 2015 in London, England

The first is titled Modular Monoliths at the Software Architect 2015 conference taking place in London. Sander Hoogendoorn is delivering the other keynote about microservices, and I hope to bring some balance to the discussion by asking this question: "if you can’t build a well-structured monolith, what makes you think microservices is the answer?". I'll also be running a new workshop at the event called Extracting software architecture from code, but more on that later. The Software Architect conference is certainly evolving from year to year, and it's fantastic to see a wide range of topics related to software architecture, design and development.

2. The Architecture Gathering in Munich, Germany

The week after, I'll be presenting another keynote titled Software architecture as code at a new conference in Munich called the The Architecture Gathering. It's a predominantly German language event and the content (from what I understand, and I don't speak German!) looks pretty interesting, as does the list of speakers.

I'm very much looking forward to both, especially as I think we've reached a turning point in our industry where people are starting to think about and appreciate the role that software architecture plays. See you there!

Categories: Architecture

Skilled for Life

A while back, a colleague challenged me to find something simple and sticky for the big idea behind Sources of Insight.  After trying several phrases, here’s the one that stuck:

Skilled for Life

He liked it because it had punch.  It also had a play on words, and you could read it two different ways.

I like it because it captured the big idea behind Sources of Insight.   The whole purpose behind the site is to help as many people improve the quality of their life as possible.

With skill.

I’ve found that skills can make or break somebody’s chance for success.   And, I don’t just mean from a career perspective.   To be effective in all areas of our life, we need skills across several domains:

  • Mind
  • Body
  • Emotions
  • Career
  • Finance
  • Relationships
  • Fun

Skilled for Life is meant to be a very simple phrase, with a very intentional outcome:

Equip you with the skills you need to survive and thrive in today’s world.

It’s all about personal empowerment.

Not everybody gets the right mentors, or the right training, or the right breaks.   So Sources of Insight is designed from the ground up to be your personal success library that helps you make your own breaks, create your opportunities, and own your destiny.

How?

By sharing the world’s best insight and action for work and life.  By providing you with very real skills for mastering emotional intelligence, intellectual horsepower, creative brilliance, interpersonal relationships, career growth, health, and happiness (yeah, happiness is a skill you can learn).  And by providing you with principles, patterns, and practices for a smarter, more creative, and more capable you.

To give you one simple example of how happiness is a skill, let me tell you about the three paths of happiness according to Dr. Martin Seligman:

  1. The Pleasant Life
  2. The Good Life
  3. The Meaningful Life

You can think of them like this:  The Pleasant Life is all about pleasures, here and now.  The Good Life is about spending more time in your values.  The Meaningful Life is about fulfillment by helping the greater good, using your unique skills.   It’s giving our best where we have our best to give, and moving up Maslow’s stack.

When you know the three paths of happiness, you can more effectively build your happiness muscles.  For example, you can Discover Your Values, so that you can spend more time in them, and live life on your terms.

That’s just one example of how you can improve your self-efficacy with skill.

There is a vast success library of everything from inspirational quotes to inspirational heroes, as well as principles, patterns, and practices for skills to pay the bills and lead a better life.  Sources of Insight is a dojo of personal development, and your jump start for realizing your potential.

I invite you to check out the following page on Sources of Insight, where I share what Skilled for Life is all about:

Skilled for Life

Skills empower you.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For July 10th, 2015

Hey, it's HighScalability time:


Spying on an ant holding its favorite microchip. (@SciencePorn)

 

  • 1,425%: malware attack ROI; 33333/sec: BP oil well datapoints generated; 8 million: mumified dogs; 5 billion: Apple map requests per week; 10 billion: parameter neural net in your basement; 1 trillion: records in Yahoo's Sherpa.
  • Quotable Quotes:
    • @warriors: "It's ironic but what the unexpected thing was that everything went exactly as we hoped. That never happens." 
    • @georgeblazer: At scale, architecture dominates material. Alan Kay #gophercon
    • Nassim Nicholas Taleb: How unpredictability rises faster and faster with dimensionality: add one variable to 100 and error doubles.
    • Elon Musk~ one of the biggest challenges in the CRS-7 event is matching events to the exact time. When you are talking a matter of milliseconds it's hard to match the ground track video to the sensors
    • @MeltingIce: NYSE halted, United grounds flights due to computer glitch, WSJ website is down. This is how it starts.
    • The Shut-In Economy: They end up asking each other which apps they work for: Postmates. Seamless. EAT24. GrubHub. Safeway.com
    • @aphyr: At a Devops Chicago meetup I asked how many people had experienced partitions in their datacenters, and over half the room raised hands.
    • @mjpt777: Simplifying code rocks. @toddlmontgomery and I are seeing new throughput highs on Aeron after simplifying and cutting indirection.
    • aphyr: Real software is fuzzier: our processes are usually not realtime, which means the network effectively extends within the node. Garbage collection, in particular, is a notorious cause of “network” partitions, because it delays messages.
    • @fgens: 2 key msgs at #AWSSummit : "Developers, we love you (IaaS is so yesterday)!" and "Go 'server-less' -- deploy code as Lambda microservices"
    • @BenedictEvans: @Jim_Edwards but the boom is caused by having 3bn people online and a generational change in the tech platform from PC to mobile.
    • @Obdurodon: "Our distributed file system doesn't work quite right and performs like crap." "OK, we'll just call it an object store then."
    • @viktorklang: Suspected main causes for slow programs: A) Doing unnecessary work B) Waiting for unnecessary work to be done
    • @jasongorman: There are only so many times one can re-learn the same patterns for remote procedure calls before one gets mightily sick of it
    • @BenedictEvans: Devices in use, end of 2014:  ~1.5bn PCs 7-800m consumer PCs 1.2-1.3bn closed Android 4-500m open Android 650-675m iOS 80m Macs, ~75m Linux.
    • vardump: When performance matters, don't chase pointers (references) and don't ever do random memory accesses if you can avoid it. Ensure variables needed about same time are near each other, so that a single cache line fill will get all. Except when other threads/cores frequently update a variable, try to keep those separately to reduce false sharing.
    • @JZdziarski: Quantum entanglement dictates if one programmer is writing productive good code, another somewhere is inexplicably porting to JavaScript.
    • There are even more quotes for your perusal, please click through to the full article.

  • Love this! jamiesonbecker: AMAZON DEPRECATES EC2 November 3, 2017, SEATTLE At the AWS Re:Invent in Las Vegas today, Amazon Web Services today announced the deprecation of Elastic Compute Cloud as it shifts toward lighter-weight, more horizontally scalable services. Amazon announced that it was giving customers the opportunity to migrate toward what it claims are lower cost "containers" and "Lambda processes".

  • Horace Dediu with an interesting view of Humanism++. This time it's preferring humans over algorithms instead of humans over faith. Curate, don't automate...or sermonize. Also interesting was the discussion on functional vs divisional organizations. Apple is the largest functional org outside of the army. Functional orgs prevent cross divisional fights for resources and power.

  • OK, this is funny: Using locks to solve your concurrency problems.

  • Do we have a precedence for the rise of walled gardens? 1492: The Year the World Began: shift of initiative—the upset in the normal state of the world—started in 1492, when the resources of the Americas began to be accessible to Westerners while remaining beyond the reach of other rival or potentially rival civilizations.

  • Here's how the new StackExchange blog was rebuilt. Pretty radical. They got rid of WordPress for a static blog built on Jekyll. While there was some contention with the move on HackerNewsjonhmchan set the record straight: Performance also wasn't the only plus here. Closing major security holes, making more of our content and technology more open, and moving to a platform that our devs liked working in are just some of the other wins. It's too early to say definitively now, but we think the change is probably a good one.

  • Should Uber be the new McDonalds? While I viscerally agree with the following statement, the idea that every place in the world must have the same laws is equally obviously wrong. Vive la difference! @paulg: Uber is so obviously a good thing that you can measure how corrupt cities are by how hard they try to suppress it.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

RebornDB: the Next Generation Distributed Key-Value Store

There are many key-value stores in the world and they are widely used in many systems. E.g, we can use a Memcached to store a MySQL query result for later same query, use MongoDB to store documents for better searching, etc.

For different scenarios, we should choose different key-value store. There is no silver-bullet key-value store for all solutions. But if you just want a simple key-value store, easy to use, very fast, supporting many powerful data structures, redis may be a good choice for your start.  

Redis is advanced key-value cache and store, under BSD license. It is very fast, has many data types(String, Hash, List, Set, Sorted Set …), uses RDB or AOF persistence and replication to guarantee data security, and supplies many language client libraries.

Most of all, market chooses Redis. There are many companies using Redis and it has proved its worth.

Although redis is great, it still has some disadvantages, and the biggest one is memory limitation.  Redis keeps all data in memory, which limits the whole dataset size and lets us save more data impossibly.

The official redis cluster solves this by splitting data into many redis servers, but it has not been proven in many practical environments yet. At the same time, it need us to change our client libraries to support “MOVED” redirection and other special commands, this is unacceptable in running production too. So redis cluster is not a good solution now.

QDB

We like redis, and want to go beyond its limitation, so we building a service named QDB, which is compatible with redis, saves data in disk to exceed memory limitation and keeps hot data in memory for performance.

Introduction

QDB is a redis like, fast key-value store.It has below good features:

Categories: Architecture

Sponsored Post: VoltDB, Datadog, Tumblr, Power Admin, Learninghouse, MongoDB, Internap, Aerospike, SignalFx, InMemory.Net, Couchbase, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Make Tumblr fast, reliable and available for hundreds of millions of visitors and tens of millions of users.  As a Site Reliability Engineer you are a software developer with a love of highly performant, fault-tolerant, massively distributed systems. Apply here now! 

  • At Scalyr, we're analyzing multi-gigabyte server logs in a fraction of a second. That requires serious innovation in every part of the technology stack, from frontend to backend. Help us push the envelope on low-latency browser applications, high-speed data processing, and reliable distributed systems. Help extract meaningful data from live servers and present it to users in meaningful ways. At Scalyr, you’ll learn new things, and invent a few of your own. Learn more and apply.

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • 90 Days. 1 Bootcamp. A whole new life. Interested in learning how to code? Concordia St. Paul's Coding Bootcamp is an intensive, fast-paced program where you learn to be a software developer. In this full-time, 12-week on-campus course, you will learn either .NET or Java and acquire the skills needed for entry-level developer positions. For more information, read the Guide to Coding Bootcamp or visit bootcamp.csp.edu.

  • The Art of Cyberwar: Security in the Age of Information. Cybercrime is an increasingly serious issue both in the United States and around the world; the estimated annual cost of global cybercrime has reached $100 billion with over 1.5 million victims per day affected by data breaches, DDOS attacks, and more. Learn about the current state of cybercrime and the cybersecurity professionals in charge with combatting it in The Art of Cyberwar: Security in the Age of Information, provided by Russell Sage Online, a division of The Sage Colleges.

  • MongoDB World brings together over 2,000 developers, sysadmins, and DBAs in New York City on June 1-2 to get inspired, share ideas and get the latest insights on using MongoDB. Organizations like Salesforce, Bosch, the Knot, Chico’s, and more are taking advantage of MongoDB for a variety of ground-breaking use cases. Find out more at http://mongodbworld.com/ but hurry! Super Early Bird pricing ends on April 3.
Cool Products and Services
  • VoltDB is a full-featured fast data platform that has all of the data processing capabilities of Apache Storm and Spark Streaming, but adds a tightly coupled, blazing fast ACID-relational database, scalable ingestion with backpressure; all with the flexibility and interactivity of SQL queries. Learn more.

  • In a recent benchmark conducted on Google Compute Engine, Couchbase Server 3.0 outperformed Cassandra by 6x in resource efficiency and price/performance. The benchmark sustained over 1 million writes per second using only one-sixth as many nodes and one-third as many cores as Cassandra, resulting in 83% lower cost than Cassandra. Download Now.

  • Datadog is a monitoring service for scaling cloud infrastructures that bridges together data from servers, databases, apps and other tools. Datadog provides Dev and Ops teams with insights from their cloud environments that keep applications running smoothly. Datadog is available for a 14 day free trial at datadoghq.com.

  • Here's a little quiz for you: What do these companies all have in common? Symantec, RiteAid, CarMax, NASA, Comcast, Chevron, HSBC, Sauder Woodworking, Syracuse University, USDA, and many, many more? Maybe you guessed it? Yep! They are all customers who use and trust our software, PA Server Monitor, as their monitoring solution. Try it out for yourself and see why we’re trusted by so many. Click here for your free, 30-Day instant trial download!

  • Turn chaotic logs and metrics into actionable data. Scalyr replaces all your tools for monitoring and analyzing logs and system metrics. Imagine being able to pinpoint and resolve operations issues without juggling multiple tools and tabs. Get visibility into your production systems: log aggregation, server metrics, monitoring, intelligent alerting, dashboards, and more. Trusted by companies like Codecademy and InsideSales. Learn more and get started with an easy 2-minute setup. Or see how Scalyr is different if you're looking for a Splunk alternative or Loggly alternative.

  • Instructions for implementing Redis functionality in Aerospike. Aerospike Director of Applications Engineering, Peter Milne, discusses how to obtain the semantic equivalent of Redis operations, on simple types, using Aerospike to improve scalability, reliability, and ease of use. Read more.

  • SQL for Big Data: Price-performance Advantages of Bare Metal. When building your big data infrastructure, price-performance is a critical factor to evaluate. Data-intensive workloads with the capacity to rapidly scale to hundreds of servers can escalate costs beyond your expectations. The inevitable growth of the Internet of Things (IoT) and fast big data will only lead to larger datasets, and a high-performance infrastructure and database platform will be essential to extracting business value while keeping costs under control. Read more.

  • SignalFx: just launched an advanced monitoring platform for modern applications that's already processing 10s of billions of data points per day. SignalFx lets you create custom analytics pipelines on metrics data collected from thousands or more sources to create meaningful aggregations--such as percentiles, moving averages and growth rates--within seconds of receiving data. Start a free 30-day trial!

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex goes beyond monitoring and measures the system's work on your MySQL and PostgreSQL servers, providing unparalleled insight and query-level analysis. This unique approach ultimately enables your team to work more effectively, ship more often, and delight more customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

How Do We Explain the Unreasonable Effectiveness of IT?

Joseph Campbell: As Schopenhauer says, when you look back on your life, it looks as though it were a plot, but when you are into it, it’s a mess: just one surprise after another. Then, later, you see it was perfect. So, I have a theory that if you are on your own path things are going to come to you. Since it’s your own path, and no one has ever been on it before, there’s no precedent, so everything that happens is a surprise and is timely.

Why is the IT industry so darn effective? Just think about these amazing advancements. A little over 30 years ago the Apple Mac went on sale. In 2020 Benedict Evans estimates 80% of adults on earth will have a smartphone. And about at that same time applications were typically monoliths that ran on one computer. Now applications can deploy with the push of a button on cloud native architectures that exploit many thousands of CPUs using datacenter scale operating systems. And software used to be this strange specialized niche only nerds cared about or understood. Now software is in everything and is so ubiquitous it’s becoming nearly invisible. The examples could go on and on and on...and on.

These advances have evolved step-by-step over time, so we don’t even realize the full weight of the transformative changes we’ve experienced. What can account for such astonishingly rapid progress?

Stepping stones.

What the heck do stepping stones have to do with anything? Here’s a clue...do you remember the Connections TV Series by the incredible James Burke?

For an explanation we turn to Ken Stanley, Computer scientist, artificial intelligence researcher, Associate Professor at the University of Central Florida, who wrote a new book Why Greatness Cannot Be Planned: The Myth of the Objective, with a fascinatingly counterintuitive premise:

The greatest achievements become less likely when they are made objectives. The best way to achieve greatness, the truest path to “blue sky” discovery or to fulfill boundless ambition, is to have no objective at all. To achieve our highest goals, we must be willing to abandon them. 

The Big Idea
Categories: Architecture

Software Architecture for Developers in Chinese

Coding the Architecture - Simon Brown - Wed, 07/01/2015 - 11:14

Although it's been on sale in China for a few months, my copies of the Chinese translation of my Software Architecture for Developers book have arrived. :-)

Software Architecture for Developers

I can't read it, but seeing my C4 diagrams in Chinese is fun! Stay tuned for more translations.

Categories: Architecture

How to create the smallest possible docker container of any image

Xebia Blog - Tue, 06/30/2015 - 10:46

Once you start to do some serious work with Docker, you soon find that downloading images from the registry is a real bottleneck in starting applications. In this blog post we show you how you can reduce the size of any docker image to just a few percent of the original. So is your image too fat, try stripping your Docker image! The strip-docker-image utility demonstrated in this blog makes your containers faster and safer at the same time!


We are working quite intensively on our High Available Docker Container Platform  using CoreOS and Consul which consists of a number of containers (NGiNX, HAProxy, the Registrator and Consul). These containers run on each of the nodes in our CoreOS cluster and when the cluster boots, more than 600Mb is downloaded by the 3 nodes in the cluster. This is quite time consuming.

cargonauts/consul-http-router      latest              7b9a6e858751        7 days ago          153 MB
cargonauts/progrium-consul         latest              32253bc8752d        7 weeks ago         60.75 MB
progrium/registrator               latest              6084f839101b        4 months ago        13.75 MB

The size of the images is not only detrimental to the boot time of our platform, it also increases the attack surface of the container.  With 153Mb of utilities in the  NGiNX based consul-http-router, there is a lot of stuff in the container that you can use once you get inside. As we were thinking of running this router in a DMZ, we wanted to minimise the amount of tools lying around for a potential hacker.

From our colleague Adriaan de Jonge we already learned how to create the smallest possible Docker container  for a Go program. Could we repeat this by just extracting the NGiNX executable from the official distribution and copying it onto a scratch image?  And it turns out we can!

finding the necessary files

Using the utility dpkg we can list all the files that are installed by NGiNX

docker run nginx dpkg -L nginx
...
/.
/usr
/usr/sbin
/usr/sbin/nginx
/usr/share
/usr/share/doc
/usr/share/doc/nginx
...
/etc/init.d/nginx
locating dependent shared libraries

So we have the list of files in the package, but we do not have the shared libraries that are referenced by the executable. Fortunately, these can be retrieved using the ldd utility.

docker run nginx ldd /usr/sbin/nginx
...
	linux-vdso.so.1 (0x00007fff561d6000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd8f17cf000)
	libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007fd8f1598000)
	libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007fd8f1329000)
	libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007fd8f10c9000)
	libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007fd8f0cce000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007fd8f0ab2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd8f0709000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fd8f19f0000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd8f0505000)
Following and including symbolic links

Now we have the executable and the referenced shared libraries, it turns out that ldd normally names the symbolic link and not the actual file name of the shared library.

docker run nginx ls -l /lib/x86_64-linux-gnu/libcrypt.so.1
...
lrwxrwxrwx 1 root root 16 Apr 15 00:01 /lib/x86_64-linux-gnu/libcrypt.so.1 -> libcrypt-2.19.so

By resolving the symbolic links and including both the link and the file, we are ready to export the bare essentials from the container!

getpwnam does not work

But after copying all essentials files to a scratch image, NGiNX did not start.  It appeared that NGiNX tries to resolve the user id 'nginx' and fails to do so.

docker run -P  --entrypoint /usr/sbin/nginx stripped-nginx  -g "daemon off;"
...
2015/06/29 21:29:08 [emerg] 1#1: getpwnam("nginx") failed (2: No such file or directory) in /etc/nginx/nginx.conf:2
nginx: [emerg] getpwnam("nginx") failed (2: No such file or directory) in /etc/nginx/nginx.conf:2

It turned out that the shared libraries for the name switch service reading /etc/passwd and /etc/group are loaded at runtime and not referenced in the shared libraries. By adding these shared libraries ( (/lib/*/libnss*) to the container, NGiNX worked!

strip-docker-image example

So now, the strip-docker-image utility is here for you to use!

    strip-docker-image  -i image-name
                        -t target-image-name
                        [-p package]
                        [-f file]
                        [-x expose-port]
                        [-v]

The options are explained below:

-i image-name           to strip
-t target-image-name    the image name of the stripped image
-p package              package to include from image, multiple -p allowed.
-f file                 file to include from image, multiple -f allowed.
-x port                 to expose.
-v                      verbose.

The following example creates a new nginx image, named stripped-nginx based on the official Docker image:

strip-docker-image -i nginx -t stripped-nginx  \
                           -x 80 \
                           -p nginx  \
                           -f /etc/passwd \
                           -f /etc/group \
                           -f '/lib/*/libnss*' \
                           -f /bin/ls \
                           -f /bin/cat \
                           -f /bin/sh \
                           -f /bin/mkdir \
                           -f /bin/ps \
                           -f /var/run \
                           -f /var/log/nginx \
                           -f /var/cache/nginx

Aside from the nginx package, we add the files /etc/passwd, /etc/group and /lib/*/libnss* shared libraries. The directories /var/run, /var/log/nginx and /var/cache/nginx are required for NGiNX to operate. In addition, we added /bin/sh and a few handy utilities, just to be able to snoop around a little bit.

The stripped image has now shrunk to an incredible 5.4% of the original 132.8 Mb to just 7.3Mb and is still fully operational!

docker images | grep nginx
...
stripped-nginx                     latest              d61912afaf16        21 seconds ago      7.297 MB
nginx                              1.9.2               319d2015d149        12 days ago         132.8 MB

And it works!

ID=$(docker run -P -d --entrypoint /usr/sbin/nginx stripped-nginx  -g "daemon off;")
docker run --link $ID:stripped cargonauts/toolbox-networking curl -s -D - http://stripped
...
HTTP/1.1 200 OK

For HAProxy, checkout the examples directory.

Conclusion

It is possible to use the official images that are maintained and distributed by Docker and strip them down to their bare essentials, ready for use! It speeds up load times and reduces the attack surface of that specific container.

Checkout the github repository for the script and the manual page.

Please send me your examples of incredibly shrunk Docker images!