Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/categories/7&page=1' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

10 Stack Benchmarking DOs and DON'Ts

An interesting question came up on the mechanical-sympathy list about how to best benchmark a stack of different queue (aeron/argona, jctools, dpdk, pony) and transport (aeron, dpdk, seastar) options.

Who better to answer than Gil Tene, Vice President of Technology and CTO, Co-Founder, of Azul Systems? Here's his usual insightful and helpful response:

If you are looking at the set of "stacks" (all of which are queues/transports), I would strongly encourage you to avoid repeating the mistakes of testing methodologies that focus entirely on max achievable throughput and then report some (usually bogus) latency stats at those max throughout modes.

The tech empower numbers are a classic example of this in play, and while they do provide some basis for comparing a small aspect of behavior (what I call the "how fast can this thing drive off a cliff" comparison, or "peddle to the metal" testing), those results are not very useful for comparing load carrying capacities for anything that actually needs to maintain some form of responsiveness SLA or latency spectrum requirements.

Rules of thumb I'd start with (some simple DOs and DON'Ts):
Categories: Architecture

Sponsored Post: TechSummit, Netflix, Aerospike, TrueSight Pulse, Redis Labs, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics
Fun and Informative Events
  • Discover the secrets of scalability in IT. The cream of the Amsterdam and Berlin tech scene are coming together during TechSummit, hosted by LeaseWeb for a great day of tech talk. Find out how to build systems that will cope with constant change and create agile, successful businesses. Speakers from SoundCloud, Fugue, Google, Docker and other leading tech companies will share tips, techniques and the latest trends in a day of interactive presentations. But hurry. Tickets are limited and going fast! No wonder, since they are only €25 including lunch and beer.

  • In today’s enterprise, new applications are being invented in droves. To cultivate this momentum, organizations must provide a fast, reliable environment that enables scalability, empowers innovation and reduces complexity. In a webinar on April 26 entitled “Contain Yourself: Development Just Got Easier”, veteran analyst Dr. Robin Bloor will discuss using containers for application and services development. He’ll be briefed by Alvin Richards of Aerospike (the flash-optimized, high-performance NoSQL database) who will showcase how Docker can simplify building and deploying multi-node Aerospike applications. Sign up here to reserve your seat!
Cool Products and Services
  • TrueSight Pulse is SaaS IT performance monitoring with one-second resolution, visualization and alerting. Monitor on-prem, cloud, VMs and containers with custom dashboards and alert on any metric. Start your free trial with no code or credit card.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

The Gig Economy Breaks Social Security

With the tax deadline looming in the US and the future of the gig economy as the engine of scaling startup workforces under fire, there's an important point to consider: In the gig economy the entire social contract is kaput. Here's why.

Everyone who works in the US pays into the Social Security system. The whole idea of Social Security is young people pay in and old people take out.

When you are an employee Social Security taxes are taken directly out of your paycheck. You don't even have to think about it.

When you work in the gig economy you get a 1099-MISC at the end of the year. A 1099 reports payments made by the hiring company during the year and it's sent by the hiring company both to the worker and the IRS.

It's up to the worker to identify their income on their tax return as self employment income, which is subject to a Social Security tax of 15.3%. Most gig workers probably won't declare this income because a lot of them don't even know they are supposed to. My wife, Linda Coleman, a respected Enrolled Agent, says from people she has talked to a lot of gig workers haven't even heard self employment tax. And there's only an ever decreasing budget for the IRS to try to enforce all the rules.

And even if a gig worker does know about the tax they might ask themselves why should should I pay 15.3% on my income when I'm making so little money and the company is capturing almost all the benefit?

The problem: if gig workers aren't contributing how is Social Security supposed to work? Gig workers simply won't have Social Security when they retire.

The way Social Security works is all your wages and self employment income are tracked by the Social Security Administration. If you aren't contributing then you aren't earning credits towards your account. And if you aren't earning credits you won't get much in the way of benefits. Social Security works like a big checking account. The amount you can take out is based on how much you put in (or close enough). If you aren't putting any money you can't take it out later. 

The whole big picture is not being communicated well to the public. Who is benefiting? It's not the worker. It's not the government. It's not even the shareholders because no dividends are being paid.

In the gig economy the entire social contract is kaput.

Categories: Architecture

The Ultimate Tester: Value Creation

Xebia Blog - Tue, 04/12/2016 - 10:10
Once upon a time, when project managers still thought that building software was a completely manageable and predictable activity, and testers were put in a seperate team to ensure independence, the result was shitty software and frustrated people. Even though the rise of the agile way of working has improved some aspects of software development,

Stuff The Internet Says On Scalability For April 8th, 2016

Hey, it's HighScalability time:


Time for a little drone envy. Sea Hunter, 132 foot autonomous surface vessel.

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • 12,000: base pairs in the largest biological circuit ever built; 3x: places GitHub data is now stored; 3.5x: Slacks daily user growth this year; 56 million: events/sec processed through BigTable; 100 Billion: requests per day served by Google App Engine

  • Quotable Quotes:
    • Horst724: #PanamaPapers is the biggest secret data leak in history. It involves 2,6 TB of data, a total of 11.5 million documents that have been leaked by an anonymous insider.
    • Amazon cloud has 1 million users and is near $10 billion in annual sales: Today, AWS offers more than 70 services for compute, storage, databases, analytics, mobile, Internet of Things, and enterprise applications. We also offer 33 Availability Zones across 12 geographic regions worldwide, with another five regions and 11 Availability Zones.
    • @CodeWisdom: "Give someone a program, you frustrate them for a day; teach them how to program, you frustrate them for a lifetime." - David Leinweber
    • @peterseibel: OH: it is amazing how many people reach for some complex distributed system when really all they need is a PC with 256 gigs of RAM in it.
    • @dschobel: once you realize that 1TB of ram costs ~$10k it changes your calculus for going distributed. I mean hopefully it does :)
    • @channingwalton: “The grid takes 8 hours so I’ll run it on my dev box, it’ll take 20 mins”, OH’d at a large bank
    • @noahsussman: First attempt at showing that CPU usage statistics of Web servers exhibit a 1/f spectral density. #devops #testing
    • @BenedictEvans: Tech spirals: Open/closed Client/server Search/curation Messaging/apps Document/service Bundle/unbundle Special/general purpose FB/Myspace
    • @Carnage4Life: Insider states Nest falling apart from constant death marches, no new products and missed revenue numbers.
Categories: Architecture

Joining an EC2 Windows instance to an AWS Directory Service domain

Agile Testing - Grig Gheorghiu - Wed, 04/06/2016 - 21:53
I've been struggling with the existing documentation on how to join an EC2 instance running Windows Server 2012 to an AWS Directory Service domain, so I am hastening to jot down some notes on how I got it to work.

1) Create an AWS Directory Service domain

There is good documentation on doing this. I chose the Microsoft Active Directory option.

A few notes on the creation of an AWS Directory Service:

  • I created a new VPC with 2 subnets for the Directory Service usage
  • I made sure each subnet has an Internet gateway associated so that it can be reachable from the outside
During the creation of the Directory Service, you'll be asked to specify an administrator-type user name and password. Make sure you remember what you specified there because you'll need this info in a subsequent step. Also make note of the DNS server IP addresses that were set during the Directory Service creation.
2) Create an IAM role to be associated with the EC2 Windows instance
  • the IAM role needs to be associated with the AmazonEC2RoleforSSM and AmazonSSMFullAccess policies
  • the IAM role also needs to have a trust relationship with ec2.amazonaws.com

3) Launch EC2 Windows instance associated with the AWS Directory Service domain
I chose the Windows_Server-2012-R2_RTM-English-64Bit-Base-2016.03.09 AMI.  In Step 3 of the AWS launch instance wizard ("Configure instance details") I made sure I specified the following:
  • Network: the VPC created in step 1) above
  • Subnet: one of the 2 subnets created in step 1) above
  • Domain join directory: the directory name for the Directory Service created in step 1) above
  • IAM role: the IAM role created in step 2) above
4) Connect to EC2 Windows instance via RDP
First get the administrator password via the AWS console (you'll need to paste the contents of the private key corresponding to the EC2 key you used when launching the Windows instance). Then connect to the Windows instance as the local administrator user.
Verify that you see the fully qualified domain name of your Directory Service (whatever you indicated in step 1) as the domain of the Windows instance (in Server Manager -> Local Server). If you don't, something went wrong with joining the domain during the instance launch. You can see the system log of that instance in the AWS console by selecting the instance, then going to Actions->Instance Settings->Get System Log. For example, in one of my failed attempts to get all of this working I saw errors related to the IAM role I was using, which at the time didn't have the correct SSM policies attached.
If the Windows instance is correctly joined to the domain, you need to install the Active Directory management tools in order to actually manage the AWS Directory Service. Here is a Rackspace article I found with good instructions.

5) Log in to the EC2 Windows instance as the domain admin to manage AD
After the EC2 Windows instance was rebooted, I managed to log in via RDP as my.aws.directory.fqdn\myusername (where both of these values are the ones chosen in Step 1 above) with the password also chosen in Step 1. At this point I was able to use the Active Directory management tools to add new AD users etc.
Here are some other good resources I found:


How to Remove Duplicates in a Large Dataset Reducing Memory Requirements by 99%

This is a guest repost by Suresh Kondamudi from CleverTap.

Dealing with large datasets is often daunting. With limited computing resources, particularly memory, it can be challenging to perform even basic tasks like counting distinct elements, membership check, filtering duplicate elements, finding minimum, maximum, top-n elements, or set operations like union, intersection, similarity and so on

Probabilistic Data Structures to the Rescue

Probabilistic data structures can come in pretty handy in these cases, in that they dramatically reduce memory requirements, while still providing acceptable accuracy. Moreover, you get time efficiencies, as lookups (and adds) rely on multiple independent hash functions, which can be parallelized. We use structures like Bloom filtersMinHashCount-min sketchHyperLogLog extensively to solve a variety of problems. One fairly straightforward example is presented below.

The Problem

We at CleverTap manage mobile push notifications for our customers, and one of the things we need to guard against is sending multiple notifications to the same user for the same campaign. Push notifications are routed to individual devices/users based on push notification tokens generated by the mobile platforms. Because of their size (anywhere from 32b to 4kb), it’s non-performant for us to index push tokens or use them as the primary user key.

On certain mobile platforms, when a user uninstalls and subsequently re-installs the same app, we lose our primary user key and create a new user profile for that device. Typically, in that case, the mobile platform will generate a new push notification token for that user on the reinstall. However, that is not always guaranteed. So, in a small number of cases we can end up with multiple user records in our system having the same push notification token.

As a result, to prevent sending multiple notifications to the same user for the same campaign, we need to filter for a relatively small number of duplicate push tokens from a total dataset that runs from hundreds of millions to billions of records. To give you a sense of proportion, the memory required to filter just 100 Million push tokens is 100M * 256 = 25 GB!

The Solution – Bloom filter
Categories: Architecture

A Language for Architecture

This is an article that I originally wrote for the Architecture Journal to walk through how we created “a language for software architecture.”  Since the article is no longer available, I’m making it available here for old time’s sake.

The goal at the time was to create a simple way to work through solution design challenges and expose some of the key architectural concerns and choices.

The idea was to make it very easy to zoom out to the broader context, and then very quickly zoom into common architecture choices, such as deployment topologies and cross-cutting concerns.

I also wanted to be able to better leverage the existing patterns in the software industry by giving them a backdrop and a canvas so architects could compose them easier and apply them in a more holistic and effective way.

Grady Booch, one of IBM’s distinguished engineers, had this to say about the Architecture Guide where we first created this “language for architecture”:

“Combine these styles and archetypes, and you have an interesting language for describing a large class of applications. While I don’t necessarily agree that these styles and archetypes are orthogonal (nor are the lists complete) for the general domain of software architecture, for Microsoft’s purposes, these styles offer an excellent operating model into which one can apply their patterns and practices.”

While a lot has changed since the original creation of our Architecture Language, a lot of the meta-frame remains the same.  If I were to update the Architecture Language, I would simply walk through the big categories and update them. 

Summary

One of the most important outcomes of the patterns & practices Application Architecture Guide 2.0 project is a language for the space. A language for application architecture. Building software applications involves a lot of important decisions. By organizing these decisions as a language and a set of mental models, we can simplify organizing and sharing information. By mapping out the architecture space, we can organize and share knowledge more effectively. By using this map as a backdrop, we can also overlay principles, patterns, technologies, and key solutions assets in meaningful and relevant ways. Rather than a sea of information, we can quickly browse hot spots for relevant solutions.
Contents

  • Overview
  • A Map of the Terrain
  • Mapping Out the Architecture Space
  • Architecture Frame
  • Application Types
  • Application Feature Frame
  • Architecture Styles
  • Quality Attributes
  • Layered Architecture Reference Example
  • Layers
  • Tiers
  • Conclusion
  • Resources
A Map of the Terrain

One of the most effective ways to deal with information overload is to frame a space. Just like you frame a picture, you can frame a problem to show it a certain way. When I started the patterns & practices Application Architecture Guide 2.0 project, the first thing I wanted to do was to frame out the space. Rather than provide step-by-step architectural guidance, I thought it would be far more valuable to first create a map of what’s important. We could then use this map to prioritize and focus our efforts. We could also use this map as a durable, evolvable backdrop for creating, organizing and sharing our patterns & practices work. This is the main map, the Architecture Frame, we created to help us organize and share principles, patterns, and practices in the application architecture space:

image
Mapping Out the Architecture Space

Creating the map was an iterative and incremental process. The first step was to break up application architecture into meaningful buckets. It first started when I created a project proposal for our management team. As part of the proposal, I created a demo to show how we might chunk up the architecture space in a meaningful way. In the demo, I included a list of key trends, a set of application types, a set of architectural styles, a frame for quality attributes, an application feature frame, a set of example deployment patterns, and a map of patterns & practices solution assets. I used examples where possible simply to illustrate the idea. It was well received and it served as a strawman for the team.

Each week, our core Application Architecture Guide 2.0 project team met with our extended development team, which primarily included patterns & practices development team members. During this time, we worked through a set of application types, created a canonical application, analyzed layers and tiers, evaluated key trends, and created technology matrix trade-off charts. To create and share information rapidly, we created a lot of mind maps and slides. The mind maps worked well. Rather than get lost in documents, we used the mind maps as backdrops for conversation and elaboration.
Key Mapping Exercises

We mapped out several things in parallel:

  • Key trends. Although we didn’t focus on trends in the guide, we first mapped out key trends to help figure out what to pay attention to. We used a mind map and we organized key trends by application, infrastructure, and process. While there weren’t any major surprises, it was a healthy exercise getting everybody on the same page in terms of which trends mattered.
  • Canonical application. This first thing we did was figure out what’s the delta from the original architecture guide. There were a few key changes. For example, we found that today’s applications have a lot more clients and scenarios they serve. They’ve matured and they’ve been extended. We also found today’s applications have a lot more services, both in terms of exposing and in terms of consuming. We also noticed that some of today’s applications are flatter and have less layers. Beyond that, many things such as the types of components and the types of layers were fairly consistent with the original model.
  • Layers and tiers. This was one of the more painful exercises. Early in the project, we met each week with our development team, along with other reviewers. The goal was to map out the common layers, tiers, and components. While there was a lot of consistency with the original application architecture guide, we wanted to reflect any learnings and changes since the original model. Once we had a working map of the layers, tiers, and components, we vetted with multiple customers to sanity check the thinking.
  • Application types. We originally explored organizing applications around business purposes or dominant functionality, customer feedback told us we were better off optimizing around technical types, such as Web application or mobile client. They were easy for customers to identify with. They also made it easy to overlay patterns, technologies, and key patterns & practices solution assets. The technical application types also made it easy to map out relevant technologies.
  • Architectural styles. This is where we had a lot of debate. While we ultimately agreed that it was helpful to have a simple language for abstracting the shapes of applications and the underlying principles from the technology, it was difficult to create a map that everybody was happy with. Things got easier once we changed some of the terminology and we organized the architectural styles by common hot spots. It then became obvious that the architectural styles are simply named sets of principles. We could then have a higher level conversation around whether to go with object-based community or message-based and SOA, for example. It was also easy to describe deployments in terms of 2-tier, 3-tier, and N-tier.
  • Hot spots for architecture. When you build applications, there’s a common set of challenges that show up again. For example, caching, data access, exception management, logging … etc. These are application infrastructure problems or cross-cutting concerns. You usually don’t want to make these decisions ad-hoc on any significant application. Instead, you want to have a set of patterns and guidelines or ideally reusable code that the team can leverage throughout the application. What makes these hot spots is that they are actionable, key engineering decisions. You want to avoid do-overs where you can. Some do-overs are more expensive than others. One of the beauties of the architecture hot spots is that they helped show the backdrop behind Enterprise Library. For example, there’s a data access block, a caching block, a validation block … etc.
  • Hot spots for application types. When you build certain classes of application, there’s recurring hot spots. For example, when you build a rich client, one of the common hot spots to figure out is how to handle occasionally disconnected scenarios. The collection of hot spots for architecture served as a baseline for finding hot spots in the other application types. For example, from the common set of hot spots, we could then figure out which ones are relevant for Web applications, or which additional hot spots would we need to include.
  • Patterns. Mapping out patterns was a lengthy process. Ultimately, we probably ended up with more information in our workspace than made it into the guide. To map out the patterns, we created multiple mind maps of various pattern depots. We summarized patterns so that we could quickly map them from problems to solutions. We then used our architecture hot spots and our hot spots for application types as a filter to find the relevant patterns. We then vetted the patterns with customers to see if the mapping was useful. We cut any patterns that didn’t seem high enough priority. We also cut many of our pattern descriptions when they started to weight the guide down. We figured we had plenty of material and insight to carve out future pattern guides and we didn’t want to overshadow the value of the main chapters in the guide. We decided the best move for now was to provide a Pattern Map at the end of each application chapter to show which patterns are relevant for key hot spots. Customers seemed to like this approach and it kept things lightweight.
  • patterns & practices solution assets. This was the ultimate exercise in organizing our catalog. We actually have a large body of documented patterns. We also have several application blocks and factories, as well as guides. By using our architecture frame, it was easier to organize the catalog. For example, the factories and reference implementations mapped to the application types. The Enterprise Library blocks mapped to the architecture hot spots. Several of the guides mapped to the quality attributes frame.
  • Microsoft platform. This was a challenge. It meant slicing and dicing the platform stack in a meaningful way as well as finding the right product team contacts. Once we had our application types in place, it got a lot easier. For example, depending on which type of application you were building (RIA, Web, mobile … etc.), this quickly narrowed down relevant technology options. We created technology matrixes for presentation technologies, integration technologies, workflow technologies, and data access technologies. Since the bulk of the guide is principle and pattern based, we kept these matrixes in the appendix for fast lookups.
Key Components of the Application Architecture Map

Over the weeks and months of the project, a very definite map of the landscape emerged. We found ourselves consistently looking for the same frames to organize information. While we tuned and pruned specific hot spots in areas, the overall model of common frames was helping us move through the space quickly.

  • Architecture frame. The architecture frame was the main organizing map. It brought together the context (scenarios, quality attributes, requirements/constraints), application types, architectural styles, and the application hot spots.
  • Application types. For application types, we optimized around a simple, technical set that resonated with customers. For example, Web application, RIA, mobile … etc.
  • Quality attributes. We organized quality attributes by key hot spots: system, runtime, design-time, and user qualities.
  • Architectural styles. We organized architectural styles by key hot spots: communication, deployment, domain, interaction, and structure.
  • Requirements and constraints. We organized requirements by key types: functional, non-functional, technological. We thought of constraints in terms of industry and organizational constraints, as well as by which concern (for example, constraints for security or privacy).
  • Application feature frame. The application feature frame became a solid backdrop for organizing many guidelines through the guide. The hot spots resonated: caching, communication, concurrency and transactions, configuration management, coupling and cohesion, data access, exception management, layering, logging and instrumentation, state management, structure, validation and workflow.
  • Application type frames. The application type frames are simply hot spots for key application types. We created frames for: Web applications, rich internet applications (RIA), mobile applications, rich client applications and services.
  • Layered architecture reference model. (Canonical application) The canonical application is actually a layered architecture reference model. It helps show the layers and components in context.
  • Layers and tiers. We used layers to represent logical partitions and tiers for physical partitions (this precedent was set in the original guide.) We identified key components within the key layers: presentation layer, business layer, data layer, and service layer.
  • Pattern Maps. Pattern maps are simply overlays of key patterns on top of relevant hot spots. We created pattern maps for the application types.
  • Product and technology maps. We created technology matrixes for relevant products and technologies. To put the technologies in context, we used application types where relevant. We also used scenarios. To help make trade-off decisions, we included benefits and considerations for each technology.
User, Business, and System Perspective

One thing that helped early on was creating a Venn diagram of the three perspectives, user, business, and system:

image

In application architecture, it’s easy to lose perspective. It helps to keep three perspectives in mind. By having a quick visual of the three perspectives, it was easy to reminder ourselves that architecture is always a trade-off among these perspectives. It also helped remind us to be clear which perspective we’re talking about at any point in time. This also helped resolve many debates. The problem in architecture debates is that everybody is usually right, but only from their perspective. Once we showed people where their perspective fit in the bigger picture, debates quickly turned from conflict to collaboration. It was easy to move through user goals, business goals, and system goals once people knew the map.
Architecture Frame

The Architecture Frame is a simple way to organize the space. It’s a durable, evolvable backdrop. You can extend it to suit your needs. The strength of the frame is that it combines multiple lenses:

image

Here are the key lenses:

  • Scenarios. This sets the context. You can’t evaluate architecture in a vacuum. You need a backdrop. Scenarios provide the backdrop for evaluation and relevancy.
  • Quality Attributes. This includes your system qualities, your runtime qualities, your design-time qualities and user qualities.
  • Requirements / Constraints. Requirements and constraints includes functional requirements, non-functional requirements, technological requirements, industry constraints and organizational constraints.
  • Application Types. This is an extensible set of common types of applications or clients. You can imagine extending for business types. You can imagine including just the types of applications your organization builds. Think of it as product-line engineering. When you know the types of applications you build, you can optimize it.
  • Architectural Styles. This is a flat list of common architectural styles. The list of architectural styles is flexible and most applications are a mash up of various styles. Architectural styles become more useful when they are organized by key decisions or concerns.
  • Application Feature Frame. The application feature frame is a concise set of hot spots that show up time and again across applications. They reflect cross-cutting concerns and common application infrastructure challenges.
Application Types

This is the simple set of technical application types we defined:

Application Type

Description

Web applications

Applications of this type typically support connected scenarios and can support different browsers running on a range of operating systems and platforms.

Rich Internet applications (RIA)

Applications of this type can be developed to support multiple platforms and multiple browsers, displaying rich media or graphical content. Rich Internet applications run in a browser sandbox that restricts access to some devices on the client.

Mobile Applications

Applications of this type can be developed as thin client or rich client applications. Rich client mobile applications can support disconnected or occasionally connected scenarios. Web or thin client applications support connected scenarios only. The device resources may prove to be a constraint when designing mobile applications.

Rich client applications

Applications of this type are usually developed as stand-alone applications with a graphical user interface that displays data using a range of controls. Rich client applications can be designed for disconnected and occasionally connected scenarios because the applications run on the client machine.

Services

Services expose complex functionality and allow clients to access them from local or remote machine. Service operations are called using messages, based on XML schemas, passed over a transport channel. The goal in this type of application is to achieve loose coupling between the client and the server.

Application Feature Frame

This is the set of hot spots for applications we defined:

Category

Description

Authentication and Authorization

Authentication and authorization allow you to identify the users of your application with confidence, and to determine the resources and operations to which they should have access.

Caching and State

Caching improves performance, reduces server round trips, and can be used to maintain the state of your application.

Communication

Communication strategies determine how you will communicate between layers and tiers, including protocol, security, and communication-style decisions.

Composition

Composition strategies determine how you manage component dependencies and the interactions between components.

Concurrency and Transactions

Concurrency is concerned with the way that your application handles conflicts caused by multiple users creating, reading, updating, and deleting data at the same time. Transactions are used for important multi-step operations in order to treat them as though they were atomic, and to recover in the case of a failure or error.

Configuration Management

Configuration management defines how you configure your application after deployment, where you store configuration data, and how you protect the configuration data.

Coupling and Cohesion

Coupling and cohesion are strategies concerned with layering, separating application components and layers, and organizing your application trust and functionality boundaries.

Data Access

Data access strategies describe techniques for abstracting and accessing data in your data store. This includes data entity design, error management, and managing database connections.

Exception Management

Exception-management strategies describe techniques for handling errors, logging errors for auditing purposes, and notifying users of error conditions.

Logging and Instrumentation

Logging and instrumentation represents the strategies for logging key business events, security actions, and provision of an audit trail in the case of an attack or failure.

User Experience

User experience is the interaction between your users and your application. A good user experience can improve the efficiency and effectiveness of the application, while a poor user experience may deter users from using an otherwise well-designed application.

Validation

Validation is the means by which your application checks and verifies input from all sources before trusting and processing it. A good input and data-validation strategy takes into account not only the source of the data, but also how the data will be used, when determining how to validate it.

Workflow

Workflow is a system-assisted process that is divided into a series of execution steps, events, and conditions. The workflow may be an orchestration between a set of components and systems, or it may include human collaboration.

Architectural Styles

For architectural styles, we first framed the key concerns to organize the architectural styles, and then we defined some common architectural styles.
Organizing Architectural Styles

These are the hot spots we used to organize architectural styles:

Hot Spots

Architectural Styles

Communication

Service-Oriented Architecture(SOA) and/or Message Bus and/or Pipes and Filters.

Deployment

Client/server or 3-Tier or N-Tier.

Domain

Domain Model or Gateway.

Interaction

Separated Presentation.

Structure

Component-Based and/or Object-Oriented and/or Layered Architecture.

Architectural Style Frame

These are some commonly recognized architectural styles:

Architectural Style

Description

Client-server

Segregates the system into two applications, where the client makes a service request to the server.

Component-Based Architecture

Decomposes application design into reusable functional or logical components that are location-transparent and expose well-defined communication interfaces.

Layered Architecture

Partitions the concerns of the application into stacked groups (layers) such as presentation layer, business layer, data layer, and services layer.

Message-Bus

A software system that can receive and send messages that are based on a set of known formats, so that systems can communicate with each other without needing to know the actual recipient.

N-Tier/3-Tier

Segregates functionality into separate segments in much the same way as the layered style, but with each segment being a tier located on a physically separate computer.

Object-Oriented

An architectural style based on division of tasks for an application or system into individual reusable and self-sufficient objects, each containing the data and the behavior relevant to the object.

Separated Presentation

Separates the logic for managing user interaction from the user interface (UI) view and from the data with which the user works.

Service-Oriented Architecture

Refers to Applications that expose and consume functionality as a service using contracts and messages.

Quality Attributes

For quality attributes, we first framed the key categories to organize the quality attributes, and then we defined some common quality attributes.
Organizing Quality Attributes

This is a simple way to organize and group quality attributes:

Type

Quality attributes

System Qualities

· Supportability

· Testability

Run-time Qualities

· Availability

· Interoperability

· Manageability

· Performance

· Reliability

· Scalability

· Security

Design Qualities

· Conceptual Integrity

· Flexibility

· Maintainability

· Reusability

User Qualities

· User Experience / Usability

Quality Attribute Frame

These are some common quality attributes:

Quality attribute

Description

Availability

Availability is the proportion of time that the system is functional and working. It can be measured as a percentage of the total system downtime over a predefined period. Availability will be affected by system errors, infrastructure problems, malicious attacks, and system load.

Conceptual Integrity

Conceptual integrity is the consistency and coherence of the overall design. This includes the way that components or modules are designed, as well as factors such as coding style and variable naming.

Flexibility

The ability of a system to adapt to varying environments and situations, and to cope with changes in business policies and rules. A flexible system is one that is easy to reconfigure or adapt in response to different user and system requirements.

Interoperability

Interoperability is the ability of diverse components of a system or different systems to operate successfully by exchanging information, often by using services. An interoperable system makes it easier to exchange and reuse information internally as well as externally.

Maintainability

Maintainability is the ability of a system to undergo changes to its components, services, features, and interfaces as may be required when adding or changing the functionality, fixing errors, and meeting new business requirements.

Manageability

Manageability is how easy it is to manage the application, usually through sufficient and useful instrumentation exposed for use in monitoring systems and for debugging and performance tuning.

Performance

Performance is an indication of the responsiveness of a system to execute any action within a given time interval. It can be measured in terms of latency or throughput. Latency is the time taken to respond to any event. Throughput is the number of events that take place within a given amount of time.

Reliability

Reliability is the ability of a system to remain operational over time. Reliability is measured as the probability that a system will not fail to perform its intended functions over a specified time interval.

Reusability

Reusability is the capability for components and subsystems to be suitable for use in other applications and in other scenarios. Reusability minimizes the duplication of components and also the implementation time.

Scalability

Scalability is the ability of a system to function well when there are changes to the load or demand. Typically, the system will be able to be extended over more powerful or more numerous servers as demand and load increase.

Security

Security is the ways that a system is protected from disclosure or loss of information, and the possibility of a successful malicious attack. A secure system aims to protect assets and prevent unauthorized modification of information.

Supportability

Supportability is how easy it is for operators, developers, and users to understand and use the application, and how easy it is to resolve errors when the system fails to work correctly.

Testability

Testability is a measure of how easy it is to create test criteria for the system and its components, and to execute these tests in order to determine if the criteria are met. Good testability makes it more likely that faults in a system can be isolated in a timely and effective manner.

Usability

Usability defines how well the application meets the requirements of the user and consumer by being intuitive, easy to localize and globalize, and able to provide good access for disabled users and a good overall user experience.

Layered Architecture Reference Model

This is our canonical application example. It’s a layered architecture showing the common components within each layer:

image

The canonical application model helped us show how the various layers and components work together. It was an easy diagram to pull up and talk through when we were discussing various design trade-offs at the different layers.
Layers

We identified the following layers:

  • Presentation layer
  • Business layer
  • Data layer
  • Service layer

They are logical layers. The important thing about layers is that they help factor and group your logic. They are also fractal. For example, a service can have multiple types of layers within it. The following is a quick explanation of the key components within each layer.
Presentation Layer Components

  • User interface (UI) components. UI components provide a way for users to interact with the application. They render and format data for users and acquire and validate data input by the user.
  • User process components. To help synchronize and orchestrate these user interactions, it can be useful to drive the process by using separate user process components. This means that the process-flow and state-management logic is not hard-coded in the UI elements themselves, and the same basic user interaction patterns can be reused by multiple UIs.
Business Layer Components
  • Application facade (optional). Use a façade to combine multiple business operations into a single message-based operation. You might access the application façade from the presentation layer by using different communication technologies.
  • Business components. Business components implement the business logic of the application. Regardless of whether a business process consists of a single step or an orchestrated workflow, your application will probably require components that implement business rules and perform business tasks.
  • Business entity components. Business entities are used to pass data between components. The data represents real-world business entities, such as products and orders. The business entities used internally in the application are usually data structures, such as DataSets, DataReaders, or Extensible Markup Language (XML) streams, but they can also be implemented by using custom object-oriented classes that represent the real-world entities your application has to work with, such as a product or an order.
  • Business workflows. Many business processes involve multiple steps that must be performed in the correct order and orchestrated. Business workflows define and coordinate long-running, multi-step business processes, and can be implemented using business process management tools.
Data Layer Components
  • Data access logic components. Data access components abstract the logic necessary to access your underlying data stores. Doing so centralizes data access functionality, and makes the process easier to configure and maintain.
  • Data helpers / utility components. Helper functions and utilities assist in data manipulation, data transformation, and data access within the layer. They consist of specialized libraries and/or custom routines especially designed to maximize data access performance and reduce the development requirements of the logic components and the service agent parts of the layer.
  • Service agents. Service agents isolate your application from the idiosyncrasies of calling diverse services from your application, and can provide additional services such as basic mapping between the format of the data exposed by the service and the format your application requires.
Service Layer Components
  • Service interfaces. Services expose a service interface to which all inbound messages are sent. The definition of the set of messages that must be exchanged with a service, in order for the service to perform a specific business task, constitutes a contract. You can think of a service interface as a façade that exposes the business logic implemented in the service to potential consumers.
  • Message types. When exchanging data across the service layer, data structures are wrapped by message structures that support different types of operations. For example, you might have a Command message, a Document message, or another type of message. These message types are the “message contracts” for communication between service consumers and providers.
Tiers

Tiers represent the physical separation of the presentation, business, services, and data functionality of your design across separate computers and systems. Some common tiered design patterns include two-tier, three-tier, and n-tier.
Two-Tier

The two-tier pattern represents a basic structure with two main components, a client and a server.

image
Three-Tier

In a three-tier design, the client interacts with application software deployed on a separate server, and the application server interacts with a database that is also located on a separate server. This is a very common pattern for most Web applications and Web services.

image
N-Tier

In this scenario, the Web server (which contains the presentation layer logic) is physically separated from the application server that implements the business logic.

image
Conclusion

It’s easier to find your way around when you have a map. By having a map, you know where the key hot spots are. The map helps you organize and share relevant information more effectively. More importantly, the map helps bring together archetypes, arch styles, and hot spots in a meaningful way. When you put it all together, you have a simple language for describing large classes of applications, as well as a common language for application architecture.

Categories: Architecture, Programming

Five leaderships lessons from the Samurai for Product Managers

Xebia Blog - Sat, 04/02/2016 - 11:54
We have covered several topics in the Product Samurai series that should make you a better product manager. But what if you are leading product management or run innovation within your enterprise? Here are five leadership lessons that make your team better. “New eras don't come about because of swords, they're created by the people

Big data in the cloud – welcome to cost oriented design

A couple of weeks ago I presented @ BDX2016

The slides are available on Slideshare

Big data in the cloud – welcome to cost oriented design from Arnon Rotem-Gal-Oz

The video is now on YouTube

Note that the first minute or so is missing – It misses the intro and  it starts toward the end of the second slide where I end my disclaimer that cost oriented design != cheap (only hopefully cheaper) costs.

Categories: Architecture

Big data in the cloud – welcome to cost oriented design

A couple of weeks ago I presented @ BDX2016

The slides are available on Slideshare

Big data in the cloud – welcome to cost oriented design from Arnon Rotem-Gal-Oz

The video is now on YouTube

Note that the first minute or so is missing – It misses the intro and  it starts toward the end of the second slide where I end my disclaimer that cost oriented design != cheap (only hopefully cheaper) costs.

Categories: Architecture

Stuff The Internet Says On Scalability For April 1st, 2016

Hey, this is no joke, it's HighScalability time:


A glorious battle in EVE. Tens of thousands of pilots fighting tens of thousands of pilots in a real time all on a single shard.

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • $9.3B: punishment for Google's temerity of using Java; 200: computer scientists and neuroscientists at Google’s DeepMind; 22: cores in Intel's new Xeon E5-2600 V4 CPU; 12: fold boost in spectrum efficiency over current 4G cellular technology using a massive antenna system; 

  • Quotable Quotes:
    • Linus Torvalds:  I’m not a big visionary. I’m a very plodding pedestrian engineer, and I try to keep my eyes firmly on the ground. I’ll let others make the big predictions about where we’ll be in 5, 10 or 25 years
    • theymos: "Core" doesn't think anything because it's not any sort of unified organization.
    • whalesalad: We are running Kubernetes in production at FarmLogs and LOVE it.
    • @StackPointCloud: The operational complexity associated with monitoring containers is multiplied given the 1:N relationship of host:containers. #NYCK8s
    • hu6Bi5To: AWS is significantly more expensive like-for-like, but it's worth remembering that you wouldn't architect your whole system that way if you were targeting AWS.
    • Demis Hassabis [DeepMind]: We don't think just observing is enough for intelligence, you also have to act. Ultimately that’s the only way you can really understand the world.
    • @inottawa: @TeslaMotors can't login to mytesla. Any chance in scaling up those servers?
    • Adrian Colyer: Cliffhanger can achieve the same hit rate with 45% less memory capacity. When memory is one of the most expensive resources in the datacenter, that’s definitely significant!
    • Google: We showed how Cloud Dataflow users no longer have to worry about specifying the number of workers or partitions, and how Cloud Dataflow dynamically adjusts the number of workers over time.
    • @PandoDaily: The switch to subscription has meant huge growth for Adobe
    • spriggan3: You're not hip enough anymore, the new good practice in the valley is femtoservices. Each statement running on its own server.
    • @adrianco: If you are confused about the Tesla Model 3 "launch" think of it as a huge $1000 Kickstarter project
    • Baidu: Our algorithm is able to use crowd data from Baidu maps to predict how many people will be [at a certain location] in the next two hours
    • @robertoglezcano: By 2020, 80% of people around the world (6 billion) will own a smartphone
    • @adrianco: Let me know when you run a 1000 node Cassandra cluster on Kubernetes :-) 
    • Seph Skerritt: The algorithm doesn’t care what you really are. It matters what you choose, and what you think you are.
    • @JimPethokoukis: "Last year, YouTube and sites like it generated $385 million in royalties ... vinyl records brought in $416 million"
    • @gigastacey: "Customers press a Dash button once every minute of the day." 
    • Julian Baggini: One of the paradoxes of creativity is that originality tends towards sameness and similarity. What makes a Wagner opera stand out from others is also what makes it unmistakably Wagnerian.
    • Grant Jensen: In this study, we revealed the beautiful complexity of this machine, [which] be the strongest motor known in nature. The machine lets M. xanthus, a predatory bacterium, move across a field to form a ‘wolf pack’ with other M. xanthus cells, and hunt together for other bacteria on which to prey

  • Chamath Palihapitiya: AWS is a tax on the compute economy.  so whether you care about mobile apps, consumer apps, IoT, SaaS etc etc, more companies than not will be using AWS vs building their own infrastructure.  ecommerce was AMZN's way to dogfood aws, and continue to do so so that it was mission grade.  if you believe that over time the software industry is a multi, deca trillion industry, then ask yourself how valuable a company would be who taxes the majority of that industry. 

  • This is spooky. Google does know everything but it's AI that makes that knowledge manifest in the world. Google shocked this man by offering sympathy on the death of his father:  Google Now was offering him condolences on the death of his dad before showing him what could be emotionally charged photos. "Mind. Blown. I'm sad, I'm amazed, I'm taken back. What a lovely moment for some automated robot voice to express it's sympathy to me," he said.

  • Stack Overflow still does the mostest with the leastest. Nick Craver with a great post on Stack Overflow: The Hardware - 2016 Edition. Sure, there's a lot of hardware porn (with pics), but Nick's mental checklist of the process he goes through to help determine what to order is really insightful. It's too big to include here, but some highlights: Is this a scale up or scale out problem? (Are we buying one bigger machine, or a few smaller ones?); How much redundancy do we need/want? (How much headroom and failover capability?); Will this server/application touch disk? (Do we need anything besides the spinny OS drives?). Also, an interesting analysis by hu6Bi5To of what Stack Overflow might look like on AWS. Less hardware redundancy, less always on capacity, more geographical redundancy. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Should Apple Build their Own Cloud?

This is one of the most interesting build or buy questions of all time: should Apple build their own cloud? Or should Apple concentrate on what they do best and buy cloud services from the likes of Amazon, Microsoft, and Google?

It’s a decision a lot of companies have to make, just a lot bigger, and because it’s Apple, more fraught with an underlying need to make a big deal out of it.

This build or buy question was raised and thoroughly discussed across two episodes of the Exponent podcast, Low Hanging Fruit and Pickaxe Retailers, with hosts Ben Thompson and James Allworth, who regularly talk about business strategy with an emphasis on tech. A great podcast, highly recommended. There’s occasional wit and much wisdom.

Dark Clouds Over Apple’s Infrastructure Efforts
Categories: Architecture

Sponsored Post: TechSummit, zanox Group, Varnish, LaunchDarkly, Swrve, Netflix, Aerospike, TrueSight Pulse, Redis Labs, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • The zanox Group are looking for a Senior Architect. We're looking for someone smart and pragmatic to help our engineering teams build fast, scalable and reliable solutions for our industry leading affiliate marketing platform. The role will involve a healthy mixture of strategic thinking and hands-on work - there are no ivory towers here! Our stack is diverse and interesting. You can apply for the role in either London or Berlin.

  • Swrve -- In November we closed a $30m funding round, and we’re now expanding our engineering team based in Dublin (Ireland). Our mobile marketing platform is powered by 8bn+ events a day, processed in real time. We’re hiring intermediate and senior backend software developers to join the existing team of thirty engineers. Sound like fun? Come join us.

  • Senior Service Reliability Engineer (SRE): Drive improvements to help reduce both time-to-detect and time-to-resolve while concurrently improving availability through service team engagement.  Ability to analyze and triage production issues on a web-scale system a plus. Find details on the position here: https://jobs.netflix.com/jobs/434

  • Manager - Performance Engineering: Lead the world-class performance team in charge of both optimizing the Netflix cloud stack and developing the performance observability capabilities which 3rd party vendors fail to provide.  Expert on both systems and web-scale application stack performance optimization. Find details on the position here https://jobs.netflix.com/jobs/860482

  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics
Fun and Informative Events
  • Discover the secrets of scalability in IT. The cream of the Amsterdam and Berlin tech scene are coming together during TechSummit, hosted by LeaseWeb for a great day of tech talk. Find out how to build systems that will cope with constant change and create agile, successful businesses. Speakers from SoundCloud, Fugue, Google, Docker and other leading tech companies will share tips, techniques and the latest trends in a day of interactive presentations. But hurry. Tickets are limited and going fast! No wonder, since they are only €25 including lunch and beer.

  • Varnish Summits are a worldwide event series where Varnish customers, partners, open source users and other enthusiasts come together to network and learn.  At the summits Varnish Software's experts and core developers do a deep dive into technical best practices and offer workshops for both new and advanced Varnish users.

  • Are you developing - or thinking about creating - UDFs to use with Aerospike? Do you want to get the most out of using UDFs within Aerospike? If so, register for our webinar on April 13th at 11am PT / 2pm ET to hear Sergey Zhemzhitsky, CTO of CleverDATA (a division of LANIT, a leading system integrator in Russia) walk through real-life use cases pertaining to UDFs – namely, how his team implemented Aerospike’s UDFs at CleverDATA. Sign up here to reserve your seat!
Cool Products and Services
  • Dev teams are using LaunchDarkly’s Feature Flags as a Service to get unprecedented control over feature launches. LaunchDarkly allows you to cleanly separate code deployment from rollout. We make it super easy to enable functionality for whoever you want, whenever you want. See how it works.

  • TrueSight Pulse is SaaS IT performance monitoring with one-second resolution, visualization and alerting. Monitor on-prem, cloud, VMs and containers with custom dashboards and alert on any metric. Start your free trial with no code or credit card.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

The Ultimate Tester

Xebia Blog - Tue, 03/29/2016 - 16:14
   

Codifying the rules used to organise your code

Coding the Architecture - Simon Brown - Tue, 03/29/2016 - 09:23

Regular readers will already know about Structurizr - a set of open source libraries to create a software architecture model as code, plus a SaaS product to visualise those models. Having created and helped create a number of models with Structurizr now, I've noticed an interesting side-effect. In the absence of architectural information being present in the code, the power of using something like Structurizr to define a software architecture model using code is in extracting information algorithmically, by codifying the rules that you've ultimately used to structure your codebase.

Let me give you an example. Imagine you're building a web-MVC web application in Java, C#, etc and you have a tens or hundreds of controller classes, each of which uses a number of other components to implement some functionality. Drawing a single diagram to visualise the static structure of the entire web application is a bad idea because it shows too much information. A better approach is to create one view per vertical slice, where there could be one vertical slice per web controller. This results in smaller, simpler diagrams like this.

A component diagram based upon a web controller

So far so good, and this is relatively easy to do using static analysis techniques. But you'll notice this diagram includes an "Authenticated User", which isn't part of the code itself. This raises the question of how the user ends up getting included on the diagram. There are a number of options:

  • Manually add the correct type of user on a case by case basis.
  • Match the controller's URL routing to the permitted user role (this is likely specified in configuration somewhere) and use this information to choose the appropriate type of user for each controller.
  • Add machine-readable metadata (e.g. Java Annotations, C# Attributes) to specify the user type (there are some Structurizr annotations I've created to help with this).
  • Codify the rules you've used to organise the controllers in your codebase.

The ability to codify the rules you've used to organise the controllers in your codebase obviously depends on how much thought you've put into doing this. For example, did you dump all of these controller classes into a single package or namespace without giving it much thought at all? Or perhaps you took Martin Fowler's advice and modularised further, creating one package/namespace per functional area or aggregate root, for example. Another possibility is that you grouped controllers together based upon whether unauthenticated users, authenticated users or other software systems are using them. Organising your code well provides you with another angle to extract architectural information, because you can codify rules such as, "the Anonymous User uses all controllers in the com.mycompany.mywebapp.unsecured package/namespace".

With hindsight this is fairly obvious, but we often don't put enough thought into how we organise our code, possibly because we perceive that it doesn't actually matter that much and modern IDEs provide powerful ways to navigate large and/or complex codebases. Trying to codify the rules used to organise a codebase certainly gets you thinking, and often refactoring too.

Categories: Architecture

How we implemented the video player in Mail.Ru Cloud

We’ve recently added video streaming service to Mail.Ru Cloud. Development started with contemplating the new feature as an all-purpose “Swiss Army knife” that would both play files of any format and work on any device with the Cloud available. Video content uploaded to the Cloud mostly falls into one of the two categories: “movies/series” and “users’ videos”. The latter are the videos that users shoot with their phones and cameras, and these videos are most versatile in terms of formats and codecs. For many reasons, it is often a problem to watch these videos on other end-user devices without prior normalization: a required codec is missing, or the file size is too big to download, or whatever.

In this article, I’ll go into detail to explain how video playback works in Mail.Ru Cloud, and how we made the Cloud player “omnivorous” and ensured support on a maximum number of end-user devices.

Storing and Caching: two approaches
Categories: Architecture

Personal Empowerment All-Stars: Jack Canfield, Ken Blanchard, and Stephen Covey at Microsoft

“You can, you should, and if you’re brave enough to start, you will.”  — Stephen King

One of the best things at Microsoft is the chance to meet extraordinary people.

Jack Canfield, Ken Blanchard, and Stephen Covey are a few that top my list.

They are personal empowerment all-stars.

As I was re-writing my posts on lessons learned from Jack Canfield, Ken Blanchard, and Stephen Covey, I noticed what they share in common.

What do Jack Canfield, Ken Blanchard and Stephen Covey have in common?

Their work has a heavy emphasis on personal-empowerment, positivity, and people.

I thought it would be interesting to write a narrative about lessons learned from each, to supplement my bullet point write ups.

Here we go …

Jack Canfield at Microsoft

Jack Canfield is all about taking full responsibility for everything that happens in your life.  And he starts with self-talk.  He says it’s not what people say or do, it’s what you say to yourself.  For example, it’s not what Jack says to Laura, it’s what Laura says to Laura.

From a personal empowerment standpoint, Jack reminds us that we have control over three responses: 1) what we say or do, 2) our thoughts, 3) the images in our head.  Jack is a big believer in the power of visualization and he reminds us that’s how athletes perform at greater levels — they see things in their minds, to guide what they can do with their bodies.

Jack shares a very simple formula for success.  Jack’s success formula is Event + Response = Outcome.  If you want to change the outcome, then change your response.  It sounds simple, but it’s empowering.

Jack Canfield also reminded us that we are the creative force in our life and to get out of victimism:

“You are not the victim of your circumstances–You are the creative force of your life.”

Grow your circle of influence and make tremendous impact.

Read more at Lessons Learned from Jack Canfield.

Ken Blanchard at Microsoft

Ken Blanchard is really about accentuating the positive.  So much of the world focuses on what’s wrong, but he wants to focus on what’s right, so we can do more of that.

Ken has an incremental model of leadership that starts with you and expands from there: you, your team, your organization.  The idea is that you can’t lead others effectively, if you can’t even lead yourself.

Ken’s model for leadership is really an adaptive model, that’s focused on the greater good, and it starts by helping everybody get an “A.”  Leaders that apply one style to all team members, aren’t very effective.  Ken suggests that leaders apply the right styles depending on what individuals need.  Ken’s 4 leadership styles are:

  1. Directive
  2. Coaching
  3. Supportive
  4. Delegating.

Perhaps, the most profound statement that Ken made is that “leadership is love.”  He said that leadership includes “loving your mission”, “loving your cusotmers”, “loving your people”, and “loving yourself — enough to get out of the way so others can be magnificent.”

Read more at Lessons Learned from Ken Blanchard.

Stephen Covey at Microsoft

Stephen Covey was really about personal effectiveness, realizing your potential, and leaving a legacy.

Covey really emphasized a whole-person approach: Body, Mind, Heart, Spirit.  His point was that if you take one of the four parts of your nature away, then you’re treating a person like a “thing” you control and manage.

Covey also emphasized the importance of a personal mission.  It gives meaning to your work and it helps you channel all of your efforts as you live and lead your legacy.  He also suggested writing your personal mission down and visualizing it to imprint it on your subconscious.

The other key to realizing your potential is finding your voice.  Use all of you, your best way, in your unique way, for your best results.  That’s how you differentiate and add value for yourself and others.

And, of course, Stephen Covey reminded us of the 7 Habits of Highly Effective People:

  1. Be proactive.
  2. Begin with the end in mind.
  3. Put first things first.
  4. Think win-win.
  5. Seek first to understand, then to be understood.
  6. Synergize.
  7. Sharpen the saw.

Habits 1,2,and 3 are the foundation for private victories and integrity.  Habits 4, 5, and 6 are the keys to public victories.

Read more at Lessons Learned from Stephen Covey.

All-in-all, I have to say that while individually each of these personal empowerment all-stars has great wisdom and insight for personal effectiveness, leadership, and success, they are actually “better together.”

Each day in the halls of Microsoft, I find myself reflecting on their one-liner reminders, whether it’s Covey’s “Put first things first,” or Canfield’s “You are the creative force of your life”, or Blanchard’s “None of us is as smart as all of us.”

You Might Also Like

How Tos for Personal Effectiveness at a Glance

Personal Effectiveness at Microsoft

Personal Effectiveness Toolbox

Categories: Architecture, Programming

Compile-Time Evaluation in Scala with macros

Xebia Blog - Sun, 03/27/2016 - 15:20
Many 'compiled' languages used to have a strict separation between what happens at 'compile-time' and what happens at 'run-time'. This distinction is starting to fade: JIT compilation moves more of the compile phase to run-time, while conversely various kinds of optimizations do 'run-time' work at compile time. Powerful type systems allow the expression things previously

Common Sense Agile Scaling (Part 1: Intro)

Xebia Blog - Sat, 03/26/2016 - 10:05
Agile is all about running experiments and see what works for you. Inspect and adapt. Grow. This also applies to scaling your agile organization. There is no out of the box scaling framework that will fit your organization perfectly and instantly. Experiment, and combine what you need from as many agile models and frameworks as