Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

10 Big Ideas from Mindset: The New Psychology of Success

My parents taught me early on to focus on growth over greatness.

The idea was that while natural ability can take you only so far, it’s things like curiosity, challenges, continuous learning, the power of persistence, taking risks, etc. that would take you further.

They also taught me that if I worried about whether I was naturally good, that I would give up on things where I didn’t start off so great.

It was great advice, even if it wasn’t scientific.

But there is science.

In fact, there’s a lot of science about how choosing a growth mindset over a fixed mindset help people to become the best in their field.  A growth mindset is what actually creates better parents, teachers, coaches, and CEOs.   A growth mindset creates better students, better artists, and even better geniuses.

Why?

Because people with a growth mindset embrace the challenges, struggles, criticisms, and setbacks as a source of growth.

And that’s how they rise above any limitation of “natural” ability.

Teaching, learning, and continuous growth takes them further than relying on talent or fear of taking risks where they might look bad or might not start off so great.

Carol S. Dweck, Ph.D.  wrote an outstanding book on how our mindsets shape us and how they can limit or enable us to realize our potential.

I wrote up my take aways using a “10 Big Ideas from …” style:

10 Big Ideas from Mindset: The New Psychology of Success

I think you'll enjoy the insights and I think you’ll appreciate how you can apply them to work and life.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For April 18th, 2014

Hey, it's HighScalability time:


Scaling to the top of "Bun Mountain" in Hong Kong
  • 44 trillion gigabytes: size of the digital universe by 2020; 6 Times: we have six times more "stuff" than the generation before us.
  • Quotable Quotes:
    • Facebook: Our warehouse stores upwards of 300 PB of Hive data, with an incoming daily rate of about 600 TB.
    • @windley: The problem with the Internet of Things is right now it’s more like the CompuServe of Things
    • Chip Overclock: If you want to eventually generate revenue, you must first optimize for developer productivity; everything else is negotiable.
    • @igrigorik: if you are gzipping your images.. you're doing it wrong: http://bit.ly/1pb8JhK  - check your server configs! and your CDN... :)
    • Seth Lloyd: The arrow of time is an arrow of increasing correlations.
    • @kitmacgillivray: When will Google enable / require all android apps to have full deep search integration so all content is visible to the engine?
    • Neal Ford: Yesterday's best practices become tomorrow's anti-patterns.
    • Rüdiger Möller: just made a quick sum up of concurrency related issues we had in a 7-15 developer team soft realtime application (clustered inmemory data grid + GUI front end). 95% of threads created are not to improve throughput but to avoid blocking (e.g. IO, don't block messaging receiver thread, avoid blocking the event thread in a GUI app, ..).
    • Ansible: When done correctly, automation tools are giving them time back -- and helping out of this problem of needing to wear many hats.

  • Amazon and Google are in an epic battle to dominate the cloud—and Amazon may already have won: If Amazon’s entire public cloud were a single computer, it would have five times more capacity than those of its next biggest 14 competitors—including Google—combined. Every day, one-third of people who use the internet visit a site or use a service running on Amazon’s cloud.

  • What books would you select to help sustain or rebuild civilization? Here's Neal Stephenson’s list. He was too shy to do so, but I would certainly add his books to my list. 

  • 12-Step Program for Scaling Web Applications on PostgreSQL. Great write up of lessons learned by wanelo.com that they used to sustain 10s of thousand of concurrent users at 3K req/sec. Highly detailed. 1) Add more cache, 2) Optimize SQL, 3) Upgrade Hardware and RAM, 4) Scale reads by replication, 5) Use more appropriate tools, 6) Move write heave table out, 7) Tune Postures and your File System, 8) Buffer and serialize frequent updates, 9) Optimize DB Schema, 10) Shard busy tables vertically, 11) Wrap busy tables with services, 12) Shard services backend horizontally.

  • Is this a soap opera? It turns out Google and not Facebook is buying Titan Aerospace, makers of high flying solar powered drones. Google's fiber network would make a great backbone network for a drone and loon powered wireless network, completely routing around the telcos.

  • Building Carousel, Part I: How we made our networked mobile app feel fast and local. Dropbox changes to an eventualy consistent / optimistic replication syncing model to make their app "feel fast, responsive, and local, even though the data on which users operate is ultimately backed by the Dropbox servers."  Lesson: don’t block the user! Instead of requiring changes to be propagated to the server synchronously. Local and remote photos are treated as equivalent objects. Actions take effect immediately locally then work there way out globally. Changes are used to stay consistent. A fast hash is used to tell what photos have not been backed up to dropbox. Remote operations happen asynchronously.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Six Lessons Learned the Hard Way About Scaling a Million User System

Ever come to a point where you feel you've learned enough to share your experiences in the hopes of helping others traveling the same road? That's what Martin Kleppmann has done in an lovingly written Six things I wish we had known about scaling, an article well worth your time.

It's not advice about scaling a Twitter, but of building a million user system, which is the sweet spot for a lot of projects. His conclusion rings true:

Building scalable systems is not all sexy roflscale fun. It’s a lot of plumbing and yak shaving. A lot of hacking together tools that really ought to exist already, but all the open source solutions out there are too bad (and yours ends up bad too, but at least it solves your particular problem).

Here's a gloss on the six lessons (plus a bonus lesson):

Categories: Architecture

Sponsored Post: Apple, HelloSign, CrowdStrike, Gengo, Layer, The Factory, Airseed, ScaleOut Software, Couchbase, Tokutek, MongoDB, BlueStripe, AiScaler, Aerospike, LogicMonitor, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • Apple is hiring a Senior Engineer in their Mobile Services team. We seek an accomplished server-side engineer capable of delivering an extraordinary portfolio of features and services based on emerging technologies to our internal customers. Please apply here

  • Apple is hiring a Software Engineer in their Messaging Services team. We build the cloud systems that power some of the busiest applications in the world, including iMessage, FaceTime and Apple Push Notifications. You'll have the opportunity to explore a wide range of technologies, developing the server software that is driving the future of messaging and mobile services. Please apply here.

  • Senior Engineer wanted for large scale, security oriented distributed systems application that offers career growth and independent work environment. Use your talents for good instead of getting people to click ads at CrowdStrike. Please apply here.

  • Ops Engineer - Are you passionate about scaling and automating cloud-based websites? Love Puppet and deployment scripts? Want to take advantage of both your sys-admin and DevOps skills? Join HelloSign as our second Ops Engineer and help us scale as we grow! Apply at http://www.hellosign.com/info/jobs

  • Human Translation Platform Gengo Seeks Sr. DevOps Engineer. Build an infrastructure capable of handling billions of translation jobs, worked on by tens of thousands of qualified translators. If you love playing with Amazon’s AWS, understand the challenges behind release-engineering, and get a kick out of analyzing log data for performance bottlenecks, please apply here.

  • We’re looking for talented and driven engineers who are passionate about scalability and reliability to help us build Layer, the open communications layer for the Internet. Layer enables app developers to easily build secure, scalable messaging, voice and video features into any app. For more information and our full list of openings, please visit: https://layer.com/jobs

  • The Factory is seeking a collaborative, talented and entrepreneurial Sr. Front End Engineer. Backed by the co-founder of Skype+Rdio and led by Rdio's founding team, we're changing the way products are built and companies get launched (think incubator/accelerator without the nagging outside influence or funding/timing constraints).  Our goal is to build a platform to better launch startups and opensource what we do along the way http://www.thefactory.com/pdfs/sr_frontend.pdf

  • Airseed -- a Google Ventures backed, developer platform that powers single sign-on authentication, rich consumer data, and analytics -- is hiring lead backend and fullstack engineers (employees #4, 5, 6). More info here: https://www.airseed.com/jobs

  • UI EngineerAppDynamics, founded in 2008 and lead by proven innovators, is looking for a passionate UI Engineer to design, architect, and develop our their user interface using the latest web and mobile technologies. Make the impossible possible and the hard easy. Apply here.

  • Software Engineer - Infrastructure & Big DataAppDynamics, leader in next generation solutions for managing modern, distributed, and extremely complex applications residing in both the cloud and the data center, is looking for a Software Engineers (All-Levels) to design and develop scalable software written in Java and MySQL for backend component of software that manages application architectures. Apply here.
Fun and Informative Events
  • Safeguard Commercial Success with a Strategic Monitoring Approach. Does your enterprise have adequate monitoring strategies in place to deal with tech challenges that can otherwise impact revenue and damage your corporate brand? Register now to learn why your business must elevate the strategic importance of technology monitoring and learn how to safeguard your business

  • The Biggest MongoDB Event Ever Is On. Will You Be There? Join us in New York City June 23-25 for MongoDB World! The conference lineup includes Amazon CTO Werner Vogels and Cloudera Co-Founder Mike Olson for keynote addresses.  You’ll walk away with everything you need to know to build and manage modern applications. Register before April 4 to take advantage of super early bird pricing.

  • April 3 Webinar: The BlueKai Playbook for Scaling to 10 Trillion Transactions a Month. As the industry’s largest online data exchange, BlueKai knows a thing or two about pushing the limits of scale. Find out how they are processing up to 10 trillion transactions per month from Vice President of Data Delivery, Ted Wallace. Register today.
Cool Products and Services
  • GigOM Interviews Aerospike at Structure Data 2014 on Application Scalability. Aerospike Technical Marketing Director, Young Paik explains how you can add rocket fuel to your big data application by running the Aerospike database on top of Hadoop for lightning fast user-profile lookups. Watch this interview.

  • Couchbase: NoSQL and the Hybrid Cloud. If a NoSQL database can be deployed on-premise or it can be deployed in the cloud, why can’t it be deployed on-premise and in the cloud? It can, and it should. Read how in this article converting three use cases for hybrid cloud deployments of NoSQL databases: master / slave, cloud burst, and multi-master.

  • Do Continuous MapReduce on Live Data? ScaleOut Software's hServer was built to let you hold your daily business data in-memory, update it as it changes, and concurrently run continuous MapReduce tasks on it to analyze it in real-time. We call this "stateful" analysis. To learn more check out hServer.

  • LogicMonitor is the cloud-based IT performance monitoring solution that enables companies to easily and cost-effectively monitor their entire IT infrastructure stack – storage, servers, networks, applications, virtualization, and websites – from the cloud. No firewall changes needed - start monitoring in only 15 minutes utilizing customized dashboards, trending graphs & alerting.

  • BlueStripe FactFinder Express is the ultimate tool for server monitoring and solving performance problems. Monitor URL response times and see if the problem is the application, a back-end call, a disk, or OS resources.

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Cloud deployable. Free instant trial, no sign-up required.  http://aiscaler.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below. Please click to read more...

Categories: Architecture

Azure Updates: Web Sites, VMs, Mobile Services, Notification Hubs, Storage, VNets, Scheduler, AutoScale and More

ScottGu's Blog - Scott Guthrie - Mon, 04/14/2014 - 21:58

It has been a really busy last 10 days for the Azure team. This blog post quickly recaps a few of the significant enhancements we’ve made.  These include:

  • Web Sites: SSL included, Traffic Manager, Java Support, Basic Tier
  • Virtual Machines: Support for Chef and Puppet extensions, Basic Pricing tier for Compute Instances
  • Virtual Network: General Availability of DynamicRouting VPN Gateways and Point-to-Site VPN
  • Mobile Services: Preview of Visual Studio support for .NET, Azure Active Directory integration and Offline support;
  • Notification Hubs: Support for Kindle Fire devices and Visual Studio Server Explorer integration
  • Autoscale: General Availability release
  • Storage: General Availability release of Read Access Geo Redundant Storage
  • Active Directory Premium: General Availability release
  • Scheduler service: General Availability release
  • Automation: Preview release of new Azure Automation service

All of these improvements are now available to use immediately (note that some features are still in preview).  Below are more details about them:

Web Sites: SSL now included at no additional charge in Standard Tiers

With Azure Web Sites you can host up to 500 web-sites in a single standard tier hosting plan.  Azure web-sites run in VMs isolated to host only your web applications (giving you predictable performance and security isolation), and you can scale-up/down the number of VMs either manually or using our built-in AutoScale functionality.  The pricing for standard tier web-sites is based on the number of VMs you run – if you host all 500 web-sites in a single VM then all you pay for is for that single VM, if you scale up your web site plan to run across two VMs then you’d pay for two VMs, etc.

Prior to this month we charged an additional fee if you wanted to enable SSL for the sites.  Starting this month, we now include the ability to use 5 SNI based SSL certificates and 1 IP based SSL certificate with each standard tier web site hosting plan at no additional charge.  This helps make it even easier (and cheaper) to SSL enable your web-sites.

Web Sites: Traffic Manager Support

I’ve blogged in the past about the Traffic Manager service we have built-into Azure. 

The Azure Traffic Manager service allows you to control the distribution of user traffic to applications that you host within Azure. This enables you to run instances of your applications across different azure regions all over the world.  Traffic Manager works by applying an intelligent routing policy engine to the Domain Name Service (DNS) queries on your domain names, and maps the DNS routes to the appropriate instances of your applications (e.g. you can setup Traffic Manager to route customers in Europe to a European instance of your app, and customers in North America to a US instance of your app).

You can use Traffic Manager to improve application availability - by enabling automatic customer traffic fail-over scenarios in the event of issues with one of your application instances.  You can also use Traffic Manager to improve application performance - by automatically routing your customers to the closet application instance nearest them.

We are excited to now provide general availability support of Traffic Manager with Azure Web Sites.  This enables you to both improve the performance and availability of your web-sites.  You can learn more about how to take advantage of this new support here.

Web Sites: Java Support

This past week we added support for an additional server language with Azure Web Sites – Java.  It is now easy to deploy and run Java web applications written using a variety of frameworks and containers including:

  • Java 1.7.0_51 – this is the default supported Java runtime
  • Tomcat 7.0.50 – the default Java container
  • Jetty 9.1.0

You can manage which Java runtime you use, as well as which container hosts your applications using the Azure management portal or our management APIs.  This blog post provides more detail on the new support and options.

With this announcement, Azure Web Sites now provides first class support for building web applications and sites using .NET, PHP, Node.js, Python and Java.  This enables you to use a wide variety of language + frameworks to build your applications, and take advantage of all the great capabilities that Web Sites provide (Easy Deployment, Continuous Deployment, AutoScale, Staging Support, Traffic Manager, outside-in monitoring, Backup, etc).

Web Sites: Support for Wildcard DNS and SSL Certificates

Azure Web Sites now supports the ability to map wildcard DNS and SSL Certficates to web-sites.  This enables a variety of scenarios – including the ability to map wildcard vanity domains (e.g. *.myapp.com – for example: scottgu.myapp.com) to a single backend web site.  This can be particularly useful for SaaS based scenarios.

Scott Cate has an excellent video that walks through how to easily set this support up.

Web Sites: New Basic Tier Pricing Option

Earlier in this post I talked about how we are now including the ability to use 5 SNI and 1 IP based SSL certificate at no additional cost with each standard tier azure web site hosting plan.  We have also recently announced that we are also including the auto-scale, traffic management, backup, staging and web jobs features at no additional cost as part of each standard tier azure web site hosting plan as well.  We think the combination of these features provides an incredibly compelling way to securely host and run any web application.

New Basic Tier Pricing Option

Starting this month we are also introducing a new “basic tier” option for Azure web sites which enables you to run web applications without some of these additional features – and at 25% less cost.  We think the basic tier is great for smaller/less-sophisticated web applications, and enables you to be successful while paying even less. 

For additional details about the Basic tier pricing, visit the Azure Web sites pricing page.  You can select which tier your web-site hosting plan uses by clicking the Scale tab within the Web Site extension of the Azure management portal.

Virtual Machines: Create from Visual Studio

With the most recent Azure SDK 2.3 release, it is now possible to create Virtual Machines from directly inside Visual Studio’s Server Explorer.  Simply right-click on the Azure node within it, and choose the “Create Virtual Machine” menu option:

image

This will bring up a “Create New Virtual Machine” wizard that enables you to walkthrough creating a Virtual Machine, picking an image to run in it, attaching it to a virtual network, and open up firewall ports all from within Visual Studio:

image

Once created you can then manage the VM (shutdown, restart, start, remote desktop, enable debugging, attach debugger) all from within Visual Studio:

image

This makes it incredibly easy to start taking advantage of Azure without having to leave the Visual Studio IDE.

Virtual Machines: Integrated Puppet and Chef support

In a previous blog post I talked about the new VM Agent we introduced as an optional extension to Virtual Machines hosted on Azure.  The VM Agent is a lightweight and unobtrusive process that you can optionally enable to run inside your Windows and Linux VMs. The VM Agent can then be used to install and manage extensions, which are software modules that extend the functionality of a VM and help make common management scenarios easier. 

At last week’s Build conference we announced built-in support for several new extensions – including extensions that enable easy support for Puppet and Chef.  Puppet and Chef allow developers and IT administrators to define and automate the desired state of infrastructure configuration, making it effortless to manage 1000s of VMs in Azure.

Enabling Puppet Support

We now have a built-in VM image within the Azure VM gallery that enables you to easily stand up a puppet-master server that you can use to store and manage your infrastructure using Puppet.  Creating a Puppet Master in Azure is now easy – simply select the “Puppet Enterprise” template within the VM gallery:

image

You can then create new Azure virtual machines that connect to this Puppet Master.  Enabling this with VMs created using the Azure management portal is easy (we also make it easy to do with VMs created with the command-line).  To enable the Puppet extension within a VM you create using the Azure portal simply navigate to the last page of the Create VM from gallery experience and check the “Puppet Extension Agent” extension within it:

image

Specify the URL of the Puppet master server to get started. Once you deploy the VM, the extension will configure the puppet agent to connect to this Puppet master server and pull down the initial configuration that should be used to configure the machine.

This new support makes it incredibly easy to get started with both Puppet and Chef and enable even richer configuration management of your IaaS infrastructure within Azure.

Virtual Machines: Basic Tier

Earlier in this blog post I discussed how we are introducing a new “Basic Tier” option for Azure Web Sites.  Starting this month we are also introducing a “Basic Tier” for Virtual Machines as well.

The Basic Tier option provides VM options with similar CPU + memory configuration options as our existing VMs (which are now called “Standard Tier” VMs) but do not include the built-in load balancing and AutoScale capabilities.  They also cost up to 27% less.  These instances are well-suited for production applications that do not require a built-in load balancer (you can optionally bring your own load balancer), batch processing scenarios, as well as for dev/test workloads.  Our new Basic tier VMs also have similar performance characteristics to AWS’s equivalent VM instances (which are less powerful than the Standard tier VMs we have today).

Comprehensive pricing information is now available on the Virtual Machines Pricing Details page.

Networking: General Availability of Azure Virtual Network Dynamic Routing VPN Gateways and Point-to-Site VPN

Last year, we previewed a feature called DynamicRouting Gateway and Point-to-Site VPN that supports Route-based VPNs and allows you to connect individual computers to a Virtual Network in Azure. Earlier this month we announced that the feature is now generally available. The DynamicRouting VPN Gateway in a Virtual Network will now carry the same 99.9% SLA as the StaticRouting VPN Gateway.

clip_image037

Now that we’re in General Availability mode, DynamicRouting Gateway will automatically incur standard Gateway charges which will take effect starting May 1, 2014. 

For further details on the service, please visit the Virtual Network website.

Mobile Services: Visual Studio Support for Mobile Services .NET Backend

With Visual Studio 2013 Update 2, you can now create your backend Mobile Service logic using .NET and the ASP.NET Web API framework in Visual Studio, using Mobile Services templates and scaffolds. Mobile Services support for .NET on the backend offers the following benefits:

  1. You can use ASP.NET Web API and Visual Studio together with Mobile Services to add a backend to your mobile app in minutes
  2. You can publish any existing Web API to Mobile Services and benefit from authentication, push notifications and other capabilities that Mobile Services provides. You can also take advantage of any Web API features like OData controllers, or 3rd party Web API-based frameworks like Breeze.
  3. You can debug your Mobile Services .NET backend using Visual Studio running locally on your machine or remotely in Azure.
  4. With Mobile Services we run, manage and monitor your Web API for you. Azure will automatically notify you if we discover you have a problem with your app.
  5. With Mobile Services .NET support you can store your data securely using any data backend of your choice: SQL Azure, SQL on Virtual Machine, Azure Table storage, Mongo, et al.

It’s easy to get started with Mobile Services .NET support in Visual Studio. Simply use the File-New Project dialog and select the Windows Azure Mobile Service project template under the Cloud node.

clip_image012

Choose Windows Azure Mobile Service in the New ASP .NET Project dialog.

clip_image014

You will see a Mobile Services .NET project, notice this is a customized ASP .NET Web API project with additional Mobile Service NuGet packages and sample controllers automatically included:

clip_image016

Running the Mobile Service Locally

You can now test your .Net Mobile Service project locally. Open the sample TodoItemController.cs in the project. The controller shows you how you can use the built-in TableController<T> .NET class we provide with Mobile Services. Set a breakpoint inside the GetAllTodoItems() method and hit F5 within Visual Studio to run the Mobile Service locally.

clip_image018

Mobile Services includes a help page to view and test your APIs. On the help page, click on the try it out link and then click the GET tables/TodoItem link. Then click try this out and send on the GET tables/TodoItem page. As you might expect, you will hit the breakpoint you set earlier.

clip_image020

Add APIs to your Mobile Service using Scaffolds

You can add additional functionality to your Mobile Service using Mobile Service or generic Web API controller scaffolds through the Add Scaffold dialog (right click on your project and choose Add -> New Scaffolded Item… command)

clip_image022

Publish your Mobile Services project to Azure

Once you are done developing your Mobile Service locally, you can publish it to Azure. Simply right click on your project and choose the Publish command. Using the publish wizard, you can publish to a new or existing Azure Mobile Service:

clip_image024

Remote debugging

Just like Cloud Services and Websites, you can now remote debug your Mobile Service to get more visibility into how your code is operating live in Azure. To enable remote debugging for a Mobile Service, publish your Mobile Service again and set the Configuration to Debug in the Publish wizard.

clip_image027

Once your Mobile Service is published and running live in the cloud, simply set a breakpoint in local source code. Then use Visual Studio’s Server Explorer to select the Mobile Service instance deployed in the cloud, right click and choose the Attach Debugger command.

clip_image028

Once the debugger attaches to the mobile service, you can use the debugging capabilities of Visual Studio to instantly and in-real time debug your app running in the cloud.

To learn more about Visual Studio Support for Mobile Services .NET backend follow tutorials at:

This new .NET backend supports makes it easy to create even better mobile applications than ever before.

Mobile Services: Offline Support

In addition to the above support, we are also introducing a preview of a new Mobile Services Offline capability with client SDK support for Windows Phone and Windows Store apps.

With this functionality, mobile applications can create and modify data even when they are offline/disconnected from a network. When the app is back online, it can synchronize local changes with the Mobile Services Table APIs. The feature also includes support for detecting conflicts when the same record is changed on both the client and the backend.

To use the new Mobile Services offline functionality, set up a local sync store. You can define your own sync store or use the provided SQLite-based implementation.  The Mobile Services SDK provides a new local table API for the sync store, with a symmetrical programming model to the existing Mobile Services Table API. You can use Optimistic Concurrency along with the offline feature to detect conflicting changes between the client and backend.

The preview of the Mobile Services Offline feature is available now as part of the Mobile Services SDK for Windows Store and Windows Phone apps. In the future, we will support all client platforms supported by Mobile Services, including iOS, Android, Xamarin, etc.

Mobile Services: Support for Azure Active Directory Sign On

We now support Azure Active Directory Single Sign On for Mobile Services.  Azure Active Directory authentication is available for both the .NET and Node.js backend options of Mobile Services.

To take advantage of the feature, first register your client app and your Mobile Service with your Azure Active Directory tenant using the Applications tab in the Azure Active Directory management portal.

clip_image030

In your client project, you will need to add the Active Directory Authentication Library (ADAL), currently available for Windows Store, iOS, and Android clients.

From there on, the token retrieved from ADAL library can be used to authenticate and access Mobile Services.  The single sign-on features of ADAL also enables your mobile service to make calls to other resources (such as SharePoint and Office 365) on behalf of the user.  You can read more about the new ADAL functionality here.

These new updates make Mobile Services an even more attractive platform for building powerful employee facing apps.

Notification Hub: Kindle Support and Visual Studio Integration

I’ve previously blogged about Azure Notification Hubs, a high scale cross platform push notification service that allows you to instantly send personalized push notifications to segments of your audience or individuals containing millions of iOS, Android, Windows, Widows Phone devices with a single API call.

Today we’ve made two important updates to Azure Notification Hubs: adding support for Amazon Kindle Fire devices, and Visual Studio support for Notification Hubs.

Support for Amazon’s Kindle

With today’s addition you can now configure your Notification Hubs with Amazon Device Messaging (ADM) service credentials on the configuration page for your Notification Hub in the Azure Management portal, and start sending push notifications to your app on Amazon’s Kindle device, in addition to iOS, Android, or Windows.

clip_image032

Testing Push Notifications with Visual Studio

Earlier I blogged about how we enabled debugging push notifications using the Azure Management Portal. With today’s Visual Studio update, you can now browse your notification hubs and send test push notifications directly from Visual Studio Server Explorer as well.

Simply select your notification hub in the Server Explorer of Visual Studio under the Notifications Hubs node.  Then right click, and choose the Send Test Notifications command:

clip_image033

In the notification hub window, you can then send a message either to a particular tag or all registered devices (broadcast). You can select from a variety of templates - Windows Store, Windows Phone, Android, iOS, or even a cross platform message using the Custom Template. After you hit Send, you’ll receive the message result instantly to help you diagnose if your message was successfully sent or not.

clip_image035

To learn more about Azure Notification Hubs, read tutorials here.

AutoScale: Announcing General Availability of Autoscale Service

Last summer we announced the preview release of our Autoscale service. I’m happy to announce that Autoscale is now generally available!  Better yet, there's no additional charge for using Autoscale.

We've added new features since we first released it as a preview version: support for both performance-and schedule-based autoscaling, along with an API and .NET SDK so you can programmatically scale using any performance counters that you define.

Autoscale supports all four Azure compute services: Cloud Services, Virtual Machines, Mobile Services and Web Sites. For Virtual Machines and Web Sites, Autoscale is included as a feature in the Standard pricing tiers, and for Mobile Services, it's included as a part of both Basic and Standard pricing tiers.

Storage: Announcing General Availability of Read Access Geo Redundant Storage (RA-GRS)

In December, we added the ability to allow customers to achieve higher read availability for their data. This feature called Read Access - Geo Redundant Storage (RA-GRS) allows you to read an eventually consistent copy of your geo-replicated data from the storage account’s secondary region in case of any unavailability to the storage account’s primary region.

Last week we announced that RA-GRS feature is now out of preview mode, and generally available. It is available to all Azure customers across all regions including the users in China.

RA-GRS SLA and Pricing

The benefit of using RA-GRS is that it provides a higher read availability (99.99+%) for a storage account over GRS (99.9+%). When using RA-GRS, the write availability continues to be 99.9+% (same as GRS today) and read availability for RA-GRS is 99.99+%, where the data is expected to be read from secondary if primary is unavailable. In terms of pricing, the capacity (GB) charge is slightly higher for RA-GRS than GRS, whereas the transaction and bandwidth charges are the same for GRS and RA-GRS. See the Windows Azure Storage pricing page here for more details about the SLA and pricing.

You can find more information on the storage blog here.

Active Directory: General Availability of Azure AD Premium

Earlier this month we announced the general availability of Azure Active Directory Premium, which provides additional identity and access management capabilities for enterprises. Building upon the capabilities of Azure AD, Azure AD Premium provides these capabilities with a guaranteed SLA and no limit on directory size. Additional capabilities include:

  • Group-based access assignment enables administrators to use groups in AD to assign access for end users to over 1200 cloud applications in the AD Application Gallery. End users can get single-sign on access to their applications from their Access Panel at https://myapps.microsoft.com or from our iOS application.
  • Self-service password reset that enables end users to reset forgotten passwords without calling your help desk.
  • Delegated group management that enables end users to create security groups and manage membership in security groups they own.
  • Multi-Factor Authentication that lets you easily deploy a Multi-Factor Authentication solution for your business without deploying new software or hardware.
  • Customized branding that lets you include your organization’s branding elements in the experiences that users see when signing in to AD or accessing their Access Panel.
  • Reporting, alerting, and analytics that increase your visibility into application usage in your organization, and potential security concerns with user accounts.

Azure AD Premium also includes usage rights for Forefront Identity Manager Server and Client Access Licenses.

To read more about AD Premium, including how to acquire it, read the Active Directory Team blog.

Active Directory: Public Preview of Azure Rights Management Service

Earlier this month we announced the public preview of the ability to manage your Azure Rights Management service within the Azure Management Portal. If your organization has Azure Rights Management either as a stand-alone service or as part of your Office 365 or EMS subscriptions you can now manage it by signing into the Azure Management Portal. Once in the Portal, select ACTIVE DIRECTORY in the left navigation bar, navigate to the RIGHTS MANAGEMENT tab, then click on the name of your directory.

clip_image039

With this preview you can now create custom rights policy templates that let you define who can access sensitive documents, and what permissions (view, edit, save, print, and more) users can have on those documents.  To begin creating a rights policy template, in the Quick Start page, click on Create an additional rights policy template option and follow the instructions on the page to define a name and description for the template, add users and rights and define other restrictions.

clip_image041

Once your template has been created and published, it will become available to users in your organization in their favorite applications.

clip_image043

To learn more managing Azure Rights Management and the benefits it offers to organizations, see the Information Protection group’s blog

Scheduler: General Availability Release Scheduler Service

This month we’ve also delivered the General Availability release of the Azure Scheduler service.  Scheduler allows you to run jobs on simple or complex recurring schedules that can invoke HTTP/S endpoints or post messages to storage queues. Scheduler has built-in high availability and can reliably call services inside or outside of Azure.

During preview customers have used it for a wide set of scenarios including for invoking services in their backend for Hadoop workloads, triggering diagnostics cleanup, and periodically checking that partners have submitted content on time. ISVs have used it to empower their applications to add scheduling capabilities such as report generation and sending reminders.

In the Scheduler portal extension you can easily create and manage your scheduler jobs. Since the initial release, Scheduler has also added the ability to update HTTP jobs with custom headers and basic authentication. It has also exposed the ability to change the recurrence schedule which will allow you to also choose to limit the execution of a job or allow the job to run infinitely.

With the general availability, new Azure Scheduler cmdlets have been released with Azure PowerShell and the Scheduler .NET API has been included in WAML 1.0.

I highly encourage you to try out the Scheduler today. You might find the following links helpful:

It makes scheduling recurring tasks really easy.

Automation: Announcing Microsoft Azure Automation Preview

Last week we announced the preview of a new Microsoft Azure service: Automation.

Automation allows you to automate the creation, deployment, monitoring, and maintenance of resources in your Azure environment using a highly scalable and reliable workflow execution engine. The service can be used to orchestrate the time-consuming, error-prone, and frequently repeated tasks you’d otherwise accomplish manually across Microsoft Azure and third-party systems to decrease operational expense for your cloud operations.

To get started with Automation, you first need to sign-up for the preview on the Azure Preview page. Once you have been approved for the preview, you can sign in to the Management Portal and start using it. Automation is currently only available in the East-US data center, but we will add the ability to deploy to additional data centers in the future.

Authoring a Runbook

Once you have the Automation preview enabled on your subscription, you can easily get started automating by following a few simple steps:

Step 1: In the Microsoft Azure management portal, click New->App Services->Automation->Runbook->Quick Create to create a new runbook. Runbooks are collections of activities that provide an environment for automating everything from diagnostic logging to applying updates to all instances of a virtual machine or web role to renewing certificates to cleaning storage accounts. Enter a name and description for the runbook, and create a new Automation account which will store your Runbooks, Assets, and Jobs.

Next time you create a runbook you can either use the same Automation account as you just created or create a separate one to if you’d like to maintain separation between a few different collections of runbooks / assets.

clip_image045

clip_image047

Step 2: Click on your runbook, then click Author->Draft. Type some PowerShell commands in the editor, then hit ‘Publish’ to make this runbook draft available for production execution.

clip_image049

Starting a Runbook and Viewing the Job

1. To start the runbook you just published, go back to the ‘Runbooks’ tab, click on your newly-published runbook, and hit ‘Start.’ Enter any required parameters for the runbook, then click the checkmark button.

clip_image051

2. Click on your runbook, then click on the ‘Jobs’ tab for this runbook. Here you can view all the instances of a runbook that have run, called jobs. You should see the job you just started.

clip_image053

3. Click on the job you just started to view more details about its execution. Here you can see the job output, as well as any exceptions that may have occurred while the job was executing.

clip_image055

Once you get familiar with the service, you’ll be able to create more sophisticated runbooks to automate your scenarios. I encourage you to try out Microsoft Azure Automation today.

For more information, click through the following links:

Summary

This most recent release of Azure includes a bunch of great features that enable you to build even better cloud solutions.  If you don’t already have a Azure account, you can sign-up for a free trial and start using all of the above features today.  Then visit the Azure Developer Center to learn more about how to build apps with it.

Hope this helps,

Scott

P.S. In addition to blogging, I am also now using Twitter for quick updates and to share links. Follow me at: twitter.com/scottgu

Categories: Architecture, Programming

How do you even do anything without using EBS?

In a recent thread on Hacker News discussing recent AWS price changes, seldo mentioned they use AWS for business, they just never use EBS on AWS. A good question was asked:

How do you even do anything without using EBS?

Amazon certainly makes using EBS the easiest path. And EBS has a better reliability record as of late, but it's still often recommended to not use EBS. This avoids a single point of failure at the cost of a lot of complexity, though as AWS uses EBS internally, not using EBS may not save you if you use other AWS services like RDS or ELB.

If you don't want to use EBS, it's hard to know where to even start. A dilemma to which Kevin Nuckolls gives a great answer:

Well, you break your services out onto stateless and stateful machines. After that, you make sure that each of your stateful services is resilient to individual node failure. I prefer to believe that if you can't roll your entire infrastructure over to new nodes monthly then you're unprepared for the eventual outage of a stateful service.

Most databases have replication but you need to make sure that the characteristics of how the database handles a node failure are well understood. Worst case you use EBS, put your state on it, snapshot it regularly, and ship those snapshots to another region because when EBS fails it fails hard.

Also, logs make every machine stateful. Use something like logstash to centralize that state.

If ELB is down in a given region then DNS failover to another region. Assuming you feel comfortable rolling your entire infrastructure monthly, have good images / configuration management, and have the state replicated in the backup region. That or sidestep ELB in your region to a team of stateless load balancers that terminate SSL.

Jeremy Edberg to a question about how to run databases without EBS, says:

By having good replication, either hand rolled or built in.

At Netflix we use Cassandra and store all data on local instance storage. We don't use EBS for databases.

Categories: Architecture

Why Have a Strategy?

To be able to change it.

Brilliant pithy advice from Professor Jason Davis’ class,Technology Strategy (MIT’s OpenCourseWare.)

Categories: Architecture, Programming

Social Intelligence and 95 Articles to Give You an Unfair Advantage

Social Intelligence is hot.

I added a new category at Sources of Insight to put the power of Social Intelligence at your fingertips:

Social Intelligence

(Note that you can get to Social Intelligence from the menu under “More Topics …”)

I wanted a simple category to capture and consolidate the wealth of insights around interpersonal communication, relationships, conflict, influence, negotiation, and more.   There are 95 articles in this category, and growing, and it includes everything from forging friendships to dealing with people you can’t stand, to building better relationships with your boss.

According to Wikipedia, “Social intelligence is the capacity to effectively negotiate complex social relationships and environments.”

There's a great book on Social Intelligence by Daniel Goleman:

Social Intelligence, The New Science of Human Relationships

According to Goleman, “We are constantly engaged in a ‘neural ballet’ that connects our brain to the brains with those around us.”

Goleman says:

“Our reactions to others, and theirs to us, have a far-reaching biological impact, sending out cascades of hormones that regulate everything from our hearts to our immune systems, making good relationships act like vitamins—and bad relationships like poisons. We can ‘catch’ other people’s emotions the way we catch a cold, and the consequences of isolation or relentless social stress can be life-shortening. Goleman explains the surprising accuracy of first impressions, the basis of charisma and emotional power, the complexity of sexual attraction, and how we detect lies. He describes the ‘dark side’ of social intelligence, from narcissism to Machiavellianism and psychopathy. He also reveals our astonishing capacity for ‘mindsight,’ as well as the tragedy of those, like autistic children, whose mindsight is impaired.”

According to the Leadership Lab for Corporate Social Innovation, by Dr. Claus Otto Scharmer  (MIT OpenCourseware), there is a relational shift:

The Rise of the Network Society

And, of course, Social is taking off as a hot technology in the Enterprise arena.  It’s changing the game, and changing how people innovate, communicate, and collaborate in a comprehensive collaboration sort of way.

Here is a sampling of some of my Social Intelligence articles to get you started:

5 Conversations to Have with Your Boss
6 Styles Under Stress
10 Types of Difficult People
Antiheckler Technique
Ask, Mirror, Paraphrase and Prime
Cooperative Controversy Over Competitive Controversy
Coping with Power-Clutchers, Paranoids and Perfectionists
Dealing with People You Can't Stand
Expectation Management
How To Consistently Build a Winning Team
How To Deal With Criticism
How Do You Choose a Friend?
How To Repair a Broken Work Relationship
Mutual Purpose
Superordinate Goals
The Lens of Human Understanding
The Politically Competent Leader, The Political Analyst, and the Consensus Builder
Work on Me First

If you really want to dive in here, you can brows the full collection at:

Social Intelligence

Enjoy, and may the power of Social Intelligence be with you.

Categories: Architecture, Programming

The Industry Life Cycle

I’m a fan of simple models that help you see things you might otherwise miss, or that help explain how things work, or that simply show you a good lens for looking at the world around you.

Here’s a simple Industry Life Cycle model that I found in Professor Jason Davis’ class, Technology Strategy (MIT’s OpenCourseWare.)

image

It’s a simple backdrop and that’s good.  It’s good because there is a lot of complexity in the transitions, and there are may big ideas that all build on top of this simple frame.

Sometimes the most important thing to do with a model is to use it as a map.

What stage is your industry in?

Categories: Architecture, Programming

An Architecture Aware VsVars.ps1

DevHawk - Harry Pierson - Sat, 04/12/2014 - 02:28

Like many in the Microsoft dev community, I’m a heavy user of Visual Studio and Powershell. And so, of course, I’ve been a heavy user Chris Tavares’ vsvars32.ps1 script. However, recently I needed the ability to specify my desired processor architecture when setting up a VS command line session. Unfortunately, Chris’s script wraps vsvars32.bat which only supports generating 32-bit apps. Luckily, VC++ includes a vcvarsall.bat script that let’s you specify processor architecture. So I updated my local copy of vsvars.ps1 to use vcvarsall.bat under the hood and added an -x64 switch to enable setting up a 64-bit command line environment. Vcvarsall.bat supports a variety of additional options, but 64-bit support is all I needed so that’s all I added. I didn’t change the name of the script because there’s WAY too much muscle memory associated with typing “vsvars” to bother changing that now.

If you want it, you can get my architecture aware version of vsvars.ps1 from my OneDrive here: http://1drv.ms/1kf8g9I.

Categories: Architecture, Programming

Brokered Component Wake On Callback Demo Video

DevHawk - Harry Pierson - Fri, 04/11/2014 - 23:43

As you might imagine, I had a pretty amazing time @ Build. The only thing that went wrong all week was when one of my demos in my session failed. It’s was pretty cool demo – the brokered WinRT component fires an event which wakes up a suspended WinRT app for a few seconds to process the event. However, I had shut off toast notifications on my machine, which messed up the demo. So here, for your enjoyment, is a short 3 minute video of the working demo.

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For April 11th, 2014

Hey, it's HighScalability time:

 Daly and Newton/Getty Images)
DNA nanobots deliver drugs in living cockroaches which have as much compute power as a Commodore 64
  • 40,000: # of people it takes to colonize a star system; 600,000: servers vulnerable to heartbleed
  • Quotable Quotes:
    • @laurencetratt: High frequency traders paid $300M to reduce New York <-> Chicago network transfers from 17 -> 13ms. 
    • @talios: People read http://highscalability.com  for sexual arousal - jim webber #cm14
    • @viktorklang: LOL RT @giltene 2nd QOTD: @mjpt777 “Some people say ‘Thread affinity is for sissies.’Those people don’t make money.”
    • @pbailis: Reminder: eventual consistency is orthogonal to durability and data loss as long as you correctly resolve update conflicts.
    • @codinghorror: Converting a low-volume educational Discourse instance from Heroku at ~$152.50/month to Digital Ocean at $10/month.
    • @FrancescoC: Scary post on kids who can't tell the diff. between atomicity & eventual consistency architecting bitcoin exchanges 
    • @jboner: "Protocols are a lot harder to get right than APIs, and most people can't get APIs right" -  @daveathomas at #reactconf
    • @vitaliyk: “Redundancy is ambiguous because it seems like a waste if nothing unusual happens. Except that something unusual happens—usually.” @nntaleb
    • Blazes: Asynchrony * partial failure is hard.
    • David Rosenthal: I have long thought that the fundamental challenge facing system architects is to build systems that fail gradually, progressively, and slowly enough for remedial action to be effective, all the while emitting alarming noises to attract attention to impending collapse. 
    • Brian Wilson: Moral of the story: design for failure and buy the cheapest components you can. :-)

  • Just damn. DNA nanobots deliver drugs in living cockroaches: Levner and his colleagues at Bar Ilan University in Ramat-Gan, Israel, made the nanobots by exploiting the binding properties of DNA. When it meets a certain kind of protein, DNA unravels into two complementary strands. By creating particular sequences, the strands can be made to unravel on contact with specific molecules – say, those on a diseased cell. When the molecule unravels, out drops the package wrapped inside.

  • Remember those studies where a guerilla walks through the middle of a basketball game and most people don't notice? Attention blindness. 1000 eyeballs doesn't mean anything will be seen. That's human nature. Heartbleed -- another horrible, horrible, open-source FAIL.

  • Remember the Content Addressable Web? Your kids won't. The mobile web vs apps is another front on the battle between open and closed systems.

  • In Public Cloud Instance Pricing Wars - Detailed Context and Analysis Adrian Cockcroft takes a deep stab at making sense of the recent price cuts by Google, Amazon, and Microsoft. AWS users should migrate to the new m3, r3, c3 instances; AWS and Google instance prices are essentially the same for similar specs; Microsoft doesn't have the latest Intel CPUs and isn't pricing against like spec'ed machines;  IBM Softlayer pricing is still higher; Moore's law dictates price curves going forward.

  • Seth Lloyd: Quantum Machine Learning - QM algorithms are a win because they give exponential speedups on BigData problems. The mathematical structure of QM, because a wave can be at two places at once, is that the states of QM systems are in fact vectors in high dimensional vector spaces. The kind of transformations that happen when particles of light bounce of CDs, for example, are linear transformations on these high dimensional vector spaces. Quantum computing is the effort to exploit quantum systems to allow these linear transformations to perform the kind of calculations we want to perform. Or something like that. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Monoidal Event Sourcing

Think Before Coding - Jérémie Chassaing - Fri, 04/11/2014 - 03:34

I Could not sleep… 3am and this idea…

 

Event sourcing is about fold but there is no monoid around !

 

What’s a monoid ?

 

First, for those that didn’t had a chance to see Cyrille Martaire’s  fabulous explanation with beer glasses, or read the great series of post on F# for fun and profit, here is a quick recap:

 

We need a set.

Let’s take a simple one, positive integers.

 

And an operation, let say + which takes two positive integers.

 

It returns… a positive integer. The operation is said to be closed on the set.

 

Something interesting is that 3 + 8 + 2 = 13 = (3 + 8) + 2 = 3 + (8 + 2).

This is associativity: (x + y) + z = x + (y + z)

 

Le last interesting thing is 0, the neutral element:

x + 0 = 0 + x = x

 

(N,+,0) is a monoid.

 

Let say it again:

(S, *, ø) is a monoid if

  • * is closed on S  (* : S –> S > S)
  • * is associative ( (x * y) * z = x * (y * z) )
  • ø is the neutral element of * ( x * ø = ø * x = x )

warning: it doesn’t need to be commutative so x * y can be different from y * x !

 

Some famous monoids:

(int, +, 0)

(int, *, 1)

(lists, concat, empty list)

(strings, concat, empty string)

 

The link with Event Sourcing

 

Event sourcing is based on an application function apply : State –> Event –> State, which returns the new state based on previous state and an event.

 

Current state is then:

fold apply emptyState events

 

(for those using C# Linq, fold is the same as .Aggregate)

 

Which is great because higher level functions and all…

 

But fold is event more powerful with monoids ! For integers, fold is called sum, and the cool thing is that it’s associative !

 

With a monoid you can fold subsets, then combine them together after (still in the right order). This is what give the full power of map reduce: move code to the data. Combine in place, then combine results. As long as you have monoids, it works.

 

But apply will not be part of a monoid. It’s not closed on a set.

 

To make it closed on a set it should have the following signature:

 

apply: State –> State –> State, so we should maybe rename the function combine.

 

Let’s dream

 

Let’s imagine we have a combine operation closed on State.

 

Now, event sourcing goes from:

 

decide: Command –> State –> Event list

apply: State –> Event –> State

 

to:

decide: Command –> State –> Event list

convert: Event –> State

combine: State –> State –> State

 

the previous apply function is then just:

apply state event = combine state (convert event)

 

and fold distribution gives:

 

fold apply emptyState events = fold combine emptyState (map convert events) 

 

(where map applies the convert fonction to each item of the events list, as does .Select in Linq)

 

The great advantage here is that we map then fold which is another name for reduce !

 

Application of events can be done in parallel by chuncks and then combined !

 

Is it just a dream ?

 

Surely not.

 

Most of the tricky decisions have been taken in the decide function which didn’t change. The apply function usually just set state members to values present in the event, or increment/decrement values, or add items to a list… No business decision is taken in the apply function, and most of the primitive types in state members are already monoids under there usual operations…

 

And a group (tuple, list..) of monoid is also a monoid under a simple composition:

if (N1,*,n1) and (N2,¤,n2) are monoids then N1 * N2 is a monoid with an operator <*> ( (x1,x2) <*> (y1,y2) = (x1*y1, x2¤y2)) and a neutral element (n1,n2)…

 

To view it more easily, the convert fonction converts an event to a Delta, a difference of the State.

 

Those delta can then be aggregated/folded to make a bigger delta.

It can then be applied to a initial value to get the result !

 

 

The idea seams quite interesting and I never read anything about this.. If anyone knows prior study of the subject, I’m interested.

 

Next time we’ll see how to make monoids for some common patterns we can find in the apply function, to use them in the convert function.

Categories: Architecture, Requirements

Paper: Scalable Atomic Visibility with RAMP Transactions - Scale Linearly to 100 Servers

We are not yet at the End of History for database theory as Peter Bailis and the Database Group at UC Berkeley continue to prove with a great companion blog post to their new paper: Scalable Atomic Visibility with RAMP Transactions. I like the approach of pairing a blog post with a paper. A paper almost by definition is formal, but a blog post can help put a paper in context and give it some heart.

From the abstract:

Databases can provide scalability by partitioning data across several servers. However, multi-partition, multi-operation transactional access is often expensive, employing coordination-intensive locking, validation, or scheduling mechanisms. Accordingly, many real-world systems avoid mechanisms that provide useful semantics for multi-partition operations. This leads to incorrect behavior for a large class of applications including secondary indexing, foreign key enforcement, and materialized view maintenance. In this work, we identify a new isolation model—Read Atomic (RA) isolation—that matches the requirements of these use cases by ensuring atomic visibility: either all or none of each transaction’s updates are observed by other transactions. We present algorithms for Read Atomic Multi-Partition (RAMP) transactions that enforce atomic visibility while offering excellent scalability, guaranteed commit despite partial failures (via synchronization independence), and minimized communication between servers (via partition independence). These RAMP transactions correctly mediate atomic visibility of updates and provide readers with snapshot access to database state by using limited multi-versioning and by allowing clients to independently resolve non-atomic reads. We demonstrate that, in contrast with existing algorithms, RAMP transactions incur limited overhead—even under high contention—and scale linearly to 100 servers.

What is RAMP?

Categories: Architecture

Microservices - Not a free lunch!

This is a guest post by Benjamin Wootton, CTO of Contino, a London based consultancy specialising in applying DevOps and Continuous Delivery to software delivery projects. 

Microservices are a style of software architecture that involves delivering systems as a set of very small, granular, independent collaborating services.

Though they aren't a particularly new idea, Microservices seem to have exploded in popularity this year, with articles, conference tracks, and Twitter streams waxing lyrical about the benefits of building software systems in this style.

This popularity is partly off the back of trends such as Cloud, DevOps and Continuous Delivery coming together as enablers for this kind of approach, and partly off the back of great work at companies such as Netflix who have very visibly applied the pattern to great effect.

Let me say up front that I am a fan of the approach. Microservices architectures have lots of very real and significant benefits:

Categories: Architecture

Why does it work in staging but not in production?

Agile Testing - Grig Gheorghiu - Mon, 04/07/2014 - 23:07
This is a question that I am sure was faced by every developer and operation engineer out there. There can be multiple answers to this question, and I'll try to offer some of the ones we arrived at, having to do mainly with our Chef workflow, but that can be applied I think to any other configuration management tool.

1) A Chef cookbook version in staging is different from the version in production

This is a common scenario, and it's supposed to work this way. You do want to test out new versions of your cookbooks in staging first, then update the version of the cookbook in production.

2) A feature flag is turned on in staging but turned off in production

We have Chef attributes defined in attributes/default.rb that serve as feature flags. If a certain attribute is true, some recipe code or template section gets included which wouldn't be included if the attribute were false. The situation can occur where a certain attribute is set to true in the staging environment but is set to false in the production environment, at which point things can get out of sync. Again, this is expected, as you do want to test new features out in staging first, but don't forget to turn them on in production at some point.

3) A block of code or template is included in staging but not in production

We had this situation very recently. Instead of using attributes as feature flags, we were directly comparing the environment against 'stg' or 'prod' inside an if block in a template, and only including that template section if the environment was 'stg'. So things were working perfectly in staging, but mysteriously the template section wasn't even there in production. An added difficulty was that the template in question was peppered with non-indented if blocks, so it took us a while to figure out what was going on.

Two lessons here:

a) Make your templates readable by indenting code blocks.

b) Use attributes as feature flags, and don't compare directly against the current environment. This way, it's easier to always look at the default attribute file and see if a given feature flag is true or false.

4) A modification is made to the cookbook version in production directly on the Chef server

I blogged about this issue in the past. Suppose you have an environments file that pins a given cookbook (let's designate it as cookbook C) to 1.0.1 in staging and to 1.0.0 in production. You want to upgrade production to 1.0.1, because it was tested in staging and it worked fine. However, instead of i) modifying the environments/prod.rb file and pinning the cookbook C to 1.0.1, ii) updating the Chef server via "knife environment from file environments/prod.rb" and iii) committing your changes in git, you modify the version of the cookbook C directly on the Chef server with "knife environment edit prod".

Then, the next time you or somebody else modifies environments/prod.rb to bump up another cookbook to the next version, the version of cookbook C in that file is still 1.0.0, so when you upload environments/prod.rb to the Chef server, it will downgrade cookbook C from 1.0.1 to 1.0.0. Chaos will ensue the next time chef-client runs on the nodes that have recipes from cookbook C. Production will be broken, while staging will still happily work.

Here are 2 other scenarios not related directly to staging vs production, but instead having the potential to break production altogether.

You forget to upload the new version of the cookbook to the Chef server

You make all of your modifications to the cookbook, you commit your code to git, but for some reason you forget to upload the cookbook to the Chef server. Particularly if you keep the same version of the cookbook that is in staging (and possibly in production), then your modifications won't take effect and you may spend some quality time pulling your hair.

You upload a cookbook to the Chef server without bumping its version

There is another, even worse, scenario though: you do upload your cookbook to the Chef server, but you realize that you didn't bump up the version number compared to what is currently pinned to production. As a consequence, all the nodes in production that have recipes from that cookbook will be updated the next time they run chef-client. That's a nasty one. It does happen. So make sure you pay attention to your cookbook versioning process and stick to it!

Google Finds: Centralized Control, Distributed Data Architectures Work Better than Fully Decentralized Architectures

For years a war has been fought in the software architecture trenches between the ideal of decentralized services and the power and practicality of centralized services. Centralized architectures, at least at the management and control plane level, are winning. And Google not only agrees, they are enthusiastic adopters of this model, even in places you don't think it should work.

Here's an excerpt from Google Lifts Veil On “Andromeda” Virtual Networking, an excellent article by Timothy Morgan, that includes a money quote from Amin Vahdat, distinguished engineer and technical lead for networking at Google:

Like many of the massive services that Google has created, the Andromeda network has centralized control. By the way, so did the Google File System and the MapReduce scheduler that gave rise to Hadoop when it was mimicked, so did the BigTable NoSQL data store that has spawned a number of quasi-clones, and even the B4 WAN and the Spanner distributed file system that have yet to be cloned.

 

"What we have seen is that a logically centralized, hierarchical control plane with a peer-to-peer data plane beats full decentralization,” explained Vahdat in his keynote. “All of these flew in the face of conventional wisdom,” he continued, referring to all of those projects above, and added that everyone was shocked back in 2002 that Google would, for instance, build a large-scale storage system like GFS with centralized control. “We are actually pretty confident in the design pattern at this point. We can build a fundamentally more efficient system by prudently leveraging centralization rather than trying to manage things in a peer-to-peer, decentralized manner.

The context of the article is Google's impressive home brew SDN (software defined network) system that uses a centralized control architecture instead of the Internet's decentralized Autonomous System model, which thinks of the Internet as individual islands that connect using routing protocols.

SDN completely changes that model as explained by Greg Ferro:

Categories: Architecture

The Growth Mindset: A Key to Resilience, Motivation, and Achievement

Your mindset holds the key to realizing your potential.

Your mindset is your way of thinking, and your way of thinking can limit or empower you, in any number of ways.

In fact, according to Carol S. Dweck, author of Mindset: The New Psychology of Success, mindset is the one big idea that helps explain the following:

  • Why brains and talent don’t bring success
  • How they can stand in the way of it
  • Why praising brains and talent doesn’t foster self-esteem and accomplishment, but jeopardizes them
  • How teaching a simple idea about the brain raises grades and productivity
  • What all great CEOs, parents, teachers, athletes know

When Dweck was a young researcher, she was obsessed with understanding how people cope with failures, and she decided to study it by watching how students grapple with heard problems.

You’re Learning, Not Failing

One of Dweck’s key insights was that a certain kind of mindset could turn  a failure into a gift.

Via Mindset: The New Psychology of Success:

“What did they know?  They knew that human qualities, such as intellectual skills could be cultivated through effort.  And that’s what they were doing – getting smarter.  Not only weren’t they discouraged by failure, they didn’t even think they were failing.  They thought they were learning.”

Your Can Change Your IQ

Believe it or not, a big believer in the idea that you can use education and practice to fundamentally change your intelligence is Alfred Binet, the inventor of the IQ test.

Via Mindset: The New Psychology of Success:

“Binet, a Frenchman working in Paris in the early twentieth century, designed this test to identify children who were not profiting from the Paris public schools, so that new educational programs could be designed to get them back on track. Without denying individual differences in children’s intellects, he believed that education and practice could bring about fundamental changes in intelligence.”

Methods Make the Difference

Here is a quote from one of Binet’s major books,  Modern Ideas About Children:

"A few modern philosophers ... assert that an individual's intelligence is a fixed quantity, a quantity which cannot be increased.  We must protest  and react against this brutal pessimism ... With practice, training, and above all, method, we manage to increase our attention, our memory, our judgment and literally to become more intelligent than we were before."

Growth Mindset vs. Fixed Mindset

The difference that makes the difference in success and achievement is your mindset.  Specifically, a Growth Mindset is the key to unleashing and realizing your potential.

To fully appreciate what a Growth Mindset is, let’s contrast it by first understanding what a Fixed Mindset is.

According to Carol Dweck, a Fixed Mindset means that you fundamentally believe that intelligence and talent are fixed traits:

“In a fixed mindset, people believe their basic qualities, like their intelligence or talent, are simply fixed traits. They spend their time documenting their intelligence or talent instead of developing them. They also believe that talent alone creates success—without effort. They’re wrong.”

In contrast, according to Dweck, a Growth Mindset means that you fundamentally believe that you can develop your brains and talent:

“In a growth mindset, people believe that their most basic abilities can be developed through dedication and hard work—brains and talent are just the starting point. This view creates a love of learning and a resilience that is essential for great accomplishment. Virtually all great people have had these qualities.”

If you want to improve your motivation, set yourself up for success, and achieve more in life, then adopt and build a growth mindset.

Here are a few articles to help you get started:

3 Mindsets that Support You

5 Sources of Beliefs for Personal Excellence

6 Sources of Beliefs and Values

Growth Mindset Over Fixed Mindset

Training Mindset and Trusting Mindset

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For April 4th, 2014

Hey, it's HighScalability time:


The world ends not with a bang, but with 1 exaFLOP of bitcoin whimpers.
  • Quotable Quotes:
    • @EtienneRoy: Algorithm:  you must encode and leverage your ignorance, not only your knowledge #hadoopsummit - enthralling
    • Chris Brenny: A material is nothing without a process. While the constituent formulation imbues the final product with fundamental properties, the bridge between material and function has a dramatic effect on its perception and use.
    • @gallifreyan: Using AWS c1, m1, m2? @adrianco says don't. c3, m3, r3 are now better and cheaper. #cloudconnect #ccevent
    • @christianhern: Mobile banking in the UK: 1,800 transactions per MINUTE. A "seismic shift" that banks were unprepared for

  • While we are waiting for that epic article deeply comparing Google's Cloud with AWS, we have Adrian Cockcroft's highly hopped slide comparing the two. Google: no enterprise customers, no reservation options, need more regions and zones, need lower inter-zone latency, no SSD options. AWS: no per minute billing, need simpler discount options, need more regions and zones, no real PaaS strategy, not instance migration.
  • Technology has change us from a demo or die culture to a deploy or die culture. We see this in software, but it's happening in hardware too says, Joi Ito (MIT Media Lab), in this interview.
  • The Curmudgeon of Truth declares I reckon your message broker might be a bad idea: Engineering in practice is not a series of easy choices, but a series of tradeoffs. Brokers aren’t bad, but the tradeoffs you’re making might not be in your favour." < Good discussion on Hacker News.
  • High phive this. Your application may already achieved a degree of consciousness. An information integration theory of consciousness: consciousness corresponds to the capacity of a system to integrate information.
  • People really really like to talk. Line, a messaging app that's big in Japansurpasses 400 Million Registered Users, 10 billion chat messages per day, 1.8 billion stickers per day, and over 12 million calls per day.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge...

Categories: Architecture

Enterprise WinRT Apps Build Roundup

DevHawk - Harry Pierson - Fri, 04/04/2014 - 16:40

Wow, it’s been a whirlwind couple of days down here in San Francisco @ Build 2014. It has certainly been a huge thrill for me, getting a chance to be a part of the day one keynote and getting 15 minutes of fame. However, as the conference winds down I wanted to pull together a summary of the stuff Microsoft announced that relates to enterprise app development and Windows 8.1 Update. I mean, it wasn’t all about my wardrobe choices…

The Windows for Business blog as a good summary post that hits the highlights. The stuff I wanted to specifically call out is:

  • We’ve changed the policy to allow side loaded apps to communicate with desktop apps. Literally every single enterprise customer, Microsoft dev consultant and enterprise technical sales rep I’ve spoken to in the last year has asked for this.
  • We’ve added a feature in Windows 8.1 Update to enable side loaded apps to run code outside of the App Container. This opens up side loaded apps to access the full power of Windows as well as all the existing code the enterprise may have in its portfolio
  • We’ve made it significantly easier to get side load rights. I’d go thru the specifics here, but Rocky Lhotka (who has been *very* vocal about the issues in this space) had a great summary: “For a maximum of around $100 virtually every organization (small to huge) can get a side loading unlock key for all their devices.”

If you want more information on how to take advantage of these new features for side loaded apps, here are some resources for you:

  • In addition to my 5 minutes in the keynote, I did a whole session where I drilled into more details on that demo. I also demos that used network loopback for interprocess communication.
  • John Vintzel and Cliff Strom had a session on deploying enterprise apps. As of this writing, the video isn’t online yet but it will be within a day or two at that URL.
  • We have published whitepapers on both Brokered WinRT Components and using network loopback in WinRT apps that go into more details on how to build solutions with this technology
  • Last but not least, we have a set of samples of sideloaded WinRT apps. This includes the keynote demo, another brokered component demo and the WCF & ASP.NET network loopback demos I did in my session. Note, the keynote demo sample is packaged oddly because of the way MSDN’s sample repo handles (or in this case doesn’t handle) VS solutions with multiple projects. When I get back to Redmond, I’m  going to see if there’s a better way to get this sample hosted.

I heard many times over the past two days from folks in person at the conference and via email, twitter, facebook, carrier pigeon, etc just how excited they are about these changes & features. As an engineer who spends most of his days in his office and or in meetings building this stuff, it is amazingly gratifying to hear directly from our users how much our work can help them.

Categories: Architecture, Programming