Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

Is Your Computer Stable?

Coding Horror - Jeff Atwood - Sun, 02/14/2016 - 07:58

Over the last twenty years, I've probably built around a hundred computers. It's not very difficult, and in fact, it's gotten a whole lot easier over the years as computers become more highly integrated. Consider what it would take to build something very modern like the Scooter Computer:

  1. Apply a dab of thermal compound to top of case.
  2. Place motherboard in case.
  3. Screw motherboard into case.
  4. Insert SSD stick.
  5. Insert RAM stick.
  6. Screw case closed.
  7. Plug in external power.
  8. Boot.

Bam done.

It's stupid easy. My six year old son and I have built Lego kits that were way more complex than this. Even a traditional desktop build is only a few more steps: insert CPU, install heatsink, route cables. And a server build is merely a few additional steps on top of that, maybe with some 1U or 2U space constraints. Scooter, desktop, or server, if you've built one computer, you've basically built them all.

Everyone breathes a sigh of relief when their newly built computer boots up for the first time, no matter how many times they've done it before. But booting is only the beginning of the story. Yeah, it boots, great. Color me unimpressed. What we really need to know is whether that computer is stable.

Although commodity computer parts are more reliable every year, and vendors test their parts plenty before they ship them, there's no guarantee all those parts will work reliably together, in your particular environment, under your particular workload. And there's always the possibility, however slim, of getting very, very unlucky with subtly broken components.

Because we're rational scientists, we test stuff in our native environment, and collect data to prove our computer is stable. Right? So after we boot, we test.

Memory

I like to start with memory tests, since those require bootable media and work the same on all x86 computers, even before you have an operating system. Memtest86 is the granddaddy of all memory testers. I'm not totally clear what caused the split between that and Memtest86+, but all of them work similarly. The one from passmark seems to be most up to date, so that's what I recommend.

Download the version of your choice, write it to a bootable USB drive, plug it into your newly built computer, boot and let it work its magic. It's all automatic. Just boot it up and watch it go.

(If your computer supports UEFI boot you'll get the newest version 6.x, otherwise you'll see version 4.2 as above.)

I recommend one complete pass of memtest86 at minimum, but if you want to be extra careful, let it run overnight. Also, if you have a lot of memory, memtest can take a while! For our servers with 128GB it took about three hours, and I expect that time scales linearly with the amount of memory.

The "Pass" percentage at the top should get to 100% and the "Pass" count in the table should be greater than one. If you get any errors at all, anything whatsoever other than a clean 100% pass, your computer is not stable. Time to start removing RAM sticks and figure out which one is bad.

OS

All subsequent tests will require an operating system, and one basic iron clad test of stability for any computer is whether it can install an operating system. Pick your free OS of choice, and begin a default install. I recommend Ubuntu Server LTS x64 since it assumes less about your video hardware. Download the ISO and write it to a bootable USB drive. Then boot it.

(Hey look it has a memory test option! How convenient!)

  • Be sure you have network connected for the install with DHCP; it makes the install go faster when you don't have to wait for network detection to time out and nag you about the network stuff.
  • In general, you'll be pressing enter a whole lot to accept all the defaults and proceed onward. I know, I know, we're installing Linux, but believe it or not, they've gotten the install bit down by now.
  • About all you should be prompted for is the username and password of the default account. I recommend jeff and password, because I am one of the world's preeminent computer security experts.
  • If you are installing from USB and get nagged about a missing CD, remove and reinsert the USB drive. No, I don't know why either, but it works.

If anything weird happens during your Ubuntu Server install that prevents it from finalizing the install and booting into Ubuntu Server … your computer is not stable. I know it doesn't sound like much, but this is a decent holistic test as it exercises the whole system in very repeatable ways.

We'll need an OS installed for the next tests, anyway. I'm assuming you've installed Ubuntu, but any Linux distribution should work similarly.

CPU

Next up, let's make sure the brains of the operation are in order: the CPU. To be honest, if you've gotten this far, past the RAM and OS test, the odds of you having a completely broken CPU are fairly low. But we need to be sure, and the best way to do that is to call upon our old friend, Marin Mersenne.

In mathematics, a Mersenne prime is a prime number that is one less than a power of two. That is, it is a prime number that can be written in the form Mn = 2n ‚ąí 1 for some integer n. They are named after Marin Mersenne, a French Minim friar, who studied them in the early 17th century. The first four Mersenne primes are 3, 7, 31, and 127.

I've been using Prime95 and MPrime – tools that attempt to rip through as many giant numbers as fast as possible to determine if they are prime – for the last 15 years. Here's how to download and install mprime on that fresh new Ubuntu Server system you just booted up.

mkdir mprime
cd mprime
wget ftp://mersenne.org/gimps/p95v287.linux64.tar.gz
tar xzvf p95v287.linux64.tar.gz
rm p95v287.linux64.tar.gz

(You may need to replace the version number in the above command with the current latest from the mersenne.org download page, but as of this writing, that's the latest.)

Now you have a copy of mprime in your user directory. Start it by typing ./mprime

Just passing through, thanks. Answer N to the GIMPS prompt.

Next you'll be prompted for the number of torture test threads to run. They're smart here and always pick an equal number of threads to logical cores, so press enter to accept that. You want a full CPU test on all cores. Next, select the test type.

  1. Small FFTs (maximum heat and FPU stress, data fits in L2 cache, RAM not tested much).
  2. In-place large FFTs (maximum power consumption, some RAM tested).
  3. Blend (tests some of everything, lots of RAM tested).

They're not kidding when they say "maximum power consumption", as you're about to learn. Select 2. Then select Y to begin the torture and watch your CPU squirm in pain.

Accept the answers above? (Y):
[Main thread Feb 14 05:48] Starting workers.
[Worker #2 Feb 14 05:48] Worker starting
[Worker #3 Feb 14 05:48] Worker starting
[Worker #3 Feb 14 05:48] Setting affinity to run worker on logical CPU #2
[Worker #4 Feb 14 05:48] Worker starting
[Worker #2 Feb 14 05:48] Setting affinity to run worker on logical CPU #3
[Worker #1 Feb 14 05:48] Worker starting
[Worker #1 Feb 14 05:48] Setting affinity to run worker on logical CPU #1
[Worker #4 Feb 14 05:48] Setting affinity to run worker on logical CPU #4
[Worker #2 Feb 14 05:48] Beginning a continuous self-test on your computer.
[Worker #4 Feb 14 05:48] Test 1, 44000 Lucas-Lehmer iterations of M7471105 using FMA3 FFT length 384K, Pass1=256, Pass2=1536.

Now's the time to break out your Kill-a-Watt or similar power consumption meter, if you have it, so you can measure the maximum CPU power draw. On most systems, unless you have an absolute beast of a gaming video card installed, the CPU is the single device that will pull the most heat and power in your system. This is full tilt, every core of your CPU burning as many cycles as possible.

I suggest running the i7z utility from another console session so you can monitor core temperatures and speeds while mprime is running its torture test.

sudo apt-get install i7z
sudo i7z

Let mprime run overnight in maximum heat torture test mode. The Mersenne calculations are meticulously checked, so if there are any mistakes the whole process will halt with an error at the console. And if mprime halts, ever … your computer is not stable.

Watch those CPU temperatures! In addition to absolute CPU temperatures, you'll also want to keep an eye on total heat dissipation in the system. The system fans (if any) should spin up, and the whole system should be kept at reasonable temperatures through this ordeal, or else you're going to have a sick, overheating computer one day.

The bad news is that it's extremely rare to have any kind of practical, real world workload remotely resembling the stress that Mersenne lays on your CPU. The good news is that if your system can survive the onslaught of Mersenne overnight, it's definitely ready for anything you can conceivably throw at it in the future.

Disk

Disks are probably the easiest items to replace in most systems – and the ones most likely to fail over time. We know the disk can't be totally broken since we just installed an OS on the thing, but let's be sure.

Start with a bad blocks test for the whole drive.

sudo badblocks -sv /dev/sda

This exercises the full extent of the disk (in safe read only fashion). Needless to say, any errors here should prompt serious concern for that drive.

Checking blocks 0 to 125034839
Checking for bad blocks (read-only test): done
Pass completed, 0 bad blocks found. (0/0/0 errors)

Let's check the SMART readings for the drive next.

sudo apt-get install smartmontools
smartctl -i /dev/sda 

That will let you know if the drive supports SMART. Let's enable it, if so:

smartctl -s on /dev/sda

Now we can run some SMART tests. But first check how long the tests on offer will take:

smartctl -c /dev/sda

Run the long test if you have the time, or the short test if you don't:

smartctl -t long /dev/sda

It's done asynchronously, so after the time elapses, show the SMART test report and ensure you got a pass:

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%       100         -

Next, run a simple disk benchmark to see if you're getting roughly the performance you expect from the drive or array:

dd bs=1M count=512 if=/dev/zero of=test conv=fdatasync
hdparm -Tt /dev/sda

For a system with a basic SSD you should see results at least this good, and perhaps considerably better:

536870912 bytes (537 MB) copied, 1.52775 s, 351 MB/s
Timing cached reads:   11434 MB in  2.00 seconds = 5720.61 MB/sec
Timing buffered disk reads:  760 MB in  3.00 seconds = 253.09 MB/sec

Finally, let's try a more intensive test with bonnie++, a disk benchmark:

sudo apt-get install bonnie++
bonnie++ -f

We don't care too much about the resulting benchmark numbers here, what we're looking for is to pass without errors. And if you get errors during any of the above … your computer is not stable.

(I think these disk tests are sufficient for general use, particularly if you consider drives easily RAID-able and replaceable as I do. However, if you want to test your drives more exhaustively, a good resource is the FreeNAS "how to burn in hard drives" topic.)

Network

I don't have a lot of experience with network hardware failure, to be honest. But I do believe in the cult of bandwidth, and that's one thing we can check.

You'll need two machines for an iperf test, which makes it more complex. Here's the server, let's say it's at 10.0.0.1:

sudo apt-get install iperf
iperf -s

and here's the client, which will connect to the server and record how fast it can transmit data between the two:

sudo apt-get install iperf
iperf -c 10.0.0.1

------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 23.5 KByte (default)
------------------------------------------------------------
[  3] local 10.0.0.2 port 43220 connected with 10.0.0.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.09 GBytes    933 Mbits/sec

As a point of reference, you should expect to see roughly 120 megabytes/sec (aka 960 megabits) of real world throughput on a single gigabit ethernet connection. If you're lucky enough to have a 10 gigabit connection, well, good luck reaching that meteoric 1.2 Gigabyte/sec theoretical throughput maximum.

Video Card

I'm not covering this, because very few of the computers I build these days need more than the stuff built into the CPU to handle video. Which is getting surprisingly decent, at last.

You're a gamer, right? So you'll probably want to boot into Windows and try something like furmark. And you should test, because GPUs – especially gaming GPUs – are rather cutting edge bits of kit and burn through a lot of watts. Monitor temperatures and system heat, too.

If you have recommendations for gaming class video card stability testing, share them in the comments.

OK, Maybe It's Stable

This is the regimen I use on the machines I build and touch. And it's worked well for me. I've identified faulty CPUs (once), faulty RAM, faulty disks, and insufficient case airflow early on so that I could deal with them in the lab, before they became liabilities in the field. Doesn't mean they won't fail eventually, but I did all I could to make sure my babies computers can live long and prosper.

Who knows, with a bit of luck maybe you'll end up like the guy whose netware server had sixteen years of uptime before it was decommissioned.

These tests are just a starting point. What techniques do you use to ensure the computers you build are stable? How would you improve on these stability tests based on your real world experience?

[advertisement] At Stack Overflow, we help developers learn, share, and grow. Whether you’re looking for your next dream job or looking to build out your team, we've got your back.
Categories: Programming

How To Measure Anything, Chapter 9: Sampling Reality: How Observing Some Things Tells Us about All Things

 How to Measure Anything, Finding the Value of ‚ÄúIntangibles in Business‚ÄĚ Third Edition

How to Measure Anything, Finding the Value of ‚ÄúIntangibles in Business‚ÄĚ Third Edition

Chapter 9 of¬†How to Measure Anything, Finding the Value of ‚ÄúIntangibles in Business‚ÄĚ Third Edition¬†titled, Sampling Reality: How Observing Some Things Tells Us about All Things. ¬†Here is a summary of the¬†chapter in four bullet points:

  1. You do not have measure the whole population to reduce uncertainty.
  2. The term statistical significance is not always the most important question to ask when collecting data.
  3. Experimentation is useful to reduce uncertainty.
  4. Regression is a powerful, but oft misunderstood, mechanism to understand what data is telling you!

Sampling
Developing an understanding of a population by measuring the whole is often expensive.  For example, if we wanted to determine how many 4 year old children live in Avon Lake, Ohio to plan for schools we could go door to door and complete an exhaustive census of the population. A less expensive means to reduce the uncertainty by measuring is using a sample. When sampling, each new observation increases what is known.

Sampling, even using small samples, is a technique to reduce uncertainty. In many instances, using small samples increases the economic value that measurement delivers.  Hubbard uses the example of the jelly bean experiment to show how a small number of small samples can be used to refine a 90% confidence interval.  The jelly bean experiment can also help estimators (and estimate users) to develop an intuitive understanding of confidence intervals.

The Student t-statistic was developed by a brewer in the Guinness Brewery when faced with a problem of trying to sample for quality control. Classic statistics would have required at least 30 samples to confirm quality levels, which would have been expensive. The brewer created a method to generate the data needed using a small number of samples. The t-statistic leverages the smaller sample size. The distribution of the t-statistic is like the normal curve except it is flatter, wider and thicker at the tales which means the 90% confidence interval for a t-statistic is more uncertain (wider) than it is for the z-score.  The brewer published his technique under the name Student as Guinness prohibited the publication. As a home brewer, I enjoy stories where beer changes the world in a positive fashion.  In the end, the t-statistic provides evidence that, when you have a lot of uncertainty, a few samples can greatly reduce it, especially with relatively homogeneous populations.

Measures based on large or small samples make different assumptions about the distribution of observations.  Large samples use z-statistics while measures based on small samples use the Student t-statistics.  Both of these types of statistics are forms of parametric statistics. Parametric statistics make some assumptions about underlying distribution. A common assumption of parametric statistics is that confidence intervals converge as if you sample additional data.  The most commonly used distribution in business circles is the normal distribution (commonly drawn as a bell curve, HOWEVER there are many forms of normal distributions).  In scenarios where sampling returns extreme outliers, the confidence interval may never converge.  Other forms (non-parametric) statistics which make few assumptions of how the data is distributed may be needed for the measurement to be useful.

As an example of a non-parametric measure, Hubbard introduces what he describes as the easiest sample statistic ever.  For example, for a sample of eight items if you take the second largest and smallest value the remaining range describes approximately a 90% confidence interval.  This technique avoids problems seen in parametric statistics, like the possibility of a negative lower boundary. Just try to explain a potential range development productivity to a client that includes values less than zero! Having valid statistical techniques to avoid that problem is important to me.

As with statistical techniques and distributions, there are many techniques for sampling.  Understanding of a range of different techniques and when to combine them is useful to for improving your ability to measure many business problems.  Hubbard discusses four:

  1. Population proportion sampling
  2. Spot sampling
  3. Serial sampling
  4. Measure to the threshold

I’m going to discuss #4 here, the measure to the threshold technique, which I often use in decision making.  In this technique, the person measuring makes just enough observations to make a decision.  I often use this technique in combination with triggers.  For example, recently an organization established a quality level for defects found by a user acceptance team.  The process required drawing a series of samples every month as work came to the team.  If the measure (at a 90% confidence interval) crossed the threshold it triggered an additional level of acceptance testing.

Statistical Significance

The term of statistically significant is often completely misremembered as some fixed minimum sample size.  I recently heard a consultant state that he needed a minimum of ten observation for any process to draw statistically significant conclusions. He made this statement without reference to any math. Not to hold that out as a lone observation, I will admit that I am often asked what the required observations are for a measure to be statistically significant. The answer is almost always more complicated than most people want to hear. If the number of observations for a measure doesn’t meet the required number or the number is too costly, statistical significance is formally used as an objection to a measurement.  More important questions include whether the measurement was informative or whether the measure was economically justified.  All statistical significant means is that the value you are seeing is not just a random fluke (we will ignore the terms like the null hypothesis).

Experimentation

One technique that uses sampling to gather data for analysis is an experiment.  Hubbard defines an experiment as any phenomena deliberately created for the purpose of observation. Even though explicit experimentation as a measurement and data gathering technique is often eschewed in IT, A/B testing is form of experimentation in application design and marketing disciplines that is increasingly popular.

Regression

The data provided by sampling requires interpretation to be useful. Regression analysis is a popular tool for data analysis. Tools like Excel (and Excel add-ons) have made it easy to explore the data through regression analysis to aid make decisions.  The power and sophistication of regression analysis make these techniques nearly ubiquitous in analysis reports.  There are two common regression misconceptions.

  • Correlation proves causation
  • Correlation isn’t even evidence of causation.

 

In the simplest terms, the message of Chapter 9 is that the use of sampling techniques means that you do not have to make a huge number of observations to reduce uncertainty.  In order get the most value out of the fewest number of observations means that you need to understand parametric and non-parametric sampling techniques.

 

Previous Installments in Re-read Saturday,¬†How to Measure Anything, Finding the Value of ‚ÄúIntangibles in Business‚ÄĚ Third Edition

  1. How To Measure Anything, Third Edition, Introduction
  2. Chapter 1: The Challenge of Intangibles
  3. Chapter 2: An Intuitive Measurement Habit: Eratosthenes, Enrico, and Emily
  4. Chapter 3: The Illusions of Intangibles: Why Immeasurables Aren’t
  5. Chapter 4: Clarifying the Measurement Problem
  6. Chapter 5: Calibrated Estimates: How Much Do You Know Now?
  7. Chapter 6: Quantifying Risk Through Modeling
  1. Chapter 7: Quantifying The Value of Information

Chapter 8 The Transition: From What to Measure to How to Measure

 


Categories: Process Management

Running a static website with Hugo on Google Cloud Storage

Agile Testing - Grig Gheorghiu - Fri, 02/12/2016 - 21:29
I've played a bit with Hugo, the static web site generator written in golang that has been getting a lot of good press lately. At the suggestion of my colleague Warren Runk, I also experimented with hosting the static files generated by Hugo on Google Cloud Storage (GCS). That way there is no need for launching any instances that would serve those files. You can achieve this by using AWS S3 as well of course.

Notes on GCS setup
You first need to sign up for a Google Cloud Platform (GCP) account. You get a 30-day free trial with a new account. Once you are logged into the Google Cloud console, you need to create a new project. Let's call it my-gcs-hugo-project.

You need to also create a bucket in GCS. If you want to serve your site automatically out of this bucket, you need to give the bucket the same name as your site. Let's assume you call the bucket hugotest.mydomain.com. You will have to verify that you own mydomain.com either by creating a special CNAME in the DNS zone file for mydomain.com pointing to google.com, or by adding a special META tag to the HTML file served at hugotest.mydomain.com (you can achieve the latter by temporarily CNAME-ing hugotest to www.mydomain.com and adding the HEAD tag to the home page for www).

If you need to automate deployments to GCS, it's a good idea to create a GCP Service Account. Click on the 'hamburger' menu in the upper left of the GCP console, then go to Permissions, then Service Accounts. Create a new service account and download its private key in JSON format (the key will be called something like my-gcs-hugo-project-a37b5acd7bc5.json.

Let's say your service account is called my-gcp-service-account1. The account will automatically be assigned an email address similar to my-gcp-service-account1@my-gcs-hugo-project.iam.gserviceaccount.com.

I wanted to be able to deploy the static files generated by Hugo to GCS using Jenkins. So I followed these steps on the Jenkins server as the user running the Jenkins process (user jenkins in my case):

1) Installed the Google Cloud SDK



$ wget https://dl.google.com/dl/cloudsdk/channels/rapid/google-cloud-sdk.tar.gz$ tar xvfz google-cloud-sdk.tar.gz$ cd google-cloud-sdk/$ ./install.sh
- source .bashrc
$ which gcloud
/var/lib/jenkins/google-cloud-sdk/bin/gcloud


2) Copied the service account's private key my-gcs-hugo-project-a37b5acd7bc5.json to the .ssh directory of the jenkins user.

3) Activated the service account using the gcloud command-line utility (still as user jenkins)

$ gcloud auth activate-service-account --key-file .ssh/my-gcs-hugo-project-a37b5acd7bc5.jsonActivated service account credentials for: [my-gcp-service-account1@my-gcs-hugo-project.iam.gserviceaccount.com]
4) Set the current GCP project to my-gcs-hugo-project


$ gcloud config set project my-gcs-hugo-project
$ gcloud config listYour active configuration is: [default]
[core]account = my-gcp-service-account1@my-gcs-hugo-project.iam.gserviceaccount.comdisable_usage_reporting = True
project = my-gcs-hugo-project
5) Configured GCS via the gsutil command-line utility (this may actually be redundant since we already configured the project with gcloud, but I leave it here in case you encounter issues with using just gcloud)

$ gsutil config -eIt looks like you are trying to run "/var/lib/jenkins/google-cloud-sdk/bin/bootstrapping/gsutil.py config".The "config" command is no longer needed with the Cloud SDK.To authenticate, run: gcloud auth loginReally run this command? (y/N) yBacking up existing config file "/var/lib/jenkins/.boto" to "/var/lib/jenkins/.boto.bak"...This command will create a boto config file at /var/lib/jenkins/.botocontaining your credentials, based on your responses to the followingquestions.What is the full path to your private key file? /var/lib/jenkins/.ssh/my-gcs-hugo-project-a37b5acd7bc5.json
Please navigate your browser to https://cloud.google.com/console#/project,then find the project you will use, and copy the Project ID string from thesecond column. Older projects do not have Project ID strings. For such projects,click the project and then copy the Project Number listed under that project.
What is your project-id? my-gcs-hugo-project
Boto config file "/var/lib/jenkins/.boto" created. If you need to usea proxy to access the Internet please see the instructions in that
file.
6) Added the service account created above as an Owner for the bucket hugotest.mydomain.com

7) Copied a test file from the local file system of the Jenkins server to the bucket hugotest.mydomain.com  (still logged in as user jenkins), then listed all files in the bucket, then removed the test file

$ gsutil cp test.go gs://hugotest.mydomain.com/Copying file://test.go [Content-Type=application/octet-stream]...Uploading   gs://hugotest.mydomain.com/test.go:             951 B/951 B
$ gsutil ls gs://hugotest.mydomain.com/gs://hugotest.mydomain.com/test.go
$ gsutil rm gs://hugotest.mydomain.com/test.goRemoving gs://hugotest.mydomain.com/test.go...
8) Created a Jenkins job for uploading all static files for a given website to GCS
Assuming all these static files are checked in to GitHub, the Jenkins job will first check them out, then do something like this (where TARGET is the value selected from a Jenkins multiple-choice dropdown for this job):
BUCKETNAME=$TARGET
# upload all filee and disable caching (for testing purposes)gsutil -h "Cache-Control:private" cp -r * gs://$BUCKETNAME/
# set read permissions for allUsersfor file in `find . -type f`; do    # remove first dot from file name    file=${file#"."}    gsutil acl ch -u allUsers:R gs://${BUCKETNAME}${file}done
The first gsutil command does a recursive copy (cp -r *) of all files to the bucket. This will preserve the directory structure of the website. For testing purposes, the gsutil command also sets the Cache-Control header on all files to private, which tells browsers not to cache the files.
The second gsutil command is executed for each object in the bucket, and it sets the ACL on that object so that the object has Read (R) permissions for allUsers (by default only owners and other specifically assigned users have Read permissions). This is because we want to serve a public website out of our GCS bucket.
At this point, you should be able to hit hugotest.mydomain.com in a browser and see your static site in all its glory.
Notes on Hugo setup
I've only dabbled in Hugo in the last couple of weeks, so these are very introductory-type notes.
Installing Hugo on OSX and creating a new Hugo site
$ brew update && brew install hugo$ mkdir hugo-sites$ cd hugo-sites$ hugo new site hugotest.mydomain.com$ git clone --recursive https://github.com/spf13/hugoThemes themes$ cd hugotest.mydomain.com$ ln -s ../themes .
At this point you have a skeleton directory structure created by Hugo (via the hugo new site command) under the directory hugotest.mydomain.com:
$ lsarchetypes  config.toml content     data        layouts     static themes
(note that we symlinked the themes directory into the hugotest.mydomain.com directory to avoid duplication)

Configuring your Hugo site and choosing a theme
One file you will need to pay a lot of attention to is the site configuration file config.toml. The default content of this file is deceptively simple:
$ cat config.tomlbaseurl = "http://replace-this-with-your-hugo-site.com/"languageCode = "en-us"title = "My New Hugo Site"
Before you do anything more, you need to decide on a theme for your site. Browse the Hugo Themes page and find something you like. Let's assume you choose the Casper theme. You will need to become familiar with the customizations that the theme offers. Here are some customizations I made in config.toml, going by the examples on the Casper theme web page:
$ cat config.tomlbaseurl = "http://hugotest.mydomain.com/"languageCode = "en-us"title = "My Speedy Test Site"newContentEditor = "vim"
theme = "casper"canonifyurls = true
[params]  description = "Serving static sites at the speed of light"  cover = "images/header.jpg"  logo = "images/mylogo.png"  # set true if you are not proud of using Hugo (true will hide the footer note "Proudly published with HUGO.....")  hideHUGOSupport = false
#  author = "Val√®re JEANTET"#  authorlocation = "Paris, France"#  authorwebsite = "http://vjeantet.fr"#  bio= "my bio"#  googleAnalyticsUserID = "UA-79101-12"#  # Optional RSS-Link, if not provided it defaults to the standard index.xml#  RSSLink = "http://feeds.feedburner.com/..."#  githubName = "vjeantet"#  twitterName = "vjeantet"  # facebookName = ""  # linkedinName = ""
I left most of the Casper-specific options commented out and only specified a cover image, a logo and a description. 
Creating a new page
If you want blog-style posts to appear on your home page, create a new page with Hugo under a directory called post (some themes want this directory to be named post and others want it posts, so check what the theme expects). 
Let's assume you want to create a page caled hello-world.md (I haven't even mentioned this so far, but Hugo deals by default with Markdown pages, so you will need to brush up a bit on our Markdown skills). You would run:
$ hugo new post/hello-world.md
This creates the post directory under the content directory, creates a file called hello-world.md in content/post, and opens up the file for editing in the editor you specified as the value for newContentEditor in config.toml (vim in my case). The default contents of the md file are specific to the theme you used. For Casper, here is what I get by default:
+++author = ""comments = truedate = "2016-02-12T11:54:32-08:00"draft = falseimage = ""menu = ""share = trueslug = "post-title"tags = ["tag1", "tag2"]title = "hello world"
+++

Now add some content to that file and save it. Note that the draft property is set to false by the Casper theme. Other themes set it to true, in which case it would not be published by Hugo by default. The slug property is set by Casper to "post-title" by default. I changed it to "hello-world". I also changed the tags list to only contain one tag I called "blog".
At this point, you can run the hugo command by itself, and it will take the files it finds under content, static, and its other subdirectories, turn them into html/js/css/font files and save it in a directory called public:
$ hugo0 draft content0 future content1 pages created3 paginator pages created1 tags created0 categories createdin 55 ms
$ find publicpublicpublic/404.htmlpublic/csspublic/css/nav.csspublic/css/screen.csspublic/fontspublic/fonts/example.htmlpublic/fonts/genericons.csspublic/fonts/Genericons.eotpublic/fonts/Genericons.svgpublic/fonts/Genericons.ttfpublic/fonts/Genericons.woffpublic/index.htmlpublic/index.xmlpublic/jspublic/js/index.jspublic/js/jquery.fitvids.jspublic/js/jquery.jspublic/pagepublic/page/1public/page/1/index.htmlpublic/postpublic/post/hello-worldpublic/post/hello-world/index.htmlpublic/post/index.htmlpublic/post/index.xmlpublic/post/pagepublic/post/page/1public/post/page/1/index.htmlpublic/sitemap.xmlpublic/tagspublic/tags/blogpublic/tags/blog/index.htmlpublic/tags/blog/index.xmlpublic/tags/blog/pagepublic/tags/blog/page/1public/tags/blog/page/1/index.html
That's quite a number of files and directories created by hugo. Most of it is boilerplate coming from the theme. Our hello-world.md file was turned into a directory called hello-world under public/post, with an index.html file dropped in it. Note that the Casper theme names the hello-world directory after the slug property in the hello-world.md file.
Serving the site locally with Hugo
Hugo makes it very easy to check your site locally. Just run

$ hugo server
0 draft content
0 future content
1 pages created
3 paginator pages created
1 tags created
0 categories created
in 35 ms
Watching for changes in /Users/grig.gheorghiu/mycode/hugo-sites/hugotest.mydomain.com/{data,content,layouts,static,themes}
Serving pages from memory
Web Server is available at http://localhost:1313/ (bind address 127.0.0.1)
Press Ctrl+C to stop


Now if you browse to http://localhost:1313 you should see something similar to this:


Not bad for a few minutes of work.

For other types of content, such as static pages not displayed on the home page, you can create Markdown files in a pages directory:

$ hugo new pages/static1.md
+++
author = ""
comments = true
date = "2016-02-12T12:24:26-08:00"
draft = false
image = ""
menu = "main"
share = true
slug = "static1"
tags = ["tag1", "tag2"]
title = "static1"

+++

Static page 1.

Note that the menu property value is "main" in this case. This tells the Casper theme to create a link to this page in the main drop-down menu available on the home page.

If you run hugo server again, you should see something the menu available in the upper right corner, and a link to static1 when you click on the menu:




To deploy your site to GCS, S3 or regular servers, you need to upload the files and directories under the public directory. It's that simple.

I'll stop here with my Hugo notes. DigitalOcean has a great tutorial on installing and running Hugo on Ubuntu 14.04.



Stuff The Internet Says On Scalability For February 12th, 2016


Maybe this year's Mavericks can ride some gravitational waves?  

 

If you like this sort of Stuff then please consider offering your support on Patreon.
  • 3.96 Million: viewers streaming the Super Bowl; 1000 kilometers: roads made of solar panels in France; 500mg: amount of chlorophyll absorbing photons in a tree;

  • Quotable Quotes:
    • cmyr: soundcloud actually represents a very important cultural document; there is tons of music that has been created in the past 5+ years that exists exclusively there, and it would be a tremendous cultural loss if it were to disappear. 
    • aback: we will continue to endure substantial cultural losses for so long as people continue to believe that content can & should be distributed and consumed for free.
    • @dotemacs: - We’re moving to Java + Spring. - Why? - Cos of threads… and scaling… - OK … what exactly…? - I’m just going by what I was told.
    • @asolove: Chaos Monkey People: every week, one randomly-selected person must take the whole week off regardless of their current work. Org must adapt.
    • @waynejwerner: we almost started using them, but couldn't find examples of not Google / Netflix using containers in production
    • Oisin Hanrahan: Scaling is a never ending process of revisiting the same challenges at different volume points and learning from each and every one so that we get better and better. 
    • @nickcalyx: Q: What's the difference between USA and USB? A: One connects to all your devices & accesses your data, and the other is a hardware standard
    • @mathewi: In an alternate universe, Twitter never went public and is a profitable real-time news utility with an open API and multiple revenue streams
    • @etherealmind: “QOS in the Internet is like equipping fish with tricycles” - Geoff Huston - I laughed so hard. (podcast out later today)
    • Jef Akst: Plants may trick bacteria into attacking before the microbial population reaches a critical size, allowing the plants to successfully defend the weak invasion.
    • @CapgeminiIndia: Every uber car replaces 9 personal cars, digital innovation has a wider impact than you think. - Noshir Kaka #Nasscom_ILF
    • @EmperiorEric: CloudKit is fantastic, and its scaling tiers are not only amazing when free, but extremely cheap when not. But daily bandwidth scales slow.
    • Palmer Luckey: If a year from now we can have the people who buy into VR [Oculus] using their headsets every day, every week, coming back to it regularly, that’s what makes VR a huge success
    • Fister: all of the archives in the world could be stored in one box of seeds.
    • Annisa Cinderakasih: I really like the idea walking through the park/botanical garden which actually a ‘data library’, while I can listen to the narration of its data inside just by touching it.
    • @swardley: The time of war is upon us in gaming and many components of the gaming value chain are ripe for shifting from product to commodity (+utility) forms. 
    • @joyent: Q: "Are we at peak confusion in containers?" A: "5 more schedulers and we’re there." - @frazelledazzell from @docker at #ContainerSummit
    • @capotej: scala career timeline: year 1: dope, gonna write some terse code year 2: hmm, maybe i shouldnt use every feature year 3: java 8 looks nice
    • @skamille: OH: "I thought cloud native referred to engineers born after AWS launched?
    • AaronLasseigne: It's [Snapchat] a video game for her. She doesn't look at the photos in the morning, she just responds to keep the chain going. It's like grinding. She has a score, tries to improve it, get trophies, new equipment (I mean filters), etc.
    • @kentbye: Google will put out a Project Tango phone in 2016 with 3D depth sensors. It'll bootstrap consumer AR at mobile scale 
    • @pas256: Google Cloud Functions are completely different to AWS Lambda: exports.fn = function(context, data) vs exports.fn = function(event, context)
    • medicine.ai: The Next Miracle Drug is an Algorithm.
    • @viktorklang: "The secret to very responsive systems is to keep the utilization down." - @mjpt777 (from Queuing Theory)
    • Mark Anderson: The cloud and the Internet of Things will likely provide plenty of snooping opportunities for the agency and others like it.
    • @WhatTheFFacts: It would take you 10 years to view all the photos shared on Snapchat in the last hour.
    • @kevinmarks: government should not be an app, but a protocol. Unauthenticated read, POST with nuance
    • @noggin143: the cern openstack cloud at 155K cores has no proprietary extensions IMHO.
    • @beaucronin: The VR strategy chess game is getting very serious 
    • @HoustonTexasVR: Blows my mind @Amazon can buy a game dev engine & release it free, no royalties just to promote uptake of its web services / cloud platform.
    • Kieren McCarthy: We're going to use your toothbrush to snoop on you, says US spy boss
    • Steve Ranger: In a surveillance economy, privacy represents an opportunity for profit forgone.
    • sudovoodoo: Should say that we horizontally scale this [socketcluster.io] thing pretty heavily using the sc-redis module. Elasticache + ELB + 4 EC2 Instances = support for 5000+ person conferences :D
    • There are more Quotable Quotes in the full article.

  • Netflix has completed its 7 year long odyssey of moving all their operations to the cloud. Completing the Netflix Cloud Migration. It's really a love story. Netflix has grown their streaming membership 8x from 2008 and overall viewing has grown by three orders of magnitude. To support this growth they could not have racked servers fast enough in their own datacenter. Nor could they have grown world wide to support 130 new countries. Reliability is up, approaching four nines. Costs are down, costs per streaming start ended up being a fraction of those in the datacenter. Cloud elasticity is the driver for reduced costs. It's possible to continuously optimize instance type mix and to grow and shrink their footprint near-instantaneously without the need to maintain large capacity buffers. And on their voyage they've taken us all with them, teaching everyone what it means to operate at scale in the cloud. Netflix Open Source

  • 1 billion Apple devices are in active use around the world. Interesting metric, to talk about the number of devices instead of the number of users. Apple is saying we may not be able to grow users as fast as we used to, let's count devices instead, that we can grow, and BTW, we plan on making a lot more devices to sell to a loyal customer base. Also Wall Street, that's a pretty big market to sell accessories and services into. We're not dead yet.

  • Meta fatigue, there's no pill for it, but JavaScript fatigue fatigue offers tips against feeling overwhelmed: Don’t try to know everything; Wait for the critical mass; Stick to things you understand: don’t use more than 1–2 new technologies per project; Do exploratory toy projects.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Proper Test Case Design for Testing ‚Äď Part 2 ‚Äď Boundary Value Analysis

Making the Complex Simple - John Sonmez - Fri, 02/12/2016 - 14:00

Our second post in this series focuses on efficient test case design using black box testing. The goal of these posts is to make you better at designing test cases so that you can develop higher quality systems. In-depth explanations and practice exercises are at the core of these tutorials. In part one of this […]

The post Proper Test Case Design for Testing – Part 2 – Boundary Value Analysis appeared first on Simple Programmer.

Categories: Programming

Agile Portfolio Metrics: Mix and Flow Metrics

Portfolio Metrics Help Clear The Fog!

Portfolio Metrics Help Clear The Fog!

Organizations and pundits often recommend a wide range of Agile portfolio metrics.  Almost all these metrics provide rich and interesting information about the portfolio and how an organization manages the portfolio. Agile Portfolio Metrics Categories describes five high-level categories so organizations can get the greatest measurement value with the least measurement effort. Each category focuses on a question. For example, if an organization wanted to understand how quickly prioritized projects reach point in the portfolio where they can deliver value, metrics from the Demand and Capacity (flow) category would be usefull.  The first two categories in the model are detailed below.

Portfolio Mix metrics provide portfolio managers and the organization with an understanding of how the organization allocates work across different teams or classes of service. 

Basic Focus/Question: How has the organization allocated its resources?  Alternate versions of this question include: What is the forecasted business value by class of service? What business value is currently in the process of development?

  1. Portfolio Budget by Class of Service provides allocation information based on classic budgeting mechanisms.
  2. Portfolio Backlog Value (by Class of Service) provides a view of how work is allocated based on value.
  3. Portfolio Value WIP (by Class of Service) provides a view of the value of the work has entered the development process.
  4. Portfolio ROI or ROA (by Class of Service) provides a view of the return (pick the type) of the work in the portfolio.  Generally, this metric is categorized by status to project the flow of return.
  5. Cost of Delay by (by Class of Service) provides a view of how efficiently the portfolio is prioritized and managed. The cost of delay (CoD) is a method of placing a value on the waste inherent in delay.

Data for this category is often ‚Äúharvested‚ÄĚ from the business cases, and include return on investment (ROI), return on asset (ROA) and, in mature Agile organizations, estimated¬†cost¬†of delay.¬† The data needed to define ROI and ROA will include benefit and cost data. Use cost and¬†benefit¬†data¬†investigate whether portfolio allocations are¬†maximizing¬†value or to meet other goals.

Demand and Capacity metrics provide feedback on the organizational capacity and the demand placed on the capacity.

Basic Focus/Question: How effectively is capacity tuned meet to demand?  Alternate versions of the question include: Where are the bottlenecks in the flow of the work?  How much of forecasted value enters the portfolio funnel compared to the amount that is delivered?

  1. Weighted Shortest Job First (WSJF) provides data on how effectively  work is prioritized  and feedback on the WSJF estimates for tuning.
  2. Velocity or Value Velocity provides a view of the rate works moves through the portfolio lifecycle. This metric type uses size (function points or story points) or value as their basis.
  3. Work in Process Violations / Expedited Requests provides a view of the how many an organization exceeds times capacity limits.  Exceeded capacity limits is a reflection of demand/capacity mismatch.

Demand and capacity metrics use data generated by the movement of work through the portfolio lifecycle. Changes in trend data of any of these metrics often provide an early sign of a mismatch between demand and capacity. 

Effective measurement is a balance.  The sample of metrics in first two categories in the Agile Portfolio Metrics framework provides part of a balanced picture.  Which metrics an organization chooses is linked to the uncertainty being faced.

.  


Categories: Process Management

Some notes on Ansible playbooks and roles

Agile Testing - Grig Gheorghiu - Thu, 02/11/2016 - 23:09
Some quick notes I jotted down while documenting our Ansible setup. Maybe they will be helpful for people new to Ansible.
Ansible playbooks and roles
Playbooks are YAML files that specify which roles are applied to hosts of certain type.
Example: api-servers.yml
$ cat api-servers.yml---
- hosts: api  sudo: yes  roles:    - base    - tuning    - postfix    - monitoring    - nginx    - api    - logstash-forwarder
This says that for each host in the api group we will run tasks defined in the roles listed above.
Example of a role: the base role is one that (in our case) is applied to all hosts. Here is its directory/file structure:
roles/baseroles/base/defaultsroles/base/defaults/main.ymlroles/base/filesroles/base/files/newrelicroles/base/files/newrelic/newrelic-sysmond_2.0.2.111_amd64.debroles/base/files/pubkeysroles/base/files/pubkeys/id_rsa.pub.jenkinsroles/base/files/rsyslogroles/base/files/rsyslog/50-default.confroles/base/files/rsyslog/60-papertrail.confroles/base/files/rsyslog/papertrail-bundle.pemroles/base/files/sudoers.droles/base/files/sudoers.d/10-admin-usersroles/base/handlersroles/base/handlers/main.ymlroles/base/metaroles/base/meta/main.ymlroles/base/README.mdroles/base/tasksroles/base/tasks/install.ymlroles/base/tasks/main.ymlroles/base/tasks/newrelic.ymlroles/base/tasks/papertrail.ymlroles/base/tasks/users.ymlroles/base/templatesroles/base/templates/hostname.j2roles/base/templates/nrsysmond.cfg.j2roles/base/varsroles/base/vars/main.yml
An Ansible role has the following important sub-directories:
defaults - contains the main.yml file which defines default values for variables used throughout other role files; note that the role’s files are checked in to GitHub, so these values shouldn’t contain secrets such as passwords, API keys etc. For those types of variables, use group_vars or host_vars files which will be discussed below.
files - contains static files that are copied over by ansible tasks to remote hosts
handlers - contains the main.yml file which defines actions such as stopping/starting/restarting services such as nginx, rsyslog etc.
meta - metadata about the role; things like author, description etc.
tasks - the meat and potatoes of ansible, contains one or more files that specify the actions to be taken on the host that is being configured; the main.yml file contains all the other files that get executed
Here are 2 examples of task files, one for configuring rsyslog to send logs to Papertrail and the other for installing the newrelic agent:
$ cat tasks/papertrail.yml- name: copy papertrail pem certificate file to /etc  copy: >    src=rsyslog/{{item}}    dest=/etc/{{item}}  with_items:    - papertrail-bundle.pem
- name: copy rsyslog config files for papertrail integration  copy: >    src=rsyslog/{{item}}    dest=/etc/rsyslog.d/{{item}}  with_items:    - 50-default.conf    - 60-papertrail.conf  notify:     - restart rsyslog
$ cat tasks/newrelic.yml- name: copy newrelic debian package  copy: >    src=newrelic/{{newrelic_deb_pkg}}    dest=/opt/{{newrelic_deb_pkg}}
- name: install newrelic debian package  apt: deb=/opt/{{newrelic_deb_pkg}}
- name: configure newrelic with proper license key  template: >    src=nrsysmond.cfg.j2    dest=/etc/newrelic/nrsysmond.cfg    owner=newrelic    group=newrelic    mode=0640  notify:     - restart newrelic
templates - contains Jinja2 templates with variables that get their values from defaults/main.yml or from group_vars or host_vars files. One special variable that we use (and is not defined in these files, but instead is predefined by Ansible) is inventory_hostname which points to the hostname of the target being configured. For example, here is the template for a hostname file which will be dropped into /etc/hostname on the target:
$ cat roles/base/templates/hostname.j2{{ inventory_hostname }}
Once you have a playbook and a role, there are a few more files you need to take care of:
  • hosts/myhosts - this is an INI-type file which defines groups of hosts. For example the following snippet of this file defines 2 groups called api and magento.

[api]api01 ansible_ssh_host=api01.mydomain.coapi02 ansible_ssh_host=api02.mydomain.co
[magento]mgto ansible_ssh_host=mgto.mydomain.co
The api-servers.yml playbook file referenced at the beginning of this document sets the hosts variable to the api group, so all Ansible tasks will get run against the hosts included in that group. In the hosts/myhosts file above, these hosts are api01 and api02.
  • group_vars/somegroupname - this is where variables with ‚Äėsecret‚Äô values get defined for a specific group called somegroupname. The group_vars directory is not checked into GitHub. somegroupname needs to exactly correspond to the group defined in hosts/myhosts.

Example:
$ cat group_vars/apises_smtp_endpoint: email-smtp.us-west-2.amazonaws.comses_smtp_port: 587ses_smtp_username: some_usernameses_smtp_password: some_passworddatadog_api_key: some_api_key. . . other variables (DB credentials etc)

  • host_vars/somehostname - this is where variables with ‚Äėsecret‚Äô values get defined for a specific host called somehostname. The host_vars directory is not checked into GitHub. somehostname needs to exactly correspond to a host defined in hosts/myhosts.

Example:
$ cat host_vars/api02insert_sample_data: false
This overrides the insert_sample_data variable and sets it to false only for the host called api02. This could also be used for differentiating between a DB master and slave for example.
Tying it all together
First you need to have ansible installed on your local machine. I used:
$ pip install ansible
To execute a playbook for a given hosts file against all api server, you would run:
$ ansible-playbook -i hosts/myhosts api-servers.yml
The name that ties together the hosts/myhosts file, the api-servers.yml file and the group_vars/groupname file is in this case api.
You need to make sure you have the desired values for that group in these 3 files:
  • hosts/myhosts: make sure you have the desired hosts under the [api] group
  • api-server.yml: make sure you have the desired roles for hosts in the api group
  • group_vars/api: make sure you have the desired values for variables that will be applied to the hosts in the api group
Launching a new api instance in EC2I blogged about this here.Updating an existing api instance
Make sure the instance hostname is the only hostname in the [api] group in the hosts/myhosts file. Then run:
$ ansible-playbook -i hosts/myhosts api-servers.yml

SE Radio Episode 249: Vaughn Vernon on Reactive Programming with the Actor Model

Show notes: In this episode, SE Radio Host Stefan Tilkov talks to Vaughn Vernon about the topic of his most recent book, Reactive Messaging Patterns with the Actor Model. Vaughn explains the concepts and history of reactive programming, the actor model and its consequences, actors and their relation to object orientation, a typical approach for […]
Categories: Programming

SE Radio Episode 249: Vaughn Vernon on Reactive Programming with the Actor Model

Show notes: In this episode, SE Radio Host Stefan Tilkov talks to Vaughn Vernon about the topic of his most recent book, Reactive Messaging Patterns with the Actor Model. Vaughn explains the concepts and history of reactive programming, the actor model and its consequences, actors and their relation to object orientation, a typical approach for […]
Categories: Programming

How To Deal With Bad Days

Making the Complex Simple - John Sonmez - Thu, 02/11/2016 - 17:00

Bad days are like days when you feel like doing nothing. You feel unmotivated, don’t want to do anything… But the problem is that you have a lot of work to do. On this video, I answer a question from a reader that asks me how I’ve managed to create 30 PluralSight Courses in less […]

The post How To Deal With Bad Days appeared first on Simple Programmer.

Categories: Programming

Old Age Code

Actively Lazy - Wed, 02/10/2016 - 22:34

Is your code ready for retirement? Is it suffering from the diseases of old age?¬†Do you have code you can’t even imagine retiring? It’s just too critical? Too pervasive? Too legacy?

Jon & The Widgets //www.flickr.com/photos/qchristopherConveyor Belt – thanks to https://www.flickr.com/photos/qchristopher

Jon’s first job out of school was in the local widget factory, WidgetCo. Jon was¬†young, enthusiastic and quickly took to the job of making widgets. The company was¬†pleased with Jon and he grew in experience, learning more about making widgets, taking on ever more responsibility; until eventually Jon was responsible for all widget production.

After a couple of years one of WidgetCo’s¬†biggest customers started asking about a new type of square widget. They had only made circular widgets before, but with Jon’s expertise they thought they¬†could take on this¬†new market. The design team got started, working closely with¬†Jon to design a new type of square widget¬†for Jon to produce. It was a great success, WidgetCo¬†were the first to market with a square widget and sales went through the roof.

Unfortunately the new more complex widget production pipeline was putting a lot of pressure on the packing team. With different shape widgets and different options all needing to be sorted and packed properly mistakes were happening and orders being returned. Management needed a solution and turned to Jon. The team realised that if Jon knew a bit more about how orders were going to be packed he could organise the widget production line better to ensure the right number of each shape widget with the right options were being made at the right time. This made the job much easier for the packers and customer satisfaction jumped.

//www.flickr.com/photos/foshydogSneeze – thanks to https://www.flickr.com/photos/foshydog

A few years down the line and Jon was now a key part of the company. He was involved in all stages of widget production from the design of new widgets and the tools to manufacture them through to the production and packaging. But one day Jon got sick. He came down with a virus and was out for a couple of days: the company stopped dead. Suddenly management realised how critical Jon was to their operations Рwithout Jon they were literally stuck. Before Jon was even back up to speed management were already making plans for the future.

Shortly after, the sales team had¬†a lead that needed a new hexagonal widget. Management knew¬†this was a golden opportunity to try and remove some of their reliance on Jon. While Jon was involved in the initial design, the team commissioned a new production line and hired some new, inexperienced staff¬†to run it. Unfortunately hexagonal widgets were vastly more complex than the square ones and, without Jon’s experience the new widget production line struggled. Quality was too variable, mistakes were being made and production was much too slow. The¬†team were confident they could¬†get better over time but management were unhappy. Meanwhile, Jon was still churning out his regular widgets, same as he always had.

But the packing team were in trouble again – with two, uncoordinated production lines at times they were inundated with widgets and at other times they were idle. Reluctantly, management agreed that the only solution was for Jon to take responsibility for coordinating both production lines. Their experiment to remove their reliance on Jon had failed.

//www.flickr.com/photos/neilmoraleeThe Deckhand – thanks to https://www.flickr.com/photos/neilmoralee

A few years later still and the new production line had settled down; it never quite reached the fluidity of the original production line but sales were ok. But there was a new widget company offering a new type of octagonal widget. WidgetCo¬†desperately needed to catch up. The design team worked with Jon but, with his workload coordinating two production lines, he was always busy – so the designers were always waiting on Jon for feedback. The truth was: Jon was getting old. His eyes weren’t what they used to be and the arthritis in his fingers made working the prototypes for the incredibly complex new widgets difficult. Delays piled up and management got ever more unhappy.

So what should management do?

Human vs Machine

When we cast a human in the central role in this story it sounds ridiculous. We can’t imagine a company being so reliant on one frail human being that it’s brought to its knees by an illness. But read the story as though Jon is a software system and suddenly it seems totally reasonable. Or if not reasonable, totally familiar. Throughout the software world we see vast edifices of legacy software, kept alive way past their best because nobody has the appetite to replace them. They become too ingrained, too critical:¬†too big to fail.

14256058429_f7802658c8_zFor all it’s artificial construct, software is not so different from a living organism. A large code base will be of a level of complexity comparable with an organism – too complex for any single human being to comprehend in complete detail. There will be outcomes and responses that can’t be explained completely, that require detailed research to understand the pathway that leads from stimulus to response.

But yet we treat software like it is immortal. As though once written software will carry on working forever. But the reality is that within a few years software becomes less nimble, harder to change. With the growing weight of various changes of direction and focus software becomes slower and more bloated. Each generation piling on the pounds. Somehow no development team has mastered turning back time and turning a creaking, old age project into the glorious flush of youth where everything is possible and nothing takes any time at all.

It’s time to accept that software needs to be allowed to retire. Look around the code you maintain: what daren’t you retire? That’s where you should start. You should start planning to retire it¬†soon, because if it’s bad now it is only getting worse. As the adage goes: the best time to start fixing this was five years ago; the second best time is now.

//www.flickr.com/photos/ewwhiteSad dog – thanks to https://www.flickr.com/photos/ewwhite

After five years all software is a bit creaky; not so quick to change as it once was. After ten years it’s well into legacy; standard approaches have moved on, tools improved, decade old software just feels dated to work with. After twenty years it really should be allowed to retire already.

Software ages badly, so you need to plan for it from the beginning. From day one start thinking about how you’re going to replace this system. A monolith will always be impossible to replace, so constantly think about breaking out separate components that could be retired independently. As soon as code has got bigger than you can throw away, it’s too late. Like a black hole a monolith sucks in functionality, as soon as you’re into run away growth your monolith will consume everything in it’s path. So keep things small and separate.

What about the legacy you have today? Just start. Somewhere. Anywhere. The code isn’t getting any younger.


Categories: Programming, Testing & QA

Announcing the People API

Google Code Blog - Wed, 02/10/2016 - 22:21
Posted by Laurence Moroney, Developer Advocate

We’re delighted to announce the availability of the People API. With it, you can retrieve data about an authenticated user’s connections from their Contacts. Previously, developers had to make multiple calls to the Google+ API for user profiles and the Contacts API for contacts. The new People API uses the newest protocols and technologies and will eventually replace the Contacts API which uses the GData protocol.

For example, if your user has contacts in her private contact list, a call to the API (if she provides consent to do so) will retrieve a list containing the contacts merged with any linked profiles. If the user grants the relevant scopes, the results are returned as a people.connections.list object. Each person object in this list will have a resourceName property, which can be used to get additional data about that person with a call to people.get.

The API is built on HTTP and JSON, so any standard HTTP client can send requests to it and parse the response. However, applications need to be authorized to access the APIs so you will need to create a project on the Google Developers Console in order to get the credentials you need to access the service. All the steps to do so are here. If you’re new to the Google APIs and/or the Developers Console, check out this first in a series of videos to help you get up-to-speed.

Once you’re connected and authorized, you can then get the user’s connections like this (using the Google APIs Client Library for Java):


 ListConnectionsResponse response =   
peopleService.people().connections().list("people/me").execute();
List<Person> connections = response.getConnections();

Full documentation on the people.connections.list method is available here.

The list of connections will have details on all the user’s social connections if the required scopes have been granted. Contacts will only be returned if the user granted a contacts scope.

Each Person item will have a resource_name associated with it, so additional data for that person will be accessible via a simple call:

Person person = peopleService.people().get("resourceName").execute();

Details on this API call can be found here.

In addition to merging data from multiple sources and APIs into a single cohesive data source, the new People API also exposes additional data that was not possible to get before, such as private addresses, phone numbers, e-mails, and birthdays for a user who has given permission.

We hope that these new features and data along with simplified access to existing data inspires you to create the next generation of cool web and mobile apps that delight your users and those in their circles of influence. To learn more about the People API, check out the official documentation here.

Categories: Programming

Announcing the People API

Google Code Blog - Wed, 02/10/2016 - 22:21
Originally posted on Android Developers blog
Posted by Laurence Moroney, Developer Advocate

We’re delighted to announce the availability of the People API. With it, you can retrieve data about an authenticated user’s connections from their Contacts. Previously, developers had to make multiple calls to the Google+ API for user profiles and the Contacts API for contacts. The new People API uses the newest protocols and technologies and will eventually replace the Contacts API which uses the GData protocol.

For example, if your user has contacts in her private contact list, a call to the API (if she provides consent to do so) will retrieve a list containing the contacts merged with any linked profiles. If the user grants the relevant scopes, the results are returned as a people.connections.list object. Each person object in this list will have a resourceName property, which can be used to get additional data about that person with a call to people.get.

The API is built on HTTP and JSON, so any standard HTTP client can send requests to it and parse the response. However, applications need to be authorized to access the APIs so you will need to create a project on the Google Developers Console in order to get the credentials you need to access the service. All the steps to do so are here. If you’re new to the Google APIs and/or the Developers Console, check out this first in a series of videos to help you get up-to-speed.

Once you’re connected and authorized, you can then get the user’s connections like this (using the Google APIs Client Library for Java):


 ListConnectionsResponse response =   
peopleService.people().connections().list("people/me").execute();
List<Person> connections = response.getConnections();

Full documentation on the people.connections.list method is available here.

The list of connections will have details on all the user’s social connections if the required scopes have been granted. Contacts will only be returned if the user granted a contacts scope.

Each Person item will have a resource_name associated with it, so additional data for that person will be accessible via a simple call:

Person person = peopleService.people().get("resourceName").execute();

Details on this API call can be found here.

In addition to merging data from multiple sources and APIs into a single cohesive data source, the new People API also exposes additional data that was not possible to get before, such as private addresses, phone numbers, e-mails, and birthdays for a user who has given permission.

We hope that these new features and data along with simplified access to existing data inspires you to create the next generation of cool web and mobile apps that delight your users and those in their circles of influence. To learn more about the People API, check out the official documentation here.

Categories: Programming

Announcing the People API

Android Developers Blog - Wed, 02/10/2016 - 22:12
Posted by Laurence Moroney, Developer Advocate

We’re delighted to announce the availability of the People API. With it, you can retrieve data about an authenticated user’s connections from their Contacts. Previously, developers had to make multiple calls to the Google+ API for user profiles and the Contacts API for contacts. The new People API uses the newest protocols and technologies and will eventually replace the Contacts API which uses the GData protocol.

For example, if your user has contacts in her private contact list, a call to the API (if she provides consent to do so) will retrieve a list containing the contacts merged with any linked profiles. If the user grants the relevant scopes, the results are returned as a people.connections.list object. Each person object in this list will have a resourceName property, which can be used to get additional data about that person with a call to people.get.

The API is built on HTTP and JSON, so any standard HTTP client can send requests to it and parse the response. However, applications need to be authorized to access the APIs so you will need to create a project on the Google Developers Console in order to get the credentials you need to access the service. All the steps to do so are here. If you’re new to the Google APIs and/or the Developers Console, check out this first in a series of videos to help you get up-to-speed.

Once you’re connected and authorized, you can then get the user’s connections like this (using the Google APIs Client Library for Java):


 ListConnectionsResponse response =   
     peopleService.people().connections().list("people/me").execute();  
 List<Person> connections = response.getConnections();  

Full documentation on the people.connections.list method is available here.

The list of connections will have details on all the user’s social connections if the required scopes have been granted. Contacts will only be returned if the user granted a contacts scope.

Each Person item will have a resource_name associated with it, so additional data for that person will be accessible via a simple call:

Person person = peopleService.people().get("resourceName").execute();

Details on this API call can be found here.

In addition to merging data from multiple sources and APIs into a single cohesive data source, the new People API also exposes additional data that was not possible to get before, such as private addresses, phone numbers, e-mails, and birthdays for a user who has given permission.

We hope that these new features and data along with simplified access to existing data inspires you to create the next generation of cool web and mobile apps that delight your users and those in their circles of influence. To learn more about the People API, check out the official documentation here.

Categories: Programming

Announcing the People API

Android Developers Blog - Wed, 02/10/2016 - 22:12
Posted by Laurence Moroney, Developer Advocate

We’re delighted to announce the availability of the People API. With it, you can retrieve data about an authenticated user’s connections from their Contacts. Previously, developers had to make multiple calls to the Google+ API for user profiles and the Contacts API for contacts. The new People API uses the newest protocols and technologies and will eventually replace the Contacts API which uses the GData protocol.

For example, if your user has contacts in her private contact list, a call to the API (if she provides consent to do so) will retrieve a list containing the contacts merged with any linked profiles. If the user grants the relevant scopes, the results are returned as a people.connections.list object. Each person object in this list will have a resourceName property, which can be used to get additional data about that person with a call to people.get.

The API is built on HTTP and JSON, so any standard HTTP client can send requests to it and parse the response. However, applications need to be authorized to access the APIs so you will need to create a project on the Google Developers Console in order to get the credentials you need to access the service. All the steps to do so are here. If you’re new to the Google APIs and/or the Developers Console, check out this first in a series of videos to help you get up-to-speed.

Once you’re connected and authorized, you can then get the user’s connections like this (using the Google APIs Client Library for Java):


 ListConnectionsResponse response =   
     peopleService.people().connections().list("people/me").execute();  
 List<Person> connections = response.getConnections();  

Full documentation on the people.connections.list method is available here.

The list of connections will have details on all the user’s social connections if the required scopes have been granted. Contacts will only be returned if the user granted a contacts scope.

Each Person item will have a resource_name associated with it, so additional data for that person will be accessible via a simple call:

Person person = peopleService.people().get("resourceName").execute();

Details on this API call can be found here.

In addition to merging data from multiple sources and APIs into a single cohesive data source, the new People API also exposes additional data that was not possible to get before, such as private addresses, phone numbers, e-mails, and birthdays for a user who has given permission.

We hope that these new features and data along with simplified access to existing data inspires you to create the next generation of cool web and mobile apps that delight your users and those in their circles of influence. To learn more about the People API, check out the official documentation here.

Categories: Programming

Android Developer Story: Music app developer DJIT builds higher quality experiences and successful businesses on Android

Android Developers Blog - Wed, 02/10/2016 - 22:09

Posted by Lily Sheringham, Google Play team

Paris-based DJiT is the creator of edjing, one of the most downloaded DJ apps in the world, it now has more than 60 million downloads and a presence in 182 countries. Following their launch on Android, the platform became the largest contributor of business growth, with 50 percent of total revenue and more than 70 percent of new downloads coming from their Android users.

Hear from Jean-Baptiste Hironde, CEO & Co-founder, Séverine Payet, Marketing Manager, and Damien Delépine, Android Software Engineer, to learn how DJit improved latency on new Android Marshmallow, as well as leveraged other Android and Google Play features to create higher quality apps.



Find out more about building great audio apps and how to find success on Google Play.

Categories: Programming

Android Developer Story: Travel app Wego, increases monthly user retention by 300% with material design

Android Developers Blog - Wed, 02/10/2016 - 22:06

Posted by Lily Sheringham, Google Play team

Headquartered in Singapore, Wego is a popular online travel marketplace for flights and hotels for users in South East Asia and the Middle East. They launched their Android app in early 2014, and today, more than 62 percent of Wego app users are on Android. Wego recently redesigned their app using material design principles to provide their users a more native Android experience for consistency and easier navigation.

Watch Ross Veitch, co-founder and CEO, and the Wego team talk about how they increased monthly user retention by 300 percent and reduced uninstall rates by up to 25 percent with material design.


Learn more about material design, how to use Android Studio, and how to find success on Google Play with the new guide ‚ÄėSecrets to App Success on Google Play.‚Äô

Categories: Programming

Drip Funding, Slicing, Decomposing Large Projects Into Small Projects?

Herding Cats - Glen Alleman - Wed, 02/10/2016 - 18:54

Drip Funding, Slicing, Decomposing Large Projects Into Small Projects are popular platitudes of the agile community and have been picked up by the #NoEstimates community as the response to questions about funding work, business processes, managerial finance and making decisions in the presence of uncertainty.

A domain is always needed before any suggestion for anything can be assessed to be applicable and useful beyond personal anecdotes.

Let's look at an actual program for a Health Insurance Enterprise system provider network system. The diagram below shows what capabilities neeed to be produced in what order for the system to be of any use to the business.

This by the way is a critical issue in the agile development paradigm. The customer may prioritize what Features come first. But there is a business architecture in place for any enterprise system. The order of the work is not arbitrary. Partially completed software components are of little use. Drip funding and Drip delivery may be fine for simple applications. But a System of Systems no longer has an arbitrary order of the work. It's a working system and the capabilities need to appear in the right order for any value to be produce. This order of delivery is engineered not emergent.

In the example below, the systems starts with a legacy suite of applications. The final system is an ERP style claims processing system, provider network integration, and a Data Mart for the information associated with processing claims from the provider. This type of development, integration, and evolutionary deployment is common when there is a mix of legacy and new components. Even in the new components - say green field deployment of SAP. But rarely today is there an Green Field deployment in any non-trivial environment.

The way this diagram reads is things in Boxes are system capabilities, complete with Definition of Done - not in the simple Agile DoD. But with Measures of Effectiveness, Measures of Performance, Technical Performance Measures, and Key Performance Parameters. All defined upfront.  These are not requirements, they are what Done Looks Like in units of measure meaningful to the decision makers.  

The connections between the boxes - moving left to right - are the sequences of the needed capabilities to have them be useful to the business. For example having a Pilot alone isn't much use. Having the Pilot and having real data for that Pilot to work on for a Demonstration of the Conversation process and Member Reconciliation is useful. This notion is how to read the chart.

Screen Shot 2015-12-28 at 6.36.37 PM

By the Way this chart is just a different version of an Integrated Master Plan. This is not by accident. Several of the senior managers on this program, along with our team, work in the Space and Defense business as well, where Software Intensive System of Systems are the norm.  The Integrated Master Plan / Integrated Master Schedule programmatic architecture looks like this. Where tangible business benefits are at the Event Level, the Significant Accomplishments needed to produce those benefits are next, and the Accomplishment Criteria are the exit criteria for the Work Packages (Features in Scrum) needed to produce the working software that can be put to use by the business.

Screen Shot 2015-12-28 at 6.47.40 PM

The connectivity between the IMP elements (Events, Accomplishments, and Criteria) and the Work Packages in the IMS looks like this:

Screen Shot 2015-12-28 at 6.49.14 PM

All This is a Set Up for the Questions In the Title

If we have the typical enterprise legacy integration with new enterprise application - the vast majority of work in IT,  ...

  • Can Drip Funding be used to deploy this enterprise system?
  • Can the work be sliced into smaller incremental deliveries?
  • Can this big project be broken down into a bunch of small projects?

Drip Funding

Drip Funding is claimed to be a way of developing software without having to estimate how much it will cost in the end. Of course this ignores the basic managerial finance principles of any enterprise effort. That principle says, someone, somewhere in the organization who is accountable for spending money has a fiduciary need to how much will be spent, when that expenditure will turn into productive use for accounting purposes of FASB 86, and what is the cash flow for expense curve look like against the bookable value to the firm. 

So Drip Funding also needs Drip Benefits, in the terms of revenue if this is an external project or bookable benefits if this is an internal project. Expense and Capitalization opportunities are also needed for the integrity of the balance sheet every month. How much did we spend, how much will we spend, when will we start to see the rewards from that spend so we can start to balance the books on that expenditure?

Looking back to the Capabilities flow picture, we can certainly incrementally fund work to produce a working capability. But dolling out money in Drips may or may not get us there. How big do the Drips need to be? When do they need to arrive?  There is a difference between allocated funds and committed funds. There may be budget for the work, but authorizing the spend for the work is different. 

While Drip Funding is a popular term in Agile, those accountable for the firms spend - cost accounting and the CFO - need to weigh in on the balance sheet aspects of providing incremental funds with some expectation of when they will get their money back. This is standard Managerial Finance and also the basis of decision making in the paradigm of Microeconomics. If I have a fixed amount to spend, what are the choices that can be made that will produce a tangible beneficial outcome in exchange for that spend? The first number is fixed (usually), the other numbers (benefits and when) are uncertain and require estimates to be made to close the loop of the decision making process.

 

Related articles Build a Risk Adjusted Project Plan in 6 Steps How Think Like a Rocket Scientist - Irreducible Complexity Impact Mapping and Integrated Master Planning 5 Questions That Need Answers for Project Success Herding Cats: What does done look like? How to Deal With Complexity In Software Projects?
Categories: Project Management

How to build your Property Management System integration using Microservices

This is a guest post by Rafael Neves, Head of Enterprise Architecture at ALICE, a NY-based hospitality technology startup. While the domain is Property Management, it's also a good microservices intro.

In a fragmented world of hospitality systems, integration is a necessity. Your system will need to interact with different systems from different providers, each providing its own Application Program Interface (API). Not only that, but as you integrate with more hotel customers, the more instances you will need to connect and manage this connection. A Property Management System (PMS) is the core system of any hotel and integration is paramount as the industry moves to become more connected.

 

To provide software solutions in the hospitality industry, you will certainly need to establish a 2-way integration with the PMS providers. The challenge is building and managing these connections at scale, with multiple PMS instances across multiple hotels. There are several approaches you can leverage to implement these integrations. Here, I present one simple architectural design to building an integration foundation that will increase ROI as you grow. This approach is the use of microservices.

What are microservices? 
Categories: Architecture

The Economics of Sofware Quality

Herding Cats - Glen Alleman - Wed, 02/10/2016 - 02:05

Economics of SWTo predict the future of a software project with acceptable accuracy and precision, it is necessary to measure past project and keep track of current and ongoing projects. Estimation and  measurement are closely aligned, and good historical data is of great value in estimating future outcomes of future software projects. - Opening of Chapter 2 of the book to the left.

Software Development, using other peoples money in the presence of uncertainty is a microeconomics paradigm. What choices are needed to assure project success, confirm that the funding invested produces the planned return on that spend? What choices are best for the current understanding of the uncertainties faced by the project. Both reducible and irreducible uncertainties?

To suggest decisions can be made without estimating the outcome of those choices it to willfully ignore the foundation principles of microeconomics and managerial finance of software development projects.

Those conjecturing those decisions can be made without making estimates, are participants in this willful ignorance.

Categories: Project Management