Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/categories/7&page=2' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

Stuff The Internet Says On Scalability For July 1st, 2016

Hey, it's HighScalability time:


If you can't explain it with Legos then you don't really understand it.

 

If you like this sort of Stuff then please support me on Patreon.

  • 700 trillion: more pixels in Google's Satellite Map; 9,000km: length of new undersea internet cable from Oregon to Japan; 60 terabits per second: that undersea internet cable again; 12%: global average connection speed increase; 76%: WeChat users who spend more than 100RMB ($15) per month; 5 liters: per day pay in beer for Pyramid workers;  680: number of rubber bands it takes to explode a watermelon; 1,000: new Amazon services this year; $15 billion: amount Uber has raised; 7 million: # of feather on on each bird in Piper; 5.8 million: square-feet in Tesla Gigafactory; 2x: full-duplex chip could double phone-network data capacity; 

  • Quotable Quotes:
    • @hyc_symas: A shame everyone is implementing on top of HTTP today. Contemporary "protocol design" is a sick joke.
    • @f3ew: Wehkamp lost dev and accept environments 5 days before launch. Shit happens.  48 hours to recovery. #devopsdays
    • Greg Linden: Ultimately, [serverless computing] this is a good thing, making compute more efficient by allowing more overlapping workloads and making it easier to move compute around. But it does seem like compute on demand could cannibalize renting VMs.
    • @viktorklang: What if we started doing only single-core chips with massive eDRAM on-package and PCI-E peer-writes MPI between? (Micro-blade machines?)
    • Robert Graham: Programmers fetishize editors. Real programmers, those who produce a lot of code, fetishize debuggers
    • @jasonhand: "Systems are becoming more ephemeral and we have to find a way to deal with it" (regarding monitoring) - @adrianco #monitorama
    • @aphyr: Queues *can* improve fault tolerance, but this only happens if they don't lose your messages. The only one I know of that doesn't is Kafka.
    • Tom Simon: There are, accordingly, two ways of reading books; but infinitely many ways to divide up the act of reading into two classes.
    • Puppet: High-performing IT organizations deploy 200 times more frequently than low performers, with 2,555 times faster lead times.
    • @benzobot: “The system scaled with the number of engineers - more engineers, more metrics.” #monitorama
    • fizx: You seem to have installation confused with administration. Off the top of my head you forgot security, monitoring, logging config, backups, handling common production issues such as splitbrains, write multiplication, garbage collection snafus, upgrades between versions with questionably compatible internal apis.
    • @Mark_J_Perry: This has to be one of the most remarkable achievements ever: Global Poverty Fell Below 10% for 1st Time in 2015
    • ewams: If you are a services company, he is right, you should be focusing on outcomes. But, if you can't tell me in 2-3 sentences what problem you are solving and how it benefits the customer you are doing it wrong.
    • @retrohack3r: Dance like nobody is watching. Encrypt like everyone is.
    • @GundersenMarius: ES6 + HTTP/2 + Service Workers + Bloom-filter = efficient module loading without bundlin mariusgundersen.net/module-pusher/ 
    • @timperrett: Distributed systems are about protocols, not implementations. Forget languages, protocols are everything.
    • steveblank: What’s holding large companies back?...companies bought into the false premise that they exist to maximize shareholder value – which said “keep the stock price high.” As a consequence, corporations used metrics like return on net assets (RONA), return on capital deployed, and internal rate of return (IRR) to measure efficiency. These metrics make it difficult for a company that wants to invest in long-term innovation.
    • Greg Linden: Like a lot of things at Amazon, this went through many stages of wrong before we got it right. As I remember it, this went through some unpleasant, heavyweight, and inefficient RPC (esp. CORBA) and pub-sub architectures before an unsanctioned skunkworks project built iquitos for lightweight http-based microservices 
    • @rawrthis:  "We all die." Except my legacy stack. That crap will live forever. #devopsdays
    • Jim Handy: The industry already has more than enough DRAM wafer capacity for the foreseeable future. Why is this happening?  The answer is relatively simple: the gigabytes per wafer on a DRAM wafer are growing faster than the market’s demand for gigabytes.
    • SwellJoe: A container doesn't consume complexity and emit order. The complexity is still in there; you still have to build your containers in a way that is replicable, reliable, and automatable. I'm not necessarily saying configuration management is the only way to address that complexity, but it does have to be addressed in a container-based environment.

  • Deep Learning desires data, so if you want to build an AI that learns how to program this is how you would go about it, you would bring all the open source code into your giant, voracious, data crunching maw. Making open source data more available: Today, [GitHub] we're delighted to announce that, in collaboration with Google, we are releasing a collection of additional BigQuery tables to expand on the GitHub Archive...This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains activity data for more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths.

  • In almost all cases, the single-threaded implementation outperforms all the others, sometimes by an order of magnitude, despite the distributed systems using 16-128 cores. Scalability! But at what COST? The paper is, at its heart, a criticism of how the performance of current research systems are evaluated. The authors focus on the field of graph processing, but their arguments extend to most distributed computation research where performance is a key factor. They observe that most systems are currently evaluated in terms of their scalability, how their performance changes as more compute resources are used, but that this metric is often both useless and misleading.

  • hinkley: The older I get the more I feel like we're an accelerate version of the fashion industry. At least in fashion you can make an excuse that the design is thirty years old and most people don't remember the last time we did this. With software it's every six or seven. It's hard not to judge my peers for having such short memories. We were in the midst of one of these upheavals when I first started, and so I learned programming in that environment. It also means I have one more cycle than most people near my age. Now it all looks the same to me, and I understand those people who wanted to be more conservative. In fact I probably owe some people an apology.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Scaling Hotjar's Architecture: 9 Lessons Learned

Hotjar offers free website analytics so they have a challenging mission: handle hundreds of millions of requests per day from mostly free users. Marc von Brockdorff, Co-Founder & Director of Engineering at Hotjar, summarized the lessons they've learned in: 9 Lessons Learned Scaling Hotjar's Tech Architecture To Handle 21,875,000 Requests Per Hour.

In response to the criticism their architecture looks like a hot mess, Erik Näslund, Chief Architect at Hotjar, gives the highlights of their architecture:

  • We use nginx + lua for the really hot code paths where python doesn't quite cut it. No language is perfect and you might have to break out of your comfort zone and use something different every now and then.
  • Redis, Memcached, Postgres, Elasticsearch and S3 are all suitable for different kinds of data storage and we eventually needed them all to be able to query and store data in a cost effective way. We didn't start out using 5 different data-stores though...it's something that we "grew into".
  • Each application server is a (majestic) monolith. Micro-services are one way of architecting things, monoliths are another - I'm still waiting to be convinced that one way is superior to the other when it comes to a smaller team of developers.

What have they learned?

Categories: Architecture

25 Ways to Get Better Results

Have you ever wondered why some people seem to be really good at getting results, while others struggle?

What are the true secrets of productivity that set one person apart from another, and help them achieve high performance?

I’ve studied the challenge of personal productivity deeply to really understand the difference that makes the difference.

I’ve put together a list of 25 Keys to Getting Better Results that cut to the chase and get to the bottom line of extreme productivity:

25 Keys to Getting Better Results

This list of productivity secrets is from my best-selling productivity book, Getting Results the Agile Way, and reflects the best of what I’ve learned in terms of what really makes some people stand out when it comes to personal productivity, high performance, and making things happen.

After all, who doesn’t want better results in work and life?

You can read the list of productivity secrets, but here I want to touch on a few key ideas.

Vision

Vision is probably the single most important starting point when it comes to getting better results. 

It’s hard to do anything if you don’t know what it’s supposed to be or do when it’s done.

If you can see the end-in-mind, then you are off to a good start.

And if you feel really good about the end-in-mind, then you have something to pull you forward versus trying to push yourself forward.

When your vision pulls you forward, you are on the right path.

Value

Value is in the eye of the beholder and in the eye of the stakeholder.

Value is also the ultimate short-cut.

If you know what good looks like or if you know what’s actually valued, you can refocus your efforts on high-value activities that produce those outcomes.

When you don’t know who the value is for, or what good looks like, you are in trouble.

That’s why it’s important to check with the people you are producing value for, whether you are actually nailing their pains, needs, and desired outcomes.

If not, no problem – learn and adapt.

Learn early, learn often, and really get curious about which of your activities produce the highest value results.

Velocity

Speed is the name of the game.

As John Wooden would often say, “Be quick, but don’t hurry.”

In other words, make your moves quickly, but with intention, and with quality.

Quality comes through practice and repetition.  It’s how you learn.

Try things.   But try then quickly, and experiment in how you product results.

Use speed as your friend to learn faster, and to build momentum.

Don’t get bogged down.  Use speed to cut through your challenges, and quickly prioritize your best bets, and create a flow of continuous value.

Sometimes you will need to slow down to speed up.

But more often than not, you will need to speed up, so you that you are effectively taking massive action.

Few challenges withstand the onslaught of massive action, as long as you keep on learning and improving.

Well, that’s about it for now.

I hope that gives you at least a big of an edge that you can use in your day, every day, to get better, faster, simpler results for work and life.

Enjoy.

Categories: Architecture, Programming

The Purpose Alignment Model

Xebia Blog - Wed, 06/29/2016 - 09:45
When scaling Agile/Scrum, we invariably run into the alignment vs. autonomy problem. In short, you cannot have autonomous, self-directing teams if they have no clue what direction they should go. Or, even shorter, alignment breeds autonomy. But how do we create alignment? And what tools can we use to quickly evaluate whether or not what

Running Gatling load tests in Docker containers via Jenkins

Agile Testing - Grig Gheorghiu - Wed, 06/29/2016 - 00:16
Gatling is a modern load testing tool written in Scala. As part of the Jenkins setup I am in charge of, I wanted to run load tests using Gatling against a collection of pages for a given website. Here are my notes on how I managed to do this.

Running Gatling as a Docker container locally

There is a Docker image already available on DockerHub, so you can simply pull down the image locally:


$ docker pull denvazh/gatling:2.2.2
Instructions on how to run a container based on this image are available on GitHub:
$ docker run -it --rm -v /home/core/gatling/conf:/opt/gatling/conf \-v /home/core/gatling/user-files:/opt/gatling/user-files \-v /home/core/gatling/results:/opt/gatling/results \ denvazh/gatling:2.2.2
Based on these instructions, I created a local directory called gatling, and under it I created 3 sub-directories: conf, results and user-files. I left the conf and results directories empty, and under user-files I created a simulations directory containing a Gatling load test scenario written in Scala. I also created a file in the user-files directory called urls.csv, containing a header named loc and a URL per line for each page that I want to load test.
Assuming the current directory is gatling, here are examples of these files:
$ cat user-files/urls.csvlochttps://my.website.comhttps://my.website.com/category1https://my.website.com/category2/product3
$ cat user-files/simulations/Simulation.scala
package my.gatling.simulation
import io.gatling.core.Predef._import io.gatling.http.Predef._import scala.concurrent.duration._
class GatlingSimulation extends Simulation {
  val httpConf = http    .baseURL("http://127.0.0.1")    .acceptHeader("text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")    .doNotTrackHeader("1")    .acceptLanguageHeader("en-US,en;q=0.5")    .acceptEncodingHeader("gzip, deflate")    .userAgentHeader("Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0")
  val scn1 = scenario("Scenario1")    .exec(Crawl.crawl)
  val userCount = Integer.getInteger("users", 1)  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)  setUp(    scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))  ).protocols(httpConf)}
object Crawl {
  val feeder = csv("/opt/gatling/user-files/urls.csv").random
  val crawl = exec(feed(feeder)    .exec(http("${loc}")    .get("${loc}")    ))}

I won't go through the different ways of writing Gatling load tests scenarios here. There are good instructions on the Gatling website -- see the Quickstart and the Advanced Tutorial. What the scenario above does is it reads the file urls.csv and randomly picks a URL from it, then runs a load test against that URL.
I do want to point out 2 variables in the above script:

  val userCount = Integer.getInteger("users", 1)  val durationInSeconds  = java.lang.Long.getLong("duration", 10L)
These variables specify the max number of users we want to ramp up to, and the duration of the ramp-up. They are used in the inject call:

scn1.inject(rampUsers(userCount) over (durationInSeconds seconds))

The special thing about these 2 variables is that they are read from JAVA_OPTS by Gatling. So if you have a -Dusers Java option and a -Dduration Java option, Gatling will know how to read them and how to set the userCount and durationInSeconds variables accordingly. This is a good thing, because it allows you to specify those numbers outside of Gatling, without hardcoding them in your simulation script. Here is more info on passing parameters via the command line to Gatling.

While pulling the Gatling docker image and running it is the simplest way to run Gatling, I prefer to understand what's going on in that image. I started off by getting the Dockerfile from GitHub:

$ cat Dockerfile
# Gatling is a highly capable load testing tool.## Documentation: http://gatling.io/docs/2.2.2/# Cheat sheet: http://gatling.io/#/cheat-sheet/2.2.2
FROM java:8-jdk-alpine
MAINTAINER Denis Vazhenin
# working directory for gatlingWORKDIR /opt
# gating versionENV GATLING_VERSION 2.2.2
# create directory for gatling installRUN mkdir -p gatling
# install gatlingRUN apk add --update wget && \  mkdir -p /tmp/downloads && \  wget -q -O /tmp/downloads/gatling-$GATLING_VERSION.zip \  https://repo1.maven.org/maven2/io/gatling/highcharts/gatling-charts-highcharts-bundle/$GATLING_VERSION/gatling-charts-highcharts-bundle-$GATLING_VERSION-bundle.zip && \  mkdir -p /tmp/archive && cd /tmp/archive && \  unzip /tmp/downloads/gatling-$GATLING_VERSION.zip && \  mv /tmp/archive/gatling-charts-highcharts-bundle-$GATLING_VERSION/* /opt/gatling/
# change context to gatling directoryWORKDIR  /opt/gatling
# set directories below to be mountable from hostVOLUME ["/opt/gatling/conf", "/opt/gatling/results", "/opt/gatling/user-files"]
# set environment variablesENV PATH /opt/gatling/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binENV GATLING_HOME /opt/gatling
ENTRYPOINT ["gatling.sh"]
I then added a way to pass JAVA_OPTS via an environment variable. I added this line after the ENV GATLING_HOME line:
ENV JAVA_OPTS ""
I dropped this Dockerfile in my gatling directory, then built a local Docker image off of it:
$ docker build -t gatling:local .
I  then invoked 'docker run' to launch a container based on this image, using the csv and simulation files from above. The current directory is still gatling.

$ docker run --rm -v `pwd`/conf:/opt/gatling/conf -v `pwd`/user-files:/opt/gatling/user-files -v `pwd`/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=10 -Dduration=60" gatling:local -s MySimulationName

Note the -s flag which denotes a simulation name (which can be any string you want). If you don't specify this flag, the gatling.sh script which is the ENTRYPOINT in the container will wait for some user input and you will not be able to fully automate your load test.

Another thing to note is the use of JAVA_OPTS. In the example above, I pass -Dusers=10 and   -Dduration=60 as the two JAVA_OPTS parameters. The JAVA_OPTS variable itself is passed to 'docker run' via the -e option, which tells Docker to replace the default value for ENV JAVA_OPTS (which is "") with the value passed with -e.

Running Gatling as a Docker container from Jenkins

Once you have a working Gatling container locally, you can upload the Docker image built above to a private Docker registry. I used a private EC2 Container Registry (ECR).  

I also added the gatling directory and its sub-directories to a GitHub repository called devops.

In Jenkins, I created a new "Freestyle project" job with the following properties:

  • Parameterized build with 2 string parameters: USERS (default value 10) and DURATION in seconds (default value 60)
  • Git repository - add URL and credentials for the devops repository which contains the gatling files
  • An "Execute shell" build command similar to this one:
docker run --rm -v ${WORKSPACE}/gatling/conf:/opt/gatling/conf -v ${WORKSPACE}/gatling/user-files:/opt/gatling/user-files -v ${WORKSPACE}/gatling/results:/opt/gatling/results -e JAVA_OPTS="-Dusers=$USERS -Dduration=$DURATION"  /PATH/TO/DOCKER/REGISTRY/gatling -s MyLoadTest 


Note that we mount the gatling directories as Docker volumes, similarly to when we ran the Docker container locally, only this time we specify ${WORKSPACE} as the base directory. The 2 string parameters USERS and DURATION are passed as variables in JAVA_OPTS.
A nice thing about running Gatling via Jenkins is that the reports are available in the Workspace directory of the project. If you go to the Gatling project we created in Jenkins, click on Workspace, then on gatling, then results, you should see directories named gatlingsimulation-TIMESTAMP for each Gatling run. Each of these directories should have an index.html file, which will show you the Gatling report dashboard. Pretty neat.

The Ultimate Tester: Sharing Knowledge

Xebia Blog - Tue, 06/28/2016 - 09:50
In the past three blog posts we have explored some aspects of being an Ultimate Tester: How we can add value, how our curiosity helps us to test the crazy stuff and how we can build quality in. We learn a lot about these things during work time (and hopefully during personal time as well),

Reimagine Your Productivity at Getting Results.com

image

Imagine if you could master your motivation, your productivity, and your time management?

It’s time to get your game on.

I’ve completely revamped Getting Results.com (http://GettingResults.com) to help you think, feel, and do your best in any situation.

At Getting Results.com you can learn all about Agile Results.  It’s more than just a time management system or a productivity system.

It’s a personal results system for work and life.

Agile Results is a simple system for meaningful results.

It helps you respond to change while making things happen.  It helps you use your best energy for your best results, while you keep learning and improving.

What Will You Find at Getting Results.com?

Getting Results.com is a source of insight, inspiration, and action for mastering productivity, motivation, time management and more.

Here are some of the key things you’ll have access to:

  1. Get Started with Agile Results.
  2. Articles on Agile Results, Goals, Motivation, and More.
  3. 7 Day Agile Results Jumpstart, helps you take Agile Results for a test-drive and potentially have one of your best weeks.  Ever.
  4. 30 Days of Getting Results, this is advanced training that will help you get better results for work and life.
  5. Resources that include Cheat Sheets, Checklists, How Tos, Visuals, and more.

You can also read Success Stories to hear about how others have implemented and are using Agile Results.

Getting Started with Agile Results

While there are a lot of resources at Getting Results.com, the most important thing you can actually do is just get started with Agile Results:

Get Started with Agile Results

Take it for a test drive and see if you can create better results at work and in your life, the Agile Way.

Categories: Architecture, Programming

How Facebook Live Streams to 800,000 Simultaneous Viewers

Fewer companies know how to build world spanning distributed services than there are countries with nuclear weapons. Facebook is one of those companies and Facebook Live, Facebook’s new live video streaming product, is one one of those services.

Facebook CEO Mark Zuckerberg

The big decision we made was to shift a lot of our video efforts to focus on Live, because it is this emerging new format; not the kind of videos that have been online for the past five or ten years...We’re entering this new golden age of video. I wouldn’t be surprised if you fast-forward five years and most of the content that people see on Facebook and are sharing on a day-to-day basis is video.

If you are in the advertising business what could better than a supply of advertising ready content that is never ending, always expanding, and freely generated? It’s the same economics Google exploited when it started slapping ads on an exponentially growing web.

An example of Facebook’s streaming prowess is a 45 minute video of two people exploding a watermelon with rubber bands. It reached a peak of over 800,000 simultaneous viewers who also racked up over 300,000 comments. That’s the kind of viral scale you can generate with a social network of 1.5 billion users.

As a comparison The 2015 Super Bowl was watched by 114 million viewers with an average 2.36 million on the live stream. On Twitch there was a peak of 840,000 viewers at E3 2015. The September 16th Republican debate peaked at 921,000 simultaneous live streams.

So Facebook is right up there with the state of the art. Keep in mind Facebook would have a large number of other streams going on at the same time as well.

A Wired article quotes Chris Cox, Facebook’s chief product officer, who said Facebook:

  • Has more than a hundred people working on Live. (it started with ~12 and now there are more than 150 engineers on the project)

  • Needs to be able to serve up millions of simultaneous streams without crashing.

  • Need to be able to support millions of simultaneous viewers on a stream, as well as seamless streams across different devices and service providers around the world.

Cox said that “It turns out it’s a really hard infrastructure problem.”

Wouldn't it be interesting if we had some details about how that infrastructure problem was solved? Woe is we. But wait, we do!

 Federico Larumbe from Facebook’s Traffic Team, which works on the caching software powering Facebook’s CDN and the Global Load Balancing system, gave an excellent talk: Scaling Facebook Live, where he shares some details about how Live works.

Here’s my gloss on the talk. It’s impressive.

Origin Story
Categories: Architecture

Stuff The Internet Says On Scalability For June 24th, 2016

Hey, it's HighScalability time:


A complete and accurate demonstration of the internals of a software system.

 

If you like this sort of Stuff then please support me on Patreon.
  • 79: podcasts for developers; 100 million: daily voice calls made on WhatsApp; 2,000; cars Tesla builds each week; 2078 lbs: weight it takes to burst an exercise ball; 500 million: Instagram users; > 100M: hours watched per day on Netflix; 400 PPM: Antarctica’s CO2 Level; 2.5 PB: New Relic SSD storage; 

  • Quotable Quotes:
    • Alan Kay: The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? The Web, in comparison, is a joke. The Web was done by amateurs.
    • @jaykreps: Actually, yes: distributed systems are hard, but getting 100+ engineers to work productively on one app is harder.
    • @adrianco: All in 2016: Serverless Architecture: AWS Lambda, Codeless Architecture: Mendix, Architectureless Architecture: SaaS
    • @AhmetAlpBalkan: "That's MS SQL Server running on Ubuntu on Docker Swarm on Docker Datacenter on @Azure" @markrussinovich #dockercon
    • Erik Darling: EVERYTHING’S EASY WHEN IT’S WORKING
    • @blueben: Bold claim by @brianl: Most of the best tech of the last 10 years has come out of work at Google. #VelocityConf
    • Joe: there is no such thing as a silver bullet … no magic pixie dust, or magical card, or superfantastic software you can add to a system to make it incredibly faster. Faster, better performing systems require better architecture (physical, algorithmic, etc.). You really cannot hope to throw a metric-ton of machines at a problem and hope that scaling is simple and linear. Because it really never works like that.
    • Eran Hammer: The web is the present and it’s a f*cking mess. Deal with it.
    • @etherealmind: If you believe in DevOps/NetOps you have to believe that leaving Europe is a difficult but better course of action. Small, fast & fluid
    • Sanfilippo: Redis is currently not good for data problems where write safety is very important. One of the main design sacrifices Redis makes in order to provide many good things is data retention. It has best effort consistency and it has a configurable level of write safety, but it’s optimized for use cases where most of the time, you have your data, but in cases of large incidents you can lose a little bit of it. 
    • David Smith: The best time you are ever going to have to make a new app is when there's a new iOS update. Go through the diffs. Go through the What's New? Find something that couldn't be possible before and make an app around that. 
    • @DanielEricLee: There was a timezone bug so I wrote a test and then the test failed because the CI runs in a different timezone and then I became a farmer
    • @jasongorman: Reminds me of someone I know who led a dev team who built something that won an award. None of team invited to awards bash. Only snr mgmt.
    • David Robinson: My advice to graduate students: create public artifacts
    • @cdixon: Because distributed clusters of commodity machines are more powerful.
    • @wmf: 128-port TORs have outrun the compute capacity of most racks, so putting two mini-TORs in 1U is a great idea.
    • msravi: I moved from AWS EC2 to Google Cloud a few days ago. Google really seems to have beaten AWS, at least in pricing and flexibility. On AWS (Singapore region) a 2-vCPU, 7.5G RAM instance costs $143/month (not including IOPS and bandwidth costs), while a similar one on GC works out to about $56/month. That's a massive difference. In addition, GC allows me to customize cores and RAM flexibly to a point, which is important for me.
    • Mobile Breakfast: What is clear that we will get rid of the classic circuit-switched technology and move to all IP networks fairly soon in the US.
    • Douglas Rushkoff: I think they all come down to how you can optimize your business or the economy for flow over growth; for the circulation of money rather than the extraction of money.
    • Alan Hastings~ [Species] that go into synchrony may be more subject to extinction because a single driver can trigger a collapse
    • @TechCrunch: More cars than phones were connected to cell service in Q1 http://tcrn.ch/28MLtmt  by @kristenhg
    • @docker: "Nobody cares about #containers, it's the application that matters!" - @solomonstre, CTO @Docker #DockerCon
    • @cmeik: The most commercially successful NoSQL database: Lotus Notes.
    • Brittany Fleit: behavior-based push results in open rates 800 percent higher than immediate blasts. Personalizing a message based on individual actions garners much more engagement.

  • Dilbert, of course, nails it on AI.

  • How will hackers be stopped from using Return-oriented programming (ROP) to execute privilege escalation attacks? ROP was created when "clever hackers realized that they could successively invoke snippets of code at the end of existing subroutines as 'proxies' to accomplish the work they needed done." Randomizing memory locations didn't stop them so Intel created new hardware magic called Control-flow Enforcement Technology. Intel added a new "ENDBRANCH" instruction and created a new "shadow stack". It turns out the immense power and beauty of the stack in von neumann architectures is also a great weakness. The story of life. Steve Gibson with an inspiring deep dive on CET in Security Now 565

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

How to Have Better Fridays

Recently, I’ve been teaching more people Agile Results. 

I teach them really fast, because I just focus on teaching them the most important tool in Agile Results:

Monday Vision, Daily Wins, Friday Reflection

But before I walk through, I share a quick story of how it all started.

Monday Vision was Born for Better Fridays

It was a warm, sunny, Friday afternoon.
My colleague and I were on our way to our favorite pizza place.
It was a beautiful day.  It should have been a great day.
But I felt like a beast of burden with the weight of the world on my back.
Our backlog was overflowing, we didn’t make it through what we thought we would, and we had been slogging away.
And for what?
Well, I caught myself looking in the rear view mirror, more than looking ahead.
Instead of feeling great, I felt like crap.
So I turned to my colleague and asked him how much we realistically have to spend on work when we get back.
We lied to each other and ourselves.  Then we got real.
We figured the best thing we could possibly do would be to prioritize the value we could deliver next week.
With that, we enjoyed our pizza, and when we got back to work, we figured out what a great next week would look like.
I never wanted us to have another Friday where we couldn’t go into weekend feeling good about what we had accomplished for the week.
And that’s how Monday Vision was born.

Monday Vision, Daily Wins, Friday Reflection for Better Fridays

One I tell that little story, people get it pretty quickly.  They’ve been there.  They feel the pain.

They slogged away all week and at the end of the week, instead of feeling good about their achievements, they feel like they haven’t done enough.

They are never done.  They are overwhelmed. 

Instead of feeling like they earned their weekend for rest and relaxation, they feel guilty that they should work on their never-ending backlog and laundry-list of To-Dos.

Monday Vision for Better Fridays

So then I walk them through how to do Monday Vision, so they can have better Fridays.

On Mondays, imagine if it were Friday.  Really step into your future Friday and feel it.   What are Three Wins you really want to have under your belt?

What would you want to be able to say to your manager or to your team or to yourself, about what you accomplished or achieved for the week?

Get clarity on that.

Use that simple story of your Three Wins that you want to be able to talk about on Friday, as your way to prioritize your focus for the week on Monday.

Now you are doing “Monday Vision.”

Daily Wins for Better Fridays

Each day, identify your Three Wins for that day.  This is the “Daily Wins” practice.

You will have those days where your Three Wins might be, “Great Breakfast,” “Great Lunch”, “Great Dinner.”

There will be days that knock you down, and you wonder how you will get back up.

But then you will also have those days where you are on top of the world and your Three Wins for today will be magnificent.

You might even say, they will be your masterpiece.

Either way, get in the habit of starting your day by identifying Three Wins you want to achieve.

That will help you focus and prioritize all that you do in a more meaningful way for results that matter.

If you don’t know how to do this, just imagine if you were closing out your day, what are Three Wins that you want to be able to say you’ve achieved, either to you, your manager, your team or that someone special.

Friday Reflection for Better Fridays

Lastly, there is Friday Reflection.

Friday Reflection is your chance to dig deep and gain some new personal productivity insights.

Ask yourself, “What are three things going well?” and ask yourself, “What are three things to improve?”

Be honest with yourself about your answers.

You are the one that will win or lose from what you learn.

This is your chance to change any long-standing patterns of your personal productivity challenges.

Do you bite off more than you can chew?  Do you get randomized during the week?

Do you have a hard time figuring out what is actually valued?

Use what you learn to feed into next week.  This is your chance to change and practice your Growth Mindset.

Don’t expect any of these exercises to be easy, but they get easier with practice.

And that’s just it.  This isn’t a one-time trick.

This is a very precise set of productivity habits and practices that you can use for your lifetime to master time management, master your energy, master your motivation, and become more of what you are capable of.

Unleash your productivity, the Agile Way.

Best wishes for better Fridays.

Categories: Architecture, Programming

How to Keep flowtype Running and Report Errors on Save

Xebia Blog - Tue, 06/21/2016 - 08:00
We use flow from Facebook to run type checking on our codebase. When you run ‘flow status’ it starts a flow server in the background and keeps it running. That way after the first run the results of each next run are almost instant. The only thing currently lacking is a watch mode, but there

The Technology Behind Apple Photos and the Future of Deep Learning and Privacy

There’s a war between two visions of how the ubiquitous AI assisted future will be rendered: on the cloud or on the device. And as with any great drama it helps the story along if we have two archetypal antagonists. On the cloud side we have Google. On the device side we have Apple. Who will win? Both? Neither? Or do we all win?

If you would have asked me a week ago I would have said the cloud would win. Definitely. If you read an article like Jeff Dean On Large-Scale Deep Learning At Google you can’t help but be amazed at what Google is accomplishing. Impressive. Wide ranging. Smart. Systematic. Dominant.

Apple has been largely absent from the trend of sprinkling deep learning fairy dust on their products. This should not be all that surprising. Apple moves at their own pace. Apple doesn’t reach for early adopters, they release a technology when it’s a win for the mass consumer market.

There’s an idea because Apple is so secretive they might have hidden away vast deep learning chops we don’t even know about yet. We, of course, have no way of knowing.

What may prove more true is that Apple is going about deep learning in a different way: differential privacy + powerful on device processors + offline training with downloadable models + a commitment to really really not knowing anything personal about you + the deep learning equivalent of perfect forward secrecy.

Photos vs Photos
Categories: Architecture

Stuff The Internet Says On Scalability For June 17th, 2016

Hey, it's HighScalability time:


You've seen the Netflix Death Star microservices map. Here's a map of microbes conversing on your skin.

 

If you like this sort of Stuff then please support me on Patreon.
  • 4281: # of unread articles in my HackerNews feed; 23%: of all corporate cash is held by Microsoft, Apple, Google; 400 million: number of new servers needed by 2020; ~25,740TB: storage Backblaze adds per month; 3 bits: IBM stores per memory cell; 488 million: faked comments by China per year; 90%: revenue Spotify makes fron 30% of users; 780 million: miles of Tesla driving data; 4 days: median time to binge watch a season on Netlix; $33: cost of Nike Air Max; $50 billion: amount Apple has paid out to app developers; $270 million: amount Line makes from selling stickers; 4,600: # of trees Apple will plant aorund the Spaceship; 200 million: Google photos users; $1.8 billion: Series F round for Snapchat; 3x: capacity of the roadway with driverless cars; 138%: growth in Alibaba's cloud; 

  • Quotable Quotes:
    • @swardley: By this time next year, AMZN should be comfortably worth more than IBM + HPQ + HPE + CISCO + VMW + ORCL + NetApp + EMC combined.
    • @pcalcado: The #serverless revolution, or as we call it in my hometown: "JavaScript folks finally find out about CGI"
    • @kellan: Those patents you filed at Y! that would "only ever be used defensively"? Up for sale
    • @mipsytipsy: oooo, the term FaaS (Functions as a Service) is WAY better than #serverless.  Accurate, doesn't irritate me at all!
    • Ian Eyberg:  We need a new paradigm. We really need to be deploying software not the systems the software lives on. We need to be configuring software not the system that they are on. This is a new way of thinking.
    • @amernetflix: 2 Million #PacketsPerSecond on a #aws public #cloudinstance. 
    • @antirez: /me hopes that because at Redis Conf there were already folks telling me “don’t bother too much with Docker, we are moving away”.
    • @swardley: Asked "Do I not like Docker?" - I like Docker but I view containers as an important but for most, ultimately invisible subsystem ...
    • @jeffjarvis: Before AMP, 51% of WaPost users returned in 7 days; after AMP that's up to 63% says David Merrell at #io16
    • @HackerNewsOnion: How LinkedIn Scaled To Billions Of Unread Messages
    • @nzben: 90% Domain Driven Design 10% Switch Statements
    • @igrigorik: 0.3s latency improvement → £8M revenue gain for thetrainline(.com): http://bit.ly/1NiFzdP  - great case study.
    • @thegrugq: A monk asked Satoshi: “Why do you not sign a challenge msg?” Satoshi answered: “signing proves nothing!” The monk was englightened
    • #bitkoans
    • @johnrobb: Stanford:  Apple could manufacture its products in the US with robotic assembly at same price as China now
    • @Khanoisseur: Amazon changes prices 2.5 million times a day. Wal-Mart and Best Buy change prices 50,000 times over entire month.
    • @Khanoisseur: Amazon pushes code live every 12 seconds and can test a feature on 5000 users by turning it on for just 45 seconds
    • southpolesteve: I have one hard example I can share. We had a node service that was running on ec2 and cost ~$2500/mo. Moved the code directly over to lambda. Now ~$400/mo.
    • Keith Chen~ The behavioural economist at UCLA said users are willing to accept surge pricing increases as high as 9.9 times the normal price of a ride if their smartphone's battery is close to dying.
    • @Bill_Gross: With its always on cellular connection, every 10 hours Tesla gets a MILLION miles of additional data
    • @timallenwagner: Nice example of converting batch system to real-time serverless analytics!
    • @danielbryantuk: "In one study 35% of catastrophic failures were caused by what I call 'dev laziness' " @caitie #qconnewyork
    • @dechampsgu~ "Improving Anything but The Bottleneck is Close to Meaningless" @ziobrando  #dddx
    • @heinrichhartman: "We [#SQLite] don't compete against Oracle, we compete against fopen(3)" Richard Hipp on @changelog https://changelog.com/201/  (47:00)
    • @Pinboard: Any complex website is an interdependent ballet of dozens of mutually supporting services. You can’t reduce it to a word like “up” or “down”
    • @kevinmarks: “The web is already decentralized,” Mr. Berners-Lee said. “The problem is the dominance of one search engine…
    • CHARITY.WTF: I’ve seen what happens when application developers think they don’t have to care about the skills associated with operations engineering.  When they forget that no matter how pretty the abstractions are, you’re still dealing with dusty old concepts like “persistent state” and “queries” and “unavailability” and so forth, or when they literally just think they can throw money at a service to make it go faster because that’s totally how services work.
    • @beaucronin: A dollar-an-hour EC2 instance today would have been at or near the top of Top500 in the early/mid 90s
    • @cloud_opinion: Is this the real ops? / Is this just devops? / Caught in a deploy / No escape from techdebt / Open your eyes / Look up to the cloud and see #serverles
    • lostcolony: In context, I'm pretty sure it was just saying "Unlike computation, bandwidth, and memory size, we haven't seen much improvement in latency, and even if we focused on it, we have a very clear limit we can't get past".
    • People have said a lot more Stuff. Read the full article to see what've they said.

  • You know all those elaborate A/B testing and new feature testing systems built into websites so companies can gather data and learn more about how their product works in the real world? It's not just for websites. Tesla Tests Self-Driving Functions with Secret Updates to Its Customers’ Cars. It appears by deploying actual cars with real drivers Tesla has the Big Data advantage when it comes to creating self-driving cars. Though like much of the modern world it's damn spooky: Tesla can pull down data from the sensors inside its customers’ vehicles to see how people are driving and the road and traffic conditions they experience. It uses that data to test the effectiveness of new self-driving features. The company even secretly tests new autonomous software by remotely installing it on customer vehicles so it can react to real road and traffic conditions, without controlling the vehicle.

  • Can you be a Libertarian and use the government as an amplification attack on an enemy? The Stunning and Expected End of Gawker.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Filtering objects to Optionals

Xebia Blog - Fri, 06/17/2016 - 16:36
A while ago I stumbled upon a Blog post by Natascha the Robot about Configuring a Constant Using Shorthand Argument Names in Swift. Which by itself is a great post, but I was most inspired by the Then library mentioned at the end of her post. Seeing how such a small amount of code could change the way we configure

Running Jenkins jobs in Docker containers

Agile Testing - Grig Gheorghiu - Fri, 06/17/2016 - 00:08
One of my main tasks at work is to configure Jenkins to act as a hub for all the deployment and automated testing jobs we run. We use CloudBees Jenkins Enterprise, mostly for its Role-Based Access Control plugin, which allows us to create one Jenkins folder per project/application and establish fine grained access control to that folder for groups of users. We also make heavy use of the Jenkins Enterprise Pipeline features (which I think are also available these days in the open source version).

Our Jenkins infrastructure is composed of a master node and several executor nodes which can run jobs in parallel if needed.

One pattern that my colleague Will Wright and I have decided upon is to run all Jenkins jobs as Docker containers. This way, we only need to install Docker Engine on the master node and the executor nodes. No need to install any project-specific pre-requisites or dependencies on every Jenkins node. All of these dependencies and pre-reqs are instead packaged in the Docker containers. It's a simple but powerful idea, that has worked very well for us. One of the nice things about this pattern is that you can keep adding various types of automated tests. If it can run from the command line, then it can run in a Docker container, which means you can run it from Jenkins!

I have seen this pattern discussed in multiple places recently, for example in this blog post about "Using Docker for a more flexible Jenkins".

Here are some examples of Jenkins jobs that we create for a given project/application:
  • a deployment job that runs Capistrano in its own Docker container, against targets in various environments (development, staging, production); this is a Pipeline script written in Groovy, which can call other jobs below
  • a Web UI testing job that runs the Selenium Python WebDriver and drives Firefox in headless mode (see my previous post on how to do this with Docker)
  • a JavaScript syntax checking job that runs JSHint against the application's JS files
  • an SSL scanner/checker that runs SSLyze against the application endpoints
We also run other types of tasks, such as running an AWS CLI command to perform certain actions, for example to invalidate a CloudFront resource. I am going to show here how we create a Docker image for one of these jobs, how we test it locally, and how we then integrate it in Jenkins.

I'll use as an example a simple Docker image that installs the AWS CLI package and runs a command when the container is invoked via 'docker run'.
I assume you have a local version of Docker installed. If you are on a Mac, you can use Docker Toolbox, or, if you are lucky and got access to it, you can use the native Docker for Mac. In any case,  I will assume that you have a local directory called awscli with the following Dockerfile in it:
FROM ubuntu:14.04
MAINTAINER You Yourself <you@example.com>
# disable interactive functionsARG DEBIAN_FRONTEND=noninteractiveENV AWS_ACCESS_KEY_ID=""ENV AWS_SECRET_ACCESS_KEY=""ENV AWS_COMMAND=""
RUN apt-get update && \    apt-get install -y python-pip && \    pip install awscli
WORKDIR /root
CMD (export AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID; export AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY; $AWS_COMMAND)
As I mentioned, this simply installs the awscli Python package via pip, then runs a command given as an environment variable when you invoke 'docker run'. It also uses two other environment variables that contain the AWS access key ID and secret access key. You don't want to hardcode these secrets in the Dockerfile and have them end up on GitHub.
The next step is to build an image based on this Dockerfile. I'll call the image awscli and I'll tag it as local:

$ docker build -t awscli:local .

Then you can run a container based on this image. The command line looks a bit complicated because I am passing (via the -e switch) the 3 environment variables discussed above:


$ docker run --rm -e AWS_ACCESS_KEY_ID=YOUR_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=YOUR_SECRET_ACCESS_KEY -e AWS_COMMAND='aws cloudfront create-invalidation --distribution-id=abcdef --invalidation-batch Paths={Quantity=1,Items=[/media/*]},CallerReference=my-invalidation-123456' awscli:local
(where distribution-id needs to be the actual ID of your CloudFront distribution, and CallerReference needs to be unique per invalidation)

If all goes well, you should see the output of the 'aws cloudfront create-invalidation' command.

In our infrastructure, we have a special GitHub repository where we check in the various folders containing the Dockerfiles and any static files that need to be copied over to the Docker images. When we push the awscli directory to GitHub for example, we have a Jenkins job that will be notified of that commit and that will build the Docker image (similarly to how we did it locally with 'docker build'), then it will 'docker push' the image to a private AWS ECR repository we have.

Now let's assume we want to create a Jenkins job that will run this image as a container. First we define 2 secret credentials, specific to the Jenkins folder where we want to create the job (there are also global Jenkins credentials that can apply to all folders). These credentials are of type "Secret text" and contain the AWS access key ID and the AWS secret access key.

Then we create a new Jenkins job of type "Freestyle project" and call it cloudfront.invalidate. The build for this job is parameterized and contains 2 parameters: CF_ENVIRONMENT which is a drop-down containing the values "Staging" and "Production" referring to the CloudFront distribution we want to invalidate; and CF_RESOURCE, which is a text variable that needs to be set to the resource that needs to be invalidated (e.g. /media/*).

In the Build Environment section of the Jenkins job, we check "Use secret text(s) or file(s)" and add 2 Bindings, one for the first secret text credential containing the AWS access key ID, which we save in a variable called AWS_ACCESS_KEY_ID, and the other one for the second secret text credential containing the AWS secret access key, which we save in a variable called AWS_SECRET_ACCESS_KEY.

The Build section for this Jenkins job has a step of type "Execute shell" which uses the parameters and variables defined above and invokes 'docker run' using the path to the Docker image from our private ECR repository:

DISTRIBUTION_ID=MY_CF_DISTRIBUTION_ID_FOR_STAGING
if [ $CF_ENVIRONMENT == "PRODUCTION" ]; then
    DISTRIBUTION_ID=MY_CF_DISTRIBUTION_ID_FOR_PRODUCTION
fi

INVALIDATION_ID=jenkins-invalidation-`date +%Y%m%d%H%M%S`

COMMAND="aws cloudfront create-invalidation --distribution-id=$DISTRIBUTION_ID --invalidation-batch Paths={Quantity=1,Items=[$CF_RESOURCE]},CallerReference=$INVALIDATION_ID"

docker run --rm -e AWS_ACCESS_KEY_ID=$ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY=$SECRET_ACCESS_KEY -e AWS_COMMAND="$COMMAND" MY_PRIVATE_ECR_ID.dkr.ecr.us-west-2.amazonaws.com/awscli


When this job is run, the Docker image gets pulled down from AWS ECR, then a container based on the image is run and then removed upon completion (that's what --rm does, so that no old containers are left around).

I'll write another post soon with some more examples of Jenkins jobs that we run as Docker containers to do Selenium test, JSHint testing and SSLyze scanning.






The Image Optimization Technology that Serves Millions of Requests Per Day

This article will touch upon how Kraken.io built and scaled an image optimization platform which serves millions of requests per day, with the goal of maintaining high performance at all times while keeping costs as low as possible. We present our infrastructure as it is in its current state at the time of writing, and touch upon some of the interesting things we learned in order to get it here.

Let’s make an image optimizer

You want to start saving money on your CDN bills and generally speed up your websites by pushing less bytes over the wire to your user’s browser. Chances are that over 60% of your traffic are images alone.

Using ImageMagick (you did read ImageTragick, right?) you can slash down the quality of a JPEG file with a simple command:

$ convert -quality 70 original.jpg optimized.jpg

$ ls -la

-rw-r--r--  1 matylla  staff  5897 May 16 14:24 original.jpg

-rw-r--r--  1 matylla  staff  2995 May 16 14:25 optimized.jpg

Congratulations. You’ve just brought down the size of that JPEG by ~50% by butchering it’s quality. The image now looks like Minecraft. It can’t look like that - it sells your products and services. Ideally, images on the Web should have outstanding quality and carry no unnecessary bloat in the form of excessively high quality or EXIF metadata.

You now open your favourite image-editing software and start playing with Q levels while saving a JPEG for the Web. It turns out that this particular image you test looks great at Q76. You start saving all your JPEGs with quality set to 76. But hold on a second… Some images look terrible even with Q80 while some would look just fine even at Q60.

Ok. You decide to automate it somehow - who wants to manually test the quality of millions of images you have the “privilege” of maintaining. So you create a script that generates dozens of copies of an input image at different Q levels. Now you need a metric that will tell you which Q level is perfect for a particular image. MSE? SSIM? MS-SSIM? PSNR? You’re so desperate that you even start calculating and comparing perceptual hashes of different versions of your input image.

Some metrics perform better than others. Some work well for specific types of images. Some are blazingly fast while the others take a long time to complete. You can get away by reducing the number of loops in which you process each image but then chances are that you miss your perfect Q level and the image will either be heavier than it could be or quality degradation will be too high.

And what about product images against white backgrounds? You really want to reduce ringing/haloing artifacts around the subject. What about custom chroma-subsampling settings on per-image basis? That red dress against white background looks all washed-out now. You’ve learned that stripping EXIF metadata will bring the file size down a bit but you’ve also removed Orientation tag and now your images are all rotated incorrectly.

And that’s only the JPEG format. For your PNGs probably you’d want to re-compress your 7-Zip or Deflate compressed images with something more cutting-edge like Google’s Zopfli. You spin up your script and watch the fan on your CPU start to melt...

Categories: Architecture

Sponsored Post: Gusto, Awake Networks, Spotify, Telenor Digital, Kinsta, Aerospike, InMemory.Net, VividCortex, MemSQL, Scalyr, AiScaler, AppDynamics, ManageEngine, Site24x7

Who's Hiring?
  • IT Security Engineering. At Gusto we are on a mission to create a world where work empowers a better life. As Gusto's IT Security Engineer you'll shape the future of IT security and compliance. We're looking for a strong IT technical lead to manage security audits and write and implement controls. You'll also focus on our employee, network, and endpoint posture. As Gusto's first IT Security Engineer, you will be able to build the security organization with direct impact to protecting PII and ePHI. Read more and apply here.

  • Awake Networks is an early stage network security and analytics startup that processes, analyzes, and stores billions of events at network speed. We help security teams respond to intrusions with super-human  efficiency and provide macroscopic and microscopic insight into the networks they defend. We're looking for folks that are excited about building systems that handle scale in a constrained environment. We have many open-ended problems to solve around stream-processing, distributed systems, machine learning, query processing, data modeling, and much more! Please check out our jobs page to learn more.

  • Site Reliability Engineer Manager. We at Spotify are looking for an engineering leader (Chapter Lead) to manage the NYC part of the Site Reliability Engineering team. This team works with the infrastructure that powers the music service used by millions of users, built by hundreds of engineers. We create tools, develop infrastructure, and teach good practices to help Spotify engineers move faster. As a Chapter Lead your primary responsibility is to the people on your team: ensuring that the members are growing as engineers, doing valuable work, performing well, and generally having a great time at Spotify. Read more and apply here

  • Site Reliability Engineer. Spotify SREs design, code, and operate tools and systems to reduce the amount of time and effort necessary for our engineers to scale the world’s best music streaming product to 40 million users. We are strong believers in engineering teams taking operational responsibility for their products and work hard to support them in this. We work closely with engineers to advocate sensible, scalable, systems design and share responsibility with them in diagnosing, resolving, and preventing production issues. Read more and apply here

  • Backend Engineer. We at Spotify are looking for senior backend engineers to join our team of talented engineers that share a common interest in distributed backend systems, their scalability and continued development.  You will build the backend systems that power our application, scale highly distributed systems, and continuously improve our engineering practices. Read more and apply here

  • Security Engineer. The security team at Spotify is a distributed team supporting autonomous development teams with a focus on raising security awareness, sharing responsibility, and building tools. We aim to constantly improve the security posture for our fast-paced, rapidly-changing environment in a manner that will keep up with our scale. We’re knowledgeable in many domains of security and are willing to teach (and learn) from anyone at the company. Read more and apply here

  • Data Architect. You will be a key figure in a rapidly growing team, where the role will highly depend on you. You must have extensive experience in Cloud Computing and AWS and deeply understand databases and/or Information Architecture (PostgreSQL, Cassandra, MongoDB, Redis, etc.). And if you also know your way in the Hadoop ecosystem (including Spark and HDFS), Kafka, Cassandra and other big data technologies, this will be more than enough. You have an understanding of how to structure the data sources and data feeds of the Data Insights big data solution, plan for integration and maintenance of the data as well as have an eye on the logical design and on how the data flows through the different stages. Please apply here at Telenor Digital.

  • Data Engineers. You know Java, and possibly Clojure or Scala, are effective in a Linux terminal (shell scripting, configuration files, etc.), have experience with some SQL database, preferably PostgreSQL, have experience with Apache Kafka, Apache Spark, Elasticsearch. You enjoy automating things and building systems. Machine learning experience is considered a plus, and Continuous Integration + delivery is important to you, and writing tests a given. You are humble and passionate; you like to listen and can understand the viewpoints of others and strive to be a good dialog partner, but you can focus on delivery once a direction is decided. Please apply here at Telenor Digital.

  • Software Engineers, Analytics. You've got strong front-end developer skills: HTML, CSS, and Javascript, with knowledge of D3.js or other charting libraries - Clojurescript is a plus; have worked with various programming languages, like Java, Clojure, or Python; have experience with SQL (PostgreSQL). You have experience with Cloud Computing, especially with AWS, a deep foundation in computer science; data structures, algorithms and programming languages, as well as networking and concurrency; exposure to architectural patterns of a large, high-scale web applications; experience with shell scripting, configuration files, etc. and enjoy automating things and building systems. Please apply here at Telenor Digital.

  • Software Engineer (DevOps). You are one of those rare engineers who loves to tinker with distributed systems at high scale. You know how to build these from scratch, and how to take a system that has reached a scalability limit and break through that barrier to new heights. You are a hands on doer, a code doctor, who loves to get something done the right way. You love designing clean APIs, data models, code structures and system architectures, but retain the humility to learn from others who see things differently. Apply to AppDynamics

  • Software Engineer (C++). You will be responsible for building everything from proof-of-concepts and usability prototypes to deployment- quality code. You should have at least 1+ years of experience developing C++ libraries and APIs, and be comfortable with daily code submissions, delivering projects in short time frames, multi-tasking, handling interrupts, and collaborating with team members. Apply to AppDynamics
Fun and Informative Events
  • NoSQL Databases & Docker Containers: From Development to Deployment. What is Docker and why is it important to Developers, Admins and DevOps when they are using a NoSQL database? Find out in this on-demand webinar by Alvin Richards, VP of Product at Aerospike, the enterprise-grade NoSQL database. The video includes a demo showcasing the core Docker components (Machine, Engine, Swarm and Compose) and integration with Aerospike. See how much simpler Docker can make building and deploying multi-node, Aerospike-based applications!  

  • Discover the secrets of scalability in IT. The cream of the Amsterdam and Berlin tech scene are coming together during TechSummit, hosted by LeaseWeb for a great day of tech talk. Find out how to build systems that will cope with constant change and create agile, successful businesses. Speakers from SoundCloud, Fugue, Google, Docker and other leading tech companies will share tips, techniques and the latest trends in a day of interactive presentations. But hurry. Tickets are limited and going fast! No wonder, since they are only €25 including lunch and beer.
Cool Products and Services
  • Kinsta provides high speed, automatically scalable managed WordPress hosting services for businesses large and small. All servers run on Google Cloud and all individual sites are completely compartmentalized using the latest LXD technology. All sites include powerful SSH access and tools like Git and WP-CLI are available out-of-the-box.

  • Turn chaotic logs and metrics into actionable data. Scalyr is a tool your entire team will love. Get visibility into your production issues without juggling multiple tools and tabs. Loved and used by teams at Codecademy, ReturnPath, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex measures your database servers’ work (queries), not just global counters. If you’re not monitoring query performance at a deep level, you’re missing opportunities to boost availability, turbocharge performance, ship better code faster, and ultimately delight more customers. VividCortex is a next-generation SaaS platform that helps you find and eliminate database performance problems at scale.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • aiScaler, aiProtect, aiMobile Application Delivery Controller with integrated Dynamic Site Acceleration, Denial of Service Protection and Mobile Content Management. Also available on Amazon Web Services. Free instant trial, 2 hours of FREE deployment support, no sign-up required. http://aiscaler.com

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network.

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

How Smart is Your City Transportation?

How easy is it to get around in your city from point A to point B?

Here’s an interesting article that rounds up some of the latest ideas:

Getting Around in European capitals: How smart is your city?

I really like this—talk about impact:

Autolib’ has taken thousands of cars off the roads, brought down driving costs by 90% and is reducing pollution by millions of metric tons per year.

Dense city + mass transit creates opportunities.

According to the article, here are what some cities are doing:

  1. In London, Transport of London implemented a contactless payment system, so users can just “touch in and out” to pay. When you’re dealing with a billion commuters a year, that’s a big deal.   Using Internet-of-Things, developers can use the sensors across London’s transport system, along with meaningful data in the Cloud, to build better transport apps that address technical incidents and protect passengers in new ways.
  2. In Paris, the Internet-of-Things made it possible to create Autolib, an electronic car-sharing solution. The fleet of electronic cars is managed centrally in the Cloud, allowing users to rent cars from kiosks and easily find charging stations. And users can easily find parking, too, with GPS-enabled parking.
  3. In Barcelona, they are using Internet-of-Things to improve Bicing, their bicycle sharing program. They can use sensors to monitor bicycle usage and detect issues between supply and demand. They can use that insight to distribute bikes better so that the bikes can be used in a more sustainable way. It’s smart logistics for bicycles in action.
  4. In Helsinki, they are using Internet-of-Things to get more value out of their 400 buses. By measuring acceleration, speed, engine temperature, fuel consumption, brake performance, and GPS location, they reduce fuel consumption, improve driver performance, and provide safer bus rides.

I also like how the article framed the challenge right up front by painting the scene of a common scenario where you have to stitch together various modes of transport to reach your destination:

“You just need to take Bus 2 for three stops,
then change to Bus 8 towards the City station,
walk for 10 minutes towards the docks,
then take Line 5 on the metro for 5 stops.
Then call a taxi.”

You can imagine all the opportunities to reimagine how people get around, and how inclusive the design can be (whether that means helping blind people safely find their next stop, or helping somebody from out of town, navigate their way around.)

Depending on how big the city is and how far out the city is spread, there is still room for Uber and Lyft to help stitch the end-to-end mass transit journey together.

And I wonder how long before Amazon’s Now drivers, go from local residents that fulfill orders, to become another ride share option (do Uber drivers become Amazon Now or do Amazon Now become Uber drivers?).

Categories: Architecture, Programming

Introducing Structurizr Express

Coding the Architecture - Simon Brown - Mon, 06/06/2016 - 13:35

I rolled out a new feature to Structurizr at the weekend called Structurizr Express, which is basically a way to create software architecture diagrams using text. Although the core concept behind Structurizr is to create a software architecture model using code, there are times when you simply want a quick diagram, perhaps for a presentation, pre-sales proposal, etc. Structurizr Express will let you do just that - quickly create a single software architecture diagram using a textual definition. Much like tools such as PlantUML, yUML, WebSequenceDiagrams, etc.

Structurizr Express

Despite the name, this is all still based around the C4 model although it only targets one diagram at a time. The three types of diagrams currently supported are System Context, Container and Component diagrams. Structurizr Express is available to use now and the help page provides a description and examples of the syntax. I hope you find it useful.

Categories: Architecture

Behind the Scenes: A Minimal Viable Setup for Creating Video Scribe

Xebia Blog - Thu, 06/02/2016 - 16:28
I'm getting a lot of questions about my previous blog post. Fortunately also about the content, but mostly about how I created the video. So in this episode we will look at the MVP (Minimal Viable Product) version of a video scribe and the lessons learned. This way you can make a better video scribe based on