Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Stuff The Internet Says On Scalability For February 24th, 2017

Hey, it's HighScalability time:

 

Great example of Latency As A Pseudo-Permanent Network Partition. A slide effectively cleaved Santa Cruz from the North Bay by slowing traffic to a crawl.
If you like this sort of Stuff then please support me on Patreon.
  • 40 TFLOPS: on Lambda; 7: new habitable planets with good beer; dozens: balloons needed in Loon network; 500 TB/sec: rate at which DNA is copied in human body; 1/2: web is encrypted; 34: regions in Azure; $8k: cost of Tesla self-driving hardware; 99.95%: DMCA takedowns are bot BS; 300 nanometers: new microscope; 7%: AMP traffic to publishers; 

  • Quotable Quotes:
    • @jasonlk: Elon Musk: Self-Driving Car Revolution Will Leave 15% of World Population Without Jobs
    • Near death Archimedes: Stand away, fellow, from my diagram!
    • rumpelstilskin21: Angular and React make for popular headlines on reddit but unless you are working for a major, large web site where such things might be deemed useful by management (and no one else) then quit trying to get educated by the amateurs on reddit.
    • StorageMojo: There is a new paradigm about to hit the industry, which will eviscerate large portions of the current storage ecosystem. Like other major shifts, it is powered by a class of users who are poorly served by existing products and technologies. But if our digital civilization is to survive and prosper, it has to happen. And it will, like it or not.
    • ThatMightBePaul: Worst case scenario: you try Go, don't like it, and you head back to Node more confident that it fits you better. That's still a pretty positive outcome, imo. So, invest the time in Go, and then see which feels right :)
    • Russ: it is the job of the application to properly figure out the network’s limits and try to live within them.
    • World's Second-Best Go Player: After humanity spent thousands of years improving our tactics, computers tell us that humans are completely wrong. I would go as far as to say not a single human has touched the edge of the truth of Go.
    • @mjpt777: After fixing a few more false sharing issues we shaved another ~350ns of Aeron's RTT between machines.
    • @thomasfuchs: 1997: Let’s make a website! *fires up vi* 2007: Let’s make a website! *downloads jQuery* *fires up vi* 2017: Let’s make a website! [very long list of tech]
    • Basho: Do not follow the ancient masters, seek what they sought.
    • hellofunk: If many years ago, someone told me that a humongous company named Alphabet was thinking about deploying balloons all over the world, I'd have told you a thing or two about having a charming imagination. 
    • Russ: Sure, the Internet is broken. But anything we invent will, ultimately, be broken in some way or another. Sure the IETF is broken, and so is open source, and so is… whatever we might invent next. We don’t need a new Internet, we need a little less ego, a lot less mud slinging, and a lot more communication. 
    • @sAbakumoff: Analyzed the sentiment of 80000 Github Commit Comments, it seems that Ruby devs tend to be pretty positive, but c++ are angriest ones!
    • Michael Sawyer: The YouTubers' common enemy is YouTube
    • @jannis_r: "Good size for a microservice: if it fits into one engineers head" @adrianco #AWSTechBreakfast
    • packagecloud: setting [TZ] environment variable can save thousands (or in some cases, tens of thousands) of unnecessary system calls that can be generated by glibc over small periods of time. 
    • @istanboolean: "Hardware has stopped getting faster. Software has not stopped getting slower." @rob_pike
    • Greg Meddles: You're out of memory on some particular Amazon instance, so you bump up to the next biggest in size. That is always the naive solution. Whatever you're doing, you'll usually end up doing more of it. Eventually, you'll end up throwing good money after bad.
    • @viktorklang: Replace the use of sequential, concurrent, and parallel with dependent, coordinated, and independent? Thoughts?
    • Coast Guard Vice Adm. Marshall Lytle: Cyberwarfare is like a soccer game with all the fans on the field with you and no one is wearing uniforms
    • CockroachDB: If you’re serious about building a company around open source software, you must walk a narrow path: introduce paid features too soon, and risk curtailing adoption. Introduce paid features too late, and risk encouraging economic free riders. Stray too far in either direction, and your efforts will ultimately continue only as unpaid open source contribution
    • Veratyr: Deployment [of k8s] is just so much harder than it should be. Fundamentally (I discovered far later on in the process), Kubernetes is comprised of roughly the following services: kube-apiserver, kubelet, kube-proxy, kube-scheduler, kube-controller-manager. The other dependencies are: A CA infrastructure for certificate based authentication, etcd, a container runtime (rkt or Docker) and CNI.
    • @jbeda: I want to go on record: the amount of yaml required to do anything in k8s is a tragedy. Something we need to solve. 

  • What do you get for $5? Quite a lot. $5 Showdown: Linode vs. DigitalOcean vs. Amazon Lightsail vs. Vultr: Linode’s new plan is not only offering the consistently better performance...Linode is still a bit behind the curve when it comes to things like block storage volumes, default SSH keys and yeah, their UI.

  • Another wonderful engineering post from Riot Games. Under the hood of the League Client's Hextech UI: Any given build of the League client is expressed as a list of units called plugins... Back-end plugins that deal purely with data are written as C++ REST microservices...front-end plugins that deal with presentation are written as Javascript client applications and run inside Chromium Embedded Framework...The League client update really is a desktop deployment of an entire constellation of microservices...APIs are thoughtfully designed, any arbitrary combination of features can run cooperatively...In the League client, the common pattern is for dependencies to flow upwards...a WebSocket that allows the front-end plugins to observe back-end plugins for changes...To make implementation of complex video-based elements simpler, we created a state machine library based on Web Components...League client is patched out to players’ local drives, it doesn’t have the same immediate bandwidth constraints...we provide a number of purpose-specific audio channels - UI SFX, Notifications, Music, Voiceover, etc. - through a plugin dedicated to managing audio...We use straight-up native Custom Elements with heavy usage of Shadow DOM.

  • Does insurance cover this? The first SHA1 collision.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

14 extensions that can enrich your daily VSTS usage

Xebia Blog - Wed, 02/22/2017 - 22:54
Using VSTS on a daily basis I find that I add a regular list of VSTS Marketplace extensions to my VSTS environment. I find them convenient and helping me to get the most out of VSTS. The list below is primarily focussed on the Work and Code area and not so much on the Build

New features in Xcode 8.2 Simulator

Xebia Blog - Wed, 02/22/2017 - 14:01
In the release notes of Xcode 8.2, Apple introduced features for their new version of Xcode. In this blog I will explain how to use these new features. Read more

New features in Xcode 8.2 Simulator

Xebia Blog - Wed, 02/22/2017 - 14:01
In the release notes of Xcode 8.2, Apple introduced features for their new version of Xcode. In this blog I will explain how to use these new features. Read more

First Steps in gRPC Bindings for React Native

Xebia Blog - Wed, 02/22/2017 - 13:54
When you want to use gRPC in your React Native app there is no official support yet, but that shouldn’t stop you! In this post I’ll show you how we designed an implementation with type safety in mind and successfully called a service remotely from React Native on Android. Read more

First Steps in gRPC Bindings for React Native

Xebia Blog - Wed, 02/22/2017 - 13:54
When you want to use gRPC in your React Native app there is no official support yet, but that shouldn’t stop you! In this post I’ll show you how we designed an implementation with type safety in mind and successfully called a service remotely from React Native on Android. Read more

Scaling @ HelloFresh: API Gateway

HelloFresh keeps growing every single day: our product is always improving, new ideas are popping up from everywhere, our supply chain is being completely automated. All of this is simply amazing us, but of course this constant growth brings many technical challenges.

Today I’d like to take you on a small journey that we went through to accomplish a big migration in our infrastructure that would allow us to move forward in a faster, more dynamic, and more secure way.

The Challenge

We’ve recently built an API Gateway, and now we had the complex challenge of moving our main (monolithic) API behind it — ideally without downtime. This would enable us to create more microservices and easily hook them into our infrastructure without much effort.

The Architecture
Categories: Architecture

Design by contract using GraphQL

Xebia Blog - Mon, 02/20/2017 - 17:20
When interfacing between systems it is good practice to think about the interface design prior to developing the systems. GraphQL can be a useful tool to write down these design decisions using its schema definition language. Even when you are not using GraphQL itself in production. GraphQL’s schema can be used to generate a mock

Stuff The Internet Says On Scalability For February 17th, 2017

Hey, it's HighScalability time:

 

Gorgeous satellite images of a thawing Greenland (NASA).
If you like this sort of Stuff then please support me on Patreon.
  • 1 cubic millimeter: computer with deep-Learning; 1,600: data on nearby stars; 40M: users for largest Parse app; 58x: Tensorflow 1.0 speedup on 64 gpus; 46%: ecommerce controlled by Amazon; 60%: IT growth in public cloud; 200 TB: one tv episode; 

  • Quotable Quotes:
    • @krishnan: Serverless will not be around in 5 years. It will be AI coding AI coding Ai....... Serverless or not doesn't matter #RunForrestRun
    • user5994461: Amazon: Create usual services and sell them. Google: Make unique products that push the boundaries of what was previously thought possible. Amazon: Don't care about inefficiencies and usage. Inefficiencies can be handled by charging more to the clients, usage doesn't matter because the users are mostly the clients and they don't feel their pain. Google: Had to make all their core technologies efficient, performant, scalable and maintainable or they couldn't sustain their business.
    • Hans Rosling: To me, the impressive thing is that people succeed at all.
    • @littleidea: Google Spanner didn't beat CAP, just mitigated the hell out of P
    • @jordw: Cloud Spanner is a very well-engineered CP database that is also very good at being available.
    • Cade Metz: The AI Threat Isn’t Skynet. It’s the End of the Middle Class
    • hosh: Four years ago, I determined that while development work might seem to be near the top of the food chain, there will at some point where my work will be replaced by AIs.
    • mi100hael:  I found Go's "simplicity" to be limiting and frustrating when it came to building production applications. Things like the weird split between functions returning errors but occasionally panicking, lack of inheritance, and poor dependency management through github links make Go a poor choice for applications within a business setting. 
    • @NathanTippy: New #Java web server clearing 1 million HTTP requests per second on 4 core box.  Can run in < 100MB of memory.
    • @kellabyte: It doesn’t matter what the founder or developer of a database tells you. It’s about the true peopeties it guarantees.
    • @swardley: Private cloud starting to drop, public cloud a three horse race - AWS 1st, MSFT 2nd, GooG 3rd ... sensible stuff 
    • @ollekullberg: Kullberg's law: when we increase the size of a microservice we increase the benefit of static typing for this microservice.
    • @swardley: ... it's not lack of engineering capability or finance or market or marketing or branding, the real story of cloud is executive failure.
    • katied: Trophic cascade is a process that starts at the top of a food chain and works its way to the bottom of it. So, even though as predators wolves survive by taking life, they also have the ability to create it.
    • @swardley: Cloud wars in IaaS - oh, please. War was well over in 2012, yes there will be price cuts as constraints are reduced but there is no battle.
    • @HenryR: 1. CAP has always said only one thing: that there is always a particular network failure that forces you to give up either C or A. 2. It has nothing at all to do with how likely that failure mode is. The failure is system-specific. 
    • throwawaydbfif: The movement from ownership to renting on the web is absolutely terrifying to me. Within the span of a few years we've gone from owning our technology to renting it out from a big players for monthly fees that we cannot completely predict or control.
    • computerex: People use cloud computing because it already is massively impractical to run your own servers. Hardware is hard to run and scale on your own and experiences economies of scale. This principle is seen everywhere and can hardly be viewed as something controversial. 
    • stuckagain: You did not ever own your own globally consistent, massively scalable, replicated database. The fact that you can now rent one by the hour is strictly an improvement for you, if you need that kind of thing
    • tedd4u: Aurora is very cool but won't help you much after you vertically scale your master and still need more write capacity. With Cloud Spanner you get horizontal write scalability out of the box. Critical difference.
    • @koivimik: REST != CRUD via HTTP #microXchg @olivergierke
    • Linus: It's almost boring how well our process works. All the really stressful times for me have been about process. They haven't been about code. When code doesn't work, that can actually be exciting ... Process problems are a pain in the ass. You never, ever want to have process problems ... That's when people start getting really angry at each other.
    • @littleidea: Almost every task run under Borg contains a built-in HTTP server that publishes information about the health of the task...
    • W. Daniel Hillis: For Richard [Feynman], figuring out these problems was a kind of a game. He always started by asking very basic questions like, “What is the simplest example?” or “How can you tell if the answer is right?” He asked questions until he reduced the problem to some essential puzzle that he thought he would be able to solve.
    • @ewolff: "Every hackathon uses Lambda. They build really complicated, production-ready systems in 12h" @adrianco at @microXchg
    • Daniel Bryant: The term "microservices" itself will probably disappear in the future, but the new architectural style of functional decomposition is here to stay.
    • @rbranson: The NoSQL movement might be a disappointment, but emerging from the rubble is the log-based (i.e. Kafka) model that actually works.
    • Chip Overclock: Surprisingly, GPS satellites actually know nothing about position. What they know about is time.
    • @codinghorror: I look at my old blog posts and think... there was a time when I believed 24GB was a lot of RAM
    • vidarh: Depending on your workloads, DO servers can come out cheaper or more expensive than AWS, but bandwidth at DO is so much cheaper than AWS that for bandwidth intensive stuff I can't serve entirely out of Europe (where Hetzner is vastly cheaper than DO again), DO is often a much cheaper alternative. Sometimes we use it as a cost-cutting do-it-yourself CDN in front of AWS for clients that insist on S3 for storage (and again where we can't just cache everything in Europe for latency reasons). For bandwidth heavy applications, you can pay for significant numbers of Droplets from the AWS bandwidth savings alone.
    • lobster_johnson: we use Google Container Engine (hosted Kubernetes), with Salt for the non-GKE VMs. This is needed because K8s is not mature enough to host all the things. In particular, stateful sets are still in beta. 
    • anonymous: The overall impact [algorithms] will be utopia or the end of the human race; there is no middle ground foreseeable. I suspect utopia given that we have survived at least one existential crisis (nuclear) in the past and that our track record toward peace, although slow, is solid.
    • keenio: In conclusion, the TCO is probably significantly lower for Kinesis. So is the risk. And in most projects, risk-adjusted TCO should be the final arbiter.
    • Adem Efe Gencer: the weekly [Bitcoin] mining power of a single miner has never exceeded the 30% of the overall mining power in 2016. Morever, in the second half of the year, the highest mining power has consistently been under the 20% range.
    • David Rosenthal: The security downside of Postel's Law is even more fundamental. The law requires the receiver to accept, and do something sensible with, malformed input. Doing something sensible will almost certainly provide an attacker with the opportunity to make the receiver do something bad.
    • douche: That's pretty much the way it has always been. You can go back at least to the Civil War and find politics has had more to do with procurement than performance of the weapon systems in question.
    • Jonathan Suen: While the brain and the Internet clearly operate using very different mechanisms, both use simple local rules that give rise to global stability. I was initially surprised that biological neural networks utilized the same algorithms as their engineered counterparts, but, as we learned, the requirements for efficiency, robustness, and simplicity are common to both living organisms and the networks we have built.
    • Bruce Johnson: Code reviews set the tone for the entire company that everything we do should be open to scrutiny from others, and that such scrutiny should be a welcome part of your workflow rather than viewed as threatening.
    • codingmyway: I think some miners are against any increase because it will lower fees. Without a blocksize limit fees tend to zero, which is fine while there is the block reward but they still want to milk the congestion fees. To say they are pro segwit or pro unlimited is bluffing. They are pro status quo and congestion and high fees.
    • edejong: Many engineers I have worked with like to throw around terms like: "CQRS", "Event sourcing", "no schema's", "document-based storage", "denormalize everything" and more. However, when pushed, I often see that they lack a basic understanding of DBMSes, and fill up this gap by basically running away from it. For 95% of the jobs, a simple, non-replicated (but backed-up) DBMS will do just fine.
    • adamu__: If China were to shut down bitcoin mining, my understanding is that the worst case scenario is much more dire. The network only adjusts the 'difficulty' relative to current network hash power every 2,016 blocks. Depending on the severity of the overall hash power reduction, new block discovery might slow down significantly. This would also delay a recalculation of the new difficulty accommodating the reduction in hash power. The network could be severely throttled for weeks.
    • boulos: Slightly off-topic, but EC2 doesn't really scale independently if you compare it to GCE. We let you combine 24 vcpus with 39 GB of RAM, 3 partitions of Local SSD and a few GPUs, all independently (though the ratio of RAM to vcpu is currently bounded between .9 and 6.5).
    • Veratyr: Personally, I settled with colocation. I pay $60/mo + $2k one-off for the initial hardware + say $150/5y/4TB HDD, which, for 80TB of storage over 5y comes out to a total of ~$88/mo, or $0.001/GBmo. 

  • Now this is object oriented programming. New software for increasingly flexible factory processes: new software that allows each individual component to tell the machine what has to be done. By breaking away from central production planning, factories can achieve unprecedented agility and flexibility... Everything would go much faster if production and the requisite machines were not rigidly set by a control program, but if every component itself knew the best way for it to be moved quickly through the process chain. 

  • Relax. Videos from TensorFlow Dev Summit 2017 are now available. Also, Learn TensorFlow and deep learning, without a Ph.D. Also also, Deep Learning book.

  • Google is Introducing Cloud Spanner: a global database service for mission-critical applications. It will be interesting to see if Spanner, as a unique hard to duplicate feature, becomes a Google Cloud differentiator. Will it make the delta between the clouds significant enough that developers choose Google? Quizlet, already running on GCP, really likes Spanner, but it's not a drop in replacement for MySQL. Like with NoSQL there's special care and feeding to make it work, but that's the sacrifice high QPS requires. Performance: "Cloud Spanner queries have higher latency at low throughputs compared with a virtual machine running MySQL. Spanner's scalability, however, means that a high-capacity cluster can easily handle workloads that stretch our MySQL infrastructure." And p90s are consistently lower than 50 ms. Cost: "For very small or low-throughput databases Cloud Spanner is overkill [min ~$8,000/yr]...Cloud Spanner comparable or slightly cheaper based on the performance in our testing."  With Spanner hitting the market maybe that will help CockroachDB? Some older articles: Spanner - It's About Programmers Building Apps Using SQL Semantics At NoSQL ScaleGoogle Spanner's Most Surprising Revelation: NoSQL Is Out And NewSQL Is InF1 And Spanner Holistically ComparedHow Google Invented An Amazing Datacenter Network Only They Could Create

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Why don’t monitoring tools monitor changes?

Xebia Blog - Fri, 02/17/2017 - 12:38
Changes in applications or IT infrastructure can lead to application downtime. This not only hits your revenue, it also has a negative impact on your reputation. Everybody in IT understands the importance of having the right monitoring solutions in place. From an infrastructure – to a business perspective, we rely on monitoring tools to get

Sponsored Post: Aerospike, GoCardless, Auth0, InnoGames, Contentful, Stream, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?
  • GoCardless is building the payments network for the internet. We’re looking for DevOps Engineers to help scale our infrastructure so that the thousands of businesses using our service across Europe can take payments. You will be part of a small team that sets the direction of the GoCardless core stack. You will think through all the moving pieces and issues that can arise, and collaborate with every other team to drive engineering efforts in the company. Please apply here.

  • InnoGames is looking for Site Reliability Engineers. Do you not only want to play games, but help building them? Join InnoGames in Hamburg, one of the worldwide leading developers and publishers of online games. You are the kind of person who leaves systems in a better state than they were before. You want to hack on our internal tools based on django/python, as well as improving the stability of our 5000+ Debian VMs. Orchestration with Puppet is your passion and you would rather automate stuff than touch it twice. Relational Database Management Systems aren't a black hole for you? Then apply here!

  • Contentful is looking for a JavaScript BackEnd Engineer to join our team in their mission of getting new users - professional developers - started on our platform within the shortest time possible. We are a fun and diverse family of over 100 people from 35 nations with offices in Berlin and San Francisco, backed by top VCs (Benchmark, Trinity, Balderton, Point Nine), growing at an amazing pace. We are working on a content management developer platform that enables web and mobile developers to manage, integrate, and deliver digital content to any kind of device or service that can connect to an API. See job description.
Fun and Informative Events
  • DBTA Roundtable Webinar: Fast Data: The Key Ingredients to Real-Time Success. Thursday February 23, 2017 | 11:00 AM Pacific Time. Join Stephen Faig, Research Director Unisphere Research and DBTA, as he hosts a roundtable discussion covering new technologies that are coming to the forefront to facilitate real-time analytics, including in-memory platforms, self-service BI tools and all-flash storage arrays. Brian Bulkowski, CTO and Co-Founder of Aerospike, will be speaking along with presenters from Attunity and Hazelcast. Learn more and register.

  • Your event here!
Cool Products and Services
  • Working on a software product? Clubhouse is a project management tool that helps software teams plan, build, and deploy their products with ease. Try it free today or learn why thousands of teams use Clubhouse as a Trello alternative or JIRA alternative.

  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • Auth0 is the easiest way to add secure authentication to any app/website. With 40+ SDKs for most languages and frameworks (PHP, Java, .NET, Angular, Node, etc), you can integrate social, 2FA, SSO, and passwordless login in minutes. Sign up for a free 22 day trial. No credit card required. Get Started Now.

  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

Azure Functions imperative bindings

Xebia Blog - Tue, 02/14/2017 - 13:09
The standard input and output bindings in Azure Functions are written in a declarative pattern using the function.json. When defining input and output declarative, you do not have the option to change some of the bindings properties like the name or make multiple outputs from one input. An imperative binding can do this for you. In

Created an open source VSTS build & release task for Azure Web App Virtual File System

Xebia Blog - Tue, 02/14/2017 - 08:15
I’ve created a new VSTS Build & Release task to help you interact with the (VFS) Virtual File System API (Part of KUDU API of your Azure Web App). Currently this task can only be used to delete specific files or directories from the web app during your build or release workflow. It will be

Part 3 of Thinking Serverless —  Dealing with Data and Workflow Issues

This is a guest repost by Ken Fromm, a 3x tech co-founder — Vivid Studios, Loomia, and Iron.io. Here's Part 1 and 2

This post is the third of a four-part series of that will dive into developing applications in a serverless way. These insights are derived from several years working with hundreds of developers while they built and operated serverless applications and functions. The platform was the serverless platform from Iron.io but these lessons can also apply to AWS LambdaGoogle Cloud FunctionsAzure Functions, and IBM’s OpenWhisk project.

Serverless Processing — Data Diagram

Thinking Serverless! The Data
Categories: Architecture

Installing a VSTS Build agent on Mac OSX

Xebia Blog - Mon, 02/13/2017 - 16:36
Today I needed to install the VSTS Build Agent on a Mac Mini in the office so we can build the IPhone App. The great part of the new build infrastructure in VSTS is that it is Cross Platform and we CAN do that. Just download the agent and run the command provided in the interface.

Force uninstall Visual Studio 2017 Release candidates

Xebia Blog - Sun, 02/12/2017 - 11:56
If you, like me, are stuck trying to upgrade Visual Studio 2017, then you may only get unblocked by removing everything and starting afresh. Since Visual Studio 2017 is still in Release Candidate and not final, this is something we may have to deal with from time to time. But when the "uninstall" button in

Add :8080 to your TFS 2017 bindings after upgrading to SSL

Xebia Blog - Sun, 02/12/2017 - 11:40
Because TFS 2017 allows authentication with Personal Access Tokens (PAT) it's recommended to upgrade to SSL if you were still on port 80. The installer will even help with the configuration and can add a redirect from port :80 to :443. It doesn't add a a redirect from port :8080 though, so your users may

Stuff The Internet Says On Scalability For February 10th, 2017

Hey, it's HighScalability time:

 

It was a game of drones.
If you like this sort of Stuff then please support me on Patreon.
  • Half a trillion: Apple’s cash machine; 4,000-5,000: collected data points per adult in US; 10 million: gallons of gas UPS saves turning right; 2.27: Tesla 0-60 time; 40: complex steps to phone security; $2.3 billion: VR/AR investment in 2016; 18%: small players make up public cloud services market; 500°C: first chip to survive on Venus; 5 billion: ever notes; 375,000: images from The Metropolitan Museum of Art in public domain; 18 million: queries per minute against Facebook's Beringei database; 159: jobs per immigrant founder; 2.5 miles: whales breach for stronger signal; 10,000x: computers faster in 2035; 

  • Quotable Quotes: 
    • @martin_casado: Chinese factory replaces 90% of human workers with robots. Production rises by 250%, defects drop by 80%
    • Jure Leskovec: It’s [trolling] a spiral of negativity. Just one person waking up cranky can create a spark and, because of discussion context and voting, these sparks can spiral out into cascades of bad behavior. Bad conversations lead to bad conversations. People who get down-voted come back more, comment more and comment even worse.
    • sudhirj: The first concrete thing I learnt is this - implement pull first, it works 100% of the time, but may be inefficient with regards to time. Then implement push, it works 99% of the time but is much faster. But always have both running.
    • Tom Randall: California’s goal is considerable, but it’s dwarfed by Tesla’s ambition to single-handedly deliver 15 gigawatt hours 1 of battery storage a year by the 2020s—enough to provide several nuclear power plants–worth of electricity to the grid during peak hours of demand
    • @aphyr: Like I can't show that it's 100% correct, but so far I haven't found a way to break 3.4.0. Opens up a bunch of new use cases for MongoDB.
    • Azethoth666: The coming fast non-volatile memory architectures will be interesting. Everything will be in memory, but it will not go away. The infection cycle will have to clean up after itself or remain in the super fast volatile memory parts.
    • StorageMojo: In five years the specter of AWS cloud dominance will be a distant memory. The potential cloud market is enormous and we are, in effect, where the computer industry was in 1965. AWS will be successful, just not dominant. No tears for AWS.
    • @johnrobb: ~ 'Bots make public conversation a synthetic conversation. This makes it very difficult to know what consensus looks like.
    • W. Daniel Hillis: One day when I was having lunch with Richard Feynman, I mentioned to him that I was planning to start a company to build a parallel computer with a million processors. His reaction was unequivocal, “That is positively the dopiest idea I ever heard.”
    • @supershabam: Every database is a message bus if you try hard enough
    • mlechha: Boltzmann machines are a stochastic version of the Hopfield network. The training algorithm simply tries to minimize the KL divergence between the network activity and real data. So it was quite surprising when it turned out that the algorithm needed a "dream phase" as they call it. Francis Crick was inspired by this and proposed a theory of sleep.
    • @benjammingh: OH "Docker is Latin for a fire consisting predominantly of tires
    • UweSchmidt: "Real" bitcoining doesn't use services like coinbase; the coins are on your computer which you have to secure yourself. At least this is what you get told in cryptocurrency forums when one of the exchanges get hacked.
    • @axleyjc: 'Think of your System as a "Set of annotated request trees"' to manage microservice complexity @adrianco @ExpediaEng
    • @happy_roman: VW CEO on Tesla: "We'll win in the end, because of our abilities to scale & spread production."
    • aaron bell: Whichever cloud provider you pick based on your needs and their specific offering, I beg of you — please don’t try hybrid
    • zebra9978: Kubernetes introduces a lot of upfront complexity with little benefit sometimes. For example, kargo is failing with Flannel, but works with Calico (and so on and so forth). Bare metal deployments with kubernetes are a big pain because the load balancer setups have not been built for it - most kubernetes configs depend on cloud based load balancers (like ELB). In fact, the code for bare metal load balancer integration has not been fully written for kubernetes.
    • a13n: This is huge. 87-99% shared code between iOS and Android. Someday companies as big as Instagram won't need to have entire separate product teams for separate platforms.
    • David Rosenthal: I've always said that the chief threat to digital preservation is economic; digital information being very vulnerable to interruptions in the money supply. 
    • YZF: There are no channels [in C++], there are no lightweight/green threads, there's no standard HTTP library, no standard crypto libraries, no standard test framework. For certain classes of applications this makes Go significantly more productive and significantly less bug/error prone. Not to mention compile times.
    • jdwyah: Kinesis firehose to S3 and then query with Athena is pretty great. I've been very happy with the combo.
    • mcherm: Your example from RethinkDB really struck home to me. The idea that superior technology might lose out due to poor marketing or (in this case) a system that is optimized for the real world rather than being optimized for benchmarks really disturbs me.
    • Aras Pranckevičius: Moral of the story is: code that used to do something with five things years ago might turn out to be problematic when it has to deal with a hundred. And then a thousand. And a million. And a hundred billion. Kinda obvious, isn’t it?
    • kordless: I've come to a hypothesis that technology's purpose is to gently erode the concept of "self"
    • Microsoft: Close to a year ago we reset and focused on how we would actually get Git to scale to a single repo that could hold the entire Windows codebase (include estimates of growth and history) and support all the developers and build machines.
    • XorNot: I've run extensive benchmarks of Hadoop/HBase in Docker containers, and there is no performance difference. There is no stability difference (oh a node might crash? Welcome to thing which happens every day across a 300 machine cluster). Any clustered database setup should recover from failed nodes. Any regular relational database should be pretty close to automated failover with replicated backups and an alert email. Containerization doesn't make this better or worse, but it helps a lot with testing and deployment.
    • Dan Luu: When I was at Google, someone told me a story about a time that “they” completed a big optimization push only to find that measured page load times increased. When they dug into the data, they found that the reason load times had increased was that they got a lot more traffic from Africa after doing the optimizations. The team’s product went from being unusable for people with slow connections to usable, which caused so many users with slow connections to start using the product that load times actually increased.

  • There's a quintessential Silicon Valley moment in The Founder, a movie about the more interesting than expected McDonald's origin story. Brothers Mac and Dick McDonald kicked around from startup to startup. Nothing stuck. Drive-ins ruled the day, but were ripe for disruption. They were slow, used lots of servers, had too many options, attracted the wrong user base, and often delivered the wrong results. Metrics told them users mostly bought burgers, fries, and milkshakes. So the brothers decided to completely rethink how burgers were made and sold. What they came up with disrupted the food industry: a serverless drive-in based on a new low latency pipeline for making burgers called the Speedy System. An order was delivered within 30 seconds of being made; metrics helped control the latency distribution. Here's a short vignette showing how it was done. You'll love it. They traced the exact dimensions of the kitchen and conducted numerous simulations to figure out the optimal configuration. Users walk up to the window to order, so no servers. The API was narrow, only a few items could be ordered. Waste was reduced because utensils were done away with using innovative packaging design. Automation and a proprietary tool chain delivered a consistent product experience. And as often happens in Silicon Valley the founders were out maneuvered. While the McDonald brothers innovated the tech, Ray Kroc innovated the business model. Guess who ended up with everything? 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Migrate a complete Git Repository from TFS to VSTS

Xebia Blog - Fri, 02/10/2017 - 11:10
I am migrating some Git repositories from a TFS server to VSTS. Moving a Git repo including all the history is very simple fo course. Since Git is a Distributed Source Control repository. But I always forget the syntax of moving the “complete” Git repository at once. With all the branches, tags etc. So here

Running Windows Containers on Azure Service Fabric - Part II

Xebia Blog - Thu, 02/09/2017 - 12:39
The previous post showed how you can create an unsecure Service Fabric test cluster in Azure, and how to run a Windows Container on it. In this follow up post, I'll show you what's going on inside the cluster, using the Docker command line. Knowledge about this can be very useful when troubleshooting. Verify your container