Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Architecture

Stuff The Internet Says On Scalability For January 20th, 2017

Hey, it's HighScalability time:

 

Absolutely. Do we agree that the cerebellum is amazingly beautiful? (@PeppeGanga)
If you like this sort of Stuff then please support me on Patreon.
  • 900 GB: data stolen in Cellebrite hack; 99.24%: users identified by cross-browser fingerprinting; 72%: intend to migrate to a hybrid cloud; 90%: Google & Facebook ad traffic is useless; 5.2 terabytes per second: data from Australian Square Kilometre Array Pathfinder; 10 billion: searches on DuckDuckGo in 2016; $330m: Amazon's loss on Alexa; 

  • Quotable Quotes:
    • @brucel: Breaking: Programmer accused of writing unreadable code refuses to comment.
    • @asymco: Remember Android first? App Annie believes the Apple’s App Store produced about twice as much revenue as Google Play
    • @bridgetkromhout: Describing your old-timer ranting as "greybeard" just makes me want to fight you with sed & awk at twenty paces. Be there tomorrow at dawn.
    • @StevenShorrock: Root Cause Analysis is: * Acceptable for simple systems * Inappropriate for complicated systems * Ludicrous for complex systems
    • @swardley: Five years ago Amazon was worth about half of Walmart, today Walmart is worth about half of Amazon.
    • Eric Raymond: In practice, I found Rust painful to the point of unusability. The learning curve was far worse than I expected; it took me those four days of struggling with inadequate documentation to write 67 lines of wrapper code for the server.
    • @swardley: past history shows many major players won't announce they're getting into the battle until some time after war has ended
    • @benthompson: Apple wasn't billed as phone maker / Amazon wasn't billed as infrastructure provider / FB wasn't billed as portal / Snapchat wasn't billed as TV
    • Jessitron: the biggest consideration in choosing whether to use libraries or services for distribution of effort / modularization is that choice of who decides when it deploys. Who controls which code is in production at a given time.
    • Hi Ben: The disruption of TV will follow a similar path: a different category will provide better live sports, better story-telling, or better escapism. Said category will steal attention, and when TV no longer commands enough attention of enough people, the entire edifice will collapse. Suddenly.
    • @leonidasfromxiv: I also don't understand why people compare Go with Rust. If you need a GC-less programming language: Rust; if you need a board game: Go.
    • Carlo Rovelli: The world isn’t just a mass of colliding atoms; it is also a web of correlations between sets of atoms, a network of reciprocal physical information between physical systems.
    • Chris Dixon: In the beginning, hardware-focused companies make gadgets with ever increasing laundry lists of features. Then a company with strong software expertise (often a new market entrant) comes along that replaces these feature-packed gadgets with full-fledged computers. 
    • Animats: The real question is "what do we do with a lot of CPUs without shared memory?" Such hardware has been built many times - Thinking Machines, Ncube, the PS2's Cell - and has not been too useful for general purpose computing.
    • @taavet: Very unfortunate that incumbents see tech only as a way to cut costs. Versus seeing tech to offer much better products.
    • NelsonMinar: This is what security looks like when your threat model is well funded government agencies.
    • Don Norman: The solution requires a different approach to the design of automation: collaboration. Instead of automating what can be automated, leaving the rest to the driver, we must develop collaborative systems so that the driver is continually involved in giving high-level guidance, thereby always staying active, always being in the loop. 
    • Thomas Frey: It took 50 years for the world to install the first million industrial robots. The next million will take only eight. Will this cause more jobs or few jobs in the future? I'm not convinced we know the answer.
    • @jtauber: "Every shot in Piper is composed of millions of grains of sand, each one of them around 5000 polygons."
    • rackforms: my point is the current situation, basically 2 companies controlling so much traffic, seems, well, bad for small business in this country. I value what they bring to the table and fully understand why they're so popular. But is things keep on this way where does that lead the guys like me? Is this just the way it has to be? Is the dream of the open Internet already dead?
    • @sheeshee: I think I know why it's called "DevOps" - "DevOops" was too obvious... ;)
    • greenspot: The open solution to a faster mobile web would have been so easy: Just penalize large and slow web pages without defining a dedicated mobile specification. That's it. This wasn't done in the past, slow pages outperformed fast ones on the SERPs because of some weird Google voodoo ranking, heck sometimes even desktop sites outperformed responsive ones on smartphones. If they had just tweaked these odd ranking rules in way that speed and size got more impact on the overall ranking there wouldn't have been any reason for AMP—the market would have regulated itself.
    • Juergen Schmidhuber: General purpose quantum computation won’t work (my prediction of 15 years ago is still standing). Related: The universe is deterministic, and the most efficient program that computes its entire history is short and fast, which means there is little room for true randomness, which is very expensive to compute. What looks random must be pseudorandom, like the decimal expansion of Pi, which is computable by a short program. Many physicists disagree, but Einstein was right: no dice. There is no physical evidence to the contrary

  • RethinkDB is shutting down and here's the post-portem. Lessons: the database market is like Mad Max fighting in the Thunderdome; it's better to optimize for useless microbenchmarks than it is to be good; optimism isn't a strategy.

  • Apple isn't alone in using custom hardware to thwart nation state level attackers. Google Infrastructure Security Design Overview. Good overview at Google reveals its servers all contain custom security silicon. Google designs "custom chips, including a hardware security chip that is currently being deployed on both servers and peripherals. These chips allow us to securely identify and authenticate legitimate Google devices at the hardware level."  Google encrypts data before it is written to disk, to make it harder for malicious disk firmware to access data. Google uses automated and manual code review techniques. Google uses automated software and code reviews to detect bugs in software its developers write. Google scans user-installed apps, downloads, browser extensions, and content browsed from the web for suitability on corp clients. Google uses a custom version of the KVMhypervisor. Good discussion on HackerNews, where a lot of the comments are on how Google needs this level sophistication to evade the prying eyes of governments.

  • What happens when you embed machine learning into a DBMS in order to continuously optimise its runtime performance? You get Self-driving database management systems. Humans suck at tuning databases so this is just one more job AIs will eventually toss into the dust bin of history. TensorFlow was integrated inside Peleton training two RNNs on 52 million queries from one month of traffic for a popular site. Does it help?: early results are promising: (1) RNNs accurately predict the expected arrival rate of queries. (2) hardware-accelerated training has a minor impact on the DBMS’s CPU and memory resources, and (3) the system deploys actions without slowing down the application. 

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Sponsored Post: Contentful, Stream, Loupe, New York Times, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?
  • Contentful is looking for a JavaScript BackEnd Engineer to join our team in their mission of getting new users - professional developers - started on our platform within the shortest time possible. We are a fun and diverse family of over 100 people from 35 nations with offices in Berlin and San Francisco, backed by top VCs (Benchmark, Trinity, Balderton, Point Nine), growing at an amazing pace. We are working on a content management developer platform that enables web and mobile developers to manage, integrate, and deliver digital content to any kind of device or service that can connect to an API. See job description.

  • The New York Times is looking for a Software Engineer for its Delivery/Site Reliability Engineering team. You will also be a part of a team responsible for building the tools that ensure that the various systems at The New York Times continue to operate in a reliable and efficient manner. Some of the tech we use: Go, Ruby, Bash, AWS, GCP, Terraform, Packer, Docker, Kubernetes, Vault, Consul, Jenkins, Drone. Please send resumes to: technicaljobs@nytimes.com
Fun and Informative Events
  • Your event here!
Cool Products and Services
  • Build, scale and personalize your news feeds and activity streams with getstream.io. Try the API now in this 5 minute interactive tutorial. Stream is free up to 3 million feed updates so it's easy to get started. Client libraries are available for Node, Ruby, Python, PHP, Go, Java and .NET. Stream is currently also hiring Devops and Python/Go developers in Amsterdam. More than 400 companies rely on Stream for their production feed infrastructure, this includes apps with 30 million users. With your help we'd like to ad a few zeros to that number. Check out the job opening on AngelList.

  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

Wouldn't it be nice if everyone knew a little queuing theory?

After many days of rain one lane of this two lane road collapsed into the canyon. It's been out for a month and it will be many more months before it will be fixed. Thanks to Google maps way too many drivers take this once sleepy local road. 

How do you think drivers go through this chokepoint? 

 

 

One hundred experience points to you if you answered one at a time.

One at a time! Through a half-duplex pipe following a first in first out discipline takes forever!

Yes, there is a stop sign. And people default to this mode because it appeals to our innate sense of fairness. What could be fairer than alternating one at a time?

The problem is it's stupid.

While waiting, stewing, growing angrier, I often think if people just knew a little queueing theory we could all be on our way a lot faster.

We can't make the pipe full duplex, so that's out. Let's assume there's no priority involved, vehicles are roughly the same size and take roughly the same time to transit the network. Then what do you do?

Why can't people figure out its faster to drive through in batches? If we went in groups of say, three, the throughput would be much higher. And when one side's queue depth grows larger because people are driving to or from work that side's batch size should increase. 

Since this condition will last a long time we have a possibility to learn because the same people take this road all the time. So what happens if you try to change the culture by showing people what a batch is by driving right behind someone as they take their turn?

You got it. Honking. There's a simple heuristic, a deeply held ethic against line cutting, so people honk, flip you off, and generally make heir displeasure known.

It's your classic battle of reason versus norms. The smart thing is the thing we can't do by our very natures. So we all just keep doing the dumb thing.

 

Categories: Architecture

Agile Results Refresher for 2017

I’ve put together a quick refresher on Agile Results for 2017:

Agile Results Refresher for 2017

I tried to keep it simple and to the point, but at the same time, help new folks that don’t know what Agile Results is, really sink their teeth into it.

For example, one important idea is that it’s effectively a system to use your best energy for your best results.

I’ve seen people struggle with getting results for years, and one of the most common patterns I see is they use their worst energy for their most important activities.

Worse, they don’t know how to change their energy.

So now they are doing work they hate, because they feel like crap,and this feeling becomes a habit.

The irony is that they would enjoy their work if they just knew how to flip the switch and reimagine their work as an opportunity to experiment and explore their full potential.

Work is actually one of the ultimate forms of self-expression.

Your work can be your dojo where you practice building your abilities, creating your competencies, and sharpening your skills in all areas of your life.

But the real key is to bridge work and life through your values.

If you can find a way to bake your values into how you show up each day, whether at home or in the office, that’s the real secret to living the good life.

But what’s the key to living the great life?

The key to living the great life is to give your best where you have your best to give in the service of others.

Agile Results is a way to help you do that.

Check out the refresher on Agile Results and use the Rule of Three to rule your day.

If you already know Agile Results, teach three people and help them live and lead a more inspired life.

Game on.

Categories: Architecture, Programming

Where to Look for Trends and Insights

“The best is yet to come.”

It can be tough creating the future among the chaos.

The key is to get a good handle on the real and durable trends that lie beneath the change and churn that’s all around you.

But how do you get a good handle on the key disruptions, the key trends, and the macro-level patterns that matter?

Draw from multiple sources that help you see the big picture in a simple way.

To get started, I’m going to share the key sources for trends and insights that I draw from (beyond my own experience and what I learn from working with customers and colleagues from around the world).

Here are the key sources for trends and insights that I draw from:

  1. Age of Context (Book), by Robert Scoble and Shel Israel.  Age of Context provides a walkthrough of 5 technological forces shaping our world: 1) mobile devices, 2) social media, 3) big data, 4) sensors, 5) location-based services.
  2. Cognizant – A global leader in business and technology services, helping clients bring the future of work to life — today.
  3. DaVini Institute – The DaVinci Institute is a non-profit futurist think tank. But unlike traditional research-based consulting organizations, the DaVinci Institute operates as a working laboratory for the future human experience A community of entrepreneurs and visionary thinkers intent on discovering the (future) opportunities created when cutting edge technology meets the rapidly changing human world.
  4. Faith Popcorn – The “Trend Oracle.”  Faith is a key strategist for BrainReserve and trusted advisor to the CEOs of The Fortune 500.  She’s identified movements such as, “Cocooning,” “AtmosFear,” “Anchoring,” “99 Lives,” “Icon Toppling” and “Vigilante Consumer.”
  5. Fjord – Fjord produces an annual report to help guide you through challenges, experiences, and opportunities you, your organization, employees, customers, and stakeholders will likely face.  Check out the Fjord Trends 2017 report on SlideShare.
  6. Foresight Factory (Formerly called Future Foundation) – Future focused, applied, global consumer insight. Universal trends that shape tastes and determine demand the world over; sector trends that are critical to success in specific industries; custom reports produced in partnership with clients and focus reports on key markets, regions and topics.
  7. Forrester – Research to help you make better decisions in a world where technology is radically changing your customer.
  8. Gartner – The the world’s leading information technology research and advisory company.
  9. Global Goals – In September 2015, 193 world leaders agreed to 17 Global Goals for Sustainable Development. If these Goals are completed, it would mean an end to extreme poverty, inequality and climate change by 2030.
  10. IBM Executive Exchange – An issues-based portal providing news, thought leadership, case studies, solutions, and social media exchange for C-level executives.
  11. Jim Carroll – A world-leading futurist, trends, and innovation expert, with a track record for strategic insight.  He is author of the book The Future Belongs to Those Who Are Fast, and he shares major trends, as well as trends by industry, on his site.
  12. Motley Fool – Motley Fool – To educate, amuse, and enrich.
  13. No Ordinary Disruption (Book) – This is a deep dive into the future, backed with data, stories, and insight.  It highlights four forces colliding and transforming the global economy: 1) the rise of emerging markets, 2) the accelerating impact of technology on the natural forces of market competition, 3) an aging world population, 4) accelerating flows of trade, capital, people, and data.
  14. O’Reilly Ideas – Insight, analysis, and research about emerging technologies.
  15. Richard Watson – A futurist author, speaker and scenario planner, and the chart maker behind The Table of Trends and Technologies for the World in 2020 (PDF). Watson is author of the What’s Next Top Trends Blog. Watson is the author of 4 books: Future Files, Future Minds, Futurevision, and The Future: 50 Ideas You Really Need to Know.
  16. Sandy Carter — Sandy Carter is IBM Vice President  of Social Business and Collaboration, and author of The New Language of Marketing 2.0, The New Language of Business, and Get Bold: Using Social Media to Create a New Type of Social Business.  She’s not just fun to read or watch – she has some of the best insight on social innovation.
  17. The Industries of the Future (Book), by Alec Ross.  Alec Ross explains what’s next for the world: the advances and stumbling blocks that will emerge in the next ten years, and how we can navigate them.
  18. The Second Machine Age, by Erik Brynjolfsson and Andrew McAfee.  Erik Brynjolfsson and Andrew McAfee identify the best strategies for survival and offer a new path to prosperity amid exponential technological change. These include revamping education so that it prepares people for the next economy instead of the last one, designing new collaborations that pair brute processing power with human ingenuity, and embracing policies that make sense in a radically transformed landscape.
  19. ThoughtWorks Technology Radar – Thoughts from the ThoughtWorks team on the technology and trends that are shaping the future.
  20. Trend Hunter – Each day, Trend Hunter features a daily dose of micro-trends, viral news and pop culture. The most popular micro-trends are featured on Trend Hunter TV and later grouped into clusters of inspiration in our Trend Reports, a series of tools for professional innovators and entrepreneurs.
  21. Trends and Technologies for the World in 2020 (PDF) – Table of trends and technologies shaping the world in 2020.
  22. Trendwatching.com – Trendwatching.com helps forward-thinking business professionals in 180+ countries understand the new consumer and subsequently uncover compelling, profitable innovation opportunities.

While it might look like a short-list, it’s actually pretty deep.

It’s like a Russian nesting doll in that each source might lead you to more sources or might be the trunk of a tree that has multiple branches.

These sources of trends and insights have served me well and continue to serve me as I look to the future and try to figure out what’s going on.

But more importantly, they all inspire me in some way to create the future, rather than wait for it to just happen.

I’m a big fan of making things happen … you play the world, or the world plays you.

You Might Also Like

All Digital Transformation Articles

Digital Transformation Books

Consumer Trend Canvas

Trend Framework

101 Hacks for a Better Year

Categories: Architecture, Programming

Don't Build That Product

Xebia Blog - Sun, 01/15/2017 - 12:06
At the Agile Chef Conference I facilitated a workshop where participants could experience how Aikido can be used to resolve conflicts on the work floor as well by applying verbal Aikido. At the end of the session someone asked me to demonstrate the best defence against a sword attack; I responded by turning around and

Consumer Trend Canvas

Consumer Trends are a key building block for innovation.

Is you are stuck coming up with innovation opportunities, part of it is that you are missing sources of insight.

And of the best sources of insight is actually consumer trends.

One tool for helping you turn consumer trends into innovation opportunities is the Consumer Trend Canvas, by Trendwatching.com.

image

 

What I like about it is the simplicity, the elegance, and the fact that it’s similar in format to the Business Model Canvas.

The Consumer Trend Canvas is broken down into to simple sections:

  1. Analyze
  2. Apply

Pretty simple.

In terms of the overall canvas, it’s actually a map of the following 7 components:

  1. Basic Needs
  2. Drivers of Change
  3. Emerging Customer Expectations
  4. Inspiration
  5. Innovation Potential
  6. Who
  7. Your Innovations

From a narrative standpoint, you can think of it in terns of pains, needs, and desired outcomes for a particular persona, along with the innovation opportunities that flow from that simple frame.

The real beauty of the Consumer Trend Canvas is that it’s a question-driven approach to revealing innovation opportunities.

Here are the questions within each of the parts of the Consumer Trend Canvas:

  1. Which deep consumer needs & desires does this trend address?
  2. Why is this trend emerging now? What’s changing?
  3. What new consumer needs, wants, and expectations are created by the changes identified above? Where and how does this trends satisfy them?
  4. How are other businesses applying this trend?
  5. How and where could you apply this trend to your business?
  6. Which (new) customer groups could you apply this trend? What would you have to change?

When you put it all together, you have a quick and simple view of how a trend can lead to some potential innovations.

The power is in the simplicity and in the consolidation.

You Might Also Like

Trend Framework

8 Big Trends

10 High-Value Activities in the Enterprise

Hack a Happy New Year

Continuous Value Delivery the Agile Way

Categories: Architecture, Programming

Stuff The Internet Says On Scalability For January 13th, 2017

Hey, it's HighScalability time:

 

So you think you're early to market! The Man Who Invented VR Goggles 50 Years Too Soon
If you like this sort of Stuff then please support me on Patreon.
  • 99.9: Percent PCs cheaper than in 1980; 300x20 miles: California megaflood; 7.5 million: articles published on Medium; 1 million: Amazon paid eBook downloads per day; 121: pages on P vs. NP; 79%: Americans use Facebook; 1,600: SpaceX satellites to fund a city on Mars; 

  • Quotable Quotes:
    • @GossiTheDog: How corporate security works: A) buy a firewall B) add a rule allowing all traffic C) the end How corporate security works:A) buy a firewall B) add a rule allowing all traffic C) the end
    • @caitie: Distributed Systems PSA: your regular reminder that the operational cost of a system should be included & considered when designing a system
    • @jimpjorps: 1998: the internet means you can "telecommute" to a tech job from anywhere on Earth 2017: everyone works in the same one square mile of SF
    • Jessi Hempel: [re: BitTorrent] Perhaps the lesson here is that sometimes technologies are not products. And they’re not companies. They’re just damn good technologies.
    • giltene: My new pet peeve: "how to make X faster: do less of X" recommendations.
    • peterwwillis: It used to be you had to actually break into a system to exfiltrate all its data. Now you just make an HTTP query.
    • Laralyn McWillams: Identify problems but focus on solutions. If you become more about problems than solutions, that negativity infects your work, your team, and how you think about your career.
    • Chris Fox: Apple is 100% a boutique retailer, meaning that a human chooses which books to promote. Without that, there was no organic discovery tool where readers could find your book.
    • vytah: In fact, the 1986 [Chernobyl] disaster happened because the engineers decided to get rid of safeguards and run tests.
    • Eric Elliott: Breaking into a user’s top 5 apps is like getting struck by lightning or winning the lottery. Don’t bank on it.
    • Peter: I say the super-intelligent aliens will be powered by hyper-computation, a technology that makes our concept of computation look like counting on your fingers; and they’ll have not only qualia, but hyper-qualia, experiential phenomenologica whose awesomeness we cannot even speak of.
    • SEJeff: LVS is pretty much the undisputed king for serious business load balancing. I've heard (anecdotally) that Uber uses gorb[1] and google has released seesaw, which are both fancy wrappers ontop of LVS for load balancing.
    • k__: I have the feeling this is haunting my life. Jobs, relationships, everything. When I got something, it didn't feel that hard to get it. When I try to get something it feels impossible.
    • Nelson Elhage: One of my favorite concepts when thinking about instrumenting a system to understand its overall performance and capacity is what I call “time utilization”. By this I mean: If you look at the behavior of a thread over some window of time, what fraction of its time is spent in each “kind” of work that it does?
    • Bart Sano (Google): I can say that we are committed to the choice of these different architectures, including X86 – and that includes AMD – as well as Power and ARM. The principle that we are investing in heavily is that competition breeds innovation, 
    • aaron-lebo: This is a larger issue with developer burnout I suspect. You master one thing and there's someone standing on the corner saying..."well, actually, I've got something better" and there's a very real anxiety in that evaluation process. Does object-oriented programming suck? Are functional languages the future? Do you really want an SPA? Should you replace your C codebase with Rust... or Go? Is Bitcoin worth getting in on? etc etc
    • StorageMojo: [re: Violin’s bankruptcy] The race is not always to the swift, nor riches to the wise. By starting with software, other companies built an early lead, and now have the money and time to optimize hardware for flash.
    • nocarrier: [Why no datacenters in India?] Cost was a smaller factor than politics; the Indian government wanted the private keys for our certs in order to let FB put a POP there. That was an absolute dealbreaker, so we served India from Singapore and other POPs in nearby countries.
    • RDX: So that original post, although long and full of real examples, was not about Javascript fatigue really. Its change fatigue. Let’s be clear, if you’re picking something new, you’re making a conscious choice to grow up with it.
    • @jamesurquhart: Amazing that emergent tech that’ll revolutionize software dev is already almost a commodity utility service. #streaming #serverless #events

  • The Ethics of Autonomous Cars. The obvious revenue model is highest bidder lives. During the first few milliseconds of a crash response a real-time bidding session is created and the lowest bidder assumes the risk. That at least captures the zeitgeist of the times.

  • First Go. Now poker. DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker. Thank the force humans are still unbeatable at Sabacc. 

  • Medium may be the first YA (Young Adult, think Hunger Games) style publishing outlet. YA is often written in first-person present. It's a good way to fake authenticity. Traditional publications use third-person past tense, but that's not what works best on Medium. What I learned from analyzing the top 252 Medium stories of 2016: The words “you” and “I” were by far the most common, which suggests that addressing the reader directly as an individual person is a better writing strategy than writing in third person.

  • Ben Kehoe says AWS Step Functions is not the cheap, high-scale state machines using an event-driven paradigm he has been looking for. FaaS is stateless, and AWS Step Functions provides state as-a-Service: at $0.025 per 1,000 executions, it’s 125 times more expensive per invocation than Lambda; it’s not going to be cost-effective to replace existing roll-your-own Lambda solutions; the default throttling limit for a state machine is two executions per second...it’s not built to handle massively scaled but transient event scheduling.

  • Ransomware has shifted to being a reproducible strategy. @SteveD3Since I fist covered the MongoDB hacking on Jan 3, the number of compromised DBs has surpassed 32,000. Now possibly Elasticsearch. Anything you can find basically with Shodan. Which is why we now have @GossiTheDog: Found out today firms have started doing legal contracts which specifically rule out liability if they get hit by ransomware, naming it.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Trend Framework

It’s that time of year when I like to take the balcony view to figure out where the world is going, at least some of the key trends.

I’ve long been a fan that while you can’t predict the future, you can take the long view and play out multiple future scenarios so you are ready for (most) anything.

But I’m an even bigger fan of the idea that rather than predict the future—create the future.

To do that, it helps to have a solid handle on the trends shaping the world.

To help make sense of the trends, I like to use mind tools and frameworks that help me see things more clearly.

One of my favorite tools for trends is the Trend Framework by Trendwatching.com

Trendwatching.com uses a framework to sort and catalog trends. 

To understand the future of consumerism, they use a framework of 16 Mega-Trends:

  1. Status Seekers.  The relentless, often subconscious, yet ever present force that underpins almost all consumer behavior.
  2. Betterment.  The universal quest for self-improvement.
  3. Human Brands.  Why personality and purpose will mean profit.
  4. Better Business.  Why “good” business will be good for business.
  5. Youniverse.  Make your consumers the center of their Youniverse.
  6. Local Love.  Why “local” is in, and will remain, loved.
  7. Ubitech.  The ever-greater pervasiveness of technology.
  8. Infolust.  Why consumers voracious appetite for (even more) information will only grow.
  9. Playsumers.  Who said business has to be boring?
  10. Ephemeral.  Why consumers will embrace the here, the now, and the soon-to-be-gone.
  11. Fuzzynomics.  The divisions between producers and consumers, brands, and customers will continue to blur.
  12. Pricing Pandemonium.  Pricing more fluid and flexible than ever.
  13. Helpful. Be part of the solution, not the problem.
  14. Joyning.  The eternal desire for connection, and the many (new) ways it can be satisfied.
  15. Post-Demographics.  The age of disrupted demographics.
  16. Remapped.  The epic power shifts in the global economy.

I’ve used these 16 Mega-Trends from the Trend Framework as a filter (well, maybe more accurately as idiot-guards and bumper-rails) for guiding how I look at consumer behaviors shaping the market.

In fact, this was one of the most helpful frameworks I used when putting together my Trends for 2016: The Year of the Bold.

As I create my master list of Trends for 2017, I’m finding this simple list of 16 Mega-Trends to be useful once again, to better understand all of the micro-trends that emerge on top of this foundation.

The Trend Framework makes it easier to see the graph of trends and to quickly make sense of why things are shaping the way they are.

Categories: Architecture, Programming

Spring Cloud Contracts and Spring Cloud Services on PCF

Ben Wilcock - Mon, 01/09/2017 - 12:07

We had a customer recently who were quite interested in the idea of using Spring Cloud Contract (SCC) in order to prevent API ‘drift’ between microservices teams where individual development teams look after the individual API’s that form part of an enterprise application.

Spring Cloud Contract is an implementation of the ‘Consumer Driven Contracts‘ concept for the Spring platform. From the docs…

Spring Cloud Contract provides support for Consumer Driven Contracts and service schemas in Spring applications. [It provides] a range of options for writing tests, publishing assets, and asserting that a contract is kept by both producers and consumers. It works with both HTTP and message-based interactions.

To help the customer get started with SCC, I created a demonstration app for them that used the 1.0 GA version of the Ssoftware. During this process, I learned that SCC is undergoing some rapid development at the moment and this meant that SCC v1.0 was occasionally a little bit ‘temperamental’ when things like filenames or folder locations change within your project. I found first few days with SCC were a learning curve but I did come to love it as the results of my effort were paid off.

I found that Spring Cloud Contract publishes very clear and helpful information about your services, improves the clarity of your testing, adds fantastic wiremock stubbing capabilities, and alerts you early to any API drift which may have occurred between projects (which is essential in multi-team microservice development environments). I’ll definitely be recommending SCC to clients in the future.

To try and help other newbies out, I used the original SCC samples but added plenty of comments into the code and the README’s to make it easier for people to just pick it up and run with it.

The code for the demo is here: https://github.com/benwilcock/spring-cloud-contracts

Extra Credit – Spring Cloud Services on PCF

The same customer also wanted a demo of the Spring Cloud Services (SCS) components for Pivotal Cloud Foundry so I built one and added additional Zipkin tracing (not part of SCS) into the mix. This demo should make it super easy for anyone giving PCF and SCS a trial run. It should even work on PCF Dev (if started with the SCS services) so any Spring developer, even those without PCF access at work can still give it a try.

https://github.com/benwilcock/pcf-spring-cloud-services-demo 

I enjoyed building them, and I hope that these are useful to you.

About the Author

Ben Wilcock works for Pivotal as a Senior Solutions Architect. Ben has a passion for microservices, cloud and mobile applications and helps Pivotal’s Cloud Foundry customers to become more responsive, innovate faster and gain greater returns from their software investments. Ben is a respected technology blogger who’s articles have featured in DZone, Java Code Geeks, InfoQ, Spring Blog and more.


Categories: Architecture, Programming

Stuff The Internet Says On Scalability For January 6th, 2017

Hey, it's HighScalability time:

 

Hot rods in space. The Smith Cloud plummets towards our galaxy at nearly 700,000 mph. Vroom!
If you like this sort of Stuff then please support me on Patreon.
  • 3 of top 5: Stackoverflow questions are about Git; 3,000: four-passenger cars could serve 98 percent of NYC taxi demand; 44%: US population lives within 20 miles of Amazon fulfillment center; 72%: Amazon customers shopped using mobile device; 110%: increase in industrial control system attacks; 455: Number of scripted television series aired this year; $28.5 billion/yr: App downloads on iOS;

  • Quotable Quotes:
    • @ValaAfshar: Number of robots working in Amazon warehouses: 2016: 45,000 / 2015: 30,000 2014: 15,000 / 2013: 1,000 — @JonErlichman
    • @jason_kint: updated duopoly #s. new IAB data came out yesterday. easy to run vs earnings for goog and fb, it's evident everyone else is zero sum game. 
    • rb2k_: I also haven't seen one [company in Germany] that isn't riddled with MBA grads that mainly push Jira tickets around.
    • Joe McCann: The best software developers I know are always hacking over the holidays. True story.
    • @kaffeecoder: Sigh. Async vs blocking protocol is irrelevant. What matters is communicating with other services outside your own req/response cycle.
    • Eric Jang: It's not a coincidence that Nvidia, the literal arms-dealer of deep learning, has had a good year in the stock market.
    • @markimbriaco: Just read a comment that said "Any good codebase has every part perfectly isolated". Oh, to be young and optimistic about software again.
    • @swardley: Asked "What do I think is the biggest impact AI would have?" ... hmmm, the largest erosion of social mobility in human history?
    • The Attention Merchants: It is therefore more effective for the State to intervene before options are seen to exist. This creates less friction with the State but requires a larger effort: total attention control.
    • StorageMojo: The cloud’s collateral damage to the legacy IT vendors continues to spread. A few billion here and a few billion there, and pretty soon you’re talking real money.
    • Janakiram MSV: The key takeaway is that Amazon wants enterprises to consume EC2 while it is pushing startups and developers towards Lambda. This move from Amazon will fuel the growth of serverless computing in the industry. 
    • Maxime Chevalier-Boisvert: Edsger Dijkstra famously said, “The question of whether machines can think is about as relevant as the question of whether submarines can swim.”
    • @karlseguin: Microservices without asynchronous messaging (queues) is actually a monolith with really slow and error prone method invocation.
    • AshleysBrain: We've been using WebRTC Datachannels for multiplayer gaming in the browser in our game editor Construct 2 (www.scirra.com) for a couple of years now. Generally they work great! However the main problem we have is switching tab suspends the game, which if you're acting as the host, freezes the game for everybody. This is really inconvenient. 
    • @lstoll: 2017: Year of the return of three tier architecture.
    • @tealtan: “I will never make a racial profiling database!” *continues working on social networks, analytics, ad tech*
    • @abt_programming: Inverse bus factor: "how many developers have to be hit by a bus before a project starts to proceed smoothly?” - @gasproni
    • M.G. Siegler: The numbers speak for themselves. 2 billion words written on Medium in the last year. 7.5 million posts during that time. 60 million monthly readers now. Pageviews galore. So step 2 is simply to slap some banner ads on the site, while step 3 is to profit, right?
    • snarf21: Writing software is hard but to me the hardest part is always taking a random abstract concept from someone's mind (or worse, several people) and converting that into something "real" in a fixed timeline and budget. There will have to be lots of tradeoffs and miscues by definition. We are always making something that doesn't already exist, it is creation and creation is hard.
    • @Pinboard: Who could have foreseen the always-on home microphone might be of interest to the cops?
    • @ThePracticalDev: I heard a rumor that Santa moved over to AWS this year. Big if true.
    • Drew Purves~ “intelligence” extends beyond brains; something as simple as self-replicating RNA exhibit intelligent behavior at the evolutionary scale. The natural world is fractal, cyclic, and fuzzy...in a biosphere, every organism is a resource to another organism. That is, learning and adaptation of each organism is not independent of other organisms
    • @meatcomputer: System clocks are always accurate and increase monotonically. Timestamps from remote machines are reliable
    • pjmlp: Everything on web development feels like an hack.
    • @kelseyhightower: In my opinion Serverless does not mean FaaS. I consider any platform that hides the management of servers from the user to be Serverless.
    • Amit: one of the lessons I learned from this journey was that the tutorials work best when I've needed that technology for a real project
    • @Carnage4Life: Snapchat' copied all the worst parts of Apple's culture & seen success. More copycats to come
    • ch: So all that's missing with the decentralized web is a centralized service to aggregate the decentralized streams?
    • @mathiasverraes: "Separation of intent and implementation" is probably a much more useful programming principle than all of SOLID combined.
    • doh: We moved back and forth between AWS and GCE (based on who gave us free credits). Once we ran out, we chose GCE and never regretted it. GCE has many quirks, for instance the inconsistency between API and the UI, it misses the richness of the services offered by AWS but everything GCE does offer is just faster, more stable and much more consistent.
    • Exponential Laws: We have argued that exponential growth would not have succeeded without sustained exponential growth at three levels of the computing ecosystem—chip, system, and adopting community. Growth (progress) feeds on itself up to the inflection point.

  • Measuring a gnat's eyebrow at a billion miles. Ivan Linscott tells the thrilling story behind the development of the New Horizons probe to Pluto. a16z Podcast: New Year, New Horizons — Pluto!  Completing the probe was a close thing. Finding enough plutonium to power system almost didn't happen. Enough wasn't found so the probe had a much lower power budget than originally spec'ed, which caused the communication system to use one FPGA instead of two. You have to use radiation hardened parts. The chips sit right next to a pile of plutonium pumping out gamma rays and neutrons. The FPGA's had a capacity of a million gates, were hardened by design, and had triple redundancy. Each gate in the array is implemented in threes. They are voted in pairs. If three agree then fine. If two agree that's the value used. They fit all the code with 5 gates margin. They also had a hero's journey sourcing a high precision oscillator. And then the frightening story of when the watchdog timer timedout and put the probe in safe mode. It turned out the JPEG compression algorithm took too long to compress an image of Pluto and that caused the timeout to fire. The reason is one of those crazy testing stories. When this feature was tested the picture of the sky was darker so it took less time to compress!

  • The impulse for folks at Twitter to delay Trump's tweets and insider trade on that information must be overwhelming.

  • 33C3 (Chaos Computer Congress) videos are now available. Great overview by Chris Hager. Lots of interesting talks. You might like: Dissecting modern (3G/4G) cellular modems; Edible Soft Robotics - An exploration of candy as an engineered material; Software Defined Emissions - A hacker’s review of Dieselgate; Rebel Cities - Towards A Global Network Of Neighbourhoods And Cities Rejecting Surveillance.

  • A compelling break down of the DNC phishing attack. Making everything viewable through a generic UI and everything programmable through a scriptable API has interesting consequences @pwnallthethings: Could have hacked? Sure. Did hack? No. Let me go through why not..The hackers weren't hacking one-by-one; so URL contraction wasn't done manually. It was done via the Bitly API...Why did the hackers include this info? Same reason they contracted links via API. Because they're not hacking 1-by-1. Are hacking at scale...When hackers hack at scale, they reuse infrastructure. They make mistakes. This isn't unusual. You can piece the bits together.

  • In the game of data you want to be at the top of the data gravity well. When your are down well nothing escapes without great cost. AWS Snowball

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

How is Writing Lord of the Rings Like Writing Software?

 

Have you ever read a book and wondered how any human could have written something so brilliant? For me it was Lord of the Rings. I despaired that in a hundred lifetimes I could never write a book so rich, so deep, so beautiful. Since then I've learned a few things about about how LoTr was created that has made me reconsider. The kick-in-the-head is that it's the same lesson I learned long ago about writing software.

I've always been amazed how a program can start as a single source file and after years of continued effort turn into a working system that is so large no human can come close to understanding it. If you had tried from the start to build the system you ended up with you would have never ever got there. That's just not how it works. Software is path dependent.

I've experienced this growth from a single cell to a Cambrian explosion many times so I know it's a thing. What I hadn't considered is how it's also a thing for writing books too. 

Creating good software is a process of evolution through the mechanism of constant iteration for the purpose of survival. This is also how good stories are made. What both have in common is creation through thought.

Thought needs an object to contemplate. Each intermediate state of a project is that object. By linking together a series of state inspired creative jumps something wonderful can be created that may contain only the faintest trace of its beginnings.

Here's how Lord of the Rings is a good example of this process...

LoTr started out as a sequel to the Hobbit. Tolkien's publisher wanted to cash in on the success of the Hobbit with a sequel. And The Silmarillion wasn't it. So Tolkien began with the intention of writing a sequel to the Hobbit. It was horrible. 

The first book title was The Return of the Shadow, not Lord of the Rings. The prose was still written for children. Frodo was called Bingo. Strider was a hobbit called Trotter. Bilbo planned to get married. And the ring was still just a ring. The story had no clear motive or direction. "What more can hobbits do?" asked Tolkein. The ideas of the Hobbit were played out. The LoTr we know and love was far far away. 

In draft after draft Tolkien probed and searched for a direction to take the story. It all turned when Tolkien wrote the scene with the Black Rider. At first the Black Rider was really a White Rider. It was Gandalf coming to talk to Bingo. But then some insight happened. A dizzying array of neurons conspired and the color of the horse changed from white to black and Gandalf transformed into a man wrapped in a great black cloak and hood. A new framework was creating itself.

How do we know? Fortunately, from Christopher Tolkien, we have the history of changes his father made to LoTr. Dr. Corey Olsen in a great series—The Return of the Shadow, Session 1 - In Search of a Sequel—walks us through what is essentially the git log for LoTr. Imagine a kind of Papers We Love treatment from a true Tolkien expert and gifted analyst. It's magical.

We see idea after idea worked through in the text. It was a continuous process of refactoring and new development. Some ideas were kept from beginning to end. Many were cut. Many morphed. Much dialogue was kept, but was given to different characters to say in different circumstances. 

The whole feeling was very much like seeing software being developed, only the result wasn't a working app, but one of the most influential stories of all time.

The lesson for me was a deep and dramatic reconfirmation of an old idea: All successful large systems started as successful small systems.

This applies to us as writers and programmers. It's easy to get down on yourself during the creation process. Neither your story or program has to start out great; greatness is something that evolves.

In this new year, that's the lesson of Lord of the Rings for me.

Categories: Architecture

Sponsored Post: Loupe, New York Times, ScaleArc, Aerospike, Scalyr, VividCortex, MemSQL, InMemory.Net, Zohocorp

Who's Hiring?
  • The New York Times is looking for a Software Engineer for its Delivery/Site Reliability Engineering team. You will also be a part of a team responsible for building the tools that ensure that the various systems at The New York Times continue to operate in a reliable and efficient manner. Some of the tech we use: Go, Ruby, Bash, AWS, GCP, Terraform, Packer, Docker, Kubernetes, Vault, Consul, Jenkins, Drone. Please send resumes to: technicaljobs@nytimes.com
Fun and Informative Events
  • Your event here!
Cool Products and Services
  • A note for .NET developers: You know the pain of troubleshooting errors with limited time, limited information, and limited tools. Log management, exception tracking, and monitoring solutions can help, but many of them treat the .NET platform as an afterthought. You should learn about Loupe...Loupe is a .NET logging and monitoring solution made for the .NET platform from day one. It helps you find and fix problems fast by tracking performance metrics, capturing errors in your .NET software, identifying which errors are causing the greatest impact, and pinpointing root causes. Learn more and try it free today.

  • ScaleArc's database load balancing software empowers you to “upgrade your apps” to consumer grade – the never down, always fast experience you get on Google or Amazon. Plus you need the ability to scale easily and anywhere. Find out how ScaleArc has helped companies like yours save thousands, even millions of dollars and valuable resources by eliminating downtime and avoiding app changes to scale. 

  • Scalyr is a lightning-fast log management and operational data platform.  It's a tool (actually, multiple tools) that your entire team will love.  Get visibility into your production issues without juggling multiple tabs and different services -- all of your logs, server metrics and alerts are in your browser and at your fingertips. .  Loved and used by teams at Codecademy, ReturnPath, Grab, and InsideSales. Learn more today or see why Scalyr is a great alternative to Splunk.

  • InMemory.Net provides a Dot Net native in memory database for analysing large amounts of data. It runs natively on .Net, and provides a native .Net, COM & ODBC apis for integration. It also has an easy to use language for importing data, and supports standard SQL for querying data. http://InMemory.Net

  • VividCortex is a SaaS database monitoring product that provides the best way for organizations to improve their database performance, efficiency, and uptime. Currently supporting MySQL, PostgreSQL, Redis, MongoDB, and Amazon Aurora database types, it's a secure, cloud-hosted platform that eliminates businesses' most critical visibility gap. VividCortex uses patented algorithms to analyze and surface relevant insights, so users can proactively fix future performance problems before they impact customers.

  • MemSQL provides a distributed in-memory database for high value data. It's designed to handle extreme data ingest and store the data for real-time, streaming and historical analysis using SQL. MemSQL also cost effectively supports both application and ad-hoc queries concurrently across all data. Start a free 30 day trial here: http://www.memsql.com/

  • ManageEngine Applications Manager : Monitor physical, virtual and Cloud Applications.

  • www.site24x7.com : Monitor End User Experience from a global monitoring network. 

If any of these items interest you there's a full description of each sponsor below...

Categories: Architecture

Matching strings in Scala

Xebia Blog - Mon, 01/02/2017 - 21:31
Over December I had a lot of fun doing the Advent of Code coding challenges with some colleagues. Many of those, such as day 21, require interpreting some kind of string input. While normally I'd probably marshall those strings into case classes before processing, in this case that seemed like overkill: a quick pattern-match should

Hack a Happy New Year!

“Whatever you can do, or dream you can, begin it. Boldness has genius, power and magic in it.” — Johann Wolfgang Von Goethe

Hack a Better New Year

It’s time to dig down, dig in, and dig deep to create a great year for yourself and others.

I’m a fan of hacks for work and life.

I find that hacking away at challenges is a great way to make progress and to eventually overcome them.

Hacking is really an a approach and a mindset where you try new things, experiment and explore while staying open-minded and learning as you go.

You never really know what’s going to work, until you’ve actually made it work.

Nothing beats personal experimentation when it comes to creating better results in your life.

Anyway, in the spirit of kicking off the new year right, I created a comprehensive collection of the ultimate hacks for a happy new year:

101 Hacks for a Happy New Year

This is no ordinary set of hacks.  It’s deep.  There are hacks for mind, body, emotions, career, finance, relationships, and fun.

There are hacks you can use everyday to change how you think, feel, and act.

There are hacks to help you change habits.

There are hacks to help you relight your fire and get back in the game, if you’ve been in a slump or waiting on the sidelines.

Jump back in the game, master your work and life, and have some fun in the process.

Here is a quick list of the hacks from 101 Hacks for a Happy New Year:

1. Get the power of a New Year’s Resolution on your side
2. Limit yourself to one big resolution at a time
3. Get specific with your goals
4. Dream bigger to realize your potential
5. If you want change, you must change
6. Guide your path with vision, values, and goals
7. Change a habit with Habit Stacking
8. Create mini-feedback loops
9. Bounce back from a setback
10. Avoid “All or Nothing” thinking
11. Choose progress over perfection
12. Reward yourself more often
13. Gamify it
14. Adopt a Tiny Habit
15. Just Start
16. Adopt a growth mindset
17. Create if-then plans to stick with your goals
18. Start with Great Expectations
19. Adopt 7 beliefs for personal excellence
20. Master the art of goal planning
21. Prime your mind for greatness
22. Use dreams, goals, and habits to pull you forward
23. Use the Exponential Results Formula to make a big change
24. Adopt the 7 Habits of Highly Motivated People
25. Use Trigger Moments to activate your higher self
26. Use Door Frame Triggers to inspire a better version of you
27. Find your purpose
28. Figure out what you really want
29. Use 3 Wins to Rule Your Year
30. Commit to your best year ever
31. Find your Signature Strengths
32. Practice a “lighter feeling”
33. Let go of regrets
34. 15-Minutes of Fulfillment
35. Create your ideal day the Tony Robbins Way
36. Master your emotions for power, passion, and strength
37. Start your year in February
38. Build your personal effectiveness toolbox
39. Write your story for the future
40. Get out of a slump
41. Give your best, where you have your best to give
42. Ask more empowering questions
43. Surround yourself with better people
44. Find better mentors
45. Do the opposite
46. Try a 30 Day Sprint
47. Grow 10 Years Younger
48. Don’t get sick this year
49. Know Thyself
50. Decide Who You Are
51. Decide Who You Want To Be
52. Cultivate an Attitude of Gratitude
53. Try 20-Minute Sprints
54. Create a vision board for your year
55. Adopt some meaningful mantras and affirmations
56. Practice your mindfulness
57. 15-Minutes of Happiness
58. Breathe better
59. Become your own gym
60. Master your wealth
61. Learn how to read faster
62. Let go of negative feelings
63. Live a meaningful life
64. Establish a routine for eating, sleeping, and exercising
65. Improve your likeability
66. Win friends and influence people
67. Improve your charisma through power, presence, and warmth
68. Fill your mind with a few good good thoughts
69. Ask for help more effectively
70. Attract everything you’ve ever wanted
71. Catch the next train
72. Unleash You 2.0
73. Learn anything in 20 hours
74. Use stress to be your best
75. Take worry breaks
76. Use the Rule of Three to rule your day
77. Have better days
78. Read 5 powerful personal development books
79. Practice the 10 Skills of Personal Leadership
80.  Develop your Emotional Intelligence
81. Cap your day with four powerful questions
82. Build mental toughness like a Navy Seal
83. Feel In Control
84. Transform your job
85. Use work as your ultimate form of self-expression
86. Be the one who gives their all
87. Live without the fear of death in your heart
88. Find your personal high-performance pattern
89. Create unshakeable confidence
90. Lead a charged life
91. Use feedback to be your best
92. Make better decisions
93. Learn how to deal with difficult people
94. Defeat decision fatigue
95. Make the most of luck
96. Develop your spiritual intelligence
97. Conquer your fears
98. Deal with tough criticism
99. Embrace the effort
100. Finding truth from the B.S.
101. Visualize more effectively

For the details of each hack, check out 101 Hacks for a Happy New Year.

I will likely tune and prune the hacks over time, and improve the titles and the descriptions.

Meanwhile, I’m not letting perfectionism get in the way of progress.

Go forth and hack a happy new year and share 101 Hacks for a Happy New Year with a friend.

Categories: Architecture, Programming

Efficient storage: how we went down from 50 PB to 32 PB

As the Russian rouble exchange rate slumped two years ago, it drove us to think of cutting hardware and hosting costs for the Mail.Ru email service. To find ways of saving money, let’s first take a look at what emails consist of.

Indexes and bodies account for only 15% of the storage size, whereas 85% is taken up by files. So, files (that is attachments) are worth exploring in more detail in terms of optimization. At that time, we didn’t have file deduplication in place, but we estimated that it could shrink the total storage size by 36% since many users receive the same messages, such as price lists from online stores or newsletters from social networks containing images and so on. In this article, I’m going to describe how we implemented a deduplication system under the supervision of Albert Galimov.

Categories: Architecture

A Guide to Generating Boring Code with Node

Xebia Blog - Fri, 12/30/2016 - 12:47
Writing boring code is not fun. Especially if that code is just a derivative of some data. You should generate it instead! A great example is to generate your HTTP client code from an API definition. This is actually very easy to do with a recent node version and requires almost no extra dependencies. In

The Best New Years Resolution: Agile Product Management

Xebia Blog - Wed, 12/28/2016 - 15:44
Agile Product Management is grounded in the Jobs to be done theory and Lean startup principles. In my book “The Product Samurai” I described how you can effectively apply these techniques to be a better Product Manager, but what I didn’t’ cover was why not everybody is doing this already? Making up for that, and

Being An Agile Security Officer: Security Stakeholdership mindset

Xebia Blog - Tue, 12/27/2016 - 16:23
This is the second part in my blog series about 'being an agile security officer'. In this blog I will focus on the mindset of security stakeholdership in Agile and DevOps environments. In the Agile world the Product Owner is the person who translates business and customer desires into work items (user stories) for the

How to integrate Jmeter-maven printing HTML reports project with DevOps CI/CD pipeline in 4 easy steps

Xebia Blog - Tue, 12/27/2016 - 08:36
(1) Fork the below github repo having pom.xml file which contains all the dependencies of Jmeter and printing the Jmeter reports in HTML format https://github.com/nishantguptaxe/JmeterMavenHtmlReports.git (2) Add your Jmeter file under the src/test/jmeter folder and checkin your code into the github (3) Install the go server, start the go server ,hit the go url in