Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Programming

Indie game developers in Latin America sustain growth after launch on Google Play

Android Developers Blog - Tue, 01/10/2017 - 01:16
Posted by Kacey Fahey, Marketing Programs Manager, Google Play

Indie game developers are some of the most exciting and innovative teams to work with. While developers large and small exist on the same field, gone are the days where you hit publish and turn your back, moving on to the next project. We've gathered a few developer stories coming out of Latin America sharing experiences and advice.



Oktagon Games

Ronaldo Cruz, Founder and CEO of Oktagon Games tells us how "reviews provide great qualitative insight on the game helping us identify problems that may not be caught by analytics."


Tiny Bytes

Tiny Bytes reduced churn by 5% using an in-game tutorial and analytics.



Impossible Apps

Cleverson Schmidt of Impossible Apps shares how introducing in-app purchases helps diversify revenue streams and "can make the game profitable and self sustainable."
How useful did you find this blogpost?


Categories: Programming

How augmented reality helps you buy furniture and capture Pokémon

Android Developers Blog - Tue, 01/10/2017 - 01:14
Posted by Jamil Moledina, Games Strategic Lead, Google Play

Online furniture seller Wayfair and Niantic's Pokémon GO have more in common than you might think. Both of these companies use augmented reality to create innovative, immersive experiences for their users. I sat down with Mike Festa, Director of Wayfair Next, and Tatsuo Nomura, Product Manager for Pokémon GO, at our recent Playtime event to discuss how developers can make the most of AR as a platform.

From 3D furniture modelling in WayfairView using Tango, to logging countless miles catching Pokémon, hear how these developers are innovating with AR, and get their advice for others looking to use AR in their apps and games.


Check out more sessions from our global Playtime events to learn best practices for your app and game businesses. Also, stay up to date with more videos from events, product news, and tips to help grow your business on Google Play with the Playbook for Developers app.

How useful did you find this blogpost?



Categories: Programming

Spring Cloud Contracts and Spring Cloud Services on PCF

Ben Wilcock - Mon, 01/09/2017 - 12:07

We had a customer recently who were quite interested in the idea of using Spring Cloud Contract (SCC) in order to prevent API ‘drift’ between microservices teams where individual development teams look after the individual API’s that form part of an enterprise application.

Spring Cloud Contract is an implementation of the ‘Consumer Driven Contracts‘ concept for the Spring platform. From the docs…

Spring Cloud Contract provides support for Consumer Driven Contracts and service schemas in Spring applications. [It provides] a range of options for writing tests, publishing assets, and asserting that a contract is kept by both producers and consumers. It works with both HTTP and message-based interactions.

To help the customer get started with SCC, I created a demonstration app for them that used the 1.0 GA version of the Ssoftware. During this process, I learned that SCC is undergoing some rapid development at the moment and this meant that SCC v1.0 was occasionally a little bit ‘temperamental’ when things like filenames or folder locations change within your project. I found first few days with SCC were a learning curve but I did come to love it as the results of my effort were paid off.

I found that Spring Cloud Contract publishes very clear and helpful information about your services, improves the clarity of your testing, adds fantastic wiremock stubbing capabilities, and alerts you early to any API drift which may have occurred between projects (which is essential in multi-team microservice development environments). I’ll definitely be recommending SCC to clients in the future.

To try and help other newbies out, I used the original SCC samples but added plenty of comments into the code and the README’s to make it easier for people to just pick it up and run with it.

The code for the demo is here: https://github.com/benwilcock/spring-cloud-contracts

Extra Credit – Spring Cloud Services on PCF

The same customer also wanted a demo of the Spring Cloud Services (SCS) components for Pivotal Cloud Foundry so I built one and added additional Zipkin tracing (not part of SCS) into the mix. This demo should make it super easy for anyone giving PCF and SCS a trial run. It should even work on PCF Dev (if started with the SCS services) so any Spring developer, even those without PCF access at work can still give it a try.

https://github.com/benwilcock/pcf-spring-cloud-services-demo 

I enjoyed building them, and I hope that these are useful to you.

About the Author

Ben Wilcock works for Pivotal as a Senior Solutions Architect. Ben has a passion for microservices, cloud and mobile applications and helps Pivotal’s Cloud Foundry customers to become more responsive, innovate faster and gain greater returns from their software investments. Ben is a respected technology blogger who’s articles have featured in DZone, Java Code Geeks, InfoQ, Spring Blog and more.


Categories: Architecture, Programming

Learn tips from Memrise to increase in-app conversions with pricing experiments

Android Developers Blog - Fri, 01/06/2017 - 22:29
Posted by Tamzin Taylor, Partner Development Manager at Google Play, & Kristina Narusk, Head of Production at Memrise

Getting people to install your app is one thing, getting them to sign up to your paid offering is quite another. It's important to understand the complete journey your users take from installing your app to paying for something. Once you do, you can experiment on the flow to try and increase conversions. Memrise has found great success in experimenting on their language learning app to increase the number of paying users.

Four experiments Memrise use to improve conversions

Memrise makes languages fun with a number of different learning modes you can play to help increase your vocabulary in a chosen language. You can download the app for free and play some of the modes or take advantage of their premium subscription offering called 'Memrise Pro' which offers new game modes and additional features like offline learning. Memrise recently ran a number of conversion experiments with the main objective of increasing the Average Revenue Per Daily Active User (ARPDAU). These experiments tested multiple user experience and pricing experiment scenarios.

1. A/B test how messaging different user benefits can impact conversion

What they did: Memrise wanted to know what motivation and call to action would convert the most users to buy a Pro subscription from a locked game mode in the app. To do this, they ran an A/B test with two similar designs, featuring different reasons for the user to upgrade, and compared the results to their original upgrade messaging.


                       

Test A: Focus on ‘difficult’ words with an orange background. Test B: Focus on ‘favorite’ words with a pink background.
Results: Test A performed the best. Conversion to Pro in Test A was 28% higher than in Test B. Pro mode usage was subsequently 9.7% higher in Test A compared to Test B too.

Next steps: After seeing how test A won the experiment, Memrise applied this creative across the board. Subscribers driven by that particular mode increased as a percentage of all subscriptions in the app by 16%. Memrise plans to run additional A/B tests at others points of conversion in the app to see if they can increase the results even further. They'll also try different text for the call to actions.

2. Test whether adapting to local price points results in sustainable uplift

In 2015, Google Play launched new minimum local price levels in countries around the world. To take advantage of the new price points, Memrise tested lowering localised prices in certain markets to better match purchasing power. Prices were an average of 6 times lower during this experiment.

Results: After 30 days, Memrise saw the following changes in conversions to paid users:


Categories: Programming

Downsizing society

Actively Lazy - Thu, 01/05/2017 - 21:38

The world is becoming increasingly automated. Jobs that were once done by people are now frequently done by machines instead. This started off in manufacturing, but the coming of self-driving cars will put huge numbers of people out of work; even lawyers are being replaced by machines. Some reports suggest that machines could do 50% of jobs within the next 30 yearsFifty percent! What will we do when half the population have been made redundant? How will we cope with such a restructuring of society?

To an economist, a job is an income. To a human being, it is much more than that. It provides a sense that you matter in society, that people beyond your family rely on you and even admire you.

If 50% of jobs are being done by machines, the question is what will people do instead? This is an important question, because we identify as our job. Our jobs define us. Let me introduce you to Louise, she’s a doctor. And this is Barry, he’s an estate agent. I bet you have different ideas of who those two people are. What about when they’re both unemployed? Unemployable. Made redundant from society.

How would so many people survive without jobs? Where’s the money to live coming from? The only possible answer seems to be some kind of universal basic income. The idea that everyone, employed or not, receives a fixed amount of money each month from the government – replacing all existing benefits and all the bureaucracy involved in administering them. With a UBI, the 50% of people without jobs would at least have some money with which to buy food and heating. But who wants to live on handouts? Who wants to be defined as unemployed? Even if the majority of people are in the same situation.

This inevitably results in a two-tier society: those that earn little to nothing on top of the basic income; and those increasingly rare few that still have jobs the machines aren’t able to do, yet. We have the non-working class, and a diminishing middle class. How can this cause anything other than resentment, envy and anger?

This is to say nothing of the challenges of funding a universal basic income. While ideas like funding it through a wealth tax make a lot of sense, could they ever succeed? Would the wealthy half ever vote for it? Would the big businesses (and their very wealthy owners) that bankroll governments stand for it?

As Brexit and Trump have shown, voters can vote for what previously seemed impossible. And we’ve barely scratched the surface of the anger and disenfranchisement that automation will bring upon us. However, with both Brexit and Trump people voted out of hope. Hope for something better. For respect. For status. For jobs. Could Farage evoke such a passionate response from a platform of “more handouts”? Of massive wealth redistribution, the likes of which no socialist government has ever even proposed? It seems improbable.

If we don’t introduce a UBI, what’s going to happen? The jobs that are left aren’t going to be spread evenly. London will always have disproportionately more jobs and lower unemployment than, say, Sunderland or Hull. What’s going to happen in these areas as unemployment becomes endemic? Rising anger seems inevitable. Riots. A public whipped into a frenzy of hatred against some group of “others”?

With so much anger, extreme politics is only going to increase. Does a majority of the public already feel left behind? Made redundant by automation? So far this has given us Brexit and Trump. It’s only going to get worse. Who are we going to wage war on? Syria? ISIS? China? Germany? This is the armageddon outcome. So many people feel so disenfranchised that we inadvertently start world war three.

What other outcomes are there? Another possibility is some kind of neo-luddite movement railing against technology. So far everyone’s anger at disappearing jobs has been directed at migrants. When people realise its actually technology taking their jobs, could that anger end up directed at technology instead? You can imagine someone like Farage standing against technology. Of banning automation. Of holding back progress to protect jobs. While this couldn’t hold back the tide forever, it could delay the inevitable march of technology for some years.

The flip-side of this neo-luddite revolution, is the inevitable flight of technology outside of such a regime. With the future being held back in one place, technologists will move somewhere else; to somewhere innovation is encouraged instead of criminalised. Some enterprising country, say Switzerland, would reduce taxes for technology companies to encourage them to relocate. The remaining jobs, and the people to do them, all move to an employment enclave. This exacerbates the split in society: between the wealthy employed few, and the many under-employed poor. This is the “Switzerland outcome”. A physical as well as economic split in society.

Of course, there is always the possibility that the jobs that disappear are replaced by new jobs. Jobs in manual labour become replaced with jobs like “social media consultant” that would have been unimaginable in Victorian England (some would say still are). But the signs aren’t good: in the wake of the 2007 financial crisis jobs aren’t returning in great numbers – they’re being done by robots instead. This might be the first economic recovery in history that hasn’t been accompanied by an increase in employment.

Eventually we’re bound to discover a universal basic income, or something like it. It might take a very long time – a time in which the social strife could be immense. But eventually, barring armageddon, we will have to find a way to move to a post-work society; that means finding a way for people without work to live happy, fulfilled lives.

But in a life without work, how will people find meaning in their lives? With the ready availability of automation, some people might move back onto the land – to become 21st century peasant farmers. With the help of machines it could be quite possible for a family to provide for itself and maintain a decent standard of living through farming alone. Of course, with our heavily urbanised society plenty of people have neither the space nor the wherewithal to do this; but for some this could be a good alternative to the slums from where jobs have vanished.

The ready availability of time frees people up for any project they wish to embark on. The utopian outcome is enabling everyone to become creative, to embark on personal projects and self-expression. A society full of people doing what people do best: being human and creative. Is this realistic? Some people would relish the opportunity to dedicate themselves to creative pursuits. But plenty of others would not, instead looking for the structure and security their jobs used to offer.

Could this usher in an era of grand projects? Where people band together to achieve some lofty goal? Not for financial compensation, but for the joy of being part of something bigger. With everyone’s basic needs met, instead of working for money people look for meaning. They will take part in activities that inspire them, for free. The cost of labour at this point is basically zero, for the first time in human history. The only constraint: the end goal has to inspire people. Improving the efficiency of a government department? Hardly. Flying to Mars? Almost certainly.

But there will still be plenty of people who feel they can’t contribute but need structure in their life. Iain M. Banks’ Culture series describes a post-work society where the machines run everything; but in Banks’ universe people dedicate themselves full-time to leisure. While no doubt attractive to some, this life seems without structure, without purpose. Can people really live like this? Work has defined us, given structure to our day, given us meaning. Without this, who are we?

It is likely that the end result is some mixture of all of these outcomes. Each individual finding their own way in an increasingly diverse world – the simple answer of finding a job, any job, is no longer possible; instead people are forced to find their own answers to hard questions: what am I going to do with my life?

As we approach this future, what are we to do? How can we prepare for the upcoming earthquake in society? It seems inevitable that this will be a tumultuous time. How can we smooth the transition to a post-work society?


Categories: Programming, Testing & QA

Some Tips for Boosting your App's Quality in 2017

Google Code Blog - Wed, 01/04/2017 - 21:59
Originally posted on the Firebase blog by Doug Stevenson /Developer Advocate


I've got to come clean with everyone: I'm making no new year's resolutions for 2017. Nor did I make any in 2016. In fact, I don't think I've ever made one! It's not so much that I take a dim view of new year's resolutions. I simply suspect that I would likely break them by the end of the week, and feel bad about it!

One thing I've found helpful in the past is to see the new year as a pivoting point to try new things, and also improve the work I'm already doing. For me (and I hope for you too!), 2017 will be a fresh year for boosting app quality.

The phrase "app quality" can take a bunch of different meanings, based on what you value in the software you create and use. As a developer, traditionally, this means fixing bugs that cause problems for your users. It could also be a reflection of the amount of delight your users take in your app. All of this gets wrapped up into the one primary metric that we have to gauge the quality of a mobile app, which is your app's rating on the store where it's published. I'm sure every one of you who has an app on a storefront has paid much attention to the app's rating at some point!

Firebase provides some tools you can use to boost your app's quality, and if you're not already using them, maybe a fresh look at those tools would be helpful this year?

Firebase Crash Reporting
The easiest tool to get started with is Firebase Crash Reporting. It takes little to no lines of code to integrate it into your iOS and Android app, and once you do, the Firebase console will start showing crashes that are happening to your users. This gives you a "hit list" of problems to fix.
One thing I find ironic about being involved with the Crash Reporting team is how we view the influx of total crashes received as we monitor our system. Like any good developer product, we strive to grow adoption, which means we celebrate graphs that go "up and to the right". So, in a strange sense, we like to see more crashes, because that means more developers are using our stuff! But for all of you developers out there, more crashes is obviously a *bad* thing, and you want to make those numbers go down! So, please, don't be like us - make your crash report graphs go down and to the right in 2017!

Firebase Test Lab for Android
Even better than fixing problems for your users is fixing those problems before they even reach your users. For your Android apps, you can use Firebase Test Lab to help ensure that your apps work great for your users among a growing variety of actual devices that we manage. Traditionally, it's been kind of a pain to acquire and manage a good selection of devices for testing. However, with Test Lab, you simply upload your APK and tests, and it will install and run them to our devices. After the tests complete, we'll provide all the screenshots, videos, and logs of everything that happened for you to examine in the Firebase console.

With Firebase Test Lab for Android now available with generous daily quotas at no charge for projects on the free Spark tier, 2017 is a great time to get started with that. And, if you haven't set up your Android app builds in a continuous integration environment, you could set that up, then configure it to run your tests automatically on Test Lab.

If you're the kind of person who likes writing tests for your code (which is, admittedly, not very many of us!), it's natural to get those tests running on Test Lab. But, for those of us who aren't maintaining a test suite with our codebase, we can still use Test Lab's automated Robo test to get automated test coverage right away, with no additional lines of code required. That's not quite that same as having a comprehensive suite of tests, so maybe 2017 would be a good time to learn more about architecting "testable" apps, and how those practices can raise the bar of quality for your app. I'm planning on writing more about this later this year, so stay tuned to the Firebase Blog for more!

Firebase Remote Config
At its core, Firebase Remote Config is a tool that lets you configure your app using parameters that you set up in the Firebase console. It can be used to help manage the quality of your app, and there's a couple neat tricks you can do with it. Maybe this new year brings new opportunities to give them a try!

First of all, you can use Remote Config to carefully roll out a new feature to your users. It works like this:
  1. Code your new feature and restrict its access to the user by a Remote Config boolean parameter. If the value is 'false', your users don't see the feature. Make 'false' the default value in the app.
  2. Configure that parameter in the Firebase console to also be initially 'false' for everyone.
  3. Publish your app to the store.
  4. When it's time to start rolling out the new feature to a small segment of users, configure the parameter to be 'true' for, say, five percent of your user base.
  5. Stay alert for new crashes in Firebase Crash Reporting, as well as feedback from your users.
  6. If there is a problem with the new feature, immediately roll back the new feature by setting the parameter to 'false' in the console for everyone.
  7. Or, if things are looking good, increase the percentage over time until you reach 100% of your users.


This is much safer than publishing your new feature to everyone with a single app update, because now you have the option to immediately disable a serious problem, and without having to build and publish a whole new version of your app. And, if you can act quickly, most of your users will never encounter the problem to begin with. This works well with the email alerts you get from Firebase Crash Reporting when a new crash is observed.

Another feature of Remote Config is the ability to experiment with some aspect of your app in order to find out what works better for the users of your app, then measure the results in Firebase Analytics. I don't know about you, but I'm typically pretty bad at guessing what people actually prefer, and sometimes I'm surprised at how people might actually *use* an app! Don't guess - instead, do an experiment and know /for certain/ what delights your users more! It stands to reason that apps finely tuned like this can get better ratings and make more money.

Firebase Realtime Database
It makes sense that if you make it easier for you user to perform tasks in your app, they will enjoy using it more, and they will come back more frequently. One thing I have always disliked is having to check for new information by refreshing, or navigating back and forward again. Apps that are always fresh and up to date, without requiring me to take action, are more pleasant to use.

You can achieve this for your app by making effective use of Firebase Realtime Databaseto deliver relevant data directly to your users at the moment it changes in the database. Realtime Database is reactive by nature, because the client API is designed for you set up listeners at data locations that get triggered in the event of a change. This is far more convenient than having to poll an API endpoint repeatedly to check for changes, and also much more respectful of the user's mobile data and battery life. Users associate this feeling of delight with apps of high quality.

What does 2017 have in store for your app?
I hope you'll join me this year in putting more effort into making our users even more delighted. If you're with me, feel free to tweet me at @CodingDoug and tell me what you're up to in 2017!
Categories: Programming

Introducing the ExifInterface Support Library

Android Developers Blog - Wed, 01/04/2017 - 18:50

Posted by Ian Lake, Developer Advocate

With the release of the 25.1.0 Support Library, there's a new entry in the family: the ExifInterface Support Library. With significant improvements introduced in Android 7.1 to the framework's ExifInterface, it only made sense to make those available to all API 9+ devices via the Support Library's ExifInterface.

The basics are still the same: the ability to read and write Exif tags embedded within image files: now with 140 different attributes (almost 100 of them new to Android 7.1/this Support Library!) including information about the camera itself, the camera settings, orientation, and GPS coordinates.

Camera Apps: Writing Exif Attributes

For Camera apps, the writing is probably the most important - writing attributes is still limited to JPEG image files. Now, normally you wouldn't need to use this during the actual camera capturing itself - you'd instead be calling the Camera2 API CaptureRequest.Builder.set() with JPEG_ORIENTATION, JPEG_GPS_LOCATION or the equivalents in the Camera1 Camera.Parameters. However, using ExifInterface allows you to make changes to the file after the fact (say, removing the location information on the user's request). Reading Exif Attributes

For the rest of us though, reading those attributes is going to be our bread-and-butter; this is where we see the biggest improvements.

Firstly, you can read Exif data from JPEG and raw images (specifically, DNG, CR2, NEF, NRW, ARW, RW2, ORF, PEF, SRW and RAF files). Under the hood, this was a major restructuring, removing all native dependencies and building an extensive test suite to ensure that everything actually works.

For apps that receive images from other apps with a content:// URI (such as those sent by apps that target API 24 or higher), ExifInterface now works directly off of an InputStream; this allows you to easily extract Exif information directly out of content:// URIs you receive without having to create a temporary file.

Uri uri; // the URI you've received from the other app
InputStream in;
try {
  in = getContentResolver().openInputStream(uri);
  ExifInterface exifInterface = new ExifInterface(in);
  // Now you can extract any Exif tag you want
  // Assuming the image is a JPEG or supported raw format
} catch (IOException e) {
  // Handle any errors
} finally {
  if (in != null) {
    try {
      in.close();
    } catch (IOException ignored) {}
  }
}

Note: ExifInterface will not work with remote InputStreams, such as those returned from a HttpURLConnection. It is strongly recommended to only use them with content:// or file:// URIs.

For most attributes, you'd simply use the getAttributeInt(), getAttributeDouble(), or getAttribute() (for Strings) methods as appropriate.

One of the most important attributes when it comes to displaying images is the image orientation, stored in the aptly-named TAG_ORIENTATION, which returns one of the ORIENTATION_ constants. To convert this to a rotation angle, you can post-process the value.

int rotation = 0;
int orientation = exifInterface.getAttributeInt(
    ExifInterface.TAG_ORIENTATION,
    ExifInterface.ORIENTATION_NORMAL);
switch (orientation) {
  case ExifInterface.ORIENTATION_ROTATE_90:
    rotation = 90;
    break;
  case ExifInterface.ORIENTATION_ROTATE_180:
    rotation = 180;
    break;
  case ExifInterface.ORIENTATION_ROTATE_270:
    rotation = 270;
    break;
}

There are some helper methods to extract values from specific Exif tags. For location data, the getLatLong() method gives you the latitude and longitude as floats and getAltitude() will give you the altitude in meters. Some images also embed a small thumbnail. You can check for its existence with hasThumbnail() and then extract the byte[] representation of the thumbnail with getThumbnail() - perfect to pass to BitmapFactory.decodeByteArray(). Working with Exif: Everything is optional

One thing that is important to understand with Exif data is that there are no required tags: each and every tag is optional - some services even specifically strip Exif data. Therefore throughout your code, you should always handle cases where there is no Exif data, either due to no data for a specific attribute or an image format that doesn't support Exif data at all (say, the ubiquitous PNGs or WebP images).

Add the ExifInterface Support Library to your project with the following dependency:

compile "com.android.support:exifinterface:25.1.0"

But when an Exif attribute is exactly what you need to prevent a mis-rotated image in your app, the ExifInterface Support Library is just what you need to #BuildBetterApps

Categories: Programming

Watch sessions from the Playtime 2016 events to learn how to succeed on Android & Google Play

Android Developers Blog - Wed, 01/04/2017 - 18:12
Posted by Patricia Correa, Head of Developer Marketing, Google Play

We’re wrapping up our annual global Playtime series of events with a last stop in Tokyo, Japan. This year Google Play hosted events in 10 cities: London, Paris, Berlin, Hong Kong, Singapore, Gurgaon, San Francisco, Sao Paulo, Seoul and Tokyo. We met with app and game developers from around the world to discuss how to build successful businesses on Google Play, share experiences, give feedback, collaborate, and get inspired. You can now watch some of the best Playtime sessions on our Android Developers YouTube Channel, as listed below. The playlist opens with a video that celebrates collaboration.


Keynote What’s next for Google Play

Learn how we're helping users discover apps in the right context, creating new ways to engage with users beyond the install, and powering innovative experiences on emerging platforms like virtual reality, wearables, and auto.

Develop and launch apps & games
Android development in 2016

Android development is more powerful and efficient than ever before. Android Studio brings you speed, smarts, and support for Android Nougat. The broad range of cross-platform tools on Firecase can improve your app on Android and beyond. Material Design and Vulkan continue to improve the user experience and increase engagement.

Daydream & Tango

Daydream View is a VR headset and controller by Google that lets people explore new worlds, or play games that put them at the center of action. Learn how we're helping users discover apps in the right context and powering new experiences with Daydream and Tango.

Fireside chat - Wayfair & Pokémon GO on augmented reality

Augmented reality engages and delights people everywhere. In this fireside chat, online furniture seller Wayfair and Niantic's Pokémon GO share their experiences with AR and discuss how other developers can make the most of the platform.

Building for billions, featuring best practices from Maliyo Games

Learn how to create apps and games for emerging markets, which are expected to drive 80% of global smartphone growth by 2020, by recognizing the key challenges and designing the right app experiences to overcome them.

At minute 16:41, hear tips from Hugo Obi, co-founder of Nigerian games developer Maliyo.

Launch smart on Google Play

Set your app up for success using experimentation and iteration. Learn best practices for soft launching and adapting your app for different markets and device types.

Apps
Sustainable growth solves most problems for apps, featuring best practices from SoundCloud & Peak

Planning and executing a great growth strategy involves a complex set of choices and mastery of many tools. In this session we discuss topics including key business objectives, tools, and techniques to help you solve the growth puzzle with our partner, SoundCloud.

Also, check out some growth best practices from Peak.

Creating sustainable user growth for startups, by Greylock

User growth isn't just about growing the number of users you have. The key to sustainability is creating and delivering core product value. In this session, VC Greylock discusses how to identify your core action to focus on and shows you how to use these insights to optimize your app for long term growth.

App engagement is the new black, featuring best practices from Lifesum

As the app marketplace becomes more competitive, developer success depends on retaining users in apps they love. Find out which Google tools and features can help you analyze your users' behaviors, improve engagement and retention in your app and hear insights from others developers including Lifesum.

Predicting lifetime value in the apps world

Deepdive into lifetime value models and predictive analytics in the apps ecosystem. Tactics to get the most out of identified segments and how to upgrade their behaviors to minimize churn.

Subscriptions update

Learn about Google's efforts to enable users, around the world, to seamlessly and safely pay for content. This session provides updates on Google Play billing and recent enhancements to our subscriptions platform.

Games
One game fits all, featuring best practices from Space Ape Games

Customize your game's experience for different users by targeting them with lifetime value models and predictive analytics. Hear how these concepts are applied by Space Ape Games to improve retention and monetization of their titles.

Promoting your game and growing your user base, featuring best practices from Seriously

Learn how to use Google's latest tools, like Firebase, for benchmarking, acquiring users and measuring your activities. Also, hear game developer Seriously share their latest insights and strategies on YouTube influencer campaigns.

Creating long-term retention, loyalty and value with engaging LiveOps events, featuring best practices from Kabam & Creative Mobile

Learn how successful developers keep their games fresh and engaging with Live Operations. In this talk, the LiveOps expert on Marvel: Contest of Champions discusses tips about the art and science of running an engaging LiveOps event.

Also check out the tips and best practices to run successful LiveOps from games developer Creative Mobile.

Panel - Play fair: Maintaining a level playing field in your game, featuring Space Ape Games and Kongregate

Ensuring that your game is fair is critical to success. Find out how game developers are achieving this and some ways Google Play can help.

Families
Why you need to build for families

Family-based households with children have higher tablet and smartphone ownership rates than the general population. These families are more likely to make purchases on their mobile devices and play games. Learn about how parents choose what to download and buy, and how you can prepare for maximum conversion.

Two keys to growth: user acquisition & app engagement, by Cartoon Network

Hear how Cartoon Network leverages their network to cross-promote new titles, acquire new users and keep them engaged through immersive experiences.

Go global: Getting ready for the emerging markets revolution, by Papumba

Papumba has a clear vision to grow a global business. Hear how they work with experts to adapt their games to local markets and leverage Google Play's developer tools to find success around the world.

Optimizing for a post install world

You've spent time and resources getting users to download your apps, but what happens after the install? Learn how to minimize churn and keep families engaged with your content long term.

Monetization best practices on freemium, by 01 Digital

Learn how 01 Digital uses In-App-Purchases (IAP) to effectively monetize their apps while maintaining a safe environment for families.

Building a subscription business that appeals to parents, by PlayKids

PlayKids has been at the forefront of the subscription business model since their inception. See how they best serve their subscribers by refreshing their content, expanding their offerings and investing in new verticals.


How useful did you find this blogpost?




Categories: Programming

Five steps to achieve sustainable growth and boost your app's long term success

Android Developers Blog - Wed, 01/04/2017 - 18:04
Maxim Mai, Business Development Manager, Google Play

Maintaining sustainable growth is difficult for even the highest quality apps. In this video and through the 5 steps below you can find out how some of our leading Android developers are tackling growth.


1) Understand and define your app's objectives Depending on your product lifecycle stage you will most likely focus on these 3 growth goals with varying intensity:
  • Acquire new users
  • Increase engagement and retention
  • Grow revenue
2) Track and measure your tactics against each of your objectives List out the tactics you're using to achieve each objective and keep track of their performance. You can visualize it using a scorecard like in the example below created by Mobile Growth Stack.


3) Apply your growth tactics. Here are a few examples of specific tactics developers have successfully used to drive sustained growth.
Tactic: Clue, a female health app, invests in the Play store listing to increase conversions.
Results: 24% aggregate increase in install conversion rate over a period of 6 months.

How they did it:



Which phone screenshot do you think drove increase in install conversion for Clue?

Tactic: ShareTheMeal, a non-profit app developed by the World Food Programme, uses public relations as a free sustainable acquisition channel.

Results: 50% of their total installs to date were driven by media coverage.

How they did it:
  • Developed an excellent messaging.
  • Boosted installs impact by combining PR with celebrity outreach and distribution partnership.
  • Learned that TV coverage has the highest impact on installs but print is a useful door opener to amplify TV coverage.
Tactic: Viral growth. Virality is a core growth tool for apps and games that focus on sharing and usually the mechanic is built into the core user experience of the product. However, even if sharing isn't a key component of your app, you can still influence two key variables to create an appropriate environment to encourage virality.

How to do it:
  • Increase the number of additional users that a single user brings to the app, by boosting the number of invitations sent.
  • Decrease your "cycle time", how long it takes between inviting a user and that user sending out the next round of invitations to their friends.
  • Offer more incentives for users to share the app or its content while they're using it will help shorten the cycle time and kickstart viral growth!
Tactic: Freeletics Nutrition, an app to adjust your nutrition to your individual needs and goals, uses cross-promotion to accelerate the launch of a new product.

Results: 96% of new Nutrition app sales generated by users who originally registered for the developer's Bodyweight training app.

How they did it:
  • Surface meal advice in the Bodyweight app's activity feed with the goal of raising awareness for the approaching launch of their new Freeletics Nutrition app.


4) Build a strong growth culture To make sustainable growth work for your app, it needs to be a part of your culture. Runtastic is one of the leading health and fitness app developers in Europe and 95% of their approximately 76M total app installs on Google Play have been generated organically. Mario Aichlseder, VP of Growth, believes this is the result of a strong growth culture and the growth principles according to which all teams operate. For example, product managers, designers and engineers at Runtastic deliberately chose a mixture of qualitative and quantitative feedback loops during the app development process to ensure they stay true to their growth principles.

5) Adjust along the way It's important to track your tactics against real metrics to measure your impact. That will help you make decisions about where to increase or decrease your efforts. Your priorities will also change based on the evolution of your business and product lifecycle as well as due to external factors such as new techniques becoming available, so be open to regularly adjusting your tactics.
Get more tips and best practices in the sessions from this year's Playtime events.

How useful did you find this blogpost?



Categories: Programming

Tips to be better found and discovered on Google Play

Android Developers Blog - Wed, 01/04/2017 - 18:00
Posted by Andrew Ahn, Product Manager, Google Play

We're constantly working on ways to make Google Play a great place for users to discover apps and games they'll love. We know this is crucial to most developers ongoing success. There are steps you can take to ensure your app is primed for success – that's why we're sharing a reminder of some of our top tips for getting your app discovered on Google Play.

Build for quality

First, build for quality. Android users expect high-quality apps. App quality directly influences the long-term success of your app - in terms of installs, user rating and reviews, engagement, and user retention. These are some of the factors that go into our search and discovery systems that help discern what apps to recommend and surface across our Google Play experiences. When building your app, check against the quality criteria, and use what you need from the material design guidelines to make sure you are delivering a highly usable experience. Also, be sure to test your app for functional quality. Opt-in to the pre-launch report for your alpha and beta apps in the Google Play Developer Console and you'll receive a report for each APK showing how it performs on real devices. This will help you identify crashes and other issues before you release your app.


Example: Designing for high usability through Google Material Design.
Request only the permissions you need

Second, be considerate on which permission settings to enable for your app. We see that there are some apps that ask for very sensitive permissions, even when the app doesn't use them. (For example, a camera app asking for read and write permissions to call logs.) Excessive app permissions may dissuade users from installing your app. In fact, one study, in which users were shown two unbranded apps with similar ratings that had the same functionality but different sets of permission requests, showed that users were, on average, 3 times more likely to install the app with fewer permissions requests. And a similar study showed that users are 1.7 times more likely, on average, to select the application with fewer permission requests. The rule of thumb is to enable permissions that are only essential to your app. Read the best practices for app permissions.


Chart: Distribution of permission groups use across Arcade Games category. If you're building an arcade game, you many only need a very few permission settings, if any.
Listen and respond to your users

Lastly, be attentive to user feedback. It's ultimately the users who drive our search and discovery systems. When you hear user feedback about bugs or other issues, we recommend engaging with the feedback and, if needed, updating your app in a timely manner. Having an up-to-date app that reflects your user's feedback can help you gain more installs, engagement, and higher ratings. Beta testing is a good way to get feedback from real users before launch. You can also check the ratings and reviews section of the Developer Console to see an analysis of what users are saying about your app and how that is affecting your rating compared to similar apps.

Review benchmarks in the Developer Console uses machine learning to give you insights about what users are saying about your app and how it affects your rating.

Google Play strives to help users find and discover the most safe, high quality, useful, and relevant apps. Building apps that put user's interest first will help you be successful in Google Play. For more tips and best practices for building a successful app business on Google Play, get the Playbook for Developers app.

How useful did you find this blogpost?

Categories: Programming

Get a glimpse of Wear 2.0’s upcoming standalone apps

Android Developers Blog - Wed, 01/04/2017 - 17:58
Kacey Fahey, Marketing Programs Manager, Google Play

The upcoming Android Wear 2.0 experience will introduce standalone apps, expanding your potential reach to both Android and iOS audiences with Wear devices. Users will be able to search, install, and use apps without ever leaving their device. See how other developers are enhancing their user experience with standalone apps for messaging, travel & local, and health & fitness.
Glide

Having a watch app further simplifies video messaging with Glide. Using the Wear Complications API, Glide is now able to live broadcast directly from the watch face. By tapping contact shortcuts from the watch face, you can now launch directly into a conversation. This experience brings speed and intimacy to the world of messaging, making wrist-based communication more accessible and effortless.

Foursquare

Travelers around the world use Foursquare's Android Wear app to discover hidden gems and be in the know about the best places to eat, drink and explore. With their upcoming 2.0 app, the team has a clean new canvas for rich notifications giving users an immersive experience with Foursquare content.

"The standalone nature of the Android Wear 2.0 app will offer a big boost in search performance and app responsiveness so you spend less time staring at the screen and more time exploring the world around you," said Kyle Fowler, Software Engineer at Foursquare.

Lifesum

Lifesum helps users make better food choices, improve their exercise, and reach health goals. The upcoming 2.0 experience complements the existing Lifesum mobile app and as a standalone app, it will allow users to more easily track water and meals throughout the day.
"It's all about increasing access and being there for the user in a quick and simple way. We believe a simplified way of tracking meals and water will make it easier for our users on their journey of becoming healthier and happier," said Joakim Hammer, Android Developer at Lifesum

Check out g.co/wearpreview for the latest builds and documentation about the recently released Android Wear Developer Preview 4.
How useful did you find this blogpost?


Categories: Programming

Four tips for startup success from a Silicon Valley VC

Android Developers Blog - Wed, 01/04/2017 - 17:55
Posted by Kacey Fahey, Marketing Programs Manager, Google Play

Working at Google Play, we’re on the front line watching developers build, polish, and launch their dreams for millions of users to experience. While it’s exciting to be a part of so much creativity, we’re often asked how small startups can stand out in such a competitive field. We recently had Josh Elman & Sarah Tavel of Greylock Partners speak at our events, sharing their experiences working in Product Marketing and Engineering at major tech companies including Twitter, Facebook and Pinterest. Below are four tips to hit the ground running and create a business built for success.

Set goals, both large and small
Every startup has an ultimate goal, but don’t forget to create micro-goals. Breaking your larger goal down into smaller milestones creates checkpoints to review progress and ensure momentum is heading in the right direction. This also allows for flexibility if teams need to course correct along the way, not to mention micro-accomplishments to celebrate!

Create stickiness
The first level in Sarah’s Hierarchy of Engagement is to identify the core action for users to perform in your app. Once you have engagement with this core action, level 2 is driving retention, getting users to come back and perform the core action more and more. The ultimate goal is to hook users with your app creating accruing benefits, whereby deeper and more frequent engagement creates habits and product dependencies.




“As companies move up the hierarchy, their products become better, harder to leave, and ultimately create virtuous loops that make the product self-perpetuating,” – Sarah Tavel, Partner at Greylock

Example: For those looking to improve on organizational skills, Evernote can be a lifesaver. The more lists users create, the more they rely on the product. Evernote becomes such an ingrained habit that it naturally transcends between personal and professional worlds.

Drive virality
When launching a new app, look for ways to achieve virality. Find hooks to make users fall in love with your app and strive to make it part of their regular habits. But watch out, not all types of virality are treated equally.

“Whenever you’re thinking about engineering virality, you need to be sure that you’re reaching the right people, getting them interested for reasons that align with the intrinsic value of your product, and leading them to the right actions,” – Josh Elman, Partner at Greylock Example: Whether you’re lucky enough to convert happy users into product evangelists or catch fire through social media, outbreak virality has driven tremendous success for apps like Pokémon GO and Prisma.

Measure cohorts
While monitoring traditional mobile metrics such as installs and DAUs provide a high level overview of app performance, cohort analysis is key to understanding user behavior and optimizing for growth. When rolling out changes in your app, make sure to track cohorts for an extended duration. Initial results may tell one story at D7, but hold tight, as things could turn a corner by D15 or even later. Give users time to adapt and get comfortable with the changes before making any final product decisions.

Read more tips on how to find success for your app or game start up in the Playbook for Developers app.

How useful did you find this blogpost?



Categories: Programming

Best practices to improve app engagement

Android Developers Blog - Wed, 01/04/2017 - 17:54
Posted by Niko Schröer, Business Development, Google Play

Driving installs is important to growing a user base, but it's not much use if your app sits on users' devices and is rarely opened. In a competitive app landscape, it's increasingly important to engage and retain users over the long term to build a successful business. Users who are using your app more will have a higher lifetime value and be more likely to share your app. Watch my Playtime session below to hear about the tools and features other developers are using to increase app engagement. You can also read the summary of my main tips below.


1. Build a high quality app to engage Android users
Building a high quality app is the foundation of a great user experience on Android. The better your app's user experience is, the more engaged your users will be. Optimizing for material design, for example, can significantly improve user engagement as well as building for Android Wear, Auto or TV where it makes sense based on your value proposition.

To achieve high quality, we recommend you to check out the latest Android features, tips, and best practices in our Playbook for Developers.

The developer of the golf app, Hole19, tailored their app's user experience thoughtfully for Android Wear and, as a result, saw a 40% increase in user engagement compared to non-Wear users. Watch a video about Hole19's success.

2. Make your users feel at home
Personalising your app experience to make users feel at home is a good way to start a long lasting relationship. Onboarding new users is a crucial step in this process. Onboarding should be fast and seamless and ask for minimal user input - after all users want to start using your app as quickly as possible. Furthermore, the onboarding should be a core part of the overall product experience. Use images and wording that's true to your brand and only ask for user input when it's actually needed, to reduce friction and avoid losing users.

Freeletics, a fitness app, created an engaging user onboarding flow in which they tailored imagery and text to male and female users respectively. They also moved the registration process to a later stage in the funnel to reduce friction. The improved onboarding flow increased user activity by 58% within the first 7 days. They also implemented Google Smart Lock to seamlessly sign-in returning users.

3. Optimize feature releases as a way to increase user engagement
Introducing new features is essential to staying ahead of competition and relevant to your users to ensure they keep coming back to your app. To make new feature launches successful drivers for user engagement, follow these simple steps:
  • Define a clear objective for each release to measure your impact, e.g. increase number of users who edit a photo by at least 10%.
  • Use beta testing to gather user feedback and iterate a feature before it's rolled out to all of your users.
  • Enable the pre-launch report in the Play developer console to spot potential flaws and ensure technical stability in your alpha and beta apps.
  • Guide users to each new feature as if it is a light onboarding experience. Visually highlight what's new and provide a short explanation why users should care.
  • Measure performance with analytics to see if the new feature drives engagement (that you've defined as your objective).
4. Use notifications wisely
Push notifications are a popular engagement tool and rightfully so. However, there is a fine line between driving engagement and annoying users (who might then uninstall your app). Follow these guidelines to ensure your notifications are on the right side of the line:
  • Be relevant and only send messages that matter to the user in context. Be creative and true to your brand, speak your users language and use an authentic tone.
  • Make notifications actionable for your users and don't forget to deep link to content where applicable to save your users time.
  • Remember that not all your users are equal so personalize your message to different user cohorts with Firebase Notifications.
  • Consider timeliness of your messages to get users the right notification at the right time and with the right frequency. For example, it might be better to send a notification about something interesting to read at a time when the user normally gets out their phone – like during their commute – instead of the middle of the day, when they might be busy and dismiss a new notification.
  • Finally, give users control over what notifications they receive so that they can opt-in and opt-out of the notifications they like and don't like respectively. If users get annoyed about certain types of notifications and don't have a way to disable them, they might uninstall your app.
The Norwegian news app Aftenposten implemented a new onboarding flow that clarified which notifications were available, allowing readers to manage their preferences. This reduced uninstalls by 9.2.% over 60 days and led to a 28% decrease in the number of users muting notifications completely. Read more about Aftenposten's success.

5. Reward your most engaged users
Last but not least, you should find ways to reward your most loyal users to retain them over time and to make it desirable to less engaged users to engage more. These rewards can come in many shapes and forms. Start by keeping it simple and make sure the reward adds real value to the user and fits in your app's ecosystem. You can do this by:
  • Giving sneak peeks of new features by inviting them to a beta group.
  • Decorating user accounts with badges based on their behaviour.
  • Offer app exclusive discounts or promo codes that can only be redeemed in your app.
Generally, the more you can personalize the reward the better it will work.

Find success with ongoing experimentation
A great Android app gives developers a unique opportunity to create a lasting relationship with users and build a sustainable business with happy customers. Therefore optimising apps to engage and retain your users by following these 5 tips should be front and centre of your development goals and company strategy. Find more tips and best practices by watching the sessions at this year's Playtime events.
How useful did you find this blogpost?




Categories: Programming

Matching strings in Scala

Xebia Blog - Mon, 01/02/2017 - 21:31
Over December I had a lot of fun doing the Advent of Code coding challenges with some colleagues. Many of those, such as day 21, require interpreting some kind of string input. While normally I'd probably marshall those strings into case classes before processing, in this case that seemed like overkill: a quick pattern-match should

Hack a Happy New Year!

“Whatever you can do, or dream you can, begin it. Boldness has genius, power and magic in it.” — Johann Wolfgang Von Goethe

Hack a Better New Year

It’s time to dig down, dig in, and dig deep to create a great year for yourself and others.

I’m a fan of hacks for work and life.

I find that hacking away at challenges is a great way to make progress and to eventually overcome them.

Hacking is really an a approach and a mindset where you try new things, experiment and explore while staying open-minded and learning as you go.

You never really know what’s going to work, until you’ve actually made it work.

Nothing beats personal experimentation when it comes to creating better results in your life.

Anyway, in the spirit of kicking off the new year right, I created a comprehensive collection of the ultimate hacks for a happy new year:

101 Hacks for a Happy New Year

This is no ordinary set of hacks.  It’s deep.  There are hacks for mind, body, emotions, career, finance, relationships, and fun.

There are hacks you can use everyday to change how you think, feel, and act.

There are hacks to help you change habits.

There are hacks to help you relight your fire and get back in the game, if you’ve been in a slump or waiting on the sidelines.

Jump back in the game, master your work and life, and have some fun in the process.

Here is a quick list of the hacks from 101 Hacks for a Happy New Year:

1. Get the power of a New Year’s Resolution on your side
2. Limit yourself to one big resolution at a time
3. Get specific with your goals
4. Dream bigger to realize your potential
5. If you want change, you must change
6. Guide your path with vision, values, and goals
7. Change a habit with Habit Stacking
8. Create mini-feedback loops
9. Bounce back from a setback
10. Avoid “All or Nothing” thinking
11. Choose progress over perfection
12. Reward yourself more often
13. Gamify it
14. Adopt a Tiny Habit
15. Just Start
16. Adopt a growth mindset
17. Create if-then plans to stick with your goals
18. Start with Great Expectations
19. Adopt 7 beliefs for personal excellence
20. Master the art of goal planning
21. Prime your mind for greatness
22. Use dreams, goals, and habits to pull you forward
23. Use the Exponential Results Formula to make a big change
24. Adopt the 7 Habits of Highly Motivated People
25. Use Trigger Moments to activate your higher self
26. Use Door Frame Triggers to inspire a better version of you
27. Find your purpose
28. Figure out what you really want
29. Use 3 Wins to Rule Your Year
30. Commit to your best year ever
31. Find your Signature Strengths
32. Practice a “lighter feeling”
33. Let go of regrets
34. 15-Minutes of Fulfillment
35. Create your ideal day the Tony Robbins Way
36. Master your emotions for power, passion, and strength
37. Start your year in February
38. Build your personal effectiveness toolbox
39. Write your story for the future
40. Get out of a slump
41. Give your best, where you have your best to give
42. Ask more empowering questions
43. Surround yourself with better people
44. Find better mentors
45. Do the opposite
46. Try a 30 Day Sprint
47. Grow 10 Years Younger
48. Don’t get sick this year
49. Know Thyself
50. Decide Who You Are
51. Decide Who You Want To Be
52. Cultivate an Attitude of Gratitude
53. Try 20-Minute Sprints
54. Create a vision board for your year
55. Adopt some meaningful mantras and affirmations
56. Practice your mindfulness
57. 15-Minutes of Happiness
58. Breathe better
59. Become your own gym
60. Master your wealth
61. Learn how to read faster
62. Let go of negative feelings
63. Live a meaningful life
64. Establish a routine for eating, sleeping, and exercising
65. Improve your likeability
66. Win friends and influence people
67. Improve your charisma through power, presence, and warmth
68. Fill your mind with a few good good thoughts
69. Ask for help more effectively
70. Attract everything you’ve ever wanted
71. Catch the next train
72. Unleash You 2.0
73. Learn anything in 20 hours
74. Use stress to be your best
75. Take worry breaks
76. Use the Rule of Three to rule your day
77. Have better days
78. Read 5 powerful personal development books
79. Practice the 10 Skills of Personal Leadership
80.  Develop your Emotional Intelligence
81. Cap your day with four powerful questions
82. Build mental toughness like a Navy Seal
83. Feel In Control
84. Transform your job
85. Use work as your ultimate form of self-expression
86. Be the one who gives their all
87. Live without the fear of death in your heart
88. Find your personal high-performance pattern
89. Create unshakeable confidence
90. Lead a charged life
91. Use feedback to be your best
92. Make better decisions
93. Learn how to deal with difficult people
94. Defeat decision fatigue
95. Make the most of luck
96. Develop your spiritual intelligence
97. Conquer your fears
98. Deal with tough criticism
99. Embrace the effort
100. Finding truth from the B.S.
101. Visualize more effectively

For the details of each hack, check out 101 Hacks for a Happy New Year.

I will likely tune and prune the hacks over time, and improve the titles and the descriptions.

Meanwhile, I’m not letting perfectionism get in the way of progress.

Go forth and hack a happy new year and share 101 Hacks for a Happy New Year with a friend.

Categories: Architecture, Programming

A Guide to Generating Boring Code with Node

Xebia Blog - Fri, 12/30/2016 - 12:47
Writing boring code is not fun. Especially if that code is just a derivative of some data. You should generate it instead! A great example is to generate your HTTP client code from an API definition. This is actually very easy to do with a recent node version and requires almost no extra dependencies. In

The Best New Years Resolution: Agile Product Management

Xebia Blog - Wed, 12/28/2016 - 15:44
Agile Product Management is grounded in the Jobs to be done theory and Lean startup principles. In my book “The Product Samurai” I described how you can effectively apply these techniques to be a better Product Manager, but what I didn’t’ cover was why not everybody is doing this already? Making up for that, and

Being An Agile Security Officer: Security Stakeholdership mindset

Xebia Blog - Tue, 12/27/2016 - 16:23
This is the second part in my blog series about 'being an agile security officer'. In this blog I will focus on the mindset of security stakeholdership in Agile and DevOps environments. In the Agile world the Product Owner is the person who translates business and customer desires into work items (user stories) for the

How to integrate Jmeter-maven printing HTML reports project with DevOps CI/CD pipeline in 4 easy steps

Xebia Blog - Tue, 12/27/2016 - 08:36
(1) Fork the below github repo having pom.xml file which contains all the dependencies of Jmeter and printing the Jmeter reports in HTML format https://github.com/nishantguptaxe/JmeterMavenHtmlReports.git (2) Add your Jmeter file under the src/test/jmeter folder and checkin your code into the github (3) Install the go server, start the go server ,hit the go url in

Go: First attempt at channels

Mark Needham - Sat, 12/24/2016 - 11:45

In a previous blog post I mentioned that I wanted to extract blips from The ThoughtWorks Radar into a CSV file and I thought this would be a good mini project for me to practice using Go.

In particular I wanted to try using channels and this seemed like a good chance to do that.

I watched a talk by Rob Pike on designing concurrent applications where he uses the following definition of concurrency:

Concurrency is a way to structure a program by breaking it into pieces that can be executed independently.

He then demonstrates this with the following diagram:

2016 12 23 19 52 30

I broke the scraping application down into four parts:

  1. Find the links of blips to download ->
  2. Download the blips ->
  3. Scrape the data from each page ->
  4. Write the data into a CSV file

I don’t think we gain much by parallelising steps 1) or 4) but steps 2) and 3) seem easily parallelisable. Therefore we’ll use a single goroutine for steps 1) and 4) and multiple goroutines for steps 2) and 3).

We’ll create two channels:

  • filesToScrape
  • filesScraped

And they will interact with our components like this:

  • 2) will write the path of the downloaded files into filesToScape
  • 3) will read from filesToScrape and write the scraped content into filesScraped
  • 4) will read from filesScraped and put that information into a CSV file.


I decided to write a completely serial version of the scraping application first so that I could compare it to the parallel version. I had the following common code:

scrape/scrape.go

package scrape

import (
	"github.com/PuerkitoBio/goquery"
	"os"
	"bufio"
	"fmt"
	"log"
	"strings"
	"net/http"
	"io"
)

func checkError(err error) {
	if err != nil {
		fmt.Println(err)
		log.Fatal(err)
	}
}

type Blip struct {
	Link  string
	Title string
}

func (blip Blip) Download() File {
	parts := strings.Split(blip.Link, "/")
	fileName := "rawData/items/" + parts[len(parts) - 1]

	if _, err := os.Stat(fileName); os.IsNotExist(err) {
		resp, err := http.Get("http://www.thoughtworks.com" + blip.Link)
		checkError(err)
		body := resp.Body

		file, err := os.Create(fileName)
		checkError(err)

		io.Copy(bufio.NewWriter(file), body)
		file.Close()
		body.Close()
	}

	return File{Title: blip.Title, Path: fileName }
}

type File struct {
	Title string
	Path  string
}

func (fileToScrape File ) Scrape() ScrapedFile {
	file, err := os.Open(fileToScrape.Path)
	checkError(err)

	doc, err := goquery.NewDocumentFromReader(bufio.NewReader(file))
	checkError(err)
	file.Close()

	var entries []map[string]string
	doc.Find("div.blip-timeline-item").Each(func(i int, s *goquery.Selection) {
		entry := make(map[string]string, 0)
		entry["time"] = s.Find("div.blip-timeline-item__time").First().Text()
		entry["outcome"] = strings.Trim(s.Find("div.blip-timeline-item__ring span").First().Text(), " ")
		entry["description"] = s.Find("div.blip-timeline-item__lead").First().Text()
		entries = append(entries, entry)
	})

	return ScrapedFile{File:fileToScrape, Entries:entries}
}

type ScrapedFile struct {
	File    File
	Entries []map[string]string
}

func FindBlips(pathToRadar string) []Blip {
	blips := make([]Blip, 0)

	file, err := os.Open(pathToRadar)
	checkError(err)

	doc, err := goquery.NewDocumentFromReader(bufio.NewReader(file))
	checkError(err)

	doc.Find(".blip").Each(func(i int, s *goquery.Selection) {
		item := s.Find("a")
		title := item.Text()
		link, _ := item.Attr("href")
		blips = append(blips, Blip{Title: title, Link: link })
	})

	return blips
}

Note that we’re using the goquery library to scrape the HTML files that we download.

A Blip is used to represent an item that appears on the radar e.g. .NET Core. A File is a representation of that blip on my local file system and a ScrapedFile contains the local representation of a blip and has an array containing every appearance the blip has made in radars over time.

Let’s have a look at the single threaded version of the scraper:

cmd/single/main.go

package main

import (
	"fmt"
	"encoding/csv"
	"os"
	"github.com/mneedham/neo4j-thoughtworks-radar/scrape"
)


func main() {
	var filesCompleted chan scrape.ScrapedFile = make(chan scrape.ScrapedFile)
	defer close(filesCompleted)

	blips := scrape.FindBlips("rawData/twRadar.html")

	var filesToScrape []scrape.File
	for _, blip := range blips {
		filesToScrape = append(filesToScrape, blip.Download())
	}

	var filesScraped []scrape.ScrapedFile
	for _, file := range filesToScrape {
		filesScraped = append(filesScraped, file.Scrape())
	}

	blipsCsvFile, _ := os.Create("import/blipsSingle.csv")
	writer := csv.NewWriter(blipsCsvFile)
	defer blipsCsvFile.Close()

	writer.Write([]string{"technology", "date", "suggestion" })
	for _, scrapedFile := range filesScraped {
		fmt.Println(scrapedFile.File.Title)
		for _, blip := range scrapedFile.Entries {
			writer.Write([]string{scrapedFile.File.Title, blip["time"], blip["outcome"] })
		}
	}
	writer.Flush()
}

rawData/twRadar.html is a local copy of the A-Z page which contains all the blips. This version is reasonably simple: we create an array containing all the blips, scrape them into another array, and then that array into a CSV file. And if we run it:

$ time go run cmd/single/main.go 

real	3m10.354s
user	0m1.140s
sys	0m0.586s

$ head -n10 import/blipsSingle.csv 
technology,date,suggestion
.NET Core,Nov 2016,Assess
.NET Core,Nov 2015,Assess
.NET Core,May 2015,Assess
A single CI instance for all teams,Nov 2016,Hold
A single CI instance for all teams,Apr 2016,Hold
Acceptance test of journeys,Mar 2012,Trial
Acceptance test of journeys,Jul 2011,Trial
Acceptance test of journeys,Jan 2011,Trial
Accumulate-only data,Nov 2015,Assess

It takes a few minutes and most of the time will be taken in the blip.Download() function – work which is easily parallelisable. Let’s have a look at the parallel version where goroutines use channels to communicate with each other:

cmd/parallel/main.go

package main

import (
	"os"
	"encoding/csv"
	"github.com/mneedham/neo4j-thoughtworks-radar/scrape"
)

func main() {
	var filesToScrape chan scrape.File = make(chan scrape.File)
	var filesScraped chan scrape.ScrapedFile = make(chan scrape.ScrapedFile)
	defer close(filesToScrape)
	defer close(filesScraped)

	blips := scrape.FindBlips("rawData/twRadar.html")

	for _, blip := range blips {
		go func(blip scrape.Blip) { filesToScrape <- blip.Download() }(blip)
	}

	for i := 0; i < len(blips); i++ {
		select {
		case file := <-filesToScrape:
			go func(file scrape.File) { filesScraped <- file.Scrape() }(file)
		}
	}

	blipsCsvFile, _ := os.Create("import/blips.csv")
	writer := csv.NewWriter(blipsCsvFile)
	defer blipsCsvFile.Close()

	writer.Write([]string{"technology", "date", "suggestion" })
	for i := 0; i < len(blips); i++ {
		select {
		case scrapedFile := <-filesScraped:
			for _, blip := range scrapedFile.Entries {
				writer.Write([]string{scrapedFile.File.Title, blip["time"], blip["outcome"] })
			}
		}
	}
	writer.Flush()
}

Let's remove the files we just downloaded and give this version a try.

$ rm rawData/items/*

$ time go run cmd/parallel/main.go 

real	0m6.689s
user	0m2.544s
sys	0m0.904s

$ head -n10 import/blips.csv 
technology,date,suggestion
Zucchini,Oct 2012,Assess
Reactive Extensions for .Net,May 2013,Assess
Manual infrastructure management,Mar 2012,Hold
Manual infrastructure management,Jul 2011,Hold
JavaScript micro frameworks,Oct 2012,Trial
JavaScript micro frameworks,Mar 2012,Trial
NPM for all the things,Apr 2016,Trial
NPM for all the things,Nov 2015,Trial
PowerShell,Mar 2012,Trial

So we're down from 190 seconds to 7 seconds, pretty cool! One interesting thing is that the order of the values in the CSV file will be different since the goroutines won't necessarily come back in the same order that they were launched. We do end up with the same number of values:

$ wc -l import/blips.csv 
    1361 import/blips.csv

$ wc -l import/blipsSingle.csv 
    1361 import/blipsSingle.csv

And we can check that the contents are identical:

$ cat import/blipsSingle.csv  | sort > /tmp/blipsSingle.csv

$ cat import/blips.csv  | sort > /tmp/blips.csv

$ diff /tmp/blips.csv /tmp/blipsSingle.csv 


The code in this post is all on github. I'm sure I've made some mistakes/there are ways that this could be done better so do let me know in the comments or I'm @markhneedham on twitter.

The post Go: First attempt at channels appeared first on Mark Needham.

Categories: Programming