Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Feed aggregator

5 tips for launching successful apps and games on Google Play

Android Developers Blog - Fri, 04/07/2017 - 17:38

Posted by Adam Gutterman, Go-To-Market Strategic Lead, Google Play Games

Last month at the Game Developers Conference (GDC), we held a developer panel focused on sharing best practices for building successful app and game businesses. Check out 5 tips for developers, both large and small, as shared by our gaming partners at Electronic Arts (EA), Hutch Games, Nix Hydra, Space Ape Games and Omnidrone.



1. Test, test, test
The best time to test, is before you launch; so test boldly and test a lot! Nix Hydra recommends testing creative, including art style and messaging, as well as gameplay mechanics, onboarding flows and anything else you're not sure about. Gathering feedback from real users in advance of launching can highlight what's working and what can be improved to ensure your game's in the best shape possible at launch.
2. Store listing experiments
Run experiments on all of your store listing page assets. Taking bold risks instead of making assumptions allows you to see the impact of different variables with your actual user base on Google Play. Test in different regions to ensure your store listing page is optimized for each major market, as they often perform differently.

3. Early Access program

Space Ape Games recently used Early Access to test different onboarding experiences and gameplay control methods in their game. Finding the right combination led them to double-digit growth in D1 retention. Gathering these results in advance of launch helped the team fine tune and polish the game, minimizing risk before releasing to the masses.

"Early Access is cool because you can ask the big questions and get real answers from real players," Joe Raeburn, Founding Product Guy at Space Ape Games.
Watch the Android Developer Story below to hear how Omnidrone benefits from Early Access using strong user feedback to improve retention, engagement and monetization in their game.


Mobile game developer Omnidrone benefits from Early Access.
4. Pre-registration

Electronic Arts has run more than 5 pre-registration campaigns on Google Play. Pre-registration allows them to start marketing and build awareness for titles with a clear call-to-action before launch. This gives them a running start on launch day having built a group of users to activate upon the game's release resulting in a jump in D1 installs.

5. Seek feedback

All partners strongly recommended seeking feedback early and often. Feedback tells both sides of the story, by pointing out what's broken as well as what you're doing right. Find the right time and channels to request feedback, whether they be in-game, social, email, or even through reading and responding to reviews within the Google Play store.

If you're a startup who has an upcoming launch on Google Play or has launched an app or game recently and you're interested in opportunities like Early Access and pre-registration, get in touch with us so we can work with you.

Watch sessions from Google Developer Day at GDC17 on the Android Developers YT channel to learn tips for success. Also, visit the Android Developers website to stay up-to-date with features and best practices that will help you grow a successful business on Google Play.


How useful did you find this blogpost?         

Categories: Programming

Stuff The Internet Says On Scalability For April 7th, 2017

Hey, it's HighScalability time:

 

Visualization of the magic system behind software infrastructure. (eyezmaze@ThePracticalDev
If you like this sort of Stuff then please support me on Patreon.
  • 10-20: aminoacids can be made per second; 64800x: faster DDL Aurora vs MySQL; 25 TFLOPS: cap for F1 simulations; 15x to 30x: Tensor Processing Unit faster than GPUs and CPUs; 100 Million: Intel transistors per square millimeter; 25%: Internet traffic generated by Google; $1 million: Tim Berners-Lee wins Turing Award; 43%: phones FBI couldn't open because of crypto;

  • Quotable Quotes:
    • @adulau: To summarize the discussions of yesterday. All tor exit nodes are evil except the ones I operate.
    • @sinavaziri: Let's say a data center costs $1-2B. Then the TPU saved Google $15-30B of capex?
    • Vinton G. Cerf: While it would be a vast overstatement to ascribe all this innovation to genetic disposition, it seems to me inarguable that much of our profession was born in the fecund minds of emigrants coming to America and to the West over the past century.
    • Alan Bundy: AI systems are not just narrowly focused by design, because we have yet to accomplish artificial general intelligence, a goal that still looks distant. 
    • JamesBarney: Soo much this, just worked on a project that sacrificed reliability, maintainability, and scalability to use a real time database to deal with loads that were on the order of 70 values or 7 writes a second.
    • bobdole1234: 3.5x faster than CPU doesn't sound special, but when you're building inference capacity by the megawatt, you get a lot more of that 3.5x faster TPU inside that hard power constraint.
    • Eugenio Culurciello: As we have been predicting for 10 years, in SoC you can achieve > 10x more performance that current GPUs and > 100x more performance per watt.
    • Google: The TPU’s deterministic execution model is a better match to the 99th-percentile response-time requirement of our NN applications than are the time-varying optimizations of CPUs and GPUs (caches, out-of-order execution, multithreading, multiprocessing, prefetching, ...) that help average throughput more than guaranteed latency. 
    • visarga: TPU excited me too at first, but when I realized that it is not related to training new networks (research) and is useful only for large scale deployment, I toned down my enthusiasm a little. 
    • Julian Friedman: Kube is being designed by system administrators who like distributed systems, not for programmers who want to focus on their apps.
    • shadowmint: Given what I've seen, I'd argue that clojure has an inherent complexity that results in poor code quality outcomes during the software maintenance cycle.
    • weberc2: I like Go, but it's not dramatically faster than Java. Any contest between the two of them will probably just be a back and forth of optimizations. They share pretty much the same upper bound.
    • adrianratnapala: All this means is that we should stop thinking of this stuff as RAM. Only the L1 cache is really RAM. Everything else is just a kind of fast, volatile, solid state disk that just happens to share an address space with the RAM.
    • pbreit: Getting a million users is infinitely harder than scaling a system to handle a million users. Most systems could run comfortably on a Raspberry Pi.
    • @sustrik: If you want your protocol to be fully reliable in the face of either peer shutting down, the terminal handshake has to be asymmetric. As we've seen above, TCP protocol has symmetric termination algorithm and thus can't, by itself, guarantee full reliability.
    • @damonedwards: Unit tests are critical for good dev, but aren't really ops concern. Integration tests are critical for good ops. Ops wants more int tests.
    • mannigfaltig: the brain appears to spend about 4.7 bits per synapse (26 discernible states, given the noisy computation environment of the brain); so it seems to be plenty enough for general intelligence. This could, of course, merely be a biological limit and on silicon more fine-grained weights might be the optimum.
    • marwanad: The main power of GraphQL is for client developers and lies in the decoupling it provides between the client and server and the ability to fulfill the client needs in a single round trip. This is great for mobile devices with slower networks.
    • kyleschiller: As a pretty good rule of thumb, a system that fails 1/nth of the time and has n opportunities to fail has ~.63 probability of failure, where n is more than ~10.
    • jjirsa: databases aren't where you want to have hipster tech. You want boring things that work. For me, Cassandra is the boring thing that works. 
    • @etherealmind: "rule #1 of Enterprise IT: easier to spend 10 million on equipment than 100k for a person. A third person would increase capacity by 30%"
    • @SwiftOnSecurity: “Just pick a good VPN” is like telling thirsty people to “go to a store and drink clear liquid.” They drank bleach, but at least you helped.
    • falsedan: There's 2 secrets to scaling to millions of users: 1. You aren't going to have millions of users so any work you do to support it is stopping you from delivering features that will make your existing 10 clients happier. 2. Write code that can be replaced (i.e. design for change). 
    • X86BSD: Have you tested running it on a FreeBSD box with ZFS? It has lz4 compression by default and makes such a great storage solution for PG. You get compression, snapshots, replication (not quite realtime but close), self healing, etc etc in a battled hardened and easy to manage filesystem and storage manager. I've found you can't beat ZFS and PG for most applications. Edge cases exist of course everywhere.

  • Worried about too much infrastructure? Only 2% of DNA codes for proteins, the other 98% codes for RNA. Harry Noller Lecture. Maybe lots of infrastructure is not a bad thing. One of they key differences in programming and biology is how in biology form completely determines function. Just amazing to watch in action: mRNA Translation (Advanced). Programming is the complete opposite.

Don't miss all that the Internet has to say on Scalability, click below and become eventually consistent with all scalability knowledge (which means this post has many more items to read so please keep on reading)...

Categories: Architecture

Test Case Points Overview: An Example, Strengths and Weaknesses

Strengths and Weaknesses are up in the air!

Jeremy Berriault provided an example from this presentation at QAI Quest 2017 for us to count test case points.  Jeremy, QA Corner,  indicated baseline data was required to effectively run the three test cases in his example

The logon, transaction

, reports and expected output blocks represent verification points.  The arrows from one test case to another represent interfaces and steps are. . .steps.  The results of the count is as follows:

Test Case Number of
Steps
Interfaces Verification
Points
Baseline
Test Data
Complexity

test case 1

4 0

4

required medium test case 2

4

0

4

required medium test case 3 3 2 3 required

medium

Deriving the complexity leverages the following chart:

The test cases generate three medium test cases or 12 unadjusted use case points (3 x 4 = 12).  This represents the output of steps 1 – 4 of the basic test case point counting process.  Step 5 requires the evaluation of the 16 adjustment factors.  For example, a set of test cases that required multi-language support would require more effort for a set of test cases that requires only one language support.

Strengths:

  1. Rules Based:  Rules increase consistency and enable cross-team or project comparison.
  2. Identifies Complexity:  Test cases that are complex based on the rules are targets for simplification.  Complex test cases are harder to execute (whether automated or manually) and evaluate.
  3. Generates thought and conversation: The process of sizing forces practitioners to think about and talk what they are testing.
  4. Testing Specific:  Test case points relate only to the activities performed by testers.

Weakness:

  1. Not Available Early: Counting test case points requires a level of detail that is often on available early in Agile efforts.
  2. Testing Specific: Test case points can only be used to predict testing activities, in cross functional teams just sizing one part of the lifecycle is not as useful.
  3. May Not Be PredictIve:  There is no published industry data that proves test case points are predictive of the testing effort.  Many test case point estimation approaches are driven by weak correlations.
  4. Only Testable Requirements: Not all requirements are “testable” however non-testable or non-functional requirements still require effort to evaluate.
  5. The Relative Adjustment Factors:  Some of the adjustments require interpretation rather than explicit criteria based evaluation.  Subjectivity will likely creep into the process.  

Test case points are a useful sizing technique.  Test case point users needs to answer two questions: whether the value of the information generated outweighs the effort needed to generate the number, and when the information to count test case points is useful.  We will return to question two next week.


Categories: Process Management

Android Things Developer Preview 3

Android Developers Blog - Thu, 04/06/2017 - 18:00
Posted by Wayne Piekarski, Developer Advocate for IoT

Today, we are releasing the Developer Preview 3 (DP3) of Android Things, bringing new features and bug fixes to the platform. This preview is part of our commitment to provide regular updates to developers who are building Internet of Things (IoT) products with our platform. Android developers can quickly build smart devices using Android APIs and Google services, while staying secure with updates directly from Google. The System-on-Module (SoM) architecture supports prototyping with development boards, and then scaling them to large production runs while using the same Board Support Package (BSP) from Google.
Android Bluetooth APIs
DP3 now includes support for all Android Bluetooth APIs in android.bluetooth and android.bluetooth.le, across all Android Things supported hardware. You can now write code that interacts with both Bluetooth classic and low energy (LE) devices just like a regular Android phone. Existing samples such as Bluetooth LE advertisements and scanning and Bluetooth LE GATT can be used unmodified on Android Things. We have also provided two new samples, Bluetooth LE GATT server and Bluetooth audio sink.
USB Host support
Android version 3.1 and later supports USB Host, which allows a regular user space application to communicate with USB devices without root privileges or support needed from the Linux kernel. This functionality is now supported in Android Things, to enable interfacing with custom USB devices. Any existing code supporting USB Host will work on Android Things, and an extra sample USB Enumerator is available that demonstrates how to iterate over and print the interfaces and endpoints for each USB device.
Feedback
Once again, thank you to all the developers who submitted feedback for the previous developer previews. Please continue to send us your feedback by filing bug reports and feature requests, and ask any questions on stackoverflow. To download images for Developer Preview 3, visit the Android Things download page, and find the changes in the release notes. You can also join Google's IoT Developers Community on Google+, a great resource to keep up to date and discuss ideas, with over 4100 new members.
Categories: Programming

Expand your color palette with new tools for Material Design

Google Code Blog - Thu, 04/06/2017 - 18:00
Posted By: Rachel Been, Creative Lead, Material Design

The Material Design Guidelines are a living documentation of visual, interactive, and motion design guidance across platforms and devices.

Beyond guidance, Material Design is a also system that supports and strengthens communication and productivity with new tools and inspiration. With today's update, Material is introducing a new way to learn about color. The new color tool helps you create, share, and apply color palettes to a sample UI and through components in codepen. The tool also supports accessibility by evaluating the legibility of text for any color combination. Specific features include:





Create color schemes

Create color schemes that include darker and lighter variations of your primary and secondary colors.











Test accessibility

Check if text is accessible on different-colored backgrounds, as measured using the Web Content Accessibility Guidelines legibility standards.










Preview your UI in color

Preview the look of your color scheme across a range of Material Design Components, with editable HTML, CSS, or JavaScript in Codepen.




With these new tools to dabble with color schemes, you'll be able to give you users a richer experience, so we can't wait to see what you come up with. To get the latest news and engage with us directly, please follow us on our new Twitter account (@materialdesign) and visit https://material.io/.
Categories: Programming

Quote of the Day

Herding Cats - Glen Alleman - Thu, 04/06/2017 - 14:34

Cost estimation is part science, part art. There are many well-defined processes within the cost estimating discipline. There is also a subjective element to cost estimating that makes the discipline an art (NASA, 2004).

Categories: Project Management

Find Root Cause, Fix It, and Only Then Make Suggestions for Improvement

Herding Cats - Glen Alleman - Thu, 04/06/2017 - 02:30

We hear all the time suggestions for improvement. Or suggestions for outright abandonment of established Processes and sometimes even established Principles.

Before listening to any of these and most important making any changes, do a Root Cause Analysis (RCA) of the problem. Most descriptions of a problem are actually descriptions of a Symptom, not the Cause.

If you don't find the cause, fix it, and monitor the fix, you will not actually be fixing anything.

Start here and learn how to actually make improvements by finding the cause of the problem and stop fixing symptoms that leave the problem in place or may even make it worse.

Screen Shot 2017-04-05 at 1.48.18 PM

Categories: Project Management

AWS Lambda: Programmatically scheduling a CloudWatchEvent

Mark Needham - Thu, 04/06/2017 - 00:49

I recently wrote a blog post showing how to create a Python ‘Hello World’ AWS lambda function and manually invoke it, but what I really wanted to do was have it run automatically every hour.

To achieve that in AWS Lambda land we need to create a CloudWatch Event. The documentation describes them as follows:

Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams.

2017 04 05 23 06 36

This is actually really easy from the Amazon web console as you just need to click the ‘Triggers’ tab and then ‘Add trigger’. It’s not obvious that there are actually three steps are involved as they’re abstracted from you.

So what are the steps?

  1. Create rule
  2. Give permission for that rule to execute
  3. Map the rule to the function

I forgot to do step 2) initially and then you just end up with a rule that never triggers, which isn’t particularly useful.

The following code creates a ‘Hello World’ lambda function and runs it once an hour:

import boto3

lambda_client = boto3.client('lambda')
events_client = boto3.client('events')

fn_name = "HelloWorld"
fn_role = 'arn:aws:iam::[your-aws-id]:role/lambda_basic_execution'

fn_response = lambda_client.create_function(
    FunctionName=fn_name,
    Runtime='python2.7',
    Role=fn_role,
    Handler="{0}.lambda_handler".format(fn_name),
    Code={'ZipFile': open("{0}.zip".format(fn_name), 'rb').read(), },
)

fn_arn = fn_response['FunctionArn']
frequency = "rate(1 hour)"
name = "{0}-Trigger".format(fn_name)

rule_response = events_client.put_rule(
    Name=name,
    ScheduleExpression=frequency,
    State='ENABLED',
)

lambda_client.add_permission(
    FunctionName=fn_name,
    StatementId="{0}-Event".format(name),
    Action='lambda:InvokeFunction',
    Principal='events.amazonaws.com',
    SourceArn=rule_response['RuleArn'],
)

events_client.put_targets(
    Rule=name,
    Targets=[
        {
            'Id': "1",
            'Arn': fn_arn,
        },
    ]
)

We can now check if our trigger has been configured correctly:

$ aws events list-rules --query "Rules[?Name=='HelloWorld-Trigger']"
[
    {
        "State": "ENABLED", 
        "ScheduleExpression": "rate(1 hour)", 
        "Name": "HelloWorld-Trigger", 
        "Arn": "arn:aws:events:us-east-1:[your-aws-id]:rule/HelloWorld-Trigger"
    }
]

$ aws events list-targets-by-rule --rule HelloWorld-Trigger
{
    "Targets": [
        {
            "Id": "1", 
            "Arn": "arn:aws:lambda:us-east-1:[your-aws-id]:function:HelloWorld"
        }
    ]
}

$ aws lambda get-policy --function-name HelloWorld
{
    "Policy": "{\"Version\":\"2012-10-17\",\"Id\":\"default\",\"Statement\":[{\"Sid\":\"HelloWorld-Trigger-Event\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"events.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:[your-aws-id]:function:HelloWorld\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:events:us-east-1:[your-aws-id]:rule/HelloWorld-Trigger\"}}}]}"
}

All looks good so we’re done!

The post AWS Lambda: Programmatically scheduling a CloudWatchEvent appeared first on Mark Needham.

Categories: Programming

Santa Tracker, open-sourced and delivered

Google Code Blog - Wed, 04/05/2017 - 18:06
Posted by Sam Thorogood, Developer Programs Engineer

Santa Tracker is a holiday tradition here at Google. Every year, you can celebrate the season with games, holiday experiences and educational content throughout December: not to mention watching Santa deliver presents on 24th.

Today, we're continuing the season of giving by delivering the updated open-source versions of both the Web and Android versions that ran in December 2016. These are large, real-world apps that show off the latest and greatest from Google—using APIs and frameworks like Firebase and Polymer.



This year, Santa's elves added even more engaging, fun and educational experiences to Santa Tracker: all while making Santa and his reindeer leaner than ever before—across both Web and Android.

On the Web, we built a reliable, offline-capable PWA-ified version of Santa Tracker that saved bandwidth and worked in environments with poor connectivity. For Android, we worked hard to save every precious byte by closely examining our visual assets and other libraries.

To get started, you can check out the code on GitHub at google/santa-tracker-weband google/santa-tracker-android. Both Web and Android versions include detailed build instructions.
On the WebIf you'd like to read about how the elves build Santa Tracker as an offline Progressive Web App, check out our Case Study on Google Developers. To download the source, head over to GitHub. Here are some highlights of the release-
  • Santa is a Progressive Web App, sporting a responsive design for mobile, desktop and tablet, supporting Add to Home Screen and offline.
    • Rather than saving the entire site offline (about 100mb, including resources needed for different browsers), Santa's Service Worker only saves the scenes you've visited at least once—icing over houses that aren't available offline.

  • Santa Tracker used Polymer 1.7+, packing code into reusable components. Every housein Santa's Village is a custom element, only loaded when needed, minimizing the startup cost of Santa Tracker.
  • The Web Share API allowed users on mobile to quickly and natively showcase their creativity—it's a modern API for interfacing with a platform's native share intent, replacing the sea of share buttons normally presented to users.
  • Santa sported a new and improved Chromecast experience that scaled well across all Cast devices—from the original Chromecast device through to the high-end Chromecast Ultra and supported TVs.

  • Users were delighted by showing some great video content from around Santa's Village, especially during Santa's long travel legs.
  • The Android client also used this Chromecast experience, so Android users joined the fun watching Santa deliver presents on the 24th on their big screen TVs.
On AndroidIn 2016, Santa went on a diet, and reduced his APK download size by over 10mb—while adding four new games and a visual refresh. To learn more about our work, check out the in-depth analysis on Android Developers—or to try it yourself, head over to GitHub. Here are some highlights of this year's app-
  • Present Quest, a new AR game where players are encouraged to explore their real-world environment to collect presents and level up!

  • Santa is smaller and faster than ever before. The download size is down 10mb from the previous release, despite including multiple new games, Santa works better on memory-constrained devices, and various sources of jank have been found and removed.
  • The app is built using split APKs - one per architecture (armv5, armv7, and x86), reducing download size. Each APK supports phones, tablets, Android TVs and provide custom watch faces on Android Wear.

Ho Ho Ho!We hope you enjoy exploring Santa Tracker and its source code, and it inspires you to leverage the same approaches to make your own magical experiences!


Categories: Programming

Thinking About Cadence vs. Iterations


Many people use an iteration approach to agile. They decide on an iteration duration, commit to work for that iteration and by definition, they are done at the end of the timebox.

I like timeboxing many things. I like timeboxing work I don’t know how to start. I find short timeboxes help me focus on the first thing of value. Back when I used staged-delivery as a way to organize projects, we had a monthly milestone (timebox) to show progress and finish features. The teams and I found that a cadence of one month was good for us. The timebox focused us and allowed us to say no to other work.

A cadence is a pulse, a rhythm for a project. In my example above, you can see I used a timebox as a cadence and as a way to focus on work. You don’t have to use timeboxes to provide a cadence.

A new reader for the Pragmatic Manager asked me about scaling their agile transformation. They are starting and a number of people are impatient to be agile already. I suggested that instead of scaling agile, they think about what each team needs for creating their own successful agile approach.

One thing many teams (but not all) is a cadence for delivery, retrospectives and more planning. Not every team needs the focus of a timebox to do that. One team I know delivers several times during the week. They plan weekly, but not the same day each week. When they’ve finished three features, they plan for the next three. It takes them about 20-30 minutes to plan. It’s not a big deal. This team retrospects every Friday morning. (I would select a different day, but they didn’t ask me.)

Notice that they have two separate cadences for planning: once a week, but not the same day; and once a week for retrospectives on the same day each week.

Contrast that with another team new to agile. They have a backlog refinement session that often takes two hours (don’t get me started) and a two-hour pre-iteration planning session. Yes, they have trouble finishing the work they commit to. (I recommended they timebox their planning to one hour each and stop planning so much. Timeboxing that work to a shorter time would force them to plan less work. They might deliver more.)

A timebox can help a team create a project cadence, a rhythm. And, the timebox can help the team see their data, as long as they measure it.

A project cadence provides a team a rhythm. Depending on what the team needs, the team might decide to use timeboxes or not.

For me, one of the big problems in scaling is that each team often needs their own unique approach. Sometimes, that doesn’t fit with what managers new to agile think. I find that when I discuss cadence and iterations and explain the (subtle) difference to people, that can help.

Categories: Project Management

Focusing our Google Play games services efforts

Android Developers Blog - Wed, 04/05/2017 - 16:19
Posted By James Smith, Product Manager, Google Play

In order to help developers make great games and build their businesses, we offer Google Play Games Services (GPGS). GPGS provides powerful tools to build, analyze and retain your audience and optimize your game. After listening to developer feedback and examining usage, we have decided to remove some of the features so we can focus on making our offering more useful.

In December, we announced the end of support for the creation of new iOS accounts given the low usage of GPGS on iOS. Additionally, our latest Native SDK release (2.3) will no longer support integration with iOS and going forward we will not be supporting or updating the iOS SDK.

We've also examined the features that GPGS offers. While developers use engagement and reporting tools extensively, there is lower usage for Gifts, Requests, and Quests. We therefore plan to stop supporting Gifts, Requests, and Quests. In order to help developers that do use these features plan for their removal, we will leave them open for 12 months, deactivating them by 31st March 2018. We'll be continuing support for other features such as Sign-in, Achievements, Leaderboards and Multiplayer.

Play games services remains an important part of the tools we provide developers, and we're working hard on future GPGS updates. We continue to be strongly committed to providing high quality services for Games, including new tools such as official Firebase support for Unity and C++ developers, and integration with Firebase Analytics. These changes allow us to focus our efforts on the services developers value most to build high quality, engaging games.

How useful did you find this blogpost? 

Categories: Programming

The Art of Software Gardening

From the Editor of Methods & Tools - Wed, 04/05/2017 - 14:56
Software development has evolved a lot the last decade. Developers are not considered code monkeys any more. They’re expected to write clean and maintainable code that continuously evolve with business needs. Some times they need to work remotely in dynamic and multi-cutltural environments using a variety of technologies and programming languages. Nevertheless they still want […]

Using field masks with Google APIs for partial response

Google Code Blog - Wed, 04/05/2017 - 14:00
Originally posted on the G Suite Developers Blog by Wesley Chun, Developer Advocate, G Suite

When you write applications using Google APIs (not just G Suite ones, but most Google APIs including YouTube or Google Cloud Platform APIs), it's important to be mindful of the data that’s returned in the response payloads from API calls. If you're not, your apps are likely getting back much more data than they need which can affect the performance of your apps whether on mobile or a server backend.

That's why most Google APIs allow you to only filter the data you need from response payloads with field masks. To get you comfortable with field masks, we’ve put together a video to demonstrate their use with various Google APIs: With field masks, you can specify exactly what fields an API should return in its response payload by providing a fields or part parameter in your calls. And once the API knows what you want, it will likely spend less time assembling your response too. Here’s an example Python call to fetch your sender addresses using the Gmail API (if GMAIL is your service endpoint):
     addresses = GMAIL.users().settings().sendAs().list(
userId='me'
).execute().get('sendAs')

Whether you’re using a Client Library (as our Python call) or using HTTP directly with GET https://www.googleapis.com/gmail/v1/users/userId/settings/sendAs, this is the payload you get back from the API:
     {
"sendAs": [{
"sendAsEmail": string,
"displayName": string,
"replyToAddress": string,
"signature": string,
"isPrimary": boolean,
"isDefault": boolean,
"treatAsAlias": boolean,
"smtpMsa": {
"host": string,
"port": integer,
"username": string,
"password": string,
"securityMode": string
},
"verificationStatus": string
}, ...]
}

The sendAs array gives you everything you need to know about each of your sender addresses. Did you know you can change a user’s email signature using the Gmail API without all of the data from above? You only need one field, or at most two: sendAsEmail and perhaps the isPrimary flag. By specifying a field mask with just those names from the sendAs attribute, you can cut out all those unneeded fields. Check it out here in Python with the field mask bolded for emphasis (versus the sample code above that doesn’t filter):
     addresses = GMAIL.users().settings().sendAs().list(
userId='me', fields='sendAs(sendAsEmail,isPrimary)'
).execute().get('sendAs')
Field masks filter our unnecessary data from Google API call responses.

In part two of this video series (coming soon), we’ll show you a different use case for field masks...for update API calls. We’ll also provide some usage tips and demonstrate how field masks can be used in both read and update calls, how both types of calls are discrete, and how in some cases, you may use both as part of a single API call. Stay tuned!

To learn more about using field masks for partial response in API payloads, check out this section of the Client Library docs. For one of the most comprehensive write-ups on both (read and update) use cases, see the guide in the Google Slides API documentation.
Categories: Programming

5 tips for building communities on mobile

Android Developers Blog - Wed, 04/05/2017 - 06:55

Posted by Dave Geffon, Partnerships Manager, Google Play Games

The most successful games usually have the strongest communities. They are a powerful force in driving additional engagement and increasing awareness for your titles. At GDC 2017, we spoke with a few game developers about best practices for successfully building their own communities. Watch the panel session below to hear advice from Seriously, Social Point, and Super Evil MegaCorp.


1. Be authentic

Community is a mindset; be honest, transparent & patient with your communications. Loyal users are extremely valuable, thus the folks at Super Evil Megacorp say that you should act like you have to earn every player.

2. Start small
Build a plan and start today. Launch your social media channels, look into influencers, and create a strategy. Whether it's sharing one piece of fan art a week across your network, or running a closed beta to gather feedback from your most valued users, take action and learn what works best for you and your users.
3. Play match-maker
When finding influencers to support your game, ensure they're a genuine match. Make sure the influencer's audience is a good fit with your game and existing community.
4. Seek feedback 
Communities are passionate. Use feedback to understand what kind of game and features your users want. Be flexible and iterative so you can react and evolve your game with the needs and desires of your community. However, don't be afraid to stay true to what you stand for as sometimes you'll need to agree to disagree with some players.
5. Build for the long-term
The lifespan of games is continuing to grow. Plan your business strategy, update cycles and community efforts to roll out over time and expand with your growing experiences and user-base.
Watch more sessions from Google Developer Day at GDC17 on the Android Developers YT channel to learn tips for success. Also, visit the Android Developers website to stay up-to-date with features and best practices that will help you grow a successful business on Google Play.

How useful did you find this blogpost?         

Categories: Programming

Quote of the Day

Herding Cats - Glen Alleman - Wed, 04/05/2017 - 02:33

It is the mark of an instructed mind to rest satisfied with the degree of precision which the nature of the subject admits and not to seek exactness when only an approximation of the truth is possible - Aristotle, 384 – 322 BC

Related articles Architecture -Center ERP Systems in the Manufacturing Domain IT Risk Management Why Guessing is not Estimating and Estimating is not Guessing
Categories: Project Management

Size As A Factor In Test Estimation: Test Case Points Overview

 

How do you measure out of the ordinary packages?

Independent testing groups are often asked how long and how much effort is required to test a piece of work.  Several size estimation techniques are actively in use in many organizations.  Each of these techniques begins by deriving size either based on a set of rules or through relative sizing.  Size, once derived, is used to estimate effort.  Effort is then used to generate cost, staffing and duration estimates.  The first sizing technique is“Test Case Points”

Test Case Points are a unit of measurement generated from the the testable requirements based on a set of rules.  The process is straightforward:

  1. Identify the testable requirements in a piece of work.  Use Cases or technical requirements documents are used for identifying testable requirements.  
  1. Identify the complexity of each testable requirement.  Test case points evaluate four factors to determine complexity:
    1. The number of test steps. The number of execution steps needed to arrive at an expected (or unexpected) outcome after all preconditions have been satisfied.
    2. The number of interfaces to the other requirements. A simple count of the number of interfaces in the test case.
    3. The number of verification points. A simple count of the points in the test case that the results are evaluated for correctness.
    4. Need for baseline test data. An evaluation of whether data needs to be created to execute the test case.  


Once all of the simple, medium and complex test cases are identified, they are summed by category.

  1. Weight each category.  

  1. Sum the weighted categories together to yield the total test case points

The goal of test case points is to use size to generate an estimate.  Every version of test case points I have worked with uses a set of factors to adjust the size as part of the sizing process.

  1. Develop an estimation adjustment weighting based on a set of factors (for those familiar with IFPUG Function Points this adjustment is a similar process to the one for determining the value adjustment factor). The factors are:
  1. Count or Single Factor Adjustment Factors
    Factor 14 – Operating System Combinations (simple count)
    Factor 15 – Browser Combinations (simple count)
    Factor 16 – Productivity Improvement from Second Iteration Onwards (percentage)
  2. Factors that leverage a combination of fixed factor and complexity weighting
    Factor 1 – Domain Knowledge & Complexity
    Factor 2 – Technical Know How
    Factor 3 – Integration with other Hardware Devices such as Handheld Devices, Scanners, Printers
    Factor 4 – Multi-lingual Support
    Factor 5 – Software/Hardware Setup
    Factor 6 – Environment Setup
    Factor 7 – Build Management
    Factor 8 – Configuration Management
    Factor 9 – Preparation of Test Bed
    Factor 10 – Stable Requirements
    Factor 11 – Offshore/Onsite Coordination
    Factor 12 – Test Data Preparation
    Factor 13 – Network Latency

 

  1. Generate an estimate using the following formula

Weighted Test Case Points X Adjustment Factor X Historical Productivity Rate

In many cases, organizations generate estimates for types of work separately using the adjustment factors that that would affect the type of work.  An example of a type of work is test case generation.  Factor 5, software/hardware setup, would not be predictive of the effort for setting up test cases.

The process for deriving test case points is fairly straightforward (steps 1 – 4).  The process of turning the test case points into an estimate is more complicated. Next, we will develop a short example and examine the strengths and weaknesses of the process – some which are very apparent and other are not.  


Categories: Process Management

Getting Santa Tracker Into Shape

Android Developers Blog - Tue, 04/04/2017 - 00:35

Posted by Sam Stern, Developer Programs Engineer

Santa Tracker is a holiday tradition at Google.  In addition to bringing seasonal joy to millions of users around the world, it's a yearly testing ground for the latest APIs and techniques in app development.  That's why the full source of the app is released on Github every year.

In 2016, the Santa team challenged itself to introduce new content to the app while also making it smaller and more efficient than ever before.  In this post, you can read about the road to a more slimmer, faster Santa Tracker. APK Bloat Santa Tracker has grown over the years to include the visual and audio assets for over a dozen games and interactive scenes.  In 2015, the Santa Tracker APK size was 66.1 MB.

The Android Studio APK analyzer is a great tool to investigate what made the 2015 app so large.

09.png

First, while the APK size is 66.1 MB, we see that the download size is 59.5MB! The majority of that size is in the resources folder, but assets and native libraries contribute a sizable piece.

The 2016 app contains everything that was in the 2015 app while adding four completely new games.  At first, we assumed that making the app smaller while adding all of that would be impossible, but (spoiler alert!) here are the final results for 2016:

23.png

The download size for the app is now nearly 10MB smaller despite the addition of four new games and a visual refresh. The rest of this section will explore how we got there. Multiple APK Support on Google Play with APK Splits The 2015 app added the "Snowdown" game by Google's Fun Propulsion Labs team.  This game is written in C++, so it's included in Santa Tracker as a native library.  The team gave us compiled libraries for armv5, armv7, and x86 architectures.  Each version was about 3.5MB, which adds up to the 10.5MB you see in the lib entry for the 2015 APK.

Since each device is only using one of these architectures, two thirds of the native libraries could be removed to save space - the tradeoff here is that we’ll publish multiple APKs.  The Android gradle build system has native support for building an APK for each architecture (ABI) with only a few lines of configuration in the app's build.gradle file:

ext.abiList = ['armeabi', 'armeabi-v7a', 'x86'] android {        // ...    splits {        abi {            // Enable ABI splits            enable true            // Include the three architectures that we support for snowdown            reset()            include(*abiList)            // Also build a "universal" APK that will run on any device            universalApk true        }    } }

Once splits are enabled, each split needs to be given a unique version code so that they can co-exist in the Play Store:

// Generate unique versionCodes for each APK variant: ZXYYSSSSS //   Z is the Major version number //   X is the Minor version number //   YY is the Patch version number //   SSSS is information about the split (default to 0000) // Any new variations get added to the front import com.android.build.OutputFile; android.applicationVariants.all { variant ->    variant.outputs.each { output ->        // Shift abi over by 8 digits        def abiFilter = output.getFilter(OutputFile.ABI)        int abiVersionCode = (abiList.indexOf(abiFilter) + 1)        // Merge all version codes        output.versionCodeOverride = variant.mergedFlavor.versionCode + abiVersionCode    } }

In the most recent version of Santa Tracker, we published versions for armv5, armv7, and x86 respectively.  With this change in place, 10.5MB of native libraries was reduced to about 4MB per variant without losing any functionality. Optimize Images The majority of the Santa Tracker APK is image resources. Each game has hundreds of images, and each image comes in multiple sizes for different screen densities. Almost all of these images are PNGs, so in past years we ran PNGCrush on all of the files and figured our job was done.  We learned in 2016 that there have been advancements in lossless PNG compression, and Google's zopfli tool is currently the state of the art.

By running zopflipng on all PNG assets we losslessly reduced the size of most images by 10% and some by as much as 30%. This resulted in almost a 5MB size reduction across the app without sacrificing any quality. For instance this image of Santa was losslessly reduced from 10KB to only 7KB.  Don't bother trying to spot the differences, there are none!


Before (10.2KB) After (7.4KB) santa-before.png santa-before-zopfli.png

Unused Resources When working on Santa Tracker engineers are constantly refactoring the app, adding and removing pieces of logic and UI from previous years. While code review and linting help to find unused code, unused resources are much more likely to slip by unnoticed.  Plus there is no ProGuard for resources, so we can't be saved by our toolchain and unused images and other resources often sneak into the app.

Android Studio can help to find resources that are not being used and are therefore bloating the APK.  By clicking Analyze > Run Inspection by Name > Unused Resources Android Studio will identify resources that are not used by any known codepaths.  It's important to first eliminate all unused code, as resources that are "used" by dead code will not be detected as unused.

After a few cycles of analysis with Android Studio's helpful tools, we were able to find dozens of unused files and eliminate a few more MB of resources from the app. Memory Usage Santa Tracker is popular all around the world and has users on thousands of unique Android devices.  Many of these devices are a few years old and have 512MB RAM or less, so we have historically run into OutOfMemoryErrors in our games.

While the optimizations above made our PNGs smaller on disk, when loaded into a Bitmap their memory footprint is unchanged.  Since each game in Santa Tracker loads dozens of images, we quickly get into dangerous memory territory.

In 2015 six of our top ten crashes were memory related. Due to the optimizations below (and others) we moved memory crashes out of the top ten altogether. Image Loading Backoff When initializing a game in Santa Tracker we often load all of the Bitmaps needed for the first scene into memory so that the game can run smoothly.  The naive approach looks like this:

private LruCache<Integer, Drawable> mMemoryCache; private BitmapFactory.Options mOptions; public void init() {  // Initialize the cache  mMemoryCache = new LruCache<Integer, Drawable>(240);  // Start with no Bitmap sampling  mOptions = new BitmapFactory.Options();  mOptions.inSampleSize = 1; } public void loadBitmap(@DrawableRes int id) {    // Load bitmap    Bitmap bmp = BitmapFactory.decodeResource(getResources(), id, mOptions);    BitmapDrawable bitmapDrawable = new BitmapDrawable(getResources(), bmp);        // Add to cache    mMemoryCache.put(id, bitmapDrawable); }

However the decodeResource function will throw an OutOfMemoryError if we don't have enough RAM to load the Bitmap into memory.  To combat this, we catch these errors and then try to reload all of the images with a higher sampling ratio (scaling by a factor of 2 each time):

private static final int MAX_DOWNSAMPLING_ATTEMPTS = 3; private int mDownsamplingAttempts = 0; private Bitmap tryLoadBitmap(@DrawableRes int id) throws Exception {    try {        return BitmapFactory.decodeResource(getResources(), id, mOptions);    } catch (OutOfMemoryError oom) {        if (mDownSamplingAttempts < MAX_DOWNSAMPLING_ATTEMPTS) {            // Increase our sampling by a factor of 2            mOptions.inSampleSize *= 2;            mDownSamplingAttempts++;        }    }    throw new Exception("Failed to load resource ID: " + resourceId); }

With this technique low-memory devices will now see more pixelated graphics, but by making this tradeoff we almost completely eliminated memory errors from Bitmap loading. Transparent Pixels As mentioned above, an image's size on disk is not a good indicator of how much memory it will use. One glaring example is images with large transparent regions.  PNG can compress these regions to near-zero disk size but each transparent pixel still demands the same RAM.

For example in the "Dasher Dancer" game, animations were represented by a series of 1280x720 PNG frames.  Many of these frames were dominated by transparency as the animated object left the screen.  We wrote a script to trim all of the transparent space away and record an "offset" for displaying each frame so that it would still appear to be 1280x720 overall. In one test this reduced runtime RAM usage of the game by 60MB! And now that we were not wasting memory on transparent pixels, we needed less downscaling and could use higher-resolution images on low-memory devices. Additional Explorations In addition to the major optimizations described above, we explored a few other avenues for making the app smaller and faster with varying degrees of success. Splash Screens The 2015 app moved to a Material Design aesthetic where games were launched from a central list of 'cards'. We noticed that half of the games would cause the card 'ripple' effect to be janky on launch, but we couldn't find the root cause and were unable to fix the issue.

When work on the 2016 version of the app we were determined to fix the game jank launch. After hours of investigation, we realized it was only the games fixed to the landscape orientation that caused jank when launched.  The dropped frames were due to the forced orientation change. To create a smooth user experience, we introduced splash screens in between the launcher Activity and game Activities.  The splash screen would detect the current device orientation and the orientation needed to play the game being loaded and rotate itself at runtime to match.  This immediately removed any noticeable jank from game launches and made the whole app feel smoother. SVG When we originally took on the task of reducing the size of our resources, using SVG images seemed like an obvious optimization.  Vector images are dramatically smaller and only need to be included once to support multiple densities.  Due to the 'flat' aesthetic in Santa Tracker, we were even able to convert many of our PNGs to tiny SVGs without much quality loss. However loading these SVGs was completely impractical on slower devices, where they would be tens or hundreds of times slower than a PNG depending on the path complexity.

In the end we decided to follow the recommendation limiting vector image sizes at 200x200 dp and only used SVG for small icons in the app rather than large graphics or game assets. Conclusions When we started building Santa Tracker 2016 we were faced with a daunting problem: how can we make the app smaller and faster while adding exciting new content? The optimizations above were discovered by constantly challenging each other to do more with less and considering resource constraints with every change we made. In the end we were able to incrementally make the Santa Tracker app as healthy as it has ever been ... our next job will be helping Mr. Claus work off all that extra cookie weight.
Categories: Programming

Noto Serif CJK is here!

Google Code Blog - Tue, 04/04/2017 - 00:00

Posted by Xiangye Xiao and Jungshik Shin, Internationalization Engineering team

Today, in collaboration with Adobe, we are responding to the call for Serif! We are pleased to announce Noto Serif CJK, the long-awaited companion to Noto Sans CJK released in 2014. Like Noto Sans CJK, Noto Serif CJK supports Simplified Chinese, Traditional Chinese, Japanese, and Korean, all in one font.

A serif-style CJK font goes by many names: Song (宋体) in Mainland China, Ming (明體) in Hong Kong, Macao and Taiwan, Minchō (明朝) in Japan, and Myeongjo (명조) or Batang (바탕) in Korea. The names and writing styles originated during the Song and Ming dynasties in China, when China's wood-block printing technique became popular. Characters were carved along the grain of the wood block. Horizontal strokes were easy to carve and vertical strokes were difficult; this resulted in thinner horizontal strokes and wider vertical ones. In addition, subtle triangular ornaments were added to the end of horizontal strokes to simulate Chinese Kai (楷体) calligraphy. This style continues today and has become a popular typeface style.

Serif fonts, which are considered more traditional with calligraphic aesthetics, are often used for long paragraphs of text such as body text of web pages or ebooks. Sans-serif fonts are often used for user interfaces of websites/apps and headings because of their simplicity and modern feeling.

Design of '永' ('eternity') in Noto Serif and Sans CJK. This ideograph is famous for having the most important elements of calligraphic strokes. It is often used to evaluate calligraphy or typeface design.
The Noto Serif CJK package offers the same features as Noto Sans CJK:

  • It has comprehensive character coverage for the four languages. This includes the full coverage of CJK Ideographs with variation support for four regions, Kangxi radicals, Japanese Kana, Korean Hangul and other CJK symbols and letters in the Unicode Basic Multilingual Plane of Unicode. It also provides a limited coverage of CJK Ideographs in Plane 2 of Unicode, as necessary to support standards from China and Japan.


Simplified ChineseSupports GB 18030 and China’s latest standard Table of General Chinese Characters (通用规范汉字表) published in 2013. Traditional ChineseSupports BIG5, and Traditional Chinese glyphs are compliant to glyph standard of Taiwan Ministry of Education (教育部國字標準字體).JapaneseSupports all of the kanji in  JIS X 0208, JIS X 0213, and JIS X 0212 to include all kanji in Adobe-Japan1-6.KoreanThe best font for typesetting classic Korean documents in Hangul and Hanja such as Humninjeongeum manuscript, a UNESCO World Heritage.Supports over 1.5 million archaic Hangul syllables and 11,172 modern syllables as well as all CJK ideographs in KS X 1001 and KS X 1002
Noto Serif CJK’s support of character and glyph set standards for the four languages
  • It respects diversity of regional writing conventions for the same character. The example below shows the four glyphs of '述' (describe) in four languages that have subtle differences.
From left to right are glyphs of '述' in S. Chinese, T. Chinese, Japanese and Korean. This character means "describe".
  • It is offered in seven weights: ExtraLight, Light, Regular, Medium, SemiBold, Bold, and Black. Noto Serif CJK supports 43,027 encoded characters and includes 65,535 glyphs (the maximum number of glyphs that can be included in a single font). The seven weights, when put together, have almost a half-million glyphs. The weights are compatible with Google's Material Design standard fonts, Roboto, Noto Sans and Noto Serif(Latin-Greek-Cyrillic fonts in the Noto family).
Seven weights of Noto Serif CJK
    • It supports vertical text layout and is compliant with the Unicode vertical text layout standard. The shape, orientation, and position of particular characters (e.g., brackets and kana letters) are changed when the writing direction of the text is vertical.



    The sheer size of this project also required regional expertise! Glyph design would not have been possible without leading East Asian type foundries Changzhou SinoType Technology, Iwata Corporation, and Sandoll Communications.

    Noto Serif CJK is open source under the SIL Open Font License, Version 1.1. We invite individual users to install and use these fonts in their favorite authoring apps; developers to bundle these fonts with your apps, and OEMs to embed them into their devices. The fonts are free for everyone to use!

    Noto Serif CJK font download:https://www.google.com/get/noto
    Noto Serif CJK on GitHub:https://github.com/googlei18n/noto-cjk
    Adobe's landing page for this release: http://adobe.ly/SourceHanSerif
    Source Han Serif on GitHub: https://github.com/adobe-fonts/source-han-serif/tree/release/
    Categories: Programming

    An investigation of Chrysaor Malware on Android

    Android Developers Blog - Mon, 04/03/2017 - 21:00
    Posted by Rich Cannings, Jason Woloz, Neel Mehta, Ken Bodzak, Wentao Chang, Megan Ruthven

    Google is constantly working to improve our systems that protect users from Potentially Harmful Applications (PHAs). Usually, PHA authors attempt to install their harmful apps on as many devices as possible. However, a few PHA authors spend substantial effort, time, and money to create and install their harmful app on one or a very small number of devices. This is known as a targeted attack.

    In this blog post, we describe Chrysaor, a newly discovered family of spyware that was used in a targeted attack on a small number of Android devices, and how investigations like this help Google protect Android users from a variety of threats.

    What is Chrysaor?
    Chrysaor is spyware believed to be created by NSO Group Technologies, specializing in the creation and sale of software and infrastructure for targeted attacks. Chrysaor is believed to be related to the Pegasus spyware that was first identified on iOS and analyzed by Citizen Lab and Lookout.

    Late last year, after receiving a list of suspicious package names from Lookout, we discovered that a few dozen Android devices may have installed an application related to Pegasus, which we named Chrysaor. Although the applications were never available in Google Play, we immediately identified the scope of the problem by using Verify Apps. We gathered information from affected devices, and concurrently, attempted to acquire Chrysaor apps to better understand its impact on users. We've contacted the potentially affected users, disabled the applications on affected devices, and implemented changes in Verify Apps to protect all users.

    What is the scope of Chrysaor?
    Chrysaor was never available in Google Play and had a very low volume of installs outside of Google Play. Among the over 1.4 billion devices protected by Verify Apps, we observed fewer than 3 dozen installs of Chrysaor on victim devices. These devices were located in the following countries:




    How we protect you
    To protect Android devices and users, Google Play provides a complete set of security services that update outside of platform releases. Users don't have to install any additional security services to keep their devices safe. In 2016, these services protected over 1.4 billion devices, making Google one of the largest providers of on-device security services in the world:
    Additionally, we are providing detailed technical information to help the security industry in our collective work against PHAs.

    What do I need to do?
    It is extremely unlikely you or someone you know was affected by Chrysaor malware. Through our investigation, we identified less than 3 dozen devices affected by Chrysaor, we have disabled Chrysaor on those devices, and we have notified users of all known affected devices. Additionally, the improvements we made to our protections have been enabled for all users of our security services.
    To ensure you are fully protected against PHAs and other threats, we recommend these 5 basic steps:
    • Install apps only from reputable sources: Install apps from a reputable source, such as Google Play. No Chrysaor apps were on Google Play.
    • Enable a secure lock screen: Pick a PIN, pattern, or password that is easy for you to remember and hard for others to guess.
    • Update your device: Keep your device up-to-date with the latest security patches.
    • Verify Apps: Ensure Verify Apps is enabled.
    • Locate your device: Practice finding your device with Android Device Manager because you are far more likely to lose your device than install a PHA.

    How does Chrysaor work?
    To install Chrysaor, we believe an attacker coaxed specifically targeted individuals to download the malicious software onto their device. Once Chrysaor is installed, a remote operator is able to surveil the victim's activities on the device and within the vicinity, leveraging microphone, camera, data collection, and logging and tracking application activities on communication apps such as phone and SMS.

    One representative sample Chrysaor app that we analyzed was tailored to devices running Jellybean (4.3) or earlier. The following is a review of scope and impact of the Chrysaor app named com.network.android tailored for a Samsung device target, with SHA256 digest:

    ade8bef0ac29fa363fc9afd958af0074478aef650adeb0318517b48bd996d5d5
    
    Upon installation, the app uses known framaroot exploits to escalate privileges and break Android's application sandbox. If the targeted device is not vulnerable to these exploits, then the app attempts to use a superuser binary pre-positioned at /system/csk to elevate privileges.

    After escalating privileges, the app immediately protects itself and starts to collect data, by:
    • Installing itself on the /system partition to persist across factory resets
    • Removing Samsung's system update app (com.sec.android.fotaclient) and disabling auto-updates to maintain persistence (sets Settings.System.SOFTWARE_UPDATE_AUTO_UPDATE to 0)
    • Deleting WAP push messages and changing WAP message settings, possibly for anti-forensic purpose.
    • Starting content observers and the main task loop to receive remote commands and exfiltrate data

    The app uses six techniques to collect user data:

    • Repeated commands: use alarms to periodically repeat actions on the device to expose data, including gathering location data.
    • Data collectors: dump all existing content on the device into a queue. Data collectors are used in conjunction with repeated commands to collect user data including, SMS settings, SMS messages, Call logs, Browser History, Calendar, Contacts, Emails, and messages from selected messaging apps, including WhatsApp, Twitter, Facebook, Kakoa, Viber, and Skype by making /data/data directories of the apps world readable.
    • Content observers: use Android's ContentObserver framework to gather changes in SMS, Calendar, Contacts, Cell info, Email, WhatsApp, Facebook, Twitter, Kakao, Viber, and Skype.
    • Screenshots: captures an image of the current screen via the raw frame buffer.
    • Keylogging: record input events by hooking IPCThreadState::Transact from /system/lib/libbinder.so, and intercepting android::parcel with the interface com.android.internal.view.IInputContext.
    • RoomTap: silently answers a telephone call and stays connected in the background, allowing the caller to hear conversations within the range of the phone's microphone. If the user unlocks their device, they will see a black screen while the app drops the call, resets call settings and prepares for the user to interact with the device normally.

    Finally, the app can remove itself through three ways:

    • Via a command from the server
    • Autoremove if the device has not been able to check in to the server after 60 days
    • Via an antidote file. If /sdcard/MemosForNotes was present on the device, the Chrysaor app removes itself from the device.

    Samples uploaded to VirusTotal
    To encourage further research in the security community, we’ve uploaded these sample Chrysaor apps to Virus Total.
    Package Name SHA256 digest SHA1 certificate com.network.android ade8bef0ac29fa363fc9afd958af0074478aef650adeb0318517b48bd996d5d5 44f6d1caa257799e57f0ecaf4e2e216178f4cb3d com.network.android 3474625e63d0893fc8f83034e835472d95195254e1e4bdf99153b7c74eb44d86 516f8f516cc0fd8db53785a48c0a86554f75c3ba
    Additional digests with links to Chrysaor
    As a result of our investigation we have identified these additional Chrysaor-related apps.

    Package Name SHA256 digest SHA1 certificate com.network.android 98ca5f94638768e7b58889bb5df4584bf5b6af56b188da48c10a02648791b30c 516f8f516cc0fd8db53785a48c0a86554f75c3ba com.network.android 5353212b70aa096d918e4eb6b49eb5ad8f59d9bec02d089e88802c01e707c3a1 44f6d1caa257799e57f0ecaf4e2e216178f4cb3d com.binary.sms.receiver 9fae5d148b89001555132c896879652fe1ca633d35271db34622248e048c78ae 7771af1ad3a3d9c0b4d9b55260bb47c2692722cf com.android.copy e384694d3d17cd88ec3a66c740c6398e07b8ee401320ca61e26bdf96c20485b4 7771af1ad3a3d9c0b4d9b55260bb47c2692722cf com.android.copy 12e085ab85db887438655feebd249127d813e31df766f8c7b009f9519916e389 7771af1ad3a3d9c0b4d9b55260bb47c2692722cf com.android.copy 6348104f8ef22eba5ac8ee737b192887629de987badbb1642e347d0dd01420f8 31a8633c2cd67ae965524d0b2192e9f14d04d016
    Lookout has completed their own independent analysis of the samples we acquired, their report can be viewed here.
    Categories: Programming

    An investigation of Chrysaor Malware on Android

    Android Developers Blog - Mon, 04/03/2017 - 21:00
    Posted by Rich Cannings, Jason Woloz, Neel Mehta, Ken Bodzak, Wentao Chang, Megan Ruthven

    Google is constantly working to improve our systems that protect users from Potentially Harmful Applications (PHAs). Usually, PHA authors attempt to install their harmful apps on as many devices as possible. However, a few PHA authors spend substantial effort, time, and money to create and install their harmful app on one or a very small number of devices. This is known as a targeted attack.

    In this blog post, we describe Chrysaor, a newly discovered family of spyware that was used in a targeted attack on a small number of Android devices, and how investigations like this help Google protect Android users from a variety of threats.

    What is Chrysaor?
    Chrysaor is spyware believed to be created by NSO Group Technologies, specializing in the creation and sale of software and infrastructure for targeted attacks. Chrysaor is believed to be related to the Pegasus spyware that was first identified on iOS and analyzed by Citizen Lab and Lookout.

    Late last year, after receiving a list of suspicious package names from Lookout, we discovered that a few dozen Android devices may have installed an application related to Pegasus, which we named Chrysaor. Although the applications were never available in Google Play, we immediately identified the scope of the problem by using Verify Apps. We gathered information from affected devices, and concurrently, attempted to acquire Chrysaor apps to better understand its impact on users. We've contacted the potentially affected users, disabled the applications on affected devices, and implemented changes in Verify Apps to protect all users.

    What is the scope of Chrysaor?
    Chrysaor was never available in Google Play and had a very low volume of installs outside of Google Play. Among the over 1.4 billion devices protected by Verify Apps, we observed fewer than 3 dozen installs of Chrysaor on victim devices. These devices were located in the following countries:




    How we protect you
    To protect Android devices and users, Google Play provides a complete set of security services that update outside of platform releases. Users don't have to install any additional security services to keep their devices safe. In 2016, these services protected over 1.4 billion devices, making Google one of the largest providers of on-device security services in the world:
    Additionally, we are providing detailed technical information to help the security industry in our collective work against PHAs.

    What do I need to do?
    It is extremely unlikely you or someone you know was affected by Chrysaor malware. Through our investigation, we identified less than 3 dozen devices affected by Chrysaor, we have disabled Chrysaor on those devices, and we have notified users of all known affected devices. Additionally, the improvements we made to our protections have been enabled for all users of our security services.
    To ensure you are fully protected against PHAs and other threats, we recommend these 5 basic steps:
    • Install apps only from reputable sources: Install apps from a reputable source, such as Google Play. No Chrysaor apps were on Google Play.
    • Enable a secure lock screen: Pick a PIN, pattern, or password that is easy for you to remember and hard for others to guess.
    • Update your device: Keep your device up-to-date with the latest security patches.
    • Verify Apps: Ensure Verify Apps is enabled.
    • Locate your device: Practice finding your device with Android Device Manager because you are far more likely to lose your device than install a PHA.

    How does Chrysaor work?
    To install Chrysaor, we believe an attacker coaxed specifically targeted individuals to download the malicious software onto their device. Once Chrysaor is installed, a remote operator is able to surveil the victim's activities on the device and within the vicinity, leveraging microphone, camera, data collection, and logging and tracking application activities on communication apps such as phone and SMS.

    One representative sample Chrysaor app that we analyzed was tailored to devices running Jellybean (4.3) or earlier. The following is a review of scope and impact of the Chrysaor app named com.network.android tailored for a Samsung device target, with SHA256 digest:

    ade8bef0ac29fa363fc9afd958af0074478aef650adeb0318517b48bd996d5d5
    
    Upon installation, the app uses known framaroot exploits to escalate privileges and break Android's application sandbox. If the targeted device is not vulnerable to these exploits, then the app attempts to use a superuser binary pre-positioned at /system/csk to elevate privileges.

    After escalating privileges, the app immediately protects itself and starts to collect data, by:
    • Installing itself on the /system partition to persist across factory resets
    • Removing Samsung's system update app (com.sec.android.fotaclient) and disabling auto-updates to maintain persistence (sets Settings.System.SOFTWARE_UPDATE_AUTO_UPDATE to 0)
    • Deleting WAP push messages and changing WAP message settings, possibly for anti-forensic purpose.
    • Starting content observers and the main task loop to receive remote commands and exfiltrate data

    The app uses six techniques to collect user data:

    • Repeated commands: use alarms to periodically repeat actions on the device to expose data, including gathering location data.
    • Data collectors: dump all existing content on the device into a queue. Data collectors are used in conjunction with repeated commands to collect user data including, SMS settings, SMS messages, Call logs, Browser History, Calendar, Contacts, Emails, and messages from selected messaging apps, including WhatsApp, Twitter, Facebook, Kakoa, Viber, and Skype by making /data/data directories of the apps world readable.
    • Content observers: use Android's ContentObserver framework to gather changes in SMS, Calendar, Contacts, Cell info, Email, WhatsApp, Facebook, Twitter, Kakao, Viber, and Skype.
    • Screenshots: captures an image of the current screen via the raw frame buffer.
    • Keylogging: record input events by hooking IPCThreadState::Transact from /system/lib/libbinder.so, and intercepting android::parcel with the interface com.android.internal.view.IInputContext.
    • RoomTap: silently answers a telephone call and stays connected in the background, allowing the caller to hear conversations within the range of the phone's microphone. If the user unlocks their device, they will see a black screen while the app drops the call, resets call settings and prepares for the user to interact with the device normally.

    Finally, the app can remove itself through three ways:

    • Via a command from the server
    • Autoremove if the device has not been able to check in to the server after 60 days
    • Via an antidote file. If /sdcard/MemosForNotes was present on the device, the Chrysaor app removes itself from the device.

    Samples uploaded to VirusTotal
    To encourage further research in the security community, we’ve uploaded these sample Chrysaor apps to Virus Total.
    Package Name SHA256 digest SHA1 certificate com.network.android ade8bef0ac29fa363fc9afd958af0074478aef650adeb0318517b48bd996d5d5 44f6d1caa257799e57f0ecaf4e2e216178f4cb3d com.network.android 3474625e63d0893fc8f83034e835472d95195254e1e4bdf99153b7c74eb44d86 516f8f516cc0fd8db53785a48c0a86554f75c3ba
    Additional digests with links to Chrysaor
    As a result of our investigation we have identified these additional Chrysaor-related apps.

    Package Name SHA256 digest SHA1 certificate com.network.android 98ca5f94638768e7b58889bb5df4584bf5b6af56b188da48c10a02648791b30c 516f8f516cc0fd8db53785a48c0a86554f75c3ba com.network.android 5353212b70aa096d918e4eb6b49eb5ad8f59d9bec02d089e88802c01e707c3a1 44f6d1caa257799e57f0ecaf4e2e216178f4cb3d com.binary.sms.receiver 9fae5d148b89001555132c896879652fe1ca633d35271db34622248e048c78ae 7771af1ad3a3d9c0b4d9b55260bb47c2692722cf com.android.copy e384694d3d17cd88ec3a66c740c6398e07b8ee401320ca61e26bdf96c20485b4 7771af1ad3a3d9c0b4d9b55260bb47c2692722cf com.android.copy 12e085ab85db887438655feebd249127d813e31df766f8c7b009f9519916e389 7771af1ad3a3d9c0b4d9b55260bb47c2692722cf com.android.copy 6348104f8ef22eba5ac8ee737b192887629de987badbb1642e347d0dd01420f8 31a8633c2cd67ae965524d0b2192e9f14d04d016
    Lookout has completed their own independent analysis of the samples we acquired, their report can be viewed here.
    Categories: Programming