Warning: Table './devblogsdb/cache_page' is marked as crashed and last (automatic?) repair failed query: SELECT data, created, headers, expire, serialized FROM cache_page WHERE cid = 'http://www.softdevblogs.com/?q=aggregator/categories/1' in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc on line 135

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 729

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 730

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 731

Warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/bootstrap.inc on line 732
Software Development Blogs: Programming, Software Testing, Agile, Project Management
Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Programming
warning: Cannot modify header information - headers already sent by (output started at /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/database.mysql.inc:135) in /home/content/O/c/n/Ocnarfparking9/html/softdevblogs/includes/common.inc on line 153.

scikit-learn: Trying to find clusters of Game of Thrones episodes

Mark Needham - Thu, 08/25/2016 - 23:07

In my last post I showed how to find similar Game of Thrones episodes based on the characters that appear in different episodes. This allowed us to find similar episodes on an episode by episode basis, but I was curious whether there were groups of similar episodes that we could identify.

scikit-learn provides several clustering algorithms that can run over our episode vectors and hopefully find clusters of similar episodes. A clustering algorithm groups similar documents together, where similarity is based on calculating a ‘distance’ between documents. Documents separated by a small distance would be in the same cluster, whereas if there’s a large distance between episodes then they’d probably be in different clusters.

The simplest variant is K-means clustering:

The KMeans algorithm clusters data by trying to separate samples in n groups of equal variance, minimizing a criterion known as the inertia or within-cluster sum-of-squares. This algorithm requires the number of clusters to be specified.

The output from the algorithm is a list of labels which correspond to the cluster assigned to each episode.

Let’s give it a try on the Game of Thrones episodes. We’ll start from the 2 dimensional array of episodes/character appearances that we created in the previous post.

>>> all.shape
(60, 638)
 
>>> all
array([[0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       ..., 
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0],
       [0, 0, 0, ..., 0, 0, 0]])

We have a 60 (episodes) x 638 (characters) array which we can now plug into the K-means clustering algorithm:

>>> from sklearn.cluster import KMeans
 
>>> n_clusters = 3
>>> km = KMeans(n_clusters=n_clusters, init='k-means++', max_iter=100, n_init=1)
>>> cluster_labels = km.fit_predict(all)
 
>>> cluster_labels
array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 2, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2, 2, 2, 2, 0, 2, 2, 2, 2, 2, 2,
       2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2], dtype=int32)

cluster_labels is an array containing a label for each episode in the all array. The spread of these labels is as follows:

>>> import numpy as np
>>> np.bincount(cluster_labels)
array([19, 12, 29])

i.e. 19 episodes in cluster 0, 12 in cluster 1, and 29 in cluster 2.

How do we know if the clustering is any good?

Ideally we’d have some labelled training data which we could compare our labels against, but since we don’t we can measure the effectiveness of our clustering by calculating inter-centroidal separation and intra-cluster variance.

i.e. how close are the episodes to other episodes in the same cluster vs how close are they to episodes in the closest different cluster.

scikit-learn gives us a function that we can use to calculate this score – the silhouette coefficient.

The output of this function is a score between -1 and 1.

  • A score of 1 means that our clustering has worked well and a document is far away from the boundary of another cluster.
  • A score of -1 means that our document should have been placed in another cluster.
  • A score of 0 means that the document is very close to the decision boundary between two clusters.

I tried calculating this coefficient for some different values of K. This is what I found:

from sklearn import metrics
 
for n_clusters in range(2, 10):
    km = KMeans(n_clusters=n_clusters, init='k-means++', max_iter=100, n_init=1)
    cluster_labels = km.fit_predict(all)
 
    silhouette_avg = metrics.silhouette_score(all, cluster_labels, sample_size=1000)
    sample_silhouette_values = metrics.silhouette_samples(all, cluster_labels)
 
    print n_clusters, silhouette_avg
 
2 0.0798610142955
3 0.0648416081725
4 0.0390877994786
5 0.020165277756
6 0.030557856406
7 0.0389677156458
8 0.0590721834989
9 0.0466170527996

The best score we manage here is 0.07 when we set the number of clusters to 2. Even our highest score is much lower than the lowest score on the documentation page!

I tried it out with some higher values of K but only saw a score over 0.5 once I put the number of clusters to 40 which would mean 1 or 2 episodes per cluster at most.

At the moment our episode arrays contain 638 elements so they’re too long to visualise on a 2D silhouette plot. We’d need to apply a dimensionality reduction algorithm before doing that.

In summary it looks like character co-occurrence isn’t a good way to cluster episodes. I’m curious what would happen if we flip the array on its head and try and cluster the characters instead, but that’s for another day.

If anyone spots anything that I’ve missed when reading the output of the algorithm let me know in the comments. I’m just learning by experimentation at the moment.

Categories: Programming

Can Software Make You Less Racist?

Coding Horror - Jeff Atwood - Thu, 08/25/2016 - 08:52

I don't think we computer geeks appreciate how profoundly the rise of the smartphone, and Facebook, has changed the Internet audience. It's something that really only happened in the last five years, as smartphones and data plans dropped radically in price and became accessible – and addictive – to huge segments of the population.

People may have regularly used computers in 2007, sure, but that is a very different thing than having your computer in your pocket, 24/7, with you every step of every day, fully integrated into your life. As Jerry Seinfeld noted in 2014:

But I know you got your phone. Everybody here's got their phone. There's not one person here who doesn't have it. You better have it … you gotta have it. Because there is no safety, there is no comfort, there is no security for you in this life any more … unless when you're walking down the street you can feel a hard rectangle in your pants.

It's an addiction that is new to millions – but eerily familiar to us.

From "only nerds will use the Internet" to "everyone stares at their smartphones all day long!" in 20 years. Not bad, team :-).

— Marc Andreessen (@pmarca) January 16, 2015

The good news is that, at this moment, every human being is far more connected to their fellow humans than any human has ever been in the entirety of recorded history.

Spoiler alert: that's also the bad news.

Nextdoor is a Facebook-alike focused on specific neighborhoods. The idea is that you and everyone else on your block would join, and you can privately discuss local events, block parties, and generally hang out like neighbors do. It's a good idea, and my wife started using it a fair amount in the last few years. We feel more connected to our neighbors through the service. But one unfortunate thing you'll find out when using Nextdoor is that your neighbors are probably a little bit racist.

I don't use Nextdoor myself, but I remember Betsy specifically complaining about the casual racism she saw there, and I've also seen it mentioned several times on Twitter by people I follow. They're not the only ones. It became so epidemic that Nextdoor got a reputation for being a racial profiling hub. Which is obviously not good.

Social networking historically trends young, with the early adopters. Facebook launched as a site for college students. But as those networks grow, they inevitably age. They begin to include older people. And those older people will, statistically speaking, be more racist. I apologize if this sounds ageist, but let me ask you something: do you consider your parents a little racist? I will personally admit that one of my parents is definitely someone I would label a little bit racist. It's … not awesome.

The older the person, the more likely they are to have these "old fashioned" notions that the mere presence of differently-colored people on your block is inherently suspicious, and marriage should probably be defined as between a man and a woman.

In one meta-analysis by Jeffrey Lax and Justin Phillips of Columbia University, a majority of 18–29 year old Americans in 38 states support same sex marriage while in only 6 states do less than 45% of 18–29 year olds support same-sex marriage. At the same time not a single state shows support for same-sex marriage greater than 35% amongst those 64 and older

The idea that regressive social opinions correlate with age isn't an opinion; it's a statistical fact.

Support for same-sex marriage in the U.S.

18 - 29 years old    65%
30 - 49 years old    54%
50 - 64 years old    45%
65+ years old        39%

Are there progressive septuagenarians? Sure there are. But not many.

To me, failure to support same-sex marriage is as inconceivable as failing to support interracial marriage. Which was not that long ago, to the tune of the late 60s and early 70s. If you want some truly hair-raising reading, try Loving v. Virginia on for size. Because Virginia is for lovers. Just not those kind of lovers, 49 years ago. In the interests of full disclosure, I am 45 years old, and I graduated from the University of Virginia.

With Nextdoor, you're more connected with your neighbors than ever before. But through that connection you may also find out some regressive things about your neighbors that you'd never have discovered in years of the traditional daily routine of polite waves, hellos from the driveway, and casual sidewalk conversations.

To their immense credit, rather than accepting this status quo, Nextdoor did what any self-respecting computer geek would do: they changed their software. Now, when you attempt to post about a crime or suspicious activity …

… you get smart, just in time nudges to think less about race, and more about behavior.

The results were striking:

Nextdoor claims this new multi-step system has, so far, reduced instances of racial profiling by 75%. It’s also decreased considerably the number of notes about crime and safety. During testing, the number of crime and safety issue reports abandoned before being published rose by 50%. β€œIt’s a fairly significant dropoff,” said Tolia, β€œbut we believe that, for Nextdoor, quality is more important than quantity.”

I'm a huge fan of designing software to help nudge people, at exactly the right time, to be their better selves. And this is a textbook example of doing it right.

Would using Nextdoor and encountering these dialogs make my aforementioned parent a little bit less racist? Probably not. But I like to think they would stop for at least a moment and consider the importance of focusing on the behavior that is problematic, rather than the individual person. This is a philosophy I promoted on Stack Overflow, I continue to promote with Discourse, and I reinforce daily with our three kids. You never, ever judge someone by what they look like. Look at what they do instead.

If you were getting excited about the prospect of validating Betteridge's Law yet again, I'm sorry to disappoint you. I truly do believe software, properly designed software, can not only help us be more civil to each other, but can also help people – maybe even people you love – behave a bit less like racists online.

[advertisement] At Stack Overflow, we help developers learn, share, and grow. Whether you’re looking for your next dream job or looking to build out your team, we've got your back.
Categories: Programming

Can Software Make You Less Racist?

Coding Horror - Jeff Atwood - Thu, 08/25/2016 - 08:52

I don't think we computer geeks appreciate how profoundly the rise of the smartphone, and Facebook, has changed the Internet audience. It's something that really only happened in the last five years, as smartphones and data plans dropped radically in price and became accessible – and addictive – to huge segments of the population.

People may have regularly used computers in 2007, sure, but that is a very different thing than having your computer in your pocket, 24/7, with you every step of every day, integrated into your life. As Jerry Seinfeld noted in 2014:

But I know you got your phone. Everybody here's got their phone. There's not one person here who doesn't have it. You better have it … you gotta have it. Because there is no safety, there is no comfort, there is no security for you in this life any more … unless when you're walking down the street you can feel a hard rectangle in your pants.

It's an addiction that is new to millions – but eerily familiar to us.

From "only nerds will use the Internet" to "everyone stares at their smartphones all day long!" in 20 years. Not bad, team :-).

— Marc Andreessen (@pmarca) January 16, 2015

The good news is that, at this moment, every human being is far more connected to their fellow humans than any human has ever been in the entirety of recorded history.

Spoiler alert: that's also the bad news.

Nextdoor is a Facebook-alike focused on specific neighborhoods. The idea is that you and everyone else on your block would join, and you can privately discuss local events, block parties, and generally hang out like neighbors do. It's a good idea, and my wife started using it a fair amount in the last few years. We feel more connected to our neighbors through the service. But one unfortunate thing you'll find out when using Nextdoor is that your neighbors are probably a little bit racist.

I don't use Nextdoor myself, but I remember Betsy specifically complaining about the casual racism she saw there, and I've also seen it mentioned several times on Twitter by people I follow. They're not the only ones. It became so epidemic that Nextdoor got a reputation for being a racial profiling hub. Which is obviously not good.

Social networking historically trends young, with the early adopters. Facebook launched as a site for college students. But as those networks grow, they inevitably age. They begin to include older people. And those older people will, statistically speaking, be more racist. I apologize if this sounds ageist, but let me ask you something: do you consider your parents a little racist? I will personally admit that one of my parents is definitely someone I would label a little bit racist. It's … not awesome.

The older the person, the more likely they are to have these "old fashioned" notions that the mere presence of differently-colored people on your block is inherently suspicious, and marriage should probably be defined as between a man and a woman.

In one meta-analysis by Jeffrey Lax and Justin Phillips of Columbia University, a majority of 18–29 year old Americans in 38 states support same sex marriage while in only 6 states do less than 45% of 18–29 year olds support same-sex marriage. At the same time not a single state shows support for same-sex marriage greater than 35% amongst those 64 and older

The idea that regressive social opinions correlate with age isn't an opinion; it's a statistical fact.

Support for same-sex marriage in the U.S.

18 - 29 years old    65%
30 - 49 years old    54%
50 - 64 years old    45%
65+ years old        39%

Are there progressive septuagenarians? Sure there are. But not many.

To me, failure to support same-sex marriage is as inconceivable as failing to support interracial marriage. Which was not that long ago, to the tune of the late 60s and early 70s. If you want some truly hair-raising reading, try Loving v. Virginia on for size. Because Virginia is for lovers. Just not those kind of lovers, 49 years ago. In the interests of full disclosure, I am 45 years old, and I graduated from the University of Virginia.

With Nextdoor, you're more connected with your neighbors than ever before. But through that connection you may also find out some regressive things about your neighbors that you'd never have discovered in years of the traditional daily routine of polite waves, hellos from the driveway, and casual sidewalk conversations.

To their immense credit, rather than accepting this status quo, Nextdoor did what any self-respecting computer geek would do: they changed their software. Now, when you attempt to post about a crime or suspicious activity …

… you get smart, just in time nudges to think less about race, and more about behavior.

The results were striking:

Nextdoor claims this new multi-step system has, so far, reduced instances of racial profiling by 75%. It’s also decreased considerably the number of notes about crime and safety. During testing, the number of crime and safety issue reports abandoned before being published rose by 50%. β€œIt’s a fairly significant dropoff,” said Tolia, β€œbut we believe that, for Nextdoor, quality is more important than quantity.”

I'm a huge fan of designing software to help nudge people, at exactly the right time, to be their better selves. And this is a textbook example of doing it right.

Would using Nextdoor and encountering these dialogs make my aforementioned parent a little bit less racist? Probably not. But I like to think they would stop for at least a moment and consider the importance of focusing on the behavior that is problematic, rather than the individual person. This is a philosophy I promoted on Stack Overflow, I continue to promote with Discourse, and I reinforce daily with our three kids. You never, ever judge someone by what they look like. Look at what they do instead.

If you were getting excited about the prospect of validating Betteridge's Law yet again, I'm sorry to disappoint you. I truly do believe software, properly designed software, can not only help people be more civil to each other, but can also help people – maybe even people you love – behave a bit less like racists online.

[advertisement] At Stack Overflow, we help developers learn, share, and grow. Whether you’re looking for your next dream job or looking to build out your team, we've got your back.
Categories: Programming

Taking the final wrapper off of Android 7.0 Nougat

Android Developers Blog - Tue, 08/23/2016 - 00:24

Posted by Dave Burke, VP of Engineering

Android Nougat

Android 7.0 Nougat

Today, Android 7.0 Nougat will begin rolling out to users, starting with Nexus devices. At the same time, we’re pushing the Android 7.0 source code to the Android Open Source Project (AOSP), extending public availability of this new version of Android to the broader ecosystem.

We’ve been working together with you over the past several months to get your feedback on this release, and also to make sure your apps are ready for the users who will run them on Nougat devices.

What’s inside Nougat

Android Nougat reflects input from thousands of fans and developers like you, all around the world. There are over 250 major features in Android Nougat, including VR Mode in Android. We’ve worked at all levels of the Android stack in Nougat — from how the operating system reads sensor data to how it sends pixels to the display — to make it especially built to provide high quality mobile VR experiences.

Plus, Nougat brings a number of new features to help make Android more powerful, more productive and more secure. It introduces a brand new JIT/AOT compiler to improve software performance, make app installs faster, and take up less storage. It also adds platform support for Vulkan, a low-overhead, cross-platform API for high-performance, 3D graphics. Multi-Window support lets users run two apps at the same time, and Direct Reply so users can reply directly to notifications without having to open the app. As always, Android is built with powerful layers of security and encryption to keep your private data private, so Nougat brings new features like File-based encryption, seamless updates, and Direct Boot.

You can find all of the Nougat developer resources here, including details on behavior changes and new features you can use in your apps. An overview of what's new for developers is available here, and you can explore all of the new user features in Nougat here.

Multi-window mode in Android Nougat

Multi-window mode in Android Nougat

The next wave of users

Starting today and rolling out over the next several weeks, the Nexus 6, Nexus 5X, Nexus 6P, Nexus 9, Nexus Player, Pixel C, and General Mobile 4G (Android One) will get an over-the-air software update to Android 7.0 Nougat. Devices enrolled in the Android Beta Program will also receive this final version.

And there are many tasty devices coming from our partners running Android Nougat, including the upcoming LG V20, which will be the first new smartphone that ships with Android Nougat, right out of the box.

With all of these new devices beginning to run Nougat, now is the time to publish your app updates to Google Play. We recommend compiling against, and ideally targeting, API 24. If you’re still testing some last minute changes, a great strategy to do this is using Google Play’s beta testing feature to get early feedback from a small group of users — including those using Android 7.0 Nougat — and then doing a staged rollout as you release the updated app to all users.

What’s next for Nougat?

We’re moving Nougat into a new regular maintenance schedule over the coming quarters. In fact, we’ve already started work on the first Nougat maintenance release, that will bring continued refinements and polish, and we’re planning to bring that to you this fall as a developer preview. Stay tuned!

We’ll be closing open bugs logged against Developer Preview builds soon, but please keep the feedback coming! If you still see an issue that you filed in the preview tracker, just file a new issue against Android 7.0 in the AOSP issue tracker.

Thanks for being part of the preview, which we shared earlier this year with an eye towards giving everyone the opportunity to make the next release of Android stronger. Your continued feedback has been extremely beneficial in shaping this final release, not just for users, but for the entire Android ecosystem.

Categories: Programming

Modernizing OAuth interactions in Native Apps for Better Usability and Security

Google Code Blog - Mon, 08/22/2016 - 22:29

Posted by William Denniss, Product Manager, Identity and Authentication

The Identity team is constantly striving to help Google users sign-in to third-party applications with their Google account in a secure and seamless way, and enable users to share select information from their account such as their calendar or contact information with other apps, when they wish to do so.

Under the hood these interactions happen via OAuth requests, and over the years Google has supported a number of ways for developers to implement OAuth flows with us. With improved security and usability in mind, we will soon be ending the support for one of these ways. In the coming months, we will no longer allow OAuth requests to Google in embedded browsers known as β€œweb-views”, such as the WebView UI element on Android and UIWebView/WKWebView on iOS, and equivalents on Windows and OS X.
Using the device browser for OAuth requests instead of an embedded web-view can improve the usability of your apps significantly: users only need to sign-in to Google once per device, improving conversion rates of sign-in and authorization flows in your app. Modern β€œin-app browser tab” patterns available on some operating systems, such as Chrome Custom Tabs on Android and SFSafariViewController on iOS offer further UX improvements for browser-based OAuth flows.

In contrast, the outdated method of using embedded browsers for OAuth means a user must sign-in to Google each time, instead of using the existing logged-in session from the device. The device browser also provides improved security as apps are able to inspect and modify content in a web-view, but not content shown in the browser.

To help you migrate, we offer libraries and samples that follow modern best practices which you can use:

  • Google Sign-In for Android and iOS, our recommended SDK for sign-in and OAuth with Google Accounts.
  • AppAuth for Android, iOS, and OS X, an open source OAuth client library that can be used with Google and other OAuth providers. We also offer GTMAppAuth (for iOS and OS X), a library which enables AppAuth support for the Google APIs Client Library for Objective-C, and the GTM Session Fetcherprojects.
  • Google Sign-in and OAuth Examples for Windows, examples demonstrating how to use the browser to authenticate Google users in various Windows environments such as Universal Windows Platform (UWP), console and desktop apps.

You can also read protocol-level documentation for our standards-based support of OAuth for Native Apps, and an IETF best current practice draft on this topic.

Versions of Google Sign-In on iOS prior to version 3.0 don’t support the current industry best practices of the in-app browser tab, and therefore are also deprecated. If you use Google Sign-In, please update to the latest version to get all the recent security and usability improvements. For now, this policy does not remove our support of WebView on iOS 8, however we may start to display notices encouraging users to upgrade their device for better security.

The rollout schedule for the deprecation of web-views for OAuth requests to Google is as follows. Starting October 20, 2016, we will prevent new OAuth clients from using web-views on platforms with a viable alternative, and will phase in user-facing notices for existing OAuth clients. On April 20, 2017, we will start blocking OAuth requests using web-views for all OAuth clients on platforms where viable alternatives exist.

If you have any questions with the migration, please post to Stack Overflow tagged with β€œgoogle-oauth”.

Categories: Programming

Neo4j/scikit-learn: Calculating the cosine similarity of Game of Thrones episodes

Mark Needham - Mon, 08/22/2016 - 22:12

A couple of months ago Praveena and I created a Game of Thrones dataset to use in a workshop and I thought it’d be fun to run it through some machine learning algorithms and hopefully find some interesting insights.

The dataset is available as CSV files but for this analysis I’m assuming that it’s already been imported into neo4j. If you want to import the data you can run the tutorial by typing the following into the query bar of the neo4j browser:

:play http://guides.neo4j.com/got

Since we don’t have any training data we’ll be using unsupervised learning methods, and we’ll start simple by calculating the similarity of episodes based character appearances. We’ll be using scitkit-learn‘s cosine similarity function to determine episode similarity.

Christian Perone has an excellent blog post explaining how to use cosine similarity on text documents which is well worth a read. We’ll be using a similar approach here, but instead of building a TF/IDF vector for each document we’re going to create a vector indicating whether a character appeared in an episode or not.

e.g. imagine that we have 3 characters – A, B, and C – and 2 episodes. A and B appear in the first episode and B and C appear in the second episode. We would represent that with the following vectors:

Episode 1 = [1, 1, 0]
Episode 2 = [0, 1, 1]

We could then calculate the cosine similarity between these two episodes like this:

>>> from sklearn.metrics.pairwise import cosine_similarity
>>> one = [1,1,0]
>>> two = [0,1,1]
 
>>> cosine_similarity([one, two])
array([[ 1. ,  0.5],
       [ 0.5,  1. ]])

So this is telling us that Episode 1 is 100% similar to Episode 1, Episode 2 is 100% similar to itself as well, and Episodes 1 and 2 are 50% similar to each other based on the fact that they both have an appearance of Character B.

Note that the character names aren’t even mentioned at all, they are implicitly a position in the array. This means that when we use our real dataset we need to ensure that the characters are in the same order for each episode, otherwise the calculation will be meaningless!

In neo4j land we have an APPEARED_IN relationship between a character and each episode that they appeared in. We can therefore write the following code using the Python driver to get all pairs of episodes and characters:

from neo4j.v1 import GraphDatabase, basic_auth
driver = GraphDatabase.driver("bolt://localhost", auth=basic_auth("neo4j", "neo"))
session = driver.session()
 
rows = session.run("""
    MATCH (c:Character), (e:Episode)
    OPTIONAL MATCH (c)-[appearance:APPEARED_IN]->(e)
    RETURN e, c, appearance
    ORDER BY e.id, c.id""")

We can iterate through the rows to see what the output looks like:

>>> for row in rows:
        print row
 
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5415 labels=set([u'Character']) properties={u'name': u'Addam Marbrand', u'id': u'/wiki/Addam_Marbrand'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5882 labels=set([u'Character']) properties={u'name': u'Adrack Humble', u'id': u'/wiki/Adrack_Humble'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=6747 labels=set([u'Character']) properties={u'name': u'Aegon V Targaryen', u'id': u'/wiki/Aegon_V_Targaryen'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5750 labels=set([u'Character']) properties={u'name': u'Aemon', u'id': u'/wiki/Aemon'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5928 labels=set([u'Character']) properties={u'name': u'Aeron Greyjoy', u'id': u'/wiki/Aeron_Greyjoy'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5503 labels=set([u'Character']) properties={u'name': u'Aerys II Targaryen', u'id': u'/wiki/Aerys_II_Targaryen'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=6753 labels=set([u'Character']) properties={u'name': u'Alannys Greyjoy', u'id': u'/wiki/Alannys_Greyjoy'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=6750 labels=set([u'Character']) properties={u'name': u'Alerie Tyrell', u'id': u'/wiki/Alerie_Tyrell'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5753 labels=set([u'Character']) properties={u'name': u'Alliser Thorne', u'id': u'/wiki/Alliser_Thorne'}> appearance=None>
<Record e=<Node id=6780 labels=set([u'Episode']) properties={u'season': 1, u'number': 1, u'id': 1, u'title': u'Winter Is Coming'}> c=<Node id=5858 labels=set([u'Character']) properties={u'name': u'Alton Lannister', u'id': u'/wiki/Alton_Lannister'}> appearance=None>

Next we’ll build a ‘matrix’ of episodes/characters. If a character appears in an episode then we’ll put a ‘1’ in the matrix, if not we’ll put a ‘0’:

episodes = {}
for row in rows:
    if episodes.get(row["e"]["id"]) is None:
        if row["appearance"] is None:
            episodes[row["e"]["id"]] = [0]
        else:
            episodes[row["e"]["id"]] = [1]
    else:
        if row["appearance"] is None:
            episodes[row["e"]["id"]].append(0)
        else:
            episodes[row["e"]["id"]].append(1)

Here’s an example of one entry in the matrix:

>>> len(episodes)
60
 
>>> len(episodes[1])
638
 
>>> episodes[1]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

From this output we learn that there are 60 episodes and 638 characters in Game of Thrones so far. We can also see which characters appeared in the first episode, although it’s a bit tricky to work out which index in the array corresponds to each character.

The next thing we’re going to do is calculate the cosine similarity between episodes. Let’s start by seeing how similar the first episode is to all the others:

>>> all = episodes.values()
 
>>> cosine_similarity(all[0:1], all)[0]
array([ 1.        ,  0.69637306,  0.48196269,  0.54671752,  0.48196269,
        0.44733753,  0.31707317,  0.42340087,  0.34989921,  0.43314808,
        0.36597766,  0.18421252,  0.30961158,  0.2328101 ,  0.30616181,
        0.41905818,  0.36842504,  0.35338088,  0.18376917,  0.3569686 ,
        0.2328101 ,  0.34539847,  0.25043516,  0.31707317,  0.25329221,
        0.33342786,  0.34921515,  0.2174909 ,  0.2533473 ,  0.28429311,
        0.23026565,  0.22310537,  0.22365301,  0.23816275,  0.28242289,
        0.16070148,  0.24847093,  0.21434648,  0.03582872,  0.21189672,
        0.15460414,  0.17161693,  0.15460414,  0.17494961,  0.1234662 ,
        0.21426863,  0.21434648,  0.18748505,  0.15308091,  0.20161946,
        0.19877675,  0.30920827,  0.21058466,  0.19127301,  0.24607943,
        0.18033393,  0.17734311,  0.16296707,  0.18740851,  0.23995201])

The first entry in the array indicates that episode 1 is 100% similar to episode 1 which is a good start. It’s 69% similar to episode 2 and 48% similar to episode 3. We can sort that array to work out which episodes it’s most similar to:

>>> for idx, score in sorted(enumerate(cosine_similarity(all[0:1], all)[0]), key = lambda x: x[1], reverse = True)[:5]:
        print idx, score
 
0 1.0
1 0.696373059207
3 0.546717521051
2 0.481962692712
4 0.481962692712

Or we can see how similar the last episode of season 6 is compared to the others:

>>> for idx, score in sorted(enumerate(cosine_similarity(all[59:60], all)[0]), key = lambda x: x[1], reverse = True)[:5]:
        print idx, score
 
59 1.0
52 0.500670191678
46 0.449085146211
43 0.448218732478
49 0.446296233312

I found it a bit painful exploring similarities like this so I decided to write them into neo4j instead and then write a query to find the most similar episodes. The following query creates a SIMILAR_TO relationship between episodes and sets a score property on that relationship:

>>> episode_mapping = {}
>>> for idx, episode_id in enumerate(episodes):
        episode_mapping[idx] = episode_id
 
>>> for idx, episode_id in enumerate(episodes):
        similarity_matrix = cosine_similarity(all[idx:idx+1], all)[0]
        for other_idx, similarity_score in enumerate(similarity_matrix):
            other_episode_id = episode_mapping[other_idx]
            print episode_id, other_episode_id, similarity_score
            if episode_id != other_episode_id:
                session.run("""
                    MATCH (episode1:Episode {id: {episode1}}), (episode2:Episode {id: {episode2}})
                    MERGE (episode1)-[similarity:SIMILAR_TO]-(episode2)
                    ON CREATE SET similarity.score = {similarityScore}
                    """, {'episode1': episode_id, 'episode2': other_episode_id, 'similarityScore': similarity_score})
 
    session.close()

The episode_mapping dictionary is needed to map from episode ids to indices e.g. episode 1 is at index 0.

If we want to find the most similar pair of episodes in Game of Thrones we can execute the following query:

MATCH (episode1:Episode)-[similarity:SIMILAR_TO]-(episode2:Episode)
WHERE ID(episode1) > ID(episode2)
RETURN "S" + episode1.season + "E" + episode1.number AS ep1, 
       "S" + episode2.season + "E" + episode2.number AS ep2, 
       similarity.score AS score
ORDER BY similarity.score DESC
LIMIT 10
 
╒═════╀════╀══════════════════╕
β”‚ep1  β”‚ep2 β”‚score             β”‚
β•žβ•β•β•β•β•β•ͺ════β•ͺ══════════════════║
β”‚S1E2 β”‚S1E1β”‚0.6963730592072543β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S1E4 β”‚S1E3β”‚0.6914173051223086β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S1E9 β”‚S1E8β”‚0.6869464497590777β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S2E10β”‚S2E8β”‚0.6869037302955034β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S3E7 β”‚S3E6β”‚0.6819943394704735β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S2E7 β”‚S2E6β”‚0.6813598225089799β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S1E10β”‚S1E9β”‚0.6796436827080401β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S1E5 β”‚S1E4β”‚0.6698105143372364β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S1E10β”‚S1E8β”‚0.6624062584864754β”‚
β”œβ”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E5 β”‚S4E4β”‚0.6518358737330705β”‚
β””β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

And the least popular?

MATCH (episode1:Episode)-[similarity:SIMILAR_TO]-(episode2:Episode)
WHERE ID(episode1) > ID(episode2)
RETURN "S" + episode1.season + "E" + episode1.number AS ep1, 
       "S" + episode2.season + "E" + episode2.number AS ep2, 
       similarity.score AS score
ORDER BY similarity.score
LIMIT 10
 
╒════╀════╀═══════════════════╕
β”‚ep1 β”‚ep2 β”‚score              β”‚
β•žβ•β•β•β•β•ͺ════β•ͺ═══════════════════║
β”‚S4E9β”‚S1E5β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E9β”‚S1E6β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E9β”‚S4E2β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E9β”‚S2E9β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E9β”‚S2E4β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S5E6β”‚S4E9β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S6E8β”‚S4E9β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E9β”‚S4E6β”‚0                  β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S3E9β”‚S2E9β”‚0.03181423814878889β”‚
β”œβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚S4E9β”‚S1E1β”‚0.03582871819500093β”‚
β””β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

The output of this query suggests that there are no common characters between 8 pairs of episodes which at first glance sounds surprising. Let’s write a query to check that finding:

MATCH (episode1:Episode)<-[:APPEARED_IN]-(character)-[:APPEARED_IN]->(episode2:Episode)
WHERE episode1.season = 4 AND episode1.number = 9 AND episode2.season = 1 AND episode2.number = 5
return episode1, episode2
 
(no changes, no rows)

It’s possible I made a mistake with the scraping of the data but from a quick look over the Wiki page I don’t think I have. I found it interesting that Season 4 Episode 9 shows up on 9 of the top 10 least similar pairs of episodes.

Next I’m going to cluster the episodes based on character appearances, but this post is long enough already so that’ll have to wait for another post another day.

Categories: Programming

Protecting Android with more Linux kernel defenses

Android Developers Blog - Sun, 08/21/2016 - 21:53

Posted by Jeff Vander Stoep, Android Security team

Android relies heavily on the Linux kernel for enforcement of its security model. To better protect the kernel, we’ve enabled a number of mechanisms within Android. At a high level these protections are grouped into two categoriesβ€”memory protections and attack surface reduction.

Memory protections

One of the major security features provided by the kernel is memory protection for userspace processes in the form of address space separation. Unlike userspace processes, the kernel’s various tasks live within one address space and a vulnerability anywhere in the kernel can potentially impact unrelated portions of the system’s memory. Kernel memory protections are designed to maintain the integrity of the kernel in spite of vulnerabilities.

Mark memory as read-only/no-execute

This feature segments kernel memory into logical sections and sets restrictive page access permissions on each section. Code is marked as read only + execute. Data sections are marked as no-execute and further segmented into read-only and read-write sections. This feature is enabled with config option CONFIG_DEBUG_RODATA. It was put together by Kees Cook and is based on a subset of Grsecurity’s KERNEXEC feature by Brad Spengler and Qualcomm’s CONFIG_STRICT_MEMORY_RWX feature by Larry Bassel and Laura Abbott. CONFIG_DEBUG_RODATA landed in the upstream kernel for arm/arm64 and has been backported to Android’s 3.18+ arm/arm64 common kernel.

Restrict kernel access to userspace

This feature improves protection of the kernel by preventing it from directly accessing userspace memory. This can make a number of attacks more difficult because attackers have significantly less control over kernel memory that is executable, particularly with CONFIG_DEBUG_RODATA enabled. Similar features were already in existence, the earliest being Grsecurity’s UDEREF. This feature is enabled with config option CONFIG_CPU_SW_DOMAIN_PAN and was implemented by Russell King for ARMv7 and backported to Android’s 4.1 kernel by Kees Cook.

Improve protection against stack buffer overflows

Much like its predecessor, stack-protector, stack-protector-strong protects against stack buffer overflows, but additionally provides coverage for more array types, as the original only protected character arrays. Stack-protector-strong was implemented by Han Shen and added to the gcc 4.9 compiler.

Attack surface reduction

Attack surface reduction attempts to expose fewer entry points to the kernel without breaking legitimate functionality. Reducing attack surface can include removing code, removing access to entry points, or selectively exposing features.

Remove default access to debug features

The kernel’s perf system provides infrastructure for performance measurement and can be used for analyzing both the kernel and userspace applications. Perf is a valuable tool for developers, but adds unnecessary attack surface for the vast majority of Android users. In Android Nougat, access to perf will be blocked by default. Developers may still access perf by enabling developer settings and using adb to set a property: β€œadb shell setprop security.perf_harden 0”.

The patchset for blocking access to perf may be broken down into kernel and userspace sections. The kernel patch is by Ben Hutchings and is derived from Grsecurity’s CONFIG_GRKERNSEC_PERF_HARDEN by Brad Spengler. The userspace changes were contributed by Daniel Micay. Thanks to Wish Wu and others for responsibly disclosing security vulnerabilities in perf.

Restrict app access to ioctl commands

Much of Android security model is described and enforced by SELinux. The ioctl() syscall represented a major gap in the granularity of enforcement via SELinux. Ioctl command whitelisting with SELinux was added as a means to provide per-command control over the ioctl syscall by SELinux.

Most of the kernel vulnerabilities reported on Android occur in drivers and are reached using the ioctl syscall, for example CVE-2016-0820. Some ioctl commands are needed by third-party applications, however most are not and access can be restricted without breaking legitimate functionality. In Android Nougat, only a small whitelist of socket ioctl commands are available to applications. For select devices, applications’ access to GPU ioctls has been similarly restricted.

Require seccomp-bpf

Seccomp provides an additional sandboxing mechanism allowing a process to restrict the syscalls and syscall arguments available using a configurable filter. Restricting the availability of syscalls can dramatically cut down on the exposed attack surface of the kernel. Since seccomp was first introduced on Nexus devices in Lollipop, its availability across the Android ecosystem has steadily improved. With Android Nougat, seccomp support is a requirement for all devices. On Android Nougat we are using seccomp on the mediaextractor and mediacodec processes as part of the media hardening effort.

Ongoing efforts

There are other projects underway aimed at protecting the kernel:

  • The Kernel Self Protection Project is developing runtime and compiler defenses for the upstream kernel.
  • Further sandbox tightening and attack surface reduction with SELinux is ongoing in AOSP.
  • Minijail provides a convenient mechanism for applying many containment and sandboxing features offered by the kernel, including seccomp filters and namespaces.
  • Projects like kasan and kcov help fuzzers discover the root cause of crashes and to intelligently construct test cases that increase code coverageβ€”ultimately resulting in a more efficient bug hunting process.

Due to these efforts and others, we expect the security of the kernel to continue improving. As always, we appreciate feedback on our work and welcome suggestions for how we can improve Android. Contact us at security@android.com.

Categories: Programming

Google Developers to open a startup space in San Francisco

Google Code Blog - Thu, 08/18/2016 - 19:10

Posted by Roy Glasberg Global Lead, Launchpad Accelerator

We’re heading to the city of San Francisco this September to open a new space for developers and startups. With over 14,000 sq. ft. at 301 Howard Street, we’ll have more than enough elbow room to train, educate and collaborate with local and international developers and startups.

The space will hold a range of events: Google Developer Group community meetups, Codelabs, Design Sprints, and Tech Talks. It will also host the third class of Launchpad Accelerator, our equity-free accelerator for startups in emerging markets. During each class, over 20 Google teams provide comprehensive mentoring to late-stage app startups who seek to scale and become leaders in their local markets. The 3-month program starts with an all-expenses-paid two week bootcamp at Google HQ.

Developers are in an ever-changing landscape and seek technical training. We’ve also seen a huge surge in the number of developers starting their own companies. Lastly, this is an unique opportunity to bridge the gap between Silicon Valley and emerging markets. To date Launchpad Accelerator has nearly 50 alumni in India, Indonesia, Brazil and Mexico. Startups in these markets are tackling critical local problems, but they often lack access to the resources and network we have here. This dedicated space will enable us to regularly engage with developers and serve their evolving needs, whether that is to build a product, grow a company or make revenue.

We can’t wait to get started and work with developers to build successful businesses that have a positive impact locally and globally.

Categories: Programming

A Google Santa Tracker update from Santa's Elves

Google Code Blog - Wed, 08/17/2016 - 00:10

Sam Thorogood, Developer Programs Engineer

Today, we're announcing that the open source version of Google's Santa Tracker has been updated with the Android and web experiences that ran in December 2015. We extended, enhanced and upgraded our code, and you can see how we used our developer products - including Firebase and Polymer - to build a fun, educational and engaging experience.

To get started, you can check out the code on GitHub at google/santa-tracker-weband google/santa-tracker-android. Both repositories include instructions so you can build your own version.

Santa Tracker isn’t just about watching Santa’s progress as he delivers presents on December 24. Visitors can also have fun with the winter-inspired experiences, games and educational content by exploring Santa's Village while Santa prepares for his big journey throughout the holidays.

Below is a summary of what we’ve released as open source.

Android app
  • The Santa Tracker Android app is a single APK, supporting all devices, such as phones, tablets and TVs, running Ice Cream Sandwich (4.0) and up. The source code for the app can be found here.
  • Santa Tracker leverages Firebase features, including Remote Config API, App Invites to invite your friends to play along, and Firebase Analytics to help our elves better understand users of the app.
  • Santa’s Village is a launcher for videos, games and the tracker that responds well to multiple devices such as phones and tablets. There's even an alternative launcher based on the Leanback user interface for Android TVs.

  • Games on Santa Tracker Android are built using many technologies such as JBox2D (gumball game), Android view hierarchy (memory match game) and OpenGL with special rendering engine (jetpack game). We've also included a holiday-themed variation of Pie Noon, a fun game that works on Android TV, your phone, and inside Google Cardboard's VR.
Android Wear

  • The custom watch faces on Android Wear provide a personalized touch. Having Santa or one of his friendly elves tell the time brings a smile to all. Building custom watch faces is a lot of fun but providing a performant, battery friendly watch face requires certain considerations. The watch face source code can be found here.
  • Santa Tracker uses notifications to let users know when Santa has started his journey. The notifications are further enhanced to provide a great experience on wearables using custom backgrounds and actions that deep link into the app.
On the web

  • Santa Tracker is mobile-first: this year's experience was built for the mobile web, including an amazing brand new, interactive - yet fully responsive, village: with three breakpoints, touch gesture support and support for the Web App Manifest.
  • To help us develop Santa at scale, we've upgraded to Polymer 1.0+. Santa Tracker's use of Polymer demonstrates how easy it is to package code into reusable components. Every housein Santa's Village is a custom element, only loaded when needed, minimizing the startup cost of Santa Tracker.

  • Many of the amazing new games (like Present Bounce) were built with the latest JavaScript standards (ES6) and are compiled to support older browsers via the Google Closure Compiler.
  • Santa Tracker's interactive and fun experience is enhanced using the Web Animations API, a standardized JavaScript APIfor unifying animated content.
  • We simplified the Chromecast support this year, focusing on a great screensaver that would countdown to the big event on December 24th - and occasionally autoplay some of the great video content from around Santa's Village.

We hope that this update inspires you to make your own magical experiences based on all the interesting and exciting components that came together to make Santa Tracker!

Categories: Programming

SE-Radio Episode 266: Charles Nutter on the JVM as a Language Platform

Charles Nutter talks to Charles Anderson about the JRuby language and the JVM as a platform for implementing programming languages. They discuss JRuby and its implementation on the JVM as an example of a language other than Java on the JVM. Venue: Skype Related Links Charles Nutter on Twitter:Β https://twitter.com/headius Charles Nutter on GitHub: https://github.com/headius JRuby […]
Categories: Programming

The Legend of the 5 Monkeys, the Doctor and the Rose

Xebia Blog - Mon, 08/15/2016 - 17:16
As Product Managers people look up to us to carry the vision, to make sure all the noses are aligned, the troops are rallied and that sort of stuff. But what is it that influences behavior? And what makes your team do what they do? The answer has more to do with you than with

Python: matplotlib/seaborn/virtualenv – Python is not installed as a framework

Mark Needham - Sun, 08/14/2016 - 19:56

Over the weekend I was following The Marketing Technologist’s content based recommender tutorial but ran into the following exception when trying to import the seaborn library:

$ python 5_content_based_recommender/run.py 
Traceback (most recent call last):
  File "5_content_based_recommender/run.py", line 14, in <module>
    import seaborn as sns
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/seaborn/__init__.py", line 6, in <module>
    from .rcmod import *
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/seaborn/rcmod.py", line 8, in <module>
    from . import palettes, _orig_rc_params
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/seaborn/palettes.py", line 12, in <module>
    from .utils import desaturate, set_hls_values, get_color_cycle
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/seaborn/utils.py", line 12, in <module>
    import matplotlib.pyplot as plt
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/matplotlib/pyplot.py", line 114, in <module>
    _backend_mod, new_figure_manager, draw_if_interactive, _show = pylab_setup()
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/matplotlib/backends/__init__.py", line 32, in pylab_setup
    globals(),locals(),[backend_name],0)
  File "/Users/markneedham/projects/themarketingtechnologist/tmt/lib/python2.7/site-packages/matplotlib/backends/backend_macosx.py", line 24, in <module>
    from matplotlib.backends import _macosx
RuntimeError: Python is not installed as a framework. The Mac OS X backend will not be able to function correctly if Python is not installed as a framework. See the Python documentation for more information on installing Python as a framework on Mac OS X. Please either reinstall Python as a framework, or try one of the other backends. If you are Working with Matplotlib in a virtual enviroment see 'Working with Matplotlib in Virtual environments' in the Matplotlib FAQ

We can see from the stacktrace that seaborn calls matplotlib so that’s where the problem lies. There’s even a page on the matplotlib website suggesting some workarounds.

I’ve come across this error before and been unable to get any of the suggestions to work, but this time I was successful. I needed to create the following function in my bash profile file:


~/.bash_profile

function frameworkpython {
    if [[ ! -z "$VIRTUAL_ENV" ]]; then
        PYTHONHOME=$VIRTUAL_ENV /usr/bin/python "$@"
    else
        /usr/bin/python "$@"
    fi
}

And call that function instead of my virtualenv’s python:

$ frameworkpython 5_content_based_recommender/run.py

This time the matplotlib visualisation works:

2016 08 14 16 16 08

#win

Categories: Programming

New features for reviews and experiments in Google Play Developer Console app

Android Developers Blog - Wed, 08/10/2016 - 20:07

Posted by Kobi Glick, Google Play team

With over one million apps published through the Google Play Developer Console, we know how important it is to publish with confidence, acquire users, learn about them, and manage your business. Whether reacting to a critical performance issue or responding to a negative review, checking on your apps when and where you need to is invaluable.

The Google Play Developer Console app, launched in May, has already helped thousands of developers stay informed of crucial business updates on the go.

We’re excited to tell you about new features, available today:

Receive notifications about new reviews

Use filters to find the reviews you want

Review and apply store listing experiment results

Increase the percent of a staged rollout or halt a bad staged rollout

Download the Developer Console app on Google Play and stay on top of your apps and games, wherever you are! Also, get the Playbook for Developers app to stay up-to-date with more features and best practices that will help you grow a successful business on Google Play.

Categories: Programming

Adding a bit more reality to your augmented reality apps with Tango

Google Code Blog - Wed, 08/10/2016 - 19:14

Posted by Sean Kirmani, Software Engineering Intern, Tango

Augmented reality scenes, where a virtual object is placed in a real environment, can surprise and delight people whether they’re playing with dominoes or trying to catch monsters. But without support for environmental lighting, these virtual objects can stick out rather than blend in with their environments. Ambient lighting should bleed onto an object, real objects should be seen in reflective surfaces, and shade should darken a virtual object.

Tango-enabled devices can see the world like we do, and they’re designed to bring mobile augmented reality closer to real reality. To help bring virtual objects to life, we’ve updated the Tango Unity SDK to enable developers to add environmental lighting to their Tango apps. Here’s how to get started:

Let’s dive in!

Before we begin, you’ll need to download the Tango Unity SDK. Then you can follow the steps below to make your reality a little brighter.

Step 1: Create a new Unity project and import the Tango SDK package into the project.

Step 2: Create a new scene. If you need help with this, check out the solar system tutorial from a previous post. Then you’ll add Tango Manager and Tango AR Camera prefabs to your scene and remove the default Main Camera game object. Also remove the artificial directional light. We won’t need that anymore. After doing this, you should see the scene hierarchy like this:

Step 3: In the Tango Manager game object, you’ll want to check Enable Video Overlay and set the method to Texture and Raw Bytes.

Step 4: Under Tango AR Camera, look for the Tango Environmental Lighting component. Make that the the Enable Environmental Lighting checkbox is checked.

Step 5: Add your game object that you’d like to be environmental lit to the scene. In our example, we’ll be using a pool ball. So let’s add a new Sphere.

Step 6: Let’s create a new material for our sphere. Go to Create > Material. We’ll be using our environmental lighting shader on this object. Under Shader, select Tango >Environmental Lighting > Standard.

Step 7: Let’s add a texture to our pool ball and tweak our smoothness parameter. The higher the smoothness, the more reflective our object becomes. Rougher objects have more of a diffuse lighting that is softer and spreads over the surface of the object. You can download the pool_ball_textureand import it into your project.

Step 8: Add your new material to your sphere, so you have a nicer looking pool ball.

Step 9: Compile and run the application again. You should able see environment lit pool ball now!

You can also follow our previous post and be able to place your pool ball on surfaces. You don’t have to worry about your sphere rolling off your surface. Here are some comparison pictures of the pool ball with a static artificial light (left) and with environment lighting (right).

We hope you enjoyed this tutorial combining the joy of environmental lighting with the magic of AR. Stay tuned to this blog for more AR updates and tutorials!

We’re just getting started!

You’ve just created a more realistically light pool ball that live in AR. That’s a great start, but there’s a lot more you can do to make a high performance smartphone AR application. Check out our Unity example code on Github (especially the Augmented Reality example) to learn more about building a good smartphone AR application.

Categories: Programming

Android Developer Story: Hole19 improves user retention with Android Wear

Android Developers Blog - Wed, 08/10/2016 - 18:48

Posted by Lily Sheringham, Google Play team

Based in Lisbon, Portugal, Hole19 is a golfing app which assists golfers before, during, and after their golfing journey with GPS and a digital scorecard. The app connects the golfing community with shared statistics for performance and golf courses, and now has close to 1 million users across all platforms.

Watch Anthony Douglas, Founder & CEO, and FΓ‘bio Carballo, Head Android Developer, explain how Hole19 doubled its number of Android Wear users in 6 months, and improved user engagement and retention on the platform. Also, hear how they are using APIs and the latest Wear 2.0 features to connect users to their golfing data and improve the user experience.


Learn more how to get started with Android Wear and get the Playbook for Developers app to stay up-to-date with more features and best practices that will help you grow a successful business on Google Play.

Categories: Programming

Expand Your Global Reach on Google Play With New Language and Country Analytics

Android Developers Blog - Tue, 08/09/2016 - 18:35

Posted by Rahim Nathwani Product Manager, App Translation Service

With users in 190 countries around the world, Google Play offers you a truly global audience for your apps and games. Localization is one of the most powerful ways to connect with people in different places, which is why we launched translation support for in-app purchase and Universal App Campaigns earlier this year. With over 30 language translation options available via the Developer Console, we updated our app translation service to help you select the most relevant languages, making it quick and easy to get started.

With the launch of new language and country analytics, you gain access to app install analysis on Google Play, including:

  • Information on the top languages and countries where apps have been installed, broken down to the level of your app’s category
  • The percentage of installs that come from users of those languages
  • Further information to help inform your go-to-market plans for these countries

To make ordering translations easier, we show language bundles that you can add to your order in a single click.

To get started, select Manage translations -> Purchase translations from the Store Listing page in the Google Play Developer Console.

Categories: Programming

Daydream Labs: positive social experiences in VR

Google Code Blog - Tue, 08/09/2016 - 17:52

Posted by Robbie Tilton, UX Designer, Google VR

At Daydream Labs, we have experimented with social interactions in VR. Just like in real reality, people naturally want to share and connect with others in VR. As developers and designers, we are excited to build social experiences that are fun and easy to useβ€”but it’s just as important to make it safe and comfortable for all involved. Over the last year, we’ve learned a few ways to nudge people towards positive social experiences.

What can happen without clear social norms

People are curious and will test the limits of your VR experience. For example, when some people join a multiplayer app or game, they might wonder if they can reach their hand through another player’s head or stand inside another avatar’s body. Even with good intentions, this can make other people feel unsafe or uncomfortable.

For example, in a shopping experiment we built for the HTC Vive, two people could enter a virtual storefront and try on different hats, sunglasses, and accessories. There was no limit to how or where they could place a virtual accessory, so some people stuck hats on friends anywhere they would stickβ€”like in front of their eyes. This had the unfortunate effect of blocking their vision. If they couldn’t remove the hat in front of their eyes with their controllers, they had no other recourse than to take off their headset and end their VR experience.


Protecting user safety

Everyone should feel safe and comfortable in VR. If we can anticipate the actions of others, then we may be able to discourage negative social behavior before it starts. For example, by designing personal space around each user, you can prevent other people from invading that personal space.

We built an experiment around playing poker where we tried new ways to discourage trolling. If someone left their seat at the poker table, their environment desaturated to black and white and their avatar would disappear from the other player’s view. A glowing blue personal space bubble would guide the person back to their seat. We found it’s enough to prevent a player from approaching their opponents to steal chips or invade personal space.


Reward positive behavior

If you want people to interact in positive waysβ€”like high-fiving

Categories: Programming

SE-Radio Episode 265: Pat Kua on Becoming a Tech Lead

Johannes ThΓΆnes talks to Patrick Kua about the role of a technical lead and how people become tech leads. The show covers the definition of a tech lead, the responsibilities of the role and the challenges of becoming a tech lead. Venue:Β Internet   Related Links Episode 228: Software Architecture Sketches with Simon Brown Article: A […]
Categories: Programming

Mapping Biases to Testing: Confirmation Bias

Xebia Blog - Mon, 08/08/2016 - 20:24
I use terminology from earlier blog posts about biases. If you have missed those posts, read part 1 here. I explain the terminology there. In the second post I wrote about the Anchoring Effect. Let me state the β€˜bad news’ up front: you cannot fully avoid the confirmation bias. That’s actually a good thing, because

Schell Games gives popular games a twist with Tango

Google Code Blog - Wed, 08/03/2016 - 18:36

Posted by Justin Quimby, Senior Product Manager Tango

At Tech World last month, our team showed off some of the latest Tango-enabled games. One crowd favorite was Domino World by Schell Games which will will be available on the first Tango-enabled device, Lenovo’s Phab 2 Pro, coming this fall. Schell Games has adapted a few classic games, including Jenga, into smartphone augmented reality, and their developers share their experience and considerations they kept in mind as they gave dominoes a new twist.

Google: How did your team first hear about Tango technology?

Schell Games: The Tango team invited us to their Game Developer Workshopwhere we learned about Tango and the types of apps we could develop for this platform.

Google: You took a classic game, and added AR elements. How did you come to dominoes?

Schell Games: At the Game Developer Workshop, we prototyped three games: a racing game, Jenga and a pet game. Of the three games, people connected the most with Jenga.

People loved sharing a device to play the game togetherβ€”and they loved that they didn’t have to pick up all the Jenga pieces when the game was over! And from a developer perspective, Jenga was great as it highlighted Tango’s ability to recognize surfaces.

Based on how much people liked Jenga, we decided that Domino World would be our second game. Domino World gives players all the fun of dominoes, but without the setup effort or mess. We were inspired by YouTube videos where people of all ages were doing really creative things with dominoes. Our goal was to bring that experience to the phone as an immersive and fun augmented-reality experience.

Google: Which Tango features did you use in Jenga and Domino World?

Schell Games: We used motion tracking, which lets people walk around their dominoes or Jenga tower. We also used surface detection with the depth camera, so that the device recognizes when objects are placed on a surface.

Google: How does your development approach differ for AR apps versus standard mobile apps?

Schell Games: With Domino World, for example, our approach to augmented reality thrives on reinforcing the feeling that the player’s display is a β€œwindow on the world.” Toys and dominoes are (virtually) placed on the actual surfaces around the player, and the game’s controls aid players in manipulating objects in the space in front of them. As a result, the player is naturally encouraged move around as they view, adjust and otherwise shape their ever-growing creations.

In contrast, traditional touchscreen controls largely work with metaphors of interacting with the screen’s image itself -- drawing on it, pinch-zooming it, etc. As a result, a more traditional touchscreen-controlled Domino World could have influenced players to remain more static and work with the existing view, as opposed to moving around to different vantage points.

Google: We noticed that you use a landscape orientation for Domino World. How did you decide to take that approach.

Schell Games: The decision to use landscape orientation for Domino World is the result of multiple smaller reasons all put together:

  • Many new players have a tendency to initially build wider versus deeper (possibly due to an instinctive desire to be able to more easily access their domino runs).
  • UI controls at the edges of a landscape layout minimizes HUD overlap when working with wider versus. deeper runs.
  • A landscape orientation naturally places players’ a hands at the device’s corners, which makes for a more stable grip during gameplay.

Google: What surprised you the most while building with Tango?

Schell Games: We were quite surprised at how easy it was to build with the Tango SDK and add Tango functionality to our apps. We used the Unity Engine which made the whole process quite seamless. It took us just over two weeks to build Jenga and 10 weeks to build Domino World from beginning to end.

Google: How do you think Tango will change the way people play games?

Schell Games: Tango makes it easy to play AR games. You don’t need to print and cut out AR trackers or markers to place throughout your room to help orient the phone. Instead, your phone always knows where it is in relation to the AR objects and you can easily start playingβ€”whether you’re in a living room or on a bus. It’s incredible to have this experience with just your mobile device.

Categories: Programming