Skip to content

Software Development Blogs: Programming, Software Testing, Agile Project Management

Methods & Tools

Subscribe to Methods & Tools
if you are not afraid to read more than one page to be a smarter software developer, software tester or project manager!

Programming

Want to preview our new DataGrid for Xamarin.Forms?

Eric.Weblog() - Eric Sink - Fri, 11/21/2014 - 19:00
tl;dr

Zumero.DataGrid is a Xamarin.Forms control for displaying data in rows and columns.

If you would be interested in testing and previewing a trial version of this product, email us:

  • datagrid-testing@zumero.com

Details
  • We plan to submit this to the Xamarin Component Store where (pending Xamarin's approval) it would be a commercial product. Pricing has not been set.

  • You'll receive a file in Xamarin Component format (a .xam file) which you will need to install into your IDE.
    (command line: xamarin-component.exe install ZumeroDataGrid-1.0.0.xam)
    The component file contains the libraries for all platforms, plus documentation and samples.

  • The test component is a trial version. It overlays the contents of some (randomly chosen) grid cells with an orange label that says "Trial". It is otherwise identical to the full version.

  • This component is unrelated to the one I mentioned in a blog entry back on 3 October. That approach was based on drawing. It was not capable of hosting Xamarin.Forms views, so it couldn't do things like in-place editing. This component is entirely different.

  • We are eager to hear feedback of any kind.

Features
  • Support for cell contents to be any Xamarin.Forms.View
  • Scrolling, both horizontal and vertical
  • Optional top frozen header row
  • Optional left frozen column
  • Separate View template for each column
  • View recycling (If you have 1000 rows but only 10 can be visible, DataGrid creates 10 Views and reuses them.)
  • Data row objects (provided to the DataGrid as an IList<object>)
  • Strong integration with the Xamarin.Forms binding mechanism
  • Automatic updates if an ObservableCollection is used for rows and columns
  • Configurable row height (for all rows in the grid)
  • Configurable column widths (can be different for each column)
  • Configurable spacing between rows and columns
  • Works with C#, F#, and XAML
  • Support for iOS, Android, and Windows Phone
  • Full API documentation in MonoDoc
  • Technical support service from Zumero
Screenshots

 

Episode 215: Gang of Four – 20 Years Later

Johannes Thönes talks with Erich Gamma, Ralph Johnson and Richard Helm from the Gang of Four about the 20th anniversary of their book Design Patterns. They discuss the following topics: the definition of a design pattern and each guest’s favorite design pattern; the origins of the book in architecture workshops; the writing of the book […]
Categories: Programming

Google Analytics Demos & Tools

Google Code Blog - Thu, 11/20/2014 - 18:05
As a member of the Google Analytics Developer Relations team, I often hear from our community that they want to do more with GA but don't always know how. They know the basics but want to see full examples and demos that show how things should be built.

Well, we've been listening, and today I'm proud to announce the launch of Google Analytics Demos & Tools, a new website geared toward helping Google Analytics developers tackle the challenges they face most often.

The site aims to make experienced developers more productive (we use it internally all the time) and to show new users what's possible and inspire them to leverage the platform to improve their business through advanced measurement and analysis.

Google Analytics Developers Image
Some highlights of the site include a full-featured Enhanced Ecommerce demo with code samples for both Google Analytics and Google Tag Manager, a new Account Explorer tool to help you quickly find the IDs you need for various Google Analytics services and 3rd party integrations, several examples of easy-to-build custom dashboards, and some old favorites like the Query Explorer.

Google Analytics Demos & Tools not only shows off Google Analytics technologies, it also uses them under the hood. All pages that require authorization use the Embed API to log users in, and usage statistics, including outbound link clicks, authorization status, client-side exceptions, and numerous other user interaction events are measured using analytics.js.

Every page that makes use of a Google Analytics technology lists that information in the footer, making it easy for developers to see how all the pieces fit together. In addition, the entire site is open sourced and available on Github, so you can dive in and see exactly how everything works.

Feedback is welcome and appreciated!

Posted By Philip Walton, Developer Programs Engineer
Categories: Programming

How Do You Estimate What You Don’t Know?

Making the Complex Simple - John Sonmez - Thu, 11/20/2014 - 16:00

In this video I answer a question about how to estimate something that you don’t know how to do yet.

The post How Do You Estimate What You Don’t Know? appeared first on Simple Programmer.

Categories: Programming

musiXmatch drives user engagement through innovation

Android Developers Blog - Thu, 11/20/2014 - 15:54

Posted by Leticia Lago, Google Play team

musiXmatch is an app that offers Android users the unique and powerful feature FloatingLyrics. FloatingLyrics pops up a floating window showing synched lyrics as users listen to tracks on their favorite player and music services. It’s achieved through a seamless integration with intents on the platform, something that’s technically possible only on Android.

As a result musiXmatch has seen “a dramatic increase in terms of engagement’, says founder Max Ciociola, “which has been two times more active users and even two times more the average time they spend in the app.”

The ability to deliver lyrics to a range of different devices — such as Chromecast, Android TV, and Android Wear — is creating opportunities for musiXmatch. It’s helping them turn their app into a smart companion for their users and getting them closer to their goal of reaching 100 million people.

In the following video, Max and Android engineer Sebastiano Gottardo talk about the unique capabilities that Android offers to musiXmatch:

To learn about achieving great user engagement and retention and reaching more users through different form factors, be sure to check out these resources:

  • Convert installs to active users — Watch this video to hear from Matteo Vallone, Partner Development Manager for Google Play, about the best practices in engaging and retaining app users using intents, identity, context, and rich notifications as well as delivering a cross-platform user experience.
  • Expanding to new form factors: Tablet, Wear & TV — Watch this panel discussion with Google experts talking about cross-platform opportunities and answering developer questions.
Join the discussion on

+Android Developers
Categories: Programming

Chinese Developers Can Now Offer Paid Applications to Google Play Users in More Than 130 countries

Android Developers Blog - Thu, 11/20/2014 - 03:07

By Ellie Powers, product manager for Google Play

Google Play is the largest digital store for Android users to discover and purchase their favorite mobile app and games, and the ecosystem is continuing to grow globally. Over the past year, we’ve expanded the list of countries where app developers can sign up to be merchants on Google Play, totaling 60 countries, including Lebanon, Jordan, Oman, Pakistan, Puerto Rico, Qatar and Venezuela most recently.

As part of that continued effort, we’re excited to announce merchant support in China, enabling local developers to export and sell their apps to Google Play users in more than 130 countries. Chinese developers can now offer both free and paid applications through various monetization models, including in-app purchasing and subscriptions. For revenue generated on Google Play, developers will receive payment to their Chinese bank accounts via USD wire transfers.

If you develop Android apps in China and want to start distributing your apps to a global audience through Google Play, visit play.google.com/apps/publish and register as a developer. If you want to sell apps and in-app products, you'll need to also sign up for a Google Wallet merchant account, which is available on the “Revenue” page in the Google Play Developer Console. After you’ve uploaded your apps, you can set prices in the Developer Console and later receive reports on your revenue. You’ll receive your developer payouts via wire transfer. For more details, please visit our developer help center.

We look forward to continuing to roll out Google Play support to developers in many more countries around the world.

中国开发者可以向全球130个国家的Google Play用户提供付费应用啦

发表者:Ellie Powers, Google Play产品经理

Google Play是一个可让Android用户发现和购买他们喜爱的移动应用程序和游戏的全球最大的应用商店,这个生态系统在全球迅速成长。过去一年中,我们已经扩展到60个国家,让应用程序开发人员可以注册成为 Google Play的商家,其中新近支持的国家包括黎巴嫩、约旦、阿曼、巴基斯坦、波多黎各、卡塔尔和委内瑞拉。

作为持续改进 Google Play努力的一部分,我们很高兴地宣布在中国增加了对商家的支持,让中国的开发者能售卖应用程序到130个国家的 Google Play 用户。中国的开发者现在可以提供通过各种盈利模式的免费和付费应用,包括应用内购买和订阅。在 Google Play 产生的营收将通过美元电汇的方式支付给开发者的中国的银行账户。

如果你在中国开发Android应用程序,并希望通过 Google Play 把应用程序推广到全球,请登录play.google.com/apps/publish 并建立你的 Google Play 开发者账户。如果你想售卖付费的应用程序和应用程序内的产品,则需要再注册一个Google 电子钱包商家帐户,通过Google Play开发者控制台里的”营收”页面进行设置。上传应用程序后,你可以通过开发者控制台设定价格,之后就可以收到营收报告,你将会通过电汇的方式获得收入。

我们将继续增加更多 Google Play 商家支持的国家,敬请关注。

更多详情,请访问我们的开发者帮助中心

Join the discussion on

+Android Developers
Categories: Programming

Keeping Your Saved Games in the Cloud

Android Developers Blog - Wed, 11/19/2014 - 19:47

Posted by Todd Kerpelman, Developer Advocate

Saved Games Are the Future!

I think most of us have at least one or two games we play obsessively. Me? I'm a Sky Force 2014 guy. But maybe you're into matching colorful objects, battling monsters, or helping avians with their rage management issues. Either way, there's probably some game where you've spent hours upon hours upgrading your squad, reaching higher and higher levels, or unlocking every piece of bonus content in the game.

Now imagine losing all of that progress when you decide to get a new phone. Or reinstall your game. Yikes!

That's why, when Google Play Games launched, one of the very first features we included was the ability for users to save their game to the cloud using a service called the AppState API. For developers who didn't need an entire server-based infrastructure to run their game, but didn't want to lose players when they upgraded their devices, this feature was a real life-saver.

But many developers wanted even more. With AppState, you were limited to 4 slots of 256k of data each, and for some games, this just wasn't enough. So this past year at Google I/O, we launched an entirely new Saved Games feature, powered by Google Drive. This gave you huge amounts of space (up to 3MB per saved game with unlimited save slots), the ability to save a screenshot and metadata with your saved games, and some nice features like showing your player's saved games directly in the Google Play app.

...But AppState is Yesterday's News

Since the introduction of Saved Games, we've seen enough titles happily using the service and heard enough positive feedback from developers that we're convinced that Saved Games is the better offering and the way to go in the future. With that in mind, we've decided to start deprecating the old cloud save system using AppState and are encouraging everybody who's still using it to switch over to the new Saved Games feature (referred to in our libraries as "Snapshots").

What does this mean for you as a game developer?

If you haven't yet added Saved Games to your game, now would be the perfect time! The holidays are coming up and your players are going to start getting new devices over the next couple of months. Wouldn't it be great if they could take your game's progress with them? Unless, I guess, "not retaining users" is part of your business plan.

If you're already using the new Saved Games / Snapshot system, put your feet up and relax. None of these changes affect you. Okay, now put your feet down, and get back to work. You probably have a seasonal update to work on, don't you?

If you're using the old AppState system, you should start migrating your player's data over to the new Saved Games service. Luckily, it's easy to include both systems in the same game, so you should be able to migrate your users' data with their ever knowing. The process would probably work a little something like this:

  • Enable the new Saved Game service for your game by
    • Adding the Drive.SCOPE_APPFOLDER scope to your list of scopes in your GoogleApiClient.
    • Turning on Saved Games for your game in the Google Play Developer Console.
  • Next, when your app tries to load the user's saved game
    • First see if any saved game exists using the new Saved Games service. If there is, go ahead and use it.
    • Otherwise, grab their saved game from the AppState service.
  • When you save the user's game back to the cloud, save it using the new Saved Games service.
  • And that should be it! The next time your user loads up your game, it will find their saved data in the new Saved Games service, and they'll be all set.
  • We've built a sample app that demonstrates how to perform these steps in your application, so we encourage you to check it out.

In a few months, we will be modifying the old AppState service to be read-only. You'll still be able to read your user's old cloud save games and transfer them to the new Saved Games service, but you'll no longer be able to save games using the old service. We are evaluating early Q2 of 2015 to make this change, which should give you enough time to push your "start using Saved Games" update to the world.

If you want to find out more about Saved Games and how they work, feel free to review our documentation, our sample applications, or our Game On! videos. And we look forward to many more hours of gaming, no matter how many times we switch devices.

Join the discussion on

+Android Developers
Categories: Programming

Coding Android TV games is easy as pie

Android Developers Blog - Wed, 11/19/2014 - 19:28
Posted by Alex Ames, Fun Propulsion Labs at Google*

We’re pleased to announce Pie Noon, a simple game created to demonstrate multi-player support on the Nexus Player, an Android TV device. Pie Noon is an open source, cross-platform game written in C++ which supports:

  • Up to 4 players using Bluetooth controllers.
  • Touch controls.
  • Google Play Games Services sign-in and leaderboards.
  • Other Android devices (you can play on your phone or tablet in single-player mode, or against human adversaries using Bluetooth controllers).

Pie Noon serves as a demonstration of how to use the SDL library in Android games as well as Google technologies like Flatbuffers, Mathfu, fplutil, and WebP.

  • Flatbuffers provides efficient serialization of the data loaded at run time for quick loading times. (Examples: schema files and loading compiled Flatbuffers)
  • Mathfu drives the rendering code, particle effects, scene layout, and more, allowing for efficient mathematical operations optimized with SIMD. (Example: particle system)
  • fplutil streamlines the build process for Android, making iteration faster and easier. Our Android build script makes use of it to easily compile and run on on Android devices.
  • WebP compresses image assets more efficiently than jpg or png file formats, allowing for smaller APK sizes.

You can download the game in the Play Store and the latest open source release from our GitHub page. We invite you to learn from the code to see how you can implement these libraries and utilities in your own Android games. Take advantage of our discussion list if you have any questions, and don’t forget to throw a few pies while you’re at it!

* Fun Propulsion Labs is a team within Google that's dedicated to advancing gaming on Android and other platforms.

Join the discussion on

+Android Developers
Categories: Programming

Launchpad Online for developers: Getting Started with Google APIs

Google Code Blog - Wed, 11/19/2014 - 19:00
Google APIs give you the power to build rich integrations with our most popular applications, including Google Maps, YouTube, Gmail, Google Drive and more. If you are new to our developer features, we want to give you a quick jumpstart on how to use them effectively and build amazing things.

As you saw last week, we’re excited to partner with the Startup Launch team to create the “Launchpad Online”, a new video series for a global audience that joins their cadre which already includes the established “Root Access” and “How I:” series, as well as a “Global Spotlight” series on some of the startups that have benefitted from the Startup Launch program. This new series focuses on developers who are beginners to one or more Google APIs. Each episode helps get new users off the ground with the basics, shows experienced developers more advanced techniques, or introduces new API features, all in as few lines of code as possible.

The series assumes you have some experience developing software and are comfortable using languages like Javascript and Python. Be inspired by a short working example that you can drop into your own app and customize as desired, letting you “take off” with that great idea. Hopefully you had a chance to view our intro video last week and are ready for more!

Once you’ve got your developers ready to get started, learn how to use the Google Developers Console to set up a project (see video to the right), and walk through common security code required for your app to access Google APIs. When you’ve got this “foundation” under your belt, you’ll be ready to build your first app… listing your files on Google Drive. You’ll learn something new and see some code that makes it happen in each future episode… we look forward to having you join us soon on Launchpad Online!

Posted by Wesley Chun, Developer Advocate, Google

+Wesley Chun (@wescpy) is an engineer, author of the bestselling Core Python books, and a Developer Advocate at Google, specializing in Google Drive, Google Apps Script, Gmail, Google Apps, cloud computing, developer training, and academia. He loves traveling worldwide to meet Google users everywhere, whether at a developers conference, user group meeting, or on a university campus!
Categories: Programming

Begin developing with Android Auto

Android Developers Blog - Tue, 11/18/2014 - 19:09

Posted by Daniel Holle, Product Manager

At Google I/O back in June, we provided a preview of Android Auto. Today, we’re excited to announce the availability of our first APIs for building Auto-enabled apps for audio and messaging. Android apps can now be extended to the car in a way that is optimized for the driving experience.

For users, this means they simply connect their Android handheld to a compatible vehicle and begin utilizing a car-optimized Android experience that works with the car’s head unit display, steering wheel buttons, and more. For developers, the APIs and UX guidelines make it easy to provide a simple way for users to get the information they need while on the road. As an added bonus, the Android Auto APIs let developers easily extend their existing apps targeting Android 5.0 (API level 21) or higher to work in the car without having to worry about vehicle-specific hardware differences. This gives developers wide reach across manufacturers, model and regions, by just developing with one set of APIs and UX standards.

There are two use cases that Android Auto supports today:

  • Audio apps that expose content for users to browse and allow audio playback from the car, such as music, podcasts, news, etc.
  • Messaging apps that receive incoming notifications, read messages aloud, and send replies via voice from the car.

To help you get started with Android Auto, check out our Getting Started guide. It’s important to note that while the APIs are available today, apps extended with Android Auto cannot be published quite yet. More app categories will be supported in the future, providing more opportunities for developers and drivers of Android Auto. We encourage you to join the Android Auto Developers Google+ community to stay up-to-date on the latest news and timelines.

We’ve already started working with partners to develop experiences for Android Auto: iHeartRadio, Joyride, Kik, MLB.com, NPR, Pandora, PocketCasts, Songza, SoundCloud, Spotify, Stitcher, TextMe, textPlus, TuneIn, Umano, and WhatsApp. If you happen to be in the Los Angeles area, stop by the LA Auto Show through November 30 and visit us in the Hyundai booth to take Android Auto and an app or two for a test drive.

Join the discussion on

+Android Developers
Categories: Programming

You’re invited to submit an app to the Google Fit Developer Challenge

Google Code Blog - Tue, 11/18/2014 - 18:00
With the recent launch of the Google Fit platform, we hope to spur innovation in the world of fitness apps and devices that encourage users to have a more active lifestyle. To expand the ecosystem and help users get a more comprehensive view of their fitness, we’re excited to announce the Google Fit Developer Challenge in partnership with adidas, Polar, and Withings.
We’re inviting you to create and showcase an app, or update an existing app, that integrates with Google Fit. The submission deadline is February 17, 2015. Our selected judges will choose up to six new apps and six existing apps for Fit, as the winners. The judges are looking for fitness apps that are innovative, fun to use, keep users coming back, offer users real benefit, and meet the Android design and quality guidelines.

Challenge winners will be featured on Google Play in front of Android users around the world. Runners-ups will also receive prizes, in the form of smart devices from Google, adidas, Polar, and Withings, so that they can continue to tinker and improve their fitness apps with the latest hardware.

Check out the challenge website to find out how to enter*. You will also find some helpful resources to find inspiration. We encourage you to join the Google Fit Developer Community on Google+ to discuss your ideas and the challenge.

* The challenge is open to registered Google Play developers in the US, UK, France, Germany, India, Israel, Hong Kong, Singapore, South Korea, Japan, and Taiwan who are over the age of 18 (or the age of majority in their country). For the full terms and conditions, read the official rules.

Posted by Angana Ghosh, Product Manager, Google Fit
Categories: Programming

Ready, Test, Go!

Xebia Blog - Tue, 11/18/2014 - 10:42

The full potential of many an agile organization is hardly ever reached. Many teams find themselves redefining user stories although they have been committed to as part of the sprint. The ‘ready phase’, meant to get user stories clear and sufficiently detailed so they can be implemented, is missed. How will each user story result in high quality features that deliver business value? The ‘Definition of Ready’ is lacking one important entry: “Automated tests are available.” Ensuring to have testable and hence automated acceptance criteria before committing to user stories in a sprint, allows you to retain focus during the sprint. We define this as: Ready, Test, Go!

Ready

Behaviour-Driven Development has proven to be a fine technique to write automated acceptance criteria. Using the Gherkin format (given, when, then), examples can be specified that need to be supported by the system once the user story is completed. When a sufficiently detailed list of examples is available, all Scrum stakeholders agree with the specification. Common understanding is achieved that when the story is implemented, we are one step closer to building the right thing.

Test

The specification itself becomes executable: at any moment in time, the gap between the desired and implemented functionality becomes visible. In other words, this automated acceptance test should be run continuously. First time, it happily fails. Next, implementation can start. This, following Test-Driven Development principles, starts with writing (also failing) unit tests. Then, development of the production code starts. When the unit tests are passing and acceptance tests for a story are passing, other user stories can be picked up; stories of which the tests happily fail. Tests thus act as a safeguard to continuously verify that the team is building the thing right. Later, the automated tests (acceptance tests and unit tests) serve as a safety net for regression testing during subsequent sprints.

Go!

That's simple: release your software to production. Ensure that other testing activities (performance tests, chain tests, etc) are as much as possible automated and performed as part of the sprint.

The (Agile) Test Automation Engineer

In order to facilitate or boost this way of working, the role of the test automation engineer is key. The test automation engineer is defining the test architecture and facilitating the necessary infrastructure needed to run tests often and fast. He is interacting with developers to co-develop fixtures, to understand how the production code is built, and to decide upon the structure and granularity of the test code.

Apart from their valuable and unique analytical skills, relevant testers grow their technical skills. If it cannot be automated, it cannot be checked, so one might question whether the user story is ready to be committed to in a sprint. The test automation engineer helps the Scrum teams to identify when they are ‘ready to test’ and urges the product owner and business to specify requirements – at least for the next sprint ahead.

So: ready, test, go!!

Seq vs Streams

Phil Trelford's Array - Tue, 11/18/2014 - 08:22

Last week Gian Ntzik gave a great talk at the F#unctional Londoners meetup on the Nessos Streams library. It’s a lightweight F#/C# library for efficient functional-style pipelines on streams of data.

The main difference between LINQ/Seq and Streams is that LINQ is about composing external iterators (Enumerable/Enumerator) and Streams is based on the continuation-passing-style composition of internal iterators, which makes optimisations such as loop fusion easier.

The slides (using FsReveal) and samples are available on Gian’s github repository.

Simple Streams

Gian started the session by live coding a simple implementation of streams in about 20 minutes:

type Stream<'T> = ('T -> unit) -> unit

let inline ofArray (source: 'T[]) : Stream<'T> =
   fun k ->
      let mutable i = 0
      while i < source.Length do
            k source.[i]
            i <- i + 1          

let inline filter (predicate: 'T -> bool) (stream: Stream<'T>) : Stream<'T> =
   fun k -> stream (fun value -> if predicate value then k value)

let inline map (mapF: 'T -> 'U) (stream: Stream<'T>) : Stream<'U> =
   fun k -> stream (fun v -> k (mapF v))

let inline iter (iterF: 'T -> unit) (stream: Stream<'T>) : unit =
   stream (fun v -> iterF v)

let inline toArray (stream: Stream<'T>) : 'T [] =
   let acc = new List<'T>()
   stream |> iter (fun v -> acc.Add(v))
   acc.ToArray()

let inline fold (foldF:'State->'T->'State) (state:'State) (stream:Stream<'T>) =
   let acc = ref state
   stream (fun v -> acc := foldF !acc v)
   !acc

let inline reduce (reducer: ^T -> ^T -> ^T) (stream: Stream< ^T >) : ^T
      when ^T : (static member Zero : ^T) =
   fold (fun s v -> reducer s v) LanguagePrimitives.GenericZero stream

let inline sum (stream : Stream< ^T>) : ^T
      when ^T : (static member Zero : ^T)
      and ^T : (static member (+) : ^T * ^T -> ^T) =
   fold (+) LanguagePrimitives.GenericZero stream

and as you can see only about 40 lines of code.

Sequential Performance

Just with this simple implementation, Gian was able to demonstrate a significant performance improvement over F#’s built-in Seq module for a simple pipeline:

#time // Turns on timing in F# Interactive

let data = [|1L..1000000L|]

let seqValue = 
   data
   |> Seq.filter (fun x -> x%2L = 0L)
   |> Seq.map (fun x -> x * x)
   |> Seq.sum
// Real: 00:00:00.252, CPU: 00:00:00.234, GC gen0: 0, gen1: 0, gen2: 0

let streamValue =
   data
   |> Stream.ofArray
   |> Stream.filter (fun x -> x%2L = 0L)
   |> Stream.map (fun x -> x * x)
   |> Stream.sum
// Real: 00:00:00.119, CPU: 00:00:00.125, GC gen0: 0, gen1: 0, gen2: 0

Note for operations over arrays, the F# Array module would be more appropriate choice and is slightly faster:

let arrayValue =
   data
   |> Array.filter (fun x -> x%2L = 0L)
   |> Array.map (fun x -> x * x)
   |> Array.sum
// Real: 00:00:00.094, CPU: 00:00:00.093, GC gen0: 0, gen1: 0, gen2: 0

Also LINQ does quite well here as it has a specialized overloads including one for summing over int64 values:

open System.Linq

let linqValue =   
   data
      .Where(fun x -> x%2L = 0L)
      .Select(fun x -> x * x)
      .Sum()
// Real: 00:00:00.058, CPU: 00:00:00.062, GC gen0: 0, gen1: 0, gen2: 0

However with F# Interactive running in 64-bit mode Streams take back the advantage (thanks to Nick Palladinos for the tip):

let streamValue =
   data
   |> Stream.ofArray
   |> Stream.filter (fun x -> x%2L = 0L)
   |> Stream.map (fun x -> x * x)
   |> Stream.sum
// Real: 00:00:00.033, CPU: 00:00:00.031, GC gen0: 0, gen1: 0, gen2: 0

Looks like the 64-bit JIT is doing some black magic there.

Parallel Performance

Switching to the full Nessos Streams library, there’s support for parallel streams via the ParStream module:

let parsStreamValue =
   data
   |> ParStream.ofArray
   |> ParStream.filter (fun x -> x%2L = 0L)
   |> ParStream.map (fun x -> x + 1L)
   |> ParStream.sum
// Real: 00:00:00.069, CPU: 00:00:00.187, GC gen0: 0, gen1: 0, gen2: 0

which demonstrates a good performance increase with little effort.

For larger computes Nessos Streams supports cloud based parallel operations against Azure.

Overall Nessos Streams looks like a good alternative to the Seq module for functional pipelines.

Nessos LinqOptimzer

For further optimization Gian recommended the Nessos LinqOptimizer:

An automatic query optimizer-compiler for Sequential and Parallel LINQ. LinqOptimizer compiles declarative LINQ queries into fast loop-based imperative code. The compiled code has fewer virtual calls and heap allocations, better data locality and speedups of up to 15x

The benchmarks are impressive:

cs-ssq

Reactive Extensions (Rx)

One of the questions in the talk and on twitter later was, given Rx is also a push model, how does the performance compare:

@sforkmann @ptrelford @dsyme how efficient memory and perf wise to @ReactiveX? Our numbers are pretty hard to beat!

— Матвій Підвисоцький (@mattpodwysocki) November 13, 2014

Clearly the Nessos Streams library and Rx have different goals (data processing vs event processing), but I thought it would be interesting to compare them all the same:

open System.Reactive.Linq

let rxValue =
   data
      .ToObservable()
      .Where(fun x -> x%2L = 0L)
      .Select(fun x -> x * x)
      .Sum()      
      .ToEnumerable()
      |> Seq.head
// Real: 00:00:02.895, CPU: 00:00:02.843, GC gen0: 120, gen1: 0, gen2: 0

let streamValue =
   data
   |> Stream.ofArray
   |> Stream.filter (fun x -> x%2L = 0L)
   |> Stream.map (fun x -> x * x)
   |> Stream.sum
// Real: 00:00:00.130, CPU: 00:00:00.109, GC gen0: 0, gen1: 0, gen2: 0

In this naive comparison you can see Nessos Streams is roughly 20 times faster than Rx.

Observable module

F# also has a built-in Observable module for operations over IObservable<T> (support for operations over events was added to F# back in 2006). Based on the claims on Rx performance made by Matt Podwysocki I was curious to see how it stacked up:

let obsValue =
   data
   |> Observable.ofSeq
   |> Observable.filter (fun x -> x%2L = 0L)
   |> Observable.map (fun x -> x * x)
   |> Observable.sum
   |> Observable.first
// Real: 00:00:00.479, CPU: 00:00:00.468, GC gen0: 18, gen1: 0, gen2: 0

As you can see Observable module comes off roughly 5 times faster.

Note: I had to add some simple combinators to make this work, you can see the full snippet here: http://fssnip.net/ow

Summary

Nessos Streams look like a promising direction for performance of functional pipelines, and for gaining raw imperative performance the Nessos LINQOptimizer is impressive.

Categories: Programming

Google Play services 6.5

Android Developers Blog - Tue, 11/18/2014 - 06:46
Posted by Ian Lake, Developer Advocate

To offer more seamless integration of Google products within your app, we’re excited to start the rollout of the latest version of Google Play services.

Google Play services 6.5 includes new features in Google Maps, Google Drive and Google Wallet as well as the recently launched Google Fit API. We are also providing developers with more granular control over which Google Play services APIs your app depends on to help you maintain a lean app.

Google Maps

We’re making it easier to get directions to places from your app! The Google Maps Android API now offers a map toolbar to let users open Google Maps and immediately get directions and turn by turn navigation to the selected marker. This map toolbar will show by default when you compile against Google Play services 6.5.

In addition, there is also a new ‘lite mode’ map option, ideal for situations where you want to provide a number of smaller maps, or a map that is so small that meaningful interaction is impractical, such as a thumbnail in a list. A lite mode map is a bitmap image of a map at a specified location and zoom level.

In lite mode, markers and shapes are drawn client-side on top of the static image, so you still have full control over them. Lite mode supports all of the map types, the My Location layer, and a subset of the functionality of a fully-interactive map. Users can tap on the map to launch Google Maps when they need more details.

The Google Maps Android API also exposes a new getMapAsync(OnMapReadyCallback) method to MapFragment and MapView which will notify you exactly when the map is ready. This serves as a replacement for the now deprecated getMap() method.

We’re also exposing the Google Maps for Android app intents available to your apps including displaying the map, searching, starting turn by turn navigation, and opening Street View so you can build upon the familiar and powerful maps already available on the device.

Drive

You can now add both public and application private custom file properties to a Drive file which can be used to build very efficient search queries and allow apps to save information which is guaranteed to persist across editing by other apps.

We’ve also made it even easier to make syncing your files to Drive both user and battery friendly with the ability to control when files are uploaded by network type or charging status and cancel pending uploads.

Google Wallet

In addition to the existing ‘Buy with Google’ button available to quickly purchase goods & services using Google Wallet, this release adds a ‘Donate with Google’ button for providing the same ease of use in collecting donations.

Google Fit

The Google Fit SDK was recently officially released as part of Google Play services and can be used to super-charge your fitness apps with a simple API for working with sensors, recording activity data, and reading the user’s aggregated fitness data.

In this release, we’ve made it easier for developers to add activity segments (predefined time periods of running, walking, cycling, etc) when inserting sessions, making it easy to support pauses or multiple activity type workouts. We’ll also be adding additional samples to help kick-start your Google Fit integration.

Granular Dependency Management

As we’ve continued to add more APIs across the wide range of Google services, it can be hard to maintain a lean app, particularly if you're only using a portion of the available APIs. Now with Google Play services 6.5, you’ll be able to depend only on a minimal common library and the exact APIs your app needs. This makes it very lightweight to get started with Google Play services.

SDK Coming Soon!

We’ll be rolling out Google Play services 6.5 over the next few days, and we’ll update this blog post, publish the documentation, and make the SDK available once the rollout completes.

To learn more about Google Play services and the APIs available to you through it, visit the Google Play Services section on the Android Developer site.

Join the discussion on

+Android Developers
Categories: Programming

How To Build a Roadmap for Your Digital Business Transformation

Let’s say you want to take your business to the Cloud --  How do you do it?

If you’re a small shop or a startup, it might be easy to just swipe your credit card and get going.

If, on the other hand, you’re a larger business that wants to start your journey to the Cloud, with a lot of investments and people that you need to bring along, you need a roadmap.

The roadmap will help you deal with setbacks, create confidence in the path, and help ensure that you can get from point A to point B (and that you know what point B actually is.)  By building an implementable roadmap for your business transformation, you can also build a coalition of the willing to help you get their faster.  And you can design your roadmap so that your journey flows continuous business value along the way.

In the book, Leading Digital: Turning Technology into Business Transformation, George Westerman, Didier Bonnet, and Andrew McAfee, share how top leaders build better roadmaps for their digital business transformation.

Why You Need to Build a Roadmap for Your Digital Transformation

If you had infinite time and resources, maybe you could just wing it, and hope for the best.   A better approach is to have a roadmap as a baseline.  Even if your roadmap changes, at least you can share the path with others in your organization and get them on board to help make it happen.

Via Leading Digital:

“In a perfect world, your digital transformation would deliver an unmatched customer experience, enjoy the industry's most effective operations, and spawn innovative, new business models.  There are a myriad of opportunities for digital technology to improve your business and no company can entertain them all at once.  The reality of limited resources, limited attention spans, and limited capacity for change with force focused choices.  This is the aim of your roadmap.”

Find Your Entry Point

Your best starting point is a business capability that you want to exploit.

Via Leading Digital:

“Many companies have come to realize that before they can create a wholesale change within their organization, they have to find an entry point that will begin shifting the needle.  How? They start by building a roadmap that leverages existing assets and capabilities.  Burberry, for example, enjoyed a globally recognized brand and a fleet of flagship retail locations around the world.  The company started by revitalizing its brand and customer experience in stores and online.  Others, like Codelco, began with the core operational processes of their business.  Caesars Entertainment combined strong capabilities in analytics with a culture of customer service to deliver a highly personalized guest experience.  There is no single right way to start your digital transformation.  What matters is that you find the existing capability--your sweet spot--that will get your company off the starting blocks.

Once your initial focus is clear, you can start designing your transformation roadmap.  Which investments and activities are necessary to close the gap to your vision?  What is predictable, and what isn't? What is the timing and scheduling of each initiative? What are the dependencies between them?  What organizational resources, such as analytics skills, are required?”

Engage Practitioners Early in the Design

If you involve others in your roadmap, you get their buy-in, and they will help you with your business transformation.

Via Leading Digital:

“Designing your roadmap will require input from a broad set of stakeholders.  Rather than limit the discussion to the top team, engage the operational specialists who bring an on-the-ground perspective.  This will minimize the traditional vision-to-execution gap.  You can crowd-source the design.  Or, you can use facilitated workshops, as as 'digital days,' as an effective way to capture and distill the priorities and information you will need to consider.  We've seen several Digital Masters do both.

Make no mistake; designing your roadmap will take time, effort, and multiple iterations.  But you will find it a valuable exercise.  it forces agreement on priorities and helps align senior management and the people tasked to execute the program.  Your roadmap will become more than just a document.  If executed well, it can be the canvas of the transformation itself.  Because your roadmap is a living document, it will evolve as your implementation progresses.”

Design for Business Outcome, Not Technology

When you create your roadmap, focus on the business outcomes.   Think in terms of adding incremental business capabilities.   Don’t make it a big bang thing.   Instead, start small, but iterate on building business capabilities that take advantage of Cloud, Mobile, Social, and Big Data technologies.

Via Leading Digital:

“Technology for its own sake is a common trap.  Don't build your roadmap as a series of technology projects.  Technology is only part of the story in digital transformation and often the least challenging one.  For example, the major hurdles for Enterprise 2.0 platforms are not technical.  Deploying the platform is relatively straightforward, and today's solutions are mature.  The challenge lies in changing user behavior--encouraging adoption and sustaining engagement in the activities the platform is meant to enable.

Express your transformation roadmap in terms of business outcomes.  For example, 'Establish a 360-degree understanding of our customers.'  Build into your roadmap the many facets of organizational change that your transformation will require customer experiences, operational processes, employee ways of working, organization, culture, communication--the list goes on.  This is why contributions from a wide variety is so critical.”

There are lots of way to build a roadmap, but the best thing you can do is put something down on paper so that you can share the path with other people and start getting feedback and buy-in.

You’ll be surprised but when you show business and IT leaders a roadmap, it helps turn strategy into execution and make things real in people’s minds.

You Might Also Like

10 High-Value Activities in the Enterprise

Cloud Changes the Game from Deployment to Adoption

Drive Business Transformation by Reenvisioning Operations

Drive Business Transformation by Reenvisioning Your Customer Experience

Dual-Speed IT Drives Business Transformation and Improves IT-Business Relationships

How To Build a Better Business Case for Digital Initiatives

How To Improve the IT-Business Relationship

How Leaders are Building Digital Skills

Management Innovation is at the Top of the Innovation Stack

Categories: Architecture, Programming

Top Developer Gifts (And Tech Geek Gifts)

Making the Complex Simple - John Sonmez - Mon, 11/17/2014 - 16:00

With the holidays around the corner, I thought it would be a good time to do another round-up of what I think are some of the best gifts for software developers, programmers and other technology geeks for this year. As you know, it can be very difficult to buy gifts for software developers, because, well, […]

The post Top Developer Gifts (And Tech Geek Gifts) appeared first on Simple Programmer.

Categories: Programming

R: ggmap – Overlay shapefile with filled polygon of regions

Mark Needham - Mon, 11/17/2014 - 01:53

I’ve been playing around with plotting maps in R over the last week and got to the point where I wanted to have a google map in the background with a filled polygon on a shapefile in the foreground.

The first bit is reasonably simple – we can just import the ggmap library and make a call to get_map:

> library(ggmap)
> sfMap = map = get_map(location = 'San Francisco', zoom = 12)
2014 11 17 00 27 11

Next I wanted to show the outlines of the different San Francisco zip codes and came across a blog post by Paul Bidanset on Baltimore neighbourhoods which I was able to adapt.

I downloaded a shapefile of San Francisco’s zip codes from the DataSF website and then loaded it into R using the readOGR and spTransform functions from the rgdal package:

> library(rgdal)
> library(ggplot2)
> sfn = readOGR(".","sfzipcodes") %>% spTransform(CRS("+proj=longlat +datum=WGS84"))
> ggplot(data = sfn, aes(x = long, y = lat, group = group)) + geom_path()
2014 11 17 00 38 32

sfn is a spatial type of data frame…

> class(sfn)
[1] "SpatialPolygonsDataFrame"
attr(,"package")
[1] "sp"

…but we need a normal data frame to be able to easily merge other data onto the map and then plot it. We can use ggplot2’s fortify command to do this:

> names(sfn)
[1] "OBJECTID" "ZIP_CODE" "ID"   
 
> sfn.f = sfn %>% fortify(region = 'ZIP_CODE')
 
SFNeighbourhoods  = merge(sfn.f, sfn@data, by.x = 'id', by.y = 'ZIP_CODE')

I then made up some fake values for each zip code so that we could have different colour shadings for each zip code on the visualisation:

> library(dplyr) 
 
> postcodes = SFNeighbourhoods %>% select(id) %>% distinct()
 
> values = data.frame(id = c(postcodes),
                      value = c(runif(postcodes %>% count() %>% unlist(),5.0, 25.0)))

I then merged those values onto SFNeighbourhoods:

> sf = merge(SFNeighbourhoods, values, by.x='id')
 
> sf %>% group_by(id) %>% do(head(., 1)) %>% head(10)
Source: local data frame [10 x 10]
Groups: id
 
      id      long      lat order  hole piece   group OBJECTID    ID     value
1  94102 -122.4193 37.77515     1 FALSE     1 94102.1       14 94102  6.184814
2  94103 -122.4039 37.77006   106 FALSE     1 94103.1       12 94103 21.659752
3  94104 -122.4001 37.79030   255 FALSE     1 94104.1       10 94104  5.173199
4  94105 -122.3925 37.79377   293 FALSE     1 94105.1        2 94105 15.723456
5  94107 -122.4012 37.78202   504 FALSE     1 94107.1        1 94107  8.402726
6  94108 -122.4042 37.79169  2232 FALSE     1 94108.1       11 94108  8.632652
7  94109 -122.4139 37.79046  2304 FALSE     1 94109.1        8 94109 20.129402
8  94110 -122.4217 37.73181  2794 FALSE     1 94110.1       16 94110 12.410610
9  94111 -122.4001 37.79369  3067 FALSE     1 94111.1        9 94111 10.185054
10 94112 -122.4278 37.73469  3334 FALSE     1 94112.1       18 94112 24.297588

Now we can easily plot those colours onto our shapefile by calling geom_polgon instead of geom_path:

> ggplot(sf, aes(long, lat, group = group)) + 
    geom_polygon(aes(fill = value))

2014 11 17 00 49 11

And finally let’s wire it up to our google map:

> ggmap(sfMap) + 
    geom_polygon(aes(fill = value, x = long, y = lat, group = group), 
                 data = sf,
                 alpha = 0.8, 
                 color = "black",
                 size = 0.2)
2014 11 17 00 50 13

I spent way too long with the alpha value set to ‘0’ on this last plot wondering why I wasn’t seeing any shading so don’t make that mistake!

Categories: Programming

Spark: Parse CSV file and group by column value

Mark Needham - Sun, 11/16/2014 - 23:53

I’ve found myself working with large CSV files quite frequently and realising that my existing toolset didn’t let me explore them quickly I thought I’d spend a bit of time looking at Spark to see if it could help.

I’m working with a crime data set released by the City of Chicago: it’s 1GB in size and contains details of 4 million crimes:

$ ls -alh ~/Downloads/Crimes_-_2001_to_present.csv
-rw-r--r--@ 1 markneedham  staff   1.0G 16 Nov 12:14 /Users/markneedham/Downloads/Crimes_-_2001_to_present.csv
 
$ wc -l ~/Downloads/Crimes_-_2001_to_present.csv
 4193441 /Users/markneedham/Downloads/Crimes_-_2001_to_present.csv

We can get a rough idea of the contents of the file by looking at the first row along with the header:

$ head -n 2 ~/Downloads/Crimes_-_2001_to_present.csv
ID,Case Number,Date,Block,IUCR,Primary Type,Description,Location Description,Arrest,Domestic,Beat,District,Ward,Community Area,FBI Code,X Coordinate,Y Coordinate,Year,Updated On,Latitude,Longitude,Location
9464711,HX114160,01/14/2014 05:00:00 AM,028XX E 80TH ST,0560,ASSAULT,SIMPLE,APARTMENT,false,true,0422,004,7,46,08A,1196652,1852516,2014,01/20/2014 12:40:05 AM,41.75017626412204,-87.55494559131228,"(41.75017626412204, -87.55494559131228)"

I wanted to do a count of the ‘Primary Type’ column to see how many of each crime we have. Using just Unix command line tools this is how we’d do that:

$ time tail +2 ~/Downloads/Crimes_-_2001_to_present.csv | cut -d, -f6  | sort | uniq -c | sort -rn
859197 THEFT
757530 BATTERY
489528 NARCOTICS
488209 CRIMINAL DAMAGE
257310 BURGLARY
253964 OTHER OFFENSE
247386 ASSAULT
197404 MOTOR VEHICLE THEFT
157706 ROBBERY
137538 DECEPTIVE PRACTICE
124974 CRIMINAL TRESPASS
47245 PROSTITUTION
40361 WEAPONS VIOLATION
31585 PUBLIC PEACE VIOLATION
26524 OFFENSE INVOLVING CHILDREN
14788 CRIM SEXUAL ASSAULT
14283 SEX OFFENSE
10632 GAMBLING
8847 LIQUOR LAW VIOLATION
6443 ARSON
5178 INTERFERE WITH PUBLIC OFFICER
4846 HOMICIDE
3585 KIDNAPPING
3147 INTERFERENCE WITH PUBLIC OFFICER
2471 INTIMIDATION
1985 STALKING
 355 OFFENSES INVOLVING CHILDREN
 219 OBSCENITY
  86 PUBLIC INDECENCY
  80 OTHER NARCOTIC VIOLATION
  12 RITUALISM
  12 NON-CRIMINAL
   6 OTHER OFFENSE
   2 NON-CRIMINAL (SUBJECT SPECIFIED)
   2 NON - CRIMINAL
 
real	2m37.495s
user	3m0.337s
sys	0m1.471s

This isn’t too bad but it seems like the type of calculation that Spark is made for so I had a look at how I could go about doing that. To start with I created an SBT project with the following build file:

name := "playground"
 
version := "1.0"
 
scalaVersion := "2.10.4"
 
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0"
 
libraryDependencies += "net.sf.opencsv" % "opencsv" % "2.3"
 
ideaExcludeFolders += ".idea"
 
ideaExcludeFolders += ".idea_modules"

I downloaded Spark and after unpacking it launched the Spark shell:

$ pwd
/Users/markneedham/projects/spark-play/spark-1.1.0/spark-1.1.0-bin-hadoop1
 
$ ./bin/spark-shell
...
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 1.1.0
      /_/
 
Using Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
...
Spark context available as sc.
 
scala>

I first import some classes I’m going to need:

scala> import au.com.bytecode.opencsv.CSVParser
import au.com.bytecode.opencsv.CSVParser
 
scala> import org.apache.spark.rdd.RDD
import org.apache.spark.rdd.RDD

Now, following the quick start example, we’ll create a Resilient Distributed Dataset (RDD) from our Crime CSV file:

scala> val crimeFile = "/Users/markneedham/Downloads/Crimes_-_2001_to_present.csv"
crimeFile: String = /Users/markneedham/Downloads/Crimes_-_2001_to_present.csv
 
scala> val crimeData = sc.textFile(crimeFile).cache()
14/11/16 22:31:16 INFO MemoryStore: ensureFreeSpace(32768) called with curMem=0, maxMem=278302556
14/11/16 22:31:16 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 32.0 KB, free 265.4 MB)
crimeData: org.apache.spark.rdd.RDD[String] = /Users/markneedham/Downloads/Crimes_-_2001_to_present.csv MappedRDD[1] at textFile at <console>:17

Our next step is to process each line of the file using our CSV Parser. A simple way to do this would be to create a new CSVParser for each line:

scala> crimeData.map(line => {
         val parser = new CSVParser(',')
         parser.parseLine(line).mkString(",")
       }).take(5).foreach(println)
14/11/16 22:35:49 INFO SparkContext: Starting job: take at <console>:23
...
4/11/16 22:35:49 INFO SparkContext: Job finished: take at <console>:23, took 0.013904 s
ID,Case Number,Date,Block,IUCR,Primary Type,Description,Location Description,Arrest,Domestic,Beat,District,Ward,Community Area,FBI Code,X Coordinate,Y Coordinate,Year,Updated On,Latitude,Longitude,Location
9464711,HX114160,01/14/2014 05:00:00 AM,028XX E 80TH ST,0560,ASSAULT,SIMPLE,APARTMENT,false,true,0422,004,7,46,08A,1196652,1852516,2014,01/20/2014 12:40:05 AM,41.75017626412204,-87.55494559131228,(41.75017626412204, -87.55494559131228)
9460704,HX113741,01/14/2014 04:55:00 AM,091XX S JEFFERY AVE,031A,ROBBERY,ARMED: HANDGUN,SIDEWALK,false,false,0413,004,8,48,03,1191060,1844959,2014,01/18/2014 12:39:56 AM,41.729576153145636,-87.57568059471686,(41.729576153145636, -87.57568059471686)
9460339,HX113740,01/14/2014 04:44:00 AM,040XX W MAYPOLE AVE,1310,CRIMINAL DAMAGE,TO PROPERTY,RESIDENCE,false,true,1114,011,28,26,14,1149075,1901099,2014,01/16/2014 12:40:00 AM,41.884543798701515,-87.72803579358926,(41.884543798701515, -87.72803579358926)
9461467,HX114463,01/14/2014 04:43:00 AM,059XX S CICERO AVE,0820,THEFT,$500 AND UNDER,PARKING LOT/GARAGE(NON.RESID.),false,false,0813,008,13,64,06,1145661,1865031,2014,01/16/2014 12:40:00 AM,41.785633535413176,-87.74148516669783,(41.785633535413176, -87.74148516669783)

That works but it’s a bit wasteful to create a new CSVParser each time so instead let’s just create one for each partition that Spark splits our file up into:

scala> crimeData.mapPartitions(lines => {
         val parser = new CSVParser(',')
         lines.map(line => {
           parser.parseLine(line).mkString(",")
         })
       }).take(5).foreach(println)
14/11/16 22:38:44 INFO SparkContext: Starting job: take at <console>:25
...
14/11/16 22:38:44 INFO SparkContext: Job finished: take at <console>:25, took 0.015216 s
ID,Case Number,Date,Block,IUCR,Primary Type,Description,Location Description,Arrest,Domestic,Beat,District,Ward,Community Area,FBI Code,X Coordinate,Y Coordinate,Year,Updated On,Latitude,Longitude,Location
9464711,HX114160,01/14/2014 05:00:00 AM,028XX E 80TH ST,0560,ASSAULT,SIMPLE,APARTMENT,false,true,0422,004,7,46,08A,1196652,1852516,2014,01/20/2014 12:40:05 AM,41.75017626412204,-87.55494559131228,(41.75017626412204, -87.55494559131228)
9460704,HX113741,01/14/2014 04:55:00 AM,091XX S JEFFERY AVE,031A,ROBBERY,ARMED: HANDGUN,SIDEWALK,false,false,0413,004,8,48,03,1191060,1844959,2014,01/18/2014 12:39:56 AM,41.729576153145636,-87.57568059471686,(41.729576153145636, -87.57568059471686)
9460339,HX113740,01/14/2014 04:44:00 AM,040XX W MAYPOLE AVE,1310,CRIMINAL DAMAGE,TO PROPERTY,RESIDENCE,false,true,1114,011,28,26,14,1149075,1901099,2014,01/16/2014 12:40:00 AM,41.884543798701515,-87.72803579358926,(41.884543798701515, -87.72803579358926)
9461467,HX114463,01/14/2014 04:43:00 AM,059XX S CICERO AVE,0820,THEFT,$500 AND UNDER,PARKING LOT/GARAGE(NON.RESID.),false,false,0813,008,13,64,06,1145661,1865031,2014,01/16/2014 12:40:00 AM,41.785633535413176,-87.74148516669783,(41.785633535413176, -87.74148516669783)

You’ll notice that we’ve still got the header being printed which isn’t ideal – let’s get rid of it!

I expected there to be a ‘drop’ function which would allow me to do that but in fact there isn’t. Instead we can make use of our knowledge that the first partition will contain the first line and strip it out that way:

scala> def dropHeader(data: RDD[String]): RDD[String] = {
         data.mapPartitionsWithIndex((idx, lines) => {
           if (idx == 0) {
             lines.drop(1)
           }
           lines
         })
       }
dropHeader: (data: org.apache.spark.rdd.RDD[String])org.apache.spark.rdd.RDD[String]

Now let’s grab the first 5 lines again and print them out:

scala> val withoutHeader: RDD[String] = dropHeader(crimeData)
withoutHeader: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[7] at mapPartitionsWithIndex at <console>:14
 
scala> withoutHeader.mapPartitions(lines => {
         val parser = new CSVParser(',')
         lines.map(line => {
           parser.parseLine(line).mkString(",")
         })
       }).take(5).foreach(println)
14/11/16 22:43:27 INFO SparkContext: Starting job: take at <console>:29
...
14/11/16 22:43:27 INFO SparkContext: Job finished: take at <console>:29, took 0.018557 s
9464711,HX114160,01/14/2014 05:00:00 AM,028XX E 80TH ST,0560,ASSAULT,SIMPLE,APARTMENT,false,true,0422,004,7,46,08A,1196652,1852516,2014,01/20/2014 12:40:05 AM,41.75017626412204,-87.55494559131228,(41.75017626412204, -87.55494559131228)
9460704,HX113741,01/14/2014 04:55:00 AM,091XX S JEFFERY AVE,031A,ROBBERY,ARMED: HANDGUN,SIDEWALK,false,false,0413,004,8,48,03,1191060,1844959,2014,01/18/2014 12:39:56 AM,41.729576153145636,-87.57568059471686,(41.729576153145636, -87.57568059471686)
9460339,HX113740,01/14/2014 04:44:00 AM,040XX W MAYPOLE AVE,1310,CRIMINAL DAMAGE,TO PROPERTY,RESIDENCE,false,true,1114,011,28,26,14,1149075,1901099,2014,01/16/2014 12:40:00 AM,41.884543798701515,-87.72803579358926,(41.884543798701515, -87.72803579358926)
9461467,HX114463,01/14/2014 04:43:00 AM,059XX S CICERO AVE,0820,THEFT,$500 AND UNDER,PARKING LOT/GARAGE(NON.RESID.),false,false,0813,008,13,64,06,1145661,1865031,2014,01/16/2014 12:40:00 AM,41.785633535413176,-87.74148516669783,(41.785633535413176, -87.74148516669783)
9460355,HX113738,01/14/2014 04:21:00 AM,070XX S PEORIA ST,0820,THEFT,$500 AND UNDER,STREET,true,false,0733,007,17,68,06,1171480,1858195,2014,01/16/2014 12:40:00 AM,41.766348042591375,-87.64702037047671,(41.766348042591375, -87.64702037047671)

We’re finally in good shape to extract the values from the ‘Primary Type’ column and count how many times each of those appears in our data set:

scala> withoutHeader.mapPartitions(lines => {
         val parser=new CSVParser(',')
         lines.map(line => {
           val columns = parser.parseLine(line)
           Array(columns(5)).mkString(",")
         })
       }).countByValue().toList.sortBy(-_._2).foreach(println)
14/11/16 22:45:20 INFO SparkContext: Starting job: countByValue at <console>:30
14/11/16 22:45:20 INFO DAGScheduler: Got job 7 (countByValue at <console>:30) with 32 output partitions (allowLocal=false)
...
14/11/16 22:45:30 INFO SparkContext: Job finished: countByValue at <console>:30, took 9.796565 s
(THEFT,859197)
(BATTERY,757530)
(NARCOTICS,489528)
(CRIMINAL DAMAGE,488209)
(BURGLARY,257310)
(OTHER OFFENSE,253964)
(ASSAULT,247386)
(MOTOR VEHICLE THEFT,197404)
(ROBBERY,157706)
(DECEPTIVE PRACTICE,137538)
(CRIMINAL TRESPASS,124974)
(PROSTITUTION,47245)
(WEAPONS VIOLATION,40361)
(PUBLIC PEACE VIOLATION,31585)
(OFFENSE INVOLVING CHILDREN,26524)
(CRIM SEXUAL ASSAULT,14788)
(SEX OFFENSE,14283)
(GAMBLING,10632)
(LIQUOR LAW VIOLATION,8847)
(ARSON,6443)
(INTERFERE WITH PUBLIC OFFICER,5178)
(HOMICIDE,4846)
(KIDNAPPING,3585)
(INTERFERENCE WITH PUBLIC OFFICER,3147)
(INTIMIDATION,2471)
(STALKING,1985)
(OFFENSES INVOLVING CHILDREN,355)
(OBSCENITY,219)
(PUBLIC INDECENCY,86)
(OTHER NARCOTIC VIOLATION,80)
(NON-CRIMINAL,12)
(RITUALISM,12)
(OTHER OFFENSE ,6)
(NON-CRIMINAL (SUBJECT SPECIFIED),2)
(NON - CRIMINAL,2)

We get the same results as with the Unix commands except it took less than 10 seconds to calculate which is pretty cool!

Categories: Programming

Building Games with Google Cast is Easy and Powerful

Google Code Blog - Fri, 11/14/2014 - 20:03
By Nathan Camarillo, Chromecast Product Manager for Games

This week, we announced several Google Cast games that were built with the Google Cast SDK. With a Google Cast endpoint like a Chromecast or Android TV, developers can now use Google Cast to bring a whole new style of multi-screen social gaming experiences to the living room.
What makes a Google Cast game unique? It enables multi-screen gameplay between mobile devices and a television, and it transforms users’ mobile devices into amazing game controllers. Anyone with a compatible Android or iOS phone or tablet can join the game. There’s no hassle or expense with extra controllers or peripherals; your very own iPhone or Android phone simply works and everyone can join in.

The innovative part of creating a Google Cast game is all the ways you can use mobile devices to create a variety of controls with the television as a shared screen. The Accelerometer can be used for motion controls ranging from subtle, to dramatic, to rhythmic. No-Look Controls and virtual controllers can allow you to focus on the television as you compete against friends. Direct target manipulation through touch controls can create intense gameplay moments that temporarily focus on the mobile device and then return the focus to the television where all players can share and compare the results. You can even use the microphone or other input methods to create games for everyone in the home. Whether cooperative or competitive, Google Cast enables you to create fun moments for everyone using the devices they already have and know how to use.

Now it’s your turn! Go make something fun with the Google Cast Design Principles. The experiences you create using Google Cast will entertain gamers and inspire a whole community of developers embracing a new revolution in multi-screen gaming.

Categories: Programming

Fixing the #1 problem with Xamarin.Forms

Eric.Weblog() - Eric Sink - Fri, 11/14/2014 - 19:00
Update

I'm not sure what I expected when I wrote this blog entry. I just figured I would "throw it out there" and see what happened. Maybe the Xamarin.Forms team would talk about my idea at their next standup.

What actually happened is that within an hour, Jason Smith (the lead developer of Xamarin.Forms) invited me to a Skype call, shared his screen, brought up The Code, and we talked about ways it could be improved.

I didn't expect that.

(But it was glorious, as nerd conversations about code so often are.)

The discussion gave me a decent idea of what kind of changes Jason intended to make, but I don't think it is my place to say more about the details. Suffice it to say that I'm probably not getting exactly what I suggested, but it looks like I'll be getting what I need. If Jason and his team end up confirming the viability of the changes he and I discussed, then it will definitely become possible to get much better performance in a Xamarin.Forms app.

Anyway, huge kudos and thanks to Jason for a response I consider way "above and beyond the call".

And the #1 problem is...

It's slow.

To be fair...

It's not always slow. There are plenty of use cases where it works fine.

And not all of Xamarin.Forms is slow. The biggest offender is the layout system.

But this can be a pretty big problem. It's not too hard to find stories of people who gave up on Xamarin.Forms simply because there was no way to make their app perform acceptably.

Positive remarks to balance my complaining

Xamarin.Forms is terribly exciting. I've spent much of the last three months working with it, and I love it. Like any young technology trying to solve very hard problems, it has plenty of rough edges. But it shows high potential for awesomeness.

And lots of people seem to agree. In fact, I daresay that the level of excitement around Xamarin.Forms has taken some people at Xamarin by surprise. I can't quote verbatim, but I'm pretty sure I heard somebody at Evolve say that they changed their plans for the training and sessions to include a lot more Xamarin.Forms content because of the buzz.

People want this technology. And they want it to be great.

That's my goal in writing this piece. I just want to help make Xamarin.Forms great.

BTW

If the code were open source, this blog entry would be a pull request. And it wouldn't be my only one.

Someone has made the decision to not open source Xamarin.Forms, and I shall not criticize or second-guess this choice. But I can't resist saying that if it were open source, I would be actively spending my time helping make it better.

This is a very difficult problem

Cross-platform stuff is really hard. And if you add "UI" into the previous sentence then you have to add a least two more "reallys".

I've been doing cross platform development for a long time. I've used dozens of different UI toolkits and APIs. After you master the first ten, they all start to look the same.

Until you try to wrap them in a common API, and then the differences stop whispering and start screaming.

Xamarin.Forms is venturing into territory where dozens of others have failed. We're going to have to be patient.

But good grief the layout code is slow

Seriously. Write something non-trivial with Xamarin.Forms and then bring it up under a profiler.

But that's okay...

... for two reasons.

  • The layout code is doing a lot of work for you. I don't mind code that takes time when it's doing stuff I need done. That's life. The layout system is powerful, and convenience always comes at a price.

  • The code will get better. I've spent an awful lot of time reading Xamarin.Forms.Core.dll in the Xamarin Studio Assembly Browser. This code simply hasn't had much attention given to optimizaiton yet. When it does, it will get faster.

I'm not complaining about the layout code being slow. I'm complaining that I always have to use it.

Child views vs. Layouts

The real problem here is that Xamarin.Forms gives you no way to have child views without using a subclass of Layout. What I want is the ability to write a subclass of View that can have child views.

Right now I can do something like this:

public class myParentView : Layout<View>
{
    private Rectangle getBox(View child)
    {
        // whatever
    }

    protected override void LayoutChildren (double x, double y, double width, double height)
    {
        foreach (View v in Children) {
            Rectangle r = getBox (v);
            v.Layout (r);
        }
    }
}

But for some situations, the Layout system is just way too much. Events propagating up and down. Height and width requests. Real estate negotiations between every child view and its parent.

And for the sake of making sure the layout always gets updated when it needs to be, the code triggers a layout cycle in response to, well, almost everything.

Add or remove a child view? Relayout everything.

Here's my favorite: Put a Label in a Frame in a StackLayout in another StackLayout in a ScrollView in a Whatever. The details don't matter. Just build a hierarchy several levels deep with a Label down near the bottom. Then change the Text property of that label from "foo" to "bar". This triggers a layout cycle, because the label might want to request more size, and the parent might want to care about that request, so the whole negotiation starts over.

For extra fun, bind that Label's Text property to something that is also bound to the Text property of an Entry. Now you can trigger a complete relayout on every keystroke.

Xamarin.Forms currently suffers from a problem that is very typical for cross-platform UI toolkits at the toddler stage: It is constantly triggering layout updates "just in case" one might be needed. In general, it is better to have your forums filled with people saying "layout is slow" rather than people saying "I changed something and layout didn't update".

We need a lower-level mechanism

I would like to rewrite my example above to look something like this:

public class myParentView : View
{
    private List myChildren;

    private Rectangle getBox(View child)
    {
        // whatever
    }

    public LayoutMyChildren()
    {
        foreach (View v in myChildren) {
            Rectangle r = getBox (v);
            v.Layout (r);
        }
    }
}
  • Instead of subclassing Layout, I have subclassed View.

  • Since I no longer inherit a Children collection, I have defined my own.

  • Whereas previously I did an override of the abstract LayoutChildren() method, now I just provide a method called LayoutMyChildren().

  • Whenever I know I need to arrange all my child views, instead of triggering a layout cycle, I call LayoutMyChildren().

  • If I only need to arrange some of my child views, I can write a method that does less work.

  • Not shown, but I probably need to override OnSizeAllocated() and call LayoutMyChildren().

  • Since I own myChildren, inserts and deletes from this collection will not automatically trigger a layout cycle. If anything needs to updated when my set of child views changes, I have to do that myself.

In other words, I don't want any help laying things out. I just want the ability to have child views, and I don't want anybody meddling with my right to parent those children however I see fit.

A full implementation of myParentView is going to be rather complicated. But it's going to be fast, because myParentView has all the knowledge it needs to allow it to make smart decisions and avoid doing things that don't need to be done.

The design of child views in Xamarin.Forms

The concept of "child views" should be be distinct from the concept of "deciding where those child views should be placed". Thinking over the various UI toolkits I have used, the separation of these two concepts is a common pattern.

But the example above doesn't work in Xamarin.Forms 1.2. My child views never get a renderer assigned to them, so they never show up. The element hierarchy on the PCL side is fine. It just never gets mirrored on the platform side.

When I explained this to a coworker, I said, "The whole idea almost works. In fact, it all works, right up until the end, when it doesn't work at all."

Interestingly, Xamarin.Forms is very close to the design I want it to have. In researching this problem, I expected to find the child view management stuff in Xamarin.Forms.Layout, but it's just not there. As it turns out, the ability to have child views is actually not specific to Xamarin.Forms.Layout. Rather, this concept is implemented in Xamarin.Forms.Element (right where it belongs, IMNSHO).

The Element class has three important members. It does not actually manage a collection of child views. Rather, it exposes a way for a subclass to do so. OnChildAdded() and OnChildRemoved() need to be called when the child views collection has changed. These methods set the Parent property of the new child view, but more importantly, they trigger an event. The platform code is listening to this event (in VisualElementPackager), and that is where child views get their renderer assigned.

So what I need to do is call OnChildAdded() whenever I add something to myChildren. Now my layout wannabe class looks like this:

public class myParentView : View
{
    private List myChildren;

    private Rectangle getBox(View child)
    {
        // whatever
    }

    private addChild(View child)
    {
        myChildren.Add(child);
        OnChildAdded(child);
    }

    public LayoutMyChildren()
    {
        foreach (View v in myChildren) {
            Rectangle r = getBox (v);
            v.Layout (r);
        }
    }
}

And this almost works, but not quite. As I said above, Element has three important members which are used to communicate with the renderer about child views. The first two (discussed above) are OnChildAdded() and OnChildRemoved(). The third one is a property called LogicalChildren. This property is, well, the collection of child views. The two OnChild* methods are just there to keep things up to do date, but LogicalChildren is the starting point.

So what I need to do in myParentView is just override the LogicalChildren property, which is virtual. No problem, right?

Here's the problem: LogicalChildren is marked "internal", so a subclass cannot override it unless it was written by Xamarin.

My request

Broadly speaking, I am asking the Xamarin.Forms team for the ability to have a View subclass which can have and manage its own child views without being a subclass of Layout.

My suggested implementation of that idea is to just change Element.LogicalChildren from "internal" to "protected internal".

And I've spent some time trying to prove that my suggestion would work. Purely as a proof of concept, I used Mono.Cecil to create a hacked-up copy of Xamarin.Forms.Core.dll which has my suggested change:

#r "Mono.Cecil.dll"

open Mono.Cecil

let a = AssemblyDefinition.ReadAssembly("Xamarin.Forms.Core.dll")
for t in a.MainModule.Types do
    if t.FullName = "Xamarin.Forms.Element" then
        for p in t.Properties do
            if p.Name = "LogicalChildren" then
                let m = p.GetMethod
                m.IsFamilyOrAssembly <- true
a.Write("Better.Xamarin.Forms.Core.dll")

And then I wrote a myParentView class which has an override for LogicalChildren. The result is a huge performance increase for my particular use case, because myParentView doesn't need all the general-purpose stuff that the Layout subclasses provide. My contention is that there are lots of similar use cases which can benefit from this approach.

So that's my suggested change, but I don't actually care how this gets done. If the Xamarin.Forms team were to ship 1.3 with my "child views without layouts" concept exposed in some other fashion, I would be just as happy.

Moving forward

The Xamarin.Forms.Layout classes still need to get faster. And they will.

But regardless of when this happens, the concept of "child views" deserves to exist on its own.