deeje.com/musings

Bansai Tree 3

Bansai Tree 3
Originally uploaded by deeje

ommmmmmmm

iPad First Impressions

Overall, I think the iPad is great, and I've already got a list of features I want to see soon…

• iPod and Airport Express - would love to play songs directly from my iPad over my home stereo, without having to plug in a wire.
• Users and/or guest accounts - want to be able to hand my iPad to my kids, spouse, or friends who are visiting, and not worry about them playing my games (in progress), sending emails or tweeting from my account, or buying things with my credit card.
• iMagazine - not sure any of the periodicals have the use case right for iPad… what I want is to subscribe, and have new content delivered automatically, all of which speaks to a background process. What I don't want is each periodical with its own background process. Podcast content has the same issues.
• Springboard 2.0 - As it is, the home screen only scales to a few pages of apps, and as with the iPhone, doesn't scale well to hundreds of apps. This is further compounded on iPad because sometimes all I want to see are iPad-specific apps… personally, I want to tag all my apps and then pivot on those tags.
• Family iPhoto Cloud - my wife and I both take pictures, and store them on different computers… how can my iPad pull all those photos in, complete with tags, faces, event info, etc?

Media, Media, Everywhere!

Here's something I've noticed about the iPod/iPad user experience… my media is scattered all over the place!

• iTunes - buy songs

• iPod - play music, videos (movies and TV shows), audiobooks, podcasts
• Videos - play TV shows and movies
• App Store - buy apps
• Home screen - run apps
• Photos - view images and play movies
• Camera - take and view images and videos
• MobileMe - view photos from the web
• iBooks - buy and read books

That's a lot of places to go to get to my media. What's more, sometimes the store for a kind of media is a feature of its related player (iBooks), and sometimes its a product in and of itself (iTunes/iPod, AppStore/Home). I think that lack of consistency is a problem that will only get worse.

FlockDraw

Draw with me!

Or, go here http://flockdraw.com/2xozbd

Just how accurate is the iPhone 3GS GPS and Compass?

I'm excited about the concept of augmented reality (AR) apps, and am curious what kinds of AR apps are *practical* with the state of the art iPhone 3GS and its accelerometer, GPS, and Compass sensors. I decided to do some field tests to see how accurate the GPS and Compass sensors are on the device. In this post, I'll describe my tests, show you the data, and offer some conclusions…

Burning Man 2009

Burning Man 2009
Originally uploaded by H Dragon

This was one of the most amazing things I saw at burning man 2009… a string of balloons, apparently 1/4 mile long steel cable, and at night each balloon also had a small light on it… simply stunning!

Notes on writing a history-driven client-server synchronization engine

I've been working on a synchronization engine for an iPhone application called Clinks, and wanted to share some notes to help others down the road. This is a hefty topic… heck, even the title had to be somewhat unwieldy. With that, lets roll up our sleeves and dive in…

Goals

First off, you're probably reading this because you recognize that native iPhone clients need to go above and beyond a polished web experience. In particular, native apps need to function regardless of the state of an internet connection. This means making sure that sufficient data is cached locally, and that any local changes get propagated to the server/cloud when an internet connection is re-established.

Second, the synchronization engine you consider building will be highly dependent on the goals of your app and on the schema of your data. I've seen basic algorithms for sync engines that attempt to sync *everything* from the client up to the server, and *everything* on the server back down to the client, but in my experience, this isn't a solution to any real-world problem, just a starting point.

By way of example, Clinks is a social wine journal that allows wine enthusiasts to keep track of the wines they try and like, and share that information with their friends. The underlying schema contains, in high level terms, tables for person, product, and experience, where experience captures the tastes and ratings of a product by a person. Users can add experience records to their journal, which may or may not create new product records (they may add an experience record for an existing product, for example). For our sync scenario, we want to push all changes to the experience table, and only push new product records. We also want to pull down friends' experience records as they change too.

We've kept things simple: sync on a record level, and collisions in the history tables are handled using simple FIFO logic. Your needs may vary, and will directly affect the complexity of your sync algorithms.

History

One way to implement record-level sync is by using a history table, which is basically a separate table in your schema that tracks all the changes made to records in the other tables. In your code, whenever a user creates, updates, or deletes a record, your code also insert a new record into the history table that captures all that information: person, table, record, action, date.

Note that each client has a history table, and the cloud/server has a master history table. With that, the sync engine on the client basically reconciles the local history table with the history table on the server on a periodic or on-demand basis. The reconciliation can be optimized by keeping track of the last history record sync'ed, and then on subsequent syncs only processing history records after the previous last history record.

GUIDs

Before we proceed, we must address the issue of primary keys. All databases support a primary key field for uniquely identifying a record, which is crucial for edit operations as well as relationships between tables. Unfortunately, almost all databases assume management of primary key (PK) fields by default, and don't have any concept of a "distributed" PK space.

To illustrate, consider two users starting with fresh databases on their iPhones. When each user creates a record, each database will assign a PK of 1 to the records. Now when those users try to sync to the cloud/server, there would be two records with the same PK value (disregarding typical checks for uniqueness, etc.), making a mess of downstream searches, edits, and sync.

The simple solution is to add an additional "GUID" field to all tables that are to be synchronized, and to generate globally unique identifiers for each new record. Many operating systems and languages have APIs to do just that, generating sufficiently long and random unique ID strings that are statistically unlikely to ever collide.

In practice what this means is that your sync engine will deal with GUIDs, while your database engines will continue to deal with PKs. One significant change you'll need to make to your server is to implement fetch based on GUID rather than by PK. For Ruby on Rails, we simply overrode the GET resource to use the GUID field, then left all the other verbs, like PUT, POST, and DELETE using the PKs.

Algorithm

Here's some pseudo code tying all this together:

// push
get lastClientHistoryDate
get clientDeltaList from client History where (date > lastClientHistoryDate)
for (clientHistoryItem in clientDeltaList)
  if (clientHistoryItem.guid isn't on server)
    switch (clientHistoryItem.action)
      case create:
        get clientItem from client where (guid = clientHistoryItem.guid)
        new remoteItem
        remoteItem.guid = clientItem.guid
        remoteItem.data = clientItem.data
        post remoteItem
      case update:
        get clientItem from client where (guid = clientHistoryItem.guid)
        get remoteItem from server where (guid = clientHistoryItem.guid)
        remoteItem.data = clientItemData
        put remoteItem
      case delete:
        get remoteItem from server where (guid = clientHistoryItem.guid)
        delete remoteItem
    add clientHistoryItem to server History table
  set lastClientHistoryDate = clientHistoryItem.date

//pull
get lastServerHistoryDate
get serverDeltaList from server History where (date > lastServerHistoryDate)
for (serverHistoryItem in serverDeltaList)
  if (serverHistoryItem.guid isn't on client)
    switch (serverHistoryItem.action)
      case create:
        get remoteItem from server where (guid = serverHistoryItem.guid)
        new clientItem
        clientItem.guid = remoteItem.guid;
        clientItem.data = remoteItem.data
        save clientItem
      case update:
        get remoteItem from server where (guid = serverHistoryItem.guid)
        get clientItem from client where (guid = serverHistoryItem.guid)
        clientItem.data = remoteItem.data
        save clientItem
      case delete:
        get clientItem from client where (guid = serverHistoryItem.guid)
        delete clientItem
    add serverHistoryItem to client History table
  set lastServerHistoryDate = serverHistoryItem.date

Note that you must fetch the record from the opposite DB using GUIDs in order to get the PK for that record for subsequent DB operations.

Note also that the algorithm appears to push changes up, then process those same changes coming back down. While this is true, the algorithm is actually designed to support *multiple* clients for a user, all syncing to the cloud/server, so there could be unprocessed history items on the server that client hasn't yet fetched.

And finally, note that this algorithm doesn't speak to relational schemas. You'll need to map the relationships between record GUIDs to the record PKs on each database accordingly.

Dates

One final thought on date fields. We use dates as a way to optimize the sync engine, allowing us to only process history records that are newer than the last sync date. Its important, therefore, to declare your date fields the same on both client and server, or to otherwise normalize the date values when comparing them.

The myth of the myth of self-commenting code

Synchronicity! I was thinking of writing on this topic, and up pops this gem…

The Accidental Businessman wrote:

The myth of the myth of self-commenting code

On Code Commenting And Technical Debt | BrandonSavage.net

Interesting post by Brandon—I disagree only on one main point:

This myth of self-documenting code needs to be squashed once and for all. Commenting is important, and unless your code does nothing other than display “Hello World!” it’s going to need some explanation.

I find this view of coding a little misguided, to be honest—in fact, I would go as far as saying that code doesn't need any comments.

Wait, did I just say that? Yes, I did, and here's why: code is a discussion between the developer and a machine. Writing code that cannot be comprehended by another human being is a bit like writing a letter to your dear friend in a language you can write but not read—strange, at best.

Good code should be self-documenting insofar as you should be able to understand what the code does without the assistance of any comments—after all, the computer gets it, so why shouldn't you? This can easily be done by the judicious use of clear logic and good programming practices like giving your variables names that are actually related to what they do.

What I suspect Brandon really means is that the comments are there to illustrate the intentions of the author when those intentions are not immediately made obvious by the code itself. This, I daresay, is the crux of Eli's argument that commenting inside the code is the most important contribution that a developer can make towards eliminating technical debt. That's because commenting inside the code helps understand the reasoning behind the choices that were made when writing the code, rather than what the code itself accomplishes…

Having re-invested myself to production level coding, and having dug into lots of existing code bases over the years, I've come to a similar conclusion… comments in code are only necessary to explain choices that are otherwise not obvious.

kinetic wave sculptures

I had the pleasure of seeing one of these sculptures at the Exploratorum some years ago… steve cooley presents wrote:

kinetic wave sculptures

This was too cool not to repost:

Rating Systems

How do you use the various rating systems you encounter in your life? Do you use the full resolution of each rating system, or have you unconsciously developed a mapping of your own classifications to the rating systems being provided?

Numerous products incorporate rating systems. Ideally, these systems allow users to develop profiles that then help recall past products and experiences, as well as discover and select new products and experiences in the future. The most common rating systems I've seen include:

• 5 stars
• 3 Stars
• Thumbs Up/Down
• 100 Point

Ratings for collections

In Apple's iLife suite, both iTunes and iPhoto incorporate the 5 star rating system, and underneath the hood, these ratings are stored on a 100-point scale.

I find it interesting how similarly and differently I use these rating systems in each of these products. In iTunes, I reserve the 5 star rating for songs I consider true classics, songs I believe will hold up over time and come to represent the best music I've ever heard. With a modest collection of 3806 items in my Music tab, only 34 items are rated 5 stars, less then 1%. 373 items are rated 4 stars, just shy of 10%, and 1547 items are rated 3 stars, which represents less than half my library. I have smart playlists defined for each of these ratings, and generally tend to play music with 3 or more stars, and I rarely mark music with less than three stars. In fact, of the 6 potential ratings available, I only really need 4: none, good, great, awesome.

In iPhoto, I use a similar 5, 4, and 3 star system. 5 star photos represent the absolute best moments of my family's life. Photo's with 4 stars and great, and photos with 3 stars are worth sharing with my family. I make little distinction between 1 and 2 stars, and only both marking photos with those low ratings as a way to indicate a photo worth preserving. As with iTunes, I have smart lists in iPhoto that find all these photos at various ratings. As with iTunes, I'm not really using all 6 rating positions (0-5), but only need: none, save, good, great, awesome.

Ratings for entertainment

Because these are collections I curate, it makes sense that these ratings to be only positive. But there are lots of places where I want to capture a negative rating because I am not collecting things but rather experiencing them.

Netflix offers a 6-position rating scale, with dislike, none, and 1-5 stars. I rarely mark something as dislike, though I do remember rating a bunch of things early on to give the recommendation system something to chew on. And I rarely mark something less than 2 stars, where 2 stars is meh, 3 stars is good, 4 stars is great, and 5 stars is something I would consider buying.

Hulu's approach to ratings mixes a negative-to-positive spectrum on top of a positive rating scale, where 1 star is "Hated it" and 5 stars is "Loved it". I personally don't consider this system to be a good design… it feels very first-grade-ish, where even the worst performance merits a "star" for effort. Unfortunately, there is no antithesis of the star symbol, and so it isn't easy to craft a rating scale with stars at the positive pole and something metaphorically connected at the negative pole.

So far, all of the rating systems I've described have suffered a bit from too much resolution, in that there are more choices available than I need to express an opinion or make a classification. I have to mentally transcode my own value onto the rating system available to me. For shared systems, this creates aggregate rating systems are are noisy and course, because even with a vocabulary of only 5 words, people may still not be saying the same things to each other.

To contrast this, consider one of the better (IMHO) approaches to rating experiences, the San Francisco Chronicle's movie reviews "little man". Except perhaps for the empty seat, each of the icons is very clear and understandable when seen alone, and makes complete sense when seen in sequence. Moreover, there is no ambiguity, and I believe most people can express their own responses to a movie using this scale with little-to-no adjustment on their part. The resolution of the rating system seems just right.

Ratings for wine

Now we turn to the rating systems for wine… Ug! What a mess! In the mid-20th century, wines were often rated on a 20-point scale, often credited to UC Davis. Then in the late 1970s' Robert Parker started publishing wine reviews using a 100-point scale. Talk about too much resolution! What's the difference between an 82 and an 83? Nothing really, but because Parker has become so powerful in the wine industry, the difference between an 89 and a 90 can make or break a wine maker. For good or bad, the rest of the industry has pretty much jumped on board.

There's no denying that the U.S. population can apply their grade-school experiences to understand the value of 90 points as representing a good wine. It's unfortunate that, unlike movies or food or music or just about any other experience, people rely far too much on wine critics' ratings and not nearly enough on their own experiences to judge and compare wines. And the sheer inanity of the 100-point resolution compounds this fact for the average wine drinker.

For Clinks, we concluded several things: first, that the 100-point scale isn't going to be replaced any time soon (certainly not by a few eager wine drinkers who decided to write a cool iPhone app); second, that the cure for the Parker monopoly is the emerging democratization of wine reviews and ratings, aka Wine 2.0; and third, that people need a way to express their experience of a wine that was easy, unambiguous, and direct. So, we came up with a rating picker that focuses on the near-universal metaphor of thumbs up and down, which maps back onto the 100-point scale.

In this way, the 100-point scale is collapsed into a manageable resolution. By no means will this quell the growing debate about the value and validity of the 100-point wine rating scale, but we think is does help average wine drinkers rank their experiences more easily, which then helps them remember the good wines next time they're at the store. It will be interested to see how people react to it.