airmail-journal

Airmail Journal #4

Timestamp Concerns

As stated in my previous post, I'm using update timestamps to provide granular conflict resolution. Every time an object is changed on the device, the timestamp of that change is logged and sent to the server as part of the change. So if two users change the name of a single Envelope before a sync occurs, the server resolves to use the newest of the two names.

As I've discovered to my pain previously: some people don't pay much attention to whether the clock on their device is set correctly, which would cause a lot of problems to this approach for conflict resolution. There are many potential solutions to a problem like this, but I've decided to do something simple.

I'm using Express.js to serve the API, so I've added a middleware step that checks a custom timestamp header on the request. If that header is more than 5 minutes off (or missing) it is considered an error.

function checkClientTimestampHeader(request, response, next) {
    var clientTimestamp = request.headers["x-airmail-client-timestamp"];
    if (!clientTimestamp) {
        // respond with error
    }
    else {
        var currentTimestamp = Math.round(Date.now() / 1000);
        var difference = Math.abs(currentTimestamp - clientTimestamp);
        if (difference > 300) {
            // respond with error
        }
        else {
            next();
        }
    }
}

I intend to have the device modify its timestamps based on the timestamp from the last request made to the server. The logic will be similar: if the device's current time was more than 5 minutes off after the last request, use the difference to adjust timestamps that are being logged. The response error is only for extreme circumstances where something actual goes wrong on the client. Plus, it gives me a chance to use 418 I'm a teapot, because it's hilarious (and I couldn't find anything more suitable but specific enough).

While we're on the subject of timestamps, can everyone just stop using strings to represent dates unless they're being shown to a user? Parsing date strings is a massive performance overhead.

Airmail Journal #3

Sync Race Conditions

It's been a while since I've posted about Airmail but I have been working on it whenever I find time outside of my usual contracting work. Since my early thoughts I've changed the design somewhat: I'm now using 2 NoSQL databases, MongoDB & CouchDB, instead of a traditional RDBMS. MongoDB handles authentication and payments, where CouchDB handles user data.

Why CouchDB?

CouchDB is something I've used before in my contracting work and it is particularly useful when it comes to syncing data. It has built in support for revision identifiers, change sequences, and persistent HTTP connections. The main advantage here is that my own API can just provide a lightweight wrapper around CouchDB's HTTP API for syncing changes.

Local Changes

The UI on the device only ever refers to its own persistent store, but there are classes that listen for changes to the data on the device, and cache them in a separate file until the next sync operation takes place—I call this the Local Changes cache. I'm using Core Data, so the Local Changes cache listens for the NSManagedObjectContextDidSaveNotification and processes the objects that have been changed.

The Local Changes cache is intelligent enough to concatenate multiple changes to the same object that happen while the device is offline, and send them all to the server as one change. It also means that I don't have to query the database for changes, rather I can just keep this cache in memory.

To put it in a simple format, the local changes cache might look like this:

{
    "envelopes": [{
        "id": "17bb2c1c-313d-41a6-ae31-c90785155687",
        "changes": [{
            "attribute": "name",
            "value": "A New Name",
            "timestamp": 1423083473
        }]
    }, {
        "id": "0cfd5804-d92d-47fa-b3c0-3845835dee25",
        "changes": [{
            "attribute": "budget",
            "value": "50.00",
            "timestamp": 1423083525
        }]
    }],
    "transfers": [{
        "id": "23590686-48a0-4f85-b0e7-3f0606cb0244",
        "changes": [{
            "attribute": ...,
            "value": ...,
            "timestamp": ...
        }, {
            "attribute": ...,
            "value": ...,
            "timestamp": ...
        }]
    }]
}

Note that each change has a timestamp associated with it, which is used for conflict resolution on the server. This allows the conflict resolution to be attribute-specific, rather than just whomever made the last change to the object wins.

I plan to go into the device sync engine in more detail in another post, but this description serves to provide context for the race condition in the following thought experiment.

The Race Condition

Presume there are 2 people sharing data, John and Marie. Both their devices are currently in Airplane Mode and they both change the name of the same envelope, Holidays. John changes it to Holiday 2015 and Marie changes it to Family Holidays (Vacations for my US friends 😝). John's change is made before Marie's, so once a full sync completes Family Holidays will be the winning change.

John disables Airplane mode and updates the server to Holiday 2015, and subsequently Marie disables Airplane mode, and herein arrives the problem: once Marie's device downloads the changes from the server, her database will be updated Holiday 2015 despite the fact that she currently has the winning change queued up in Local Changes. The Local Changes and the database are now out of sync, and when Marie's device pushes its own changes up to the server: Family Holidays will become the value on the server, and on John's device, but not on her own.

The Way Around

The only solution I can see is to do some conflict resolution on the device, something I was hoping to avoid. When changes are downloaded from the server the application will have to check its Local Changes before writing to the database. Annoyingly.

Airmail Journal #2

The problem with accounts

As I mentioned in my first journal post, I don't much care for usernames and passwords. In fact, I've been systematically planning their demise since I started in software development - I'll let you know how that goes. This is especially true in mobile development. We each have our own device,1 so why not just associate an account with the device and leave it at that? Nobody needs to create a username and password for that to happen.

User defined passwords are a security disaster waiting to happen. There's only so much that we, as developers, can do to prevent our users from making mistakes. The most obvious of which is the general population's penchant for using hideously insecure sequences of characters to prevent unauthorised access to their data, but what I consider to be the much greater threat is password reuse. Despite how much I try to convince all of the people in my life that they're putting themselves at risk with this practice, it's still something I see all the time. I definitely don't want to be blamed for someone's online credit card account being breached because they used the same password for my lowly sync engine.

What's the solution?

Those of us with good sense know to use something like 1Password to make sure that all of our passwords are different, and hopefully just a series of random characters. But do you know what's even better than random characters? Time-limited random characters. And do you know what's really good at generating random characters? Computers. It's almost as if this is what we were supposed to do all along...

One notable, recent, example of an attempt to circumvent usernames and passwords is Marco Arment's login system for The Magazine's website.2 It works by sending a temporary URL to the email address associated with your account, which then grants you access to the account on that machine. My issue with this idea is that it's effectively delegating authentication to your email provider; if an attacker compromises a user's email account, they can compromise the user's account on your service.

This is how I envisage the login process for Airmail:

  • A user creates a new account, which they are immediately connected to. They don't need to give me anything, I can generate a UUID to refer to the device they're using.
  • The user requests an invitation for their significant other to join the account.
  • The Airmail server generates a random sequence of characters, which is securely returned to the user. Let's call it an invitation code.
  • The user shares this invitation code with their significant other.
  • The user's significant other elects to join an account and uses the invitation code to gain access to the same data.
  • The invitation code is invalidated.

Simple. Secure. And of course: the invitation codes would expire if not used within a reasonable timeframe.

By handling the process like this, the user is never asked to give any personal information that could potentially be compromised. There is also a much lesser chance that an attacker would be able to gain access to the account even if they did get hold of an invitation code. But there are still issues that need to be solved.

Recovery

Imagine that Adam creates an account and invites Betty to join. Then if Adam needs to delete and reinstall the app for any reason,3 Betty can invite him back to the account and everything continues as normal. However, in the unlikely event that both Adam and Betty delete the app simultaneously, nobody is left to generate invitations and the account is abandoned. This is a problem.

There needs to be a way for the Airmail server to generate an invitation code for the account in this situation, but without compromising the security of the account - multi-factor authentication is required. My current thinking is that I can accomplish this with a recovery key (possession factor) and some information about the account data (knowledge factor) to verify the user. I'm still ironing out the details in my head, but I'm relatively sure that I can handle this manually via support requests. It should be a rare occurrence in any case.

Ownership

Continuing with Adam and Betty, it stands to reason that Adam is the account owner because he created the account and invited Betty to join. There are a few maintenance tasks that will need to be possible, which would usually be handled by an account owner:

  • Extending the subscription.
  • Inviting new collaborators.
  • Revoking access for collaborators.

We need to consider the first scenario in account recovery again: Adam has deleted and reinstalled the app, and needs to regain his access to the account.

By assuming that Adam is the account owner and only allowing the account owner to generate invitations, Betty wouldn't be able to invite him back onto the account and he would have to go through multi-factor authentication. I don't like this.

By allowing all users to generate invitations, but only the account owner to perform the other maintenance tasks, Betty can still invite Adam back onto the account. However, the device that was considered the account owner now isn't active anymore, so nobody can perform the other maintenance tasks (without potentially going through multi-factor authentication). I don't like this either.

My preferred solution is that nobody is the account owner. The server will not distinguish between the user that created the account and any subsequently invited users, and the only difference for Adam and Betty is that Adam would have the recovery key for the account - Betty can still perform all maintenance tasks. This falls down when imagining a scenario in which someone with malicious intent has gained access to the account, and can now revoke the owner's access to their own account. I consider this to be another rare occurrence, and the existence of the recovery key should provide an adequate workaround.

Conclusion

I don't need usernames and passwords, I can create a much more secure system by taking away the user's ability to define any of the keys that grant access to their account. I feel I should clarify: I'm in no way a security expert - far from it. I welcome anyone to poke holes in this system or suggest better solutions to the problems I've described already. Hit me up on twitter.


  1. iOS doesn't support multiple user accounts, and neither do I.

  2. Judging by this and the original Instapaper no-password login mechanism: Marco shares my disdain.

  3. Access to Airmail would be granted by an authorisation token, which would be erased if the user deleted the app.

Airmail Journal #1

The story [begins | continues] - delete as appropriate

Inspired by Brent Simmons' series of blog posts about implementing syncing in Vesper, I have decided to chronicle my own work in building a sync engine for Envelopes. I intend to use these posts as a way to keep track of all my ideas as I have them. And also as a way to hold myself accountable for the development - I often have motivation issues when it comes to writing my own software.

Introduction

I have already decided that the sync engine will be called Airmail (get it?), hence the post title. This will be my second attempt at writing it; I have previously completed around 50% only for circumstances to make finishing it impossible. The work I already have is mostly useless in light of what I have learned (and Apple has changed) since then.

Since I released Envelopes in 2010, syncing is by far my most requested feature.

Requirements

Before starting again, it feels appropriate to list what exactly I hope to accomplish with the sync engine.

It will be a premium feature.

The cost will be subscription based rather than a single payment. This introduces the concept of account expiry and all the headaches that go with it (informing the user, restricting access, eventual removal etc.).

Payments will be taken initially via in-app purchase, but in the future it should be possible to take payments in other ways. It should also be possible to grant promotional free periods to customers at my discretion.

The concept of an account should be exclusive to the server.

Usernames and passwords are tedious, I don't like them. I also don't like the idea of storing a password on my servers that is potentially used for an online banking account or the like (not everybody uses 1Password) - it seems like every other week another story comes out about leaked account details and password hashes. I don't even want to go near something like this.

The goal is sharing data.

Customers of Envelopes typically want their spouse or significant other to be able to modify the same dataset, rather than struggling with the logistics of having a day phone and a night phone.

So, a user needs to be able to invite someone else to share the data in their account. This mechanism allows me to avoid usernames and passwords, but still solve the day/night phone problem if needed. The one issue with this approach I've thought of (so far!) is account recovery: say a user accidentally deletes Envelopes from all the devices connected to their account, they now can't invite themselves to join again.

It needs to be secure.

Envelopes is a financial management tool; the data is inherently sensitive. Ok, so I'm not exactly storing credit card details, but most people wouldn't like the idea of an attacker seeing how much they spent on sex toys last month.

Tools

Platforms

In my first attempt at the web service I used Rails, but I'm now planning to use Node.js much like Brent. I like the single process model, and I like closures. I'll be using it with a relational database system, rather than a trendy NoSQL alternative.

On the device I use Core Data for persistence and I have no intention to change this. Despite the fact that a lot of people dislike Core Data (for perfectly valid reasons), over the years I've become quite adept at making it perform even with huge datasets - and Envelopes isn't exactly the kind of app that creates huge datasets in any case.

Editors

I use Xcode for Cocoa development, and Sublime Text 2 for everything else. If you're wondering why I use Sublime Text 2: I made this decision some time ago for several reasons (mainly personal taste).

Sublime Text 2 is also not the kind of editor I would usually entertain, but it works, I know the keyboard shortcuts, and it's very powerful once you learn how to use it.