Categories
App Architecture Cloud

Sync is a difficult problem to solve, challenges we don’t always realise

Sync across devices can be challenging where a large numbe of small files are involved.

If you have got a very large number of files stored on one device, and you expect your cloud sync to seamlessly copy them across to other devices, you’re bound to be surprised. The surprise may turn into frustration if one of your devices is newly setup or fresh, hence needing to download everything in one go. You might even think that it will never work. Well of course, if it were so trivial then companies like Box.com and Dropbox.com wouldn’t exist. Before you indulge in ranting, it may help to pay attention to what is going on.

We live in an age of instant reward, we want things at the fingertips and exactly when we want them. This isn’t easy to pull off though, particularly when it comes to getting our data synced across multiple devices. Let’s say I take some pictures with my iPhone, I want these to be seamlessly available on all my devices – almost immediately, though, I’ll come to that later, in the case of pictures we are actually quite lucky.

sync, a difficult problem to solve

Consider the illustration, I’ve added some sequence numbers that aren’t even realistic. I snap a photo with the iPhone (1). Since I have an iCloud account, my new photos are quitely and transparently sent to the Cloud (2). The speed at which this operation completes depends on the quality of my Internet connection at the time and throughout. The transfer may also be delayed if my iPhone decides to save battery for some (always reasonable) reason.

What happens next is that, any number of devices that I may possess and that happen to be actively checking my iCloud account, may get notified that new photo(s) are available. This is step (3). Usually, since we tend to be on the move when we take photos with the smartphone, we aren’t likely to open a computer, (Macbook, iMac, etc), too quickly, so there is a chance that the new photos make it on time to iCloud and to the device before we open the Photo app (or icloud web page) to consult it. I’ve added a lettered numbering to depict that my devices may sync at any time, the progression (a, b, c, …) isn’t acurate, in reality the sequence is non-deterministic. A given device may actually never sync up until I trigger it by attempting to view my photos on it.

What I’ve very briefly described, skipping a lot of nuances and details, will be familiar to those who own multiple personal computing devices, PCs, Macs, tablets and smartphones. Imagine the scale at which this needs to be done, over 1.2 billion people own smartphones and it is now common knowledge that we take and share more photos than any other things. The systems got better over time, you barely see people complain nowadays about their photos not syncing where they expect to see them.

How difficult can it be?

Although we tend to take of lot of photos, their numbers are relatively small and the sync often goes well. For those who deal with very large number of files on their device, typically IT tech people in my experience, this can quickly go sour. I see fairly reguarly someone complain on Twitter, usually Apple is mentioned, about a bad sync experience – I’ve certainly experienced some teething problems with Dropbox sync in the past. They go like, How come Apple can’t get this right?

User may unknowingly have stored a huge number of files and perhaps didn’t give their computer enough time to sync it cross to the Cloud. Hopping from one device to another, it could be that while one sync job isn’t complete yet, user has made some more changes in several places, now the behind-the-scenes sync task gets significantly more complicated. A folder appears empty when you are sure it shouldn’t be. Your apps start throwing strange error messages, confusing you even more. But, in appearance everything seems normal, you never got notified that something was in progress or went wrong. That, latter, is where I think some low hanging fruits may lie.

Vendors such as Apple and Dropbox, in their attempt to make things transparent to users, don’t clearly indicate the sync status of your files. Apple, in particular, makes it harder because their intelligent sync feature would remove less frequently accessed files from the device, keeping a copy in the cloud until a file is requested again. This clever idea can turn into a nightmare, from user perspective, because people just aren’t aware or have forgotten about it. By improving this user experience, giving a rich but simple and cogent status to the user, people might get to appreciate what’s going on. I guess this isn’t getting attention because the number of users impacted is small.

Sync is a tough problem because there are too many factors at play and the trade-offs are quite hard to expose to users in simple terms. All the computing parts keep improving, maybe the issue will fade out faster than user annoyances grow. This is certainly a lot more complex to address than people realise. Not much get written about it on user level computing, the best discussions tend to be very technical, at distributed computing level, a specialist topic.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.