Suddenly my MacBookPro becomes unresponsive, tempted to fling it out the window, and curse Apple. But just wait, what’s actually going on?

This article was actually written just before new year. So the tone may reflect that. But I only managed to add a small bit then publish it now.

A recent craze on my Twitter timeline was triggered by someone’s rather ill-advised tweet (no direct link, hoping to spare the poor lad). The guy has CEO as title in his Twitter profile. He tweeted:

The best software developers I know are always hacking over the holidays.

The twitterati started mocking him, sometimes in creative ways. Hopefully the author’s learned his lesson.

Anyway, that aside, I did take a break, turned off my computer for a few days so that I could dedicate most of my attention to my little family. Then, just today, it happened. I turned on my Mac, it became totally unresponsive. I wasn’t sure what I did wrong. I only remembered recently updating the CUDA driver and Parallels Desktop, but nothing else. A strange dialog popped up, giving me the shudder:

your system has run out of applicaition memory

What was up with this? How come, a system that run smoothly for two years, would suddenly behave like this? Puzzling anyone, except perhaps those versed with the inner-workings of MacOS – I’m not one. At this point it would be reasonable to consider a system reinstall, would cost a day’s effort.

Too long, Don’t read [TL; DR]

Two processes showing high CPU usage, Dropbox app for Mac, a called bird which seems to stand in for iCloud daemon. iCloud also showing memory leaks in the system log. Additionally, Docker for Mac Beta, Parallels Desktop, Parallels Access and Veertu are all installed. All of these installed kernel extensions, something in that mix was apparently causing havoc. I couldn’t tell which one. After a long quest, I resorted to 1.) Backup my Mac, 2.) create a bootable USB stick and reinstall the OS. Only then it was back to normal operation. Without losing any files, or buying a new computer, I was able to fully recover my system.

Now the whole story

I took a deep sigh, started investigating. I used the troubleshooting tips that I know, checked the disk with Disk Utility, it found no errors. Uninstalled CUDA, removed items from the LaunchDaemon and LaunchAgent lists. To no avail, nothing helped.

Boot MacOs in Safe Mode

I started googling at this point. I eventually found a suggestion to restart the computer in Safe Mode. I wasn’t sure why that should help, but I tried nonetheless. I followed the procedure:

  1. Restart the Mac
  2. Immediately press and hold the Shift button.
  3. It took so long that I actually put something heavy on the button to keep it depressed and used my iPad while waiting. Later I woul find out it wasn’t necessary to hold the key depressed. I didn’t time it, could’ve been anything between 3 to 5 minutes, but eventually …
  4. When the login screen showed, I released the SHIFT button, then restarted the computer.
  5. The problem seem to be resolved. But I was going to find out soon enough.

Then something dawned on me: Caching screwed up my machine, again!

For a little while I used my computer, then it started to slow down steadily to a crawl once more. It became totally unusable. It got so bad that something seemed to be turning the camera on without user intervention! I checked, didn’t see Photo Booth or FaceTime running, what the heck was turning the camera on then? Now I really started to worry that some malware has crept in, ruining my system.

Chasing malware (or ghosts)

Now thinking that my computer was malware infected, I decided to investigate. First, I covered the camera with a tape, started doing packet capture and analysis with Wireshark. This is a tedious task, it’s forensic work, it takes forever and you don’t know upfront if you’ll get anywhere! I briefly had a go, sampled various connections, scanned and tracked them. I found nothing, not sure how the camera how was triggered anyway. After about an hour of prodding here and there, I stopped and decided to should try something else.

The system became slow to the point that nothing worked. Activity Monitor was stalling with blank screens. Finder wasn’t responding. Trying to launch a terminal session wasn’t working, I would invoke the system search box and it would take 2 minutes to show the search input box! I would try to type something, one character shows, not the rest, I stop typing wait and after much delay, a couple of characters show and then again no response. In short, all the signs of an absolutely hosed computer. My trusted workhorse was just letting me down, without a warning, and all the usual troubleshooting tools keep showing me green, a clean bill of health for my computer. What the heck is going on?

Getting angry

At this point my patience run out, starting to get angry. I have no idea how I managed to trip the system up in this way. I felt I wasn’t actually doing anything special or tricky, just running Parallels Desktop and getting my Windows VM updated. Other than that, nothing else. I can’t remember such a terrible experience with a Mac since the days before I got rid of Adobe Flash, a few months before the famous thoughts on flash letter was published. Simply cursing Apple is now tempting, given the perplexing decline in quality at Apple. But I thought better of it, I wasn’t just going to fall for a cargo cult. I had to fix this, it’d already cost much time but I had to find out.

Boot in recovery mode

I rebooted it in recovery mode, intending to restore from a backup. Then I found out, rather was reminded that I had explicitly excluded the OS from my backups. Had I resorted to the online restore or other means, I would be going back to an ancient version, this means OSX Maverick! No way!

I left it alone, went back to socialise some more.

A few hours later, I returned, not sure if it’d be worthwhile or not. I run a systems diagnostic test, boot with ⌘ + D. The diagnostic completed signalling no errors or problems at all! This isn’t making any sense at all, no problems or errors anywhere, yet my system has suddenly become unusable, how is this possible ?!?

I had to look in another direction now, perhaps I should have done right from the start. I booted it up in recovery mode again, this time I opened up a terminal session to examine the system differently. I had a hunch that perhaps some software has setup a kernel extension hijacking my system.

Kernel extensions misbehaving?

On terminal, I looked for “.kext” files in the system Library. There were a bunch of them. I couldn’t recognise most, logical since I’m mostly just a user. However, I was beginning to doubt if Docker for Mac or Dropbox was causing the issue. The reason for thinking that is because, I know they are architected to directly hook into the system, listen to low level events and catch some of those on the fly. Dropbox is a particularly aggressive one, I had to remember to turn it off before ever upgrading Xcode for example. Whenever I fail to do that, Xcode installation or upgrade would take forever.

So I had my potential culprit. I deleted every kernel extension associated with Dropbox, Docker for Mac, and Veertu. I also deleted the apps themselves, emptied my account’s Login Items, then restarted the computer. And tah dah!, there it was, back in business again, all responsive and smooth! Phew! This cost me hours I hadn’t planned. I decided to write about it, use it normally for a couple of days before reaching any definite (temporary) conclusion.

Are we there yet?

After all that effort, I thought I was safe. I was wrong. After a reboot, the system performance would slowly degrade. Before it became totally unusable, I would reboot it, gain some relief only to go back in the same state in less than an hour of normal use. I was baffled. Alas, having tired of trial and error, I decided it was time to call Apple Support. As it happen, my standard Apple Care support  had expired, so I had to purchase a one time € 29,- support call. I decided to go for that.

I called Apple, the friendly voice at the other hand took me to the usual drill, paths that  I’d already exhausted earlier. We went through reboots, diagnostic, SMC Reset and all that, to no avail. Apple Support suggested that I reinstall the OS from the recovery mode session, and call them back should that fail. I hung up, took a break. I came back and tried to reinstall the OS from the recovery mode. I booted in recovery mode, tried to reinstall the OS, it would start for a little while then fail. I tried this several times, no success. I eventually gave up for the day.

The next day, I called Apple Support again. Apple Support now suggested a remote session, I were to download and install a program. That process also stalled, I got a zero length file that wouldn’t install. Apple Support now suggested that I book an appointment with Genius Bar. I reluctantly thought I’d try, though I doubted it would help. I called Genius Bar, they were super busy, the next free slot was a full week away. I couldn’t be without my laptop for that long, I decided against this. I had to figure it out by myself, buying another computer seemed like a real possibility now. But which one, given that it was allegedly ‘no longer for professionals‘ ?  😉

I went to my Mac Mini, an mid-2011 model, still working fine though noticeably slower. I checked the time stamp of my last successful Time Machine backup, it was one day earlier, in the interim I hadn’t created anything new on my computer. So I could afford to rebuild it. Using the Mac Mini, I launched App Store app to download a fresh copy of macOS Sierra. When it finished, I created a bootable USB with it. I tried to boot my MBP with it, that didn’t work. So I booted it up normally, logged in as one of my spare admin user, then mounted the USB stick and launched the installer app. The first attempt failed, but the second attempt worked. Finally, I could let it go through the long running installation process to its completion.

Once the OS re-installed successfully, I rebooted the machine and logged in with my regular account. I found that everything was where I left it, nothing lost, nothing broken, and my computer was back to its shiny best! I had expected to have to restore from backup, I didn’t have to. And since, for a long while, I took the habit of spreading my work files between iCloud and  Dropbox, I was sure my work documents were available, intact.

Conclusions

I am not sure what triggered the problem, but it was really startling and irritating to see my computer suddenly become unresponsive. I still haven’t figure out how the camera seemingly turned itself on. Was I hacked? Is there some command line tool or keystroke sequence to start Camera without using FaceTime and Photo Booth or some kind of camera enabled app? I don’t know, it is not reassuring at all. It might as well have been a malware, I had no patience to do the forensic work, rebuilding it has got me rid of any troublesome bit there was. As I was contemplating this issue, I saw a tweet in my timeline, on the security aspect. It’s a product I once run into but might now give a try.

When our computers become unstable, we are often quick to blame the vendor. This behaviour is even more acute when it’s open season for bashing a vendor. By this, I mean the cargo cult habit of thrashing a company for it’s alleged failures. Whether justified or not, that’s how people typically behave. I’ve linked a few articles above on this subject. You can’t always blame people, it’s frustration due to feeling powerless and a sense that you’re getting a service below par. Yes, some are definitely out for grabbing headlines, juicy click-baits, I-was-first and what not, kind of pursuit. In my particular situation, the system was healthy with regular apps running. Having installed a bunch of stuff over time, I eventually reached a point where some system extension caused trouble. So, once trouble hit, backtracking some of what I did, I was able to eventually recover my system. It did take a lot of time, as I hope I’ve detailed enough. Does this say something about Apple? I am not so sure. Years ago, when I only used Windows PCs and laptops, I also reached such situations at times. Does this say something about software reliability in relation to extensibility? Absolutely!

It’s often a question of trade-offs. I could have pursued the path of getting external support, it would have cost me time, money and I’m not even sure the fix wouldn’t have unnecessarily resulted in even more expenditure. By doing it myself, it cost me a lot of personal productivity and leisure time, but I didn’t spend any extra money to solve it. Maybe I was just lucky that there was no hidden hardware or other serious issue.

It could have been a hardware issue, though the diagnostic tools didn’t report any problems, I have to rule that out. So, yes it’s possible that Apple software contains some annoying bugs, every software has those. Maybe I was just unfortunate to have hit one. The same thing is possible with any number of third-party software I have installed over time. It is also possible that I was infected by some malware. Whatever the case, in this particularly situation, having found the resources to troubleshoot my problem and get back to a normal operation, it would be harsh to only blame Apple.

Mac, Windows or other Unix/Linux are all susceptible to get corrupted and become unstable eventually, as you install more and more software on it. We, the users, most often find ourselves in the latter situation. The non-technical user, the user in a rush, often don’t have the resources to fully recover their machines in such situations.

If you’ve hang around this far, I thank you for your patience and I truly hope this tale could help save you some time or frustration or even unnecessary expenditure, some day.

In the department of Too Subtle UX, your Apple ID is the unique passport for iCloud on every device

I was triggered by the following tweet:

I quickly read the article and saw what was going on. Indeed I did notice it, but I knew to expect the kind of behaviour. The good news is, the feature seems to be working correctly: with a single Apple ID, you can login to any device and get access to your desktop and documents thanks to iCloud. The bad news is, most people will easily overlook the single Apple ID part, they’d only think about each device they’re using and would not make the mental link to a single iCloud account. Apple, with their clout, are bound to know this and should be expected to anticipate on it. However, as far as I can tell, people weren’t explicitly forewarned, and they’re probably supposed to be obviously aware of owning a single iCloud account (as in Duh, what else are you thinking?). And that is precisely the problem, this kind of logical behaviour, although intuitive at first sight, doesn’t take into account the long established mindset that we all have. Therefore, while upgrading the OS you wouldn’t think twice before enabling iCloud sync.

In my personal experience, iCloud did the right thing. The first computer I upgraded was my MacBook. My Desktop and Document were all properly sync’ed to iCloud. Then a few days later, I upgraded my Mac Mini, this time iCloud added a new folder on my ubiquitous iCloud backed-up desktop, prefixing the folder name with my Mac Mini’s hostname. In an instant I could recognise what had happened and weren’t surprised. It seems that the gentleman who posted the article didn’t get the same nice experience that I did. I won’t speculate on that particular case, but had Apple said something about this topic loudly and clearly as they geared up for the official launch, people wouldn’t get startled and maybe some would have thought things through before upgrading.

To my mind, this is a perfect illustration of the kind of problem I was referring to when I wrote the following post: When the UX interaction can be too subtle.

Maybe this article won’t go anywhere, many having got a good experience. Or maybe, since the twitterati is always ready for some outrage-tweetstorm, there will be plenty of chatter and not just compliments to go about. We’ll see. Funny enough, I was actually pondering this weekend, the apparently lack of something-gate to do with MacOs and iOS10 in social media. I didn’t have to wait for too long. 😉

The original article is here, exactly as it’s title reads:

maybe be careful with osx sierra”

Four days with my Upgraded MBP to MacOs Sierra, a few minor tweaks, but so far no complaints.

Yes, I could’ve waited a couple more days. But had I wanted to wait a couple of days, then I wouldn’t actually want to upgrade anytime this year. The reason is simple, anything that still breaks in a significant way right now will probably not see a good fix until a few months down the line. This has been my previous experience. That’s why I decided this time round, if something should fail to install or run then I’d look for a Docker container and not bother about it much.

Docker

Yay! It runs without me doing anything at all. I simply let the upgrade run its course and the restart all happen, when that was all done then Docker continued to work as before. Now I can’t recall, I might have downloaded Docker for Mac once again. In any case, I never tend to remember technical things that went to plan. This flawless upgrade gave me the assurance that my fallback plan, running things in containers, is going to work.

Here is what I did to keep it humming.

Scala and Java

There was an issue with TLS, I fixed it by pointing the JVM to the new location of the CA Certs with this setting:

-Djavax.net.ssl.trustStore=/Library/Java/Home/lib/security/cacerts

I did this for both JAVA_OPTS and SBT_OPTS, then I could run Java apps again.

IntelliJ IDEA

If this wouldn’t work then I might seriously consider downgrading. But nope, I didn’t have to downgrade, it worked first time. The one issue I run into was with running a Java app that tried to establish a TLS connection to the outside world. This issue was resolved as per the previous fix. I run all the JetBrains tools that I use, AppCode, IDEA, DataGrip, and so on, no problems with any of them. So This meant that whatever breaks from this point on will be re-homed in Docker and I won’t deal with it.

Ruby

I’m using RVM, as expected it complained and wouldn’t install or re-install a Ruby binary. I checked the version that ships with MacOS Sierra, it was good enough for me, ruby 2.3.0p0 (2015-12-25 revision 53290) [x86_64-darwin16]. so I settled with that. Here is what I had to get RVM and the ruby stack going:

  • run, to get the command line tools installed (shouldn’t have to, given that I have Xcode installed, but I was at this spot before and didn’t want to spend any time on it:
  • $ xcode-select --install
  • Switched to System Ruby distro
  • $ rvm use system
  • Restored the install gems to pristine, then everything worked from there.
  • $ gem pristine

Haskell Stack

GHC installs fine, but Haskell Stack doesn’t. I run brew install ghc, but I gave up on Haskell Stack as it just wouldn’t succeed. I run out of my budgeted time, so I stopped trying that.

Rust

I got a few misleading error messages from this, but it turns out the real culprit was a prior failed installation via brew. I run brew doctor, looked for and removed all libraries that obviously look related to rust. You can’t miss them. Once I was done with this, I just built Rust from source and installed it. I haven’t used it in any anger as yet, but it seems to work fine now.

GO

Not a single problem. I didn’t even have to touch it, it worked as it always did.

Homebrew

I run brew doctor, to find out what was broken. I found out I had a partially installed Rust version. Rust would no longer install nor uninstall. I solved that the hard way, got rid of all related libraries as reported by brew. After that everything was working fine.

My daily tools

I found out I actually use lots of tools, none of them showed any issues. Omnigraffle, Sketch, Android Studio, Atom, TextMate, Parallels Desktop, Postgresql and a half dozen other command line tools, not a single glitch with any of those.

Forward with Sierra

This is the chance to do a clean slate on one topic that I’ve had in mind for a while. If you follow this blog, you’d remember my statement about containers providing a new chance to compartmentalising software components. With MacOs Sierra, this is my chance to pursue this idea. I will stop installing development components via Homebrew. Instead, I’ll look for Docker containers. This is what I plan to do for everything like DBMS, ElasticSearch, and other cluster native stuff like Riak, Spark, and so on. Basically, it no longer makes sense to install these locally, it’s best to use a container-based cluster manager as that is the only likely use for them.

Closing notes

Please use your own judgment, decide if you want to risk installing a bleeding edge software on your production machine. Remember that Leslie Lamport statement that I often paraphrase, no two computers will likely have the same state. This is important because the slightest difference, and there will be a lot of them always, could result in different behaviours between two seemingly equivalent environments. If nothing else, ensure you have backups that you can verifiably restore your machine from, before attempting this kind of upgrade. If you still go ahead and break your machine, don’t say that I didn’t warn you.

That being out of away now, if you decide to give it a go and it goes well, you might enjoy the experience. Since the upgrade I noticed that I actually gained about 40GB of additional free space. And this is probably thanks to a feature that promise to move rarely used items off of my computer and on to iCloud. Yes, you guessed right, I have an iCloud account.