The Aurigma Up 1.10 update was the largest one since its birth date and as many people experienced firsthand, it did not go well. A lot of end users complained that the app crashes on the launch while we could not reproduce the problem locally. We spend more than a week to figure out what is wrong. The app is on the way to the Apple's approval and meanwhile I am going to share what we learned from this incident.
Aurigma Up exists about 3 years. When it was first created, iOS API was quite immature. Aurigma Up used deprecated approaches and with each iOS update the code became looking like a spaghetti. Another problem is that each new version of iOS changes behavior in certain aspects, so it was a bit challenging to manage things.
Renovation
When iOS8 was about to be released, we decided to renovate Aurigma Up - update the user interface and refactor old code. We realized that we have to do too radical changes and that's why we spent almost a week to create test suites to cover most scenarios. It was really useful to catch bugs on the early stages and without them the product were way too more raw.
To test the product we use a couple of iPads and iPhones with different iOS versions installed + iOS simulators built into XCode. We felt quite confident that these devices cover most use cases because iOS platform is not so fragmented as Android. However unfortunately it was not enough.
After 1.10 release
After the product was released, we started receiving complaints from end users. As the problem could not be reproduced on our devices and we even did not have an access to websites where users open Aurigma Up, we did not have any ideas what might be the reason.
Several days later some of our paid customers contacted us regarding this problem which gave us a bit more information, but we still had to guess. These customers volunteered to be our beta testers and without their assistance we could not do anything about the problem. Using the opportunity, I want to thank everybody whom I contacted with!
Not a common bug but a new challenge
We never had this issue before. We were lucky enough to be able to reproduce whatever problem our customers reported. However in theory this problem should be familiar to almost every mobile app vendor. I have found the following:
- When an app crashes, it generates a crash report file.
- At the first glance the crash reports file gives as much information as a transmission from an alien spaceship from Mars:
{"app_name":"AurigmaUp","app_version":"1.10","bundleID":"com.aurigma.aurigmauploader","adam_id":432611633,"os_version":"iPhone OS 8.0.2 (12A405)","slice_uuid":"d26078ca-a516-3e37-8889-a46a3872d300","share_with_app_devs":false,"build_version":"1.10.2","is_first_party":false,"bug_type":"109","name":"AurigmaUp"}
Incident Identifier: 62800AD4-27BE-46E5-A18B-A81A5A802092
CrashReporter Key: 4bd35bc5476910aa9bfafc05b974e0833aaa616f
Hardware Model: iPhone7,2
Process: AurigmaUp [1235]
Path: /private/var/mobile/Containers/Bundle/Application/9F25E1EF-A46A-4708-B130-DB88D63045EA/AurigmaUp.app/AurigmaUp
Identifier: com.aurigma.aurigmauploader
Version: 1.10.2 (1.10)
Code Type: ARM-64 (Native)
Parent Process: launchd [1]
Date/Time: 2014-10-21 21:00:06.764 -0500
Launch Time: 2014-10-21 21:00:06.061 -0500
OS Version: iOS 8.0.2 (12A405)
Report Version: 105
Exception Type: EXC_CRASH (SIGSEGV)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Triggered by Thread: 1
Thread 0 name: Dispatch queue: com.apple.main-thread
Thread 0:
0 Foundation 0x000000018928e5b0 0x189264000 + 173488
1 Foundation 0x000000018928e594 0x189264000 + 173460
2 AurigmaUp 0x000000010004c9f8 0x10001c000 + 199160
3 AurigmaUp 0x000000010004b19c 0x10001c000 + 192924
4 AurigmaUp 0x0000000100023eb0 0x10001c000 + 32432
5 UIKit 0x000000018cc82140 0x18cc00000 + 532800
... etc ...
- Fortunately it is possible decipher the crash log, although it is not "just works" as you may expect from Apple.
- Gathering crash logs is a more serious challenge - although Apple allows sending them automatically to iTunes, it is turned off by default. Also, for some reason I saw only two crash logs in my iTunes Connect account which may mean that this functionality is not well done.
Another problem is how to send a version which is presumable fixed to a beta tester to check if it helps. In the past we used to use TestFlight service. However recently Apple acquired them and integrated them into iTunes Connect and completely reworked the beta testing process.
In theory it should become much more simple and "just work", but again, on practice it is not that smoothly. It does not work for iOS7 users. You have to add beta testers as employees (or "submit beta version for review" which does not seem to make a lot sense). Many beta testers don't receive emails from iTunes Connect. Et cetera. So it looks like this acquisition was very good for TestFlight shareholders, but not their customers.
Solution
Fortunately there are a number of TestFlight alternatives. A great news is that all of them offer a solution for the crash report gathering (and even deciphering). As the app crash is a really critical issue and we were under a pressure, I could not afford to spend a lot of time choosing between the best service and overcome the steep learning curve. So after reading few comparison articles, I decided to stick to a Crashlytics.
The unexpected obstacle was the invite system used on Crashlytics. You cannot just register there, you leave your email address and wait. I did not know how long to wait and contacted them to understand the status of my request.
Fortunately, guys from Crashlytics responded quickly and expedited things up. My request was approved in the end of the business day and on the next day by the lunch time I have finished to set it up. Few hours later the version 1.11 has been submitted for review to Apple and all we need is to wait several days (last time they approved it within only 6 days).
Conclusion
As a conclusion, I would like to highlight one important thing. The version 1.11 is just yet another quick fix. Integrating Aurigma Up with Crashlytics answered a question which kept me on the alert - what if my fix helps only my beta testers, but other users have some other reason of the crash? Since Apple are not very fast approving the app to the AppStore, I cannot afford making a lot of tries.
With the crash reporting system I will be able to know the reason of the crash even before the user starts dialing our support phone number. So we will be able to address the issue before it becomes really critical.
That's why if anyone is disappointed with Aurigma Up 1.10 stability, let me assure you that since 1.11 things will change. Now you can be more confident in Aurigma Up than ever!