Getting back into it

May 9th is Grocery's 6th birthday, and there's a lot to celebrate. The app has almost a million downloads and continues to be used by more people year over year. I'm so thankful to everyone who has tried the app and taken it shopping. Grocery has surpassed all of my original expectations and continues to be the project that I am the most proud of.

Usually a Grocery blog post would be all about a new update I'm shipping to make the app better, but not this time. I am actually working on several updates to Grocery, so keep a look out for those coming soon. But I wanted to take some time to reflect on a different kind of story about the updates I haven't shipped, and why that’s been especially challenging for me as a solo app developer.

On January 16th I underwent a spine surgery procedure known as an Anterior Lumbar Interbody Fusion (ALIF). If that sounds familiar, it’s the same surgery that Tiger Woods had a few years before his most recent victory at The Masters. I underwent the surgery as the last resort to try and resolve chronic low back pain that I’ve been dealing with for many years.

Dealing with chronic pain is extremely challenging and confusing. I really didn’t know what to do when my lower back started to hurt five years ago. I tried stretching, yoga, and physical therapy. I tried improving my posture. I tried a standing desk. Back pain is extremely common but it’s usually mercifully temporary. But in my case it just got progressively worse until it was too painful to sit in any kind of chair for more than an hour.

I wish I knew what the cause was. There’s no specific injuring moment I can point to that caused my pain to start. When a runner tears their ACL there's no mystery involved. They feel the exact moment that the injury occurs, and there’s a clear path to follow immediately afterwards to resolve the issue. But you don’t expect to tear your ACL sitting at your desk, and I certainly couldn’t imagine doing anything to injure my back severely enough that I couldn’t sit anymore.

As it turns out though, my back was injured. The lowest lumbar disc was essentially gone, and two of my vertebrae were no-longer aligned. That caused my spine to be unstable and arthritis was setting in. That’s not a great situation to be in for anyone, and it was rendering all of my other efforts to relieve the pain moot.

I figured all of this out last year, when the chronic pain became effectively permanent. Sitting at my desk for more than an hour was extremely uncomfortable, which was starting to make things harder for me at work, and was impacting the time I could spend working on Grocery too.

It’s safe to say that my pain was affecting nearly all aspects of my life, but where I felt the loss most acutely was in my work. Even the temporary loss of something that has become your life is tough to deal with. I know that for athletes who are injured, their sport and their training are what they miss the most. For me as a software engineer who is injured, I know that sitting and feeling productive writing code is what I miss the most too.

Back pain and spine surgery can affect anyone in any field, but I haven't seen many people talk about what it's like to deal with as a solo app developer. I could write a book about how back pain has affected other aspects of my life that nearly anyone could relate to. But I think Grocery is the most interesting part of my recovery journey because it’s something that I can’t step away from; when I do step away from Grocery there’s no one else who can step in.

There’s a lot of great things about being the solo developer of an indie app. You are in the ultimate position of flexibility. You get to build exactly the app you want, with exactly the technology you love, on whatever schedule you choose. You get to wear all the hats, write every line of code, and enjoy all of the satisfaction knowing that people love using the thing that YOU made!

But there are some less great things about being a solo developer too, and a big one is that when you’re injured, having major surgery, and recovering from surgery, no one else is developing your app for you. I think that it’s pretty common for solo indie developers to take breaks from writing code - no one should be writing code nonstop all of the time anyways. But I think that those breaks are usually because it’s just time to do something else for a while, not because of an injury that forces you to step away entirely.

When my back started to get pretty bad last year I was in the middle of Grocery’s biggest release ever: 3.0. I think it’s worth celebrating that I actually did ship such a major release for Grocery while dealing with all of this, because it wasn’t easy. A lot of my time working on the app was done laying down on the couch or sitting in a chair with an ice pack to reduce the pain in my back. I was motivated to finish the work though because this release was a big deal. The complete design overhaul and new features really felt like they put the app in a great place. Seeing that work taking shape gave me the motivation to push through the pain and get it done.

One of the key features of 3.0 that I added at the very end was a completely re-written User Guide. Looking back, I think that was the most important feature of the release, and going forward will continue to be a major area that I invest time in. While I was working on 3.0 I spent a lot of time fixing bugs that users had reported, and trying to resolve confusing user experience issues that I heard about in emails. I tried to capture anything that I thought users would need to know about the app in the user guide so that people would have a place to go to look for answers if I couldn’t respond to support email.

That would prove to be prophetic, because it’s been very difficult for me to respond to support email during my surgery recovery period. The way that I usually handle support email for Grocery is that I’ll take a Saturday or Sunday here and there to respond to everything I’ve received in the last few weeks. That takes a few hours though and I just couldn’t spend that kind of time in front of a computer after my surgery. As a result, my support responses have been much slower and I know there are questions that my users are waiting for answers on. But because I spent time filling out the user guide, I know that there is at least a place for users to find answers to most of what they should need to be able to use the app. It’s not a replacement for responding to support email, but at least email isn’t the only avenue for users to get answers.

If I had to do this whole surgery thing over again though, I would probably look into hiring someone to help part time with support. It’s a tricky problem because some support requires code level knowledge that another person wouldn’t have. But it would be better than not being able to do anything myself, which is where I was after surgery for several months.

Throughout this period I’ve had to remind myself that the reason normal people are using an app is for what the app does right now. People aren’t looking at my roadmap and deciding if they should use Grocery based on that. I feel extremely thankful that the app was in a pretty good spot before I went under the knife, but that doesn’t stop me from feeling guilty and sad that I’ve made no progress on the improvements and new features I want to work on.

There’s a physical side of combating any injury and there’s also a mental side. They are equally important and in a lot of ways the mental part is the toughest to handle. When I’m sad about the lack of progress that I’ve made on Grocery during this period I’ve found it helps me to focus on _why_ I am doing this. Why am I working on this app in the first place and what am I hoping to get out of it?

I recently read a support email from a husband and wife that love using the app to shop. I recently used the app myself to plan and cook a whole meal of enchiladas from scratch for my partner and family (the first time I’ve done that since before surgery). The love for Grocery is still there because I know it has a place in my life and the lives of many others.

Recovering from an injury requires a singular focus on getting better. A dear friend has been helping me focus on each of the incremental milestones I achieve during recovery. The first time I walked around the block, made my own meals, drove a car, went to a restaurant, watched a movie out of bed, played a computer game, and went back to work. We call them medals. I didn’t earn very many during those first few weeks but over three months later and I’m earning a few every week. Progress is definitely happening and it’s critical to celebrate it.

I’m nearly back to where I was before surgery, and I have a lot of recovery left to focus on to get my back to where it needs to be. I think I’m also nearly ready to focus just a little more on the work that I would like to get back to with Grocery. I think that earning a few Grocery medals might just give me some of the motivation I need to prepare my mind for working on the app regularly once my body is ready to do so.

I’ve always liked to start small when I’m trying to get back into the swing of anything, so I figured my first step coming back should be something simple like fixing a bug. Thankfully, I just checked that one off. Two minor issues were reported via email, and both turned out to be very easy fixes. Medals awarded! First Grocery code changes, bugs fixed, and support emails responded to since Surgery 💪.

I’m hoping to continue checking off a few more smaller medals while working towards some larger ones: clearing out my support queue (earned 🥇), adding a small requested feature (earned 🥇) and shipping my first bug fix update after surgery (coming soon 🏆). I don’t have a timeline for the release but I know that I’ll feel really good about completing it when I do.

I’ll be looking for other ways to build excitement and motivation along the way. The biggest excitement booster I can think of has always been WWDC in June, and I won the ticket lottery so I'll be there in person. If you're a developer reading this and you'll be in town, reach out! I would love to say hi. I've always come home full of excitement after WWDC and I think it'll be just what I need this year too.

This injury, surgery, and recovery is up there with any major challenge I’ve faced in my life. This experience has been a great reminder that there are some things you just can’t do alone. I know that I have benefited greatly from the love and support of my family and friends, and honestly don’t think that I could have recovered without them. I’ve needed to be realistic with myself that I can’t do all of the things I would like to, yet determined to progress back to a place where I can do those things eventually.

That I think is the key for solo developers in a situation like this. Be patient with yourself and deal with what life is throwing at you, while persistently knowing that your app will be there for you when you’re ready.

Blocks, Operations, and Retain Cycles

There's been some great discussions in the iOS community lately about the pitfalls of Objective-C and things to watch out for while developing iOS apps.  One of our projects at Mutual Mobile recently encountered a very difficult to diagnose issue that I wanted to describe so that hopefully others can avoid this happening to them.  The issue involved leaking images which resulted in a memory pressure warning and subsequent crash.  We knew what the symptoms were and how to reproduce it, but the root cause of the leak was extremely hard to pin down.  It used to be the case that most memory management issues were the result of programmer oversight.  But as you will see below, this issue would be extremely easy to overlook which is why I felt it was important to share a detailed explanation of it here.  

 

You can't start off a conversation about memory management anymore without mentioning Automatic Reference Counting (ARC).  Apple knew that managing memory is a big deal and that a lot of people in the iOS community were doing it wrong.  At WWDC in 2011, they mentioned that around 90% of the crashes on the app store were due to memory management issues.  ARC was their attempt to combat that.  Essentially ARC synthesizes calls to retain and release for you so that you the developer can focus on actually building your app.  It seemed too good to be true at the time, but I think I can safely say now that ARC has been a huge success.  It's now extremely easy for us to manage memory in Objective-C because, well, we really don't have to do anything.  As long as we follow the simple conventions the compiler takes care of the rest.  It really does work quite well.

 

There are a few gotchas with ARC though.  For starters, ARC doesn't really apply to C references (though it can help with converting from C to Objective-C, but that's another topic).  You do have to deal with C in iOS projects, because several libraries like CoreGraphics have a C interface.  Instead of method calls to retain and release, C libraries include retain and release functions, such as CGContextRelease(context c), to manage their memory.  ARC does not synthesize those for you, so you still have to call them.  CoreGraphics can be a tough framework to use, so my first thought was that the issue would be somewhere in this layer.  But it didn't take long to discover that this wasn't the case.  All it took was reading the code.  All of the calls to retain and release were balanced out correctly where the images were created.  Here's an example below taken from the method that we thought may have been at fault:

 

    CGContextRef context = CGBitmapContextCreate(NULL, target_w, target_h, 8, 0, rgb, bmi);

    

    CGColorSpaceRelease(rgb);

    

    UIImage *pdfImage = nil;

    

    if (context != NULL) {

        CGContextDrawPDFPage(context, page);

        

        CGImageRef imageRef = CGBitmapContextCreateImage(context);

        CGContextRelease(context);

 

        pdfImage = [UIImage imageWithCGImage:imageRef scale:screenScale orientation:UIImageOrientationUp];

        CGImageRelease(imageRef);

    } else {

       CGContextRelease(context);

    }

 

So as you can see, at the top of the method the context is being created, so by convention it has a retain count of +1.  Then the context is used to draw an pdf into it, which is then copied out to an image so that it can be returned by the method.  The image ref also has a retain count of +1, because that's how the creation convention works.  It is then the developer's responsibility to release those references, which you can see is done in this example.  So far so good.  We knew from instruments that the image context was being leaked, but we still didn't know how or why.  But since we could tell both from our own inspection, and reading Apple's sample code for this exact use case, that it was being done correct, we went back to the drawing board.

The other main gotcha that remains with ARC is the retain cycle.  A retain cycle can take a few forms, but it typically means that object A retains object B, and object B retains object A, but nothing else retains object A or B.  They are still holding on to each other, so they never get dealloc'd, but nothing else is holding on to them so they both leak.  A lot of the time this can happen with a block, where the block retains the thing that created the block, and never gets released, so that the thing that created the block never gets released either.  That's clearly a problem.  It's also a problem that is tough to solve.  The static analyzer isn't well equipped to point these out to you, and Instruments isn't super effective at nailing them down either.  It will tell you something is leaking, but that's about it.  You do get a stack trace for the leak (or cycle if it detects one), but it's still up to you to pin down exactly what is causing the cycle.  You have to have a really strong understanding of how this stuff works and then do the detective work to parse through the code to find out what the cause could be.

In this case the team did understand retain cycles and did a lot to prevent them.  The view controller in question was responsible for showing all of the images that we are dealing with here.  That view controller creates dozens of blocks to perform image conversion operations from a PDF to an image.  The concern initially was that perhaps one of these blocks was holding onto the view controller, which holds the view hierarchy...and so maybe the view hierarchy isn't ever getting released.  There were indeed a few places where a block could have been "capturing" the view controller, and we took steps to prevent those.  Here's how you typically address that issue, by changing any reference to "self" to be a weak reference:

    __weak typeof(self) weakSelf = self;

    

    void (^ loadThumbnailBlock)(NSIndexPath *indexPath) = ^ (NSIndexPath *indexPath) {

        Page *page = [weakSelf.fetchedResultsController objectAtIndexPath:indexPath];

        [weakSelf loadThumbnailForPage:page forIndexPath:indexPath];

    };

    

So we eventually proved that the view controller was always being released, dealloc'd, and that all of it's views were going away.  Still the problem persisted, so that was still not the root cause.  The view controller was also handling the memory pressure warning correctly.  Whenever the app received the warning that it was using too much memory, it would remove the images it wasn't displaying from memory, as well as unload any views it wasn't using.  This was only a small fraction of the memory that had already been leaked though, so this only delayed the inevitable.

 

Finally, we nailed it down though.  We knew that the image contexts were being leaked both from checking instruments and because that's the only thing that could eat up that much memory.  The images are being rendered in blocks that are kicked off by the view controller.  We knew now that the view controller wasn't being held on to, because it was being dealloc'd, and so thus it also couldn't be holding onto the blocks that were rendering the images.  So something else must be holding onto those blocks...

Enter NSOperationQueue and NSBlockOperation.  NSOperationQueue and NSBlockOperation are built around Grand Central Dispatch to provide conveniences such as the ability to "cancel" an operation.  We were using this convenience to allow the app to cancel image conversion blocks if you closed the view controller before they finished.  Makes sense right?  Well, it was actually this optimization that was killing us in terms of memory leaks.  Take a look at the block of code below:

    __block NSBlockOperation *operation = [[NSBlockOperation alloc] init];

    __weak typeof(self)weakSelf = self;

 

    MMVoidBlock thumbnailOperationBlock = ^ {

        if (!operation.isCancelled) {

            workerBlock();

        }

        

        [weakSelf.thumbnailOperationList removeObjectForKey:key];

    };

 

    [operation addExecutionBlock:thumbnailOperationBlock];

Notice any problems?  In this case we are building a thumbnail operation block that in turn calls a worker block.  The workerBlock() is actually what goes off and renders the PDF into a graphics context, converts that into an image, and saves the image.  But take a look at what else it's doing.  The block has a reference to the operation.  That sounds fine, until you look at the last line of that snippet.  The block, which holds a strong reference to the operation, is then being added to the same operation.  That addExecutionBlock method is going to retain the block, so now we have ourselves a retain cycle.  The block is holding onto the operation, so the operation won't be released.  But when the operation finishes the queue is going to release the operation, so now the operation has leaked because we don't have anything that holds a reference to it.  But the operation also has a reference to the block, so the block is never going to get released.  And finally, the block has a reference to the image and graphics contexts, which will now never be released either.  All because that silly block captured the operation it was added to.

Now the plot thickens even further.  The way you would typically solve this problem is by declaring the variable you want to use in the block as being weak by using the __weak specifier.  The example above inside the view controller illustrates that method.  But in this case, we can't do that because it will fail with ARC.  If you were to change __block operation to __weak operation in the example above, the operation would be released immediately.  So the performance optimizations of ARC bite us badly here too, because the operation will be nil in the last line of this function, which will cause the app to not work at all.  The compiler knows this and will actually warn you not to use __weak there.  In this case, what the compiler is telling you to do is actually the wrong thing to do, which is why this problem is so hard to solve.  By being a good sport and doing what the compiler tells you, you are lulled into a false sense of security that you will not have a retain cycle.

 

Here is the actual solution below.

 

    __block NSBlockOperation *operation = [[NSBlockOperation alloc] init];

    __weak typeof(self)weakSelf = self;

    __weak typeof(operation)weakOp = operation;

 

    MMVoidBlock thumbnailOperationBlock = ^ {

        if (!weakOp.isCancelled) {

            workerBlock();

        }

        

        [weakSelf.thumbnailOperationList removeObjectForKey:key];

    };

    [operation addExecutionBlock:thumbnailOperationBlock];

 

What we had to do is just make the operation referenced by the block into a weak reference.  That way, the block isn't holding onto the operation, it just has a weak reference to it.  It's worth pointing out that pre-ARC, __block was sufficient to prevent a retain cycle.  But with ARC, __block no longer carries the same meaning and __weak must be used to prevent a retain cycle.  When the operation is finished and gets released by the operation queue, it will go away.  That will then release the block, which releases the image and graphics contexts just like we see it's supposed to in the sample above.  That resulted in dramatically improved memory performance and completely normal memory behavior.  Literally just that 2 line fix there.

 

It is worth pointing out that, in some ways, this represents a possible bug (or questionable behavior) on the part of NSBlockOperation.  What appears to be happening is that when you add an execution block, it retains it.  So far so good.  But when the block executes, it does not release the block, which would have actually broken the cycle.  Instead, the block won't be dropped until the operation is dealloc'd because, for whatever reason, NSBlockOperation does not release its associated execution blocks until dealloc gets called.  What the reason for that is, I do not know, but that's just the icing on the cake for what is already a tremendously ridiculous issue.

 

There is always a risk when dealing with scarce resources and expensive operations if you are not careful about how you use them.  Retain cycles are preventable, but it takes a lot of thought to consider how they may be caused by the code we write.  This situation represents a perfect storm in terms of memory issues on iOS.  It's easy for retain cycles to go unnoticed.  This one became so critical because we were leaking images, not strings and numbers.  Leaking images will cause an app to get killed due to memory pressure, where as strings and numbers probably never will.  Dealing with retain cycles can be extremely painful, and we should be especially careful to avoid them when dealing with expensive objects like Images and other graphical assets.  In this case, even with extreme care it was still nearly impossible to avoid a cycle.  This was a really tough problem to solve and I hope you all will remember it and avoid it in the future.