Tweaking a web server with Nginx

A few weeks ago, I started playing with nginx as a replacement for Apache. The exact reasons I decided to do this are a bit unclear, but I think it had something to do with me playing with node.js and MongoDB. The initial setup was pretty smooth and the configuration files are much easier to understand than Apache for my sites. However, I also decided to offload the PHP processing to php-fpm and that’s where the problems began. My VPS is pretty lightweight and doesn’t have a lot of RAM. My blog was running out of memory all the time and php-fpm stopped responding.

After a lot of tweaking to reduce memory usage, I managed to stabilize the system and get much faster response times for my blog; it doesn’t get a lot of traffic, but I think I’ve done a decent job with performance. If I ever get hit with a massive load, I’m sure my server will keel over.

Is there a real reason to use nginx vs Apache? I’m sure some people will say yes, but for me the jury is still out. Things are going smoothly, but was it worth the trouble? I have no idea.

Thoughts on Storyboards

When Apple came out with Storyboards in Xcode, I thought that they would be quite useful in making it easier to see how an app was laid out. However, after attempting one project with them, I determined that they had limited utility as they wouldn't scale and wouldn't work well for multi person teams. I left storyboards for a year and didn't revisit them.

I started another project about a year after my first experience and decided to give it another go. This project was a small project and I was the only developer, so it was well suited for storyboards. The project grew and grew. Storyboards worked OK, but the layout in Interface Builder was almost too hard to manage. I finally decided to see about splitting up the storyboards into multiple storyboards.

The app was a tab bar based app, so the logical breakdown was one storyboard per tab. It only took about 15 minutes for me to do this and undo a poor initial decision. However, there were still one or two links between storyboards that were easy to fix in code. Some peopl have come up with ways to link storyboards like RBStoryboardLink which looks interesting, but I haven't explored it.

Multiple storyboards seems scary at first, but it is worth doing to keep difference parts of an app more contained and easier to visualize.

I haven't used XIBs in about a year and I'm not sure there is a good reason to use them as storyboards are easy to work with and make it simple to use custom UITableViewCells and UICollectionViewCells.

Code Signing Woes on Xcode Server

Following up on my post from yesterday, I’ve discovered that Xcode Server does something very strange in the build process. After the build, it creates a .ipa file that is suitable to install on an iOS device and a .xcarchive file that contains the .dSYM file as well as the .app file which is the .ipa file before it has been packaged up. The packaging process is basically putting the .app file in a Payload directory, zipping it up and changing the .zip extension to .ipa. Sounds simple enough and unzipping the .ipa should yield a file identical to the .app. Unfortunately this is not the case. If you run the following command:

codesign --display --entitlements -

on the .app in the xcarchive and on the .app that is extracted from the .ipa file, you will get different results.

From the .ipa, I get:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

from the .app, I get:

??qq?<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

It appears that the code signing entitlement, aps-environment is missing from the .ipa Xcode Server built. I can’t figure out why this is the case, but it does show that I’m not crazy and my setup works. The .ipa I create myself and upload to TestFlight works because my simple packaging script doesn’t re-codesign the .app.

So I have 2 choices when uploading to the app store:
1. Do what I described yesterday and that is resign the app on my desktop machine

  1. Take the .app file out of the xcarchive, zip it up and upload that.

Xcode Server does mangle the product and I’m not sure that is intentional. I should file a bug with Apple to see if they can fix it in the future.

I’d love to be proven wrong on this whole setup. If you have more information, please let me know.

Using Xcode Server for Builds

At last year’s WWDC, Apple introduced Xcode Server for doing continuous integration. Lots of people have tried to use it with mixed results. Recently after dealing with our build server for work (we use Jenkins), I decided to give Xcode server a try for one of my projects.

I was hesitant at first to use OS X Server (which includes Xcode Server) as I’ve had poor results with it in the past. It intrigued me often that I decided to install server and see where it went. OS X server install was a breeze and so was setting up Xcode Server.

I created a “Bot” from Xcode on my development machine and saw that it failed when the server tried to build it. No problem, I searched the Internet and all the results said to add the repository from the server instead of when creating the bot. Perfect, I did that and then the build succeeded. Yeah! I also added my Team to Xcode server so it could pull down provisioning profiles and create a certificate to build.

I checked the log and found 2 problems. First, I have a run script build phase that puts the version number in the Settings.bundle that failed and second, it was being signed using the wrong mobile provisioning profile.

I decided to tackle the second problem first. Through my searches, I found that I needed to put the mobile provisioning profile (an AdHoc one) in /Library/Server/Xcode/Data/ProvisioningProfiles. No problem. Next I also found that I needed to put the certificate and private key for the profile in the System keychain as the Xcode Server runs under _teamserver which doesn’t have its own keychain. I did that as well and found the problem didn’t go away. I started thinking about it and found that sometimes Xcode ignores the Automatic Distribution Code Signing Setting. So, I removed my Team and deleted all the other mobile provisioning profiles. Bingo, now my app was building and being signed correctly.

Next step was to figure out my build phase. After dumping the environment variables, I found that in order to put files in the .app after the build, I had to set my script phase to be:

shortversionnumber=`/usr/libexec/PlistBuddy -c "print CFBundleShortVersionString" ${BUILT_PRODUCTS_DIR}/${INFOPLIST_PATH}`
bundleversionnumber=`/usr/libexec/PlistBuddy -c "print CFBundleVersion" ${BUILT_PRODUCTS_DIR}/${INFOPLIST_PATH}`
shortversionnumber=`echo $shortversionnumber '('$bundleversionnumber')'`

echo ${shortversionnumber}


echo $dest_file

/usr/libexec/PlistBuddy -c "delete :PreferenceSpecifiers" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers array" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:0:Title string About" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:0:Type string PSGroupSpecifier" "$dest_file"

/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:Key string VersionKey" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:DefaultValue string ${shortversionnumber}" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:Title string Version" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:Type string PSTitleValueSpecifier" "$dest_file"

/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:2:File string Acknowledgements" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:2:Title string Acknowledgements" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:2:Type string PSChildPaneSpecifier" "$dest_file"

The key was the ${BUILT_PRODUCTS} variable.

So I was almost done with the build server. I setup 1 bot for the master branch that builds when code changes and another on the build branch that also builds when code changes; however, I only push to build when I’m ready for a build. The build branch also had to upload to TestFlight, so I researched that and came up with very convoluted solutions.

I created a second scheme for the TestFlight upload and set my build branch bot to use that scheme. Then I added a Post Action to the Archive phase that looks like this (make sure you select Provide Build Settings From “Your Target”):

rm -rf ${IPA}
/usr/bin/xcrun -sdk iphoneos PackageApplication -v ${ARCHIVE_PRODUCTS_PATH}${INSTALL_PATH}/${WRAPPER_NAME} -o ${IPA}

RELEASE_NOTES=`cat ${SRCROOT}/CurrentRelease.txt`
echo "*** Uploading ${IPA} to TestFlight ***"
/usr/bin/curl "" \
-F file=@"${IPA}" \
-F api_token="${API_TOKEN}" \
-F team_token="${TEAM_TOKEN}" \
-F distribution_lists="${DISTRIBUTION_LISTS}" \
-F notify=True \
-F notes="${RELEASE_NOTES}"

echo "TestFlight upload finished!"

I thought I was all done; builds worked and were uploaded to TestFlight. The moment of truth came when it was time to upload to the AppStore. I have push notifications enabled in this app and it is properly provisioning. After the upload, I got email saying that the ape-environment key wasn’t in the entitlements. Very strange, I checked and rechecked the build I got out of Xcode Server and it was correct. I tried a number of things and in the end sort of gave up. I took the archive that the build server created, put it on my main machine, imported into Xcode, exported for AdHoc distribution, resigned it using the same profile it was already signed with and submitted it. For some strange reason, this worked. I can live with that as I don’t submit to the app store all the time.

Things to note

  • You do NOT have to create an AppStore distribution profile. You can submit the AdHoc build to the AppStore without problems. This is key as it lets you test the exact version going to the AppStore without rebuilding or resigning.
  • You have to manually move your mobile provisioning profiles for AdHoc builds to your build server.
  • You have to manually add your private key and certificate to the System Keychain on your build machine.
  • Be careful about your build scripts as the paths are different on Xcode Server.
  • You have to add your repositories from Xcode Server and not from the bot you create.
  • Read the logs carefully when there is a problem.
  • Apple appears to have created Xcode Server for testing and not necessarily for release builds; I hope some of this changes in the future. I’m not a huge fan of unit tests, so I may just be abusing the server by doing what I’m doing.

Overall, I’m pretty plea with having my own continuous integration server. It will catch issues where I have something different from Debug builds to Release builds. In addition, it will upload to TestFlight along with the release notes saving me time. I am a little disappointed about having to resign for the app store, but it isn’t a huge deal. I could probably modify my script to do that during the archive, but I really don’t like having to hard code in the ID of a mobile provisioning profile. I could probably check the profile into source control and reference that, but I’m OK with this solution. As I already want to move the archives to my main machine, the extra signing process isn’t a big deal.

I know that there is a lot here, but hopefully it helps someone else in the future (that person may be me when I forget what I’ve done!).

Flaws in RESTful response codes

Anyone that uses or develops APIs is familiar with REST. It is widely used and by some developers considered superior to other ways of doing APIs. Since REST was developed alongside the HTTP 1.1 spec, HTTP status codes are generally used for responses to REST calls.

To me, it seems that using the standard HTTP response codes (a nice list is given here) is overloading a standard server response with the response of a query. While the spec says that for client errors (400 series errors), the response should contain the reason why in the response. I’ve looked at a lot of APIs and the best ones at handling errors return the error response inside of a successful 200 HTTP response. For instance, if you are creating an API for dealing with support tickets and you ask for a ticket ID that doesn’t exist or you don’t have permission to access, returning a 400 series error generally tells the client that something bad has happened when, in fact, the error is kind of minor. Embedding the error in the response seems like a better way to go.

The problem, however, with embedding the error in the response is that the API developer has to come up with his own error codes and convey that to those that use the APIs. Many developers that implement APIs treat anything but a 200 series response as a complete failure and give up. So, the API developer has to make sure that the client implementors properly handle the responses. In general, I find it easier to have one set of parsing code and deal with the errors inside of a successful response.

My argument, of course, is not very compelling as it is simply to swap one way of handling responses and errors with another. Last night, however, I came up with a potential argument that leads credence to what I’m saying. If there is a proxy server or a load balancer in front of the server, it could potentially return an error code which the client may not expect. Let’s take the case where our support ticket server is behind a proxy server and the client requests a particular ticket. If the proxy server or load balancer is misconfigured and we make a request, the proxy server could return a 400 series error, even a 404 error. The client could interpret this as “the ticket doesn’t exist”. This may not be the case; the ticket could exist, but the proxy returned an error. A better way of handling this would be to use 200 series status codes when the request succeeds and embed the API error code inside of the response. If the proxy server returned a 400 series error, the client could always treat it as an error and wouldn’t have to figure out if the support ticket really existed or not.

In addition, the HTTP status codes are not granular enough for many APIs. This would require the API developer to have a secondary set of error codes that would be returned with the 400 series error codes anyway. If that is the case, why not just create one set of error codes and always return 200 series success codes unless something major has happened?

I know that some of this may be confusing (it is crystal clear in my head), but it is worth some thought to consider the best way to handle response codes in APIs. Many people have opinions on this and may think that I have no idea what I’m talking about and HTTP response codes are perfectly fine for RESTful APIs.

Wrong definition of a pioneer?

The other day I was looking at a website of a contracting company that specializes in mobile development. The site said that they were pioneers in the field. The site then had as its next line that the company was started in 2009. I normally think of pioneers as people who initially settle an area or who are first in a particular field. Unless my history of mobile is a little off, 2009 is a bit late to the party. If we just consider the “modern” era of mobile development that began with the iPhone and Android, they are still late as the iPhone was released in 2007 and the SDK in 2008.

However, saying that mobile development began in 2007/2008 is doing a disservice to all the predecessors to iOS. PalmOS came out in 1996, the Newton came out in 1993, Windows CE came out in 1996 and the Psion was about the same time.

I have no idea what meaning of pioneer this company had in mind, but they are certainly not pioneers in mobile applications. I’ve been writing mobile applications since 1994 when I first got a Newton (the Apple Personal Interactive Electronics group which made the Newton licensed my NotifyMail program and sent me a Newton MessagePad 110 in exchange for the license) and I don’t consider myself a pioneer. I know a number of people that wrote mobile applications prior to this.

If the company considers themselves pioneers, what do they call their developers that have 1-2 years of experience? Experts?

The wrong way to use asserts

One of the ways programmers use to help ensure that inputs to methods are correct is to use asserts. Basically these will test conditions and stop execution if a test fails. By default, Xcode suppresses these asserts in release builds so that the code will simply continue.

This past weekend, I was helping out on a project and found that the app stopped because of an assert indicating that a condition was not satisfied. As I looked into it closer, the programmer had put in asserts to test the results of what came back from the server. The author then didn’t do runtime checks to validate the data, he simply assumed that if the assert passed, that the data was valid and continued. The problem is that, in this case, the server returned data that the code wasn’t expecting, so in a release build, the code would have continued and then crashed because the data wasn’t in the correct format.

While I’m not a huge fan of asserts, they do have their place, just not here. The code should have done runtime checks and gracefully handle bad server data.

I suspect that the author hadn’t done enough client-server communication to know that servers return bad data more often than you’d expect. I’ve worked on so many projects where a simple server change causes clients to have hiccups, that I’ve gotten pretty good at defensively programming against this.

If you choose to use asserts, make sure that you also use runtime checks and gracefully handle conditions that you test for in the asserts as the asserts go away in release builds.

Bluetooth Knowledge Followup – Apple Radar Filed

In my last article, I wrote about an issue I had with my Bluetooth heart rate monitor and my Bluetooth headset. After analyzing the issue and trying to reproduce the issue, I was unable to do so in my office and then on my next run, everything working perfectly. However, on my run this morning, the problems came back. When I got home, I turned off RunKeeper and ran my test app. My test app either disconnected or I heard an audio dropout every 6 seconds. The big difference between my initial tests and this test was that my heart rate was up and the contacts on the heart rate monitor were conducting quite well. This would indicate that the issue is not a RunKeeper issue, but either an iOS issue or an iPhone 5 issue.

I’ve gone ahead and reported it to Apple as radar://13004711. Based on my past history with Apple bug reports, I don’t expect a resolution any time soon. This would be quite unfortunate as more and more Bluetooth devices come on the market.

Using my Bluetooth knowledge

In November, I saw that TI announced a product called SensorTag which is basically a development board for working with Bluetooth 4.0. I was immediately intrigued as iOS 5.0 supports Bluetooth 4.0 (LE) and lets me write apps that aren’t certified by Apple via the Made for iPhone program. I ordered the device (it was $25 which I consider a bargain) and then in early December, it arrived.

While I had no real use for it, I decided to learn how to write code to use Bluetooth LE and wrote a app that monitors all the sensors in the SensorTag and reports them. The TI app did this, but wasn’t very elegant and I also wanted to add alarms for high/low temperature. A few weeks later (I worked on it an hour at a time in the evenings), I finished my code and had a working app. Since I don’t have any plans to monetize my work, I’ve made all the code available under an MIT license. One of the comments I’ve received about my work is that my code is better than the code provided by TI! It might be that I’ve been writing Objective-C and iOS code for awhile or my goal was to produce an app that anyone could use. In any case, I may release it to the app store, but the audience is quite limited as it requires the SensorTag which is basically a developer/hacker tool.

Now, where was I going? With all this Bluetooth 4.0 knowledge, what should I do next? I wanted to learn something new and I did that; I likely won’t make any money from my new found knowledge, but it is fun playing around with it. However, today while I was running, I determined what I could do with the knowledge.

Let’s rewind a few months. When I run, I take my iPhone, use a pair of Motorola S10-HD Bluetooth headphones and a Wahoo Fitness Blue HR heart rate monitor. I use RunKeeper to log all the data. I have been using this combination successfully for many months and have been quite happy with it. When iOS 6 came out and I got an iPhone 5, things started going downhill. RunKeeper was no longer able to talk to my heart rate monitor and they blamed Wahoo Fitness as they use a Wahoo Fitness library to talk to the heart rate monitor (the library also supports ANT+ heart rate monitors via a Wahoo Fitness dongle). After a few app updates, RunKeeper appeared to support the heart rate monitor again.

After hurting my foot and then being sick, I started running again on Sunday. During that run, I noticed that the heart rate monitor reported accurate results via RunKeeper, but my music kept cutting out periodically. I initially blamed the Motorola headset as I’ve already had to return one pair due to the buttons failing. Today I ran again and noticed the same problem. However, I decided to count how often the drop outs occurred. The drop outs occurred every 6 seconds. For anyone doing sampling of data, this should ring a bell. Every 6 seconds is 10 minutes per minute which is a reasonable period to be sampling data. I took off my heart rate monitor, put it in my pocket and after a few minutes when the Bluetooth connection disconnected, my music continued without drop outs!

So now I knew the problem wasn’t necessarily with the Motorola headset, but there were too many variables:

  • Switched from iPhone 4S to iPhone 5
  • Went from iOS 5 to iOS 6
  • Updated RunKeeper

The only pieces of the equation that hadn’t changed since everything was working smoothly were the Bluetooth Heart Rate Monitor (it is a Bluetooth 4.0 device) and the Motorola headset. Having worked with earlier versions of Bluetooth, I thought that maybe there was a conflict between the Bluetooth devices. However, the devices remained the same and only the phone had changed. How was I going to track down the problem? Since I now knew had to write a Bluetooth LE app, I decided to write an app to talk to the heart rate monitor and get readings. This turned out to be a simple task and took about an hour to write the app. I ran the app and played music through the headsets; I was expecting to hear dropouts, but no such dropouts occurred. So, this little test told me that the iPhone 5 with the Wahoo Bluetooth 4.0 Heart Rate Monitor and Motorola Bluetooth headset could all be operating at the same time without problems.

So what next? I ran RunKeeper again and tried to duplicate my results from my run, but was unable to do so. Granted I wasn’t running, but I expected to have dropouts. I’ll go for a run tomorrow and see what happens. I’m a bit suspicious about the Wahoo Fitness library as Bluetooth 4.0 is quite simple to talk to the heart rate monitor. I turned on the option for the device to notify (and not poll) the heart rate monitor and received an update every 2 seconds. So if the Wahoo Library was using the notify option, I wouldn’t have heard dropouts every 6 seconds.

I think that this issue is worth a look as the Wahoo Fitness library is doing something funky and that RunKeeper can directly talk to the heart rate monitor without using the library. If the library is polling, it should be changed to use the notify feature of Bluetooth 4.0.

Love/hate relationship with Cocoa bindings

When I started ReceiptWallet over 6 years ago, I wanted to use the latest and greatest Mac OS X technologies, so I used Cocoa bindings to make it easier to bring the UI to life. Bindings allowed me to write less code and have UI elements updated automatically based on what was selected and other variables. Bindings work great for simple cases, but once things start getting complex, bindings almost become a problem. Bindings are quite hard to debug as there is no one place where you can see all the bindings and one slight change to the code can cause a crash; tracking down the crash can take hours. I had to write custom debugging routines to examine the bindings and eventually shipped a stable product.

Fast forward 6 years. This past week I wanted to write a simple app and choose to use bindings again to link up the user interface. This app didn’t do much, but it took me hours to figure out bindings again discovering that I had forgotten almost all I learned about bindings and the documentation wasn’t much help. My biggest problem was that I accidentally setup a binding for an NSTableView in addition to setting up a binding for the NSTableColumn. Of course, nothing warns you of this and debugging is a nightmare. I’m sure if I started using bindings again I’d get better at them, but with as infrequently as I write Mac applications, I’ll probably struggle the next time I attempt to use them.

Is there a simple guide I’m missing? Should I abandon bindings completely? I have no idea.