Determination or insanity?

For the last week I’ve been tracking down a bug in my current project. I’ve spent just about every spare minute including the weekend trying to figure out why my code works in a test app, but not the main app. I rewrote a huge piece of some communications code to no avail. I tried threading the communications, putting in proper locks, changed how I processed the bytes, etc. Basically when I was receiving large amounts of data from a Bluetooth device, some data was lost in the transmission and the checksum was incorrect; for small amounts of data, this problem didn’t show up. At this moment in time, I believe I’m the only person in the world working with this combination of hardware and software which made using Stackoverflow useless.

A co-worker of mine kept sending me ideas to try as I was completely out of ideas. His latest idea was to put the files from my test app into the main app to make sure the project settings weren’t the problem. That didn’t solve the problem, so I started ripping out pieces of the code and came across some code we had put in for a demo.

All of the communications code that I was working on was using the External Accessory protocol over Bluetooth. The demo code we had in there was using Multipeer Connectivity which uses WiFi and Bluetooth to discover peers. Once I removed that code, all of my communication issues went away! Now I should file a Radar issue with Apple on issue, but I can’t reproduce it outside of the particular hardware I’m using, networking library I’ve written, and a few other pieces so a bug report would be quite lacking.

I must have spent 40+ hours on this one issue which has caused me to lose sleep and be a bit on edge all week. As failure was not an option and I had no place to turn, I had to solve this. I’m sure that others would have given up as 40+ hours is far too long to spend on a bug. For me, however, I actually spent 4 months full time chasing down a bug in a network router for a cellular base station; like this one, I did eventually discover the fix.

Lesson learned…be careful of using different communication libraries in an app.

Swift Lessons

As I wrote in my last post, I’ve been learning Swift. This has been interesting. Chris Lattner and his team have been working on Swift for years and have done something that is amazing. Not only have I learned a bit about programming in Swift, I’ve learned a few things about using Swift.

  • Swift isn’t ready for prime time. I’ve seen references to this from Apple.
  • There are lots of bugs in the compiler; SourceKit crashes all the time and auto complete dies.
  • The syntax, while flexible, is a bit confusing at times. For instance, I have a framework in my app called Vera. In some places I access classes of that framework using the Vera namespace, i.e. Vera.Device.Category.Audio to get an enum value and in other places, I use an array of Device. This is confusing and inconsistent.
  • In some places, parameters have names and others they don’t. While you can specifically require the parameter names, sometimes you don’t have to do that and sometimes they’re required.
  • While Swift is extremely flexible, it is confusing as anything once you start getting into wacky things go haywire. Take a look at this code with AnyObject and Unmanaged.
  • There are bugs in the optimization such that turning on optimizations breaks certain code (like the one referenced in the last bullet). I can’t remember the last time that turning on optimizations broke code that I didn’t discover until I tried to deploy an app.

I’m sure that things will get better, but at the moment more work needs to be done. I’ll keep poking with Swift and enhancing my app.

Learning Swift

When Apple announced Swift at WWDC, I knew that if I was going to keep up with the youngsters, I had to learn Swift. Unfortunately it has been a long time since I learned a new language; I learned Objective-C in 2000, I think. While I am capable of learning, finding the time to learn it and use it are getting harder and harder. However, I know that I didn’t have a choice.

I started reading Apple’s books on Swift and made it part way through when I got distracted. I’ve restarted a few times, but still haven’t finished. So I decided to take my automation app and completely rewrite it in Swift. While I could have re-used my existing code, I decided to do my implementation completely in Swift using a few open source libraries (there may have been better libraries to use, but the ones I picked seemed reasonable for now). While I’m still learning Swift, I thought it was a decent attempt to learn it. I’ve put my rewrite on GitHub for all to mock.

I’m going to keep learning and I think that reading the books will make a lot more sense now that I have a project under my belt.

Giving up on Xcode Server

Several weeks ago, I started having problems with my media center where recorded shows started stuttering in playback. There were a huge number of variables that could have caused it as lots of things changed, like an update to EyeTV, update to my HDHomeRun, Xcode update, OS X Server update, etc. While I’m still trying to identify the cause, I think I’ve narrowed down the problem to Xcode and Xcode Server.

Why am I blaming Xcode? Well, I went into OS X Server and looked to disable it. It wasn’t turned on and didn’t have Xcode selected. Since I hadn’t turned it off and had selected a version of Xcode, I knew something went crazy. I selected Xcode again, restarted the Xcode Server and then turned it off. I noticed that even after I did this, there were still log entries related to Xcode Server. Huh? If I disabled it, why was it still doing stuff? I found a blog entry which identified a way to completely reset Xcode Server.

Before I had completely disabled Xcode Server, I had noticed periodic log entries about the server doing stuff which wasn’t a complete surprise as it had to check source code and do other things. However, the status showed a huge number of little spikes in CPU usage. After I disabled Xcode Server, the spikes stopped. When it wasn’t doing builds, it really shouldn’t have done much to warrant the CPU spikes.

I can’t say that I’m going to miss fighting Xcode server; I installed Jenkins and got it configured how I wanted my jobs configured. Jenkins, itself, was easy to setup and do the builds; the tricky part was getting my release notes to TestFlight and to get the archives named the way I wanted. Even though Jenkins is written in Java, I’m not seeing spikes in CPU usage; in fact, when idle, I’m seeing a decrease in CPU usage.

Fun with Xcode Server – Round 2

Earlier this year, I wrote about Xcode Server as well as some issues I had with it. My Xcode Server has been running well for months spitting out builds for testing as well as ones that goto the AppStore.

Unfortunately, Apple changed Xcode Server as part of Xcode 6/Server 3.2.1 (the Xcode Server part of OS X Server primarily is found in Xcode itself as far as I can tell; feel free to correct me if I’m wrong). There were a number of things I had to do in order to get my server building again due to a change in the location of the provisioning profiles and where the key/certificate pair are stored for the profile. These steps are long and not guaranteed to work, but give them a try if you’re feeling lucky.

  1. Switch to root user
    sudo su -
  2. Switch to the Xcode Server keychain directory
    cd /Library/Developer/XcodeServer/Keychains
  3. Copy the Portal keychain to your Desktop
    cp Portal.keychain /Users/<youruser>/Desktop/
  4. Make a backup copy of the keychain.
    cp Portal.keychain Portal.keychain.bu
  5. Change the owner of the keychain to your user.
    chown <youruser> /Users/<youruser>/Desktop/Portal.keychain
  6. Record the password of the Portal keychain
    cat /Library/Developer/XcodeServer/SharedSecrets/PortalKeychainSharedSecret
  7. Leave this terminal window open.

  8. Open Keychain Access as your user.

  9. Add the keychain you just copied to Keychain Access – /Users//Desktop/Portal.keychain

  10. Open a new terminal window

  11. Unlock the Portal keychain

    security unlock-keychain -p \
    <secret you copied from the PortalKeychainSharedSecret File> /Users/<youruser>/Desktop/Portal.keychain
  12. Copy the key and certificate used for the provisioning profile to the Portal Keychain in Keychain Access.

  13. Lock the Portal Keychain

  14. Go back to the Terminal window where you are the root user. You’ll still be in the Keychains directory for Xcode Server.

  15. Copy the Portal Keychain back to the directory.

    cp /Users/<youruser>/Desktop/Portal.keychain .
  16. Change the owner of the keychain
    chown _xcsbuildd Portal.keychain
  17. Switch to the Provisioning Profiles directory
    cd /Library/Developer/XcodeServer/ProvisioningProfiles
  18. Copy your provisioning profile.
    cp /Users/<youruser>/Desktop/MyProfile.mobileprovision
  19. Edit your bot in Xcode and reset git credentials. I always let the bot create a new SSH key pair and add that to either GitHub or Bitbucket depending on the project.

  20. Cross your fingers and build.

It appears to me that Xcode Server has gotten worse where more functionality has moved to Xcode and less is in the web interface. This means I can’t change the git branch I use for building and I can’t edit my bots using the web interface. I think that there is a lot of potential with Xcode Server, but there is a long way to go in terms of usability. The server must be able to do more than run tests and must be able to let the small development shop do builds that can’t be distributed via TestFlight and uploaded to the app store.

Automation is key in any development project as the manual process is for the birds. Many developers don’t need the power of Jenkins, but could easily use Xcode Server if it worked a little better. Doing builds locally and then uploading for testers is cumbersome and a process that should be avoided. I’d be more than happy to talk to the Xcode Server folks about build practices. At work, I have our Jenkins based build system crank out 13 builds at a time and upload to TestFlight with minimal effort; we just push code to a build branch in git and out pops the builds. We can do several builds a day if need be without blinking an eye. For my personal projects, I was doing a build nightly for someone I was working with and without the automation, he wouldn’t have been able to see the progress as quickly as I would have spent 5-10 minutes a day doing builds which really adds up.

Tweaking a web server with Nginx

A few weeks ago, I started playing with nginx as a replacement for Apache. The exact reasons I decided to do this are a bit unclear, but I think it had something to do with me playing with node.js and MongoDB. The initial setup was pretty smooth and the configuration files are much easier to understand than Apache for my sites. However, I also decided to offload the PHP processing to php-fpm and that’s where the problems began. My VPS is pretty lightweight and doesn’t have a lot of RAM. My blog was running out of memory all the time and php-fpm stopped responding.

After a lot of tweaking to reduce memory usage, I managed to stabilize the system and get much faster response times for my blog; it doesn’t get a lot of traffic, but I think I’ve done a decent job with performance. If I ever get hit with a massive load, I’m sure my server will keel over.

Is there a real reason to use nginx vs Apache? I’m sure some people will say yes, but for me the jury is still out. Things are going smoothly, but was it worth the trouble? I have no idea.

Thoughts on Storyboards

When Apple came out with Storyboards in Xcode, I thought that they would be quite useful in making it easier to see how an app was laid out. However, after attempting one project with them, I determined that they had limited utility as they wouldn't scale and wouldn't work well for multi person teams. I left storyboards for a year and didn't revisit them.

I started another project about a year after my first experience and decided to give it another go. This project was a small project and I was the only developer, so it was well suited for storyboards. The project grew and grew. Storyboards worked OK, but the layout in Interface Builder was almost too hard to manage. I finally decided to see about splitting up the storyboards into multiple storyboards.

The app was a tab bar based app, so the logical breakdown was one storyboard per tab. It only took about 15 minutes for me to do this and undo a poor initial decision. However, there were still one or two links between storyboards that were easy to fix in code. Some peopl have come up with ways to link storyboards like RBStoryboardLink which looks interesting, but I haven't explored it.

Multiple storyboards seems scary at first, but it is worth doing to keep difference parts of an app more contained and easier to visualize.

I haven't used XIBs in about a year and I'm not sure there is a good reason to use them as storyboards are easy to work with and make it simple to use custom UITableViewCells and UICollectionViewCells.

Code Signing Woes on Xcode Server

Following up on my post from yesterday, I’ve discovered that Xcode Server does something very strange in the build process. After the build, it creates a .ipa file that is suitable to install on an iOS device and a .xcarchive file that contains the .dSYM file as well as the .app file which is the .ipa file before it has been packaged up. The packaging process is basically putting the .app file in a Payload directory, zipping it up and changing the .zip extension to .ipa. Sounds simple enough and unzipping the .ipa should yield a file identical to the .app. Unfortunately this is not the case. If you run the following command:

codesign --display --entitlements -

on the .app in the xcarchive and on the .app that is extracted from the .ipa file, you will get different results.

From the .ipa, I get:

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

from the .app, I get:

??qq?<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "">
<plist version="1.0">

It appears that the code signing entitlement, aps-environment is missing from the .ipa Xcode Server built. I can’t figure out why this is the case, but it does show that I’m not crazy and my setup works. The .ipa I create myself and upload to TestFlight works because my simple packaging script doesn’t re-codesign the .app.

So I have 2 choices when uploading to the app store:
1. Do what I described yesterday and that is resign the app on my desktop machine

  1. Take the .app file out of the xcarchive, zip it up and upload that.

Xcode Server does mangle the product and I’m not sure that is intentional. I should file a bug with Apple to see if they can fix it in the future.

I’d love to be proven wrong on this whole setup. If you have more information, please let me know.

Using Xcode Server for Builds

At last year’s WWDC, Apple introduced Xcode Server for doing continuous integration. Lots of people have tried to use it with mixed results. Recently after dealing with our build server for work (we use Jenkins), I decided to give Xcode server a try for one of my projects.

I was hesitant at first to use OS X Server (which includes Xcode Server) as I’ve had poor results with it in the past. It intrigued me often that I decided to install server and see where it went. OS X server install was a breeze and so was setting up Xcode Server.

I created a “Bot” from Xcode on my development machine and saw that it failed when the server tried to build it. No problem, I searched the Internet and all the results said to add the repository from the server instead of when creating the bot. Perfect, I did that and then the build succeeded. Yeah! I also added my Team to Xcode server so it could pull down provisioning profiles and create a certificate to build.

I checked the log and found 2 problems. First, I have a run script build phase that puts the version number in the Settings.bundle that failed and second, it was being signed using the wrong mobile provisioning profile.

I decided to tackle the second problem first. Through my searches, I found that I needed to put the mobile provisioning profile (an AdHoc one) in /Library/Server/Xcode/Data/ProvisioningProfiles. No problem. Next I also found that I needed to put the certificate and private key for the profile in the System keychain as the Xcode Server runs under _teamserver which doesn’t have its own keychain. I did that as well and found the problem didn’t go away. I started thinking about it and found that sometimes Xcode ignores the Automatic Distribution Code Signing Setting. So, I removed my Team and deleted all the other mobile provisioning profiles. Bingo, now my app was building and being signed correctly.

Next step was to figure out my build phase. After dumping the environment variables, I found that in order to put files in the .app after the build, I had to set my script phase to be:

shortversionnumber=`/usr/libexec/PlistBuddy -c "print CFBundleShortVersionString" ${BUILT_PRODUCTS_DIR}/${INFOPLIST_PATH}`
bundleversionnumber=`/usr/libexec/PlistBuddy -c "print CFBundleVersion" ${BUILT_PRODUCTS_DIR}/${INFOPLIST_PATH}`
shortversionnumber=`echo $shortversionnumber '('$bundleversionnumber')'`

echo ${shortversionnumber}


echo $dest_file

/usr/libexec/PlistBuddy -c "delete :PreferenceSpecifiers" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers array" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:0:Title string About" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:0:Type string PSGroupSpecifier" "$dest_file"

/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:Key string VersionKey" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:DefaultValue string ${shortversionnumber}" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:Title string Version" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:1:Type string PSTitleValueSpecifier" "$dest_file"

/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:2:File string Acknowledgements" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:2:Title string Acknowledgements" "$dest_file"
/usr/libexec/PlistBuddy -c "add :PreferenceSpecifiers:2:Type string PSChildPaneSpecifier" "$dest_file"

The key was the ${BUILT_PRODUCTS} variable.

So I was almost done with the build server. I setup 1 bot for the master branch that builds when code changes and another on the build branch that also builds when code changes; however, I only push to build when I’m ready for a build. The build branch also had to upload to TestFlight, so I researched that and came up with very convoluted solutions.

I created a second scheme for the TestFlight upload and set my build branch bot to use that scheme. Then I added a Post Action to the Archive phase that looks like this (make sure you select Provide Build Settings From “Your Target”):

rm -rf ${IPA}
/usr/bin/xcrun -sdk iphoneos PackageApplication -v ${ARCHIVE_PRODUCTS_PATH}${INSTALL_PATH}/${WRAPPER_NAME} -o ${IPA}

RELEASE_NOTES=`cat ${SRCROOT}/CurrentRelease.txt`
echo "*** Uploading ${IPA} to TestFlight ***"
/usr/bin/curl "" \
-F file=@"${IPA}" \
-F api_token="${API_TOKEN}" \
-F team_token="${TEAM_TOKEN}" \
-F distribution_lists="${DISTRIBUTION_LISTS}" \
-F notify=True \
-F notes="${RELEASE_NOTES}"

echo "TestFlight upload finished!"

I thought I was all done; builds worked and were uploaded to TestFlight. The moment of truth came when it was time to upload to the AppStore. I have push notifications enabled in this app and it is properly provisioning. After the upload, I got email saying that the ape-environment key wasn’t in the entitlements. Very strange, I checked and rechecked the build I got out of Xcode Server and it was correct. I tried a number of things and in the end sort of gave up. I took the archive that the build server created, put it on my main machine, imported into Xcode, exported for AdHoc distribution, resigned it using the same profile it was already signed with and submitted it. For some strange reason, this worked. I can live with that as I don’t submit to the app store all the time.

Things to note

  • You do NOT have to create an AppStore distribution profile. You can submit the AdHoc build to the AppStore without problems. This is key as it lets you test the exact version going to the AppStore without rebuilding or resigning.
  • You have to manually move your mobile provisioning profiles for AdHoc builds to your build server.
  • You have to manually add your private key and certificate to the System Keychain on your build machine.
  • Be careful about your build scripts as the paths are different on Xcode Server.
  • You have to add your repositories from Xcode Server and not from the bot you create.
  • Read the logs carefully when there is a problem.
  • Apple appears to have created Xcode Server for testing and not necessarily for release builds; I hope some of this changes in the future. I’m not a huge fan of unit tests, so I may just be abusing the server by doing what I’m doing.

Overall, I’m pretty plea with having my own continuous integration server. It will catch issues where I have something different from Debug builds to Release builds. In addition, it will upload to TestFlight along with the release notes saving me time. I am a little disappointed about having to resign for the app store, but it isn’t a huge deal. I could probably modify my script to do that during the archive, but I really don’t like having to hard code in the ID of a mobile provisioning profile. I could probably check the profile into source control and reference that, but I’m OK with this solution. As I already want to move the archives to my main machine, the extra signing process isn’t a big deal.

I know that there is a lot here, but hopefully it helps someone else in the future (that person may be me when I forget what I’ve done!).

Flaws in RESTful response codes

Anyone that uses or develops APIs is familiar with REST. It is widely used and by some developers considered superior to other ways of doing APIs. Since REST was developed alongside the HTTP 1.1 spec, HTTP status codes are generally used for responses to REST calls.

To me, it seems that using the standard HTTP response codes (a nice list is given here) is overloading a standard server response with the response of a query. While the spec says that for client errors (400 series errors), the response should contain the reason why in the response. I’ve looked at a lot of APIs and the best ones at handling errors return the error response inside of a successful 200 HTTP response. For instance, if you are creating an API for dealing with support tickets and you ask for a ticket ID that doesn’t exist or you don’t have permission to access, returning a 400 series error generally tells the client that something bad has happened when, in fact, the error is kind of minor. Embedding the error in the response seems like a better way to go.

The problem, however, with embedding the error in the response is that the API developer has to come up with his own error codes and convey that to those that use the APIs. Many developers that implement APIs treat anything but a 200 series response as a complete failure and give up. So, the API developer has to make sure that the client implementors properly handle the responses. In general, I find it easier to have one set of parsing code and deal with the errors inside of a successful response.

My argument, of course, is not very compelling as it is simply to swap one way of handling responses and errors with another. Last night, however, I came up with a potential argument that leads credence to what I’m saying. If there is a proxy server or a load balancer in front of the server, it could potentially return an error code which the client may not expect. Let’s take the case where our support ticket server is behind a proxy server and the client requests a particular ticket. If the proxy server or load balancer is misconfigured and we make a request, the proxy server could return a 400 series error, even a 404 error. The client could interpret this as “the ticket doesn’t exist”. This may not be the case; the ticket could exist, but the proxy returned an error. A better way of handling this would be to use 200 series status codes when the request succeeds and embed the API error code inside of the response. If the proxy server returned a 400 series error, the client could always treat it as an error and wouldn’t have to figure out if the support ticket really existed or not.

In addition, the HTTP status codes are not granular enough for many APIs. This would require the API developer to have a secondary set of error codes that would be returned with the 400 series error codes anyway. If that is the case, why not just create one set of error codes and always return 200 series success codes unless something major has happened?

I know that some of this may be confusing (it is crystal clear in my head), but it is worth some thought to consider the best way to handle response codes in APIs. Many people have opinions on this and may think that I have no idea what I’m talking about and HTTP response codes are perfectly fine for RESTful APIs.