I hate the change back to standard time as it gets dark so much earlier and kind of cuts my days short. However, what is even worse than the change is having to deal with daylight savings time as a software developer. Most of the software I deal with lately is synchronization and each device handles daylight savings time differently which results in major headaches. Too bad it isn’t all standardized to make life easy. Oh well, these issues keep me employed.
Backup scheme
I think that I finally have my backup scheme worked out. My server meltdown had me re-think the strategy. However, my meltdown, while time consuming to get everything working again, really only cost me one day’s worth of data that I was able to restore from another machine. My backups sort of worked, except for the monolithic “dump” archive that I created. This archive got corrupted and caused the restore to fail. Luckily I had created tar/gzipped backups and had the files.
So my new backup strategy is quite involved. First off, I have 2 drives doing RAID 1 which will protect against hard drive failure (I hope). Next, I have a third drive in my server that every hour does an rsync of the main drive. Next, I every night, a nightly rsync is done of the main drive to the third drive. Next, I have a TrayDock that I do an rsync to every few days. Then I take the dock to my safe deposit box. I have a second tray for it and rotate those backups. I also do nightly tar/gzipped backups of important stuff to the third drive which then get copied to the TrayDocks. Lastly, I periodically copy the tar/gzipped archives to a TrayDock attached to my PowerBook (I have 2 trays for my PowerBook).
I think that I should be covered in terms of backups. While there is still the potential for downtime, I’ll sleep better knowing that I can restore any one file or files without having to rely on a massive dump file still working. Yes, I may be paranoid, but my livelihood depends on data on my server.
dspam
dspam is a great product for catching spam, but as I’ve complained about in the past, setting it up is a nightmare. I went to install the latest version today and wanted to install it “correctly” and have the web user interface operate how it was designed. I spent awhile futzing around, but ended up using my hacked up CGI to get the web interface working. Uggh.
Server Recovery
This sure has been a nightmare to get my server running again adequately. I got almost everything working yesterday and today I tackled converting to software RAID1 so that I have a mirror. With most Linux tasks, there is some help on the web. A co-worker pointed me to a document for “crazy sysadmins”. I didn’t think that applied to me, until I re-read it several times and realized that it is almost what I need. I followed the directions and was stoked that things were going smoothly. Then came the hard part, rebooting. I always have problems with grub, fstab, etc. After much Google searching and futzing, I figured out the solution…I had to rebuild the ram disk image that got loaded so that it knows to boot off the RAID. This normally wouldn’t be necessary, but the default Fedora Core 3 install used an LVM volume and the old initrd file was based on that. So, I figured out that:
mkinitrd -v –preload=raid1 –fstab=/mnt/newroot/etc/fstab initrd-2.6.12-1.1378_FC3.img 2.6.12-1.1378_FC3
worked. It’s hard to tell from the documentation what is going on, but if you don’t specify the fstab file, it uses the current active one which happens to have the LVM mess in it.Just to make sure I didn’t screw anything up, I removed the original drive and setup a clean drive as the second drive for the RAID (I bought 4 drives with the idea that 2 were for the RAID and 2 were hot swappable spares).In about 40 minutes when the drives finish mirroring, I’ll restart the server and see what happens.I’m now convinced more than ever that sysadmins (at least those that run Linux/UNIX machines) don’t make enough money. It is extremely frustrating to have a server crash and then to have trouble restoring it. I also forgot to mention that one of the times I was restarting the server, it tripped my UPS and somehow killed the UPS. The UPS definitely has enough capacity for the server, but something went haywire and I have to get the UPS replaced. A new one will be here in 5-7 business days. I do have a spare, but it’s significantly smaller.
Rapidweaver 3.2 Public Beta
I just downloaded the new version and must say that some of the changes are interesting. My only real complaint is that their beta feedback requires me to use a forum and post. Can’t I just fill out a form? There are a number of issues I’ve discovered, but right now I’m not inclined to fill out the form. (Removing a blog category screws up every blog entry that is in a category after it; there is no way to rearrange blog categories; the copyright symbol isn’t uploading properly even if I replace it with & copy ;., etc.). Hopefully other people will report them. I realize I’m quite a useless tester, but the harder it is for my to report, the less inclined I am to help.
Server crashed
When I went to check my email this morning, it failed which was odd. When I started investigating, my server said that the file system was mounted read-only. I lugged a monitor and keyboard into the other room and started taking a look at the server. I was unable to repair the file system, so I went down the path of reformatting the primary drive and restoring from the secondary drive. I’ve never restored the drive before, so I had to use my Mac to do a ‘man restore’ to figure out what I needed to do. The good news is that the corruption appears to have started around 6 am and my backup is done daily at 4 am. At most, I should only lose a few hours worth of stuff. However, this kind of file corruption is worrisome and I’ll have to keep an eye on things. Nothing like having to spend hours fixing a server on a Monday morning.
Email Spam
Email spam, as everyone knows, has gotten completely out of hand. I use a system called DSPAM which works amazingly well. However, today I decided to upgrade to the latest version. The configuration on the program is a royal nightmare. I wanted to remove the hacks I put in place for retraining false positives as they’re hard to maintain each time I upgrade. This turned out to be a loser proposition. After several hours trying to find hints on the net and reconfiguring, I finally gave up and put my hacks back in. (Basically the web CGI for re-training doesn’t seem to work right and puts the mail into a blackhole instead of delivering it.) If anyone knows an easy way to install DSPAM on Linux with sendmail, please let me know. However, I want to invoke it with procmail and not directly from sendmail as I want the ability to have certain filters act on mail before DSPAM gets it; DSPAM wants to classify certain senders as spam even though I know they’re not.)
User Interface
Yesterday I had the privilege (or was it?) of having Missing Sync for Palm OS reviewed by a user interface guru at Apple. He had some good ideas on how we can improve the user interface that I think will be quite beneficial to the product. It is quite intimidating having the product I’ve been working on for more than a year and a half critiqued and ripped apart. After thinking about some of the things said, I have to respectfully disagree with some of the comments made as there isn’t one type of user interface for all applications and even Apple’s own applications seem to contradict many of the things the guru said.
I really appreciated the time that was spent reviewing the app; however, it was only slightly more enjoyable than going to the dentist.
Apple’s Transition to Intel
In June, Apple announced the move to Intel processors with Steve Jobs saying on stage that the transition would be relatively easy for most developers. While this is true for simple applications and applications that have been cross platform for ages, it is not true for what I develop. I took a simple program, TimeRecord, and had it running natively on Intel in about 10 minutes. Not bad, but the application doesn’t have hooks into any other program, doesn’t connect to websites, doesn’t transfer data, nor take advantage of any cool features.
The main program I work on these days is Missing Sync for Palm OS and one of my tasks is to figure out how to move it to Intel. I came up with a brilliant (if I do say so myself) whitepaper on the transition. Implementing what I wrote, however, is going to be quite difficult as endian issues had never been taken into account and a requirement is that old, CFM conduits are supported. Apple’s current strategy doesn’t permit us to transition to an Intel native binary and support CFM conduits. So, we’re going to have to come up with some ingenious way to tackle this. One way, outlined in the way paper, may not work. So it could be back to the drawing board.
Missing Sync for Palm OS has upwards of 50 separate components that get compiled and will have to be tested on Intel. This will be no easy task. I’m always up for a challenge, but this one gives me a headache.