With all my problems with my server, you’d think I’d give up and just host my server somewhere else and make it someone else’s problem. After a little thinking, I’ve come up with the pros and cons of running my own server.Pros————–Physical security of serverMusic server – I need a server to run my SqueezeboxesUnlimited bandwidth (at 1 MBps upstream, but unlimited)Full backups – I backup the server to an external drive and take it offsiteLots of storage (currently at 300 GB, but only about 15% used)It’s a learning experience5 static IP addressesFull control over serverAbility to run my own PBX and have good voice qualityCons————Electricity usageI have to keep it runningLimited upstream bandwidthPotentially more costly (it costs about $50 per month for me to host my own server); however for the hardware I have, a hosting service might charge more for RAID 1 and backupsWhile there are more items I’m missing here, it’s pretty clear to me that running my own server is the right way to go. My issues last week appeared to stem from drives that were failing; it just happened that when I was putting in the new drives, the drive I left running was failing causing some issues. I’m now up and running on my 2 new drives and have been for about 5 days now. Lesson for the future…replace drives about once a year if they need it or not as the drives are relatively cheap ($150 for both drives) and run 24/7 which puts a lot of wear and tear on the drives. Now the question is, what do I do with the old drives? Would a reformat on the drives do the trick and keep them running? What less mission critical role can I put them in?
The new world of IMAP
I realize that IMAP isn’t new, but I just started using it yesterday. I’ve resisted for years as I’ve always had one main store for my email. However, lately, I’m checking email using web based email and everything I’ve heard about IMAP make it sound like it is perfect for using in multiple locations. So far it’s working OK, but I’m getting used to it. For instance, if Mail on my Mac filters messages out of the Inbox, they’re pulled off the IMAP server which kind of defeats the purpose of using IMAP. Most of my email is filtered. I created a rule in Mail that moves all messages < 1 day old into my Inbox and made it my first filter. This should leave everything in my inbox, but when I run the filters over my inbox manually later, it should put everything in the right spot.
Hopefully I’ll get used to this soon; I’ve been using POP3 for close to 15 years ever since I started using Eudora in college. It’s hard to change my thinking, but since I started using POP3 over SSL, I could no longer telnet into my server and issue commands (I know the POP3 commands as I’ve written 2 POP3 email clients in my life), there really is no advantage to me using POP3.
Speeding up Ruby
When I setup my store, I sued FastCGI to get acceptable performance which was fine as I don’t get many hits. Today I installed BrowseBack because it came as part of the software bundle I purchase through MacHeist/MacUpdate. As I hit my store a lot to check stats and such, BrowseBack kept loading the store which spiked the server load to 4 and required me to kill BrowseBack to get my server back to normal. While this isn’t normally a problem, it exposed a potential kink in my server’s armor. So I went looking for an alternative. I’ve read about using lighttpd by proxying requests from Apache. Setting it up was straightforward and seemed to work well, until a customer complained that he couldn’t purchase. I tried it myself (again) and it worked fine. I had someone else try again and it failed. Hmmm. After much tinkering, I figured out the problem, my store code required an https connection otherwise it redirected to https which was fine, except that coming from the proxy, it was always an http connection. Since I already do a redirect on my store so that any http://store.receiptwallet.com requests goes to https://store.receiptwallet.com, I can be assured that all requests are secure. So I commented that line out of my store and everything is working fine.
(On a side note, the reason I couldn’t see it myself is that it was checking for local requests and since I access my server using a private IP address range, it basically got flagged as local, so it never required the SSL connection.)
Am I being told something?
I went to install a new hard drive in my server today to pre-emptively avoid a disaster as the drives in my server have been running 24/7 for 1.5 years and the drives are so cheap, that replacing it should save me time later. I thought all I had to do was power down, yank one drive, power up, partition the new drive, and rebuild the RAID. That is great in theory, but my RAID had errors so when I rebooted, it failed to reboot. I ran fsck on it (after failing to read what the screen told me to do for too long) and let it repair lots of little problems. After that, I was able to reboot and rebuild the RAID. In about an hour, my RAID will be rebuilt and I’ll let it sit for a few days before I go ahead and replace the second drive. With all the problems I have with my server, you’d think I’d learn to just use a hosting service and let someone else manage the hardware. Oh well.
A week of server woes
Last week was another one of those weeks where my server decided not to cooperate. On Wednesday morning, I copied 7 GB of photos to my server so my wife could tag them and upload some of them (not 7 GB, of course) around 8:15 am. Every 2 hours, my server automatically backs up to a secondary hard drive. Apparently this process (using rsync) combined with other stuff going on (that’s my current theory) caused my server to have a fit and crashed around 10:30 am. Then the process kept continuing as it kept trying to copy the files every 2 hours. Then at around 1 am, my server backs up everything to tar/gzipped files which caused it to crash again due to the load getting way too high combined with higher external temperatures. I finally figured out what was going on around 2:15 am when I was up with the little tike and got my server stablized by excluded the photos directory. Then on Friday when I was doing backups, there was some corruption on my backup drive which caused the CPU load to spike to 14 (normal is about 0.25 or less) and prevented me from unmounting the drive. A quick reboot got me up and running again. I reformatted the backup drive, did backups to it and everything has been running smoothly. During this whole fiasco, I increased the case fan speed to the max and brought the temperature down a bit.
Phew, everything is working again with very little downtime. I still refuse to co-locate my server somewhere as I like to have complete control over things and am extremely paranoid about backups (see my post on what I actually do.
Soon I should probably replace the main drives in my machine as they’re over a year and a half old. While that may not sound old, they’ve been running 24/7 since I installed them. Since the drives are in a RAID 1 configuration, replacing them should just be a matter of shutting down, yanking one drive, putting in a new drive, formatting it, let the RAID rebuild and then repeating the process. Anything I can do to prevent another crisis is well worth it.
Keeping good on a promise
In a pervious post, I said that I’d purchase the products from Potion Factory even though I currently have no use for them because Andy Kim’s open source PotionStore is a perfect solution for my own store. Well, I kept forgetting about this and finally got around to making the purchase this morning. Thanks again, Andy, for the store; not only is it working beautifully, it has also gotten me interested in Ruby on Rails.
Yet another new store
I wasn’t quite happy with my online store and was pleased to see a clean, open source store released. Andy Kim of Potion Factory released his store as open source. It’s Ruby on Rails, so it was a learning experience for me to get it running again. I learned a great deal in the 2+ days that I spent getting it running, but it is so much cleaner than my old stores. Hopefully I don’t have any problems with it as I just put the finishing touches on it this morning. It’s at store.receiptwallet.com. Things I learned from installing this on my server:
- Even though there is an RPM for mod_fcgid on Fedora Core 5 with Apache2, FastCGI still needs to be installed from the source
- The Apache configuration for FastCGI needs to have:
DefaultInitEnv RAILS_ENV production SetEnv RAILS_ENV productionin it. The first line for FastCGI to work and the second to work without FastCGI
- ruby-debug gem has to be installed to get the store to work in development mode and MUST be commented out to work under Apache
- To use multiple databases in RoR, the model has to look something like:
class User < ActiveRecord::Base require 'yaml' set_table_name "users" # we need to set this explicitly extra_coord = YAML.load(File.open(File.join(RAILS_ROOT,"config/database.yml"),"r"))["userdatabase"] self.establish_connection(extra_coord) end
(my pool of serial numbers is stored in a separate database and so are my users; I modified the store to interact with these other databases)
- On PayPal’s developer site, if you want to test Website Payments Pro, you need to use a Social Security Number of 111xxxxxx where x is any number, but this number can’t be used by another developer
- STDERR.puts is your friend. It’s great for logging stuff to the console
- RoR running under Apache must use FastCGI in a production environment or it is way too slow
- In application.rb, it MUST have:
class ApplicationController < ActionController::Base filter_parameter_logging "cc_number|cc_code|cc_month|cc_year" end
otherwise you have important stuff getting logged which is bad.
- Turn off IPN on PayPal or you’ll have an error when processing orders and people will order multiple times because they think it fails
That’s about it for now. I’ll update this as I remember more stuff so that I don’t waste time in the future and it may help others.
Thanks, Andy, for the store! While I don’t have a use for your products right now, I’ll be purchasing them anyway as your store is such an elegant solution, it is the least I can do to repay you.
Apache MultiViews option is dangerous
I was poking through my web logs and saw requests to pages that didn’t exist on my server and the server returned something (200 return code), so I started looking at them. I was able to hit the pages with a browser and after much searching, I figured out how pages that obviously didn’t exist could return a valid page that was slightly messed up (image references were hosed). Turns out my sites had
Options MultiViews
in the Apache configuration files. MultiViews, as I have come to find out, guesses what page to load if the page doesn’t exist. Well, its guesses are whacked. So, disabling this feature fixed the problem and now pages that don’t exist get redirected to the home page.
Dumped Sirius!
Today we finally dumped Sirius radio! We got it as a gift from my in-laws a few years back and they’ve been paying the fee since then which was fine, however, we got real annoyed at the DJs. In January, we got my wife a new car with a built-in XM receiver and she enjoyed the 3 months free. However, I held out subscribing until they sent me an offer I couldn’t refuse (the dealer said to wait until they sent a good offer and they sent several offers). When I subscribed, it also came with free online access which was pretty neat. I found an XM Radio plugin for my Squeezebox and was pleased. I discovered that I could stream 2 XM stations at once, so I picked up another Squeezebox today (used) and hacked the XM plugin code to let me connect twice. Now we can stream 2 XM stations at once or have XM on one Squeezebox and MP3s on the other; this also works out well so that we can play children’s music on one box and whatever we want on the other. Very cool. Goodbye, Sirius and the crappy DJs! (I hope things don’t go downhill when/if the Sirius/XM merger is finalized.)
Sales tax on downloads
Today I went to purchase the professional pictures that were taken of me at the marathon last week and one of the options was electronic download as opposed to CD. This wasn’t offered last year and would save me $16 (the $84 for the electronic download is already extortion). I got to the checkout page and saw that they wanted to charge me sales tax. Whoa…that doesn’t fly in California. According to the California Board of Equalization’s publication about downloads
Products electronically transmitted to customers
Your sale of electronic data products such as software, data, and digital images is generally not taxable when you transmit the data to your customer over the Internet or by modem. However, if as part of the sale you provide your customer with a printed copy of the electronically transferred information or a backup data copy on a physical storage medium such as a CD-ROM, your entire sale is usually taxable.
So I sent the company email and hopefully they’ll fix the problem. I suspect that when they started offering electronic downloads, they didn’t re-tool their store to reflect this or don’t have a clue about sales tax on non-tangible goods.
This is a bit frustrating as I’ve had to do this in the past with another company. Let’s get with the picture, folks! I’ve known about the sales tax on electronic downloads for 13 years when I first had to do research on it; things, luckily, haven’t changed with respect to this.