So I went back to The Chip Merchant to exchange my RAM. The guy said that it was unlikely that both RAM modules were bad and went off to talk to his tech.He came back and said that my report of bad RAM wasn’t the first and there appears to be an issue with the combination of RAM, motherboard, and processor they sold me, so they gave me a different type of RAM and told me that the DDR 400 could only operate at 333, but the BIOS’s setting of Auto should detect it. I popped the RAM in, set the BIOS to a RAM frequency of 333 and will cross my fingers. It only took me a few days of futzing around to come to the same conclusion.
Stable server?
Now that my server is rebuilt, my problem is that it keeps crashing kernel panicking and I saw segmentation faults all over the place. All roads point to hardware problems. So how do i solve this? Well, first off, my old memory modules work in the new machine. I installed one of them (512 MB) and the machine seemed to stay up all night with one exception. I noticed that it had rebooted at 5:32 am. In all the other crashing, it never once rebooted. That got me thinking that the UPS I plugged the machine into (an old one) wasn’t powerful enough and a surge that put the system on battery failed to move it to battery and the server restarted. At least, that’s what I hope happened. So I got to thinking, how could 2 brand new memory modules fail. I remembered that when I was handed the memory, they were in adjoining pouches. I checked the serial numbers and they were 12 apart meaning that they most likely came from the same batch and if a batch was bad, both modules could be bad. So this evening I used a program called Memtest86 which supposedly thoroughly tests RAM. I popped in each new RAM modules one at a time and after less than a minute, each module showed thousands of errors. Then I put both in and after 20 minutes I saw 500+ errors; I’m not sure why the results were different with 1 vs. 2, but it convinced me that there was a real problem. I then tested my 2 old memory modules (slower, but the same capacity) and after an hour, they showed no errors.
Now I’m running the server with the old RAM and will see what happens. On Monday, I’ll go back to The Chip Merchant and get the RAM replaced.
I wish all this just worked and I didn’t have to futz with it.
Server Recovery
This sure has been a nightmare to get my server running again adequately. I got almost everything working yesterday and today I tackled converting to software RAID1 so that I have a mirror. With most Linux tasks, there is some help on the web. A co-worker pointed me to a document for “crazy sysadmins”. I didn’t think that applied to me, until I re-read it several times and realized that it is almost what I need. I followed the directions and was stoked that things were going smoothly. Then came the hard part, rebooting. I always have problems with grub, fstab, etc. After much Google searching and futzing, I figured out the solution…I had to rebuild the ram disk image that got loaded so that it knows to boot off the RAID. This normally wouldn’t be necessary, but the default Fedora Core 3 install used an LVM volume and the old initrd file was based on that. So, I figured out that:
mkinitrd -v –preload=raid1 –fstab=/mnt/newroot/etc/fstab initrd-2.6.12-1.1378_FC3.img 2.6.12-1.1378_FC3
worked. It’s hard to tell from the documentation what is going on, but if you don’t specify the fstab file, it uses the current active one which happens to have the LVM mess in it.Just to make sure I didn’t screw anything up, I removed the original drive and setup a clean drive as the second drive for the RAID (I bought 4 drives with the idea that 2 were for the RAID and 2 were hot swappable spares).In about 40 minutes when the drives finish mirroring, I’ll restart the server and see what happens.I’m now convinced more than ever that sysadmins (at least those that run Linux/UNIX machines) don’t make enough money. It is extremely frustrating to have a server crash and then to have trouble restoring it. I also forgot to mention that one of the times I was restarting the server, it tripped my UPS and somehow killed the UPS. The UPS definitely has enough capacity for the server, but something went haywire and I have to get the UPS replaced. A new one will be here in 5-7 business days. I do have a spare, but it’s significantly smaller.
Server crashed again
This time, my backup was corrupted and the server seemed hosed, so I got a new one and started rebuilding from backups. Unfortunately the backup appears to be corrupt (I think it was the drive as I restored parts later from another backup from last week and the files came across fine). I still have a long way to go, but mail and web are back up. I hate computers.
All wires pulled!
Well, my installer decided to come back today and finish the job. (Granted I had to get in the attic when it was 80 degrees out to pull the wire.) I just have to drop down 3 wires, finish the termination and patch the walls. Victory is almost here!
Audio System Installed
The system is finally done. (OK, I still have to patch some holes.) I pulled the last wires and terminated the connections. It feels good to finally be done. My wife and I are really going to enjoy the system. Lots of work, but it will be worth it. I’ve posted pictures of the system all wrapped up in it’s temporary home in the gallery. We just have to move our entertainment center into the other room and then I can put all the components in the cabinet.
Rapidweaver 3.2 Public Beta
I just downloaded the new version and must say that some of the changes are interesting. My only real complaint is that their beta feedback requires me to use a forum and post. Can’t I just fill out a form? There are a number of issues I’ve discovered, but right now I’m not inclined to fill out the form. (Removing a blog category screws up every blog entry that is in a category after it; there is no way to rearrange blog categories; the copyright symbol isn’t uploading properly even if I replace it with & copy ;., etc.). Hopefully other people will report them. I realize I’m quite a useless tester, but the harder it is for my to report, the less inclined I am to help.
Almost done!
My dad came over today and helped me install 2 pair of outdoor speakers, the control unit for those speakers as well as putting the Sirius radio antenna on the roof. It was an exhausting day, but we now have everything working in all but one room. I still have a bunch of patching to do (and my dad added some stucco patching to the mix), but I can almost taste victory over this project! I have to call the installers tomorrow and find out when they can find time to finish the job they should have finished on Friday. (My dad was quite surprised when I connected the control unit and the speakers worked on the first try. I only had one control module not work on the first try and re-punching down the connections cleared up all the problems.)
Audio System Installation
Well, the installers showed up today at 9 am and said that they could only stay only 2 pm as they had another job that had to be done before the weekend. I guess my money isn’t as good as someone else’s. Anyway, The first room progressed quite quickly with only a couple of holes to patch. The next room was extremely difficult and took several hours. I just had them run the wires and only install speakers in the locations that I couldn’t reach with helped speed things along. So, 2 pm rolls around and they don’t have to go to another job, but one guy says he’s not feeling well and going him; they’d finish the job the end of next week. I asked if the other guy could stay to finish another room with me and he did; that room went quickly because I knew exactly how to fish the wires. While I’m pleased that the wires got run, I’m a little disappointed that the job wasn’t completed today (2 controls and 2 speakers are left). I did spend the entire day either helping them fish wires, installing speakers, or terminating the connections (I didn’t even have them pull the wires the last 8 feet down the wall as I had already done that a number of times and I could do it again). So finally around 6:30 pm when my wife came home, I had music playing in 6 rooms in the house! I still have to patch a bunch of holes, texture, and then paint them. My dad is coming over on Sunday to help me install the outdoor speakers. Maybe at the end of next week this project will be done. (Pictures of the installed system will come when it’s done.)
Dog freaked us out
While I was on a conference call today, my wife brought our dog in and had me feel his underside where he was neutered. She was panicked because it was hard down there, like some type of abnormal mass. She quickly called the vet and they told her that when some boy dogs get excited, their glands swell up and it gets hard. Sure enough, a few minutes later things were back to normal. I’m so glad that we didn’t have to take the dog to the vet; I love my dog and will do whatever we have to for him, vets scare me. We spent way too much time at the vet with our last dog.