RAIDers of the lost archives

I don't know about other IT support people elsewhere, but for me, we always have computer problems after a long break, like the first day after the New Year, or, often worse, the first day after Songkran festival.

On the last day of the long Songkran festival vacation, the employee RFID scanner's UPS died and somehow took out the scanner's power supply, I had to quickly find a replacement power supply before people started coming in to work. At 8 AM on the first day of work, everyone started turning on the lights and computers at the same time, and all the breakers automatically cut off. Normally someone just had to manually reset the breakers but for some reason resetting didn't work this time, so they just called the building maintenance people then switched all the breakers off, without notifying the IT people, and then the power to the servers went out.

After the power came back, the affected RAID drives (the SQL server has an SSD RAID1 array) started rebuilding themselves, and I started getting support calls that users can't login to the ERP database. I looked at the SQL server and saw alert level 024: Hardware Error. Unfortunately, eventhough I have a nightly backup routine, it failed to activate since earlier last week, and the SQL Agent didn't notify me of its failure. If I restored from last good backup, I lose two days of work and get killed by the users and the boss. If I repair the database, I lose less work but my ERP support guys have to fix it. DBCC CHECKDB REPAIR_ALLOW_FOR_DATA_LOSS it is then.

Fortunately, there wasn't much data loss and internal consistency wasn't compromised, but my ERP guys are still working on the problem as I'm typing this. Later in the afternoon, more support calls started coming in from users using non-standard clients such as Windows 98 and things like scanners that automatically scan and save to shared folders on the server. I turned my attention to the new domain controller to check user and folder rights. Then I notice a little blinking icon on the task bar: one drive on the RAID array had failed.

One of the first posts of this blog was about putting Western Digital RE2 drives into a Buffalo TeraStation. That was in early 2008. Three years later, five of the original fourteen RE2 drives I bought have already failed. The reason I got fourteen drives at the time was to put four each into two Terastations, and three each into two Windows servers. I thought these are supposed to be special drives designed for RAID use, but the stupidly cheap "consumer" drives that I also use for RAID have not had any problems, and these expensive RAID drives are dropping like flies.

One time it was so bad because one drive failed in a TeraStations' RAID5 array, and when I put in a new drive to rebuild the RAID array, another drive went bad took out the entire RAID array. I had to reflash the firmware to bring it back. Last year, one drive in a server's RAID5 array went bad, imagine how bad I felt when it took 30 hours to rebuilt the array.

Luckily I happened to have a test server in an isolated network, so I quickly joined into into the domain and promoted it as a domain controller, then proceeded to replace the failed drive.

During Songkran vacation, while I worked to replace the old server, which I'll write about another time, there was a bunch of half-naked ladyboys outside making noise and splashing water. Had I known I was gonna have all these problems, I would've invited them to come in and just blow up my servers and preemptively end all this misery.

My SQL data and failed hard drive have gone to Bit Heaven. They served well and died without thanks. May they rest in peace.

Throwing hardware at the problem

My ERP project had been live for a few months, but I never managed to resolve the slowness issue. I'm not very good with SQL Server, and all the optimizations I did probably had limited effects on the performance since Dynamics NAV's C/SIDE code is executed client side and the idiots I have for consultants don't know the difference between native database and SQL Server. I couldn't get to the ERP's code, and even if I could, I probably wouldn't want to hack the code anyway.

Few months ago my boss suddenly told me that we made huge profits this month, and to avoid paying huge amounts of tax on the huge profits, he's giving me special permission to buy stuff. A lot of stuff.

Old server specs (purchased in October 2007): Intel Core 2 Quad Q600 (2.4 GHz), 8 GB DDR2 RAM, ASUS P5K Premium, and 2x500 GB SATA (RAID1). (Was 3x500 GB SATA RAID5, but I listened to the consultants and changed the RAID5 to RAID1, with no difference in speed whatsoever.)

New server specs (purchased in late 2010):

Oh wait...

New server specs: Intel Core i7-950 (3.06 GHz), 24 GB DDR3 RAM, ASUS P6X58D Premium, and 2x160 GB SSD (RAID1).

Windows 7's experience index score is 7.5 on the CPU and RAM, and 7.9 on the hard drive. I wanted to buy a faster CPU, but the 950 is most cost effective, and RAM is maximized for Core i7.

Oh, and the ERP didn't run any faster than before. None whatsoever.

I've talked about my Core 2 Quad Q6600 servers many times before. They were originally bought to replace the older domain controllers and run the ERP database. But I used them for SQL only due to performance recommendations, not that it made any difference since my database is so small. Now that I have the Core i7 dedicated to SQL, I decided to replace domain controllers with the Q6600 servers, and also upgrade the entire domain to 2008 R2 functional level.

Arduino relays

I'm putting my newly learned Arduino knowledge to practical use. Since I don't really know any electronics, I went to a local electronics supplier called ETT to see what pre-made boards they have. While looking around, I discovered that they make a custom Arduino board with a mouthful name called ET-EASY MEGA1280 Duino Mega that easily connects to their boards using a simple 10-pin connector.

This is the Duino Mega connected to a 5V relay board. The 5V board can be powered directly from the Duino Mega board which is powered from USB. Also connected is a Sharp GP2Y0D21YK infrared sensor. The sensor is a digital sensor with a 24 cm range.

Here's another relay board also produced by ETT. This is a 12V relay board that needs to be externally powered. Actually I got the 12V board first, but then I saw the 5V board that can be powered from the Arduino. But after getting the 5V board I was afraid that it might draw too much power. But both seem to work perfectly fine.

I wrote a little sketch that controls the relays using commands transferred over USB serial as well as reads the status of the infrared sensor. My first Arduino project is a motorized computer scale system. One relay is used to control the motors of the conveyor belt and the second relay is used to toggle a warning siren. The infrared sensor (which will be replaced by a laser sensor in production) is used to sense when to stop the conveyor belt in order to weigh the product.

Of course, the system has a load cell and a weight indicator that also has to be connected to work. The load cell is connected to the indicator through a special load cell connector, and the indicator is connected to the computer using a simple USB serial port converter.

It's complicated

I bet it is.

Linksys WRT160NL and DD-WRT

A few years ago, I wrote about the Linksys WRT310N and DD-WRT. I no longer have that router since a friend decided to borrow it permanently, and I ended up selling it to him.

When my ASUS WL500gP died last year, I tried to buy another unit only to find it wasn't available locally any more. So I went looking for an alternative, and the Linksys WRT160NL was what I found. I've had one for a few months already, and recently I bought a few more as it's quickly turning into my favorite router.

WRT160NL has a 400 MHz CPU, 32 MB RAM, and 8 MB Flash. The It has a USB port ("Storage Link") for attaching USB storage devices. However, with DD-WRT, it can also be using for USB printing. The box has a penguin logo and it's just inviting me to put a third-party firmware on it.

I'm running the most recent DD-WRT (16214 as of this writing) on it. With this version, I can enable USB Printer Support from the web interface and it automatically works with my inkjet printer. With earlier versions I had to manually install p910nd to get printing to work. DD-WRT doesn't have a detailed changelog so I don't know when this happened, but I'm glad it works "out-of-the-box" now.

The DD-WRT router database currently shows version 14896 for download. This version seems to have wireless problems since I seem to lose wireless connectivity after a while. 16214 also seems to have fixed that problem.

While trying to figure out the wireless issues, I also followed the recommendations in this thread and set my TX Antenna to 1+2+3 and RX Antenna to 1+3 and Antenna Gain to 2, which seems to greatly increase my wireless stability and connectivity.

Oh, while I was playing with upgrading the different firmwares, the WRT160NL locked up completely. It would no longer finish booting and the power light just flashes. However, upon closer inspection, I discovered that the router wasn't actually "bricked", and I managed to recover it simply by using tftp to upload the linksys-to-ddwrt-firmware.bin to it. After it rebooted, then upload the actual wrt160nl-firmware.bin to it using the web interface.

Just to be sure that it wasn't pure luck that I recovered the router, I purposely crashed the router several more times, and each time I managed to recover it by using tftp.

The WRT160NL also works with Gargoyle, which I'll talk about next time.

Giant iPod touch

iPad 2 was released earlier today and it looks almost exactly like the leaked photos. I was one of the few people that thought the leaked iPad 2 photos were real, since it follows closely Apple's recent design paradigm: no more round backs. The iPad 2 looks almost exactly like the fourth generation iPod touch, which I really like.

But what I like most about the iPad 2 is that it's much lighter! I've had the iPad 3G for a few months and I no longer really notice the weight, but whenever I pick up iPads without 3G I notice how much lighter they are. The iPad 2 with or without 3G are both lighter than the original iPad. But I probably won't upgrade unless I can get one for free. Besides, my iPad is jailbroken.