This post describes how to build a 8 drive storage server for under a hundred bucks (drives not included).

Last summer I bought a Dell PowerEdge 2800, which I converted to use silent fans and SATA drives instead of SCSI drives. See the project page for this server for more information on how to hack the BMC’s firmware in order to swap the fans agains silent ones.

This post is the writeup of the conversion of the server’s backplane from SCSI to SATA. My main reason for this conversion was money: servers that offer a lot of swappable SATA drive slots are quite expensive. SCSI-based servers, on the other hand, are quite cheap – nobody uses plain SCSI anymore I guess. After searching a bit on the internet, I’ve bought a Dell PowerEdge 2800 that supports 8 SCSI drive slots for 25 bucks.

I gutted the SCSI backplane and replaced the connectors with SATA connectors. Adding in a 3Ware 9500S-12 PCI-X card, I’ve ended up with a raid system that supports 8 SATA drives. (Cold-swappable only, which is what I aimed for. This server is not a production server, obviously, but just our home storage).

In the following, I illustrate the steps how I converted the PowerEdge 2800 to SATA. It’s quite easy to do, so I hope I can inspire others. Apropos others: there is a similar project that was featured on Hackaday around the same time I’ve started with my project.

Assumptions:

• you do know how to use a soldering iron and which end of a screwdriver is the front
• you have a Dell PowerEdge generation 8 (or similar, this should also work with generation 7 and 9 servers as the 2900)
• you have a SATA hardware raid card (I used a 3Ware 9500S-12, but any other will do)

Required material:

• 90 degree angled all-in-one SATA connector (I used Delock type 84307)
• cables for connecting the drive LEDs
• connector(s) maching the pin headers of your raid controller
• glue, solder, time

So, let’s start! Open up your server and remove the backplane. Obviously, you have to remove all drives before being able to detach the backplane. I say obviously after I ripped of the connector of my floppy drive by brute force – I removed the harddrives but forgot completely about the other drives. D’oh!

I first thought about replacing the whole backplane, but after I bought the server I realized that all the front-panel buttons, the optical drive etc are connected to the backplane. The backplane is therefore essential to the server, and the server does not even boot without it. So we have to leave the backplane in, fair enough.

The first step is to remove the old SCSI connectors from the backplane. Remove first the protective plasic shield from the backplane. Keep it, we will stick it back on later! The SCSI connectors are 68-pin bastards, so unsoldering them is out of the question. In the picture you can see the first connector removed – I pulled of the plasic part with some heavy pliers and stripped the pins one by one with a cutter.

Next, remove all connectors as shown in the picture. After removing the first ones by hand, I ended up using a Dremel to remove the pins. It does not have to be perfect, but pay attention to remove any short circuits. You do not want to disturb the underlying SCSI system that is still active in the server, or (even worse) short circuit the power planes!

Next, we are going to solder the harddisk access LEDs. This step is optional, but as everybody loves flashing LEDs this will give your project some street cred. No, seriously, the LEDs help you to see access on the harddisks and (if your controller supports it) to identify a specific disk/controller port.

The PowerEdge features 2 LEDs for each drive, of which we will use only one (does not matter which one). First, identify the traces on the backside of the backlane for each of the LEDs. Mark them with a pen according to drive number and if it’s the anode and the cathode. Cut the traces leading to the controller of the backplane leaving enough copper for you to solder a cable on.

In the picture you can see how I did this for one of the ports: one cable connected to the anode of the LED, and the other to the resistor, which in turn is connected to the cathod of the LED. Don’t forget to include the limiting resistor into the mix, because else you might burn your LEDs!

Here you can see how I connected all ports on one side of the backplane. Note that I soldered the cable on the back of the backplane (is that the frontplane?? ;) Remember to fix the cables in place with a little hot-glue. No hacking project can call itself a proper hacking project without a little hot-glue, right?

Next, we prepare the connectors to the raid controller. Most raid controllers support pin headers for connecting the drive LEDs. You’ve checked that your’s does support this before starting to solder the cables to the LEDs, right? Along the same lines, you’ve made the cables long enough to go all the way to the raid controller, even when all the fans etc are in the system? Ok, good.

Configure your connector according to your raid controllers pinout. In my case, the pinout was given in the manual of the raid controller: here’s the relevant page. Pay attention not to swap pins.

Reattach the isolating protective plastic shield and put back the power cable. Your backplane should look like in the picture above. Note that I’ve cut away the two bars at the bottom of the backplane’s PCB (at the bottom of the two big cutouts, which were closed before). This allows me to insert the backplane easier with all the cables in place, but it is not really required. If you do this on another type of system, make sure that there are no traces running in this part of the PCB.

Now we start doing the actual conversion to SATA. Mount all drives in their drive cages and stick them in your server. With the backplane still unmounted, you should be able to see the back of the drives. Here, I plugged in the angled all-in-one SATA connectors (I used Delock type 84307, which worked really good).

As the PowerEdge servers do not have many drive power connectors and we cannot reuse the power connectors of the backplane, we have to connect the power cables of the all-in-one connectors together. This also reduces the cable mess. In the picture you can see how I did it: add the connectors one by one: plug the all-in-one connector in the drive, cut away the power cable that is longer than the distance to the next drive, add the next connector and solder the two cables together. Repeat until you have the whole column completed. It’s hard to explain but easy enough to do, so have a look at the pictures.

This is how the end result should look like. Remember to properly protect all solder joints by shrink-tubes or similar. Use zip-ties to clean things up. In the end you’ll have a single power connector for each column of drives.

Next, we need to attach the SATA connectors to the back plane. The idea is that the connectors stay on the backplane when you pull a drive out – without this you would have to dismount the backplane each time you want to swap a drive. I thought about many different ways to do this, but in the end I chose to use a kind of construction glue to do it. There are for sure ways to do this in a cleaner way, but this serves the purpose and is quick and cheap.

The glue should have the following attributes:

• stick on plastic
• must remain (at least a bit) flexible
• must be able to cover distances of a few millimeters
• must not shrink upon drying (will pull the connector back)
• must be viscose (not too liquid)

Apply the glue to the back of the SATA connectors as shown in the picture (the white stuff). Be sure to clean the connectors and the backplane from any fat residues before doing so. Additionally, push your drives as far as possible to the front (there’s usually a little play in the cage mounting mechanism, and you’ll want it to be maxed out).

Then, put the backplane in. Wait.

This is how your server should now look at the interior. You can see the 8 SATA cables coming out of the cutouts at the bottom of the backplane, just along the two power connectors. Additionally, you can see the cables that will connect the drive LEDs to the RAID controller (the two gray cables).

After you’ve waited long enough (see glue instructions), you can remove and re-insert your drives to check if everything is in order. This is how your empty drive slots should look like.

Plug the power connectors and connect the SATA and LED cables to the RAID controller. Fire up your machine and check the drive status, and the drive LEDs. Remember that your drives are only cold-swappable, so you have to shut down your server before changing them.

Welcome to your new SATA-based storage server!

Update 4/11/2011: I managed to find out some more info about the packaging scheme Dell uses for their BMC firmware files. I deciphered most of the container format. I am in the process of testing modifications right now, but for the moment I updated the version of the tool below with a new version. You can also download the program directly here: dell-extract-bmc-firmware.tar.gz.

Deciphering the firmware header.

I had a look into the firmware flash file, and noticed that it seems to consist of several files (as usual for BIOS/firmwares). As this might increase my chances not to brick my BMC, I decided to I separate the individual files for starters. I couldn’t find a program that does that (the firmware tools of the Dell linux community are closed-source, unfortunately), so I grabbed a hex-editor and deciphered (more or less) the firmware’s header. Here’s the corresponding C program:

The names of the individual files are listed below. They are organized in blocks (that’s what I call them), and apparently by function. Get the latest BMC firmware (30/6/2009, v1.83, A10) and apply my program to retrieve the individual files.

• block 0 (code, big files):
  • BB.FLC
  • OB.FLC
  • ID.FLC
  • OEM_DEF.FLC
• block 1 (*_BB files):
  • SD_BB.FLC
  • FI_BB.FLC
  • TOC_BB.FLC
  • IO_BB.FLC
  • IS_BB.FLC
  • OEM_BB.FLC
• block 2 (*_K_C files):
  • SD_K_C.FLC
  • FI_K_C.FLC
  • TOC_K_C.FLC
  • IO_K_C.FLC
  • IS_K_C.FLC
  • OEM_K_C.FLC

The BMC seems to be little-endian (makes only sense I guess). I’ve scanned the different files for appearances of the threshold values (900 and 2025/0x07e9 and 0x0384 in int and 0x6144000 in float). No avail. Darn. Either I am doing something wrong or the thesholds are not hard-coded in the firmware (I had my hopes up when I saw the OEM_DEF.FLC file, which actually contains the default BMC password and the like). Maybe the thresholds are stored in the configuration flash after all – only how can we access it?

Update

I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.

Earlier, I wrote about the problem of the noisy fans in my Dell PowerEdge 2800. Since then, I investigated a bit more. Just as a reminder: I can’t run silent fans because they have a lower RPM than a hardcoded panic-threshold of the PowerEdge. *grrr*

Brainstorm:

1. make the fans faster/buy faster fans
2. make the system hotter
3. hack the fans into reporting more RPM than they actually do
   1. hack the fans themselves
   2. alter their tacho signal
4. hack the BMC
5. find out how the OEM sets these thresholds

Well, as I mentioned earlier 1. and 2. are for obvious reasons dissatisfactory.

A couple of fans taken apart. Notice the blob of brown paint on the ring magnet of the fan on the right.

Concerning 3A, I took a couple of fans apart, looking at how they create the tacho signal. Almost all fans I opened (luckily I have a whole stack of noisy, throw-away fans lying around) have a sensor sitting just under the ring magnet, which is part of the rotor as can be seen in the photo on the right. I had no idea what this sensor might be, but I noticed on all of the fans one or more blows of brownish-red paint. I figured that this paint might be used to create a signal for the sensor – and I did some tests with other paint in order to replicate the effect (left fan on the photo with magnetic paint applied). Well, nice idea, but total bullshit as it turns out. The sensor is a hall-sensor that senses the change in the magnetic field of the ring-magnet, and thus changes inevitable 2 times per revolution. I figure that the paint applied on the rotor is used for calibrating the fans… Well, it was a nice idea.

3B might be an option, but it would require either a microcontroller or some analogue circuit – not really what I want to fiddle into the fan trays of 6 fans.

Concerning option 5, I thought that there may be hidden ipmi OEM commands for configuring the thresholds. I dug around the Dell OEM extensions for ipmitool (can be retrieved from the Dell Linux Community Repositories). This code officially earned worst code of the year – I completely understand why the ipmitool maintianers flatout refuse to integrate that piece of crap. It’s a hacked-up collection of extensions, seemingly done on the fly to fix customer problems. Horrible. Even more so as it does not seem to be able to set the thresholds either. After a few hours of digging in the code I managed to query the BMC sensors with Dell’s OEM commands, and the returned capability flags do indicate that the thesholds cannot be changed. Darn.

Now I am back to hacking the BMC firmware – but that’s for another post…

Update

I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.

As I explained before, I changed all the fans of my PowerEdge 2800 to low-noise, low-RPM models.

The fans of my PowerEdge 2800. Front the cpu fans (stock Delta left, replacement everflow right); at the back the 120mm case fans (left the Artic, right the stock Nidec)

This was nice and shiny, and the system was quiet and quite cool. Too cool unfortunately, because the BMC throttles the fans too much so that their RPM values go below the configured thresholds. Then, the system goes into panic mode, spins up all fans at 100%, and lowers them successively below the threshold again. Repeat. :)

I ended up putting the Delta’s back in, because you cannot configure the lower failure threshold of the fans. So, lots of work for nothing, and the system is still way too loud. Very unsastifactory.

Thus, I started thinking how to get around this problem. Possible ways I see:

1. make the fans faster/buy faster fans
2. make the system hotter
3. hack the fans into reporting more RPM than they actually do
4. hack the BMC
5. find out how the OEM sets these thresholds

Well, 1. and 2. are for obvious reasons dissatisfactory.

Regarding 3: I thought about getting the sensor in the fan to report a higher value. Most fans actually consist of a ring-magnet. The sensor lies just below it, with the rotating magnet passing over it. On 1-4 places on the magnet, the manufacturers apply some paint. I guess it’s some sort of EMI shield and the sensor detects the change in the field – but I couldn’t find any paint that would reproduce the effect. This would be a very neat and nice solution.

Regarding 4: The BMC is implemented in some microcontroller on the motherboard of the PowerEdge. I haven’t found out yet where it is. I am not even sure what type of architecture it is, so reverse engineering the BMC firmware wasn’t possible. Damn, I don’t even know if this is little-endian or not.

Regarding 5: The manual states that these values are read-only and are to be configured by the OEM. But how? I doubt that they create a new firmware for each combination of fan manufacturers that they use. So my guess is that there are some hidden OEM IPMI commands that allow to set the threshold. I asked the guys over at FreeIPMI if they’ve got a clue, but they don’t know about any such functionality.

Anyone out there with some hints?

Update: I continued my search for a solution

Update 2

I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.