I am slowly moving most of my projects to GitHub. GitHub is great because people can easily collaborate, fork, submit issues and patches and so on. Additionally, stuff gets shared in an open and unified way, making it much easier to others to find and reuse existing code.
I started with the firmware hack of the Dell PowerEdge BMC:
https://github.com/arnuschky/dell-bmc-firmware.git.
See here for the post explaining what it’s for.
This article described how to boot Ubuntu from a bootable CD without installing anything on the harddrive. It is meant for the many users of my script that described how to adjust the fan thresholds of a Dell PowerEdge.
Step 1
Download Ubuntu and burn it on a CD. The latest version with 32-bit will do fine.
Step 2
Boot from the CD and select “Try Ubuntu” (don’t install for obvious reasons). This allows you to try Ubuntu without installing anything. You can find more detailed instructions on the download page.
Ubuntu Search Menu
Step 3
After Ubuntu booted, click on the Ubuntu button on the top left. A search menu as seen on the right should open.
Step 4
Type “terminal” into the search box. Ubuntu should give you an icon for the corresponding application. Click on it to start the terminal.
Screenshot of the terminal
This should install the software necessary to query your server using IPMI. IPMI is a protocol for server management. See this link for more information regarding IPMI on Ubuntu.
That’s it! You can continue here: How to adjust the fan thresholds of a Dell PowerEdge.
This post describes how to build a 8 drive storage server for under a hundred bucks (drives not included).
Last summer I bought a Dell PowerEdge 2800, which I converted to use silent fans and SATA drives instead of SCSI drives. See the project page for this server for more information on how to hack the BMC’s firmware in order to swap the fans agains silent ones.
This post is the writeup of the conversion of the server’s backplane from SCSI to SATA. My main reason for this conversion was money: servers that offer a lot of swappable SATA drive slots are quite expensive. SCSI-based servers, on the other hand, are quite cheap – nobody uses plain SCSI anymore I guess. After searching a bit on the internet, I’ve bought a Dell PowerEdge 2800 that supports 8 SCSI drive slots for 25 bucks.
I gutted the SCSI backplane and replaced the connectors with SATA connectors. Adding in a 3Ware 9500S-12 PCI-X card, I’ve ended up with a raid system that supports 8 SATA drives. (Cold-swappable only, which is what I aimed for. This server is not a production server, obviously, but just our home storage).
In the following, I illustrate the steps how I converted the PowerEdge 2800 to SATA. It’s quite easy to do, so I hope I can inspire others. Apropos others: there is a similar project that was featured on Hackaday around the same time I’ve started with my project.
Assumptions:
Required material:
So, let’s start! Open up your server and remove the backplane. Obviously, you have to remove all drives before being able to detach the backplane. I say obviously after I ripped of the connector of my floppy drive by brute force – I removed the harddrives but forgot completely about the other drives. D’oh!
I first thought about replacing the whole backplane, but after I bought the server I realized that all the front-panel buttons, the optical drive etc are connected to the backplane. The backplane is therefore essential to the server, and the server does not even boot without it. So we have to leave the backplane in, fair enough.
The first step is to remove the old SCSI connectors from the backplane. Remove first the protective plasic shield from the backplane. Keep it, we will stick it back on later! The SCSI connectors are 68-pin bastards, so unsoldering them is out of the question. In the picture you can see the first connector removed – I pulled of the plasic part with some heavy pliers and stripped the pins one by one with a cutter.
Next, remove all connectors as shown in the picture. After removing the first ones by hand, I ended up using a Dremel to remove the pins. It does not have to be perfect, but pay attention to remove any short circuits. You do not want to disturb the underlying SCSI system that is still active in the server, or (even worse) short circuit the power planes!
Next, we are going to solder the harddisk access LEDs. This step is optional, but as everybody loves flashing LEDs this will give your project some street cred. No, seriously, the LEDs help you to see access on the harddisks and (if your controller supports it) to identify a specific disk/controller port.
The PowerEdge features 2 LEDs for each drive, of which we will use only one (does not matter which one). First, identify the traces on the backside of the backlane for each of the LEDs. Mark them with a pen according to drive number and if it’s the anode and the cathode. Cut the traces leading to the controller of the backplane leaving enough copper for you to solder a cable on.
In the picture you can see how I did this for one of the ports: one cable connected to the anode of the LED, and the other to the resistor, which in turn is connected to the cathod of the LED. Don’t forget to include the limiting resistor into the mix, because else you might burn your LEDs!
Here you can see how I connected all ports on one side of the backplane. Note that I soldered the cable on the back of the backplane (is that the frontplane?? ;) Remember to fix the cables in place with a little hot-glue. No hacking project can call itself a proper hacking project without a little hot-glue, right?
Next, we prepare the connectors to the raid controller. Most raid controllers support pin headers for connecting the drive LEDs. You’ve checked that your’s does support this before starting to solder the cables to the LEDs, right? Along the same lines, you’ve made the cables long enough to go all the way to the raid controller, even when all the fans etc are in the system? Ok, good.
Configure your connector according to your raid controllers pinout. In my case, the pinout was given in the manual of the raid controller: here’s the relevant page. Pay attention not to swap pins.
Reattach the isolating protective plastic shield and put back the power cable. Your backplane should look like in the picture above. Note that I’ve cut away the two bars at the bottom of the backplane’s PCB (at the bottom of the two big cutouts, which were closed before). This allows me to insert the backplane easier with all the cables in place, but it is not really required. If you do this on another type of system, make sure that there are no traces running in this part of the PCB.
Now we start doing the actual conversion to SATA. Mount all drives in their drive cages and stick them in your server. With the backplane still unmounted, you should be able to see the back of the drives. Here, I plugged in the angled all-in-one SATA connectors (I used Delock type 84307, which worked really good).
As the PowerEdge servers do not have many drive power connectors and we cannot reuse the power connectors of the backplane, we have to connect the power cables of the all-in-one connectors together. This also reduces the cable mess. In the picture you can see how I did it: add the connectors one by one: plug the all-in-one connector in the drive, cut away the power cable that is longer than the distance to the next drive, add the next connector and solder the two cables together. Repeat until you have the whole column completed. It’s hard to explain but easy enough to do, so have a look at the pictures.
This is how the end result should look like. Remember to properly protect all solder joints by shrink-tubes or similar. Use zip-ties to clean things up. In the end you’ll have a single power connector for each column of drives.
Next, we need to attach the SATA connectors to the back plane. The idea is that the connectors stay on the backplane when you pull a drive out – without this you would have to dismount the backplane each time you want to swap a drive. I thought about many different ways to do this, but in the end I chose to use a kind of construction glue to do it. There are for sure ways to do this in a cleaner way, but this serves the purpose and is quick and cheap.
The glue should have the following attributes:
Apply the glue to the back of the SATA connectors as shown in the picture (the white stuff). Be sure to clean the connectors and the backplane from any fat residues before doing so. Additionally, push your drives as far as possible to the front (there’s usually a little play in the cage mounting mechanism, and you’ll want it to be maxed out).
Then, put the backplane in. Wait.
This is how your server should now look at the interior. You can see the 8 SATA cables coming out of the cutouts at the bottom of the backplane, just along the two power connectors. Additionally, you can see the cables that will connect the drive LEDs to the RAID controller (the two gray cables).
After you’ve waited long enough (see glue instructions), you can remove and re-insert your drives to check if everything is in order. This is how your empty drive slots should look like.
Plug the power connectors and connect the SATA and LED cables to the RAID controller. Fire up your machine and check the drive status, and the drive LEDs. Remember that your drives are only cold-swappable, so you have to shut down your server before changing them.
Welcome to your new SATA-based storage server!
Adjusted lower critical thresholds for the fans of a PowerEdge 2800
Note: you MUST change your fans against slower, quieter ones to reduce the noise. The threshold adjustment discussed in this article only allows you to do so – without new fans, it’s useless!
In order to swap the fans on a Dell PowerEdge with slower, more quiet ones you have to adjust the lower critical threshold (LCR). If you don’t, the server’s firmware actually lowers the fan’s speed under it’s own LCR, panics, spins them back up a 100%, lowers them again etc. Very noisy, very annoying.
Previous, related posts:
This behavior is controlled by the BMC, an embedded management controller. You can configure many parameters of the BMC using the IPMI protocol. Unfortunately, the BMC’s firmware of a Dell PowerEdge does not allow to change the thresholds mentioned above. I contacted Dell support, and they refused to change the thresholds for such an old server.
So I had no choice but to change them myself. It took me quite a while to isolate the proper setting in the BMC’s firmware, the checksums etc. But I managed, and the server’s running now very quiet with adjusted thresholds.
Below, I explain how to adjust these thresholds with a python script I wrote. Note that you’ll need Python 2.6 in order to run the script. In case someone is interested I can also write up how I did it, but this is for another post.
Update: I created a project page for my server.
First, here’s the result: my PowerEdge 2800 with swapped fans and patched fan thresholds.
This has been recorded with my laptop, 10cm/4in in front of the server. The system is now more silent than my desktop!
I assume that you have a sufficiently recent Linux distribution up and running, with python installed and IPMI set up. If you don’t, have a look at this article that explains how to get a recent Ubuntu version running (without installing anything on your harddisk!).
I assume that you have now FreeIPMI installed, the BMC configured and that you can query the BMC using IPMI.
You have to note the part about the fans (d’oh). Record sensor numbers, fan names and thresholds (the value in brackets). You’ll need it later to identify your system.
Then execute:
This should extract your firmware. Check that you have a file called extracted/payload/bmcflsh.dat. If not, game over, your system isn’t compatible. If yes, yay!
The program is a python (version >= 2.6) script, that first lets you choose a system from the ones available in the firmware and the adjust the fan thresholds of this system. Yes, there can be support for multiple systems in a single firmware. You recorded the fan values before? Now you know why: you have to use it to identify your system from the ones the script shows to you. Just use the number of fans, their names and thresholds to identify your system. Maybe you’re lucky and the system name has already been found and is directly displayed.
In the next step you can select fans and change their threshold. Just remember that the result is a multiple of 75. Half the usual speed has proven to be a good value. I’ve never tested what happened if you set it to 0, but this would be quite stupid as you can’t detect broken fans.
If the program display a code at the end and asks you to report back, please do so! That way we can identify the other systems using their code (for example, the code of a PowerEdge 2800 is “K_C”).
Disclaimer: I am not responsible for any damage you do to your system! If you flash this firmware, you might render your PowerEdge server unusable. It might even be unrecoverable. Additionally, badly set thresholds might cause overheating.
Additionally, use the usual caution when flashing (do not interrrupt power, do not flash other a network link, do not be stupid).
Cross your fingers. The flasher should accept the firmware. If not and it complains about the CRC, something went wrong. Don’t worry if the fans speed up fully and go dead afterwards during the flash, that’s normal. The system should stabilize afterwards. There is not need to reboot.
That’s it. Enjoy your silent PowerEdge!
Some things that I learned while messing with the firmware:
Update 1: I created a project page for my server.
Update 2: I wrote this article that explains how to get a recent Ubuntu version running (without installing anything on your harddisk!). This is for all the Windows users out there!
Update 3: I moved the code of this project into a GitHub repository: http://projects.nuschkys.net/2012/04/06/how-to-get-ubuntu-live-running/ GitHub is great because people can easily collaborate, fork, submit issues and patches and so on.
Please don’t ask me basic Linux questions! Google is your friend. If you don’t know what you are doing, you shouldn’t be doing it as you might damage your server!
Adjusted lower critical thresholds for the fans of a PowerEdge 2800
More to follow soon…
Update: Here’s the howto. Additionally, I created a project page for my server.
This guide describes an easy method how to make your XBox360 DVD drive accessible from the outside so that you can easily reflash it whenever you want without going through the pain of unbuilding your XBox each time. Ah, and you’ll loose your warranty.
I guess others did this before, but I haven’t found anything on the web, so I figured I describe how to do it.
You need:
Disconnect the existing SATA cable from the DVD and the mainboard. Remove the DVD drive. It might be hard to find a longer cable with the proper angled connector, so here’s a closeup of it. It took me 3 tries to get the right one, so here are the DeLOCK model numbers:
Search for a good position to cut a hole into the casing and mark it. Protect the rest of your XBox with paper that you tape into position. In the end, only the soon-to-be-hole should be visible; all the rest should be under a thick layer of paper.
Take your Dremel and cut a hole into the XBox as marked, carefully avoiding to damage any of the components. Make the whole big enough that the two SATA cables can fit through.
Or, in my case: take your el-cheapo Dremel, start it up, notice the smell, realize the thing is going into meltdown, curse, burn yourself, unplug the thing, wait for the smoke to clear, open it up, yep, it actually melted, order a new one, be too impatient to wait, take out the drill and pipe wrench, and go ahead mistreat your XBox.
Carefully remove all the metal dust with a vaccum cleaner; remove the paper. End up with a nice and clean hole. Or in my case the worst exectured casemod ever.
Connect the angled SATA connector to the mainboard (just under the DVD drive, see first picture). Then, fiddle the cable through the hole. Fiddle the SATA extension cable through the hole as well, with the extension connector on the outside (d’oh). Plug the SATA connector into the DVD drive and put the drive back into your XBox. Connect the two SATA cables on the outside. It should now look like the image above.
Attach the SATA cables to the case using a zip-tie. This prevents the you from pulling the cable off the DVD drive when messing with the connectors on the outside. Use a strong wirecutter to cut a hole in the outer plastic casing. Reassemble your XBox.
You’re done! If you want to flash your drive, just connect a normal SATA cable from the extension plug to your PC (remember to have your XBox switched on and the video connector plugged in!)
Update 4/11/2011: I managed to find out some more info about the packaging scheme Dell uses for their BMC firmware files. I deciphered most of the container format. I am in the process of testing modifications right now, but for the moment I updated the version of the tool below with a new version. You can also download the program directly here: dell-extract-bmc-firmware.tar.gz.
Deciphering the firmware header.
I had a look into the firmware flash file, and noticed that it seems to consist of several files (as usual for BIOS/firmwares). As this might increase my chances not to brick my BMC, I decided to I separate the individual files for starters. I couldn’t find a program that does that (the firmware tools of the Dell linux community are closed-source, unfortunately), so I grabbed a hex-editor and deciphered (more or less) the firmware’s header. Here’s the corresponding C program:
The names of the individual files are listed below. They are organized in blocks (that’s what I call them), and apparently by function. Get the latest BMC firmware (30/6/2009, v1.83, A10) and apply my program to retrieve the individual files.
The BMC seems to be little-endian (makes only sense I guess). I’ve scanned the different files for appearances of the threshold values (900 and 2025/0x07e9 and 0x0384 in int and 0x6144000 in float). No avail. Darn. Either I am doing something wrong or the thesholds are not hard-coded in the firmware (I had my hopes up when I saw the OEM_DEF.FLC file, which actually contains the default BMC password and the like). Maybe the thresholds are stored in the configuration flash after all – only how can we access it?
Update
I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.
Earlier, I wrote about the problem of the noisy fans in my Dell PowerEdge 2800. Since then, I investigated a bit more. Just as a reminder: I can’t run silent fans because they have a lower RPM than a hardcoded panic-threshold of the PowerEdge. *grrr*
Brainstorm:
Well, as I mentioned earlier 1. and 2. are for obvious reasons dissatisfactory.
A couple of fans taken apart. Notice the blob of brown paint on the ring magnet of the fan on the right.
Concerning 3A, I took a couple of fans apart, looking at how they create the tacho signal. Almost all fans I opened (luckily I have a whole stack of noisy, throw-away fans lying around) have a sensor sitting just under the ring magnet, which is part of the rotor as can be seen in the photo on the right. I had no idea what this sensor might be, but I noticed on all of the fans one or more blows of brownish-red paint. I figured that this paint might be used to create a signal for the sensor – and I did some tests with other paint in order to replicate the effect (left fan on the photo with magnetic paint applied). Well, nice idea, but total bullshit as it turns out. The sensor is a hall-sensor that senses the change in the magnetic field of the ring-magnet, and thus changes inevitable 2 times per revolution. I figure that the paint applied on the rotor is used for calibrating the fans… Well, it was a nice idea.
3B might be an option, but it would require either a microcontroller or some analogue circuit – not really what I want to fiddle into the fan trays of 6 fans.
Concerning option 5, I thought that there may be hidden ipmi OEM commands for configuring the thresholds. I dug around the Dell OEM extensions for ipmitool (can be retrieved from the Dell Linux Community Repositories). This code officially earned worst code of the year – I completely understand why the ipmitool maintianers flatout refuse to integrate that piece of crap. It’s a hacked-up collection of extensions, seemingly done on the fly to fix customer problems. Horrible. Even more so as it does not seem to be able to set the thresholds either. After a few hours of digging in the code I managed to query the BMC sensors with Dell’s OEM commands, and the returned capability flags do indicate that the thesholds cannot be changed. Darn.
Now I am back to hacking the BMC firmware – but that’s for another post…
Update
I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.
The system has usually a limited number of users, and if you don’t logout properly multiple times you find yourself in the position that all slots have been taken. In this case, you have to wait for a timeout…
On Linux, the only supported browser is Firefox. On standard Ubunut, this does not work in combination with the DRAC as Ubuntu installs the Icetea Java plugin by default. Switch to the Sun Java plugin:
On many systems, you will have problems to access the DRAC (or at least the remote console) because of firewalls in your way. You can forward the whole thing by using a host inside the network as an proxy. For this, you have to tunnel the connections through SSH as follows:
Update: I created a project page for my server.
As I explained before, I changed all the fans of my PowerEdge 2800 to low-noise, low-RPM models.
The fans of my PowerEdge 2800. Front the cpu fans (stock Delta left, replacement everflow right); at the back the 120mm case fans (left the Artic, right the stock Nidec)
This was nice and shiny, and the system was quiet and quite cool. Too cool unfortunately, because the BMC throttles the fans too much so that their RPM values go below the configured thresholds. Then, the system goes into panic mode, spins up all fans at 100%, and lowers them successively below the threshold again. Repeat. :)
I ended up putting the Delta’s back in, because you cannot configure the lower failure threshold of the fans. So, lots of work for nothing, and the system is still way too loud. Very unsastifactory.
Thus, I started thinking how to get around this problem. Possible ways I see:
Well, 1. and 2. are for obvious reasons dissatisfactory.
Regarding 3: I thought about getting the sensor in the fan to report a higher value. Most fans actually consist of a ring-magnet. The sensor lies just below it, with the rotating magnet passing over it. On 1-4 places on the magnet, the manufacturers apply some paint. I guess it’s some sort of EMI shield and the sensor detects the change in the field – but I couldn’t find any paint that would reproduce the effect. This would be a very neat and nice solution.
Regarding 4: The BMC is implemented in some microcontroller on the motherboard of the PowerEdge. I haven’t found out yet where it is. I am not even sure what type of architecture it is, so reverse engineering the BMC firmware wasn’t possible. Damn, I don’t even know if this is little-endian or not.
Regarding 5: The manual states that these values are read-only and are to be configured by the OEM. But how? I doubt that they create a new firmware for each combination of fan manufacturers that they use. So my guess is that there are some hidden OEM IPMI commands that allow to set the threshold. I asked the guys over at FreeIPMI if they’ve got a clue, but they don’t know about any such functionality.
Anyone out there with some hints?
Update: I continued my search for a solution
Update 2
I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.
