Note: you MUST change your fans against slower, quieter ones to reduce the noise. The threshold adjustment discussed in this article only allows you to do so – without new fans, it’s useless!
In order to swap the fans on a Dell PowerEdge with slower, more quiet ones you have to adjust the lower critical threshold (LCR). If you don’t, the server’s firmware actually lowers the fan’s speed under it’s own LCR, panics, spins them back up a 100%, lowers them again etc. Very noisy, very annoying.
Previous, related posts:
- The battle againt the BMC – Part 2
- The battle againt the BMC – Part 1
- Those darn fans!
- Replacing the fans on a Poweredge 2800
This behavior is controlled by the BMC, an embedded management controller. You can configure many parameters of the BMC using the IPMI protocol. Unfortunately, the BMC’s firmware of a Dell PowerEdge does not allow to change the thresholds mentioned above. I contacted Dell support, and they refused to change the thresholds for such an old server.
So I had no choice but to change them myself. It took me quite a while to isolate the proper setting in the BMC’s firmware, the checksums etc. But I managed, and the server’s running now very quiet with adjusted thresholds.
Below, I explain how to adjust these thresholds with a python script I wrote. Note that you’ll need Python 2.6 in order to run the script. In case someone is interested I can also write up how I did it, but this is for another post.
Update: I created a project page for my server.
First, here’s the result: my PowerEdge 2800 with swapped fans and patched fan thresholds.
This has been recorded with my laptop, 10cm/4in in front of the server. The system is now more silent than my desktop!
I assume that you have a sufficiently recent Linux distribution up and running, with python installed and IPMI set up. If you don’t, have a look at this article that explains how to get a recent Ubuntu version running (without installing anything on your harddisk!).
Adjusting the fan thresholds
I assume that you have now FreeIPMI installed, the BMC configured and that you can query the BMC using IPMI.
- Query the sensors
First, you have to query the sensors of your server using IPMI. The output should look a bit like this:you@server$
You have to note the part about the fans (d’oh). Record sensor numbers, fan names and thresholds (the value in brackets). You’ll need it later to identify your system.
- Download the latest BMC firmware
Got to http://support.dell.com/support/downloads/ and get the latest BMC firmware for your system. Select any Linux OS; the BMC firmware should be listed under something like Embedded Server Management. On the download page, select the .BIN package. In my case the file was called BMC_FRMW_LX_R223079.BIN. Download it!
- Fix and extract .BIN package
In my case the .BIN package did not properly work. I had to fix it first, and then extract it. For this, open a terminal and go to the folder you’ve downloaded the package to.
Then execute:you@server$ sed -i 's/#!\/bin\/sh/#!\/bin\/bash/' BMC_FRMW_LX_R223079.BIN # fix interpreter bug
you@server$ chmod 755 BMC_FRMW_LX_R223079.BIN # make executable
you@server$ sudo mkdir bmc_firmware # create dir as root
you@server$ sudo ./BMC_FRMW_LX_R223079.BIN --extract bmc_firmware # yes, you have to do this as root! :(
you@server$ cd bmc_firmware
This should extract your firmware. Check that you have a file called extracted/payload/bmcflsh.dat. If not, game over, your system isn’t compatible. If yes, yay!
- Patch firmware
Next, download the program I wrote for patching the firmware. Then, use the program on the firmware as shown below:you@server$ wget https://raw.github.com/arnuschky/dell-bmc-firmware/master/adjust-fan-thresholds/dell-adjust-fan-thresholds.py
you@server$ chmod 755 dell-adjust-fan-thresholds.py
you@server$ ./dell-adjust-fan-thresholds.py payload/bmcflsh.dat
The program is a python (version >= 2.6) script, that first lets you choose a system from the ones available in the firmware and the adjust the fan thresholds of this system. Yes, there can be support for multiple systems in a single firmware. You recorded the fan values before? Now you know why: you have to use it to identify your system from the ones the script shows to you. Just use the number of fans, their names and thresholds to identify your system. Maybe you’re lucky and the system name has already been found and is directly displayed.
In the next step you can select fans and change their threshold. Just remember that the result is a multiple of 75. Half the usual speed has proven to be a good value. I’ve never tested what happened if you set it to 0, but this would be quite stupid as you can’t detect broken fans.
If the program display a code at the end and asks you to report back, please do so! That way we can identify the other systems using their code (for example, the code of a PowerEdge 2800 is “K_C”).
- Flash firmware
Finally, flash the firmware like as shown below.
Disclaimer: I am not responsible for any damage you do to your system! If you flash this firmware, you might render your PowerEdge server unusable. It might even be unrecoverable. Additionally, badly set thresholds might cause overheating.
Additionally, use the usual caution when flashing (do not interrrupt power, do not flash other a network link, do not be stupid).you@server$ LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f
Cross your fingers. The flasher should accept the firmware. If not and it complains about the CRC, something went wrong. Don’t worry if the fans speed up fully and go dead afterwards during the flash, that’s normal. The system should stabilize afterwards. There is not need to reboot.
- Check the sensors
Check that everything is in order:you@server$ ipmi-sensors
That’s it. Enjoy your silent PowerEdge!
Some things that I learned while messing with the firmware:
- There can be multiple systems per firmware
- Generally it’s quite well engineered
- I’ve found Dell’s default password root/calvin. What is the 444444 for?
- Dell server systems seem to be named internally after cities. BER, LOND, OSLO etc are easy enough to guess. But what the hell is K_C??? (my system)
- The firmware package is probably the most horrible over-engineered script I’ve ever met on Linux
- Dell uses CRC-16 for checksum – two different algorithms in the same firmware!
Update 1: I created a project page for my server.
Update 2: I wrote this article that explains how to get a recent Ubuntu version running (without installing anything on your harddisk!). This is for all the Windows users out there!
Update 3: I moved the code of this project into a GitHub repository: http://projects.nuschkys.net/2012/04/06/how-to-get-ubuntu-live-running/ GitHub is great because people can easily collaborate, fork, submit issues and patches and so on.
Please don’t ask me basic Linux questions! Google is your friend. If you don’t know what you are doing, you shouldn’t be doing it as you might damage your server!