Those darn fans!

As I explained before, I changed all the fans of my PowerEdge 2800 to low-noise, low-RPM models.

Fans of a PowerEdge 2800

The fans of my PowerEdge 2800. Front the cpu fans (stock Delta left, replacement everflow right); at the back the 120mm case fans (left the Artic, right the stock Nidec)

This was nice and shiny, and the system was quiet and quite cool. Too cool unfortunately, because the BMC throttles the fans too much so that their RPM values go below the configured thresholds. Then, the system goes into panic mode, spins up all fans at 100%, and lowers them successively below the threshold again. Repeat. :)

I ended up putting the Delta’s back in, because you cannot configure the lower failure threshold of the fans. So, lots of work for nothing, and the system is still way too loud. Very unsastifactory.

Thus, I started thinking how to get around this problem. Possible ways I see:

  1. make the fans faster/buy faster fans
  2. make the system hotter
  3. hack the fans into reporting more RPM than they actually do
  4. hack the BMC
  5. find out how the OEM sets these thresholds

Well, 1. and 2. are for obvious reasons dissatisfactory.

Regarding 3: I thought about getting the sensor in the fan to report a higher value. Most fans actually consist of a ring-magnet. The sensor lies just below it, with the rotating magnet passing over it. On 1-4 places on the magnet, the manufacturers apply some paint. I guess it’s some sort of EMI shield and the sensor detects the change in the field – but I couldn’t find any paint that would reproduce the effect. This would be a very neat and nice solution.

Regarding 4: The BMC is implemented in some microcontroller on the motherboard of the PowerEdge. I haven’t found out yet where it is. I am not even sure what type of architecture it is, so reverse engineering the BMC firmware wasn’t possible. Damn, I don’t even know if this is little-endian or not.

Regarding 5: The manual states that these values are read-only and are to be configured by the OEM. But how? I doubt that they create a new firmware for each combination of fan manufacturers that they use. So my guess is that there are some hidden OEM IPMI commands that allow to set the threshold. I asked the guys over at FreeIPMI if they’ve got a clue, but they don’t know about any such functionality.

Anyone out there with some hints?

UpdateI continued my search for a solution

Update 2

I finally managed to adjust the critical fan thresholds by patching the BMC firmware! Here’s the howto. Additionally, I created a project page for my server.

6 thoughts on “Those darn fans!

  1. This was an interesting read, going to read your continuation now. I have a similar issue with my re-purposed Google Search Appliance (basically a PowerEdge 2950).

  2. There seem to be quite some people out there with the same issue – all those re-purposed PowerEdges seem to end up in people’s offices and living rooms. :)

  3. Pingback: How to adjust the fan thresholds of a Dell PowerEdge | tüfteln

  4. So interesting thing happened just moments ago… My server over-heated and one DIMM reported as faulty. I cracked opened the case and found the memory to be scolding hot! Any idea how further adjust the BMC FW to get a better control over the fan speeds vs temp? I don’t mind the fans running faster as long as I can keep this server cool enough.

    • Well I didn’t have any choice but to revert back to the older 1.77 BMC firmware. The fans are all spinning now at 3000 rpm which is where they were originally. With the newer 2.50 firmware, the fans settled at 1500 rpm which was just too low to cool the server. I’m still happy with the noise level at 3k rpm. I’ll bet the OEM fans would still be louder at 1500 rpm than my aftermarket replacements at 3k rpm. Catch 22, huh?

Leave a Reply

Your email address will not be published. Required fields are marked *