How to adjust the fan thresholds of a Dell PowerEdge

Adjusted lower critical thresholds for the fans of a PowerEdge 2800

Note: you MUST change your fans against slower, quieter ones to reduce the noise. The threshold adjustment discussed in this article only allows you to do so – without new fans, it’s useless!

Intro

In order to swap the fans on a Dell PowerEdge with slower, more quiet ones you have to adjust the lower critical threshold (LCR). If you don’t, the server’s firmware actually lowers the fan’s speed under it’s own LCR, panics, spins them back up a 100%, lowers them again etc. Very noisy, very annoying.

Previous, related posts:

This behavior is controlled by the BMC, an embedded management controller. You can configure many parameters of the BMC using the IPMI protocol. Unfortunately, the BMC’s firmware of a Dell PowerEdge does not allow to change the thresholds mentioned above. I contacted Dell support, and they refused to change the thresholds for such an old server.

So I had no choice but to change them myself. It took me quite a while to isolate the proper setting in the BMC’s firmware, the checksums etc. But I managed, and the server’s running now very quiet with adjusted thresholds.

Below, I explain how to adjust these thresholds with a python script I wrote. Note that you’ll need Python 2.6 in order to run the script. In case someone is interested I can also write up how I did it, but this is for another post.

Update: I created a project page for my server.

The result

First, here’s the result: my PowerEdge 2800 with swapped fans and patched fan thresholds.

This has been recorded with my laptop, 10cm/4in in front of the server. The system is now more silent than my desktop!

Prerequisites

I assume that you have a sufficiently recent Linux distribution up and running, with python installed and IPMI set up. If you don’t, have a look at this article that explains how to get a recent Ubuntu version running (without installing anything on your harddisk!).

Adjusting the fan thresholds

I assume that you have now FreeIPMI installed, the BMC configured and that you can query the BMC using IPMI.

  1. Query the sensors
    First, you have to query the sensors of your server using IPMI. The output should look a bit like this:

    you@server$ ipmi-sensors
    1: Temp (Temperature): NA (NA/125.00): [NA]
    2: Temp (Temperature): NA (NA/125.00): [NA]
    3: Ambient Temp (Temperature): NA (3.00/47.00): [NA]
    4: Planar Temp (Temperature): NA (3.00/72.00): [NA]
    5: Riser Temp (Temperature): NA (3.00/62.00): [NA]
    6: Temp (Temperature): NA (NA/NA): [NA]
    7: Temp (Temperature): NA (NA/NA): [NA]
    8: Temp (Temperature): 71.00 C (NA/125.00): [OK]
    9: Temp (Temperature): NA (NA/125.00): [NA]
    10: Ambient Temp (Temperature): 27.00 C (3.00/47.00): [OK]
    11: Planar Temp (Temperature): 46.00 C (3.00/72.00): [OK]
    12: Riser Temp (Temperature): 50.00 C (3.00/62.00): [OK]
    13: Temp (Temperature): NA (NA/NA): [NA]
    14: Temp (Temperature): NA (NA/NA): [NA]
    15: CMOS Battery (Voltage): NA (2.64/NA): [NA]
    16: ROMB Battery (Voltage): [NA]
    17: VCORE (Voltage): [State Deasserted]
    18: VCORE (Voltage): [NA]
    19: PROC VTT (Voltage): [State Deasserted]
    20: 1.5V PG (Voltage): [State Deasserted]
    21: 1.8V PG (Voltage): [State Deasserted]
    22: 3.3V PG (Voltage): [State Deasserted]
    23: 5V PG (Voltage): [State Deasserted]
    24: 5V Riser PG (Voltage): [State Deasserted]
    25: Riser PG (Voltage): [State Deasserted]
    26: CMOS Battery (Voltage): 3.11 V (2.64/NA): [OK]
    27: Presence  (Entity Presence): [Entity Present]
    28: Presence  (Entity Presence): [Entity Absent]
    29: Presence  (Entity Presence): [Entity Present]
    30: Presence  (Entity Presence): [Entity Absent]
    31: ROMB Presence (Entity Presence): [Entity Present]
    32: FAN 1 RPM (Fan): NA (1575.00/NA): [NA]
    33: FAN 2 RPM (Fan): NA (1575.00/NA): [NA]
    34: FAN 3 RPM (Fan): NA (1575.00/NA): [NA]
    35: FAN 4 RPM (Fan): NA (1575.00/NA): [NA]
    36: FAN 5 RPM (Fan): NA (1575.00/NA): [NA]
    37: FAN 6 RPM (Fan): NA (1575.00/NA): [NA]
    38: FAN 1 RPM (Fan): NA (2025.00/NA): [NA]
    39: FAN 2 RPM (Fan): NA (2025.00/NA): [NA]
    40: FAN 3 RPM (Fan): 4875.00 RPM (2025.00/NA): [OK]
    41: FAN 4 RPM (Fan): 4800.00 RPM (2025.00/NA): [OK]
    42: FAN 5 RPM (Fan): 1800.00 RPM (900.00/NA): [OK]
    43: FAN 6 RPM (Fan): 1950.00 RPM (900.00/NA): [OK]
    44: FAN 7 RPM (Fan): 1875.00 RPM (900.00/NA): [OK]
    45: FAN 8 RPM (Fan): 1875.00 RPM (900.00/NA): [OK]
    46: Status  (Processor): [Processor Presence detected]
    47: Status  (Processor): [NA]
    48: Status  (Power Supply): [Presence detected]
    49: Status  (Power Supply): [NA]
    50: VRM  (Power Supply): [Presence detected]
    51: VRM  (Power Supply): [Presence detected]
    52: OS Watchdog (Watchdog 2): [OK]
    53: SEL (Event Logging Disabled): [Unknown]
    54: Intrusion (Physical Security): [OK]
    55: PS Redundancy (Power Supply): [NA]
    56: Fan Redundancy (Fan): [Fully Redundant]
    73: SCSI Connector A (Cable/Interconnect): [NA]
    74: SCSI Connector B (Cable/Interconnect): [NA]
    75: SCSI Connector A (Cable/Interconnect): [NA]
    76: Drive (Slot/Connector): [NA]
    77: Drive (Slot/Connector): [NA]
    78: 1x2 Drive (Slot/Connector): [NA]
    79: Secondary (Module/Board): [NA]
    80: ECC Corr Err (Memory): [Unknown]
    81: ECC Uncorr Err (Memory): [Unknown]
    82: I/O Channel Chk (Critical Interrupt): [Unknown]
    83: PCI Parity Err (Critical Interrupt): [Unknown]
    84: PCI System Err (Critical Interrupt): [Unknown]
    85: SBE Log Disabled (Event Logging Disabled): [Unknown]
    86: Logging Disabled (Event Logging Disabled): [Unknown]
    87: Unknown (System Event): [Unknown]
    88: CPU Protocol Err (Processor): [Unknown]
    89: CPU Bus PERR (Processor): [Unknown]
    90: CPU Init Err (Processor): [Unknown]
    91: CPU Machine Chk (Processor): [Unknown]
    92: Memory Spared (Memory): [Unknown]
    93: Memory Mirrored (Memory): [Unknown]
    94: Memory RAID (Memory): [Unknown]
    95: Memory Added (Memory): [Unknown]
    96: Memory Removed (Memory): [Unknown]
    97: PCIE Fatal Err (Critical Interrupt): [Unknown]
    98: Chipset Err (Critical Interrupt): [Unknown]
    99: Err Reg Pointer (OEM Reserved): [Unknown]

    You have to note the part about the fans (d’oh). Record sensor numbers, fan names and thresholds (the value in brackets). You’ll need it later to identify your system.

  2. Download the latest BMC firmware
    Got to http://support.dell.com/support/downloads/ and get the latest BMC firmware for your system. Select any Linux OS; the BMC firmware should be listed under something like Embedded Server Management. On the download page, select the .BIN package. In my case the file was called BMC_FRMW_LX_R223079.BIN. Download it!

  3. Fix and extract .BIN package
    In my case the .BIN package did not properly work. I had to fix it first, and then extract it. For this, open a terminal and go to the folder you’ve downloaded the package to.

    Then execute:

    you@server$ sed -i 's/#!\/bin\/sh/#!\/bin\/bash/' BMC_FRMW_LX_R223079.BIN  # fix interpreter bug
    you@server$ chmod 755 BMC_FRMW_LX_R223079.BIN                              # make executable
    you@server$ sudo mkdir bmc_firmware                                        # create dir as root
    you@server$ sudo ./BMC_FRMW_LX_R223079.BIN --extract bmc_firmware          # yes, you have to do this as root! :(
    you@server$ cd bmc_firmware

    This should extract your firmware. Check that you have a file called extracted/payload/bmcflsh.dat. If not, game over, your system isn’t compatible. If yes, yay!

  4. Patch firmware
    Next, download the program I wrote for patching the firmware. Then, use the program on the firmware as shown below:

    you@server$ wget https://raw.github.com/arnuschky/dell-bmc-firmware/master/adjust-fan-thresholds/dell-adjust-fan-thresholds.py
    you@server$ chmod 755 dell-adjust-fan-thresholds.py
    you@server$ ./dell-adjust-fan-thresholds.py payload/bmcflsh.dat

    The program is a python (version >= 2.6) script, that first lets you choose a system from the ones available in the firmware and the adjust the fan thresholds of this system. Yes, there can be support for multiple systems in a single firmware. You recorded the fan values before? Now you know why: you have to use it to identify your system from the ones the script shows to you. Just use the number of fans, their names and thresholds to identify your system. Maybe you’re lucky and the system name has already been found and is directly displayed.

    In the next step you can select fans and change their threshold. Just remember that the result is a multiple of 75. Half the usual speed has proven to be a good value. I’ve never tested what happened if you set it to 0, but this would be quite stupid as you can’t detect broken fans.

    If the program display a code at the end and asks you to report back, please do so! That way we can identify the other systems using their code (for example, the code of a PowerEdge 2800 is “K_C”).

  5. Flash firmware
    Finally, flash the firmware like as shown below.

    Disclaimer: I am not responsible for any damage you do to your system! If you flash this firmware, you might render your PowerEdge server unusable. It might even be unrecoverable. Additionally, badly set thresholds might cause overheating.

    Additionally, use the usual caution when flashing (do not interrrupt power, do not flash other a network link, do not be stupid).

    you@server$ LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

    Cross your fingers. The flasher should accept the firmware. If not and it complains about the CRC, something went wrong. Don’t worry if the fans speed up fully and go dead afterwards during the flash, that’s normal. The system should stabilize afterwards. There is not need to reboot.

  6. Check the sensors
    Check that everything is in order:

    you@server$ ipmi-sensors

    That’s it. Enjoy your silent PowerEdge!

Trivia

Some things that I learned while messing with the firmware:

  • There can be multiple systems per firmware
  • Generally it’s quite well engineered
  • I’ve found Dell’s default password root/calvin. What is the 444444 for?
  • Dell server systems seem to be named internally after cities. BER, LOND, OSLO etc are easy enough to guess. But what the hell is K_C??? (my system)
  • The firmware package is probably the most horrible over-engineered script I’ve ever met on Linux
  • Dell uses CRC-16 for checksum – two different algorithms in the same firmware!

Update 1: I created a project page for my server.

Update 2: I wrote this article that explains how to get a recent Ubuntu version running (without installing anything on your harddisk!). This is for all the Windows users out there!

Update 3: I moved the code of this project into a GitHub repository: http://projects.nuschkys.net/2012/04/06/how-to-get-ubuntu-live-running/ GitHub is great because people can easily collaborate, fork, submit issues and patches and so on.

Please don’t ask me basic Linux questions! Google is your friend. If you don’t know what you are doing, you shouldn’t be doing it as you might damage your server!

309 thoughts on “How to adjust the fan thresholds of a Dell PowerEdge

  1. Pingback: Dell PowerEdge Fan Replacement | tüfteln

  2. Pingback: Those darn fans! | tüfteln

  3. Pingback: Dell PowerEdge BMC Firmware Fan Thresholds Adjustment | tüfteln

  4. I found your article on changing the Dell PowerEdge system fan thresholds and would like your help on changing them on my dell PowereEge 2900.
    I have replaced the system fans and experience the same issue with the MIN RPM threshold (1275 RPM) being triggered and spinning up at 100%
    I am at work and dont have a linux machine available to extract the BMC Firmware to check if there is a bmcflsh.dat
    The output of ipmish on the fan sensors with the original fans installed are below, the new quiet and optimal fans run at a much lower speed pushing the same CFM.
    Could you please help me build a new BMC Firmware for my server?
    Thanks

    Index : 18
    Status : Normal
    Probe Name : System Board FAN 1 RPM
    Reading : 3300 RPM
    Minimum Warning Threshold : N/A
    Maximum Warning Threshold : N/A
    Minimum Failure Threshold : 1275 RPM
    Maximum Failure Threshold : N/A

    Index : 19
    Status : Normal
    Probe Name : System Board FAN 2 RPM
    Reading : 3300 RPM
    Minimum Warning Threshold : N/A
    Maximum Warning Threshold : N/A
    Minimum Failure Threshold : 1275 RPM
    Maximum Failure Threshold : N/A

    Index : 20
    Status : Normal
    Probe Name : System Board FAN 3 RPM
    Reading : 3300 RPM
    Minimum Warning Threshold : N/A
    Maximum Warning Threshold : N/A
    Minimum Failure Threshold : 1275 RPM
    Maximum Failure Threshold : N/A

    Index : 21
    Status : Normal
    Probe Name : System Board FAN 4 RPM
    Reading : 3225 RPM
    Minimum Warning Threshold : N/A
    Maximum Warning Threshold : N/A
    Minimum Failure Threshold : 1275 RPM
    Maximum Failure Threshold : N/A

    Index : 22
    Status : Normal
    Probe Name : System Board FAN 5 RPM
    Reading : 3225 RPM
    Minimum Warning Threshold : N/A
    Maximum Warning Threshold : N/A
    Minimum Failure Threshold : 1275 RPM
    Maximum Failure Threshold : N/A

    Index : 23
    Status : Normal
    Probe Name : System Board FAN 6 RPM
    Reading : 3300 RPM
    Minimum Warning Threshold : N/A
    Maximum Warning Threshold : N/A
    Minimum Failure Threshold : 1275 RPM
    Maximum Failure Threshold : N/A

    • You have to be able to flash the standalone bmcflsh.dat under Windows – if you are not you need Linux in any way, Is this possible in Windows? It might well be that Windows does not allow you to do this, because it gives you only a one-in-all package.

      If you can extract your Windows package so that you can flash the bmcflsh.dat file separately, you can send it to me and I can adjust the thresholds for you without any problem.

      Please not that I do not offer to modify firmware files.

      • hi
        the dell bmc download is an executable but can be unpacked with winrar. In the unpacked folder can be found a bmcfl32w.exe and a folder called payload with the bmcflsh.dat. how nice of them. obviously the dat file has to be modified, so there you go.
        red

    • Hello,
      I’ve bought DELL POWEREDGE 2950 III with 2x CPU and 32Gb RAM.
      I’ve encounter the same problem as all people here… noise, noise, noise.

      As this server is for studying CCIE using CSR1000V routers as a home LAB, I let you imagine how I can concentrate on the video course, exercises and so on having this Noisy thing just near me.
      I then followed this procedure which run from first steps to last steps without problem, but also without results.
      My fans are still running at 8400 RPM…. which seems to be half the speed this turboreactor can do (proof of this at each power on).

      So I’ve used FC17 Live CD, ran the procedure, and here are my values :

      - Firmware for my system is : 2950_ESM_Firmware_4NNNG_LN32_2.50_A00
      - lpmi-sensors –group=Fan returns me :
      22 | FAN 1 RPM | Fan | 8925.00 | RPM | ‘OK’
      23 | FAN 2 RPM | Fan | 9225.00 | RPM | ‘OK’
      24 | FAN 3 RPM | Fan | 8925.00 | RPM | ‘OK’
      25 | FAN 4 RPM | Fan | 8775.00 | RPM | ‘OK’
      26 | FAN 5 RPM | Fan | N/A | RPM | N/A
      27 | FAN 6 RPM | Fan | N/A | RPM | N/A
      46 | Fan Redundancy | Fan | N/A | N/A | ‘Fully Redundant’

      When I run the python script, I get the following choices :

      Systems found in firmware file:

      1) PowerEdge 2900
      Number of fans: 8
      Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM, FAN 7 RPM, FAN 8 RPM
      Fan speeds : 1275, 1275, 1275, 1275, 1275, 1275, 2400, 2400
      Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55

      2) unknown PowerEdge (code: BRCL)
      Number of fans: 0
      Fan names :
      Fan speeds :
      Sensor numbers:

      3) PowerEdge 2950
      Number of fans: 6
      Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
      Fan speeds : 2025, 2025, 2025, 2025, 2250, 2250
      Sensor numbers: 48, 49, 50, 51, 52, 53

      4) unknown PowerEdge (code: BRLN)
      Number of fans: 16
      Fan names : FAN MOD 1A RPM, FAN MOD 1B RPM, FAN MOD 1C RPM, FAN MOD 1D RPM, FAN MOD 2A RPM, FAN MOD 2B RPM, FAN MOD 2C RPM, FAN MOD 2D RPM, FAN MOD 3A RPM, FAN MOD 3B RPM, FAN MOD 3C RPM, FAN MOD 3D RPM, FAN MOD 4A RPM, FAN MOD 4B RPM, FAN MOD 4C RPM, FAN MOD 4D RPM
      Fan speeds : 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575
      Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63

      5) PowerEdge 1900 (Gen I)
      Number of fans: 6
      Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
      Fan speeds : 1275, 1275, 1275, 1275, 1275, 1275
      Sensor numbers: 48, 49, 50, 51, 52, 53

      6) unknown PowerEdge (code: FT)
      Number of fans: 6
      Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
      Fan speeds : 1275, 1275, 1275, 1275, 2400, 2400
      Sensor numbers: 48, 49, 50, 51, 54, 55

      7) unknown PowerEdge (code: GNS)
      Number of fans: 8
      Fan names : FAN MOD 1A RPM, FAN MOD 1C RPM, FAN MOD 1B RPM, FAN MOD 1D RPM, FAN MOD 2A RPM, FAN MOD 2C RPM, FAN MOD 2B RPM, FAN MOD 2D RPM
      Fan speeds : 2175, 2175, 2175, 2175, 2175, 2175, 2175, 2175
      Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55

      8) unknown PowerEdge (code: BULN)
      Number of fans: 6
      Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
      Fan speeds : 1275, 1275, 1275, 1275, 2400, 2400
      Sensor numbers: 48, 49, 50, 51, 54, 55

      Select (1-8):

      First, I’ve tried choice ’3′ (2950). Flash is ok but no effect on the noise. I’ve tested with half the original values, and a quarter of originals values.

      Then I’ve also modified section 4, 5, 6 and 8 to set values half of their default values and I’ve then flashed the system, in case it was another system code

      Each time, the script result is ok :

      Writing sensor CRC 0x9bff
      Writing file CRC 0xba58
      Verifying sensor block…
      Verifying file…

      All done.

      Each time the flash is ok :

      [root@localhost bmc_firmware]# LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

      Baseboard Management Controller Flash Utility Version: 1.33
      Copyright 2003-2007 Dell Inc. All Rights Reserved.
      Please wait… -./disneyinstall.sh: line 592: /var/log/dell/disneyinstall.log: No such file or directory
      Loading file completed CRC OK

      Firmware Versions Current Image
      BMC …………………………………… 2.50 2.50

      Force flash BMC
      Proceed with flash? Y/N: Y
      Processing BMC
      Flash Completed
      Checking Revision completed

      So my questions are :

      - Why do I have almost 9000 RPM where the max configured in the BMC firmware is around 2200 RPM ?
      - Am I using the good choice in the python script, as my system runs with only 4 FANS, not 6, 8 or 16 ?
      - How can I identify my system code to be sure to use ’2950′ ? And what would be the good values ?
      - Is it possible that there are other system code that the scripts doesn’t identify correctly in the firmware ?
      - Should I run half the value in each of the eight sections ? so at least one will be correct ?
      - Why do I have in Section 2 something empty (Code : BRCL), is my system BRCL so it can’t be adjusted ?

      Many thanks for your help.
      Franck.

      • I would be surprised if you can get the fans to go any slower. Although I would not be happy running low fan speeds with dual cpus and that much ram. My 2900 for example with only 16GB and a single cpu hits 110oC on the memory under heavy load.

  5. Hi Arnuschky.

    I’ve tried following your instructions (also for a PE2900). I’ve extracted the bmcflsh.dat in RHEL and downloaded your python script, however when I run it I get:

    [root@localhost user]# ./dell-adjust-fan-thresholds.py bmcflsh.dat
    File “./dell-adjust-fan-thresholds.py”, line 141
    class FileHeader():
    SyntaxError: invalid syntax

    (I have moved the bmcflsh.dat from the extracted folder into the same dir as the script).

    Any ideas? Getting this working would be awesome! I can send you the bmcflsh.dat if required. (This will be same one that Sam is needing also)

  6. Just checked on this RHEL5 and the Python is 2.4.3.

    I’ve got an Ubuntu box running Python 2.6.5 and the script appears to properly execute and I can change the values. Upon writing (w) however I get a CRC error from the py script and subsequently cannot re-open the dat file once it has been edited.
    Is this expected due to the firmware file being modified or would it indicate a problem?

    • Just for more information, the message I get when I try to re-open the edited file (changing the 6x fan values stated above):

      [user@localhost fw]$ su -c ‘./dell-adjust-fan-thresholds.py bmcflsh.dat’
      Password:
      Opening file ‘bmcflsh.dat’
      Verifying file…
      Error: file CRC failed (0xfed7 vs. 0xc4a6)!

    • Hey,

      I sorry, this clearly indicates that my script did not work. As I wasn’t able to test it with other systems, there might be something different in a generation 9 system (2900 vs. 2800). Do NOT flash this firmware (the tool should not allow you to, but don’t take risks).

      I’ve downloaded your firmware already; if you tell me what exactly you’ve changed (system code and new thresholds) I can maybe backtrack your steps and debug the problem. Can’t promise though, but I might have some time over the holidays.

      • Hi Again, – No I’m not Sam, just another guy who’s been following your project with interest :)

        There were 8x different models in there. I presumed PE2900 was either 1 or 5 (Sam stated that there are 6x fans at 1275RPM). I think one option has 6x 1275RPM + 2x 2400RPM and another has 6 x 1275 RPM.

        I tried changing 5 (6×1275 RPM) to a value of 10 (750RPM) for all 6 fans and then w to write changes.

        Hope this helps and thanks in advance for your hard work!

        Just a note to anyone else who is swapping these fans; I checked the temperature of my SAS drives after fitting the nexus fans and they were red hot (too hot to hold). I have replaced the two modded fans behind the HDD module back with the original units.

        Hope this helps,

        • I’ve checked, my script definately does something wrong there – there’s a bug somewhere, I will try to track it down.

          Regarding the temperature of the HDDs: I’ve also tracked it for a while after changing my fans, and I do not have any problems (8x 1GB SATA):

          3Ware RAID5 Port0 temp: 47 degrees C
          3Ware RAID5 Port1 temp: 57 degrees C
          3Ware RAID5 Port2 temp: 53 degrees C
          3Ware RAID5 Port3 temp: 46 degrees C
          3Ware RAID5 Port4 temp: 41 degrees C
          3Ware RAID5 Port5 temp: 50 degrees C
          3Ware RAID5 Port6 temp: 51 degrees C
          3Ware RAID5 Port7 temp: 44 degrees C

          Which is well inside the specs, if not the optimal I have to admit…

          • I think it’s mainly for the 15K SAS drives as they will generate more heat – I have a 4x 300GB 15K SAS drives in the top 4 bays and 4x 1TB SATA drives in the bottom bays. The SATA drives were running at normal temperature with the modded fans while the SAS drives were too hot to hold :)

        • Good news! I found and fixed the bug! Luckily, it was a very simple bug in the script and not a problem with my understanding of the firmware’s structure.

          Your CRC error occurred just because the script terminated before it could update the CRC properly. Redownload the script and use it on an unchanged bmcflsh.dat; it should work alright.

          I would appreciate it if you could report back if you’re successful with this change, and tell me which system code corresponds to your PE 2900.

          Disclaimer: I am not responsible for any damage you do to your system! If you flash this firmware, you might render your PowerEdge server unusable. It might even be unrecoverable. Additionally, badly set thresholds might cause overheating.

          Good luck!

          • Works a treat! Tried the code MEC with the 6 x 1275RPM fans first, but that didn’t make a difference so went back to the MNTRL option with the 8 x fans. – Presumably the 2 x 2400RPM are the PSU fans. After a second flashing the fans are running nice and quiet with no more flashing orange warnings!

            I can therefore say that the PE2900 code is MNTRL

            Many thanks – You are a true gent and a coding genius!

            FYI Sam, the bmcflash.dat is the same in the Windows package as the linux package and can be force flashed (I was already on latest version) from a DOS prompt with the -f switch.

          • Happy to hear that it works – thanks a lot for your feedback and motivating words!

            I updated the script so future users will have it easier to identify their system (and I added a check for the python version).

  7. Hey guys Sam here!
    Looks like we have made some progress, nice work.
    Im going to give this a try soon, I just need to figure out how im going to flash my PE2900. As its currently running ESXi5.
    Im thinking about using a linux boot DVD, do you see any problem with this?
    It would be so much easier if Dell had a bootable cd for this flash.

      • Hi Sam,

        The download options on Dell should give you a Dos option which has its own dos flash utility. – I ran this from a bootable DOS usb pendrive and had no problems. You’ll need to use it with the F switch if you’re already on the latest version.

        Just another warning to anyone with 15K SAS drives; they WILL bake with the slower nexus fans. They must rely on the airflow from the two lower front fans. SATA drives seem to be OK.

        • Thanks for that info Dave.
          I already have a USB boot drive with DOS on it, so will use this method. Im just having some problems getting this file patched (linux problems), because we have the same Dell PE 2900 can I use your already patched bmcflash.dat file?

  8. Good job. Perhaps K_C stands for “Kansas City”, the underscore being there because it’s a two word name. If so, it would be an odd inclusion next to London and Berlin,

    • Well, there seems also to be Melbourne, Montreal and San Francisco, so it’s not only Europe based. Anyways, just a curiosity… Thanks for the hint, and the feedback!

    • Thanks for that :)
      Hrm now you have me nervous… ok I will try patch my own file. Maybe you can help me out with this.. When I try to extract the bin file downloaded from Dell I get the following error;
      Cannot find utilities on the system to execute package.
      Make sure the following utilities are in the path:
      rpm stat stty cut fmt tty tar gzip tail rm mkdir mktemp chmod ls basename dirname wc sleep
      This is a fresh ubuntu install (in a VM)

      • Update ::
        OK I have flashed my PE 2900 successfully. I first used the option 5 but this didnt have any effect on my machine, then used option 1 from your script and changed the first 6 fans.. left the last two at default values.
        Thank-You very much arnuschky and DaveG
        Also congrats on the recent popularity arnuschky, I have noticed a few articles on different sites I read referring to your project page. Probably why your web server was down for a bit there.

          • Many thanks everyone – I was struggling for a while I’m guessing as I was running an older version of python.

            I’ve used the modded file kindly provided, and now my 2900 II also has the thresholds for all the 6 fans reduced to 750rpm.

            Many thanks arunschky and Sam!

        • Ok, just to followup, “sudo apt-get rpm” was the only thing I needed under Ubuntu 12.04 to resolve this issue. So it seems you simply have to go through the list and make sure each is installed.

          • not to correct you as such aaronls, but i noticed something
            “sudo apt-get rpm” , is it an install?
            if so for anyone who reads this it is for help purposes

            if so the coding / syntax to install rpm would be
            “sudo apt-get install rpm”

            as a deb based system would complain without one of its key words before the product / app / tool you want to install , and would confuse it by trying to take “rpm” as a command word ;-)

            hope this helps

  9. I have a 1900 gen 2.
    1) Can that be done by using this method
    2) Can it be done under windows using server 2008
    3) Do you have to change the fans or are you slowing down the fans already in the system.
    Thanks
    Glen (UK)

    • Hey Glen,

      1) I think it can be done, but as I don’t have a 1900 myself I can’t guarantee it.
      2) What you can do it like Sam and others: use a linux live cd (doesn’t install anything) and program to create the adjusted firmware file, then flash it with a dos bootdisk or similar.
      3) You have to change fans. What I do only allows to put in fans that turn slower than the panic thresholds for the existing ones. It does not affect the the speed at which the fans turn.

      Hope that helped,
      Arnuschky

      • Thanks for your prompt reply Amuschky.
        Ok so from what I can see I need to download a live cd type some commands and see what it gives me back…. I can from what I can see set the fans lower even with the fans that are noisey in it now before I change them….
        Ill try that and report back then.. before I flash however.

        Thanks

        Glen

      • I dropped the changes on a few of my “production” (if you can call it that) servers this morning before heading out to work- they SOUNDED quieter, but when i check the fan RPM in vSphere client, they still show as around 5-6k rpm…

        the ones i did the initial setup on were done in linux, and ipmi-sensors didn’t seem to reflect a change for the running speed, but the threshold seemed changed-
        (2025.00/NA) was the original value for the 4 “real” fans- I changed them to 10x (750) and they most definitely ARE slower, but the ipmi sensors command still shows 5-6k to the left of the /NA value… is everything ok, or is ipmi somehow flawed?

        My ESXi 5.0 install is done using the dell OEM discs, so I’m unsure if it’s padding the values?

        • Daniel,

          Forgive me for asking the obvious but you have changed the fans as well haven’t you?
          All this script does is modify the failure thresholds, rather than the RPM the fans run at. The RPM would depend on the model of the fans fitted I imagine. My stock Dell ones idled at around 2500RPM wheras the replacements run at ~1500RPM.

          Regards,
          Dave

        • You did exchage the fans against quieter ones, right? My patch only allows to swap fans – the stock firmware does not allow to do this because of the hardcoded thresholds.

          If you flash the patched firmware without changing the fans it should not change anything.

          • Yes- I saw that thread, that was more or less my next plan- the odd part is that the system definitely seems substantially quieter…

            The 2950 has been criticized for horrendously loud fans whereas the 2850 is apparently quite quiet- I’m not sure if this did something directly but it does seem to have made the system quite a bit quieter. I’m still looking for some decent replacement fans as they need to be 60mm…

            I guess I’ll check into replacing the fans- I had been hoping (and somehow there has been a bit of the desired effect (or a damn good placebo) that this would lower the normal idling speed, as this is this main issue on the 2950.

          • Yes, you are right, it’s quite hard to find quiet 60mm PWM fans as they are out of scope for the average home/desktop user. In the end I’ve settled for some Delta AFB0612EHE, which seem to be quite sturdy (especially in comparison with the flimsy Arctics). They don’t have the same depth as the original fans but this does not matter.

            Weird that you have the impression that your server is a bit quieter with the patched firmware – I have no explanation for this…

          • Sorry, my bad. I replaced the stock Deltas AFB0612EHE with Everflows F126025BU. See the other post; in my reply I confused the models. Hope nobody went to buy the Deltas!

          • awesome- thanks, i’ll try a few of these- I think the recent cold temperatures have been the reason for my quiet servers after the change, nothing to do with the thresholds as you suggested, but those fans look nice!

            thanks again for all the help!

          • A lower ambient temperature would certainly explain the reduced noise. :)

            Happy to help – it’s gratifying to know that I didn’t do all that work just for a single system (and a lot of work it was…)

          • Daniel

            Did you ever replace the fans on the PowerEdge 2950? I’m wanting to do it also, but it looks like the fans that are referenced are 60x60x25mm. The stock fans are 60x60x38mm. I’ve had a very hard time finding any 38mm fans that support PWM except for the Deltas that come stock. Please let me know if you have had any luck with this.

            Thanks

  10. I dropped the changes on a few of my “production” (if you can call it that) servers this morning before heading out to work- they SOUNDED quieter, but when i check the fan RPM in vSphere client, they still show as around 5-6k rpm…

    the ones i did the initial setup on were done in linux, and ipmi-sensors didn’t seem to reflect a change for the running speed, but the threshold seemed changed-
    (2025.00/NA) was the original value for the 4 “real” fans- I changed them to 10x (750) and they most definitely ARE slower, but the ipmi sensors command still shows 5-6k to the left of the /NA value… is everything ok, or is ipmi somehow flawed?

    My ESXi 5.0 install is done using the dell OEM discs, so I’m unsure if it’s padding the values?

  11. Thanks a lot for great job!!!!!

    “Dell 1900 Gen I” seems to be MEC (6×1275 RPM). I reduced RPM to 600 and flashed using USB flash disk with DOS and HDD BMC image files from Dell, just replaced .dat file.

    When checking via Dell Remote Access new thresholds are identified correctly. Can not not confirm with Esxi 5 warnings yet though. All functions work ok.

    As of project: I replaced 6 main fans with nexus 1209 PWM and PSU fan with 2 Noiseblocker XR2-s 60x60mm. I was really afraid to replace by only one xr2 since PSU generates lots of heat and original fun is like 4 x NB XR2 power. The second one installed on outer end grill near power socket (which i had to modify). Now it is really silent, even with 3 15000rpm hdds.

    P.S.
    Esxi fun works flwaless with 1900 Gen I although not in HCL from Vmware.

    • Ace I have one of the 1900 Gen 2…downloaded PE1900BmcA10 which has 4 files in it including bmcflsh.dat dated 19/9/2008. Were do I get the new modified .bin file from or do I have to make it somehow?

      Thanks

      Glen

    • 1)Sorry should of read were do I get the modified bmcflsh.dat file for the 1900 server.
      2)Also when I flash the moidified file from a usb stick with dos on it do I just run the bmcfl16d.exe or another command?

      Thanks

      Glen

  12. Just as a follow up to my comments earlier regarding the internal component temperatures; I’ve been having some stability issues with my 2900 with the Nexus PWM fans fitted. Upon checking my memory temperature today it was SCALDING hot. This server is currently situated in my garage with an ambient air temperature of around 10 degrees celsius.

    If any of you guys are running production servers or have invested in expensive RAM please double check the temperature of your DIMMS!

    I’ve refitted one of the original Delta fans in the memory cooling position for the time being.

    I believe the PowerEdge 2800 ram may be different (ie non heatsinked, at least my 2800 in the office is) and so may be able to run with less forced cooling.

    • Ouch! I haven’t tested mine yet (is there a way to read the temperature by software?) I only get the following readings by IPMI, which seem to be fine to me. I also tested the harddisks, which are also ok (see above for readings).

      8: Temp (Temperature): 69.00 C (NA/125.00): [OK]
      10: Ambient Temp (Temperature): 28.00 C (3.00/47.00): [OK]
      11: Planar Temp (Temperature): 47.00 C (3.00/72.00): [OK]
      12: Riser Temp (Temperature): 49.00 C (3.00/62.00): [OK]

      Just for the record: ipmi-sensors give hardware sensor readings, as temperatures. For a perl script that allows to read temperatures from raid controllers (i.e., 3ware) see http://josephlo.wordpress.com

      • I don’t think so no.

        I can get ambient and CPU temps and these are fine, as are the SATA hard drives I ‘ve now got in there but the memory was way too hot for my liking.

        I’m running ESXi on mine so I can’t query the sensors directly and have to rely on the health monitoring in VMWare and OpenManage for information.

  13. Hello all, I have a Dell Poweredge 2900 server purchased in Europe, I purchached 6 Nexus PWM fans and I would like to know exactly step by step, what is the procedure to change the alarm threshold 1250 RPM of 6 fans. I’ve read everything above but I’m confused because there is too much data.

    I have Xenserver, linux. I hope you can help me. a greeting y Thanks

    Daniel

      • Thanks arnuschky, then the code of mi Server is MNTRL and the correct option is 8 Fans?

        I have a doubt, this Nexus fan, has a range of 700 rpm to 2500 rpm, so I think it’s best to leave the threshold at 675 (9x 75) instead of 750.

        Another thing, a screenshot of the register of the fans:

        Severity
        OK 02/22/2012 21:16:08 The fans are redundant.
        OK 02/22/2012 21:16:04 Fan 1 RPM is operating within the allowable range.
        Crítical 02/22/2012 21:16:03 Fan redundancy is lost.
        Crítical 02/22/2012 21:15:54 Fan 1 RPM is operating less than the lower critical threshold.

        Can you solve the problem “Allowable range”? Thanks

        Daniel

        • Yes, that should be the correct code. Just execute the script on the proper firmware and it should present you with the correct option in the menu.

          “Crítical 02/22/2012 21:15:54 Fan 1 RPM is operating less than the lower critical threshold.”

          That means that you should patch the firmware using my script. The error will be solved by this.

          • Hello, i have the same problem that Dave:

            File “./dell-adjust-fan-thresholds.py”, line 141
            class FileHeader():
            SyntaxError: invalid syntax

            But i have installed the last version of python 3.2, How he solved the problem?

            My OS is Xenserver, is similar to CentOS, identical. Thanks

            Daniel

          • > Xenserver, is similar to CentOS, identical

            It is based on CentOS? Are you sure? The most recent CentOS uses Python 2.6.6. Are you sure it’s python 3.2? The error you are seeing looks like a python version that is too old. Exectue ‘python -V’ and see what it reports to you. You are doing this on your dom0 and not a domU – meaning, you are executing your commands on the main system and not on one of the virtualized guests, right?

            Anyways, if you have python 3.2 installed, there should exist a parallel install for a python 2.x for backwards compatibility. Try executing ‘python2.6 -V’ or 2.7…

  14. Yes arnuschky, this is the problem,i installed python 3.2 but Xenserver (CentoOS) only detect the old version 2.4, if you have CentOS and you had to update python to 3.2 with yum command. How would you do?

    thanks

    Daniel

    • If you successfully installed it, it should live somewhere. Try ‘python3.2 -V’ and see if it returns the correct version. If yes, execute ‘python3.2 ./dell-adjust-fan-thresholds.py’ and you should be fine.

  15. Hi arnuschky,

    Thankyou for sorting the firmware out, I am running my Dell 2900 on windows 2008 server.

    Would it be possible to send me a mod file in which I can execute the fireware mod?

    I do not have linux and am really stuck!/

    Thanks

    Andrew

  16. Hi, I havnt been able to do this on my 1900 gen 2 yet. I will be able to chage the fans no problem but as for the software patch need to work me way throu that one as dont have linux and not sure how to do it with the linux boot disk etc… There is one problem which may be a slight issue. Has anyone checked the temp of the perc controller as im told these get vary hot. Many gamers who use these cards have added extra heat sinks on the chips on the perc cards. So it may be worth a check.
    Glen

  17. Pingback: How to get Ubuntu Live running | tüfteln

  18. Ok I downloaded and have hit try ubantu am at the terminal screen and typed in
    sudo line however I get this message
    reading package lists done
    Building dependency tree
    reading state information….Done
    E: Unable to locate package freeimpi
    E: Unable to locate package freeimpi-tools
    E: Unable to locate package impitools

    Glen

  19. Hi Guys,
    I need some assistance I amusing Dell 2900 currently as an ESX, change recently to Nexus fan server PWM keeps pushing the fan speed high and low every few min.
    I got Ubuntu Live cd boot up the system load the tools use the python file thanks to you guys and patch the firmware but now when i go about flashing the image i am getting this error message failed to load library i used the exact command as above, when i access the hapi folder there is no Opt directory so where should the library files do i need to load from.
    many thanks

    Baseboard Management Controller Flash Utility Version: 1.33
    Copyright 2003-2007 Dell Inc. All Rights Reserved.
    Please wait… -./disneyinstall.sh: 27: typeset: not found
    ./disneyinstall.sh: 35: typeset: not found
    ./disneyinstall.sh: 36: typeset: not found
    ./disneyinstall.sh: 37: typeset: not found
    ./disneyinstall.sh: 39: typeset: not found
    ./disneyinstall.sh: 40: typeset: not found
    ./disneyinstall.sh: 41: typeset: not found
    ./disneyinstall.sh: 42: typeset: not found
    ./disneyinstall.sh: 43: typeset: not found
    ./disneyinstall.sh: 44: typeset: not found
    ./disneyinstall.sh: 45: typeset: not found
    ./disneyinstall.sh: 46: typeset: not found
    ./disneyinstall.sh: 47: typeset: not found
    ./disneyinstall.sh: 50: typeset: not found
    ./disneyinstall.sh: 54: typeset: not found
    ./disneyinstall.sh: 55: typeset: not found
    ./disneyinstall.sh: 56: typeset: not found
    ./disneyinstall.sh: 57: typeset: not found
    ./disneyinstall.sh: 58: typeset: not found
    ./disneyinstall.sh: 59: typeset: not found
    ./disneyinstall.sh: 60: typeset: not found
    ./disneyinstall.sh: 62: Syntax error: “(” unexpected
    Failed to Load Library

    • This might be a mistake Dell did in their scripts. Try the following:

      sed -i "s/bin\/sh/bin\/bash/g" hapi/disneyinstall.sh

      This script fails while extracting the hapi/opt directory. You could also extract it yourself by executing

      cd hapi
      tar zxvf hapi.tgz
      cd ..

      Hope this helps…

  20. Thanks arnuschky, I manage to flash it via dos mode boot it up with a pen drive and flash the bmc, although it took almost 3 hours to flash it did change the speed of the fan but i got failure at the end.
    currently i have all six fans running at 750RPM with nexus, I am thinking of lowering them to 500 not sure if it wold be wise to do that, as I got 15K SAS drives.

    • Can you explain in more detail what you mean dos mode? You mean by “via dos mode boot it up with a pen drive”. Do you mean a DOS operating system isntead of linux? How did you resolve the typeset errors you were having.
      I tried both of these commands suggested(sed on disneyinstall.sh and then extracting hapi) and still getting “Failed to Load Library”:
      LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

      Baseboard Management Controller Flash Utility Version: 1.33
      Copyright 2003-2007 Dell Inc. All Rights Reserved.
      Failed to Load Library

      I can naviaget to ./hapi/opt/dell/dup/lib.

      • I ended up doing like another commentor suggested further down and booted into Fedora, and I could then successfully flash it(I just kept and copied over the bmc_firmware folder I had already patched while working under ubuntu, it was just the finaly LIBRARY… command that was failing under Ubuntu). I also had to chmod 755 *.sh in the root and then also in the hapi folder to resolve permission denied errors. I also “chmod 755 bmcfl32l”. Also put “sudo su” or “sudo LIBRARY…” to the beginning of the flash command. I had already done a couple other workarounds arnuschky suggested, but in the end I don’t know if they were necessary or if Fedora was the solution: http://projects.nuschkys.net/2011/11/15/how-to-adjust-the-fan-thresholds-of-a-dell-poweredge/#comment-325

        Since my poweredge 2950 doesn’t have DVD boot, and fedora is too big for CD, I had to use the Fedora Live USB Creator. I went through several tries to get a USB that would boot on my poweredge 2950:
        Make sure the fedora you choose is a i386/i686/32bit, not one of the 64bit.
        I don’t remember if I ended up with Fedora 18 or 19, but it was one of the desktop options from the “Download” drop down in the USB creator. Some of them failed sha check. Some burned successfully but the poweredge wouldn’t boot. I just kept trying. I also tried different USB drives because I had seen some people say they had certain USB drives they could never get to boot(some have funky proprietary partitions). Also note, my poweredge 2950 BIOS lists the USB drives under the HDD Boot Sequence option, and everytime you change USB drives and reboot, it will reorder them and I had to make sure to go into the bios and move the USB above the HDD each time. For the USB boot option where you can choose Auto, Floppy, HDD, I chose Auto.

        Also when you run the python script that lists the options to choose your system, mine had 8 and scrolled off screen, so I ran it one time with ” | less” on the end of the command first to scroll through and find the number for my system.

        I set my thresholds to 900. The Cooljag Everflow 60mm x 25mm PWM Fan (F126025BU) I got from frozencpu settled and stayed at 1125 RPM, and the issue with speeds ramping up and down was resolved. I’ve only replaced one so far. I’m hoping after replacing the others I don’t find that any drop below 900 (maybe after aging) and cause the ramping issue again. I probably should have done a lower threshold. The originals run at ~5800rpm. So the new fan runs about 1/5th the originals.

        Thanks arnuschky, for posting this guide and all the followup comments you’ve made helping work through people’s issues.

  21. Hi arnuschky, I now have linux on the dell 2900, I have downloaded the BIN file for both the 2800 and 2900, where does the file need to sit before I execute the terminal work? Do I need to install the firmware Bin file first or does it just need to be located on the desktop? Thanks in advance

    • Where did you save the bin file? Open a terminal and go to this directory. For example, if you saved the bin file on your desktop type cd Desktop in the terminal. Then continue the how-to by executing the commands given in step 3.

  22. Hi again,

    would it be possible to go on my server remotely and help me out with this, im nearly there but some of the paths are not being found. a youtube video would be great to support your work.

    Feel really stupid

    Andrew

  23. I am trying to lower the thresholds on a power edge 2950 III.
    I’m using Ubuntu Live CD and couldn’t get IPMI to work so i used LOND to change the thresholds as suggested by Daniel for the PE 2950 series.
    Basically, going to follow the trial and error approach without IPMI.

    I followed the entire procedure and everything went well except for the last part (flashing) it first loads for a few seconds and comes back with just one message “Fail to load library” even after extracting the hapi.tgz as suggested manually and can see the content in opt/dell/dup/lib.

    Any other suggestions? I feel like i’m so close to quiet this baby down :).
    Thanks in advance for your help…

      • I have an SC1425 that I was wanting to quiet down. I do not have the bmcfl32l library file. I have several *.so files, but nothing else.

          • Sorry, I ran out of time previously.

            Anyways, I started from scratch using Ubuntu 12.04. I had to make a few modifications to the apt-get.

            sudo apt-get install freeipmi-common freeipmi-tools openipmi python wget firmware-tools ipmitool

            I successfully change the fan speeds and write it to the firmware config.

            Flashing, I ran into the problem helix had which was:
            ./disneyinstall.sh: 35: typeset: not found

            I followed your instructions ran the script that was supposed to extract, that still gave an error of:
            root@ubuntu:/home/ubuntu/Downloads/bmc_firmware# sed -i “s/bin\/sh/bin\/bash/g” hapi/disneyinstall.sh
            root@ubuntu:/home/ubuntu/Downloads/bmc_firmware# LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

            Baseboard Management Controller Flash Utility Version: 1.33
            Copyright 2003-2007 Dell Inc. All Rights Reserved.
            Please wait… -./disneyinstall.sh: line 592: /var/log/dell/disneyinstall.log: No such file or directory
            Failed to Load Library

            So I went ahead and extracted manually, and now I only get a error of Failed to load library.

            root@ubuntu:/home/ubuntu/Downloads/bmc_firmware# LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

            Baseboard Management Controller Flash Utility Version: 1.33
            Copyright 2003-2007 Dell Inc. All Rights Reserved.
            Failed to Load Library
            root@ubuntu:/home/ubuntu/Downloads/bmc_firmware#

            I have a Dell PE SC1425
            At a bit of a standstill, I’ll keep messing with it, hopefully I can get it to work. I’m wanting to install this in the hall closest where the AC unit is located.

          • I’ve double and triple checked everything, I’m beginning to wonder if Ubuntu 12 is incompatible with this process. I’m going to get an 11.04 or earlier and try again.

          • I have tried with 10.04 and 9.04, still a no go, same issues. Seems to stem from the issues of not finding a particular library file. Although, I do not know which one it is looking for.
            I am using 32Bit versions of the live CDs. I have double and triple checked that the LD_LIBRARY_PATH is pointing to the correct location and that there (what would seem to be) the correct files in the folder. I’m at a standstill, Any ideas?

          • Hey, sorry for the late reply, I am in the middle of moving and it’s a mess (as always :)

            Are you sure that you have the versions of Linux and flash tool matched? That is, 32-bit Linux with 32-bit flashthingy? Maybe there’s another library missing. Do a “ldd ./bmcfl32l” and see what you get. Can you post your directory contents?

            If you can’t flash, you can always use one of the support CDs (bootable, available from Dell) to boot a flashtool. Or you use Windows or DOS for flashing, it does not matter as long as you have the modified firmware already.

          • @Andy: I had the exact same issue with the not found error.

            The solution is to use the 32bit Fedora 18 desktop LiveCD. Boot into that, open terminal, get your files across there and unzipped. su – to get root, then run the command to flash bios. It worked first time for me in Fedora – was a total pain in Ubuntu.

            Personally, I like Ubuntu, but for this it seems Fedora (or RedHat really) is your friend.

  24. Well i hope this comment gets posted – I have tried posing before (several times) about my performing the project on a PE2900 and my posts never appear.

    Anyway, after replacing my fans with Scythe Kama 92mm fans things were going pretty well. My fan speeds were between like 1100-1300. But, we’ve been having some cooler days and the speeds started dropping until one morning they were falling below the 700RPM threshold I set, causing all the fans to kick on high speed for a couple minutes. This repeated continuously every 10 minutes.

    I lowered the threshold to 500 RPM and have been doing ok, except I started to get temp warning for the CPU and Memory and am worried about component failure. It looks like the server hardware runs too hot with the fans only spinning at 500-650 RPM since the overall case temperature is what is used to control the fan.

    When i first hooked the fans up, I did a bad solder job on one fan’s PWM cable. This caused all the fans to spin at top speeds. What I am wondering is – if I put in fans that have max RPMs of say 1500 – but either don’t use/don’t connect the PWM cable. This should cause all the fans to spin at their 1500 max (which should still be pretty quite). Can I adjust the thresholds down to 0 in the bmc to turn off all alerts?

    Other than the fact that I won’t truly know the fan RPMs (or if the fans are running at all perhaps?) will this work?

    thanks.

    • Hey Ben, thanks for all the useful and detailed comments! I have no idea why the first comments weren’t posted – I could neither see them in the queue nor the spam folder. Weird.

      It’s weird that you get warnings for the CPU and Memory temperature. In my system, the BMC spins up/lowers fan speeds depending on these temperatures. That is, as long as the system is capable of reading the temperature of a component, it should not overheat as overheating would cause a rise in fan speeds. People have reported high disk or memory temperatures, but these components usually can’t be queried through the BMC.

      What I am wondering is – if I put in fans that have max RPMs of say 1500 – but either don’t use/don’t connect the PWM cable. This should cause all the fans to spin at their 1500 max (which should still be pretty quite).

      Yes, that’s correct. I did the same initially, but was pretty annoyed by the noise so I went looking for another solution.

      Can I adjust the thresholds down to 0 in the bmc to turn off all alerts? Other than the fact that I won’t truly know the fan RPMs (or if the fans are running at all perhaps?) will this work?

      I don’t know, I have never tested this. You should still be able to query the fan RPM through the BMC, so in a pinch you can add a cronjob for checking the temps.

  25. Well it looks like my above comment got posted, so I’ll try again to post my original comments. For anyone reading these please be sure to read my comment above about problems with the fan speeds going TOO low and causing overheating issues and cpu/memory alerts on my system. At this point, I am just trying to keep the room the server is in closed to make it a little warmer to keep the fan speeds at a higher RPM.

    (also it looks like the problem is that there is a maximum # of characters per post…so I am going to try and break this up into a few posts)

    [POST 1]
    arnuschky,

    Thanks so much for the work you did for this. I found your site via Brent Ozar’s similar project (http://www.brentozar.com/archive/2010/01/how-to-make-a-dell-poweredge-quieter/).

    I would just like to leave a few comments for you and for others who might be looking to do this. I undertook this project on a Dell Power Edge 2900 server. I used Dell BMC Firmware A12 2.37 (dated 11/20/2008) with the filename BMC_FRMW_LX_R202148.BIN. I have uploaded a copy of my bmcflsh.dat to http://www.mediafire.com/?5vd7lvsqw5547cv incase anyone else with a 2900 wants to save some time (thresholds have been modified from 1275 to 600 in my .dat).

    Many of my issues were due to the fact that my 2900 runs ESX 4.1 which introduced some logistical challenges…

    1) Problems building free-ipmi on ESX:

    The main thing that gave me a problem was I was unable to install free-ipmi on it because I was unable to configure the package since there is no compiler available on ESX. After some quick research into free-ipmi though I saw that it is simply an ipmi client and can be installed on ANY system. You can then query a remote machine using the syntax:

    “ipmi-sensors -h 10.10.10.10 -u root -p calvin”

    (replace with IP of Dell system to query).

    So, for anyone else trying to do this on an ESX system you can install free-ipmi on another linux box and query remotely.

    • Hi Ben

      Can you please re-upload the file, the link in your post is dead ?

      I have a PE2900 ii running Windows Server 2008 if I have your modified bmcflsh.dat file I can simply update my system from windows (saves having to do the the Linux thing).

      Thanks in advance, Kal

      PS or can you email it to: octaviamanager@hotmail.co.uk

  26. [POST 2]

    2) Problems with Python version on ESX:

    Additionally, since you will have an older version of Python on ESX I suggest that one download the ActivePython 2.6.7.20 package (newer may work) as you can install this into its own location and not mess with the 2.4.3 Python shipped with ESX.

    3) Problems getting data from ipmi-sensor:

    Now this one isn’t ESX specific, and I hope you will clarify some of this in your post. You mention above in regards to running ipmi-sensor:

    “Record sensor numbers, fan names and thresholds (the value in brackets)”. In your snippet of output you also show something that looks like the following:

    32: FAN 1 RPM (Fan): NA (1575.00/NA): [NA]
    33: FAN 2 RPM (Fan): NA (1575.00/NA): [NA]
    34: FAN 3 RPM (Fan): NA (1575.00/NA): [NA]
    35: FAN 4 RPM (Fan): NA (1575.00/NA): [NA]
    36: FAN 5 RPM (Fan): NA (1575.00/NA): [NA]
    37: FAN 6 RPM (Fan): NA (1575.00/NA): [NA]
    38: FAN 1 RPM (Fan): NA (2025.00/NA): [NA]
    39: FAN 2 RPM (Fan): NA (2025.00/NA): [NA]
    40: FAN 3 RPM (Fan): 4875.00 RPM (2025.00/NA): [OK]
    41: FAN 4 RPM (Fan): 4800.00 RPM (2025.00/NA): [OK]
    42: FAN 5 RPM (Fan): 1800.00 RPM (900.00/NA): [OK]
    43: FAN 6 RPM (Fan): 1950.00 RPM (900.00/NA): [OK]
    44: FAN 7 RPM (Fan): 1875.00 RPM (900.00/NA): [OK]
    45: FAN 8 RPM (Fan): 1875.00 RPM (900.00/NA): [OK]

    Now your server undoubtedly has different values, but my ipmi-sensor output doesn’t look like yours. I started with free-ipmi 1.1.3 and back-revved a couple times all the way to 1.0.1 and couldn’t match your output. My output (on CentOS 5.6) looked like this:

    22 | FAN 1 RPM | Fan | 1275.00 | RPM | ‘OK’
    23 | FAN 2 RPM | Fan | 1275.00 | RPM | ‘OK’
    24 | FAN 3 RPM | Fan | 3150.00 | RPM | ‘OK’
    25 | FAN 4 RPM | Fan | 3225.00 | RPM | ‘OK’
    26 | FAN 5 RPM | Fan | 3225.00 | RPM | ‘OK’
    27 | FAN 6 RPM | Fan | 3075.00 | RPM | ‘OK’
    28 | FAN 7 RPM | Fan | N/A | RPM | N/A
    29 | FAN 8 RPM | Fan | N/A | RPM | N/A

    So you can see that the syntax is quite different. I have no brackets, no thresholds are listed (the values shown are the actual speeds), and there are no sensor numbers shown (the number to the left do not equate to the sensor value).

    Since I just saved this output out for the later steps, I didn’t realize this until it came time to run your script. Luckily, this wasn’t a major problem because your script listed the following:

    1) PowerEdge 2900
    Number of fans: 8
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM, FAN 7 RPM, FAN 8 RPM
    Fan speeds : 1275, 1275, 1275, 1275, 1275, 1275, 2400, 2400
    Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55

    Not only did it mention the model of the server (2900) but it also listed the fan speed thresholds at 1275 which I knew were correct based on information from Dell OpenManage. There was also an entry for a 1900 and some other “unknown PowerEdge” servers listed – but I felt pretty good about using the one labeled 2900.

    It is also strange that the ipmi-sensor output lists 8 fans, when only 6 exist in the system. I modified the values for FAN7 & FAN8 as well, but I don’t think it was necessary.

    • So you can see that the syntax is quite different. I have no brackets, no thresholds are listed (the values shown are the actual speeds), and there are no sensor numbers shown (the number to the left do not equate to the sensor value). Since I just saved this output out for the later steps, I didn’t realize this until it came time to run your script.

      Yes, the ipmi-sensor output is just meant for helping users to identify their system. Once the system is labeled in the script, it’s basically just a safety measure that can be omitted…

      It is also strange that the ipmi-sensor output lists 8 fans, when only 6 exist in the system. I modified the values for FAN7 & FAN8 as well, but I don’t think it was necessary.

      That is usually the case if your server isn’t fully equipped with all possible extensions. In my case, FAN7 and FAN8 are the fans for the second processor, which my system doesn’t have.

  27. [POST 3]
    4) Verifying changes:

    So after modifying the .dat file and re-flashing successfully, I tried to use ipmi-sensor to verify my changes. But, since it didn’t display any threshold values (as I mention in #3 above), it was difficult for me to determine if the changes were successful. I finally discovered the -v (verbose) switch in ipmi-sensor that allowed me to output my data in the following manner:

    Record ID: 22
    ID String: FAN 1 RPM
    Sensor Type: Fan (4h)
    Sensor Number: 48
    IPMB Slave Address: 10h
    Sensor Owner ID: 20h
    Sensor Owner LUN: 0h
    Channel Number: 0h
    Entity ID: system board (7)
    Entity Instance: 1
    Entity Instance Type: Physical Entity
    Event/Reading Type Code: 1h
    Lower Critical Threshold: 600.000000 RPM
    Upper Critical Threshold: N/A
    Lower Non-Critical Threshold: N/A
    Upper Non-Critical Threshold: N/A
    Lower Non-Recoverable Threshold: N/A
    Upper Non-Recoverable Threshold: N/A
    Sensor Min. Reading: 0.000000 RPM
    Sensor Max. Reading: 19125.000000 RPM
    Normal Min.: 10425.000000 RPM
    Normal Max.: 14775.000000 RPM
    Nominal Reading: 2400.000000 RPM
    Sensor Reading: 1200.000000 RPM
    Sensor Event: ‘OK’

    This showed not only the correct sensor number (48), but also listed the new Lower Critical Threshold (600 – as modified from 1275). At this point I was pretty confident that everything had worked.

    5) Dell OpenManage

    As a final check I looked in DellOpenManage which is installed on my Server (management software web interface http://IP:1311). It was reporting that my replaced fans were spinning at lower temperatures however it still showed the Minimum Failure Threshold at the original value of 1275 RPM. This was obviously a discrepancy with the date returned from ipmi-sensor. In order to solve this I simply needed to restart the OpenManage services (srvadmin-services.sh restart) and when I logged back into the web console and checked my fans, they all reported the new threshold value!

  28. You may need to enable the Universe package set when using an Ubuntu live CD. Also, you need to install the firmware-tools package.

    Additionally, the code: NAG is a Poweredge SC1425

  29. Hello,
    Thank you very much for this great documentation. I haven’t tried it myself yet but I intend to do so (I have a Dell PE T410 with a single/noisy 120x120x38mm Delta monster).. I wish I could throttle it down a bit as I’m using low power components inside the server.
    When adding a new PE model to your script, is there anything special that needs to be done?
    For reference, here’s the file I got for the BMC:
    -rwxr-xr-x 1 root root 11107049 Jun 11 11:32 PET410_BMC_FRMW_LX_R295869.bin
    -rw-r–r– 1 root root 65 Jun 11 11:22 PET410_BMC_FRMW_LX_R295869.bin.md5
    # bash ./PET410_BMC_FRMW_LX_R295869.bin –list
    ….
    /tmp/PET410_BMC_FRMW_LX_R295869.bin-17250-27594/payload:
    total 7244
    7244 -rwxrwxrwt 1 root root 7402040 Jan 31 2011 firmimg.d6

    so it appears that the firmware image has a different name..
    Vincent

    • Sorry, the script does not work for your server. Dell changed the firmware between generation 9 and 10 (yours is gen 11), and I do not know the format of this new firmware type. Anyways, you might need to try first to replace your fan. Maybe an adjustment of the fan thresholds isn’t even necessary with your firmware type of can even be done with the stock firmware. Good luck!

  30. Hi,
    I would just like to report that even for 11th gen servers, it appears that the BMC would do a big difference:
    I was running BMC 1.33 and I had this:
    # ipmitool sdr|grep -i fan
    FAN 1 RPM | 1680 RPM | ok
    Fan Redundancy | Not Readable | ns

    After upgrading to BMC 1.70, I now have this:
    # ipmitool sdr |grep -i fan
    FAN 1 RPM | 1440 RPM | ok
    Fan Redundancy | Not Readable | ns

    (Note the drop by 240 rpms).. It’s still better but then again I wouldn’t mind something even a little bit slower/quieter (I have low power components in this system).

    Regards,

    Vincent

  31. Fantastic, Thanks so much for this… you legend! I have a 2900 and bought myself 8X 92mm Artic cooling PWM fans after finding your article online. Everything worked like a charm and my server is now silent and bearable! fantastic. Thanks again

    • Unfortunately I spoke to soon… CPU’s and RAM started overheating even when the system was idle. The fans sit around 600 RPMS, which is obviously not enough for a PE2900.

      I wonder if the ambient temperature failure threshhold could be lowered to make the fans run a little faster say around 1000RPM? Otherwise it’s back to the drawing board… bummer.

      Excerpt from logs

      OK System Boot CPU 1 is operating correctly.
      OK System Boot CPU 1 is operating correctly.
      Critical System Boot CPU 1 has an internal error (IERR).
      OK System Boot CPU 2 is operating correctly.
      OK System Boot CPU 1 is operating correctly.
      Critical System Boot CPU 2 has an internal error (IERR).
      Critical System Boot CPU 1 has an internal error (IERR).
      Critical System Boot Memory device at location DIMM1 is over heating.
      Critical System Boot Multi-bit memory errors detected on a memory device at location
      OK 06/25/2012 13:00:00 The fans are redundant.
      OK 06/25/2012 12:59:56 Fan 1 RPM is operating within the allowable range.

      • Sorry to hear that it didn’t work out for you. DaveG. (above) reported similar problems with his PE2900 and SAS drives, I think that his solution was to mix original stock fans with silent ones.

        I fear that we don’t have the possibility to modify the RPM of the fans directly, we can only change the failure thresholds. If the failure threshold is hit, the fan goes back to 100% RPM, and lowered again, resulting in this annoying oscillating noise pattern

        What you could try is to disconnect the PWM cable (the blue one), at which the fan would be turning at 100%. If you do this only on some of the Artics, the overall system should be still quite silent.

        • Follow up:

          I replaced the fan in memory shroud with the stock fan and this seems to be sufficient. It’s a little noisier but still better than 6 stock fans. I am soldering to the points on the fan so *may* try the Artic again without PWM but the solder is starting to look a bit rough…

          I ran “stress” on the system for 30 mins and when the CPU’s started heating up the fans increased in RPM and successfully counteracted the temp increases. It looks as though the memory temperature is not monitored.

          So if you want to do this to your PE2900 I would suggest a PWM fan with a RPM range starting at around 1000RPM and either the stock fan on the memory shroud or a higher RPM (bu quieter than the stock) fan on that. Becasue the shround is in the centre of the server the noise level is ok.

          I might look at whether I can wrestle a 120CM fan on the memory shroud, but not sure if I can be bothered.

        • Ok so I did bother to replace the memory shround fan with the Artic fan and disconnected the PWM cable. Results are good! God those stock fans are loud, I was kidding myself that even the single one I placed in the server was “OK”.

          Now most of the noise is coming from what I can only assume are the power supply fans.

          Anyway, the temperature results of a 30 min stress were:

          i5k_amb-isa-0000
          Adapter: ISA adapter
          Ch. 0 DIMM 0: +90.0°C (low = +105.0°C, high = +120.0°C)
          Ch. 0 DIMM 1: +74.0°C (low = +105.0°C, high = +120.0°C)
          Ch. 1 DIMM 0: +75.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 1 DIMM 1: +67.0°C (low = +105.0°C, high = +120.0°C)
          Ch. 2 DIMM 0: +80.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 2 DIMM 1: +80.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 3 DIMM 0: +86.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 3 DIMM 1: +78.5°C (low = +105.0°C, high = +120.0°C)

          coretemp-isa-0000
          Adapter: ISA adapter
          Core 0: +68.0°C (high = +82.0°C, crit = +100.0°C)
          Core 1: +68.0°C (high = +82.0°C, crit = +100.0°C)
          Core 2: +65.0°C (high = +82.0°C, crit = +100.0°C)
          Core 3: +66.0°C (high = +82.0°C, crit = +100.0°C)

          coretemp-isa-0001
          Adapter: ISA adapter
          Core 0: +81.0°C (high = +82.0°C, crit = +100.0°C)
          Core 1: +80.0°C (high = +82.0°C, crit = +100.0°C)
          Core 2: +78.0°C (high = +82.0°C, crit = +100.0°C)
          Core 3: +75.0°C (high = +82.0°C, crit = +100.0°C)

          Idle temperatures were:

          i5k_amb-isa-0000
          Adapter: ISA adapter
          Ch. 0 DIMM 0: +89.0°C (low = +105.0°C, high = +120.0°C)
          Ch. 0 DIMM 1: +71.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 1 DIMM 0: +75.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 1 DIMM 1: +70.0°C (low = +105.0°C, high = +120.0°C)
          Ch. 2 DIMM 0: +78.0°C (low = +105.0°C, high = +120.0°C)
          Ch. 2 DIMM 1: +71.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 3 DIMM 0: +74.5°C (low = +105.0°C, high = +120.0°C)
          Ch. 3 DIMM 1: +66.5°C (low = +105.0°C, high = +120.0°C)

          coretemp-isa-0000
          Adapter: ISA adapter
          Core 0: +50.0°C (high = +82.0°C, crit = +100.0°C)
          Core 1: +50.0°C (high = +82.0°C, crit = +100.0°C)
          Core 2: +45.0°C (high = +82.0°C, crit = +100.0°C)
          Core 3: +50.0°C (high = +82.0°C, crit = +100.0°C)

          coretemp-isa-0001
          Adapter: ISA adapter
          Core 0: +55.0°C (high = +82.0°C, crit = +100.0°C)
          Core 1: +55.0°C (high = +82.0°C, crit = +100.0°C)
          Core 2: +51.0°C (high = +82.0°C, crit = +100.0°C)
          Core 3: +49.0°C (high = +82.0°C, crit = +100.0°C)

          Shows that DIMM0 gets a raw deal due ot the shround design and that CPU1 also gets a raw deal from not having a fan (from the memory shround) directly behind it.

  32. Also just a little tip. I haven’t tried it myself (was an after thought) but instead of editing the interpreter to be bash in the scripts you could try:

    sudo dpkg-reconfigure dash

    And choose “No” to use bash as the system shell before running the BMC firmware program.

  33. Does this work on the PowerEdge 2850 model? I’m completely knew at this and as far as I know it is a stock model with 2x 36GB SCSI drives, 4GB of RAM, 2x Intel Xeon 3.2Ghz (Dual-Core) and 1 power supply. It currently runs Ubuntu 12.04 LTS and I have everything downloaded and working but I want to know before I flash this, will it work. Thanks

    • It should work, yes. Just note that this firmware hack does only allow you to use silent fans – it does not make the stock models any quieter!

      • ok thanks for the last post was just about to try to make the stock fans quieter ! so i DO need to change the fans… anyone done a PE2950 ? and can advise on suitable fans and i guess BMC firmware version?

        Will

  34. I used the ubuntu boot cd as you said but apt-get can’t seem to find freeipmi package… what gives… you didn’t mention loading a repository…

  35. I know this is more of a Dell’s issue, but.. I can’t seem to find a BMC firmware for my PE2900… The closest I can find with my service tag is located in “Firmware” and named FRMW_LX_R165842.BIN , but the page info says it’s for a PowerVault MD1000??

    Can the firmware of the PowerVault somehow fit the PE2900?

    • I doubt that. On the other hand, the service tags are usually reliable. Are you sure you’ve got the right one? Another option is not to use the service tags but try to select a PE2900 by hand, or extract the firmware from one of Dell’s service cds (Dell publishes cdroms with firmwares and drivers for many systems. You can download them and browse the html catalog to find the proper firmware.)

  36. Dell PE2900 replaced with Artic F9 fans. Firmware 2.37 patch and installed successfully.
    Used option 1 (8 fans). Had to drop Fan 2 threshold to 525rpm since there was no CPU installed and it kept dropping below threshold of 600 and turning on all fans to full.
    Looks like Dell just updated BMC from 2.37 to 2.50 today.

  37. Dell PowerEdge 2950 2U 3rd Generation
    CPU: 1x3ghz Dual-Core @ 1333mhz bus
    RAM: 4gb 2x2gb @ 533mhz (low power mode)
    PSU: 1x750watt
    HD: 6x2tb SATA WD Caviar Green in RAID5

    Modified the ESM > BMC Firmware 2.50 released on 2012-10-15
    Fan 1-4 RPM multiplier was 27 x 75, 2025 RPM. Changed this to 14.
    Fan 5-6 RPM multiplier was 30 x 75, 2250 RPM. Changed this to 15.

    Replaced the 4 front fans with Everflow F126025BU. The PSU fan is still on stock and majority of noise comes from that one now. I will wait on temperatures before cracking PSU open and swapping.

    Thanks again for all this work it makes the server bearable in a home environment. I hope people find this information useful. I will report back with temperatures when I have them. I’m going for a low power solution file server here, very low projected CPU activity.

    • Jake,
      I have the same model Dell PowerEdge 2950 2U 3rd Generation as yours.
      And I am going to change these 4 noisy fans with 4 Everflow F126025BU, can you please send me your BMC Firmware 2.50 file?
      Thanks,

    • Jake,
      Have you had a chance to check your temperatures under load?
      I also have the same model 2950 III. Could you please share your modded BMC firmware 2.50 file. I am not a Linux expert and I don’t want to mess anything up!

      Thanks, Greg greg at wildoaksranch dot c**

  38. I just set this up on a poweredge 2850. I used the everflow fans, but i ran into two small issues. The first was that the fan tach and pwm were flipped. The second was that changing the fan values for the 2850 in the firmware did not work. I had to change the fan values for the 2800 section of the firmware. Anyway, thanks a bunch for this, it has turned a 120 dollar machine off of ebay into a reasonably quiet media sharing powerhouse.

  39. works like a charm, thanks so much for this. 2900 is running smoooooth and quiet, like a dream (no really i can sleep now with this thing on, finally)

    I was having trouble with the dell site not giving me the BMC download, after searching I used BMC_FRMW_LX_R202148.BIN – for anyone trying to get this done.

  40. Hi, I can confirm this works on a PE 860 for changing the thresholds.
    Default values before patching were 750/750/1450 on sensors 9,10,11
    My issue was that after boot, fans would stay at 100% but only intermittently. sometimes after a cold reboot they would slow down but never after a hot reboot.
    After the flash they stayed at 100% so I rebooted and all seems fine however seeing as my issue was intermittent, I can’t guarantee these thresholds work yet.

    Sensor data (After patched fw) as follows:

    1: Temp (Temperature): NA (5.00/125.00): [NA]
    2: Planar Temp (Temperature): 35.00 C (3.00/53.00): [OK]
    3: CMOS Battery (Voltage): 3.04 V (2.64/NA): [OK]
    4: VCORE (Voltage): [State Deasserted]
    5: PROC VTT (Voltage): [State Deasserted]
    6: 1.5V PG (Voltage): [State Deasserted]
    7: 1.8V PG (Voltage): [State Deasserted]
    8: Presence (Entity Presence): [Entity Present]
    9: PROC Fan (Fan): 3825.00 RPM (375.00/NA): [OK]
    10: DIMM Fan (Fan): 3675.00 RPM (375.00/NA): [OK]
    11: PCI Fan (Fan): NA (675.00/NA): [NA]
    12: Status (Processor): [NA]
    13: VRM (Power Supply): [NA]
    14: OS Watchdog (Watchdog 2): [OK]
    15: SEL (Event Logging Disabled): [Unknown]
    16: Intrusion (Physical Security): [General Chassis Intrusion]
    17: Temp Interface (Temperature): [NA]
    23: ECC Corr Err (Memory): [Unknown]
    24: ECC Uncorr Err (Memory): [Unknown]
    25: I/O Channel Chk (Critical Interrupt): [Unknown]
    26: PCI Parity Err (Critical Interrupt): [Unknown]
    27: PCI System Err (Critical Interrupt): [Unknown]
    28: SBE Log Disabled (Event Logging Disabled): [Unknown]
    29: Logging Disabled (Event Logging Disabled): [Unknown]
    30: Unknown (System Event): [Unknown]
    31: PROC Protocol (Processor): [Unknown]
    32: PROC Bus PERR (Processor): [Unknown]
    33: PROC Init Err (Processor): [Unknown]
    34: PROC Machine Chk (Processor): [Unknown]
    35: Memory Spared (Memory): [Unknown]
    36: Memory Mirrored (Memory): [Unknown]
    37: Memory RAID (Memory): [Unknown]
    38: Memory Added (Memory): [Unknown]
    39: Memory Removed (Memory): [Unknown]
    40: PCIE Fatal Err (Critical Interrupt): [Unknown]
    41: Chipset Err (Critical Interrupt): [Unknown]
    42: Err Reg Pointer (OEM Reserved): [Unknown]

  41. Hey there, ubuntu now still running and all works. BUT its not possible to flash the firmware. The feedback i get ist that no root permission is avaible but the hole bmc_firmware diretory has chmod 777
    If i wan to flash, i get this message

    Baseboard Management Controller Flash Utility Version: 1.33
    Copyright 2003-2007 Dell Inc. All Rights Reserved.
    Please wait… -./disneyinstall.sh : Script execution requires ‘root’ user privileges.
    Please wait… -./disneyinstall.sh : Script execution requires ‘root’ user privileges.
    Please wait… -./disneyinstall.sh : Script execution requires ‘root’ user privileges.
    Failed to Load Library

    If i want do the same but with sudo before, i get

    Baseboard Management Controller Flash Utility Version: 1.33
    Copyright 2003-2007 Dell Inc. All Rights Reserved.
    Failed to Load Library

    So how can i change the permission? Thanks for Help!!

  42. Hey, that brings no effect.
    I`ve changed the fan speed in ubuntu and send me the bmcflsh.dat
    In the next step i unpacked the firmware updater in win2k3 and overwrite the “normal” bmcflsh.dat from dell with the bmcflsh, which i had update under linux.
    After these long way it still works ;)
    Many thanks for help!

    It works with ubuntu 12.10 still but you have to do sudo -s first!

  43. If you are having trouble with the
    typeset: not found
    error, then read this blog entry to resolve. Basicall, because the Dell update script is written for Red Hat and not Ubuntu, you need to make a few system tweaks to get it to work.
    http://wherethebitsroam.com/blogs/jeffw/running-dell-update-package-red-hat-linux-debian

    Also, do note that with the latest PE2900 firmware, the ‘opt’ directory was ZIPped up and needed extracting before the script could be successfully run.

    And finally, to those of you having trouble installing IPMI – you can do it manually by going to the source site and compiling it yourself. The instructions are fairly straightforward – BUT do take note of the comment about “the options passed into
    `./configure’ must be adjusted”. You DO need to do that.

    • For the googley-challenged amonst you, here is the site with the freeipmi source code: http://www.gnu.org/software/freeipmi/
      and instructions for compiling, which were quite straight-forward (noting the ./configure options): http://www.gnu.org/software/freeipmi/README.build

      In other news, I have successfully used the Arctic F9 PWM CO fans in my PE2900 III. Note two things: (a) I went for the ‘CO’ Continuous Operation model rather than the standard one because they have a slightly better ball bearing inside as they are meant for 24×7 systems, and they were only a few pence more expensive than the already cheap standard model, and (b) the wire colourings are the same as the Dell wires BUT you must swap over the yellow and blue wires.

  44. I found this page googling for a silent Dell server. Basically most home NAS don’t have RAID-6 or multiple hotspares with RAID-5, but for about the same price you can pick up a used 2900 on eBay with a PERC6i that does. Except as stock configured, it would be rather noisy. I wouldn’t want a regular 2900 in a bedroom.

    From looking at the fifty or so followup comments I see a lot of changes and updates since the original post a year ago. Would it be possible for an expert to distill and condense all the changes into a new instruction summary?

  45. Would anyone like to help out a Linux challenged guy with modified BMC firmware for a 2950? I gave it a go and just got to many errors along the way. I suppose I can try a full vm Linux install with all the dependencies to see fi I can make more headway.

    I have replaced the fans and now they are cycling. The system is nice and quiet just before the fan failure detection spins them up again.

    Thanks, Greg greg at wildoaksranch d0t com

  46. Hi,

    Im using a Dell PE 2900 and I have followed and completed all of the above information and got to the last step 5. Flash firmware, after doing all the steps it comes up with

    Baseboard Management Controller Flash Utility Version: 1.33
    Copyright 2003-2007 Dell Inc. All Rights Reserved.
    Failed to Load Library

    I have flashed it by a USB drive but by using the same bios version 2.7.0 to 2.7.0 but the fans are still running really loud after the flashing. I always flashed back bios version to 2.6.1 and still didn’t work. I have used this guide to flash the bios http://www.egroup-us.com/how-to-upgrade-a-dell-server-bios-using-a-bootable-flash-drive

    Can you tell me what I am doing wrong? that would be much appreciated.

    Thanks.

    • Did you change the fans with silent ones? The flashing of the firmware only allows you to use fans which produce less noise. The hack does NOT reduce the noise in itself!

      • Thanks for the reply arnuschky but yes I replaced the fans with the exact models shown in the guide from brentoza the Nexus 92mm PWM Series fan. They go really quite once left on for a while but then it hits the threshold and all of them spin up to max speed again. Just not to sure what im doing wrong. Its a PE 2900 and im selecting option 1 from the script that specifies what model server and I believe thats correct.

        • There’s a slight confusion as there are 2 generations of PE 2900. When using the script, try changing the values of all the models in the firmware. That way, you can check if one of the models works.

          After you flashed the firmware, could you see the changed thresholds using ipmitool?

          • Hi,

            Thanks for the advice but even after changing every threshold in the script 1-8 to 600 RPM and flashing the bios the fans still spin up when they go under 1275 rpm. I checked the fans using impitool and when it hits 1275 the fans say Below (<=) Lower Critical Threshold so the thresholds arnt changing. Any other ideas y this is the case?

            Thank you

  47. Hi there,

    First up, great tutorial. I have a couple of Dell 2950′s (gen III) that are desperately in need of quietening down.
    I’ve completed a fan swap on one of them, the new fans are much quieter. Because of where I live in the world, without spending heaps of money, I really only had the DeepCool 60x60x25′s available to me. 3pin, 3,000rpm. I just dont wire the PWM, and they work just fine – even max rpm is miles quieter than the ~65dBA that each of the stock fans puts out!

    But, I’m a bit nervous to try the firmware. I have the latest 2.5 BMC for my system, but its name is a bit different from what people have reported above: “2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN”.

    I can extract it, and run your script, and it finds 8 different sets of systems inside. Trouble is, none of them say LOND at all – as reported by other people with 2950′s, and none of the sensor numbers match what ipmi gives me. I know that the standard thresholds on my 2950′s are up around 3,500rpm (!!!). None of the default thresholds that your script finds in the payload are even close to that.
    A quick hexedit into the payload file confirms in the headers that there are multiple LOND FLC sections… Just this doesn’t seem to reflect in your little python piece of magic when it runs through it.

    Is there something I’m missing, or has the BMC firmwares from Dell become different enough since you wrote your program that it no longer works quite right?

    I understand you’re busy, but any help or push in the right direction would be greatly appreciated.

  48. I posted previously, but have since worked on the issues and have success!

    2.5 BMC firmware for a Dell 2950 (III), is option “3″ from the script, even though the default thresholds dont match what actually seems to be used, and “LOND” isn’t actually specified anywhere (but it does say Dell 2950)

    So, all good, and all working. :-)

    Things I learned that might help others:

    * PWM pin not needed on fans, only Tach required (yellow wire on Dell plug)
    * RPM thresholds in firmware dont seem to match whats actually used when server is running.
    * While Ubuntu seems to be preferred by most people, only the Fedora 32bit LiveCD was successful at flashing the firmware for me – and the least painful by FAR.
    * remember to unzip the hapi.tgz!!!
    * execute flag didn’t seem to be set on /hapi/disneyinstall.sh or the bmcfl32l file by default – may have been my fault when I was moving files around though.

    – Stoo

  49. Howdy,

    I was wondering if anyone has a solution for a T710 server, just did the fan upgrade recently and when the system is cold, fans go to 100% then back down. Seems like the BMC patch does not work with idrac6 configs, wondering if you have any suggestions or can work towards a universal patch, i think this project is great for home labs and would love to see it work on newer dell PE servers.

    Tim

    • The structure of the BMC firmware changed completely after the generation 9 servers. I have no idea if this only concerns the packaging format, or if actually the code changed completely. I assume this is due to a major change in the BMC hardware. Anyways, it would mean reading all the work on a >= gen 10 system (your T710 is gen 11). If you’re up to it, I can try to provide you with some pointers. I can’t do it myself as I do not have such a server available.

      • Please any pointers would help me greatly. Ive tried to change the critical warning threshold via ipmitools but im not doing it right it seems or those values are read only. Im batting around the idea of finding higher idle RPM fans. Purchased the Nexas fans who’s idle is 700rpm, critical threshold for the fans on my bmc are 1080rpm, might see if we can find a happy medium.

        Tim

        • Hey Tim, sorry for the late reply (work’s horrible right now). You are right, the critical thresholds are read-only. I outline what the firmware hack involved for < = gen 9 servers. Basically, all servers share the same firmware format. Each file consists of several sub-files that are either data or code, and can be applied to multiple systems. Part of the sub-files are shared among systems, and others are present for each different model. For example, a 1900 and 2900 might be present in the same firmware file, both sharing the same primary code bits, but having different resource tables for model-specific data such as fan values.

          If you want to do something similar with the newer models, here some pointers:

          • understand if dell has a 1:1 matching of firmware to model, or if one firmware applies to a whole family
          • get a bunch of firmwares; different versions same model and same version different models
          • find out if the file is compressed somehow; if yes, try to uncompress it
          • check if there are multiple files contained in that file; usually, that requires a table at the beginning of the file (or, rarely, the end)
          • try to decode/reverse engineer that table; write a program that can unpack that firmware file into its separate parts
          • have a look at the different part; are there similar files that occur multiple times?
          • find out what’s data and what isn’t; search for human readable strings
          • check which of the different bits change across different versions; and which change across different models
          • That’s it for now. I wish you good luck and lot’s of fun – that stuff’s basically like a very nice riddle. :)

          • Did you get anywhere with this Tim? (I’m hoping you get notified of comments.) I want to switch out the 92mm PWM fan in my t310 with iDrac express in it. Might try a question on the Dell support forum, but the rule seems to be that when the server is no longer in sale (or at least this old) help like this from them is not likely.

          • ended up putting the stock fans back in, might revisit the bmc hacking at a later date, the fans throttling all night long was driving me nuts.

          • Hey Tim,

            I, too have a T710, am looking for a solution to the fan noise issue. While I know that you put the stock fans back in, I was curious a) what replacement fans you used initially b) if you recall what the lowest RPM they were running at?

            I am noting in iDRAC 6, with 1.97b2 (current) that the failure threshold now shows as 1080RPM (warning is listed as ‘N/A’). Vincent listed in this thread that previously these thresholds were at 1680, then 1440 (and now it appears 1080). So, I’m trying to understand if any of these ‘slower’ fans still have a low end of 1080 or higher.

  50. Any ideas about changing other bmc settings? I replaced the perc 6i in my 2900 III and it is complaining at startup about the flexbay sas connector being disconnected.

    Any ideas would be helpful.

    • Sorry, I do not think that this is possible. I decoded the overall structure of the firmware, and was able to identify some values in the resource tables of it. That is, I am changing data, not functions. There’s no possibility to change the code. If you can figure out where this might be saved, or if you have access to two firmwares (one with, the other without the functionality), you might have a chance.

  51. Hello – this is probably a silly question but does anyone know where I can get this python script? Thank you in advance!

    Also if anyone has an altered firmware for a 2950 I’d really appreciate a copy to save me having to do it!!

    Thank you

  52. Pingback: Moving code to GitHub | tüfteln

  53. Hello again. Thank you for your help with this but I have another problem:

    When I run:

    sudo ./2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN –extract bmc_firmware

    I just get:

    sudo ./2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN: command not found

    I am sorry for what is probably a silly question but do you think the firmware is invalid?

    Thank you!

  54. Hello again – just wanted to add: I see that you have made this script available:

    dell-extract-bmc-firmware.c

    but there is no mention of it here; is it needed?

    Thank you.

  55. Thank you once again.

    Just to let everyone know: I got it working it worked great until the memory fried. The ambient temp sensor was only reading 25c but the DIMMs were too hot to touch. Might have had something to do with the six 15k SAS disks!

    Cheers

  56. Firstly, thanks for the great work. This is very interesting.

    I have a Poweredge 1955 Blade rack chassis with 3 blades. I was looking into ways of making it quieter. As it is deafening on startup (100 dbA @ 1 foot, peak frequency 500hz), and I came across an undocumented command on another blog: ‘racadm fanctrl (0-5)’. The ‘racadm fanctrl 0′ command is quiet enough for me to use them, but I would still be annoying the neighbors, and it would be ideal to use these afterhours, without disturbing anyone.

    I am trying to figure out how to do two things, the first is remove the warning, and critical thresholds by lowering them as you have done for other models. The second is to modify the ‘fanctrl’ setpoint to be something along the lines of:

    fanctrl 0: 25%
    fanctrl 1: 50%
    fanctrl 2: 75%
    fanctrl 3: 100%

    4, and 5 are apparently enable / disable of dynamic control.

    I ran your Python program against the bmcflsh.dat (as extracted via 7-zip on Win 7 x64, Python v2.7.3), and it throws crc errors for every checksum computed. I have a bit of familiarity with Python, and I am wanting to learn alot more, so this seems like the perfect project. Also learning a bit of firmware hacking would be awesome.

    To that end I have been trying to stumble through the firmware files for the 1955s (both for the bmc, and the drac). I haven’t found anything that seems logically like an rpm setpoint.

    The fans: All fans are 38mm thick.
    Main fans are 4 x 140mm
    PSU fans are 1 x 92mm & 2 x 60mm
    There are 4 PSUs in all so that makes:
    4x140s
    4x92s
    8x60s

    The (bmc) firmware is oddly divided into either listing 8 fans, 6 fans, or 16 fans, and the most confusing thing is that the symbols nearby the entries for them can’t be the rpm setpoints, as the entries don’t correspond logically; they seem random. I was thinking they are pointers to the location of setpoints, but I am lost as to where those values are actually stored. I was also thinking that perhaps the listing of 6 fans is the setpoint entries for fanctrl 0-5, but the values nearby don’t seem logical either. I tried to read through your program to see what data it was actually reading to get the setpoints of the warning values, but I wasn’t able to get it.

    The setpoints for the fans are:

    Main 140mm 1700 RPM 13000 RPM 1450 RPM 15000 RPM
    PSU 92mm 3150 RPM 13000 RPM 2700 RPM 15000 RPM
    PSU 60mm 4550 RPM 13000 RPM 3900 RPM 15000 RPM

    Searching for these values (in hex) didn’t yield anything. Also I don’t know how the ipmi multiplier of 75 comes into play?

    Here is a dump from the drac webpage, and then the bmc / drac files (from a hex editor). If you can point me in the right direction to solving this I would be very grateful.

    Fan1-Fan-1 N/A 1700 RPM 13000 RPM 1450 RPM 15000 RPM
    Fan1-Fan-2 N/A 1700 RPM 13000 RPM 1450 RPM 15000 RPM
    Fan2-Fan-1 N/A 1700 RPM 13000 RPM 1450 RPM 15000 RPM
    Fan2-Fan-2 N/A 1700 RPM 13000 RPM 1450 RPM 15000 RPM
    PS1-Fan-1 N/A 3150 RPM 13000 RPM 2700 RPM 15000 RPM
    PS1-Fan-2 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS1-Fan-3 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS2-Fan-1 N/A 3150 RPM 13000 RPM 2700 RPM 15000 RPM
    PS2-Fan-2 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS2-Fan-3 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS3-Fan-1 N/A 3150 RPM 13000 RPM 2700 RPM 15000 RPM
    PS3-Fan-2 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS3-Fan-3 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS4-Fan-1 N/A 3150 RPM 13000 RPM 2700 RPM 15000 RPM
    PS4-Fan-2 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM
    PS4-Fan-3 N/A 4550 RPM 13000 RPM 3900 RPM 15000 RPM

    I was just resizing these in notepad to get the columns to line up:

    ………………………….ÎFAN MOD 1A RPMTÍ..Q.9 .1…Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 1B RPM/…Q.9 .2…Ô… ……..K…0..GÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 1C RPM×%..Q.9 .3…Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 1D RPMªp..Q.9 .4…Ô… ……..K…0..GÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 2A RPM²†..Q.9 .5…Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 2B RPMÉZ..Q.9 .6..}Ô… ……..K…0..GÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 2C RPM2v..Q.9 .7..}Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 2D RPMO#..Q.9 .8..}Ô… ……..K…0..GÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 3A RPMžg..Q.9 .9..}Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 3B RPMå» .Q.9 .:..}Ô… ……..K…0..GÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 3C RPMUÈ!.Q.9 .;..}Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 3D RPM(.”.Q.9 …}Ô… ……..K…0..GÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 4C RPM‘k%.Q.9 .?..}Ô… ……..K…0..hÅ‹ÿ.ÿÿÿ……..ÎFAN MOD 4D RPMì>&.Q.# .P..mÀ

    Å‹ÿ.ÿÿÿ……..ÉFAN 1 RPM.š..Q.4 .1…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 2 RPMGù..Q.4 .2…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 3 RPMTŠ..Q.4 .3…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 4 RPM.`..Q.4 .4…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 5 RPM….Q.4 .5…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 6 RPMOf..Q.# .P..mÀ%o……………..

    ……..K…0..TÅ‹ÿ.ÿÿÿ..5…..ÉFAN 1 RPM<I..Q.4 .1…Ô… ……..K…0..TÅ‹ÿ.ÿÿÿ..5…..ÉFAN 2 RPM}*..Q.4 .2…Ô… ……..K…0..TÅ‹ÿ.ÿÿÿ..5…..ÉFAN 3 RPMÂæ..Q.4 .3…Ô… ……..K…0..TÅ‹ÿ.ÿÿÿ..5…..ÉFAN 4 RPM……Q.4 .4…ÿ… ……..K…0..8Å‹ÿ.ÿÿÿ..#…..ÉFAN 5 RPM‰—..Q.4 .5…ÿ… ……..K…0..8Å‹ÿ.ÿÿÿ..#…..ÉFAN 6 RPMÈô..Q.# .P..mÀ

    ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 1 RPMÐÅ..Q.4 .1…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 2 RPM‘¦..Q.4 .2…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 3 RPM.j..Q.4 .3…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 4 RPMi€..Q.4 .4…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 5 RPMtå..Q.4 .5…Ô… ……..K…0.. Å‹ÿ.ÿÿÿ……..ÉFAN 6 RPM5†..Q.4 .6…ÿ… ……..K…0..<Å‹ÿ.ÿÿÿ. &…..ÉFAN 7 RPM´ù..Q.4 .7…ÿ… ……..K…0..<Å‹ÿ.ÿÿÿ. &…..ÉFAN 8 RPMÞ…Q.#

    E1310.RPM Fan..E1311.RPM Fan Mod.E1313.Fan Redundancy.E1410

    • Hey, first off, which file are you looking at? The firmware files of the gen 9 servers consist of multiple sub-files, and often contain the data files for multiple systems. There’s a separate C-program in the GitHub that allows you to extract the separate firmware files. Makes your life with the hex-editor a lot easier! Try to extract these files first, and then we’ll see. AS my script throws a lot of CRC errors, I have the impression that you’re facing a structure/firmware totally different from the others.

  57. I thought I’d share my own experiences of replacing the fans on a PowerEdge 2850 so that anyone else reading this can avoid some of the pitfalls that I came across along the way.

    Firstly, I like to say thanks to Arnuschky for his hard work in patching the firmware. If it wasn’t for him, I wouldn’t have stood a chance in patching it myself.

    So, I bought the following spec server from eBay for £85 or so to be used as an office fileserver. We’re a startup, and as every penny counts, this looked ideal for fast/reliable storage, but as we sadly don’t have our own server room, had to do something about the fan noise.

    PowerEdge 2850
    2 x 3.6GHz Xeon
    8Gb RAM
    6 x 72Gb disks

    I was worried that replacing the fans would increase the heat produced to unacceptable levels, but as it will be lightly loaded for most of the time I think the risk of this is small.

    I decided replace the stock fans with Nidec U60T12MUA7-57 60mm x 25mm fans giving 23 CFM at a max 4600 rpm producing 32.5 db’s. There are probably quieter/better fans available, but I couldn’t find any in the UK quickly/cheaply/easily so settled on these.

    My PowerEdge has 6 x 60mm internal case fans, four cooling the cpu’s and two outlet fans. I decided initially to just replace the outlet fans to see what the drop in noise was like, the rationale being that the noise from the cpu fans was inside the case and would be muffled rather than the outlet fans which would be more audible. If it was still very noisy, I’d replace two of the cpu fans and monitor the noise/heat and finally replace the other two cpu fans if needed. I had a number of issues along the way including:-

    1. Although the fans have Black/Red/Yellow/Blue cables the same as the stock Dell ones, the Blue & Yellow are swapped (control & fan speed ones), so the first time I wired them up, the new fan wasn’t recognised and the other five case fans ran at full speed (= really loud!). Swapping the Blue/Yellow wires fixed this ok.

    2. As mentioned in the original article, the fan threshold is hit for fans 5 & 6 at 1500rpm, meaning all fans spool up to their full speed and slowly decrease until this repeats every 10 mins or so – time to patch the firmware!

    3. I downloaded the latest BMC firmware from Dell (also BMC_FRMW_LX_R223079.) but when I ran the dell-adjust-fan-thresholds.py script the description for the 2850 didn’t really fit?

    a. 2850 with 8 fans labelled 1A/1B/2A/2B/3A/3B/4A/4B
    b. 2800 with 14 fans labelled Fans 1-6 & Fans 1-8

    The 2800 seemed more suitable as an ipmi-sensors lists two banks of fans 1-6 & 1-8, but only Fan 1-6 actually listed any values. I’ve no idea what Fans 1-8 are, but guessed/hoped that Fans 1-6 for a 2800 is what I needed. The sensor numbers were also slightly different as well, but went ahead and patched the firmware anyway.

    4. My PowerEdge’s running Ubuntu 12.10, and running bmcfl32l to patch the firmware throws an error

    bash: ./bmcfl32l: No such file or directory

    I Googled round for an answer to this, with no joy, and then read in one of the comments to try patching from a Fedora LiveCD which I dutifully downloaded and tried. I then wasted an hour or so trying to boot from the DVD only to realise that my old 2850 only has a cd rom – doh! Downloading the Fedora 18 LXDE Spin version burns to cd ok, and was then able to patch the BMC firmware successfully.

    5. The moment of truth – I’d set the fan thresholds to 15×75 = 1125 rpm, and the fan speeds slowly dropped below the original 1500rpm without everything speeding up, so it looked like it had worked ok. The fan speed then kept dropping even further, until it hit the new 1125 threshold, and everything sped up again. Time to drop the thresholds even lower, which I was slightly worried about as the speeds seem very low at this point. Reset the threshold to 750 and with all fingers crossed, watched as the fan drop dropped to 900 and kept steady at this – hurrah!

    Here’re the new steady state fan speeds from ipmi:-
    33 | FAN 1 RPM | Fan | 4875.00 | RPM | ‘OK’
    34 | FAN 2 RPM | Fan | 4950.00 | RPM | ‘OK’
    35 | FAN 3 RPM | Fan | 4875.00 | RPM | ‘OK’
    36 | FAN 4 RPM | Fan | 4800.00 | RPM | ‘OK’
    37 | FAN 5 RPM | Fan | 975.00 | RPM | ‘OK’
    38 | FAN 6 RPM | Fan | 1050.00 | RPM | ‘OK’

    The overall noise is quite acceptable with just fans 5 & 6 replaced, and as I thought, Fans 1-4 are muffled inside the case. The temperature is still fairly low:-

    1 | Temp | Temperature | 41.00 | C | ‘OK’
    2 | Temp | Temperature | 44.00 | C | ‘OK’
    3 | Ambient Temp | Temperature | 26.00 | C | ‘OK’
    4 | Planar Temp | Temperature | 44.00 | C | ‘OK’
    5 | Riser Temp | Temperature | 41.00 | C | ‘OK’

    so I may go ahead replace the others to make it quieter, but I’ll monitor things over the coming days to see how it goes. In a busy office, the noise fades into the background so may not be needed.

    Thanks once again to Arnuschky and everyone else who pasted info the comments.

    Andy.

  58. I have the windows install firmware package for windows that drops the thresholds on a 2950 to 150rpm. If anyone wants a copy let me know.

  59. I tried to compile the C program on Win 7 x64 to no avail. I found, and downloaded various libraries as it asked for them: getopt.h, strings.h, unistd.h, _ansi.h. I finally gave up on Windows, as I am thinking with these various cobbled together libraries it may not work, and I don’t know how many it needs (getopt.h is a rather large file) I will retry it on my Ubuntu rig when I get a moment, and post back.

    • Try on Ubuntu, I haven’t written that program with portability in mind. This command should install all required files:

      sudo apt-get install build-essential
      • Thanks for this. I have been swamped at work, but I was finally able to get back to this. I was able to extract files on Ubuntu, and I have analyzed them. I don’t see anything regarding racadm, or what look like a setpoint to my eyes yet…

        After running strings.exe (SysInternals) against the OB.FLC file extracted from the BMC firmware for the Poweredge 1955 blades(bmcflash.dat):
        I found the E codes for some fan errors (maybe I can use these to find addresses in other files?):

        E1310
        RPM Fan
        E1311
        RPM Fan Mod
        E1313
        Fan Redundancy

        And some more interesting items regarding fans:

        SETRPM
        GETRPM
        FAN

        Some other interesting tidbits:

        Copyright (c) 1996-2001 Express Logic Inc. * ThreadX H8/300H-Hitachi Version G4.0.4.0 *
        G-GB-GL-M-D-DL-KML-CMR-HMR-ML2-GZ-KH2-CM-RP-TC-NH-TD-AP-HA-GF-DD-AT-MF-MS-DW-USA-CA-SD-SDSU
        OSA Tech. Taiwan
        Copyright (c) 1996-2005 Express Logic Inc. * NetX Hitachi H8/300H Version G2.1d.2.1c *

        Also here is a list of apparent Servernames:

        Unknown
        Montreal
        Barcelona
        MontrealEC
        London
        Berlin
        Oslo
        FatTireo
        Guinnesso
        Bullion
        Veso
        Vanguard
        Calypso
        Miranda

        Fat tire, and Guinness being beers, prehaps K_C is Kentucky Fried Chicken!

  60. Hi,
    I picked up a PE2900 for use as a home NAS, and quickly learned how loud it is! I’m interested in lowering the noise, but rather than swap out the fans, I would like to know how Dell’s fan control algorithm works. If there’s a PID controller, then perhaps we could lower it’s setpoint. I looked at the bmcflash.dat file using a hex editor but found nothing mentioning a PID. Did you decompile the file? I noticed that the file lists Hitachi H8/300H, which I assume is the BMC controller. I’m looking for a suitable decompiler to try to understand how the BMC works. By the way, great page!

    • Great idea! If you have some technical expectise, go for it! I didn’t find the reference to the Hitachi processor; that is certainly a helpful starting point. (I didn’t decompile the code of the firmware as I didn’t know the architecture/processor, and as the thresholds are stored in a data file I didn’t need to do it in the end.)

      For starters, get the “extract firmware” program that I wrote (you can find it on GitHub). It extracts the different files that are contained in the
      bmcflash.dat file; this allows you to separate data from code. Read the comments in the program and the python script, but the code files should be the ones of which there’s only a single version (data files are included multiple times, each for a different model of PowerEdge). \

      Good luck!

      • OK, finally found sometime to try to compile the dell-extract-bmc-firmare.c file and somethings missing between the c file and the bmcflsh.dat file. I keep getting an “Header is not valid” message. I modified the original c file to try to find out what the problem was and turns out the header.zero field is non zero! I looked at the dat file using a hex editor and the header is fine : 02 35 68 7F 08 00 00 00. I then modified the code to dump header.hex02, numBlocks, filesize and zero field values using fprintf and strangely the filesize value is being reported as 0×00000008 instead of 0x00087F68. Any idea why the 68 7F fields are being skipped? I can’t seem to make the program read those!

        • A bit more digging. The c code is definitely having some sort of an issue on Ubuntu 12.04 LTS x64. The uint32_t filesize in the header structure is not picking up the right value, I modified the header structure as follows:
          …….
          uint16_t filesize;
          uint16_t dummy;
          uint16_t zero;
          char dellHeaderStr[9];
          …….
          Naturally the filesize is incorrect but the zero and dellHeaderStr values are. Here’s what I get ( I modified the code to dump the header field values and the filesize calculated by fseek ):

          Header:
          header.hex02= 2
          header.numBlocks= 35
          header.filesize= 7F68
          header.zero= 0
          header.dellStr= DELL_INC
          File Size Expected = 556904

          Valid Dell PowerEdge BMC firmware header found:
          – number of blocks : 53
          – oemstr (fixed) : DELL_INC
          – total file size : 32616
          – total header size: 2774
          – header CRC16 : 0x344b
          – total file CRC16 : 0x9cf8

          * CRC16 check FAILED, actual CRC16 is 0x9cf8 instead of 0x207d

          Error: Block 0 not valid.

          If I restore the original code with uint32_t filesize, the program does fails at the “Header is not valid” stage. Something is not right with uint32_t on ubuntu x64.
          I will try again using a 32 bit live CD to check if this problem appears.
          Did you test your code on a 64 bit system?

          • No, I think I wrote that code before I changed all my systems to 64bits. It might well be the case that the code does not work on 64bits. Maybe try playing with endianess?

          • Hi again,
            I finally managed to find a workaround the nasty problem that I ran into! I’m not entirely sure of the reason I had the problem, I was able to reproduce the problem on 2 Ubuntu x64 machines and even my macbookpro. I believe the problem appears to be the compiler padding the header struct, leading to wrong data being assigned to the struct elements. In retrospect, I should have tried compiling with no padding, but I had gone too far by then and I was too lazy to search for the no padding compiler flag!
            I would like to send you the updated file, in the hope that somebody may find it useful. I don’t want to paste the code here, how do I send you the modified file?
            Now on to some decompiling and perhaps we can get the fans running with a PID controller!

          • Great that you found it! I had actually exactly the same problem when developing my code. The Makefile in the git actually includes the appropriate compiler flag, but I guess that this is architecture dependent – or didn’t you use that Makefile?

            I happily include your code in the git – check your email!

  61. Hello,
    I am tring to flash a PE 2900 III using your script but get some issues finding the right model…
    ipmi-sensors returns 6 fans:

    22 | FAN 1 RPM | Fan | 1650.00 | RPM | ‘OK’
    23 | FAN 2 RPM | Fan | 1725.00 | RPM | ‘OK’
    24 | FAN 3 RPM | Fan | 1725.00 | RPM | ‘OK’
    25 | FAN 4 RPM | Fan | 1800.00 | RPM | ‘OK’
    26 | FAN 5 RPM | Fan | 1950.00 | RPM | ‘OK’
    27 | FAN 6 RPM | Fan | 1875.00 | RPM | ‘OK’
    28 | FAN 7 RPM | Fan | N/A | RPM | N/A
    29 | FAN 8 RPM | Fan | N/A | RPM | N/A

    When I run the script, here is what I partially get:

    Class 1 info block sensor id 48
    –> found valid fan sensor info block: FAN MOD 1A RPM: 29/2175 RPM
    Sensor info header blockId 16 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 49
    –> found valid fan sensor info block: FAN MOD 1C RPM: 29/2175 RPM
    Sensor info header blockId 17 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 50
    –> found valid fan sensor info block: FAN MOD 1B RPM: 29/2175 RPM
    Sensor info header blockId 18 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 51
    –> found valid fan sensor info block: FAN MOD 1D RPM: 29/2175 RPM
    Sensor info header blockId 19 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 52
    –> found valid fan sensor info block: FAN MOD 2A RPM: 29/2175 RPM
    Sensor info header blockId 20 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 53
    –> found valid fan sensor info block: FAN MOD 2C RPM: 29/2175 RPM
    Sensor info header blockId 21 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 54
    –> found valid fan sensor info block: FAN MOD 2B RPM: 29/2175 RPM
    Sensor info header blockId 22 class 1 length 57
    -> CRC ok
    Class 1 info block sensor id 55
    –> found valid fan sensor info block: FAN MOD 2D RPM: 29/2175 RPM

    Reading block header #46
    – verifying block header and block data integrity CRC16
    Verifying sensor block…
    – Block header: type=11 systemId=22 crc=0x7fb7 length=4258 offset=540554 filename=’SD_BULN.FLC’
    – checking for sensor block…
    > found sensor block!
    > found an unknown system.
    Sensor info block with 97 sensors, whatever is 0x9f

    Then a few lines after:

    Class 1 info block sensor id 48
    –> found valid fan sensor info block: FAN 1 RPM: 17/1275 RPM
    Sensor info header blockId 31 class 1 length 52
    -> CRC ok
    Class 1 info block sensor id 49
    –> found valid fan sensor info block: FAN 2 RPM: 17/1275 RPM
    Sensor info header blockId 32 class 1 length 52
    -> CRC ok
    Class 1 info block sensor id 50
    –> found valid fan sensor info block: FAN 3 RPM: 17/1275 RPM
    Sensor info header blockId 33 class 1 length 52
    -> CRC ok
    Class 1 info block sensor id 51
    –> found valid fan sensor info block: FAN 4 RPM: 17/1275 RPM
    Sensor info header blockId 34 class 1 length 52
    -> CRC ok
    Class 1 info block sensor id 54
    –> found valid fan sensor info block: FAN 5 RPM: 32/2400 RPM
    Sensor info header blockId 35 class 1 length 52
    -> CRC ok
    Class 1 info block sensor id 55
    –> found valid fan sensor info block: FAN 6 RPM: 32/2400 RPM

    And I am then presented the choice for 8 systems:

    Systems found in firmware file:

    1) PowerEdge 2900
    Number of fans: 8
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM, FAN 7 RPM, FAN 8 RPM
    Fan speeds : 1275, 1275, 1275, 1275, 1275, 1275, 2400, 2400
    Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55

    2) unknown PowerEdge (code: BRCL)
    Number of fans: 0
    Fan names :
    Fan speeds :
    Sensor numbers:

    3) PowerEdge 2950
    Number of fans: 6
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
    Fan speeds : 2025, 2025, 2025, 2025, 2250, 2250
    Sensor numbers: 48, 49, 50, 51, 52, 53

    4) unknown PowerEdge (code: BRLN)
    Number of fans: 16
    Fan names : FAN MOD 1A RPM, FAN MOD 1B RPM, FAN MOD 1C RPM, FAN MOD 1D RPM, FAN MOD 2A RPM, FAN MOD 2B RPM, FAN MOD 2C RPM, FAN MOD 2D RPM, FAN MOD 3A RPM, FAN MOD 3B RPM, FAN MOD 3C RPM, FAN MOD 3D RPM, FAN MOD 4A RPM, FAN MOD 4B RPM, FAN MOD 4C RPM, FAN MOD 4D RPM
    Fan speeds : 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575, 1575
    Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63

    5) PowerEdge 1900 (Gen I)
    Number of fans: 6
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
    Fan speeds : 1275, 1275, 1275, 1275, 1275, 1275
    Sensor numbers: 48, 49, 50, 51, 52, 53

    6) unknown PowerEdge (code: FT)
    Number of fans: 6
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
    Fan speeds : 1275, 1275, 1275, 1275, 2400, 2400
    Sensor numbers: 48, 49, 50, 51, 54, 55

    7) unknown PowerEdge (code: GNS)
    Number of fans: 8
    Fan names : FAN MOD 1A RPM, FAN MOD 1C RPM, FAN MOD 1B RPM, FAN MOD 1D RPM, FAN MOD 2A RPM, FAN MOD 2C RPM, FAN MOD 2B RPM, FAN MOD 2D RPM
    Fan speeds : 2175, 2175, 2175, 2175, 2175, 2175, 2175, 2175
    Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55

    8) unknown PowerEdge (code: BULN)
    Number of fans: 6
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
    Fan speeds : 1275, 1275, 1275, 1275, 2400, 2400
    Sensor numbers: 48, 49, 50, 51, 54, 55

    Select (1-8):

    I tried several and alwazs had to flash rebooting on a DOS usb key. I decided to change them all this stime, and to flash / Iwill post result.

    Oh the script also ends saying:

    Please also report this string ‘unknown PowerEdge (code: BULN)’
    along with your type of PowerEdge! Thank you.

    I will give feedback once flashed…

    • I flashed with the 8 proposed system modified at 225 RPM (I have 2 noctua NF-B9 PWM that can run as slow as 300 RPM – the rest are Arctic Cooling F9 PWM that can run at 600 RPM) and no change.
      The fans start full speed, and never reduce speed.

      I tried to remove one fan at a time and boot, since I read in the previous comments that a bad solded fan bring all the others to full speed for someone, without any change.

      Any suggestion is welcome, as this machine is now more Noisy than a combat helicopter :(

      Thanks in advance,
      Félix

  62. If somebody uses ubuntu 12.04 do the trick with reconfigure dash
    afterwards extract hapi.tgz
    enter to hapi/dell/dup

    make cp -r * /
    an error wil ocure but skip it
    go back to hapi folder ant then copy dell to /opt
    cd hapi
    cp -r dell /opt/

    this will make the trick in ubuntu live :) ( 3 hours of missing things ) but i have the AAB FAN 9 PWM 800-2000 RPM and i think they all goes under 800RPM’s and it spins up – I have to lower the threshold for one point more – DRAC says that critical is 750 ( 10 in python script)

    while flushing i saw that my BMC is 1.77v and I was searching does PE2900 is compatible with that revision but it is :) I can upload ready dat file with tresholds 750 and 675RPM’s

    • And HOW to use ubuntu 12.04?( 12.10 probably too )
      First install go to /etc/apt
      nano sources.list
      comment out with # CD media
      ctrl+x to end
      y to commit
      enter

      next

      apt-get update
      apt-get install aptitude ( resolves conficts better than apt-get )

      go back to sources.list
      edit in 3 deb starting packages last two words to “universe multiverse”

      should look like this

      # /etc/apt/sources.list

      #deb cdrom:[Ubuntu 12.04.2 LTS _Precise Pangolin_ - Release i386 (20130213)]/ precise main restricted
      deb http://archive.ubuntu.com/ubuntu/ precise universe multiverse
      deb http://security.ubuntu.com/ubuntu/ precise-security universe multiverse
      deb http://archive.ubuntu.com/ubuntu/ precise-updates universe multiverse

      ctrl+x ect.

      after that again but with aptitude

      aptitude update
      and aptitude install freeipmi-tools openipmi ipmitools python wget

      doesn’t want anybody to ship me to Poland a PE 2900 heatsink? I’ve foun in country estimating 18$ another xeon 5335 to double my power in 2900:)) but the heatsink is about 65$ ( sic! ) while on whole PE with spare 6 fans I’ve spent 550$ :) ( with win 2k3 CAL! and 3x 146GB SAS and 8GB RAM, DRAC and PERC 5/i with cashe and failed BBU :( )

  63. I will definitely try the 2 above steps to make the flash possible in Ubuntu – Just need to figure out how to get the modified dat file first – I currently did modify the flash filoe, but the fans all run full speed as mentionned earlier.
    And as said, the script tells me to report the unknown PE 2900 III as code “BULN” but even modifiying the rpoposed system did not change anything…
    Shall the python script be updated?
    I checked and it already contains an entry for the BULN model, just stating it is an unknown model…
    Thanks in advance for help,
    Félix

  64. I modified the BMC on my Dell R300 today. It’s the early model with single power supply. Here’s the boring stuff:

    Firmware: R300_ESM_Firmware_V8N40_LN32_2.50_A00.BIN
    Code: CAL

    My Experience:
    I had some problems. I tried using a bootable USB of Linux Mint 14. The R300 didn’t like it, and I found out that i needed to set the USB emulation in the BIOS to Hard Drive. It still didn’t take it, but I didn’t pursue why. I put in a DVD of Linux Mint 13 I had lying around. That one booted fine.

    My second problem was setting up the environment. The command to install the IPMI tools said there were missing packages. The first one freeimpi is because of two reasons. It is misspelled in the instructions (it should be freeipmi). Second, it was listed as freeipmi-common in the repositories. I forget what the second package was, but I don’t remember it being difficult to find.

    My third problem was that it would not flash the firmware. No matter what I did, it wouldn’t do it. It did not give me any errors in terminal. It seemed to have a problem with executing bmcfl32l.

    My copy of Mint 13 is 64bit. I thought that may be the problem. I read people found Fedora 18 worked well for them. I have never used it, so thought this was a good opportunity. I wouldn’t suggest it to others if you are not familiar with it or Linux. Some of the packages were named different (although freeipmi was just that), I had to set up sudo, the interface is different than what I was used to (I use MATE), yum (Fedora’s installer) doesn’t have a GUI to look for the differently named packages (I installed yumex), and python was only version 2.2.

    However, after I got everything set up and edited my firmware file, it did finally flash the BMC. My fans had been spinning at 4k rpms and now they are around 2k rpms. It’s still louder than I would like, but it’s much better.

    When I initially set the firmware, I had the fan speeds set for 600, but I had two set for 0. I am going to remove one fan and another one isn’t sending a tach signal, so setting them to 0 won’t generate an error. Although during the time of the firmware changes, I do have 8 working fans installed.

    I wanted more fan speed reduction so I changed it to 300 and reflashed. The rpms stayed the same. I thought that maybe the 0 values were causing a issue, so I set all of them to 300. This made no difference. There appears to be some minimum value that the server will not go below.

    My fan speeds stay roughly 2100, 3100, 3700, 2600, 2250, 2250, 2100, 2200. My ambient temp is 29 and my planar temp is 30.

    Some background stuff you should know:
    The Dell R300 is a 1U rack mount server. Inside are banks of tiny 40mm that spin at high speed. There are 8 fans and they are very, very loud.

    I took the guts out of my R300 and installed them in an Apple G5 case. The case I got was used and partially stripped, but it still had the fans in it. The plan was to use the G5 fans as it came with 7 fans and I was going to install a CPU heatsink/fan.

    Because the R300 has custom fan headers, I cut the small fans off and put on standard 3 pin male connectors. I found the pinouts for the G5 fans, rewired them to work like normal fans, and installed 3 pin female connectors. I also installed the heatsink on the CPU with its own dedicated fan.

    Because the fans in the case are different sizes and styles, I believe that is why the fan rpms differ so much. The first 4 are vastly different than the last 4 which are matching pairs. Still, i’d like even lower speeds and a quieter server.

    I had hoped to save some money by reusing the G5 fans, but now I think I am going to replace them. Well, some of them to see if a different kind is much quieter. I may take some fans out. The CPU now has a heatsink with dedicated fan, and there are two 92mm fans feeding the CPU air as well.

    If I can’t get the fans slower/quieter, I may end up getting a fan controller. I was trying to avoid that, but at least I won’t have amber lights blinking at me if I do.

    Thanks for giving us options with your hard work, arnuschky.

  65. Hello – I have a number of 2950s I’ve done this with which have dual quad core 3Ghz processors, 8 x 2GB, and no hard disks (ESXi off USB), and the memory gets very hot and fails. Has anyone else experienced this?

    The ambient temp only shows as about 27c but the underside of the server where the memory is is almost too hot to touch!

    Is it just me? Anyone else got this??

  66. @Dan, I don’t have a 2950, but i know after repackaging my R300, the north bus got very hot. I put a fan facing it and it cooled off a lot. Since i have an extra fan header, i ordered a fan just for it. The reported temperature did not get hot.

    My point is, Dell doesn’t monitor the components directly and they sometimes put the temperature sensors in less than ideal areas. It may be that with the reduced rpms, the fans aren’t blowing enough hot air to the sensor for it to read it. You must also be wary of how changes you make to the server alter the airflow inside. Obviously, running without the access cover is an issue, but maybe not having the internal hard drives and/or blank covers in place is causing air to escape out the front. Check for any other changes such as a missing or incorrectly installed shroud. My R300 had one for directing air over the CPU and memory.

    Lastly, the ambient temperature is usually taken near the front display panel. It measures the outside room temperature as it goes in the server. What you should be looking at is planar temperature. This is what is measured at the motherboard inside the server.

  67. Success!! It took a little longer than planned, but got the thresholds changed on my PE 2950III running Ubuntu 13.04. The volume is a much more manageable level. The fans tend to run at 2,000 RPM–not silent, but manageable. I’m still watching to see what temperatures do.

    It’s probably somewhere in the info, but when I changed the fans, they still came on at 7,000 RPM and sounded the same, making me think I had screwed up the wiring. It took them a while to step down to the slower and quieter speed. Then they’d hit the low RPM threshold and spike back up to 7,000.

    I started out running the scripts on Ubuntu 13.04. In order to run the BIN file downloaded from Dell, I had to install ‘rpm’ with apt-get. From there everything went well until I ran bmcfl32l and got No Such File or Library. I messed with it for a while, before giving up and making a Live Fedora 32-Bit Disc. Not sure if the issue with bmcfl32l is a 64-bit vs 32-bit issue or a Debian/Ubuntu vs Red Hat.

    In Fedora, I had to install wget using yum. Also, when I ran bmcfl32l, I got an error for disneyinstall.sh, similar to others. It offered me to continue but I hit no. I checked and the hapi.gz had been unzipped properly and everything looked good. So I reran bmcfl32l without any changes and this time it didn’t give me an error. Now I have a quiet server!

    Thanks again for sharing all your work! You may want to consider a donate button–this is valuable info.

    • At 2,000RPM the RAM is overheating and causing system errors. 7,000RPM is too loud. It seems like 4,000RPM should be a happy medium. Any idea how to modify the BMC firmware to make the fans run faster? Thanks!

      • Sorry, but that’s not possible. The logic of the BMC just scales the fan speeds in terms of a percentage, starting from 100%. So there is no fixed RPM speed set in the data tables which my hack can change. I would need to change the logic of the controller which I cannot do.

        It’s unfortunate that the 2950 seems to be prone to overheating RAM (others reported similar results in the comments). You options are basically:

        – swap the fan that blows on the memory for a stock fan – louder, but at least no overheating
        – disconnect the PWM pin on the fan that blows on the memory. that way, this fan should run at 100% while the others are still quiet
        – buy another type of silent fan that spins a bit faster than your current ones
        – lower RAM frequency (results in performance loss, though)

        • Thanks for the suggestions. I was hoping there was some multiplier that could be adjusted that the BMC uses between the temp, PWM, and fan voltage to achieve faster fans, but sounds like not.

          I tried adding back in some of the stock fans–it took two to keep it cool enough and I was surprised how loud those two fans were after having the server near silent!

          Without the PMW, the replacement fan runs at 7,000 RPM, which is outside it’s operating specs, and while not as loud as stock, still pretty loud. I’d probably need to do this to two fans to keep it cool enough.

          The solution that has worked best so far is I set up two 9″ box fans from WalMart blowing into the front and over the case. Big blades, so not very loud.

          I’ll look into Dan’s suggestion of adding exhaust fans instead. However, I like the box fans because they also blow over the case which adds additional cooling.

  68. I think I have a solution – or at least it seems to work so far for a 2950:

    Use electricians tape to seal up all the holes at the back of the server.

    Buy two 230v AC 120mm fans and some rubber fan gaskets and place them over the holes at the back of the lid, so they blow upwards.

    They will draw the warm air out of the server.

    Because they are bigger than the farty little whiny fans in the server they shift more air and make less noise so using these with the slower fans and BMC upgrade seems to work a treat.

    I’ve one just done this but it seems to work. I’ll keep you posted.

    Also I have a package up firmware upgrade for windows that sets the minimum to 150rpm. If anyone wants a copy instead of having to mess about with Linux please let me know.
    Cheers..

    • Thanks Dan. Did you run external power for the 120mm fans? I looked and the only useful internal power I saw was a drive power cable at the front of the case, which is a bit long to run. Also the width of the holes is good for two 120mm fans, but the height is only about half that (approx 50mm). Did you add holes or just have half the fan blowing out the holes?

  69. I don’t think you’d be able to update the BMC to tell the fans to run at 4krpm – this is down to the fans not the firmware. With the firmware you can only set the maximum / minimum rpm.

  70. Hello,

    I continued trying to silent my PowerEdge 2900 Gen III – I had wrongly wired connections – I was in fact reading your pinout table the wrong way (Thinking you were showing from the pluggable part of the connector, not from the side the cables go in). I tried multiple possible combinations, breaking 3 fans by the way, but those were cheap ones, and now it is really silent and I understood my mistake.
    I cannot however tell if the fans start at high speed then lower, but at least they’re all running (I tried to cross pwm and tacho, but they then do not start at all).

    Now I have the problem of the low rpm alarms.

    As reported previously, the scripts gives me the choice of 8 models, and ends up saying:

    Please also report this string ‘unknown PowerEdge (code: BULN)’
    along with your type of PowerEdge! Thank you.

    Did anyone successfuly flashed a modded BMC for this model?
    Or do the script need to be adapted for this?

    Thanks in advance for your help,
    Félix

    • The “unknown PowerEdge” message is only for display and does not affect the functioning of the chip. Have you tried to change the thresholds of the fans and then tried to flash the firmware? There have been other people reporting success with PE2900 Gen IIIs.

      • Hello,
        Thanks for your answer…
        I tried to patch the firmware on a live ubuntu as you explained it, but have not been able to flash from this OS. I therefore copied the modified dat file to a usb key and flashed from windows.
        I even tried to lower the values of all the fans of all the 8 proposed systems before flashing, but it still complains about fan speed to low.
        I did this a few weeks ago, and have not retried since, I will retry to patch the dat file and flash, if you have recommandations regarding which live distro shall be used, so both the patching and the flashing can be done at once, I would be very happy…
        I’ll post my findings once I’ll have done some few more tests anyway.
        And thank you very much for sharing your experience on this – I was searching on a way to transform this combat copter into a silent machine for weeks :)
        Félix

  71. Hey guys! Quick question, I know its a super old thread but I just got a poweredge 2600. I was wondering if the tut on here would work for my machine.. I dont want to brick … well a working brick

    • No idea if someone ever tried it (can’t remember). Check the comments, if you can’t find anything check if the script recognizes the firmware file. If it does, you should be fine.

  72. Hello,

    Thanks for the great advice and script. I have a poweredge 2850 that came with 6 Delta AFB0612EHE case fans and 2 Delta AFB0612EH power supply fans. I was able to swap the power supply fans with two Fractal FD-FAN-SSR2-60 fans (same wire colors, different connector) and the case fans with 6 Nidec U60T12MUA7-57 fans (same color wires except PWM is yellow and tachometer is blue).

    The steps for re-flashing the BMC worked great with a couple notes:

    – the “bmcfl32l” file that is called to flash the controller has the same bug as the “BMC_FRMW_LX_R223079.BIN” file where the header reads “#!/bin/sh’ when it should read “#!/bin/bash”

    – the “bmcfl32l” file is a 32 bit binary so make sure that libc6-i386 is installed

  73. Have you seen or had any success stories with a poweredge 1950? I tried your script but my fans was in the late teens and 20s while your script showed the 16 fan sets in the 40s. So it did not work for me. Still looking for some kind of solution

    • I don’t understand your problem, sorry. Please note that the script does not affect the speed of the fans itself. It just allows you to drop in fans with a very low speed.

  74. I’ve been reading this without comprehension for a while. I have a 2900 III with the replacement fans and like all the others, it drifts downward and hits the low limit and goes back up high again. I don’t have a running Linux box. Is there a modified BMC somewhere that I’m not finding, or do I need to build a Linix box, and learn how to modify the script? Sorry for such dumb questions, but I’m confused.

    Thanks in advance

    • You don’t need a Linux box, you can use a boot cd for booting into a live Linux (no installation required). Alternatively, you can try to contact some other posters on this thread that have a 2900.

  75. Whoops, posted this comment on another site, here it is anyway…

    I cannot thank you enough for posting this! The instructions worked incredibly well and my server has gone from something that terrified my cats, to a large box no more loud than my previous server. The only hiccups I had were on Dell’s side, namely that their flashing suite (at least the only one I could find for the 2950) refused to do anything on a non-rpm (Fedora, CentOS, etc.) system and the utility itself would only run on a 32-bit architecture.

    It’s guys like you that make Linux and its associated distros, the greatest collection of operating systems in existence. Thank you.

  76. Few issues I had with my Dell power edge 2600:
    First of all (L)Ubuntu 13.04, 12.04 wont start at all.
    Have no idea why ( video is suspect number 1)
    But pretty new distro called SolusOS 1.3 non pae – started.

    #1. ipmi-sensors command show only two lines – with cpu temperature.
    ipmi-sensors -g fan <- do the trick ( collect IDs here

    #2. BMC firmware for 2600 is not exists. May be I am just a stupid person who can't find and download a simple file form a web site … may be dell web-dev team did something wrong… who knows but anything I can download from there stopped on #3 – yay. There is no YAY there :(
    I mean I can't find bmcflsh.dat for Dell Power Edge 2600.

    So I was not able to proceed to #4.

    Here is my workaround:

    #1- ipmi-sensors-config –checkout –filename=/home//ipmi.cfg
    We get everything related to sensor in the file ipmi.cfg
    Open it , change whatever you want ( I wanted XXXX_Fan_1 .. ZZZZ_Fan_5)
    Lower_Non_Critical_Threshold
    Lower_Critical_Threshold

    I set them to 940 both

    #2- ipmi-sensors-config –diff –filename=/home//ipmi.cfg
    to make sure you don’t change something else

    #3- ipmi-sensors-config –commit –filename=/home//ipmi.cfg
    Write values.

    #4- ipmi-sensors-config –diff –filename=/home//ipmi.cfg
    To double check what values we have active.
    * Here I am found out that I have 960 in use instead of 940 (Don’t know why)

    BUT !!!
    After reboot I get my fan speed back ;(( So I put it #3 to autoload

    • You managed to change your critical thresholds using IMPI? Must be the 2600 then – never worked for my 2800; the thresholds are read-only. Hence this hack. :) Congratulations on finding an easier solution!

  77. Greetings – my Linux tech tried to use the Red Hat BMC firmware download on an Ubuntu system. When he ran the flasher, libraries could not be found. But, now the 2900 hangs on power up, just about where BMC prompt should appear. Does this mean the motherboard is toast?

  78. Update: the system boots now that I removed the DRAC 5. Previously, the DRAC would be listed but without any of the additional lines about IP address or key combinations for configuring the card. Still don’t have the fans configured but don’t have to make it boot. I think that the issue is that we can’t expand/decompress the Linux format firmware. My friend says he was able to at his house with a system not running Ubuntu. How do you open the file with Ubuntu so you can then run the script to change the thresholds?

  79. Thanks for you hard work with this! I highly recommend the Fedora 19 32-bit Live CD as the best way to go about making these modifications. Centos 6.4 64-bit struggled, and I couldn’t even get Debian 7.1 64-bit to boot up on my Poweredge (likely a driver module not being loaded early enough!).

    Another question, however, does anyone know of anyway simple way to reduce the CPU frequency e.g. is there any way to turn the multiplier down to save power and heat?

  80. Hi

    Great site BTW

    I see you offered the ability to modify the bin file directly for another user who uses windows, is that offer still open?

    I have ESXi installed on this 2900 and is a BMC update possible from within a VM or does it need to be installed on a HD on its own?

    • Thanks. Sorry, I never offered that service – it would be illegal. Maybe one of the others commenters did?

      I do not not if the BMC can be updated from within a VM. I doubt it. No need to install it on a HD though, just use a boot cd.

    • You can boot from a Fedora 19 32-bit live CD and perform ALL of the necessary adjustments from there. No messing with VM’s or other installations.

  81. December 14 2011

    “If you can extract your Windows package so that you can flash the bmcflsh.dat file separately, you can send it to me and I can adjust the thresholds for you without any problem.”

    Can you adjust it for me as i am a linux noob.

  82. As with previous comments, I had trouble using Ubuntu with my Poweredge 2900. So do the following.

    I used Arctic F9 PWM fans:

    * Use the latest firmware “Dell BMC Firmware, v.2.50, A00″
    * Use “Fedora-Live-Desktop-i686-19-1.iso” and create a live cd or usb (I used Lili Usb Creator for this)
    * Follow the instructions posted here changing “apt-get” to “yum” (Fedora vs Ubuntu command), also when you load up the terminal, run “sudo su” to change to root user.
    * I set the each of the fans to 5 (which is 375 RPM) as the Arctic fans run at 500~600 rpm under light load.

    Perfect. Thanks to Arnuschky for this great post.

  83. Ok, ive managed to adjust the bmflash.dat file, however for some reason i cannot extract the firmware manually in Linux. Ive got the dat file but i cant extract the bin file to enable flashing as well.

    Does anyone have the already extracted files that enable flashing? Im also confused as to the last step as well.

  84. Scratch that alotgether.

    You dont need to run a fan mod at all on the PE 2900

    Just cut the blue wire which is the PWM and the Arctic F9′s run at full speed withour triggering the LSF alarm. Sorted!

  85. Have a 2950 III (Version3). I have original fans that I modified with 30ohm 5W resistor. Is it any good of me flashing modified BMC. ?? Also concerned about sensor numbers not matching up when running modify script. I have no problem getting to the point of have the firmware modified but concerned that sensor numbers did not match up. I used option 3 and lowered thersholds and saved .dat but have not flashed yet. Anyone please let me know what I should do next. IPMI says ID’s 22 to 27 (6 fans) but script says 48 to 53??

      • as for the ids, i wouldnt bother. you’ll see, if you scroll up a little bit, the script found the right ids for the fans. ( had the same problem, but it worked fine 1h ago on my 2950iii )

  86. I have a 2950 III . I modded the original fans with 30ohm 5w resistor. Is it still ok for me to mod the threshold in the firmware while using the original modded fans?? I have managed to mod the BMC firmware file to the point that I am ready to flash. However I am concerned as the ipmi-sensors data showed different IDs that what was shown when the modify script was run. I used option 3 in the script. The ipmi-sensors data show IDs 22 to 27 as Fan 1 to 6 but the script shows IDs 48 to 53 as Fans 1 to 6. Show this be of concern before I flash.

  87. After 3 days of head scratching, here is the exact way i did it, and it worked.

    for a PowerEdge2950 iii

    on ubuntu 13.04 64bit live-usb

    boot from the usb and choose try ubuntu.

    with the Gui, navigate to GearIcon (System Settings) then click Software&Updates, check everything, and close window

    then in terminal:

    sudo su root

    apt-get update

    apt-get install freeipmi freeipmi-tools openipmi ipmitool python wget rpm

    wget http://downloads.dell.com/FOLDER00928606M/1/2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN

    chmod 755 2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN

    chmod +x 2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN

    rm /bin/sh && ln -s /bin/bash /bin/sh

    mkdir bmc_firmware

    ./2950_ESM_Firmware_4NNNG_LN32_2.50_A00.BIN –extract bmc_firmware

    cd bmc_firmware/

    cd payload

    wget https://raw.github.com/arnuschky/dell-bmc-firmware/master/adjust-fan-thresholds/dell-adjust-fan-thresholds.py

    chmod 755 dell-adjust-fan-thresholds.py

    ./dell-adjust-fan-thresholds.py bmcflsh.dat

    ( choose option for poweredge 2950 )

    cd ..

    chmod o+x bmcfl32l

    dpkg-reconfigure dash

    ( Choose Yes )

    apt-get install libc6-i386

    LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

    • Just followed your write up on my 2950 using Linux Mint 17 booted off usb. Thank Phil and arnuschky for providing the how to’s. The noise level is dramatically lower.

      I have trouble also running ipmi-sensors like a few other posters. It crashes after it reads 20-30 things. I went along with the install anyhow and used half values like arnuschky recommended.

      Next step is the fans inside the PSU’s. Do people have much luck with the resistor slowing the speed down in the PSU? or should I also replace those fans.

      • An update on my 2950.

        I installed resistors on the PSU’s. I ended up getting memory errors. I think it was error E2111. I removed the resistors and no more errors. The fans in the PSU must act as some sort of active cooling to vent some hot air.

        I did notice that the memory still seemed pretty hot though. I decided to install some active cooling in the back being this is the longest thing on my rack and there is nothing above it. I put a 80mm fan on it, and used the aluminum tape that is for duct work to seal all the vent holes on the top and back. To power the fan I used the power connector in the front that is for a tape drive I believe.

        http://www.imagegainer.com/images/BillyC333/img20140820204852.jpg

  88. I just wanted to come back and revisit this site. I have several comment posted up top from about a year ago when I originally did this mod on my PE2900 running ESXi 4.1.

    I had mentioned some heat issues I was having and, seeing how the temperatures are dropping out in my neck of the woods, I began having some issues again. Let me explain the problem(s) for those of you looking to do this mod. (I might have to break this up again over a couple of posts):

    The BMC appears to control the installed fans at a percentage of what the system board ambient temperature is. This temperature measurement is the only visible temperature indicator exposed on the PE2900. There is no direct way to know how hot your RAM or HDD or running, only the “ambient” temperature of the system board.

    So, when the room your server in is warm, the ambient temperature is generally around 25-30 degrees Celsius – at least this is my measurement in my office when the temps range from high 60′s to low 80s (Fahrenheit). At these temperatures, the fans seem to spin anywhere from about 30-50% of their max RPM. For my 2500RPM fans, this means between about 800-1300 RPM. At those speeds, things seem sufficient to keep all my components cool enough to function.

    However, when the temperatures in my office drop below about 68F, the “ambient” temperature inside the case also drops below 25C. This causes the fans to spin slower, since they believe the case is being kept cool. Unfortunately, since the system isn’t monitoring the RAM or HDD temps separately, it is getting a false view of things. It is like your hallway thermostat saying it is 63F in your home because you don’t have the heat on, but underneath all your covers in your room it is really 75F!.

    When my ambient system temps drop to about 24/25C, the fans are spinning at only about 5-600RPM. This isn’t enough to keep the box cool and I get RAM and CPU failures and the box shuts down. The reason my fans aren’t/weren’t spinning up is because I set my threshold to only 450. The reason I did this is that as I was tuning it I was finding late at night/early morning when it is colder in my home the fans would start whirring up to their max when they hit the minimum threshold due to low ambient temps. This was quite annoying and loud so I kept tuning down the threshold until this didn’t happen. Unfortunately, when you get low enough (450) that it doesn’t do this, it isn’t enough to keep the box warm. (This is what arnuschky was describing as “Panic Mode” here: http://projects.nuschkys.net/2011/08/03/these-darn-fans/)

    In the end, my solution is three fold:
    1) Set the minimum threshold at 600 – I found this to be the best compromise as any lower than this I risk overheating. Under normal temps in my office I usually am spinning at 800-1300 RPM

    2) Regulate my office temperatures: Because, if it gets to cold, the fans go in and out of their whir cycle when they fall below 600RPM. This means I can’t let my office fall below about 68F or else I’ll get the fans whirring up every 5-30 mins (depending on just how cold it goes). This is pretty easy for me to do – as long as I remember to close the door to my office on cold nights – the heat from my machines keeps it toasty. But, if/when I forget – I have an overheated and downed server.

    3) Because I don’t like to keep risking the overheating and the fan noise, I have set up a system using OpenManage’s SNMP traps to a linux box running snmptrad that will trigger the server to shutdown when the RPM alert is triggered. I was initially planning on doing something similar using the ambient temps as a trigger – but it isn’t very reliable as my system may not have enough time to shut down when it begins to overheat. With the RPM trigger, and a 600RPM threshold, the server may whir up once or twice to keep things cool while my system shuts things down. This way – I don’t need to worry about the fans over working and the noise waking us up.

    I have read all te literature available about this hack from the several people who have written articles, and every single comment. I do believe that everyone (at least PE2900 owners) will have these same problems. Either you will need to set the RPM threshold high enough to keep your server cool – in which case “Panic Mode” will happen when the fans spin too low, or you will have to set the RPMs so low as to not trigger the “Panic Mode”, but doing this risks overheating the other system components. Lesson: If you can’t control the ambient temps where your server is stored within an acceptable range – it is probably best not to do this hack. Of course, most people looking to do it likely have their server in a “livable” area where they want to cut down on noise….just don’t let your basement closet get too cold!

    Finally, I want to mention that I just updated to the latest BMC firmware from Dell (2.50?) and used this hack and it worked again great. The only problem I had is I just also updated to ESXi 5.5 and it is impossible (or at least I couldn’t find a way) to update the firmware from within ESX (I was able to in 4.1). I believe this is due to a lack of a bash shell within ESXi. Anyway, I build a Knoppix live CD and did the update from there.

    Once again, thanks to arnuschky for writing this hack and providing a forum to discuss. While I may have lessened the life of some of my system by the occasional overheats I have gotten, the overall experience of a quieter system has probably been worth it.

    • another solution on pe2900 is cut pwm (control) fan wire. Then all fans run at 100% 24/7 or solution 2, intsall fans with higher RPM spec than artic F9s. I used Asetek Vapochill Micro 92mm PWM 3800RPM x 6 and df126025bm 7000rpm x 2 (psu) coz @ ambient temp of approx. 20C my F9′s ran at about 250rpm, while the asetek vapochill’s run at 850rpm , so they move more air.

  89. hi

    your page did help . it clarified a few issues and pointed me in the right direction. thou not the the one listed on your website for flashing. I have a dell 2900 ver. iii which takes the 54xx cpu’s (London), which I installed a small HDD 80gb on which I installed 2008 r2 x64 server then later Ubuntu 12.04 lts as dual boot. So knowing my machine and having its service tags I could download the correct BMC update package to my windows machine. The package arrived as an Executable. I also download your python script and saved as fanmod.py.

    so enough history, now some meat.

    I unpacked the downloaded dell EXE with WinRAR to a desktop folder and inside found the payload folder with an unvarnished bmcflsh.dat (the BIN download from Dell has extra crap) and transferred to USB stick along with the python script(fanmod.py).

    Booted to Ubuntu , copied bmcflsh.dat(firmware) and fanmod.py (python script) to Home folder from usb and opened Terminal.

    >python fanmod.py bmcflsh.dat

    chose option 1 as it fits my machine, your python script ran well on Ubuntu and modified the original bmcflsh.dat with the new LCR threshold in situ.

    I then copied the new modified DAT file to usb (bmcflsh.dat original 534kb now 544kb).

    Reboot to windows

    opened my extracted folder and deleted file in payload folder and replaced with the modified file on usb(files have same names).

    Inside this extracted folder is a file called spsetup.exe just double clik and your all done. 8mins 52secs later that for the dell package prog. 2 days study later.

    This worked for me and it is all thanks to Arnuschky. Hail Arnuschky….

  90. Pingback: Silence the Dell PowerEdge 2950 » Martin DENIZET

  91. RE: “However I am concerned as the ipmi-sensors data showed different IDs that what was shown when the modify script was run. I used option 3 in the script. The ipmi-sensors data show IDs 22 to 27 as Fan 1 to 6 but the script shows IDs 48 to 53 as Fans 1 to 6. Show this be of concern before I flash.”

    I’ve got a 2950 that reported the same thing – 6 fans on sensor IDs 22-27:
    ipmi-sensors:
    22 | FAN 1 RPM | Fan | 7875.00 | RPM | ‘OK’
    23 | FAN 2 RPM | Fan | 7725.00 | RPM | ‘OK’
    24 | FAN 3 RPM | Fan | 7425.00 | RPM | ‘OK’
    25 | FAN 4 RPM | Fan | 7425.00 | RPM | ‘OK’
    26 | FAN 5 RPM | Fan | N/A | RPM | N/A
    27 | FAN 6 RPM | Fan | N/A | RPM | N/A

    I flashed anyway… I don’t know if the sensor_id
    ./dell-adjust-fan-thresholds.py payload/bmcflsh.dat:
    3) PowerEdge 2950
    Number of fans: 6
    Fan names : FAN 1 RPM, FAN 2 RPM, FAN 3 RPM, FAN 4 RPM, FAN 5 RPM, FAN 6 RPM
    Fan speeds : 975, 975, 975, 975, 975, 975
    Sensor numbers: 48, 49, 50, 51, 52, 53

    I flashed as the instructions indicate “Just use the number of fans, their names and thresholds to identify your system.” – Nothing about the fan id.

  92. Does the python script run OK under Windows? I haven’t been able to test it under linux yet, but in attempting to patch the .dat file, I just get CRC error during verify.

    C:\Temp>python dell-adjust-fan-thresholds.py C:\Temp\2950_ESM_Firmware_4NNNG_WN32_2.50_A00\payload\bmcflsh.dat
    Opening file
    Verifying file…
    Error: file CRC failed (0×0780 vs. 0xfc85)!

      • I’ll try the Ubuntu Live and Linux packged version later tonight… the Windows version is an EXE from which I just pulled the .dat file. The Linux version has all the shell script stuff attached.
        I know the .dat files SHOULD be identical, but one never knows.
        Thanks

        • I thought the command was…

          C:\Temp>python dell-adjust-fan-thresholds.py bmcflsh.dat

          with the bmcflsh.dat extracted from the downloaded EXE and place in the same directory as dell-adjust-fan-thresholds.py. eg. your Temp dir

  93. Pingback: Harrier Jumpjets! | Hexland's Blog

  94. Many thanks to the OP and Phil. I thoroughly read and understood what the purpose and method was. I first attempted to run the Python script on my Windows machine using the .dat file I extracted from the Windows .EXE download. Unfortunately, just like David Cowan report, I also got the same CRC mismatch. Then I attempted to perform your extraction steps using Ubuntu 12.04, and the extract part never completed successfully. Finally, I followed the step provided by Phil, and it worked finally :)

    As a follow-up, I would like to know more about what causes certain fans to run faster than others. For example, I’m using a PE2950 Gen3 and there are 4 fans inside, all of which have been replaced with quieter fans. Two of them on the right run at ~4000rpm sometimes while the other two remain at ~1500rpm. What sensors are being leveraged to determine this?

  95. Pingback: Hacking Fan Speed on Dell PowerEdge Servers | jafrro.net

  96. Hello,

    I have problem after upgrad fan-s and BMC. I over heat my PE 2900 :( Fan are set to 600RPM but they not spinn to full. I have Arctic Coling F9. Now i have open the case but that don’t solve this problem.

    help

    • Depending on which component overheats, you can swap one of the stock fans back in. Of course it’s louder, but it’s a compromise between noise and cooling.

      • Hello,

        Radiators on motherboard ar very nigh:
        ch 0 dimm 0: 99
        ch 0 dimm 1: 82 …
        ch 3 dimm 1: 84
        core 0 :00 ..
        core 7: 83′

        There are no software fix i bmc to speed fan over temp ?

        • Unfortunately no. My hack only changed the values of thresholds (ie, the data) rather than the logic. Changing the logic would require to decompile the firmware and actually understand what they are doing in there. Lots of work! :(

  97. Getting this working using Fedora 20 turned out to be more annoying then anticipated. Following the steps listed above will yield an error about no srvadmin-hapi. To solve this simply run wget -q -O – http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash
    That will download the dell repo’s. Then edit the dell repo file /etc/yum.repos.d/dell-osma-repository.repo.

    My final mirrorlist within that file was http://linux.dell.com/repo/hardware/latest/mirrors.cgi?osname=el5&basearch=i386&native=1&sys_ven_id=$sys_ven_id. after a yum clean all, this allowed me to do yum install srvadmin-hapi. This installed fine. to /opt/dell/srvadmin. Copy that whole directory to dup (cp -R srvadmin/ dup/) and then try running bmcfl321 with the options specified in the tutorial again.

    This worked great for a 2950 with fedora 20 i686 (assume it will work with x86_64 with the right compiler installed). I used 975 as my threshold. Thanks!

  98. Thank you for your work!
    Successfully did that trick for my 2900. Tried in Ubuntu initially, but finally did in Fedora Live

  99. I figured out the Failed to load library problem on ubuntu server 12.04.3 LTS amd64.
    I successfully flashed my sc1425 and my 2950 by simply running
    “sudo dpkg-reconfigure dash”
    and selecting no. Hope this helps.

  100. Hi,
    I tried your method on Powerege 2950 III and ubuntu 12.04.4. BMC version is 2.50
    I did everything that you wrote in this article, flashing was correct but fans’ speed cannot change.
    What should I do?

    ipmi sensors before modifing

    22 | FAN 1 RPM | Fan | 7725.00 | RPM | ‘OK’
    23 | FAN 2 RPM | Fan | 7650.00 | RPM | ‘OK’
    24 | FAN 3 RPM | Fan | 7725.00 | RPM | ‘OK’
    25 | FAN 4 RPM | Fan | 7800.00 | RPM | ‘OK’
    26 | FAN 5 RPM | Fan | N/A | RPM | N/A
    27 | FAN 6 RPM | Fan | N/A | RPM | N/A

    after the flashing

    22 | FAN 1 RPM | Fan | 7725.00 | RPM | ‘OK’
    23 | FAN 2 RPM | Fan | 7650.00 | RPM | ‘OK’
    24 | FAN 3 RPM | Fan | 7725.00 | RPM | ‘OK’
    25 | FAN 4 RPM | Fan | 7800.00 | RPM | ‘OK’
    26 | FAN 5 RPM | Fan | N/A | RPM | N/A
    27 | FAN 6 RPM | Fan | N/A | RPM | N/A

    • You did exchage the fans against quieter ones, right? My patch only allows to swap fans – the stock firmware does not allow to do this because of the hardcoded thresholds.

      If you flash the patched firmware without changing the fans it should not change anything.

      • root@ubuntu4:~# ./2950.BIN –extract bmc_firmware
        Successfully extracted to bmc_firmware
        root@ubuntu4:~# cd bmc_firmware
        root@ubuntu4:~/bmc_firmware# wget https://raw.github.com/arnuschky/dell-bmc-firmware/master/adjust-fan-thresholds/dell-adjust-fan-thresholds.py

        Select fan to adjust:

        1) fan sensor number 48, threshold 2025, name ‘FAN 1 RPM’
        2) fan sensor number 49, threshold 2025, name ‘FAN 2 RPM’
        3) fan sensor number 50, threshold 2025, name ‘FAN 3 RPM’
        4) fan sensor number 51, threshold 2025, name ‘FAN 4 RPM’
        5) fan sensor number 52, threshold 2250, name ‘FAN 5 RPM’
        6) fan sensor number 53, threshold 2250, name ‘FAN 6 RPM’

        LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH ./bmcfl32l -i=payload/bmcflsh.dat -f

        Baseboard Management Controller Flash Utility Version: 1.33
        Copyright 2003-2007 Dell Inc. All Rights Reserved.
        Loading file completed CRC OK

        Firmware Versions Current Image
        BMC …………………………………… 2.37 2.50

        Force flash BMC
        Proceed with flash? Y/N: Y
        Processing BMC
        Flash Completed
        Checking Revision completed
        After flashing:

        ipmi-sensors |grep FAN Mon Mar 3 16:46:10 2014

        22 | FAN 1 RPM | Fan | 7725.00 | RPM | ‘OK’
        23 | FAN 2 RPM | Fan | 7650.00 | RPM | ‘OK’
        24 | FAN 3 RPM | Fan | 7950.00 | RPM | ‘OK’
        25 | FAN 4 RPM | Fan | 7950.00 | RPM | ‘OK’

  101. Tested in PE2950
    Conclusion is can flash the BMC but none of effect. The fans run 5xxxrpm before flash, and the threshold is 2250rpm before flash. And no matter what value you flashed, it keeps above 5000rpm
    Found Necessary Dependencies: compat-libstdc++-33.i686 wget libc6-i386
    in Ubuntu 12.04LTS x64 won’t work at all, even installed all libraries it need. you can try to run ./xxxx.BIN -v so it will check dependencies(I got them by this way too).
    in Fedora 18 x32 works but in a different way
    if i run ./bmcfl32l -f -i=xxxx it will require gcc to compile dell_rbu(i dun know what is it too). as mentioned by sb else in comments, you need to (add repo first) manaually install srvadmin-hapi and copy all things from /opt/dell/srvadmin -> /path_to_bmc_fw/hapi/opt/dell/dup
    then the compiler will compile it. i did it but compile failed. i can solved the problem by reading error log of make. so i tried any method.
    i found if i run ./spsetup.sh will get same result(same text output in terminal) of ./XXXX.BIN it means the .BIN extract the resources to a temp folder and then execute ./spsetup.sh automatically. and since the resources in the BIN file are hard to phrase, so i stopped to try modify the BIN file with modified firmware file
    I found that the spsetup.sh will move(NOT copy) the bmcflsh.dat from ../payload/ to ../ and then execute “./bmcfl32l -f -fml” and bmcfl32l will flash the bmcflsh.dat from default path(which is the same working path) if all things get well(passed inventory check and confirmed by user). and it will also take care anything about the $LD_LIBRARY_PATH and disneyinstall.sh, even you didnt installed srvadmin-hapi
    so the best way to run the bmc fw update is using ./spsetup , no need to get crazy about ./bmcfl32l
    also, it’s odd that if i try to run ./bmcfl32l -f -fml -i=xxxx.dat, it got failed(Failed to load library). i can’t figure out the reason and i dun want to know now(coz it wasted me 2 days).

    • You have to physically change fans: replace the original fans with quieter ones. My script only allows you to do that.

      Please read all the articles again.

      • Oh I m sorry but even i set to 0, i can’t remove any fans or fill in fans with tissue to stop it. otherwise, the bmc will turn all fans to full speed(including fans in power supply)
        im wondering what’s the definition of threshold in bmc.

  102. Well I went through and replace the fans in my 2950 III along with alot of people on this site. I am having problems with cpu1 temps. When I put a load on the server, fan1,2 they go up to full, 3,4 stay around 2000 or so. CPU1 starts boiling. The other is nice and cool. If anyone has seen anything like this please post. They rev up to full speed just fine with the cover off and cool everything back down nicely. They just do not seem to respond to the bmc anymore without an error.

  103. Worked on a 2950 I got second-hand. Swapped Fans with some 25mm ones. I had to grab the dell centOS from linux.dell.com. Ubuntu had too many missing dependencies, and I’m not linux smart.

    I used the steps and commands from above. I just had to wget from “http” instead of “https” for the python script.

    Dell OMSA on the top level does not show any fan “health” issues. But If you dig into the individual fan probes it shows as an error, but does not cycle the fans up.

    Thanks Much for this tool! My file server is quieter then my desktop!
    Cheers!

    • Oh with dell centOS there was no need to install any additional packages. Also did not have to do and “sh” and “bash” swapping.

      My first step was making the folder to extract to and extracting the .BIN.

      All in all fan swap and flashing took 3 hours.

  104. what about hacking the program that monitors the temperature and controls the fan speed instead of replacing the fans? it should be somewhere in BIOS or BMC … this has been done for thinkpad laptops before

    • Sure, would certainly be possible. But it would involve decompiling/reverse engineering the firmware, which is quite a lot of work, and I don’t know how to do it…
      If you do, have a go at it!

      • I found this text on Dell’s website:
        “Dell BMC Firmware, v.1.22, A04
        The BMC firmware monitors various sensors on the system board and adjusts certain parameters to maintain normal operation. This update is designed to improve the fan speed algorithm which impacts the overall power draw of the system. The BMC firmware checks the configuration of system components” ref: http://ftp.dell.com/Pages/Drivers/poweredge-1950-all.html

        this means BMC controls the fan speed and ,now i’am 100% sure BMC contains the fan speed algorithm, so we need to mod BMC to make the fans go slower

  105. I think what you posted was actually very logical. But,
    think on this, what if you wrote a catchier post title?
    I am not suggesting your information isn’t solid, however suppose you added something to possibly grab folk’s attention?

    I mean How to adjust the fan thresholds of a Dell PowerEdge | tüfteln is kinda boring.
    You might peek at Yahoo’s home page and note how they write
    news titles to get viewers to click. You might add a video or
    a pic or two to grab readers interested about
    everything’ve got to say. Just my opinion, it would bring your website a little bit more interesting.

  106. Tried it on my dell R210, did not work……
    Any idea how i can mod. my R210 BMC to accept a lower rpm silent fan?

    Z69

  107. Finally got this working on my 2900, using hints from Chaim Feb 9 post (Thanks, Chaim!).
    I’m not a linux expert, so a couple things that helped debug was to remove the -q from his wget, which clued me in. There is a long dash character in his wget line, and the script wasn’t getting piped to the shell and didn’t run. Changed it and ran it directly.

    Also, the trick was to edit the dell-osma-repository.repo file and find the last ‘mirrorlist=’ line and replace its value with the link in his post. (Again, this might be obvious to folks with more experience in this).

    I did this using Fedora 32 bit, using the Ubuntu instructions but replacing apt-get with yum.

    Now my system is quiet – finally!
    Thanks to the OP for putting together the mod.

  108. First thanks so much for this!

    I was able to flash my BMC on my Dell PowerEdge 2900 in CentOS 6.5 x86_64. I had to make some changes to your procedures
    The latest BMC package is:
    ESM_Firmware_C6H4X_LN32_2.50_A00.BIN

    First you need to install the OMSA repo via:
    wget -q -O – http://linux.dell.com/repo/hardware/latest/bootstrap.cgi | bash

    Second install compat-libstdc++-33 libstdc++ libxml2 32-bit glibc freeipmi ipmitools OpenIPMI OpenIPMI-libs and srvadmin-all:
    yum install OpenIPMI freeipmi ipmitool glibc.i686 srvadmin-all compat-libstdc++-33.i686 libstdc++.i686 libxml2.i686

    Start the ipmi service
    chkconfig ipmi on
    service ipmi start

    Extract the bin the same way but you can skip the sed command as centos will interpret the typeset commands correctly in the bin.

    Patch the payload/bmcflash.dat the same. I chose choice 1.

    When attempting to flash you need to extract the hapi.tgz in the hapi dir
    cd $EXTRACTED_BMC_BIN_DIR/hapi
    tar xvzpf hapi.tgz

    Then you need to export the path and LD_LIBRARY_PATH
    export PATH=./hapi/opt/dell:$PATH
    export LD_LIBRARY_PATH=./hapi/opt/dell/dup/lib:$LD_LIBRARY_PATH

    Finally you can flash the bmc image:
    ./bmcfl32l -i=payload/bmcflash.dat -f

    It will start the dell_rbu and ipmi drivers, compare current and image version numbers and then ask to proceed with flash.

  109. FYI, I am selling my machine as I can’t run it anymore at home.

    It’s the Dell Poweredge 2800 that I used in my posts, that is, it has silent fans and a hacked BIOS. Additionally, I converted it to SATA: it has 8 cold-swap SATA slots, and a 12-port SATA hardware raid controller (see http://projects.nuschkys.net/2011/12/19/converting-a-poweredge-scsi-backplane-to-support-sata/ for more info).

    For sale. Ships from Belgium, price to discuss. If desired I can also sell 8 matching 1TB SATA disks.

  110. I’ve got a 2950 here, and when i try to run ipmi-sensors (sometimes it just crashes), it shows me the fans are on 22 to 27, nothing else shows up.

    I’m guessing this mod won’t work for this server :/.

    Certainly not going to try and flash it to see what happens.

  111. Hey man, nice write-up. Im trying to use your script on my dell pe1950 gen 2. Im at the point where i choose the system for editting, but i do not see my system. I see a similar system with the same amount of fans and the fan names are the same but the sensor numbers are different. My question is, can i change specific settings and number assignments within your script or the bmc bin file?
    In addition, can you change actual fan speeds under specific loads with your method, or just the critical speeds?

    Thanx,
    qdellinger

      • Well now i have a new/better idea. Im going to play with resistor values until i get a fan speed that is nearly half my normal speed, as long as the cpu is fin with that, then i will try to use your method to adjust the fan thresholds.

        But, im still unsure if i should even flash the editted bmc firmware as my sensors numbers did not align with your python script.
        My system is closest to mathing your
        #4) unknown PowerEdge (code: BRLN)
        Number of fans: 16
        Fan names : FAN MOD 1A RPM, FAN MOD 1B RPM, FAN MOD 1C RPM, FAN MOD 1D RPM, FAN MOD 2A RPM, FAN MOD 2B RPM, FAN MOD 2C RPM, FAN MOD 2D RPM, FAN MOD 3A RPM, FAN MOD 3B RPM, FAN MOD 3C RPM, FAN MOD 3D RPM, FAN MOD 4A RPM, FAN MOD 4B RPM, FAN MOD 4C RPM, FAN MOD 4D RPM
        Fan speeds : 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125, 1125
        Sensor numbers: 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63

        Unfortunately my system fans look like this:
        22 | FAN MOD 1A RPM | Fan | 7425.00 | RPM | ‘OK’
        23 | FAN MOD 1B RPM | Fan | 7425.00 | RPM | ‘OK’
        24 | FAN MOD 1C RPM | Fan | 4800.00 | RPM | ‘OK’
        25 | FAN MOD 1D RPM | Fan | 4575.00 | RPM | ‘OK’
        26 | FAN MOD 2A RPM | Fan | 7275.00 | RPM | ‘OK’
        27 | FAN MOD 2B RPM | Fan | 7425.00 | RPM | ‘OK’
        28 | FAN MOD 2C RPM | Fan | 4725.00 | RPM | ‘OK’
        29 | FAN MOD 2D RPM | Fan | 4575.00 | RPM | ‘OK’
        30 | FAN MOD 3A RPM | Fan | 7725.00 | RPM | ‘OK’
        31 | FAN MOD 3B RPM | Fan | 7725.00 | RPM | ‘OK’
        32 | FAN MOD 3C RPM | Fan | 4800.00 | RPM | ‘OK’
        33 | FAN MOD 3D RPM | Fan | 4650.00 | RPM | ‘OK’
        34 | FAN MOD 4A RPM | Fan | 7500.00 | RPM | ‘OK’
        35 | FAN MOD 4B RPM | Fan | 7425.00 | RPM | ‘OK’
        36 | FAN MOD 4C RPM | Fan | 4800.00 | RPM | ‘OK’
        37 | FAN MOD 4D RPM | Fan | 4725.00 | RPM | ‘OK’

        Is there any way to change the sensor numbers allocated in your python script. I have ran through your code on gtihub, and i cant figure out how it works. – Please Help.

        -qdellinger

        • There are some internal sensor numbers as displayed by my script. Some impi-tools show other numbers, if I remember correctly because they simply index them rather than displaying the internal number. In any case, their order should match, that is, 22=48 and 37=63. Just fiddle with it, you can’t break anything by changing the wrong value.

  112. Thank you for the two part write up, just sorted out my 2950.

    The only place I could find 60mm PWM fans was from an EBay Chinese supplier, they are 12.5mm wide fans so 25% the width of the original fans.

    I had two goes at sorting out the firmware using your script, the first attempt I just updated one entry your script found, that didn’t seem to do anything, so the second time I reprogrammed all the fans for all the models in the firmware.

    I never did get the firmware flashed in Ubuntu, had to copy the patches firmware to a USB stick, reboot into Windows, download the Dell Windows update and flash package, I didn’t run it as provided as that would just flash the unmodified firmware. Used 7zip to unzip the contents of the Dell Windows update and flash package and having a look at the files inside it contained (amongst other stuff) the (unmodified) firmware and the Windows flasher utility.

    I simply replaced the unmodified BIN file with the modified one and ran the flasher.

    Worked no problem.

    Of course after that it STILL wasn’t working, all fans going flat out.

    Turned out that if you leave the top off the machine, it whacks all the fans upto maximum speed.

    Turned off, put top on, turned on, all fans lovely and quiet and spinning just fast enough to keep things a bit above warm.

    I did have to do a couple of other steps to install freeipmi, I had to go into Ubuntu software update, then go into it’s settings (do not download the updates that it finds), go into the types of packages that it checks, select them ALL apart from ‘Sources’, close that Settings window and it will refresh the list of packages and offer a number of updates. Just close it then, you don’t want to install any updates. Then freeipmi can be installed using apt-get.

    (trying to get the flasher to work in linux I did go through all the binaries, checking them all using ‘ldd’, it found a couple of dependancies that were missing but even after sorting those out it still came up with the “Could not load library” error, not a problem, just use the Windows flasher as described above)

    Many thanks for the script to patch the BIN file, and for all your efforts in making these instructions!

  113. Regarding the temperatures of various bits inside the server, I left it running for a few (3) hours with all the fans going that slow they were barely audible. It was sitting idle doing nothing.

    Powered it down, pulled the power plug, removed the top and measured various temperatures, listed from hottest to coolest:-

    (all temps in degrees C, ambient room temp approx 22 degrees C)

    * Small black heatsink on top of RAID card CPU : 67
    * Both black heatsinks on the motherboard : 65
    * Network controller chips (x 2, back of motherboard, no heatsinks) : 60
    * Memory : 52
    * USB 3.0 daughter card CPU : 55
    * Power supply (top of PSU case off, max temp from measuring all over) : 48
    * 15K SAS drives:-
    * Bottom left : 48
    * Top left : 45
    * Bottom middle : 43
    * Top middle : 41
    * CPUs : 33
    * 7.2K SATA 3TB drives:-
    * Bottom right : 27
    * Top right : 31

    Everything else was 35 or less.

    Normal max temp of things like the network and mobo chipsets are around 85 deg C, and the only thing that’s going to heat up under load is the CPU.

    15K SAS drives seem to have a common max operating temp of 55 deg C which these are close too. As they never power down, the only difference when they’re working hard is the head seeking a lot more and some additional heat from the drives controller, can’t see that making much difference.

    I do note that on the front plate of the server, there are ventilation holes at the top of the front plate that have nothing behind them, these are the easiest route for the fresh air to get in, and I’m going to try blocking off half of those which I hope will force more air to come in via the drive bay holes, therefore cooling the drives better.

    Blocking off too much will just cause everything else to heat up, not good, but it’s easy to experiment with sellotape covering up some of the holes and see what happens.

    (this is a home server and does nothing crucial, plus all the important data is on the SATA drives which are in RAID1, so I’m not particularly worried if I end up replacing all the SAS drives)

  114. Thanks for the write up. It would seem that the Fedora 18 i386 live cd works well for this, and there is no modification of any script required. You’ll need freeipmi and wget, so yum install wget freeipmi.

    I’ve spent a few hours digging through the comments, and I may have missed it, but how does one set the maximum allowed RPM? I see that the script sets the minimum, which is helpful.

    Thanks

  115. Hi Arnuschky,

    Thanks for all your work on this and for posting your findings, scripts, and recommendations. It was a pretty big surprise buying a used T710 for a home virtualization lab to find it was as loud as a jet engine!

    Since you’ve been following/researching this for quite some time (and it’s easy to miss a comment or some of the history), I’m wondering if you wouldn’t mind answering a few questions? That would help me get jump started.

    - Across the various PowerEdge generations you’ve seen, what are typical delivered Fan Failure RPMs configured to? I notice in Vincent’s comment that for 11th generation servers, it was 1680, then 1440. In the latest iDRAC, I now see it as 1080.

    - On the low end, how slow do most of these ‘slow’ fans spin (say, the ones you replaced in your server). I’m assuming they do have some sort of minimum speed that they can effectively operate at.

    - Do you know what minimum voltage/current Dell typically applies to the fans?

    - Do you think there are fans that would natively (with typical Dell control) not drop below 1080, or do you think there is a way to configure a fan (perhaps a ‘smart fan’) to not drop below 1100 RPM? I’ve heard of some folks cutting the PWM wire, but it seems that the ideal situation would be to buy a slow/quiet fan that always runs at 1100 or faster and can go up from there, if needed. Thoughts?

    Thanks in advance!
    Avery

    • Avery — it seems I’m in the same boat as you are with a T710. Have you found anything that works for you yet?

      Justin

  116. Pingback: Dell DSC6005 cloud server | Perkele.net

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>