How to Safely Update IBM X3650 M4 Firmware: A Step-by-Step Guide to Reviving Your Home/Business Data Center

Overview

I recently purchased a few second hand IBM X3650 M4 servers in order to set up a small home data center using the Proxmox operating system for some side projects. All was going well and I had ordered additional hard drives and some other components to begin the project of setting up this small data center.

With one of the servers set up with a full dev ops and project management stack as well as some open source alternatives to common cloud computing web services, I was ready to start tackling some projects.

It was then that a friend reached out to me to inform me that one of the two IBM X3650 servers had crashed due to a critical firmware bug that results in a fried component (VT261WF). Having invested the money and time upfront to get one of these servers operational, the last thing I wanted was to have to order parts and start messing around with soldering them onto the motherboard.

The bug affects a small piece of hardware on the motherboard that forms part of the Integrated Management Module (IMM) and has the potential to blow a small chip due to unregulated voltage bursts when the machine is booting up. Luckily, IBM did release firmware updates that address this bug. So providing your server can still start up and you can get into the BIOS, you can update the firmware to bring your server to a happy and healthy state, hopefully, for years to come.

Although various youtube videos exist on the topic of firmware updates for these machines, I found it difficult to follow along due to a variety of reasons:

The machines were released in 2011, making the documentation quite old.
They were released while IBM and Lenovo were getting their software and products a little muddied, leading to confusion regarding where to find certain resources (documentation and drivers).
Some of the features required by the IMM web interface for remote firmware updates have been deprecated (specifically Java’s Web Start and Microsoft’s ActiveX).
The latest versions of the software/firmware updater and bootable media creator from Lenovo do not include the IBM X3650 M4 models (Type 7915)

The rest of this post includes the exact steps I took to update the firmware on each of my IBM X3650 machines.

How To

The process for updating the firmware can be divided into three main acitivities.

Downloading and creating bootable media
Setting up Web UI access for the IMM
a. BIOS updates
b. Browser and internet configurations
Installing the updates.

Downloading and creating bootable media

The latest version of the Lenovo XClarity Essentials Bootable Medial Creator tool (v13.1.0 – 01k) does not list the IBM X3650 system so an earlier version of the tool is required.

The most recent version that I could find that listed the IBM X3650 can be found here under the heading ‘BoMC 13.0.1’.

Download the tool for the operating system of the machine that you will be using to create the bootable media (ie. not for the operating system you will be or are using for the server). Open the tool and click next at the ‘Welcome’ screen. Select your server type from the IBM platforms list for the ‘Targeted Systems’.

Select only the ‘Update’ radio button for the ‘Media Purpose’ screen. This allows some finer grained control during the install process that we will cover in ‘Installing the updates’. At the ‘Acquire Location’ screen leave the defaults selected as shown below.

Click next on the ‘Network Access’ screen without changing any of the defaults.

The next screen is for ‘Target Directory’. Although we are only interested in the final ISO files, the tool downloads files for each of the updates that exist in the express update pack. I highly recommend creating a new empty directory for the target directory.

On the ‘Media Format’ page we’ll select a device type of ‘CD/DVD’ for this guide as we will mount this as a drive via a KVM session in a later section. Give your ISO file a meaningful name and use whatever directory you would like. We will use X3650M4.iso.

On the ‘Unattended Mode’ screen leave the ‘Do not use unattended mode’ radio box selected and click next.

Confirm your configurations on the ‘Confirm Choices’ screen and click next.

The tool will then fetch the updates from the Lenovo repository. Once complete, you can leave the defaults selected and click next. Do the same for the ‘Progress’ screen. When everything completes, you should have a bootable ISO in the directory you specified with the filename that you specified.

Setting up Web UI access for the IMM

Before starting with the following steps, ensure that the network cable you are using is plugged into the server in the port furthest to the right if you are looking at it from the back. Both the rightmost and leftmost are labelled for IMM but I found, on all of my machines, only the rightmost worked correctly for the IMM web UI.

BIOS updates

Press F1 at the screen below to enter the BIOS of the server.

The first thing we do here is to reload the system defaults.

Once the system defaults have been loaded, go to ‘System Settings’.

In the ‘System Settings’ menu select ‘Integrated Management Module’. You will then see the screen below. First select ‘Reset IMM to Defaults’. This will take a few minutes and will cause the fans to run at 100%. The IMM is responsible for the fan spin rate based on the operating conditions but doesn’t have the data it needs during the reset so the system runs the fans at 100% to be safe. Once the IMM has been reset to defaults, head over to the ‘Network Configration’ screen by selecting ‘Network Configuration’ shown below.

On the ‘Network Configuration’ screen make sure that the NIC Port is set to ‘Dedicated’. Some people have recommended using a static IP here but I have found that the default populated ‘DHCP with Failover’ configurations worked best for me. Ensure that the DHCP network configurations (IP addess, Subnet Mask, and Default Gateway) correspond to the network that the machine you’ll be using to remotely access the IMM console is running on. If it is not then, depending on your network configuration, your browser may not be able to access the web UI.

It is up to you whether you want to disable IP v6. It makes no difference to accesssing the web UI. We will access the web UI through IP v4.

Browser and internet configurations

The machines firmware update mechanism I am outlining is from a not so distant past, a past when Java’s Web Start and Microsoft’s ActiveX were prominant web technologies. Both of these technologies have since been deprecated in favour of newer more secure technologies. The remainder of this guide will walk you through using ActiveX from Microsoft Edge browser in Internet Explorer mode to remotely access the KVM session to mount the update ISO created in step one and to progress through the remainder of the update process.

First open the ‘Internet Options’ from the Windows start menu on the machine that you’ll be using to remotely access the server.

You will be presented with the following window. In this window, click the ‘Custom level…’ button as shown below.

This will open up a menu as show below. Set your ActiveX options to reflect the screenshots below.

The final piece of this puzzle is to add a button in the Microsoft Edge toolbar that allows us to view the current tab in Internet Explorer mode.

Open up the Edge browser and select the three dots in top left corner and select ‘Settings’ from this menu. Then search for ‘explorer mode’.

Enable the ‘Internet Explrer mode (IE mode) button’ under the ‘Select which buttons to show on the toolbar’ section of the results.

We are now ready to access the IMM web UI and do the firmware updates.

Installing the updates.

Vist the IP address from the DHCP configuration settings from the IMM network configuration step we performed earlier. This must use SSL. From the example shown earlier, we would visit https://192.168.1.126. We will then be greeted with the following page. Be sure to click the IE mode button on the top right toolbar to enter IE mode so that we can use ActiveX.

You can ignore this and click on ‘Go on to the webpage (not recommended)’. We will then be greeted by the login page for the IMM console.

The default username and password for the console are:

Username: USERID
Password: PASSW0RD

Note that the ‘o’ in password is replaced with the number zero. Here you can check the server’s current firmware if you want before we remotely connect and install the updates.

Let’s move on to remote access and mounting the ISO. Back on the home page click the ‘Remote Control…’ button.

This will take us to a page where we can configure the remote session using ActiveX or Java Web Start. We’re going to use ActiveX. Just select the ActiveX radio button and start the remote session. You should see a window pop up and you may need to click ‘Allow’ before the connection completes and you can see the screen.

Next, to mount the ISO we need to select ‘Virtual Media’ and ‘Activate’ from the top menu. This will enable the ‘Select Devices to Mount’ option in the same dropdown.

Click ‘Select Devices to Mount’. In the window that pops up, add the ISO that you created in the first step. Enable the ‘Mapped’ option and then click ‘Mount this device’ button.

Once this is done, go back to the BIOS screen and select ‘Exit Setup’. This will cause the machine to reboot. During this reboot you will see various checks and loading screens as the bootable media is read.

Examples:

After all of these, we will be presented with the UI for the firmware updates. It shold look like the screen below. Click the ‘Begin’ button.

After clicking the begin button, the tool will detect the current firmware versions and the updates that are available on the ISO that you have created.

Leave the default selections from the tool which should include the most important one to us, the IMM2 update (usually near the bottom).

This will take some time to complete and once completed, you will need to reboot the server and it will cycle through the boot up process a number of times (depending on the number and nature of the firmware updates your machine required). I typically experienced between 3 and 5 reboots on my servers.

And that’s it! Comparing the before and after firmware versions should look something like the below. Note that the ‘Firmware update successful’ message appears in the IMM web UI after the update has completed but before the reboots have completed.

Before Update

After Update

Conclusion

With that your servers should no longer be at risk of the firmware bug that can fry the hardware component.

Good luck and have fun!