My friend has a TalosII and a blackbird computer. He would like to update both computers' firmware. More recent version of the TalosII's firmware, allow one to navigate a web browser to https://OpenBMCIpAddress. On that web page, you can click a few buttons to update your firmware. Unfortunately, my friend is using an old firmware version that does not offer that feature. So how can we update the firmware?

The recommended method, is to ssh into the BMC (base management controller). With power to the motherboard, but before you press the power on button, you can do something like this:

ssh root@OpenBmcIPaddress
 # lots of magic commands that update the firmware

Well let's give that a try. First, we'll need the OpenBMC's IP address. Interestingly, both computers have multiple ethernet ports, so both machines' have multiple IP addresses. Each machine has one IP address that is exclusively used by the OpenBMC. The easiest way to find that address, is to log into the router, and select Network Settings -> LAN (we are using the turris omni router). All of the Talos II's and the Blackbird's mac addresses start with "2c:09:4d". You then try to ssh into all of those IP addresses. One will eventually ask you for a password. Note that our blackbird's BMC's hostname was "blackbird," while the TalosII's hostname was "talos".

Unfortunately, we do not know the root password (the default password is "0penBmc" not "OpenBmc"). Well apparently there is a sneaky way to reset the root password with a serial port connection.

What is serial port eh? Serial ports are a slightly outdated but extremely reliable way to communicate to a device, particularly during early boot, in this case even before you press the power button. With a serial port, one can watch the talos start uboot, which starts booting the OpenBMC, which eventually starts petitboot, which lets the user select the operating system to boot.

Here's an ASCII art depicting the boot up sequence:

plug in power to the motherboard
     ||
     \/
Uboot starts
     ||
     \/
OpenBMC starts
     ||
     \/
petiboot starts
     ||
     \/
The user selects which operating system to boot.

Required hardware

Ok what hardware does one need to use a serial connection? You will need to hook a serial bracket (be sure that you buy the right serial bracket: the Intel/DTK standard) to the motherboard (consult the talos and blackbird manual for where to plug this in), a null modem, and a serial to usb adapter. Do ask questions in the #talos-workstation on the OFTC network for help, which I had to many times.

If you are like me and have no idea what a serial bracket is?it looks like this:

Setting up the com port

Since I am running OpenBSD, I want to enable my regular user to use a serial connection: doas usermod -G dialer joshua. Then I plugged in the serial cord to the talos, plugged in the serial to usb adapter, and plugged the usb in my laptop.

joshua@dobby.lan:~/ $ sysctl | grep hw.ucom

hw.ucomnames=ucom0:usb5.0.00001.0

Then I needed to read man cu to learn about how to establish the serial connection.

joshua@dobby.lan:~/ $ cu -s 115200 -l usb5.0.00001.0

The cu command is specifying connection details: "115200 8n1", which means that the speed of the connection is 115200 with 8 data bits, 1 stop bit, and flow control off and parity none.

When the OpenBMC has completely booted up, it should give you a login prompt, over the serial connection. And it should say "OpenBMC" somewhere. Mine looked like "Phosphor OpenBMC (Phosphor OpenBMC Project Reference Distro) 0.1.0 talos ttyS4". I did not know the BMC's root password, so I could not log in. I reset the BMC password, which this blog post explains better.

The first thing I did, was to disable the watchdog. Essentially, due to a rarely occurring hardware issue, Uboot tries to start the OpenBMC every 40 seconds. If you fail to run a command in that time or fail to stop the watchdog, then uboot will boot the OpenBMC.

root@talos $ loads 83000000
root@talos $ # Now paste in the this binary: disable-watchdog-binary
root@talos $ go 83000000 # this runs the program

With the watchdog disabled, I can finally reset the root password to "0penBmc".

root@talos:~/ $ printenv

baudrate=115200
bootargs=console=ttyS4,115200n8 root=/dev/ram rw
bootcmd=bootm 20080000
bootdelay=2
eth1addr=2c:09:4d:00:08:a0
ethact=aspeednic#0
ethaddr=2c:09:4d:00:08:a0
spi_dma=yes
stderr=serial
stdin=serial
stdout=serial
verify=yes

I copy the value of bootargs into this next command, which I modify via adding in overlay-filesystem-in-ram. I need to run the setenv command twice. You may need to physically pull the power out of the computer and put it back in so that you can run this command twice.

root@talos:~/ $ setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw
root@talos:~/ $ setenv bootargs console=ttyS4,115200n8 root=/dev/ram overlay-filesystem-in-ram rw
root@talos:~/ $ # watchdog is now disabled! woo!
root@talos:~/ $ # let's reset the default password to 0penBmc
root@talos:~/ $ flash_eraseall /dev/mtd/rwfs
root@talos:~/ $ reboot

Now I was able to ssh root@talosIPAddress, which makes updating the firmware straightforward.

We then were able to log in to the BMC via https://BMCIpAddress. That was pretty cool:

Notice the highlighted gf6295c9c text. Consulting this table, (and lots of help on irc), we confirmed that our Talos II is using firmware version 2.00, which means that our BMC is up to date, but the PNOR could be updated. I even made a short video showing my attempt to update the firmware via the web interface.

Since the web interface didn't work, I then went and updated the PNOR firmware via ssh.

    joshua@dobby.lan:~/ $ wget \
Downloads/https://wiki.raptorcs.com/w/images/6/61/Talos-ii-pnor-v2.10-bundle.tar

Updating the PNOR firmware

joshua@dobby.lan:~/Downloads/ $ tar -xf Talos-ii-pnor-v2.10-bundle.tar
joshua@dobby.lan:~/Downloads/ $ # chmod u+w shell_upgrade
joshua@dobby.lan:~/Downloads/ $ bzip2 -d ./shell_upgrade/talos-ii-v2.10.pnor.bz2
joshua@dobby.lan:~/Downloads/ $ scp -O ./shell_upgrade/talos-ii-v2.10.pnor root@192.168.1.226:/tmp/

root@192.168.1.226:~/ $ pflash -E -p /tmp/talos-ii-v2.10.pnor
root@192.168.1.226:~/ $ pflash -P CVPD -c
root@192.168.1.226:~/ $ rm /tmp/talos-ii-v2.10.pnor

Updating the BMC firmware

For future reference, here is how one updates the BMC firmware:

joshua@dobby.lan:~/Downloads/ $ scp image-{kernel,rofs} root@$TALOS_BMC_ADDR:/run/initramfs/

root@192.168.1.226:~/ $ reboot

Updating FPGA firmware

You have to physically flash this the FPGA. It's not possible to flash this via software.

Or this may help: https://forums.raptorcs.com/index.php/topic,349.msg4232.html#msg4232

https://wiki.raptorcs.com/wiki/Debricking_the_BMC#Reset_persistent_storage

 [Libera] <helpfulBro> regarding the flashing, IIRC
            buspirate was recommended as an open and cheap option
            for flashing the FPGA, but for writing the BMC (and
            PNOR) out of the system you can use any flash programmer
            with the right adapter (eg. TL866) [15:47]
  <gnucode> the adapter is the TL866 ? [15:57]
  <gnucode> I just don't know what to buy to flash the BMC and PNOR
                                                              [16:01]
<ljrBridge> [Libera] <helpfulBro> TL866 is a programmer, see eg.
            https://www.ebay.de/itm/376505717827, they new versions
            like T48, T56, ... [16:13]

https://www.ebay.de/itm/376505717827

On rasberrypi OS, you'll have to enable SPI

$ sudo raspi-config

Select "Interface Options". Then select "Enable/disable automatic loading of SPI kernel module". Click "Yes".

this forum page has a link to the OpenBMC on PNOR flash chip, which should contain the massive data sheet.

I got this command from a google ai. -p ch341a_spi should actually be something like -p linux_spi:dev=/dev/ flashrom -p ch341a_spi -w filename-of-the-firmware.bin -V

This video may be helpful: https://www.youtube.com/watch?v=L35AD0LS70g

Debricking the BMC

If somehow, you have a bricked BMC, that does not let you ssh into it, BUT you do have a working uboot, you can still debrick the BMC!

You definitely want to remember the machine's mac addresses. Here's how you would copy those out.

joshua@192.168.1.220:~/ $ ip link show | grep -A 2 enP4

2: enP4p1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 2c:09:4d:00:08:9e brd ff:ff:ff:ff:ff:ff
    altname enx2c094d00089e
3: enP4p1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 2c:09:4d:00:08:9f brd ff:ff:ff:ff:ff:ff
    altname enx2c094d00089f

:ID:       b120decc-a9f8-48f5-8985-aaaa2a80a0ad