MX3 not detected after SDK 2.2 upgrade. Deevice missing from lspci (bricked?)

General
1

I upgraded from MemryX SDK 2.1 to 2.2 on Ubuntu 24.04 (kernel 6.17.0-20-generic).

sudo apt upgrade memx-accl memx-drivers mxa-manager

The upgrade process hung - I let it run for 20 minutes - with a progress bar stuck at 55%. Eventually, I had to kill the process and fix the apt install with:

sudo dpkg --configure -a

After the upgrade, and a reboot, Frigate started failing with:

[error] [Client] No devices in system, please check the server
[DFPRunner] Error in client->init_conenction local mode for device: FIXME
Failed to initialize MemryX model: Init DFP Runner failed!

At this point I discovered that 2.2 isn’t recommended for Frigate, which I assumed to be the problem, so I downgraded back to SDK 2.1:

sudo apt install memx-drivers=2.1.1-1.1 memx-accl=2.1.2-1 mxa-manager=2.1.1-1
dpkg -l | grep -E "memx|mxa" | awk '{print $2, $3}'
memx-accl 2.1.2-1
memx-drivers 2.1.1-1.1
mxa-manager 2.1.1-1

I tried restarting the systemd service, and tried reflashing the device:

sudo systemctl restart mxa-manager.service

sudo mxfw_pcie_update_flash -f /lib/firmware/cascade_4chips_flash.bin
[sudo] password for andornaut:

########## Memryx Flash UpdateTool - Directed (V1.6) ###############
****Found 0 MemryX Devices.****

Memryx PCIe EP not found!

Now the memryx device is no longer listed in lspci:

lspci -nn|grep -i -E "memry|1ed9"

Other observations:

  • No /dev/memx0 device

  • No kernel messages related to memryx in dmesg

  • DKMS shows the driver built and installed: memx-drivers/2.1.1, 6.17.0-20-generic, x86_64: installed

I’ve also tried a cold start (poweroff, unplug power, and restart).

The card was working fine on SDK 2.1 before the upgrade. The M.2 module is a PCIe Gen 3 MX3.

I assume the problem is that apt upgrade tried to flash the hardware and got stuck. Any hope at recovery?

Edit: Looks like other users have run into the same. Like this poster, the suggested remediation doesn’t work for me, because the problem is that the memryx device is no longer detected to begin with.

2

Hi @andornaut ,

Yea, if it’s not showing up in lspci, on a system where it used to show up, the flash might have failed. Unfortunately, recovering it in this state requires an external flashing tool over SPI.

Apologies for encountering this issue, and we’d be happy to send a replacement.

I see you’ve already emailed us, so we can work out the details there. Thanks!

3

Update for posterity: I received an excellent response. Appreciate the good customer service.

4

Unfortunately this lines up almost exactly with what I’m seeing too. Same missing-from-lspci / non-enumerating state, and I’ve now reproduced it across four different machines. Really appreciate this thread and the clarification from Tim here.