Hi @lindsay.morel and team:
Update, same day: I kept measuring and the cause is not what this post says. These DFPs are not broken — they run 3000 frames straight with one inference in flight. The failure needs concurrent streaming and it is probabilistic. The reproducer, the sweep and the recovery below all still hold; the diagnosis does not. See my follow-up further down.
I have been benchmarking models on an M.2 module (4 chips, Cascade+ at 600 MHz, SDK 2.2.5, driver 1.3.13.1, firmware 2.2 / FW_CommitID=0x196bb59f) and I hit a failure that took me a couple of days to pin down. I am sharing it because the reproducer is small, the behaviour is deterministic, I have swept it far enough to give the exact range, and there is a recovery that is not in the docs.
The symptom
Some DFPs compile cleanly (Assembling DFP (Done), 4 MXAs) and then wedge the board at runtime. They do not fail on load, which took me a while to see: /sys/memx0/hitcount counts the ingress-done interrupt per chip, and it keeps going up while a bad DFP runs. Between 127 and 168 frames get acknowledged by chip 0 before anything goes wrong. So the group gets configured, the model gets downloaded, the stream gets enabled, and more than a hundred frames go through normally. Only after that the data path stops.
At that moment the firmware is still answering. /sys/memx0/temperature gives normal values (56 C / 51 C, not the -274 C of a dead chip), so the admin channel is alive, but every DFP already fails and systemctl restart mxa-manager does not help, so it is not a stale device lock either. A few minutes later the admin channel goes too, and that is when you get this:
Driver reqruied firmware anti_rollback cnt >= 6
Cur firmware: cnt 0 ver 0x0
ioctl MEMX_CONFIG_MPU_GROUP failed
RuntimeError: Init DFP Runner failed!
and dmesg shows this:
memryx: fops_write: wait timeout 1(s), retrying again
memryx: wait_for_fw_ack: failed!
memryx: memx_firmware_init: get hardware info from fw failed
memryx: admin timeout device status 1 subop 4 chip 0..3
Two of those lines are easy to misread, so it is worth saying. memx_firmware_init: get hardware info from fw failed is not printed by the driver probe — the string lives inside memx_get_hw_info, and the MEMX_CONFIG_MPU_GROUP ioctl calls that function too, so here it comes from the ioctl hitting a firmware that is already gone. And fops_write: wait timeout has no retry limit, the loop in memx_cascade_pciemain.c only exits on abort or on a signal, so “212 timeouts” means 212 seconds of waiting and not 212 attempts.
From that point the board is dead for every other DFP, including ones that measured fine seconds before. /dev/memx0 and the kernel module are still loaded, and the PCIe link is still healthy (8 GT/s x2, no fatal errors), so it is the firmware that dies, not the card dropping off the bus.
Minimal reproducer
A chain of N identical 3x3 convolutions, 64 channels, 64x64 input. Nothing else:
# N Conv 3x3, c=64, 64x64 -> ONNX -> mx_nc -c 4 -e hard
for i in range(n):
nodes.append(helper.make_node("Conv", [prev, f"W{i}", f"B{i}"], [f"v{i}"],
kernel_shape=[3,3], pads=[1]*4))
prev = f"v{i}"
Then mx_bench -d fF<N>_hard.dfp -f 300. I swept 35 values of N, with a known-good control run after every single point. Results on my board:
| N | result | N | result |
|---|---|---|---|
| 24 | 1443.79 FPS | 94 | hangs |
| 36 | 961.70 | 95 | hangs |
| 42 | 963.15 | 96 | hangs |
| 46 | 963.14 | 97 | hangs |
| 47 | 953.98 | 98 | hangs |
| 48 | hangs the firmware | 100 | hangs |
| 49 | 552.60 | 102 | hangs |
| 50 | 552.25 | 104 | 291.03 |
| 60 | 553.54 | 108 | 288.59 |
| 72 | 553.00 | 114 | 289.35 |
| 78 | 477.97 | 120 | 290.21 |
| 80 | 479.46 | 130 | 213.50 |
| 82 | 478.22 | 144 | 171.55 |
| 84 | hangs | 145 | 160.80 |
| 90 | hangs | 168 | 95.54 |
| 185 | 117.27 | ||
| 192 | 74.46 | ||
| 216 | 76.77 | ||
| 240 | 70.33 | ||
| 288 | 31.38 |
So the failures are not scattered: there is one isolated value at N=48, and a contiguous band from 84 to 102. Both edges are tight — 47 and 49 run either side of 48, 82 runs and 84 hangs, 102 hangs and 104 runs. And larger is not worse: 288 layers run fine and 84 do not.
It is repeatable
I ran the two sides of each boundary several times, with a known-good control DFP after each run to tell “this DFP does not execute” apart from “the chip was already dead”:
- fF48: killed the firmware 5 out of 5
- fF60: ran 5 out of 5 (553.58, 553.70, 549.37, 553.57, 553.57)
- fF96: killed 3 out of 3
- fF120: ran 3 out of 3 (290.25, 290.20, 290.12)
That control after each run matters. Without it I wrongly flagged several DFPs as broken (fF145, fF185 and a modified yolo11n) that actually run perfectly — they had simply been measured after a hang.
What it is not
- Not thermal. I measured fF48 as the first load after a cold boot, entering at 54 C with
ThermalThrottlingState: 0in/sys/memx0/temperature. It hung with the same ~211 timeouts as when the board was at 67 C. For reference, healthy runs reach 93 C and only then does throttling start, so 54 C is nowhere near a thermal limit. - Not old firmware. Read while the chip is alive,
verinforeportsFirmware version: 2.2,FW_CommitID=0x196bb59f. Thecnt 0 ver 0x0in the error is a consequence of the firmware not answering, not proof that it is old. Worth mentioning because a similar report was answered with a firmware reflash, and here that would not be the fix. - Not size or complexity. The largest DFP I have (57.5 MB) runs fine, and fF48 is 1.9 MB.
- Not the effort setting. fF48 compiled with
-e normalproduces exactly the same mapping, 24 core-groups of (16,16), and hangs the same way. - Not visible in the compiler estimate. fF97 and fF120 both get
estimated fps = 288.3. One kills the board, the other runs at 290.21.
That last one goes the other way too: fF36, fF42, fF46 and fF48 all report the same estimated fps = 991.0, and only fF48 hangs. So I compiled all 35 with -vvv and compared the mappings. Outside the M-Unit and core-group tables the logs are identical line for line between a DFP that hangs and one that runs: no warning, no extra field, same ports, same inter-chip transfer line, same 768 / 768 M-Cores, same 36.31 KByte of WMem per layer. The one thing that does change is how the mapper splits the 32 M-Cores of each core-group:
| N | core-group split | result |
|---|---|---|
| 24 | 24 x (32) | runs |
| 47 | 23 x (16,16) + 1 x (32) | runs |
| 48 | 24 x (16,16) | hangs |
| 72 | 24 x (10,11,11) | runs |
| 82 | 14 x (10,11,11) + 10 x (8,8,8,8) | runs |
| 84 | 12 x (10,11,11) + 12 x (8,8,8,8) | hangs |
| 96 | 24 x (8,8,8,8) | hangs |
| 102 | 6 x (6,6,6,7,7) + 18 x (8,8,8,8) | hangs |
| 104 | 8 x (6,6,6,7,7) + 16 x (8,8,8,8) | runs |
| 108 | 12 x (6,6,6,7,7) + 12 x (8,8,8,8) | runs |
| 192 | 24 x (4,4,4,4,4,4,4,4) | runs |
Every DFP that hangs has core-groups split into exactly equal parts of 16 or 8 M-Cores, and every DFP whose core-groups never get that kind of split runs. That said, it is not sufficient, and this is where I got stuck: fF84 and fF108 have the same twelve core-groups of (8,8,8,8), and one hangs while the other runs at 288.59. I tried four different rules on this table and each of them died on a specific measured point, so I am not going to claim a predictor. That pair is the cleanest bad/good comparison I have.
On scope, this looks like a toy-model problem. The YOLO DFPs I compiled have 0 or 1 core-groups with that kind of exactly equal split, because a real network has layers of many different sizes.
Recovering without a reboot
This is the part I would like to pass on, because it saved me a lot of time. Reloading the kernel module does not bring the board back:
sudo rmmod memx_cascade_plus_pcie && sudo modprobe memx_cascade_plus_pcie # does not work
Removing the device from the bus and rescanning does:
sudo systemctl stop mxa-manager.service
sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:06:00.0/remove"; sleep 3
sudo sh -c "echo 1 > /sys/bus/pci/rescan"; sleep 5
sudo systemctl start mxa-manager.service
After that the board measures 145.43 FPS on my control model, against 146.58 before the hang. So the firmware is reloaded on the PCIe re-probe. Takes about 15 seconds and no reboot, which turned days of power cycles into unattended sweeps .. one warning though, check that the board actually executes a DFP afterwards, because the temperature reading and /dev/memx0 both come back before the firmware is really usable.
One thing to do before that, which I found late. The firmware keeps its own log in a ring buffer in host DRAM, so it can be read even while the board is wedged:
for c in 0 1 2 3; do echo "fwlog $c" | sudo tee /sys/memx0/cmd; done && sudo dmesg
The remove calls memx_fw_log_deinit and clears that buffer, so every recovery destroys the post-mortem. In my case that log has no crash trace at all, only the boot messages, so the firmware never got to write anything down — which may be useful to you on its own.
My question
Is this a known limitation of the allocation for these shapes, or should I file it? And is there anything in the mapping I should be checking that separates fF84 from fF108, since the reported numbers clearly do not. Happy to share the ONNX files, the DFPs and the full -vvv logs for any point in that table.