# MX3 wedges under concurrent streaming with some DFPs — reproducer, workarounds, and a way to recover without rebooting **URL:** \ **Category:** Technical Support\ **Tags:** sdk\ **Created:** [August 1, 2026, 4:33pm UTC](https://community.memryx.com/t/mx3-wedges-under-concurrent-streaming-with-some-dfps-reproducer-workarounds-and-a-way-to-recover-without-rebooting/232 "2026-08-01T16:33:05Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![JESUSROYETH](https://yyz2.discourse-cdn.com/flex008/user_avatar/community.memryx.com/jesusroyeth/32/84_2.png) [@JESUSROYETH](https://community.memryx.com/u/JESUSROYETH)\ **Post date:** [August 1, 2026, 4:33pm UTC](https://community.memryx.com/t/mx3-wedges-under-concurrent-streaming-with-some-dfps-reproducer-workarounds-and-a-way-to-recover-without-rebooting/232/1 "2026-08-01T16:33:05Z")
Hi @lindsay.morel and team: **Update, same day:** I kept measuring and the cause is not what this post says. These DFPs are not broken — they run 3000 frames straight with one inference in flight. The failure needs concurrent streaming and it is probabilistic. The reproducer, the sweep and the recovery below all still hold; the diagnosis does not. See my follow-up further down. I have been benchmarking models on an M.2 module (4 chips, Cascade+ at 600 MHz, SDK 2.2.5, driver 1.3.13.1, firmware 2.2 / `FW_CommitID=0x196bb59f`) and I hit a failure that took me a couple of days to pin down. I am sharing it because the reproducer is small, the behaviour is deterministic, I have swept it far enough to give the exact range, and there is a recovery that is not in the docs. **The symptom** Some DFPs compile cleanly (`Assembling DFP (Done)`, 4 MXAs) and then wedge the board at runtime. They do not fail on load, which took me a while to see: `/sys/memx0/hitcount` counts the ingress-done interrupt per chip, and it keeps going up while a bad DFP runs. Between 127 and 168 frames get acknowledged by chip 0 before anything goes wrong. So the group gets configured, the model gets downloaded, the stream gets enabled, and more than a hundred frames go through normally. Only after that the data path stops. At that moment the firmware is still answering. `/sys/memx0/temperature` gives normal values (56 C / 51 C, not the -274 C of a dead chip), so the admin channel is alive, but every DFP already fails and `systemctl restart mxa-manager` does not help, so it is not a stale device lock either. A few minutes later the admin channel goes too, and that is when you get this: ```auto Driver reqruied firmware anti_rollback cnt >= 6 Cur firmware: cnt 0 ver 0x0 ioctl MEMX_CONFIG_MPU_GROUP failed RuntimeError: Init DFP Runner failed! ``` and dmesg shows this: ```auto memryx: fops_write: wait timeout 1(s), retrying again memryx: wait_for_fw_ack: failed! memryx: memx_firmware_init: get hardware info from fw failed memryx: admin timeout device status 1 subop 4 chip 0..3 ``` Two of those lines are easy to misread, so it is worth saying. `memx_firmware_init: get hardware info from fw failed` is not printed by the driver probe — the string lives inside `memx_get_hw_info`, and the `MEMX_CONFIG_MPU_GROUP` ioctl calls that function too, so here it comes from the ioctl hitting a firmware that is already gone. And `fops_write: wait timeout` has no retry limit, the loop in `memx_cascade_pciemain.c` only exits on abort or on a signal, so “212 timeouts” means 212 seconds of waiting and not 212 attempts. From that point the board is dead for every other DFP, including ones that measured fine seconds before. `/dev/memx0` and the kernel module are still loaded, and the PCIe link is still healthy (8 GT/s x2, no fatal errors), so it is the firmware that dies, not the card dropping off the bus. **Minimal reproducer** A chain of N identical 3x3 convolutions, 64 channels, 64x64 input. Nothing else: ```python # N Conv 3x3, c=64, 64x64 -> ONNX -> mx_nc -c 4 -e hard for i in range(n): nodes.append(helper.make_node("Conv", [prev, f"W{i}", f"B{i}"], [f"v{i}"], kernel_shape=[3,3], pads=[1]*4)) prev = f"v{i}" ``` Then `mx_bench -d fF_hard.dfp -f 300`. I swept 35 values of N, with a known-good control run after every single point. Results on my board: | N | result | N | result | | --- | --- | --- | --- | | 24 | 1443.79 FPS | **94** | **hangs** | | 36 | 961.70 | **95** | **hangs** | | 42 | 963.15 | **96** | **hangs** | | 46 | 963.14 | **97** | **hangs** | | 47 | 953.98 | **98** | **hangs** | | **48** | **hangs the firmware** | **100** | **hangs** | | 49 | 552.60 | **102** | **hangs** | | 50 | 552.25 | 104 | 291.03 | | 60 | 553.54 | 108 | 288.59 | | 72 | 553.00 | 114 | 289.35 | | 78 | 477.97 | 120 | 290.21 | | 80 | 479.46 | 130 | 213.50 | | 82 | 478.22 | 144 | 171.55 | | **84** | **hangs** | 145 | 160.80 | | **90** | **hangs** | 168 | 95.54 | | | | 185 | 117.27 | | | | 192 | 74.46 | | | | 216 | 76.77 | | | | 240 | 70.33 | | | | 288 | 31.38 | So the failures are not scattered: there is one isolated value at N=48, and a contiguous band from 84 to 102. Both edges are tight — 47 and 49 run either side of 48, 82 runs and 84 hangs, 102 hangs and 104 runs. And larger is not worse: 288 layers run fine and 84 do not. **It is repeatable** I ran the two sides of each boundary several times, with a known-good control DFP after each run to tell “this DFP does not execute” apart from “the chip was already dead”: - fF48: killed the firmware **5 out of 5** - fF60: ran **5 out of 5** (553.58, 553.70, 549.37, 553.57, 553.57) - fF96: killed **3 out of 3** - fF120: ran **3 out of 3** (290.25, 290.20, 290.12) That control after each run matters. Without it I wrongly flagged several DFPs as broken (`fF145`, `fF185` and a modified yolo11n) that actually run perfectly — they had simply been measured after a hang. **What it is not** - **Not thermal.** I measured fF48 as the first load after a cold boot, entering at 54 C with `ThermalThrottlingState: 0` in `/sys/memx0/temperature`. It hung with the same ~211 timeouts as when the board was at 67 C. For reference, healthy runs reach 93 C and only then does throttling start, so 54 C is nowhere near a thermal limit. - **Not old firmware.** Read while the chip is alive, `verinfo` reports `Firmware version: 2.2`, `FW_CommitID=0x196bb59f`. The `cnt 0 ver 0x0` in the error is a _consequence_ of the firmware not answering, not proof that it is old. Worth mentioning because a similar report was answered with a firmware reflash, and here that would not be the fix. - **Not size or complexity.** The largest DFP I have (57.5 MB) runs fine, and fF48 is 1.9 MB. - **Not the effort setting.** fF48 compiled with `-e normal` produces exactly the same mapping, 24 core-groups of (16,16), and hangs the same way. - **Not visible in the compiler estimate.** fF97 and fF120 both get `estimated fps = 288.3`. One kills the board, the other runs at 290.21. That last one goes the other way too: fF36, fF42, fF46 and fF48 all report the same `estimated fps = 991.0`, and only fF48 hangs. So I compiled all 35 with `-vvv` and compared the mappings. Outside the M-Unit and core-group tables the logs are identical line for line between a DFP that hangs and one that runs: no warning, no extra field, same ports, same inter-chip transfer line, same `768 / 768` M-Cores, same 36.31 KByte of WMem per layer. The one thing that does change is how the mapper splits the 32 M-Cores of each core-group: | N | core-group split | result | | --- | --- | --- | | 24 | 24 x (32) | runs | | 47 | 23 x (16,16) + 1 x (32) | runs | | **48** | **24 x (16,16)** | **hangs** | | 72 | 24 x (10,11,11) | runs | | 82 | 14 x (10,11,11) + 10 x (8,8,8,8) | runs | | **84** | **12 x (10,11,11) + 12 x (8,8,8,8)** | **hangs** | | **96** | **24 x (8,8,8,8)** | **hangs** | | **102** | **6 x (6,6,6,7,7) + 18 x (8,8,8,8)** | **hangs** | | 104 | 8 x (6,6,6,7,7) + 16 x (8,8,8,8) | runs | | 108 | 12 x (6,6,6,7,7) + 12 x (8,8,8,8) | runs | | 192 | 24 x (4,4,4,4,4,4,4,4) | runs | Every DFP that hangs has core-groups split into exactly equal parts of 16 or 8 M-Cores, and every DFP whose core-groups never get that kind of split runs. That said, it is not sufficient, and this is where I got stuck: **fF84 and fF108 have the same twelve core-groups of (8,8,8,8), and one hangs while the other runs at 288.59**. I tried four different rules on this table and each of them died on a specific measured point, so I am not going to claim a predictor. That pair is the cleanest bad/good comparison I have. On scope, this looks like a toy-model problem. The YOLO DFPs I compiled have 0 or 1 core-groups with that kind of exactly equal split, because a real network has layers of many different sizes. **Recovering without a reboot** This is the part I would like to pass on, because it saved me a lot of time. Reloading the kernel module does **not** bring the board back: ```bash sudo rmmod memx_cascade_plus_pcie && sudo modprobe memx_cascade_plus_pcie # does not work ``` Removing the device from the bus and rescanning does: ```bash sudo systemctl stop mxa-manager.service sudo sh -c "echo 1 > /sys/bus/pci/devices/0000:06:00.0/remove"; sleep 3 sudo sh -c "echo 1 > /sys/bus/pci/rescan"; sleep 5 sudo systemctl start mxa-manager.service ``` After that the board measures 145.43 FPS on my control model, against 146.58 before the hang. So the firmware is reloaded on the PCIe re-probe. Takes about 15 seconds and no reboot, which turned days of power cycles into unattended sweeps .. one warning though, check that the board actually **executes** a DFP afterwards, because the temperature reading and `/dev/memx0` both come back before the firmware is really usable. One thing to do _before_ that, which I found late. The firmware keeps its own log in a ring buffer in host DRAM, so it can be read even while the board is wedged: ```bash for c in 0 1 2 3; do echo "fwlog $c" | sudo tee /sys/memx0/cmd; done && sudo dmesg ``` The remove calls `memx_fw_log_deinit` and clears that buffer, so every recovery destroys the post-mortem. In my case that log has no crash trace at all, only the boot messages, so the firmware never got to write anything down — which may be useful to you on its own. **My question** Is this a known limitation of the allocation for these shapes, or should I file it? And is there anything in the mapping I should be checking that separates fF84 from fF108, since the reported numbers clearly do not. Happy to share the ONNX files, the DFPs and the full `-vvv` logs for any point in that table. ---
**Author:** ![JESUSROYETH](https://yyz2.discourse-cdn.com/flex008/user_avatar/community.memryx.com/jesusroyeth/32/84_2.png) [@JESUSROYETH](https://community.memryx.com/u/JESUSROYETH)\ **Post date:** [August 2, 2026, 1:55am UTC](https://community.memryx.com/t/mx3-wedges-under-concurrent-streaming-with-some-dfps-reproducer-workarounds-and-a-way-to-recover-without-rebooting/232/2 "2026-08-02T01:55:58Z")
Follow-up, after another day of measuring. I had the cause in the wrong place, and the corrected version is much easier to act on. **The DFP is not broken** fF48, the one I flagged as killing the firmware, runs **3000 out of 3000 frames** with a single inference in flight: median 5.137 ms, 194.66 FPS, and my control model measures 141.62 right after. It also passes 20 out of 20 short runs at that depth. The mapping is valid and the maths comes out. What fails is the streaming pattern. **It is a race, with a probability rather than a threshold** Sending K frames before draining any output, 20 runs per point, control after every single attempt: | K | fF48 | fF84 | fF108 | fF50 | | --- | --- | --- | --- | --- | | 1 | **0/20** | **0/20** | — | — | | 2 | 12/20 | — | — | — | | 3 | 19/20 | 4/20 | **0/20** | **0/20** | | 8 | — | — | **1/20** | 0/5 | The important line is fF108: clean over 20 runs at K=3 and then failing 1 in 20 at K=8. So there are no “bad” and “good” DFPs, there is one race whose rate moves with the mapping, the frame size and the depth. Every zero above means “low”, not “impossible” — including my own. Two more things about the shape of it. The point where it wedges moves: with fF48 at K=3 it stalled waiting for output number 83, 5, 1 and 7 on successive tries. And the rate is not stationary — the same configuration gave me 3/20 and 8/20 in two batches an hour apart, which is why I am quoting pooled numbers. My guess is chip temperature, since the later batches ran 15 C warmer, but I did not measure that. **What actually breaks** `/sys/memx0/hitcount` across five different DFPs, each in its own wedge: | DFP | sends HSK(5) | ingress-done MSI(2) | egress MSI(7) | out/in | | --- | --- | --- | --- | --- | | fF48 | +147 | +146 | **+62** | 42% | | fF94 | +155 | +154 | **+40** | 26% | | fF95 | +133 | +132 | **+40** | 30% | | fF97 | +133 | +132 | **+42** | 32% | | fF98 | +134 | +133 | **+42** | 32% | Sends and ingress-done track each other almost exactly, so the input side completes essentially every frame. The egress side stops between 26% and 42%, and in absolute terms it always stops somewhere around 40 to 62 outputs no matter how many went in. The host then sits in `fops_read: wait timeout 10(s)`. Worth correcting one thing from my first post while I am here: the board does not fail when you load the DFP. It processes over a hundred frames first, and right after the wedge the firmware is still answering — `/sys/memx0/temperature` reads normal values, not -274 C. The `admin timeout` lines and `FW_CommitID=0xffffffff` only show up minutes later. So the dmesg block I pasted at the top is the end state, not the failure. **Four things move the rate, and I checked each by removing it** _The mapping._ Every DFP that wedges has core-groups split into exactly equal parts of 16 or 8 M-Cores. fF50 has the same 262.1k transfer on all five hops as fF48 but 22 of 24 evenly split, and it is 0 out of 20 at K=3. _The frame size._ Holding the mapping at 24 x (16,16) and K=3, changing only the resolution: 256 KB → 0.00, 529 KB → 0.00, 576 KB → 0.00, 1 MB → 0.80, 4 MB → 1.00. Changing only the output channels, same input and same mapping: 4 ch → 0.40, 8 → 0.55, 16 → 0.50, 32 → 0.80, 64 → 0.95. A 16x smaller output more than halves the rate but never gets near zero. _The cascade._ The **same ONNX** compiled with `-c 2` instead of `-c 4` goes from 0.80 to 0.00 and stays clean at K=8. fF24 says the same from the other side. _Where the evenly split core-groups sit._ Looking at which of the four MPUs end up entirely made of them, fF84 has MPU2 and MPU3 — the tail, next to the output — and fF108 has MPU0 and MPU1. I wrote that prediction down and then built two models to test it: fF86 and fF106 have the **same fourteen** evenly split core-groups, at the tail and at the head, and they came out 6 out of 10 against 0 out of 10. Two more landed where predicted, so four for four. I owe you two caveats on that last one. The rule as I programmed it predicts exactly the same set of layer counts I had already measured, so those points confirm the range more than the position idea itself, and I could not build a model with them in the _middle_ because the compiler packs by cost and scattered them. And fF48 on 2 chips has both MPUs entirely even including the output one, and never fails — so it does not survive a shorter cascade. **A guess at the mechanism, which you can check and I cannot** An exactly equal split means every M-Unit in that core-group has the same amount of work, so they should finish on the same cycle and write out together. An uneven split — (10,11,11), (6,6,6,7,7) — staggers them for free. If that is what happens, it accounts for all of it: why only even splits matter, why their position matters and the output-side MPU counts, why depth and frame size both raise the rate, why 2 chips is clean, why the egress side is what stops, and why it is probabilistic with the stall landing somewhere different each time. I cannot see from outside whether those M-Units really do complete in lockstep. **Workarounds, all measured** Compiling for 2 chips is the cleanest — 0 out of 5 at K=3 and at K=8, same model. One inference in flight is next, 3000 frames clean, at the cost of an empty pipeline. Shrinking the output or keeping the frame under about 576 KB both help a lot but do not reach zero. None of these is available through the SDK, which is the part I would like to raise. I tried five combinations of `frame_limit`, `ifmap_queue_size` and `ofmap_queue_size` through AsyncAccl, including all three at their minimum together, and every one of them wedged 5 out of 5. `frame_limit` is documented as frames before the DFP is rescheduled, so it is a scheduler time slice and not a depth bound, which matches what I see. So today the only way to bound the number of inferences in flight is doing send/receive by hand with SyncAccl. **On scope, so nobody worries unnecessarily** Real models look fine. My control model, a YOLO26n, ran after every single attempt I made — **443 runs of 300 frames, about 133,000 frames** , all through the normal threaded path — without one failure, and it passes 20 runs with 8 and with 16 frames in flight. Structurally it fits: the YOLO DFPs I compiled have 0 or 1 core-groups with that kind of exactly equal split, because a real network has layers of many different sizes. This looks like a toy-model problem. **What I would ask** Is this a known issue in the streaming path? Is there a supported way to bound the inferences in flight that I am missing? Can you decode the status the daemon returns when it gives up, `0x13000022` — the public header only defines OK and OTHERS, so I cannot tell whether it comes from libmemx, the kdriver or the firmware. And do the M-Units of an evenly split core-group complete in lockstep? fF84 against fF108 is the pair I would look at first: same twelve evenly split core-groups, same 262.1k transfer on all five hops, and different rates. I compared their full `-vvv` dumps by line shape with all numbers normalised, and the only templates unique to either one are cosmetic — progress bars and column widths. The compiler reports nothing that separates them. Happy to send the ONNX files, the DFPs, the mapping sections of the `-vvv` output and the raw per-run data for any point above. ---
**Author:** ![lindsay.morel](https://avatars.discourse-cdn.com/v4/letter/l/59ef9b/32.png) [@lindsay.morel](https://community.memryx.com/u/lindsay.morel)\ **Post date:** [August 17, 2026, 1:19pm UTC](https://community.memryx.com/t/mx3-wedges-under-concurrent-streaming-with-some-dfps-reproducer-workarounds-and-a-way-to-recover-without-rebooting/232/3 "2026-08-17T13:19:52Z")
This query was resolved over direct message, but I wanted to reply here in case someone else faces a similar issue. At a high level, the custom models developed and tested in this case mapped to the MX3 hardware such that they consumed a large amount of power when run at the default x86 frequency of 600 MHz. When the M.2 module’s power consumption surpasses a set limit, the firmware will hang, preventing further usage of the MX3 chips as a safeguard until the host system is rebooted. Reducing the operating frequency of the MX3 was enough to bypass this issue by lowering the overall power consumption of the M.2 module. To adjust the frequency of your MX3 device, simply execute the following command and select the desired frequency for your configuration: ```auto sudo mx_set_powermode ``` The power consumption of the M.2 module varies and is highly dependent on the model architecture and workload being run. In this case, the observed sequence appears to be: 1. The workload causes the module to exceed its power limit. 2. The firmware hangs and egress completions stop. 3. The application continues submitting inputs into the buffered ingress path. 4. Outstanding inputs accumulate because the device is no longer consuming them. 5. The internal input queue eventually fills, resulting in status `0x13000022`. Reducing the module frequency prevents the workload from exceeding the power limit and therefore avoids the firmware hang and the subsequent queue timeout. Hope this helps!