Skip to main content

Debugging

Mirrored from the FEMU repository

This page is hw/femu/docs/guides/debugging.md at FEMU d8b29b1e0 (2026-11-16), licensed GPL-2.0-or-later. Send corrections to the FEMU repository.

Where FEMU reports problems, how to get more output, and how to run it under a debugger. For answers to common problems, see troubleshooting.

Where messages go​

FEMU prints its messages on QEMU's console:

PrefixStreamMeaning
[FEMU] Log:stdoutinformation, such as a vendor command taking effect
[FEMU] Err:stderran error FEMU recovered from, such as a CSD program that failed to load
[FEMU] FTL-Log:, [FEMU] FTL-Err:stdout, stderrthe BlackBox FTL
[Misao] ZFTL-Log:, [Misao] ZFTL-Err:stdout, stderrthe ZNS FTL
[FEMU] FDP-Log:, [FEMU] FDP-Trace:stderrFDP setup; placement and reclaim traces with FEMU_FDP_DEBUG
qemu-system-x86_64: -device femu,...: MESSAGEstderrthe device refused its options and QEMU did not start

run-blackbox.sh, run-zns.sh and run-csd.sh copy the console to build-femu/log, and run-blackbox-fdp.sh to /tmp/femu-fdp.log. run-nossd.sh, run-whitebox.sh and run-cxlssd.sh print only to the terminal. Through the tee pipe, standard output is buffered, so stdout lines can reach the log later than stderr lines. Put stdbuf -oL in front of ./qemu-system-x86_64 on the launcher's sudo line to write them at once.

Check the guest's side too. A guest that gives up on the device logs it in dmesg:

sudo dmesg | grep -i nvme

FEMU defines no QEMU trace events and does not use QEMU's -d log categories, so -d and -trace show nothing from FEMU itself. They still help with the QEMU code around it. -d guest_errors,unimp -D qemu.log logs bad register accesses that QEMU's MSI-X and CXL code detect. Add them to the QEMU command line in the launcher:

-d guest_errors,unimp -D qemu.log

Run FEMU under gdb​

Do not use gdb-run.sh; it starts QEMU's stock nvme device, not FEMU. Instead, run a launcher's own command line under gdb. From build-femu/, make a copy of the launcher that starts QEMU through gdb, and run it:

sed -e 's|\./qemu-system-x86_64|gdb -ex "handle SIGUSR1 nostop noprint pass" --args ./qemu-system-x86_64|' \
-e 's/ 2>&1 | tee .*$//' run-blackbox.sh > gdb-blackbox.sh
bash gdb-blackbox.sh

At the gdb prompt, set breakpoints and start QEMU:

(gdb) break femu_realize
(gdb) run

KVM uses SIGUSR1 to kick vCPU threads, which is why gdb is told to pass it on. The guest's serial console shares the terminal with gdb; press Ctrl-C to get back to the gdb prompt.

To attach to a QEMU that is already running instead, on the host:

sudo gdb -p "$(pgrep -x qemu-system-x86)" -ex "handle SIGUSR1 nostop noprint pass"

When QEMU crashes, thread apply all bt in gdb prints every thread's stack. The launchers pass -name NAME,debug-threads=on, so info threads shows FEMU's thread names (femu-poller, FEMU-FTL-Thread, femu-csd-cu, femu-cxl-ftl, femu-cxl-cca); add it to a QEMU command line of your own (performance tuning).

Debug builds and compile-time switches​

femu-compile.sh builds with -O2 -g. gdb works on that build, but many variables are optimized out. To single-step, configure an unoptimized build yourself from build-femu/:

../configure --enable-kvm --target-list=x86_64-softmmu --enable-slirp \
--disable-libnfs --disable-libiscsi --disable-curl \
--enable-debug --enable-debug-info
make -j"$(nproc)"

The debug messages (Dbg:) have no run-time switch. These macros, passed with --extra-cflags, compile them in:

MacroEffect
FEMU_DEBUG_NVMEcontroller debug messages ([FEMU] Dbg:), and the per-command log that vendor command 0xEF with CDW10 6 and 7 turns on and off
FEMU_DEBUG_FTLBlackBox FTL debug messages ([FEMU] FTL-Dbg:), and the FTL invariant checks of BlackBox and ZNS
FEMU_DEBUG_ZFTLZNS FTL debug messages ([Misao] ZFTL-Dbg:)
FEMU_FTL_ASSERTthe FTL invariant checks only, without the messages

For example:

../configure --enable-kvm --target-list=x86_64-softmmu --enable-slirp \
--disable-libnfs --disable-libiscsi --disable-curl \
--extra-cflags="-DFEMU_DEBUG_FTL -DFEMU_DEBUG_NVME"
make -j"$(nproc)"

The invariant checks (ftl_assert) are compiled out of a normal build. A mapping or bound error there corrupts FTL state quietly instead of stopping QEMU. When a BlackBox or ZNS run produces wrong data or impossible counters, rebuild with -DFEMU_FTL_ASSERT first. The debug messages print on the I/O path, so use them with small workloads.

The sanitizer build adds AddressSanitizer and UndefinedBehaviorSanitizer on top of the FTL checks. Run the failing workload on it to find memory errors.

Run-time switches​

FEMU reads a few environment variables (environment variables):

VariablePrints to stderr
FEMU_FDP_DEBUG (any value)FDP placement and reclaim traces
FEMU_EXP_LOG=1 with FEMU_SECRET=TEXTwrites, overwrites, deallocations, GC moves and erases of pages whose data contains TEXT
FEMU_DUMP_LPN=Na hex dump of logical page N on every read not served from the write buffer
FEMU_KV_SELFTEST (any value)the result of a KV FTL self-test run once at start
FEMU_KV_CHECK (any value)stops QEMU with a message when a KV page owner check fails

The launchers other than run-cxlssd.sh start QEMU with sudo, which drops your environment. run-blackbox.sh passes FEMU_EXP_LOG, FEMU_SECRET and FEMU_DUMP_LPN through, so this works:

FEMU_EXP_LOG=1 FEMU_SECRET=MARKER ./run-blackbox.sh

For the others, add the variable to the launcher's sudo line, for example sudo FEMU_FDP_DEBUG=1 ./qemu-system-x86_64 \ in run-blackbox-fdp.sh.

On a running BlackBox controller, the vendor admin command 0xEF switches GC time and NAND time on and off and prints poller counts (timing model).

Check a configuration without a guest​

hw/femu/scripts/femu-config-check.py tells you, in about a second, whether FEMU accepts a set of device options, and what it builds from them. It needs no guest image and no KVM.

options --> qemu-system-x86_64 -accel qtest (no guest)
realize fails? --> FEMU's message exit 1
realize works --> enable each controller (qtest)
Identify, query-femu
--> derived configuration exit 0

Give the options as on the QEMU command line, or the value of one -device option. From build-femu/:

../femu-scripts/femu-config-check.py -device femu,femu_mode=1,devsz_mb=4096
femu-config-check: accepted
device /machine/peripheral-anon/device[0] bbssd PCI 00:02.0
controller: FEMU BlackBox-SSD Controller, enabled with CC 0x460061 (4096 B memory page, command sets all), admin queues of 64 entries
ns 1 bbssd, NVM command set: 4.0GiB (8388608 blocks of 512 B)
NAND: 8 channels x 8 LUNs x 1 planes x 256 blocks x 256 pages x 4096 B = 16.0GiB
over-provisioning: 300.0% (NAND / capacity - 1)
lines: 256 of 64.0MiB, free 255, spare 0
media: bus phases not charged

A configuration that FEMU refuses gives FEMU's message and exit status 1. The example uses a small geometry, because FEMU allocates the memory backend before it checks the size:

../femu-scripts/femu-config-check.py -device femu,femu_mode=1,devsz_mb=80,nchs=2,luns_per_ch=2,blks_per_pl=64,pgs_per_blk=64
femu-config-check: REFUSED
qemu-system-x86_64: -device femu,femu_mode=1,devsz_mb=80,nchs=2,luns_per_ch=2,blks_per_pl=64,pgs_per_blk=64: FEMU bbssd: namespace 1 exposes 80 MiB of the 64 MiB this geometry has, leaving garbage collection no room; expose at most 60 MiB per namespace (lower devsz_mb, or set op_pcent)

FDP needs its femu-subsys, and ZNS takes the bare -device value:

../femu-scripts/femu-config-check.py -device femu-subsys,id=fdp0,fdp=on,fdp.nruh=4 -device femu,femu_mode=1,devsz_mb=4096,subsys=fdp0
../femu-scripts/femu-config-check.py femu,femu_mode=3,devsz_mb=4096
femu-config-check: accepted
device /machine/peripheral-anon/device[0] znssd PCI 00:02.0
controller: FEMU ZMS-SSD Controller [by Misao], enabled with CC 0x460061 (4096 B memory page, command sets all), admin queues of 64 entries
ns 1 znssd, zoned command set: 4.0GiB (8388608 blocks of 512 B)
zones: 16 of 256.0MiB, capacity 256.0MiB, max open 0, max active 0
media: bus phases not charged

The report shows, for each namespace:

ItemSourceModes
mode, command set, capacity, block sizeIdentifyall
NAND geometry, NAND size, over-provisioning, line size, line countsquery-femu and Identifybbssd
reclaim groups, units, unit size, handlesquery-femubbssd with FDP
zone count, zone size and capacity, open and active limitsquery-femuznssd
NAND bus timingquery-femuall except nossd
warnings that QEMU printedstderrall

Over-provisioning is NAND size / capacity - 1, per namespace. Each bbssd namespace has its own FTL and NAND.

How the check runs:

  1. It drops the options that need a host resource or a monitor, for example KVM, -cpu, -qmp, -serial and -kernel.
  2. It replaces each -drive by a null block device and each -netdev by user networking. The report lists each change.
  3. It keeps -machine, -m, -smp, -numa, -object and -global. Without -machine, it uses pc.
  4. It enables each controller as a guest driver does. It uses the smallest memory page the controller takes, and all command sets.

The configuration it reports is the one after the enable. A controller behind a PCI bridge (a root port, a CXL port) is not enabled. Then the check reports what it can and exits with status 2.

The exit status is one of these:

  • 0: QEMU accepts the configuration and the controllers come ready.
  • 1: QEMU or a controller refuses the configuration.
  • 2: the check itself fails. A QEMU that dies of a signal is a check failure.

Options: --json prints one JSON object, --qemu BINARY names QEMU (default ./qemu-system-x86_64, then build-femu/ in the source tree), and --timeout SEC (default 60). Put -- before QEMU options when you give tool options.

Record an experiment manifest​

--manifest FILE also writes one JSON file for each run. Keep it with the results, so a reader can find the exact device behind each number.

manifest.json
exit, created (UTC)
host: kernel, CPU model, CPUs, memory, NUMA nodes, cpufreq scaling
build: QEMU path, sha256, version
tool-commit, tool-dirty: the checkout of this script, which
need not be the source of the QEMU binary
report: the --json report; each femu device also lists
properties: name, type, value, default, source

The source of each property value comes from the live device:

SourceRule
giventhe -device option names the property; given holds its text
defaultthe value equals the class default, or the property has no default and is empty
derivedneither of the above: realize, -global or another device set it
unknownthe check could not match the device to its option

How the check matches devices and options:

  • A device with id= matches its option by id. The first id= counts, as in QEMU.
  • The other femu devices match their options in creation order: the key=value options first, then the JSON ones. When the counts differ, the sources are unknown and the report notes why.
  • Options follow the QEMU rules: ,, is a comma, a bare flag is on and noflag is off. Property names stay as given.

The manifest leaves out x- test properties, child and link properties, and the properties of femu-subsys and CXL devices. A failed property read keeps its error, and the records read before it stay.

Inspect a running device​

QEMU's monitor shows what QEMU built. With the QMP socket the launchers other than run-cxlssd.sh create (build-femu/qmp-sock), on the host:

printf '%s\n' '{"execute":"qmp_capabilities"}' '{"execute":"query-pci"}' |
sudo socat - UNIX-CONNECT:./qmp-sock

qom-list and qom-get on /machine/peripheral/<id> read a device's properties and counters (runtime properties). run-nossd.sh names its controller nvme0 and run-cxlssd.sh names its device cxlssd. The other launchers give the controller no id=, so add one to address it by name.

Common crash reports​

SymptomWhat it meansWhat to do
qemu-system-x86_64: -device femu,...: MESSAGE and QEMU exitsA property value was refused at startRead the message; the mode guide's "Limits and refusals" table explains it
[FEMU] Err: cannot pin the N MiB memory backendNot a crash. QEMU runs without sudo and the memory lock limit is too lowIgnore it, or raise ulimit -l (requirements)
failed to allocate N bytes, then QEMU stops with a trap or abort and may dump coreThe host has less free memory than the device needsLower devsz_mb or free memory (host sizing)
Guest dmesg shows nvme nvme0: I/O ... timeout, then a controller resetThe guest's NVMe driver waited too long for a completion and reset the controllerCheck the QEMU console for an error at the same time. If there is none, check that the pollers have cores (performance tuning). If it repeats on an idle host, report it
The device is missing in the guest, and the console shows [FEMU] Err: linesFEMU refused a command or a configuration the guest usedRead the console; for ZNS, see the ZNS troubleshooting
QEMU exits with a segmentation fault or an assertionA FEMU bugRun it under gdb or the sanitizer build, and report it with the steps below

Older reports of crashes in FDP garbage collection (#186, #189, #191) and of controllers disabled under load (#22, #30) were made against older FEMU versions. Reproduce on current master before you report one.

Reporting a bug​

Open an issue with:

  • the FEMU commit (git log -1 --oneline) and how you built it;
  • the full QEMU command line, or the launcher and every change you made to it;
  • the guest kernel (uname -r) and the commands you ran in the guest;
  • the QEMU console output and the guest's dmesg;
  • for a crash, the thread apply all bt output from gdb, or the sanitizer report.

Report a security problem privately, as SECURITY.md says.

Related issues: #18, #22, #30, #105; discussion #133.