Tutorial 07: key-value SSD
This page is hw/femu/docs/tutorials/07-kv.md at FEMU 39a55eeb6 (2026-10-02), licensed GPL-2.0-or-later. Send corrections to the FEMU repository.
You start an SSD with the NVMe Key Value command set, then store, retrieve, check, list and delete values with nvme-cli's passthrough command, and read the status codes the device returns. It takes about ten minutes.
You need: the variables from Before you start,
about 5 GiB of free host memory, and a guest kernel of Linux 6.0 or newer.
Older kernels skip a namespace whose command set they do not know, so the
commands below fail. The Ubuntu 24.04 image from make-guest-image.sh
runs Linux 6.8.
Background
A KV namespace stores values under keys instead of blocks at addresses.
Linux has no driver for the key-value command set, so the namespace gets no
block device, only the generic node /dev/ngXnY, and you send each command
as a passthrough command. The fields of a command:
| Field | Holds |
|---|---|
| opcode | 0x01 Store, 0x02 Retrieve, 0x06 List, 0x10 Delete, 0x14 Exist |
| CDW2, CDW3 | bytes 0 to 7 of the key, little-endian |
| CDW14, CDW15 | bytes 8 to 15 of the key |
| CDW11 bits 7:0 | key length, 1 to 16 bytes |
| CDW11 bits 15:8 | Store only: bit 8 stores only if the key exists, bit 9 only if it does not |
| CDW10 | the value size for Store, the buffer size for Retrieve and List |
| NSID | the namespace; it must be given |
Values are up to 2 MiB and live on emulated NAND with the BlackBox geometry and timing (the KV design page).
1. Start the guest (on the host)
./qemu-system-x86_64 -name femu-tut07,debug-threads=on \
-enable-kvm -cpu host -smp 4 -m 4G \
-device virtio-scsi-pci,id=scsi0 -device scsi-hd,drive=hd0 \
-drive file=$OSIMGF,if=none,cache=none,format=qcow2,id=hd0 \
-net user,hostfwd=tcp::$SSH_PORT-:22 -net nic,model=virtio \
-device femu,femu_mode=5,devsz_mb=1024 \
-nographic
femu_mode=5 is KV; the namespace is 1 GiB.
2. Find the namespace
In the guest (./run-guest-ssh.sh):
sudo nvme list
ls /dev/ng0n1 /dev/nvme0n1
nvme0n1 /dev/ng0n1 vKVSSD0 FEMU KV-SSD Controller 0x1 1.07 GB / 0.00 B 512 B + 0 B 1.0
/dev/ng0n1
ls: cannot access '/dev/nvme0n1': No such file or directory
There is a generic node and no block device. On a controller with one
namespace you may also send passthrough commands to the controller node
/dev/nvme0, which this tutorial does. With several namespaces use
/dev/ngXnY (tutorial 06).
3. Store and retrieve
Store a 64-byte value under the 4-byte key BBBB (0x42424242), then read
it back:
head -c 64 /dev/urandom > value.bin
sudo nvme io-passthru /dev/nvme0 -O 0x01 -n 1 --cdw10=64 --cdw11=4 \
--cdw2=0x42424242 -l 64 -w -i value.bin
sudo nvme io-passthru /dev/nvme0 -O 0x02 -n 1 --cdw10=64 --cdw11=4 \
--cdw2=0x42424242 -l 64 -r -b > out.bin
cmp value.bin out.bin && echo match
IO Command Write is Success and result: 0x00000040
IO Command Read is Success and result: 0x00000040
match
nvme-cli names opcodes 0x01 and 0x02 after the NVM command set's Write and
Read; the device treats them as Store and Retrieve. The result (completion
Dword 0) of a Retrieve is the full size of the value, 0x40. -n 1 is
required: nvme-cli sends namespace 0 by default, which Linux refuses with
Invalid argument.
A buffer smaller than the value gets the first bytes, and the result still reports the full size, so the host can retry with a larger buffer:
sudo nvme io-passthru /dev/nvme0 -O 0x02 -n 1 --cdw10=16 --cdw11=4 \
--cdw2=0x42424242 -l 16 -r -b | od -An -tx1
The command prints result: 0x00000040 and 16 bytes of the value.
4. Conditional stores, Exist and List
Check that the key exists, then try to store it again with bit 9 set
("only if the key does not exist"), which CDW11 0x204 encodes with the
key length 4:
sudo nvme io-passthru /dev/nvme0 -O 0x14 -n 1 --cdw11=4 --cdw2=0x42424242
printf hello > v2.bin
sudo nvme io-passthru /dev/nvme0 -O 0x01 -n 1 --cdw10=5 --cdw11=0x204 \
--cdw2=0x42424242 -l 5 -w -i v2.bin
IO Command Vendor Specific is Success and result: 0x00000000
NVMe status: unrecognized(0x4089)
nvme-cli does not know the key-value status codes. 0x4089 is status 0x89,
Key Exists, with the Do Not Retry bit (0x4000). Store a second key, CCCC,
and list the keys. List walks the device's key index from its first slot,
so keys come back in index order, not sorted:
sudo nvme io-passthru /dev/nvme0 -O 0x01 -n 1 --cdw10=5 --cdw11=4 \
--cdw2=0x43434343 -l 5 -w -i v2.bin
sudo nvme io-passthru /dev/nvme0 -O 0x06 -n 1 --cdw10=4096 --cdw11=0 \
-l 4096 -r -b | od -An -tx1 -N 24
02 00 00 00 04 00 43 43 43 43 00 00 04 00 42 42
42 42 00 00 00 00 00 00
The buffer starts with a 4-byte key count (2), then one entry per key: a 2-byte key length (4), the key, and padding to a multiple of 4 bytes.
5. Delete
sudo nvme io-passthru /dev/nvme0 -O 0x10 -n 1 --cdw11=4 --cdw2=0x42424242
sudo nvme io-passthru /dev/nvme0 -O 0x14 -n 1 --cdw11=4 --cdw2=0x42424242
IO Command Vendor Specific is Success and result: 0x00000000
NVMe status: unrecognized(0x4087)
0x87 is Key Does Not Exist. Deleting a key that does not exist succeeds unless the host sets the EDNEK bit of the Key Value Configuration feature (20h).
| Status | Meaning |
|---|---|
| 0x85 | Invalid Value Size: the value is larger than 2 MiB |
| 0x86 | Invalid Key Size: a key length of 0 on Store, Retrieve, Delete or Exist |
| 0x87 | Key Does Not Exist |
| 0x89 | Key Exists, from a Store with bit 9 set |
6. Counters
The KV namespace adds to log page C0h like a BlackBox one:
sudo nvme get-log /dev/nvme0 --log-id=0xc0 --log-len=512 -b | od -An -t u8 -j 8 -N 24 -w24
2 0 2
Two Stores of small values, two pages programmed. The key index lives in device DRAM, but each command is charged a base cost of one NAND page read time, which overlaps the media time of its value (KV timing).
What you learned
- A KV namespace has a generic node and no block device; Linux 6.0 or newer is needed to see it.
nvme io-passthruwith the opcode, the key in CDW2 and CDW3 (and CDW14 and CDW15), the key length in CDW11 and an explicit namespace drives every command.- nvme-cli prints KV status codes as
unrecognized; read the low byte.
Next
- KV mode has
kv-probe.c, a C program that runs the whole command cycle and checks the results. - The KV design page explains the key index and the value store.