Computational storage (CSD)
This page is hw/femu/docs/modes/csd.md at FEMU d8b29b1e0 (2026-11-16), licensed GPL-2.0-or-later. Send corrections to the FEMU repository.
CSD mode (femu_mode=4) emulates a computational storage drive. On top of a
normal NVMe namespace, the device has its own memory (function data memory,
FDM) and compute units that run programs next to the data. The guest
allocates device memory, copies namespace data into it, loads and runs a
program, and reads the result back, all with NVMe commands. Ordinary reads
and writes go through the BlackBox FTL, so they have SSD
timing.
The mode is a port of CEMU; if you use
it, cite CEMU as well as FEMU (citation). It does not need
CEMU's modified guest kernel, FDMFS or a fixed VM image. CEMU's VM freezing
and virtual clock changes are left out of this port. The opt-in
csd_pause does that job on an
unmodified QEMU and guest.
Use it to study offloading filters, scans, compression or other kernels to the drive, and what that does to latency and host CPU use.
Requirements
- Host and guest: see the mode table and requirements.md. Any guest kernel with the NVMe driver works.
- Guest tools:
hw/femu/tests/csd, built in the guest with a C compiler. The image frommake-guest-image.shhas none; add one with./make-guest-image.sh --packages build-essential, or runsudo apt install build-essentialin the guest. - Programs: shared-library programs are built on the host and placed in the
directory named by
csd_program_dir. eBPF programs also need a FEMU build with uBPF (./femu-compile.sh --enable-csd-ubpf, see build.md) and clang with the BPF target on the host.
Security: programs run inside QEMU on the host
A program load names a file. QEMU, on the host, opens that file and runs
code from it inside the QEMU process. The run-*.sh launchers start QEMU
with sudo, so that code runs as root on the host.
FEMU limits what the guest can name:
- With
csd_program_dirunset, only the built-in phantom program type loads. Shared-library and eBPF loads fail. - With it set, the guest may name only a file in that directory: no
/, no.., and the path must still resolve inside the directory after symbolic links are followed.
That protects the rest of the host file system. It does not make the programs in the directory safe: they run on QEMU's compute unit threads with all of QEMU's privileges and memory. Use CSD only with guests you trust, put only programs you built and trust in the directory, and make sure no one else can write to it.
Launch
From build-femu/:
./run-csd.sh
The FEMU device in that script is:
-device femu,devsz_mb=4096,namespaces=1,femu_mode=4,secsz=512,secs_per_pg=8,pgs_per_blk=256,blks_per_pl=256,pls_per_lun=1,luns_per_ch=8,nchs=8,pg_rd_lat=40000,pg_wr_lat=200000,blk_er_lat=2000000,ch_xfer_lat=0,gc_thres_pcent=75,gc_thres_pcent_high=95,fdm_size=64,nr_cu=4,nr_thread=4,time_slice=200000,context_switch_time=200,csf_runtime_scale=3
The preset hw/femu/scripts/configs/csd.conf describes the same device
without the three scheduler properties that have no effect.
ssd-config.sh expands it to:
-device femu,id=nvme0,devsz_mb=4096,namespaces=1,secsz=512,secs_per_pg=8,pgs_per_blk=256,blks_per_pl=256,pls_per_lun=1,luns_per_ch=8,nchs=8,pg_rd_lat=40000,pg_wr_lat=200000,blk_er_lat=2000000,gc_thres_pcent=75,gc_thres_pcent_high=95,fdm_size=64,nr_cu=4,csf_runtime_scale=3,femu_mode=4
run-csd.sh does not set csd_program_dir, so it runs phantom programs
only. To load your own programs, add the directory after the other
FEMU_OPTIONS lines in run-csd.sh:
FEMU_OPTIONS=${FEMU_OPTIONS}",csd_program_dir=$HOME/csd-programs"
The shortest device line with programs enabled is:
-device femu,devsz_mb=4096,femu_mode=4,fdm_size=64,csd_program_dir=/home/you/csd-programs
FEMU does not check the directory at start-up. A directory that does not exist makes every program load fail.
Configuration
Properties: CSD. The BlackBox geometry, timing, GC and FTL properties also apply to the namespace; see the BlackBox guide.
fdm_size: device memory in MiB. Required.nr_cu: compute units, 1 to 64. Each is a host thread,femu-csd-cu, that runs programs. A program waits for the first free unit.csf_runtime_scale: a program that declares no run time and no scale of its own holds its unit for its measured host run time times this value (default 3). A completion carries the program's result, so it never arrives before the host has run the program. Withoutcsd_pause, a run time, declared or scaled, shorter than the host's own is not reached, and QEMU warns once when that happens.csd_pause: stop the guest while the host runs a program, so the guest sees only the modelled run time (default off). See below.csd_pause_max_ms: withcsd_pause, the longest stop, 1 to 60000 ms (default 1000).csd_program_dir: the host directory programs load from.nr_thread,time_sliceandcontext_switch_timeare accepted so CEMU configurations still start. They have no effect, and a value other than the default prints a warning at realize.
A copy from the namespace into device memory costs one pg_rd_lat,
whatever its size, when any page of the range has been written, and nothing
otherwise. See the
timing model.
Program types
| Type | What runs | Needs |
|---|---|---|
| Phantom | Built in: copies the input memory range to the output range. Useful to test the command flow and timing. | nothing |
| Shared library | A function with the signature int64_t fn(struct femu_csd_args *args) from a .so file, named in the load command. | csd_program_dir |
| uBPF | An eBPF ELF object, interpreted or JIT-compiled. | csd_program_dir and a build with --enable-csd-ubpf |
How programs run
An Execute command is checked on the poller and then handed to one of the
nr_cu compute unit threads, which runs the program and posts the
completion. While a program runs, I/O on every queue, other CSD commands
and admin commands go on as usual; only that compute unit is busy.
- At most
nr_cuprograms run at once. Runs of one program take turns, so a shared library needs no locking of its own; different programs run side by side, and a run waiting for its own program leaves the compute unit free for another. - A program that never returns keeps its compute unit for good, and QEMU waits for it when the device is removed. FEMU cannot stop native code it has called.
- Device memory a program is using stays allocated, and counted against
fdm_size, until the program returns, even after Free device memory. - Deleting the I/O queue an Execute came from, or resetting the controller, while the program runs drops its result; the program still runs to the end.
Hide the host run time (csd_pause)
A real CSD runs a program on its own cores while the host goes on. FEMU runs it on a host thread, and the completion carries the result. So without help the guest waits for the host run, also when the model says the device is faster.
With csd_pause=on, FEMU stops the guest while a program runs, and hides
that time from the guest clocks. The device clock of the controller stops
with them. The guest then sees the modelled run time.
host time |--Execute--|======= host runs program =======|--CQE--|
guest time |--Execute--| |--CQE--|
<- guest stopped, clocks hidden ->
guest sees Execute .......... modelled run time .......... CQE
- Under KVM, FEMU stops the vCPUs, sets the kvmclock back and moves the TSC of each vCPU back by the same span. Guest readers of the kvmclock, the vDSO and the TSC all skip the stop.
- Overlapped runs on several compute units share one stop.
- All I/O of the controller runs on the device clock, so other commands keep their modelled latency.
- A stop has a cost: the guest runs for a short time before the vCPUs stop. On a 20-CPU host with 4 vCPUs, the guest saw the modelled run time plus 30 to 50 us at the median.
- Time outside the guest does not stop. Network peers, the wall clock and other devices, also other FEMU controllers, move on during a stop.
- A vCPU that stays halted through many stops keeps an old pvclock page. The page can stray from the clock by about 1 ns a stop until that vCPU runs again.
- A LAPIC timer due during a stop fires once, early. Linux with high resolution timers sets it again, so a sleep does not end early.
- A program that runs past
csd_pause_max_mslets the guest run again, and the guest sees the rest of the host run. QEMU warns once. - A VM stop, suspend, reset or hot plug during a program makes that one stop visible to the guest.
- If KVM fails a clock call after the kvmclock is set, the vCPUs would run on clocks that disagree. QEMU then stops the VM with an internal error.
It needs KVM on x86 with Linux 6.0 or later on the host. The guest TSC must
run at the host rate, and realize refuses a tsc-frequency that changes
it. Every namespace of the controller must be CSD or bbssd, and the
controller must not share namespaces. The script
hw/femu/scripts/csd-pause-check/run.sh boots a guest with and without the
option, and prints the latency each sees.
-device femu,devsz_mb=4096,femu_mode=4,fdm_size=64,csd_program_dir=/home/you/csd-programs,csd_pause=on
Commands
I/O commands on the namespace. The opcodes are in the vendor specific range 80h to FFh. Bits 1:0 give the data direction (Base 2.3 Figure 91), and a command that moves no data may carry any value there:
| Opcode | Command | Data |
|---|---|---|
| 0xb0 | Allocate device memory | none |
| 0xc0 | Free device memory | none |
| 0xd0 | Copy namespace data into device memory | none |
| 0xe1 | Execute a program | host to device |
| 0xf2 | Read device memory | device to host |
| 0xf5 | Write device memory | host to device |
| 0xf6, 0xf7, 0xf8 | Create a group, set its QoS, delete it | none |
Admin commands, in the vendor specific range C0h to FFh. As a mnemonic, 0xc5, 0xc8 and 0xc9 are the CP 1.2 opcodes 0x85, 0x88 and 0x89 of the matching commands plus 40h. CP has no Load Data command, so 0xcd has no CP pair:
| Opcode | Command | Data |
|---|---|---|
| 0xc5 | Program load and unload | host to device |
| 0xc8 | Program activate and deactivate | none |
| 0xc9 | Memory range set management | host to device |
| 0xcd | Load more program data | host to device |
The command layouts follow CEMU; hw/femu/tests/csd/csd-passthru.c builds
each one.
Migration from the CEMU opcodes
CEMU and FEMU before this change used admin opcodes 0x21, 0x22, 0x23 and 0x25. NVMe Base 2.3 gives 0x21, 0x22 and 0x25 to fabrics commands, which a PCIe controller must not support, and reserves 0x23. 0x22 also declares data to the host, but it sends program bytes to the device. The Linux driver reads that bit to map the data buffer. The I/O opcodes do not change.
old new command
0x21 0xc9 memory range set management
0x22 0xc5 program load and unload
0x23 0xc8 program activate and deactivate
0x25 0xcd load more program data
The device now answers the old opcodes with Invalid Command Opcode. To run
an old tool unchanged, set csd_legacy_opcodes=on. The device then takes
the old opcodes and refuses the new ones. The Commands Supported and
Effects log (05h) lists the layout in use. The passthrough helper takes
CSD_LEGACY_OPCODES=1 in its environment to send the old opcodes.
NVMe CP and SLM command sets
csd_slm_size adds a memory namespace (SLM 1.2) and csd_cp=on a compute
namespace (CP 1.2) after the CSD namespace. They take the ratified opcodes,
and their programs, range sets and memory are separate from the CEMU ones
above. The CEMU commands stay on, on the CSD namespace. See
CP and SLM.
CSD namespace (CSI 00h) NVM commands + CEMU b0h..f8h, admin c5h..cdh
memory ns (CSI 03h) Memory Copy 01h, Read 02h, Fill 04h, Write 05h
compute ns (CSI 04h) Execute Program 01h, admin 85h, 88h, 89h
Use it from the guest
Check the device:
sudo nvme list
sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|sn) '
The model is FEMU Computational Storage Controller and the serial number
starts with vCSD.
Build the guest tool
From build-femu/ on the host, copy the tools into the guest and build the
passthrough helper:
scp -P 8080 -i ~/images/femu-guest-key -r ../hw/femu/tests/csd femu@localhost:
./run-guest-ssh.sh make -C csd csd-passthru
Run the phantom smoke test
This needs no program directory. It allocates, writes and reads device memory, loads and runs a phantom program, and frees everything:
./run-guest-ssh.sh sudo ./csd/csd-passthru /dev/nvme0n1 smoke
Run a shared-library program
On the host, build the example programs and put them in the program
directory (csd-original-kernels.so needs the lz4 development package):
cd hw/femu/tests/csd # in the FEMU source tree
make csd-vadd.so csd-original-kernels.so
mkdir -p ~/csd-programs
cp csd-vadd.so csd-original-kernels.so ~/csd-programs/
Start FEMU with csd_program_dir set to that directory, then name the file
alone in the guest:
sudo ./csd/csd-passthru /dev/nvme0n1 smoke-so csd-vadd.so
sudo ./csd/csd-passthru /dev/nvme0n1 smoke-so-all csd-original-kernels.so
sudo ./csd/csd-passthru /dev/nvme0n1 bench 4096 32
bench reports the average latency of device memory writes, reads and
namespace-to-memory copies. hw/femu/tests/csd/README.md
lists every subcommand, the eBPF steps and the program ABI.
The vendor log page C0h counts the namespace's NAND traffic as for BlackBox (log pages and counters).
Limits and refusals
| Message | Cause and fix |
|---|---|
CSD mode requires fdm_size to be non-zero | Set fdm_size. |
CSD nr_cu must be in range [1, 64] | nr_cu out of range. |
CSD nr_thread must be non-zero | nr_thread=0. |
CSD csf_runtime_scale must be non-zero | csf_runtime_scale=0. |
CSD csd_pause_max_ms must be in range [1, 60000] | csd_pause_max_ms out of range. |
csd_pause needs a CSD namespace | csd_pause on a controller with no CSD namespace. |
csd_pause: namespace N runs a mode other than bbssd or CSD | Other modes keep their own clocks; use them on another controller. |
csd_pause cannot be used with shared namespaces: ... | Two controllers would time one FTL on two clocks. |
csd_pause needs KVM | Another accelerator; csd_pause needs -enable-kvm. |
csd_pause needs KVM_CLOCK_HOST_TSC and KVM_CAP_VM_TSC_CONTROL ... | The host kernel is older than Linux 6.0. |
csd_pause needs the guest TSC at the host rate; ... | Remove tsc-frequency from -cpu. |
csd supports at most one namespace per controller | Two CSD namespaces, from namespace_modes or from femu_mode=4 with namespaces above 1. Other namespaces of the controller may use other modes. |
FEMU bbssd: namespace 1 exposes ... | The namespace does not fit the NAND geometry; see the BlackBox limits. |
FEMU csd: hot_cold_sep is not supported under FDP: ... | A knob the FDP write path does not take, on a controller in an FDP subsystem; the same list as for BlackBox (FDP). |
A program load that fails returns Invalid Field to the guest, or Capacity
Exceeded when the program table is full. For a missing or bad program file,
QEMU prints the reason on its console after [FEMU] Err:, for example:
CSD: loading a program needs csd_program_dir to be setCSD: a program name must be a file in csd_program_dir, got "..."CSD: <path> does not resolve inside csd_program_dirCSD: failed to load shared library <path>: <reason>
An eBPF load on a build without uBPF, a malformed load descriptor, or an unknown program type fails with Invalid Field and prints nothing.
Verify
sudo nvme id-ctrl /dev/nvme0reportsFEMU Computational Storage Controller.csd-passthru /dev/nvme0n1 smokecompletes without errors.- With
csd_program_dirset,smoke-so csd-vadd.socompletes; without it, the same command fails and the QEMU console says why.
Troubleshooting
- A program load fails with Invalid Field. Read the QEMU console (or
build-femu/log). Usuallycsd_program_diris unset, the name contains a path, or the file is not in the directory. No message means an eBPF load on a build without uBPF, or a malformed load command. - An eBPF load fails. The FEMU build has no uBPF support. Rebuild with
./femu-compile.sh --enable-csd-ubpf. makein the guest fails oncsd-original-kernels.so. It needs the lz4 development package and is meant to be built on the host. Build onlycsd-passthruin the guest.
Related issues: #60, #143, #188.
Citation
CSD mode is derived from CEMU. We thank the CEMU authors, Qiuyang Zhang, Jiapin Wang, You Zhou, Peng Xu, Kai Lu, Jiguang Wan, Fei Wu and Tao Lu, and Emilio (@Emilio597), who ported it to FEMU in #188. If you use the CSD mode, please also cite:
@inproceedings{Zhang+26-CEMU,
author = {Qiuyang Zhang and Jiapin Wang and You Zhou and Peng Xu and
Kai Lu and Jiguang Wan and Fei Wu and Tao Lu},
title = {{CEMU: Enabling Full-System Emulation of Computational Storage
Beyond Hardware Limits}},
booktitle = {Proceedings of the 31st ACM International Conference on
Architectural Support for Programming Languages and Operating
Systems (ASPLOS '26), Volume 2},
pages = {323--341},
year = {2026},
doi = {10.1145/3779212.3790137},
}
Related pages
- hw/femu/tests/csd/README.md: the guest tool and program ABI
- BlackBox SSD: the FTL under the namespace
- Timing model: KV and CSD