Skip to main content

Computational storage (CSD)

Mirrored from the FEMU repository

This page is hw/femu/docs/modes/csd.md at FEMU d8b29b1e0 (2026-11-16), licensed GPL-2.0-or-later. Send corrections to the FEMU repository.

CSD mode (femu_mode=4) emulates a computational storage drive. On top of a normal NVMe namespace, the device has its own memory (function data memory, FDM) and compute units that run programs next to the data. The guest allocates device memory, copies namespace data into it, loads and runs a program, and reads the result back, all with NVMe commands. Ordinary reads and writes go through the BlackBox FTL, so they have SSD timing.

The mode is a port of CEMU; if you use it, cite CEMU as well as FEMU (citation). It does not need CEMU's modified guest kernel, FDMFS or a fixed VM image. CEMU's VM freezing and virtual clock changes are left out of this port. The opt-in csd_pause does that job on an unmodified QEMU and guest.

Use it to study offloading filters, scans, compression or other kernels to the drive, and what that does to latency and host CPU use.

Requirements​

  • Host and guest: see the mode table and requirements.md. Any guest kernel with the NVMe driver works.
  • Guest tools: hw/femu/tests/csd, built in the guest with a C compiler. The image from make-guest-image.sh has none; add one with ./make-guest-image.sh --packages build-essential, or run sudo apt install build-essential in the guest.
  • Programs: shared-library programs are built on the host and placed in the directory named by csd_program_dir. eBPF programs also need a FEMU build with uBPF (./femu-compile.sh --enable-csd-ubpf, see build.md) and clang with the BPF target on the host.

Security: programs run inside QEMU on the host​

A program load names a file. QEMU, on the host, opens that file and runs code from it inside the QEMU process. The run-*.sh launchers start QEMU with sudo, so that code runs as root on the host.

FEMU limits what the guest can name:

  • With csd_program_dir unset, only the built-in phantom program type loads. Shared-library and eBPF loads fail.
  • With it set, the guest may name only a file in that directory: no /, no .., and the path must still resolve inside the directory after symbolic links are followed.

That protects the rest of the host file system. It does not make the programs in the directory safe: they run on QEMU's compute unit threads with all of QEMU's privileges and memory. Use CSD only with guests you trust, put only programs you built and trust in the directory, and make sure no one else can write to it.

Launch​

From build-femu/:

./run-csd.sh

The FEMU device in that script is:

-device femu,devsz_mb=4096,namespaces=1,femu_mode=4,secsz=512,secs_per_pg=8,pgs_per_blk=256,blks_per_pl=256,pls_per_lun=1,luns_per_ch=8,nchs=8,pg_rd_lat=40000,pg_wr_lat=200000,blk_er_lat=2000000,ch_xfer_lat=0,gc_thres_pcent=75,gc_thres_pcent_high=95,fdm_size=64,nr_cu=4,nr_thread=4,time_slice=200000,context_switch_time=200,csf_runtime_scale=3

The preset hw/femu/scripts/configs/csd.conf describes the same device without the three scheduler properties that have no effect. ssd-config.sh expands it to:

-device femu,id=nvme0,devsz_mb=4096,namespaces=1,secsz=512,secs_per_pg=8,pgs_per_blk=256,blks_per_pl=256,pls_per_lun=1,luns_per_ch=8,nchs=8,pg_rd_lat=40000,pg_wr_lat=200000,blk_er_lat=2000000,gc_thres_pcent=75,gc_thres_pcent_high=95,fdm_size=64,nr_cu=4,csf_runtime_scale=3,femu_mode=4

run-csd.sh does not set csd_program_dir, so it runs phantom programs only. To load your own programs, add the directory after the other FEMU_OPTIONS lines in run-csd.sh:

FEMU_OPTIONS=${FEMU_OPTIONS}",csd_program_dir=$HOME/csd-programs"

The shortest device line with programs enabled is:

-device femu,devsz_mb=4096,femu_mode=4,fdm_size=64,csd_program_dir=/home/you/csd-programs

FEMU does not check the directory at start-up. A directory that does not exist makes every program load fail.

Configuration​

Properties: CSD. The BlackBox geometry, timing, GC and FTL properties also apply to the namespace; see the BlackBox guide.

  • fdm_size: device memory in MiB. Required.
  • nr_cu: compute units, 1 to 64. Each is a host thread, femu-csd-cu, that runs programs. A program waits for the first free unit.
  • csf_runtime_scale: a program that declares no run time and no scale of its own holds its unit for its measured host run time times this value (default 3). A completion carries the program's result, so it never arrives before the host has run the program. Without csd_pause, a run time, declared or scaled, shorter than the host's own is not reached, and QEMU warns once when that happens.
  • csd_pause: stop the guest while the host runs a program, so the guest sees only the modelled run time (default off). See below.
  • csd_pause_max_ms: with csd_pause, the longest stop, 1 to 60000 ms (default 1000).
  • csd_program_dir: the host directory programs load from.
  • nr_thread, time_slice and context_switch_time are accepted so CEMU configurations still start. They have no effect, and a value other than the default prints a warning at realize.

A copy from the namespace into device memory costs one pg_rd_lat, whatever its size, when any page of the range has been written, and nothing otherwise. See the timing model.

Program types​

TypeWhat runsNeeds
PhantomBuilt in: copies the input memory range to the output range. Useful to test the command flow and timing.nothing
Shared libraryA function with the signature int64_t fn(struct femu_csd_args *args) from a .so file, named in the load command.csd_program_dir
uBPFAn eBPF ELF object, interpreted or JIT-compiled.csd_program_dir and a build with --enable-csd-ubpf

How programs run​

An Execute command is checked on the poller and then handed to one of the nr_cu compute unit threads, which runs the program and posts the completion. While a program runs, I/O on every queue, other CSD commands and admin commands go on as usual; only that compute unit is busy.

  • At most nr_cu programs run at once. Runs of one program take turns, so a shared library needs no locking of its own; different programs run side by side, and a run waiting for its own program leaves the compute unit free for another.
  • A program that never returns keeps its compute unit for good, and QEMU waits for it when the device is removed. FEMU cannot stop native code it has called.
  • Device memory a program is using stays allocated, and counted against fdm_size, until the program returns, even after Free device memory.
  • Deleting the I/O queue an Execute came from, or resetting the controller, while the program runs drops its result; the program still runs to the end.

Hide the host run time (csd_pause)​

A real CSD runs a program on its own cores while the host goes on. FEMU runs it on a host thread, and the completion carries the result. So without help the guest waits for the host run, also when the model says the device is faster.

With csd_pause=on, FEMU stops the guest while a program runs, and hides that time from the guest clocks. The device clock of the controller stops with them. The guest then sees the modelled run time.

host time |--Execute--|======= host runs program =======|--CQE--|
guest time |--Execute--| |--CQE--|
<- guest stopped, clocks hidden ->
guest sees Execute .......... modelled run time .......... CQE
  • Under KVM, FEMU stops the vCPUs, sets the kvmclock back and moves the TSC of each vCPU back by the same span. Guest readers of the kvmclock, the vDSO and the TSC all skip the stop.
  • Overlapped runs on several compute units share one stop.
  • All I/O of the controller runs on the device clock, so other commands keep their modelled latency.
  • A stop has a cost: the guest runs for a short time before the vCPUs stop. On a 20-CPU host with 4 vCPUs, the guest saw the modelled run time plus 30 to 50 us at the median.
  • Time outside the guest does not stop. Network peers, the wall clock and other devices, also other FEMU controllers, move on during a stop.
  • A vCPU that stays halted through many stops keeps an old pvclock page. The page can stray from the clock by about 1 ns a stop until that vCPU runs again.
  • A LAPIC timer due during a stop fires once, early. Linux with high resolution timers sets it again, so a sleep does not end early.
  • A program that runs past csd_pause_max_ms lets the guest run again, and the guest sees the rest of the host run. QEMU warns once.
  • A VM stop, suspend, reset or hot plug during a program makes that one stop visible to the guest.
  • If KVM fails a clock call after the kvmclock is set, the vCPUs would run on clocks that disagree. QEMU then stops the VM with an internal error.

It needs KVM on x86 with Linux 6.0 or later on the host. The guest TSC must run at the host rate, and realize refuses a tsc-frequency that changes it. Every namespace of the controller must be CSD or bbssd, and the controller must not share namespaces. The script hw/femu/scripts/csd-pause-check/run.sh boots a guest with and without the option, and prints the latency each sees.

-device femu,devsz_mb=4096,femu_mode=4,fdm_size=64,csd_program_dir=/home/you/csd-programs,csd_pause=on

Commands​

I/O commands on the namespace. The opcodes are in the vendor specific range 80h to FFh. Bits 1:0 give the data direction (Base 2.3 Figure 91), and a command that moves no data may carry any value there:

OpcodeCommandData
0xb0Allocate device memorynone
0xc0Free device memorynone
0xd0Copy namespace data into device memorynone
0xe1Execute a programhost to device
0xf2Read device memorydevice to host
0xf5Write device memoryhost to device
0xf6, 0xf7, 0xf8Create a group, set its QoS, delete itnone

Admin commands, in the vendor specific range C0h to FFh. As a mnemonic, 0xc5, 0xc8 and 0xc9 are the CP 1.2 opcodes 0x85, 0x88 and 0x89 of the matching commands plus 40h. CP has no Load Data command, so 0xcd has no CP pair:

OpcodeCommandData
0xc5Program load and unloadhost to device
0xc8Program activate and deactivatenone
0xc9Memory range set managementhost to device
0xcdLoad more program datahost to device

The command layouts follow CEMU; hw/femu/tests/csd/csd-passthru.c builds each one.

Migration from the CEMU opcodes​

CEMU and FEMU before this change used admin opcodes 0x21, 0x22, 0x23 and 0x25. NVMe Base 2.3 gives 0x21, 0x22 and 0x25 to fabrics commands, which a PCIe controller must not support, and reserves 0x23. 0x22 also declares data to the host, but it sends program bytes to the device. The Linux driver reads that bit to map the data buffer. The I/O opcodes do not change.

old new command
0x21 0xc9 memory range set management
0x22 0xc5 program load and unload
0x23 0xc8 program activate and deactivate
0x25 0xcd load more program data

The device now answers the old opcodes with Invalid Command Opcode. To run an old tool unchanged, set csd_legacy_opcodes=on. The device then takes the old opcodes and refuses the new ones. The Commands Supported and Effects log (05h) lists the layout in use. The passthrough helper takes CSD_LEGACY_OPCODES=1 in its environment to send the old opcodes.

NVMe CP and SLM command sets​

csd_slm_size adds a memory namespace (SLM 1.2) and csd_cp=on a compute namespace (CP 1.2) after the CSD namespace. They take the ratified opcodes, and their programs, range sets and memory are separate from the CEMU ones above. The CEMU commands stay on, on the CSD namespace. See CP and SLM.

CSD namespace (CSI 00h) NVM commands + CEMU b0h..f8h, admin c5h..cdh
memory ns (CSI 03h) Memory Copy 01h, Read 02h, Fill 04h, Write 05h
compute ns (CSI 04h) Execute Program 01h, admin 85h, 88h, 89h

Use it from the guest​

Check the device:

sudo nvme list
sudo nvme id-ctrl /dev/nvme0 | grep -E '^(mn|sn) '

The model is FEMU Computational Storage Controller and the serial number starts with vCSD.

Build the guest tool​

From build-femu/ on the host, copy the tools into the guest and build the passthrough helper:

scp -P 8080 -i ~/images/femu-guest-key -r ../hw/femu/tests/csd femu@localhost:
./run-guest-ssh.sh make -C csd csd-passthru

Run the phantom smoke test​

This needs no program directory. It allocates, writes and reads device memory, loads and runs a phantom program, and frees everything:

./run-guest-ssh.sh sudo ./csd/csd-passthru /dev/nvme0n1 smoke

Run a shared-library program​

On the host, build the example programs and put them in the program directory (csd-original-kernels.so needs the lz4 development package):

cd hw/femu/tests/csd # in the FEMU source tree
make csd-vadd.so csd-original-kernels.so
mkdir -p ~/csd-programs
cp csd-vadd.so csd-original-kernels.so ~/csd-programs/

Start FEMU with csd_program_dir set to that directory, then name the file alone in the guest:

sudo ./csd/csd-passthru /dev/nvme0n1 smoke-so csd-vadd.so
sudo ./csd/csd-passthru /dev/nvme0n1 smoke-so-all csd-original-kernels.so
sudo ./csd/csd-passthru /dev/nvme0n1 bench 4096 32

bench reports the average latency of device memory writes, reads and namespace-to-memory copies. hw/femu/tests/csd/README.md lists every subcommand, the eBPF steps and the program ABI.

The vendor log page C0h counts the namespace's NAND traffic as for BlackBox (log pages and counters).

Limits and refusals​

MessageCause and fix
CSD mode requires fdm_size to be non-zeroSet fdm_size.
CSD nr_cu must be in range [1, 64]nr_cu out of range.
CSD nr_thread must be non-zeronr_thread=0.
CSD csf_runtime_scale must be non-zerocsf_runtime_scale=0.
CSD csd_pause_max_ms must be in range [1, 60000]csd_pause_max_ms out of range.
csd_pause needs a CSD namespacecsd_pause on a controller with no CSD namespace.
csd_pause: namespace N runs a mode other than bbssd or CSDOther modes keep their own clocks; use them on another controller.
csd_pause cannot be used with shared namespaces: ...Two controllers would time one FTL on two clocks.
csd_pause needs KVMAnother accelerator; csd_pause needs -enable-kvm.
csd_pause needs KVM_CLOCK_HOST_TSC and KVM_CAP_VM_TSC_CONTROL ...The host kernel is older than Linux 6.0.
csd_pause needs the guest TSC at the host rate; ...Remove tsc-frequency from -cpu.
csd supports at most one namespace per controllerTwo CSD namespaces, from namespace_modes or from femu_mode=4 with namespaces above 1. Other namespaces of the controller may use other modes.
FEMU bbssd: namespace 1 exposes ...The namespace does not fit the NAND geometry; see the BlackBox limits.
FEMU csd: hot_cold_sep is not supported under FDP: ...A knob the FDP write path does not take, on a controller in an FDP subsystem; the same list as for BlackBox (FDP).

A program load that fails returns Invalid Field to the guest, or Capacity Exceeded when the program table is full. For a missing or bad program file, QEMU prints the reason on its console after [FEMU] Err:, for example:

  • CSD: loading a program needs csd_program_dir to be set
  • CSD: a program name must be a file in csd_program_dir, got "..."
  • CSD: <path> does not resolve inside csd_program_dir
  • CSD: failed to load shared library <path>: <reason>

An eBPF load on a build without uBPF, a malformed load descriptor, or an unknown program type fails with Invalid Field and prints nothing.

Verify​

  1. sudo nvme id-ctrl /dev/nvme0 reports FEMU Computational Storage Controller.
  2. csd-passthru /dev/nvme0n1 smoke completes without errors.
  3. With csd_program_dir set, smoke-so csd-vadd.so completes; without it, the same command fails and the QEMU console says why.

Troubleshooting​

  • A program load fails with Invalid Field. Read the QEMU console (or build-femu/log). Usually csd_program_dir is unset, the name contains a path, or the file is not in the directory. No message means an eBPF load on a build without uBPF, or a malformed load command.
  • An eBPF load fails. The FEMU build has no uBPF support. Rebuild with ./femu-compile.sh --enable-csd-ubpf.
  • make in the guest fails on csd-original-kernels.so. It needs the lz4 development package and is meant to be built on the host. Build only csd-passthru in the guest.

Related issues: #60, #143, #188.

Citation​

CSD mode is derived from CEMU. We thank the CEMU authors, Qiuyang Zhang, Jiapin Wang, You Zhou, Peng Xu, Kai Lu, Jiguang Wan, Fei Wu and Tao Lu, and Emilio (@Emilio597), who ported it to FEMU in #188. If you use the CSD mode, please also cite:

@inproceedings{Zhang+26-CEMU,
author = {Qiuyang Zhang and Jiapin Wang and You Zhou and Peng Xu and
Kai Lu and Jiguang Wan and Fei Wu and Tao Lu},
title = {{CEMU: Enabling Full-System Emulation of Computational Storage
Beyond Hardware Limits}},
booktitle = {Proceedings of the 31st ACM International Conference on
Architectural Support for Programming Languages and Operating
Systems (ASPLOS '26), Volume 2},
pages = {323--341},
year = {2026},
doi = {10.1145/3779212.3790137},
}