Skip to main content

NAND media and timing model

Mirrored from the FEMU repository

This page is hw/femu/docs/design/nand-timing.md at FEMU d8b29b1e0 (2026-11-16), licensed GPL-2.0-or-later. Send corrections to the FEMU repository.

This chapter describes the component that decides how long a flash operation takes in FEMU. It covers the geometry the model works on, the time it charges for a page read, a page program and a block erase, how operations on the same die, plane or channel wait for each other, and how those times become the completion time of an NVMe command. It also covers the two models that sit after the media: the host link and the controller firmware CPU.

The timing model concept page explains the same model from the user's side. This page is for readers who want the algorithms, the data structures and the source.

Contents​

Purpose and place in FEMU​

FEMU copies a command's data as soon as it parses the command. Timing is computed separately: a model works out when the command would have finished on the emulated device, and the poller holds the completion until the host clock reaches that time. The NAND media model is the part of that computation that charges flash operations.

The media layer, nand_media_op() in hw/femu/nand/nand-media.c, is the one media engine. BBSSD, CSD, KV, ZNS, the FTL behind a femu-cxl-ssd, and Open-Channel 1.2 and 2.0 use it. It never includes a controller header. Each mode decodes its own address into a NandLoc, gives the layer a configuration, and lends it pointers to its busy-until fields. hw/femu/ocssd/oc-timing.c keeps the Open-Channel table times, the geometry check and 0xEE (OCSSD timing model).

guest NVMe command
|
v
+--------------------+ stamps stime, copies data
| poller (nvme-io.c) |------------------------------------------+
+--------------------+ |
| to_ftl ring |
v |
+--------------------+ per mode: page list -> NAND ops |
| FTL | bbssd/ftl-datapath.c, zns/zftl.c, |
| (FTL thread, or | kvssd/kvssd-ftl.c (in the poller) |
| poller for KV) | |
+--------------------+ |
| NandLoc + op + stime |
v |
+--------------------+ array time, rules, bus phases, |
| NAND media layer | suspend, ECC; returns done - stime |
| nand/nand-media.c | |
+--------------------+ |
| per-command latency = max over its operations |
v |
expire_time = stime + latency |
| |
v |
+--------------------+ host link, firmware CPU, then the |
| poller completion | priority queue by expire_time <-------+
+--------------------+
|
v
CQE posted on the first sweep at or after expire_time
ModeEngineBuilt inCalled from
BBSSD, CSDmedia layerbb_nand_media_init() in bbssd/ftl-media.cFTL thread
KVmedia layer, through the BBSSD wrapperssd_init() from kvssd/kvssd-ftl.cpoller
ZNSmedia layerzns_nand_media_init() in zns/zftl.cFTL thread
femu-cxl-ssdmedia layer, through the BBSSD wrapperfemu_cxl_start() in cxlssd/cxlssd.cfemu-cxl-ftl worker
OCSSD 1.2, 2.0media layeroc12_media_init(), oc20_media_init() in ocssd/poller
NoSSDnone

Geometry​

The NAND hierarchy (channel, LUN, plane, block, page, sector), a line as one block index across every plane of every LUN, and the order in which the write pointer hands out pages.

Figure: The NAND hierarchy (channel, LUN, plane, block, page, sector), a line as one block index across every plane of every LUN, and the order in which the write pointer hands out pages.

Every mode that models NAND describes it with the same hierarchy. Only the names of the properties and the field widths differ.

device
+-- channel 0 .. nchs-1 one shared bus per channel
+-- LUN 0 .. luns_per_ch-1 a die: runs one array operation at a time
| (one per plane under the ZNS plane rule)
+-- plane 0 .. pls_per_lun-1
+-- block 0 .. blks_per_pl-1 erase unit
+-- page 0 .. pgs_per_blk-1 program and read unit
+-- sector 0 .. secs_per_pg-1 (secsz bytes each)

BBSSD line (superblock) k = block k on every (channel, LUN, plane)
lines = blks_per_pl pages per line = nchs * luns_per_ch * pls_per_lun * pgs_per_blk

Address formats​

ModeAddressFields, low bit firstWhere
BBSSD, CSD, KVstruct ppablk 16, pg 16, sec 8, pl 4, lun 7, ch 12, reserved 1bbssd/ftl.h
ZNSstruct ppaspg 2, pg 16, blk 16, fc (LUN) 8, pl 3, ch 7, valid 1, reserved 8zns/zns.h
OCSSD 1.264-bit PPA with masks and offsets from the geometrych, lun, pln, blk, pg, secPPA_* macros in nand/nand.h
OCSSD 2.064-bit LBA with group, parallel unit, chunk, sector fieldsgroup, PU, chunk, sectorOC20_LBA_GET_* in ocssd/

A field width is an upper bound on its axis. bb_check_geometry() in bbssd/ftl-geom.c refuses a count larger than its field can hold, and refuses a geometry whose total sector count does not fit in a signed 32-bit integer. zns_check_params() bounds each ZNS axis by its field. oc_timing_geometry_ok() requires every OCSSD axis to be non-zero, lnum_ch at most 32 and lnum_ch * lnum_lun at most 128, the sizes of the per-channel and per-LUN arrays.

How pages land on the geometry​

The timing a workload sees depends on where the FTL puts its pages, because the media model only charges what it is asked to.

  • BBSSD, CSD, KV allocate from a write pointer that moves channel first, then LUN, then plane, then page (plane first with plane_first on bbssd and CSD) (ssd_advance_write_pointer_common() in bbssd/ftl-line-gc.c). Consecutive pages of a large write therefore land on different channels, then different LUNs, and are programmed in parallel.
  • ZNS buffers writes in a per-zone write cache. A flush programs one program unit per plane: zns_flash_type pages of 16 KiB (ZNS_PAGE_SIZE) per plane, charged as one program operation per plane (zns_wc_flush() in zns/zftl.c). A 4 KiB logical page is a sub-page (spg) of a 16 KiB page.
  • OCSSD takes physical addresses from the host, so the host chooses the parallelism.

Data structures​

The media layer's types are in hw/femu/nand/nand-media.h.

TypeFields that matterRole
NandLocch, lun, pl, blk, pg, flash_type, page_type, pe_cycles, age_sec, xfer_secsOne operation's position and the facts timing depends on. Filled by the mode's decoder (bb_decode_loc(), zns_advance_status(), oc12_media_op()).
NandMediaTimingrd_ns, wr_ns, er_ns (flat); rd_table_ns, wr_table_ns, er_table_ns (by cell and page type); pgtype_mult; cmd_addr_ns, page_xfer_ns, status_ns; tplpbsy_ns, tplrbsy_ns, tplebsy_ns; ecc_*; tsusp_nsEvery duration.
NandMediaPolicyarray_contend, channel_mode, pe_suspend, ecc_on_read, use_flat_timingWhich mechanisms are on.
NandTimelineOpsch_avail, lun_avail, plane_avail, lock_lun, unlock_lunAccessors that return pointers into the mode's own busy-until fields.
NandMediacfg, bus_res (per-channel booked windows), susp (per-position suspend state)One instance per namespace FTL.
NandOpCompletiondone_ns, latency_nsAbsolute end time, and done_ns - stime.

The busy-until times ("availability timestamps") live in the mode's structures, not in the media layer:

ResourceBBSSD, CSD, KVZNSOCSSD
Channel busssd_channel.next_ch_avail_timezns_ch.next_ch_avail_timeOc12Ctrl.ch_avail[] (1.2 only)
LUN (die)nand_lun.next_lun_avail_timezns_fc.next_fc_avail_time (only with zns_lun_contention)Oc12Ctrl.lun_avail[], Oc20Ctrl.lun_avail[]
Planenot keptzns_plane.next_plane_avail_timenot kept

Each mode configures the layer differently:

SettingBBSSD, CSD, KV, CXLZNS
array_contendNAND_CONTEND_LUN_ONLYNAND_CONTEND_PLANE_ONLY; NAND_CONTEND_LUN_AND_PLANE with zns_lun_contention
channel_modeNAND_CH_STAGED when any bus phase is non-zero, else NAND_CH_OFFsame rule
use_flat_timingtrue when nand_cell_type=0, false when it is setfalse (per-type table; page type 0, or pg % zns_flash_type for reads with zns_page_types)
pe_suspendpe_suspend propertyzns_pe_suspend property
ecc_on_readtrue; the adder is still 0 unless ecc_step_ns is settrue; the adder is still 0 unless zns_ecc_step_ns is set

The enum also has NAND_CH_NOOP. No mode selects it today.

Operation timing​

How the media layer times an operation: the phases of a read, program and erase, and four reads issued together showing LUN serialization, a shared channel bus and parallel channels.

Figure: How the media layer times an operation: the phases of a read, program and erase, and four reads issued together showing LUN serialization, a shared channel bus and parallel channels.

Array time​

array_lat() picks the time the die spends on the operation:

erase : flat ? er_ns : er_table_ns[flash_type]
read : (flat ? rd_ns : rd_table_ns[flash_type][page_type]) + ecc_extra
program : flat ? wr_ns : wr_table_ns[flash_type][page_type]
if flat and pgtype_lat: * pgtype_mult[page_type] / 1000

Phase sequence of one operation​

With the channel bus off (the default), an operation is one array phase:

start = max(stime, rule busy-until)
done = start + array time
rule busy-until = done

With the bus on (NAND_CH_STAGED), each operation also has bus phases. "Now" phases queue on the bus in arrival order; a read's data-out happens after the array finishes, so it is booked as a future window (Channel bus).

READ |cmd/addr|..wait for die..|=== tR array ===|data-out|status|
bus now die bus later bus later

PROGRAM |cmd/addr|data-in|..wait for die..|======= tPROG array =======|
bus now bus now die (done = array end)

ERASE |cmd/addr|..wait for die..|=========== tBERS array ===========|status|
bus now die bus later

A program completes when its array phase ends. A read completes after its status phase, an erase after its status phase. A zero-length phase touches nothing, so a bus with only pg_xfer_lat set has no command or status cost.

Default times​

ModeReadProgramEraseSource
BBSSD, CSD, KV flat (nand_cell_type=0)pg_rd_lat 40 uspg_wr_lat 200 usblk_er_lat 2 msproperty defaults in femu.c
femu-cxl-ssdread-ns 40 usprogram-ns 200 userase-ns 2 mscxlssd/qemu-adapter.c
ZNS SLC4 us75 us2 mszns/zns.h, erase from nand/nand.h
ZNS TLC32 us937.5 us3 mssame
ZNS QLC (default zns_flash_type)85 us12.196 ms3 mssame
ZNS MLC, PLCnone built innone built innone built inrealize refuses them unless all three zns_*_lat are set

The ZNS program time covers a whole program unit (zns_flash_type pages of 16 KiB on one plane). The header cites the ISSCC papers the figures come from.

Cell types and page types​

BBSSD, CSD, KV: nand_cell_type​

nand_cell_typeCellRead by page type (us)Program by page type (us)Erase
0flatpg_rd_latpg_wr_lat, optionally scaled by pgtype_latblk_er_lat
1SLC408002 ms
2MLClower 48, upper 64lower 850, upper 23003 ms
3TLC56.5, 77.5, 106820.5, 2225, 57343 ms
4QLC59.325, 85.25, 127.2, 169.6861.525, 2447.5, 6880.8, 9174.43 ms

Values come from nand/nand.h: MLC from a profile of Micron L95B parts, TLC from SimpleSSD, QLC scaled from the TLC values. Any other value is reset to 0 with a message (ssd_init() in bbssd/ftl.c); PLC has no table here. With a cell type set, pg_rd_lat, pg_wr_lat and blk_er_lat no longer set NAND times (pg_rd_lat still sets the write buffer and read cache hit cost), and pgs_per_blk must be at most 512, the size of the page-type tables.

The page type of page pg in a block comes from the pairing tables mlc_tbl, tlc_tbl and qlc_tbl, built by init_nand_flash() in nand/nand.c; slc_tbl is all zeros. They follow a shadow programming order: the first pages of a block are lower pages, then consecutive pairs of pages cycle through the page types from lower to upper.

BBSSD, CSD, KV: pgtype_lat and cell_pages​

With nand_cell_type=0 and pgtype_lat non-zero, the flat program time is multiplied by a factor that depends on the page's position in its wordline. The page type is pg % cell_pages; cell_pages 0 is set to 3 when pgtype_lat is on (ssd_init()). The factors, in thousandths, are in bb_nand_media_init():

cell_pagesPage type 01234
11000
26001400
360010001400
450085011501500
5400750100012501600

Each row averages to 1000, so the mean program time stays pg_wr_lat when pgs_per_blk is a multiple of cell_pages. Reads and erases are not scaled.

ZNS and OCSSD​

ZNS keeps one read, program and erase time per flash type. It uses page type 0, except for reads with zns_page_types (see ZNS). Each zns_*_lat property replaces the built-in value for the configured zns_flash_type only. OCSSD uses the nand_cell_type tables above, selected by flash_type; Open-Channel 1.2 looks up the page type of each page. Open-Channel 2.0 uses page type 0. With oc20_page_types=on a chip operation covers one NAND page and uses its page type.

Channel bus​

The bus is modelled only when a phase has a duration: cmd_addr_lat, pg_xfer_lat (or ch_xfer_lat when pg_xfer_lat is 0) or status_lat for BBSSD, CSD and KV; zns_cmd_addr_lat, zns_pg_xfer_lat or zns_status_lat for ZNS. Otherwise transfers take no time and channels never contend.

Two kinds of phase share a channel:

  • bus_now(): command/address, a program's data-in. It starts at max(earliest, ch_avail), moves past any booked window it would overlap, and advances ch_avail to its end.
  • bus_later(): a read's data-out and status, which happen after the array finishes. It starts no earlier than ch_avail and is booked as a window [start, end) in bus_res[ch] and does not advance ch_avail, so another die's "now" phase can use the bus before the window opens.

Windows that ended before the current operation's stime are pruned. A channel holds at most NAND_BUS_RES_MAX (32) windows; when it is full, the data-out is charged FIFO from ch_avail instead. With policy.bus_res_unbounded (OCSSD 1.2) the list grows and nothing falls back.

A data phase moves one page in page_xfer_ns. When the mode sets secs_per_page and the operation's NandLoc.xfer_secs, it moves only those sectors and takes DIV_ROUND_UP(page_xfer_ns * xfer_secs, secs_per_page), as OCSSD 1.2 charges a partial page.

channel 0 bus, cmd/addr 1 us, data 10 us (times in us)

0 1 11 12 13 52 62
|P c|P data-in|R1c|R2c|.... idle .....|R1 out |
^ booked window, R1's array ended at 52
a phase that arrives at 20 fits in the idle gap; one that would overlap
[52,62) moves to 62

ONFI 6.0 profile (BBSSD, CSD, KV)​

onfi_mode (0 to 26) sets every channel bus phase from an ONFI 6.0 NV-LPDDR4 timing mode. ONFI 6.0 gives no array times, so pg_rd_lat, pg_wr_lat, blk_er_lat and nand_cell_type still set tR, tPROG and tBERS. The default, -1, turns the profile off and changes nothing.

onfi_mode, onfi_mts --> tDSC --> page_xfer = bytes * tDSC / 2 (x8, DDR)
Table 4-34 constants --> per-op command, status and burst phases
pg_rd_lat, pg_wr_lat, blk_er_lat ------------------> array times (unchanged)

read |00h addr 30h tWB|..... tR .....|70h status|00h pre| page |post|
program |80h addr tADL|edges| page |10h tWB|...... tPROG ......|70h status|
erase |60h addr D0h tWB|.............. tBERS ..............|70h status|
^ sent only when the LUN is ready (RDY) LUN held to here ^

The profile derives these phases (nand/onfi.c). Each one is a sum in ps, rounded up to whole ns once. The table shows mode 26 with 2 column and 3 row address cycles.

PhaseSequenceMode 26
read command00h, 5 address cycles, 30h, tWB275 ns
program command80h, 5 address cycles, tADL562 ns
data in edgestWPRE2 + tWPST2 + tWPSTH257 ns
program confirm10h, tWB125 ns
erase command60h, 3 address cycles, D0h, tWB225 ns
status70h (78h + 3 cycles with several LUNs), tWHR, tRPRE2, one byte, tRPSTH, tRHW246 ns (321 ns)
data out lead00h, tRPRE2; with several LUNs 06h, 5 cycles, E0h, tCCS, tRPRE250 ns (200 ns)
data out tailtRPSTH, tRHW115 ns
page of 4 KiB4096 bytes at 0.417 ns per 2 bytes855 ns
  • onfi_mts replaces the mode's rate for the page only: ceil(bytes * 1000 / mts) ns. The status byte keeps the mode's tDSC.
  • onfi_spare_bytes adds spare bytes to each page on the bus.
  • onfi_tccs_ns gives tCCS. ONFI 6.0 has no value, so 0 charges nothing.
  • A tplpbsy, tplrbsy or tplebsy left at 0 takes 500 ns, the Table 4-30 typical value.
  • A read sends its status before its data out (Section 6.14). A program, erase or copyback ends at its status.
  • An op sends its command only when its LUN is ready, because the page register is in use until then. The LUN stays busy to the end of the read's data out, or of the program, erase or copyback status.
  • A command that waits for a busy LUN books a future window, so an op on an idle LUN can use the bus first.
  • Op start times are not monotonic. A copy's write starts at the end of its reads, and garbage collection starts at the wall clock. An op with an older start time can still use an idle bus gap before the latest start time on its channel.
latest start time on the channel: high
dropped | kept history (ends <= high) | future windows
[=] [==] | [==] [=] [===] ... [==] | [====] [==]
low high
an op with start t >= low: can use any idle gap after t
an op with start t < low: starts no earlier than low (the fallback)
  • At the start of each op, a grown list drops old windows. When more than 1024 windows end before high in time, it keeps the newest 512. The NAND_BUS_HISTORY_MAX value sets this cap. Then low moves to the end of the newest dropped window, and the op can add a few windows of its own.
  • A window that ends before ch_avail also goes, because no phase can start before it. A bounded 32-entry list has no room for history, so it keeps low at high.
  • An op that starts at or after low gets the gaps it would get from a list that never drops a window. Its LUN and ch_avail can still delay it.
  • A phase that happens now (a command to a ready LUN) still starts after ch_avail in arrival order, as without the profile.
  • Future windows stay until they end, so the list grows with the media ops in flight: one set of windows per queued op.
  • A multi-plane program sends every plane's command and data, with the tPLPBSY gaps, as one bus window (Section 3.1.3). A multi-plane read or erase books each later plane command after its gap.
  • With several LUNs, a multi-plane program or erase reads one 78h status per plane. With one LUN, one 70h status reads the composite.

Approximations:

  • The status and data out phases charge tRPSTH (15 ns) and not tRPST. tRPST is tDQSRE plus half a tRC, and tDQSRE is 1 to 25 ns by device (Table 4-38).
  • Warmup cycles are not modelled. Section 4.12 calls them optional, but Section 4.15 note 12 asks for them above 800 MT/s.
  • The profile assumes the program page register clear enhancement is on.
  • With the profile, the channel keeps every future window it books, so a full list never books the bus from now.

Realize refuses these settings:

  • onfi_mode outside -1 to 26, or onfi_mts above 100000;
  • a negative onfi_mts, onfi_spare_bytes or onfi_tccs_ns;
  • an odd page size on the bus (secsz * secs_per_pg + onfi_spare_bytes), because a DDR burst moves bytes in pairs;
  • cmd_addr_lat, pg_xfer_lat, ch_xfer_lat or status_lat with the profile;
  • a namespace that is not bbssd, CSD or KV;
  • cxl_ssd, ns_mgmt or shared namespaces;
  • pe_suspend, because ONFI 6.0 has no suspend and a held page register has no suspend rule.

onfi_mts, onfi_spare_bytes or onfi_tccs_ns without onfi_mode warns.

SCA protocol (onfi_sca)​

onfi_sca=1 selects the Separate Command Address protocol (Section 5). Commands, addresses and status then move to a CA bus. Each channel gets its own CA bus. The channel bus (DQ) carries only data bursts, so a command of one LUN can run during the data of another.

read, P = 12 ns CA packet
CA |00h A*5 30h tWB| |70h/78h status|06h A*5 E0h|tCCS|SCE| |SCT|
DQ |SCE|pre page post|SCT|hold|
LUN ..RDY..|=== tR ===|.................................................held|
program
CA |80h A*5 12h| |SCE| |SCT|10h tWB| |status|
DQ |<-- gap --->|SCE|pre page|SCT|hold|
|<------ tCDL ----------->| (to the end of the preamble)
LUN |=== tPROG ===|.held|
  • A packet is a 4-bit header and an 8-bit body on two DDR pins: 3 tCACI, 12 ns (Section 5.7.1, Table 5-9).
  • A data burst owns the DQ bus from its SCE packet to the 50 ns hold after its SCT packet. A new SCE cannot come during another burst (Section 5.9.2).
  • FEMU looks for the first start where DQ is free for the whole burst and the CA bus is free for both packets. Then it books all three.
  • Every other CA phase takes the first idle CA gap after its earliest time.
  • Array data out needs Change Read Column Enhanced (06h-E0h) before SCE, also with one LUN (Section 5.9.6). onfi_tccs_ns gives tCCSR.
  • tCDL (400 ns) runs from the 12h packet to the end of the write preamble. It replaces tADL.
  • A status read is a CA output: the header, tW2R_CA, one byte at tCACO 10 ns (two bits per cycle), then tR2W_CA.
  • The CA bus keeps its own window history and its own low mark, as the channel bus does.
PhaseSequenceMode 26
read command (CA)00h, 5 address packets, 30h, tWB184 ns
program command (CA)80h, 5 address packets, 12h84 ns
tCDL gaptCDL - (SCE + tSCD + tWPRE2)313 ns
write burst (DQ)SCE, tSCD, tWPRE2, page, SCT, tWPSTH_CA149 ns + page
program confirm (CA)10h, tWB112 ns
erase command (CA)60h, 3 address packets, D0h, tWB160 ns
status (CA)70h (78h + 3 packets), tWHR, header, tW2R_CA, 4 tCACO, tR2W_CA241 ns (277 ns)
column select (CA)06h, 5 address packets, E0h84 ns, then tCCSR
read burst (DQ)SCE, tSCR, tRPRE2, page, then SCT ending tRPST_CA after the page, tRPSTH_CA168 ns + page

tRPST_CA is tDQSRE plus half a tRC. FEMU takes tDQSRE at its 25 ns maximum.

A multi-plane program sends 11h after each plane but the last, then waits tPLPBSY. The SCA figures call this busy tDBSY (Sections 5.12.5 to 5.12.7), and ONFI 6.0 gives it no value. So tplpbsy sets it, and 0 takes the 500 ns FEMU default. A value of 0 cannot model a device with no busy at all. Another LUN can use both buses between the planes (Figure 5-34).

The confirm opcodes are 11h and 10h (program), 32h and 30h (read), and D1h and D0h (erase). Each pair has the same length.

Approximations:

  • tCDL ends at the end of the write preamble. With an idle bus the write burst, SCE first, starts 313 ns after the 12h packet.
  • The 10h confirm waits for the end of the hold after SCT.
  • LUNSel and NTO packets, ODT and CA training are not modelled.
  • CA_CE# setup and hold appear only in the status turnaround.

With a parameter page, SCA needs two things:

  • Change Read Column Enhanced (bytes 8-9 bit 6);
  • at most 16 LUNs per target (byte 100), because SCE and SCT carry 4 LUN bits.

Read cache sequence (onfi_cache_read)​

onfi_cache_read=1 reads the pages of one read command that land on one LUN as a read cache sequence (Section 6.15). The array reads the next page while the bus moves the current one.

00h addr 30h read page 0 into the data register (tR)
31h copy page i to the page register, start page i + 1
(the next page of the same block and plane)
00h addr 31h the same, for any other page of the LUN
3Fh copy the last page, start nothing
each copy keeps SR[6] busy for tRCBSY after the array read ends

array |== tR 0 ==|cp|== tR 1 ==|cp|== tR 2 ==|cp|
bus st 31h --^ |st out 0|31h |st out 1|3Fh |st out 2|
  • A sequence sends the command of the next page before the data out of the current page. So the media layer needs every page of the sequence at once. The bbssd read path holds the media reads of one command, groups them by LUN in LPN order, and gives each group to nand_media_read_seq().
  • A group of one page is a plain read. Reads of two commands never join one sequence. With 8 channels of 8 LUNs, a command needs more than 64 pages before a LUN gets two of them.
  • Each copy ends at max(trigger, array end) + tRCBSY. The next array read starts there. trcbsy sets tRCBSY; 0 takes 3000 ns (Table 4-30 typical).
  • With several LUNs, a 78h status selects the LUN before each 31h or 3Fh. Each page reads its status, then its data out, after its copy.
  • The LUN is held from the 30h command to the last data out.
  • Garbage collection reads, mapping reads and copyback stay plain reads.
  • A KV namespace reads a value one page at a time, so it keeps plain reads. Realize warns, and query-femu reports cache-read false for it.
  • The read counts and the wear checks of the held pages run in LPN order, as without the sequence.

The period per page on one LUN, with no other traffic:

plain read = cmd + tR + status + data out
in sequence = max(tR, data out + status + 31h + select) + tRCBSY

A sequence is faster only when the bus part of a plain read is longer than tRCBSY. tRCBSY as a copy time after the array read is a model choice. ONFI gives it only as a typical busy time.

FEMU reads a group as plain reads when the sequence would end later, as a controller that knows the timing parameters would choose:

nand_media_read_seq(group of one LUN)
|
+-- one page, or no cache read --> plain read
+-- cost on an idle LUN and idle buses (closed form, no booking):
| PLAIN = sum (cmd + tR(i) + status + data out(i))
| SEQ = the sequence recurrence above, with every phase at its
| earliest time
+-- SEQ <= PLAIN --> one read cache sequence
+-- SEQ > PLAIN --> plain reads of the group, in LPN order
Mode, page8 pages, one LUN, plainas a sequenceFEMU uses
0, 4 KiB817240 ns560590 nsthe sequence
26, 4 KiB332328 ns345912 nsplain reads
  • The choice uses only the timing fields and the pages of the group. These give tR with the ECC time of each page, the data out with its partial transfer, and the 31h or full read command of each step.
  • The choice does not look at the bus, so the same group always gets the same choice.
  • With other traffic both forms can end later than their idle cost. The choice does not change then.
  • The plain reads stay inside the LUN group. With several LUNs the FTL still books group by group, so the order differs from a device with onfi_cache_read off.
  • The choice needs the ONFI order and the page register hold (the profile sets both) and LUN contention, as bbssd uses.

Realize refuses onfi_cache_read with these settings:

  • mp_read: FEMU does not model the multi-plane cache read;
  • mapping=dftl: a mapping read can make a page ready after the command start;
  • read_cache_mb: a DRAM hit can come before the NAND read of the page ends;
  • a negative trcbsy.

Parameter page (onfi_param_page)​

onfi_param_page names a file with an ONFI parameter page image (Section 6.7.1, Table 6-4). The page turns the profile on, also with onfi_mode=-1. Then the mode is the highest one that the page lists.

file --> copy 0, 1, 2, ... (256 bytes each)
signature: 2 or more of "ONFI" (6.7.1.1)
CRC-16: poly 8005h, init 4F4Eh, bytes 0-253 (6.7.1.42)
--> first good copy
--> checks against the device --> mode, spare bytes, address cycles
BytesFieldUse
4-5revisionbit 15 (ONFI 6.0) must be set
6-7featuresbit 9 (NV-LPDDR4) must be set and bit 0 (x16) clear; see the rules below
8-9optional commandsRead Status Enhanced, Change Read Column Enhanced and Copyback, when the device needs them
80-83, 84-85data and spare bytes per pagedata must equal secsz * secs_per_pg; spare bytes go on the bus
92-95, 96-99pages per block, blocks per LUNmust equal pgs_per_blk and blks_per_pl * pls_per_lun
100LUNs per targetmust divide luns_per_ch
101address cyclescolumn cycles (bits 4-7) and row cycles (bits 0-3) set the command lengths
102bits per memory entrymust equal nand_cell_type when both are set
113, 114plane address bits, multi-plane attributespls_per_lun must be 1 << bits; several planes need the concurrent form
118-121NV-LPDDR4 modesbit 0 is modes 0 to 3, bit k is mode k + 3

Realize refuses the page in these cases. The error names the field and both values.

  • The file is missing or shorter than 256 bytes, or no copy has a signature and a good CRC.
  • The data bytes per page are not a power of two of 512 or more.
  • The pages per block are not a multiple of 32.
  • A geometry field differs from the device, or an address field is too short for it (Section 3.1).
  • onfi_spare_bytes is set and differs from bytes 84-85.
  • onfi_mode is set and the page does not list it, or the page lists no mode.
  • The listed modes are not one range, as Section 6.7.1.30 asks.
  • Byte 102 (bits per memory entry) is 0.
  • Several LUNs per target (byte 100) without multiple LUN operations (bytes 6-7 bit 1) and the program page register clear enhancement (bit 8). The held page register model needs both.
  • Several LUNs per channel without Read Status Enhanced (bytes 8-9 bit 3), or several LUNs or planes without Change Read Column Enhanced (bit 6).
  • Several planes without multi-plane program and erase (bytes 6-7 bit 3), which garbage collection uses, or with overlapped multi-plane only (byte 114 bit 0). FEMU models the concurrent form.
  • mp_read without multi-plane read (bytes 6-7 bit 6), or gc_copyback without Copyback (bytes 8-9 bit 4).

A page can clear odd to even copyback (bytes 6-7 bit 4). Then gc_copyback moves a page on the die only when the source and the destination page have the same parity. Other moves go off the die. The page gives no array times in ONFI 6.0, so pg_rd_lat, pg_wr_lat and blk_er_lat still set them. Pages of older ONFI revisions are refused.

QMP query-femu reports the phases a namespace charges in its media member, for every mode with a media layer, with the profile on or off. The values come after the fallbacks: a command phase of 0 takes cmd_addr_ns, and with the bus off every bus phase is 0.

Plane parallelism and multi-plane operations​

The array rule decides which operations exclude each other:

LUN rule (BBSSD, CSD, KV) plane rule (ZNS)
LUN 0: [pl0 op][pl1 op][pl0 op] LUN 0 pl0: [op][op]
one at a time per LUN pl1: [op][op] in parallel

Under the LUN rule, all planes of a LUN are busy together, so pls_per_lun > 1 adds capacity and line width but no single-operation parallelism, except for the batched operations below.

nand_media_multiplane() runs one operation on several planes of a LUN:

per plane i: cmd/addr on bus (+ data-in for a program); tplXbsy between planes
array: start = max(bus end, rule of every plane); done = start + array time
read: status, then per plane cmd/addr + data-out (booked windows)
program/erase: status (if set)
OperationBatched across planesCaller
Erase of a line's block on every plane of a LUNyes, one erase time plus tplebsy per extra planeBBSSD GC (bbssd/ftl-line-gc.c), FDP GC (bbssd/ftl-fdp.c), KV reclaim (kvssd/kvssd-ftl.c)
Host programs, and buffer write-backwith mp_program: pages of the same block and page on distinct planes of a LUN, one program time plus tplpbsy per extra planessd_program_lpn() in bbssd/ftl-datapath.c
Host readswith mp_read: as above, one read time plus tplrbsy per extra planessd_read() in bbssd/ftl-datapath.c
FDP reads and writes, KV writesno, one operation per pagebbssd/ftl-fdp.c, kvssd/kvssd-ftl.c
GC page movesnogc_read_page(), gc_write_page()
ZNS zone resetno batching needed: each plane has its own rule, so per-plane erases overlapzns_zone_reset()
Copyback (nand_media_copyback())no: one read and one program on one die, no data phase, no ECC adder (it skips the controller's ECC)BBSSD line GC and FDP GC with gc_copyback (bbssd/ftl-line-gc.c, bbssd/ftl-fdp.c)

trcbsy applies only to the read cache sequence of onfi_cache_read (see the ONFI 6.0 profile). Without it, a value warns at realize. With the defaults (no bus, no tplebsy) a two-plane erase takes exactly one erase time.

Multi-plane program and read (BBSSD, CSD)​

mp_program and mp_read are off by default. They have no effect with one plane per LUN, in other modes, or under FDP, whose reads and writes keep their single-page timing; each setting warns at realize in those cases. By default the write pointer walks channel, LUN, plane, page, so the pages of one block and page on the planes of a LUN arrive nchs * luns_per_ch pages apart. With plane_first it walks plane, channel, LUN, page, and those pages arrive in a row. plane_first follows the same rules and warnings as mp_program. The datapath holds each LUN's pages in a batch and charges the batch through ssd_advance_status_multiplane() when one of these happens:

  • the batch has one page on every plane;
  • the next page for that LUN is in another block or page, or on a plane the batch already has;
  • the command ends (write, Write Zeroes, buffer write-back, read);
  • garbage collection or a mapping merge is about to run.

A batch of one page is an ordinary single-page operation. The media layer counts a batch as one command, so the read and program command counts drop while host, NAND, relocated-page and erase counts stay the same. Garbage collection moves pages one at a time.

tplpbsy and tplrbsy are durations: the busy time between the planes of a multi-plane program or read. Each warns at realize when set without its enable, and realize refuses a negative value.

Program and erase suspend​

Program suspend on one LUN: with pe_suspend=1 two reads go ahead of a running program, the first pays tsusp_ns, and the program finishes later by that overhead plus both reads.

Figure: Program suspend on one LUN: with pe_suspend=1 two reads go ahead of a running program, the first pays tsusp_ns, and the program finishes later by that overhead plus both reads.

With pe_suspend (BBSSD, CSD, KV) or zns_pe_suspend (ZNS), a read that finds its LUN (ZNS: plane) running a program or erase goes ahead of it. The layer keeps, per position, pe_end (when the newest program or erase there ends) and rd_end (when the newest read there ends). suspend_read_start() decides:

read arrives at t, rule busy until B
if t >= B or t >= pe_end: ordinary rule (nothing to suspend)
elif rd_end > t: join the open suspension
start = rd_end, shift = tR
else: open a suspension
start = t + tsusp, shift = tR + tsusp
done = start + tR
pe_end += shift; every rule busy-until > t += shift; rd_end = done

Only programs and erases are suspended: a read that finds the die busy with another read waits for it. Reads that queue behind an open suspension pay tsusp_ns once. A read that arrives after the suspension's reads have ended opens a new one and pays it again.

Worked example, one LUN, pg_wr_lat 200 us, pg_rd_lat 40 us, tsusp_ns 5 us, bus off. Numbers produced by nand_media_op():

time (us) 0 50 55 95 135 150 155 195 330 530
LUN busy |== P ===|ss| R1 | R2 |= P =|ss| R3 |====== P ======|===== W2 =====|

P program arrives 0 runs 0-50, 135-150, 195-330 (suspended twice)
R1 read arrives 50 opens a suspension: starts 50 + 5, done 95
R2 read arrives 60 joins it behind R1: starts 95, done 135, no tsusp
R3 read arrives 150 R1 and R2 are over, so it opens a new one: 155-195
W2 program arrives 160 waits for the LUN until P ends: 330-530
(ss = tsusp_ns)

P's die time: 200 alone; with the reads 200 + (5+40) + 40 + (5+40) = 330

P's own command still completes at 200. Its latency was computed when it was issued and is never revisited. Suspension moves only the die's busy-until time, from 200 to 330, so operations issued afterwards, such as W2, start after 330. A suspended program or erase never makes its own command complete afterwards; it delays the commands that follow.

nand_media_multiplane() does not consult the suspend state, so a read issued through it never suspends; this applies to erases and to the batches of mp_program and mp_read. The state is read and written without a lock, so operations on one position must be serialized. BBSSD and ZNS run them on the single FTL thread. A device with suspend on also leaves the lock-free fast path described in Concurrency.

Read priority​

With read_prio (BBSSD, CSD, KV), a read goes ahead of the programs and erases queued on its LUN. It waits only for the one that runs when it arrives, and for reads that went ahead before it. The layer keeps, per position, up to 16 program and erase windows [start, end), sorted by start. prio_read_start() decides:

read arrives at t, rule busy until B
drop windows with end <= t
rs = max(t, rd_end, end of the windows the full list dropped)
for each window: start < rs (or start == t == rs) -> rs = max(rs, end)
no window starts at or after rs, or rs >= B: ordinary rule
else: done = rs + tR; cur = done
each later window: push it to start at cur if it starts before;
cur = its end (a gap absorbs the push)
every rule busy-until > rs += the push of the last window
rd_end = done
off: [P1 run][P2][P3][R] read waits 3 programs
on: [P1 run][R][P2][P3] read waits P1; P2, P3 slide by tR
on: [R]..gap..[P1] the gap absorbs the read; P1 stays

With pe_suspend on as well, a suspension comes first. Read priority takes the read only when the suspend rule declines it, for example past pe_suspend_max. A slid program keeps the completion time the host was given, as with a suspension. Multi-plane reads keep the ordinary rule.

Realize refuses read_prio with the bus phases (cmd_addr_lat, pg_xfer_lat, ch_xfer_lat, status_lat) and ONFI timing. Their status phases stay booked at the old end of a slid operation. It also refuses gc_copyback, plane_reads, rber_ppb and the dynamic pSLC tier.

Die parity​

Datacenter drives keep one parity page per stripe across dies, so a failed die loses no data. With die_parity (BBSSD, CSD), a stripe is one page per die at a plane and page index of a line. The write pointer steps the channel first, then the LUN, so a program on the last die completes the stripe. The FTL then books a parity program on die (pg + pl) % dies, after parity_xor_ns.

page pg, plane pl: ch0/lun0 ch1/lun0 ... chN/lunM
data data data <- completes the stripe
parity program on die (pg + pl) % dies, + XOR time

The parity program takes die time and counts in WAF (log C0h). It takes no page: the over-provisioning stays as configured. With failed_die, a read of that die reads the same page of every other die in parallel and adds parity_xor_ns. Those reads add no read count to their blocks. The data itself stays in the backend, so a failed die loses nothing in FEMU. failed_die is set at device creation. With ocp=on, OCP event injection 0016h (NAND Die Offline, Feature C0h) takes a die offline while the guest runs. The die is inject_die, and the clear brings it back:

Set Features C0h, NUME=1, type 0016h, NRTDE n
--> after n Read commands complete: die inject_die offline
die_parity on: reads of the die take the stripe, as above
die_parity off: a host read of the die fails (Unrecovered Read
Error) and counts as a media error
Set Features C0h, NUME=0, or simulate-power-loss --> die back

The event fails only host Read, Compare and Verify commands. Programs to an offline die still run. Garbage collection, Copy source reads, translation pages and CSD transfers to device memory still read it. A page in the write buffer, the read cache or a staged wordline still reads.

Log C0h counts the degraded reads as XOR Recovery Count (SMART-5). FEMU refuses the event on a controller whose namespaces live in shared subsystem storage, and with die_parity when failed_die names another die. With rain_parity the event takes the die out of reads only; see RAIN for when it is accepted. query-femu reports parity-write-pages and degraded-reads, and its WAF counts the parity programs.

Two-pass program​

QLC drives program a wordline twice: a coarse pass, then a fine pass after later wordlines. The data of a wordline is not readable from the array until its fine pass ends, so the controller keeps it in its buffer.

With prog_pass2_ns (BBSSD, CSD), the program time of a page is the coarse pass. When a program completes a wordline w, the FTL books the fine pass of wordline w - prog_pass2_lag on the same die with nand_media_hold(). That call books array time only, after the coarse program. A block that fills books the fine passes still due. A read of a wordline before its fine pass returns at once, as a buffer hit, and the die does no work for it.

lag 1, one page per wordline
program: [c0] [c1][f0] [c2][f1] [c3][f2][f3] block full after c3
read wl1 after c1: buffer, no array time
read wl1 after c2: array read, after f1 on the die

Pages per wordline come from nand_cell_type (or cell_pages), as for the program disturb terms. Realize refuses the paths that program or read in batches or through another allocator: mp_program, mp_read, onfi_cache_read, gc_copyback, KV, FDP, the pSLC tier and the hybrid and fast mappings.

ECC read time​

ecc_step_ns adds read time for worn or old data (ecc_extra_ns()):

tiers = pe_cycles / 750 + age_sec / ecc_retention_sec (second term only if set)
tiers = min(tiers, 4)
extra read time = tiers * ecc_step_ns

pe_cycles is the block's erase count, age_sec the time since its line was filled (line.close_time). The constants are NAND_ECC_PE_PER_TIER and NAND_ECC_MAX_TIERS in nand/nand-media.h. A multi-plane read pays the largest extra of its planes. Read-retry as a sequence of separate array reads is not modelled; this adder stands in for it. ZNS has the wear term with zns_ecc_step_ns, and no retention term. OCSSD has no ECC time.

Program disturb terms​

Four opt-in properties add a program disturb sensitivity model to the same adder. They apply to bbssd, CSD, FDP and KV. These modes share the decoder bb_decode_loc() in bbssd/ftl-media.c. The FTL computes extra tiers. ecc_extra_ns() adds them before the shared cap of 4.

bb_decode_loc() (FTL) ecc_extra_ns() (media)
-------------------------------- --------------------------------
pe_cycles = block erase count -----> tiers = pe / 750
age_sec = age of the line -----> + age / ecc_retention_sec
extra_tiers = edge + open -----> + extra_tiers
tiers = min(tiers, 4)
extra = tiers * ecc_step_ns
  • Edge term. A read of one of the bottom pd_edge_wls wordlines of a block pays pd_edge_tiers. A wordline is B contiguous pages. B is the pages per wordline of nand_cell_type (1 SLC to 4 QLC), else the effective cell_pages, else 1. The effective cell_pages is 3 when pgtype_lat is on and cell_pages is unset. The pairing tables give page types, not wordlines, so B is a synthetic grouping.
  • Open term. A read of a block that is only part programmed pays pd_open_tiers. nand_block.wp counts the data pages programmed since the last erase. Invalidation and a stream seal do not change it.
  • Open age. With pd_open_age, the age counts from the first program of the line (line.open_time), not from the line close. Open data ages, and a close does not start the age again. One time per line makes later pages look older than they are.
mark_page_valid(), mark_page_valid_fdp() --> blk->wp++, open_time once
mark_block_free() (erase) --> blk->wp = 0
mark_line_free(), wear retirement, FDP --> open_time = 0
0 < blk->wp < pgs_per_blk --> block part programmed

The tier counts are chosen values for sensitivity studies. No source measurement maps an error rate to a number of tiers. Translation page reads of dftl and the KV index address a placeholder block, so they pay the wear and close-time age only. A copyback needs a source with no ECC cost, so a page with a program disturb term moves off chip. The model does not carry error tiers to the copyback destination.

Program disturb actions​

Two more opt-in properties add the firmware actions of the model. They apply to bbssd and CSD, and to FDP reclaim units. Both are 0 (off) by default, and neither needs ecc_step_ns.

trigger action
--------------------------------------- ----------------------------------
a read of a part-programmed block takes queue the line; the next Write
its read count to pd_open_read_limit seals the frontier that fills it
and rewrites the line
a frontier stays open pd_open_close_sec program pad pages into its free
after its first program pages and close it
  • Open-block read limit. pd_open_read_limit is a read limit for a block with 0 < wp < pgs_per_blk. It must not be above a non-zero read_reclaim_limit. The read path queues the line in the same single slot as read_reclaim_limit and retention_limit_sec. It is checked first, and it takes over a request already queued for the same line. The next Write command serves the slot, as on master. If a write pointer still fills the line, the FTL seals its free pages with no program, as a stream release does. It then rewrites the line with no copyback and counts it in C0h pd_open_reclaims. On master, a request for a line under a write pointer is dropped.
  • Pad and close. pd_open_close_sec limits how long a frontier stays open after its first program (line.open_time, also set for this property). The time is device time, so age_scale speeds it up. When the FTL takes a request, it checks every write pointer: data, hot, stream and stream collection. For each line past the limit, it programs every free page with pad data and closes the line into the victim queue. The data pointer takes a new line at once. The others take one at their next write.
bb_ftl_process_req() (every request, inside the streams lock)
pd_open_close_sec set?
for each write pointer with a line:
age(open_time) >= limit, not at the high watermark, 2 free lines
for each free page: PG_FREE -> PG_INVALID, wp++, ipc++,
one single-plane program at the request time
close the line --> victim queue; C0h pd_pad_pages, pd_pad_closes

A pad is a NAND program with no data and no mapping. It counts in the plane programs and the energy, in the write amplification and in the media bytes. It does not count as a host write. The pads are not added to the latency of the request that finds the old line. They hold the LUNs, so that request and later ones can wait behind them. After the pad the blocks are full, so their reads no longer pay pd_open_tiers.

The check runs only when the FTL takes a request. A device that gets no request pads nothing, as a device that gets no read refreshes nothing for age. The single queue slot is best effort: while one line waits, reads of other lines queue nothing. Write and the write phase of Copy serve the slot. In line mode, Write Zeroes and Flush do not. Under FDP, Write Zeroes without Deallocate serves it too. KV, ZNS, OCSSD and NoSSD namespaces, mapping=fast and mapping=hybrid are refused. The ZNS form of this action, a finish recommended limit or a zone active excursion, is not modelled.

From operations to a completion time​

The per-operation rule​

For every resource the operation needs, it starts no earlier than that resource's busy-until time, and it moves the resource's busy-until time to when it stops using it. With the bus off and the LUN rule this reduces to:

done(op) = max(stime, lun_avail) + array_time(op)
lun_avail = done(op)
latency(op) = done(op) - stime

stime is the command's submission time, stamped by the poller from QEMU_CLOCK_REALTIME (nvme-io.c; a test-only property switches it to the virtual clock). Background work (GC) passes stime = 0, which the wrappers replace with the current time.

The per-command rule​

The FTL issues all of a command's NAND operations with the command's stime and returns the largest latency:

latency(cmd) = max over the command's operations of latency(op)
(and of write buffer, read cache and mapping cache costs)
expire_time = stime + latency(cmd) femu_ftl_thread(), femu.c

Then the poller adds the optional host link and firmware CPU models and queues the request in a priority queue ordered by expire_time. The completion is posted on the first poller sweep at or after it.

Worked example: operations overlapped across LUNs and channels​

Geometry 2 channels by 2 LUNs, 1 plane, flat 40 us read and 200 us program, staged bus with cmd_addr_lat=1000 and pg_xfer_lat=10000. Four operations arrive at time 0 in this order. The numbers are what nand_media_op() returns.

op target arrives
P ch0 LUN0 0 program
R1 ch0 LUN1 0 read
R2 ch0 LUN0 0 read (same LUN as P)
R3 ch1 LUN0 0 read (other channel)

time (us) 0 1 11 12 13 51 52 62 211 251 261
ch0 bus |c|-d--|c |c | . |o1-| . |o2-|
P P R1 R2 R1 . R2
ch0 LUN0 . [===== P tPROG 200 ======================)[R2 tR )
ch0 LUN1 [R1 tR 40)
ch1 bus |c| |o3-|
ch1 LUN0 [R3 tR 40)

done: P 211 R1 62 R2 261 R3 51
  • P: command 0-1, data-in 1-11, array 11-211.
  • R1: its command waits for P's data-in (11-12), its array runs on the idle LUN1 (12-52), its data-out is booked 52-62.
  • R2: command 12-13, then waits for LUN0 until P ends at 211, array 211-251, data-out 251-261.
  • R3: on channel 1, which nothing else uses: 0-1, 1-41, 41-51.

With the bus off, the same four operations end at P 200, R1 40, R2 240, R3 40. With pe_suspend=1 and tsusp_ns=5000, R2 suspends P instead: it starts at 13 + 5 = 18, ends its array at 58, waits for R1's booked window and moves its data at 62-72 (done 72), and the die stays busy until 256. P's own completion stays at 211.

If R1 and R2 were the two pages of one 8 KiB read command, its media latency would be max(62, 261) = 261 us.

Worked example: full command completion​

A host read of two written 4 KiB pages on idle LUNs of the default BBSSD (bus off), with pcie_bandwidth_mbps=4000, pcie_prop_delay_ns=500 and fw_cpu_ns=1000, submitted at time S:

S stime stamped by the poller
S + 40 000 FTL: max(40 000, 40 000) -> expire_time
S + 40 000 .. 42 048 link: 8192 B * 1000 / 4000 MB/s = 2 048 ns on the
device-to-host queue (starts at max(queue free, expire))
S + 42 548 + propagation 500 ns
S + 42 548 .. 43 548 firmware core: max(core free, expire) + 1 000 ns
S + 43 548 expire_time; CQE posted on the first sweep at or after it

The link and firmware queues are shared by every command of the controller, so under load each command also waits for the ones queued before it on those queues.

Concurrency​

  • BBSSD and ZNS call the layer from the single FEMU-FTL-Thread, so their timelines need no lock. KV calls it from the poller and femu-cxl-ssd from its femu-cxl-ftl worker, each under its own device lock.
  • With the LUN rule, no bus and no suspend, nand_media_op() updates next_lun_avail_time with a compare-and-swap loop that gives the same result as a locked read-max-add-store. Any other configuration takes the general path, which calls lock_lun/unlock_lun when the mode provides them; BBSSD and ZNS do not.
  • OCSSD 1.2 with channel timing takes the channel's spinlock (chnl_locks[]) through lock_lun/unlock_lun; without it, and OCSSD 2.0, use the compare-and-swap path.

How garbage collection is charged​

GC operations go through the same timelines as host operations, with stime = 0 (the current time) and type = GC_IO:

  • gc_read_page() charges a read of each valid page, gc_write_page() a program at its new location, and reclaim_line() one multi-plane erase per LUN (bbssd/ftl-line-gc.c). All of a victim's pages are moved before any of its blocks is erased, so the erases follow the last relocation. FDP GC does the same in bbssd/ftl-fdp.c.
  • The returned latency is discarded. GC never adds to a command's latency directly. It makes LUNs and channels busy, and the host operations that need them afterwards wait.
  • Foreground GC runs when the share of used lines has reached gc_thres_pcent_high: at the start of ssd_write(), and before each page that a write, a write buffer destage or a Write Zeroes programs (bbssd/ftl-datapath.c), until the share drops below it. The command's own programs then queue behind the GC operations on the LUNs they share.
  • Read reclaim (read_reclaim_limit, retention_limit_sec) rewrites one queued line through reclaim_line() at the start of a write, charged the same way as GC.
  • Background GC runs one victim after a request finishes in bb_ftl_process_req() (bbssd/ftl.c), once the latency of that request has been computed. It delays the commands that follow.
  • Because GC is stamped with the current time and a host command with its submission time, a command submitted before a GC pass can still wait for it when the FTL thread gets to the command after the pass.
  • GC time can be turned off with 0xEF code 2 (Runtime switches). GC still moves pages and erases blocks; its operations are then not charged at all.
LUN 3 timeline during foreground GC for one write (bus off)
|-- GC read --|-- GC program --| ... |== multi-plane erase ==|-- host program --|
^ write arrives ^ write's done

KV is different. Its reclaim erases and compaction moves are stamped with the triggering command's stime, the command completes when the last of them and of its own operations does (kvssd/kvssd-ftl.c), and 0xEF does not apply to KV, so they cannot be switched off.

ZNS has no device GC. A Zone Reset charges one erase per block of the zone on every plane it spans, with the command's stime, and returns the largest (zns_zone_reset()).

Costs that are not NAND operations​

The FTL adds these to a command's latency alongside its NAND operations:

  • BBSSD write buffer (buffer_size > 0): a write the buffer accepts costs dram_hit_ns, or pg_rd_lat / 16 when that is 0 (1 us when pg_rd_lat is 0 too), plus the programs of any destage it forces. A Flush pays for programming everything buffered (bbssd/ftl.c). A read of a buffered page costs the same DRAM access.
  • BBSSD read cache and DFTL mapping cache: a hit costs a DRAM access; a mapping miss adds a NAND read of the translation page, after a program if the evicted entry is dirty.
  • ZNS write cache: a write only enters the zone's write cache and pays SRAM_WRITE_LATENCY_NS (1 us) per 4 KiB page, summed over the command (zns_write() in zns/zftl.c). It pays program time only when the cache fills, or when it has to evict another zone's cache. A cache holds one stripe: 16 KiB * zns_flash_type * zns_num_plane * zns_num_ch * zns_num_lun. A read of data still in the cache is not mapped yet and costs nothing.

Both are off by default. A read pays the link in nvme_process_cq_cpl() in hw/femu/nvme-io.c, after the media latency is in expire_time. A write that passes the rule below pays it at submit, before the FTL books its programs. The firmware cost is charged in nvme_process_cq_cpl() for every command.

ModelApplies toComputation
Link transfer (pcie_bandwidth_mbps)NVMe Read and Write (opcodes 01h and 02h) in BBSSD, CSD, KV, ZNS and NoSSD; KV store and retrieve share those opcodes and use the bytes moved. Not Zone Append, and not the Open-Channel vector commands, so OCSSD 1.2 never pays ittrans = bytes * 1000 / MBps ns. One queue per direction: writes on pcie_rx_next_avail_time, reads on pcie_tx_next_avail_time. Read: start = max(queue, expire). Write, where the rule below allows it: start = max(queue, submit), and the FTL start moves to the transfer end; other writes as a read. Both: queue = start + trans
Propagation (pcie_prop_delay_ns)sameexpire = queue + delay; does not occupy the queue
Firmware CPU (fw_cpu_ns, fw_read_ns, fw_write_ns, fw_other_ns, fw_cores)Read (fw_read_ns), Write and Zone Append (fw_write_ns), each falling back to fw_cpu_ns when 0; every other I/O command, the Open-Channel vector commands included, pays fw_other_nsfw_cores modelled cores: the command takes the core free soonest, start = max(core, expire), core = start + cost, expire = core

A write's data crosses the link before the media can program it. So the write books its transfer when it passes its checks, and the FTL starts at the transfer end. A read's transfer starts after its NAND time.

The early booking applies only where the FTL books the media later, from the request start time. Every other write keeps the old order.

Write (01h) onOrder
bbssd, FDP or CSD block namespace, buffer_size=0link first
NoSSDlink first
a write buffer (buffer_size > 0), ZNS (zone write cache)old: buffered pages keep no arrival time
KV store, OCSSD 2.0old: nvme_io_cmd() books the media itself
power_loss=onold: the FTL thread checks the command
a controller linked to a CXL SSD, SLM, CP Executeold
write: submit --> link (rx queue) --> FTL programs --> firmware --> CQE
read: submit --> FTL reads --> link (tx queue) --> firmware --> CQE
pcie_write_first=off: write: submit --> FTL programs --> link --> firmware --> CQE

The stage counters of query-femu-stages count from the submit time. The FTL handoff time is real time, so a write's link time shows in the hold stage. The firmware cost is charged at the end of a command, which caps the command rate at about fw_cores per firmware time but does not delay the start of the media operations. The link model is enabled when either link property is non-zero (pcie_enabled in femu.c).

OCSSD timing model​

Open-Channel devices run on the media layer, each with its own NandMedia and the LUN rule (oc12_media_init(), oc20_media_init()):

  • Per chip (flat LUN id ch * num_lun + lun), an operation ends at max(now, busy) + time. Times come from the flash_type table: 1.2 as a table by page type, 2.0 as flat lower-page times (by page type with oc20_page_types).
  • Open-Channel 1.2 with channel timing uses the staged bus with no command, address or status phase: bus_now() for a program's data-in and bus_later() for a read's data-out.
  • A write moves its data over the channel, then programs the chip. A read occupies the chip, then moves its data out. Erase charges each chip in the address list.
  • Open-Channel 1.2 charges a channel transfer only with oc12_channel_timing=on: ch_xfer_lat ns per page, or the table value for flash_type when it is 0, scaled by the sectors used out of lsecs_per_pg. Open-Channel 2.0 charges no channel time and uses page type 0 unless oc20_page_types is on.
  • Open-Channel 1.2 charges one chip operation per NAND page of the address list. Open-Channel 2.0 groups addresses that differ only in the sector field, which spans the whole chunk, so a run of consecutive sectors in one chunk pays one chip operation whatever its length.
  • The latency is max over operations of (done - stime), written straight into expire_time in the poller (oc12_advance_status(), oc20_advance_status()).
  • Open-Channel 1.2 keeps its data-out windows in an unbounded list per channel (policy.bus_res_unbounded), where the other modes cap it at 32.

Runtime switches​

A BBSSD controller accepts the vendor admin command 0xEF (bb_flip() in bbssd/bb.c). The action is in CDW10. For codes 1 to 4 it pauses the pollers, applies the change to every namespace of the controller that has a BBSSD FTL, and resumes them. Controllers of every other mode return Invalid Opcode.

CDW10Constant (bbssd/ftl.h)Effect on timing
1FEMU_ENABLE_GC_DELAYGC operations are charged (default)
2FEMU_DISABLE_GC_DELAYGC operations are not charged
3FEMU_ENABLE_DELAY_EMUFlat read, program and erase set to the built-in 40 us, 200 us and 2 ms, not to the command-line values
4FEMU_DISABLE_DELAY_EMUFlat read, program and erase set to 0
5accounting resetPrints and resets the per-poller completion counters; no timing change
6, 7FEMU_ENABLE_LOG, FEMU_DISABLE_LOGPer-command debug log on or off; no timing change

Codes 3 and 4 refresh only the flat rd_ns, wr_ns and er_ns in the media layer (bb_nand_media_refresh_timing()). With nand_cell_type 1 to 4 the flat times are not used, so the codes have no effect on NAND time. They never change the channel bus phases, pgtype_lat, ECC time or trim_lat_ns. Code 4 makes a write buffer hit cost 1 us; the read cache keeps the hit cost it computed at start. On a controller linked to a femu-cxl-ssd, codes 1 to 4 change the medium's FTL instead, and code 3 restores its read-ns, program-ns and erase-ns.

From the guest:

sudo nvme admin-passthru /dev/nvme0 --opcode=0xef --cdw10=2 # stop charging GC
sudo nvme admin-passthru /dev/nvme0 --opcode=0xef --cdw10=4 # zero NAND times

The vendor command 0xEE (NVME_ADM_CMD_FEMU_DEBUG) sets an Open-Channel controller's NAND times at run time, in nanoseconds: CDW10 upper page read, CDW11 lower page read, CDW12 upper page program, CDW13 lower page program, CDW14 block erase and CDW15 channel transfer per page. The lower page is page type 0 and the upper page the highest page type of flash_type; the centre pages of TLC and QLC keep their table times, and SLC, with one page type, takes the lower page values. The times belong to that controller (FemuCtrl.oc_pg_rd_lat and the fields beside it, filled from the flash_type table by set_latency()), so other devices are not affected. Other modes refuse 0xEE with Invalid Command Opcode, and log 05h does not list it.

sudo nvme admin-passthru /dev/nvme0 --opcode=0xee --cdw10=64000 --cdw11=48000 \
--cdw12=2300000 --cdw13=850000 --cdw14=3000000 --cdw15=52433

Parameters​

The generated property reference is the authority for types, defaults and ranges. This table groups the timing properties and states how they interact.

GroupPropertiesInteractions
Geometrysecsz, secs_per_pg, pgs_per_blk, blks_per_pl, pls_per_lun, luns_per_ch, nchsnchs * luns_per_ch is the number of independent dies. pls_per_lun adds no read or program parallelism under the LUN rule unless mp_program or mp_read is set.
Array timespg_rd_lat, pg_wr_lat, blk_er_latIgnored when nand_cell_type is 1 to 4. 0xEF codes 3 and 4 overwrite them. With dram_hit_ns at 0, pg_rd_lat / 16 (1 ns at least) is also the cost of a write buffer hit.
Cell typenand_cell_type, cell_pages, pgtype_latpgtype_lat applies only with nand_cell_type=0. nand_cell_type limits pgs_per_blk to 512.
Bus phasescmd_addr_lat, pg_xfer_lat, ch_xfer_lat, status_latAny non-zero value turns the bus on. pg_xfer_lat takes precedence over ch_xfer_lat.
Multi-planetplebsy, mp_program, tplpbsy, mp_read, tplrbsy, plane_first (trcbsy applies only with onfi_cache_read)Only with pls_per_lun > 1. tplebsy applies to the multi-plane erase of GC; tplpbsy only with mp_program; tplrbsy only with mp_read.
Suspendpe_suspend, tsusp_nstsusp_ns matters only with pe_suspend.
ECCecc_step_ns, ecc_retention_sececc_retention_sec matters only with ecc_step_ns.
Program disturbpd_edge_wls, pd_edge_tiers, pd_open_tiers, pd_open_ageAll need ecc_step_ns; pd_open_age also needs ecc_retention_sec.
Other FTL coststrim_lat_nsCharged per Dataset Management range by the FTL, not by the media layer, also under FDP.
ZNSzns_num_ch, zns_num_lun, zns_num_plane, zns_num_blk, zns_flash_type, zns_pg_rd_lat, zns_pg_wr_lat, zns_blk_er_lat, zns_cmd_addr_lat, zns_pg_xfer_lat, zns_status_lat, zns_pe_suspend, zns_tsusp_nsThe zns_*_lat overrides apply to the configured cell type only. zns_num_plane scales the program unit.
OCSSDflash_type, oc12_channel_timing, ch_xfer_lat, lnum_ch, lnum_lun, lnum_pln, lsecs_per_pg, lpgs_per_blkch_xfer_lat is shared with BBSSD. lnum_ch is at most 32 and lnum_ch * lnum_lun at most 128.
Host link and firmwarepcie_bandwidth_mbps, pcie_prop_delay_ns, fw_cpu_nsApplied after the media time, to the opcodes listed in Host link and firmware CPU.
femu-cxl-ssdchannels, luns-per-channel, pages-per-block, blocks-per-plane, read-ns, program-ns, erase-ns, channel-nsOne plane per LUN; feeds the same media layer.

A BBSSD configuration with the bus and suspend on:

-device femu,devsz_mb=1024,femu_mode=1,cmd_addr_lat=1000,pg_xfer_lat=10000,pe_suspend=1,tsusp_ns=5000

A BBSSD with TLC page-type timing:

-device femu,devsz_mb=1024,femu_mode=1,nand_cell_type=3

Statistics​

The media layer keeps no counters of its own. What it charges shows up in:

  • Latency seen by the guest: fio completion latency. This is the primary output of the model.
  • Vendor log page C0h (BBSSD, CSD): host, NAND and GC page counts, write amplification, buffer hits. See log pages and counters.
  • Per-block counters used by the model: nand_block.erase_cnt (ECC tiers, SMART Percentage Used) and nand_block.read_cnt (read reclaim), updated in ssd_advance_status() and on erase.
  • femu-cxl-ssd: the QOM counter media-time-ns, the sum of the latencies the FTL returned. See runtime properties.
  • Debug: nand_lun.gc_endtime records when the last GC operation on a LUN ends; nothing reports it.

Calibrating against a real device​

  1. Take array times from the datasheet. pg_rd_lat is tR, pg_wr_lat is tPROG, blk_er_lat is tBERS. For a part with different lower and upper page times, either use nand_cell_type if one of the built-in tables is close, or set the mean tPROG and turn on pgtype_lat with cell_pages equal to bits per cell.
  2. Derive the bus phases from the interface rate. pg_xfer_lat is the page size divided by the interface rate: a 16 KiB page on an 8-bit bus at 1600 MT/s takes 16384 / 1.6e9 s, about 10.2 us. cmd_addr_lat and status_lat are the command, address and status cycle times, usually well under 1 us.
  3. Match the parallelism. Set nchs and luns_per_ch to the device's channels and dies per channel, and secs_per_pg * secsz to its page size.
  4. Check the unloaded latency. Write the range first (a read of an unwritten page costs nothing), then run a queue depth 1 4 KiB random read. The median completion latency should be about tR plus the data-out and status phases plus guest and poller overhead. Measure that overhead once with flat timing, no bus phases, no caches and 0xEF code 4, and subtract it.
  5. Check the saturated throughput. A random read at high queue depth should approach nchs * luns_per_ch / tR operations per second when the bus is not the limit, or nchs / (pg_xfer_lat + cmd_addr_lat + status_lat) when it is. Compare with the real device at the same queue depth.
  6. Add the host side. If the real device's peak is below what the media allows, set pcie_bandwidth_mbps to its link rate and fw_cpu_ns to the inverse of its peak command rate.
  7. Check GC separately. Precondition the device, run sustained random writes, and compare the write amplification in log page C0h and the steady-state write bandwidth. Toggle 0xEF codes 1 and 2 to see how much of the gap GC time explains.

Give each poller and the FTL thread its own host core while measuring; performance tuning and measuring cover the setup.

Validation status​

  • Unit tests: hw/femu/tests/unit/test-nand-media.c checks the media layer's arithmetic without QEMU: ECC tiers, the staged bus and its booked windows, the plane rule with the bus, multi-plane erase against serial erases, copyback, and program/erase suspend. Run it with make -C hw/femu/tests check.
  • qtests: hw/femu/tests/qtest/femu-test.c covers the OCSSD 1.2 channel model (oc12-channel-timing, oc12-channel-gap, oc12-channel-default, oc12-channel-off, oc12-ppa-timing, oc12-flash-type, oc12-page-count, oc12-transfer-cost, and the exact per-command times of oc12-trace-on and oc12-trace-off), the warning for the timing properties that have no effect (ignored-props), the 0xEF flips on a linked femu-cxl-ssd (cxl-nvme-flip), the two example configurations on this page (doc-examples), and program/erase suspend on a BBSSD, CSD, KV and ZNS device (pe-suspend-<mode>-on and pe-suspend-<mode>-off): a read that arrives during a program completes before the program only when suspend is on.
  • The firmware CPU model per opcode and per core (fw-cost-read, fw-cost-other, fw-cost-cores1, fw-cost-cores2).
  • Not covered by automated tests: other BBSSD and ZNS timing inside QEMU, the host link, the 0xEF flips on a plain BBSSD controller, and end-to-end guest latency against a reference device. The built-in MLC table is a profile of real parts; the other tables are from published papers, as cited in the headers. Check numbers that matter to you with the calibration steps above.
make -C hw/femu/tests check

What is not modelled​

  • Read-retry as repeated array reads, soft-decision decoding, and read disturb errors. The ECC adder is a fixed time per tier, and data is never corrupted.
  • Multi-plane reads and programs outside BBSSD and CSD. mp_read and mp_program batch host pages there; they are off by default and refused with FDP. Line erases are always batched.
  • Cache read outside the ONFI profile, and cache program. onfi_cache_read models the read cache sequence of one command (Read cache sequence).
  • Copyback outside BBSSD line GC and FDP GC. Paced collection, read reclaim and retention refresh move pages off-chip.
  • Bit errors of a copyback. A real copyback skips ECC, so its bit errors are not corrected and can add up; FEMU models no bit errors, so a copyback changes only timing.
  • Program suspend for programs, or erase suspend by another erase. Only reads suspend.
  • Per-die power or thermal limits, and temperature-dependent NAND physics. The package thermal model (thermal_tau_ms, thermal_r) reports a temperature; with throttle=on it stretches the array time of bbssd and CSD media as a device-wide pacing factor. Energy is charged when an operation is booked, and the policy stops while the VM pauses.
  • Plane-level parallelism for BBSSD and CSD programs, and for KV reads and programs (the rule is per LUN). plane_reads=on lets BBSSD and CSD reads on different planes overlap, with the plain channel model only. The channel bus for OCSSD 2.0.
  • PLC outside ZNS. ZNS accepts PLC with times you supply; BBSSD resets nand_cell_type=5 to 0 and OCSSD refuses flash_type=5.
  • Queueing inside the controller other than the link and the single firmware core: there is no DRAM bandwidth or NAND controller queue model.
  • Timing accuracy below the poller sweep. A completion is posted on the first sweep after it is due, and a poller without its own host core adds delay.

Adding a timing feature​

  1. Decide the resource. If the feature makes an existing resource busy for longer, it is an array_lat() or bus-phase change. If it adds a resource, add an accessor to NandTimelineOps and keep the state in the mode's structures, like the existing busy-until fields.
  2. Add the duration to NandMediaTiming and a switch to NandMediaPolicy in nand/nand-media.h. Default it to 0 or off so that existing configurations keep their exact timing. The layer must not include a controller header or branch on the mode.
  3. Implement it in nand-media.c. Keep the max() ordering of the existing paths. A phase that happens at issue goes through bus_now(); one that happens after the array goes through bus_later().
  4. Wire the property. Add it with DEFINE_PROP_* in femu.c and a description in femu-props.c, copy it into ssdparams in ssd_init_params() (bbssd/ftl-geom.c), validate it in bb_check_geometry(), and set the media config in bb_nand_media_init(). For ZNS, the same in zns_init_params() and zns_nand_media_init().
  5. Test the arithmetic in tests/unit/test-nand-media.c. Include a case that fails if the feature is ignored, so the test cannot pass with the feature off.
  6. Update the docs. Regenerate the property reference with gen-property-docs.py, add the parameter to this page and to the timing model concept page, and add a CHANGELOG entry. Keeping the documentation correct lists the checks.

Source map​

FileWhat it holds
hw/femu/nand/nand-media.hMedia layer types and API: NandLoc, NandMediaTiming, NandMediaPolicy, NandTimelineOps, nand_media_op(), nand_media_multiplane(), nand_media_copyback()
hw/femu/nand/nand-media.cArray time, rules, bus booking (bus_now(), bus_later()), suspend, ECC, multi-plane, copyback
hw/femu/nand/nand.h, nand.cCell-type timing tables, page pairing tables, rated P/E cycles
hw/femu/bbssd/ftl-media.cBBSSD adapter: bb_decode_loc(), bb_nand_media_init(), ssd_advance_status(), ssd_advance_status_multiplane(), bb_nand_media_refresh_timing()
hw/femu/bbssd/ftl-geom.cGeometry checks and parameter copy (bb_check_geometry(), ssd_init_params())
hw/femu/bbssd/ftl-datapath.cHost read and write: per-page operations and max latency
hw/femu/bbssd/ftl-line-gc.cWrite pointer order, GC reads, programs and multi-plane erase
hw/femu/bbssd/bb.c0xEF handler (bb_flip(), bb_flip_apply())
hw/femu/zns/zftl.cZNS adapter, write cache flush, zone reset erase
hw/femu/zns/zns.c, zns.hZNS timing values and property overrides (zns_init_params())
hw/femu/ocssd/oc-timing.cOCSSD flash_type times, geometry check, 0xEE
hw/femu/ocssd/oc12.c, oc20.cOCSSD per-command timing (oc12_advance_status(), oc20_advance_status())
hw/femu/femu.cFTL thread: expire_time += latency; timing properties
hw/femu/nvme-io.cstime stamp, host link and firmware CPU models, priority queue and completion
hw/femu/tests/unit/test-nand-media.cMedia layer unit tests