NAND media and timing model
This page is hw/femu/docs/design/nand-timing.md at FEMU d8b29b1e0 (2026-11-16), licensed GPL-2.0-or-later. Send corrections to the FEMU repository.
This chapter describes the component that decides how long a flash operation takes in FEMU. It covers the geometry the model works on, the time it charges for a page read, a page program and a block erase, how operations on the same die, plane or channel wait for each other, and how those times become the completion time of an NVMe command. It also covers the two models that sit after the media: the host link and the controller firmware CPU.
The timing model concept page explains the same model from the user's side. This page is for readers who want the algorithms, the data structures and the source.
Contents
- Purpose and place in FEMU
- Geometry
- Data structures
- Operation timing
- Cell types and page types
- Channel bus
- Plane parallelism and multi-plane operations
- Program and erase suspend
- ECC read time
- From operations to a completion time
- How garbage collection is charged
- Host link and firmware CPU
- OCSSD timing model
- Runtime switches
- Parameters
- Statistics
- Calibrating against a real device
- Validation status
- What is not modelled
- Adding a timing feature
- Source map
Purpose and place in FEMU
FEMU copies a command's data as soon as it parses the command. Timing is computed separately: a model works out when the command would have finished on the emulated device, and the poller holds the completion until the host clock reaches that time. The NAND media model is the part of that computation that charges flash operations.
The media layer, nand_media_op() in hw/femu/nand/nand-media.c, is
the one media engine. BBSSD, CSD, KV, ZNS, the FTL behind a femu-cxl-ssd,
and Open-Channel 1.2 and 2.0 use it. It never includes a controller header.
Each mode decodes its own address into a NandLoc, gives the layer a
configuration, and lends it pointers to its busy-until fields.
hw/femu/ocssd/oc-timing.c keeps the Open-Channel table times, the geometry
check and 0xEE (OCSSD timing model).
guest NVMe command
|
v
+--------------------+ stamps stime, copies data
| poller (nvme-io.c) |------------------------------------------+
+--------------------+ |
| to_ftl ring |
v |
+--------------------+ per mode: page list -> NAND ops |
| FTL | bbssd/ftl-datapath.c, zns/zftl.c, |
| (FTL thread, or | kvssd/kvssd-ftl.c (in the poller) |
| poller for KV) | |
+--------------------+ |
| NandLoc + op + stime |
v |
+--------------------+ array time, rules, bus phases, |
| NAND media layer | suspend, ECC; returns done - stime |
| nand/nand-media.c | |
+--------------------+ |
| per-command latency = max over its operations |
v |
expire_time = stime + latency |
| |
v |
+--------------------+ host link, firmware CPU, then the |
| poller completion | priority queue by expire_time <-------+
+--------------------+
|
v
CQE posted on the first sweep at or after expire_time
| Mode | Engine | Built in | Called from |
|---|---|---|---|
| BBSSD, CSD | media layer | bb_nand_media_init() in bbssd/ftl-media.c | FTL thread |
| KV | media layer, through the BBSSD wrapper | ssd_init() from kvssd/kvssd-ftl.c | poller |
| ZNS | media layer | zns_nand_media_init() in zns/zftl.c | FTL thread |
femu-cxl-ssd | media layer, through the BBSSD wrapper | femu_cxl_start() in cxlssd/cxlssd.c | femu-cxl-ftl worker |
| OCSSD 1.2, 2.0 | media layer | oc12_media_init(), oc20_media_init() in ocssd/ | poller |
| NoSSD | none |
Geometry
Figure: The NAND hierarchy (channel, LUN, plane, block, page, sector), a line as one block index across every plane of every LUN, and the order in which the write pointer hands out pages.
Every mode that models NAND describes it with the same hierarchy. Only the names of the properties and the field widths differ.
device
+-- channel 0 .. nchs-1 one shared bus per channel
+-- LUN 0 .. luns_per_ch-1 a die: runs one array operation at a time
| (one per plane under the ZNS plane rule)
+-- plane 0 .. pls_per_lun-1
+-- block 0 .. blks_per_pl-1 erase unit
+-- page 0 .. pgs_per_blk-1 program and read unit
+-- sector 0 .. secs_per_pg-1 (secsz bytes each)
BBSSD line (superblock) k = block k on every (channel, LUN, plane)
lines = blks_per_pl pages per line = nchs * luns_per_ch * pls_per_lun * pgs_per_blk
Address formats
| Mode | Address | Fields, low bit first | Where |
|---|---|---|---|
| BBSSD, CSD, KV | struct ppa | blk 16, pg 16, sec 8, pl 4, lun 7, ch 12, reserved 1 | bbssd/ftl.h |
| ZNS | struct ppa | spg 2, pg 16, blk 16, fc (LUN) 8, pl 3, ch 7, valid 1, reserved 8 | zns/zns.h |
| OCSSD 1.2 | 64-bit PPA with masks and offsets from the geometry | ch, lun, pln, blk, pg, sec | PPA_* macros in nand/nand.h |
| OCSSD 2.0 | 64-bit LBA with group, parallel unit, chunk, sector fields | group, PU, chunk, sector | OC20_LBA_GET_* in ocssd/ |
A field width is an upper bound on its axis. bb_check_geometry() in
bbssd/ftl-geom.c refuses a count larger than its field can hold, and refuses
a geometry whose total sector count does not fit in a signed 32-bit integer.
zns_check_params() bounds each ZNS axis by its field. oc_timing_geometry_ok()
requires every OCSSD axis to be non-zero, lnum_ch at most 32 and
lnum_ch * lnum_lun at most 128, the sizes of the per-channel and per-LUN
arrays.
How pages land on the geometry
The timing a workload sees depends on where the FTL puts its pages, because the media model only charges what it is asked to.
- BBSSD, CSD, KV allocate from a write pointer that moves channel first,
then LUN, then plane, then page (plane first with
plane_firston bbssd and CSD) (ssd_advance_write_pointer_common()inbbssd/ftl-line-gc.c). Consecutive pages of a large write therefore land on different channels, then different LUNs, and are programmed in parallel. - ZNS buffers writes in a per-zone write cache. A flush programs one
program unit per plane:
zns_flash_typepages of 16 KiB (ZNS_PAGE_SIZE) per plane, charged as one program operation per plane (zns_wc_flush()inzns/zftl.c). A 4 KiB logical page is a sub-page (spg) of a 16 KiB page. - OCSSD takes physical addresses from the host, so the host chooses the parallelism.
Data structures
The media layer's types are in hw/femu/nand/nand-media.h.
| Type | Fields that matter | Role |
|---|---|---|
NandLoc | ch, lun, pl, blk, pg, flash_type, page_type, pe_cycles, age_sec, xfer_secs | One operation's position and the facts timing depends on. Filled by the mode's decoder (bb_decode_loc(), zns_advance_status(), oc12_media_op()). |
NandMediaTiming | rd_ns, wr_ns, er_ns (flat); rd_table_ns, wr_table_ns, er_table_ns (by cell and page type); pgtype_mult; cmd_addr_ns, page_xfer_ns, status_ns; tplpbsy_ns, tplrbsy_ns, tplebsy_ns; ecc_*; tsusp_ns | Every duration. |
NandMediaPolicy | array_contend, channel_mode, pe_suspend, ecc_on_read, use_flat_timing | Which mechanisms are on. |
NandTimelineOps | ch_avail, lun_avail, plane_avail, lock_lun, unlock_lun | Accessors that return pointers into the mode's own busy-until fields. |
NandMedia | cfg, bus_res (per-channel booked windows), susp (per-position suspend state) | One instance per namespace FTL. |
NandOpCompletion | done_ns, latency_ns | Absolute end time, and done_ns - stime. |
The busy-until times ("availability timestamps") live in the mode's structures, not in the media layer:
| Resource | BBSSD, CSD, KV | ZNS | OCSSD |
|---|---|---|---|
| Channel bus | ssd_channel.next_ch_avail_time | zns_ch.next_ch_avail_time | Oc12Ctrl.ch_avail[] (1.2 only) |
| LUN (die) | nand_lun.next_lun_avail_time | zns_fc.next_fc_avail_time (only with zns_lun_contention) | Oc12Ctrl.lun_avail[], Oc20Ctrl.lun_avail[] |
| Plane | not kept | zns_plane.next_plane_avail_time | not kept |
Each mode configures the layer differently:
| Setting | BBSSD, CSD, KV, CXL | ZNS |
|---|---|---|
array_contend | NAND_CONTEND_LUN_ONLY | NAND_CONTEND_PLANE_ONLY; NAND_CONTEND_LUN_AND_PLANE with zns_lun_contention |
channel_mode | NAND_CH_STAGED when any bus phase is non-zero, else NAND_CH_OFF | same rule |
use_flat_timing | true when nand_cell_type=0, false when it is set | false (per-type table; page type 0, or pg % zns_flash_type for reads with zns_page_types) |
pe_suspend | pe_suspend property | zns_pe_suspend property |
ecc_on_read | true; the adder is still 0 unless ecc_step_ns is set | true; the adder is still 0 unless zns_ecc_step_ns is set |
The enum also has NAND_CH_NOOP. No mode selects it today.
Operation timing
Figure: How the media layer times an operation: the phases of a read, program and erase, and four reads issued together showing LUN serialization, a shared channel bus and parallel channels.
Array time
array_lat() picks the time the die spends on the operation:
erase : flat ? er_ns : er_table_ns[flash_type]
read : (flat ? rd_ns : rd_table_ns[flash_type][page_type]) + ecc_extra
program : flat ? wr_ns : wr_table_ns[flash_type][page_type]
if flat and pgtype_lat: * pgtype_mult[page_type] / 1000
Phase sequence of one operation
With the channel bus off (the default), an operation is one array phase:
start = max(stime, rule busy-until)
done = start + array time
rule busy-until = done
With the bus on (NAND_CH_STAGED), each operation also has bus phases.
"Now" phases queue on the bus in arrival order; a read's data-out happens
after the array finishes, so it is booked as a future window
(Channel bus).
READ |cmd/addr|..wait for die..|=== tR array ===|data-out|status|
bus now die bus later bus later
PROGRAM |cmd/addr|data-in|..wait for die..|======= tPROG array =======|
bus now bus now die (done = array end)
ERASE |cmd/addr|..wait for die..|=========== tBERS array ===========|status|
bus now die bus later
A program completes when its array phase ends. A read completes after its
status phase, an erase after its status phase. A zero-length phase touches
nothing, so a bus with only pg_xfer_lat set has no command or status cost.
Default times
| Mode | Read | Program | Erase | Source |
|---|---|---|---|---|
BBSSD, CSD, KV flat (nand_cell_type=0) | pg_rd_lat 40 us | pg_wr_lat 200 us | blk_er_lat 2 ms | property defaults in femu.c |
femu-cxl-ssd | read-ns 40 us | program-ns 200 us | erase-ns 2 ms | cxlssd/qemu-adapter.c |
| ZNS SLC | 4 us | 75 us | 2 ms | zns/zns.h, erase from nand/nand.h |
| ZNS TLC | 32 us | 937.5 us | 3 ms | same |
ZNS QLC (default zns_flash_type) | 85 us | 12.196 ms | 3 ms | same |
| ZNS MLC, PLC | none built in | none built in | none built in | realize refuses them unless all three zns_*_lat are set |
The ZNS program time covers a whole program unit (zns_flash_type pages of
16 KiB on one plane). The header cites the ISSCC papers the figures come from.
Cell types and page types
BBSSD, CSD, KV: nand_cell_type
nand_cell_type | Cell | Read by page type (us) | Program by page type (us) | Erase |
|---|---|---|---|---|
| 0 | flat | pg_rd_lat | pg_wr_lat, optionally scaled by pgtype_lat | blk_er_lat |
| 1 | SLC | 40 | 800 | 2 ms |
| 2 | MLC | lower 48, upper 64 | lower 850, upper 2300 | 3 ms |
| 3 | TLC | 56.5, 77.5, 106 | 820.5, 2225, 5734 | 3 ms |
| 4 | QLC | 59.325, 85.25, 127.2, 169.6 | 861.525, 2447.5, 6880.8, 9174.4 | 3 ms |
Values come from nand/nand.h: MLC from a profile of Micron L95B parts, TLC
from SimpleSSD, QLC scaled from the TLC values. Any other value is reset to 0
with a message (ssd_init() in bbssd/ftl.c); PLC has no table here. With a
cell type set, pg_rd_lat, pg_wr_lat and blk_er_lat no longer set NAND
times (pg_rd_lat still sets the write buffer and read cache hit cost), and
pgs_per_blk must be at most 512, the size of the page-type tables.
The page type of page pg in a block comes from the pairing tables
mlc_tbl, tlc_tbl and qlc_tbl, built by init_nand_flash() in
nand/nand.c; slc_tbl is all zeros. They follow a shadow programming order: the first pages of
a block are lower pages, then consecutive pairs of pages cycle through the
page types from lower to upper.
BBSSD, CSD, KV: pgtype_lat and cell_pages
With nand_cell_type=0 and pgtype_lat non-zero, the flat program time is
multiplied by a factor that depends on the page's position in its wordline.
The page type is pg % cell_pages; cell_pages 0 is set to 3 when
pgtype_lat is on (ssd_init()). The factors, in thousandths, are in
bb_nand_media_init():
cell_pages | Page type 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| 1 | 1000 | ||||
| 2 | 600 | 1400 | |||
| 3 | 600 | 1000 | 1400 | ||
| 4 | 500 | 850 | 1150 | 1500 | |
| 5 | 400 | 750 | 1000 | 1250 | 1600 |
Each row averages to 1000, so the mean program time stays pg_wr_lat when
pgs_per_blk is a multiple of cell_pages.
Reads and erases are not scaled.
ZNS and OCSSD
ZNS keeps one read, program and erase time per flash type. It uses page type
0, except for reads with zns_page_types (see ZNS). Each zns_*_lat property replaces the built-in value for the
configured zns_flash_type only. OCSSD uses the nand_cell_type tables
above, selected by flash_type; Open-Channel 1.2 looks up the page type of
each page. Open-Channel 2.0 uses page type 0. With oc20_page_types=on a
chip operation covers one NAND page and uses its page type.
Channel bus
The bus is modelled only when a phase has a duration: cmd_addr_lat,
pg_xfer_lat (or ch_xfer_lat when pg_xfer_lat is 0) or status_lat for
BBSSD, CSD and KV; zns_cmd_addr_lat, zns_pg_xfer_lat or zns_status_lat
for ZNS. Otherwise transfers take no time and channels never contend.
Two kinds of phase share a channel:
bus_now(): command/address, a program's data-in. It starts atmax(earliest, ch_avail), moves past any booked window it would overlap, and advancesch_availto its end.bus_later(): a read's data-out and status, which happen after the array finishes. It starts no earlier thanch_availand is booked as a window[start, end)inbus_res[ch]and does not advancech_avail, so another die's "now" phase can use the bus before the window opens.
Windows that ended before the current operation's stime are pruned. A
channel holds at most NAND_BUS_RES_MAX (32) windows; when it is full, the
data-out is charged FIFO from ch_avail instead. With
policy.bus_res_unbounded (OCSSD 1.2) the list grows and nothing falls back.
A data phase moves one page in page_xfer_ns. When the mode sets
secs_per_page and the operation's NandLoc.xfer_secs, it moves only those
sectors and takes DIV_ROUND_UP(page_xfer_ns * xfer_secs, secs_per_page),
as OCSSD 1.2 charges a partial page.
channel 0 bus, cmd/addr 1 us, data 10 us (times in us)
0 1 11 12 13 52 62
|P c|P data-in|R1c|R2c|.... idle .....|R1 out |
^ booked window, R1's array ended at 52
a phase that arrives at 20 fits in the idle gap; one that would overlap
[52,62) moves to 62
ONFI 6.0 profile (BBSSD, CSD, KV)
onfi_mode (0 to 26) sets every channel bus phase from an ONFI 6.0
NV-LPDDR4 timing mode. ONFI 6.0 gives no array times, so pg_rd_lat,
pg_wr_lat, blk_er_lat and nand_cell_type still set tR, tPROG and
tBERS. The default, -1, turns the profile off and changes nothing.
onfi_mode, onfi_mts --> tDSC --> page_xfer = bytes * tDSC / 2 (x8, DDR)
Table 4-34 constants --> per-op command, status and burst phases
pg_rd_lat, pg_wr_lat, blk_er_lat ------------------> array times (unchanged)
read |00h addr 30h tWB|..... tR .....|70h status|00h pre| page |post|
program |80h addr tADL|edges| page |10h tWB|...... tPROG ......|70h status|
erase |60h addr D0h tWB|.............. tBERS ..............|70h status|
^ sent only when the LUN is ready (RDY) LUN held to here ^
The profile derives these phases (nand/onfi.c). Each one is a sum in ps,
rounded up to whole ns once. The table shows mode 26 with 2 column and 3
row address cycles.
| Phase | Sequence | Mode 26 |
|---|---|---|
| read command | 00h, 5 address cycles, 30h, tWB | 275 ns |
| program command | 80h, 5 address cycles, tADL | 562 ns |
| data in edges | tWPRE2 + tWPST2 + tWPSTH2 | 57 ns |
| program confirm | 10h, tWB | 125 ns |
| erase command | 60h, 3 address cycles, D0h, tWB | 225 ns |
| status | 70h (78h + 3 cycles with several LUNs), tWHR, tRPRE2, one byte, tRPSTH, tRHW | 246 ns (321 ns) |
| data out lead | 00h, tRPRE2; with several LUNs 06h, 5 cycles, E0h, tCCS, tRPRE2 | 50 ns (200 ns) |
| data out tail | tRPSTH, tRHW | 115 ns |
| page of 4 KiB | 4096 bytes at 0.417 ns per 2 bytes | 855 ns |
onfi_mtsreplaces the mode's rate for the page only:ceil(bytes * 1000 / mts)ns. The status byte keeps the mode's tDSC.onfi_spare_bytesadds spare bytes to each page on the bus.onfi_tccs_nsgives tCCS. ONFI 6.0 has no value, so 0 charges nothing.- A
tplpbsy,tplrbsyortplebsyleft at 0 takes 500 ns, the Table 4-30 typical value. - A read sends its status before its data out (Section 6.14). A program, erase or copyback ends at its status.
- An op sends its command only when its LUN is ready, because the page register is in use until then. The LUN stays busy to the end of the read's data out, or of the program, erase or copyback status.
- A command that waits for a busy LUN books a future window, so an op on an idle LUN can use the bus first.
- Op start times are not monotonic. A copy's write starts at the end of its reads, and garbage collection starts at the wall clock. An op with an older start time can still use an idle bus gap before the latest start time on its channel.
latest start time on the channel: high
dropped | kept history (ends <= high) | future windows
[=] [==] | [==] [=] [===] ... [==] | [====] [==]
low high
an op with start t >= low: can use any idle gap after t
an op with start t < low: starts no earlier than low (the fallback)
- At the start of each op, a grown list drops old windows. When more than
1024 windows end before
highin time, it keeps the newest 512. TheNAND_BUS_HISTORY_MAXvalue sets this cap. Thenlowmoves to the end of the newest dropped window, and the op can add a few windows of its own. - A window that ends before
ch_availalso goes, because no phase can start before it. A bounded 32-entry list has no room for history, so it keepslowathigh. - An op that starts at or after
lowgets the gaps it would get from a list that never drops a window. Its LUN andch_availcan still delay it. - A phase that happens now (a command to a ready LUN) still starts after
ch_availin arrival order, as without the profile. - Future windows stay until they end, so the list grows with the media ops in flight: one set of windows per queued op.
- A multi-plane program sends every plane's command and data, with the tPLPBSY gaps, as one bus window (Section 3.1.3). A multi-plane read or erase books each later plane command after its gap.
- With several LUNs, a multi-plane program or erase reads one 78h status per plane. With one LUN, one 70h status reads the composite.
Approximations:
- The status and data out phases charge tRPSTH (15 ns) and not tRPST. tRPST is tDQSRE plus half a tRC, and tDQSRE is 1 to 25 ns by device (Table 4-38).
- Warmup cycles are not modelled. Section 4.12 calls them optional, but Section 4.15 note 12 asks for them above 800 MT/s.
- The profile assumes the program page register clear enhancement is on.
- With the profile, the channel keeps every future window it books, so a full list never books the bus from now.
Realize refuses these settings:
onfi_modeoutside -1 to 26, oronfi_mtsabove 100000;- a negative
onfi_mts,onfi_spare_bytesoronfi_tccs_ns; - an odd page size on the bus (
secsz * secs_per_pg + onfi_spare_bytes), because a DDR burst moves bytes in pairs; cmd_addr_lat,pg_xfer_lat,ch_xfer_latorstatus_latwith the profile;- a namespace that is not bbssd, CSD or KV;
cxl_ssd,ns_mgmtor shared namespaces;pe_suspend, because ONFI 6.0 has no suspend and a held page register has no suspend rule.
onfi_mts, onfi_spare_bytes or onfi_tccs_ns without onfi_mode warns.
SCA protocol (onfi_sca)
onfi_sca=1 selects the Separate Command Address protocol (Section 5).
Commands, addresses and status then move to a CA bus. Each channel gets
its own CA bus. The channel bus (DQ) carries only data bursts, so a
command of one LUN can run during the data of another.
read, P = 12 ns CA packet
CA |00h A*5 30h tWB| |70h/78h status|06h A*5 E0h|tCCS|SCE| |SCT|
DQ |SCE|pre page post|SCT|hold|
LUN ..RDY..|=== tR ===|.................................................held|
program
CA |80h A*5 12h| |SCE| |SCT|10h tWB| |status|
DQ |<-- gap --->|SCE|pre page|SCT|hold|
|<------ tCDL ----------->| (to the end of the preamble)
LUN |=== tPROG ===|.held|
- A packet is a 4-bit header and an 8-bit body on two DDR pins: 3 tCACI, 12 ns (Section 5.7.1, Table 5-9).
- A data burst owns the DQ bus from its SCE packet to the 50 ns hold after its SCT packet. A new SCE cannot come during another burst (Section 5.9.2).
- FEMU looks for the first start where DQ is free for the whole burst and the CA bus is free for both packets. Then it books all three.
- Every other CA phase takes the first idle CA gap after its earliest time.
- Array data out needs Change Read Column Enhanced (06h-E0h) before SCE,
also with one LUN (Section 5.9.6).
onfi_tccs_nsgives tCCSR. - tCDL (400 ns) runs from the 12h packet to the end of the write preamble. It replaces tADL.
- A status read is a CA output: the header, tW2R_CA, one byte at tCACO 10 ns (two bits per cycle), then tR2W_CA.
- The CA bus keeps its own window history and its own
lowmark, as the channel bus does.
| Phase | Sequence | Mode 26 |
|---|---|---|
| read command (CA) | 00h, 5 address packets, 30h, tWB | 184 ns |
| program command (CA) | 80h, 5 address packets, 12h | 84 ns |
| tCDL gap | tCDL - (SCE + tSCD + tWPRE2) | 313 ns |
| write burst (DQ) | SCE, tSCD, tWPRE2, page, SCT, tWPSTH_CA | 149 ns + page |
| program confirm (CA) | 10h, tWB | 112 ns |
| erase command (CA) | 60h, 3 address packets, D0h, tWB | 160 ns |
| status (CA) | 70h (78h + 3 packets), tWHR, header, tW2R_CA, 4 tCACO, tR2W_CA | 241 ns (277 ns) |
| column select (CA) | 06h, 5 address packets, E0h | 84 ns, then tCCSR |
| read burst (DQ) | SCE, tSCR, tRPRE2, page, then SCT ending tRPST_CA after the page, tRPSTH_CA | 168 ns + page |
tRPST_CA is tDQSRE plus half a tRC. FEMU takes tDQSRE at its 25 ns maximum.
A multi-plane program sends 11h after each plane but the last, then waits
tPLPBSY. The SCA figures call this busy tDBSY (Sections 5.12.5 to
5.12.7), and ONFI 6.0 gives it no value. So tplpbsy sets it, and 0
takes the 500 ns FEMU default. A value of 0 cannot model a device with no
busy at all. Another LUN can use both buses between the planes (Figure
5-34).
The confirm opcodes are 11h and 10h (program), 32h and 30h (read), and D1h and D0h (erase). Each pair has the same length.
Approximations:
- tCDL ends at the end of the write preamble. With an idle bus the write burst, SCE first, starts 313 ns after the 12h packet.
- The 10h confirm waits for the end of the hold after SCT.
- LUNSel and NTO packets, ODT and CA training are not modelled.
- CA_CE# setup and hold appear only in the status turnaround.
With a parameter page, SCA needs two things:
- Change Read Column Enhanced (bytes 8-9 bit 6);
- at most 16 LUNs per target (byte 100), because SCE and SCT carry 4 LUN bits.
Read cache sequence (onfi_cache_read)
onfi_cache_read=1 reads the pages of one read command that land on one
LUN as a read cache sequence (Section 6.15). The array reads the next page
while the bus moves the current one.
00h addr 30h read page 0 into the data register (tR)
31h copy page i to the page register, start page i + 1
(the next page of the same block and plane)
00h addr 31h the same, for any other page of the LUN
3Fh copy the last page, start nothing
each copy keeps SR[6] busy for tRCBSY after the array read ends
array |== tR 0 ==|cp|== tR 1 ==|cp|== tR 2 ==|cp|
bus st 31h --^ |st out 0|31h |st out 1|3Fh |st out 2|
- A sequence sends the command of the next page before the data out of
the current page. So the media layer needs every page of the sequence
at once. The
bbssd read path holds the media reads of one command, groups them by
LUN in LPN order, and gives each group to
nand_media_read_seq(). - A group of one page is a plain read. Reads of two commands never join one sequence. With 8 channels of 8 LUNs, a command needs more than 64 pages before a LUN gets two of them.
- Each copy ends at
max(trigger, array end) + tRCBSY. The next array read starts there.trcbsysets tRCBSY; 0 takes 3000 ns (Table 4-30 typical). - With several LUNs, a 78h status selects the LUN before each 31h or 3Fh. Each page reads its status, then its data out, after its copy.
- The LUN is held from the 30h command to the last data out.
- Garbage collection reads, mapping reads and copyback stay plain reads.
- A KV namespace reads a value one page at a time, so it keeps plain
reads. Realize warns, and
query-femureportscache-readfalse for it. - The read counts and the wear checks of the held pages run in LPN order, as without the sequence.
The period per page on one LUN, with no other traffic:
plain read = cmd + tR + status + data out
in sequence = max(tR, data out + status + 31h + select) + tRCBSY
A sequence is faster only when the bus part of a plain read is longer than tRCBSY. tRCBSY as a copy time after the array read is a model choice. ONFI gives it only as a typical busy time.
FEMU reads a group as plain reads when the sequence would end later, as a controller that knows the timing parameters would choose:
nand_media_read_seq(group of one LUN)
|
+-- one page, or no cache read --> plain read
+-- cost on an idle LUN and idle buses (closed form, no booking):
| PLAIN = sum (cmd + tR(i) + status + data out(i))
| SEQ = the sequence recurrence above, with every phase at its
| earliest time
+-- SEQ <= PLAIN --> one read cache sequence
+-- SEQ > PLAIN --> plain reads of the group, in LPN order
| Mode, page | 8 pages, one LUN, plain | as a sequence | FEMU uses |
|---|---|---|---|
| 0, 4 KiB | 817240 ns | 560590 ns | the sequence |
| 26, 4 KiB | 332328 ns | 345912 ns | plain reads |
- The choice uses only the timing fields and the pages of the group. These give tR with the ECC time of each page, the data out with its partial transfer, and the 31h or full read command of each step.
- The choice does not look at the bus, so the same group always gets the same choice.
- With other traffic both forms can end later than their idle cost. The choice does not change then.
- The plain reads stay inside the LUN group. With several LUNs the FTL
still books group by group, so the order differs from a device with
onfi_cache_readoff. - The choice needs the ONFI order and the page register hold (the profile sets both) and LUN contention, as bbssd uses.
Realize refuses onfi_cache_read with these settings:
mp_read: FEMU does not model the multi-plane cache read;mapping=dftl: a mapping read can make a page ready after the command start;read_cache_mb: a DRAM hit can come before the NAND read of the page ends;- a negative
trcbsy.
Parameter page (onfi_param_page)
onfi_param_page names a file with an ONFI parameter page image (Section
6.7.1, Table 6-4). The page turns the profile on, also with
onfi_mode=-1. Then the mode is the highest one that the page lists.
file --> copy 0, 1, 2, ... (256 bytes each)
signature: 2 or more of "ONFI" (6.7.1.1)
CRC-16: poly 8005h, init 4F4Eh, bytes 0-253 (6.7.1.42)
--> first good copy
--> checks against the device --> mode, spare bytes, address cycles
| Bytes | Field | Use |
|---|---|---|
| 4-5 | revision | bit 15 (ONFI 6.0) must be set |
| 6-7 | features | bit 9 (NV-LPDDR4) must be set and bit 0 (x16) clear; see the rules below |
| 8-9 | optional commands | Read Status Enhanced, Change Read Column Enhanced and Copyback, when the device needs them |
| 80-83, 84-85 | data and spare bytes per page | data must equal secsz * secs_per_pg; spare bytes go on the bus |
| 92-95, 96-99 | pages per block, blocks per LUN | must equal pgs_per_blk and blks_per_pl * pls_per_lun |
| 100 | LUNs per target | must divide luns_per_ch |
| 101 | address cycles | column cycles (bits 4-7) and row cycles (bits 0-3) set the command lengths |
| 102 | bits per memory entry | must equal nand_cell_type when both are set |
| 113, 114 | plane address bits, multi-plane attributes | pls_per_lun must be 1 << bits; several planes need the concurrent form |
| 118-121 | NV-LPDDR4 modes | bit 0 is modes 0 to 3, bit k is mode k + 3 |
Realize refuses the page in these cases. The error names the field and both values.
- The file is missing or shorter than 256 bytes, or no copy has a signature and a good CRC.
- The data bytes per page are not a power of two of 512 or more.
- The pages per block are not a multiple of 32.
- A geometry field differs from the device, or an address field is too short for it (Section 3.1).
onfi_spare_bytesis set and differs from bytes 84-85.onfi_modeis set and the page does not list it, or the page lists no mode.- The listed modes are not one range, as Section 6.7.1.30 asks.
- Byte 102 (bits per memory entry) is 0.
- Several LUNs per target (byte 100) without multiple LUN operations (bytes 6-7 bit 1) and the program page register clear enhancement (bit 8). The held page register model needs both.
- Several LUNs per channel without Read Status Enhanced (bytes 8-9 bit 3), or several LUNs or planes without Change Read Column Enhanced (bit 6).
- Several planes without multi-plane program and erase (bytes 6-7 bit 3), which garbage collection uses, or with overlapped multi-plane only (byte 114 bit 0). FEMU models the concurrent form.
mp_readwithout multi-plane read (bytes 6-7 bit 6), orgc_copybackwithout Copyback (bytes 8-9 bit 4).
A page can clear odd to even copyback (bytes 6-7 bit 4). Then
gc_copyback moves a page on the die only when the source and the
destination page have the same parity. Other moves go off the die. The page gives no array times in
ONFI 6.0, so pg_rd_lat, pg_wr_lat and blk_er_lat still set them.
Pages of older ONFI revisions are refused.
QMP query-femu reports the
phases a namespace charges in its media member, for every mode with a
media layer, with the profile on or off. The values come after the
fallbacks: a command phase of 0 takes cmd_addr_ns, and with the bus off
every bus phase is 0.
Plane parallelism and multi-plane operations
The array rule decides which operations exclude each other:
LUN rule (BBSSD, CSD, KV) plane rule (ZNS)
LUN 0: [pl0 op][pl1 op][pl0 op] LUN 0 pl0: [op][op]
one at a time per LUN pl1: [op][op] in parallel
Under the LUN rule, all planes of a LUN are busy together, so
pls_per_lun > 1 adds capacity and line width but no single-operation
parallelism, except for the batched operations below.
nand_media_multiplane() runs one operation on several planes of a LUN:
per plane i: cmd/addr on bus (+ data-in for a program); tplXbsy between planes
array: start = max(bus end, rule of every plane); done = start + array time
read: status, then per plane cmd/addr + data-out (booked windows)
program/erase: status (if set)
| Operation | Batched across planes | Caller |
|---|---|---|
| Erase of a line's block on every plane of a LUN | yes, one erase time plus tplebsy per extra plane | BBSSD GC (bbssd/ftl-line-gc.c), FDP GC (bbssd/ftl-fdp.c), KV reclaim (kvssd/kvssd-ftl.c) |
| Host programs, and buffer write-back | with mp_program: pages of the same block and page on distinct planes of a LUN, one program time plus tplpbsy per extra plane | ssd_program_lpn() in bbssd/ftl-datapath.c |
| Host reads | with mp_read: as above, one read time plus tplrbsy per extra plane | ssd_read() in bbssd/ftl-datapath.c |
| FDP reads and writes, KV writes | no, one operation per page | bbssd/ftl-fdp.c, kvssd/kvssd-ftl.c |
| GC page moves | no | gc_read_page(), gc_write_page() |
| ZNS zone reset | no batching needed: each plane has its own rule, so per-plane erases overlap | zns_zone_reset() |
Copyback (nand_media_copyback()) | no: one read and one program on one die, no data phase, no ECC adder (it skips the controller's ECC) | BBSSD line GC and FDP GC with gc_copyback (bbssd/ftl-line-gc.c, bbssd/ftl-fdp.c) |
trcbsy applies only to the read cache sequence of onfi_cache_read
(see the ONFI 6.0 profile). Without it, a value warns at realize. With the
defaults (no bus, no tplebsy) a two-plane erase takes exactly one erase
time.
Multi-plane program and read (BBSSD, CSD)
mp_program and mp_read are off by default. They have no effect with one
plane per LUN, in other modes, or under FDP, whose reads and writes keep
their single-page timing; each setting warns at realize in those cases.
By default the write pointer walks channel, LUN, plane, page, so the pages
of one block and page on the planes of a LUN arrive nchs * luns_per_ch
pages apart. With plane_first it walks plane, channel, LUN, page, and
those pages arrive in a row. plane_first follows the same rules and
warnings as mp_program. The datapath holds each
LUN's pages in a batch and charges the batch through
ssd_advance_status_multiplane() when one of these happens:
- the batch has one page on every plane;
- the next page for that LUN is in another block or page, or on a plane the batch already has;
- the command ends (write, Write Zeroes, buffer write-back, read);
- garbage collection or a mapping merge is about to run.
A batch of one page is an ordinary single-page operation. The media layer counts a batch as one command, so the read and program command counts drop while host, NAND, relocated-page and erase counts stay the same. Garbage collection moves pages one at a time.
tplpbsy and tplrbsy are durations: the busy time between the planes of
a multi-plane program or read. Each warns at realize when set without its
enable, and realize refuses a negative value.
Program and erase suspend
Figure: Program suspend on one LUN: with pe_suspend=1 two reads go ahead of a running program, the first pays tsusp_ns, and the program finishes later by that overhead plus both reads.
With pe_suspend (BBSSD, CSD, KV) or zns_pe_suspend (ZNS), a read that
finds its LUN (ZNS: plane) running a program or erase goes ahead of it. The
layer keeps, per position, pe_end (when the newest program or erase there
ends) and rd_end (when the newest read there ends). suspend_read_start()
decides:
read arrives at t, rule busy until B
if t >= B or t >= pe_end: ordinary rule (nothing to suspend)
elif rd_end > t: join the open suspension
start = rd_end, shift = tR
else: open a suspension
start = t + tsusp, shift = tR + tsusp
done = start + tR
pe_end += shift; every rule busy-until > t += shift; rd_end = done
Only programs and erases are suspended: a read that finds the die busy with
another read waits for it. Reads that queue behind an open suspension pay
tsusp_ns once. A read that arrives after the suspension's reads have ended
opens a new one and pays it again.
Worked example, one LUN, pg_wr_lat 200 us, pg_rd_lat 40 us,
tsusp_ns 5 us, bus off. Numbers produced by nand_media_op():
time (us) 0 50 55 95 135 150 155 195 330 530
LUN busy |== P ===|ss| R1 | R2 |= P =|ss| R3 |====== P ======|===== W2 =====|
P program arrives 0 runs 0-50, 135-150, 195-330 (suspended twice)
R1 read arrives 50 opens a suspension: starts 50 + 5, done 95
R2 read arrives 60 joins it behind R1: starts 95, done 135, no tsusp
R3 read arrives 150 R1 and R2 are over, so it opens a new one: 155-195
W2 program arrives 160 waits for the LUN until P ends: 330-530
(ss = tsusp_ns)
P's die time: 200 alone; with the reads 200 + (5+40) + 40 + (5+40) = 330
P's own command still completes at 200. Its latency was computed when it was issued and is never revisited. Suspension moves only the die's busy-until time, from 200 to 330, so operations issued afterwards, such as W2, start after 330. A suspended program or erase never makes its own command complete afterwards; it delays the commands that follow.
nand_media_multiplane() does not consult the suspend state, so a read
issued through it never suspends; this applies to erases and to the batches
of mp_program and mp_read. The state is
read and written without a lock, so operations on one position must be
serialized. BBSSD and ZNS run them on the single FTL thread. A device
with suspend on also leaves the lock-free fast path described in
Concurrency.
Read priority
With read_prio (BBSSD, CSD, KV), a read goes ahead of the programs and
erases queued on its LUN. It waits only for the one that runs when it
arrives, and for reads that went ahead before it. The layer keeps, per
position, up to 16 program and erase windows [start, end), sorted by
start. prio_read_start() decides:
read arrives at t, rule busy until B
drop windows with end <= t
rs = max(t, rd_end, end of the windows the full list dropped)
for each window: start < rs (or start == t == rs) -> rs = max(rs, end)
no window starts at or after rs, or rs >= B: ordinary rule
else: done = rs + tR; cur = done
each later window: push it to start at cur if it starts before;
cur = its end (a gap absorbs the push)
every rule busy-until > rs += the push of the last window
rd_end = done
off: [P1 run][P2][P3][R] read waits 3 programs
on: [P1 run][R][P2][P3] read waits P1; P2, P3 slide by tR
on: [R]..gap..[P1] the gap absorbs the read; P1 stays
With pe_suspend on as well, a suspension comes first. Read priority
takes the read only when the suspend rule declines it, for example past
pe_suspend_max. A slid program keeps the completion time the host was
given, as with a suspension. Multi-plane reads keep the ordinary rule.
Realize refuses read_prio with the bus phases (cmd_addr_lat,
pg_xfer_lat, ch_xfer_lat, status_lat) and ONFI timing. Their status
phases stay booked at the old end of a slid operation. It also refuses
gc_copyback, plane_reads, rber_ppb and the dynamic pSLC tier.
Die parity
Datacenter drives keep one parity page per stripe across dies, so a
failed die loses no data. With die_parity (BBSSD, CSD), a stripe is one
page per die at a plane and page index of a line. The write pointer steps
the channel first, then the LUN, so a program on the last die completes
the stripe. The FTL then books a parity program on die (pg + pl) % dies,
after parity_xor_ns.
page pg, plane pl: ch0/lun0 ch1/lun0 ... chN/lunM
data data data <- completes the stripe
parity program on die (pg + pl) % dies, + XOR time
The parity program takes die time and counts in WAF (log C0h). It takes
no page: the over-provisioning stays as configured. With failed_die, a
read of that die reads the same page of every other die in parallel and
adds parity_xor_ns. Those reads add no read count to their blocks. The
data itself stays in the backend, so a failed die loses nothing in FEMU.
failed_die is set at device creation. With ocp=on, OCP event
injection 0016h (NAND Die Offline, Feature C0h) takes a die offline while
the guest runs. The die is inject_die, and the clear brings it back:
Set Features C0h, NUME=1, type 0016h, NRTDE n
--> after n Read commands complete: die inject_die offline
die_parity on: reads of the die take the stripe, as above
die_parity off: a host read of the die fails (Unrecovered Read
Error) and counts as a media error
Set Features C0h, NUME=0, or simulate-power-loss --> die back
The event fails only host Read, Compare and Verify commands. Programs to an offline die still run. Garbage collection, Copy source reads, translation pages and CSD transfers to device memory still read it. A page in the write buffer, the read cache or a staged wordline still reads.
Log C0h counts the degraded reads as XOR Recovery Count (SMART-5).
FEMU refuses the event on a controller whose namespaces live in shared
subsystem storage, and with die_parity when failed_die names another
die. With rain_parity the event takes the die out of reads only; see
RAIN for when it is accepted.
query-femu reports parity-write-pages and degraded-reads, and its
WAF counts the parity programs.
Two-pass program
QLC drives program a wordline twice: a coarse pass, then a fine pass after later wordlines. The data of a wordline is not readable from the array until its fine pass ends, so the controller keeps it in its buffer.
With prog_pass2_ns (BBSSD, CSD), the program time of a page is the coarse
pass. When a program completes a wordline w, the FTL books the fine pass
of wordline w - prog_pass2_lag on the same die with nand_media_hold().
That call books array time only, after the coarse program. A block that
fills books the fine passes still due. A read of a wordline before its fine
pass returns at once, as a buffer hit, and the die does no work for it.
lag 1, one page per wordline
program: [c0] [c1][f0] [c2][f1] [c3][f2][f3] block full after c3
read wl1 after c1: buffer, no array time
read wl1 after c2: array read, after f1 on the die
Pages per wordline come from nand_cell_type (or cell_pages), as for
the program disturb terms. Realize refuses the paths that program or read
in batches or through another allocator: mp_program, mp_read,
onfi_cache_read, gc_copyback, KV, FDP, the pSLC tier and the hybrid
and fast mappings.
ECC read time
ecc_step_ns adds read time for worn or old data (ecc_extra_ns()):
tiers = pe_cycles / 750 + age_sec / ecc_retention_sec (second term only if set)
tiers = min(tiers, 4)
extra read time = tiers * ecc_step_ns
pe_cycles is the block's erase count, age_sec the time since its line was
filled (line.close_time). The constants are NAND_ECC_PE_PER_TIER and
NAND_ECC_MAX_TIERS in nand/nand-media.h. A multi-plane read pays the largest
extra of its planes. Read-retry as a sequence of separate array reads is not
modelled; this adder stands in for it. ZNS has the wear term with
zns_ecc_step_ns, and no retention term. OCSSD has no ECC time.
Program disturb terms
Four opt-in properties add a program disturb sensitivity model to the same
adder. They apply to bbssd, CSD, FDP and KV. These modes share the decoder
bb_decode_loc() in bbssd/ftl-media.c. The FTL computes extra tiers.
ecc_extra_ns() adds them before the shared cap of 4.
bb_decode_loc() (FTL) ecc_extra_ns() (media)
-------------------------------- --------------------------------
pe_cycles = block erase count -----> tiers = pe / 750
age_sec = age of the line -----> + age / ecc_retention_sec
extra_tiers = edge + open -----> + extra_tiers
tiers = min(tiers, 4)
extra = tiers * ecc_step_ns
- Edge term. A read of one of the bottom
pd_edge_wlswordlines of a block payspd_edge_tiers. A wordline isBcontiguous pages.Bis the pages per wordline ofnand_cell_type(1 SLC to 4 QLC), else the effectivecell_pages, else 1. The effectivecell_pagesis 3 whenpgtype_latis on andcell_pagesis unset. The pairing tables give page types, not wordlines, soBis a synthetic grouping. - Open term. A read of a block that is only part programmed pays
pd_open_tiers.nand_block.wpcounts the data pages programmed since the last erase. Invalidation and a stream seal do not change it. - Open age. With
pd_open_age, the age counts from the first program of the line (line.open_time), not from the line close. Open data ages, and a close does not start the age again. One time per line makes later pages look older than they are.
mark_page_valid(), mark_page_valid_fdp() --> blk->wp++, open_time once
mark_block_free() (erase) --> blk->wp = 0
mark_line_free(), wear retirement, FDP --> open_time = 0
0 < blk->wp < pgs_per_blk --> block part programmed
The tier counts are chosen values for sensitivity studies. No source
measurement maps an error rate to a number of tiers. Translation page
reads of dftl and the KV index address a placeholder block, so they pay
the wear and close-time age only. A copyback needs a source with no ECC
cost, so a page with a program disturb term moves off chip. The model
does not carry error tiers to the copyback destination.
Program disturb actions
Two more opt-in properties add the firmware actions of the model. They
apply to bbssd and CSD, and to FDP reclaim units. Both are 0 (off) by
default, and neither needs ecc_step_ns.
trigger action
--------------------------------------- ----------------------------------
a read of a part-programmed block takes queue the line; the next Write
its read count to pd_open_read_limit seals the frontier that fills it
and rewrites the line
a frontier stays open pd_open_close_sec program pad pages into its free
after its first program pages and close it
- Open-block read limit.
pd_open_read_limitis a read limit for a block with0 < wp < pgs_per_blk. It must not be above a non-zeroread_reclaim_limit. The read path queues the line in the same single slot asread_reclaim_limitandretention_limit_sec. It is checked first, and it takes over a request already queued for the same line. The next Write command serves the slot, as on master. If a write pointer still fills the line, the FTL seals its free pages with no program, as a stream release does. It then rewrites the line with no copyback and counts it in C0hpd_open_reclaims. On master, a request for a line under a write pointer is dropped. - Pad and close.
pd_open_close_seclimits how long a frontier stays open after its first program (line.open_time, also set for this property). The time is device time, soage_scalespeeds it up. When the FTL takes a request, it checks every write pointer: data, hot, stream and stream collection. For each line past the limit, it programs every free page with pad data and closes the line into the victim queue. The data pointer takes a new line at once. The others take one at their next write.
bb_ftl_process_req() (every request, inside the streams lock)
pd_open_close_sec set?
for each write pointer with a line:
age(open_time) >= limit, not at the high watermark, 2 free lines
for each free page: PG_FREE -> PG_INVALID, wp++, ipc++,
one single-plane program at the request time
close the line --> victim queue; C0h pd_pad_pages, pd_pad_closes
A pad is a NAND program with no data and no mapping. It counts in the
plane programs and the energy, in the write amplification and in the
media bytes. It does not count as a host write. The pads are not added to
the latency of the request that finds the old line. They hold the LUNs,
so that request and later ones can wait behind them. After the pad the
blocks are full, so their reads no longer pay pd_open_tiers.
The check runs only when the FTL takes a request. A device that gets no
request pads nothing, as a device that gets no read refreshes nothing for
age. The single queue slot is best effort: while one line waits, reads of
other lines queue nothing. Write and the write phase of Copy serve the
slot. In line mode, Write Zeroes and Flush do not. Under FDP, Write Zeroes
without Deallocate serves it too.
KV, ZNS, OCSSD and NoSSD namespaces, mapping=fast and mapping=hybrid
are refused. The ZNS form of this action, a finish recommended limit or a
zone active excursion, is not modelled.
From operations to a completion time
The per-operation rule
For every resource the operation needs, it starts no earlier than that resource's busy-until time, and it moves the resource's busy-until time to when it stops using it. With the bus off and the LUN rule this reduces to:
done(op) = max(stime, lun_avail) + array_time(op)
lun_avail = done(op)
latency(op) = done(op) - stime
stime is the command's submission time, stamped by the poller from
QEMU_CLOCK_REALTIME (nvme-io.c; a test-only property switches it to the
virtual clock). Background work (GC) passes stime = 0,
which the wrappers replace with the current time.
The per-command rule
The FTL issues all of a command's NAND operations with the command's stime
and returns the largest latency:
latency(cmd) = max over the command's operations of latency(op)
(and of write buffer, read cache and mapping cache costs)
expire_time = stime + latency(cmd) femu_ftl_thread(), femu.c
Then the poller adds the optional host link and firmware CPU
models and queues the request in a priority queue ordered by expire_time.
The completion is posted on the first poller sweep at or after it.
Worked example: operations overlapped across LUNs and channels
Geometry 2 channels by 2 LUNs, 1 plane, flat 40 us read and 200 us program,
staged bus with cmd_addr_lat=1000 and pg_xfer_lat=10000. Four operations
arrive at time 0 in this order. The numbers are what nand_media_op()
returns.
op target arrives
P ch0 LUN0 0 program
R1 ch0 LUN1 0 read
R2 ch0 LUN0 0 read (same LUN as P)
R3 ch1 LUN0 0 read (other channel)
time (us) 0 1 11 12 13 51 52 62 211 251 261
ch0 bus |c|-d--|c |c | . |o1-| . |o2-|
P P R1 R2 R1 . R2
ch0 LUN0 . [===== P tPROG 200 ======================)[R2 tR )
ch0 LUN1 [R1 tR 40)
ch1 bus |c| |o3-|
ch1 LUN0 [R3 tR 40)
done: P 211 R1 62 R2 261 R3 51
- P: command 0-1, data-in 1-11, array 11-211.
- R1: its command waits for P's data-in (11-12), its array runs on the idle LUN1 (12-52), its data-out is booked 52-62.
- R2: command 12-13, then waits for LUN0 until P ends at 211, array 211-251, data-out 251-261.
- R3: on channel 1, which nothing else uses: 0-1, 1-41, 41-51.
With the bus off, the same four operations end at P 200, R1 40, R2 240,
R3 40. With pe_suspend=1 and tsusp_ns=5000, R2 suspends P instead: it
starts at 13 + 5 = 18, ends its array at 58, waits for R1's booked window and
moves its data at 62-72 (done 72), and the die stays busy until 256. P's
own completion stays at 211.
If R1 and R2 were the two pages of one 8 KiB read command, its media latency would be max(62, 261) = 261 us.
Worked example: full command completion
A host read of two written 4 KiB pages on idle LUNs of the default BBSSD (bus off),
with pcie_bandwidth_mbps=4000, pcie_prop_delay_ns=500 and
fw_cpu_ns=1000, submitted at time S:
S stime stamped by the poller
S + 40 000 FTL: max(40 000, 40 000) -> expire_time
S + 40 000 .. 42 048 link: 8192 B * 1000 / 4000 MB/s = 2 048 ns on the
device-to-host queue (starts at max(queue free, expire))
S + 42 548 + propagation 500 ns
S + 42 548 .. 43 548 firmware core: max(core free, expire) + 1 000 ns
S + 43 548 expire_time; CQE posted on the first sweep at or after it
The link and firmware queues are shared by every command of the controller, so under load each command also waits for the ones queued before it on those queues.
Concurrency
- BBSSD and ZNS call the layer from the single
FEMU-FTL-Thread, so their timelines need no lock. KV calls it from the poller andfemu-cxl-ssdfrom itsfemu-cxl-ftlworker, each under its own device lock. - With the LUN rule, no bus and no suspend,
nand_media_op()updatesnext_lun_avail_timewith a compare-and-swap loop that gives the same result as a locked read-max-add-store. Any other configuration takes the general path, which callslock_lun/unlock_lunwhen the mode provides them; BBSSD and ZNS do not. - OCSSD 1.2 with channel timing takes the channel's spinlock
(
chnl_locks[]) throughlock_lun/unlock_lun; without it, and OCSSD 2.0, use the compare-and-swap path.
How garbage collection is charged
GC operations go through the same timelines as host operations, with
stime = 0 (the current time) and type = GC_IO:
gc_read_page()charges a read of each valid page,gc_write_page()a program at its new location, andreclaim_line()one multi-plane erase per LUN (bbssd/ftl-line-gc.c). All of a victim's pages are moved before any of its blocks is erased, so the erases follow the last relocation. FDP GC does the same inbbssd/ftl-fdp.c.- The returned latency is discarded. GC never adds to a command's latency directly. It makes LUNs and channels busy, and the host operations that need them afterwards wait.
- Foreground GC runs when the share of used lines has reached
gc_thres_pcent_high: at the start ofssd_write(), and before each page that a write, a write buffer destage or a Write Zeroes programs (bbssd/ftl-datapath.c), until the share drops below it. The command's own programs then queue behind the GC operations on the LUNs they share. - Read reclaim (
read_reclaim_limit,retention_limit_sec) rewrites one queued line throughreclaim_line()at the start of a write, charged the same way as GC. - Background GC runs one victim after a request finishes in
bb_ftl_process_req()(bbssd/ftl.c), once the latency of that request has been computed. It delays the commands that follow. - Because GC is stamped with the current time and a host command with its submission time, a command submitted before a GC pass can still wait for it when the FTL thread gets to the command after the pass.
- GC time can be turned off with 0xEF code 2 (Runtime switches). GC still moves pages and erases blocks; its operations are then not charged at all.
LUN 3 timeline during foreground GC for one write (bus off)
|-- GC read --|-- GC program --| ... |== multi-plane erase ==|-- host program --|
^ write arrives ^ write's done
KV is different. Its reclaim erases and compaction moves are stamped with
the triggering command's stime, the command completes when the last of
them and of its own operations does (kvssd/kvssd-ftl.c), and 0xEF does not
apply to KV, so they cannot be switched off.
ZNS has no device GC. A Zone Reset charges one erase per block of the zone on
every plane it spans, with the command's stime, and returns the largest
(zns_zone_reset()).
Costs that are not NAND operations
The FTL adds these to a command's latency alongside its NAND operations:
- BBSSD write buffer (
buffer_size> 0): a write the buffer accepts costsdram_hit_ns, orpg_rd_lat / 16when that is 0 (1 us whenpg_rd_latis 0 too), plus the programs of any destage it forces. A Flush pays for programming everything buffered (bbssd/ftl.c). A read of a buffered page costs the same DRAM access. - BBSSD read cache and DFTL mapping cache: a hit costs a DRAM access; a mapping miss adds a NAND read of the translation page, after a program if the evicted entry is dirty.
- ZNS write cache: a write only enters the zone's write cache and pays
SRAM_WRITE_LATENCY_NS(1 us) per 4 KiB page, summed over the command (zns_write()inzns/zftl.c). It pays program time only when the cache fills, or when it has to evict another zone's cache. A cache holds one stripe: 16 KiB *zns_flash_type*zns_num_plane*zns_num_ch*zns_num_lun. A read of data still in the cache is not mapped yet and costs nothing.
Host link and firmware CPU
Both are off by default. A read pays the link in nvme_process_cq_cpl() in
hw/femu/nvme-io.c, after the media latency is in expire_time. A write
that passes the rule below pays it at submit, before the FTL books its
programs. The firmware cost is charged in nvme_process_cq_cpl() for every
command.
| Model | Applies to | Computation |
|---|---|---|
Link transfer (pcie_bandwidth_mbps) | NVMe Read and Write (opcodes 01h and 02h) in BBSSD, CSD, KV, ZNS and NoSSD; KV store and retrieve share those opcodes and use the bytes moved. Not Zone Append, and not the Open-Channel vector commands, so OCSSD 1.2 never pays it | trans = bytes * 1000 / MBps ns. One queue per direction: writes on pcie_rx_next_avail_time, reads on pcie_tx_next_avail_time. Read: start = max(queue, expire). Write, where the rule below allows it: start = max(queue, submit), and the FTL start moves to the transfer end; other writes as a read. Both: queue = start + trans |
Propagation (pcie_prop_delay_ns) | same | expire = queue + delay; does not occupy the queue |
Firmware CPU (fw_cpu_ns, fw_read_ns, fw_write_ns, fw_other_ns, fw_cores) | Read (fw_read_ns), Write and Zone Append (fw_write_ns), each falling back to fw_cpu_ns when 0; every other I/O command, the Open-Channel vector commands included, pays fw_other_ns | fw_cores modelled cores: the command takes the core free soonest, start = max(core, expire), core = start + cost, expire = core |
A write's data crosses the link before the media can program it. So the write books its transfer when it passes its checks, and the FTL starts at the transfer end. A read's transfer starts after its NAND time.
The early booking applies only where the FTL books the media later, from the request start time. Every other write keeps the old order.
| Write (01h) on | Order |
|---|---|
bbssd, FDP or CSD block namespace, buffer_size=0 | link first |
| NoSSD | link first |
a write buffer (buffer_size > 0), ZNS (zone write cache) | old: buffered pages keep no arrival time |
| KV store, OCSSD 2.0 | old: nvme_io_cmd() books the media itself |
power_loss=on | old: the FTL thread checks the command |
| a controller linked to a CXL SSD, SLM, CP Execute | old |
write: submit --> link (rx queue) --> FTL programs --> firmware --> CQE
read: submit --> FTL reads --> link (tx queue) --> firmware --> CQE
pcie_write_first=off: write: submit --> FTL programs --> link --> firmware --> CQE
The stage counters of query-femu-stages count from the submit time. The
FTL handoff time is real time, so a write's link time shows in the hold
stage. The firmware cost is charged at the end of a command,
which caps the command rate at about fw_cores per firmware time but does not delay
the start of the media operations. The link model is enabled when either link property is non-zero
(pcie_enabled in femu.c).
OCSSD timing model
Open-Channel devices run on the media layer, each with its own NandMedia
and the LUN rule (oc12_media_init(), oc20_media_init()):
- Per chip (flat LUN id
ch * num_lun + lun), an operation ends atmax(now, busy) + time. Times come from theflash_typetable: 1.2 as a table by page type, 2.0 as flat lower-page times (by page type withoc20_page_types). - Open-Channel 1.2 with channel timing uses the staged bus with no command,
address or status phase:
bus_now()for a program's data-in andbus_later()for a read's data-out. - A write moves its data over the channel, then programs the chip. A read occupies the chip, then moves its data out. Erase charges each chip in the address list.
- Open-Channel 1.2 charges a channel transfer only with
oc12_channel_timing=on:ch_xfer_latns per page, or the table value forflash_typewhen it is 0, scaled by the sectors used out oflsecs_per_pg. Open-Channel 2.0 charges no channel time and uses page type 0 unlessoc20_page_typesis on. - Open-Channel 1.2 charges one chip operation per NAND page of the address list. Open-Channel 2.0 groups addresses that differ only in the sector field, which spans the whole chunk, so a run of consecutive sectors in one chunk pays one chip operation whatever its length.
- The latency is
max over operations of (done - stime), written straight intoexpire_timein the poller (oc12_advance_status(),oc20_advance_status()). - Open-Channel 1.2 keeps its data-out windows in an unbounded list per
channel (
policy.bus_res_unbounded), where the other modes cap it at 32.
Runtime switches
A BBSSD controller accepts the vendor admin command 0xEF (bb_flip() in
bbssd/bb.c). The action is in CDW10. For codes 1 to 4 it pauses the
pollers, applies the change to every namespace of the controller that has a
BBSSD FTL, and resumes them. Controllers of every other mode return Invalid
Opcode.
| CDW10 | Constant (bbssd/ftl.h) | Effect on timing |
|---|---|---|
| 1 | FEMU_ENABLE_GC_DELAY | GC operations are charged (default) |
| 2 | FEMU_DISABLE_GC_DELAY | GC operations are not charged |
| 3 | FEMU_ENABLE_DELAY_EMU | Flat read, program and erase set to the built-in 40 us, 200 us and 2 ms, not to the command-line values |
| 4 | FEMU_DISABLE_DELAY_EMU | Flat read, program and erase set to 0 |
| 5 | accounting reset | Prints and resets the per-poller completion counters; no timing change |
| 6, 7 | FEMU_ENABLE_LOG, FEMU_DISABLE_LOG | Per-command debug log on or off; no timing change |
Codes 3 and 4 refresh only the flat rd_ns, wr_ns and er_ns in the
media layer (bb_nand_media_refresh_timing()). With nand_cell_type 1 to 4
the flat times are not used, so the codes have no effect on NAND time. They
never change the channel bus phases, pgtype_lat, ECC time or
trim_lat_ns. Code 4 makes a write buffer hit cost 1 us; the read cache
keeps the hit cost it computed at start.
On a controller linked to a femu-cxl-ssd, codes 1 to 4 change the medium's
FTL instead, and code 3 restores its read-ns, program-ns and erase-ns.
From the guest:
sudo nvme admin-passthru /dev/nvme0 --opcode=0xef --cdw10=2 # stop charging GC
sudo nvme admin-passthru /dev/nvme0 --opcode=0xef --cdw10=4 # zero NAND times
The vendor command 0xEE (NVME_ADM_CMD_FEMU_DEBUG) sets an Open-Channel
controller's NAND times at run time, in nanoseconds: CDW10 upper page read,
CDW11 lower page read, CDW12 upper page program, CDW13 lower page program,
CDW14 block erase and CDW15 channel transfer per page. The lower page is
page type 0 and the upper page the highest page type of flash_type; the
centre pages of TLC and QLC keep their table times, and SLC, with one page
type, takes the lower page values. The times belong to that controller
(FemuCtrl.oc_pg_rd_lat and the fields beside it, filled from the
flash_type table by set_latency()), so other devices are not affected.
Other modes refuse 0xEE with Invalid Command Opcode, and log 05h does not list it.
sudo nvme admin-passthru /dev/nvme0 --opcode=0xee --cdw10=64000 --cdw11=48000 \
--cdw12=2300000 --cdw13=850000 --cdw14=3000000 --cdw15=52433
Parameters
The generated property reference is the authority for types, defaults and ranges. This table groups the timing properties and states how they interact.
| Group | Properties | Interactions |
|---|---|---|
| Geometry | secsz, secs_per_pg, pgs_per_blk, blks_per_pl, pls_per_lun, luns_per_ch, nchs | nchs * luns_per_ch is the number of independent dies. pls_per_lun adds no read or program parallelism under the LUN rule unless mp_program or mp_read is set. |
| Array times | pg_rd_lat, pg_wr_lat, blk_er_lat | Ignored when nand_cell_type is 1 to 4. 0xEF codes 3 and 4 overwrite them. With dram_hit_ns at 0, pg_rd_lat / 16 (1 ns at least) is also the cost of a write buffer hit. |
| Cell type | nand_cell_type, cell_pages, pgtype_lat | pgtype_lat applies only with nand_cell_type=0. nand_cell_type limits pgs_per_blk to 512. |
| Bus phases | cmd_addr_lat, pg_xfer_lat, ch_xfer_lat, status_lat | Any non-zero value turns the bus on. pg_xfer_lat takes precedence over ch_xfer_lat. |
| Multi-plane | tplebsy, mp_program, tplpbsy, mp_read, tplrbsy, plane_first (trcbsy applies only with onfi_cache_read) | Only with pls_per_lun > 1. tplebsy applies to the multi-plane erase of GC; tplpbsy only with mp_program; tplrbsy only with mp_read. |
| Suspend | pe_suspend, tsusp_ns | tsusp_ns matters only with pe_suspend. |
| ECC | ecc_step_ns, ecc_retention_sec | ecc_retention_sec matters only with ecc_step_ns. |
| Program disturb | pd_edge_wls, pd_edge_tiers, pd_open_tiers, pd_open_age | All need ecc_step_ns; pd_open_age also needs ecc_retention_sec. |
| Other FTL costs | trim_lat_ns | Charged per Dataset Management range by the FTL, not by the media layer, also under FDP. |
| ZNS | zns_num_ch, zns_num_lun, zns_num_plane, zns_num_blk, zns_flash_type, zns_pg_rd_lat, zns_pg_wr_lat, zns_blk_er_lat, zns_cmd_addr_lat, zns_pg_xfer_lat, zns_status_lat, zns_pe_suspend, zns_tsusp_ns | The zns_*_lat overrides apply to the configured cell type only. zns_num_plane scales the program unit. |
| OCSSD | flash_type, oc12_channel_timing, ch_xfer_lat, lnum_ch, lnum_lun, lnum_pln, lsecs_per_pg, lpgs_per_blk | ch_xfer_lat is shared with BBSSD. lnum_ch is at most 32 and lnum_ch * lnum_lun at most 128. |
| Host link and firmware | pcie_bandwidth_mbps, pcie_prop_delay_ns, fw_cpu_ns | Applied after the media time, to the opcodes listed in Host link and firmware CPU. |
femu-cxl-ssd | channels, luns-per-channel, pages-per-block, blocks-per-plane, read-ns, program-ns, erase-ns, channel-ns | One plane per LUN; feeds the same media layer. |
A BBSSD configuration with the bus and suspend on:
-device femu,devsz_mb=1024,femu_mode=1,cmd_addr_lat=1000,pg_xfer_lat=10000,pe_suspend=1,tsusp_ns=5000
A BBSSD with TLC page-type timing:
-device femu,devsz_mb=1024,femu_mode=1,nand_cell_type=3
Statistics
The media layer keeps no counters of its own. What it charges shows up in:
- Latency seen by the guest:
fiocompletion latency. This is the primary output of the model. - Vendor log page C0h (BBSSD, CSD): host, NAND and GC page counts, write amplification, buffer hits. See log pages and counters.
- Per-block counters used by the model:
nand_block.erase_cnt(ECC tiers, SMART Percentage Used) andnand_block.read_cnt(read reclaim), updated inssd_advance_status()and on erase. femu-cxl-ssd: the QOM countermedia-time-ns, the sum of the latencies the FTL returned. See runtime properties.- Debug:
nand_lun.gc_endtimerecords when the last GC operation on a LUN ends; nothing reports it.
Calibrating against a real device
- Take array times from the datasheet.
pg_rd_latis tR,pg_wr_latis tPROG,blk_er_latis tBERS. For a part with different lower and upper page times, either usenand_cell_typeif one of the built-in tables is close, or set the mean tPROG and turn onpgtype_latwithcell_pagesequal to bits per cell. - Derive the bus phases from the interface rate.
pg_xfer_latis the page size divided by the interface rate: a 16 KiB page on an 8-bit bus at 1600 MT/s takes 16384 / 1.6e9 s, about 10.2 us.cmd_addr_latandstatus_latare the command, address and status cycle times, usually well under 1 us. - Match the parallelism. Set
nchsandluns_per_chto the device's channels and dies per channel, andsecs_per_pg * secszto its page size. - Check the unloaded latency. Write the range first (a read of an unwritten page costs nothing), then run a queue depth 1 4 KiB random read. The median completion latency should be about tR plus the data-out and status phases plus guest and poller overhead. Measure that overhead once with flat timing, no bus phases, no caches and 0xEF code 4, and subtract it.
- Check the saturated throughput. A random read at high queue depth
should approach
nchs * luns_per_ch / tRoperations per second when the bus is not the limit, ornchs / (pg_xfer_lat + cmd_addr_lat + status_lat)when it is. Compare with the real device at the same queue depth. - Add the host side. If the real device's peak is below what the media
allows, set
pcie_bandwidth_mbpsto its link rate andfw_cpu_nsto the inverse of its peak command rate. - Check GC separately. Precondition the device, run sustained random writes, and compare the write amplification in log page C0h and the steady-state write bandwidth. Toggle 0xEF codes 1 and 2 to see how much of the gap GC time explains.
Give each poller and the FTL thread its own host core while measuring; performance tuning and measuring cover the setup.
Validation status
- Unit tests:
hw/femu/tests/unit/test-nand-media.cchecks the media layer's arithmetic without QEMU: ECC tiers, the staged bus and its booked windows, the plane rule with the bus, multi-plane erase against serial erases, copyback, and program/erase suspend. Run it withmake -C hw/femu/tests check. - qtests:
hw/femu/tests/qtest/femu-test.ccovers the OCSSD 1.2 channel model (oc12-channel-timing,oc12-channel-gap,oc12-channel-default,oc12-channel-off,oc12-ppa-timing,oc12-flash-type,oc12-page-count,oc12-transfer-cost, and the exact per-command times ofoc12-trace-onandoc12-trace-off), the warning for the timing properties that have no effect (ignored-props), the 0xEF flips on a linkedfemu-cxl-ssd(cxl-nvme-flip), the two example configurations on this page (doc-examples), and program/erase suspend on a BBSSD, CSD, KV and ZNS device (pe-suspend-<mode>-onandpe-suspend-<mode>-off): a read that arrives during a program completes before the program only when suspend is on. - The firmware CPU model per opcode and per core (
fw-cost-read,fw-cost-other,fw-cost-cores1,fw-cost-cores2). - Not covered by automated tests: other BBSSD and ZNS timing inside QEMU, the host link, the 0xEF flips on a plain BBSSD controller, and end-to-end guest latency against a reference device. The built-in MLC table is a profile of real parts; the other tables are from published papers, as cited in the headers. Check numbers that matter to you with the calibration steps above.
make -C hw/femu/tests check
What is not modelled
- Read-retry as repeated array reads, soft-decision decoding, and read disturb errors. The ECC adder is a fixed time per tier, and data is never corrupted.
- Multi-plane reads and programs outside BBSSD and CSD.
mp_readandmp_programbatch host pages there; they are off by default and refused with FDP. Line erases are always batched. - Cache read outside the ONFI profile, and cache program.
onfi_cache_readmodels the read cache sequence of one command (Read cache sequence). - Copyback outside BBSSD line GC and FDP GC. Paced collection, read reclaim and retention refresh move pages off-chip.
- Bit errors of a copyback. A real copyback skips ECC, so its bit errors are not corrected and can add up; FEMU models no bit errors, so a copyback changes only timing.
- Program suspend for programs, or erase suspend by another erase. Only reads suspend.
- Per-die power or thermal limits, and temperature-dependent NAND
physics. The package thermal model (
thermal_tau_ms,thermal_r) reports a temperature; withthrottle=onit stretches the array time of bbssd and CSD media as a device-wide pacing factor. Energy is charged when an operation is booked, and the policy stops while the VM pauses. - Plane-level parallelism for BBSSD and CSD programs, and for KV reads and
programs (the rule is per LUN).
plane_reads=onlets BBSSD and CSD reads on different planes overlap, with the plain channel model only. The channel bus for OCSSD 2.0. - PLC outside ZNS. ZNS accepts PLC with times you supply; BBSSD resets
nand_cell_type=5to 0 and OCSSD refusesflash_type=5. - Queueing inside the controller other than the link and the single firmware core: there is no DRAM bandwidth or NAND controller queue model.
- Timing accuracy below the poller sweep. A completion is posted on the first sweep after it is due, and a poller without its own host core adds delay.
Adding a timing feature
- Decide the resource. If the feature makes an existing resource busy for
longer, it is an
array_lat()or bus-phase change. If it adds a resource, add an accessor toNandTimelineOpsand keep the state in the mode's structures, like the existing busy-until fields. - Add the duration to
NandMediaTimingand a switch toNandMediaPolicyinnand/nand-media.h. Default it to 0 or off so that existing configurations keep their exact timing. The layer must not include a controller header or branch on the mode. - Implement it in
nand-media.c. Keep the max() ordering of the existing paths. A phase that happens at issue goes throughbus_now(); one that happens after the array goes throughbus_later(). - Wire the property. Add it with
DEFINE_PROP_*infemu.cand a description infemu-props.c, copy it intossdparamsinssd_init_params()(bbssd/ftl-geom.c), validate it inbb_check_geometry(), and set the media config inbb_nand_media_init(). For ZNS, the same inzns_init_params()andzns_nand_media_init(). - Test the arithmetic in
tests/unit/test-nand-media.c. Include a case that fails if the feature is ignored, so the test cannot pass with the feature off. - Update the docs. Regenerate the property reference with
gen-property-docs.py, add the parameter to this page and to the timing model concept page, and add a CHANGELOG entry. Keeping the documentation correct lists the checks.
Source map
| File | What it holds |
|---|---|
hw/femu/nand/nand-media.h | Media layer types and API: NandLoc, NandMediaTiming, NandMediaPolicy, NandTimelineOps, nand_media_op(), nand_media_multiplane(), nand_media_copyback() |
hw/femu/nand/nand-media.c | Array time, rules, bus booking (bus_now(), bus_later()), suspend, ECC, multi-plane, copyback |
hw/femu/nand/nand.h, nand.c | Cell-type timing tables, page pairing tables, rated P/E cycles |
hw/femu/bbssd/ftl-media.c | BBSSD adapter: bb_decode_loc(), bb_nand_media_init(), ssd_advance_status(), ssd_advance_status_multiplane(), bb_nand_media_refresh_timing() |
hw/femu/bbssd/ftl-geom.c | Geometry checks and parameter copy (bb_check_geometry(), ssd_init_params()) |
hw/femu/bbssd/ftl-datapath.c | Host read and write: per-page operations and max latency |
hw/femu/bbssd/ftl-line-gc.c | Write pointer order, GC reads, programs and multi-plane erase |
hw/femu/bbssd/bb.c | 0xEF handler (bb_flip(), bb_flip_apply()) |
hw/femu/zns/zftl.c | ZNS adapter, write cache flush, zone reset erase |
hw/femu/zns/zns.c, zns.h | ZNS timing values and property overrides (zns_init_params()) |
hw/femu/ocssd/oc-timing.c | OCSSD flash_type times, geometry check, 0xEE |
hw/femu/ocssd/oc12.c, oc20.c | OCSSD per-command timing (oc12_advance_status(), oc20_advance_status()) |
hw/femu/femu.c | FTL thread: expire_time += latency; timing properties |
hw/femu/nvme-io.c | stime stamp, host link and firmware CPU models, priority queue and completion |
hw/femu/tests/unit/test-nand-media.c | Media layer unit tests |