Vimthusiast/tempest
1
0
Fork 0
Code Issues 1 Activity

use io_uring fixed file indices to bypass file descriptor table lookups in LinuxIo #1

New issue
Open
opened 2026-03-25 23:34:19 +01:00 by Vimthusiast · 0 comments
Vimthusiast commented 2026-03-25 23:34:19 +01:00 (Migrated from codeberg.org)
Copy link

Summary

Register SSTable (and other long-lived) file descriptors into io_uring's fixed file table, so the kernel can skip fd table lookups and refcount churn on every SQE.

Background

io_uring supports pre-registering a set of fds via IORING_REGISTER_FILES / IORING_REGISTER_FILES_UPDATE. SQEs can then reference them by slot index (types::Fixed(slot)) instead of by fd (types::Fd(fd)), which avoids a file table lookup and atomic refcount bump/drop on every operation.

This is entirely a LinuxIo implementation detail - the Io trait and VirtualIo need no changes.

Implementation

New fields on LinuxIo

fixed_files: Vec<Option<RawFd>>,     // slot index -> fd, pre-allocated at construction
fd_to_slot: HashMap<RawFd, u32>,     // fd -> slot index for fast SQE construction
pending_opens: HashSet<OpHandle>,    // to detect open CQEs in poll()
pending_closes: HashMap<OpHandle, RawFd>, // to detect close CQEs and free slots

Construction

  • Add max_fixed_files: u32 parameter to LinuxIo::new
  • Pre-register the table filled with -1 placeholder fds via IORING_REGISTER_FILES
  • fixed_files is initialized as vec![None; max_fixed_files as usize]

open()

  • Insert handle into pending_opens after a successful SQE push

poll()

  • On CQE arrival, check pending_opens:
    • If present and result is Ok(fd), call register_fixed(fd) to find a free slot and call IORING_REGISTER_FILES_UPDATE for that slot
  • Check pending_closes:
    • If present, clear the slot in fixed_files and fd_to_slot, update the kernel table with -1 for that slot

close()

  • Insert (handle, fd) into pending_closes after a successful SQE push

SQE construction (read_at, write_at, fsync, fstat)

  • Check fd_to_slot.get(&fd):
    • If registered: use types::Fixed(slot)
    • If not (e.g. table is full): fall back to types::Fd(fd)

Notes

  • Fixed file table size must be declared upfront at ring creation - choose a reasonable default (e.g. 1024)
  • If the table is full, fall back gracefully to types::Fd rather than erroring
  • VirtualIo requires zero changes
  • Benchmark before and under compaction load to verify the win is real - this is most impactful under high sustained I/O, less so at low concurrency -> performance increase is likely
## Summary Register SSTable (and other long-lived) file descriptors into io_uring's fixed file table, so the kernel can skip fd table lookups and refcount churn on every SQE. ## Background io_uring supports pre-registering a set of fds via `IORING_REGISTER_FILES` / `IORING_REGISTER_FILES_UPDATE`. SQEs can then reference them by slot index (`types::Fixed(slot)`) instead of by fd (`types::Fd(fd)`), which avoids a file table lookup and atomic refcount bump/drop on every operation. This is entirely a `LinuxIo` implementation detail - the `Io` trait and `VirtualIo` need no changes. ## Implementation ### New fields on `LinuxIo` ```rust fixed_files: Vec<Option<RawFd>>, // slot index -> fd, pre-allocated at construction fd_to_slot: HashMap<RawFd, u32>, // fd -> slot index for fast SQE construction pending_opens: HashSet<OpHandle>, // to detect open CQEs in poll() pending_closes: HashMap<OpHandle, RawFd>, // to detect close CQEs and free slots ``` ### Construction - Add `max_fixed_files: u32` parameter to `LinuxIo::new` - Pre-register the table filled with `-1` placeholder fds via `IORING_REGISTER_FILES` - `fixed_files` is initialized as `vec![None; max_fixed_files as usize]` ### `open()` - Insert `handle` into `pending_opens` after a successful SQE push ### `poll()` - On CQE arrival, check `pending_opens`: - If present and result is `Ok(fd)`, call `register_fixed(fd)` to find a free slot and call `IORING_REGISTER_FILES_UPDATE` for that slot - Check `pending_closes`: - If present, clear the slot in `fixed_files` and `fd_to_slot`, update the kernel table with `-1` for that slot ### `close()` - Insert `(handle, fd)` into `pending_closes` after a successful SQE push ### SQE construction (read_at, write_at, fsync, fstat) - Check `fd_to_slot.get(&fd)`: - If registered: use `types::Fixed(slot)` - If not (e.g. table is full): fall back to `types::Fd(fd)` ## Notes - Fixed file table size must be declared upfront at ring creation - choose a reasonable default (e.g. 1024) - If the table is full, fall back gracefully to `types::Fd` rather than erroring - `VirtualIo` requires zero changes - Benchmark before and under compaction load to verify the win is real - this is most impactful under high sustained I/O, less so at low concurrency -> performance increase is likely
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/forgejo-action
feat/enums-prototype
prototype/syntax
v0.0.2
v0.0.1
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Vimthusiast/tempest#1
No description provided.