Plugin Architecture
A three-tier extensibility model that allows server vendors to supply hardware-specific implementations for every nabustore subsystem — without modifying core code.
Design Principles
The current architecture uses compile-time build tags (-tags spdk, -tags dpdk) controlled at compile time. This is a hard fork model — a vendor cannot add a backend without modifying core source. The plugin architecture replaces this with three well-defined extensibility contracts.
BlobBackend, ec.Engine, and index.Index interfaces are already well-designed. The problem is composition — how backends are selected and wired — not the interfaces themselves.
Static Registry
In-process, zero overhead. Vendors ship a Go package with an init() function. The binary imports what it wants via blank imports.
gRPC Out-of-Process
Vendor ships a standalone binary. Communicates over a Unix domain socket using the existing BlobService proto. No Go required.
Auto-Detection
Hardware probed at startup. CXL, NVMe, hugepages, DPDK PMD, ISA-L CPUID — all detected automatically. No flags required for standard hardware.
The selection rule is simple: Tier 1 is used for any hot-path operation (sub-microsecond reads, KV cache streaming). Tier 2 is for bulk-put paths and control-plane operations where one UDS round-trip is acceptable. Tier 3 is not a tier at all — it is the mechanism that selects between Tier 1 implementations at startup.
Directory Layout
Each subsystem is split into an interface package and one or more implementation sub-packages. A central plugin/ package owns the registry. The driver/ package owns auto-detection.
-
nabustore/
-
plugin/ new Central registry
- registry.go
-
blob/ BlobBackend interface
- blob.go interface
-
localfs/ moved
- localfs.go
- register.go init() → RegisterBackend("localfs")
-
spdk/ moved //go:build spdk
- spdk.go
- register.go
-
grpc/ new Tier 2 out-of-proc wrapper
- grpc_backend.go
-
transport/ new package
- transport.go interface
-
tcp/
- tcp.go
- register.go
-
dpdk/ //go:build dpdk
- dpdk.go
- register.go
-
ec/ Codec interface
- ec.go interface
-
reedsolomon/ moved Pure-Go GF(2^8)
- rs.go
- register.go
-
isal/ new //go:build isal, CGO → ISA-L
- isal.go
- register.go
-
index/ Index interface
- index.go interface
-
mmap/ moved Robin Hood hash map
- mmap.go
- register.go
-
cxl/ new DAX-backed, auto-detects NUMA
- cxlmem.go
- register.go
-
driver/ new package
- autodetect.go Hardware probe → DriverSet
-
cxl/ CXL util (feeds autodetect)
- util.go
-
cmd/nabustore/
- main.go calls driver.Detect + plugin.Open*
- plugins.go new Blank imports, build-tag controlled
-
Tier 1 — Static Registry
Modelled after database/sql and Docker's volume driver registry. Each subsystem has a named factory registered via init(). The binary composition is controlled entirely by blank imports in cmd/nabustore/plugins.go.
Registry API
The registry lives in plugin/registry.go. It is the only package that imports from all subsystem interfaces simultaneously.
package plugin import ( "fmt" "sync" "github.com/nabustore/nabustore/blob" "github.com/nabustore/nabustore/ec" "github.com/nabustore/nabustore/index" "github.com/nabustore/nabustore/transport" ) type ( BackendFactory func(cfg map[string]string) (blob.BlobBackend, error) TransportFactory func(cfg map[string]string) (transport.Transport, error) ECFactory func(cfg map[string]string) (ec.Engine, error) IndexFactory func(cfg map[string]string) (index.Index, error) ) // Registration — called from init() in each driver package. func RegisterBackend(name string, f BackendFactory) { reg(&backends, name, f) } func RegisterTransport(name string, f TransportFactory) { reg(&transports, name, f) } func RegisterEC(name string, f ECFactory) { reg(&ecEngines, name, f) } func RegisterIndex(name string, f IndexFactory) { reg(&indexes, name, f) } // Lookup — called during node startup via driver.DriverSet. func OpenBackend(name string, cfg map[string]string) (blob.BlobBackend, error) { mu.RLock() f, ok := backends[name] mu.RUnlock() if !ok { return nil, fmt.Errorf("plugin: unknown backend %q (registered: %v)", name, Keys(&backends)) } return f(cfg) } // OpenTransport, OpenEC, OpenIndex follow the same pattern.
A vendor calls one of the Register* functions from a package-level init(). The binary only needs to blank-import the package:
package main import ( // ── Always compiled in ────────────────────────────────── _ "github.com/nabustore/nabustore/blob/localfs" _ "github.com/nabustore/nabustore/transport/tcp" _ "github.com/nabustore/nabustore/ec/reedsolomon" _ "github.com/nabustore/nabustore/index/mmap" _ "github.com/nabustore/nabustore/index/cxl" // auto-detects at Open time // ── Build-tag gated ───────────────────────────────────── _ "github.com/nabustore/nabustore/blob/spdk" // -tags spdk _ "github.com/nabustore/nabustore/transport/dpdk" // -tags dpdk _ "github.com/nabustore/nabustore/ec/isal" // -tags isal // ── Vendor plugins ────────────────────────────────────── // _ "github.com/kioxia/nabustore-kioxia" // Kioxia FlashArray // _ "github.com/hpe/nabustore-alletra" // HPE Alletra MP )
if backend == "kioxia" conditional.
Subsystem Interfaces
Each subsystem exposes a minimal Go interface. Implementations are completely decoupled from each other. All methods must be safe for concurrent use.
blob.BlobBackend (existing)
| Method | Signature | Notes |
|---|---|---|
| Put | (ctx, id, r io.Reader, meta) error | Write-once. Atomic — blob visible or not at all. |
| Get | (ctx, id) (io.ReadCloser, BlobMeta, error) | Returns stream; caller closes. |
| GetRange | (ctx, id, off, n int64) (io.ReadCloser, error) | Byte-range read for KV cache partial fetches. |
| Stat | (ctx, id) (BlobMeta, error) | Size + hash without body transfer. |
| Delete | (ctx, id) error | Best-effort; not required for KV workloads. |
| List | (ctx, prefix, limit) ([]BlobMeta, error) | Prefix scan. Used by rebalancer. |
| Capacity | (ctx) (used, total int64, err error) | (0,0,nil) is valid for NVMe-oF targets. |
| Close | () error | Called on clean shutdown. |
transport.Transport (new)
| Method | Signature | Notes |
|---|---|---|
| Dial | (ctx, addr) (Conn, error) | Returns a bidirectional message-oriented connection. |
| Listen | (addr) (Listener, error) | Accept loop for inbound replication/DHT traffic. |
| Name | () string | Returns "tcp", "dpdk", "rdma", etc. |
| Close | () error |
ec.Engine (existing, already abstracted)
| Method | Signature | Notes |
|---|---|---|
| Encode | (shards [][]byte) error | Fills parity shards in-place. Caller allocates. |
| Reconstruct | (shards [][]byte) error | nil entries = missing. Restores data + parity. |
| ReconstructData | (shards [][]byte) error | Data only — faster when parity is not needed. |
| Verify | (shards [][]byte) (bool, error) | Check parity consistency without reconstruct. |
index.Index (interface to be extracted)
| Method | Signature | Notes |
|---|---|---|
| Put | (id BlobID, e BlobIndexEntry) error | Upsert. Auto-resize at 0.72 load factor. |
| Get | (id BlobID) (BlobIndexEntry, error) | ErrNotFound triggers ring-based fallback. |
| Delete | (id BlobID) error | Uses backward-shift deletion. |
| Scan | (fn func(BlobID, BlobIndexEntry) bool) | Full scan for rebalancer. fn returns false to stop. |
| Count | () int64 | |
| LoadFactor | () float64 | Prometheus metric exposure. |
| Close | () error | Flushes CXL clwb/msync before return. |
Tier 2 — gRPC Out-of-Process
For vendors who cannot or will not ship Go source. The vendor delivers a compiled binary that listens on a Unix domain socket and speaks the existing BlobService protobuf contract. A thin adapter in blob/grpc/grpc_backend.go wraps the gRPC client as a BlobBackend.
Discovery Protocol
Modelled after the Kubernetes device plugin pattern. The vendor binary drops a socket file into the well-known plugin directory. The plugin manager watches this directory and automatically wraps each new socket as a registered backend.
/var/lib/nabustore/plugins/ ├── kioxia.sock ← vendor binary listens here ├── hpe-alletra.sock └── pure-fb.sock ← auto-discovered on startup + inotify watch
// GRPCBackend wraps a vendor gRPC process as a BlobBackend. // The vendor binary need not be written in Go — it only needs to // implement BlobService as defined in proto/nabustore.proto. type GRPCBackend struct { conn *grpc.ClientConn client pb.BlobServiceClient } func NewGRPCBackend(socketPath string) (*GRPCBackend, error) { conn, err := grpc.Dial( "unix://"+socketPath, grpc.WithTransportCredentials(insecure.NewCredentials()), grpc.WithBlock(), ) if err != nil { return nil, fmt.Errorf("grpc_backend: dial %s: %w", socketPath, err) } return &GRPCBackend{conn: conn, client: pb.NewBlobServiceClient(conn)}, nil } // Put, Get, GetRange, Stat, Delete, List, Capacity, Close // all delegate to the gRPC client — proto messages already match // BlobBackend semantics 1:1.
Tier 3 — Auto-Detection
Tier 3 is not a separate plugin tier — it is the startup mechanism that selects among registered Tier 1 plugins based on what hardware is present. The -backend flag becomes an override, not a requirement.
DriverSet
driver.Detect() probes the system and returns a DriverSet naming the best available implementation for each subsystem.
CXL Index
Checks /sys/bus/cxl/devices, loaded kernel modules (cxl_mem, cxl_port), and DAX device presence at /dev/dax*.
SPDK Backend
Verifies socket at --spdk-socket is alive, and that at least one NVMe device is bound to vfio-pci or uio_pci_generic.
DPDK Transport
Checks 1GB hugepage allocation via /proc/meminfo, and that a ConnectX DPDK PMD device is present under /dev/vfio/.
ISA-L EC Engine
Runtime CPUID check for AVX-512 (x86) or NEON + SVE (ARM). Falls back to pure-Go klauspost/reedsolomon when unavailable.
package driver type DriverSet struct { Backend string // "spdk" | "localfs" | "kioxia" | ... Transport string // "dpdk" | "tcp" | "rdma" EC string // "isal" | "reedsolomon" Index string // "cxl" | "mmap" Config map[string]string // driver-specific key/value config } func Detect(hints *Config) DriverSet { ds := DriverSet{Config: make(map[string]string)} // ── Index: CXL DIMM via kernel DAX driver ──────────────── if info := cxlinfo.Detect(); info.Available { ds.Index = "cxl" ds.Config["cxl_numa"] = strconv.Itoa(info.PreferredNUMA()) ds.Config["cxl_dax"] = info.DAXDevice // e.g. /dev/dax0.0 } else { ds.Index = "mmap" ds.Config["mmap_path"] = hints.IndexPath } // ── Backend: SPDK if socket alive + devices bound ──────── if hints.SPDKSocket != "" && spdk.SocketAlive(hints.SPDKSocket) && spdk.HasBoundDevices() { ds.Backend = "spdk" ds.Config["socket"] = hints.SPDKSocket ds.Config["bdev"] = hints.SPDKBdev } else { ds.Backend = "localfs" ds.Config["data_dir"] = hints.DataDir } // ── Transport: DPDK if hugepages + PMD device present ──── if dpdk.HugepagesAvailable() && dpdk.HasPMDDevice() { ds.Transport = "dpdk" } else { ds.Transport = "tcp" } // ── EC: ISA-L via runtime CPUID ────────────────────────── if isal.Available() { ds.EC = "isal" } else { ds.EC = "reedsolomon" } // Explicit flag overrides take final precedence if hints.BackendOverride != "" { ds.Backend = hints.BackendOverride } if hints.IndexOverride != "" { ds.Index = hints.IndexOverride } return ds }
In main.go the startup sequence becomes a clean three-step pipeline:
Blob Backends
The three built-in storage backends and their registration patterns. Each lives in its own sub-package and registers itself without any dependency on core.
package localfs import "github.com/nabustore/nabustore/plugin" func init() { plugin.RegisterBackend("localfs", func(cfg map[string]string) (blob.BlobBackend, error) { return NewLocalFSBackend(cfg["data_dir"]) }) }
//go:build spdk package spdk import "github.com/nabustore/nabustore/plugin" func init() { plugin.RegisterBackend("spdk", func(cfg map[string]string) (blob.BlobBackend, error) { return NewSPDKBackend(cfg["socket"], cfg["bdev"], cfg["core_mask"]) }) }
package grpcbackend import "github.com/nabustore/nabustore/plugin" func init() { // The gRPC backend name is the socket path prefix. // plugin.OpenBackend("grpc:kioxia", cfg) → dials unix:///var/lib/nabustore/plugins/kioxia.sock plugin.RegisterBackend("grpc", func(cfg map[string]string) (blob.BlobBackend, error) { return NewGRPCBackend(cfg["socket_path"]) }) }
Transport
The transport subsystem handles intra-cluster replication traffic and DHT ring messages. It is separated from the blob backend because DPDK NICs and SPDK NVMe devices use fundamentally different programming models that must not be mixed.
rdma-core/libibverbs and cannot simultaneously be driven by DPDK's mlx5 PMD. SPDK owns IB NICs (NVMe-oF RDMA path); DPDK owns ConnectX Ethernet NICs (replication/RoCE path). These must never share a physical NIC.
//go:build dpdk package dpdk import "github.com/nabustore/nabustore/plugin" func init() { plugin.RegisterTransport("dpdk", func(cfg map[string]string) (transport.Transport, error) { return NewDPDKTransport(cfg["pci_addr"], cfg["core_mask"], cfg["mem_size"]) }) }
EC Engines
The EC engine interface is already implicit in ec/ec.go. The migration moves the klauspost/reedsolomon implementation into ec/reedsolomon/ and adds an ISA-L CGO backend for AVX-512 / NEON production paths.
//go:build isal // +build isal // // CGO linkage: -lisal (Intel Intelligent Storage Acceleration Library) // ISA-L must be installed: dnf install isa-l-devel / apt install libisal-dev package isal import "github.com/nabustore/nabustore/plugin" func init() { plugin.RegisterEC("isal", func(cfg map[string]string) (ec.Engine, error) { data, _ := strconv.Atoi(cfg["data_shards"]) parity, _ := strconv.Atoi(cfg["parity_shards"]) return NewISALEngine(data, parity) // CGO call into gf_vect_mul / ec_encode_data }) }
The isal.Available() function used by driver/autodetect.go performs a runtime CPUID check — it is always compiled in even without the isal build tag, so the autodetector can query it on any binary.
Index Backends
index/index.go currently exports a concrete Robin Hood hashmap rather than an interface. Extracting index.Index is a hard prerequisite for the mmap/CXL split. This is the first task in the migration path below.
// CXLIndex is a Robin Hood hash map backed by a Linux DAX device. // No vendor software required — uses the standard kernel cxl_mem driver // (in from 5.12, production-stable from 6.5). // // Detection path: // 1. /sys/bus/cxl/devices/* — enumerate CXL DIMMs // 2. /dev/dax* — find corresponding DAX character device // 3. mmap(O_RDWR|MAP_SHARED) — map the DAX range directly // 4. clflushopt/clwb — persistence without msync() overhead // // The layout on the DAX device is identical to the mmap/file backend // so a node can be rebooted without re-indexing. type CXLIndex struct { base *mmapIndex // shared Robin Hood logic dax *os.File numaNode int } func New(daxPath string, numaNode int) (*CXLIndex, error) { f, err := os.OpenFile(daxPath, os.O_RDWR, 0600) if err != nil { return nil, fmt.Errorf("cxl_index: open %s: %w", daxPath, err) } // ... mmap + validate magic header ... }
Writing a Vendor Plugin
A vendor supplying, for example, a Kioxia XD7P FlashArray backend follows these steps to produce a Tier 1 plugin with no changes to nabustore core.
Tier 1 Step-by-Step
Create your Go module
Module path github.com/kioxia/nabustore-kioxia. Add github.com/nabustore/nabustore as a dependency — only interface packages, no core.
Implement the interface
Create a struct that satisfies blob.BlobBackend. All methods must be goroutine-safe. Return blob.ErrBlobExists, blob.ErrBlobNotFound, etc. for correct error signalling.
Register in init()
Call plugin.RegisterBackend("kioxia-xd7p", factory) in a package-level init(). The factory receives a map[string]string config from the DriverSet.
Publish the package
Tag a release. The operator adds one blank import to cmd/nabustore/plugins.go: _ "github.com/kioxia/nabustore-kioxia". No other changes required.
package kioxia import ( "github.com/nabustore/nabustore/blob" "github.com/nabustore/nabustore/plugin" ) func init() { plugin.RegisterBackend("kioxia-xd7p", func(cfg map[string]string) (blob.BlobBackend, error) { return NewXD7PBackend( cfg["endpoint"], // e.g. "192.168.1.10:4420" cfg["nqn"], // NVMe Qualified Name for the subsystem cfg["ns_id"], // Namespace ID ) }) }
Build Composition
Build tags remain the mechanism for gating hardware-specific CGO dependencies. The difference is that tags now only affect which packages get blank-imported in plugins.go — they never appear in core logic.
| Target | Build command | Included backends |
|---|---|---|
| Development / CI | go build ./cmd/nabustore | localfs, tcp, reedsolomon, mmap |
| Production NVMe | go build -tags spdk ./cmd/nabustore | + spdk |
| Production full | go build -tags "spdk dpdk isal" ./cmd/nabustore | + spdk, dpdk, isal |
| Vendor (Kioxia) | go build -tags kioxia ./cmd/nabustore | + kioxia-xd7p (Tier 1) |
| ARM (Grace) | GOARCH=arm64 go build -tags isal ./cmd/nabustore | isal → NEON path via ISA-L ARM |
Compatibility Matrix
Which tiers are appropriate for each subsystem and workload type.
| Subsystem | Tier 1 (in-process) | Tier 2 (gRPC) | Hot path safe | CXL/auto-detect |
|---|---|---|---|---|
| Blob backend | ✓ Required for reads | △ Bulk-put only | Tier 1 only | N/A |
| Transport | ✓ | ✗ Not supported | ✓ | △ hugepages probe |
| EC engine | ✓ | ✗ Not supported | ✓ | ✓ CPUID at startup |
| Index | ✓ | ✗ Not supported | ✓ | ✓ DAX probe |
Migration Path
The migration is fully additive — no existing interface is changed. Each step can be merged independently and is safe to deploy without completing the full migration.
Prerequisites
index.Index interface from index/index.go. Currently exports a concrete Robin Hood hashmap. All callers use the concrete type directly — this must become an interface before the mmap/CXL split can proceed cleanly.
Effort
~2 hours. Existing contract tests in blob/local_test.go already exercise the full interface surface — they provide the test harness for the new interface.
Risk
Low. Pure refactor — no behaviour change, no new dependencies.
Ordered Steps
Extract index.Index interface
Define interface in index/index.go, move concrete impl to index/mmap/mmap.go. Update all call sites. Required before any further split.
Create plugin/registry.go
New package. No dependencies on existing code except interfaces. Does not break anything — nothing imports it yet.
Move localfs and spdk into sub-packages
Add register.go with init() to each. Update cmd/nabustore/plugins.go with blank imports. Remove the if backend == "spdk" branches from server/server.go.
Create transport package
Define Transport interface. Move existing TCP replication logic to transport/tcp/. Move DPDK build-tag code to transport/dpdk/.
Create driver/autodetect.go
Wraps existing cxl/util.go detection. Adds SPDK socket probe and DPDK hugepage check. Returns DriverSet. Simplify main.go to the three-step pipeline.
Add CXL index plugin
Implement index/cxl/ backed by kernel DAX (/dev/dax*). Register as "cxl". Autodetector selects it when a DAX device is present.
Add ISA-L EC engine
Implement ec/isal/ with CGO bindings to ISA-L. Register as "isal". isal.Available() compiled into all binaries for the autodetector.
Add Tier 2 gRPC backend
Implement blob/grpc/grpc_backend.go. Add plugin manager with inotify watch on /var/lib/nabustore/plugins/. Register any discovered .sock files as "grpc:<name>" backends.