Design Principles

The current architecture uses compile-time build tags (-tags spdk, -tags dpdk) controlled at compile time. This is a hard fork model — a vendor cannot add a backend without modifying core source. The plugin architecture replaces this with three well-defined extensibility contracts.

Core insight: The existing BlobBackend, ec.Engine, and index.Index interfaces are already well-designed. The problem is composition — how backends are selected and wired — not the interfaces themselves.
Tier 1

Static Registry

In-process, zero overhead. Vendors ship a Go package with an init() function. The binary imports what it wants via blank imports.

Zero overhead Go source Hot path safe
Tier 2

gRPC Out-of-Process

Vendor ships a standalone binary. Communicates over a Unix domain socket using the existing BlobService proto. No Go required.

Any language UDS socket Control path
Tier 3

Auto-Detection

Hardware probed at startup. CXL, NVMe, hugepages, DPDK PMD, ISA-L CPUID — all detected automatically. No flags required for standard hardware.

CXL 3.0 native Kernel modules Zero-config

The selection rule is simple: Tier 1 is used for any hot-path operation (sub-microsecond reads, KV cache streaming). Tier 2 is for bulk-put paths and control-plane operations where one UDS round-trip is acceptable. Tier 3 is not a tier at all — it is the mechanism that selects between Tier 1 implementations at startup.

Directory Layout

Each subsystem is split into an interface package and one or more implementation sub-packages. A central plugin/ package owns the registry. The driver/ package owns auto-detection.

  • nabustore/
    • plugin/ new Central registry
      • 📄registry.go
    • blob/ BlobBackend interface
      • 📄blob.go interface
      • localfs/ moved
        • 📄localfs.go
        • 📄register.go init() → RegisterBackend("localfs")
      • spdk/ moved //go:build spdk
        • 📄spdk.go
        • 📄register.go
      • grpc/ new Tier 2 out-of-proc wrapper
        • 📄grpc_backend.go
    • transport/ new package
      • 📄transport.go interface
      • tcp/
        • 📄tcp.go
        • 📄register.go
      • dpdk/ //go:build dpdk
        • 📄dpdk.go
        • 📄register.go
    • ec/ Codec interface
      • 📄ec.go interface
      • reedsolomon/ moved Pure-Go GF(2^8)
        • 📄rs.go
        • 📄register.go
      • isal/ new //go:build isal, CGO → ISA-L
        • 📄isal.go
        • 📄register.go
    • index/ Index interface
      • 📄index.go interface
      • mmap/ moved Robin Hood hash map
        • 📄mmap.go
        • 📄register.go
      • cxl/ new DAX-backed, auto-detects NUMA
        • 📄cxlmem.go
        • 📄register.go
    • driver/ new package
      • 📄autodetect.go Hardware probe → DriverSet
    • cxl/ CXL util (feeds autodetect)
      • 📄util.go
    • cmd/nabustore/
      • 📄main.go calls driver.Detect + plugin.Open*
      • 📄plugins.go new Blank imports, build-tag controlled

Tier 1 — Static Registry

Modelled after database/sql and Docker's volume driver registry. Each subsystem has a named factory registered via init(). The binary composition is controlled entirely by blank imports in cmd/nabustore/plugins.go.

Registry API

The registry lives in plugin/registry.go. It is the only package that imports from all subsystem interfaces simultaneously.

goplugin/registry.go
package plugin

import (
    "fmt"
    "sync"

    "github.com/nabustore/nabustore/blob"
    "github.com/nabustore/nabustore/ec"
    "github.com/nabustore/nabustore/index"
    "github.com/nabustore/nabustore/transport"
)

type (
    BackendFactory   func(cfg map[string]string) (blob.BlobBackend, error)
    TransportFactory func(cfg map[string]string) (transport.Transport, error)
    ECFactory        func(cfg map[string]string) (ec.Engine, error)
    IndexFactory     func(cfg map[string]string) (index.Index, error)
)

// Registration — called from init() in each driver package.
func RegisterBackend(name string, f BackendFactory)     { reg(&backends, name, f) }
func RegisterTransport(name string, f TransportFactory) { reg(&transports, name, f) }
func RegisterEC(name string, f ECFactory)               { reg(&ecEngines, name, f) }
func RegisterIndex(name string, f IndexFactory)         { reg(&indexes, name, f) }

// Lookup — called during node startup via driver.DriverSet.
func OpenBackend(name string, cfg map[string]string) (blob.BlobBackend, error) {
    mu.RLock()
    f, ok := backends[name]
    mu.RUnlock()
    if !ok {
        return nil, fmt.Errorf("plugin: unknown backend %q (registered: %v)", name, Keys(&backends))
    }
    return f(cfg)
}
// OpenTransport, OpenEC, OpenIndex follow the same pattern.

A vendor calls one of the Register* functions from a package-level init(). The binary only needs to blank-import the package:

gocmd/nabustore/plugins.go
package main

import (
    // ── Always compiled in ──────────────────────────────────
    _ "github.com/nabustore/nabustore/blob/localfs"
    _ "github.com/nabustore/nabustore/transport/tcp"
    _ "github.com/nabustore/nabustore/ec/reedsolomon"
    _ "github.com/nabustore/nabustore/index/mmap"
    _ "github.com/nabustore/nabustore/index/cxl"     // auto-detects at Open time

    // ── Build-tag gated ─────────────────────────────────────
    _ "github.com/nabustore/nabustore/blob/spdk"      // -tags spdk
    _ "github.com/nabustore/nabustore/transport/dpdk" // -tags dpdk
    _ "github.com/nabustore/nabustore/ec/isal"        // -tags isal

    // ── Vendor plugins ──────────────────────────────────────
    // _ "github.com/kioxia/nabustore-kioxia"   // Kioxia FlashArray
    // _ "github.com/hpe/nabustore-alletra"     // HPE Alletra MP
)
Zero changes to core: adding a vendor backend is a single blank import line. Removing it is deleting that line. The core never contains an if backend == "kioxia" conditional.

Subsystem Interfaces

Each subsystem exposes a minimal Go interface. Implementations are completely decoupled from each other. All methods must be safe for concurrent use.

blob.BlobBackend (existing)

MethodSignatureNotes
Put(ctx, id, r io.Reader, meta) errorWrite-once. Atomic — blob visible or not at all.
Get(ctx, id) (io.ReadCloser, BlobMeta, error)Returns stream; caller closes.
GetRange(ctx, id, off, n int64) (io.ReadCloser, error)Byte-range read for KV cache partial fetches.
Stat(ctx, id) (BlobMeta, error)Size + hash without body transfer.
Delete(ctx, id) errorBest-effort; not required for KV workloads.
List(ctx, prefix, limit) ([]BlobMeta, error)Prefix scan. Used by rebalancer.
Capacity(ctx) (used, total int64, err error)(0,0,nil) is valid for NVMe-oF targets.
Close() errorCalled on clean shutdown.

transport.Transport (new)

MethodSignatureNotes
Dial(ctx, addr) (Conn, error)Returns a bidirectional message-oriented connection.
Listen(addr) (Listener, error)Accept loop for inbound replication/DHT traffic.
Name() stringReturns "tcp", "dpdk", "rdma", etc.
Close() error

ec.Engine (existing, already abstracted)

MethodSignatureNotes
Encode(shards [][]byte) errorFills parity shards in-place. Caller allocates.
Reconstruct(shards [][]byte) errornil entries = missing. Restores data + parity.
ReconstructData(shards [][]byte) errorData only — faster when parity is not needed.
Verify(shards [][]byte) (bool, error)Check parity consistency without reconstruct.

index.Index (interface to be extracted)

MethodSignatureNotes
Put(id BlobID, e BlobIndexEntry) errorUpsert. Auto-resize at 0.72 load factor.
Get(id BlobID) (BlobIndexEntry, error)ErrNotFound triggers ring-based fallback.
Delete(id BlobID) errorUses backward-shift deletion.
Scan(fn func(BlobID, BlobIndexEntry) bool)Full scan for rebalancer. fn returns false to stop.
Count() int64
LoadFactor() float64Prometheus metric exposure.
Close() errorFlushes CXL clwb/msync before return.

Tier 2 — gRPC Out-of-Process

For vendors who cannot or will not ship Go source. The vendor delivers a compiled binary that listens on a Unix domain socket and speaks the existing BlobService protobuf contract. A thin adapter in blob/grpc/grpc_backend.go wraps the gRPC client as a BlobBackend.

Hot path restriction: Tier 2 is appropriate for bulk-put paths and control-plane operations only. The UDS round-trip overhead (~2–5 µs) is incompatible with sub-microsecond KV cache read requirements. For hot-path operations, vendors must provide a Tier 1 Go package.

Discovery Protocol

Modelled after the Kubernetes device plugin pattern. The vendor binary drops a socket file into the well-known plugin directory. The plugin manager watches this directory and automatically wraps each new socket as a registered backend.

textDiscovery path
/var/lib/nabustore/plugins/
├── kioxia.sock       ← vendor binary listens here
├── hpe-alletra.sock
└── pure-fb.sock      ← auto-discovered on startup + inotify watch
goblob/grpc/grpc_backend.go (sketch)
// GRPCBackend wraps a vendor gRPC process as a BlobBackend.
// The vendor binary need not be written in Go — it only needs to
// implement BlobService as defined in proto/nabustore.proto.
type GRPCBackend struct {
    conn   *grpc.ClientConn
    client pb.BlobServiceClient
}

func NewGRPCBackend(socketPath string) (*GRPCBackend, error) {
    conn, err := grpc.Dial(
        "unix://"+socketPath,
        grpc.WithTransportCredentials(insecure.NewCredentials()),
        grpc.WithBlock(),
    )
    if err != nil {
        return nil, fmt.Errorf("grpc_backend: dial %s: %w", socketPath, err)
    }
    return &GRPCBackend{conn: conn, client: pb.NewBlobServiceClient(conn)}, nil
}

// Put, Get, GetRange, Stat, Delete, List, Capacity, Close
// all delegate to the gRPC client — proto messages already match
// BlobBackend semantics 1:1.

Tier 3 — Auto-Detection

Tier 3 is not a separate plugin tier — it is the startup mechanism that selects among registered Tier 1 plugins based on what hardware is present. The -backend flag becomes an override, not a requirement.

DriverSet

driver.Detect() probes the system and returns a DriverSet naming the best available implementation for each subsystem.

CXL Index

Checks /sys/bus/cxl/devices, loaded kernel modules (cxl_mem, cxl_port), and DAX device presence at /dev/dax*.

Selects: index/cxl

SPDK Backend

Verifies socket at --spdk-socket is alive, and that at least one NVMe device is bound to vfio-pci or uio_pci_generic.

Selects: blob/spdk

DPDK Transport

Checks 1GB hugepage allocation via /proc/meminfo, and that a ConnectX DPDK PMD device is present under /dev/vfio/.

Selects: transport/dpdk

ISA-L EC Engine

Runtime CPUID check for AVX-512 (x86) or NEON + SVE (ARM). Falls back to pure-Go klauspost/reedsolomon when unavailable.

Selects: ec/isal
godriver/autodetect.go
package driver

type DriverSet struct {
    Backend   string            // "spdk" | "localfs" | "kioxia" | ...
    Transport string            // "dpdk" | "tcp" | "rdma"
    EC        string            // "isal" | "reedsolomon"
    Index     string            // "cxl" | "mmap"
    Config    map[string]string // driver-specific key/value config
}

func Detect(hints *Config) DriverSet {
    ds := DriverSet{Config: make(map[string]string)}

    // ── Index: CXL DIMM via kernel DAX driver ────────────────
    if info := cxlinfo.Detect(); info.Available {
        ds.Index = "cxl"
        ds.Config["cxl_numa"] = strconv.Itoa(info.PreferredNUMA())
        ds.Config["cxl_dax"]  = info.DAXDevice   // e.g. /dev/dax0.0
    } else {
        ds.Index = "mmap"
        ds.Config["mmap_path"] = hints.IndexPath
    }

    // ── Backend: SPDK if socket alive + devices bound ────────
    if hints.SPDKSocket != "" && spdk.SocketAlive(hints.SPDKSocket) && spdk.HasBoundDevices() {
        ds.Backend = "spdk"
        ds.Config["socket"] = hints.SPDKSocket
        ds.Config["bdev"]   = hints.SPDKBdev
    } else {
        ds.Backend = "localfs"
        ds.Config["data_dir"] = hints.DataDir
    }

    // ── Transport: DPDK if hugepages + PMD device present ────
    if dpdk.HugepagesAvailable() && dpdk.HasPMDDevice() {
        ds.Transport = "dpdk"
    } else {
        ds.Transport = "tcp"
    }

    // ── EC: ISA-L via runtime CPUID ──────────────────────────
    if isal.Available() {
        ds.EC = "isal"
    } else {
        ds.EC = "reedsolomon"
    }

    // Explicit flag overrides take final precedence
    if hints.BackendOverride != "" { ds.Backend = hints.BackendOverride }
    if hints.IndexOverride != ""   { ds.Index   = hints.IndexOverride }

    return ds
}

In main.go the startup sequence becomes a clean three-step pipeline:

flag.Parse()
driver.Detect(hints)
plugin.Open*(ds.*)
server.New()

Blob Backends

The three built-in storage backends and their registration patterns. Each lives in its own sub-package and registers itself without any dependency on core.

goblob/localfs/register.go — always compiled
package localfs

import "github.com/nabustore/nabustore/plugin"

func init() {
    plugin.RegisterBackend("localfs", func(cfg map[string]string) (blob.BlobBackend, error) {
        return NewLocalFSBackend(cfg["data_dir"])
    })
}
goblob/spdk/register.go — build tag: spdk
//go:build spdk

package spdk

import "github.com/nabustore/nabustore/plugin"

func init() {
    plugin.RegisterBackend("spdk", func(cfg map[string]string) (blob.BlobBackend, error) {
        return NewSPDKBackend(cfg["socket"], cfg["bdev"], cfg["core_mask"])
    })
}
goblob/grpc/register.go — Tier 2 out-of-process
package grpcbackend

import "github.com/nabustore/nabustore/plugin"

func init() {
    // The gRPC backend name is the socket path prefix.
    // plugin.OpenBackend("grpc:kioxia", cfg) → dials unix:///var/lib/nabustore/plugins/kioxia.sock
    plugin.RegisterBackend("grpc", func(cfg map[string]string) (blob.BlobBackend, error) {
        return NewGRPCBackend(cfg["socket_path"])
    })
}

Transport

The transport subsystem handles intra-cluster replication traffic and DHT ring messages. It is separated from the blob backend because DPDK NICs and SPDK NVMe devices use fundamentally different programming models that must not be mixed.

DPDK / IB RDMA incompatibility: A ConnectX-7 NIC in native InfiniBand mode is owned by rdma-core/libibverbs and cannot simultaneously be driven by DPDK's mlx5 PMD. SPDK owns IB NICs (NVMe-oF RDMA path); DPDK owns ConnectX Ethernet NICs (replication/RoCE path). These must never share a physical NIC.
gotransport/dpdk/register.go — build tag: dpdk
//go:build dpdk

package dpdk

import "github.com/nabustore/nabustore/plugin"

func init() {
    plugin.RegisterTransport("dpdk", func(cfg map[string]string) (transport.Transport, error) {
        return NewDPDKTransport(cfg["pci_addr"], cfg["core_mask"], cfg["mem_size"])
    })
}

EC Engines

The EC engine interface is already implicit in ec/ec.go. The migration moves the klauspost/reedsolomon implementation into ec/reedsolomon/ and adds an ISA-L CGO backend for AVX-512 / NEON production paths.

goec/isal/register.go — build tag: isal
//go:build isal
// +build isal
//
// CGO linkage: -lisal  (Intel Intelligent Storage Acceleration Library)
// ISA-L must be installed: dnf install isa-l-devel / apt install libisal-dev

package isal

import "github.com/nabustore/nabustore/plugin"

func init() {
    plugin.RegisterEC("isal", func(cfg map[string]string) (ec.Engine, error) {
        data, _ := strconv.Atoi(cfg["data_shards"])
        parity, _ := strconv.Atoi(cfg["parity_shards"])
        return NewISALEngine(data, parity)  // CGO call into gf_vect_mul / ec_encode_data
    })
}

The isal.Available() function used by driver/autodetect.go performs a runtime CPUID check — it is always compiled in even without the isal build tag, so the autodetector can query it on any binary.

Index Backends

Prerequisite: index/index.go currently exports a concrete Robin Hood hashmap rather than an interface. Extracting index.Index is a hard prerequisite for the mmap/CXL split. This is the first task in the migration path below.
goindex/cxl/cxl.go — DAX-backed, zero vendor software required
// CXLIndex is a Robin Hood hash map backed by a Linux DAX device.
// No vendor software required — uses the standard kernel cxl_mem driver
// (in from 5.12, production-stable from 6.5).
//
// Detection path:
//   1. /sys/bus/cxl/devices/*  — enumerate CXL DIMMs
//   2. /dev/dax*               — find corresponding DAX character device
//   3. mmap(O_RDWR|MAP_SHARED) — map the DAX range directly
//   4. clflushopt/clwb         — persistence without msync() overhead
//
// The layout on the DAX device is identical to the mmap/file backend
// so a node can be rebooted without re-indexing.

type CXLIndex struct {
    base    *mmapIndex  // shared Robin Hood logic
    dax     *os.File
    numaNode int
}

func New(daxPath string, numaNode int) (*CXLIndex, error) {
    f, err := os.OpenFile(daxPath, os.O_RDWR, 0600)
    if err != nil {
        return nil, fmt.Errorf("cxl_index: open %s: %w", daxPath, err)
    }
    // ... mmap + validate magic header ...
}

Writing a Vendor Plugin

A vendor supplying, for example, a Kioxia XD7P FlashArray backend follows these steps to produce a Tier 1 plugin with no changes to nabustore core.

Option A — Tier 1: Go package
Recommended. Zero overhead. Compiled into the nabustore binary.
✓ Sub-microsecond hot path
✓ No IPC overhead
✓ Full interface coverage
Option B — Tier 2: gRPC binary
No Go required. Ships as a compiled binary. Suitable for control path.
△ ~2–5µs UDS round-trip
△ Control path / bulk-put only
△ BlobService proto only

Tier 1 Step-by-Step

1

Create your Go module

Module path github.com/kioxia/nabustore-kioxia. Add github.com/nabustore/nabustore as a dependency — only interface packages, no core.

2

Implement the interface

Create a struct that satisfies blob.BlobBackend. All methods must be goroutine-safe. Return blob.ErrBlobExists, blob.ErrBlobNotFound, etc. for correct error signalling.

3

Register in init()

Call plugin.RegisterBackend("kioxia-xd7p", factory) in a package-level init(). The factory receives a map[string]string config from the DriverSet.

4

Publish the package

Tag a release. The operator adds one blank import to cmd/nabustore/plugins.go: _ "github.com/kioxia/nabustore-kioxia". No other changes required.

gogithub.com/kioxia/nabustore-kioxia — register.go
package kioxia

import (
    "github.com/nabustore/nabustore/blob"
    "github.com/nabustore/nabustore/plugin"
)

func init() {
    plugin.RegisterBackend("kioxia-xd7p", func(cfg map[string]string) (blob.BlobBackend, error) {
        return NewXD7PBackend(
            cfg["endpoint"],   // e.g. "192.168.1.10:4420"
            cfg["nqn"],        // NVMe Qualified Name for the subsystem
            cfg["ns_id"],      // Namespace ID
        )
    })
}

Build Composition

Build tags remain the mechanism for gating hardware-specific CGO dependencies. The difference is that tags now only affect which packages get blank-imported in plugins.go — they never appear in core logic.

TargetBuild commandIncluded backends
Development / CIgo build ./cmd/nabustorelocalfs, tcp, reedsolomon, mmap
Production NVMego build -tags spdk ./cmd/nabustore+ spdk
Production fullgo build -tags "spdk dpdk isal" ./cmd/nabustore+ spdk, dpdk, isal
Vendor (Kioxia)go build -tags kioxia ./cmd/nabustore+ kioxia-xd7p (Tier 1)
ARM (Grace)GOARCH=arm64 go build -tags isal ./cmd/nabustoreisal → NEON path via ISA-L ARM

Compatibility Matrix

Which tiers are appropriate for each subsystem and workload type.

SubsystemTier 1 (in-process)Tier 2 (gRPC)Hot path safeCXL/auto-detect
Blob backend ✓ Required for reads △ Bulk-put only Tier 1 only N/A
Transport ✗ Not supported △ hugepages probe
EC engine ✗ Not supported ✓ CPUID at startup
Index ✗ Not supported ✓ DAX probe

Migration Path

The migration is fully additive — no existing interface is changed. Each step can be merged independently and is safe to deploy without completing the full migration.

Prerequisites

Blocker Extract index.Index interface from index/index.go. Currently exports a concrete Robin Hood hashmap. All callers use the concrete type directly — this must become an interface before the mmap/CXL split can proceed cleanly. Effort ~2 hours. Existing contract tests in blob/local_test.go already exercise the full interface surface — they provide the test harness for the new interface. Risk Low. Pure refactor — no behaviour change, no new dependencies.

Ordered Steps

1

Extract index.Index interface

Define interface in index/index.go, move concrete impl to index/mmap/mmap.go. Update all call sites. Required before any further split.

2

Create plugin/registry.go

New package. No dependencies on existing code except interfaces. Does not break anything — nothing imports it yet.

3

Move localfs and spdk into sub-packages

Add register.go with init() to each. Update cmd/nabustore/plugins.go with blank imports. Remove the if backend == "spdk" branches from server/server.go.

4

Create transport package

Define Transport interface. Move existing TCP replication logic to transport/tcp/. Move DPDK build-tag code to transport/dpdk/.

5

Create driver/autodetect.go

Wraps existing cxl/util.go detection. Adds SPDK socket probe and DPDK hugepage check. Returns DriverSet. Simplify main.go to the three-step pipeline.

6

Add CXL index plugin

Implement index/cxl/ backed by kernel DAX (/dev/dax*). Register as "cxl". Autodetector selects it when a DAX device is present.

7

Add ISA-L EC engine

Implement ec/isal/ with CGO bindings to ISA-L. Register as "isal". isal.Available() compiled into all binaries for the autodetector.

8

Add Tier 2 gRPC backend

Implement blob/grpc/grpc_backend.go. Add plugin manager with inotify watch on /var/lib/nabustore/plugins/. Register any discovered .sock files as "grpc:<name>" backends.