RFC 0001: Repository structure and conventions

Date: 2026-07-08

Summary

Moraine brings a SlateDB backend to DuckLake: a DuckLake catalog implemented on a transactional KV store over object storage, instead of the usual relational catalog database. This RFC establishes the repository layout, crate boundaries, and engineering conventions for the project.

Goals

Open-source best practices from the start (licensing, CI, semver, docs), growing into a production quality dependency.
Core catalog logic developed and tested as pure Rust, independent of DuckDB.
Conventions enforced by tooling wherever possible; prose only for what can’t be mechanized.

Crate structure

A Cargo workspace with a virtual root (no root crate):

Crate	Type	Role
`moraine`	lib	Core: DuckLake catalog semantics on SlateDB. The flagship crate.
`moraine-duckdb`	cdylib (+ linked C++ shim)	DuckDB extension wrapping the core. Thin by policy — language-agnostic: no domain logic, only `StorageExtension` registration, C-ABI marshalling, and the sync↔async bridge. Composition and the DuckLake catalog contract are specified by RFC 0006. If logic accumulates here, it belongs in the core.
`xtask`	bin (unpublished)	Automation: build/package the extension, orchestrate the e2e suite. Rust instead of shell scripts/Makefiles — cross-platform and type-checked.

Library-first: the catalog logic lives in moraine, free of the FFI/build tax and embeddable in any Rust host. The extension is one consumer of it.

Repository layout

moraine/
├── Cargo.toml                    # workspace root: [workspace.dependencies], [workspace.lints], shared metadata
├── rust-toolchain.toml           # pinned stable toolchain + components
├── rustfmt.toml
├── deny.toml                     # cargo-deny: licenses, advisories, sources
├── AGENTS.md                     # conventions source of truth (for agents and humans)
├── CLAUDE.md -> AGENTS.md        # symlink
├── README.md                     # badges (CI/crates.io/docs.rs), what/why, status banner, quickstart, architecture sketch, license
├── ROADMAP.md                    # phased checkbox roadmap
├── CONTRIBUTING.md
├── CHANGELOG.md                  # generated by release-plz; never hand-edited
├── SECURITY.md                   # vulnerability reporting policy (GitHub private advisories)
├── CODE_OF_CONDUCT.md            # Contributor Covenant
├── LICENSE-MIT / LICENSE-APACHE  # dual license (Rust ecosystem standard)
├── Cargo.lock                    # committed (current guidance even for libraries); CI runs --locked
├── .github/
│   ├── workflows/
│   │   ├── ci.yml                # fmt, clippy, test, doc, deny, e2e
│   │   └── release.yml           # release-plz (dormant until first release)
│   ├── dependabot.yml            # weekly updates: cargo + npm + github-actions ecosystems
│   ├── ISSUE_TEMPLATE/           # bug report + feature request
│   └── PULL_REQUEST_TEMPLATE.md
├── crates/
│   ├── moraine/
│   │   ├── Cargo.toml
│   │   ├── src/
│   │   ├── examples/             # runnable usage examples, compiled by CI
│   │   └── tests/                # Integration tests
│   └── moraine-duckdb/
│       ├── Cargo.toml
│       ├── src/
│       └── tests/                # e2e tests
├── xtask/
│   └── src/main.rs
├── benchmark/
│   └── clickbench/               # a ClickBench system entry, run by `cargo xtask clickbench`
└── docs/
    └── rfcs/
        ├── README.md             # the RFC process
        └── 0000-template.md

benchmark/ holds benchmark suites that are not ours: a third-party benchmark defines a contract its own driver calls, and a directory per suite keeps that shape intact instead of bending it into an xtask. The automation around one — building what it needs, scoring its output — stays in xtask. A suite’s own queries are not vendored: ClickBench is CC BY-NC-SA, which does not mix with this repository’s licensing, so they are read out of a checkout at run time.

Document roles: README = what this is and how to use it. ROADMAP = where it’s going. RFCs = why it is the way it is. AGENTS.md = how to work in it.

Community files (SECURITY.md, CODE_OF_CONDUCT.md, issue/PR templates) exist from day one: they cost minutes to add, GitHub surfaces them in its community UI, and a security-reporting channel especially must predate the first external user — a catalog backend is exactly the kind of dependency people will report vulnerabilities against.

RFC process

Files are docs/rfcs/NNNN-kebab-title.md. RFCs carry no status field: every RFC in the directory is the current, binding design; superseding one means updating or replacing it (with a pointer back), not re-labeling it.
An RFC is required for decisions that are expensive to reverse: on-disk/KV key layout, commit/transaction protocol, public API shape. Optional for everything else.
RFCs double as an ADR log. RFC 0002 is expected to be the SlateDB key encoding for DuckLake catalog state.
Design documents produced by brainstorming/design sessions are written directly as RFCs in this directory. There is no separate specs directory; this location overrides any tooling default (e.g. docs/superpowers/specs/).

Core crate skeleton (`crates/moraine`)

src/
├── lib.rs            # crate docs + re-exports only; no logic
├── error.rs          # thiserror-based Error enum + crate Result alias
├── catalog.rs        # DuckLake domain: snapshots, schemas, tables, data-file metadata
├── data_file.rs      # Parquet layer: scoped reads of registered data files
├── store.rs          # SlateDB layer: key layout, value codec
└── transaction.rs            # commit protocol: catalog transaction → atomic SlateDB write

Layering: catalog never touches SlateDB directly; store knows nothing about DuckLake semantics; data_file is the only module that reads Parquet, and knows nothing about either. The small API between them keeps catalog logic testable against an in-memory store and concentrates key-encoding decisions in one reviewable place.
Two kinds of read: store reads the catalog’s own keyspace out of SlateDB; data_file reads the Parquet objects DuckLake registers, fetching only the columns and rows an index entry needs. Keeping them separate is what makes the second one auditable — a bounded projection is a different thing from a scan, and the boundary is where that stays visible.
Module growth: start as foo.rs; split into foo.rs + foo/ submodules when needed (no mod.rs files — enforced via clippy mod_module_files).
Visibility: private by default; pub only what lib.rs deliberately re-exports. #![warn(missing_docs)] on the core crate.
Docs teach by example: the crate root doc is a worked example, not a summary; key public items carry doctests (compiled by CI, so they can’t rot); examples/ holds at least one runnable end-to-end program. This is the single strongest habit of the best crates.
Feature flags: additive-only (enabling a feature never breaks or removes API), each documented in the crate root. docs.rs builds with all-features = true and --cfg docsrs so feature-gated items render with their requirement badge.
Async: the core API is async end-to-end (SlateDB requires tokio). The core never creates a runtime; moraine-duckdb owns a tokio runtime and blocks on core futures at the FFI boundary.
Errors: one crate-level Error enum (thiserror) with variants per failure domain: store I/O, corrupt/unexpected state, commit conflict, invalid catalog operation. No panics in library code (mechanized below).

This skeleton is a starting guess at the seams. If an implementation RFC (key layout, commit protocol) reveals better boundaries, the skeleton bends to the RFC, not vice versa.

Testing strategy

Tier	Home	What	CI cadence
Unit	colocated `#[cfg(test)]` mods	Tricky internals: key encoding, codecs, conflict resolution. Property-based tests (`proptest`) are mandatory for anything in `store` that encodes/decodes: roundtrips, key-ordering preservation. Encoding bugs in a catalog are data-corruption bugs.	every push/PR
Integration	`crates/moraine/tests/`	Public API against real SlateDB on in-memory `object_store`. No mocks of the store layer. Covers snapshot visibility, concurrent-commit conflicts, and the named crash cases of RFC 0011 — `CommitNotDurable` and `CommitDurableNotAcknowledged` for the ordinary commit, the rest for drop, genesis, takeover, and the resumable operations — driven as data rather than as whichever sequences an author imagined.	every push/PR
E2E	`crates/moraine-duckdb/tests/`, via `cargo xtask e2e`	Build the cdylib, load into real DuckDB, run actual DuckLake SQL. Validates our assumptions about what DuckLake demands, not just that the code does what we think.	separate required PR job
Real object storage	`crates/moraine/tests/object_storage.rs`, via `cargo xtask s3`	Public-API round-trips against an S3-compatible endpoint. `#[ignore]`d by default; the xtask starts a pinned RustFS, creates the bucket, and runs them un-ignored. The same tests point at a real bucket through `MORAINE_S3_ENDPOINT`.	separate required PR job
Fuzzing	(future) `fuzz/` via `cargo-fuzz`	Fuzz targets for `store` codecs and the commit protocol’s read-path (decode arbitrary bytes without panicking). The project’s worst failure mode is catalog corruption; fuzzing is the cheapest insurance against it. Added once the codecs stabilize.	scheduled/nightly

Process: TDD (test first, watch it fail, implement). Every bugfix lands with a regression test. Tests assert on behavior through public APIs, except Unit.

Lint and format policy

Lives in [workspace.lints], inherited by every crate:

clippy::pedantic at warn, with a small allow-list — each entry documented where it’s allowed.
clippy::unwrap_used, clippy::expect_used, clippy::panic denied in library code; allowed in tests and xtask.
unsafe_code forbidden in moraine; allowed in moraine-duckdb (FFI), where every unsafe block carries a // SAFETY: comment (clippy::undocumented_unsafe_blocks at deny).
rustfmt.toml: near-defaults plus imports_granularity = "Crate" and grouped import ordering.

C/C++ formatting and analysis use the LLVM version in .clang-tools-version. cargo xtask fmt-cpp [--check] applies or checks .clang-format on handwritten sources and headers; dependencies and the generated ABI header are excluded. cargo xtask tidy-cpp configures the release build and applies the explicit checks in .clang-tidy to both shim configurations and the standalone drivers. Formatting runs in pre-commit and CI; static analysis runs in CI and the full local gate. Both commands reject mismatched tool versions.

CI

One workflow (ci.yml), parallel jobs, all required:

Job	Enforces
`fmt`	`cargo +nightly-2026-08-29 fmt --check`, `cargo xtask fmt-cpp --check`
`clippy`	`cargo clippy --workspace --all-targets`, warnings denied
`test`	`cargo test --workspace --locked` (Unit + Integration, doctests, examples)
`doc`	`cargo doc` with `RUSTDOCFLAGS="-D warnings"`
`deny`	`cargo deny check -D advisory-not-detected` — a stale `ignore` (one whose advisory no longer matches any crate) is otherwise only a warning and still exits 0, so a suppression can outlive the vulnerability it was written for
`e2e`	`cargo xtask tidy-cpp`, `cargo xtask e2e` (C/C++ analysis and E2E; separate job so it doesn’t gate fast feedback)

A fuzz/ directory holds the cargo-fuzz targets: key_decode and record_decode over the store codecs and the read path, and crash_recovery over the commit protocol’s atomicity guarantee (RFC 0011). It is its own workspace — the targets need nightly’s sanitizer and coverage instrumentation, and must stay out of the ordinary build, test, and lint pass — and runs on a nightly schedule rather than per-PR, a fuzz run being a search rather than a check. Its corpus is cached between runs so each night resumes the search instead of restarting it.

Toolchain pinned via rust-toolchain.toml. rust-version declared in workspace metadata and gated by an MSRV job that reads the declared value and compiles the workspace with that toolchain, so the promise cannot drift from what builds. Rust build caching through mbx (jdx/mr-boxington-action), the same store that serves a developer’s worktrees, for CI speed; the nightly fuzz job keeps Swatinem/rust-cache, which saves on a scheduled run where that action does not. Workflows set a concurrency group with cancel-in-progress so superseded pushes don’t burn CI minutes. Dependabot keeps both cargo dependencies and pinned GitHub Actions current (weekly, so deny advisories surface as PRs rather than red CI).

Release and git conventions

Conventional commits (feat:, fix:, docs:, refactor:, …). Load-bearing, not cosmetic: release-plz derives changelogs and version bumps from them.
release-plz on release.yml: opens a release PR with version bumps + CHANGELOG; merging publishes to crates.io. Configured from the start, dormant until the first release.
Both published crates share a workspace version ([workspace.package] version) and release in lockstep. Decoupling is a post-1.0 problem.
Pre-1.0 semver: breaking changes bump the minor version. Stated in the README. From the first release onward, release-plz runs cargo-semver-checks, catching accidental breaking changes mechanically, not by reviewer vigilance.
PRs into main; squash-merge so main stays a clean conventional-commit history.

Alternatives considered

Extension-first (cdylib as the primary crate): rejected — pays the FFI/build/version-pinning tax during the phase where all the hard problems are pure Rust. The E2E suite mitigates the risk of drift from real DuckLake behavior instead.
Single crate, split later: rejected — both crates are already committed; converting a root crate to a workspace later touches paths, CI, and metadata for no savings.
Lean scaffold, add tooling later (no xtask/release config/templates up front): rejected in favor of full scaffold, given the stated open-source-release ambition; cargo-deny and licensing especially are cheap now and painful to retrofit.