Multi-Core Cluster

GSys LibreCore multi-core is a SoC product, not a second copy of the core pipeline. Each core instance stays a single-tile CPU with a unique mhartid base; the cluster wires N masters through a coherence hub into shared L2/L3.

Authority: architecture/multi-core/README.md (U6.2).

Intent

N coherent cores (parameterized 1…8), shared LLC, write-invalidate coherence, scaled CLINT/PLIC, and SMP Linux. SMT threads per core remain NrHarts ≤ 2, so software hart count ≈ NrCores × NrHarts.

N × ariane (core tile)
        │ AXI masters
        ▼
  coherence hub / snoop filter   (corev_apu/coherence/)
        ▼
  shared L2 → optional L3 → PF → DRAM

Live building blocks

N=1 remains identity: hub and multi-master paths stay inert.

Coherence policies

cva6_cfg_t.CohPolicy selects the invalidation strategy (coh_policy_t):

COH_OOO is the envelope for out-of-order cores: check_cfg admits it only when OoOEn && L2En && NrCores > 1 && DCacheType == WT and the I$/D$ line widths match (the !FpPresent term left with M4 — FP loads replay like any load, and g6lc64_ooo_int2_l3 runs RV64GC+B on it). On this fabric every load is revalidated against its accepted physical address until it retires — a modification event holds load retirement for one cycle and matching loads replay through the existing precise cancel path; PMA non-idempotent loads never replay. Verified with directed LSQ/scoreboard tests, mutations, a bounded formal ledger and the strict four-hart OpenSBI boots of g6lc64_ooo_int2 (17,870,562 cycles) and g6lc64_ooo_int2_l3 (18,297,381 at ring 16).

Hub invariants held by g6lc_coherence_hub.sv:

• AW/AR slot ownership; a writer’s B is withheld until the invalidation is applied (slot_applied), tracked by enqueue/dequeue sequence numbers on g6lc_inval_bus.sv.
• Atomic slot lifetime: a slot is retained until both its B and required RLAST complete, in either order.
• Per-core offered-B lock and per-slot same-(core, original-ID) B predecessor masks preserve same-ID B ordering; a younger same-(core, id) AR is withheld while the older read slot is live, so re-tagged reads keep same-ID R order.
• ATOP/lock classified as invalidating writers; full-width response-ID bounds.
• Signature presence is acquired on AR and AW: a core whose first contact with a line is a write still holds a forwarding copy (write buffer, post-ACK repair entry), so it is a sharer. On the WT side those repair copies are dropped by an invalidation of their index, and a write acknowledged after an invalidation hit it in flight retains no copy — the directed cross-core shared-line program passes in the isolated two-core route with both cores retired (plan T7i).
• The hub’s transaction credits (CohMaxOutstanding, default 4, legal 2–14; 8 on the int2 packages with L2MshrDepth/L3MshrDepth 8) bound L2 fills plus the posted-write and bypass-read trackers; the credits bench shows the eight-read burst using all 8 slots and the mixed-burst write stall dropping to 0 at 8.
• Same-line ordering (M1a): an AR to a line with a pending or live same-line write is ineligible until that write’s B (closes the stale-refill counterexample that the standalone hub scenario 23 documented), and an invalidation to a core with a same-line fill in flight is held until the fill lands. The ATOP R that this hold could deadlock against is parked in its AW slot and replayed once the invalidation is applied — the same treatment the B channel already had (hub scenario 27, lifetime lane 50/50).
• CBO reaches every level (M1a): g6lc_cmo_engine in the cluster broadcasts the L1 invalidation to all cores, match-invalidates L2 and L3, and completes clean/flush on the write-idle trackers; see L2 / L3 Cache.
• The hub never sees the L3. g6lc64_ooo_int2_l3 places a non-inclusive L3 below the L2 (L3InclusiveEn=0); the composed stack bench carries the stalled-write / self-invalidation / CMO scenarios through hub → L2 → L3, and the four-hart strict boot passes on it. The WT boundary emits allocate attributes (WtAxiAllocEn) — before it the L2 in this envelope was a bypass.
• Write-back L1 types are refused on multi-core packages (check_cfg): the hub/L2 track presence, not dirty data, so a WB victim could carry the only copy.

The composed hub + real g6lc_l2_top bench (tb_g6lc_coherence_l2 in verif/tb/l2/tb_g6lc_coherence_hub.sv) shows the L2 refusing a reader’s AR during a stalled write and admitting it after memory B; with posted writes the same contract (late_ar ≥ mem_b) holds because the slave B is still memory’s B.

Local AMO/CAS operations raise the same apply event as external invalidations (tb_g6lc_wt_amo_apply), PMU group 2 exposes “invalidation applied” (event 5) and “COH_OOO load replay” (event 6), and the signature SRAM carries ImplKey("g6lc_coh_signature") with its macro/MBIST plan in corev_apu/coherence/g6lc_ooo_snoop_filter.tech-spec.md.

Qualified today (source-profiled strict OpenSBI, single-threaded Verilator, per-hart traces): g6lc64_ooo_int2 four harts, g6lc64_ooo_int2_l3 four harts (RV64GC+B, ring 16), g6lc64_smt2_l3, and the in-order two-core HPDCACHE packages g6lc64_stream8_l3 / g6lc64_server_math_l3 (M6). Deferred (tracked, not closed): Linux boot and riscv-arch-test compliance on these packages, RVWMO litmus coverage across harts, foundry macro selection and MBIST insertion, STA/power/area sign-off — see AGENTS-todo.md. HPDCACHE packages use COH_FILTERED; COH_OOO stays a WT-L1 envelope.

Config and inference

When geometry fields are 0, build_config_pkg can infer L2/L3/prefetch and snoop-filter sizing from NrCores (e.g. L2 ≈ max(256 KiB, N×128 KiB)). Exact formulas live in core/include/build_config_pkg.sv — do not hardcode in docs.

check_cfg enforces multi-core legality (e.g. S when N>1, MMU present, snoop filter entries when filter enabled).

Contention optimisations (product view)

• Snoop filter guided invalidation
• Invalidate coalesce
• Multi-master AXI RR + anti-starve
• Non-cacheable bypass for MMIO
• Inclusive L3→L2/L1 when configured

Software / DTS

• N× cpu@ nodes with consecutive hart ids (account for SMT)
• PLIC contexts matching M/S per core
• CLINT extents covering all harts
• next-level-cache = <&l2> (and L3 when present)

Validate with AGENTS-dts-validation.md / build-platform/scripts/validate-cva6-dts.*.

Verification

g6lc-build verify --target g6lc64_ooo_int2         # two-core COH_OOO coherence cluster
g6lc-build verify --target g6lc64_ooo_int2_l3      # + L3, FP, ring 16
g6lc-build verify --target g6lc64_stream8_l3       # in-order two-core HPDCACHE + L3
g6lc-build verify --target g6lc64_ooo_server       # opt-in / unqualified (COH_FILTERED leg)
g6lc-build test --suite mc-stream-tests

Zacas (AMOCAS) is partial on selected packages (W/D); see multicore directed tests under verif/tests/custom/multicore/.

Invariants

• RVWMO across harts
• Cluster-wide LR/SC + AMO discipline
• MMIO never cached
• Precise traps per hart

Related

• L2 / L3 Cache
• SMT — threads × cores
• Server Profiles
• SKU Matrix
• Upgrade Program