Cxl

https://www.arm.com/architecture/system-architectures/amba/amba-specifications

why CXL

similar to message passing vs shared memory in programming. But applied in hardware.

Without cohenrency, it is unclear if a memory is being modified. Forcing a ownership transfer at some point.

where CXL is not applicable

Traditional non-coherent I/O devices mainly rely on standard Producer-Consumer ordering models and execute against Host-attached memory. For such devices, there is little interaction with the Host except for work submission and signaling on work completion boundaries. Such accelerators also tend to work on data streams or large contiguous data objects. These devices typically do not need the advanced capabilities provided by CXL, and traditional PCIe* is sufficient as an accelerator-attached medium.

what CXL is good for

CXL allow for other ordering than just Producer-Consumer. It also allows a large amount of atomic operations.

how concurrency can be done

1. snoopy bus

send all requests to all processors (boardcast)
processor snoops the request and decide if it needs to respond
boardcast works well with small number of processors
bus contention makes sure only write will go through

write invalidate: when one processor writes, all other processor’s cache line is invalidated write boadcast: when one processor writes, all other processor’s cache line is updated

snooping refers to the processor silently monitoring the bus and update its own status based on the bus transaction.

2. directory-based cache coherence

no boardcast, no longer need to hold the bus

Local node: the node that the request originates from Home node: the node that holds the memory (may just hold a mapping considering that SN will perform actual memory access)

Subordinate node: service the underlying request, typically memory controller or peripheral device

AMBA CHI

HN-F: Fully coherent Home Node

https://developer.arm.com/documentation/101481/0200/Components-and-configuration/Components/Fully-coherent-Home-Node–HN-F-

HN-I: I/O coherent Home Node (HN-I) -> converts CHI transactions to AMBA transactions

without DMT (data memory transfer)

RN -> HN -> SN -> HN -> RN

with DMT, helps skip the HN

RN -> HN -> SN -> RN

with out DCT

HN node can snoop the bus and notice that a prior RN has cached the data, so it can retrieve from Cache instead of going to memory.

RN -> HN -> another processor -> HN -> RN

Direct Cache Transfer, helps skip the HN

RN -> HN -> another processor -> RN

CXL type

type 1: device without memory = cxl.cache + cxl.io type 2: device with memory = cxl.mem + cxl.cache + cxl.io type 3: memory expansion = cxl.mem + cxl.io

cxl.io

non-coherent protocol for IO access basically PCIe

cxl.cache

protocol for transferring one 64 byte cache line

D2H, H2D

bias based coherency

Host-bias: meaning that host has cache line so a coherency protocol needs to be followed Device-bias: meaning that device do not need to worry about host cache line and can operate on the memory very fast

cxl.mem

• HDM-H (Host-only Coherent): Used only for Type 3 Devices • HDM-D (Device Coherent): Used only for legacy Type 2 Devices that rely on CXL.cache to manage coherence with the Host • HDM-DB (Device Coherent using Back-Invalidate): Can be used by Type 2 Devices or Type 3 Devices

M2S, S2M : master to subordinate, subordinate to master

HDM-DB allows type 2 device to operate on other type 3 device’s memory. (this is called direct P2P)

HDM

Host-managed Device Memory.

Device-attached memory that is mapped to system coherent address space and accessible to the Host using standard write-back semantics.

Memory located on a CXL device can be mapped as either HDM or PDM.

HDM decoder

HDM decoder enables the routing of both CXL.mem and UIO transactions targeting HDM. HDM Decoders are responsible for translating incoming Host Physical Addresses (HPA) into corresponding Device Physical Addresses (DPA), allowing memory accesses to be correctly directed to the appropriate device

(mainly for memory interleaving access)

HPA

Host Physical Address.

PDM

Private Device Memory.

DPA

Device Physical Address. DPA forms a device-scoped flat address space. An LD-FAM device presents a distinct DPA space per LD. A G-FAM device presents the same DPA space to all hosts. The CXL HDM decoders or GFD decoders map HPA into DPA space.

LD: logical device

FAM

Fabric-Attached Memory. HDM within a Type 2 or Type 3 device that can be made accessible to multiple hosts concurrently. Each HDM region can either be pooled (dedicated to a single host) or shared (accessible concurrently by multiple hosts).

APN

Alternate Protocol Negotiation -> transtion PCIe protocol to CXL protocol

register space

There are two sets of register space in CXL, one that resides under configuration space and the another set that resides under memory mapped space.

Configuration space defined register space Memory mapped register space

All configuration space registers are identified as PCIe defined designated vendor-specific extended capabilities (DVSEC) registers with DVSEC Vendor ID as h1E98 (reserved for CXL). DVSEC ID associated with capability helps determine the type of CXL capability

atomic operations

POC - point of coherency - i.e. the device owning the cache LCN - locally caching node - the requester-local caceh that might hold a remote copy

POC == GL2 for GPU , SLC for CPU

note there is GMEM vs R-GMEM

R-GMEM - remote GPU memory for different socket (in the atomic context)

H-SLC vs R-SLC H is home, R is remote indicating different CPU socket

SMMU

IO coherent vs fully coherent

https://developer.arm.com/documentation/109242/0100/System-architecture-considerations/I-O-coherency

A device is I/O coherent with the PE caches if its transactions snoop the PE caches for cacheable regions of memory. This improves performance by avoiding Cache Maintenance Operation (CMO).

The device does not need to access the external memory.

The PE does not snoop the device cache.

atomic vs reduction

reduction means to not return the old value to caller vs atomic means to return the old value to caller

how to check PCIe atomic capabilities

DevCap2: AtomicOpsCap: Routing+ 32bit+ 64bit+ 128bitCAS+ -> determine if PCIe can complete atomic requests DevCtl2: AtomicOpsCtl: ReqEn- EgressBlck+ -> determine if PCIe can issue atomic requests -> determine if PCIe can block on atomic requests to be forwarded to other devices