d-Matrix

Sid Sheth, Founder & CEO of d-Matrix, discusses how his inference computing company builds silicon, subsystems, and software for ultra-low latency computing, completing two acquisitions in its first year to scale systems and software expertise. He highlights d-Matrix's partnership with NVIDIA to embed its Raptor XPU—the world's first 3D stacked DRAM-based XPU with co-packaged DRAM on compute—into NVIDIA's rack ecosystem, enabling 144 Raptor XPUs to communicate in large-scale domain racks as AI interconnects become critical for scaling workloads.

Sree Ganesan, VP of Product at d-Matrix, announces the company's Squadron cluster-scale solution and demonstrates the industry's first live attention feed-forward disaggregation on heterogeneous compute, running a 20 billion parameter model with attention on NVIDIA H100 GPUs while experts execute on d-Matrix's Corsair nodes at over 800 tokens per second per user. The AFD approach optimizes performance by leveraging Corsair's memory bandwidth architecture for memory-bound expert workloads while keeping attention operations on GPUs, enabling higher batch processing and efficient node utilization.

Sid Sheth, CEO of d-Matrix, introduces their Jetream IO accelerator product that works alongside their Corsair compute accelerator to enable multi-node AI workload scaling. The solution uses PCI cards supporting up to eight Corsair accelerators per node, with the Jetream IO accelerator enabling cross-node scaling through an Ethernet switch.

Max Sbabo, Senior Staff Engineer at d-Matrix, presents the company's rack scale AI inference accelerator at the OCP Summit, highlighting their collaboration with OCP through ODSA workgroups, Bunch of Wires standard implementation, and pioneering block floating-point numerics. He announces d-Matrix's SquadRack product developed with Arista, Super Micro, and Broadcom following OCP's Open Rack Specification V3 standards, along with support for the new EON (Ethernet scaleup network) on their roadmap.