D-Matrix’s unique compute platform, known as the Corsair C8, can make a big claim to displace Nvidia’s industry-leading H100 GPU — at least according to some surprising test results the startup has published.
Designed specifically for generative AI workloads, the Corsair C8 differs from GPUs in that it uses d-Matrix’s unique digital-in-memory computer (DIMC) architecture.
The result? A nine-fold increase in throughput versus the industry-leading Nvidia H100 and a 27-fold increase versus the A100.
Corsair C8 power
The startup is one of the hottest in Silicon Valley, raising $110 million from investors in its latest funding round, including funding from Microsoft. It came in April 2022 with a $44 million investment round from backers including Microsoft, SK Hynix and others.
Its flagship Corsair C8 card includes 2,048 DIMC cores with 130 billion transistors and 256GB of LPDDR5 RAM. It boasts 2,400 to 9,600 TFLOPS of computing performance and has a chip-to-chip bandwidth of 1TB/s
These unique cards can deliver 20x higher throughput for generative inference on large language models (LLMS), 20x lower inference latency for LLM, and 30x cost savings compared to traditional GPUs.
As generative AI expands rapidly, the industry is caught in a race to build increasingly powerful hardware to power future generations of technology.
The leading components are the GPUs and specifically Nvidia’s A100 and new H100 units. But according to d-Matrix, GPUs are not optimized for LLM inference and require too many GPUs to handle AI workloads, leading to high energy consumption.
This is because the bandwidth demands of running AI inference cause GPUs to spend too much time idle, waiting for data to arrive from DRAM. Moving data out of DRAM also means higher power consumption along with lower throughput and additional latency. This means the cooling demand is then increased.
The solution, the firm claims, is its special DIMC architecture that alleviates many of the problems with GPUs. D-Matrix claims its solution can reduce costs by 10 to 20 times – and in some cases 60 times.
Beyond d-Matrix’s technology, other players are emerging in the race to overtake Nvidia’s H100. Presented by IBM New analog AI chip Aug. which mimics the human brain and can work 14 times more efficiently.