DeepX has unveiled a second-generation chip project designed to handle AI inference for models with up to 100 billion parameters while targeting a power profile of less than 5 W. By integrating LPDDR5X Processing-in-Memory (PIM) technology, the company claims a 6.7x performance gain over its first-generation hardware, which was primarily utilized for vision tasks.
Hardware Architecture and Performance Claims
The core of DeepX's new project is the shift to LPDDR5X PIM, a technology that performs computations directly within the memory modules to reduce data movement bottlenecks. According to the primary source, this architecture is intended to support large-scale model inference with a focus on power efficiency.
The company reports performance targets of 80 TOP/s and 150 GB/s of memory bandwidth. These specifications are intended to facilitate inference speeds of 10 to 30 tokens per second, though the source does not specify the exact precision (such as INT4 or FP16) used to achieve these figures.
MoE Optimization and Parameter Scaling
While the headline highlights 100B parameters, the technical feasibility of running such a large dense model on this hardware is a point of discussion. Commenters on Hacker News noted that the chip is likely designed for Mixture-of-Experts (MoE) models, which only activate a fraction of their total parameters during inference.
Specifically, the hardware is expected to handle models with 20 billion to 100 billion total parameters by utilizing 3 billion to 20 billion active parameters at any given time. This distinction is critical because it allows for the scale of a 100B model without the linear compute requirements of a dense architecture.
Technical Caveats and Power Realities
Several technical limitations were identified in the Hacker News discussion regarding the current state of the project. One commenter, writing as augment_me, pointed out that the reported performance numbers are simulated and that the compiler is a prototype rather than a production-ready tool.
Furthermore, there is a discrepancy between the marketing claim of 5 W and the actual power requirements for high-performance tasks. A commenter noted that the performance figures may actually require approximately 13.5 W (2.7x the base power), and that the 5 W figure might only apply to smaller models under 4 billion parameters.
Trade-offs and Future Outlook
The move toward 'compute-in-memory' offers significant efficiency gains but may come with increased costs. One commenter suggested that this architecture could lead to higher memory prices due to the complexity of the PIM modules.
Additionally, the relative performance of this chip compared to consumer hardware remains a point of analysis. Commenters observed that while the chip is efficient, its memory bandwidth might limit its ability to handle large models in FP16 precision, potentially making it more suitable for specialized home AI applications or specific MoE configurations.