Abstract: As computational devices continue to diversify and computational power grows rapidly, the increasing number of large language models (LLMs) has made efficient multi-model inference across heterogeneous platforms a complex and formidable challenge. To address this, we propose  OpenLM, a high-performance inference framework to support efficient deployment of multiple LLMs on diverse hardware platforms. The OpenLM framework boasts extensive model compatibility, providing efficient performance support for a wide range of models. It incorporates high-performance computing operators optimized for multiple platforms and architectures to maximize hardware performance. Meanwhile, OpenLM features a flexible framework architecture that facilitates rapid integration and support for the latest models. To further optimize memory (both GPU and CPU memory) consumption, task scheduling, and system stability during the inference process, the framework introduces features such as Paged- Attention mechanisms, dynamic batching, weight quantization, and KV cache quantization. According to the experimental results, these optimization strategies effectively enhance inference efficiency, reduce resource overhead, and bolster overall framework performance.

Key words: deep learning, large language model (LLM), high-performance computing (HPC), LLM inference framework