论文

• 论文地址 A Survey on Efficient Inference for Large Language Models

  Tsinghua University    Jul 2024
  

1 INTRODUCTION [1, 2, 3]

• Computational cost: the amount of float-point operations
• Memory access cost: the amount of data that read or written between off-chip HBM and GPU chip
• Memory usage: the amount of weights/KV cache/activation that stored in HBM

                         *图注：大语言模型推理效率瓶颈分析图示   图1：大模型部署挑战*

2 PRELIMINARIES