推理llama3[1]

LLaMA-Factory 安装

# 安装llamafactory-cli命令
git clone <https://github.com/hiyouga/LLaMA-Factory.git>
conda create -n llama_factory python=3.10
conda activate llama_factory
cd LLaMA-Factory
pip install -e .[metrics]

推理llama3

#模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('LLM-Research/Meta-Llama-3-8B-Instruct')

# 配置
$ vim examples/inference/llama3.yaml
model_name_or_path: /home/wei/models/model/LLM-Research/Meta-Llama-3-8B-Instruct
template: llama3

# 阿里云必须加这句，不然页面会报异常
$ export GRADIO_ROOT_PATH=/${JUPYTER_NAME}/proxy/7860/

# 启动
$ llamafactory-cli webchat examples/inference/llama3.yaml

• Command

  

• WebUI

  

分布训练[3]

DDP

DeepSpeed

FSDP

参考

1. LLaMA-Factory QuickStart