---
layout: single
title:  "利用多张 GPU 训练大语言模型"
date:   2025-04-04 12:00:00 +0800
categories: [AI 与大模型, 硬件加速]
tags: [Train, LLM, DeepSpeed, ZeRO, FlashAttention, Quantization, 李宏毅, 2025]
---

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.002.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.003.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.004.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.005.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.006.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.007.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.008.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.009.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.010.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.011.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.012.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.013.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.014.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.015.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.016.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.017.jpeg)

![](/images/2025/TrainingLargeLanguageModels/训练大语言模型.018.jpeg)


## 参考资料
- [李宏毅生成式 AI 时代下的机器学习（2025）助教课：利用多张 GPU 训练大型语言模型——从零开始介绍 DeepSpeed、Liger Kernel、Flash Attention 及 Quantization](https://www.bilibili.com/video/BV1GsZRYtEUY)
- [【生成式AI時代下的機器學習(2025)】助教課：利用多張GPU訓練大型語言模型—從零開始介紹DeepSpeed、Liger Kernel、Flash Attention及Quantization](https://www.youtube.com/watch?v=mpuRca2UZtI)
- [Excalidraw](https://excalidraw.com/#json=3csTedqWVrLNRaESA8Z8i,XWDzmtI4xomLwmCB23BvFg)
- [The Ultra-Scale Playbook: Training LLMs on GPU Clusters](https://huggingface.co/spaces/nanotron/ultrascale-playbook)