华为 Atlas 800I A2 服务器的大模型推理性能压测

大模型推理性能压测工具
- 安装 EvalScope
- 压测命令的使用
数据集
实验结果对比
- 🏆 vLLM ⚔️ XInference (T4: 4X16G)
- 🏆 MindIE (910B4: 8X32G) ⚔️ vLLM (T4: 4X16G)
实验结果（MindIE）
实验结果（vLLM）
- Qwen1.5-7B-Chat
- Qwen2.5-72B-Chat
实验结果（XInference - MindIE）
实验结果（Nvidia T4: XInference - vLLM）
- Qwen2-7B-Instruct
实验结果（Nvidia T4: vLLM）
- Qwen2-7B-Instruct
- Qwen2-7B-Instruct (long.jsonl)
参考资料

大模型推理性能压测工具

安装 EvalScope

git clone https://github.com/modelscope/evalscope
cd evalscope

pip install -e .

压测命令的使用

evalscope perf \
    --api openai \
    --url 'http://127.0.0.1:1025/v1/chat/completions' \
    --model 'qwen' \
    --dataset openqa \
    --dataset-path './datasets/open_qa.jsonl' \
    --max-prompt-length 8000 \
    --stop '<|im_end|>' \
    --read-timeout=120 \
    --parallel 100 \
    -n 1000

❌ –stream 不要加，经常出问题。

--read-timeout: 网络读取超时
--parallel: 并发数
-n: 请求数

数据集

中文聊天 HC3-Chinese

mkdir datasets
wget https://modelscope.cn/datasets/AI-ModelScope/HC3-Chinese/resolve/master/open_qa.jsonl \
    -O datasets/open_qa.jsonl

压测命令

evalscope perf \
    --api openai \
    --url 'http://127.0.0.1:1025/v1/chat/completions' \
    --model 'qwen' \
    --dataset openqa \
    --dataset-path './datasets/open_qa.jsonl' \
    --max-prompt-length 8000 \
    --stop '<|im_end|>' \
    --read-timeout=120 \
    --parallel 1 \
    -n 1

代码问答 Codefuse-Evol-Instruct-Clean

wget https://modelscope.cn/datasets/Banksy235/Codefuse-Evol-Instruct-Clean/resolve/master/data.json \
    -O datasets/Codefuse-Evol-Instruct-Clean-data.jsonl

# 修改数据集格式，将 "input" 改为 "question"，以适应 EvalScope 的数据集格式 openqa
sed -i 's/"input"/"question"/g' datasets/Codefuse-Evol-Instruct-Clean-data.jsonl

压测命令

evalscope perf \
    --api openai \
    --url 'http://127.0.0.1:1025/v1/chat/completions' \
    --model 'qwen' \
    --dataset openqa \
    --dataset-path './datasets/Codefuse-Evol-Instruct-Clean-data.jsonl' \
    --max-prompt-length 4000 \
    --stop '<|im_end|>' \
    --read-timeout=120 \
    --parallel 1 \
    -n 1

构造长输入和输出的数据集

编辑文件：datasets/long.jsonl

{"question":"Learning to Reason with LLMs\nWe are introducing OpenAI o1, a new large language model trained with reinforcement learning to perform complex reasoning. o1 thinks before it answers—it can produce a long internal chain of thought before responding to the user.\n\nContributions\nOpenAI o1 ranks in the 89th percentile on competitive programming questions (Codeforces), places among the top 500 students in the US in a qualifier for the USA Math Olympiad (AIME), and exceeds human PhD-level accuracy on a benchmark of physics, biology, and chemistry problems (GPQA). While the work needed to make this new model as easy to use as current models is still ongoing, we are releasing an early version of this model, OpenAI o1-preview, for immediate use in ChatGPT and to trusted API users⁠(opens in a new window).\n\nOur large-scale reinforcement learning algorithm teaches the model how to think productively using its chain of thought in a highly data-efficient training process. We have found that the performance of o1 consistently improves with more reinforcement learning (train-time compute) and with more time spent thinking (test-time compute). The constraints on scaling this approach differ substantially from those of LLM pretraining, and we are continuing to investigate them.\n\nThe image shows two scatter plots comparing o1 AIME accuracy during training and at test time. Both charts have pass@1 accuracy on the y-axis and compute (log scale) on the x-axis. The dots indicate increasing accuracy with more compute time.\no1 performance smoothly improves with both train-time and test-time compute\n\nEvals\nTo highlight the reasoning improvement over GPT-4o, we tested our models on a diverse set of human exams and ML benchmarks. We show that o1 significantly outperforms GPT-4o on the vast majority of these reasoning-heavy tasks. Unless otherwise specified, we evaluated o1 on the maximal test-time compute setting.\n\nCompetition evals for Math (AIME 2024), Code (CodeForces), and PhD-Level Science Questions (GPQA Diamond)\no1 greatly improves over GPT-4o on challenging reasoning benchmarks. Solid bars show pass@1 accuracy and the shaded region shows the performance of majority vote (consensus) with 64 samples.\nBreakdown of the accuracy and raw score of gpt-4o vs. o1 on various competition evals\no1 improves over GPT-4o on a wide range of benchmarks, including 54/57 MMLU subcategories. Seven are shown for illustration.\nIn many reasoning-heavy benchmarks, o1 rivals the performance of human experts. Recent frontier models1 do so well on MATH2 and GSM8K that these benchmarks are no longer effective at differentiating models.\nA score of 13.9 places it among the top 500 students nationally and above the cutoff for the USA Mathematical Olympiad.\n\nWe also evaluated o1 on GPQA diamond, a difficult intelligence benchmark which tests for expertise in chemistry, physics and biology. In order to compare models to humans, we recruited experts with PhDs to answer GPQA-diamond questions. We found that o1 surpassed the performance of those human experts, becoming the first model to do so on this benchmark. These results do not imply that o1 is more capable than a PhD in all respects — only that the model is more proficient in solving some problems that a PhD would be expected to solve. On several other ML benchmarks, o1 improved over the state-of-the-art. With its vision perception capabilities enabled, o1 scored 78.2 on MMMU, making it the first model to be competitive with human experts. It also outperformed GPT-4o on 54 out of 57 MMLU subcategories.\n\nChain of Thought\nSimilar to how a human may think for a long time before responding to a difficult question, o1 uses a chain of thought when attempting to solve a problem. Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses. It learns to recognize and correct its mistakes. It learns to break down tricky steps into simpler ones. It learns to try a different approach when the current one isn’t working. This process dramatically improves the model’s ability to reason. To illustrate this leap forward, we showcase the chain of thought from o1-preview on several difficult problems below.\n\nCoding\nWe trained a model that scored 213 points and ranked in the 49th percentile in the 2024 International Olympiad in Informatics (IOI), by initializing from o1 and training to further improve programming skills. This model competed in the 2024 IOI under the same conditions as the human contestants. It had ten hours to solve six challenging algorithmic problems and was allowed 50 submissions per problem.\n\nFor each problem, our system sampled many candidate submissions and submitted 50 of them based on a test-time selection strategy. Submissions were selected based on performance on the IOI public test cases, model-generated test cases, and a learned scoring function. If we had instead submitted at random, we would have only scored 156 points on average, suggesting that this strategy was worth nearly 60 points under competition constraints.\n\nWith a relaxed submission constraint, we found that model performance improved significantly. When allowed 10,000 submissions per problem, the model achieved a score of 362.14 – above the gold medal threshold – even without any test-time selection strategy.  \n\nFinally, we simulated competitive programming contests hosted by Codeforces to demonstrate this model’s coding skill. Our evaluations closely matched competition rules and allowed for 10 submissions. GPT-4o achieved an Elo rating3 of 808, which is in the 11th percentile of human competitors.\nHuman preference evaluation\nIn addition to exams and academic benchmarks, we also evaluated human preference of o1-preview vs GPT-4o on challenging, open-ended prompts in a broad spectrum of domains. In this evaluation, human trainers were shown anonymized responses to a prompt from o1-preview and GPT-4o, and voted for which response they preferred. o1-preview is preferred to gpt-4o by a large margin in reasoning-heavy categories like data analysis, coding, and math. However, o1-preview is not preferred on some natural language tasks, suggesting that it is not well-suited for all use cases.\n\nThe image shows a horizontal bar chart comparing five models scores with error bars representing confidence intervals. The x-axis ranges from 0 to 100, with a dashed line as a reference point for performance.\nSafety\nChain of thought reasoning provides new opportunities for alignment and safety. We found that integrating our policies for model behavior into the chain of thought of a reasoning model is an effective way to robustly teach human values and principles.\nWhat does all of this mean for founders in the AI market? What does this mean for incumbent software companies? And where do we, as investors, see the most promising layer for returns in the Generative AI stack?\nIn our latest essay on the state of the Generative AI market, we’ll explore how the consolidation of the foundational LLM layer has set the stage for the race to scale these higher-order reasoning and agentic capabilities, and discuss a new generation of “killer apps” with novel cognitive architectures and user interfaces.\nThis is where System 2 thinking comes in, and it’s the focus of the latest wave of AI research. When a model “stops to think,” it isn’t just generating learned patterns or spitting out predictions based on past data. It’s generating a range of possibilities, considering potential outcomes and making a decision based on reasoning. \n\nTranslate to France."}

输入和输出 Tokens 大约在 3500

压测命令

evalscope-perf http://127.0.0.1:1025/v1/chat/completions qwen \
     ./datasets/long.jsonl \
     --max-prompt-length 8000 \
     --read-timeout=120 \
     --parallels 1 \
     --n 1

实验结果对比

🏆 vLLM ⚔️ XInference (T4: 4X16G)

从结果看，在生产环境中还是要使用 vLLM，推理性能更好且稳定更棒。

代码

import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties

# 设置中文字体
font_path = '/System/Library/Fonts/Hiragino Sans GB.ttc'  # 替换为你的字体文件路径
font_prop = FontProperties(fname=font_path)

# 数据
batch_sizes = [8, 16, 32, 64, 100, 128, 150, 200, 300, 400, 500]

vllm_qps = [0.970, 1.588, 2.443, 3.503, 3.593, 3.580, 3.509, 3.076, 2.847, 2.820, 3.060]
xinf_qps = [0.783, 1.288, 1.958, 2.472, 2.353, 2.334, 2.046, 1.750, 1.664, 1.254, 1.163]

vllm_latency = [8.213, 9.944, 12.846, 16.913, 19.182, 19.831, 17.806, 16.743, 16.285, 16.285, 17.649]
xinf_latency = [10.128, 12.307, 15.749, 19.225, 19.235, 19.151, 17.479, 15.949, 16.718, 14.750, 15.771]

vllm_throughput = [298.860, 496.458, 753.176, 1073.697, 1039.610, 1005.881, 1032.794, 925.476, 839.437, 816.049, 875.565]
xinf_throughput = [254.695, 424.701, 631.260, 753.852, 700.681, 700.324, 637.260, 550.200, 510.211, 392.697, 362.167]

vllm_failures = [0, 0, 0, 15, 73, 115, 26, 11, 19, 23, 26]
xinf_failures = [1, 0, 12, 71, 101, 72, 32, 25, 66, 47, 89]

# 创建子图
fig, axs = plt.subplots(2, 2, figsize=(15, 10))

# QPS
axs[0, 0].plot(batch_sizes, vllm_qps, label='vLLM', marker='o')
axs[0, 0].plot(batch_sizes, xinf_qps, label='XInferencev(LLM)', marker='o')
axs[0, 0].set_title('QPS', fontproperties=font_prop)
axs[0, 0].set_xlabel('并行数', fontproperties=font_prop)
axs[0, 0].set_ylabel('QPS', fontproperties=font_prop)
axs[0, 0].legend()

# 延迟
axs[0, 1].plot(batch_sizes, vllm_latency, label='vLLM', marker='o')
axs[0, 1].plot(batch_sizes, xinf_latency, label='XInference(vLLM)', marker='o')
axs[0, 1].set_title('延迟', fontproperties=font_prop)
axs[0, 1].set_xlabel('并行数', fontproperties=font_prop)
axs[0, 1].set_ylabel('延迟 (秒)', fontproperties=font_prop)
axs[0, 1].legend()

# 吞吐量
axs[1, 0].plot(batch_sizes, vllm_throughput, label='vLLM', marker='o')
axs[1, 0].plot(batch_sizes, xinf_throughput, label='XInference(vLLM)', marker='o')
axs[1, 0].set_title('吞吐量', fontproperties=font_prop)
axs[1, 0].set_xlabel('并行数', fontproperties=font_prop)
axs[1, 0].set_ylabel('吞吐量 (每秒Tokens)', fontproperties=font_prop)
axs[1, 0].legend()

# 失败率
axs[1, 1].plot(batch_sizes, vllm_failures, label='vLLM', marker='o')
axs[1, 1].plot(batch_sizes, xinf_failures, label='XInference(vLLM)', marker='o')
axs[1, 1].set_title('失败率', fontproperties=font_prop)
axs[1, 1].set_xlabel('并行数', fontproperties=font_prop)
axs[1, 1].set_ylabel('失败数', fontproperties=font_prop)
axs[1, 1].legend()

# 调整布局
plt.tight_layout()
plt.show()

🏆 MindIE (910B4: 8X32G) ⚔️ vLLM (T4: 4X16G)

和我们现有服务器 T4 的性能对比

代码

import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties

# 设置中文字体
font_path = '/System/Library/Fonts/Hiragino Sans GB.ttc'  # 替换为你的字体文件路径
font_prop = FontProperties(fname=font_path)

# 数据
batch_sizes_mindie = [8, 16, 32, 64, 128, 150, 200, 256, 300, 400, 512, 720]
batch_sizes_vllm = [8, 16, 32, 64, 100, 128, 150, 200, 300, 400, 500]

mindie_qps = [2.474, 4.649, 8.273, 12.065, 18.924, 20.457, 22.294, 23.392, 22.868, 23.328, 24.007, 24.643]
vllm_qps = [0.970, 1.588, 2.443, 3.503, 3.593, 3.580, 3.509, 3.076, 2.847, 2.820, 3.060]

mindie_latency = [3.213, 3.391, 3.724, 4.974, 6.108, 6.517, 7.805, 9.208, 10.904, 13.628, 16.031, 18.790]
vllm_latency = [8.213, 9.944, 12.846, 16.913, 19.182, 19.831, 17.806, 16.743, 16.285, 16.285, 17.649]

mindie_throughput = [594.929, 1119.102, 1989.159, 2903.023, 4559.489, 4920.856, 5354.701, 5636.846, 5506.567, 5618.110, 5772.230, 5940.622]
vllm_throughput = [298.860, 496.458, 753.176, 1073.697, 1039.610, 1005.881, 1032.794, 925.476, 839.437, 816.049, 875.565]

mindie_failures = [0] * len(batch_sizes_mindie)  # 假设 MindIE(910B4 8*32) 没有失败数据
vllm_failures = [0, 0, 0, 15, 73, 115, 26, 11, 19, 23, 26]

# 创建子图
fig, axs = plt.subplots(2, 2, figsize=(15, 10))

# QPS
axs[0, 0].plot(batch_sizes_mindie, mindie_qps, label='MindIE(910B4 8*32)', marker='o')
axs[0, 0].plot(batch_sizes_vllm, vllm_qps, label='vLLM(T4 4*16)', marker='o')
axs[0, 0].set_title('QPS', fontproperties=font_prop)
axs[0, 0].set_xlabel('并行数', fontproperties=font_prop)
axs[0, 0].set_ylabel('QPS', fontproperties=font_prop)
axs[0, 0].legend()

# 延迟
axs[0, 1].plot(batch_sizes_mindie, mindie_latency, label='MindIE(910B4 8*32)', marker='o')
axs[0, 1].plot(batch_sizes_vllm, vllm_latency, label='vLLM(T4 4*16)', marker='o')
axs[0, 1].set_title('延迟', fontproperties=font_prop)
axs[0, 1].set_xlabel('并行数', fontproperties=font_prop)
axs[0, 1].set_ylabel('延迟 (秒)', fontproperties=font_prop)
axs[0, 1].legend()

# 吞吐量
axs[1, 0].plot(batch_sizes_mindie, mindie_throughput, label='MindIE(910B4 8*32)', marker='o')
axs[1, 0].plot(batch_sizes_vllm, vllm_throughput, label='vLLM(T4 4*16)', marker='o')
axs[1, 0].set_title('吞吐量', fontproperties=font_prop)
axs[1, 0].set_xlabel('并行数', fontproperties=font_prop)
axs[1, 0].set_ylabel('吞吐量 (每秒Tokens)', fontproperties=font_prop)
axs[1, 0].legend()

# 失败率
axs[1, 1].plot(batch_sizes_mindie, mindie_failures, label='MindIE(910B4 8*32)', marker='o')
axs[1, 1].plot(batch_sizes_vllm, vllm_failures, label='vLLM(T4 4*16)', marker='o')
axs[1, 1].set_title('失败率', fontproperties=font_prop)
axs[1, 1].set_xlabel('并行数', fontproperties=font_prop)
axs[1, 1].set_ylabel('失败数', fontproperties=font_prop)
axs[1, 1].legend()

# 调整布局
plt.tight_layout()
plt.show()

实验结果（MindIE）

Qwen1.5-7B-Chat

指标	8	16	32	64	128	150	200	256	300	400	512	720
用时	404.284	215.085	120.876	82.884	52.844	48.884	44.856	42.750	43.729	42.866	41.655	40.580
QPS	2.474	4.649	8.273	12.065	18.924	20.457	22.294	23.392	22.868	23.328	24.007	24.643
延迟	3.213	3.391	3.724	4.974	6.108	6.517	7.805	9.208	10.904	13.628	16.031	18.790
吞吐量	594.929	1119.102	1989.159	2903.023	4559.489	4920.856	5354.701	5636.846	5506.567	5618.110	5772.230	5940.622
p50	3.2461	3.4271	3.7514	5.0248	6.1491	6.5487	7.7782	9.1754	10.9164	13.9850	17.0675	20.4161
p90	4.9771	5.2905	5.8484	7.7522	9.5705	10.1980	12.3493	13.6320	15.8667	19.2667	22.7640	26.8183

平均每个请求的输入 token 数: 40
平均每个请求的输出 token 数: 240

parallel 8

Benchmarking summary: 
 Time taken for tests: 404.284 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 2.474
 Average latency: 3.213
 Throughput(average output tokens per second): 594.929
 Average time to first token: 3.213
 Average input tokens per request: 40.296
 Average output tokens per request: 240.520
 Average time per output token: 0.00168
 Average package per request: 1.000
 Average package latency: 3.213
 Percentile of time to first token: 
   p50: 3.2461
   p66: 3.7587
   p75: 4.2213
   p80: 4.4208
   p90: 4.9771
   p95: 5.6460
   p98: 6.3678
   p99: 6.8545
 Percentile of request latency: 
   p50: 3.2461
   p66: 3.7587
   p75: 4.2213
   p80: 4.4208
   p90: 4.9771
   p95: 5.6460
   p98: 6.3678
   p99: 6.8545

parallel 16

Benchmarking summary: 
 Time taken for tests: 215.085 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 4.649
 Average latency: 3.391
 Throughput(average output tokens per second): 1119.102
 Average time to first token: 3.391
 Average input tokens per request: 40.296
 Average output tokens per request: 240.702
 Average time per output token: 0.00089
 Average package per request: 1.000
 Average package latency: 3.391
 Percentile of time to first token: 
   p50: 3.4271
   p66: 3.9816
   p75: 4.3792
   p80: 4.6188
   p90: 5.2905
   p95: 5.9389
   p98: 6.7555
   p99: 7.2478
 Percentile of request latency: 
   p50: 3.4271
   p66: 3.9816
   p75: 4.3792
   p80: 4.6188
   p90: 5.2905
   p95: 5.9389
   p98: 6.7555
   p99: 7.2478

parallel 32

Benchmarking summary: 
 Time taken for tests: 120.876 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 8.273
 Average latency: 3.724
 Throughput(average output tokens per second): 1989.159
 Average time to first token: 3.724
 Average input tokens per request: 40.296
 Average output tokens per request: 240.442
 Average time per output token: 0.00050
 Average package per request: 1.000
 Average package latency: 3.724
 Percentile of time to first token: 
   p50: 3.7514
   p66: 4.3989
   p75: 4.8352
   p80: 5.1087
   p90: 5.8484
   p95: 6.5664
   p98: 7.3057
   p99: 8.0644
 Percentile of request latency: 
   p50: 3.7514
   p66: 4.3989
   p75: 4.8352
   p80: 5.1087
   p90: 5.8484
   p95: 6.5664
   p98: 7.3057
   p99: 8.0644

parallel 64

Benchmarking summary: 
 Time taken for tests: 82.884 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 12.065
 Average latency: 4.974
 Throughput(average output tokens per second): 2903.023
 Average time to first token: 4.974
 Average input tokens per request: 40.296
 Average output tokens per request: 240.615
 Average time per output token: 0.00034
 Average package per request: 1.000
 Average package latency: 4.974
 Percentile of time to first token: 
   p50: 5.0248
   p66: 5.8985
   p75: 6.4676
   p80: 6.8351
   p90: 7.7522
   p95: 8.8266
   p98: 9.9534
   p99: 10.6036
 Percentile of request latency: 
   p50: 5.0248
   p66: 5.8985
   p75: 6.4676
   p80: 6.8351
   p90: 7.7522
   p95: 8.8266
   p98: 9.9534
   p99: 10.6036

parallel 128

Benchmarking summary: 
 Time taken for tests: 52.844 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 18.924
 Average latency: 6.108
 Throughput(average output tokens per second): 4559.489
 Average time to first token: 6.108
 Average input tokens per request: 40.296
 Average output tokens per request: 240.943
 Average time per output token: 0.00022
 Average package per request: 1.000
 Average package latency: 6.108
 Percentile of time to first token: 
   p50: 6.1491
   p66: 7.2622
   p75: 7.9560
   p80: 8.3894
   p90: 9.5705
   p95: 10.7209
   p98: 12.2300
   p99: 13.1657
 Percentile of request latency: 
   p50: 6.1491
   p66: 7.2622
   p75: 7.9560
   p80: 8.3894
   p90: 9.5705
   p95: 10.7209
   p98: 12.2300
   p99: 13.1657

parallel 150

Benchmarking summary: 
 Time taken for tests: 48.884 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 20.457
 Average latency: 6.517
 Throughput(average output tokens per second): 4920.856
 Average time to first token: 6.517
 Average input tokens per request: 40.296
 Average output tokens per request: 240.550
 Average time per output token: 0.00020
 Average package per request: 1.000
 Average package latency: 6.517
 Percentile of time to first token: 
   p50: 6.5487
   p66: 7.7580
   p75: 8.4394
   p80: 8.9248
   p90: 10.1980
   p95: 11.4446
   p98: 13.0906
   p99: 13.7333
 Percentile of request latency: 
   p50: 6.5487
   p66: 7.7580
   p75: 8.4394
   p80: 8.9248
   p90: 10.1980
   p95: 11.4446
   p98: 13.0906
   p99: 13.7333

parallel 200

Benchmarking summary: 
 Time taken for tests: 44.856 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 22.294
 Average latency: 7.805
 Throughput(average output tokens per second): 5354.701
 Average time to first token: 7.805
 Average input tokens per request: 40.296
 Average output tokens per request: 240.188
 Average time per output token: 0.00019
 Average package per request: 1.000
 Average package latency: 7.805
 Percentile of time to first token: 
   p50: 7.7782
   p66: 9.2457
   p75: 10.0596
   p80: 10.8689
   p90: 12.3493
   p95: 13.7108
   p98: 15.1361
   p99: 16.3464
 Percentile of request latency: 
   p50: 7.7782
   p66: 9.2457
   p75: 10.0596
   p80: 10.8689
   p90: 12.3493
   p95: 13.7108
   p98: 15.1361
   p99: 16.3464

parallel 256

Benchmarking summary: 
 Time taken for tests: 42.750 seconds
 Expected number of requests: 1000
 Number of concurrency: 256
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 23.392
 Average latency: 9.208
 Throughput(average output tokens per second): 5636.846
 Average time to first token: 9.208
 Average input tokens per request: 40.296
 Average output tokens per request: 240.975
 Average time per output token: 0.00018
 Average package per request: 1.000
 Average package latency: 9.208
 Percentile of time to first token: 
   p50: 9.1754
   p66: 10.6423
   p75: 11.5348
   p80: 12.1507
   p90: 13.6320
   p95: 14.9237
   p98: 16.5329
   p99: 18.0215
 Percentile of request latency: 
   p50: 9.1754
   p66: 10.6423
   p75: 11.5348
   p80: 12.1507
   p90: 13.6320
   p95: 14.9237
   p98: 16.5329
   p99: 18.0215

parallel 300

Benchmarking summary: 
 Time taken for tests: 43.729 seconds
 Expected number of requests: 1000
 Number of concurrency: 300
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 22.868
 Average latency: 10.904
 Throughput(average output tokens per second): 5506.567
 Average time to first token: 10.904
 Average input tokens per request: 40.296
 Average output tokens per request: 240.795
 Average time per output token: 0.00018
 Average package per request: 1.000
 Average package latency: 10.904
 Percentile of time to first token: 
   p50: 10.9164
   p66: 12.5896
   p75: 13.6076
   p80: 14.2442
   p90: 15.8667
   p95: 17.2967
   p98: 18.8841
   p99: 20.2304
 Percentile of request latency: 
   p50: 10.9164
   p66: 12.5896
   p75: 13.6076
   p80: 14.2442
   p90: 15.8667
   p95: 17.2967
   p98: 18.8841
   p99: 20.2304

parallel 400

Benchmarking summary: 
 Time taken for tests: 42.866 seconds
 Expected number of requests: 1000
 Number of concurrency: 400
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 23.328
 Average latency: 13.628
 Throughput(average output tokens per second): 5618.110
 Average time to first token: 13.628
 Average input tokens per request: 40.296
 Average output tokens per request: 240.828
 Average time per output token: 0.00018
 Average package per request: 1.000
 Average package latency: 13.628
 Percentile of time to first token: 
   p50: 13.9850
   p66: 15.7791
   p75: 16.8451
   p80: 17.6249
   p90: 19.2667
   p95: 20.8091
   p98: 22.5674
   p99: 23.6675
 Percentile of request latency: 
   p50: 13.9850
   p66: 15.7791
   p75: 16.8451
   p80: 17.6249
   p90: 19.2667
   p95: 20.8091
   p98: 22.5674
   p99: 23.6675

parallel 512

Benchmarking summary: 
 Time taken for tests: 41.655 seconds
 Expected number of requests: 1000
 Number of concurrency: 512
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 24.007
 Average latency: 16.031
 Throughput(average output tokens per second): 5772.230
 Average time to first token: 16.031
 Average input tokens per request: 40.296
 Average output tokens per request: 240.440
 Average time per output token: 0.00017
 Average package per request: 1.000
 Average package latency: 16.031
 Percentile of time to first token: 
   p50: 17.0675
   p66: 18.9757
   p75: 20.0632
   p80: 20.8715
   p90: 22.7640
   p95: 24.0828
   p98: 25.3913
   p99: 26.6549
 Percentile of request latency: 
   p50: 17.0675
   p66: 18.9757
   p75: 20.0632
   p80: 20.8715
   p90: 22.7640
   p95: 24.0828
   p98: 25.3913
   p99: 26.6549

parallel 720

Benchmarking summary: 
 Time taken for tests: 40.580 seconds
 Expected number of requests: 1000
 Number of concurrency: 720
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 24.643
 Average latency: 18.790
 Throughput(average output tokens per second): 5940.622
 Average time to first token: 18.790
 Average input tokens per request: 40.296
 Average output tokens per request: 241.071
 Average time per output token: 0.00017
 Average package per request: 1.000
 Average package latency: 18.790
 Percentile of time to first token: 
   p50: 20.4161
   p66: 22.8199
   p75: 24.1332
   p80: 24.8298
   p90: 26.8183
   p95: 28.8718
   p98: 30.2479
   p99: 31.1723
 Percentile of request latency: 
   p50: 20.4161
   p66: 22.8199
   p75: 24.1332
   p80: 24.8298
   p90: 26.8183
   p95: 28.8718
   p98: 30.2479
   p99: 31.1723

Qwen1.5-7B-Chat (long.jsonl)

指标	32	64	80	100
用时	227.466	176.405	177.402	176.430
QPS	0.879	1.134	1.127	1.134
延迟	34.012	51.059	62.032	73.359
吞吐量	1534.689	1889.768	1864.759	1869.820
p50	34.7112	48.4820	59.2757	80.3454
p90	36.6736	68.2014	84.9306	93.7045

平均每个请求的输入 token 数: 1614
平均每个请求的输出 token 数: 1654

parallel 32

Benchmarking summary: 
 Time taken for tests: 227.466 seconds
 Expected number of requests: 200
 Number of concurrency: 32
 Total requests: 200
 Succeed requests: 200
 Failed requests: 0
 Average QPS: 0.879
 Average latency: 34.012
 Throughput(average output tokens per second): 1534.689
 Average time to first token: 34.012
 Average input tokens per request: 1614.000
 Average output tokens per request: 1745.450
 Average time per output token: 0.00065
 Average package per request: 1.000
 Average package latency: 34.012
 Percentile of time to first token: 
   p50: 34.7112
   p66: 36.2749
   p75: 36.4008
   p80: 36.4714
   p90: 36.6736
   p95: 36.7247
   p98: 36.7508
   p99: 36.7635
 Percentile of request latency: 
   p50: 34.7112
   p66: 36.2749
   p75: 36.4008
   p80: 36.4714
   p90: 36.6736
   p95: 36.7247
   p98: 36.7508
   p99: 36.7635

parallel 64

Benchmarking summary: 
 Time taken for tests: 176.405 seconds
 Expected number of requests: 200
 Number of concurrency: 64
 Total requests: 200
 Succeed requests: 200
 Failed requests: 0
 Average QPS: 1.134
 Average latency: 51.059
 Throughput(average output tokens per second): 1889.768
 Average time to first token: 51.059
 Average input tokens per request: 1614.000
 Average output tokens per request: 1666.820
 Average time per output token: 0.00053
 Average package per request: 1.000
 Average package latency: 51.059
 Percentile of time to first token: 
   p50: 48.4820
   p66: 52.8716
   p75: 58.2624
   p80: 60.0386
   p90: 68.2014
   p95: 74.7256
   p98: 78.7428
   p99: 79.3227
 Percentile of request latency: 
   p50: 48.4820
   p66: 52.8716
   p75: 58.2624
   p80: 60.0386
   p90: 68.2014
   p95: 74.7256
   p98: 78.7428
   p99: 79.3227

parallel 80

Benchmarking summary: 
 Time taken for tests: 177.402 seconds
 Expected number of requests: 200
 Number of concurrency: 80
 Total requests: 200
 Succeed requests: 200
 Failed requests: 0
 Average QPS: 1.127
 Average latency: 62.032
 Throughput(average output tokens per second): 1864.759
 Average time to first token: 62.032
 Average input tokens per request: 1614.000
 Average output tokens per request: 1654.060
 Average time per output token: 0.00054
 Average package per request: 1.000
 Average package latency: 62.032
 Percentile of time to first token: 
   p50: 59.2757
   p66: 71.6039
   p75: 74.4594
   p80: 76.9160
   p90: 84.9306
   p95: 91.9959
   p98: 95.0497
   p99: 98.3784
 Percentile of request latency: 
   p50: 59.2757
   p66: 71.6039
   p75: 74.4594
   p80: 76.9160
   p90: 84.9306
   p95: 91.9959
   p98: 95.0497
   p99: 98.3784

parallel 100

Benchmarking summary: 
 Time taken for tests: 176.430 seconds
 Expected number of requests: 200
 Number of concurrency: 100
 Total requests: 200
 Succeed requests: 200
 Failed requests: 0
 Average QPS: 1.134
 Average latency: 73.359
 Throughput(average output tokens per second): 1869.820
 Average time to first token: 73.359
 Average input tokens per request: 1614.000
 Average output tokens per request: 1649.460
 Average time per output token: 0.00053
 Average package per request: 1.000
 Average package latency: 73.359
 Percentile of time to first token: 
   p50: 80.3454
   p66: 85.7741
   p75: 88.7250
   p80: 90.7941
   p90: 93.7045
   p95: 97.3420
   p98: 99.5837
   p99: 101.2294
 Percentile of request latency: 
   p50: 80.3454
   p66: 85.7741
   p75: 88.7250
   p80: 90.7941
   p90: 93.7045
   p95: 97.3420
   p98: 99.5837
   p99: 101.2294

Qwen1.5-14B-Chat

指标	8	16	32	64	128	150	200	256	512
用时	578.571	361.169	253.040	204.961	170.001	169.981	162.999	159.840	153.937
QPS	1.727	2.766	3.952	4.874	5.882	5.877	6.129	6.250	6.490
延迟	3.712	3.928	4.581	5.628	7.223	8.004	9.205	11.446	22.695
吞吐量	480.043	897.511	915.133	2333.955	1363.656	3621.096	4310.525	4333.013	3806.748
p50	3.7038	3.9261	4.4544	5.6047	7.0534	7.9484	9.0271	11.3066	23.6591
p90	5.7184	6.0562	6.9198	8.7597	11.1194	12.5181	14.5017	16.8646	33.3052

parallel 8

Benchmarking summary: 
 Time taken for tests: 578.571 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 1.727
 Average latency: 3.712
 Throughput(average output tokens per second): 480.043
 Average time to first token: 3.712
 Average input tokens per request: 40.287
 Average output tokens per request: 224.138
 Average time per output token: 0.00208
 Average package per request: 1.000
 Average package latency: 3.712
 Percentile of time to first token: 
   p50: 3.7038
   p66: 4.4215
   p75: 4.8051
   p80: 5.0476
   p90: 5.7184
   p95: 6.2956
   p98: 7.0707
   p99: 7.4415
 Percentile of request latency: 
   p50: 3.7038
   p66: 4.4215
   p75: 4.8051
   p80: 5.0476
   p90: 5.7184
   p95: 6.2956
   p98: 7.0707
   p99: 7.4415

parallel 16

Benchmarking summary: 
 Time taken for tests: 361.169 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 2.766
 Average latency: 3.928
 Throughput(average output tokens per second): 897.511
 Average time to first token: 3.928
 Average input tokens per request: 40.287
 Average output tokens per request: 223.750
 Average time per output token: 0.00111
 Average package per request: 1.000
 Average package latency: 3.928
 Percentile of time to first token: 
   p50: 3.9261
   p66: 4.6890
   p75: 5.1204
   p80: 5.3290
   p90: 6.0562
   p95: 6.6784
   p98: 7.3113
   p99: 7.8980
 Percentile of request latency: 
   p50: 3.9261
   p66: 4.6890
   p75: 5.1204
   p80: 5.3290
   p90: 6.0562
   p95: 6.6784
   p98: 7.3113
   p99: 7.8980

parallel 32

Benchmarking summary: 
 Time taken for tests: 253.040 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 3.952
 Average latency: 4.581
 Throughput(average output tokens per second): 915.133
 Average time to first token: 4.581
 Average input tokens per request: 40.296
 Average output tokens per request: 231.565
 Average time per output token: 0.00109
 Average package per request: 1.000
 Average package latency: 4.581
 Percentile of time to first token: 
   p50: 4.4544
   p66: 5.2905
   p75: 5.8235
   p80: 6.1074
   p90: 6.9198
   p95: 7.5185
   p98: 8.5143
   p99: 9.3296
 Percentile of request latency: 
   p50: 4.4544
   p66: 5.2905
   p75: 5.8235
   p80: 6.1074
   p90: 6.9198
   p95: 7.5185
   p98: 8.5143
   p99: 9.3296

parallel 64

Benchmarking summary: 
 Time taken for tests: 204.961 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 4.874
 Average latency: 5.628
 Throughput(average output tokens per second): 2333.955
 Average time to first token: 5.628
 Average input tokens per request: 40.287
 Average output tokens per request: 223.930
 Average time per output token: 0.00043
 Average package per request: 1.000
 Average package latency: 5.628
 Percentile of time to first token: 
   p50: 5.6047
   p66: 6.6600
   p75: 7.3423
   p80: 7.7040
   p90: 8.7597
   p95: 9.6390
   p98: 10.7844
   p99: 11.6003
 Percentile of request latency: 
   p50: 5.6047
   p66: 6.6600
   p75: 7.3423
   p80: 7.7040
   p90: 8.7597
   p95: 9.6390
   p98: 10.7844
   p99: 11.6003

parallel 128

Benchmarking summary: 
 Time taken for tests: 170.001 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 5.882
 Average latency: 7.223
 Throughput(average output tokens per second): 1363.656
 Average time to first token: 7.223
 Average input tokens per request: 40.296
 Average output tokens per request: 231.823
 Average time per output token: 0.00073
 Average package per request: 1.000
 Average package latency: 7.223
 Percentile of time to first token: 
   p50: 7.0534
   p66: 8.4098
   p75: 9.3191
   p80: 9.7640
   p90: 11.1194
   p95: 12.2800
   p98: 13.8248
   p99: 14.6733
 Percentile of request latency: 
   p50: 7.0534
   p66: 8.4098
   p75: 9.3191
   p80: 9.7640
   p90: 11.1194
   p95: 12.2800
   p98: 13.8248
   p99: 14.6733

parallel 150

Benchmarking summary: 
 Time taken for tests: 169.981 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 5.877
 Average latency: 8.004
 Throughput(average output tokens per second): 3621.096
 Average time to first token: 8.004
 Average input tokens per request: 40.287
 Average output tokens per request: 224.225
 Average time per output token: 0.00028
 Average package per request: 1.000
 Average package latency: 8.004
 Percentile of time to first token: 
   p50: 7.9484
   p66: 9.3969
   p75: 10.5065
   p80: 11.0508
   p90: 12.5181
   p95: 13.6289
   p98: 15.3693
   p99: 16.5225
 Percentile of request latency: 
   p50: 7.9484
   p66: 9.3969
   p75: 10.5065
   p80: 11.0508
   p90: 12.5181
   p95: 13.6289
   p98: 15.3693
   p99: 16.5225

parallel 200

Benchmarking summary: 
 Time taken for tests: 162.999 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 6.129
 Average latency: 9.205
 Throughput(average output tokens per second): 4310.525
 Average time to first token: 9.205
 Average input tokens per request: 40.287
 Average output tokens per request: 223.865
 Average time per output token: 0.00023
 Average package per request: 1.000
 Average package latency: 9.205
 Percentile of time to first token: 
   p50: 9.0271
   p66: 10.8104
   p75: 12.0957
   p80: 12.8212
   p90: 14.5017
   p95: 15.7233
   p98: 17.6891
   p99: 19.2401
 Percentile of request latency: 
   p50: 9.0271
   p66: 10.8104
   p75: 12.0957
   p80: 12.8212
   p90: 14.5017
   p95: 15.7233
   p98: 17.6891
   p99: 19.2401

parallel 256

Benchmarking summary: 
 Time taken for tests: 159.840 seconds
 Expected number of requests: 1000
 Number of concurrency: 256
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 6.250
 Average latency: 11.446
 Throughput(average output tokens per second): 4333.013
 Average time to first token: 11.446
 Average input tokens per request: 40.287
 Average output tokens per request: 224.384
 Average time per output token: 0.00023
 Average package per request: 1.000
 Average package latency: 11.446
 Percentile of time to first token: 
   p50: 11.3066
   p66: 13.1698
   p75: 14.4698
   p80: 15.2733
   p90: 16.8646
   p95: 18.3524
   p98: 20.0468
   p99: 21.3758
 Percentile of request latency: 
   p50: 11.3066
   p66: 13.1698
   p75: 14.4698
   p80: 15.2733
   p90: 16.8646
   p95: 18.3524
   p98: 20.0468
   p99: 21.3758

parallel 512

Benchmarking summary: 
 Time taken for tests: 153.937 seconds
 Expected number of requests: 1000
 Number of concurrency: 512
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 6.490
 Average latency: 22.695
 Throughput(average output tokens per second): 3806.748
 Average time to first token: 22.695
 Average input tokens per request: 40.287
 Average output tokens per request: 224.177
 Average time per output token: 0.00026
 Average package per request: 1.000
 Average package latency: 22.695
 Percentile of time to first token: 
   p50: 23.6591
   p66: 27.2753
   p75: 29.3330
   p80: 30.5309
   p90: 33.3052
   p95: 35.7777
   p98: 37.4897
   p99: 38.1186
 Percentile of request latency: 
   p50: 23.6591
   p66: 27.2753
   p75: 29.3330
   p80: 30.5309
   p90: 33.3052
   p95: 35.7777
   p98: 37.4897
   p99: 38.1186

Qwen2-72B-Chat

指标	8	16	32	64	128	150	200	256	512
用时	1569.707	909.001	567.479	382.247	179.015	270.054	251.060	237.063	206.734
QPS	0.636	1.099	1.759	2.613	5.586	3.699	3.975	4.214	4.832
延迟	11.705	12.806	14.526	17.296	21.041	23.856	28.198	33.176	59.912
吞吐量	188.589	342.764	588.262	973.795	1549.069	1595.176	1748.784	1866.155	1828.363
p50	11.8443	12.8275	14.7115	17.5290	21.3863	23.8551	28.4407	32.7329	63.0404
p90	16.2106	17.5466	20.0371	23.9890	29.5537	33.7459	40.4210	45.7666	81.6602

平均每个请求的输入 token 数: 40
平均每个请求的输出 token 数: 277

parallel 8

Benchmarking summary: 
 Time taken for tests: 1569.707 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 0.636
 Average latency: 11.705
 Throughput(average output tokens per second): 188.589
 Average time to first token: 11.705
 Average input tokens per request: 40.303
 Average output tokens per request: 277.618
 Average time per output token: 0.00530
 Average package per request: 1.000
 Average package latency: 11.705
 Percentile of time to first token: 
   p50: 11.8443
   p66: 13.1671
   p75: 13.9665
   p80: 14.5981
   p90: 16.2106
   p95: 17.8844
   p98: 20.0471
   p99: 23.0309
 Percentile of request latency: 
   p50: 11.8443
   p66: 13.1671
   p75: 13.9665
   p80: 14.5981
   p90: 16.2106
   p95: 17.8844
   p98: 20.0471
   p99: 23.0309

parallel 16

Benchmarking summary: 
 Time taken for tests: 909.001 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 1.099
 Average latency: 12.806
 Throughput(average output tokens per second): 342.764
 Average time to first token: 12.806
 Average input tokens per request: 40.303
 Average output tokens per request: 278.224
 Average time per output token: 0.00292
 Average package per request: 1.000
 Average package latency: 12.806
 Percentile of time to first token: 
   p50: 12.8275
   p66: 14.3998
   p75: 15.3983
   p80: 16.1443
   p90: 17.5466
   p95: 19.6906
   p98: 22.2533
   p99: 25.1283
 Percentile of request latency: 
   p50: 12.8275
   p66: 14.3998
   p75: 15.3983
   p80: 16.1443
   p90: 17.5466
   p95: 19.6906
   p98: 22.2533
   p99: 25.1283

parallel 32

Benchmarking summary: 
 Time taken for tests: 567.479 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 998
 Failed requests: 2
 Average QPS: 1.759
 Average latency: 14.526
 Throughput(average output tokens per second): 588.262
 Average time to first token: 14.526
 Average input tokens per request: 40.297
 Average output tokens per request: 277.259
 Average time per output token: 0.00170
 Average package per request: 1.000
 Average package latency: 14.526
 Percentile of time to first token: 
   p50: 14.7115
   p66: 16.2993
   p75: 17.4013
   p80: 18.2002
   p90: 20.0371
   p95: 21.8216
   p98: 24.5539
   p99: 27.3373
 Percentile of request latency: 
   p50: 14.7115
   p66: 16.2993
   p75: 17.4013
   p80: 18.2002
   p90: 20.0371
   p95: 21.8216
   p98: 24.5539
   p99: 27.3373

parallel 64

Benchmarking summary: 
 Time taken for tests: 382.247 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 2.613
 Average latency: 17.296
 Throughput(average output tokens per second): 973.795
 Average time to first token: 17.296
 Average input tokens per request: 40.303
 Average output tokens per request: 276.968
 Average time per output token: 0.00103
 Average package per request: 1.000
 Average package latency: 17.296
 Percentile of time to first token: 
   p50: 17.5290
   p66: 19.5218
   p75: 20.7063
   p80: 21.6443
   p90: 23.9890
   p95: 26.0998
   p98: 29.7887
   p99: 32.3975
 Percentile of request latency: 
   p50: 17.5290
   p66: 19.5218
   p75: 20.7063
   p80: 21.6443
   p90: 23.9890
   p95: 26.0998
   p98: 29.7887
   p99: 32.3975

parallel 128

Benchmarking summary: 
 Time taken for tests: 179.015 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 5.586
 Average latency: 21.041
 Throughput(average output tokens per second): 1549.069
 Average time to first token: 21.041
 Average input tokens per request: 40.296
 Average output tokens per request: 277.307
 Average time per output token: 0.00065
 Average package per request: 1.000
 Average package latency: 21.041
 Percentile of time to first token: 
   p50: 21.3863
   p66: 23.7687
   p75: 25.1877
   p80: 26.3636
   p90: 29.5537
   p95: 32.3925
   p98: 36.5261
   p99: 39.3924
 Percentile of request latency: 
   p50: 21.3863
   p66: 23.7687
   p75: 25.1877
   p80: 26.3636
   p90: 29.5537
   p95: 32.3925
   p98: 36.5261
   p99: 39.3924

parallel 150

Benchmarking summary: 
 Time taken for tests: 270.054 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 3.699
 Average latency: 23.856
 Throughput(average output tokens per second): 1595.176
 Average time to first token: 23.856
 Average input tokens per request: 40.303
 Average output tokens per request: 277.760
 Average time per output token: 0.00063
 Average package per request: 1.000
 Average package latency: 23.856
 Percentile of time to first token: 
   p50: 23.8551
   p66: 26.6484
   p75: 28.7350
   p80: 29.8586
   p90: 33.7459
   p95: 36.6390
   p98: 41.2772
   p99: 47.8515
 Percentile of request latency: 
   p50: 23.8551
   p66: 26.6484
   p75: 28.7350
   p80: 29.8586
   p90: 33.7459
   p95: 36.6390
   p98: 41.2772
   p99: 47.8515

parallel 200

Benchmarking summary: 
 Time taken for tests: 251.060 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 1000
 Succeed requests: 998
 Failed requests: 2
 Average QPS: 3.975
 Average latency: 28.198
 Throughput(average output tokens per second): 1748.784
 Average time to first token: 28.198
 Average input tokens per request: 40.308
 Average output tokens per request: 276.789
 Average time per output token: 0.00057
 Average package per request: 1.000
 Average package latency: 28.198
 Percentile of time to first token: 
   p50: 28.4407
   p66: 31.6658
   p75: 34.0785
   p80: 35.5489
   p90: 40.4210
   p95: 43.0363
   p98: 48.1876
   p99: 52.8204
 Percentile of request latency: 
   p50: 28.4407
   p66: 31.6658
   p75: 34.0785
   p80: 35.5489
   p90: 40.4210
   p95: 43.0363
   p98: 48.1876
   p99: 52.8204

parallel 256

Benchmarking summary: 
 Time taken for tests: 237.063 seconds
 Expected number of requests: 1000
 Number of concurrency: 256
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 4.214
 Average latency: 33.176
 Throughput(average output tokens per second): 1866.155
 Average time to first token: 33.176
 Average input tokens per request: 40.303
 Average output tokens per request: 276.399
 Average time per output token: 0.00054
 Average package per request: 1.000
 Average package latency: 33.176
 Percentile of time to first token: 
   p50: 32.7329
   p66: 37.0212
   p75: 39.6246
   p80: 41.3947
   p90: 45.7666
   p95: 49.4765
   p98: 54.4858
   p99: 58.0206
 Percentile of request latency: 
   p50: 32.7329
   p66: 37.0212
   p75: 39.6246
   p80: 41.3947
   p90: 45.7666
   p95: 49.4765
   p98: 54.4858
   p99: 58.0206

parallel 512

Benchmarking summary: 
 Time taken for tests: 206.734 seconds
 Expected number of requests: 1000
 Number of concurrency: 512
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 4.832
 Average latency: 59.912
 Throughput(average output tokens per second): 1828.363
 Average time to first token: 59.912
 Average input tokens per request: 40.303
 Average output tokens per request: 277.592
 Average time per output token: 0.00055
 Average package per request: 1.000
 Average package latency: 59.912
 Percentile of time to first token: 
   p50: 63.0404
   p66: 69.5273
   p75: 73.3044
   p80: 75.7455
   p90: 81.6602
   p95: 87.3050
   p98: 92.2078
   p99: 97.6189
 Percentile of request latency: 
   p50: 63.0404
   p66: 69.5273
   p75: 73.3044
   p80: 75.7455
   p90: 81.6602
   p95: 87.3050
   p98: 92.2078
   p99: 97.6189

Qwen2-72B-Chat (long.jsonl)

指标	8	12	20	30	40	50
用时	3091.468	2385.800	1598.805	1542.828	1509.713	1408.587
QPS	0.032	0.042	0.063	0.063	0.056	0.043
延迟	238.540	268.873	294.636	382.291	414.746	403.061
吞吐量	158.986	206.011	307.418	299.114	268.391	199.552
p50	239.3327	270.9418	292.3093	348.5759	396.9046	350.8392
p90	239.6762	271.3905	313.2425	514.8233	567.8100	597.3057

平均每个请求的输入 token 数: 6385
平均每个请求的输出 token 数: 4915

parallel 8

Benchmarking summary: 
 Time taken for tests: 3091.468 seconds
 Expected number of requests: 100
 Number of concurrency: 8
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.032
 Average latency: 238.540
 Throughput(average output tokens per second): 158.986
 Average time to first token: 238.540
 Average input tokens per request: 6385.000
 Average output tokens per request: 4915.000
 Average time per output token: 0.00629
 Average package per request: 1.000
 Average package latency: 238.540
 Percentile of time to first token: 
   p50: 239.3327
   p66: 239.5209
   p75: 239.5622
   p80: 239.6446
   p90: 239.6762
   p95: 240.0079
   p98: 240.0081
   p99: 240.0121
 Percentile of request latency: 
   p50: 239.3327
   p66: 239.5209
   p75: 239.5622
   p80: 239.6446
   p90: 239.6762
   p95: 240.0079
   p98: 240.0081
   p99: 240.0121

parallel 12

Benchmarking summary: 
 Time taken for tests: 2385.800 seconds
 Expected number of requests: 100
 Number of concurrency: 12
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.042
 Average latency: 268.873
 Throughput(average output tokens per second): 206.011
 Average time to first token: 268.873
 Average input tokens per request: 6385.000
 Average output tokens per request: 4915.000
 Average time per output token: 0.00485
 Average package per request: 1.000
 Average package latency: 268.873
 Percentile of time to first token: 
   p50: 270.9418
   p66: 271.2864
   p75: 271.3597
   p80: 271.3610
   p90: 271.3905
   p95: 271.3978
   p98: 271.4502
   p99: 271.4786
 Percentile of request latency: 
   p50: 270.9418
   p66: 271.2864
   p75: 271.3597
   p80: 271.3610
   p90: 271.3905
   p95: 271.3978
   p98: 271.4502
   p99: 271.4786

parallel 20

Benchmarking summary: 
 Time taken for tests: 1598.805 seconds
 Expected number of requests: 100
 Number of concurrency: 20
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.063
 Average latency: 294.636
 Throughput(average output tokens per second): 307.418
 Average time to first token: 294.636
 Average input tokens per request: 6385.000
 Average output tokens per request: 4915.020
 Average time per output token: 0.00325
 Average package per request: 1.000
 Average package latency: 294.636
 Percentile of time to first token: 
   p50: 292.3093
   p66: 293.7218
   p75: 296.3762
   p80: 296.4460
   p90: 313.2425
   p95: 323.9384
   p98: 334.5282
   p99: 357.6738
 Percentile of request latency: 
   p50: 292.3093
   p66: 293.7218
   p75: 296.3762
   p80: 296.4460
   p90: 313.2425
   p95: 323.9384
   p98: 334.5282
   p99: 357.6738

parallel 30

Benchmarking summary: 
 Time taken for tests: 1542.828 seconds
 Expected number of requests: 100
 Number of concurrency: 30
 Total requests: 97
 Succeed requests: 97
 Failed requests: 0
 Average QPS: 0.063
 Average latency: 382.291
 Throughput(average output tokens per second): 299.114
 Average time to first token: 382.291
 Average input tokens per request: 6385.000
 Average output tokens per request: 4757.546
 Average time per output token: 0.00334
 Average package per request: 1.000
 Average package latency: 382.291
 Percentile of time to first token: 
   p50: 348.5759
   p66: 378.9814
   p75: 420.5971
   p80: 443.5496
   p90: 514.8233
   p95: 548.8156
   p98: 559.4441
   p99: 590.6074
 Percentile of request latency: 
   p50: 348.5759
   p66: 378.9814
   p75: 420.5971
   p80: 443.5496
   p90: 514.8233
   p95: 548.8156
   p98: 559.4441
   p99: 590.6074

parallel 40

Benchmarking summary: 
 Time taken for tests: 1509.713 seconds
 Expected number of requests: 100
 Number of concurrency: 40
 Total requests: 87
 Succeed requests: 85
 Failed requests: 2
 Average QPS: 0.056
 Average latency: 414.746
 Throughput(average output tokens per second): 268.391
 Average time to first token: 414.746
 Average input tokens per request: 6385.000
 Average output tokens per request: 4766.976
 Average time per output token: 0.00373
 Average package per request: 1.000
 Average package latency: 414.746
 Percentile of time to first token: 
   p50: 396.9046
   p66: 458.9677
   p75: 482.9745
   p80: 521.3878
   p90: 567.8100
   p95: 580.6507
   p98: 586.6928
   p99: 587.2930
 Percentile of request latency: 
   p50: 396.9046
   p66: 458.9677
   p75: 482.9745
   p80: 521.3878
   p90: 567.8100
   p95: 580.6507
   p98: 586.6928
   p99: 587.2930

parallel 50

Benchmarking summary: 
 Time taken for tests: 1408.587 seconds
 Expected number of requests: 100
 Number of concurrency: 50
 Total requests: 78
 Succeed requests: 60
 Failed requests: 18
 Average QPS: 0.043
 Average latency: 403.061
 Throughput(average output tokens per second): 199.552
 Average time to first token: 403.061
 Average input tokens per request: 6385.000
 Average output tokens per request: 4684.783
 Average time per output token: 0.00501
 Average package per request: 1.000
 Average package latency: 403.061
 Percentile of time to first token: 
   p50: 350.8392
   p66: 386.2204
   p75: 516.6384
   p80: 558.8747
   p90: 597.3057
   p95: 597.3098
   p98: 597.5291
   p99: 598.8612
 Percentile of request latency: 
   p50: 350.8392
   p66: 386.2204
   p75: 516.6384
   p80: 558.8747
   p90: 597.3057
   p95: 597.3098
   p98: 597.5291
   p99: 598.8612

DeepSeek-Coder-6.7B-Instruct

指标	8	16	32	64	128	150	200	300	400	500	600
用时	621.642	325.248	178.007	109.977	70.124	67.204	61.252	63.928	70.753	72.668	75.559
QPS	1.609	3.075	5.618	9.093	14.261	14.880	16.326	15.643	12.494	7.527	5.294
延迟	4.967	5.153	5.590	6.847	8.644	9.535	12.010	17.516	20.797	18.579	21.424
吞吐量	643.457	1229.830	2247.103	3637.131	5704.202	5952.054	6530.446	6257.033	4997.658	3010.939	2117.557
p50	4.9568	5.1310	5.5455	6.9095	8.6159	9.7244	12.1211	14.5084	20.3585	17.9382	22.6927
p90	5.0456	5.3116	6.0241	6.9913	8.9456	10.0026	12.3709	23.8041	28.1569	28.4858	24.4160
失败	0	0	0	0	0	0	0	0	0	184	216

平均每个请求的输入 token 数: 157
平均每个请求的输出 token 数: 400

parallel 8

Benchmarking summary: 
 Time taken for tests: 621.642 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 1.609
 Average latency: 4.967
 Throughput(average output tokens per second): 643.457
 Average time to first token: 4.967
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00155
 Average package per request: 1.000
 Average package latency: 4.967
 Percentile of time to first token: 
   p50: 4.9568
   p66: 4.9916
   p75: 5.0074
   p80: 5.0173
   p90: 5.0456
   p95: 5.0796
   p98: 5.1067
   p99: 5.1223
 Percentile of request latency: 
   p50: 4.9568
   p66: 4.9916
   p75: 5.0074
   p80: 5.0173
   p90: 5.0456
   p95: 5.0796
   p98: 5.1067
   p99: 5.1223

parallel 16

Benchmarking summary: 
 Time taken for tests: 325.248 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 3.075
 Average latency: 5.153
 Throughput(average output tokens per second): 1229.830
 Average time to first token: 5.153
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00081
 Average package per request: 1.000
 Average package latency: 5.153
 Percentile of time to first token: 
   p50: 5.1310
   p66: 5.1707
   p75: 5.2303
   p80: 5.2545
   p90: 5.3116
   p95: 5.4090
   p98: 5.4987
   p99: 5.5322
 Percentile of request latency: 
   p50: 5.1310
   p66: 5.1707
   p75: 5.2303
   p80: 5.2545
   p90: 5.3116
   p95: 5.4090
   p98: 5.4987
   p99: 5.5322

parallel 32

Benchmarking summary: 
 Time taken for tests: 178.007 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 5.618
 Average latency: 5.590
 Throughput(average output tokens per second): 2247.103
 Average time to first token: 5.590
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00045
 Average package per request: 1.000
 Average package latency: 5.590
 Percentile of time to first token: 
   p50: 5.5455
   p66: 5.5729
   p75: 5.6628
   p80: 5.7434
   p90: 6.0241
   p95: 6.0393
   p98: 6.1004
   p99: 6.1034
 Percentile of request latency: 
   p50: 5.5455
   p66: 5.5729
   p75: 5.6628
   p80: 5.7434
   p90: 6.0241
   p95: 6.0393
   p98: 6.1004
   p99: 6.1034

parallel 64

Benchmarking summary: 
 Time taken for tests: 109.977 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 9.093
 Average latency: 6.847
 Throughput(average output tokens per second): 3637.131
 Average time to first token: 6.847
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00027
 Average package per request: 1.000
 Average package latency: 6.847
 Percentile of time to first token: 
   p50: 6.9095
   p66: 6.9292
   p75: 6.9419
   p80: 6.9551
   p90: 6.9913
   p95: 7.0102
   p98: 7.0201
   p99: 7.0224
 Percentile of request latency: 
   p50: 6.9095
   p66: 6.9292
   p75: 6.9419
   p80: 6.9551
   p90: 6.9913
   p95: 7.0102
   p98: 7.0201
   p99: 7.0224

parallel 128

Benchmarking summary: 
 Time taken for tests: 70.124 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 14.261
 Average latency: 8.644
 Throughput(average output tokens per second): 5704.202
 Average time to first token: 8.644
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00018
 Average package per request: 1.000
 Average package latency: 8.644
 Percentile of time to first token: 
   p50: 8.6159
   p66: 8.7015
   p75: 8.7256
   p80: 8.8329
   p90: 8.9456
   p95: 8.9532
   p98: 8.9652
   p99: 8.9727
 Percentile of request latency: 
   p50: 8.6159
   p66: 8.7015
   p75: 8.7256
   p80: 8.8329
   p90: 8.9456
   p95: 8.9532
   p98: 8.9652
   p99: 8.9727

parallel 150

Benchmarking summary: 
 Time taken for tests: 67.204 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 14.880
 Average latency: 9.535
 Throughput(average output tokens per second): 5952.054
 Average time to first token: 9.535
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00017
 Average package per request: 1.000
 Average package latency: 9.535
 Percentile of time to first token: 
   p50: 9.7244
   p66: 9.8244
   p75: 9.8670
   p80: 9.8891
   p90: 10.0026
   p95: 10.0508
   p98: 10.0876
   p99: 10.1092
 Percentile of request latency: 
   p50: 9.7244
   p66: 9.8244
   p75: 9.8670
   p80: 9.8891
   p90: 10.0026
   p95: 10.0508
   p98: 10.0876
   p99: 10.1092

parallel 200

Benchmarking summary: 
 Time taken for tests: 61.252 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 16.326
 Average latency: 12.010
 Throughput(average output tokens per second): 6530.446
 Average time to first token: 12.010
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00015
 Average package per request: 1.000
 Average package latency: 12.010
 Percentile of time to first token: 
   p50: 12.1211
   p66: 12.2211
   p75: 12.2520
   p80: 12.2641
   p90: 12.3709
   p95: 12.3958
   p98: 12.4472
   p99: 12.4868
 Percentile of request latency: 
   p50: 12.1211
   p66: 12.2211
   p75: 12.2520
   p80: 12.2641
   p90: 12.3709
   p95: 12.3958
   p98: 12.4472
   p99: 12.4868

parallel 300

Benchmarking summary: 
 Time taken for tests: 63.928 seconds
 Expected number of requests: 1000
 Number of concurrency: 300
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 15.643
 Average latency: 17.516
 Throughput(average output tokens per second): 6257.033
 Average time to first token: 17.516
 Average input tokens per request: 157.292
 Average output tokens per request: 400.000
 Average time per output token: 0.00016
 Average package per request: 1.000
 Average package latency: 17.516
 Percentile of time to first token: 
   p50: 14.5084
   p66: 21.8597
   p75: 22.9786
   p80: 23.3034
   p90: 23.8041
   p95: 25.4313
   p98: 25.8759
   p99: 26.0190
 Percentile of request latency: 
   p50: 14.5084
   p66: 21.8597
   p75: 22.9786
   p80: 23.3034
   p90: 23.8041
   p95: 25.4313
   p98: 25.8759
   p99: 26.0190

parallel 400

Benchmarking summary: 
 Time taken for tests: 70.753 seconds
 Expected number of requests: 1000
 Number of concurrency: 400
 Total requests: 884
 Succeed requests: 884
 Failed requests: 0
 Average QPS: 12.494
 Average latency: 20.797
 Throughput(average output tokens per second): 4997.658
 Average time to first token: 20.797
 Average input tokens per request: 157.958
 Average output tokens per request: 400.000
 Average time per output token: 0.00020
 Average package per request: 1.000
 Average package latency: 20.797
 Percentile of time to first token: 
   p50: 20.3585
   p66: 25.7757
   p75: 26.3887
   p80: 27.0304
   p90: 28.1569
   p95: 28.6731
   p98: 29.6462
   p99: 29.8135
 Percentile of request latency: 
   p50: 20.3585
   p66: 25.7757
   p75: 26.3887
   p80: 27.0304
   p90: 28.1569
   p95: 28.6731
   p98: 29.6462
   p99: 29.8135

parallel 500

Benchmarking summary: 
 Time taken for tests: 72.668 seconds
 Expected number of requests: 1000
 Number of concurrency: 500
 Total requests: 731
 Succeed requests: 547
 Failed requests: 184
 Average QPS: 7.527
 Average latency: 18.579
 Throughput(average output tokens per second): 3010.939
 Average time to first token: 18.579
 Average input tokens per request: 156.399
 Average output tokens per request: 400.000
 Average time per output token: 0.00033
 Average package per request: 1.000
 Average package latency: 18.579
 Percentile of time to first token: 
   p50: 17.9382
   p66: 19.5846
   p75: 20.2549
   p80: 20.4220
   p90: 28.4858
   p95: 29.5889
   p98: 29.9512
   p99: 30.0487
 Percentile of request latency: 
   p50: 17.9382
   p66: 19.5846
   p75: 20.2549
   p80: 20.4220
   p90: 28.4858
   p95: 29.5889
   p98: 29.9512
   p99: 30.0487

parallel 600

Benchmarking summary: 
 Time taken for tests: 75.559 seconds
 Expected number of requests: 1000
 Number of concurrency: 600
 Total requests: 616
 Succeed requests: 400
 Failed requests: 216
 Average QPS: 5.294
 Average latency: 21.424
 Throughput(average output tokens per second): 2117.557
 Average time to first token: 21.424
 Average input tokens per request: 157.625
 Average output tokens per request: 400.000
 Average time per output token: 0.00047
 Average package per request: 1.000
 Average package latency: 21.424
 Percentile of time to first token: 
   p50: 22.6927
   p66: 23.5150
   p75: 24.1050
   p80: 24.2716
   p90: 24.4160
   p95: 24.5849
   p98: 29.1271
   p99: 30.2161
 Percentile of request latency: 
   p50: 22.6927
   p66: 23.5150
   p75: 24.1050
   p80: 24.2716
   p90: 24.4160
   p95: 24.5849
   p98: 29.1271
   p99: 30.2161

安装依赖库

pip install evalscope-perf
pip install evalscope

执行命令

evalscope-perf http://127.0.0.1:1025/v1/chat/completions qwen \
    ./datasets/Codefuse-Evol-Instruct-Clean-data.jsonl \
    --parallels 32 \
    --parallels 64 \
    --parallels 100 \
    --parallels 128 \
    --parallels 150 \
    --parallels 200 \
    --parallels 256 \
    --parallels 300 \
    --parallels 400 \
    --parallels 500 \
    --parallels 600 \
    --parallels 700 \
    --parallels 800 \
    --parallels 900 \
    --parallels 1000 \
    --n 2000

绘图代码

import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties

# 设置中文字体
font_path =  '/System/Library/Fonts/Hiragino Sans GB.ttc' # 替换为你的字体文件路径
font_prop = FontProperties(fname=font_path)

# 数据: Qwen1.5-7B-Chat
concurrency = [8, 16, 32, 64, 128, 150, 200, 256, 300, 400, 512, 720]
time = [404.284, 215.085, 120.876, 82.884, 52.844, 48.884, 44.856, 42.750, 43.729, 42.866, 41.655, 40.580]
qps = [2.474, 4.649, 8.273, 12.065, 18.924, 20.457, 22.294, 23.392, 22.868, 23.328, 24.007, 24.643]
latency = [3.213, 3.391, 3.724, 4.974, 6.108, 6.517, 7.805, 9.208, 10.904, 13.628, 16.031, 18.790]
throughput = [594.929, 1119.102, 1989.159, 2903.023, 4559.489, 4920.856, 5354.701, 5636.846, 5506.567, 5618.110, 5772.230, 5940.622]
p50 = [3.2461, 3.4271, 3.7514, 5.0248, 6.1491, 6.5487, 7.7782, 9.1754, 10.9164, 13.9850, 17.0675, 20.4161]
p90 = [4.9771, 5.2905, 5.8484, 7.7522, 9.5705, 10.1980, 12.3493, 13.6320, 15.8667, 19.2667, 22.7640, 26.8183]

# 数据: Qwen1.5-14B-Chat
# concurrency = [8, 16, 32, 64, 128, 150, 200, 256, 512]
# time = [578.571, 361.169, 253.040, 204.961, 170.001, 169.981, 162.999, 159.840, 153.937]
# qps = [1.727, 2.766, 3.952, 4.874, 5.882, 5.877, 6.129, 6.250, 6.490]
# latency = [3.712, 3.928, 4.581, 5.628, 7.223, 8.004, 9.205, 11.446, 22.695]
# throughput = [480.043, 897.511, 915.133, 2333.955, 1363.656, 3621.096, 4310.525, 4333.013, 3806.748]
# p50 = [3.7038, 3.9261, 4.4544, 5.6047, 7.0534, 7.9484, 9.0271, 11.3066, 23.6591]
# p90 = [5.7184, 6.0562, 6.9198, 8.7597, 11.1194, 12.5181, 14.5017, 16.8646, 33.3052]

# 数据: Qwen2-72B-Chat
# concurrency = [8, 16, 32, 64, 128, 150, 200, 256, 512]
# time = [1569.707, 909.001, 567.479, 382.247, 179.015, 270.054, 251.060, 237.063, 206.734]
# qps = [0.636, 1.099, 1.759, 2.613, 5.586, 3.699, 3.975, 4.214, 4.832]
# latency = [11.705, 12.806, 14.526, 17.296, 21.041, 23.856, 28.198, 33.176, 59.912]
# throughput = [188.589, 342.764, 588.262, 973.795, 1549.069, 1595.176, 1748.784, 1866.155, 1828.363]
# p50 = [11.8443, 12.8275, 14.7115, 17.5290, 21.3863, 23.8551, 28.4407, 32.7329, 63.0404]
# p90 = [16.2106, 17.5466, 20.0371, 23.9890, 29.5537, 33.7459, 40.4210, 45.7666, 81.6602]


# 绘制曲线
plt.figure(figsize=(12, 8))

# 用时 vs 并行数
plt.subplot(2, 3, 1)
plt.plot(concurrency, time, marker='o')
plt.title('用时 vs 并行数', fontproperties=font_prop)
plt.xlabel('并行数', fontproperties=font_prop)
plt.ylabel('用时 (秒)', fontproperties=font_prop)

# QPS vs 并行数
plt.subplot(2, 3, 2)
plt.plot(concurrency, qps, marker='o')
plt.title('QPS vs 并行数', fontproperties=font_prop)
plt.xlabel('并行数', fontproperties=font_prop)
plt.ylabel('QPS', fontproperties=font_prop)

# 延迟 vs 并行数
plt.subplot(2, 3, 3)
plt.plot(concurrency, latency, marker='o')
plt.title('延迟 vs 并行数', fontproperties=font_prop)
plt.xlabel('并行数', fontproperties=font_prop)
plt.ylabel('延迟 (秒)', fontproperties=font_prop)

# 吞吐量 vs 并行数
plt.subplot(2, 3, 4)
plt.plot(concurrency, throughput, marker='o')
plt.title('吞吐量 vs 并行数', fontproperties=font_prop)
plt.xlabel('并行数', fontproperties=font_prop)
plt.ylabel('吞吐量 (每秒输出的token数)', fontproperties=font_prop)

# p50 vs 并行数
plt.subplot(2, 3, 5)
plt.plot(concurrency, p50, marker='o')
plt.title('p50 vs 并行数', fontproperties=font_prop)
plt.xlabel('并行数', fontproperties=font_prop)
plt.ylabel('p50 (秒)', fontproperties=font_prop)

# p90 vs 并行数
plt.subplot(2, 3, 6)
plt.plot(concurrency, p90, marker='o')
plt.title('p90 vs 并行数', fontproperties=font_prop)
plt.xlabel('并行数', fontproperties=font_prop)
plt.ylabel('p90 (秒)', fontproperties=font_prop)

# 显示图表
plt.tight_layout()
plt.show()

实验结果（vLLM）

Qwen1.5-7B-Chat

指标	8	16	32	64	100	128	150	200
用时	2555.302	1355.736	800.953	515.309	403.138	375.187	386.202	355.307
QPS	0.391	0.738	1.249	1.941	2.481	2.660	2.569	2.730
延迟	20.326	21.475	24.877	31.015	37.778	43.181	52.514	61.304
吞吐量	94.803	177.014	300.603	469.235	595.103	638.980	612.597	640.172
p50	20.5326	21.6749	24.9076	31.2051	37.8700	42.3652	52.1732	61.0796
p90	31.8381	33.7150	38.8008	48.5248	59.2335	68.8249	84.6935	96.6051
❌							6	28

平均每个请求的输入 token 数: 40
平均每个请求的输出 token 数: 240

# 数据
concurrency = [8, 16, 32, 64, 100, 128, 150, 200]
time = [2555.302, 1355.736, 800.953, 515.309, 403.138, 375.187, 386.202, 355.307]
qps = [0.391, 0.738, 1.249, 1.941, 2.481, 2.660, 2.569, 2.730]
latency = [20.326, 21.475, 24.877, 31.015, 37.778, 43.181, 52.514, 61.304]
throughput = [94.803, 177.014, 300.603, 469.235, 595.103, 638.980, 612.597, 640.172]
p50 = [20.5326, 21.6749, 24.9076, 31.2051, 37.8700, 42.3652, 52.1732, 61.0796]
p90 = [31.8381, 33.7150, 38.8008, 48.5248, 59.2335, 68.8249, 84.6935, 96.6051]

parallel 8

Benchmarking summary: 
 Time taken for tests: 2555.302 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 0.391
 Average latency: 20.326
 Throughput(average output tokens per second): 94.803
 Average time to first token: 20.326
 Average input tokens per request: 40.296
 Average output tokens per request: 242.251
 Average time per output token: 0.01055
 Average package per request: 1.000
 Average package latency: 20.326
 Percentile of time to first token: 
   p50: 20.5326
   p66: 23.7691
   p75: 26.5282
   p80: 28.1502
   p90: 31.8381
   p95: 35.6152
   p98: 40.9497
   p99: 45.7076
 Percentile of request latency: 
   p50: 20.5326
   p66: 23.7691
   p75: 26.5282
   p80: 28.1502
   p90: 31.8381
   p95: 35.6152
   p98: 40.9497
   p99: 45.7076

parallel 16

Benchmarking summary: 
 Time taken for tests: 1355.736 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 0.738
 Average latency: 21.475
 Throughput(average output tokens per second): 177.014
 Average time to first token: 21.475
 Average input tokens per request: 40.296
 Average output tokens per request: 239.984
 Average time per output token: 0.00565
 Average package per request: 1.000
 Average package latency: 21.475
 Percentile of time to first token: 
   p50: 21.6749
   p66: 25.2886
   p75: 27.4429
   p80: 29.0391
   p90: 33.7150
   p95: 37.1781
   p98: 42.4568
   p99: 45.9629
 Percentile of request latency: 
   p50: 21.6749
   p66: 25.2886
   p75: 27.4429
   p80: 29.0391
   p90: 33.7150
   p95: 37.1781
   p98: 42.4568
   p99: 45.9629

parallel 32

Benchmarking summary: 
 Time taken for tests: 800.953 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 1.249
 Average latency: 24.877
 Throughput(average output tokens per second): 300.603
 Average time to first token: 24.877
 Average input tokens per request: 40.296
 Average output tokens per request: 240.769
 Average time per output token: 0.00333
 Average package per request: 1.000
 Average package latency: 24.877
 Percentile of time to first token: 
   p50: 24.9076
   p66: 29.1147
   p75: 32.1535
   p80: 34.1477
   p90: 38.8008
   p95: 43.0980
   p98: 48.4589
   p99: 53.6278
 Percentile of request latency: 
   p50: 24.9076
   p66: 29.1147
   p75: 32.1535
   p80: 34.1477
   p90: 38.8008
   p95: 43.0980
   p98: 48.4589
   p99: 53.6278

parallel 64

Benchmarking summary: 
 Time taken for tests: 515.309 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 1.941
 Average latency: 31.015
 Throughput(average output tokens per second): 469.235
 Average time to first token: 31.015
 Average input tokens per request: 40.296
 Average output tokens per request: 241.801
 Average time per output token: 0.00213
 Average package per request: 1.000
 Average package latency: 31.015
 Percentile of time to first token: 
   p50: 31.2051
   p66: 36.5305
   p75: 40.1962
   p80: 42.1270
   p90: 48.5248
   p95: 53.4686
   p98: 60.3826
   p99: 65.7931
 Percentile of request latency: 
   p50: 31.2051
   p66: 36.5305
   p75: 40.1962
   p80: 42.1270
   p90: 48.5248
   p95: 53.4686
   p98: 60.3826
   p99: 65.7931

parallel 100

Benchmarking summary: 
 Time taken for tests: 403.138 seconds
 Expected number of requests: 1000
 Number of concurrency: 100
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 2.481
 Average latency: 37.778
 Throughput(average output tokens per second): 595.103
 Average time to first token: 37.778
 Average input tokens per request: 40.296
 Average output tokens per request: 239.909
 Average time per output token: 0.00168
 Average package per request: 1.000
 Average package latency: 37.778
 Percentile of time to first token: 
   p50: 37.8700
   p66: 44.4992
   p75: 49.1464
   p80: 52.3330
   p90: 59.2335
   p95: 64.9315
   p98: 74.1723
   p99: 80.4544
 Percentile of request latency: 
   p50: 37.8700
   p66: 44.4992
   p75: 49.1464
   p80: 52.3330
   p90: 59.2335
   p95: 64.9315
   p98: 74.1723
   p99: 80.4544

parallel 128

Benchmarking summary: 
 Time taken for tests: 375.187 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 999
 Succeed requests: 998
 Failed requests: 1
 Average QPS: 2.660
 Average latency: 43.181
 Throughput(average output tokens per second): 638.980
 Average time to first token: 43.181
 Average input tokens per request: 40.297
 Average output tokens per request: 240.217
 Average time per output token: 0.00156
 Average package per request: 1.000
 Average package latency: 43.181
 Percentile of time to first token: 
   p50: 42.3652
   p66: 51.6154
   p75: 56.2693
   p80: 59.2260
   p90: 68.8249
   p95: 75.4546
   p98: 85.2362
   p99: 93.3652
 Percentile of request latency: 
   p50: 42.3652
   p66: 51.6154
   p75: 56.2693
   p80: 59.2260
   p90: 68.8249
   p95: 75.4546
   p98: 85.2362
   p99: 93.3652

parallel 150

Benchmarking summary: 
 Time taken for tests: 386.202 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 998
 Succeed requests: 992
 Failed requests: 6
 Average QPS: 2.569
 Average latency: 52.514
 Throughput(average output tokens per second): 612.597
 Average time to first token: 52.514
 Average input tokens per request: 40.303
 Average output tokens per request: 238.494
 Average time per output token: 0.00163
 Average package per request: 1.000
 Average package latency: 52.514
 Percentile of time to first token: 
   p50: 52.1732
   p66: 61.4450
   p75: 67.9417
   p80: 72.0179
   p90: 84.6935
   p95: 92.2286
   p98: 101.7002
   p99: 107.7996
 Percentile of request latency: 
   p50: 52.1732
   p66: 61.4450
   p75: 67.9417
   p80: 72.0179
   p90: 84.6935
   p95: 92.2286
   p98: 101.7002
   p99: 107.7996

parallel 200

Benchmarking summary: 
 Time taken for tests: 355.307 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 998
 Succeed requests: 970
 Failed requests: 28
 Average QPS: 2.730
 Average latency: 61.304
 Throughput(average output tokens per second): 640.172
 Average time to first token: 61.304
 Average input tokens per request: 40.287
 Average output tokens per request: 234.493
 Average time per output token: 0.00156
 Average package per request: 1.000
 Average package latency: 61.304
 Percentile of time to first token: 
   p50: 61.0796
   p66: 74.1724
   p75: 80.9575
   p80: 84.8420
   p90: 96.6051
   p95: 105.2873
   p98: 114.3178
   p99: 115.7257
 Percentile of request latency: 
   p50: 61.0796
   p66: 74.1724
   p75: 80.9575
   p80: 84.8420
   p90: 96.6051
   p95: 105.2873
   p98: 114.3178
   p99: 115.7257

Qwen2.5-72B-Chat

指标	16	32	64
用时	406.618	223.379	159.293
QPS	0.236	0.448	0.609
延迟	53.359	54.293	60.959
吞吐量	69.242	129.193	185.140
p50	52.8675	56.3690	58.7318
p90	80.6415	87.1464	90.6065

平均每个请求的输入 token 数: 50
平均每个请求的输出 token 数: 290

# 数据
concurrency = [16, 32, 64]
time = [406.618, 223.379, 159.293]
qps = [0.236, 0.448, 0.609]
latency = [53.359, 54.293, 60.959]
throughput = [69.242, 129.193, 185.140]
p50 = [52.8675, 56.3690, 58.7318]
p90 = [80.6415, 87.1464, 90.6065]

parallel 1

Benchmarking summary: 
 Time taken for tests: 80.089 seconds
 Expected number of requests: 1
 Number of concurrency: 1
 Total requests: 1
 Succeed requests: 1
 Failed requests: 0
 Average QPS: 0.012
 Average latency: 79.353
 Throughput(average output tokens per second): 5.819
 Average time to first token: 79.353
 Average input tokens per request: 44.000
 Average output tokens per request: 466.000
 Average time per output token: 0.17186
 Average package per request: 1.000
 Average package latency: 79.353

parallel 16

Benchmarking summary: 
 Time taken for tests: 406.618 seconds
 Expected number of requests: 100
 Number of concurrency: 16
 Total requests: 98
 Succeed requests: 96
 Failed requests: 2
 Average QPS: 0.236
 Average latency: 53.359
 Throughput(average output tokens per second): 69.242
 Average time to first token: 53.359
 Average input tokens per request: 50.156
 Average output tokens per request: 293.281
 Average time per output token: 0.01444
 Average package per request: 1.000
 Average package latency: 53.359
 Percentile of time to first token: 
   p50: 52.8675
   p66: 62.7250
   p75: 70.2725
   p80: 73.0569
   p90: 80.6415
   p95: 91.7408
   p98: 96.5681
   p99: 114.6619
 Percentile of request latency: 
   p50: 52.8675
   p66: 62.7250
   p75: 70.2725
   p80: 73.0569
   p90: 80.6415
   p95: 91.7408
   p98: 96.5681
   p99: 114.6619

parallel 32

Benchmarking summary: 
 Time taken for tests: 223.379 seconds
 Expected number of requests: 100
 Number of concurrency: 32
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.448
 Average latency: 54.293
 Throughput(average output tokens per second): 129.193
 Average time to first token: 54.293
 Average input tokens per request: 49.890
 Average output tokens per request: 288.590
 Average time per output token: 0.00774
 Average package per request: 1.000
 Average package latency: 54.293
 Percentile of time to first token: 
   p50: 56.3690
   p66: 61.8036
   p75: 73.1621
   p80: 77.8210
   p90: 87.1464
   p95: 97.1933
   p98: 102.0623
   p99: 107.4325
 Percentile of request latency: 
   p50: 56.3690
   p66: 61.8036
   p75: 73.1621
   p80: 77.8210
   p90: 87.1464
   p95: 97.1933
   p98: 102.0623
   p99: 107.4325

parallel 64

Benchmarking summary: 
 Time taken for tests: 159.293 seconds
 Expected number of requests: 100
 Number of concurrency: 64
 Total requests: 99
 Succeed requests: 97
 Failed requests: 2
 Average QPS: 0.609
 Average latency: 60.959
 Throughput(average output tokens per second): 185.140
 Average time to first token: 60.959
 Average input tokens per request: 50.093
 Average output tokens per request: 296.392
 Average time per output token: 0.00540
 Average package per request: 1.000
 Average package latency: 60.959
 Percentile of time to first token: 
   p50: 58.7318
   p66: 72.0510
   p75: 77.6354
   p80: 82.2268
   p90: 90.6065
   p95: 98.0913
   p98: 108.4727
   p99: 112.9350
 Percentile of request latency: 
   p50: 58.7318
   p66: 72.0510
   p75: 77.6354
   p80: 82.2268
   p90: 90.6065
   p95: 98.0913
   p98: 108.4727
   p99: 112.9350

实验结果（XInference - MindIE）

❌ 部署时间长了，请求无响应。
❌ 部署多副本，压测一次后，服务挂掉。

实验结果（Nvidia T4: XInference - vLLM）

Qwen2-7B-Instruct

指标	8	16	32	64	100	128	150	200	300	400	500
用时	1276.077	776.170	504.480	372.118	363.016	365.024	418.874	459.912	401.385	458.450	375.796
QPS	0.783	1.288	1.958	2.472	2.353	2.334	2.046	1.750	1.664	1.254	1.163
延迟	10.128	12.307	15.749	19.225	19.235	19.151	17.479	15.949	16.718	14.750	15.771
吞吐量	254.695	424.701	631.260	753.852	700.681	700.324	637.260	550.200	510.211	392.697	362.167
p50	10.3076	12.5569	16.3129	20.0818	20.1519	20.0230	17.9699	16.2526	17.0123	14.8173	15.8847
p90	14.3874	17.6026	22.5020	26.8947	26.5725	26.8284	24.7138	23.0809	24.4670	21.4278	23.7326
失败	1	0	12	71	101	72	32	25	66	47	89

平均每个请求的输入 token 数: 40
平均每个请求的输出 token 数: 300

evalscope-perf http://172.16.33.66:9997/v1/chat/completions gpt-4-32k \
    ./datasets/open_qa.jsonl \
    --parallels 8 \
    --parallels 16 \
    --parallels 32 \
    --parallels 64 \
    --parallels 100 \
    --parallels 128 \
    --parallels 150 \
    --parallels 200 \
    --parallels 300 \
    --parallels 400 \
    --parallels 500 \
    --n 1000

parallel 8

Benchmarking summary: 
 Time taken for tests: 1276.077 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 999
 Failed requests: 1
 Average QPS: 0.783
 Average latency: 10.128
 Throughput(average output tokens per second): 254.695
 Average time to first token: 10.128
 Average input tokens per request: 40.295
 Average output tokens per request: 325.335
 Average time per output token: 0.00393
 Average package per request: 1.000
 Average package latency: 10.128
 Percentile of time to first token: 
   p50: 10.3076
   p66: 11.5736
   p75: 12.3458
   p80: 12.6923
   p90: 14.3874
   p95: 15.7724
   p98: 17.3099
   p99: 18.2167
 Percentile of request latency: 
   p50: 10.3076
   p66: 11.5736
   p75: 12.3458
   p80: 12.6923
   p90: 14.3874
   p95: 15.7724
   p98: 17.3099
   p99: 18.2167

parallel 16

Benchmarking summary: 
 Time taken for tests: 776.170 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 1.288
 Average latency: 12.307
 Throughput(average output tokens per second): 424.701
 Average time to first token: 12.307
 Average input tokens per request: 40.296
 Average output tokens per request: 329.640
 Average time per output token: 0.00235
 Average package per request: 1.000
 Average package latency: 12.307
 Percentile of time to first token: 
   p50: 12.5569
   p66: 13.8731
   p75: 14.9178
   p80: 15.6629
   p90: 17.6026
   p95: 19.0937
   p98: 21.7158
   p99: 22.9113
 Percentile of request latency: 
   p50: 12.5569
   p66: 13.8731
   p75: 14.9178
   p80: 15.6629
   p90: 17.6026
   p95: 19.0937
   p98: 21.7158
   p99: 22.9113

parallel 32

Benchmarking summary: 
 Time taken for tests: 504.480 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 988
 Failed requests: 12
 Average QPS: 1.958
 Average latency: 15.749
 Throughput(average output tokens per second): 631.260
 Average time to first token: 15.749
 Average input tokens per request: 40.277
 Average output tokens per request: 322.326
 Average time per output token: 0.00158
 Average package per request: 1.000
 Average package latency: 15.749
 Percentile of time to first token: 
   p50: 16.3129
   p66: 18.2424
   p75: 19.2717
   p80: 19.9721
   p90: 22.5020
   p95: 24.8820
   p98: 26.8200
   p99: 27.5026
 Percentile of request latency: 
   p50: 16.3129
   p66: 18.2424
   p75: 19.2717
   p80: 19.9721
   p90: 22.5020
   p95: 24.8820
   p98: 26.8200
   p99: 27.5026

parallel 64

Benchmarking summary: 
 Time taken for tests: 372.118 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 991
 Succeed requests: 920
 Failed requests: 71
 Average QPS: 2.472
 Average latency: 19.225
 Throughput(average output tokens per second): 753.852
 Average time to first token: 19.225
 Average input tokens per request: 40.204
 Average output tokens per request: 304.915
 Average time per output token: 0.00133
 Average package per request: 1.000
 Average package latency: 19.225
 Percentile of time to first token: 
   p50: 20.0818
   p66: 22.5724
   p75: 23.7237
   p80: 24.5372
   p90: 26.8947
   p95: 28.2781
   p98: 29.2713
   p99: 29.5592
 Percentile of request latency: 
   p50: 20.0818
   p66: 22.5724
   p75: 23.7237
   p80: 24.5372
   p90: 26.8947
   p95: 28.2781
   p98: 29.2713
   p99: 29.5592

parallel 100

Benchmarking summary: 
 Time taken for tests: 363.016 seconds
 Expected number of requests: 1000
 Number of concurrency: 100
 Total requests: 955
 Succeed requests: 854
 Failed requests: 101
 Average QPS: 2.353
 Average latency: 19.235
 Throughput(average output tokens per second): 700.681
 Average time to first token: 19.235
 Average input tokens per request: 40.218
 Average output tokens per request: 297.843
 Average time per output token: 0.00143
 Average package per request: 1.000
 Average package latency: 19.235
 Percentile of time to first token: 
   p50: 20.1519
   p66: 22.4520
   p75: 23.9718
   p80: 24.6514
   p90: 26.5725
   p95: 28.0242
   p98: 29.1395
   p99: 29.5591
 Percentile of request latency: 
   p50: 20.1519
   p66: 22.4520
   p75: 23.9718
   p80: 24.6514
   p90: 26.5725
   p95: 28.0242
   p98: 29.1395
   p99: 29.5591

parallel 128

Benchmarking summary: 
 Time taken for tests: 365.024 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 924
 Succeed requests: 852
 Failed requests: 72
 Average QPS: 2.334
 Average latency: 19.151
 Throughput(average output tokens per second): 700.324
 Average time to first token: 19.151
 Average input tokens per request: 40.188
 Average output tokens per request: 300.041
 Average time per output token: 0.00143
 Average package per request: 1.000
 Average package latency: 19.151
 Percentile of time to first token: 
   p50: 20.0230
   p66: 22.2805
   p75: 23.6442
   p80: 24.5745
   p90: 26.8284
   p95: 28.1978
   p98: 29.3051
   p99: 29.6282
 Percentile of request latency: 
   p50: 20.0230
   p66: 22.2805
   p75: 23.6442
   p80: 24.5745
   p90: 26.8284
   p95: 28.1978
   p98: 29.3051
   p99: 29.6282

parallel 150

Benchmarking summary: 
 Time taken for tests: 418.874 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 889
 Succeed requests: 857
 Failed requests: 32
 Average QPS: 2.046
 Average latency: 17.479
 Throughput(average output tokens per second): 637.260
 Average time to first token: 17.479
 Average input tokens per request: 40.272
 Average output tokens per request: 311.473
 Average time per output token: 0.00157
 Average package per request: 1.000
 Average package latency: 17.479
 Percentile of time to first token: 
   p50: 17.9699
   p66: 20.0651
   p75: 21.5481
   p80: 22.2761
   p90: 24.7138
   p95: 27.1316
   p98: 28.8103
   p99: 29.3229
 Percentile of request latency: 
   p50: 17.9699
   p66: 20.0651
   p75: 21.5481
   p80: 22.2761
   p90: 24.7138
   p95: 27.1316
   p98: 28.8103
   p99: 29.3229

parallel 200

Benchmarking summary: 
 Time taken for tests: 459.912 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 830
 Succeed requests: 805
 Failed requests: 25
 Average QPS: 1.750
 Average latency: 15.949
 Throughput(average output tokens per second): 550.200
 Average time to first token: 15.949
 Average input tokens per request: 40.337
 Average output tokens per request: 314.340
 Average time per output token: 0.00182
 Average package per request: 1.000
 Average package latency: 15.949
 Percentile of time to first token: 
   p50: 16.2526
   p66: 18.0521
   p75: 19.4394
   p80: 20.1756
   p90: 23.0809
   p95: 25.5287
   p98: 28.5592
   p99: 29.4651
 Percentile of request latency: 
   p50: 16.2526
   p66: 18.0521
   p75: 19.4394
   p80: 20.1756
   p90: 23.0809
   p95: 25.5287
   p98: 28.5592
   p99: 29.4651

parallel 300

Benchmarking summary: 
 Time taken for tests: 401.385 seconds
 Expected number of requests: 1000
 Number of concurrency: 300
 Total requests: 734
 Succeed requests: 668
 Failed requests: 66
 Average QPS: 1.664
 Average latency: 16.718
 Throughput(average output tokens per second): 510.211
 Average time to first token: 16.718
 Average input tokens per request: 40.383
 Average output tokens per request: 306.573
 Average time per output token: 0.00196
 Average package per request: 1.000
 Average package latency: 16.718
 Percentile of time to first token: 
   p50: 17.0123
   p66: 19.2765
   p75: 20.8575
   p80: 21.9712
   p90: 24.4670
   p95: 26.6873
   p98: 28.2457
   p99: 29.1315
 Percentile of request latency: 
   p50: 17.0123
   p66: 19.2765
   p75: 20.8575
   p80: 21.9712
   p90: 24.4670
   p95: 26.6873
   p98: 28.2457
   p99: 29.1315

parallel 400

Benchmarking summary: 
 Time taken for tests: 458.450 seconds
 Expected number of requests: 1000
 Number of concurrency: 400
 Total requests: 622
 Succeed requests: 575
 Failed requests: 47
 Average QPS: 1.254
 Average latency: 14.750
 Throughput(average output tokens per second): 392.697
 Average time to first token: 14.750
 Average input tokens per request: 40.310
 Average output tokens per request: 313.099
 Average time per output token: 0.00255
 Average package per request: 1.000
 Average package latency: 14.750
 Percentile of time to first token: 
   p50: 14.8173
   p66: 16.7375
   p75: 18.0656
   p80: 18.7752
   p90: 21.4278
   p95: 24.3040
   p98: 28.1092
   p99: 29.1603
 Percentile of request latency: 
   p50: 14.8173
   p66: 16.7375
   p75: 18.0656
   p80: 18.7752
   p90: 21.4278
   p95: 24.3040
   p98: 28.1092
   p99: 29.1603

parallel 500

Benchmarking summary: 
 Time taken for tests: 375.796 seconds
 Expected number of requests: 1000
 Number of concurrency: 500
 Total requests: 526
 Succeed requests: 437
 Failed requests: 89
 Average QPS: 1.163
 Average latency: 15.771
 Throughput(average output tokens per second): 362.167
 Average time to first token: 15.771
 Average input tokens per request: 40.268
 Average output tokens per request: 311.444
 Average time per output token: 0.00276
 Average package per request: 1.000
 Average package latency: 15.771
 Percentile of time to first token: 
   p50: 15.8847
   p66: 17.4772
   p75: 18.8302
   p80: 20.0079
   p90: 23.7326
   p95: 26.3385
   p98: 28.6426
   p99: 29.0919
 Percentile of request latency: 
   p50: 15.8847
   p66: 17.4772
   p75: 18.8302
   p80: 20.0079
   p90: 23.7326
   p95: 26.3385
   p98: 28.6426
   p99: 29.0919

实验结果（Nvidia T4: vLLM）

Qwen2-7B-Instruct

指标	8	16	32	64	100	128	150	200	300	400	500
用时	1030.632	629.843	409.251	280.896	252.708	236.597	254.208	274.376	256.799	222.726	175.166
QPS	0.970	1.588	2.443	3.503	3.593	3.580	3.509	3.076	2.847	2.820	3.060
延迟	8.213	9.944	12.846	16.913	19.182	19.831	17.806	16.743	16.285	16.285	17.649
吞吐量	298.860	496.458	753.176	1073.697	1039.610	1005.881	1032.794	925.476	839.437	816.049	875.565
p50	8.4703	10.2975	13.3060	17.5850	20.1962	20.9986	18.6905	17.0203	16.6503	16.4550	18.0830
p90	11.8119	14.2264	18.5003	23.7261	27.0515	27.5461	25.4481	24.0718	23.5927	24.0938	25.7348
失败	0	0	0	15	73	115	26	11	19	23	26

平均每个请求的输入 token 数: 40
平均每个请求的输出 token 数: 300

部署

python -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 --port 8008 \
    --model /data/models/llm/qwen/Qwen2-7B-Instruct/ \
    --served-model-name qwen2-7b \
    --tensor-parallel-size 4 \
    --dtype=float16 \
    --max-model-len 16000

压测

evalscope-perf http://172.16.33.66:8008/v1/chat/completions qwen2-7b \
    ./datasets/open_qa.jsonl \
    --parallels 8 \
    --parallels 16 \
    --parallels 32 \
    --parallels 64 \
    --parallels 100 \
    --parallels 128 \
    --parallels 150 \
    --parallels 200 \
    --parallels 300 \
    --parallels 400 \
    --parallels 500 \
    --n 1000

parallel 8

Benchmarking summary: 
 Time taken for tests: 1030.632 seconds
 Expected number of requests: 1000
 Number of concurrency: 8
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 0.970
 Average latency: 8.213
 Throughput(average output tokens per second): 298.860
 Average time to first token: 8.213
 Average input tokens per request: 40.296
 Average output tokens per request: 308.015
 Average time per output token: 0.00335
 Average package per request: 1.000
 Average package latency: 8.213
 Percentile of time to first token: 
   p50: 8.4703
   p66: 9.3882
   p75: 10.1059
   p80: 10.5404
   p90: 11.8119
   p95: 12.8131
   p98: 14.0085
   p99: 15.0006
 Percentile of request latency: 
   p50: 8.4703
   p66: 9.3882
   p75: 10.1059
   p80: 10.5404
   p90: 11.8119
   p95: 12.8131
   p98: 14.0085
   p99: 15.0006

parallel 16

Benchmarking summary: 
 Time taken for tests: 629.843 seconds
 Expected number of requests: 1000
 Number of concurrency: 16
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 1.588
 Average latency: 9.944
 Throughput(average output tokens per second): 496.458
 Average time to first token: 9.944
 Average input tokens per request: 40.296
 Average output tokens per request: 312.691
 Average time per output token: 0.00201
 Average package per request: 1.000
 Average package latency: 9.944
 Percentile of time to first token: 
   p50: 10.2975
   p66: 11.4685
   p75: 12.2523
   p80: 12.7172
   p90: 14.2264
   p95: 15.5556
   p98: 17.0453
   p99: 18.0745
 Percentile of request latency: 
   p50: 10.2975
   p66: 11.4685
   p75: 12.2523
   p80: 12.7172
   p90: 14.2264
   p95: 15.5556
   p98: 17.0453
   p99: 18.0745

parallel 32

Benchmarking summary: 
 Time taken for tests: 409.251 seconds
 Expected number of requests: 1000
 Number of concurrency: 32
 Total requests: 1000
 Succeed requests: 1000
 Failed requests: 0
 Average QPS: 2.443
 Average latency: 12.846
 Throughput(average output tokens per second): 753.176
 Average time to first token: 12.846
 Average input tokens per request: 40.296
 Average output tokens per request: 308.238
 Average time per output token: 0.00133
 Average package per request: 1.000
 Average package latency: 12.846
 Percentile of time to first token: 
   p50: 13.3060
   p66: 14.7707
   p75: 15.6738
   p80: 16.5227
   p90: 18.5003
   p95: 20.2770
   p98: 22.3723
   p99: 23.2546
 Percentile of request latency: 
   p50: 13.3060
   p66: 14.7707
   p75: 15.6738
   p80: 16.5227
   p90: 18.5003
   p95: 20.2770
   p98: 22.3723
   p99: 23.2546

parallel 64

Benchmarking summary: 
 Time taken for tests: 280.896 seconds
 Expected number of requests: 1000
 Number of concurrency: 64
 Total requests: 999
 Succeed requests: 984
 Failed requests: 15
 Average QPS: 3.503
 Average latency: 16.913
 Throughput(average output tokens per second): 1073.697
 Average time to first token: 16.913
 Average input tokens per request: 40.278
 Average output tokens per request: 306.501
 Average time per output token: 0.00093
 Average package per request: 1.000
 Average package latency: 16.913
 Percentile of time to first token: 
   p50: 17.5850
   p66: 19.7422
   p75: 20.8753
   p80: 21.5526
   p90: 23.7261
   p95: 25.7272
   p98: 28.0227
   p99: 28.5129
 Percentile of request latency: 
   p50: 17.5850
   p66: 19.7422
   p75: 20.8753
   p80: 21.5526
   p90: 23.7261
   p95: 25.7272
   p98: 28.0227
   p99: 28.5129

parallel 100

Benchmarking summary: 
 Time taken for tests: 252.708 seconds
 Expected number of requests: 1000
 Number of concurrency: 100
 Total requests: 981
 Succeed requests: 908
 Failed requests: 73
 Average QPS: 3.593
 Average latency: 19.182
 Throughput(average output tokens per second): 1039.610
 Average time to first token: 19.182
 Average input tokens per request: 40.166
 Average output tokens per request: 289.337
 Average time per output token: 0.00096
 Average package per request: 1.000
 Average package latency: 19.182
 Percentile of time to first token: 
   p50: 20.1962
   p66: 22.6797
   p75: 23.9937
   p80: 24.8991
   p90: 27.0515
   p95: 28.3231
   p98: 29.3312
   p99: 29.6593
 Percentile of request latency: 
   p50: 20.1962
   p66: 22.6797
   p75: 23.9937
   p80: 24.8991
   p90: 27.0515
   p95: 28.3231
   p98: 29.3312
   p99: 29.6593

parallel 128

Benchmarking summary: 
 Time taken for tests: 236.597 seconds
 Expected number of requests: 1000
 Number of concurrency: 128
 Total requests: 962
 Succeed requests: 847
 Failed requests: 115
 Average QPS: 3.580
 Average latency: 19.831
 Throughput(average output tokens per second): 1005.881
 Average time to first token: 19.831
 Average input tokens per request: 40.038
 Average output tokens per request: 280.979
 Average time per output token: 0.00099
 Average package per request: 1.000
 Average package latency: 19.831
 Percentile of time to first token: 
   p50: 20.9986
   p66: 23.7704
   p75: 25.2322
   p80: 25.8841
   p90: 27.5461
   p95: 28.7326
   p98: 29.5878
   p99: 29.8692
 Percentile of request latency: 
   p50: 20.9986
   p66: 23.7704
   p75: 25.2322
   p80: 25.8841
   p90: 27.5461
   p95: 28.7326
   p98: 29.5878
   p99: 29.8692

parallel 150

Benchmarking summary: 
 Time taken for tests: 254.208 seconds
 Expected number of requests: 1000
 Number of concurrency: 150
 Total requests: 918
 Succeed requests: 892
 Failed requests: 26
 Average QPS: 3.509
 Average latency: 17.806
 Throughput(average output tokens per second): 1032.794
 Average time to first token: 17.806
 Average input tokens per request: 40.286
 Average output tokens per request: 294.333
 Average time per output token: 0.00097
 Average package per request: 1.000
 Average package latency: 17.806
 Percentile of time to first token: 
   p50: 18.6905
   p66: 20.7610
   p75: 22.0715
   p80: 23.0934
   p90: 25.4481
   p95: 27.6829
   p98: 29.1659
   p99: 29.6321
 Percentile of request latency: 
   p50: 18.6905
   p66: 20.7610
   p75: 22.0715
   p80: 23.0934
   p90: 25.4481
   p95: 27.6829
   p98: 29.1659
   p99: 29.6321

parallel 200

Benchmarking summary: 
 Time taken for tests: 274.376 seconds
 Expected number of requests: 1000
 Number of concurrency: 200
 Total requests: 855
 Succeed requests: 844
 Failed requests: 11
 Average QPS: 3.076
 Average latency: 16.743
 Throughput(average output tokens per second): 925.476
 Average time to first token: 16.743
 Average input tokens per request: 40.307
 Average output tokens per request: 300.863
 Average time per output token: 0.00108
 Average package per request: 1.000
 Average package latency: 16.743
 Percentile of time to first token: 
   p50: 17.0203
   p66: 19.1086
   p75: 20.4436
   p80: 21.6162
   p90: 24.0718
   p95: 26.2962
   p98: 28.3664
   p99: 29.4007
 Percentile of request latency: 
   p50: 17.0203
   p66: 19.1086
   p75: 20.4436
   p80: 21.6162
   p90: 24.0718
   p95: 26.2962
   p98: 28.3664
   p99: 29.4007

parallel 300

Benchmarking summary: 
 Time taken for tests: 256.799 seconds
 Expected number of requests: 1000
 Number of concurrency: 300
 Total requests: 750
 Succeed requests: 731
 Failed requests: 19
 Average QPS: 2.847
 Average latency: 16.285
 Throughput(average output tokens per second): 839.437
 Average time to first token: 16.285
 Average input tokens per request: 40.453
 Average output tokens per request: 294.893
 Average time per output token: 0.00119
 Average package per request: 1.000
 Average package latency: 16.285
 Percentile of time to first token: 
   p50: 16.6503
   p66: 18.6659
   p75: 19.8392
   p80: 20.8286
   p90: 23.5927
   p95: 26.8649
   p98: 28.0731
   p99: 29.3460
 Percentile of request latency: 
   p50: 16.6503
   p66: 18.6659
   p75: 19.8392
   p80: 20.8286
   p90: 23.5927
   p95: 26.8649
   p98: 28.0731
   p99: 29.3460

parallel 400

Benchmarking summary: 
 Time taken for tests: 222.726 seconds
 Expected number of requests: 1000
 Number of concurrency: 400
 Total requests: 651
 Succeed requests: 628
 Failed requests: 23
 Average QPS: 2.820
 Average latency: 16.285
 Throughput(average output tokens per second): 816.049
 Average time to first token: 16.285
 Average input tokens per request: 40.247
 Average output tokens per request: 289.419
 Average time per output token: 0.00123
 Average package per request: 1.000
 Average package latency: 16.285
 Percentile of time to first token: 
   p50: 16.4550
   p66: 18.4089
   p75: 20.1452
   p80: 20.9587
   p90: 24.0938
   p95: 26.5017
   p98: 28.0750
   p99: 28.6327
 Percentile of request latency: 
   p50: 16.4550
   p66: 18.4089
   p75: 20.1452
   p80: 20.9587
   p90: 24.0938
   p95: 26.5017
   p98: 28.0750
   p99: 28.6327

parallel 500

Benchmarking summary: 
 Time taken for tests: 175.166 seconds
 Expected number of requests: 1000
 Number of concurrency: 500
 Total requests: 562
 Succeed requests: 536
 Failed requests: 26
 Average QPS: 3.060
 Average latency: 17.649
 Throughput(average output tokens per second): 875.565
 Average time to first token: 17.649
 Average input tokens per request: 40.192
 Average output tokens per request: 286.136
 Average time per output token: 0.00114
 Average package per request: 1.000
 Average package latency: 17.649
 Percentile of time to first token: 
   p50: 18.0830
   p66: 20.6764
   p75: 22.3065
   p80: 23.0424
   p90: 25.7348
   p95: 27.9985
   p98: 29.2158
   p99: 29.5974
 Percentile of request latency: 
   p50: 18.0830
   p66: 20.6764
   p75: 22.3065
   p80: 23.0424
   p90: 25.7348
   p95: 27.9985
   p98: 29.2158
   p99: 29.5974

Qwen2-7B-Instruct (long.jsonl)

指标	4	8	12	16	20	25	30	35	40
用时	1501.129	831.393	661.167	553.051	492.972	482.926	503.931	708.094	1708.086
QPS	0.066	0.120	0.151	0.181	0.203	0.197	0.169	0.102	0.036
延迟	58.530	63.844	75.483	81.761	93.514	95.232	82.990	67.363	55.090
吞吐量	150.200	268.802	340.991	411.723	450.299	437.290	369.642	224.586	79.384
p50	61.1869	67.7709	79.9282	85.2388	101.0449	100.2105	84.4174	67.4104	57.0599
p90	63.1877	70.4871	84.9531	89.7831	106.6341	113.0055	106.2575	81.7474	59.3524
失败	1	0	0	0	0	5	15	28	38

平均每个请求的输入 token 数: 1600
平均每个请求的输出 token 数: 2200

部署

python -m vllm.entrypoints.openai.api_server \
    --host 0.0.0.0 --port 8008 \
    --model /data/models/llm/qwen/Qwen2-7B-Instruct/ \
    --served-model-name qwen2-7b \
    --tensor-parallel-size 4 \
    --dtype=float16 \
    --max-model-len 16000

压测

evalscope-perf http://172.16.33.66:8008/v1/chat/completions qwen2-7b \
    ./datasets/open_qa.jsonl \
    --parallels 4 \
    --parallels 8 \
    --parallels 12 \
    --parallels 16 \
    --parallels 20 \
    --parallels 25 \
    --parallels 30 \
    --parallels 35 \
    --parallels 40 \
    --n 100

parallel 4

Benchmarking summary: 
 Time taken for tests: 1501.129 seconds
 Expected number of requests: 100
 Number of concurrency: 4
 Total requests: 100
 Succeed requests: 99
 Failed requests: 1
 Average QPS: 0.066
 Average latency: 58.530
 Throughput(average output tokens per second): 150.200
 Average time to first token: 58.530
 Average input tokens per request: 1614.000
 Average output tokens per request: 2277.475
 Average time per output token: 0.00666
 Average package per request: 1.000
 Average package latency: 58.530
 Percentile of time to first token: 
   p50: 61.1869
   p66: 61.9394
   p75: 62.4250
   p80: 62.7114
   p90: 63.1877
   p95: 63.8551
   p98: 64.0589
   p99: 64.3522
 Percentile of request latency: 
   p50: 61.1869
   p66: 61.9394
   p75: 62.4250
   p80: 62.7114
   p90: 63.1877
   p95: 63.8551
   p98: 64.0589
   p99: 64.3522

parallel 8

Benchmarking summary: 
 Time taken for tests: 831.393 seconds
 Expected number of requests: 100
 Number of concurrency: 8
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.120
 Average latency: 63.844
 Throughput(average output tokens per second): 268.802
 Average time to first token: 63.844
 Average input tokens per request: 1614.000
 Average output tokens per request: 2234.800
 Average time per output token: 0.00372
 Average package per request: 1.000
 Average package latency: 63.844
 Percentile of time to first token: 
   p50: 67.7709
   p66: 68.7757
   p75: 69.2135
   p80: 69.4998
   p90: 70.4871
   p95: 71.4362
   p98: 74.7053
   p99: 77.3827
 Percentile of request latency: 
   p50: 67.7709
   p66: 68.7757
   p75: 69.2135
   p80: 69.4998
   p90: 70.4871
   p95: 71.4362
   p98: 74.7053
   p99: 77.3827

parallel 12

Benchmarking summary: 
 Time taken for tests: 661.167 seconds
 Expected number of requests: 100
 Number of concurrency: 12
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.151
 Average latency: 75.483
 Throughput(average output tokens per second): 340.991
 Average time to first token: 75.483
 Average input tokens per request: 1614.000
 Average output tokens per request: 2254.520
 Average time per output token: 0.00293
 Average package per request: 1.000
 Average package latency: 75.483
 Percentile of time to first token: 
   p50: 79.9282
   p66: 81.6170
   p75: 82.3043
   p80: 82.8302
   p90: 84.9531
   p95: 87.1365
   p98: 94.9953
   p99: 96.4239
 Percentile of request latency: 
   p50: 79.9282
   p66: 81.6170
   p75: 82.3043
   p80: 82.8302
   p90: 84.9531
   p95: 87.1365
   p98: 94.9953
   p99: 96.4239

parallel 16

Benchmarking summary: 
 Time taken for tests: 553.051 seconds
 Expected number of requests: 100
 Number of concurrency: 16
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.181
 Average latency: 81.761
 Throughput(average output tokens per second): 411.723
 Average time to first token: 81.761
 Average input tokens per request: 1614.000
 Average output tokens per request: 2277.040
 Average time per output token: 0.00243
 Average package per request: 1.000
 Average package latency: 81.761
 Percentile of time to first token: 
   p50: 85.2388
   p66: 86.6946
   p75: 87.8569
   p80: 88.2254
   p90: 89.7831
   p95: 91.9183
   p98: 93.1188
   p99: 94.1187
 Percentile of request latency: 
   p50: 85.2388
   p66: 86.6946
   p75: 87.8569
   p80: 88.2254
   p90: 89.7831
   p95: 91.9183
   p98: 93.1188
   p99: 94.1187

parallel 20

Benchmarking summary: 
 Time taken for tests: 492.972 seconds
 Expected number of requests: 100
 Number of concurrency: 20
 Total requests: 100
 Succeed requests: 100
 Failed requests: 0
 Average QPS: 0.203
 Average latency: 93.514
 Throughput(average output tokens per second): 450.299
 Average time to first token: 93.514
 Average input tokens per request: 1614.000
 Average output tokens per request: 2219.850
 Average time per output token: 0.00222
 Average package per request: 1.000
 Average package latency: 93.514
 Percentile of time to first token: 
   p50: 101.0449
   p66: 103.1628
   p75: 104.5066
   p80: 105.2498
   p90: 106.6341
   p95: 109.0586
   p98: 112.6580
   p99: 114.0485
 Percentile of request latency: 
   p50: 101.0449
   p66: 103.1628
   p75: 104.5066
   p80: 105.2498
   p90: 106.6341
   p95: 109.0586
   p98: 112.6580
   p99: 114.0485

parallel 25

Benchmarking summary: 
 Time taken for tests: 482.926 seconds
 Expected number of requests: 100
 Number of concurrency: 25
 Total requests: 95
 Succeed requests: 95
 Failed requests: 0
 Average QPS: 0.197
 Average latency: 95.232
 Throughput(average output tokens per second): 437.290
 Average time to first token: 95.232
 Average input tokens per request: 1614.000
 Average output tokens per request: 2222.937
 Average time per output token: 0.00229
 Average package per request: 1.000
 Average package latency: 95.232
 Percentile of time to first token: 
   p50: 100.2105
   p66: 103.2044
   p75: 104.4999
   p80: 105.5968
   p90: 113.0055
   p95: 117.3441
   p98: 119.4187
   p99: 119.9363
 Percentile of request latency: 
   p50: 100.2105
   p66: 103.2044
   p75: 104.4999
   p80: 105.5968
   p90: 113.0055
   p95: 117.3441
   p98: 119.4187
   p99: 119.9363

parallel 30

Benchmarking summary: 
 Time taken for tests: 503.931 seconds
 Expected number of requests: 100
 Number of concurrency: 30
 Total requests: 85
 Succeed requests: 85
 Failed requests: 0
 Average QPS: 0.169
 Average latency: 82.990
 Throughput(average output tokens per second): 369.642
 Average time to first token: 82.990
 Average input tokens per request: 1614.000
 Average output tokens per request: 2191.459
 Average time per output token: 0.00271
 Average package per request: 1.000
 Average package latency: 82.990
 Percentile of time to first token: 
   p50: 84.4174
   p66: 86.3143
   p75: 88.0736
   p80: 91.9183
   p90: 106.2575
   p95: 109.8099
   p98: 118.7324
   p99: 119.7411
 Percentile of request latency: 
   p50: 84.4174
   p66: 86.3143
   p75: 88.0736
   p80: 91.9183
   p90: 106.2575
   p95: 109.8099
   p98: 118.7324
   p99: 119.7411

parallel 35

Benchmarking summary: 
 Time taken for tests: 708.094 seconds
 Expected number of requests: 100
 Number of concurrency: 35
 Total requests: 72
 Succeed requests: 72
 Failed requests: 0
 Average QPS: 0.102
 Average latency: 67.363
 Throughput(average output tokens per second): 224.586
 Average time to first token: 67.363
 Average input tokens per request: 1614.000
 Average output tokens per request: 2208.722
 Average time per output token: 0.00445
 Average package per request: 1.000
 Average package latency: 67.363
 Percentile of time to first token: 
   p50: 67.4104
   p66: 67.9889
   p75: 68.3605
   p80: 68.5264
   p90: 81.7474
   p95: 116.1246
   p98: 118.3541
   p99: 119.0251
 Percentile of request latency: 
   p50: 67.4104
   p66: 67.9889
   p75: 68.3605
   p80: 68.5264
   p90: 81.7474
   p95: 116.1246
   p98: 118.3541
   p99: 119.0251

parallel 40

Benchmarking summary: 
 Time taken for tests: 1708.086 seconds
 Expected number of requests: 100
 Number of concurrency: 40
 Total requests: 62
 Succeed requests: 62
 Failed requests: 0
 Average QPS: 0.036
 Average latency: 55.090
 Throughput(average output tokens per second): 79.384
 Average time to first token: 55.090
 Average input tokens per request: 1614.000
 Average output tokens per request: 2187.000
 Average time per output token: 0.01260
 Average package per request: 1.000
 Average package latency: 55.090
 Percentile of time to first token: 
   p50: 57.0599
   p66: 57.4411
   p75: 58.2631
   p80: 58.5435
   p90: 59.3524
   p95: 62.2720
   p98: 96.8205
   p99: 99.6312
 Percentile of request latency: 
   p50: 57.0599
   p66: 57.4411
   p75: 58.2631
   p80: 58.5435
   p90: 59.3524
   p95: 62.2720
   p98: 96.8205
   p99: 99.6312

目录

大模型推理性能压测工具

安装 EvalScope

压测命令的使用

数据集

中文聊天 HC3-Chinese

代码问答 Codefuse-Evol-Instruct-Clean

构造长输入和输出的数据集

实验结果对比

🏆 vLLM ⚔️ XInference (T4: 4X16G)

🏆 MindIE (910B4: 8X32G) ⚔️ vLLM (T4: 4X16G)

实验结果（MindIE）

Qwen1.5-7B-Chat

Qwen1.5-7B-Chat (long.jsonl)

Qwen1.5-14B-Chat

Qwen2-72B-Chat

Qwen2-72B-Chat (long.jsonl)

DeepSeek-Coder-6.7B-Instruct

绘图代码

实验结果（vLLM）

Qwen1.5-7B-Chat

Qwen2.5-72B-Chat

实验结果（XInference - MindIE）

实验结果（Nvidia T4: XInference - vLLM）

Qwen2-7B-Instruct

实验结果（Nvidia T4: vLLM）

Qwen2-7B-Instruct

Qwen2-7B-Instruct (long.jsonl)

参考资料