After optimizing lately my vllm startup I started optimizing performance of vLLM during inference. Thing it had abysmal performance spanning like 2-4ts. Or even 0.5ts with 180k context. That was really terrible and I started to think that I wont be able to get better performance out of Strix Halo inside my Desktop Framework.
But… I remembered people praising it and its great usability with agents and Qwen models. How is that possible since it have so terrible performance.
I tested new version of Qwen 3.6 MOE in GGUF format with MTP (it was not available till recently in llama.cpp). On Vulkan I was able to achieve even 60t/s! Wow!
How is that possible? I understand that llama.cpp is optimized for single user usage but still… 60t/s vs 4? That is gigantic difference.
And then I tried it on agent and it was often almost all the time with more complex tool usage like regex replace for example.
I did another web search and some people were praising dense model as more correct. I decided to give it a go and it works!
It is much slower achieving only 20-30t/s in llama.cpp, but it is better to have slower but very often outputting correct text model instead of the wrong one like with MOE.
Just to have comparison I tested both llama.cpp with Qwen 3.6 27B:
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Maximum request concurrency: 1
Benchmark duration (s): 3473.48
Total input tokens: 2915
Total generated tokens: 25600
Request throughput (req/s): 0.03
Output token throughput (tok/s): 7.37
Peak output token throughput (tok/s): 4.00
Peak concurrent requests: 2.00
Total token throughput (tok/s): 8.21
---------------Time to First Token----------------
Mean TTFT (ms): 703.98
Median TTFT (ms): 718.68
P99 TTFT (ms): 737.81
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 133.42
Median TPOT (ms): 124.01
P99 TPOT (ms): 158.99
---------------Inter-token Latency----------------
Mean ITL (ms): 367.61
Median ITL (ms): 355.41
P99 ITL (ms): 1000.07
==================================================
Less than 10t/s. And this is for small prompts about 20-30t. For over 100k context it would be most probably 4t/s.
But then I tested llama.cpp.
============ Serving Benchmark Result ============
Successful requests: 100
Failed requests: 0
Maximum request concurrency: 1
Benchmark duration (s): 1029.14
Total input tokens: 2915
Total generated tokens: 25600
Request throughput (req/s): 0.10
Output token throughput (tok/s): 24.88
Peak output token throughput (tok/s): 52.00
Peak concurrent requests: 2.00
Total token throughput (tok/s): 27.71
---------------Time to First Token----------------
Mean TTFT (ms): 315.49
Median TTFT (ms): 194.85
P99 TTFT (ms): 1046.75
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 39.10
Median TPOT (ms): 29.51
P99 TPOT (ms): 120.42
---------------Inter-token Latency----------------
Mean ITL (ms): 38.82
Median ITL (ms): 0.02
P99 ITL (ms): 362.10
==================================================
27 t/s. This is huge difference. And llama.cpp can have similar throughput of tokens even with 200k context. Yes only for one user but on the other hand I can run two models like that and have agent use one and my AI assistant use another one.
Still this is very peculiar situation, why there is such difference. I will test it more but even if I messed up some of kernel parameters for GTT, which I do not think I did, with docker image that I am using it should rather OOM instead of getting so slow. Also with docker image all necessary libraries should be bundled there.
I will keep testing it but right now I am again using llama mostly for inference.
