Speculative Decoding on InternetException

Quick review of Laguna S 2.1

Fri, 24 Jul 2026 15:06:50 +0200

Just couple of days ago poolside released their new model: Laguna S 2.1. I was happy to try it since it have native support for 1M tokens in context. I am doing development of hobby projects on my phone, via SSH prompting models to do stuff and 250k tokens of context using Qwen 3.6 27B usually means going through few rounds of compaction before I am happy with the results. Having longer context would be better - in theory - without those ugly interruptions and reiterating stuff to agent that were lost in previous session.

Running Qwen 3.6 on vLLM with 6 draft tokens

Wed, 20 May 2026 20:13:55 +0200

Just out of curiosity I tested Qwen3.6-27B model with MTP speculative decoding with 6 draft tokens. Whole script looked like this:

docker run --rm --name="${2:-qwen3.6-medium-vllm}" \
 --group-add=video \
 --cap-add=SYS_PTRACE \
 --security-opt seccomp=unconfined \
 --device /dev/kfd \
 --device /dev/dri \
 -v /cache/huggingface:/root/.cache/huggingface \
 -v /cache/vllm:/root/.cache/vllm \
 --env "HF_TOKEN=$3" \
 -e TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 \
 -e VLLM_ROCM_USE_AITER=1 \
 -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
 -p "${1:-8000}":8000 \
 --ipc=host \
 --entrypoint '/bin/sh' \
 vllm/vllm-openai-rocm:nightly \
 -c "pip install fastokens; vllm serve Lorbus/Qwen3.6-27B-int4-AutoRound \
 --gpu-memory-utilization 0.48 \
 --dtype half \
 --optimization-level 1 \
 --enable-prefix-caching \
 --performance-mode interactivity \
 --kv-cache-dtype fp8_e4m3 \
 --language-model-only \
 --reasoning-parser qwen3 \
 --enable-auto-tool-choice \
 --tool-call-parser qwen3_coder \
 --speculative-config '{ \"method\": \"mtp\", \"num_speculative_tokens\": 2}' \
 --max-num-seqs 1 \
 --tokenizer-mode fastokens \
 --max-num-batched-tokens 262144 \
 #--max-model-len 262144"

Those are results with vLLM bench tool:

Lllama.cpp vs vLLM running Qwen 3.6 27B

Wed, 20 May 2026 17:27:57 +0200

After optimizing lately my vllm startup I started optimizing performance of vLLM during inference. Thing it had abysmal performance spanning like 2-4ts. Or even 0.5ts with 180k context. That was really terrible and I started to think that I wont be able to get better performance out of Strix Halo inside my Desktop Framework.

But… I remembered people praising it and its great usability with agents and Qwen models. How is that possible since it have so terrible performance.

Optimizing vLLM startup time

Thu, 07 May 2026 20:38:15 +0200

vLLM has very slow time to first token (TTFT). On AMD Ryzen AI Max+ 395 it can take even 10 minutes depending on exact settings and model size.

To make my user experience of using those things myself I need to have comparison of llama.cpp and vLLM startup time and performance. But to do this I needed to perform optimizations of startup of vLLM since it is even slower to start then it was with v0.17.