Posts

I posted already about me selfhosting LLM models. Right now I have two devices:

• Framework Desktop
• Custom Ryzen Threadripper server with R9700 32GB GPU

Both are decent with llama.cpp. The biggest issue is prefill times.

I have from work subscription for Claude and Codex which I tested too for last couple of months. It is OK but lack of thinking traces really hurt to keep up with what models are doing.

I know that LLM or Large Language Models are designed to mimic what people write, but still from time to time it surprises me and make me chuckle.

I am working on small tool that runs agents in confinement that gives those programs limited set of tools that you would want them to use only. Otherwise they may try to use sudo to change system settings or kill all your docker containers instead of only few were created by them. I had several times the same problem with codex not being able to start in such sandbox because of permissions issues. I am using Qwen 3.8 for now and it thinks a lot and this particular case it was thinking about those permissions.

Qwen 3.8 is out since Friday, 17 of August. I am running it on my own server since Saturday. I am running Q4 quants from unsloth .

So far it feels like a step up from Qwen 3.6 27B, which sometimes had problems figuring stuff in my project that I am developing almost entirely using models - just as a test of how far I can push LLMs in agentic coding - mostly local models, I must add. Qwen 3.8 seems like having no problem at all.

Just couple of days ago poolside released their new model: Laguna S 2.1. I was happy to try it since it have native support for 1M tokens in context. I am doing development of hobby projects on my phone, via SSH prompting models to do stuff and 250k tokens of context using Qwen 3.6 27B usually means going through few rounds of compaction before I am happy with the results. Having longer context would be better - in theory - without those ugly interruptions and reiterating stuff to agent that were lost in previous session.

I bought new phone and I started installing applications from previous device. When I installed termux I tried to copy ssh keys and ssh config from previous installation.

This is where problems started. I could not reach any local address. Nothing in 10.x.x.x was working. No pings no ssh.

I could do that to public IPs just fine.

I tried to install new version of packages but it also did not work.

I remember when I started selfhosting my own instance of GitLab, which had simple code editor that allows you to edit code quickly inline. Then they integrated with fork of Visual Studio Code and allowed you to code in real repository directly in GL.

I was thinking of doing quick code changes to applications at this way, change some configuration of my API, quick fix of my mobile phone for home automation and etc.

I do not know… I find it funny.

After few days fighting with local LLM to force it to write documentation and comments in code, in good quality I must add, and the forcing it to write readable unit tests, after I wrote:

File is untracked (new). Build passes. Ready for review.

I just answered.

awesome mate. It is friday we deserve a brake. High five.

I was thinking for a bit and answered:

Just out of curiosity I tested Qwen3.6-27B model with MTP speculative decoding with 6 draft tokens. Whole script looked like this:

docker run --rm --name="${2:-qwen3.6-medium-vllm}" \
    --group-add=video \
    --cap-add=SYS_PTRACE \
    --security-opt seccomp=unconfined \
    --device /dev/kfd \
    --device /dev/dri \
    -v /cache/huggingface:/root/.cache/huggingface \
    -v /cache/vllm:/root/.cache/vllm \
    --env "HF_TOKEN=$3" \
    -e TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 \
    -e VLLM_ROCM_USE_AITER=1 \
    -e FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE \
    -p "${1:-8000}":8000 \
    --ipc=host \
    --entrypoint '/bin/sh' \
    vllm/vllm-openai-rocm:nightly \
    -c "pip install fastokens; vllm serve Lorbus/Qwen3.6-27B-int4-AutoRound \
    --gpu-memory-utilization 0.48 \
    --dtype half \
    --optimization-level 1 \
    --enable-prefix-caching \
    --performance-mode interactivity \
    --kv-cache-dtype fp8_e4m3 \
    --language-model-only \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --speculative-config '{ \"method\": \"mtp\", \"num_speculative_tokens\": 2}' \
    --max-num-seqs 1 \
    --tokenizer-mode fastokens \
    --max-num-batched-tokens 262144 \
    #--max-model-len 262144"

Those are results with vLLM bench tool:

After optimizing lately my vllm startup I started optimizing performance of vLLM during inference. Thing it had abysmal performance spanning like 2-4ts. Or even 0.5ts with 180k context. That was really terrible and I started to think that I wont be able to get better performance out of Strix Halo inside my Desktop Framework.

But… I remembered people praising it and its great usability with agents and Qwen models. How is that possible since it have so terrible performance.

AI hype is calming down in some companies , but in others like in the one I am currently consulting for, we still on the hype.

Few days ago I was asked by my peer about some problem with the tool that I was mostly developing. This is fine are there are always some bugs in every software, though bug was concerning behavior on bad data and therefore out of scope really. Still I offered to help, regardless of the fact that correct fix would be to fix the bad data.

I have been running few coding agents in my spare time on my personal projects and Junie does not look great when paired with local model

First of all, local models are capable but their ability to answer prompts in sustainable speeds degrades very quickly. I was experimenting with Qwen3.6-35B-A3B and Qwen3.5-35B-A3B. Both seems to be fairly capable, which is great for a local model that can run on decent GPU that is few years old (I have Radeon 7900 XTX with 24GB of VRAM) - that is outstanding considering what we thought about computers i.e. ten years ago. If someone then would say to me that we will be able to converse with our applications in human language and instruct them to do useful stuff, I would say: ‘Impossible!’. But we can and it does work but it have limitation, mainly in context size.

vLLM has very slow time to first token (TTFT). On AMD Ryzen AI Max+ 395 it can take even 10 minutes depending on exact settings and model size.

To make my user experience of using those things myself I need to have comparison of llama.cpp and vLLM startup time and performance. But to do this I needed to perform optimizations of startup of vLLM since it is even slower to start then it was with v0.17.

I was playing over the weekend trying to write POC of C# interactive console. It is not ready yet but I wanted to have some cute hamster as logo for this project - I will probably incorporate it to some degree into Hamster Wheel , so cute hamster seemed appropriate.

Just for fun I decided to use LLMs to generate it since maybe I will not finish this project ever so trying to do it myself would be waste of time. And result probably would be terrible since I do not have any artistic skills.

I have Desktop Framework for 2 months now. It is pretty capable for interefence with small contexts. With longer contexts it degrades quickly. It can for example can run 100k tokens contexts for 10minutes. Because of those limitations I decided to try vllm again - maybe it will be quicker. Two months is long time in LLM frameworks last time I was using vllm it has version 0.17. Now stable is 0.19.

Vibe was the first coding agent I tried to use with local model. It works and it is fun to have ability to to just tell the computer to do something you want in plain English or even in your own language if your model support it.

Installation and Configuration

There are several installation methods for vibe that are available along to the usual, security nightmare of running of piping curl-downloaded scripts directly to bash :)
I advise using other installation methods for example via uv: