Docker on InternetException

Running Qwen 3.5 on Strix Halo using vLLM and Docker

Sun, 26 Apr 2026 22:20:40 +0200

I have Desktop Framework for 2 months now. It is pretty capable for interefence with small contexts. With longer contexts it degrades quickly. It can for example can run 100k tokens contexts for 10minutes. Because of those limitations I decided to try vllm again - maybe it will be quicker. Two months is long time in LLM frameworks last time I was using vllm it has version 0.17. Now stable is 0.19.

Desktop Framework AI assistant part 1

Wed, 04 Mar 2026 16:06:41 +0100

I ordered Desktop Framework motherboard last week and today it arrived. It was neatly packaged and did not come with anything else then some piece of plastic to mount your own fan on top of the radiator. But I did not need much else. At least for first tests.

I had some PSU laying around that was compatible. I have some old fan that I was able connect to board even if it was a bit to small but would cool it a bit at least.

Running LLM on your daily PC and why it is not good

Fri, 27 Feb 2026 15:11:00 +0100

Two days ago I started playing a bit with my own AI assistant. It has been very great and funny experience so far but with few wrinkles. For example:

performance of new models on previous generation GPU (Radeon RX 7900 XTX) is surprisingly good with average of 28t/s but it takes whole VRAM and put a load on PC overall.
It takes whole lot of disk space. I.e. one vllm docker image to download and extract needs about 18gb; this seems a bit excessive.
Rocm installation needed a bit like 9gb to install all of its binaries. How much code do you need to produce 9gb of compiled code? This is crazy.
I have 16c/32t CPU and it seems to be enough for most of my workload but with llm in the background sometimes most of it is in use.
32gb is not enough, but with current prices I can suffer for now. :)
If you want to swap models a lot you need a lot of disk space since bigger models takes several gigabytes.
You need decent internet speed to be able to download it.

Yes you may say that it is fine in 2026, everybody have fiber connection now! Yes but when you want to test something and new docker image is downloading in with 1mb/s speed, you may have a walk in that time.

Adding the bot to Matrix server

Thu, 26 Feb 2026 15:04:00 +0100

I have my own matrix server for my own use. Usually it is used by me a bit by my family and most of it by my own services to sent me some notifications.

Lately I started playing with nanobot. It is personal AI assistant like OpenClaw. I wanted to be able to chat with it in my own Matrix via my own phone. To do that I needed to create new user dedicated