GPU on InternetException

I built an app and it cost almost $40

Fri, 28 Aug 2026 17:33:25 +0200

I posted already about me selfhosting LLM models. Right now I have two devices:

Framework Desktop
Custom Ryzen Threadripper server with R9700 32GB GPU

Both are decent with llama.cpp. The biggest issue is prefill times.

I have from work subscription for Claude and Codex which I tested too for last couple of months. It is OK but lack of thinking traces really hurt to keep up with what models are doing.

Junie Coding Agent short review

Wed, 13 May 2026 07:40:03 +0200

I have been running few coding agents in my spare time on my personal projects and Junie does not look great when paired with local model

First of all, local models are capable but their ability to answer prompts in sustainable speeds degrades very quickly. I was experimenting with Qwen3.6-35B-A3B and Qwen3.5-35B-A3B. Both seems to be fairly capable, which is great for a local model that can run on decent GPU that is few years old (I have Radeon 7900 XTX with 24GB of VRAM) - that is outstanding considering what we thought about computers i.e. ten years ago. If someone then would say to me that we will be able to converse with our applications in human language and instruct them to do useful stuff, I would say: ‘Impossible!’. But we can and it does work but it have limitation, mainly in context size.

Bonding Network interfaces in Debian Forky

Tue, 31 Mar 2026 21:18:00 +0200

I am experimenting lately on with Nanobot and running self hosted language models on Desktop Framework.

I was trying to run lately Qwen 3.5 397B model on this small PC and it was a bit hard because I just do not have enough disk space on it. I bought just small NVMe SSD disk. With current prices, I planned to use disk space on other server I have at my hope as storage

Swapping vLLM to Llama.cpp

Thu, 26 Mar 2026 20:04:51 +0100

Yesterday I posted about vLLM being nice but optimized for different usage I am being really interested in . I commented on Llama Swap GH issue asking for help my problem of vLLM being shutdown before it actually started. During writing that comment I was thinking about Llama.cpp and it inability to run Qwen 3.5. It was at the beginning of March when I started (again) experimenting with LLMs. I tried to run Qwen models with release version of Llama.cpp and it was failing with ‘model architecture of Qwen3 is not found’ (or something like that; I do not really remember). Because of that I switched to using vLLM, even if it could not run Qwen 3.5, it could run Qwen 3 just fine from the docker image. Sglang seemed like not a good choice for my needs.

vLLM is fast but kind of slow

Wed, 25 Mar 2026 18:53:05 +0100

I was using vLLM to run my AI assistant models for few weeks now. It is quite nice, production ready framework that have few bugs . ROCm support and by extent Strix Halo support is quite nice, even if AMD GPU docker image does not work on Strix Halo without aiter in custom version.

Interference speeds are quite nice. You can get 20-30t/s out of new AMD APU. Which is totally usable. Here is sample test I done on this device via LLM API Throughput Benchmark:

Teaching my AI assistant to surf the web part 2

Fri, 20 Mar 2026 12:01:32 +0100

Few days ago I thought my assistant to do web search via Browsh and Duck Duck Go. It was working for some time with small problems, like for example I changed host name of the machine hosting nanobot and forget to adjust the host name in skill URL. Another small pain point is that sometimes returns timeout instead of web page content and not falls back to Brave Search API, and since I do not like that service, there is no API key and it falls back to DDG via python code. So it is mostly slow but I am still happy with that.

Assigning VRAM on AMD AI Max+ 395

Sun, 08 Mar 2026 11:33:00 +0100

Few days ago I started playing with AI assistant and I decided to buy new hardware dedicated to running LLM. I bought Framework Desktop board. So far it is really great but running more than one model is a bit difficult and I wanted to test few things while my nanobot is running undisrupted.

I did quick search on my phone before buying and people were saying that it is possible to set it up via BIOS settings up to 96GB.

How I taught my AI assistant to read websites

Tue, 03 Mar 2026 22:16:00 +0100

Main problem with running your own AI assistant is speed. Nanobot is lightweight but main problem it time od the reponses generated by the model. I am running right now Qwen 3 27B parameters. Average reponse varies but it is at least few seconds. If I would have faster GPU than Radeon 7900 XTX than probably it would faster but prices of GPUs are crazy now.

One of the things I have been testing my little chatbot with was searching for stuff on the internet. For example I like to check prices of some PC parts from time to time. Just out of curiosity and to have a knowledge how crazy it is (NVMe SSDs of 8TB sizes are at least 2x times pricier than last year, on the other hand salary for my profession dropped to 60-70% last 3 years, without inflation, with inflation it would be around 58%). Things are crazy.

Self hosting your AI assistant efficiently

Sun, 01 Mar 2026 06:27:00 +0100

I wrote previously why running models that you want to use on your daily driver is not great. Basically it is not convenient and it is fine for testing but if you want to run while doing something else it not great experience.

Of course you can have more powerful device so you will not notice few models running in the background but this is like running a fan with heater under your desk.

Running LLM on your daily PC and why it is not good

Fri, 27 Feb 2026 15:11:00 +0100

Two days ago I started playing a bit with my own AI assistant. It has been very great and funny experience so far but with few wrinkles. For example:

performance of new models on previous generation GPU (Radeon RX 7900 XTX) is surprisingly good with average of 28t/s but it takes whole VRAM and put a load on PC overall.
It takes whole lot of disk space. I.e. one vllm docker image to download and extract needs about 18gb; this seems a bit excessive.
Rocm installation needed a bit like 9gb to install all of its binaries. How much code do you need to produce 9gb of compiled code? This is crazy.
I have 16c/32t CPU and it seems to be enough for most of my workload but with llm in the background sometimes most of it is in use.
32gb is not enough, but with current prices I can suffer for now. :)
If you want to swap models a lot you need a lot of disk space since bigger models takes several gigabytes.
You need decent internet speed to be able to download it.

Yes you may say that it is fine in 2026, everybody have fiber connection now! Yes but when you want to test something and new docker image is downloading in with 1mb/s speed, you may have a walk in that time.