Desktop Framework on InternetException

I built an app and it cost almost $40

Fri, 28 Aug 2026 17:33:25 +0200

I posted already about me selfhosting LLM models. Right now I have two devices:

Framework Desktop
Custom Ryzen Threadripper server with R9700 32GB GPU

Both are decent with llama.cpp. The biggest issue is prefill times.

I have from work subscription for Claude and Codex which I tested too for last couple of months. It is OK but lack of thinking traces really hurt to keep up with what models are doing.

Running Qwen 3.5 397B on Desktop Framework

Mon, 30 Mar 2026 17:53:10 +0200

Today I finally was able to download and run largest Qwen 3.5 model. I downloaded the smallest quant even if a bit bigger probably would fit too, but I want to run also few smaller models at the same time, so 107GB for UD-IQ1_M is the biggest I was able to spare.

I downloaded and was able to run it with llama.cpp with the following command:

./llama-server \
 -hf unsloth/Qwen3.5-397B-A17B-GGUF:UD-IQ1_M\
 --ctx-size 32000 \
 --no-warmup \
 --no-mmap \
 --flash-attn on \
 --n-gpu-layers 1 \
 --host 0.0.0.0\
 --port $1 \
 --direct-io \
 --jinja --chat-template-file /data/apps/vllm/qwen3_nonthinking.jinja

My intention was to test it with nanobot to see if this will be capable agent for my AI assistant.

Desktop Framework AI assitant part 3

Mon, 16 Mar 2026 21:31:03 +0100

Recently I started playing with Nanobot which is a bit like OpenClaw but, in my opinion much better since it is small and simple and have native integration with custom OpenAI APIs providers. I decided to use vLLM because I wanted to use Qwen 3.5 because according to my quick reaserach it is pretty good in such agentic usage and Qwen models are built with integration with SgLang and vLLM frameworks in mind. Also AMD is testing their own drivers and libraries, ROCm on docker images with SgLang and vLLM and Desktop Framework is AMD APU. So all of it since like a good idea since I decided to go with Strix Halo unifies memory architecture for my AI assistant. I bought Desktop Framework motherboard and played with it a little to test models performance on ROCm inside the docker. It was not blazing fast but enough to actually have working solution.

Assigning VRAM on AMD AI Max+ 395

Sun, 08 Mar 2026 11:33:00 +0100

Few days ago I started playing with AI assistant and I decided to buy new hardware dedicated to running LLM. I bought Framework Desktop board. So far it is really great but running more than one model is a bit difficult and I wanted to test few things while my nanobot is running undisrupted.

I did quick search on my phone before buying and people were saying that it is possible to set it up via BIOS settings up to 96GB.

Desktop Framework AI assistant part 2

Thu, 05 Mar 2026 18:59:13 +0100

Today my new NVMe disk arrived. I bought it for my new Desktop Framework mini PC on new AMD APU intended for AI workloads. Certainly it will be better suited for running my AI assistant then my daily working PC. Though performance will be a bit worse. But still better than CPU.

Anyway I decided to split my assistant into 2 parts. The nanobot part will be working on my main server and model (or models) that will be working on Desktop Framework. This way more powerful machine will be running AI assitant UI and operations and LLM capable device will do faster interference - that will be the Framework device. Nanobot communicate with model via OpenAI API anyway so it is not a problem. Maybe a bit of security in terms of HTTPS with some auth would not hurt. Or maybe I will put it in private subnetwork that can’t be accessed from outside and only have internet when necessary? Running dockers and all of those python AI firework is not exactly secure but maybe I will deal with that later.

Desktop Framework AI assistant part 1

Wed, 04 Mar 2026 16:06:41 +0100

I ordered Desktop Framework motherboard last week and today it arrived. It was neatly packaged and did not come with anything else then some piece of plastic to mount your own fan on top of the radiator. But I did not need much else. At least for first tests.

I had some PSU laying around that was compatible. I have some old fan that I was able connect to board even if it was a bit to small but would cool it a bit at least.