Ryzen AI Max · 5 posts (all tags)
#vLLM #Qwen #Ryzen AI Max #Docker
I have Desktop Framework for 2 months now. It is pretty capable for interefence with small contexts. With longer contexts it degrades quickly. It can for example can run 100k tokens contexts for 10minutes. Because of those limitations I decided to try vllm again - maybe it will be quicker. Two months is long time in LLM frameworks last time I was using vllm it has version 0.17. Now stable is 0.19.
#vLLM #Qwen #Ryzen AI Max #GPU
I was using vLLM to run my AI assistant models for few weeks now. It is quite nice, production ready framework that have few bugs . ROCm support and by extent Strix Halo support is quite nice, even if AMD GPU docker image does not work on Strix Halo without aiter in custom version.
Interference speeds are quite nice. You can get 20-30t/s out of new AMD APU. Which is totally usable. Here is sample test I done on this device via LLM API Throughput Benchmark:
#Qwen #Ryzen AI Max #vLLM #Nanobot
I bough lately Desktop Framework with intention of running Qwen 3.5 as model for my AI assistant on Nanobot . At first I could not run this model on this hardware for some weird bug in one of the libraries. I explained why and how I fixed it here . In this post I will just put the list of packages that I used to run Qwen finally and vLLM command switches and parameters.
#Ryzen AI Max #VRAM #Desktop Framework #GPU
Few days ago I started playing with AI assistant and I decided to buy new hardware dedicated to running LLM. I bought Framework Desktop board. So far it is really great but running more than one model is a bit difficult and I wanted to test few things while my nanobot is running undisrupted.
I did quick search on my phone before buying and people were saying that it is possible to set it up via BIOS settings up to 96GB.
#self-hosting #Ryzen AI Max #VRAM #GPU
I wrote previously why running models that you want to use on your daily driver is not great. Basically it is not convenient and it is fine for testing but if you want to run while doing something else it not great experience.
Of course you can have more powerful device so you will not notice few models running in the background but this is like running a fan with heater under your desk.
