Few days ago I started playing with AI assistant and I decided to buy new hardware dedicated to running LLM. I bought Framework Desktop board. So far it is really great but running more than one model is a bit difficult and I wanted to test few things while my nanobot is running undisrupted.
I did quick search on my phone before buying and people were saying that it is possible to set it up via BIOS settings up to 96GB.
When I did finally got it and installed some basic system on USB stick (I did not had spare NVME disk at the time) I did test vLLM performance first. It was OK.
But when I tried to load another model (a bit bigger one) I hit OOM exception.
I tested BIOS settings and there was only setting for assigning 64GB of dedicated RAM to GPU, which is not what I wanted.
I reset those settings to default and tried to use this solution with AMD helper script. Solution looked sensible even if installation is via PIP which is always a bit problematic on Debian.
After installation I started the command:
amd-ttm 💻 Current TTM pages limit: 16469033 pages (62.82 GB) 💻 Total system memory: 125.65 GB
So far so good!
I tried changing it to other value:
❯ amd-ttm --set 100 🐧 Successfully set TTM pages limit to 26214400 pages (100.00 GB) 🐧 Configuration written to /etc/modprobe.d/ttm.conf ○ NOTE: You need to reboot for changes to take effect. Would you like to reboot the system now? (y/n): y
And rebooted!
And guess what? It did not work!
I should have known. From amount of emojis in there the whole thing smells with vibe coding. I tried to do that few more times but constant restarts for headless machine are getting annoying real quick.
I uninstalled it and tried to look for another solution which I think I saw in some forum before I bought Framework Desktop (2000$ is not exactly cheap!) with usage of options command.
But I could not find it and I gave for few days.
Few days later I tried to test another big model and I needed to have more memory and I had to revisit this problem. Luckily I was able to find this thread and this actually worked. I created file called: /etc/modprobe.d/amdgpu_llm_optimized.conf with following content:
options amdgpu gttsize=120000 options ttm pages_limit=31457280 options ttm page_pool_size=15728640
After that I did run:
sudo update-grub
though I must say I am not really sure if this was necessary but since it is really quick and does not brakes anything I am including it in this solution.
After that I just rebooted the machine and it worked. Running the amd-smi showed:
+------------------------------------------------------------------------------+ | AMD-SMI 26.2.1+fc0010cf6a amdgpu version: Linuxver ROCm version: 7.2.0 | | VBIOS version: 00107962 | | Platform: Linux Baremetal | |-------------------------------------+----------------------------------------| | BDF GPU-Name | Mem-Uti Temp UEC Power-Usage | | GPU HIP-ID OAM-ID Partition-Mode | GFX-Uti Fan Mem-Usage | |=====================================+========================================| | 0000:c1:00.0 AMD Radeon Graphics | N/A N/A 0 N/A | | 0 0 N/A N/A | N/A N/A 153/512 MB | +-------------------------------------+----------------------------------------+ +------------------------------------------------------------------------------+ | Processes: | | GPU PID Process Name GTT_MEM VRAM_MEM MEM_USAGE CU % | |==============================================================================| | 0 4927 python3.12 5.9 MB 62.5 KB 16.0 EB N/A | | 0 5497 python3.12 106.3 GB 5.6 MB 108.8 GB N/A | +------------------------------------------------------------------------------+
I did uninstalled amd-ttm so I do not know if this would be shown by this tool but I have more trust in amd-smi as of now and it really works by reading /sys/module/ttm/parameters/pages_limit file which you can read yourself:
and it was showing correct value now.
And that is it! Happy playing with your models!