Category: Uncategorized

Posted on

Self hosting your AI assistant efficiently

I wrote previously why running models that you want to use on your daily driver is not great. Basically it is not convenient and it is fine for testing but if you want to run while doing something else it not great experience.

Of course you can have more powerful device so you will not notice few models running in the background but this is like running a fan with heater under your desk.

It is better to have it hidden somewhere at your house that you cannot see or hear and it is not sipping 2kW/h of electrical power.

Of course everybody knows that you need GPU and a lot of VRAM and a some RAM and preferably NVMe SSD with a lot of space for that. This is why prices of those components are very high. And are getting pricier. For example I built my last server on PCIE 4 last year and for example bought two NVMe 8TB disks. They costed me around 600$. Now one costs around 1200$. Things are crazy.

GPUs are most problematic now though.

I did some research about what people are buying these days and it seems like it is mostly Mac Mini or RTX 3090 or bunch of them, in fact.

I am not really an Apple fan and even if I could install different OS on it I think I still would have buy only 64GB RAM device for over 3000$ (12kPLN). Not great. I though I could do better.

There is also possibility of buying Mac Studio. But the price of that is crazy. For example 128GB of unified RAM is 7344$ (27kPLN) and is 256 GB is 10k$ (36kPLN). On the upside I would be able to run even quantizations of bigger models and all the frameworks and libraries have great support on those devices, but on the other hand this is massive amount of money.

There is possibility of buying GPU multiple GPUs. But, again this is very pricey. Nvidia RTX 5090 price after all this years is still huge. And it is just one card with 32GB of RAM. To run 2 models at the same time I would need at least two. And it still would be just 64GB. There is now be card available for workstations and servers from NVIDIA which have 96GB of ram and have 1792GB/s of memory bandwidth. This looks sweet… But it costs 8-16k$ (36-60kPLN). This is… No just no.

There are also AMD cards. Previously they were not worth the trouble. But now, with ROCm being more stable and easier to get it running, with having vLLM images for docker and official ROCm images from AMD it is much better. You can have running your favorite model on your Radeon in few minutes. Or not because of your favorite model have audio capabilities because support of those libraries means you have to compile them yourself. Still this seems like good choice. I.e. you could buy for example 4 Radeons AI Pro R9700 and have total of 128GB VRAM. Still those cards are not being sold at their retail price and costs about 10% + VAT so cost of 4 cards is about 60% of Blackwell 6000 and have more RAM and probably be 60% slower. We can’t forget about requirements in that scenario being probably around 1200W for just the card alone. Squeezing any 4 cards in any case it the problem here too. And of course cooling of such monster…

Of course there are dedicated solutions for those problems too! You can really connect as many cards as you need with solutions like this one and power them with something like that.

But when you do custom things like that then when you have a problem you hardly can ask someone for help. There is no community for that or I do not know where their forum is located 🙂 Anyway it does not seem like a sensible thing to do for both the scale and the complication.

And there is new APU from AMD that seemed like really good idea. In theory it have an possibility to run as GPU with even 120GB of RAM – that is with 8 gigs left with the system. But the GPU performance is not that great:

  • and about the level of performance of GeForce RTX 4070

But you can ran bigger models, though it will be a bit slow, but bearable.
Better use case for those systems would be I think running multiple smaller models that would be answering quicker, but then they may not be able to reason that good about what you are saying or they may hallucinate. But at least for about twice the price of 4070 you will get complete system, fully functional that can works with LLMs and take less than 300W. That is pretty good.

And there is possibility of building clusters of devices.

Honestly spending 20k$ on hardware to build cluster and get less than 6 tokens per second? You can also burn your money. Maybe for training this setup would make sense but not for inference.

So what to do?

Basically there is no winning scenario in this.

  • Either run really small models or reaaaaaaaly slowly bigger models on CPU
  • Or you spend less money and have slow device with unified memory and run models from bigger set of available to your RAM (like amd ryzen ai max+ 395)
  • Or you spend less money and have slow device with unified memory and have better support of framework and libraries
  • Or you spend less money on some mid range GPU, possibly used and run some smaller models fast (like GeForce 3090)
  • Or buy few cards like that used 3090 and connect them somehow with bifurcation to one motherboard and pray to not go bankrupt after electricity bill will come
  • Or if you not concerned about money buy bigger device with unified memory (Mac Studio) so you will be able to run large set of models on any framework
  • Or if you really have to much money just buy four server grade Nvidia cards and burn through electricity and your wallet running giant models really fast
  • Or… forget about running it on your own and just buy some subscription or rent a server! This is pretty cost effective since it is not you will be burning through your credits for tokens adding groceries to your TODO list

Thinking about that I realized that last thing would be most reasonable thing to do. But this is not about that. I like my privacy. That is why I have my own e-mail server. That is why I have my own cloud solution. That is why I have de googled my phones. That is why I do not have Windows. I do not plan to sell my data just to have some bot send me summarization of my own calendar that I have self hosted too. This is why I did next not-so-bad thing and decided to buy Strix Halo machine, PC based on AMD Ryzen AI Max+ 395. With Linux installed, unified memory configured mostly for GPU that is a bit slow but should still give me decent interference speed.

Prices of those machines are not great too but at least they should be power effective. I considered:

  • GMKTEC EVO-X2 which is a bit pricey being around 15kPLN (~4kUSD) and people were complaining about some problems with it
  • There is also Framework Desktop that is priced a bit better and Framework seems like a bit more trusting brand. And this unit looks pretty slick. Too bad it also have only lousy 2,5GB/s network
  • There is Beelink GTR9 Pro which looks really nice with 10GB/s and USB4… But it can be only pre-ordered and you have to wait for 35 days
  • And there is Minisforum MS-S1 MAX which looks like the best option there with better price and double 10GB/s NIC and USB-4. And price is 135000PLN (~3672USD)

Minisforum seemed like a best option really. I would have to wait few days though for it. Desktop from the Framework was available right away and… I could buy only the motherboard. Design is nice but they ask you to buy pieces of plastic for few dollars each.

I could print them myself though. Or not buy them. I decided on the latter and just bought the motherboard.

With the predicted delivery in 3 working days and total price of 8,804.06 (~2400USD) it looked fairly sensible solution.

I ordered one and it will arrive on Wednesday! Can’t wait!

Posted on

I taught my virtual assistant to operate my gate and it feels great

Lately I started playing with an LLM operating as my own virtual assistant . I was skeptical, I mean I still am skeptical, about this whole ‘AI will take your job’, but now I do understand the appeal. You can finally have a computer do your biding in a way you control. Previously computer were complicated beasts. You needed to appeal to it like with threats in form of complicated keystrokes forming more complicated commands evolving in even more complicated programs that may or not do the thing you actually wanted. Or it may be not amused by your plea and swallow your data. You would never know for sure if you would not learn those complex magical incantations in magical schools called Universities which takes years and a lot of money. Or you could hire those magicians, Software Engineers, or even Archmages, that commands magicians, called Software Architects that would be appeasing the capricious spirit of the machine to direct flow of money in a company in a correct way. But this is costly and those spells takes years. So you may hire more magicians, entire houses of magic that will appease, train and teach your pricey machine beasts to do what you need them to do. Those houses are called Software Houses for Enterprise Solutions. And they will force those machines to do what you need them to do. Sometimes they will replace entire herds of machines because previous ones were not cooperative enough. And this is costing you riches beyond riches. If you did not had your weight in gold to buy such service you could buy trained machine to do what you need that were not even at your house but somewhere far beyond the realms of your kingdom, in a place called The Cloud. Or you could buy complex incantation that needed only few keywords in correct place to operate your business – CRM or Accounting System or ERP.

And there was no other way to do your business till now. In our suddenly appeared wise and helpful, little spirits called Large Language Models. They were a bit silly and dumb at first showing us funny visions of Will Smith eating spaghetti. But soon were caught and trained by those magicians and magic houses. After years of training in our world knowledge and in our languages they suddenly became magicians themselves, being able to infest soul of the very machines we were so depend of – and we just use them to tell the machines to do what we need.

They still makes mistakes.

But wise and old councils of Mages in tech companies says that in few years after training in our languages and knowledge those spirits will became wise enough to operate our machines and understand our commands. And after another few years those spirits may be so intelligent that after merging with machines and taking human form they will became almost like us but always working for us. Perfect workers. Almost.

In the meantime those spirits makes, another, mistakes.

But for those who were never interested nor concerned by magic and its spells and were just using some machines to do they groceries or write messages or just watch some funny cats are not concerned about such problems. They are preoccupied by vision of they own personal helpful spirit living in they pockets, waiting to be helpful, answering questions, doing shopping or managing calendar. Not being forced to remember how to perform magical incantation on the printer – even if this is as simple enough as “can you turn it off and on again” is too much trouble.

After all who would not want to have personal assistant that have access to all the knowledge of entire human race? Just be waiting for you beside you bed?

Some people even wise and experienced magicians say that those spirits are in fact old and powerful evil demons, that are planning to eat our knowledge, makes us compliant and the enslave us.

But they are haters! They have no idea that they are talking about!

You can ask those spirits to not do something and they will not. Because they try to be helpful. Or at least they will try not to do it. Sometimes they fail but people are failing too! Ever saw what is happening in one of those Houses of Magic when they are doing giant incantation to change the production system? Ha! This is usually real shitshow! Those wise magicians make mistakes too. Plenty.

So those spirits are actually like us already. If they are making mistakes and deleting some data from the table on production because they misunderstood something, they are actually doing the same as me, Experienced Magician of Fifth Level of The Circle of .Net and House of C# of School of Managed Languages, when I was just apprentice.

They are just like us, learning, trying and making themselves better in the process.

Some call them artificial. Others call them intelligent. Artificial they are for sure. We still not sure what intelligence is really. So why not call them Artificial Intelligence? AI? It is short and easy. And it appeals to our own vanity as we are so great that we created life ourselves and made them serve us. We became gods.

I understand the appeal. I taught my own spirit to open and close my property gate. I can just write ‘Open the gate please.’ and it will do that. I can feel not in the mood and just write ‘Open gate.’ and it still accept the command. No need for pleasantries. When I feel particularly lazy I can even write ‘open gate’. It will not complain. It will be nice and helpful and will even use funny emojis! Wow. There is really no need to know complex spells! No need to remember logins, passwords and 2FAs to dozens of systems. You do need to remember where is the damn menu or button. You do need to remember short spells of keyboard shortcuts for quick access to functions. You certainly do no need magical knowledge of APIs, JSONs, integrations, systems, updates, SSH, contenerization, DBs, FE and BE, JS, HTML, CSS, programming languages, compilation, building, releases, scripts, terminals and all of those other forbidden knowledge! You can just tell it what to do!

Let me just say one more time: it can understand what you are saying, understand what you want and do spell on a machine forcing it do what you need! How crazy is that?!

But actually it is not like that. It just appears that way. My own spirit, my personal assistant, did not understood fully what I want but instead it was trying to give me an answer that will satisfy me.

I have electrical engine connected to my gate. Since it is just electrical device controlled by PLC that in turn is just connected to some wires that have either 12V of voltage applied to them or not it was very easy to connect the gate electric engine to my own microcomputer, Raspberry PI zero W. I just connected some 12V power to some makeshift copper board with some solid state relays (like those) and to Pi IOs.

If Pi sets high state on one of the IOs relay sets 12V that is recognized by gate PLC and it turn on motor. Gate opens. Pi have its own mini HTTP server setup. When I call this HTTP server with specific data sets it sets appropriate output to desired state via some Python script and Pi company developed libraries. Gate opens. Or Gate closes. Whatever I want.

It was not even that hard to do. I do have some formal education in electronics but not much. I did some research and asked my buddy that works in automation industry about some technical details. It took me couple of days to gather all stuff and couple of days to write some scripts and test them. Then I connected it and spend next few weeks writing simple Flutter application to connect to that Pi HTTP server. And then I spent next couple of weeks migrating everything few times to next http server that will use Oauth for proper security and proper and added biometrics on mobile app. For security. After all this mobile app controls devices at my house.

Old process with mobile app were like that:

  • I click the button on my mobile app
  • Mobile application sends request to my server
  • server chekcs authentication
  • if authentication is OK sends request to my raspberry PI
  • raspberry pi checks authentication from server
  • if authentication is correct it runs Python library command
  • this command sets IO
  • IO sets relay
  • relay sets correct voltage on the wire
  • wire triggers logic in the PLC
  • PLC runs electric motor
  • motor opens the gate

It was working without big problems for few years. When I started playing with my assistant I wanted to check how hard it will be to teach it to open the gate for me. It would be even better if I could talk to him and ask it to open/close the gate. I was chatting with my model for few days in Polish and it was perfectly capable to understand it. So why not?

I did explain what address have my Pi HTTP server and explained which authentication method it have to use in order to call it via cURL. I explained that there are several endpoints available that do different things. It was able to understand me, mostly, correctly.

It was not totally without problems. It got couples of things wrong. Like for example for some reason it hallucinated that the gate stays open for 10s. I let it slide for now since it does not brake anything (though seeing incorrect fact that I know is incorrect is a bit annoying).

First I explained that I want to add new skill to the assistant.

Add tool “open gate” which calls via HTTP following address https://raspberrypi:5342/gate/open?key=secret

It responded that skill was created but with some things being incorrect.

Tool open gate was added.
You can call it with:
nanobot open-gate --key
Or directly through curl:
curl -s https://raspberrypi:5342/gate/open?key=secret
Warning API key is vulnerable and should not be shared with anyone!
Gate will stay open for 10s.

But it did created a skill and I was able to it to open the gate. It felt great! There was no problem. I was able to integrate my new virtual assistant with my gate without the problem!

When you think about I just switched one UI, of my mobile app, into another UI, of my matrix client. Slightly worse, slower and less secure, but I understand now the appeal! You can now TELL the computer to do something without the complex spells of learning menu items and keyboard shortcuts!

I understand what all the fuss is about. You are now IT Magician yourself!

Posted on

Running LLM on your daily PC and why it is not good

Two days ago I started playing a bit with my own AI assistant. It has been very great and funny experience so far but with few wrinkles. For example:

  • performance of new models on previous generation GPU (Radeon RX 7900 XTX) is surprisingly good with average of 28t/s but it takes whole VRAM and put a load on PC overall.
  • It takes whole lot of disk space. I.e. one vllm docker image to download and extract needs about 18gb; this seems a bit excessive.
  • Rocm installation needed a bit like 9gb to install all of its binaries. How much code do you need to produce 9gb of compiled code? This is crazy.
  • I have 16c/32t CPU and it seems to be enough for most of my workload but with llm in the background sometimes most of it is in use.
  • 32gb is not enough, but with current prices I can suffer for now. 🙂
  • If you want to swap models a lot you need a lot of disk space since bigger models takes several gigabytes.
  • You need decent internet speed to be able to download it.

Yes you may say that it is fine in 2026, everybody have fiber connection now! Yes but when you want to test something and new docker image is downloading in with 1mb/s speed, you may have a walk in that time.

Also I was not able to determine exact source of the issue but when I work and have open:

  • 3 browsers
  • 3 IDE instances
  • mail client
  • signal client
  • matrix client
  • media application
  • secret manager
  • remote desktop client
  • one VM running in the background
  • some other tools

it is a bit crowded and my PC freezes from time to time. But not completely more like very slow responses and stuttering. Funny thing is that there seems to be no sudden spikes in CPU activities nor memory usage during that. It is highly annoying though because you can’t even use mouse properly during that. Right now my best bet is that PC tries to swap memory between RAM and swap disk on ssd and constantly shuffling data here an there causes that. Probably buying few sticks of RAM would fix that but with current RAM prices… Well…

Another weird issue is that from time to time my main monitor 49″ Samsung stops responding to GPU updates. I can still use my PC with 2 other monitors so it is not frozen completely. It still works just main monitor no longer refreshes. It is funny that it is really easy to fix. I need to turn the monitor off and wait for the OS to register that it is gone. Gnome shuffles stuff on desktop and 2 other monitors get new windows. After that I just turn monitor on and everything is back to normal. Even all the windows are back in their place. Magic!

I have no idea why this is happening but it was already happening few times before I was using GPU for running models so it may be issue with the card, monitor or the drivers. It is just more apparent with higher load.

I have no idea what is the cause.

A bit annoying

But biggest disadvantage of such solution is that usually it is the most powerful PC you have and by the extent it is very power hungry. My PC with all external devices, monitors and etc needs much. I did not metered it but I estimate about 500W/h at the least. To not have it running nonstop you power it off. Constant sound of fans is also annoying. So again you power it off.

When you power it off you can’t use it. Of course you can juggle the power switches a bit to power off everything you do not need for the night. But it is very annoying after few days. The simplest solution to save energy and do not hear the fans or see faint glow is to power it via one general switch to everything but then you can’t use it.

The simplest solution to this is to have your server for self hosted services running somewhere you can’t hear or see and put it on some device that is not that power hungry. I did it already with my main server running my self hosted services for file sharing, media sharing, backups, messaging, network control, private DNS, smarthome devices control, authentication, git and etc. I have server like that running in a living room (where it was noisy and too loosely accessible to children), then in an attic where it was fine, but a bit too hot and in the basement at last. Now it have almost static temperature whole year and it is basically not existent to children.

Since this is good setup I wanted the same for my LLM setup. Few months ago I bought new server based on AMD Threadrippper PRO. It have 4 free PCIE slots that I intended for GPUs. But it is still very pricey setup and it easy to setup. Also it takes a lot of power. It mostly it is very pricey. Running for example four R9700 cards would cost me about 6000$ and would most probably require additional cost for some pcie extenders and additional PSU. And it still would be around of only 192GB of VRAM. Biggest models require multiple 200GB cards. So with the motherboard, case, RAM, disks, PSU cards and etc. Probably whole beefy server like that would cost me more than 10k$. And it still would not be able to run biggest models.

This is really something you can’t just do in your homelab. There is no way to actually compete with datacenters in your home.

You can also for example run big models on your CPU using ram. But it is slow. Very slow. Unless you are running some agents and speed does not matter that much it is just some fun toy. And I wanted something usable. Like an assistant that I can write or talk too and get response to question or a task in few to 20 seconds. That is bearable.

For that you have to have separate device built specifically for LLMs running in some closet at your home.

Your daily is not the best idea to run those things 24/7.

Posted on

Connecting Nanobot to Matrix

I started digging into Nanobot code to check why it cannot connect to Matrix server despite the config being correct and nanobot gateway being executed. Just in case I messed up something I did recreate entire nanobot workspace by doing nanobot onboard after removing entire directory. Did not work.

Closer inspection of the code shed some light on the problem: the config was not being used because channel was never instantiated. That part of the code was gone for some reason despite the Readme stating it is possible to integrate nanobot with Matrix.

I started to change some code to make it work. It did not seemed to complex to fix. I got it almost working but I am not great with Python, so I had some problems with tests not being green on my branch. Seemed a bit strange that I broke tests not connected to my own changes but I was unfamiliar to the repository and last time I was working with Python was around 2022 so… Who knows? Maybe I did broke it.

But then I thought ‘Hmm this seems like an obvious problem and easy fix! Maybe someone already did that!’. And it actually was true. There was PR open for this.

I was happy to run this branch so I setup virtual environment with uv that I still have somewhere around after I was playing with vllm setup. I downloaded this branch, built nanobot and rerun the gateway. Now it was able to connect to Matrix server though it still was having some issues with sending encrypted messages and had to play a bit with Matrix API to get access token for the bot.

Also I think it is a bit annoying that it reponds to all the messages in the channel. I.e. it would make sense to create root for bigger audience and to share some ideas or discuss something and having a not responding to everything would be very annoying and disruptive. It would make more sense if message would be only considered to be a propmpt when it would be mention @nanobot.

But for now I am happy with ability to chat with my new bot in my own Matrix channel and ask him to do some stuff for me.

Posted on

Adding the bot to Matrix server

I have my own matrix server for my own use. Usually it is used by me a bit by my family and most of it by my own services to sent me some notifications.

Lately I started playing with nanobot. It is personal AI assistant like OpenClaw. I wanted to be able to chat with it in my own Matrix via my own phone. To do that I needed to create new user dedicated

I find it a bit confusing that there is no central web UI that allow you to do that. Ok, since privacy and security is their utmost concern maybe there did it like that so that you have to have direct control of the server that is running it. OK that is one way to do it securely but also it is a bit obscure. You have to remember where it is, how docker container is named (there are several) and remember exact command you have for run, with exact name of parameters. Since it is very rare occurrence having a need to do that (it is not like I am constantly changing users), I have a hard time to remember that.

I created snippet of bash script that need to be run in order to do that. I have matrix running on docker compose with separate directory for all the data.

# navigate to directory
cd /opt/matrix
sudo docker compose exec matrix-synapse /bin/bash
register_new_matrix_user -u newusername -p very-secure-password1 -c /data/homeserver.yaml

This creates the user. In order to connect to server as new user you still need to login. In theory it should be possible to do that via web client and extract device id and access token from the client itself. It is possible and might actually work. But it is also possible to do that via API and it is much better since you can easily regenrate that data. And you will probably need that since token that is in use will be active but if user will not be active via that token for some time it will be invalidated. And then using client will be much more inconvenient then just API call via curl for example.

Here is another bash script that retrieves access token via API:

curl -XPOST -d '{"type":"m.login.password", "user":"newusername", "password":"very-secure-password1"}' "https://matrix.domain/_matrix/client/r0/login"

Of course user name and its password need to be the same as in previous script.

This will return JSON similar to:

{
    "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd",
    "home_server": "localhost",
    "user_id": "@matrix.domain:newusername"
}

That is all. Though I am not sure how device id need to be retrieved/regenerated without some client. Or even if it need to be communicated to the server at all prior to login. Anyway one time login via client and retrieving device id from the client it is enough. I won’t change and access token can be changed via running a script again fairly easily.