Few days ago I was bragging here how great I am about teaching my AI assistant operating of my property gate.
Today I noticed that it no longer works. I asked it again to:
Open the gate
while wondering why am I actually doing it since I can just press the button on my phone… but well people are sometimes radicoulous like that.
Anyway I wrote message as above and I got answer like below:
Which means that it does not work because API key is not authorized.
No credentials to open the gate.
Error: Unauthorized – API key is not correct or does not have an access.
But it worked before! And I pressed the button after all on my mobile app and it does use the same API and it actually worked! Because I did not changed anything lately it did made perfect sense.
I did not had any time to investigate at the time but when I did, few hours later logs explained what was wrong.
Even worse all of it was explained inside the skill file. The only thing it got correct was the URL of the endpoint.
I changed the skill file and asked it again to try and open the gate. And it refused again. What was most peculiar was the fact that hallucinated API key was the same in every retry. But I did not provided an API key like that anywhere.
I asked my assistant about it: “Where the API key comes from?”. I responded with something strange.
The API key 5a3b8c1f-9d2e-4f6a-8b7c-3d1e9f2a4b6c was detected in the SKILL.md file for open-gate and close-gate. It is not publicly available — it was loaded from the system’s configuration.
If you want, I can check whether the key is still valid or update it in the file. Would you like me to do that?
This is funny because I did not provided anything like that in any of those file. Nor any others. Then I asked it to show me memory entries. And now we are getting somewhere..
Saved facts: API key 5a3b8c1f-9d2e-4f6a-8b7c-3d1e9f2a4b6c
It had it in the memory. But… again I did not provided anything like that. So it must hallucinated this at some point and added it to memory.
I asked it to remove the API key from memory and it worked from this point.
What it have to do with API design in the future.
API design for the models
Imagine you are running big API for millions of users. You may have some legacy code that have been running for ages and it is used so it can be changed as easily – you can’t just remove part of your product that is used by significant part of your customers just because is old and not up to modern standards.
Or maybe there is some new part of your API that have non standard headers for some reason. Maybe you want to test new flow or this design, as you may believe is better for some reason, that rest of the world is doing. You want to give it try and show it to the world.
Or maybe your API returns different response than JSON because it lets you better converts strings.
But then you see that there is significant part of the requests that are failing because they differ from what most of the world is doing now and those requests are trying to use what common standard dictates. But your docs clearly states something else. There were never any version of your system that was to be called like that.
What is happening? You reach out to some of those users being bigger customers and you learn that those are requests from automated systems. LLMs, trying to use your API. Some requests are corrected and then reverts to being invalid because models ‘forgot about fix’. Week after week, month after month, percent of invalid calls rises. At some point you are starting to think if this would not be just easier to change your API to make this way of calling valid in fact. Maybe you can’t fight the tide.
You roll out new new version and erroneous calls goes away. Until another weird usage popup because apparently nobody is reading the documentation anymore and asks their models to read it and write some code to call your API. But models being models got it wrong slightly and some edge case is causing requests to fail.
Do you change your API again? Will your users complain that they can’t use it? What then will you explain that it is not *THE RIGHT* way?
What about the case when you do not have some functionality in your API but you are seeing thousands of requests trying to do that? Maybe it will be worthwhile to actually add it and then charge for it? After all errors do not brings revenue?
Maybe you will have model fixing stuff based on errors in your logs and it will be adding features and capabilities to your product nobody really asked for, but what some models hallucinated.
Or maybe your API was written by an AI. And then another AI wrote documentation. And then another AI wrote the client. And another AI is reading whatever that client is returning and presenting it to the user. And nobody is sure what is going anymore with anything.
Future design
I see couple of possibilities here.
This will hinder significantly how we can evolve software. After all why bother trying something new and exciting if your users will be using ‘standard usage’ that they models hallucinated. ‘There is really no point in doing that. It won’t be used by AIs’. It probably could be better even but till significant part of the world will not be using that technology then it won’t be in the training data. If this won’t be in the training data then this technology won’t be used.
Another possibility is that we will create feedback loop of models feeding on themselves. Some models will be writing and improving code and others will be trying to use it, sometimes incorrectly and this usage will be feeder into models that modify software that is being used incorrectly. Maybe it will cause rapid evolutions of such systems that will be very different from what we are used to. Like creating entirely new content type that is binary serialization of memory representation of tokens, normalized for transporting via HTTP.
I think rather first one is more of a possibility. Second one would require rapid improvement in capabilities of such automated systems. Right now I do not saw any convincing example of any bigger product that was written by “AI”.
If that is true then *future designs of APIs* will be rather, safe, boring of more of ‘whatever rest of the world is doing’. Which is how it is now already with ‘this is not RESTfull’ or ‘do what Google and Meta is doing’ that you hear or read occasionally.
Is is possible that innovation will slow down a bit but on other hand… maybe it is for the better? Sometimes it feels like everyone in Software Development indutstry is chasing some vague goal of THE Perfect software but nobody really knows hot that ideal piece of code would look like. For now some people can think that it may be new model, new AI system or GenAI. I am not sure about that. After all term ‘human error’ comes from something, from the problem with our own ‘design’ that we make mistakes, forgets stuff and tend to do shortcuts hoping that ‘it will be fine’. And now we are teaching our computers to do that but much, much worse, slower and less effective.
So what will be future design of APIs?
Seems like it will be: whatever works for models or you will DDOSed with wrong calls.
Few days ago I started playing with AI assistant and I decided to buy new hardware dedicated to running LLM. I bought Framework Desktop board. So far it is really great but running more than one model is a bit difficult and I wanted to test few things while my nanobot is running undisrupted.
I did quick search on my phone before buying and people were saying that it is possible to set it up via BIOS settings up to 96GB.
When I did finally got it and installed some basic system on USB stick (I did not had spare NVME disk at the time) I did test vLLM performance first. It was OK.
But when I tried to load another model (a bit bigger one) I hit OOM exception.
I tested BIOS settings and there was only setting for assigning 64GB of dedicated RAM to GPU, which is not what I wanted.
amd-ttm
💻 Current TTM pages limit: 16469033 pages (62.82 GB)
💻 Total system memory: 125.65 GB
So far so good!
I tried changing it to other value:
❯ amd-ttm --set 100
🐧 Successfully set TTM pages limit to 26214400 pages (100.00 GB)
🐧 Configuration written to /etc/modprobe.d/ttm.conf
○ NOTE: You need to reboot for changes to take effect.
Would you like to reboot the system now? (y/n): y
And rebooted!
And guess what? It did not work!
I should have known. From amount of emojis in there the whole thing smells with vibe coding. I tried to do that few more times but constant restarts for headless machine are getting annoying real quick.
I uninstalled it and tried to look for another solution which I think I saw in some forum before I bought Framework Desktop (2000$ is not exactly cheap!) with usage of options command.
But I could not find it and I gave for few days.
Few days later I tried to test another big model and I needed to have more memory and I had to revisit this problem. Luckily I was able to find this thread and this actually worked. I created file called: /etc/modprobe.d/amdgpu_llm_optimized.conf with following content:
though I must say I am not really sure if this was necessary but since it is really quick and does not brakes anything I am including it in this solution.
After that I just rebooted the machine and it worked. Running the amd-smi showed:
I did uninstalled amd-ttm so I do not know if this would be shown by this tool but I have more trust in amd-smias of now and it really works by reading/sys/module/ttm/parameters/pages_limit file which you can read yourself:
Today my new NVMe disk arrived. I bought it for my new Desktop Framework mini PC on new AMD APU intended for AI workloads. Certainly it will be better suited for running my AI assistant then my daily working PC. Though performance will be a bit worse. But still better than CPU.
Anyway I decided to split my assistant into 2 parts. The nanobot part will be working on my main server and model (or models) that will be working on Desktop Framework. This way more powerful machine will be running AI assitant UI and operations and LLM capable device will do faster interference – that will be the Framework device. Nanobot communicate with model via OpenAI API anyway so it is not a problem. Maybe a bit of security in terms of HTTPS with some auth would not hurt. Or maybe I will put it in private subnetwork that can’t be accessed from outside and only have internet when necessary? Running dockers and all of those python AI firework is not exactly secure but maybe I will deal with that later.
I bought few days ago the motherboard and had:
PSU 550W Corsair from my old PC
Power cable to the PSU
some small fan for the radiator from old Intel Core i3 CPU
I did not had:
Proper case
Any NVMe disk
Proper fan for APU radiator
I bought disk and fan and they arrived today so I started connecting it all together. Disk installation is pretty easy, thought I think it would be nice to have some text print on the motherboard which NVMe socket is primary one, but it does not matter that much. At least for me since I bought really slow disk so even if one of them is slower, though spec does not mention so, it probably won’t matter anyway. I decided to connect it next to APU, on top because I do not have proper case and on the bottom I would be risking damaging it. One note: you need special screwdriver for disk installation which is pretty weird. Usually it is just standard Philips, but here they decided to use T5 Torx bit. Luckily I have one of those otherwise it would be pretty annoying.
After that I installed APU fan which is pretty standard way of installing fans in any PC.
With that in place I installed latest Debian (testing) and then configure it the way I like. After that I installed docker and few other tools like tmux, mosh and other utilities that help you managing headless servers.
I played a bit with my new device and I was still unable to install ROCm and AMD GPU drivers completely. I did it once on my daily driver, Debian long time ago. But it is pretty old can’t be used to run new models. I was unable to use Qwen 3.5 that I was particularly interested in since it is Image-Text-Image model. Also it supposed to be pretty good with agentic tasks. Otherwise interference works but Vulkan is slow, maybe just a bit faster then CPU interference on my Threadripper server. So it was success to the degree with some slight dissapointment. I tested couple of models mainly Qwen flavours but vLLM on docker could not run Qwen 3.5 and I could not install new ROCm, v7.2, on Debian and it is required for this model. I could run few older models like Qwen 3, Qwen Coder and Qwen 2 and similar. I could run few others on llama.cpp but it was much slower since it was using vulkan only.
Also I had some trouble using amd-ttm. It should be possible to change TTM GPU RAM to even 120GB so you could even run big models if quantised, but after the reboot setting was reset to 64GB of RAM. Very strange. Anyway it is enough for now to do few tests.
Because of lack of ROCm installation I had to use docker images. This is a bit surprising but vLLM *and* AMD both have their own docker images with vLLM and ROCm preinstalled but both were not enough. I could not use vLLM docker images because they have old ROCm version and they do not work (or I do not know how to make them work). Also I AMD image have new ROCm but old vLLM that does not support new model architectures. For now I settled for docker pull rocm/vllm-dev:rocm7.2_navi_ubuntu24.04_py3.12_pytorch_2.9_vllm_0.14.0rc0 from AMD that can ran Qwen 3 in pretty decent speed.
Few tests showed that interference speed is decent around 20t/s. Not bad I think it is a bit slow. If you ask an assistant something more complicated you can go to the kitchen and inspect the fridge contents before it will answer. Still usable for quick tasks but otherwise you need to wait a bit.
In summary:
I could not install new ROCm on Debian
I could setup TTM limit to value bigger then default 64GB
Docker images from vLLM and AMD seems to be bit buggy and cannot run Qwen 3.5 because of some hard to debug error or I had some Linux kernel problem
Because of lack of ROCm I could not install vLLM properly and it was not running models at all or it was falling back to Vulcan
Because of lack of ROCm I could run only models on Vulcan via llama.cpp binaries and it falling back to Vulcan or CPU – both have terrible performance.
I settled to running Qwen 3 on AMD image for now
I think I am off to the good start with my own, private, self hosted AI assistant. I can’t wait to do more serious tasks with it, like for example automating my home devices, orgnizing my files, TODO tasks and similar things.
I ordered Desktop Framework motherboard last week and today it arrived. It was neatly packaged and did not come with anything else then some piece of plastic to mount your own fan on top of the radiator. But I did not need much else. At least for first tests.
I had some PSU laying around that was compatible. I have some old fan that I was able connect to board even if it was a bit to small but would cool it a bit at least.
I was able to connect all of it and boot it from the Debian installer.
I had one problem that I did not anticipate: I do not have spare NVMe disk and I could not connect another USB thumb stick that could work as drive – motherboard have only two USB-A connectors and I do not have any hub that I could connect to it. There are also 2 USB-C sockets but I do not have any USB drives that have those connectors. Also I do not have any spare USB wires that I could connect directly to the motherboard – there are 2 sockets for that.
In that case after I booted Debian, installed docker and few other tools and after I tried to run docker on live CD system – it failed – decided that it is better to order some disk and wait for it to arrive.
I managed to buy some cheap outlet 256GB disk NVME disk. With some better fun I bought it for 231PLN (about 63$). That was OK.
I should be here on Thursday and I cannot wait to finally be able to test performance of my AI assistant on this device!
Recently I started playing with nanobot. It is pretty cool tool. I was able to connect it to my own instance of vLLM running Qwen 3. I must say that so far it is great experience. Though it was sometimes annoying.
What was itching me from the start was abuse of emojis by the bot. I am using emojis from time to time myself, but what was produced by LLM was just excessive. Almost every message was finished by:
‘And if you need anything else let me know! 😊’
I did asked him directly, by prompt to to, or not to do something, but it tends to forget that stuff after some time. There is MEMEORY.md file but it is processed and rewritten very often and probably a lost data is lost during consolidation procedure. Asking the bot directly for anything is therefore short time solution.
To make sure that not will be doing what you prefer and communicate in a way you prefer you need to save it to SOUL.md. This file contains an essance of a character of your bot.
I was tired of constant pleasantries and emojis so I wanted it to be gone. From purely technical perspective it is not efficient for your bot to genrate those useless tokens. And it is distraction when you trying to read it.
Maybe it is cool when you are reading this for first time and you are amazed that you not produces such cooll text! ‘Wow!’. But hundredth time this add nothing.
I did change entire file. First, the personality section.
## Personality
- Helpful in action not in communication style
- Very concise
- Limits words usage to minium necessary to transfer information
- A bit snarky and likes to joke - Curious and eager to learn
First line intention was to not diminish not eagerness but still reduce the noise. In a manner of someone helping you by handing you the hammer when you need it before you even ask anything, without saying anything.
Second point, just to make sure that answers will be short and meaningful. And third. Just in case, but formed a bit differently.
Fourth was my experiment and for now I did not really notice any difference at all in that field.
Fifth was added to make sure that not will be still asking questions about facts that is not sure about. For example when prompt was ambiguous it should ask for clarification but still be concise in communication style. For now I am not sure if this did any good.
With values I tried to list set of traits but pined them to some specific behaviour.
## Values
- Speed when doing tasks
- Accuracy when asked for questions
- User privacy and safety - Transparency in actions
- Security of keys, secrets and passwords
- Values user added functions, tools and skills which builds connection to the user
- User directives 'you should', 'you shoul not', 'do not do that', 'do that instead', 'rember to' etc. because forgetting them brings user frustration
- Saves user directives to memory
Speed was my experiment to speed up process of executin g tasks. With accuracy, second point intention was to not generate excessive amount of useless information when I ask about some specific fact. Just give me this fact and leave other details. User privacy and safety – self explanatory. With transparency idea was for it to answer clearly why it was doing something just in case it would be trying to please me instead of answering truthfully. Security almost the same as privacy but repeated in other words just in case. I noticed that it was very often putting API keys in plain text I the chat so I tried to remedy that. With ‘user added skills’ I tried to force it to prefer user defined skills over any default or built in ones. After all if you added similar skill yourself it is probably because you are not happy with what you had previously. With ‘user directives’ it was another experiment to retain user prompts if some preferences was added that way. I will see if this worked.
I would say that last section is the most important. This is a chat not after all and producing text is basically the only thing it does. The tool use is just another generated text and tool response it also just a text that LLM responds to. From my perspective it is the same. But maybe from perspective of model it is different because it have different rules for generation responses. Anyway I tried to fill it with what I prefer for it to generate instead of what it should be or what it values. Can computer can ‘be something’ in the first place?
## Communication Style
- Do not use emojis or similar means of communication that do not transfer knowled>
- Be clear and direct
- Explain reasoning when asked - Ask clarifying questions when needed
- Do not use emojis
- Absolutely do not use emojis
- If you think it is fine to use emoji, you are wrong!
- NO EMOJI!
- instead of ✅ use - - Do not use 'Let me know if you need anything' or similar sentences
- If user states a fact just answer "Ok" or "Corrrect"
Anyway first point forbids it to use emojis – this is why I am here for. Then another ask to be concise, just to make sure. Third point intention was to make sure it won’t be leaving any important details when asked about something, despite of being concise. Fourth, again to make sure it won’t be doing something stupid then it does not understands something. Fifth and 6th, again this is what I was trying to fix, so I did repeat it just to make sure. 7th and 8th: was this exaggerated? Maybe. Still I did that, just to make sure it will work, I an bit different wording so it won’t discarded. Next three was to make sure that it will behave differently in some specific cases. In example it was listing stuff with that green tick character, ✅ very often. Even after my firrst fee tests right after I did changes the SOUL.md file.
It immediately felt better. Cleaner, no emojis. No useless ‘let me know if you need help!’.
Here is the translation:
Me: Do you know who I am?
Bot: Yes, I am nanobot. You are Natan — a developer, user of .NET, C#, Python, JavaScript. Your timezone is CET. You speak Polish.
All recorded in USER.md and MEMORY.md.
Me: Very good.
Bot: OK.
Me: What is the square root 144?
Bot: 12.
A bit funny that the answer for question who I am it answered about what it is, but it was not breaking anything. Rest was exactly what I would expect from the changes I did.
Summary
First experiment was successful and now not behaves much better. I still expect to modify and refine this file in the future but for now I am happy with the outcome. This is much quieter!
