I am blogging last few weeks about my new aventure with nanobot. I am running model for that little assistant of mine on Desktop Framework motherboard I bought couple of days ago.
Few days ago I was working on my own fork of nanobot. I was not happy how few things are not entirely correctly implemented in this project. For example command support in Matrix channel. I created PR for that one here. Unfortunately it was not merged yet. I am not sure why. Maybe it is mising few unit tests.
Yesterday my my another PR was merged though! I created change request that added kind of streaming support to nanobot Matrix channel. It is not entirely streaming, it is series of edits to original message that was sent from first set of tokens to nanobot from LLM.
After the first set the message is created and after next tokens are generated nanobot sends another matrix event that changes original one. This is not really edit either as Matrix does not edit messages. It is just another event that is updating previous event data. Actually when I was testing that with small model, Qwen 3.5 0.6B, which is often generating repeating or incorect messages, but it does it very fast. On one accassion it did generated a lot of tokens which generates a lot of edits. I think it was about 90 lines edited few times each. Which caused entire message to be edited probably about 1000 times. This actually caused my mobile client to hang and stop getting updates altogether. It was just to much for it to handle.
I had to remove this message entirely 🙂
So it is not ideal but I still think it is beneficial because it is better to have quicker response and read through updates then wait for entire LLM response to appear in Element.
I think it is great success to be part of new and interesting project. And it is great to be able to benefit from this project by running my self hosted AI assistant.