Running AI assistant on Threadripper PRO
#Threadripper #vLLM #llama.cpp #Qwen
The answer is: don’t do that to yourself.
Recently I started playing with AI assistant using nanobot. It is good but so far I am running it on my daily working PC. And it is not great experience so far . So I am exploring some other way that will allow me to do that, especially when I am not at home but still want to use my little helper to do stuff. Running big PC in your office next to your bedroom is not the best idea. And electricity bill would also not be that great.