Anyone else running local LLM? What do you think?

[2009Caraman]

Ive been messing with Ollama and some models. Mostly Gemma3. Its pretty neat stuff. I dont know if it provides anything more than a bit of research can provide but it can answer questions quickly. I dont code so I dont have use for them in that case.

Im a bit of a privacy nut so really have no use for online or intgrated AI. I run it at home and access it through tailscale if I want to use it.

If anyone here knows a good front-end ios app or something to link back to ollama I'm all ears.

 

[chorty]

how much storage does such a model require?

 

[LDM]

I setup one a while back but didn't really get into playing with it much. Just haven't had the time. I threw it together from an old couple of decommed servers we were throwing out from work. Its a a little old but I managed to get 128GB of RAM into it and a few TB of hard drive space as well. Couple of old enterprise level graphics cards too. It works but again, haven't played with it enough to really get anything interesting out of it. Just been way to busy with work and life stuff lately. I used Ollama 3.2 for my setup, currently only have it setup for text only mode.

 

[jcy]

reddit: /r/LocalLLaMA

 

[2009Caraman]

how much storage does such a model require?

Anywhere from 5GB to hundreds of gigs.

 

[2009Caraman]

I setup one a while back but didn't really get into playing with it much. Just haven't had the time. I threw it together from an old couple of decommed servers we were throwing out from work. Its a a little old but I managed to get 128GB of RAM into it and a few TB of hard drive space as well. Couple of old enterprise level graphics cards too. It works but again, haven't played with it enough to really get anything interesting out of it. Just been way to busy with work and life stuff lately. I used Ollama 3.2 for my setup, currently only have it setup for text only mode.

Yeah I am text only mode. i have a 9070xt and was trying to get StableDiffusion running with AMD but ran into some issues. I tried AMD supported pic generation models. They work okay but lots of room for improvement.

 

[chorty]

That's not bad.

I just setup a 80TB zfs array. Looking to play around then nuke it when done.

 

[Brons2]

I wouldn't mind running a security focused LLM and letting it chew on syslog from my (home) network. Don't know if anything exists like that. Haven't really looked into it much.

I could feed it external blacklists and such and tell it to look for hits.

 

[andrew_j]

I tried on my work computer but is 24 GB+ and MacBook Pro m4.