For the longest time, I’ve been trying to figure out a way to “survive” in this new AI age without having to fork over a ton of money just to keep up. I’ve tried using local models via Ollama, and while they definitely work to a degree, they’re (unsurprisingly) not as good as the big model providers.
The local models tend to
- Forget what they’re doing
- Struggle to break larger tasks into smaller ones
- Lose focus easily
- Have weaker coding performance
- Drift over longer sessions
So to improve the reliability of fully local, smaller models (and to keep all my data local and in my own network), I created Loki.
It’s a local-first, batteries-included command line tool and runtime for building and running LLM workflows locally. It’s model agnostic and supports things like
- Agents and agent delegation
- Roles/personas
- MCP Servers
- RAG
- Custom tools
- Macros
- Workflow Scripting
A lot of the features it supports are specifically designed to compensate for weaknesses in smaller local models. For example:
- Auto continuation to keep pushing models to completion instead of stopping halfway through problems
- Parallel agent delegation so tasks can be split into smaller, focused scopes
- Workflow-based execution (“If this, do that”) for building more reliable and repeatable automations
It also supports the major cloud providers if you want them (which definitely helped while testing 😄), but my long-term goal is simple:
Get as close as possible to Claude Code-style reliability using fully local models.
I’m always open to feedback, questions, or ideas.
I got a project brewing myself, looks good!
Looks sick, gonna check this out
The demo would be a lot more impressive if the questions you were asking weren’t longer than the extremely simple SQL queries it generates.
Tried building something like this myself and it was surprisingly complicated. Sounds exactly what I needed!
Naming an AI tool after a god of mischief seems like tempting the fates. 😆
Seems like a great idea to me…
I can respect the self-awareness though.
Can it be used with horses?
Looks cool! I really like some aspects of LLMs but cloud hosting always turns me off.
You might want to rethink the name though as Loki already is a well known log server: https://github.com/grafana/loki
Yeah… 😅 I originally named it Loki because, well…if you leave LLMs unsupervised they just create mischief. Any ideas of a good rename? I’ve gotten this comment before and I just couldn’t think of anything good.
Weapons like Gungnir are popular names, Lævateinn is a little hard to spell though.
Maybe Coyote? Coyote is the trickster spirit in a lot of Native American mythologies.
Do you have a GitHub and would you be willing to share with me so I could credit you with the name? No worries if not, I can just link to your Lemmy profile instead of you prefer. I just don’t want to change it without giving credit.
Sure! My GitHub is github.com/erichury/ I’m just barely getting into software development, so It’d be nice to have some love on my GitHub page to show off :).
Ooh I like Coyote! That’s definitely in the running now. Not to mention that’s really a really cool allusion to Native American mythology!
Or Coyode (mixture between code and coyote, could be written co[yo]de for extra yo). Only has 4 duckduckgo results so easily searchable and distinguishable.
ChatGPT generated logo example
![co[yo]de logo: coyote with a hoodie and the aforementioned spelling](https://feddit.org/pictrs/image/2287d9db-d8b8-4193-83d0-2dbe7de66c50.png)
After sitting with Coyote for a while, I’m really liking the name. Before I get too attached, any other ideas? (Just to make sure I stay objective 😛)
“Frank”, it I don’t think that name is in the same ballpark as you are looking for.
Stop overthinking it and use it. It seems like the consensus is in the approval of your choice.
Works for me. I’ll refactor that and rename it tomorrow and hopefully have a new minor release sometime this week. It’ll be another baking change release so I’ll need to attach a couple commands to the release notes to make it easy to migrate.
Not to mention road-runners (humans) and ACME (OpenAi, Anthropic etc.) extending the metaphor in a different direction… Wile.E. (coyote, suupergenius) might be another name option.
Ahh, ninja’d, I’ll leave it as another vote.
Hermes is also the trickster god in Greek mythology, but not sure if the makers of that project were thinking of that, or his role as messenger. Or the one who guides souls to Hades. Dude’s got a lot of jobs…
I feel like that well describes a border collie.
Wants to do stuff, but if you don’t attend they’ll find stuff to do.
Could call it Sylvie as well (the Marvel gender bent Loki)
Sounds like some shepherd reference could be good, since it’s herding the agents.
I’m curious how well this would work with speckit
I’m a little confused about this thing’s use case.
What does it do differently/better than OpenCode ?
OpenCode is specific to coding workflows. Loki is built to be a general LLM runtine/workflow engine for any problem domain, not just code. An example use I have for it is a cron job that runs at boot to
- See if the cause of the reboot was power loss (LLM)
- If it was, check all services to ensure they’re up and running (tool)
- If a service isn’t up, then use an LLM to see what happened (LLM)
- Try out the usual methods for getting that service started (tool + RAG)
- If none of those work, try figuring out what’s ultimately wrong (LLM)
- Send me a ntfy notification on my phone to let me know what service isn’t running, and the suspected cause with some context (tool)
Opencode isn’t very fun to set up with local LLMs and I’ve had issues with tool calling, but it’s very doable! That said, OpenCode is my go-to, absolutely love it compared to all alternatives I’ve tried
Haven’t configured much beyond this but what’s wrong with
ollama launch opencode <model>? Haven’t had an issue yet.yo… what? i was configuring json files n shit with custom sources. granted i used lmstudio as ollama doesn’t support mlx models or something for mac. This is definitely the easy way if you’re using ollama.
Opencode needs like 10k tokens just to get started
Is that included in the token & cost counter? I haven’t really noticed that yet. It’s just the most reliable and best harness i’ve yet used. For context i’ve only otherwise tried claude, gemini, and aider. More if you count non cli apps
Yes of course. It’s all tools and skills and system stuff
But 10k isn’t much in the grand scope. But it can be a big hurdle if you want to use opencode with local small models
When it comes to writing code, OpenCode is my go-to as well. It’s my ultimate benchmark for how well optimized and reliable I can make local models function in Loki.
Ditto. I don’t see how this is different/better from existing harnesses such as Opencode, Pi, and even “commercial” open source offerings such as the CLIs for Codex, Copilot, and Gemini, especially once tricked out with plugins and extensions.
did you even try pi or opencode?
I mean, maybe they just wanted to build something?
Yeah, reinventing the wheel can be fun.
fair hit.
I like that you are so focused on local models but I can’t find any info on setting up local models in the clients setup https://github.com/Dark-Alex-17/loki/wiki/Clients
What am I missing?
Edit: well it seems this post is an entirely fictional origin story. Here is the first time OP posted about his project 6 months ago https://piefed.zip/c/rust/p/663115/loki-an-all-in-one-batteries-included-llm-cli
So actually, this was the original purpose of it. But all the help I tried to get on it didn’t really have much interest in doing anything outside of the usual big model providers, so I tried advertising a more general use case to attract more input. I can’t deny that agnostic support for even the big providers is helpful when you’re trying to stay current with the rapid advances in LLMs.
After that, I kind of gave up on getting feedback on local-first models. So, instead, I just dove in head-first the way I wanted;Trying new things, building new agents to try and rival Claude Code, adding features as I found them useful and necessary to improve that reliability, etc., and iterating. Then, with the most recent release on Friday, I had done so many changes and improvements specifically for local models that I thought I finally had a strong enough tool to maybe pique enough people’s interest to get some feedback and input. 🙂
Oh, and the config example shows how to add Ollama models here
Ollama is enshittifying at a rate of knots, have you got a way to use llama-server (or preferably llama-swap) instead ?
Crap. I was just starting to play with Ollama and thought it might be a good balance between running local models and using one of the proprietary services.
Could you elaborate on what’s happening with them / what to watch out for?
I suggest using unsloth studio to get a friendly GUI for not just downloading models and running inference but also finetuning and such. Underneath it just uses llama.cpp which is supported by a lot of apps but it also adds other APIs IIRC. You can run claude code, github codex, mistral vibe off either the llama.cpp API or the unsloth API depending on which agent you’re using and they’ve got tutorials for setting those up. Other tools too.
That’s not to say it’s the only one or the best one, but I really like the UI, because it’s both simple and advanced (if you look for it, you can set KV cache type, temperature, etc, but you can also run default settings without ever looking at the advanced stuff).
If it gets you started with local models, by all means go ahead, their onboarding is the easiest and it works. Also a lot of 3rd party stuff uses it as a first class citizen allowing you to try out other things (e.g. Open WebUI) easily as you explore what’s possible. Currently try the Qwen 3.6 and Gemma4 models as best bang for buck, somewhere there’s a does it fit in my machine website that can help (search for it).
That said, basically all roads in local LLM lead to llama.cpp, which gets the innovations first and then others copy their homework. Ollama (looks like they’re angling to go commercial) for a long time used it internally without attribution, now they use a bodged up engine of their own that is less performant and almost certainly a copy (possibly vibe coded) of llama.cpp. They heavily encourage using their own models / quantizations and don’t let you play with a lot of parameters without a lot of friction (possibly because they’re not implemented yet, but who knows, low transparency). You get the picture, wannabe techbros. That’s off the top of my head, search for more authoritative sources.
After you’ve gotten the hang of things, have a look at llama-swap which just wraps llama.cpp, lemonade if you’re on AMD, vLLM for nvidia, LM Studio for mac.
Looking at Llama-swap, since it says it supports OpenAI-compatible API, it should just work natively already. Just set up the client to be
type: openai-compatibleand fill in the URL and provide the models. Should work out of the box!Hope so, bet it doesn’t without some tweaking though, OpenAI-compatible seldom is, and ollama is bad for that. Still, worth checking out, I’ll have a go at it sometime soonish and perhaps you’ll see a PR (or some doco in the best case scenario).
Looking forward to it! Heads up in case you missed it: I had settled on renaming it to Coyote, so sometime this week will be a breaking change and release to get that done.
Biggest pains are just going to be updating the repo tokens for Crates.io and renaming the homebrew repo.
K, I’ll circle back in a week or so…
Just an fyi, Loki is also an extremely popular logging system by Grafana, might want a rename if you don’t want to deal with people not finding your project due to having a larger project named the same thing
This was my first thought
Loki, check my Loki logs!
Does it have built-in protections so it doesn’t randomly decide to delete every file it has permissions to?
Yes it does. By default, any of the execute_command or fs_write/fs_patch/etc. tools all have guards around them that prompt for user confirmation before doing things. They can be disabled via the
AUTO_APPROVEenvironment variable if necessary (like they are when using thesisyphusagent). For bash tools, I’ve included functions that can help do this when you write your own tools. For Python tools, you can use the usualinputmethods.As usual, leave it to the random developers on the internet to put more care and thought into something than the multibillion dollar companies.
The multibillion dollar companies do this too but people find the permission prompts annoying and use the full dumbass mode instead lol
Very cool idea! Going with the Coyote theme maybe name it Wile E?
Certified Genius: Have Brain, Will Travel
I’m confused. You say in post title you don’t want to send code to the cloud but the image you attached shows openai gpt4o. So what’s the deal?
It was just the one gif I had available and also the model that worked fast enough to fit into a gif without taking forever between prompts so I could demo Loki well. You make a good point though. It’s an old build and is slightly outdated. I’ll update that. Thanks for pointing that out.
What local model are you using?
I’m using a ton of different ones but the main ones I use daily are
gemma4:26bdeepseek-coderdeepseek-r1:32bdevstral:24bgranite-code:34bopenthinker:latestphi4:latestqwen3:30bmixtral:8x22b
I’m also going to use this opportunity to plug an amazing project to help figure out which models will work well on my hardware: https://github.com/AlexsJones/llmfit Is amazing!
Isn’t it a huge delay to swap out to a different ~30b model every few minutes depending on the use case?
Unfortunately, yes. It’s one reason I’m trying to figure out a good mechanism to maybe do something like multiple ollama hosts. So like: you can specify what model to use specifically in an agent. But if an agent delegates to a sub-agent, it unloads that model and loads the new one. I’m trying to figure out if there’s a way to “alternate” between multiple hosts (say, ollama running locally and one running on your server), so that when a switch happens, it does it on the secondary host while also looking ahead to see what needs to be switched, if anything, on the primary host.
It supports multiple Ollama hosts right now as-is so what I’ve honestly been doing for the time being is specify which model on which host each agent uses so there’s only loading of one model at the beginning of a session. Then there’s no unloading/loading/etc. The other thing I’ve been trying is to see how small I can get the models to be without losing performance. While the tricks implemented in Loki help dramatically, I know there’s still a lot more I can do to improve it further.
