Ollaya: An Open-Source Ollama-Style Runner for Local Decision Models

Ollaya is a new open-source tool that lets you send typed questions — as plain text or JSON — to a decision model running entirely on your own hardware, and get a calibrated answer back. It borrows the local-first, developer-friendly packaging that made Ollama popular for LLMs, but applies it to a different kind of model.

What Ollaya actually does

Ollaya is built to run what the project describes as 'Jev-style' decision models locally. You send a typed question — text or structured JSON — and the tool returns a calibrated answer computed on your own GPU, with no cloud round-trip. The name is clearly a nod to Ollama, the tool that popularized one-command local LLM serving, but the target workload is different: decisions rather than free-form generation.

Why the architecture is different from LLM inference

Standard LLM inference is autoregressive: each output token depends on every token generated before it, so response time grows with answer length. Ollaya's decision models skip that entirely. Instead of generating tokens one at a time, they answer in a single forward pass — the model runs once per question and produces its output directly. That means compute cost per query stays roughly fixed, regardless of how 'long' an equivalent generative answer would have been.

The benchmark context

The source material references a benchmark run on an RTX 4090, processing a batch of five questions against a model called Laya through Ollaya's HTTP API. The exact completion time wasn't fully specified in the available material, so it shouldn't be quoted as a hard number here — but the setup itself is telling: it's testing whether a single consumer GPU can handle decision-style queries at a latency that's usable in a real request/response loop, not just as an offline batch job.

How this differs from just calling an LLM API

Many teams currently get structured decisions out of general-purpose LLMs by prompting them and parsing the text response — asking a chat model to output 'approve' or 'deny' with a JSON wrapper, for instance. That approach inherits all the latency and cost of full generative inference for what is, functionally, a single classification. Ollaya's pitch is narrower and more specialized: it's closer in spirit to serving a calibrated classifier as a local API, but wrapped in the same kind of frictionless developer experience Ollama brought to LLMs — pull a model, run it, hit an HTTP endpoint.

Who this is actually useful for

Teams building backend logic that needs fast, structured judgments — fraud flags, approval/denial decisions, content categorization, risk scoring — are the obvious audience, particularly where sending data to an external API isn't an option for privacy or compliance reasons (finance, healthcare, internal enterprise tooling). For those cases, a fixed-cost single-pass decision model running on local hardware can be both faster and cheaper than routing every request through a hosted LLM.

What's still unclear

How many pretrained decision models will be available in this ecosystem, how well calibration generalizes across domains, and how accurate these models are compared to purpose-built classifiers are all open questions not answered by the material available so far. As with any new open-source runner, adoption will likely track how quickly a usable model library grows around it.

Takeaway

Ollaya doesn't replace LLMs — it targets a workload LLMs are arguably overkill for. If the project gains traction, it could become a lightweight local alternative for structured decision-making tasks that currently get routed through generative APIs simply because that's what's convenient.

Reference: https://ollaya.dev/