Tobias Krug
All writing

Kev: open decision models that answer with probabilities

Kev is an Apache-2.0 family of small models that take typed questions and return probabilities instead of text. How it works, how to try it, what to check first.

Find6 min readFacts checked

Found: github.com/jaredpalmer/kev

Contents

A lot of what code asks a language model is classification, not writing. Is this ticket about billing? Which team should get it? How urgent is it? The usual approach is to prompt a chat model for JSON and parse whatever comes back. Kev takes a narrower route: you ask typed questions and get probabilities back, not text.

What it is

Kev is a family of open-weight decision models, licensed Apache-2.0 and built on Qwen3.5 and Qwen3.8. It copies the interface of Jev, a hosted model from TypeSafe that is in early access: a Kev server speaks TypeSafe's System One API. Each request is one POST /v1/systemone with a state (the text) and named questions of three types:

  • noul is a yes/no question and returns the probability of yes.
  • choice takes 1–255 options and returns a distribution plus a confidence.
  • score takes ordered levels and returns the mean level, counting from 0, plus a distribution over the levels and a confidence.

The README says TypeSafe's Python SDK works against a Kev server unchanged.

The repo was created on 17 September 2026 and reached about 9,000 stars in its first four weeks. Kev 1.0 followed on 1 October and pins four sizes (0.8B, 4B, 9B, 27B) together. Nothing in it was newly trained; it is the fixed baseline the next generation is measured against.

How it works

Each question sees the state and its own text, never the other questions. On an attention-only model you get that with a mask: pack the state and all the questions into one sequence, and stop each question from attending to the others. That trick fails here. Qwen3.5 and Qwen3.8 mix attention layers with Gated DeltaNet layers, which are recurrent and ignore attention masks. So each question runs as its own row, the state plus that one question, and the server computes the state once and reuses its cache for every row.

The prompt wraps each option in <opt>…</opt> and ends each question with a <decide> token. A small pointer head scores each option's </opt> hidden state against the question's <decide> hidden state; because <decide> comes last, it can attend to every option. A softmax with one fitted temperature per checkpoint turns the scores into probabilities. That temperature is what the README means by "calibrated".

The sizes differ:

  • Kev-0.8B, 4B and 9B are rank-16 LoRA adapters plus the pointer head on frozen Qwen3.5 base models. The 0.8B runs on any Apple Silicon Mac or a 4 GB GPU.
  • Kev-27B (v2) is a full fine-tune of Qwen's post-trained Qwen3.8-27B, averaged 0.85 to 0.15 with the earlier LoRA version. It ships as 51 GB of bf16 weights plus the head and serves in bf16 only, about 66 GB resident, so it needs an 80 GB card.

The design follows Jev's Architecture Unmasked, a write-up that infers Jev's architecture from 10,000 API calls and calls itself speculative. The README states that no Jev outputs were used for training.

Trying it

The Hugging Face Space lets you try Kev-4B and Kev-0.8B from the browser with nothing to install; it runs them on Hugging Face's GPUs. Locally you need Python 3.12 or 3.13 and uv. These are the README's commands, with the weights pinned to the Hub tag v1.0. The clone is main; add --branch kev-1.0 if you want the code that matches the release.

git clone https://github.com/jaredpalmer/kev.git && cd kev
uv sync --extra serve
uv run --extra serve python -m kev.serve --run jaredpalmer/kev-4b@v1.0 --port 8009

It runs on CUDA, ROCm, or MLX on Apple Silicon; for Kev-4B the README lists an L40S or H100, or a 32 GB Mac. Then ask it something:

curl -s localhost:8009/v1/systemone -H 'content-type: application/json' -d '{
  "state": "Shoes arrived two weeks late and in the wrong size. Also I see two charges on my card.",
  "model": "kev-latest",
  "questions": {
    "department": {"type": "choice", "instructions": "Which team should handle this?",
                   "criteria": {"returns": "Exchanges, refunds, wrong or damaged items",
                                "shipping": "Delivery status, delays, lost packages",
                                "billing": "Charges, invoices, payment problems"}}
  }}'

In the README's example response for this ticket, Kev-4B put 0.47 on returns, 0.28 on shipping and 0.25 on billing, and the README reads that as the ticket being about all three. But choice ends in a softmax, so the three teams have to share 1.0, and a spread like this can't tell "the model is unsure" from "this ticket needs three teams". When a ticket can belong to several teams, ask one noul question per team instead: the questions are isolated, so each can come back high. Keep choice for questions with exactly one right answer.

What I'd watch out for

  • pip install kev installs a different package. On PyPI that name belongs to an unrelated key-value ORM, last released in 2021; the README warns that Kev isn't published there. Remember that when an agent installs dependencies for you.
  • The local server has no auth by default. It binds to 127.0.0.1 and accepts requests without a key unless KEV_API_KEY is set. Start it with --host 0.0.0.0 and no key, and anyone who can reach the port can use it.
  • The quick deploy runs a script from main. The README's quick deploy to Modal downloads kev_serve.py from the default branch. It pins the Kev package to a commit, but the script itself is whatever main held at download time. Read it first, or fetch it from the Git tag kev-1.0. Then set a key and keep it: run export KEV_API_KEY=$(openssl rand -hex 24) before modal deploy kev_serve.py. The README's one-shot form sets the key for that one command only, and the script never prints it. Without a key, the docstring says, "the URL is the only secret".
  • Validated context is not accepted context. The server accepts states of up to 65,536 tokens, but Kev-0.8B, 4B and 9B are validated only to 8,192: at 16k the measurement can no longer rule out an accuracy drop of more than 3 points. Kev-27B holds to 65,536.
  • Known weak spots, from the release notes and README:
    • Kev-0.8B scores below chance on When2Call (0.133), so don't use it for tool routing.
    • Date arithmetic is the weakest task family at every size below 27B.
    • Kev-27B v2 is overconfident on long contracts (CUAD ECE 0.053, against 0.007 for v1). The release notes suggest refitting the temperature on your own documents, or using @v1-lora for contract review.
    • On 14 public datasets no Kev trained on (breadth-v1, scored with a chance-corrected index), Kev-27B is 1.7 points below Jev and the smaller sizes are 13 to 31 points below.
    • Changing the option order can change an answer, so fix the order in code and version it like the prompt. The server's /v1/systemone/permute endpoint reruns a choice question under several orders, so you can see how much yours move.

What I'd copy

The README says its numbers are checked in CI against the reports they come from. Most of them are. docs/claims.json maps each printed number to a committed JSON or JSONL report and the keys that hold it: 1,019 records, 150 of them for the README and almost all the rest for the model cards and release notes. A test in the Python CI job asserts that scripts/verify_claims.py finds no mismatch, and CI is green on commit 0a6d3fa (run 38441554314).

The script is candid about its limits: its docstring says the text check "is presence, not position". That limit shows. The only 0.47 in the README is in the example response above, and the claim record that matches it traces to something else: a 9B training run's coverage at a 5% error budget. Some README figures, such as the 0.52–0.69 automation rates at a 5% error budget and the fine-tuning gain from 67.7% to 73.6%, have no README claim record at all.

Precision is the other limit. The script averages with plain sum(), and one record checks a printed 0.67 against the mean of four accuracies from a 4B evaluation report: 0.45, 0.6, 0.95 and 0.7. In decimal that mean is exactly 0.675, but the exact mean of those four floats sits just below it. Since Python 3.12, float sum() is more accurate: it gets 0.6749999999999999, which formats as 0.67. Python 3.9's plain summation lands on the float nearest 0.675, a hair above it, which formats as 0.68. So the same arithmetic on Python 3.9, which Kev doesn't support, fails that one claim, and Kev's requires-python = ">=3.12,<3.14" is doing real work. The same record's text check passes only because the 4B model card prints 0.67 only as the first digits of 0.677.

Locked test sets are read once per released model, the 0.8B, 4B and 9B release tarballs ship with SHA-256 checksums, and the release notes say where test margins are optimistic. One gap: the SDK compatibility tests need a running server, so CI doesn't run them. The claim that TypeSafe's SDK works unchanged rests on tests/test_api.py, which you run yourself against a local server.

Why I'm looking at it

OKF Vault Kit (alpha, v0.1.0) pairs a vault with a local 2B-class model that answers in text and can search and edit notes over MCP. Kev-0.8B only answers typed questions, with probabilities, so I'd like to try it next to the 2B for yes/no and pick-one questions about a note. Not for tool routing, though: its release notes rule that out.