Decision models in local: JEV-35B and the 3B-active System 1

One pass, no generation

There's a task that big LLMs do, and you don't need them to do it that way: decide. Classify a ticket, pick which team to route it to, score the risk, say yes or no. For that you don't need a model that writes sentences; you need one that answers a precise question with a probability per option, in a single pass, without decoding a single token.

That's a decision model. It doesn't generate text: it reads the state, reads the question, and returns a probability distribution over the options. It's the System 1 — fast, instinctive, calibrated — against the LLM's System 2, which reasons, writes, and takes its time.

The idea isn't new, but two October releases make it concrete in local. JEV-35B from Autotrust is a 35B-parameter MoE (3B active per token) that runs quantized in NVFP4 on a 32GB GPU. ej is the opposite: 11.4MB, a 128MB resident-memory peak, runs on CPU. Two extremes, same category: models that decide, not write.

What a decision model is (and isn't)

A generative LLM takes a prompt and produces a sequence of tokens, one by one. A decision model does one thing: given a state and typed questions, it returns a probability per option.

The typed questions come in three forms:

Type	Input	Output
choice	instructions + list of options	a probability per option
noul (yes/no)	a statement	P(false), P(true)
score	instructions + ordered levels	a distribution over levels

The advantage is double. First, speed and cost: no generation loop, a single forward pass. JEV-35B answers in a 204ms median on one GPU; ej in 214ms on a 4-core CPU. Second, no formatting to lose: the output is already the answer (the most probable option), not a text you have to extract it from.

It's not a chatbot. JEV-35B doesn't hold conversations, doesn't generate free text. If you need to write, this model isn't the one.

JEV-35B: the 32GB decision-maker

JEV-35B is a fine-tune of Qwen3.5-35B-A3B (MoE, 35B total, ~3B active per token), Apache-2.0. A 264-slot decision head gives one logit per slot; the active slots of the question are soft-maxed with a per-kind temperature.

The point that changes the hardware is the quantization. The vendor's numbers:

Version	Download	GPU memory (weights)	Decision Index 0.3
JEV-35B bf16	72GB	66.5 GiB	59.73
JEV-35B-NVFP4	25.6GB	23.3 GiB	59.18

Moving to NVFP4 takes the weights from 66.5 to 23.3 GiB: it fits on a 32GB GPU or larger, and the answers stay identical to bf16 in 96.6% of cases. The math is simple: 35B parameters at 16 bits weigh ~72GB; at 4 bits (NVFP4) you drop to a quarter of the bits per weight, and with overhead you stay under 25.6GB of download.

Where it shines is where it should: classification and routing. BANKING77 at 0.921, CLINC150+OOS at 0.943, intent recognition in general. Its strong area is Retrieval & Classification (skill 0.710); the weak one is Knowledge & Reasoning (GPQA 0.360). For that, you need a real LLM.

To try it:

hf download autotrust/JEV-35B-NVFP4 --local-dir JEV-35B-NVFP4
bash JEV-35B-NVFP4/serve.sh   # vLLM on :8000

If it doesn't work: it needs vLLM (a September 2026 development build) and a 32GB+ GPU. Below that, it doesn't fit.

ej: the pocket System 1

ej is the other end: 11.4MB (one file, model.ejpack), built on an e5-small-v2 text encoder. Resident-memory peak 128MB (35MB in low-memory mode); the Python process reaches ~558MB, but 204MB is just import torch. Latency 214ms per record on a 4-core Xeon, single thread.

It's not an LLM. The "teachers" are used only at fit time; at inference it runs alone, without generating tokens. Given a state and typed questions, it returns a distribution per question, calibrated (measured, not promised).

Its honest numbers: 0.832 on known intents (zs_massive), 0.712 on in-domain tickets. But on never-seen workflows it drops to 0.419. It's built for a workflow you know and measure on your own labelled cases, not zero-shot classification of new things.

Locally (CPU):

pip install -e .   # from the 5ak3t/ej repo

import ej
model = ej.load('5ak3t/ej', revision='v0.0.1')
(probs,) = model.predict([ej.EXAMPLE_RECORD])

If it doesn't work: it's Python + CPU only, and English-only. For other languages, it's not the right model.

When a decision model is enough (and when it isn't)

The selection criterion is this: does your question have finite options and a right label? If so, the decision model beats a generative LLM on three counts — cost, latency, and the fact that the output is already the answer.

	JEV-35B (NVFP4)	ej	Generative LLM (e.g. 8B)
Size	25.6GB	11.4MB	~5GB (Q4)
Hardware	32GB+ GPU	CPU, 128MB	GPU/CPU
Median latency	204ms	214ms	30–100ms/token
Generates text	no	no	yes
Strength	GPU classification	on-device routing	any open-ended task
Languages	English	English	depends on the model

Practical rules:

• Routing and triage on a server with a GPU (ticket → team, risk 0–5) → JEV-35B. Strong classification, calibrated probabilities, a single request.
• On-device triage, no GPU, no cloud (intent in an app, gating before escalation) → ej. 128MB, runs anywhere.
• The answer is open, or there's no right label → generative LLM. The decision model doesn't exist for that.
• High-stakes decisions about people (credit, hiring, health) → neither alone: you need human review and a confidence gate certified on your own data.

The limits, stated plainly

• English. Both are English-centric. For other-language content, there's no open decision model equivalent today.
• New workflows = weak numbers. ej drops to 0.419 on never-seen workflows; JEV-35B is strong where it's trained (classification) and weak on open reasoning (GPQA 0.360).
• Calibration is per suite. A confidence threshold certified on one workflow can fail on another. Certify it on your own labelled cases.
• No text. If the task produces words, the decision model isn't the one.

Related

• How much a token costs: five routes, the math, and the break-even — the full calculation when a generative model is really what you need.
• Ollama vs llama.cpp: under the hood — the two engines that serve a decision model like JEV-35B.

In summary

• A decision model doesn't write: it answers a typed question with a probability per option, in a single pass, without decoding tokens.
• JEV-35B is the GPU route: 35B (3B active) in NVFP4, 25.6GB download, 23.3 GiB of memory, on a 32GB GPU. Very strong at classification and routing.
• ej is the CPU route: 11.4MB, 128MB of resident memory, for on-device triage. Weak on new workflows.
• The rule: question with finite options and a right label → decision model. Open answer → generative LLM. Never alone on high-stakes decisions.

Sources

• autotrust/JEV-35B — model card on Hugging Face
• autotrust/JEV-35B-NVFP4 — quantized variant
• 5ak3t/ej — model card on Hugging Face
• 5ak3t/ej — GitHub repository

Article generated with AI