PROMPIT

Data

Data

*Data is based on hardware at hand. More will come once hardware is available.

How it is measured

How to read it. A few runs per setup: a first look, not a ranking. Compare within one machine, since the two ran different models and versions.

Setup. A model file at one context size, with its settings. In a name, 9B or 27B is the model’s size, Q4_K_M its quantisation, and 16k or 32k its context in tokens.

Run. One setup on one task: the signed PROMPIT, in a copy on the machine, driven through its real page in Pump. Each run starts from a clean folder, has no network and stops at 30 minutes.

Passed. A run passes only when every check passes, never on the model’s word. Checks read a kept copy of the run’s folder, and some cases are hidden from the model. The checks were first run on folders that must pass and folders that must fail.

Checks or runs. On the Steam Deck each point is one round of the eight tasks, so its charts count checks; v12.97 ran two rounds, the second with the tasks in reverse order. Linux mostly ran two rounds, with the order of runs changed in the second, so its charts count runs passed.

Speed. Generation tokens per second, from the timings llama.cpp reports for every reply of the run. Nothing is estimated.

Time. Minutes from a run’s start to its end. The Steam Deck’s total adds up the eight tasks. Linux shows the median run.

GPU memory. What the GPU reports in use, so more than the model alone. Peak is the highest reading during a run, read every 20 seconds; once loaded is a reading taken when the model was ready. The Steam Deck counts VRAM and shared memory together, about 9 GB at most.

The green line. A setup is on it when no other setup beats it on both counts.

The grey bars. The 95% range of a pass rate. With this few runs, the true rate could be anywhere along the bar.

Run-card score. The mean of PROMPIT’s four run-card marks, where GOOD counts one, WATCH a half and POOR nothing.

MTP. A draft of the next tokens from the model’s own extra layer, which llama.cpp checks and keeps when it is right.

Routing. A small decision model reads the prompt first. A request for a picture goes straight to the image model, and the text model is not asked or loaded.

Search by meaning. A small embedding model indexes the folder, and the running model may search it. A chart that says first index counted adds the time to build that index.