clef

A web UI and API for Clef-Flash, Cloudflare's 9B decision model, running locally on llama.cpp. Tested on an AMD Radeon RX 9070 XT under WSL2 with ROCm.

A decision model does not write text. It takes a state and typed questions and returns a probability for every allowed answer. This project serves it through llama.cpp's /v1/systemone endpoint (the Jev/SystemOne API) and adds a UI with two tabs:

• Ask: enter a state and one or more questions, each with a list of candidate answers. Each question is sent to the model as a choice question. Results show the predicted answer and every answer's probability, with questions ranked by confidence.
• Evaluate CSV: paste labelled examples, or load them from a file. Each row is sent to the model and its prediction is compared with the row's correct answer. Shows accuracy overall, by confidence band and by question, and every row ranked by confidence, with a results CSV to download.

Tested on

With this setup, all 33/33 layers are offloaded to the GPU (8,169 MiB of model weights in VRAM, about 5 GB left free), and the server is ready about 20 seconds after start once the model is downloaded.

Setup: AMD Radeon GPU on WSL2

These are the steps used on the machine above. Under WSL2 the GPU is reached through /dev/dxg (DirectX), not the Linux amdgpu driver, so ROCm needs the librocdxg bridge, and the Linux amdgpu DKMS driver must not be installed. Use HIP/ROCm rather than Vulkan: inside WSL, Vulkan only gets the slower Dozen (Vulkan-on-D3D12) layer.

1. Windows

1. Install the AMD Software: Adrenalin Edition driver for your GPU from AMD Drivers and Support. AMD lists the driver versions supported for WSL in the WSL support matrices.
2. Update WSL from PowerShell: wsl --update.
3. In Ubuntu, check that the GPU device exists:

   ls -l /dev/dxg

If it is missing, WSL needs updating.

2. ROCm 10.0

Add AMD's ROCm repository and install the runtime for your GPU target, following Install AMD ROCm 10.0.0:

sudo mkdir --parents --mode=0755 /etc/apt/keyrings
wget https://stable.repo.amd.com/rocm/gpg/packages.gpg -O - | \
  gpg --dearmor | sudo tee /etc/apt/keyrings/amdrocm.gpg > /dev/null

sudo tee /etc/apt/sources.list.d/amdrocm-stable.sources << EOF
X-Repo-Id: amdrocm-stable
Types: deb
URIs: https://stable.repo.amd.com/rocm/core/packages/ubuntu2404/
Suites: stable
Components: main
Architectures: amd64
Signed-By: /etc/apt/keyrings/amdrocm.gpg
Enabled: yes
EOF

sudo apt update
sudo apt install amdrocm10.0-gfx1201

gfx1201 is the RX 9070 XT. For another GPU, use its target (rocminfo shows it once the GPU is visible, and AMD's support matrices list them).

3. librocdxg (the WSL GPU bridge)

ROCm on WSL loads librocdxg to reach the GPU through /dev/dxg. Install the prebuilt package from its releases:

wget https://github.com/ROCm/librocdxg/releases/download/v1.2.2/rocdxg-roct_1.2.2_amd64.deb
sudo apt install ./rocdxg-roct_1.2.2_amd64.deb

It installs /opt/rocm/lib/librocdxg.so.1.2.2. See librocdxg issue #74 for another report of the RX 9070 XT working this way.

Check that ROCm sees the GPU:

/opt/rocm/bin/rocminfo | grep -E 'gfx1201|Marketing Name'

Expected:

  Name:                    gfx1201
  Marketing Name:          AMD Radeon RX 9070 XT

rocm-smi does not work under WSL, so don't use it to check.

4. ROCm development files

The runtime package alone is not enough to compile against HIP. Building llama.cpp needs the HIP and hipBLAS headers and CMake configs, which are in the development metapackage:

sudo apt install amdrocm-core-dev10.0-gfx1201

Without it, CMake fails with does not contain the HIP runtime CMake package.

5. Build llama.cpp with HIP

sudo apt install cmake ninja-build build-essential libssl-dev

git clone https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
cd ~/src/llama.cpp
HIPCXX="$(/opt/rocm/bin/hipconfig -l)/clang" HIP_PATH="$(/opt/rocm/bin/hipconfig -R)" \
  cmake -S . -B build -G Ninja -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1201 -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server -j

libssl-dev lets llama-server download models from Hugging Face over HTTPS (-hf). Clef support is in llama.cpp main (it was added in PR #29831).

HSA_OVERRIDE_GFX_VERSION was not needed.

6. Run this app

Requires uv.

git clone https://github.com/kshitizwagle/clef && cd clef
uv run clef

Open http://127.0.0.1:8000. The first start downloads the model (about 10 GB) into ~/.cache/huggingface.

To confirm the GPU is used, look for these lines in ~/clef-server.log:

using device ROCm0 (AMD Radeon RX 9070 XT) (0000:03:00.0) - 13753 MiB free
load_tensors: offloaded 33/33 layers to GPU
load_tensors:        ROCm0 model buffer size =  8168.77 MiB

How it runs

On startup the app runs llama-server (~/src/llama.cpp/build/bin/llama-server -hf ggml-org/Clef-Flash-GGUF:Q8_0 -ngl 99 -lv 4) as a child process and waits until the model is loaded before accepting requests. llama-server's output is shown in the terminal and written to ~/clef-server.log. Stopping the app (Ctrl+C) stops llama-server; if the app is killed, llama-server receives SIGTERM and exits too.

If a model server is already answering at CLEF_UPSTREAM, the app uses it instead of starting one.

The app has no authentication. It listens on all interfaces by default; set CLEF_HOST=127.0.0.1 to keep it local-only unless the network is trusted.

Input size limit

By default a request can be at most 512 tokens, counting the state, the question and the options. Clef evaluates the whole prompt in one physical batch, and llama-server's default batch size (--ubatch-size) is 512. Larger requests fail with:

input (1007 tokens) is too large to process. increase the physical batch size (current batch size: 512)

The 262,144-token n_ctx in the log is the base model's training context and does not apply here. To accept longer inputs, raise the batch size, at the cost of more GPU memory:

CLEF_LLAMA_ARGS="-b 4096 -ub 4096" uv run clef

How far the batch size can go on a 16 GB card has not been measured yet; the compute buffer is 138 MiB at the default 512.

For comparison, Cloudflare's reference code defaults to 16,384 tokens (max_length), Clef on Workers AI has a 64,000-token context, and Jev accepts 64k tokens for the state plus all questions.

Known issues

• Startup crash under WSL. The ROCm runtime sometimes segfaults while creating the GPU context and prints [CreateContext] fail 11. It happened in 2 of about 12 starts on the machine above. When llama-server crashes like this during startup, the app starts it again, up to CLEF_START_ATTEMPTS times. If llama-server exits while the app is running, the app logs the error and shuts down rather than serving without a model.
• No image input. Clef-Flash reads images and video, but llama.cpp does not support image input for Clef yet.

CSV format

The file is read with pandas. Columns can be separated by any punctuation or a tab: common separators (, tab ; |) are tried first, and the first one that yields the context, question and answer headers is used. Rows copied from a spreadsheet are tab-separated, so they can be pasted directly. A field containing the separator must be in double quotes, with any quotes inside it doubled (""). Extra fields at the end of a row, such as trailing separators, are ignored.

In the options cell, the parser picks the separator whose split includes the row's answer, so a comma inside an answer like Yes, definitely is not mistaken for a separator. Header names are case-insensitive, and answers are matched to options case-insensitively. Invalid rows (an empty field, an answer that is not one of the options, fewer than 2 options) are all reported before anything is sent to the model. Row numbers match a spreadsheet: the header is row 1. See `src/clef/static/sample.csv`.

API

POST /api/predict

{
  "state": "Checkout has been failing for every customer for the last hour.",
  "questions": [
    {"question": "Which team should handle this?", "answers": ["billing", "technical", "shipping"]}
  ]
}

Returns results: one entry per question, sorted by confidence (highest first). Each entry has question, prediction, confidence, and answers (each answer with its probability, highest first).

POST /api/evaluate takes a CSV as the request body (Content-Type: text/csv) and streams NDJSON: a start line with the row count, a row line per row as it finishes, then a summary line. An invalid CSV returns 400 with every problem listed.

GET /api/health reports whether the model server is reachable.

Test

uv run pytest

License

This project is released under the MIT License.

It does not include or redistribute model weights or llama.cpp; both are downloaded or built separately and have their own licenses:

• Clef-Flash model weights: Apache-2.0 (post-trained from Qwen3.5-9B, also Apache-2.0)
• llama.cpp: MIT
• ROCm and librocdxg: see AMD's repositories