Measured scope, not a generic speed promise
Validated on 26 October 2026. This page reads the version manifest, performance rows and capability matrix directly from their machine-readable sources. The benchmark uses real 0.8B model weights with a small fixed text case and synthetic diagnostic image/video inputs. These are not representative task-accuracy or production-load benchmarks.
| Component | Pinned version / revision | P1 scope |
|---|---|---|
| Native model | a0a10112716f54457bad6f60b792ad334f9e89fd | 0.8B CPU FP32 |
| Python SDK | valen-sdk==0.2.0 | ONNX local + HTTP |
| JavaScript SDK | @valen-team/sdk@0.2.0 | HTTP |
| ONNX Runtime | 1.31.0 | CPU FP32 |
| GGUF runtime | 0.1.0 | 0.8B Q8_0 text and fixed images; direct video not compiled |
| GGUF source build | 1a0b56cec73d + Valen patch | Separate real regression including direct video; no pooled performance rows |
Full version and model-revision manifest · Executed tutorial outputs and tested source hashes · Capability manifest
Same-worker CPU measurements
All eight successful rows below came from one allocation on one machine: Intel Xeon Platinum 8558, Linux x86-64, four CPU threads, CPU affinity [45, 46, 141, 142], no GPU. Native and ONNX use FP32; GGUF uses Q8_0 backbone weights with FP32 Mixer/vision, so its memory/latency tradeoff includes a precision difference. The source model revision is a0a10112716f54457bad6f60b792ad334f9e89fd.
Each backend/modality gets a fresh server process, one cold request, two additional warmups and seven measured sequential requests. p50/p95 use linear interpolation on sorted samples. Concurrency is one; this is latency, not throughput. Startup includes interpreter/runtime imports, model initialization and health readiness; ONNX can initialize graph sessions during the first request. OS page caches were not cleared: “cold” means process-cold, not disk-cold. Native text was first in the allocation and had a longer startup; do not treat its first startup as a stable backend ratio.
| Backend / precision | Input | Startup s | Cold s | Warm p50 s | Warm p95 s | Peak RSS GiB |
|---|---|---|---|---|---|---|
| native-fp32 | text | 28.471 | 3.900 | 0.640 | 0.655 | 2.728 |
| native-fp32 | image | 6.011 | 1.363 | 0.999 | 1.055 | 3.128 |
| native-fp32 | video | 6.113 | 1.772 | 1.191 | 1.258 | 3.148 |
| onnx-fp32 | text | 1.204 | 3.147 | 1.567 | 1.605 | 2.108 |
| onnx-fp32 | image | 1.103 | 4.956 | 2.855 | 2.945 | 2.481 |
| onnx-fp32 | video | 1.203 | 5.031 | 2.798 | 2.872 | 2.519 |
| gguf-q8 | text | 2.011 | 1.263 | 1.285 | 1.301 | 1.390 |
| gguf-q8 | image | 1.306 | 2.327 | 2.276 | 2.373 | 1.410 |
| gguf-q8 | video | 1.406 | Failed: video not compiled | — | — | — |
Peak memory is the isolated server process's Linux VmHWM resident-memory high-water mark, including resident mapped pages, libraries and the loaded model pipeline. It is not the model file size, the allocation limit, GPU memory, or a minimum RAM specification. The failed video probe does not provide a usable video-memory measurement.
The three backends receive the same bytes, questions, candidate order, temperature 1.0 and media budgets. Successful token counts match: text 166, image 212, video 271 (native/ONNX). Image budgets are 65,536–262,144 pixels; direct video uses 16 uniform frames with a 131,072–2,097,152 total-pixel budget. Exact input/media hashes and raw samples are in the JSON. Media bytes are prepared before the request timer; upload from a remote client or network asset fetch is not included.
The native/ONNX latency server is a small localhost measurement adapter around real predict, not SGLang/vLLM or the production SDK HTTP stack. Follow-up checks separately exercised the production ONNX SDK service. The original benchmark was not rerun after fixing the measurement service's model alias for Cookbook checks.
Client time versus server time
Client time includes JSON encoding, localhost HTTP, service work and response decoding. The native/ONNX adapter additionally measures the predict call, including compilation and media preprocessing. The prebuilt GGUF server does not expose an equivalent measurement, so the table leaves it unavailable rather than relabeling HTTP time as inference.
| Backend | Input | Client warm p50 s | Server predict warm p50 s |
|---|---|---|---|
| native-fp32 | text | 0.640 | 0.639 |
| native-fp32 | image | 0.999 | 0.998 |
| native-fp32 | video | 1.191 | 1.190 |
| onnx-fp32 | text | 1.567 | 1.567 |
| onnx-fp32 | image | 2.855 | 2.854 |
| onnx-fp32 | video | 2.798 | 2.797 |
| gguf-q8 | text | 1.285 | — |
| gguf-q8 | image | 2.276 | — |
Benchmark config JSON · Raw performance report (including the failed probe) · Preserved failures and resolutions
Choose within the measured scope
| Need | A useful starting path | Evidence and limits |
|---|---|---|
| Try without downloading weights | Hosted API / website | Separate online service; no speed or memory comparison with the local worker |
| Lower warm latency on these small CPU inputs | Native FP32 | Fastest measured warm requests here, with higher RSS and heavier startup/dependencies |
| Python/JS integration without Torch | ONNX SDK | Python CPU FP32 measured here; Node/WebGPU local performance not rerun |
| Smaller RSS on the measured CPU cases | GGUF Q8_0 + custom runtime | Lowest RSS in these rows; lower precision; CPU text/fixed images qualified |
| Direct video | Native / Python ONNX; video-enabled GGUF source build | Prebuilt 0.1.0 has no direct video support; source-build regression is separate and its times are not pooled into this table |
| CUDA, other model sizes/devices, concurrency | Use the deployment guide and measure your workload | No new P1 performance data; earlier quality/device evidence must not be mixed into this same-worker comparison |
Interface and media capabilities
| Backend | Decisions | SystemOne | Text | Image | Direct video | Evidence scope |
|---|---|---|---|---|---|---|
| Native Transformers Python | Yes | Yes | Yes | Yes | Yes | P1: 0.8B CPU FP32, fixed fixtures + full local tutorial |
| Python ONNX local / SDK service | Yes | Yes | Yes | Yes | Yes | P1: 0.8B CPU FP32, real service and cookbook; CUDA not rerun |
| JavaScript ONNX local / WebGPU | Yes | Yes | Yes | Yes | Yes | Prior qualified local path; P1 tests HTTP clients only |
| C++ ONNX integration | No | Yes | Yes | No | No | Existing FP32 CPU text integration; not rerun in P1 |
| GGUF prebuilt 0.1.0 | Yes | Yes | Yes | Yes | No | P1: exact existing archive, 0.8B Q8_0 text/fixed images; video probe failed |
| Valen GGUF source build | Yes | Yes | Yes | Yes | Yes | P1: cached video-enabled source binary, 0.8B Q8_0 fixed fixtures |
| SGLang Valen plugin | Yes | Yes | Yes | Yes | No | Existing single-GPU text/image path; no new P1 GPU run |
| vLLM Valen plugin | Yes | Yes | Yes | Yes | No | Existing single-GPU text/image path; no new P1 GPU run |
| Valen custom Ollama | Yes | Yes | Yes | Yes | Yes | Prior custom CPU import/protocol/media evidence; not rerun in P1 |
| Hosted API | Yes | Yes | Yes | Yes | No | Reference supports text/one image; P1 recipes execute on pinned local backends |
Both protocols remain supported where listed. Decisions uses input, an ordered question list, predicate, typed string/boolean choice values and labeled score levels. SystemOne uses state, an ID-keyed question mapping, noul, a string-keyed criteria mapping and ordered score criteria. Scores are probability-weighted zero-based level indices. Numeric choice values are rejected; numeric-looking string keys in JavaScript objects follow JavaScript object-order rules. Use Decisions arrays when candidate order is significant; the JS HTTP client rejects Map rather than silently changing it.
A finite positive scalar temperature is the portable intersection. Native/ONNX legacy per-type temperature mappings are not a GGUF or Decisions portability guarantee. Candidate order is preserved within a request; changing the order can change the model's prompt and probabilities. No permutation-invariance promise is made.
Local native/ONNX calls can read local files. HTTP media uses inline image/video data URLs where supported; the production ONNX service disables client-local paths. The hosted API and GPU plugins use a single image, so extract a frame explicitly for the video-tagging application. Valen.local is an ONNX loader; use Valen.remote for GGUF HTTP services. Generic upstream llama.cpp/Ollama does not become compatible through these SDKs.
The exact downloadable GGUF 0.1.0 binary was reused and checked against its existing SHA256; no asset was downloaded again. It now has additional P1 fixed-image evidence. Its direct video probe returned HTTP 500, and the log states video support not compiled in. This failure remains published; use a sampled image or a separately built video-enabled runtime. The source-build video result is not qualification of the downloadable binary.
Fixed regression and release checks
One fixture in the main repository, tests/fixtures/backend-contract.json, covers choice, predicate/noul, score, mixed boolean/string candidates, reversed candidate order, temperatures 1.0/0.5 and invalid requests. Real checks compare each backend's Decisions and SystemOne answers, usage, candidate types/order and errors. The raw Python and JavaScript SDK packages are also exercised against the production ONNX service, the prebuilt GGUF text/image service and the video-enabled source service.
| Compared backend | Variants | Choice agreement | Probability MAE | Max probability error | Input tokens match |
|---|---|---|---|---|---|
| onnx-production | 12 | 100.0% | 0.000114 | 0.001887 | ✓ |
| gguf-prebuilt | 9 | 100.0% | 0.002311 | 0.023910 | ✓ |
| gguf-source-video | 12 | 100.0% | 0.003300 | 0.035390 | ✓ |
The numerical gates reuse the previous frozen GGUF regression's choice/probability/normalized-score thresholds. This small protocol fixture is not a replacement for the earlier 292-text/30-image quality set, and consistency with the native model is not accuracy against ground truth. All compared input token counts match. The follow-up allocation contributes numerical checks only; its timings are not merged with the earlier performance worker.
| Validation layer | Actual P1 result | What it proves |
|---|---|---|
| Mock / constructed answers | Python SDK 28 tests, JavaScript SDK 12 tests; shared adapter check: 16 valid variants + 5 invalid cases | Client/adapter contracts, lifecycle, geometry and failure handling; no model quality claim |
| Synthetic model | Not executed for P1 | No synthetic-model result substitutes for real weights |
| Real model, fixed diagnostic fixtures | Native/ONNX/GGUF supported scopes passed both protocols; six live SDK-client reports passed | Small reproducible protocol and numerical regression, with real weights |
| Real model, complete tutorials | Five maintained scripts: direct local ONNX/native basics and four HTTP scripts on ONNX and GGUF; 10 successful CLI executions | End-to-end sample workflows and actual business outputs |
| Publication | Checked separately after deployment | A build or local test alone is not a live-publication pass |
Fast synthetic mapping checks run in the main repository CI, pinned to the stable SDK source commits. Website CI tests source/output hashes, builds both languages and checks links. Code/sample changes invalidate the stored real-execution hashes until rerun. Before a relevant model/runtime/SDK release, run the real checks on cluster caches and validate the pinned release evidence; CI does not download model weights. See the release and reproduction procedure.
The first four HTTP tutorial attempts failed because the benchmark adapter served Valen-0.8B while the commands requested the deployment alias valen. The alias was corrected and affected checks rerun against the production SDK service. Both original failures and the successful replacement checks are retained. The prebuilt video failure was not “fixed” by relabeling a source build.
Release check summary · Numerical comparison · Synthetic adapter check · Migration map
P1 does not newly qualify 2B/4B, GPU/CUDA performance, browser WebGPU local execution, the C++ integration, SGLang/vLLM GPU serving or custom Ollama import. Existing qualified evidence remains linked from their deployment/model references. The original media lacks a separately recorded dataset/license attribution; the tutorials state that limitation and preserve the shared URLs.