llmprobe
Synthetic monitoring and CI smoke tests for LLM inference endpoints. Measure TTFT, latency, throughput, and errors. Single binary, zero SDKs.
llmprobe is a CLI tool for LLM serving reliability. It probes hosted APIs or OpenAI-compatible inference servers, then reports the metrics that matter for production user experience: time to first token (TTFT), total latency, generation throughput (tokens/sec), and error rates.
Use it as a one-off health check, a continuous monitor, or a CI gate that blocks deploys when your LLM provider is degraded.
Public benchmark
llm-bench uses llmprobe to run a continuous public benchmark of major LLM APIs. It publishes a live dashboard at bench.jonathanwrede.de and raw JSONL data in Jwrede/llm-bench-data.
This is the intended use case: repeated synthetic probes that make LLM latency, TTFT regressions, throughput drops, and provider degradation visible before users report them.
Install
Download a prebuilt binary from the latest release (Linux, macOS, Windows; amd64 and arm64).
