LLMOps Tools on GitHub: The Open-Source Stack

Search for “llmops github” and you get a strange mix of results: a handful of awesome-lists, some course repos, and a few vendor projects that happen to rank. What almost nobody publishes is the thing the query is actually reaching for — a map of which open-source repositories occupy which layer of a production LLM stack, and what each one is genuinely responsible for.

That map is worth having because LLMOps tooling has stratified fast. Two years ago the category was mostly one monolithic framework plus a vector database. It is now at least six distinct layers, each with two or three credible open-source projects and a very different set of adoption risks. This is a documentation-based survey of that landscape: what sits where, what each layer must do, and what to check in a repo before you build on it.

The layers, and the question each one answers

Layer	Representative open-source projects	The question it decides
Serving	vLLM, SGLang, TensorRT-LLM, Ray Serve, TGI	How many tokens per second per GPU, at what tail latency
Gateway / proxy	LiteLLM, Envoy-class AI gateways	One API surface, keys, quotas, failover across providers
Tracing & prompt management	Langfuse, Arize Phoenix, OpenLLMetry	Why did this specific answer come out wrong
Evaluation	promptfoo, DeepEval, Ragas, Inspect AI	Does this change ship, or does it get blocked
Guardrails	NeMo Guardrails, Guardrails AI, classifier models	What never reaches the user, in either direction
Retrieval & state	Qdrant, Milvus, Weaviate, pgvector, GPTCache	What the model is allowed to know, and what gets reused
Registry & control plane	MLflow, ZenML, KServe	Which artifact version is live, and who approved it

The layers are not optional in the sense of “pick three.” They are optional in the sense that a small system can collapse several into one process. What is not optional is knowing which layer a failure belongs to, because that is what turns a two-day outage into a twenty-minute rollback.

Serving

This is the layer with the clearest winner and the most measurable output. vLLM’s PagedAttention memory manager and continuous batching set the throughput baseline everything else is compared against, and the vLLM documentation covers the OpenAI-compatible server that makes it a drop-in for hosted APIs. SGLang’s RadixAttention targets prefix reuse, which matters disproportionately for agent loops and long shared system prompts. TensorRT-LLM trades portability for NVIDIA-specific kernel performance.

The important repo-health story in this layer is Text Generation Inference, whose own repository now carries a maintenance-mode notice directing new deployments elsewhere. That single banner should reweight any benchmark chart you are looking at, which is the argument made in detail in the vLLM vs TGI serving comparison. For the wider field including SGLang, TensorRT-LLM and Ray Serve, the LLM serving frameworks roundup covers the selection criteria.

Gateway and proxy

The gateway is the most under-adopted layer relative to how much pain it removes. LiteLLM normalises dozens of provider APIs behind one OpenAI-shaped interface and adds the operational furniture nobody wants to write twice: virtual keys, per-team budgets, retries, provider failover, and a single place to emit cost and token metrics.

The reason this layer earns its place is not convenience, it is instrumentation. Once every model call passes one process, per-feature and per-tenant spend becomes a query rather than an archaeology project. That is the precondition for everything in token cost observability in production, and it is also where a semantic cache belongs architecturally rather than inside application code.

Tracing and prompt management

Langfuse is the reference implementation of this layer: nested traces across chains and tool calls, prompt versioning with runtime fetch, scores attached to individual generations, and a datasets feature that turns production traffic into eval fixtures. The Langfuse documentation describes the tracing model and the prompt-management API that lets a prompt change ship without a code deploy — the workflow examined in prompt versioning and deployment.

Arize Phoenix takes the OpenTelemetry-native route, and OpenLLMetry provides OTel instrumentation for LLM SDKs. The reason to care about the OTel route specifically is portability: the OpenTelemetry generative-AI attribute registry fixes standard names for the operation (gen_ai.operation.name), the model on each side of the call (gen_ai.request.model, gen_ai.response.model), token usage including cache reads (gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.usage.cache_read.input_tokens) and why generation stopped (gen_ai.response.finish_reasons). Instrumenting against those conventions means the traces survive a change of backend. Instrumenting against a vendor SDK means they do not. For the retrieval-specific side of tracing, see RAG observability in production, and for the hosted-platform comparison, Arize vs Evidently.

Evaluation

Evaluation is where open source is furthest ahead of commercial tooling, because the hard part is a test harness rather than a dashboard. The promptfoo documentation describes a declarative YAML config that runs a matrix of prompts against a matrix of providers with assertion types ranging from exact string match to model-graded rubrics — which is exactly the shape you want in CI. DeepEval takes the pytest-native approach, so eval failures surface as ordinary test failures. Ragas specialises in retrieval metrics such as faithfulness, answer relevancy and context precision. Inspect AI, from the UK AI Security Institute, targets structured evaluations with solvers and scorers.

The pattern that matters more than the tool choice is where the harness runs. An eval suite that only runs on a developer’s laptop is a demo. The gating setup is described in LLM eval pipelines in CI/CD.

Guardrails

NeMo Guardrails implements conversational rails in a dedicated modelling language, Guardrails AI validates structured output against declared schemas and correction policies, and classifier models handle the input-side detection work. All three sit in the serving path, which means they inherit a latency budget, and that budget is the entire design constraint — the tradeoffs are laid out in guardrails in the serving path.

Retrieval and state

The vector store choice is more consequential than the serving choice for RAG systems, because the failure modes are silent: stale indexes, wrong distance metric, filter pushdown that quietly degrades recall. Qdrant, Milvus, Weaviate and pgvector each make a different bet on operational complexity versus scale, compared side by side in choosing a vector database for RAG. GPTCache belongs in the same layer conceptually: it is state reuse rather than state storage, and the hit-rate and staleness behaviour is covered in semantic caching for LLM serving.

Registry and control plane

MLflow remains the anchor here, with a model registry, stage transitions, and an LLM evaluation module. ZenML and KServe cover pipeline orchestration and Kubernetes-native serving respectively. The registry question in an LLM stack is subtly different from classic MLOps because the deployable artifact is often a prompt plus a model pointer plus a retrieval index version, not a single binary — which is why model registry patterns is worth reading before you decide the registry is a solved problem you already own.

Vetting a repo before you build on it

Adoption risk in this ecosystem is not mainly about code quality. Five checks catch most of it:

1. Maintenance signals over star counts. A repository can carry tens of thousands of stars and a maintenance-mode banner at the same time. Read the README notice, the last release date, and whether recent merged commits are features or only dependency bumps.
2. The open-core boundary. Several projects in the tracing, gateway and vector layers ship an OSS core alongside a separately licensed enterprise tier. Confirm which side of that line the specific feature you are planning around sits on — SSO, RBAC and multi-tenant quotas are the usual paid entries.
3. Telemetry defaults. Some tools phone home by default. Find the opt-out flag and set it explicitly before anything touches production traffic, rather than discovering it during a compliance review.
4. Standards compliance. Preferring OTel-native instrumentation and OpenAI-compatible serving APIs is a portability decision. Both are exit ramps you may need later.
5. Security patch behaviour. Check whether the project has a published security policy and a history of shipping fixes rather than only disclosing them. A frozen dependency in the serving path is a supply-chain problem, not a performance one.

A minimal stack that actually works

For a team going to production for the first time, the smallest configuration that covers every layer is roughly: vLLM for serving, LiteLLM as the gateway, Langfuse or Phoenix for traces and prompt versions, promptfoo wired into CI as a merge gate, one guardrail pass on input and output, pgvector if you already run Postgres, and MLflow only once you have more than one deployable artifact to track. Everything else is an upgrade you make when a specific metric tells you to.

If the open question is cost rather than architecture, the LLM cost and latency estimator will sketch monthly spend and p95 for a given traffic shape before any of this gets installed, and the operating habits that hold the stack together are collected in LLMOps best practices.

Caveats

This is a landscape survey compiled from project documentation and repository metadata, not a benchmark. Project positioning in this ecosystem moves faster than almost any other part of infrastructure — maintenance status, licensing boundaries and feature parity have all shifted within single quarters — so treat every claim here as a pointer to a repository you should open and verify against its current README before committing. Nothing above is a recommendation to run a specific version, and no numbers are asserted for projects whose published benchmarks were produced by their own maintainers.