LLMOps tools on GitHub cover several different jobs: serving models, routing requests, retrieving context, tracking releases, and checking outputs before deployment. This guide maps official repositories to those jobs and links each to its documentation, rather than offering an undifferentiated list of projects.
The scope is the production serving stack. For framework selection and scoring methodology, use Evals ML’s LLM evaluation frameworks. For the infrastructure decisions, start with vLLM vs TGI, self hosting LLM vs API cost, and the best vector database for RAG.
LLMOps GitHub repositories by layer
| Layer | Official repository | Documented role | Adoption question |
|---|---|---|---|
| Serving | vLLM | LLM inference and serving | Does the target release support the model, hardware, and request shape? |
| Gateway | LiteLLM | Model API access through a proxy or SDK | How will routing, retries, and accounting be configured? |
| Traces and prompts | Langfuse | Observability, prompt management, and evaluation workflows | What data is captured, retained, and exportable? |
| Release evaluation | promptfoo | Declarative prompt evaluations and CI integration | Which assertions block a release, and how are results retained? |
| Retrieval | pgvector | Vector search in PostgreSQL | Can the filtered query meet relevance and latency requirements? |
This table is a starting stack, not a requirement to deploy all five projects. An application using a hosted model endpoint may not need an inference server. Existing application infrastructure may already provide routing or telemetry. Add a component when it owns a clear responsibility that would otherwise be missing.
The linked repositories document capabilities; they do not establish benchmark superiority or that every enterprise feature is included without a paid license. Check the exact release and license boundaries before adoption.
Serving: turn model support into an operating decision
The serving layer controls how requests reach the model and how its execution consumes capacity. Choose it against the exact model revision, tokenizer, hardware, and workload rather than repository popularity.
vLLM documents its serving engine and supported capabilities in its repository. For an alternative already in use, inspect its current status. TGI’s official documentation carries its maintenance notice: Hugging Face put TGI in maintenance mode on 2025-12-11 and archived the GitHub repository (read-only) on 2026-03-21, so no further upstream fixes, security fixes included, can be merged; plan existing deployments around a pinned, unpatched release. That belongs in a dependency decision alongside performance and migration work.
The vLLM vs TGI comparison explains a workload matrix, metric checks, and release acceptance criteria. Use it before translating advertised tokens per second into a capacity commitment. For a wider shortlist, use the serving framework selection guide to define engine and orchestration requirements.
Gateway and proxy: account for every attempt
LiteLLM’s repository describes both a proxy server and a Python SDK. Decide which integration owns provider selection, request identifiers, retries, and budgets in your architecture. A library inside each application and a shared service have different operational boundaries.
Document failure behavior explicitly. If a call is retried or routed to another model, preserve the relationship between the user request and each attempt. Otherwise, successful-request counts and billed model calls can diverge without a useful explanation.
Budget enforcement also needs a defined accounting source and response to unavailable storage. Verify the selected configuration instead of assuming that enabling a gateway automatically enforces every cost limit. Token cost observability describes the ledger the application needs.
Tracing and prompt management: preserve release context
Langfuse documents tracing, prompt management, and evaluations. Use release identifiers to connect these features: a generation should be attributable to the prompt version, model route, and retrieval configuration that produced it.
Choose content-capture and retention settings deliberately. Request metadata may answer an operational question without retaining the complete prompt or answer. An integration trial should check data export, deletion, and the behavior of the application if the telemetry destination is unavailable.
The prompt-versioning workflow covers deployment mechanics. LLM and prompt registry patterns explains the release record needed to reproduce an incident. For selecting the operational signals themselves, use LLM observability tools for serving.
Evaluation: connect a result to a release
Promptfoo documents declarative test configurations and CI integration. The operational requirement is a result tied to an identifiable candidate: prompt version, model revision, dataset, scorer configuration, and application release.
Define what a failure does before wiring a job into CI. A report that never blocks promotion serves a different purpose from a release gate. Keep the report accessible after deployment so a rollback decision can refer to the evidence that originally permitted the release.
LLM eval pipelines in CI/CD covers gate placement and release workflow. The evaluation-framework comparison at Evals ML handles harness and methodology selection. Avoid treating one tool’s default metric as a universal definition of acceptable output.
Retrieval and state: choose with the real filters
The pgvector repository documents exact and approximate vector search and the behavior of filtered approximate scans. Existing PostgreSQL operations can make it a useful candidate, but the query plan, data size, and competing application workload still need evaluation.
The vector database comparison covers Pinecone, Weaviate, Qdrant, pgvector, Chroma, and Milvus with official sources. It separates index recall from relevance and includes a capacity-planning procedure.
Keep the corpus revision, embedding model, chunking configuration, and permissions tied to each retrieval release. A cache adds another versioned state boundary; semantic caching for LLM serving covers similarity thresholds, freshness, and invalidation.
Registry and control plane: record the bundle
MLflow’s Prompt Registry documentation describes prompt versions, aliases, and associated metadata. In an LLM application, the release record should connect a prompt with the model and retrieval dependencies it was evaluated against.
Treat an environment alias as a deployment pointer, and retain the immutable versions it resolved to. A rollback needs the bundle, not just the prompt’s text. The LLM/prompt registry guide develops that operational record without duplicating a general model-registry comparison.
Vetting a repository before adoption
Use a short, repeatable review:
- Read the maintenance notice, release history, and supported-version policy.
- Check the license for the components and features you will actually deploy.
- Identify persistence, backup, upgrade, and recovery requirements.
- Inspect content capture and telemetry defaults against your data requirements.
- Check export formats and client compatibility with a small representative fixture.
- Record the pinned version, configuration owner, and rollback procedure.
Stars and download counts can help discovery, but do not establish compatibility, support, or operating cost. Keep the review attached to a release decision so it can be repeated when the dependency changes.
Where Azure Prompt Flow fits
Microsoft documents an end-to-end GenAIOps workflow that runs prompt flow with GitHub (GenAIOps with prompt flow and GitHub).
That page also carries retirement and migration guidance and says Prompt Flow is no longer recommended for new development. Readers maintaining an existing deployment should verify the current notice and migration path before extending it. It is a distinct lifecycle topic, rather than another generic serving engine in this repository table.
A minimal operating plan
Start with an application boundary, a model route, a retrieval choice if needed, and a release record. Decide how requests are identified, how failures are handled, and what evidence permits promotion. Add a gateway, prompt service, or registry only when its responsibility and operating cost are clear.
Compare the resulting capacity plan with self hosting LLM vs API cost. The cost and latency estimator can explore assumptions, while the cost guide accounts for idle capacity, retrieval, and operations. A smaller stack with explicit owners is easier to evaluate than a collection of overlapping dashboards.
