Project 04 / Production inference
Inference
as a service.
For an AI API provider serving two of India’s largest consumer-internet apps.
The project
The platform between
models and applications.
- Client
- An AI API provider serving two of India’s largest consumer-internet apps
- Our work
- Inference as a service
- Environment
- Bare metal, in the client’s data centre
- Our role
- Design, build and 24×7 operation
- Hardware
- Accelerator nodes, two scheduling models
- Across the stack
- Infrastructure · Applied AI
The request path
Requests in.
Metered results out.
Every hop emits a span. Every request becomes one metered row with 15 stage timings.
Engineering spotlight / Inference platform
From bare metal
to live inference.
We designed, built and operate the end-to-end inference platform behind a commercial AI API.
Metered requests
Since July 2026, each one metered and traceable end to end.
Requests a day
Peaking near 17 requests per second.
Requests without a platform fault
Fewer than 0.01% platform-caused failures over 7 days.
Dropped requests
During model and platform rollouts.
Platform overhead
Per request, under 1 s regardless of model. Inference itself takes 2.7–4.1 s.
Model workers
On 25 accelerator nodes, operated 24×7.
Reported production results, as of 28 October 2026. Request reliability is not an uptime SLA.
The challenge
Three workloads.
One commercial API.
Image, video and language models behave differently on the same hardware, and every request has to be billable.
High concurrency.
Short requests. Bursty traffic.
Long-running jobs.
Async queues. Large outputs.
Streaming tokens.
OpenAI-compatible API. Per-token metering.
The operating stack
Every layer,
working together.
Kubernetes
PostgreSQL
PrometheusSix layers, from hardware to the application. Observability and security run across all of them.
Application
OpenAI-compatible API · Operator and customer consoles
Inference platform
Gateway · Auth · Quotas · Routing · Async jobs · Metering
Model serving
Worker pools · Accelerator profiles · Weights · vLLM
Kubernetes platform
Kubespray · Cilium · Rook-Ceph · CNPG Postgres · ClickHouse · Redis · Vault
Hardware enablement
In-house device plugin · Kubernetes DRA · Device telemetry
Hardware
Accelerator servers · Ansible bare-metal provisioning
The fleet
Two scheduling models.
One fleet.
64 model workers on 25 accelerator nodes, run from a control plane on 3 nodes.
Gateway and state.
Envoy gateway, Postgres, ClickHouse, Redis, Vault and full observability.
32 slices.
Split by an in-house device plugin. Serving image-edit models.
Kubernetes DRA.
Scheduled with Dynamic Resource Allocation. Serving image-edit models, with one server spare.
Observability
Prometheus · OpenTelemetry · Tempo · VictoriaLogs · Pyroscope · ClickHouse
60 dashboards covering device, model and request health.
Operated 24×7, with tested alert rules and a runbook per alert group.
Security
Vault · Kyverno · Falco · Cilium policies · Signed digests
Policy enforced at admission. Runtime behaviour watched on every node.
ISO 27001 and SOC 2 assurance.
Journey to production
Every release follows
the same path.
From automated provisioning to real-request tests and signed, stage-to-prod promotion.
- 01
Preflight
Checks pass
- 02
Provision
Ansible
- 03
Orchestrate
Kubespray · Kubernetes
- 04
Devices
Device plugin · Kubernetes DRA
- 05
Deploy
Argo CD ApplicationSets
- 06
Serve
vLLM
- 07
Validate
Real requests
- 08
Promote
Signed digests, stage to prod
How upgrades keep traffic moving
Workers drain gracefully before they are replaced, while a hot spare takes traffic. Model and platform rollouts have dropped zero requests.
How the platform handles failures
If a worker crashes mid-request, the request is retried on another free worker within its deadline and the failed worker is quarantined. Devices that trigger health alerts are quarantined and re-probed.
How overload is handled
Traffic beyond capacity waits in a first-come, first-served queue per model, with a bounded wait, then a fast shed. Priority tenants keep reserved slots.
How state and deliveries are protected
Customer webhooks leave through an outbox with leases, backoff, a circuit breaker per target and a full delivery trail. Postgres runs on 3 nodes with synchronous replicas, beside replicated ClickHouse, Redis Sentinel and Ceph, all with verified backups.
How performance is benchmarked
A reproducible harness runs on real hardware. It replays load from anonymised production traffic, uses per-stage timings from metering and produces comparable reports: throughput, latency per stage and cost per request.
The outcome
A commercial API.
Run as a platform.
The client sells inference. We run the platform underneath it, around the clock.
Live at scale.
A production inference platform serving two of India’s largest consumer-internet apps.
Every request accounted for.
100% of requests metered and traceable end to end: one row per request, with 15 stage timings.
Two weeks to production.
For a new deployment: discovery on day 1, install on days 2–3, a pilot with real tenants by day 10 and production on day 14.
Bring this capability to your team
Inference & GPU platforms
Explore Infer8: the inference platform family built from this work, as a Blueprint, a self-hosted Stack or a managed Cloud.
Explore the service Discuss your project
Next project / 05