Book a call ↗

Illustrative engineering bench with an open GPU server and compact compute cabinet

Project 03 / On-premises GPU cloud

GPU as
a service.

For a $2B edge-AI company.

← All our work

The project

From accelerator hardware
to a usable service.

Client
A $2B edge-AI company
Our work
GPU as a service
Environment
On-premises: the client’s own GPU data centre
Technology
NVIDIA H100, A100 and L40S · GPU Operator · MIG · DCGM
Across the stack
Infrastructure

The service model

Make compute available
to AI workloads.

GPU as a service provides a platform between accelerator capacity and the applications that use it.

GPU capacityNVIDIA GPU servers
GPU serviceQuotas · Partitioning
AI workloadsTeams and tenants
Conceptual service model

Engineering spotlight / NVIDIA GPUs

From GPU servers
to self-service compute.

A multi-tenant GPU platform on Kubernetes. Teams request GPUs through an API or console, receive an isolated share of the fleet and are metered for what they use.

480

NVIDIA GPUs

H100, A100 and L40S on 60 servers, under one control plane.

40+

Tenant teams

Each in its own namespace, with quotas and single sign-on.

31→74%

Average GPU utilisation

Within two quarters, as MIG and time-slicing filled idle capacity.

<2min

From request to running GPU

Through the API or console, replacing a ticket queue measured in days.

7

Isolated instances per GPU

MIG splits each A100 and H100 into as many as seven hardware-isolated slices.

Zero

Workloads cut off by upgrades

GPU nodes are drained before every driver and operator upgrade.

Client details are anonymised.

The challenge

Expensive hardware.
Many teams.

GPU capacity had to be shared safely between teams, and between workloads of very different sizes.

01 / Sizing

Right-sized GPUs.

Small experiments and large jobs on the same fleet.

02 / Isolation

Tenants kept apart.

One team’s workload must not see or slow another’s.

03 / Accounting

Usage on record.

Quotas and metering, so capacity can be planned and charged back.

The operating stack

Every layer,
working together.

Tenancy, scheduling and GPU partitioning on one Kubernetes platform. Observability and security run across it.

Self-service

API and console · Projects · Quotas

Tenancy

Namespaces · RBAC · Resource quotas · Network policies

Scheduling and metering

GPU-aware scheduling · Priorities · Usage per tenant

GPU enablement

NVIDIA GPU Operator · MIG · Time-slicing · DCGM

Kubernetes platform

Upstream Kubernetes · Argo CD · Container registry

Hardware

NVIDIA H100, A100 and L40S servers

The fleet

Three pools.
One control plane.

480 NVIDIA GPUs on 60 servers, pooled by the way each workload uses a GPU.

Training pool / 16 servers

Whole H100s.

128 GPUs on NVLink, allocated whole for training and fine-tuning.

Serving pool / 28 servers

A100s with MIG.

224 GPUs, split into hardware-isolated instances for inference and evaluation.

Development pool / 16 servers

L40S, time-sliced.

128 GPUs, shared between notebooks and experiments that tolerate contention.

Observability

DCGM · Prometheus · Grafana

GPU utilisation, memory and health, per node and per tenant.

Tested alert rules, with a runbook for each alert group.

Security

SSO/OIDC · RBAC · Network policies · Signed images

Every tenant works inside its own namespace, with access granted through single sign-on.

Request to release

Every GPU request follows
the same path.

From a request in the console to capacity returned to the pool, each step is automated and recorded.

  1. 01

    Request

    API or console

  2. 02

    Authorise

    SSO and quotas

  3. 03

    Place

    GPU-aware scheduler

  4. 04

    Partition

    MIG or time-slice

  5. 05

    Run

    Tenant workload

  6. 06

    Observe

    DCGM telemetry

  7. 07

    Meter

    Usage per tenant

  8. 08

    Release

    Capacity returned

How a GPU is shared

MIG splits supported NVIDIA GPUs into hardware-isolated instances, each with its own memory. Time-slicing shares a GPU between lighter workloads that can tolerate contention. Whole GPUs remain available for jobs that need them.

How capacity is governed

Each tenant has quotas for GPUs and GPU instances. Requests beyond quota wait or are refused, and usage is metered per tenant for planning and charge-back.

How platform upgrades roll out

Changes ship through GitOps with Argo CD and reach a staging cluster first. GPU nodes are drained before driver or operator upgrades, so running workloads are not cut off.

The outcome

One fleet.
Shared with confidence.

GPU capacity became a service teams request, rather than hardware they wait for.

01 / Self-service

GPUs on request.

Teams get GPU capacity through an API or console instead of a ticket.

02 / Utilisation

Fewer idle cards.

MIG and time-slicing fit smaller workloads onto shared GPUs.

03 / Visibility

Usage you can plan from.

Per-tenant metering and DCGM telemetry show who uses what.

← All our work