A compute pool API coordinates machines and schedulable capacity across GPU workers, CPU nodes, RAM-heavy hosts, VPS instances, containers, and specialized accelerators. Clients submit workload requirements instead of hard-coding a machine address. The control plane validates policy, finds capacity, creates a lease, and returns a route or execution handle. The data plane starts and monitors the workload.
Good compute pooling improves utilization and failure recovery, but it also concentrates risk. A mistake in the scheduler can overcommit memory, place sensitive workloads together, starve small tenants, or burn expensive accelerator time. The API should expose explicit resource shapes, lifecycle states, queue policy, and security boundaries rather than promising generic “compute.”
Describe a workload with a resource shape
Express minimums and preferences
A resource shape defines the minimum and preferred CPU, memory, GPU count and type, local storage, network class, architecture, operating system, container runtime, and geographic constraints. Keep requests declarative. The client should say “one compatible accelerator with at least this memory” when exact hardware is not required, while still allowing a hard pin for validated workloads.
Separate allocatable capacity from raw machine totals. The operating system, runtime, monitoring, and safety margin consume resources. Accelerator topology matters: two GPUs on the same interconnect are different from two remote devices. RAM may be technically available but fragmented across NUMA boundaries. The member inventory should publish scheduler-relevant capacity, not marketing specifications.
Register members with verified capabilities
A node joins the pool through a controlled registration process. It presents an identity, attestation or enrollment credential, software version, resource inventory, labels, and supported runtime features. The control plane verifies the record before the node becomes schedulable. Heartbeats update health and current capacity, while periodic inventory reconciliation catches drift.
Do not trust a self-reported GPU model or memory total without validation when the pool spans untrusted contributors. In a private fleet, an authenticated agent may be enough. In a marketplace, stronger attestation, benchmarking, and isolation checks may be required. Keep “registered,” “verified,” “ready,” “draining,” “quarantined,” and “retired” as distinct states.
Allocate with leases
A lease is a time-bounded claim on a resource shape. The scheduler atomically selects a member, reserves capacity, and returns a lease identifier with expiry. The client or execution controller commits the lease when startup begins. Heartbeats can extend long-running jobs. If the lease expires before commit, capacity returns to the pool. If the worker disappears, the control plane marks the lease uncertain until reconciliation confirms whether the workload is still running.
Leases prevent double scheduling and create a clean cancellation path. They also support reservations for scheduled work. A future reservation should specify a window and tolerance, not block a machine indefinitely without policy. Distinguish capacity held for a job from capacity actually consumed so utilization and billing remain accurate.
Treat the container runtime as a data-plane boundary
Containers provide packaging and process isolation, but they are not a complete multi-tenant security model. Docker documents a client-server architecture in which the client communicates with the daemon through an API; the official Docker overview is a useful reference for that control relationship. A compute pool should avoid exposing a privileged daemon directly to tenants. Put a narrow execution service between the pool API and the runtime.
Allow only approved image registries or signed images for sensitive environments. Resolve tags to immutable digests before scheduling. Set CPU, memory, process, filesystem, and network limits. Run as a non-root user where possible, drop unnecessary capabilities, and isolate secrets from the image. For stronger boundaries, consider dedicated virtual machines, sandboxed runtimes, or separate node pools based on workload trust.
Design the queue and fairness policy
When demand exceeds capacity, the system needs more than first-in, first-out. A strict FIFO queue can let one large job block many small jobs. Weighted fair sharing, tenant quotas, priority classes, and maximum wait targets provide better control. State whether priority can preempt running work or only reorder queued work. Preemption is safe for checkpointable batch jobs and dangerous for stateful services unless recovery is designed.
Prevent starvation. Reserve some capacity for short jobs, enforce per-tenant concurrency, and age waiting jobs so they gain priority. Expose queue position cautiously because exact positions change quickly and can leak other tenants’ activity. A predicted start window or queue class is often more useful. Let clients cancel queued requests without creating a running job during a race.
Use filters, scores, and placement constraints
Separate eligibility from ranking
The scheduler first filters nodes that cannot satisfy hard requirements. It then scores eligible members for locality, utilization, energy or price class, failure domain, warm image cache, and affinity. Keep anti-affinity rules for replicas that should not share a host or zone. Co-locate tightly coupled tasks only when the network and topology support it.
Version the scheduling policy and record the decision factors. Operators need to explain why a job landed on a specific node or waited despite apparent capacity. Capacity may be reserved, fragmented, quarantined, or incompatible with the image architecture. A transparent reason code reduces support load and helps clients choose better shapes.
Handle GPU and accelerator details explicitly
GPU pools need fields for model or capability class, memory, driver and runtime compatibility, partitioning support, topology, and health. A device can be present but unsuitable because of error state, temperature, degraded links, or incompatible software. Run health checks that exercise the actual execution path, not just device enumeration.
Decide whether fractional GPU allocation is supported and what it means. Memory partitioning, time slicing, and process sharing have different isolation and performance characteristics. Expose them as distinct resource classes. Do not advertise fractional units unless the scheduler and runtime can enforce them. Record peak and average utilization so the pool can improve packing without guessing from requested limits alone.
Do not hide storage and network
Compute jobs often fail because the scheduler focused on CPU and GPU while ignoring data. Include local scratch size, persistent volume needs, object-store access, ingress and egress constraints, and network locality. Large model weights may take longer to load than the computation itself. A warm cache signal can improve startup time, but it should be treated as a preference unless the job requires local data.
Network policy should limit which services a workload can reach. Egress controls protect credentials and reduce data exfiltration risk. If tenants bring datasets, specify encryption, retention, and cleanup. Leases should include storage lifecycle: what is ephemeral, what persists, and how the client retrieves outputs after the compute allocation ends.
A scheduler-oriented API surface
GET /compute-pools
GET /compute-pools/{id}/shapes
POST /compute-leases
POST /compute-leases/{id}/commit
POST /compute-leases/{id}/renew
POST /compute-leases/{id}/release
POST /jobs
GET /jobs/{id}
POST /jobs/{id}/cancel
GET /jobs/{id}/eventsA job request references a resource shape, image digest, command or workload manifest, input locations, network policy, maximum runtime, retry policy, and output destination. The response should distinguish queued, leased, starting, running, checkpointing, succeeded, failed, cancelled, and lost states. Every transition gets a timestamp and reason.
Measure allocation and execution separately
Allocation metrics include queue time, scheduling attempts, unsatisfied constraints, lease expiry, and fairness by tenant. Execution metrics include startup time, runtime, CPU and memory pressure, accelerator utilization, retries, eviction, and exit reason. Capacity metrics include allocatable, reserved, used, fragmented, draining, and unhealthy resources by shape and failure domain.
Trace the path from request to scheduler decision, node admission, image pull, container start, and result upload. A generic “job failed” status is not enough. Clients need to know whether the image was invalid, capacity vanished, startup timed out, the process exited, or the node was lost. Operators need the same event sequence with infrastructure details.
Protect the control plane
Separate permissions for registering nodes, defining shapes, submitting jobs, viewing logs, retrieving outputs, and changing policy. A compromised worker should not be able to register more workers or alter quotas. Rotate node credentials and bind them to identity. Keep the scheduler’s credentials away from tenant containers. Validate every callback from a node against the lease and job it claims to update.
Use quotas for concurrent jobs, reserved GPU time, storage, and network egress in addition to credit balances. A well-funded tenant should not be able to exhaust all control-plane threads or queue memory. Rate-limit expensive scheduling searches and reject impossible shapes early.
Pool compute around enforceable contracts
A compute pool API is a scheduler contract: declarative requirements in, verifiable allocation and execution events out. The strongest designs model capacity honestly, allocate with leases, isolate the runtime, define fairness, and keep accelerator, storage, and network constraints visible. Start with a few shapes and one trusted fleet before adding marketplace nodes or complex preemption.
Use the Compute Pools hub for the broader resource map and the node-pool guide for registration and health patterns.
