A node pool API manages a fleet of machines or runtime workers that can accept scheduled work. A node may be a physical server, virtual machine, edge device, container host, database replica, blockchain participant, or specialized inference worker. The control plane needs to know what each node can do, whether it is trustworthy, whether it is healthy, how much capacity remains, and when it should stop receiving work.
The design becomes reliable when identity, health, readiness, and capacity are separate concepts. A node can be authenticated but unhealthy. It can be healthy but intentionally draining. It can report free capacity while being incompatible with a workload. Compressing all of that into an “online” boolean produces unsafe scheduling and confusing operations.
Give every node a durable identity
Bind identity to attestation
A node should have an immutable identifier independent of hostname or IP address. Registration binds that identifier to credentials, ownership, pool membership, labels, supported features, and enrollment evidence. Mutable network addresses and display names belong in metadata. Reimaging a machine may create a new node identity or a new incarnation number, depending on how the system handles trust and history.
Credentials should be short-lived or renewable and scoped to node operations. A node can update its own heartbeat and job state, but it should not add another node, change pool policy, or increase its capacity limit. Record the credential issuer, enrollment time, software version, and last rotation. A compromised node credential must be revocable without taking down the whole fleet.
Use an admission workflow
Registration is not the same as admission. A newly enrolled node starts in a pending state while the control plane validates identity, software version, runtime configuration, capability inventory, and any required attestation. Automated checks can confirm that the node can pull an approved image, reach required services, enforce resource limits, and report telemetry. Only then should it become schedulable.
For contributor or marketplace nodes, add benchmarking and fraud controls. Self-reported hardware should not determine payment or high-value placement by itself. Keep measured performance, declared capability, and verified capability as separate fields. A node can remain visible to its owner while quarantined from production workloads.
Model liveness, readiness, and health separately
Liveness answers whether the node agent is communicating. Readiness answers whether the node is willing and configured to accept new work. Health summarizes whether hardware, runtime, storage, network, and required dependencies are within policy. A heartbeat can carry current observations, but the control plane should calculate the authoritative state using timestamps and rules.
Kubernetes documents nodes as worker machines managed by a control plane and containing services needed to run workloads; its node concept provides a useful reference for separating node status and management. A custom node pool can use similar principles without copying every Kubernetes object. The important part is a clear state model and a controller that reconciles desired state with observations.
Design heartbeats for loss and delay
Handle stale and partitioned nodes
Heartbeats should include node identity, incarnation, monotonic sequence, observed time, agent version, readiness, active leases, and bounded health summaries. The server records receipt time. Do not trust the node clock as the only signal because clock skew can make stale data appear fresh. Use a grace period before marking a node unreachable, then a separate threshold before declaring work lost.
Heartbeats may be duplicated or arrive out of order. Ignore a lower sequence from the same incarnation. When a node restarts, increment the incarnation so old messages cannot overwrite new state. Keep payloads small and use separate endpoints or streams for detailed metrics. Heartbeat traffic must remain available during load spikes because it protects scheduling decisions.
Publish allocatable capacity and active commitments
Raw capacity is not the same as allocatable capacity. Reserve resources for the operating system, runtime, monitoring, and failure margin. Report active leases and measured use. The scheduler should account for reservations before a workload starts so two clients do not claim the same slot. Capacity updates and lease creation must be coordinated atomically in the control plane.
Represent heterogeneous resources explicitly: CPU, RAM, accelerator type and memory, local disk, network class, architecture, runtime, and attached services. Boolean labels are useful for features, while quantities need units and precision. A generic “size: large” label can be offered as a convenience but should resolve to a versioned shape.
Schedule with constraints and failure domains
Filter nodes by hard requirements, policy, readiness, and current capacity. Score the remaining set for load, locality, warm caches, energy or price class, and failure-domain spread. Workloads with replicas should use anti-affinity across hosts, racks, zones, or providers. Tightly coupled workers may need affinity to a fast network or shared storage.
Return a decision reason and policy version to operators. When a job remains queued despite free aggregate capacity, the reason may be fragmentation, an incompatible accelerator, a region constraint, or a draining node. Exposing a stable reason code is more useful than showing a rapidly changing queue number.
Make maintenance a first-class state
Draining means the node accepts no new ordinary work while existing work completes or moves. The drain request should include deadline, eviction policy, and reason. Some workloads can checkpoint and relocate. Others need graceful completion. The control plane tracks which leases remain and whether the deadline is at risk. At the end, the node becomes maintenance or offline rather than automatically returning to ready.
Cordon and drain are different operations. Cordon stops new placement. Drain also initiates workload handling. Emergency quarantine is different again: it may immediately revoke access, stop tasks, and block logs or artifacts until investigation. Keep those actions separate so routine maintenance does not trigger incident behavior and incidents do not masquerade as maintenance.
Reconcile leases after partitions
Network partitions create uncertainty. The control plane may stop hearing from a node while the node continues running jobs. If the scheduler immediately reallocates stateful work, two copies may act at once. Use leases with fencing tokens. A replacement receives a newer token, and shared services reject writes from an older token. For jobs without shared side effects, retry may be simpler.
When connectivity returns, the node reports active leases and their tokens. The controller reconciles them against authoritative state. Unknown or expired work should be stopped. Do not let a stale node mark an already-replaced job as successful. Every state update must include the job, lease, node incarnation, and fencing value expected by the control plane.
A node-pool API surface
POST /node-enrollments
POST /nodes/{id}/admit
GET /nodes/{id}
POST /nodes/{id}/heartbeats
POST /nodes/{id}/cordon
POST /nodes/{id}/drain
POST /nodes/{id}/quarantine
POST /node-leases
POST /node-leases/{id}/renew
POST /node-leases/{id}/releaseAdministrative actions should require an expected version to prevent one operator from overwriting another. Heartbeat and lease endpoints can be optimized for machine traffic, but they still need authentication, bounded payloads, idempotency where appropriate, and protection against replay.
Observe the fleet as a pool
Track ready capacity, reserved capacity, active use, fragmentation, unhealthy resources, unreachable nodes, drain duration, lease expiry, placement failures, and work lost by failure domain. A fleet with ninety percent “healthy” nodes can still be unable to serve a rare accelerator shape. Pool-level health should answer whether expected request classes can be satisfied.
Logs and traces should connect node enrollment, policy changes, scheduling decisions, job starts, heartbeat gaps, and recovery. Use metrics with bounded labels and retain high-cardinality identifiers in logs. Alert on missing heartbeats, repeated incarnation changes, capability drift, clock anomalies, and nodes that report capacity inconsistent with active leases.
Assume a node can be compromised
Limit a node’s control-plane permissions. Validate every job callback against the assigned lease. Keep tenant secrets short-lived and scoped to one job. Network policy should prevent a worker from reaching control-plane administration or unrelated tenant services. Signed images, measured boot, sandboxing, and dedicated pools can add assurance depending on risk.
Quarantine automatically when integrity checks fail, impossible capacity appears, or the node runs an unapproved version. Automation should be reversible by an authorized operator with an audit trail, but it should fail safe. Marketplace payouts or credits should not be finalized until usage evidence passes reconciliation.
Nodes are actors, not rows
A node pool API is a distributed control system. Durable identity, admission, heartbeats, readiness, leases, fencing, draining, and reconciliation matter more than a polished list of servers. Design for delayed messages and partitions from the beginning. Make maintenance and quarantine explicit, and schedule only against verified, current capacity.
Continue with the Node & Endpoint Pools hub or connect this design to the compute pool scheduler guide.
