Ai2 changes GPU scheduling incentives

Ai2 has published an October 9 account of GPU scheduling changes rolled out from late July. Managers allocate time budgets through a research hierarchy. A seven day lookback favors groups below their shares. Budgeted jobs receive a declared minimum runtime of up to eight hours before becoming interruptible. Work outside a budget can use spare GPUs but receives no protection from preemption.

Its internal measurements show teams received 98% of the GPU hours owed over 30 days, with entitlement capped by actual demand. Cluster occupancy remained 98%. Debugging jobs saw their 90th percentile wait drop from two hours to 30 seconds, although the earlier sample was smaller and more variable.

Ai2 reports disrupted interactive sessions and is investigating whether protected jobs fragment capacity for large runs. Reported results concern budget compliance and faster access. Funding priorities reflect management judgment. Whether that produces better science requires separate evaluation.

Fairness depends on the hierarchy

Slurm’s Fair Tree documentation provides a useful comparison. The algorithm became its default fair share method in version 19.05. It ranks accounts by assigned shares relative to recorded usage, then carries that ordering down the account tree. Users beneath a higher ranked sibling account receive higher fair share factors than users beneath a lower ranked one.

For a research organization, that makes the budget tree consequential. A project’s queue position can reflect the allocation and consumption of its wider branch. Teams evaluating such a policy should ask who can adjust those shares when an experiment needs more time. Slurm also documents other priority weights alongside fair share. Administrators therefore need to inspect the complete priority policy before treating a share as a promise about an exact start time.

Slurm also restarts requeued batch scripts from their beginning. For jobs that accumulate valuable state, that makes saving and restoring progress an important part of planning for interruption.

A guarantee needs an interruption policy

Apache Hadoop’s Fair Scheduler documentation helps explain why a resource entitlement can still leave a job waiting. Queues can have guaranteed minimum shares, and spare capacity is distributed to other applications when those shares are unused. But an application arriving below its minimum might still wait because existing work holds the resources. Configurable preemption timeouts determine when queues can reclaim capacity.

Apache exposes separate waiting thresholds for queues below their minimum share and below a chosen fraction of their fair share. Administrators can decide how quickly an imbalance should trigger intervention. Waiting longer gives existing work more time to finish. Acting sooner lets another queue reclaim resources earlier. For a team evaluating a policy, those settings belong alongside the headline allocation. A useful test reports how often resources were reclaimed and which classes of application spent longer waiting.

Measure the work inside the allocation

NVIDIA’s DCGM documentation separates several device signals. Streaming multiprocessor activity counts time with at least one active warp, including warps waiting on memory. Tensor activity measures how often the tensor pipeline is active. Memory bandwidth utilization tracks cycles transferring data to or from device memory. These measure different parts of execution, so a single busy percentage cannot explain whether a training job is spending its allocation effectively.

There is a terminology trap, too. NVIDIA’s streaming multiprocessor occupancy describes resident warps relative to the hardware maximum. It is a device metric with a different denominator from a scheduler’s assigned time. NVIDIA explicitly warns that higher occupancy does not always mean more effective GPU use.

For teams testing a scheduling change, the implication is to pair access metrics with workload progress and execution telemetry. Keep the experiment, GPU type and observation period comparable. A shorter queue is valuable, but the stronger result is more completed useful work from the same available hardware.

ByteForward’s reporting on Oxide examines the related question of how hardware choices and resource limits affect usable capacity.