Labels and Annotations

This page serves as a reference for all labels and annotations in Kueue.

kueue.x-k8s.io/admission-gated-by

Type: Annotation

Example: kueue.x-k8s.io/admission-gated-by: "example.com/mygate,example.com/mygate2"

Used on: Kueue-managed Jobs.

This annotation requires the AdmissionGatedBy feature that is disabled by default.

When it contains a value, the annotation prevents the Job from entering the Quota Reservation phase of Kueue’s Admission Flow. The annotation value is a comma-separated list of 1 or more gate names and can only be added during Job creation. After creation, the annotation may only be deleted or modified to remove 1 or more gates.

kueue.x-k8s.io/cluster-queue-name

Type: Label

Example: kueue.x-k8s.io/cluster-queue-name: "my-cluster-queue"

Used on: Pods.

This label requires AssignQueueLabelsForPods feature that is enabled by default.

The label key in all Pods created by Kueue workloads or managed by Kueue. It indicates which cluster queue admitted the Kueue workload that the Pod corresponds to. It can be used, for example to aggregate the actual resource usage (cpu/mem) coming from workloads admitted to a given cluster queue.

kueue.x-k8s.io/component-workload-index

Type: Annotation

Example: kueue.x-k8s.io/component-workload-index: "0"

Used on: Workload.

The annotation stores the numeric index for component workloads created by multi-workload jobs (e.g., LeaderWorkerSet creates one workload per replica). Used by MultiKueue to determine primary workload ordering when dispatching component workloads to worker clusters atomically.

kueue.x-k8s.io/elastic-job

Type: Annotation

Example: kueue.x-k8s.io/elastic-job: "true"

Used on: Kueue-managed Jobs and Pods.

The annotation key present on Jobs that support workload slicing. To enable workload slicing for a given Job, both the key and the value "true" must be set.

The same key is also used as a scheduling gate name applied to Pods to delay their scheduling until the associated workload slice has been admitted.

This annotation is alpha-level for the ElasticJobsViaWorkloadSlices feature gate.

kueue.x-k8s.io/is-group-workload

Type: Annotation

Example: kueue.x-k8s.io/is-group-workload: "true"

Used on: Workload.

The annotation key indicates that this workload is used for a group of Pods.

kueue.x-k8s.io/job-completions-equal-parallelism

Type: Annotation

Example: kueue.x-k8s.io/job-completions-equal-parallelism: "true"

Used on: batch/Job.

The annotation key is used to keep completions and parallelism in sync.

kueue.x-k8s.io/job-min-parallelism

Type: Annotation

Example: kueue.x-k8s.io/job-min-parallelism: "5"

Used on: batch/Job.

The annotation key indicates the minimum parallelism acceptable for the job in the case of partial admission.

kueue.x-k8s.io/job-owner-gvk

Type: Annotation

Example: kueue.x-k8s.io/job-owner-gvk: "leaderworkerset.x-k8s.io/v1, Kind=LeaderWorkerSet"

Used on: Workload.

The annotation holds the GVK (GroupVersionKind) of the owner job. Used for MultiKueue adapter lookup in workloads with multiple owner references (e.g., LeaderWorkerSet). This is an annotation rather than a label because the GVK string format contains characters invalid in label values.

kueue.x-k8s.io/job-owner-name

Type: Annotation

Example: kueue.x-k8s.io/job-owner-name: "my-leaderworkerset"

Used on: Workload.

The annotation holds the name of the owner job. Used when the owner reference has been removed by Kubernetes garbage collection but the job name is still needed for MultiKueue operations.

kueue.x-k8s.io/job-uid

Type: Label

Example: kueue.x-k8s.io/job-uid: "46ef6b23-a7d9-42b1-b0f8-071bbb29a94d"

Used on: Workload.

The label key in the workload resource holds the UID of the owner job.

kueue.x-k8s.io/local-queue-name

Type: Label

Example: kueue.x-k8s.io/local-queue-name: "my-local-queue"

Used on: Pods.

This label requires AssignQueueLabelsForPods feature that is enabled by default.

The label key in all Pods created by Kueue workloads or managed by Kueue. It indicates to which local queue a workload corresponding to the Pod has been assigned. In the case of managed Pods, the value is equal to the name of the local queue to which the Pod has been assigned.

It can be used, for example to aggregate the actual resource usage (cpu/mem) coming from workloads assigned to a given local queue.

kueue.x-k8s.io/managed

Type: Label

Example: kueue.x-k8s.io/managed: "true"

Used on: Plain Pods and ProvisioningRequest.

The label key that indicates which Pods and ProvisioningRequests are managed by Kueue.

kueue.x-k8s.io/max-exec-time-seconds

Type: Label

Example: kueue.x-k8s.io/max-exec-time-seconds: "120"

Used on: Kueue-managed Jobs.

The value of this label is passed in the Job’s Workload spec.maximumExecutionTimeSeconds and used by the Maximum execution time feature.

kueue.x-k8s.io/multikueue-origin

Type: Label

Example: kueue.x-k8s.io/multikueue-origin: "true"

Used on: MultiKueue.

The label key is used to track the creator of MultiKueue remote objects in Worker Cluster.

kueue.x-k8s.io/multikueue-origin-uid

Type: Annotation

Example: kueue.x-k8s.io/multikueue-origin-uid: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"

Used on: MultiKueue.

The annotation stores the UID of the original object from the management cluster on remote objects in worker clusters. Used by job types (e.g., LeaderWorkerSet) that generate workload names based on their UID, ensuring workload names match between management and worker clusters.

kueue.x-k8s.io/pod-group-fast-admission

Type: Annotation

Example: kueue.x-k8s.io/pod-group-fast-admission: "true"

Used on: Plain Pods.

The annotation key is used to allow admitting a PodGroup as soon as the first Pod in the group is created.

kueue.x-k8s.io/pod-group-name

Type: Label

Example: kueue.x-k8s.io/pod-group-name: "my-pod-group-name"

Used on: Plain Pods.

The label key indicates the name of the group of Pods that should be admitted together.

kueue.x-k8s.io/pod-group-pod-index

Type: Label

Example: kueue.x-k8s.io/pod-group-pod-index: "0"

Used on: Plain Pods.

The label key indicates the Pod’s index within the Pod group it belongs to.

kueue.x-k8s.io/pod-group-pod-index-label

Type: Annotation

Example: kueue.x-k8s.io/pod-group-pod-index-label: "apps.kubernetes.io/pod-index"

Used on: Plain Pods.

The annotation key indicates a label name used to retrieve the Pod’s index within the group.

kueue.x-k8s.io/pod-group-serving

Type: Annotation

Example: kueue.x-k8s.io/pod-group-serving: "true"

Used on: Plain Pods.

The annotation key is used to indicate whether the Pod group is being used as a serving workload.

kueue.x-k8s.io/pod-group-total-count

Type: Annotation

Example: kueue.x-k8s.io/pod-group-total-count: "2"

Used on: Plain Pods.

The annotation key is used to indicate how many Pods to expect in the group.

kueue.x-k8s.io/pod-index-offset

Type: Annotation

Example: kueue.x-k8s.io/pod-index-offset: "1"

Used on: Kueue-managed Jobs (e.g., MPIJob).

The annotation indicates a starting index offset for Pod replicas within a PodSet. For example, when an MPIJob uses runLauncherAsWorker mode, Worker replica indexes start from 1 instead of 0. This annotation is set by the Kueue webhook and used by the TAS topology ungater to correctly map Pod indexes to topology assignments.

kueue.x-k8s.io/pod-suspending-parent

Type: Annotation

Example: kueue.x-k8s.io/pod-suspending-parent: "deployment"

Used on: Plain Pods.

The annotation key is used to indicate the integration name of the Pod owner.

kueue.x-k8s.io/podset

Type: Label

Example: kueue.x-k8s.io/podset: "main"

Used on: Kueue-managed Jobs.

The label key is used on the Job’s PodTemplate to indicate the name of the PodSet of the admitted Workload corresponding to the PodTemplate. The label is set when starting the Job, and removed on stopping the Job.

kueue.x-k8s.io/podset-group-name

Type: Annotation

Example: kueue.x-k8s.io/podset-group-name: "workers"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation indicates the name of the group of PodSets. A PodSet Group is a unit of flavor assignment and topology domain fitting.

Used with Topology Aware Scheduling to group PodSets that should share the same topology domain and flavor assignment.

kueue.x-k8s.io/podset-preferred-topology

Type: Annotation

Example: kueue.x-k8s.io/podset-preferred-topology: "cloud.provider.com/topology-rack"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation indicates that a PodSet requires Topology Aware Scheduling, but scheduling all pods on nodes within the same topology domain is a preference rather than a requirement.

The levels are evaluated one-by-one going up from the level indicated by the annotation. If the PodSet cannot fit within a given topology domain then the next topology level up is considered. If the PodSet cannot fit at the highest topology level, then it gets admitted as distributed among multiple topology domains.

kueue.x-k8s.io/podset-required-topology

Type: Annotation

Example: kueue.x-k8s.io/podset-required-topology: "cloud.provider.com/topology-block"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation indicates that a PodSet requires Topology Aware Scheduling, and requires scheduling all pods on nodes within the same topology domain corresponding to the topology level indicated by the annotation value (e.g. within a rack or within a block).

kueue.x-k8s.io/podset-slice-required-topology

Type: Annotation

Example: kueue.x-k8s.io/podset-slice-required-topology: "cloud.provider.com/topology-rack"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation indicates that a PodSet requires Topology Aware Scheduling, and requires scheduling each PodSet slice on nodes within the topology domain corresponding to the topology level indicated by the annotation value (e.g. within a rack or within a block).

kueue.x-k8s.io/podset-slice-required-topology-constraints

Type: Annotation

Example: kueue.x-k8s.io/podset-slice-required-topology-constraints: '[{"topologyLevel":"cloud.provider.com/topology-rack","sliceSize":4}]'

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation contains a JSON-encoded array of topology constraints for multi-level slice subdivision. Each entry specifies a topology level and slice size.

This annotation is alpha-level for the TASMultiLayerTopology feature gate.

kueue.x-k8s.io/podset-slice-size

Type: Annotation

Example: kueue.x-k8s.io/podset-slice-size: "4"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation describes the requested size of a PodSet slice for which Kueue finds a requested topology domain.

This annotation is required if kueue.x-k8s.io/podset-slice-required-topology is defined.

kueue.x-k8s.io/podset-unconstrained-topology

Type: Annotation

Example: kueue.x-k8s.io/podset-unconstrained-topology: "true"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation indicates that a PodSet does not have any topology requirements. Kueue admits the PodSet if there is enough free capacity available.

Recommended for PodSets that don’t need low-latency or high-throughput pod-to-pod communication, but want to leverage Topology Aware Scheduling capabilities to improve accuracy of admitting jobs.

kueue.x-k8s.io/prebuilt-workload-name

Type: Label

Example: kueue.x-k8s.io/prebuilt-workload-name: "my-prebuilt-workload-name"

Used on: Kueue-managed Jobs.

The label key of the job holds the name of the pre-built workload to be used. The intended use of prebuilt workload is to create the Job once the workload is created. In other scenarios the behavior is undefined.

kueue.x-k8s.io/priority-boost

Type: Annotation

Example: kueue.x-k8s.io/priority-boost: "10"

Used on: Workload.

An optional signed integer that adjusts a workload’s effective priority: effectivePriority = workloadPriority + priorityBoost. Positive values increase priority; negative values decrease it. The effective priority is used for both scheduling order and preemption candidate ordering. This annotation is intended to be set directly on Workloads by external controllers (not propagated from Jobs). If the value is missing or empty, it is treated as 0. If the value is invalid, Workload create or update is rejected by the admission webhook.

kueue.x-k8s.io/priority-class

Type: Label

Example: kueue.x-k8s.io/priority-class: "my-priority-class-name"

Used on: Kueue-managed Jobs.

The label key in the workload holds the workloadPriorityClass name. This label is always mutable, as it may be useful for preemption. For more details, see Workload Priority Class.

kueue.x-k8s.io/queue-name

Type: Label

Example: kueue.x-k8s.io/queue-name: "my-local-queue"

Used on: Kueue-managed Jobs.

The label key in the workload holds the queue name.

kueue.x-k8s.io/retriable-in-group

Type: Annotation

Example: kueue.x-k8s.io/retriable-in-group: "false"

Used on: Plain Pods.

The annotation key is used to finalize the group if at least one terminated Pod (either Failed or Succeeded) has the retriable-in-group: false annotation.

kueue.x-k8s.io/role-hash

Type: Annotation

Example: kueue.x-k8s.io/role-hash: "b54683bb"

Used on: Plain Pods.

The annotation key is used as the name for a Workload podSet.

kueue.x-k8s.io/safe-to-forcefully-delete

Type: Annotation

Example: kueue.x-k8s.io/safe-to-forcefully-delete: "true"

Used on: Pods.

The annotation key controls whether a Pod has opted in to FailureRecoveryPolicy. When the value is "true", Kueue may forcefully delete the Pod as part of the failure recovery process.

kueue.x-k8s.io/stopping

Type: Annotation

Example: kueue.x-k8s.io/stopping: "true"

Used on: batch/Job.

The annotation is used internally during the Job stop process. When Kueue suspends a Job, it sets this annotation to "true" to track that the stop operation is in progress. The annotation is removed once all stop operations complete successfully.

kueue.x-k8s.io/workload

Type: Annotation

Example: kueue.x-k8s.io/workload: "my-workload-name"

Used on: Kueue-managed Jobs (Job PodTemplate).

The annotation is set on the Job’s PodTemplate to indicate the name of the admitted Workload corresponding to the Job. The annotation is set when starting the Job, and removed on stopping the Job.

kueue.x-k8s.io/workload-slice-name

Type: Annotation

Example: kueue.x-k8s.io/workload-slice-name: "my-workload-original"

Used on: Workload and Pods.

The annotation identifies the original workload name in a slice chain. It is set on every Workload created in the chain of workloads, as well as on the Pods associated with that Workload.

This annotation is alpha-level for the ElasticJobsViaWorkloadSlices feature gate.

kueue.x-k8s.io/workload-slice-replacement-for

Type: Annotation

Example: kueue.x-k8s.io/workload-slice-replacement-for: "default/my-old-workload-slice"

Used on: Workload.

The annotation is set on a new workload slice to indicate the key (namespace/name) of the workload slice it is intended to replace (i.e., the old slice being preempted).

This annotation is alpha-level for the ElasticJobsViaWorkloadSlices feature gate.

provreq.kueue.x-k8s.io/

Type: Annotation prefix

Example: provreq.kueue.x-k8s.io/MyParameter: "my-value"

Used on: Kueue-managed Jobs, Workload, and ProvisioningRequest.

This is a prefix for annotations that are passed to a ProvisioningRequest as Parameters. Any annotation with this prefix on the Job or Workload will be forwarded to the corresponding ProvisioningRequest.