KEP-5598: Opportunistic batching

Implementation History
BETA Implementable
Created 2025-10-01
Latest v1.36
Milestones
Beta v1.35
Stable v1.37
Ownership
Owning SIG
SIG Scheduling
Primary Authors
@bwsalmon @romanbaron

KEP-5598: Opportunistic batching

Release Signoff Checklist

Items marked with (R) are required prior to targeting to a milestone / release.

  • (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
  • (R) KEP approvers have approved the KEP status as implementable
  • (R) Design details are appropriately documented
  • (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
    • e2e Tests for all Beta API Operations (endpoints)
    • (R) Ensure GA e2e tests meet requirements for Conformance Tests
    • (R) Minimum Two Week Window for GA e2e tests to prove flake free
  • (R) Graduation criteria is in place
  • (R) Production readiness review completed
  • (R) Production readiness review approved
  • “Implementation History” section is up-to-date for milestone
  • User-facing documentation has been created in kubernetes/website , for publication to kubernetes.io
  • Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes

Summary

This KEP proposes an opportunistic batching mechanism in the scheduler to improve performance of scheduling many compatible pods at once, and to begin building the infrastructure required for gang scheduling. To implement this mechanism we propose the following additions:

  • Pod scheduling signature: A signature that captures the properties of a pod that impact scoring and feasibility.
  • Batching mechanism: A mechanism to reuse the scheduling output from one pod to provide node hints for multiple subsequent pods with matching scheduling signatures.
  • Opportunistic batching: Transparent inclusion of the batching mechanism in the scheduler to improve the performance of targeted workloads that could benefit from it.

Motivation

Today our scheduling algorithm is O(num pods x num nodes). As the size of clusters and jobs continue to increase, this leads to low performance when scheduling or rescheduling large jobs. This increases user cost and slows down user jobs, both unpleasant impacts. Optimizations like this one have the potential to dramatically reduce the cost of scheduling in these scenarios.

We are also working on gang scheduling (in addition to other forms of multi-pod scheduling), which will give us a way to consider multiple pods at the same time. “Opportunistic batching” provides a starting point for these mechanisms by providing signatures and batching, both necessary foundational mechanisms, and including them initially in a simple way.

Another change is the shift towards 1-pod-per-node in batch and ML environments. Many of these environments (among others) only attempt to run a single user pod on each node, along with a complement of daemon set pods. This simplifies our scheduling needs significantly, as it allows to reuse not only filtering, but also scoring results.

Goals

  • Improve the performance of scheduling large jobs on large clusters where the constraints are simple.
  • Begin building infrastructure to support gang scheduling and other “multi-pod” scheduling mechanisms.
  • Ensure that the infrastructure we build is maintainable as we update, add and remove plugins.
  • Provide improved performance for a targeted set of workloads in this release.
  • Provide a path where we can expand batching to apply to most or all workloads over the next few releases.
  • Allow users to continue to use out-of-tree plugins. For this KEP we need to ensure that out-of-tree plugins continue to work without requiring edits, although they may not be able to take advantage of the new feature without some edits.

Non-Goals

  • We are not attempting to apply this optimization to all pods in this release. We will make the addition of batching transparent, but only applicable to a reduced set of workloads in this KEP.
  • We are not adding gang scheduling of any kind in this KEP. This is purely a performance improvement without adding dependency on the Workload API KEP-4671 , although we hope the work on this KEP will help us with gang scheduling as we build it.

Proposal

We discuss each of the added items: pod scheduling signature, batching mechanism and opportunistic batching in turn.

Pod scheduling signature

The pod scheduling signature is used to determine if two pods are “the same” from a scheduling perspective. In specific, what this means is that any pod with the given signature will get the same scores / feasibility results from any arbitrary set of nodes. Also, assigning a pod to a node will not change the feasibility or scoring of other nodes. This is necessary for the cache to work, since we need to be able to reuse the previous work.

Note that some pods will not have a signature, because the scoring uses not just pod and node attributes, but other pods in the system, global data about pod placement, etc. These pods get a nil signature, and we fall back to the existing path.

To allow non in-tree plugins to construct a signature, we add a new framework interface to implement. Each plugin returns a set of signature fragments that capture the pod attributes relevant to that plugin’s scheduling decisions. To construct a full signature, the framework collects fragments from all plugins, and marshals them into a single PodSignature (a byte slice). If any plugin cannot generate a signature for a given pod (because it depends on information other than the pod and node), it returns an error status, and we generate a nil signature for that pod, skipping batching.

If an enabled plugin that does Scoring, Prescoring, Filtering or Prefiltering does not implement this interface, batching is turned off for all pods.

The signature interface and types are defined as follows:

// A portion of a pod signature. The sign fragments from all plugins are combined
//  to create a unified signature.
type SignFragment struct {
	// Key identifies this fragment. Fragments with the same key should contain
	// the same value for the same pod.
	Key string

	// Value must be JSON-marshallable.
	Value any
}

// The signature for a given pod after all fragments are consolidated.
type PodSignature []byte

// SignPlugin is an interface that should be implemented by plugins that either filter
// or score pods to enable batching and gang scheduling optimizations.
type SignPlugin interface {
	Plugin
	// SignPod returns SignFragments for this pod.
	//
	// Return values:
	//   - Success: plugin can sign the pod, returns signature fragments
	//   - Unschedulable: plugin cannot sign pod (pod not eligible for batching)
	//   - Error: unexpected failure (pod not eligible for batching, error logged)
	SignPod(ctx context.Context, pod *v1.Pod) ([]SignFragment, *Status)
}

Batching mechanism

The second component of this KEP is a batching mechanism. The batching mechanism provides two main operations that are invoked during the scheduling cycle:

  • GetNodeHint: Returns a node hint for a pod with a valid signature by validating that cached scheduling results can be reused.
  • StoreScheduleResults: Stores the sorted scheduling results from a “canonical” pod for potential reuse with subsequent matching pods.

GetNodeHint

The GetNodeHint operation is called during the filtering phase, after PreFilter but before evaluating individual nodes, to determine if we can reuse cached scheduling results from a previous pod. It takes a pod with a signature and attempts to provide a node hint that will allow the scheduler to take a fast path.

Before returning a hint, the operation validates that the cached batch state is compatible with the current pod by checking:

  1. Cycle continuity: The current scheduling cycle must be exactly one greater than the last cycle (no other pods were scheduled).
  2. Signature match: The pod’s signature must exactly match the cached signature.
  3. Cache freshness: The cached data must be sufficiently recent to avoid relying on stale scheduling decisions.
  4. Last node is full: The node chosen in the previous scheduling cycle must now be infeasible for the new pod. This is verified by running filter plugins against that node. This validation ensures the 1-pod-per-node constraint that allows us to reuse scoring results without rescoring.

If all validations pass, the operation pops the next best node from the cached sorted list and returns it as a hint (see “Integration with Scheduling Cycle” below for how the hint is used).

If any validation fails, the batch state is invalidated (with the reason recorded in metrics), no hint is provided, and the scheduler proceeds with normal full evaluation of all nodes.

StoreScheduleResults

The StoreScheduleResults operation is called after a pod has been scheduled (after filtering, scoring, and node selection). It stores scheduling results for potential reuse with subsequent pods.

The operation first records information about the last scheduling cycle (cycle number and chosen node) for use in the next cycle’s validation.

Then it determines whether to store new batch state:

If a hint was used (hintedNode == chosenNode):

  • This means the batch worked! We successfully reused cached results.
  • No new batch state is stored; we continue using the existing batch.
  • Statistics are recorded (batchedPods counter incremented).

If no hint was provided or the hint was not used:

  • If the pod has a valid signature and there are remaining nodes in the sorted list, new batch state is created containing:
    • The pod’s signature
    • The sorted list of remaining feasible nodes
    • Creation timestamp (for the expiration check)
  • If the pod has no signature or no remaining nodes, no batch state is stored.

The batch state is kept in memory only and is constrained to a short-lived validity window to prevent stale data from affecting scheduling decisions.

Integration with Scheduling Cycle

The GetNodeHint operation returns a hint string (node name) that is plugged directly into the scheduling cycle.

During the scheduling cycle, if a node was hinted by the batching mechanism, the scheduler evaluates that specific node first before iterating over all nodes. This “try one node first” is the fast path. If the hinted node passes all filters, it is immediately returned as the only feasible node, bypassing evaluation of all other nodes and scoring entirely.

If the hinted node fails filtering, the scheduler falls back to the normal path of evaluating all nodes. This ensures correctness while providing significant performance benefits when the hint is valid. The batching mechanism can be used in multiple places, including future gang scheduling implementations, without requiring changes to the Pod API.

Opportunistic batching

We will then apply the batching mechanism to simple cases in the current code. We will target providing incremental value with minimal code changes in this KEP, and leave the more complex integration questions to gang scheduling. This involves using the same batch state (and snapshot and potentially plugin cyclestate) across multiple pods using the batching mechanism.

Notes/Constraints/Caveats (Optional)

Risks and Mitigations

Plugins need to keep signatures up to date

The cache will only work if plugin maintainers are able to keep their portion of the signature up-to-date. We believe this should be doable because the logic is put into the plugin interface itself, and we are restricting it to portions of the pod spec, but there is still risk of subtle dependencies creeping in.

If plugin changes prove to be an issue, we could codify the signature as a new “Scheduling” object that only has a subset of the fields of the pod. Plugins that “opt-in” could only be given access to this reduced scheduling object, and we could then use the entire scheduling object as the signature. This would make it more or less impossible for the signature and plugins to be out of sync, and would naturally surface new dependencies as additions to the scheduling object. However, as we expect plugin changes to be relatively modest, we don’t believe the complexity of making the interface changes is worth the risk today.

We are narrowing the feature set where batching will work

Because we are explicitly limiting the functionality that this cache will support, we run the risk of designing something that will not work for enough users for it to be useful. To mitigate this risk we are actively engaging with users and doing analysis of data available on K8s users to ensure we are still capturing a large enough number of user use cases. We also will address this by expanding over time; we expect to have a few interested parties up front, but will then evaluate expansions that could onboard more.

We don’t have experience with batching in production

Because we haven’t deployed batching in production before, we are still somewhat limited in the information we have about user workloads. To mitigate this concern we will build in tracing / analytics to help us understand how frequently we see specific patterns, how often we are able to batch, and the most common reasons we are unable to batch. When possible we will collect this information even when the feature itself is disabled, to allow us to approach our next iterations with more data.

Design Details

Pod signature

A pod scheduling signature is a hash of the pod’s scheduling requirements. It is used to identify pods that can be scheduled together. To optimize the scheduling cycle, the signature is calculated and cached when a pod first enters the scheduling queue. By pre-calculating the signature during the queuing phase, the scheduler can avoid doing extra work during the time-sensitive scheduling process and helping the system handle larger batches of pods more smoothly.

The following section outlines the attributes we are currently proposing to use as the signature for each of the plugins in the scheduler. We need the plugin owners to validate that these signatures are correct, or help us find the correct signature.

Note that the signature does not need to be stable across versions, or even invocations of the scheduler. It only needs to be comparable between pods on a given running scheduler instance.

  • DynamicResources: For now we mark a pod unsignable if it has dynamic resource claims. We should improve this in the future, since most DRA claims are node specific and we should be able to determine this with a little effort. We will attempt to pull forward at least some integration of simple DRA claims with batching into this version as well.
  • ImageLocality: We use the canonicalized image names from the Volumes as the signature.
  • InterPodAffinity: If either the PodAffinity or PodAntiAffinity fields are set, the pod is marked unsignable, otherwise we need to include the pod labels in the signature.
  • NodeAffinity: We use the NodeAffinity and NodeSelector fields, plus any defaults set in configuration as the signature.
  • NodeName: We use the NodeName field as the signature.
  • NodePorts: We use the results from util.GetHostPorts(pod) as the signature.
  • NodeResourcesBalancedAllocation: We use the output of calculatePodResourceRequestList as the signature.
  • NodeResourcesFit: We use the output of the computePodResourceRequest function as the signature.
  • NodeUnschedulable: We use the Tolerations field as the signature.
  • NodeVolumeLimits: We use all Volume information except from Volumes of type ConfigMap or Secret.
  • PodTopologySpread: If the PodTopologySpead field is set, or it is not set but a default set of rules are applied, we mark the pod unsignable, otherwise it returns an empty signature. Because the plugin itself is creating the signature, it knows whether and what kind of default it will apply.
  • TaintToleration: We use the Tolerations field as the signature.
  • VolumeBinding: Same as NodeVolumeLimits.
  • VolumeRestrictions: Same as NodeVolumeLimits.
  • VolumeZone: Same as NodeVolumeLimits.

Test Plan

[X] I/we understand the owners of the involved components may require updates to existing tests to make this code solid enough prior to committing the changes necessary to implement this enhancement.

Prerequisite testing updates
Unit tests
Coverage of existing packages

Will add an extra function and test for plugins we touch.

  • k8s.io/kubernetes/pkg/scheduler: 2025-10-7 - 86.1
  • k8s.io/kubernetes/pkg/scheduler/framework: 2025-10-7 - 51.8
  • k8s.io/kubernetes/pkg/scheduler/framework/runtime: 2025-10-7 - 84.3
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/dynamicresources: 2025-10-7 - 80.5
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/imagelocality: 2025-10-7 - 86.2
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/interpodaffinity: 2025-10-7 - 89.7
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodeaffinity: 2025-10-7 - 85.8
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodename: 2025-10-7 - 50
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodeports: 2025-10-7 - 83.7
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/noderesources: 2025-10-7 - 89.5
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodeunschedulable: 2025-10-7 - 87.1
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/nodevolumelimits: 2025-10-7 - 73.7
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/podtopologyspread: 2025-10-7 - 87.8
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/tainttoleration: 2025-10-7 - 86.9
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumebinding: 2025-10-7 - 83.9
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumerestrictions: 2025-10-7 - 74
  • k8s.io/kubernetes/pkg/scheduler/framework/plugins/volumezone: 2025-10-7 - 84.8
New unit tests

The code draft has first versions of most of these, will add more as we get through the discussion process.

  • schedule_one_test.go - Add test cases for opportunistic batching.
  • signature_test.go - Test cases for the framework signature call and the helper class
  • signature_consistency_test.go - Test cases to ensure the signature captures all the necessary information. We will take a range of pod specs and node definitions, run them through the filtering / scoring code, then ensure that the pods with matching signatures always get equivalent results.
  • batching_test.go - Test cases for the batching mechanism, separate from the actual integration into the scheduling pipeline.
Integration tests

Will add a few integration tests:

  • Perf tests: Add a few tests into scheduler_perf that look at performance with batching enabled and disabled for a few target scenarios.
  • End-to-end consistency: Add tests that run a set of pods through the scheduler end-to-end with batching enable and disabled. Ensure the scheduling decisions are the same. Hopefully use same pod spec and node definitions from the signature_consistency_test.
e2e tests
  • We will run the existing scheduling e2e tests with batching enabled and disabled, to ensure they pass in both cases.
  • We will also add e2e tests ensuring that pod configurations we expect to be batched are in fact batched.

Graduation Criteria

Beta

  • Feature implemented behind a feature flag
  • Initial signature implementations for all in-tree plugins (note that some, as described in the section, will always return unsignable if the pod is configured to use them).
  • Monitoring
  • Hand-done perf test runs
  • Integration tests
  • Initial e2e tests completed and enabled
  • Handle common 1-pod-per-node batches: host ports and resources
  • Parameter tuning (batch sizes, etc.)
  • Excluded: batching for non “1-pod-per-node” workloads

GA

  • At least 1 test user with experience running the feature

Upgrade / Downgrade Strategy

Users should continue to see the same behavior, just with better performance. If the feature has bugs, they can use the feature gates to disable it.

Users should be able to take advantage of batching without any change to their behavior, other than ensuring the feature gate is enabled. Batching will not speed up all workloads, but workloads it can improve will be improved transparently.

Version Skew Strategy

This feature should be localized to the scheduler. So long as the scheduler is correctly built, we should not require other interactions from components in the system. Scheduler plugins will need to implement new methods to take advantage of the feature, but if they do nothing the feature will simply end up disabled.

Production Readiness Review Questionnaire

Feature Enablement and Rollback

How can this feature be enabled / disabled in a live cluster?
  • Feature gate (also fill in values in kep.yaml)
    • Feature gate name: SchedulerOpportunisticBatching
    • Components depending on the feature gate: kube-scheduler
Does enabling the feature change any default behavior?

No, it should not. Batching will improve the performance of some workloads, but should be transparent otherwise.

Can the feature be disabled once it has been enabled (i.e. can we roll back the enablement)?

Yes, it can be disabled. Because it only keeps in-memory state, setting the flag to false and restarting the scheduler should clear any previous state.

What happens if we reenable the feature if it was previously rolled back?

This feature only maintains in-memory, in-flight state, so changing the feature gate, which restarts the scheduler, should not cause issues with a running system.

Are there any tests for feature enablement/disablement?

Not needed, as described above.

Rollout, Upgrade and Rollback Planning

How can a rollout or rollback fail? Can it impact already running workloads?

Rollout can fail if the feature is faulty, causing pods to either no schedule or schedule incorrectly.

What specific metrics should inform a rollback?

Existing metrics:

  • pod_scheduling_sli_duration_seconds
  • schedule_attempts_total - specifically unschedulable and error cases
  • pending_pods
  • unschedulable_pods

New metrics:

  • Pods that cannot be batched.
  • Pod batch failure reasons
Were upgrade and rollback tested? Was the upgrade->downgrade->upgrade path tested?

Upgrade and downgrade should be simple due the feature being in-memory. But we will test the path before GA.

Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?

No.

Monitoring Requirements

How can an operator determine if the feature is in use by workloads?
  • We will log statistics about how often pods are batched vs not batched.
How can someone using this feature know that it is working for their instance?

Operator can query metrics noting how many pods are batched or not. This will be a new metric added by this KEP.

What are the reasonable SLOs (Service Level Objectives) for the enhancement?

We should continue to maintain the existing scheduler SLOs.

What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
  • Metrics
  • Metric name: pod_scheduling_sli_duration_seconds
    • Components exposing the metric: kube-scheduler
Are there any missing metrics that would be useful to have to improve observability of this feature?

We will add new metrics to identify behavior. This includes:

  • Batched vs non-batched pods - Identify how often a pod can be batched vs not.
  • Counts of non-batching reasons - This can include plugin signatures, feasibility checks, etc.

Dependencies

Does this feature depend on any specific services running in the cluster?

No.

Scalability

Will enabling / using this feature result in any new API calls?

No.

Will enabling / using this feature result in introducing new API types?

No.

Will enabling / using this feature result in any new calls to the cloud provider?

No.

Will enabling / using this feature result in increasing size or count of the existing API objects?

No.

Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?

No, it should result in decreased time for scheduling operations.

Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, …) in any components?

No, it should not.

Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

No, it should not.

Troubleshooting

How does this feature react if the API server and/or etcd is unavailable?

The same as the scheduler does today.

What are other known failure modes?

  • [Increased pod scheduling latencies]

    • Detection: pod scheduling latencies rise.
    • Mitigations: turn off the opportunistic batching feature.
    • Diagnostics: look for batching failures (in metrics), look for batching related log messages.
    • Testing: No, because it is unclear why specifically this would happen.

  • [Pods scheduled on incorrect nodes]

    • Detection: pods on nodes where they should not exist (affinity rules, etc)
    • Mitigations: turn off the opportunistic batching feature.
    • Diagnostics: look for batching metrics, look for batching related log messages.
    • Testing: yes, we will run tests to catch these kinds of issues before rolled out to production.
What steps should be taken if SLOs are not being met to determine the problem?
  • Check to see if pods are being batched.
  • If they are being batched, look at latencies for batched pods vs non
  • If nominated nodes are failing feasibility, check if these pods were batched.

Implementation History

Drawbacks

Alternatives

Comparison with Equivalence Cache (circa 2018)

This KEP is addressing a very similar problem to the Equivalence Cache (eCache), an approach suggested in 2018 and then retracted because it became extremely complex. While this KEP addresses a similar problem it does so in a very different way, which we believe avoids the issues experienced by the eCache

The issues experienced by eCache were:

  • eCache performance was still O(num nodes).
  • eCache was complex
  • eCache was tightly coupled with plugins.

We’ll address each in turn, but at a high level the differences stem from our scope reduction in this cache, where we focus on simple constraints in a 1-pod-per-node world, and are comfortable extending our “race” period slightly.

eCache performance was still O(num nodes)

The eCache was caching a fundamentally different result than this cache. In the case of the eCache they were caching the results of a predicate p, (which is sounds like was one of a number of ops for a given plugin) for a specific pod and node. This meant the number of cache lookups per pod was O(num nodes * num predicates) where num predicates was O(num plugins). Because the cache was so fine-grained, the cache lookups were, in many cases, more expensive than the actual computation. This also meant that while the cache could improve performance, it fundamentally did not remove the O(num nodes) nature of the per pod computation.

In the case of this cache, we are looking up the entire host filtering and scoring for a single pod, so the number of cache lookups per pod is 1. We are caching the entire filtering / scoring result, so the map lookup is guaranteed to be faster even than just iterating over the plugins themselves, let alone the computation needed to filter / score. As the number of nodes go up, the fact that the cache lookup is O(1) per pod will make it an increasingly perfromant alternative to the full computation.

We can cache this more granular data because we only cache for simple plugins, and in fact avoid the complex plugins entirely. Thus we do not need to be concerned about cross pod dependencies, meaning we do not need to try to keep detailed information up-to-date. Because we assume 1-pod-per-node and some amount of “staleness” we simply need to invalidate whole hosts, rather than requiring upkeep of complex predicate results required to keep the eCache functional.

eCache was complex

Because the eCache cached predicates, the logic for computing these results went into the cache as well. This meant that significant amount of the plugin functionality was replicated in the cache layer. This added significant complexity to the cache, and also made keeping the cache results themselves up to date complex, involving multiple pods, etc. Because the eCache only improved performance for complex queries, it needed to include this complexity to provide value.

In contrast, the signature used in this cache is just a subset of the pod object, without complex logic. It is static and as the pod object changes slowly, it will change slowly as well. In addition, we explicitly avoid all the complex plugins in this cache because they are rarely used. Thus we do not have the same complexity needed in the cache.

eCache was tightly coupled with plugins

Because a significant amount of the plugin complexity made into the eCache, it was difficult for plugin owners to keep the things in sync. Since in this cache the signature is just parts of the pod object, and the pod object is fairly stable, this makes keeping the signature up to date a much simpler task. The creation of the signature is also spread across the plugins themselves, so instead of needing to keep the cache up to date, plugin owners simply have a new function they need to manage within their plugin, which the cache only aggregates.

We will also provide tests that evaluate different pod configurations against different node configurations and ensure that any time the signatures match the results do as well. This will help us catch issues in the future, in addition to providing testing opportunities in other areas.

See https://github.com/kubernetes/kubernetes/pull/65714#issuecomment-410016382 as starting point on eCache.

Future work

Today we have the ability to determine if a given node would still be feasible after we added a specific pod to it. This is powerful and will be used by this feature. However, we do not have the same capability when it comes to scoring. Adding this capability would make it much easier for us to do batching (and many other things) on a wider range of workloads. This work is not required for this KEP, but would increase the number of use cases where we could apply batching.

Infrastructure Needed (Optional)