Introduction

In the world of Kubernetes, resource management is more than just creating, deleting, or updating objects. It's an intricate dance involving numerous tools, operators, and users. As our infrastructure grows, it becomes increasingly challenging to maintain control, necessitating the adoption of more advanced and sophisticated resource management and control systems and approaches.

In this article, I will not delve into the basic methods of creating resources, as that is a rather trivial task. Instead, I would like to share my experience in optimizing resource update paths, which have proven to be immensely valuable in managing resources within large and complex Kubernetes clusters, as well as during the development of operators.

Update

"How do I update a resource?" This is often the first question that comes to mind for every developer, DevOps engineer, or anyone who has interacted with Kubernetes.

And the first answer that comes to mind is UPDATE - or, in Kubernetes terminology, APPLY.

This approach is entirely correct.

The kubectl apply command is an incredibly powerful and convenient tool: you simply modify the desired part of the resource manifest and apply it, and Kubernetes handles the rest.

Let's apply the following deployment manifest as an example:

> cat test-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpa-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gpa-app
  template:
    metadata:
      labels:
        app: gpa-app
    spec:
      containers:
        - name: gpa-app
          image: nginx:1.21.0
          ports:
            - containerPort: 80

> kubectl apply -f test-deployment.yaml

However, the complexity comes when we want to automate the regular checking and updating of a resource, for example, by encapsulating the logic within a microservice.

In such a scenario, you always need to keep the source manifest to introduce changes and apply them. Storing the manifest directly within the microservice isn't an ideal solution for several reasons:

A more robust solution is to implement a logic that first retrieves (GET) the current state of the resource's manifest, updates the necessary fields, and then applies the changes back to the cluster:

> kubectl get deployment gpa-app -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ...
  creationTimestamp: "2025-08-25T23:43:37Z"
  generation: 1
  name: gpa-app
  namespace: default
  resourceVersion: "495672"
  uid: 2dec1474-988d-431b-9cf2-e8f41a624517
spec:
  replicas: 1
  ...
  selector:
    matchLabels:
      app: gpa-app
  ...
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: gpa-app
    spec:
      containers:
      - image: nginx:1.21.0
        imagePullPolicy: IfNotPresent
        name: gpa-app
        ports:
        - containerPort: 80
          protocol: TCP
        ...
      ...
status:
  ...


> cat test-deployment-upd.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpa-app
spec:
  replicas: 2   # new replicas count
  selector:
    matchLabels:
      app: gpa-app
  template:
    metadata:
      labels:
        app: gpa-app
    spec:
      containers:
        - name: gpa-app
          image: nginx:1.21.0
          ports:
            - containerPort: 80


> kubectl apply -f test-deployment-upd.yaml

And this is a very popular solution that's sufficient for most use cases.

However, we're discussing large, complex, and high-load systems where an extra request to the Kubernetes API can be an expensive operation. In the example above, we are making two separate requests every time: a GET followed by an APPLY.

To delve even deeper into this situation, let's consider the presence of microservices or other systems that subscribe to and react to resource events. In this scenario, we would constantly be spamming the cluster with events, even when no actual changes are being made to the resource.

This situation does not apply to the standard kubectl apply command because the utility itself performs a validation of the applied manifest by using information stored in the resource's annotations. Specifically, it uses the kubectl.kubernetes.io/last-applied-configuration annotation to intelligently determine what has changed and send only the necessary updates.

One of the most straightforward solutions is to first check for changes and then apply the new manifest only if the resource has genuinely been modified.

To summarize everything discussed above, the logic for implementing a microservice or an operator for resource updates should be as follows:

Let's name this approach GET-CHECK-APPLY

Collaborative Resource Management

Everything we've discussed so far represents a simple and elegant solution for a scenario where a microservice/user manages a resource. But what if multiple contributors are involved in modifying that same resource?

This brings us to the main topic of this article: how to resolve the challenges of shared resource management politely and diplomatically.

The first obvious step is to distribute attribute ownership among the contributors. For example, one service might be responsible for watching and updating the image, while another manages the number of replicas.

“service-a”

> cat test-deployment-service-a.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpa-app
spec:
  ...
  template:
    ...
    spec:
      containers:
        - name: gpa-app
          image: nginx:1.21.0 # belongs to service-A
          ...

“service-b”

> cat test-deployment-service-b.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: gpa-app
spec:
  replicas: 3  # belongs to service-B
  ...

Unfortunately, the GET-CHECK-APPLY approach will not be effective in this scenario. Since it operates on the entire resource manifest, a collision can occur when multiple services are working concurrently. Specifically, between the GET and APPLY steps of one service, another service might apply its own changes, which would then be overwritten by the first service's final APPLY.

Patch as a solution for collaborative resource management

The most straightforward and obvious path to solve the problem of collaborative resource management is to use PATCH. This approach works well for two main reasons:

> cat test-deployment-service-a.yaml
spec:
  replicas: 3

> kubectl patch deployment gpa-app --patch-file test-deployment-service-a.yaml


> cat test-deployment-service-b.yaml
spec:
  template:
    spec:
      containers:
        - name: gpa-app
          image: nginx:1.22.0

> kubectl patch deployment gpa-app --patch-file test-deployment-service-b.yaml


> cat kubectl get deployment gpa-app -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ...
  generation: 4  # one of the indicators that the resource has changed
  resourceVersion: "520392"
  name: gpa-app
  ...
spec:
  replicas: 3  # changed replicas count by service-A
  ...
  template:
    ...
    spec:
      containers:
      - image: nginx:1.22.0 # changed image by service-B
        ...
      ...
status:
  ...

Unfortunately, we still can't abandon the GET-CHECK step. Like APPLY, PATCH also triggers a resource version change, which generates an event and creates noise, bothering our "neighbors" (other services and systems).

As a result, we've found that GET-CHECK-PATCH is more convenient than GET-CHECK-APPLY for collaborative work`.

However, despite these improvements, this logic still feels quite cumbersome:

In Kubernetes circles, this GET-CHECK-PATCH(APPLY) approach is known as Client-Side Apply (CSA), where all the logic for merging, conflict resolution, and validation is performed on the client side, and only the final result is applied.

While the client has significant control over the resource management process, many tools remain unavailable to it. For example, a client cannot prevent another from overwriting a set of fields that it owns.

Kubernetes SSA

Starting with Kubernetes v1.22, a highly effective and powerful declarative mechanism was introduced: Server-Side Apply (SSA).

SSA significantly simplifies collaborative resource management by moving the responsibility for updating, validating, and consolidating logic to the API server itself. The client only sends the desired state of the resource, and the Kubernetes API server handles all the complex logic under the hood.

A key feature introduced with SSA is the mechanism of shared field management. The Kubernetes API server now knows which client is managing which field within a resource's specification. When a client sends a manifest using SSA, the API server checks if that client owns the fields it's trying to modify. If the field is unowned or already belongs to that client, the change is applied successfully. However, if another client owns the field, the API server will return an error, alerting you to a conflict or overwriting the owner based on your settings.

SSA usage completely eliminates the need to use the GET-CHECK-PATCH(APPLY) approach. With SSA, you send the desired state, specify the client's name (field manager), and you receive the server's response.

It's important to note that using PATCH instead of APPLY of the entire manifest is still a best practice, as it allows your service to "claim" ownership of only the specific fields it manages.

We can use the same patch files from the previous example, changing the replicas and image, and apply them using SSA.

> kubectl patch deployment gpa-app --patch-file test-deployment-service-a.yaml --field-manager=service-a
> kubectl patch deployment gpa-app --patch-file test-deployment-service-b.yaml --field-manager=service-b


> kubectl get deployment gpa-app -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ...
  generation: 6  # one of the indicators that the resource has changed
  resourceVersion: "534637"
  name: gpa-app
  ...
spec:
  replicas: 1  # changed replicas count by service-A
  ...
  template:
    ...
    spec:
      containers:
      - image: nginx:1.21.0 # changed image by service-B
        ...
      ...
status:
  ...

To view a list of all managed fields, you need to expand the command kubectl get, by adding the --show-managed-fields flag

> kubectl get deployment gpa-app -o yaml --show-managed-fields
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    ...
  ...
  managedFields:
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:replicas: {}   # confirmation that the spec.replicas belong to service-a
    manager: service-a
    operation: Update
    time: "2025-08-26T00:23:50Z"
  - apiVersion: apps/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        f:template:
          f:spec:
            f:containers:
              k:{"name":"gpa-app"}:
                f:image: {}  # confirmation that the spec.template.spec.containers.[0].image belong to service-b
    manager: service-b
    operation: Update
    time: "2025-08-26T00:24:05Z"
  ...
  name: gpa-app
  ...
spec:
  ...

As you've seen, Kubernetes has "claimed" the replicas field for "service-a" and the image field for "service-b".

This is the core of SSA's field management. If you now try to override the entire manifest again with SSA, Kubernetes will return an error because it detects a conflict.

> kubectl apply -f test-deployment.yaml --field-manager=service-c --server-side
error: Apply failed with 1 conflict: conflict with "service-a" using apps/v1: .spec.replicas
Please review the fields above--they currently have other managers. Here
are the ways you can resolve this warning:
* If you intend to manage all of these fields, please re-run the apply
  command with the `--force-conflicts` flag.
* If you do not intend to manage all of the fields, please edit your
  manifest to remove references to the fields that should keep their
  current managers.
* You may co-own fields by updating your manifest to match the existing
  value; in this case, you'll become the manager if the other manager(s)
  stop managing the field (remove it from their configuration).
See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts

It correctly identifies that you, as the new applicant, do not own the fields that are already claimed by "service-a" and "service-b." This behavior is a key advantage of SSA, as it prevents unintentional overwrites and ensures that shared resources are updated collaboratively and safely.

Conclusion

By diving deep into Kubernetes resource management, it becomes clear that the evolution from Client-Side Apply to Server-Side Apply is not just a command-line change. The SSA is a fundamental shift in the philosophy of interacting with a cluster. While SSA may seem like a silver bullet, it does have its complexities, requiring a deeper understanding of Kubernetes architecture for a successful implementation.

For a long time, CSA was our reliable companion. It gets the job done but comes with certain limitations. Its reliance on the kubectl.kubernetes.io/last-applied-configuration annotation makes it vulnerable to conflicts and errors, especially in complex, automated environments. In the hands of a single developer, CSA can be an effective tool for quick, simple operations. However, as soon as multiple systems or individuals try to manage the same resource simultaneously, its fragility becomes obvious. CSA can lead to unpredictable results, race conditions, and, as a consequence, cluster instability.

SSA solves these problems by moving complex logic to the API server. The field ownership management feature is a game-changer. The API server is no longer just executing commands; it becomes an intelligent arbiter that knows exactly who is responsible for which fields. This makes collaboration safe by preventing accidental overwrites and conflicts. For developers building operators and controllers, SSA is not just an option but a necessity. It unleashes the creation of robust and scalable systems that can coexist within the same cluster without interfering with each other.

So, when should you use each approach?

Ultimately, understanding these two approaches is key to working effectively and without errors in Kubernetes. By choosing Server-Side Apply, you're not just using a new command; you're adopting a modern, reliable, and more innovative way to manage your infrastructure.

Thank you!