Applying changes to annotated Kubernetes resources

Quick disclaimer, I work for Google but this is not a Google sponsored or affiliated blog post. The following blog is a short small solution to to a specific situation.

tl;dr — You need to find resources in kubernetes that have a specific annotation and apply a change to them. Using a combination ofjsonpath, tr, xargs and kubectl, commands can be applied to KRM resources with a specific annotation. This blog might be a bit verbose on the output, by my intention is to explain both how and why to a diverse audience.

Where this “solution” fits on the spectrum

Annotations in Kubernetes should be used for identifiable information about the specific resource. Due to the cardinality, annotations are not indexed and are not directly search or sortable, but this does not mean that there are not cases where searching for resources with a specific annotation is not desirable.

One example is changing the ownership of all services developed by a specific team. (hint: best practices for annotations includes adding an annotation for the team that owns a particular service).

Note: In this example, one could still argue a case for using labels, given you have a small number of teams, the cardinality would be low and therefore a target for labels. Let’s just go with what we have here, or if you need, assume this the “owner” is an individual and you have 10k developers in your company.

Kubectl does not provide a way to search by annotations, so our approach will be to query for the resource type we want to target, then filter responses using jsonpath. Note, this is “post-processing” and not a fast operation, so use this technique/hack for less frequent queries (ie, don’t make this a production-level or frequently run command).

Here’s the plan: Query for resource type, filter those responses into individual names, apply some string manipulation (optionally) then pipe into a command.

1. First, we need to query for all resources of the target type:

○ → kubectl get svc
NAME                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
echo-service-app-app   ClusterIP   192.168.1.2    <none>        8080/TCP   20h
my-app-name-app        ClusterIP   192.168.1.46   <none>        8080/TCP   44h

2. Filter the results using jsonpath. This took a long time to get right as jsonpath is kinda a pain. The purpose for using jsonpath versus jq or other applications is to reduce requirements for use within an individual environment or in a CICD pipeline (ex: gcr.io/cloud-builders/gke-builders:stable does not have jq installed by default).

Here is the jsonpath used:

# Replace "TARGET_ANNOTATION" with your annotation
{range.items[?(@.metadata.annotations.TARGET_ANNOTATION)]}{.metadata.name}{':'}{end}

To query for ANY service that has an annotation of “owner”:

# Query for ANY service with an annotation of "owner"
○ → kubectl get svc -o jsonpath="{range .items[?(@.metadata.annotations.owner)]}{.metadata.name}{‘:’}{end}"
echo-service-app-app:my-app-name-app:

To query for ANY service that is owned by “team-one@acme.com”:

# Query for ANY service with owner='team-one@acme.com'
○ → kubectl get svc -o jsonpath="{range .items[?(@.metadata.annotations.owner == 'team-one@acme.com')]}{.metadata.name}{‘:’}{end}"
echo-service-app-app:

Couple of comments so far. First, the delimiter used in this example is “:”. As I will explain a the bottom, the delimiter might need to change based on the availability of tooling. Second, the delimiter is appended to the end of the name with the {':'} in the above examples, therefore there is no way to only append if there is a “next” token, so there will always be an appended delimiter at the end of the output.

3. Now to apply those results to a command. We need to assign those values to something. In this case, we will use a bash variable. It’s possible to write to a temp file or even chain further with pipe redirections.

export SVC_WITH_OWNERS=$(kubectl get svc -o jsonpath=”{range .items[?(@.metadata.annotations.owner)]}{.metadata.name}{‘:’}{end}”)
# Note the $(...) in bash to take the results of the command and place into the variable

4. Use the output as input for a command. This is ultimately completed using the xargs command. The xargs command MAY have different functionality for different containers or environments (see note below). The key feature we will exploit is the -I {} switch. This allows us to place the input anywhere we need in the command. The other is the -d \#DELIMITER# that looks for a delimiter from the input and splits up tokens. We can take the output of echo $SVC_WITH_OWNERSand use as input to xargs which then runs the command kubectl annotate svc <INPUT> owner=SOME_NEW_TEAM@acme

echo $SVC_WITH_OWNERS | xargs -d \: -I {} kubectl annotate svc {} owner=SOME_NEW_TEAM@acme.com

6. Grab a drink, you’re done!…well…There are a few small flaws that need cleaning up. You might have noticed that your output includes an extra command run for the “empty” service/token. This is because there is an extra \n appended to the string and xargs sees this as a token to split on. To remove this, we need to change the token output. We can use trto remove the \n

echo $SERVICENAME | tr -d '\n' | xargs ...omitted for brevity...

7. So, we’re fully done and want to run this in a CICD pipeline. Well, again, not yet. Many containers derived from busybox have a limited feature set when it comes to included tools. xargs on busybox containers does not have -d \:switch available, so you’re limited to only \n. Good news, there’s one more simple fix to get around the limitation. Again, tr to the rescue. We can replace the : with \0 (NULL character) and then take advantage of xarg -0 which converts NULL character as a delimiter. The result looks like:

echo $SERVICENAME | tr -d '\n' | tr ':' '\0' | xargs -0 -I {}...

8. Are we done now? Yes, pretty much…the final solution looks like this:

export SVC_WITH_OWNER=$(kubectl get svc -o jsonpath="{range .items[?(@.metadata.annotations.owner)]}{.metadata.name}{':'}{end}")
echo $SVC_WITH_OWNER | tr -d '\n' | tr ':' '\0' | xargs -0 -I {} kubectl annotate svc {} owner=NEW_TEAM@acme.com

To wrap up, it is possible to apply changes to kubernetes resources with specific annotations using jsonpath and a small group of linux-based tools. Keep in mind, the solution is a “post-processing” filter, not a reducing query set and annotations are not indexed, therefore this could be an expensive operation.