I’ve recently done a Twitter poll and only 20% of the participants accurately predicted that it takes Kubernetes 60-90 seconds to propagate changes to Secrets and ConfigMaps on the mounted volumes. So I want to take you on a journey in the codebase on how the mechanics of these volume types work and why it takes so long.

Before going on this journey, I would answer the poll “nearly instantly” (like the majority 40% did). After all, I know there’s a kubelet setting called configMapAndSecretChangeDetectionStrategy to control how kubelet detects changes to Secrets/ConfigMaps, and its documentation reads:

configMapAndSecretChangeDetectionStrategy is a mode in which ConfigMap and Secret managers are running:

  • Get: kubelet fetches necessary objects directly from the API server;
  • Cache: kubelet uses TTL cache for object fetched from the API server;
  • Watch: kubelet uses watches to observe changes to objects that are in its interest.

Default: Watch.

and the fact that Watch is the default on Kubernetes, it made me think that kubelet probably finds out a Secret is updated through a “watch”, and then rebuilds the volume in reaction to the event.

This assumption is incorrect. This setting above only controls “how kubelet finds the contents of the Secret/ConfigMap” and has nothing to do with how/when the mounted volume will be updated (or rather, rebuilt, since it uses an atomic write).

So what’s this configMapAndSecretChangeDetectionStrategy setting is for then? It is merely a scalability configuration as establishing a watch on individual objects put more load on the kube-apiservers, and some users may benefit from “Cache” setting with a long TTL better if they run too many Pods per Node, or mount too many Secrets per Pod etc.

With that in mind, I think we can paraphrase the options above in a way that sounds more accurate:

  • Get: every time kubelet needs the ConfigMap/Secret contents, it will make a GET request
  • Cache: same as above, but the results are cached for cacheTTL;
  • Watch: kubelet maintains a “watch” for each referenced ConfigMap/Secret, so it already has the object contents without making an API request when it needs.

In other words, when you use a Secret/ConfigMap volume, kubelet already knows the most up-to-date contents of that object since it’s watching resource [code] —it’s just not reflecting the updated value to the Pod. Why?

What’s taking so long?

It turns out there’s no mechanism in the kubelet’s secret manager or configmap manager that informs the volume reconciliation logic to re-run when these resources update.

Due to the lack of a notification channel, the secret/configMap volumes are only updated is when pod is “synced” in the syncPod function which gets called under these circumstances:

  1. something about the pod changes (pod event, update to Pod object on the API, containe lifecycle change on the container runtime) —which do not happen very often while the pod is running, and

  2. periodically, roughly every minute or so.

That periodical syncing of the Pod is what’s updating the secret/configMap volumes here: kubelet periodically requeues the Pod [code] after 60-90 seconds. (Exactly how long it takes Secret/ConfigMaps updates to be reflected to the volumes). This delay is calculated from syncInterval kubelet setting (default: 60 seconds) plus a constant jitter factor of 0.5 (so, it’s 1.0-1.5 x syncInterval).

If you want to observe this delay, see my proof of concept repository, which shows this 60-90s delay clearly.

Can we make the updates go out faster? It turns out, yes. Remember the #1 bullet point above? What if we trigger some change to the Pod status to generate an event to get our pod to the “pod sync loop” quicker?

For example, if you modify Pod object annotations on the API, it won’t restart the Pod, but it will immediately add the Pod to the reconciliation queue. Then, the secret volume will be immediately re-calculated and the change will be picked up in less than a second. (But as you can imagine, updating annotations is not practical, as most users don’t manage Pod objects directly.)

How the secret/configMap volumes are built?

During the syncPod function, many things are actuated or double-checked about the pod, including calling into the volumemanager to ensure Pod’s volumes are attached and mounted [code]. The reconciliation in the volumemanager calls MountVolume if the volume requires a “remount”. [code]

As you might expect, both configmap volume plugin and secret volume plugin indicates such volumes “require remount”.

Due to this choice, mounted secret/configMap volumes are recalculated every minute or so. (This doesn’t cause any filesystem updates if the Secret/ConfigMap object contents have not changed, since the AtomicWriter compares the volume contents to the files on disk. [code])

So this is roughly the mechanics behind how pods get their secret/configMap volume contents updated periodically and how it’s completely disconnected from the update event that happens on these objects.

Conclusion

I went into this journey because I mistakenly recalled Kubernetes was able to reflect Secret/ConfigMap updates nearly instantly back in the 1.0 release or so (and that’s what the “Watch” mode in the kubelet is for). It turns out this was never correct.

Furthermore, the fact that such volume updates are delayed is actually called out in both Secret updates and [ConfigMap updates] documentation (more clearly in the latter):

The kubelet checks whether the mounted ConfigMap is fresh on every periodic sync. However, it uses its local TTL-based cache for getting the current value of the ConfigMap. As a result, the total delay from the moment when the ConfigMap is updated to the moment when new keys are projected to the pod can be as long as kubelet sync period (1 minute by default) + TTL of ConfigMaps cache (1 minute by default) in kubelet.

I don’t think there’s inherently a need for kubelet to pick up secret/configMap updates nearly instantly, though introducing that notification channel may not have any downsides, other than possibly destabilizing an already complex codebase.

Also, using mutable Secrets is a thing of the past: Updating Secrets on a live system makes it harder to reason about the system and can easily cause outages. Though, kubelet updating Secret/ConfigMap volumes periodically has neat use cases like picking up rotated TLS certificates from the filesystem conveniently.

I hope you enjoyed reading this. If this kind of this is interesting to you, you might also enjoy my write-up on how Kubernetes updates Secret/ConfigMap volumes atomically.