Kubernetes secrets are frequently used to securely store sensitive data, including passwords, API credentials, and certificates. In some cases, it may be necessary to compare secrets across several namespaces, clusters, or even two sets deployed by different systems.
In this article, we will look at easy and effective methods for analyzing two sets of Kubernetes secrets.
Problem in the wild
I can't claim that comparing Kubernetes secrets is part of the usual routine for DevOps or cloud engineers; however, it may be necessary for the following reasons:
Maintaining consistency across environments
- Teams may need to make a replica of the current environment for testing, including all configurations and secrets. In these cases, it's important to ensure that the two sets of secrets are equivalent in terms of structure and values.
- To avoid any structural difference during the deployment of an application to different environments (such as development, staging, and production), it's important to maintain consistency in underlying secret structures (not values). Otherwise, any differences could lead to security vulnerabilities or application failures (e.g., due to missing attributes).
Maintaining consistency across provisioners
- This use case frequently happens when two secret delivery systems are tested concurrently or when you migrate from one to the other (for example, from SealedSecrets to ExternalSecrets). To verify the secret provisioning mechanism, it is necessary to compare two sets of secrets that were deployed by different systems and represent the same K8S objects.
Versioning and updates
- During upgrades, you may need to ensure that the updated versions of secrets are properly propagated across the application. Comparing secrets helps to verify that no obsolete or stale secrets remain, reducing the risk of using invalid or expired credentials in the system.
Audit and security
- Secret comparison can also play a role in security audits, checking that sensitive information, such as passwords or API keys, hasn't been mistakenly exposed, updated, or leaked across environments. For example, production environments must always have an individual set of secrets.
To deal with any of these scenarios, a structural (keys and formats) or combination (structure and values) comparison of Kubernetes secrets may be required. Let's see how to do it.
How to compare Kubernetes secrets
To compare Kubernetes secrets, we need to perform the steps listed below, which we will implement as a script further:
- Retrieve a list of Kubernetes secrets with the first prefix (e.g.:
kvendingoldo1) using the Kubernetes API (kubectl). - Retrieve a list of Kubernetes secrets with the second prefix (e.g.:
alextest) using the Kubernetes API (kubectl). - Iterate through the first list, checking each secret:
- Attempt to find a matching secret in the second list.
- If found, compare both structure and values using tools like jq.
- Repeat the process for the second list to ensure completeness.
- Generate a report in the terminal, highlighting:
- Missing secrets
- Missing fields within secrets
- Differences in values
The basic implementation of the algorithm is something like this:
#!/bin/bash
#
# inputs
#
SECRET1_PREFIX="${1}"
SECRET1_NS="${2}"
SECRET2_PREFIX="${3}"
SECRET2_NS="${4}"
#
# ANSI color codes
#
COLOR_GREEN="\033[0;32m"
COLOR_YELLOW="\033[1;33m"
COLOR_RED="\033[0;31m"
COLOR_RESET="\033[0m"
function get_secrets_with_prefix() {
local prefix="${1}"
kubectl get secrets -n="${SECRET1_NS}" -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name"
}
function mask_value() {
local value=${1}
local length=${#value}
local mask_length=$((length / 2))
local visible_part=${value:2:mask_length}
echo "${visible_part}******"
}
function compare_secrets() {
local secret1=${1}
local secret2=${2}
# Get keys (structure) for both secrets
keys1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]')
keys2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]')
# Find common keys between the two secrets
common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d)
if [ -z "${common_keys}" ]; then
echo -e "No matching keys between ${secret1} and ${secret2}"
return
fi
# Compare values of each common key
for key in ${common_keys}; do
value1=$(kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)
value2=$(kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)
if [ "${value1}" != "${value2}" ]; then
echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}"
echo -e " - ${secret1}: $(mask_value ${value1})"
echo -e " - ${secret2}: $(mask_value ${value2})"
else
echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}"
fi
done
# Check for keys that are missing in one of the secrets
for key in ${keys1}; do
if ! echo -e "${keys2}" | grep -q "${key}"; then
echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}"
fi
done
for key in ${keys2}; do
if ! echo -e "${keys1}" | grep -q "${key}"; then
echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}"
fi
done
}
function main() {
secrets1=$(get_secrets_with_prefix "${SECRET1_PREFIX}")
secrets2=$(get_secrets_with_prefix "${SECRET2_PREFIX}")
for s1 in ${secrets1}; do
suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix
s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace
# Check if the corresponding secret exists
if echo -e "${secrets2}" | grep -q "${s2}"; then
echo -e "Comparing ${s1} and ${s2}:"
compare_secrets "${s1}" "${s2}"
echo -e
else
echo -e "No corresponding secret for ${s1} in ${SECRET2_PREFIX}."
fi
done
}
main "${@}"
As you can see, the basic script supports different Kubernetes clusters, but it’s not difficult to implement.Let's do it with easy modifications:
-
Add KUBECONFIG variables for each secret set
KUBECONFIG1="${1}" SECRET1_PREFIX="${2}" SECRET1_NS="${3}" KUBECONFIG2="${4}" SECRET2_PREFIX="${5}" SECRET2_NS="${6}" -
Add kubeconfig variable into
get_secrets_with_prefixfunctionfunction get_secrets_with_prefix() { local kubeconfig=${1} local prefix=${2} local namespace=${3} KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name" } -
Add
kubeconfig1andkubeconfig2variables tocompare_secrets
As a result, you will get the following script that can work with different clusters:
#!/bin/bash
#
# inputs
#
KUBECONFIG1="${1}"
SECRET1_PREFIX="${2}"
SECRET1_NS="${3}"
KUBECONFIG2="${4}"
SECRET2_PREFIX="${5}"
SECRET2_NS="${6}"
#
# ANSI color codes
#
COLOR_GREEN="\033[0;32m"
COLOR_YELLOW="\033[1;33m"
COLOR_RED="\033[0;31m"
COLOR_RESET="\033[0m"
function mask_value() {
local value=${1}
local length=${#value}
local mask_length=$((length / 2))
local visible_part=${value:2:mask_length}
echo "${visible_part}******"
}
function get_secrets_with_prefix() {
local kubeconfig="${1}"
local prefix="${2}"
local namespace="${3}"
KUBECONFIG=${kubeconfig} kubectl get secrets -n=${namespace} -o json | jq -r ".items[] | select(.metadata.name | startswith(\"${prefix}\")) | .metadata.name"
}
function compare_secrets() {
local kubeconfig1="${1}"
local kubeconfig2="${2}"
local secret1=""${3}"
local secret2="${4}"
# Get keys (structure) for both secrets
keys1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r '.data | keys[]')
keys2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r '.data | keys[]')
# Find common keys between the two secrets
common_keys=$(echo -e "${keys1}\n${keys2}" | sort | uniq -d)
if [ -z "${common_keys}" ]; then
echo -e "No matching keys between ${secret1} and ${secret2}"
return
fi
# Compare values of each common key
for key in ${common_keys}; do
value1=$(KUBECONFIG=${kubeconfig1} kubectl get secret -n="${SECRET1_NS}" "${secret1}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)
value2=$(KUBECONFIG=${kubeconfig2} kubectl get secret -n="${SECRET2_NS}" "${secret2}" -o json | jq -r ".data[\"${key}\"]" | base64 --decode)
if [ "${value1}" != "${value2}" ]; then
echo -e "${COLOR_YELLOW}Difference found in key '${key}':${COLOR_RESET}"
echo -e " - ${secret1}: $(mask_value ${value1})"
echo -e " - ${secret2}: $(mask_value ${value2})"
else
echo -e "${COLOR_GREEN}Key '${key}' is identical in both secrets.${COLOR_RESET}"
fi
done
# Check for keys that are missing in one of the secrets
for key in ${keys1}; do
if ! echo -e "${keys2}" | grep -q "${key}"; then
echo -e "${COLOR_RED}Key '${key}' is missing in ${secret2}.${COLOR_RESET}"
fi
done
for key in ${keys2}; do
if ! echo -e "${keys1}" | grep -q "${key}"; then
echo -e "${COLOR_RED}Key '${key}' is missing in ${secret1}.${COLOR_RESET}"
fi
done
}
function main() {
secrets1=$(get_secrets_with_prefix "${KUBECONFIG1}" "${SECRET1_PREFIX}" "${SECRET1_NS}")
secrets2=$(get_secrets_with_prefix "${KUBECONFIG2}" "${SECRET2_PREFIX}" "${SECRET2_NS}")
for s1 in ${secrets1}; do
suffix=${s1#$SECRET1_PREFIX} # Get the suffix by removing the prefix
s2="${SECRET2_PREFIX}${suffix}" # Construct the corresponding secret in the second namespace
# Check if the corresponding secret exists
if echo -e "${secrets2}" | grep -q "${s2}"; then
echo -e "Comparing ${s1} in Cluster1 and ${s2} in Cluster2:"
compare_secrets "${KUBECONFIG1}" "${KUBECONFIG2}" "${s1}" "${s2}"
echo -e
else
echo -e "No corresponding secret for ${s1} in Cluster2 (${SECRET2_PREFIX})."
fi
done
}
main "${@}"
Testing the script
To test the script, let's generate test data using another bash script. The script source code:
#!/bin/bash
#
# inputs
#
SECRET1_PREFIX="${1}"
SECRET1_NS="${2}"
SECRET2_PREFIX="${3}"
SECRET2_NS="${4}"
NUM_SECRETS=5 # Number of secrets to generate
NUM_KEYS=4 # Number of keys per secret
function generate_random_value() {
head -c 8 /dev/urandom | base64
}
function create_test_secrets() {
local prefix=${1}
local namespace=${2}
for i in $(seq 1 $NUM_SECRETS); do
secret_name="${prefix}test-secret-${i}"
data=""
if (( i % 3 == 1 )); then
# Generate identical values
data=" key1: $(echo -n 'fixedValue' | base64)\n key2: $(echo -n 'fixedValue' | base64)\n key3: $(echo -n 'fixedValue' | base64)"
elif (( i % 3 == 2 )); then
# Generate different values
data=" key1: $(generate_random_value)\n key2: $(generate_random_value)\n key3: $(generate_random_value)"
else
# Generate different keys
data=" keyA: $(generate_random_value)\n keyB: $(generate_random_value)"
fi
echo "Creating secret ${secret_name} in namespace ${namespace}"
echo -e "apiVersion: v1\nkind: Secret\nmetadata:\n name: ${secret_name}\n namespace: ${namespace}\ntype: Opaque\ndata:\n${data}" | kubectl apply -f -
done
}
function main() {
create_test_secrets "${SECRET1_PREFIX}" "${SECRET1_NS}"
create_test_secrets "${SECRET2_PREFIX}" "${SECRET2_NS}"
}
main "${@}"
To run it, do the following commands:
$ bash generate_test_data.sh kvendingoldo test-secrets-demo alextest test-secrets-demo$ bash compare_secrets.sh kvendingoldo test-secrets-demo alextest test-secrets-demo
Following that, you will see the output log, which shows that some secrets are identical while others have different values:
Conclusion
Comparing Kubernetes secrets is essential to maintaining consistency, security, and versioning in your application secrets. The approach described in this post is an excellent solution for rapid manual activities, which CloudOps/DevOps teams may need. You may need to modify the script a little bit for your task, but this should be easy thanks to the Bash language.
Feel free to experiment, and keep your secrets safeguarded!