Helm is being used broadly to deploy Kubernetes applications as it is an easy way to publish and consume them via a couple of commands, as well as integrate them in your GitOps pipeline. But is Helm secure enough? Can you trust it blindly?
This post explains the benefits of using Helm, the pitfalls, and offers a few recommendations for how to secure it. Let’s get started!
Why Helm?
Helm is a graduated open-source CNCF project originally created by DeisLabs. If you want to know more about how it works, we recommend you read the Helm 101 article of our Learning Cloud Native hub.
Let’s face it, managing an application lifecycle on Kubernetes is hard. If you want to just deploy an application, it requires at least a Deployment, usually tweaking the application configuration via modifications performed to a ConfigMap or a Secret, deploying CRDs if needed, and more. Clearly, it’s not really straightforward.
You can create your own deployment files with some environment variables and use envsubst
to substitute them at runtime (envsubst < deploy.yml | kubectl apply -f -
) to have a “DIY templating engine,” but that is probably not an optimal solution.
Kustomize improves the previous DIY solution but it has some limitations as well (it is mainly focused on templating, not packaging). Jsonnet can also be used for templating.
Helm is not perfect, but it tries to make that process easier by providing a simple command interface, a repository with more than 9000 charts available called Artifact Hub (and the ability to host your own charts on your own repository), and a templating engine (with over 60 available functions, mostly based on the Go template language). That allows you to package complex applications to make them easily deployable by just providing specific parameters.
For example, you can deploy a whole MySQL cluster with replication enabled (a non-trivial task, let’s be honest) by just using the architecture=replication
parameter.
It also has some advanced features, such as hooks (to run specific tasks at specific points on the deployment process such as ‘pre-install’), and can be integrated with GitOps tools, such as ArgoCD or Flux. You can leverage library charts or named templates, and even run post-render tasks (e.g., to run Kustomize).
How to secure Helm
We’ve covered a lot of ground, but we didn’t pay any attention to any security aspects and most charts are not secure by default.
There are a few angles to tackle depending on the process we want to cover. Are we just consuming the Helm charts, the Kubernetes objects created by the charts, or are we talking about custom Helm charts?
Custom Helm charts
If writing your own Helm charts, a few general recommendations apply, as well as some security focused ones:
- Store the charts in a Git repository. This may seem obvious in 2022, but Git will give you some benefits just by using it, such as easy rollbacks or the ability to track changes.
- Store the Helm charts in a proper repository. Charts can be served via HTTP but everything is HTTPS these days, right?
-
Use
helm lint
or any other linter you prefer to verify the Helm charts are properly formed. You don’t want to break the production environment for a silly typo.For example, in this basic Helm chart file without a proper
version
, Helm lint complains about it:Code language: JavaScript (javascript)apiVersion: v2 name: hello-world description: A Helm chart for Kubernetes type: application appVersion: "0.0.1" $ helm lint --strict ==> Linting . [ERROR] Chart.yaml: version is required [INFO] Chart.yaml: icon is recommended [ERROR] templates/: validation: chart.metadata.version is required [ERROR] : unable to load chart validation: chart.metadata.version is required Error: 1 chart(s) linted, 1 chart(s) failed
-
Use consistent versioning on your charts (Helm follows the SemVer2 standard). It is helpful for reproducibility and to be able to respond quickly in a situation where you need to update your charts because a vulnerability has been found. If your charts are unversioned or using “latest”, which one would you update?
There are two different versions you can use: the version of the chart itself (version
in theChart.yaml
file) and the version of the application (appVersion
).Code language: JavaScript (javascript)$ helm show chart falcosecurity/falco | grep -E '^version|^appVersion' appVersion: 0.33.0 version: 2.2.0
Don’t forget to Keep a Changelog (like Falco does).
-
Create test scenarios for your Helm charts to cover your use cases. The idea is to validate the success of the Helm deployment by creating Kubernetes objects (as in Helm templates) that will test your deployed chart by running
helm test <RELEASE_NAME>
. For example, a test can be just a simple pod running on the same namespace where your application has been deployed, that queries your application API to see if it has been deployed properly:
Code language: JavaScript (javascript)apiVersion: v1 kind: Pod metadata: … annotations: "helm.sh/hook": test spec: containers: - name: wget image: busybox command: ['wget'] args: ['{{ .Values.service.name }}:{{ .Values.service.port }}'] restartPolicy: Never
Usually, tests are stored in the templates/tests/
folder and are required to have the “helm.sh/hook": test
annotation to identify themselves as tests.
Code language: JavaScript (javascript)$ helm test hello-world NAME: hello-world ... Phase: Succeeded $ kubectl get po -n hello-world NAME READY STATUS RESTARTS AGE hello-world-78b98b4c85-kbt58 1/1 Running 0 91s hello-world-test 0/1 Completed 0 67s
-
Sign your charts easily with
helm package –sign
(and verify them withhelm install --verify
). Asserting the integrity of the software components is the most common task when securing the software supply chain. This usually means verifying a digital signature (either included with the software itself or close to it). Helm uses a PGP-based digital signature to create provenance records stored in provenance files (.prov
), which are stored alongside a packaged chart. Let’s see an example:
Code language: JavaScript (javascript)$ helm package --sign --key 'Eduardo Minguez' hello-world --keyring ~/.gnupg/secring.gpg Password for key "Eduardo Minguez (gpg key) <[email protected]>" > Successfully packaged chart and saved it to: /home/edu/git/my-awesome-stuff/hello-world-0.0.1.tgz
And this is what the provenance file looks like:
$ cat hello-world-0.0.1.tgz.prov -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 ... name: hello-world ... files: hello-world-0.0.1.tgz: sha256:b3f75d753ffdd7133765c9a26e15b1fa89784e18d9dbd8c0c51037395eeb332e -----BEGIN PGP SIGNATURE----- wsFcB… -----END PGP SIGNATURE-----%
If the signature doesn’t match, Helm will complain:
Code language: JavaScript (javascript)$ helm verify hello-world-0.0.1.tgz Error: openpgp: invalid signature: hash tag doesn't match
Running helm install --verify
will automatically check the provenance files:
Code language: JavaScript (javascript)$ helm install --verify myrepo/mychart-1.2.3
Or, you can just pull the chart and verify it:
Code language: JavaScript (javascript)$ helm pull --verify myrepo/mychart-1.2.3
The public key needs to be trusted beforehand for --verify
to work, so you must make it publicly available somewhere, otherwise it will fail:
Code language: JavaScript (javascript)$ helm pull --verify myrepo/mychart-1.2.3 Error: openpgp: signature made by unknown entity $ cat security/pubkey.gpg | gpg --import --batch $ helm pull --verify myrepo/mychart-1.2.3 Signed by:..
There is also a sigstore Helm plugin to use Rekor as signature storage, which is even better.
-
Automate all the previous steps (testing, versioning, signing, and releasing) in a CI/CD pipeline to make sure they are consistent with the best practices on every change, and to avoid potential problems when doing manual changes.
You can use the helm/charts-repo-actions-demo for inspiration on how to create a GitHub actions workflow to test and release a chart:
Code language: JavaScript (javascript)- name: Run chart-releaser uses: helm/[email protected] with: charts_dir: charts config: cr.yaml env: CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
Kubernetes objects
When creating the Kubernetes objects via templates, Helm doesn’t provide any security measures out of the box. You are on your own and can apply any bad practice you want, such as deploying a container with a root user or with full capabilities (OK, you may want to do it). Let’s talk about some recommendations:
-
Use Role-based access control (RBAC) to limit the object’s permissions (don’t use
cluster-admin
for everything). For example, the falcosidekick Helm chart creates a Role, ServiceAccount, and RoleBinding to minimize the required permissions used in the K8s Deployment:
Code language: JavaScript (javascript)--- apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "falcosidekick.fullname" . }} … --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: {{ include "falcosidekick.fullname" . }} … rules: - apiGroups: - "" resources: - endpoints verbs: - get … --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding … roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: {{ include "falcosidekick.fullname" . }} subjects: - kind: ServiceAccount name: {{ include "falcosidekick.fullname" . }} …
- Provide sane defaults. For example, if your chart includes a MySQL pod, don’t use a default password so any user could know about it. Instead, generate it randomly or force the user to specify it. However, there are a couple of things to consider, including how to deal with upgrades in this GitHub issue and this blog post. You can use the lookup function and the resource policy annotation to prevent overwriting when upgrading, as follows:
Code language: JavaScript (javascript){{- if not (lookup "v1" "Secret" .Release.Namespace "hello-world") }} apiVersion: v1 kind: Secret metadata: name: mysecret annotations: "helm.sh/resource-policy": "keep" type: Opaque stringData: password: {{ randAlphaNum 24 }} {{- end }}
- Reduce the attack surface. Make the deployment small and enable components and features using flags or values if needed. For example, the Falco Helm chart doesn’t deploy falcosidekick by default, but you can enable it easily:
Code language: JavaScript (javascript)falcosidekick: # -- Enable falcosidekick deployment. enabled: false
Which is then used in the Chart:
Code language: JavaScript (javascript)dependencies: - name: falcosidekick condition: falcosidekick.enabled
All the rest of Kubernetes recommendations apply (including the CIS benchmarks, for example), so make sure you scan your Kubernetes object definitions for best practices. If your preferred tool doesn’t support Helm charts, don’t worry. You can always render the Kubernetes objects in a previous step with the Helm template command, as follows:
Code language: JavaScript (javascript)$ helm template falco falcosecurity/falco --namespace falco --create-namespace --set driver.kind=ebpf > all.yaml
And then verify them as:
Code language: JavaScript (javascript)$ myawesometool --verify all.yaml
Using Helm charts
-
Don’t trust the Helm charts blindly, especially third-party ones. Fortunately, as we’ve seen, the
helm template
command renders and outputs the Kubernetes objects created by the Helm chart, so it is a good practice to at least take a quick look at the results before deploying it in your Kubernetes cluster. You probably don’t want to use joaquinito2051‘s charts. -
As explained before, use
helm verify
to check the digital signatures of the charts you use to make sure you are using the charts you are supposed to. - Uninstall unused releases: If you’re no longer using a Helm release, uninstall it to reduce your attack surface.
-
Always try to keep the Helm Charts you use updated (as well as the
helm
binary and their plugins!). Let’s face it, mistakes and bugs happen so it is a good idea to always use the latest version with the latest fixes for both the Helm chart itself or for the objects the Helm chart creates (for example, if it uses a container image that has been found vulnerable). This also applies to subcharts. There are a couple of options to verify what an upgrade will change, including using the helm diff plugin:
Code language: JavaScript (javascript)$ helm diff --install foo --set image.tag=1.14.0 .
Or the ability to render the manifests via helm template
and use kubectl
to make the diffs:
Code language: JavaScript (javascript)$ helm template --is-upgrade --no-hooks --skip-crds foo --set image.tag=1.14.0 . | kubectl diff --server-side=false -f -
However, there are some corner cases when using both approaches, and ideally you should check both to cover all the scenarios. See the kubectl and Helm diff challenges article for more details.
- Store Kubernetes secrets encrypted. Base64 is an encoding algorithm, not an encryption one, and despite its name, Kubernetes secrets are not secret. This is a convoluted topic to discuss because some folks prefer to store encrypted secrets along with the code, while others prefer to keep them in a different location. There are a few alternatives worth mentioning, including helm-secrets plugin, Hashicorp Vault, Bitnami’s Sealed secrets, Mozilla’s sops, DIY solutions, and more. Let’s see a simple example of helm-secrets with AWS SSM via vals.
Create an AWS SSM SecureString object:
Code language: JavaScript (javascript)$ aws ssm put-parameter --name mysecret --value "secret0" --type SecureString
Check the Helm parameter required. In this example, “secretdata
“:
Code language: JavaScript (javascript)$ cat hello-world/templates/secret.yaml apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque stringData: password: {{ .Values.secretdata }}
Verify it:
Code language: JavaScript (javascript)$ helm secrets --backend vals template hello-world -s templates/secret.yaml --set secretdata="ref+awsssm://mysecret" --- # Source: hello-world/templates/secret.yaml apiVersion: v1 kind: Secret metadata: name: mysecret type: Opaque stringData: password: secret0
- Finally, you can even run arbitrary commands right after rendering the chart, like:
Code language: JavaScript (javascript)$ helm install mychart my-chart --post-renderer my-script.sh
Where the my-script.sh
script can be mostly everything, including running kustomize to apply environment variables, verifying a specific parameter has not been used, a script that calls a webhook to get some data, Windows batch scripts, and more! Your imagination is the limit!
Conclusion
Helm is a useful tool to manage the Kubernetes applications lifecycle. While the security aspect is not enforced by default, this article has covered some best practices and security recommendations for consuming and creating Helm charts.
The post How to secure Helm appeared first on Sysdig.
Article Link: How to secure Helm – Sysdig