Programmatically restart pods in a Kubernetes cluster with Python
I’ve been using Google Cloud Kubernetes (GKE) in staging for just over three months for a client project. It’s incredible how far this technology has come.
But this is not a critique of Kubernetes. It doesn’t need one, just look at which cloud companies support Kubernetes - hint, it’s all of them. (Azure, IBM, Amazon, Google)
There is a learning curve, especially since SSH is not the method of deployment or communication. When done correctly, like the way that Google authenticates has integrated authentication with kubectl - it’s a lot more powerful than anything I’ve used before. It reduces security risks by a factor since there is no chance of SSH key’s going out of sync or a colleague that left before you joined a year ago key is found on an machine that was somehow left off the ssh targets - it happens.
There are a few tools that I find indispensable for using Kubernetes:
- kex. An amazing tool that allows you to quickly run a command on a pod. It is prompt based, asking first which namespace and pod to use. Check out the gif on the site.
- kpoof Another great tool from farmotive. Allows you to quickly create a tunnel from a pod to a port on your machine. Great for debugging connection issues.
- kubernetes-deploy Although this is written in Ruby, it’s a great tool to automate deployment and running of once-off containers. This tooling saved me countless hours and is production ready. They use it at Shopify. I just wish there was a Python port.
I also like the CRON jobs that can be setup in Kubernetes. I had been using Django and Celery with the Database backed scheduler, but it kept failing at the oddest of times.
My CRON jobs in Kubernetes
I migrated them in a morning and have not had a problem since. I’m still using Celery, but all the tasks are kicked off by a Kubernetes CRON job that just enqueues the jobs in Celery.
Sorry, I got sidetracked. Back to the issue at hand, how to programmatically restart pods in a cluster.
Firstly, if you didn’t know, Kubernetes exposes credentials and environment variables into each pod that allows it to interact with the k8s API.
The token is exposed at /var/run/secrets/kubernetes.io/serviceaccount/token
, while the environment
variables are KUBERNETES_SERVICE_HOST
and KUBERNETES_PORT_443_TCP_PORT
.
This is Python 3 code below that restarts all the Celery pods in a namespace.
The starmap is a pretty cool concept. Basically I wanted to pass two variables into the function
restart_kubernetes_pod
but there wasn’t an easy way to do it without rewriting the function
which was used by other tools as well. #annoying
This is accomplished by using starmap and zip.
In the end, you end up with two arrays as follows:
As you can see, it’s not too difficult to quickly build some interesting tooling for Kubernetes.