Introduction 🔗
Inspektor Gadget is a collection of tools (or gadgets) to debug and inspect Kubernetes applications.
Inspektor Gadget is deployed to each node as a privileged DaemonSet. It uses in-kernel BPF helper programs to monitor events mainly related to syscalls from userspace programs in a pod. The BPF programs are run by the kernel and gather the log data. Inspektor Gadget’s userspace utilities fetch the log data from ring buffers and display it.
The architecture explains how it works under the hood, but to me it means to be able to run eBPF programs easily on Kubernetes.
Inspektor Gadget in OpenShift 🔗
There are a few minor details currently that ‘prevents’ to run it in OpenShift:
- It needs to be installed in the
kube-systemnamespace (it is not recommended to deploy things there) - It uses
/run/to store thegadgettracermanagersocket (instead its own directory)
But mainly, it doesn’t recognize RHCOS and fails to be deployed.
Fortunately, there is a fork created by Philippe Huet that circunvent those issues and works pretty well!
In the meantime, I opened a Github issue to notify the Kinvolk folks about this so in the future it can be deployed without any forks.
NOTE: Obviously this is totally unsupported.
OCP Requirements 🔗
AFAIK, there is no special requirements to run it, besides being cluster-admin (well, the workers operating system is a limitation as well but we are using Philippe’s fork that allows it to run it in OCP)
In this post I’ll deploy it using crc but I’ve been able to make it work using a baremetal IPI cluster as well.
NOTE: In case you don’t know what crc is, check this blog post or the official documentation , but tl;dr.- it is a stripped version of OpenShift so you can run it in your beefy laptop (4 vCPUs, 9 GB of free memory & 35 GB of storage space is required) in Linux/Windows/OSX as it is basically a VM with a preconfigured single node OCP cluster in.
Deploy OCP (using crc) 🔗
A Red Hat account is required to access the user pull secret. You can create a
developer account that will give you access to crc, RHEL and some other
goodies in the developers.redhat.com site.
- Download
crc - Download the pull-secret file
- Extract the
crcbinarytar Jxvf crc-linux-amd64.tar.xz - Optionally, copy it to your
$PATH - Run
crc setupand paste the pull-secret content when asked - Run
crc startand wait for the VM to be up - Run
eval $(crc oc-env)andexport KUBECONFIG=~/.crc/machines/crc/kubeconfig
You are now cluster-admin of your own OCP cluster running on your laptop, pretty cool right?
Deploy Inspektor Gadget 🔗
oc apply -f https://raw.githubusercontent.com/clustership/inspektor-gadget/master/openshift/deployment.yaml
NOTE: There is a bug currently where the daemonset can fail with sigsegv. In that case, delete the daemonset and redeploy it again:
oc delete ds gadget -n gadget-tracing
oc apply -f https://raw.githubusercontent.com/clustership/inspektor-gadget/master/openshift/deployment.yaml
Compile the client 🔗
It is required to have make, git and golang already installed. RHEL8:
sudo dnf install -y make git golang
Clone the inspektor-gadget fork repository:
git clone https://github.com/clustership/inspektor-gadget.git
Build the client:
make kubectl-gadget-linux-amd64
Copy the binary to the /usr/local/bin/ directory so oc and kubectl can
recognize it as a plugin:
chmod a+x ./kubectl-gadget-linux-amd64
sudo mv kubectl-gadget-linux-amd64 /usr/local/bin/kubectl-gadget
sudo restorecon /usr/local/bin/kubectl-gadget
NOTE: It is required to compile your own cli because it includes fixes to make
it work with OpenShift (basically querying the inspketor-gadget pods running in
the gadget-tracing namespace instead of kube-system)
Usage (get capabilities requested by a pod) 🔗
Deploy a demo application. Basically this one but in the gadget-tracing namespace:
cat <<EOF | oc apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: set-priority
namespace: gadget-tracing
labels:
k8s-app: set-priority
spec:
selector:
matchLabels:
name: set-priority
template:
metadata:
labels:
name: set-priority
spec:
containers:
- name: set-priority
image: busybox
command: [ "sh", "-c", "while /bin/true ; do nice -n -20 echo ; sleep 5; done" ]
EOF
Then, get the pod name and the worker name where the pod is running. In this case, we will be focused on crc-tnpk6-master-0:
oc get po -o wide -n gadget-tracing
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gadget-cf8hx 1/1 Running 0 18h 192.168.126.11 crc-tnpk6-master-0 <none> <none>
set-priority-5646554d9d-jcmch 1/1 Running 0 18h 10.116.0.27 crc-tnpk6-master-0 <none> <none>
Run the kubectl-gadget capabilities command as:
kubectl-gadget capabilities --unique --verbose -p set-priority-5646554d9d-jcmch --node crc-tnpk6-master-0
or
oc gadget capabilities --verbose --unique -p set-priority-5646554d9d-jcmch --node crc-tnpk6-master-0
The output looks like:
Node numbers: 0 = crc-tnpk6-master-0
Running command: exec /opt/bcck8s/bcc-wrapper.sh --tracerid 20201203092226-55d305646fcd --gadget /usr/share/bcc/tools/capable --namespace gadget-tracing --podname set-priority-5646554d9d-jcmch -- --unique -v
NODE TIME UID PID COMM CAP NAME AUDIT
[ 0] 08:22:30 1000580000 16178 sh 21 CAP_SYS_ADMIN 0
[ 0] 08:22:30 1000580000 20501 sh 21 CAP_SYS_ADMIN 0
[ 0] 08:22:30 1000580000 20501 nice 6 CAP_SETGID 1
[ 0] 08:22:30 1000580000 20501 nice 7 CAP_SETUID 1
[ 0] 08:22:30 1000580000 20501 nice 23 CAP_SYS_NICE 1
[ 0] 08:22:30 1000580000 20500 sh 21 CAP_SYS_ADMIN 0
[ 0] 08:22:30 1000580000 20500 true 6 CAP_SETGID 1
[ 0] 08:22:30 1000580000 20500 true 7 CAP_SETUID 1
[ 0] 08:22:30 1000580000 20502 sh 21 CAP_SYS_ADMIN 0
[ 0] 08:22:30 1000580000 20502 sleep 6 CAP_SETGID 1
[ 0] 08:22:30 1000580000 20502 sleep 7 CAP_SETUID 1
^C
Terminating...
Running command: exec /opt/bcck8s/bcc-wrapper.sh --tracerid 20201123115948-c2009dae9253 --stop
References 🔗
Notes 🔗
Yes, I’ve tried to resist the temptation to put an Inspector gadget image in the post to make it look more professional… but I guess nobody reads it until completion, so here it is! :D
