Sometimes we need to run additional commands in a pod in order to troubleshoot or accomplish a certain task, let me show you how.

Background

Ever felt you…

• need to do packet capture (tcpdump) from inside an Kubernetes Pod.
• want to use commands in a container via nsenter to capture information with tools that are not built into the pod
• require to run iperf and other Linux tools that are not in the container’s built-in command list

Instructions

Here is a method using nsenter on the node where the pod is hosted.

• SSH to node where pod is hosted.
• Identify Container ID using: docker ps.

There will be two entries in docker ps output for your application. You may use either one since both should be in same network namespace.

Copy the Container ID (series of numbers and letters) and paste into this command:

$ cid="put Container ID here"

$ nsenter -n -t $(sudo docker inspect --format "{{ .State.Pid }}" "$cid") \
   tcpdump -s 0 -n -i eth0 -w /tmp/$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap

The step above will start the packet capture so at this point reproduce the network issue requiring the packet capture.

Example

Example steps to perform packet capture on a pod named ruby-ex-1-2bdp8:

# Get the pods
$ oc get pods -o wide
NAME              READY     STATUS      RESTARTS   AGE       IP           NODE
ruby-ex-1-2bdp8   1/1       Running     0          1d        10.130.0.8   node-0

# Login to the local node running the pod
$ ssh node-0

# Lookup the pod container ID
$ sudo su -
$ docker ps
CONTAINER ID        IMAGE                                                                                                                            COMMAND                  CREATED             STATUS              PORTS               NAMES
7a2ef74a4a06        docker-registry.default.svc:5000/test/ruby-ex@sha256:6a7e9874154f70c0c2997eeb07c2d9a4d67e70d7bc3faea8adf63f8844a25aeb            "container-entrypoint"   41 hours ago        Up 41 hours                             k8s_ruby-ex_ruby-ex-1-2bdp8_test_a9474054-68fc-11e8-8d13-fa163e721eb0_0
19025a64e5e7        openshift3/ose-pod:v3.7.46                                                                                                       "/usr/bin/pod"           41 hours ago        Up 41 hours                             k8s_POD_ruby-ex-1-2bdp8_test_a9474054-68fc-11e8-8d13-fa163e721eb0_0
53a71ceb3dfe        registry.access.redhat.com/openshift3/registry-console@sha256:1a1230f359de42d07fc011669bd7895fa1d3d92abc6de79b8c067b402459daa6   "/usr/libexec/cockpit"   42 hours ago        Up 42 hours                             k8s_registry-console_registry-console-1-6g4th_default_ca148abc-68f2-11e8-8d13-fa163e721eb0_0
c4f704644a36        openshift3/ose-pod:v3.7.46                                                                                                       "/usr/bin/pod"           42 hours ago        Up 42 hours                             k8s_POD_registry-console-1-6g4th_default_ca148abc-68f2-11e8-8d13-fa163e721eb0_0

# Set the container ID and connect using nsenter
$ cid=19025a64e5e7
$ nsenter -n -t $(sudo docker inspect --format "{{ .State.Pid }}" "$cid") \
   tcpdump -s 0 -n -i any -w /tmp/$(hostname)-$(date +"%Y-%m-%d-%H-%M-%S").pcap

And then from another terminal, run the test:

# Login to the pod
$ oc rsh ruby-ex-1-2bdp8

# Fire up an example request
$ curl http://localhost:8080 #<-- This HTTP request will appear in the packet capture started above.