Certified Kubernetes Administrator Study Guide – Logging and Monitoring – Understand how to monitor applications

• Applications should not be monitored in isolation, need to consider the performance of the containers, pods, services and the overall cluster
• Container level –
  1. livenessProbe – checks whether the container is running. If the probe fails the kubelet kills the container and the container restart policy then determines whether the container is restarted. Limited in scope as if a container does not provide a liveness probe the result will always be success. Also only keeps if the container is running, not the state of the container contents
  2. readinessProbe – checks whether the container is ready to service requests. If the probe fails the endpoints controller removes the Pod’s IP address from the endpoints of the Services that match the Pod. Limited in scope as if a container does not provide a readiness probe the results will always be success.

• Liveness probes ensure healthy pods by killing off and replacing unhealthy containers.
• Readiness probes ensure only healthy pods receive incoming requests by preventing unhealthy pods being routed to by the Services.
• Probes have an initialDelaySeconds field which controls how long Kubernetes waits before starting the probes. Use this to allow containers to have a spin up time before you start checking for liveness or readiness.
• Probes are executed periodically
• Liveness probes have three different mechanisms:
  1. An HTTP GET probe performs an HTTP GET request on the container’s IP address, a port and path you specify. If the probe receives a response, and the response code doesn’t represent an error (in other words, if the HTTP response code is 2xx or 3xx), the probe is considered successful. If the server returns an error response code or if it doesn’t respond at all, the probe is considered a failure and the container will be restarted as a result.
  2. A TCP Socket probe tries to open a TCP connection to the specified port of the container. If the connection is established successfully, the probe is successful. Otherwise, the container is restarted.
  3. An Exec probe executes an arbitrary command inside the container and checks the command’s exit status code. If the status code is 0, the probe is successful. All other codes are considered failures.

• There are three types of Readiness probes:
  1. An Exec probe, where a process is executed. The container’s status is determined by the process’ exit status code.
  2. An HTTP GET probe, which sends an HTTP GET request to the container and the HTTP status code of the response determines whether the container is ready or not.
  3. A TCP Socket probe, which opens a TCP connection to a specified port of the container. If the connection is established, the container is considered ready.

• New feature in Kubernetes 1.14 called Pod ready ++ which introduces a new field ReadinessGate in the PodSpec to specify additional conditions to be evaluated for Pod readiness, for example a url that must be accessible.
• This means for a pod to be evaluated by Kubernetes as ready:
  1. All containers in the Pod are in the ready state
  2. All conditions specified in the readinessGates fields of the Pod are True

Kind: Pod

...

spec:

  readinessGates:

    - conditionType: "www.example.com/feature-1"

status:

  conditions:

    - type: Ready  #
this is a builtin PodCondition

      status: "False"

      lastProbeTime: null

      lastTransitionTime: 2018-01-01T00:00:00Z

    - type: "www.example.com/feature-1"   # an extra PodCondition

      status: "False"

      lastProbeTime: null

      lastTransitionTime: 2018-01-01T00:00:00Z

  containerStatuses:

    - containerID: docker://abcd...

      ready: true

...