Certified Kubernetes Administrator Study Guide – Logging and Monitoring – Manage cluster component logs

  • Containers by default will use the standard output and error streams
  • This means if a container dies the logs are lost, if a pod is evicted or a node dies the logs are also lost
  • Therefore logs should have a separate storage and lifecycle independent of the nodes, pods or containers – this is cluster level logging
  • Cluster level logging requires a separate architecture to provide a backend storage and analytical engine like log insight for the logs collected
  • To view the standard output and error streams from a container or pod, if there is only a single container use
kubectl logs <pod name> 
  • If there are multiple containers in the pod you need to append the container name you are interested in
kubectl logs <pod name> -c <container name> or kubectl logs <pod name> --all-containers=true 
  • To view the logs of a previous instance of a container
kubectl logs -p -c <container name> <pod name>
  • By default the kubelet keeps one terminated container with its logs if a container restarts. If a pod is evicted all containers are also evicted along with their logs and are not accessible.
  • When running kubectl logs command the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response
  • The command only queries the latest log file, so if logs have been rotated and the latest log file is empty kubectl logs will return an empty response
  • kubelet and container runtime write their logs to journald
  • Kubernetes scheduler and kube-proxy run in a container and write their logs to .log files in the /var/log directory
  • Different log files are present on Master and Worker nodes:
    • Master:
      • /var/log/kube-apiserver.log – API server responsible for serving the API
      • /var/log/kube-scheduler.log – Scheduler responsible for making scheduler decisions
      • /var/log/kube-controller-manager.log – Controller that manages replication controllers
    • Worker:
      • /var/log/kubelet.log – kubelet responsible for running containers on the node
      • /var/log/kube-proxy.log – kube proxy responsible for service load balancing

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this:
search previous next tag category expand menu location phone mail time cart zoom edit close