Certified Kubernetes Administrator Study Guide – Logging and Monitoring – Manage cluster component logs

• Containers by default will use the standard output and error streams
• This means if a container dies the logs are lost, if a pod is evicted or a node dies the logs are also lost
• Therefore logs should have a separate storage and lifecycle independent of the nodes, pods or containers – this is cluster level logging
• Cluster level logging requires a separate architecture to provide a backend storage and analytical engine like log insight for the logs collected

• To view the standard output and error streams from a container or pod, if there is only a single container use

kubectl logs <pod name> 

• If there are multiple containers in the pod you need to append the container name you are interested in

kubectl logs <pod name> -c <container name> or kubectl logs <pod name> --all-containers=true 

• To view the logs of a previous instance of a container

kubectl logs -p -c <container name> <pod name>

• By default the kubelet keeps one terminated container with its logs if a container restarts. If a pod is evicted all containers are also evicted along with their logs and are not accessible.

• When running kubectl logs command the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response
• The command only queries the latest log file, so if logs have been rotated and the latest log file is empty kubectl logs will return an empty response
• kubelet and container runtime write their logs to journald
• Kubernetes scheduler and kube-proxy run in a container and write their logs to .log files in the /var/log directory
• Different log files are present on Master and Worker nodes:
  • Master:
    • /var/log/kube-apiserver.log – API server responsible for serving the API
    • /var/log/kube-scheduler.log – Scheduler responsible for making scheduler decisions
    • /var/log/kube-controller-manager.log – Controller that manages replication controllers
  • Worker:
    • /var/log/kubelet.log – kubelet responsible for running containers on the node
    • /var/log/kube-proxy.log – kube proxy responsible for service load balancing