To view the logs of a previous instance of a container
kubectl logs -p -c <container name> <pod name>
By default the kubelet keeps one terminated container with its logs if a container restarts. If a pod is evicted all containers are also evicted along with their logs and are not accessible.
When running kubectl logs command the kubelet on the node handles the request and reads directly from the log file, returning the contents in the response
The command only queries the latest log file, so if logs have been rotated and the latest log file is empty kubectl logs will return an empty response
kubelet and container runtime write their logs to journald
Kubernetes scheduler and kube-proxy run in a container and write their logs to .log files in the /var/log directory
Different log files are present on Master and Worker nodes:
Master:
/var/log/kube-apiserver.log – API server responsible for serving the API
/var/log/kube-scheduler.log – Scheduler responsible for making scheduler decisions
/var/log/kube-controller-manager.log – Controller that manages replication controllers
Worker:
/var/log/kubelet.log – kubelet responsible for running containers on the node
/var/log/kube-proxy.log – kube proxy responsible for service load balancing