Debian上Kubernetes日志分析

Logging in Kubernetes on Debian: Tools, Configurations, and Best Practices

Kubernetes clusters running on Debian require a robust logging strategy to collect, store, and analyze logs from containers, nodes, and cluster components (e.g., kubelet, kube-apiserver). This ensures observability, helps debug issues, and meets compliance requirements. Below is a structured guide to logging in Debian-based Kubernetes environments.

1. Core Logging Concepts in Kubernetes

Kubernetes logs are generated from three primary sources:

• Container Logs: Standard output (stdout) and error (stderr) from application containers (managed by kubelet).
• Cluster Component Logs: Logs from Kubernetes control plane components (e.g., kube-apiserver, kube-controller-manager) and node agents (kubelet).
• Node OS Logs: Logs from the Debian host OS (e.g., systemd journal, auth logs).

The default log driver for containers is json-file, which writes logs to /var/log/containers on each node. For long-term storage and analysis, these logs need to be forwarded to a centralized system.

2. Popular Logging Solutions for Debian+Kubernetes

A. EFK Stack (Elasticsearch + Fluentd + Kibana)

The EFK stack is the most widely used solution for Kubernetes logging. It offers scalable log collection, storage, and visualization:

• Fluentd: A daemonset that runs on each Debian node, collecting logs from /var/log/containers and forwarding them to Elasticsearch.
• Elasticsearch: A distributed search engine that stores and indexes logs for fast querying.
• Kibana: A visualization tool that provides dashboards, search, and alerting for log data.

Key Steps for Deployment:

1. Deploy Elasticsearch: Use a StatefulSet to create a highly available Elasticsearch cluster. Configure resource limits (e.g., 20Gi storage per node) to avoid disk exhaustion.
2. Deploy Fluentd as a DaemonSet: Use the official fluent/fluentd-kubernetes-daemonset image. Mount /var/log/containers and /var/lib/docker/containers as read-only volumes to access container logs. Configure environment variables (e.g., FLUENT_ELASTICSEARCH_HOST, FLUENT_ELASTICSEARCH_PORT) to connect to Elasticsearch.
3. Deploy Kibana: Use a Deployment to expose Kibana via a LoadBalancer or NodePort. Configure elasticsearch.hosts in kibana.yml to point to your Elasticsearch cluster.

Advantages: Full-text search, real-time analytics, and rich visualization. Suitable for large-scale clusters with complex log parsing needs.

B. Loki + Promtail

Loki is a lightweight log aggregation system developed by Grafana Labs. It is optimized for cloud-native environments and integrates seamlessly with Grafana for visualization:

• Promtail: A client that runs on each Debian node, scraping logs from /var/log/containers and sending them to Loki.
• Loki: Stores logs with labels (e.g., namespace, pod name) for efficient querying. Unlike Elasticsearch, Loki does not index log content, reducing storage costs.
• Grafana: Provides pre-built dashboards for Loki logs and supports alerting.

Key Steps for Deployment:

1. Deploy Loki: Use a StatefulSet or Helm chart to deploy Loki. Configure storage (e.g., local SSDs or cloud storage like S3) and retention policies.
2. Deploy Promtail as a DaemonSet: Mount /var/log/containers and configure promtail.yml to scrape logs and send them to Loki. Use relabeling rules to filter logs (e.g., exclude kube-system namespace).
3. Configure Grafana: Add Loki as a data source and import pre-built dashboards (e.g., “Kubernetes Cluster Monitoring”).

Advantages: Lower resource usage, simpler architecture, and cost-effective for small-to-medium clusters. Ideal for teams already using Grafana for monitoring.

C. Filebeat (Lightweight Alternative)

Filebeat is a log shipper from Elastic that runs as a DaemonSet on Debian nodes. It collects logs from /var/log/containers and forwards them to Elasticsearch or Logstash:

• Configuration: Define inputs (e.g., paths: ["/var/log/containers/*.log"]) and outputs (e.g., Elasticsearch endpoint). Use modules (e.g., kubernetes) to parse structured logs.
• Advantages: Lightweight and easy to configure. Suitable for environments where Elasticsearch is already in use.

3. Key Logging Best Practices for Debian+Kubernetes

A. Use Structured Logs

Format logs as JSON (e.g., {"timestamp": "2025-10-04T12:00:00Z", "level": "INFO", "message": "Application started"}). Structured logs enable easier parsing, filtering, and analysis in tools like Elasticsearch and Loki. Most applications (e.g., Go, Python) support JSON logging libraries.

B. Implement Log Rotation

Prevent disk space exhaustion by configuring log rotation for container logs (stored in /var/log/containers). Use logrotate with settings like:

/var/log/containers/*.log {
  daily
  rotate 7
  compress
  missingok
  notifempty
}

This keeps 7 days of compressed logs and deletes older files.

C. Centralize Logs

Avoid relying on local logs (e.g., /var/log/syslog) by forwarding all logs to a centralized system (Elasticsearch, Loki). This ensures logs are accessible across nodes and persist after node failures.

D. Secure Logs

• Encrypt Log Transmissions: Use TLS between Fluentd/Promtail and Elasticsearch/Loki to prevent eavesdropping.
• Mask Sensitive Data: Use tools like Fluentd’s record_transformer or Loki’s replace filter to redact sensitive information (e.g., passwords, API keys) from logs.
• Restrict Access: Use RBAC in Kubernetes to limit access to log data (e.g., only allow the monitoring service account to query Elasticsearch).

E. Monitor Log Infrastructure

Use Prometheus to monitor the health of your logging stack:

• Track Fluentd/Promtail resource usage (CPU, memory).
• Monitor Elasticsearch/Loki disk space and indexing performance.
• Set alerts for anomalies (e.g., Fluentd restarts, Elasticsearch high CPU).

4. Essential Log Analysis Commands

Even with centralized logging, you’ll often need to retrieve logs directly from Kubernetes:

A. View Logs for a Specific Pod

kubectl logs <pod-name> --timestamps

Add --follow (-f) to stream logs in real-time.

B. View Logs for a Label Selector

kubectl logs -f -l app=my-app

This retrieves logs for all pods with the label app=my-app.

C. View Logs from a Previous Container Instance

kubectl logs <pod-name> -p

Useful for debugging crashed containers.

D. Stream Logs from Multiple Pods

Use kubetail (install via brew install kubetail or apt-get install kubetail):

kubetail my-app

Aggregates logs from all pods with the label app=my-app into a single stream.

E. View Cluster Component Logs

Check kubelet logs (on Debian nodes):

journalctl -u kubelet --no-pager -n 50

View kube-apiserver logs (if running on Debian):

kubectl logs -n kube-system kube-apiserver-<node-name> --timestamps

By following this guide, you can implement a scalable and secure logging strategy for your Debian-based Kubernetes cluster. Choose the right tools (EFK for advanced analytics, Loki for cost efficiency) and adhere to best practices to ensure your logs are actionable and reliable.