Docker Architecture: The components and processes

Introduction

In Part 1 of our Docker Architecture series, we explored the Docker architecture, the role of the Docker Daemon, and the internal processes involved in managing containers and images. Building on that foundation, Part 2 will focus on two critical aspects of Docker which is Security and Monitoring.

As Docker continues to gain traction in modern software development and deployment, it's essential to grasp how to secure your Docker environment and monitor its performance effectively. In this article, we'll dive into practical insights and examples designed to help you bolster the security of your Docker setup while implementing robust monitoring solutions. Let's get started on enhancing your Docker experience!

Communication and Security

Securing your Docker environment has become more important than ever. With the increasing reliance on Docker for application deployment, it's essential to implement robust security measures to safeguard your Docker Daemon and containers. Let's explore some key security measures you can implement to protect your Docker setup effectively.

1. Daemon-level Security Configurations

Rootless mode:

Docker supports running the Daemon in rootless mode, where the Daemon runs as an unprivileged user, reducing the risk of privilege escalation attacks. Running in rootless mode is particularly beneficial in shared environments or when deploying applications that handle sensitive data. However, it may not be suitable for all scenarios, such as when you need to access certain privileged operations or resources on the host system.

To enable rootless mode, you can use this script and then do:

dockerd-rootless-setuptool.sh install

After installation, start the daemon:

dockerd --host=unix:///run/user/1000/docker.sock

Note: Rootless mode may have some limitations, such as reduced performance and incompatibility with certain Docker features. Evaluate your specific use case before implementing it in production.

Transport Layer Security (TLS)

TLS ensures secured communication between the Docker CLI and the Daemon. This ensures that API requests are authenticated and encrypted, preventing unauthorized access.

How to enable TLS in Docker?

To configure TLS:

Generate CA, Server, and Client Keys with OpenSSL

Create a Certificate Authority (CA)
‍First, generate a private key for the CA:
```
openssl genrsa -aes256 -out ca-key.pem 4096
```
Then, create the CA certificate
```
openssl req -new -x509 -days 365 -key ca-key.pem -sha256 -out ca.pem
```
During this process, you will be prompted to enter information for the certificate. Ensure that the "Common Name" (CN) matches the DNS name of your Docker daemon's host.
Create a Server Key and Certificate Signing Request (CSR):
Generate a private key for the server:
openssl genrsa -out server-key.pem 4096
Create a CSR for the server:
```
openssl req -subj "/CN=$HOST" -sha256 -new -key server-key.pem -out server.csr
```

Sign the Server Certificate:
Create an extensions file to specify allowed IPs and DNS names:

echo subjectAltName = DNS:$HOST,IP:10.10.10.20,IP:127.0.0.1 >> extfile.cnf
echo extendedKeyUsage = serverAuth >> extfile.cnf

Generate the signed certificate:

openssl x509 -req -days 365 -sha256 -in server.csr -CA ca.pem -CAkey ca-key.pem \
  -CAcreateserial -out server-cert.pem -extfile extfile.cnf

Create a Client Key and Certificate Signing Request (CSR):
Generate a private key for the client:
```
openssl genrsa -out key.pem 4096
```
Create a CSR for the client:
```
openssl req -subj '/CN=client' -new -key key.pem -out client.csr
```

Sign the Client Certificate:
Create an extensions file for client authentication:

echo extendedKeyUsage = clientAuth > extfile-client.cnf

Generate the signed client certificate:

openssl x509 -req -days 365 -sha256 -in client.csr -CA ca.pem -CAkey ca-key.pem \
  -CAcreateserial -out cert.pem -extfile extfile-client.cnf

Clean Up:
Remove unnecessary files:

rm client.csr server.csr extfile.cnf extfile-client.cnf

Set Permissions:
Protect your keys by changing their permissions:

chmod 0400 ca-key.pem key.pem server-key.pem
chmod 0444 ca.pem server-cert.pem cert.pem

Configure Docker Daemon to Use TLS
Now that you have your certificates ready, you can configure the Docker Daemon to use TLS:

dockerd \
    --tlsverify \
    --tlscacert=ca.pem \
    --tlscert=server-cert.pem \
    --tlskey=server-key.pem \
    -H=0.0.0.0:2376

Connect to Docker with TLS
To connect securely to your Docker Daemon from a client machine, you need to provide your CA certificate, client certificate, and client key:
```
docker --tlsverify \
    --tlscacert=ca.pem \
    --tlscert=cert.pem \
    --tlskey=key.pem \
    -H=$HOST:2376 version
```
Secure by Default
For convenience, you can move your certificates to the .docker directory in your home directory and set environment variables to use them by default:
```
mkdir -pv ~/.docker
cp {ca,cert,key}.pem ~/.docker

export DOCKER_HOST=tcp://$HOST:2376 DOCKER_TLS_VERIFY=1
```
Now, Docker commands will connect securely by default without needing to specify TLS options each time.
‍

Important Notes:

Using TLS and managing a CA is an advanced topic; familiarize yourself with OpenSSL, x509, and TLS before using it in production.
Ensure that your certificates are stored securely; anyone with access to these keys can control your Docker Daemon.
While enabling TLS is a critical step in securing communication between the Docker CLI and the Daemon, it is important to take this a step further by implementing mutual TLS (mTLS)

What is mTLS (Mutual TLS):

Mutual TLS (mTLS) is an extension of TLS that not only encrypts the communication but also verifies both the client and server identities.. This is crucial for preventing man-in-the-middle attacks and ensuring both parties are who they claim to be.Here’s how mTLS works in Docker:
- The Docker Daemon uses a server certificate to authenticate itself to clients.
- The Docker CLI uses a client certificate to authenticate itself to the Daemon.
- Both certificates are verified during the handshake process, ensuring secure communication.

By implementing mTLS, you can significantly enhance the security of your Docker environment by ensuring that only authorized clients can communicate with the Daemon.

2. User Namespace Remapping

User namespace remapping enhances security by mapping the container's root user (UID 0) to a non-root user on the host. This prevents the container's root user from having direct access to the host's root user, effectively reducing potential security risks.

To enable user namespace remapping, you need to configure it in the Docker Daemon with a specific UID and GID range in the /etc/subuid and /etc/subgid files. Here’s how: To enable user namespace remapping, you need to configure the Docker Daemon with a specific UID and GID range in the /etc/subuid and /etc/subgid files

Edit /etc/docker/daemon.json:
You have to address the user and group by ID or name
```
{
    "userns-remap": "default"
}
```
Restart the Docker Daemon:
```
sudo systemctl restart docker
```
Verify the remapping:
```
docker run -it --rm alpine id
```
When you run the container, you'll notice that the root user (UID 0) inside the container is mapped to a non-root UID on the host system.

By implementing these security measures, you can harden your Docker environment and reduce the attack surface for potential threats. It's recommended to regularly review and update your security configurations as new vulnerabilities and best practices emerge.

3. Container Runtime Security

Since container runtime directly interacts with the host system's resources, securing it is crucial to ensure the safety and stability of your Docker environment. Let's explore the steps you can take to secure the container runtime.

Use Official Images
Always use official or verified images from trusted sources. These images are regularly updated and scanned for vulnerabilities. Official images are regularly updated and scanned for vulnerabilities, reducing the risk of security issues in your applications. For example, using images from well-known organizations ensures that they follow best practices for security and maintenance.‍
Implement Resource Limits
‍Prevent container resource exhaustion by setting CPU and memory limits:
```
docker run -d --name myapp --cpus 0.5 --memory 512m myapp:latest
```
Enable Read-Only Containers
Whenever possible run containers in read-only mode to prevent modifications to the container filesystem
```
docker run -d --name myapp --read-only myapp:latest
```
‍Use Security Options
Leverage Docker's security options to enhance container isolation:
```
docker run -d --name myapp \
  --security-opt=no-new-privileges \
  --cap-drop=ALL \
  --cap-add=NET_BIND_SERVICE \
  myapp:latest
```
This command drops all capabilities, adds only the necessary ones, and prevents privilege escalation.‍
Utilize Tools like Chainguard:
You could consider using tools like Chainguard, Sysdig Secure, Aqua Security, etc, to enhance image security further. These tools not only help in securing the images but also in maintaining compliance requirements throughout the application lifecycle. Tools like Chainguard also provide you with secure base images designed for production use, ensuring that your applications are built on a foundation that prioritizes security and compliance

Logs and Monitoring

Monitoring and troubleshooting are essential aspects of managing Docker environments. Understanding how to access logs, diagnose issues, and monitor performance can significantly improve the reliability and efficiency of your containerized applications. This section will cover how to access Docker Daemon logs, common troubleshooting techniques, and tools for performance monitoring.

Daemon Logs and Diagnostics

How to Access and Interpret Docker Daemon Logs:
The Docker Daemon logs provide valuable insights into the operations of Docker and can help diagnose issues. By default, Docker logs are stored in the systemd logging service (like journald on many Linux distributions) or in the /var/log/docker.log file. To view the logs, you can use the following command:
```
journalctl -u docker.service
```
If your Docker installation uses a log file, you can view it with:
```
tail -f /var/log/docker.log
```
This command will show the latest entries in real-time. Look for error messages or warnings that can indicate problems with container creation, image pulls, or network issues.
How to access and Interpret Container Logs:
To view logs:
```
docker logs <container_id>
```
To follow logs in real-time
```
docker logs -f <container_id>
```
To show logs with timestamps
```
docker logs -t <container_id>
```
Common Issues and Troubleshooting Techniques:‍
1. Container Fails to Start: If a container fails to start, check the logs for that specific container with:
```
docker logs <container_id>
```
  This will provide output from the container's main process, helping you identify issues like missing dependencies or configuration errors. For example, the logs might look like this:
```
2024-10-13T12:00:00.000Z  INFO  Starting application...
2024-10-13T12:00:01.000Z ERROR  Missing configuration file: /app/config.yml
2024-10-13T12:00:02.000Z INFO  Application failed to start.
```
2. Image Pull Errors: If you encounter issues pulling images, verify your network connection and ensure that the Docker Daemon has access to the internet. Check for any proxy settings that might be interfering with the connection.‍
3. Resource Constraints: Containers may fail to start if there are insufficient resources (CPU, memory) available on the host. Use commands like docker stats to monitor resource usage across all running containers.
  The output might look like this:
```
CONTAINER ID   NAME         CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
abc123def456   my_container 0.15%     50MiB / 1GiB          4.88%     1.5kB / 2kB       10MB / 5MB        2
```
4. Networking Issues: If containers cannot communicate with each other or external services, check the network configuration. Use docker network ls to list networks and docker inspect <network_name> to view details about a specific network.‍
5. Out of Disk Space: One common issue is running out of disk space, which can prevent containers from starting or cause them to crash. To check disk usage, you can run:
```
df -h
```
  If you find that your disk is full, consider cleaning up unused images and containers with. We have a detailed blog post on Docker space management.
```
docker system prune
```
  This command will remove all stopped containers, unused networks, dangling images, and build cache.‍
Log Rotation: To maintain optimal performance and prevent log files from consuming excessive disk space, it's essential to configure log rotation for your Docker containers. You can set this up in your Docker daemon configuration file (/etc/docker/daemon.json). Here’s an example configuration:
```
{
    "log-driver": "json-file",
    "log-opts": {
        "max-size": "10m",
        "max-file": "3"
    }
}
```

Monitoring with Prometheus and Grafana

Integrating Prometheus and Grafana into your Docker environment allows for robust monitoring and visualization of metrics from your containers and services. Here’s how to set up monitoring with Prometheus and Grafana along with OpenTelemetry:

Create Prometheus configuration file:

Create a configuration file named

prometheus.yml

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['cadvisor:8080']

Create a Docker Compose file:‍

Create a

docker-compose.ymlversion: '3'

services:
  cadvisor:
      image: gcr.io/cadvisor/cadvisor:v0.47.1
      hostname: cadvisor
      platform: linux/aarch64
      volumes:
        - "/:/rootfs:ro"
        - "/var/run:/var/run:ro"
        - "/sys:/sys:ro"
        - "/var/lib/docker/:/var/lib/docker:ro"
        - "/dev/disk/:/dev/disk:ro"
      ports:
        - "8080:8080"
  
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  
  grafana:
    image: grafana/grafana
    environment: 
      GF_SECURITY_DISABLE_INITIAL_ADMIN_CREATION: "true"
      GF_AUTH_ANONYMOUS_ENABLED: "true"
      GF_AUTH_ANONYMOUS_ORG_ROLE: "Admin"
      GF_AUTH_DISABLE_SIGNOUT_MENU: "true"
      GF_AUTH_DISABLE_LOGIN_FORM: "true"
    ports:
      - "9100:3000"

The above docker-compose.yml file sets up a monitoring stack consisting of three services: cAdvisor, Prometheus, and Grafana. It pulls the necessary Docker images for each service, mounts essential host directories to cAdvisor in read-only mode to allow it to collect system metrics, and exposes ports for external access—port 8080 for cAdvisor, port 9090 for Prometheus, and port 9100 for Grafana.‍

Run the compose file
docker compose -f docker-compose.yaml --detach

Verify configuration of Prometheus with cAdvisor‍
1. Visit http://localhost:9090/targets?search and check if the cAdvisor target is up

b. Visit this sample query and check if its working correctly. The expected output should be:

‍Configure Prometheus as a Data Source in Grafana:
1. Log in to Grafana.
2. Go to Configuration > Data Sources > Add data source.
3. Select Prometheus and set the URL to http://host.docker.internal:9090.
4. Click Save & Test.

We will use this Grafana dashboard for our use case, which you can download and import.

You should see the logs in your Grafana dashboard once the above dependencies are setup.

Conclusion

In conclusion, this deep dive into Docker Daemon architecture has showcased the intricate workings behind containerized applications. We've explored underlying processes and security considerations that form the backbone of Docker's ecosystem. Additionally, we highlighted the significance of logging and monitoring for maintaining the reliability and performance of your containerized applications. By integrating tools like Prometheus and Grafana, you can gain valuable insights into your Docker setup, allowing for proactive management and troubleshooting.

Key takeaways include the importance of securing the Docker Daemon through measures like rootless mode and TLS encryption, the value of effective monitoring for maintaining healthy Docker environments, and the power of extensibility through plugins and custom configurations. These insights equip administrators and developers with the knowledge to optimize Docker deployments, troubleshoot issues efficiently, and tailor the platform to specific needs.

As container technology evolves, a thorough understanding of Docker Daemon's architecture remains crucial. By mastering these concepts, you're better positioned to leverage Docker's full potential, creating more secure, efficient, and scalable containerized applications.

Thank you! We will notify you when we publish new posts!

Oops! Something went wrong while submitting the form.

Docker Architecture: The components and processes - Part 2

Recent Posts

Deployments still suck, but maybe they don’t have to

Blacksmith Achieves SOC 2 Type 2 Compliance

Stacked Diffs Can Be Expensive: Why You Should Do It More

Recent Guides

How to reduce spend in GitHub Actions

Matrix Builds with GitHub Actions

Building Multi-Platform Docker Images for ARM64 in GitHub Actions