Docker Optimization

1---
2name: docker-optimization
3description: Docker image optimization patterns including multi-stage builds, layer caching, security hardening, and size reduction techniques. Use when building Docker images, optimizing container size, improving build performance, or implementing Docker security best practices. Reduces image sizes by 70-90% and build times by 50-80%.
4---
5 
6# Docker Optimization Patterns
7 
8Comprehensive guide to optimizing Docker images for size, build speed, and security. Covers multi-stage builds, layer caching strategies, security hardening, and production deployment patterns.
9 
10---
11 
12## Quick Reference
13 
14**When to use this skill:**
15- Building production Docker images
16- Optimizing image size (reducing from 500MB+ to <100MB)
17- Improving Docker build times
18- Implementing Docker security best practices
19- Debugging slow builds or large images
20- Setting up Docker for microservices
21 
22**Common triggers:**
23- "My Docker image is too large"
24- "Docker builds take forever"
25- "How do I optimize this Dockerfile"
26- "Docker security best practices"
27- "Multi-stage build pattern"
28 
29---
30 
31## Part 1: Multi-Stage Builds
32 
33### The Problem: Bloated Images
34 
35**Typical single-stage Dockerfile** (800MB+ image):
36```dockerfile
37FROM python:3.11
38WORKDIR /app
39COPY . .
40RUN pip install -r requirements.txt
41CMD ["python", "app.py"]
42```
43 
44**Problems:**
45- Includes build tools (gcc, make, etc.) - 300MB+
46- Includes pip cache - 100MB+
47- Includes source .git directory - 50MB+
48- Includes test files and dev dependencies - 50MB+
49- **Total: 800MB+ for simple Python app**
50 
51### The Solution: Multi-Stage Pattern
52 
53**Optimized multi-stage Dockerfile** (120MB image):
54```dockerfile
55# Stage 1: Builder
56FROM python:3.11-slim AS builder
57WORKDIR /app
58 
59# Install build dependencies in separate layer
60RUN apt-get update && apt-get install -y --no-install-recommends \
61    gcc \
62    && rm -rf /var/lib/apt/lists/*
63 
64# Copy only requirements first (cache optimization)
65COPY requirements.txt .
66RUN pip install --user --no-cache-dir -r requirements.txt
67 
68# Stage 2: Runtime
69FROM python:3.11-slim
70WORKDIR /app
71 
72# Copy only Python packages from builder
73COPY --from=builder /root/.local /root/.local
74 
75# Copy only application code
76COPY app.py .
77COPY src/ ./src/
78 
79# Make sure scripts in .local are usable
80ENV PATH=/root/.local/bin:$PATH
81 
82# Run as non-root user
83RUN useradd -m appuser && chown -R appuser:appuser /app
84USER appuser
85 
86CMD ["python", "app.py"]
87```
88 
89**Result: 800MB → 120MB (85% reduction)**
90 
91### Multi-Stage Pattern Breakdown
92 
93**Stage 1: Builder** (Throw away after build)
94- Install build tools
95- Compile dependencies
96- Run tests (optional)
97- Generate artifacts
98 
99**Stage 2: Runtime** (Final image)
100- Minimal base image
101- Copy only artifacts from builder
102- No build tools
103- No source files (only compiled/necessary files)
104 
105---
106 
107## Part 2: Layer Caching Optimization
108 
109### Understanding Docker Layer Caching
110 
111Each instruction creates a layer. Docker caches unchanged layers.
112 
113**Bad Order** (cache invalidated on every code change):
114```dockerfile
115FROM python:3.11-slim
116COPY . .                        # ❌ Copies everything
117RUN pip install -r requirements.txt  # ❌ Runs on every code change
118```
119 
120**Good Order** (cache preserved):
121```dockerfile
122FROM python:3.11-slim
123COPY requirements.txt .         # ✅ Only requirements
124RUN pip install -r requirements.txt  # ✅ Cached if requirements unchanged
125COPY . .                        # ✅ Code changes don't invalidate pip cache
126```
127 
128### Layer Caching Best Practices
129 
130**1. Order by change frequency** (least to most):
131```dockerfile
132# 1. System dependencies (rarely change)
133RUN apt-get update && apt-get install -y curl
134 
135# 2. Language runtime (rarely changes)
136FROM python:3.11-slim
137 
138# 3. Dependencies (change occasionally)
139COPY requirements.txt .
140RUN pip install -r requirements.txt
141 
142# 4. Application code (changes frequently)
143COPY . .
144```
145 
146**2. Separate COPY operations**:
147```dockerfile
148# ❌ Bad: Invalidates cache on any file change
149COPY . .
150 
151# ✅ Good: Cache preserved unless specific files change
152COPY package.json package-lock.json ./
153RUN npm ci
154COPY src/ ./src/
155COPY public/ ./public/
156```
157 
158**3. Use .dockerignore**:
159```
160# .dockerignore
161.git
162.gitignore
163node_modules
164npm-debug.log
165Dockerfile
166.dockerignore
167.env
168.venv
169__pycache__
170*.pyc
171tests/
172docs/
173```
174 
175---
176 
177## Part 3: Image Size Optimization
178 
179### Choose Minimal Base Images
180 
181**Image Size Comparison**:
182```
183python:3.11          → 1.01GB
184python:3.11-slim     → 130MB   (87% smaller)
185python:3.11-alpine   → 50MB    (95% smaller)
186```
187 
188**When to use each**:
189- **Full image** (`python:3.11`): Never for production
190- **Slim** (`python:3.11-slim`): Default choice, good compatibility
191- **Alpine** (`python:3.11-alpine`): Smallest, but can have glibc issues
192 
193### Multi-Stage Size Optimization
194 
195**Node.js Example** (900MB → 150MB):
196```dockerfile
197# Builder stage
198FROM node:20 AS builder
199WORKDIR /app
200COPY package*.json ./
201RUN npm ci --only=production
202 
203# Production stage
204FROM node:20-alpine
205WORKDIR /app
206COPY --from=builder /app/node_modules ./node_modules
207COPY . .
208EXPOSE 3000
209CMD ["node", "server.js"]
210```
211 
212**Result**: 900MB → 150MB (83% reduction)
213 
214### Clean Up in Same Layer
215 
216**❌ Bad** (creates large intermediate layers):
217```dockerfile
218RUN apt-get update
219RUN apt-get install -y build-essential
220RUN rm -rf /var/lib/apt/lists/*
221```
222 
223**✅ Good** (single layer, no intermediate garbage):
224```dockerfile
225RUN apt-get update && \
226    apt-get install -y --no-install-recommends build-essential && \
227    rm -rf /var/lib/apt/lists/*
228```
229 
230### Remove Build Dependencies After Use
231 
232```dockerfile
233RUN apt-get update && \
234    apt-get install -y --no-install-recommends \
235        gcc \
236        g++ \
237        make \
238    && pip install --no-cache-dir -r requirements.txt \
239    && apt-get purge -y --auto-remove \
240        gcc \
241        g++ \
242        make \
243    && rm -rf /var/lib/apt/lists/*
244```
245 
246---
247 
248## Part 4: Security Best Practices
249 
250### Don't Run as Root
251 
252**❌ Bad** (runs as root):
253```dockerfile
254FROM python:3.11-slim
255COPY app.py .
256CMD ["python", "app.py"]
257```
258 
259**✅ Good** (runs as non-root user):
260```dockerfile
261FROM python:3.11-slim
262 
263# Create non-root user
264RUN useradd -m -u 1000 appuser && \
265    mkdir -p /app && \
266    chown -R appuser:appuser /app
267 
268WORKDIR /app
269USER appuser
270 
271COPY --chown=appuser:appuser app.py .
272CMD ["python", "app.py"]
273```
274 
275### Never Include Secrets in Image
276 
277**❌ Bad** (secrets baked into image):
278```dockerfile
279ENV DATABASE_PASSWORD=secret123
280COPY .env .
281```
282 
283**✅ Good** (secrets provided at runtime):
284```dockerfile
285# Pass secrets via environment variables at runtime
286# docker run -e DATABASE_PASSWORD=$DB_PASS myapp
287```
288 
289**✅ Also Good** (Docker secrets):
290```dockerfile
291# Use Docker secrets (Swarm/Kubernetes)
292CMD ["sh", "-c", "python app.py"]
293# Secrets mounted at /run/secrets/
294```
295 
296### Scan Images for Vulnerabilities
297 
298```bash
299# Using Docker Scout
300docker scout cves myapp:latest
301 
302# Using Trivy
303trivy image myapp:latest
304 
305# Using Snyk
306snyk container test myapp:latest
307```
308 
309### Use Specific Image Tags
310 
311**❌ Bad** (unpredictable):
312```dockerfile
313FROM python:latest
314```
315 
316**✅ Good** (reproducible):
317```dockerfile
318FROM python:3.11.9-slim-bookworm
319```
320 
321---
322 
323## Part 5: Build Performance Optimization
324 
325### BuildKit (Modern Docker Builder)
326 
327Enable BuildKit for faster builds:
328```bash
329export DOCKER_BUILDKIT=1
330docker build -t myapp .
331```
332 
333**Benefits**:
334- Parallel layer building
335- Skip unused stages
336- Better caching
337- 30-50% faster builds
338 
339### Build Cache Mounts
340 
341**With BuildKit** (cache pip downloads):
342```dockerfile
343RUN --mount=type=cache,target=/root/.cache/pip \
344    pip install -r requirements.txt
345```
346 
347**Benefits**:
348- Pip packages cached between builds
349- No need to clear cache (doesn't bloat image)
350- Significantly faster rebuilds
351 
352### Parallel Multi-Stage Builds
353 
354BuildKit automatically parallelizes independent stages:
355```dockerfile
356# Stage 1: Frontend build (runs in parallel)
357FROM node:20 AS frontend
358WORKDIR /app/frontend
359COPY frontend/package*.json ./
360RUN npm ci
361COPY frontend/ ./
362RUN npm run build
363 
364# Stage 2: Backend build (runs in parallel)
365FROM python:3.11-slim AS backend
366WORKDIR /app/backend
367COPY backend/requirements.txt .
368RUN pip install -r requirements.txt
369 
370# Stage 3: Final image (waits for both stages)
371FROM python:3.11-slim
372COPY --from=frontend /app/frontend/dist /app/static
373COPY --from=backend /app/backend /app
374```
375 
376---
377 
378## Part 6: Production Patterns
379 
380### Health Checks
381 
382```dockerfile
383FROM python:3.11-slim
384COPY app.py .
385 
386# Add health check
387HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
388    CMD curl -f http://localhost:8000/health || exit 1
389 
390CMD ["python", "app.py"]
391```
392 
393### Proper Signal Handling
394 
395```dockerfile
396# Use exec form to ensure proper signal handling
397CMD ["python", "app.py"]  # ✅ Receives SIGTERM
398 
399# Not shell form
400CMD python app.py  # ❌ Shell doesn't forward signals
401```
402 
403### Labels for Metadata
404 
405```dockerfile
406LABEL org.opencontainers.image.title="MyApp"
407LABEL org.opencontainers.image.version="1.2.3"
408LABEL org.opencontainers.image.authors="team@example.com"
409LABEL org.opencontainers.image.source="https://github.com/org/repo"
410```
411 
412---
413 
414## Part 7: Language-Specific Patterns
415 
416### Python Optimization
417 
418```dockerfile
419FROM python:3.11-slim AS builder
420 
421# Prevent Python from writing pyc files
422ENV PYTHONDONTWRITEBYTECODE=1
423# Prevent Python from buffering stdout/stderr
424ENV PYTHONUNBUFFERED=1
425 
426# Install system dependencies
427RUN apt-get update && apt-get install -y --no-install-recommends \
428    gcc \
429    && rm -rf /var/lib/apt/lists/*
430 
431WORKDIR /app
432COPY requirements.txt .
433 
434# Install to user site-packages
435RUN pip install --user --no-cache-dir -r requirements.txt
436 
437# Runtime stage
438FROM python:3.11-slim
439ENV PYTHONDONTWRITEBYTECODE=1 \
440    PYTHONUNBUFFERED=1 \
441    PATH=/root/.local/bin:$PATH
442 
443WORKDIR /app
444COPY --from=builder /root/.local /root/.local
445COPY . .
446 
447RUN useradd -m appuser && chown -R appuser:appuser /app
448USER appuser
449 
450CMD ["python", "-m", "uvicorn", "app:app", "--host", "0.0.0.0"]
451```
452 
453### Node.js Optimization
454 
455```dockerfile
456FROM node:20-alpine AS builder
457 
458WORKDIR /app
459COPY package*.json ./
460 
461# Install production dependencies only
462RUN npm ci --only=production && \
463    # Remove npm cache
464    npm cache clean --force
465 
466# Runtime stage
467FROM node:20-alpine
468 
469# Create non-root user
470RUN addgroup -g 1001 -S nodejs && \
471    adduser -S nodeuser -u 1001
472 
473WORKDIR /app
474COPY --from=builder --chown=nodeuser:nodejs /app/node_modules ./node_modules
475COPY --chown=nodeuser:nodejs . .
476 
477USER nodeuser
478EXPOSE 3000
479 
480CMD ["node", "server.js"]
481```
482 
483### Go Optimization
484 
485```dockerfile
486# Builder stage
487FROM golang:1.21-alpine AS builder
488 
489WORKDIR /app
490COPY go.mod go.sum ./
491RUN go mod download
492 
493COPY . .
494RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .
495 
496# Runtime stage - minimal scratch image
497FROM scratch
498 
499# Copy CA certificates for HTTPS
500COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
501 
502# Copy binary
503COPY --from=builder /app/main /main
504 
505EXPOSE 8080
506ENTRYPOINT ["/main"]
507```
508 
509**Result**: 1GB → 15MB (98.5% reduction!)
510 
511---
512 
513## Part 8: Common Mistakes to Avoid
514 
515### Mistake 1: Installing Recommended Packages
516 
517**❌ Bad** (installs hundreds of unnecessary packages):
518```dockerfile
519RUN apt-get install curl
520```
521 
522**✅ Good** (minimal installation):
523```dockerfile
524RUN apt-get install -y --no-install-recommends curl && \
525    rm -rf /var/lib/apt/lists/*
526```
527 
528### Mistake 2: Using ADD Instead of COPY
529 
530**❌ Bad** (ADD has implicit behavior):
531```dockerfile
532ADD requirements.txt .  # Can extract tarballs, fetch URLs
533```
534 
535**✅ Good** (COPY is explicit):
536```dockerfile
537COPY requirements.txt .  # Only copies files
538```
539 
540### Mistake 3: Multiple FROM Without AS
541 
542**❌ Bad** (can't reference previous stages):
543```dockerfile
544FROM python:3.11
545RUN pip install -r requirements.txt
546FROM python:3.11-slim
547# Can't copy from previous stage!
548```
549 
550**✅ Good** (named stages):
551```dockerfile
552FROM python:3.11 AS builder
553RUN pip install -r requirements.txt
554FROM python:3.11-slim
555COPY --from=builder /root/.local /root/.local
556```
557 
558### Mistake 4: Not Using .dockerignore
559 
560Without .dockerignore:
561- Copies .git directory (50MB+)
562- Copies node_modules (100MB+)
563- Copies test files
564- Invalidates cache on any file change
565 
566### Mistake 5: Hardcoding Versions Incorrectly
567 
568**❌ Bad** (no control over patch versions):
569```dockerfile
570FROM python:3.11
571```
572 
573**✅ Good** (pin exact version):
574```dockerfile
575FROM python:3.11.9-slim-bookworm
576```
577 
578---
579 
580## Part 9: Before/After Examples
581 
582### Example 1: Python FastAPI App
583 
584**Before** (1.2GB image, 5min build):
585```dockerfile
586FROM python:3.11
587WORKDIR /app
588COPY . .
589RUN pip install -r requirements.txt
590CMD ["uvicorn", "app:app"]
591```
592 
593**After** (140MB image, 2min build):
594```dockerfile
595FROM python:3.11-slim AS builder
596WORKDIR /app
597RUN apt-get update && apt-get install -y --no-install-recommends gcc && \
598    rm -rf /var/lib/apt/lists/*
599COPY requirements.txt .
600RUN pip install --user --no-cache-dir -r requirements.txt
601 
602FROM python:3.11-slim
603ENV PATH=/root/.local/bin:$PATH
604WORKDIR /app
605COPY --from=builder /root/.local /root/.local
606COPY app.py .
607COPY src/ ./src/
608 
609RUN useradd -m appuser && chown -R appuser:appuser /app
610USER appuser
611 
612CMD ["uvicorn", "app:app", "--host", "0.0.0.0"]
613```
614 
615**Results**:
616- Size: 1.2GB → 140MB (88% reduction)
617- Build time: 5min → 2min (60% faster)
618- Security: Now runs as non-root
619- Cache: Code changes don't rebuild dependencies
620 
621---
622 
623## Part 10: Quick Optimization Checklist
624 
625**Image Size**:
626- [ ] Use slim or alpine base images
627- [ ] Multi-stage build (build tools in first stage only)
628- [ ] Clean up in same layer (`apt-get install && rm -rf`)
629- [ ] Use `--no-install-recommends` with apt-get
630- [ ] Remove package manager cache (`pip --no-cache-dir`, `npm cache clean`)
631- [ ] Use .dockerignore
632 
633**Build Speed**:
634- [ ] Order COPY by change frequency
635- [ ] Copy dependency files before code
636- [ ] Enable BuildKit
637- [ ] Use build cache mounts
638 
639**Security**:
640- [ ] Run as non-root user
641- [ ] Pin specific image versions
642- [ ] Scan for vulnerabilities
643- [ ] Never include secrets in image
644- [ ] Use minimal base images
645 
646**Production**:
647- [ ] Add HEALTHCHECK
648- [ ] Use exec form for CMD
649- [ ] Add metadata labels
650- [ ] Proper signal handling
651- [ ] Set up proper logging
652 
653---
654 
655## Resources
656 
657**Official Docker Documentation**:
658- Multi-stage builds: https://docs.docker.com/build/building/multi-stage/
659- Best practices: https://docs.docker.com/develop/dev-best-practices/
660- BuildKit: https://docs.docker.com/build/buildkit/
661 
662**Security Scanning**:
663- Docker Scout: https://docs.docker.com/scout/
664- Trivy: https://github.com/aquasecurity/trivy
665- Snyk: https://snyk.io/product/container-vulnerability-management/
666 
667**Base Images**:
668- Docker Hub: https://hub.docker.com/
669- Google Distroless: https://github.com/GoogleContainerTools/distroless
670- Alpine Linux: https://alpinelinux.org/
671