This page documents the Kubernetes cluster running on the home server. Diagrams are generated from D2 source files in ~/git/k8s-infra/docs/.
Last updated: 2026-03-29
¶ Infrastructure Overview
Physical and virtual infrastructure: internet connectivity, router, host server, KVM virtual machines, storage, and network topology.
- Server: Ubuntu 25.04, AMD Ryzen AI 9 HX 370, 53 GiB RAM (10.0.20.9)
- Router: OpenWrt 21.02 (10.0.20.254), dnsmasq for local *.grogg.org resolution
- KVM VMs: k8s-control (.119), k8s-worker1 (.187), k8s-worker2 (.101), openclaw (.211) on virbr0 (192.168.122.0/24)
- Storage: 2x NVMe in LVM (vg0), 12.7T NAS at /nas
¶ Cluster Architecture
Kubernetes components, namespaces, workloads, monitoring, and backup systems.
¶ Control Plane
- K8s v1.32.13, containerd 1.7.28
- Single control-plane node (k8s-control)
- etcd backup: systemd timer daily at 03:00, rsync to /nas at 03:30
¶ Infrastructure Services
- CNI: Cilium v1.19.1 (eBPF) with Hubble observability
kubeProxyReplacement: true— Cilium handles all service routing via eBPF, kube-proxy removed (2026-03-28)- Previously ran with
kubeProxyReplacement: falsealongside kube-proxy, which caused a dual data path race condition leading to intermittent API server unreachability from pods
- Ingress: ingress-nginx (2 replicas) behind MetalLB L2 (VIP 192.168.122.200)
- TLS: cert-manager with Let's Encrypt (letsencrypt-prod ClusterIssuer)
- Storage: local-path-provisioner, NFS for Velero/MinIO
¶ Application Workloads
| Namespace | Service | Replicas | Domain(s) | Storage |
|---|---|---|---|---|
| davidata | nginx:alpine | 2 | davidata.se, davidata.com | - |
| grogg | grogg-web | 2 | grogg.org | - |
| blog | Grav CMS | 1 | servant.grogg.org, thedailythinkingmachine.grogg.org | 5Gi PVC |
| dokuwiki | DokuWiki | 1 | wiki.davidata.se | 5Gi PVC |
| vaultwarden | Vaultwarden | 1 | vault.davidata.se | 2Gi PVC |
| obsidian-search | Qdrant + FastAPI | 1+1 | obsidian.grogg.org (HTTP, internal) | 1Gi PVC (Qdrant) + NFS (vault) |
¶ Monitoring
- Prometheus + Grafana (grafana.grogg.org, internal HTTP only)
- kube-proxy monitoring disabled in Helm values (kubeProxy.enabled: false) — no scrape target or KubeProxyDown alert since kube-proxy was removed
- Loki (20Gi) + Promtail for log aggregation
- Tempo (10Gi) for distributed tracing
- Alertmanager, kube-state-metrics, node-exporter
¶ Alerting
Alertmanager emails david@davidata.se via Gmail SMTP (km@grogg.org).
Design principle: Alert noise is handled via Alertmanager routing (receiver selection), not by modifying Prometheus rules. This keeps signal detection intact and survives Helm upgrades.
Silenced via routing to null receiver:
| Alert | Reason |
|---|---|
| Watchdog | Heartbeat alert, not actionable |
| InfoInhibitor | Informational, not actionable |
| etcdMembersDown, etcdInsufficientMembers | False positive — Prometheus can't scrape etcd metrics port (2381 refused) |
| TargetDown (kube-system) | Known scrape targets not exposed |
| KubeJobFailed (velero, kopia-maintain jobs) | Kopia repo maintenance jobs have hardcoded backoffLimit: 0; intermittent failures are non-critical since next cycle self-heals. Actual backup failures are covered by VeleroBackupFailed/VeleroBackupNotRun |
Active custom alerts:
- VeleroBackupFailed, VeleroBackupPartialFailure, VeleroBackupNotRun, VeleroBackupTargetDown
- EtcdBackupNotRun, EtcdBackupTargetDown
- CertificateExpiringSoon (14 days), CertificateExpiringVerySoon (3 days)
- Plus all default kube-prometheus-stack alerts (KubeJobFailed, KubeJobNotCompleted, etc.)
¶ Backup
- Velero + MinIO (NFS-backed, 50Gi)
- Daily at 04:00, 30-day retention
- node-agent DaemonSet with kopia for PVC file-system backup
- Kopia is the backup engine Velero uses for file-level PVC snapshots (replaced Restic)
- Repo maintenance jobs run every ~65 minutes (compacting indexes, cleaning epoch markers)
backoffLimit: 0is hardcoded in Velero source — cannot be changed- Maintenance jobs have resource requests: 50m CPU, 128Mi/512Mi memory
keepLatestMaintenanceJobs: 3— Velero's own cleanup mechanism (not K8s TTL)
- Covers: blog, dokuwiki, vaultwarden namespaces
¶ Traffic Flow
Request path from user to backend service, showing DNS resolution, TLS termination, and routing.
- Internet -> public IP (37.123.171.156) -> router port-forward -> host nginx (L4 TCP passthrough with PROXY protocol)
- Host nginx -> MetalLB VIP (192.168.122.200) -> ingress-nginx (TLS termination) -> ClusterIP services
- All service routing handled by Cilium eBPF (no kube-proxy/iptables)
- cert-manager manages Let's Encrypt certificates as K8s Secrets
- Vaultwarden has a NetworkPolicy restricting ingress to ingress-nginx namespace only
¶ Security
- NetworkPolicy: Vaultwarden (ingress-only from ingress-nginx), enforced by Cilium
- PodDisruptionBudgets: vaultwarden, dokuwiki, grav, ingress-nginx, obsidian-search
- TLS: All public sites via cert-manager/Let's Encrypt
- Resource limits: All application deployments have requests/limits set
¶ Diagram Sources
D2 source files are version-controlled:
~/git/k8s-infra/docs/overview.d2~/git/k8s-infra/docs/k8s-architecture.d2~/git/k8s-infra/docs/traffic-flow.d2
Regenerate with: d2 <file>.d2 <file>.svg && rsvg-convert -w 2000 <file>.svg -o <file>.png