Kubernetes Homelab

A hands-on Kubernetes cluster built from scratch using 3 Beelink SER5 mini PCs, featuring automated deployment scripts, enterprise-grade networking, monitoring, and database services.

Type
Homelab/Learning
Hardware
3x Beelink SER5
OS
Ubuntu 24.04 LTS
Kubernetes
v1.34
View Source Code Documentation
Kubernetes kubeadm Cilium CNI MetalLB Prometheus Grafana PostgreSQL Redis RabbitMQ Bash

Project Overview

This homelab project demonstrates hands-on experience building and managing a production-like Kubernetes cluster from bare metal. Rather than using managed Kubernetes services, this implementation uses kubeadm to create a cluster from scratch, providing deep understanding of Kubernetes internals and operations.

Why This Matters:

  • Hands-on experience with Kubernetes cluster bootstrapping and configuration
  • Understanding of networking, storage, and security at a fundamental level
  • Real hardware constraints and troubleshooting experience
  • Cost-effective learning environment for testing and experimentation
  • Foundation for understanding managed Kubernetes services like EKS, AKS, GKE

Hardware Setup

Master Node

  • Beelink SER5 mini PC
  • AMD Ryzen 5 5560U
  • 16GB RAM
  • 256GB NVMe SSD
  • Ubuntu 24.04 LTS

Worker Node 1

  • Beelink SER5 mini PC
  • AMD Ryzen 5 5560U
  • 16GB RAM
  • 256GB NVMe SSD
  • Ubuntu 24.04 LTS

Worker Node 2

  • Beelink SER5 mini PC
  • AMD Ryzen 5 5560U
  • 16GB RAM
  • 256GB NVMe SSD
  • Ubuntu 24.04 LTS

Network Configuration:

  • Pod CIDR: 10.244.0.0/16
  • Service CIDR: 10.96.0.0/12
  • MetalLB IP Pool: 192.168.1.240-192.168.1.250
  • All nodes on same local network

Automated Deployment

One of the key features of this homelab is the comprehensive automation through bash scripts. The entire cluster can be deployed with a single command, or step-by-step for learning purposes.

Quick Deployment:

# Deploy everything automatically sudo ./scripts/99-deploy-all.sh

Step-by-Step Deployment:

# 1. Prepare all nodes (install dependencies) ./scripts/01-prepare-nodes.sh # 2. Initialize master node ./scripts/02-init-master.sh # 3. Join worker nodes to cluster ./scripts/03-join-workers.sh # 4. Deploy networking (Cilium CNI + MetalLB) ./scripts/04-deploy-networking.sh # 5. Setup Helm package manager ./scripts/05-setup-helm.sh # 6. Deploy monitoring stack ./scripts/06-deploy-monitoring.sh # 7. Deploy database services ./scripts/07-deploy-databases.sh

The scripts handle all the complexity including KUBECONFIG setup, dependency checks, and proper deployment ordering.

Core Features

Cilium CNI

Advanced networking with eBPF-based packet processing, network policies, and service mesh capabilities.

MetalLB

LoadBalancer implementation for bare metal, providing external IPs to services in the homelab.

Prometheus & Grafana

Complete monitoring stack with metrics collection, visualization, and alerting capabilities.

PostgreSQL

Production-grade relational database for persistent data storage and application backends.

Redis

In-memory data store for caching, session management, and real-time applications.

RabbitMQ

Message broker for asynchronous communication between microservices.

Technical Implementation

Cluster Architecture:

  • 1 Master Node running control plane components (API server, scheduler, controller manager, etcd)
  • 2 Worker Nodes for application workloads
  • kubeadm for cluster bootstrapping
  • containerd as the container runtime
  • Cilium for pod networking and security policies

Project Structure:

├── scripts/ # Automated deployment scripts ├── manifests/ # Kubernetes YAML manifests ├── configs/ # Configuration files ├── helm/ # Helm charts and values └── docs/ # Documentation and guides

Key Learnings Implemented:

  • Script Ordering: Understanding dependency chains and proper deployment sequences
  • KUBECONFIG Management: Proper certificate and config file handling
  • CNI Deployment: Why nodes stay "NotReady" until CNI is deployed
  • LoadBalancer Services: How MetalLB enables LoadBalancer type in bare metal
  • Persistent Storage: Working with local persistent volumes

Troubleshooting & Documentation

A critical part of this project is comprehensive documentation of common issues and solutions:

Common Issues Addressed:

  • Nodes NotReady: Expected behavior before CNI deployment, resolved by Cilium installation
  • kubectl Certificate Errors: KUBECONFIG environment variable configuration
  • LoadBalancer No External IP: MetalLB configuration and IP pool setup
  • Helm Installation Timing: Proper sequence after cluster initialization

Testing & Validation:

# Test deployment order and cluster health ./scripts/test-deployment-order.sh # Verify cluster status kubectl get nodes kubectl get pods -A kubectl get svc -A

The project includes detailed documentation for deployment ordering, troubleshooting, quick reference commands, and application deployment guides.

Skills Demonstrated

3
Physical Nodes
7
Deployment Scripts
100%
Automated
6+
Services Deployed

DevOps Skills:

  • Kubernetes Administration: Cluster setup, management, and troubleshooting
  • Networking: CNI configuration, service discovery, LoadBalancer implementation
  • Automation: Bash scripting for repeatable deployments
  • Monitoring: Prometheus metrics and Grafana dashboards
  • Database Management: PostgreSQL, Redis, and RabbitMQ in Kubernetes
  • Documentation: Clear guides and troubleshooting resources
  • Problem Solving: Real-world issues with bare metal Kubernetes

Why Homelab Matters

While cloud platforms like AWS, Azure, and GCP offer managed Kubernetes services, building a homelab provides invaluable hands-on experience with the underlying technology:

  • Deep Understanding: Building from scratch reveals how Kubernetes really works
  • Cost-Effective Learning: One-time hardware investment vs. ongoing cloud costs
  • Safe Experimentation: Break things and learn without production consequences
  • Transferable Skills: Knowledge applies directly to managed services
  • Interview Advantage: Demonstrates genuine passion and self-directed learning
  • Platform Agnostic: Understanding fundamentals that work anywhere

This homelab serves as a continuous learning environment for testing new technologies, practicing disaster recovery, and developing automation skills.

Future Enhancements

  • Add distributed storage with Rook/Ceph
  • Implement GitOps with ArgoCD or Flux
  • Deploy service mesh (Istio or Linkerd)
  • Add ingress controller with cert-manager for TLS
  • Implement backup and disaster recovery procedures
  • Configure HA for critical services
  • Add CI/CD pipelines for application deployments
  • Implement security scanning and policy enforcement