Moving a dozen-plus applications to our own k3s cluster — problems and payoffs

Kubernetes on your own hardware has a reputation as a project for large teams. We built and operate such a cluster for a SaaS platform and its internal tools. Here is an honest summary — including what went wrong.

Starting point

Before the migration, applications were deployed by hand directly on the servers. Every redeployment required a maintenance window or a manual blue-green switch between environments — laborious, and dependent on the person who remembered every step.

The client's cluster was our third bare-metal Kubernetes cluster — we had built and tested two earlier ones in-house. That is why the production rollout in November 2023 was automated from the very first node.

Architecture

• 3 server nodes and 6 workers as virtual machines across several physical Proxmox hosts, managed with Ansible.
• Cluster state in MariaDB Galera instead of etcd — more stable for a topology spread across physical servers.
• Two HAProxy load balancers with Keepalived in front of the cluster API, sharing a virtual address.
• MetalLB for exposing services, with a dedicated routing table.
• Storage: local SSD for databases and distributed storage (Longhorn) with replication for everything else.
• Automation: cert-manager for certificates, external-dns for DNS records, Rancher as a console, Prometheus, Loki and Grafana for metrics and logs.

The cluster runs about 20 applications and about 170 pods: platform services across test, beta, canary and production, databases (MySQL, highly available PostgreSQL, Valkey), analytics tools, a mailing system, GitLab with CI runners, and supporting services.

Seven problems that surprised us

1. MTU on the overlay network. Traffic between nodes on different physical servers runs through a VXLAN tunnel. The default MTU dropped large packets — the symptom: connections that open but hang on larger responses. Fix: MTU 1450 and a dedicated Ansible routine.
2. A missing storage package. Nodes added months later lacked the iSCSI component Longhorn needs. Volumes failed to mount only on the new nodes. Since then every node is built from the same Ansible role.
3. Return traffic with MetalLB. Responses from services exposed by MetalLB went back the wrong way. A separate routing table and kernel ARP settings fixed it.
4. An incomplete certificate chain. When cert-manager and the ingress competed for the same certificate, one without the intermediate CA was served. Browsers forgive that; Java clients do not. The result: a freshly deployed BPM engine could not reach the API and started no processes on its first night. We now check the certificate chain automatically.
5. SSL enforced after a database upgrade. After a PostgreSQL cluster upgrade, the operator began enforcing encrypted connections. One application was not ready and stopped working.
6. Manifest drift. Quick fixes applied directly on the cluster meant the repository stopped describing reality. Today changes go only through the repository and the pipeline.
7. Memory limits and 503s. A Java application sporadically returned 503 — not because of a bug, but because the container's memory limit mechanism was throttling it. Diagnosis meant looking at cgroup statistics, not application logs.

What we gained

• Zero-downtime deployments. Rolling updates and health checks instead of maintenance windows and manual blue-green switches.
• A complete CI/CD chain. Commit → build → image → deploy, in every repository.
• Four environments, including canary. A new version can serve part of the traffic and be compared with the previous one.
• Certificates and DNS without manual work.
• Speed for new services. A new service — for example the BPM process engine — reached production four days after its first commit, because the infrastructure was already waiting.

Is it worth it?

With steady load and a dozen-plus applications — yes, provided someone on the team understands every layer, from the physical network to container memory limits. A bare-metal cluster is not hard to build. It is hard to diagnose when something goes wrong — and that knowledge is paid for with experience, not a licence.