On-Prem to AWS + Kubernetes: The Migration Gotchas Nobody Mentions

In 2023 we rebuilt a multiplayer gaming platform with 20+ microservices from an on-prem datacenter into an AWS + Kubernetes architecture. On paper the plan looked straightforward: migrate services in this order, write the Helm charts, push them to EKS, shift the traffic. Reality never played like that.

This post walks through seven things that aren't in the plan documents but will absolutely slow you down. If you're staring at a similar migration, read this first.

1. Finish observability migration before the workload migration

Pre-migration, our dashboards were bespoke per service. No unified Grafana love — each service had its own monitoring. Move to Kubernetes and suddenly:

• Pods come and go dynamically
• Logs vanish in short-lived containers
• The old dashboard references stop working

So we were migrating 20 services and rebuilding observability infra from scratch. Bad combination.

Lesson: centralized logging (ELK for us), tracing (Jaeger), unified metrics (Prometheus) first. Get those running for every service. Then start migration. Reverse the order and "what's happening, no one knows" eats 2–3 weeks mid-migration.

2. JVM warm-up needs a custom readiness probe

Our Spring-based Java services spend the first 30 seconds warming up the JIT. The instant Kubernetes brings a pod up and the TCP port is open, Kubernetes sends traffic. Java isn't ready yet.

Result: every scale-up event, the first traffic hit got slow responses. Users saw it as "the system is slow."

Fix: readiness probe checks not just the port but an application-level warm-up endpoint. Until the JVM is warm, the pod doesn't go "ready."

This isn't in the plan documents but it applies to every Java service in practice. If you're moving anything JVM-based, write the readiness probe yourself.

3. Keep the database outside Kubernetes

Arguable, but for us a firm rule: RDS MySQL and DocumentDB clusters stay outside Kubernetes, managed. Kubernetes runs stateless workloads only.

Reason: running a DB on Kubernetes is possible but operationally noisy. Storage, backup, failover, monitoring, upgrades — all your responsibility. Managed DB services generally handle these already.

We were tempted mid-migration ("one place is simpler"). We resisted. That was the right call.

4. Service mesh must come at the migration's start

Istio was added at the start of the migration — a deliberate choice. Adding it afterward is much harder. Why:

• Injecting sidecars into every service changes runtime behavior
• Enabling mTLS changes auth behavior
• Traffic splitting requires revisiting every deploy path

If you're adding a service mesh, do it during the migration. After is a separate migration, far more expensive.

5. DNS and service discovery during the bridge period

The migration wasn't one-shot — for 6 months some services were on-prem, some were in AWS, talking to each other. The most critical piece of this bridge is service discovery. How does an on-prem service reach an AWS one? How does AWS find on-prem MySQL?

In our case:

• VPN tunnel between on-prem and AWS VPC
• On-prem had its own DNS; AWS had Route53
• Cross-over names (e.g. game-server.internal) resolved from both sides

This kind of bridge looks "simple" on paper. Setting it up eats three weeks.

6. Cost control from day one

In AWS, cost is something you notice too late. First month's bill: "fine." Second month: "wait, what?" Third: "nobody deleted that EBS volume, it's been billing for 6 months."

Set up on migration day one:

• Daily Cost Explorer alerts
• A tag strategy (every resource has at least team, service, env)
• Unused resource script (weekly, lists zombies)

Setting this up later is 3× the work, because by then you have tens of thousands of resources accumulated.

7. Ownership: the team adopts after

The most important reality: migration is led by the people who know Kubernetes first. But the application will be owned day-to-day by the dev teams.

What happened with us: for the first 3 months, I and a 2–3 person platform team drove the migration. In month 4, a service slowed down; the dev team couldn't read the Helm chart, so it fell back to us. We were both migrating and running operations. Burnout near.

Fix: during migration, have each service's owning team write the chart, open the PR, run the deploy. Platform team guides but hands off. Feels slow early; after 3 months you have genuinely distributed ownership.

Takeaway

Cloud native transformation is often framed as "a technical job," but it actually has three dimensions: technical, human, process. The technical is solved — docs are plentiful, examples are plentiful. Human and process dimensions aren't written; they're discovered.

Notice none of these seven items are technology choices. They're all team + process. That's why they're absent from plan documents — they can't easily be written down. Yet they're where the real work happens.

The strongest signal of migration success is the team's well-being six months after. If everyone is exhausted, something went wrong.