How Do I Prioritize a Migration Queue Without Boiling the Ocean?

Migrating workloads to new infrastructure or cloud platforms is a complex, resource-intensive task. When your migration backlog balloons to hundreds or thousands of candidates, Click here to find out more it’s tempting to try tackling everything at once — but that approach leads to wasted time, disrupted services, and missed savings opportunities.

The secret to maximizing impact while minimizing risk and engineering effort is prioritization. In this post, we’ll break down how to create a candidate shortlist and focus on quick wins. We’ll also challenge some common assumptions about cloud sizing and performance metrics — especially around always-on small services and CPU utilization — and discuss how tools like AWS Compute Optimizer and Azure Advisor can guide your decisions.

Why Prioritization Matters More Than Ever

At first glance, migration projects sound straightforward: identify older resources, pick new instance types or platforms, move workloads, and reap the cost or performance benefits.

But reality quickly diverges when:

You have a large, heterogeneous estate composed of shared CPU instances, burstable services, and always-on background jobs
Your business is cost-sensitive, so trade-offs between engineering effort and savings must be explicit
You’re under pressure to show progress without causing downtime or performance regressions

Throwing a big party for all migrations at once (“boiling the ocean”) tends to backfire:

Engineering resources spread thin across teams and projects
High-risk, high-effort migrations push out quick wins
Suboptimal selection criteria lead to chasing savings that aren’t real

Know Your Migration Candidates: The Importance of a Candidate Shortlist

Prioritization begins with identifying and refining a candidate shortlist: the subset of workloads that can deliver meaningful impact within reasonable effort and risk parameters.

Here’s how to get from a noisy migration backlog to an actionable shortlist:

Inventory and tag. Make sure every resource is associated with business units, environment (prod vs staging), and migration readiness.
Eliminate no-gos and deferred work. Identify resources with compliance or architectural restrictions and carve them out for later phases.
Assess potential savings and engineering effort using automation tools. Use cloud-native services like AWS Compute Optimizer and Azure Advisor to scan resource utilization and recommendations.
Apply your organization’s prioritization filters. For example, focus on non-critical environments that provide low-risk pilot targets and offset engineering effort.

Don’t Fall for Hand-Wavy Cost Estimates

Many teams lean heavily on upfront cost savings estimates, but fail to account for hidden costs:

Storage and egress expenses
Engineering time spent migrating and validating workloads
Performance risks that manifest as downtime or business impact

Instead, ground your shortlist in realities: tune your candidate criteria based on historical migration timelines, validate recommendation quality with spot checks, and always build rollback criteria ahead of migrations.

Why Always-On Small Services Hide Cloud Waste

One of the most overlooked sources of cloud waste is small, burstable instances cpu limits always-on services running at the edge of your estate. These might be monitoring agents, internal tooling, sidecar containers, or low-volume API handlers. Individually, these services seem inexpensive, but collectively they often:

Consume a disproportionate number of compute hours
Use burstable or shared CPU instances that mask underlying inefficiencies
Complicate lightweight migration efforts due to poor utilization baselines

Because these workloads run all the time, even slight overprovisioning on instance sizes or configurations adds up to wasted spend. The challenge is recognizing opportunity here without forcing a drain on valuable engineering cycles.

What To Do Instead

Group always-on small services into their own migration buckets
Apply automation tools like AWS Compute Optimizer to identify right-sizing recommendations
Set up guardrails so migration don’t degrade availability of non-critical jobs

Shared CPU Definitions Vary Dramatically by Provider

When evaluating candidate instances, many teams get tripped up by confusing or inconsistent definitions of “shared CPU”.

Important to note:

Cloud Provider Shared CPU Definition Performance Implications AWS Burstable Performance Instances (e.g., T3, T4g) that accrue CPU credits Instances run at baseline CPU levels with credit-based bursting; sustained high CPU usage requires ample credits to avoid throttling Azure B-series VMs - also burstable with credit systems Similar to AWS, but with different credit accrual and consumption policies Google Cloud Shared core instances (e.g., E2) built on hyperthreading and overcommitment Performance differs based on workload burstiness and underlying host utilization; no explicit CPU credit system

Assuming all shared CPU instances behave similarly leads to incorrect sizing decisions, potential performance regressions, and unanticipated downtime.

Pro tip: Always consult provider-specific documentation and KPI dashboards before downgrading or migrating shared CPU workloads.

Measure Peaks with the Right Observation Window

Performance engineers often rely on average CPU utilization as a sizing guide. This is a trap. Average utilization ignores important performance characteristics related to spikes, burst duration, and real-world SLA requirements.

Before changing instance types or scaling parameters, ask yourself:

What does the P95 and P99 CPU utilization look like over a meaningful window (e.g., 7 days or 28 days)?
How long do spikes last, and how frequent are they?
What is the distribution of latency or error rates during peak CPU usage?

Obtaining these metrics requires robust monitoring and observability pipelines that can export quantiles rather than simple averages.

Why Percentiles and Spike Duration Matter More Than Averages

Averages flatten out spike information—making a workload seem underutilized even though it occasionally maxes out CPU for several seconds or minutes. Using the P95 or P99 utilization lets you model worst-case demand, which influences migration decisions:

Short bursts: Can often be handled by burstable or smaller instances without throttling
Long spikes: May require right-sizing to larger or dedicated CPU instances

Ignoring spike duration and frequency means risking latency or downtime post-migration, especially for latency-sensitive workloads.

Using AWS Compute Optimizer and Azure Advisor to Scale Smartly

Manual performance analysis across hundreds of migration candidates is impractical. Cloud-native tools speed up discovery while reducing the risk of hand-wavy estimates:

AWS Compute Optimizer analyzes historical utilization metrics for EC2, Auto Scaling groups, and EBS volumes, providing recommended instance types for cost savings and performance improvements.
Azure Advisor offers similar recommendations for VM right-sizing, reserved instance purchases, and operational improvements tailored to Azure workloads.

How to Integrate These Tools Into Your Prioritization Workflow

Run Compute Optimizer or Azure Advisor across your estate and export the recommendation reports
Cross-reference with your migration backlog to identify candidates with high cost savings and low risk (e.g., non-prod environments, isolated services)
Incorporate percentiles and spike duration into your evaluation by validating the recommended instance sizes against raw metrics for P95 and P99 CPU usage
Rank candidates by expected savings divided by estimated engineering effort. For instance, prioritize moving a service with $1,500 monthly savings requiring 1 engineer-week versus one with $5,000 savings requiring 8 engineer-weeks

This prioritization matrix helps identify quick wins—low-hanging fruit that free up budget and capacity for more complex migrations.

Balancing Engineering Effort and Impact: Writing Rollback Criteria Before Running a Pilot

No migration succeeds without a clear rollback plan. Before kicking off a pilot migration:

Define success criteria: Performance thresholds, error rates, and SLA adherence that must hold post-migration
Document rollback triggers: Metrics or business signals that automatically escalate rollback or investigation
Plan communication: Who owns the rollback, how to execute it swiftly, and impact on dependent services

Pilot migrations should focus on candidates that provide low risk but still significant insights — ideally from your shortlisted quick wins.

Summary: Prioritize Smart, Avoid Boiling the Ocean

In summary, effective migration prioritization springs from data-driven selection combined with thoughtful risk management. Key takeaways:

Create a targeted candidate shortlist using tags, usage metrics, and tool recommendations
Focus on always-on small services as a stealth source of cloud waste
Know the nuances of shared CPU across providers and validate assumptions
Use P95 and P99 percentiles with spike duration to model real workload demands, not just averages
Leverage AWS Compute Optimizer and Azure Advisor reports as a foundation, but validate with raw telemetry
Write rollback criteria before pilot runs to preserve uptime and trust
Seek quick wins with high impact-to-effort ratios before expanding to more complex workloads

By applying these principles, you reduce migration risk, deliver faster ROI, and build a sustainable framework for ongoing cloud optimization.