How to plan a data migration with OpenContext

Monarch butterflies migrate from Canada to Mexico to avoid freezing to death. Salmon use the Earth’s magnetic field to migrate from the ocean to freshwater to reproduce. Migrating is a part of life and evolution. 

Migrations have also been a part of my decades spent working in tech where things are constantly evolving.

As a product management veteran, I was used to executives making decisions to migrate and it being my job to figure out how we meet the customer needs from a feature functionality perspective while meeting the executive needs from a timing perspective.

Whether you’re moving to the cloud or going from monolith to microservices or Terraform to Pulumi, the name of the migration game is keeping track of all the moving parts while simultaneously keeping the lights on to avoid service disruptions and making sure no one is left unaware to any and all major changes—especially customers… 

Okay, so there’s more than one name of the game and clearly a lot of pressure from all sides.  

Let’s go over how OpenContext, along with some other handy tools, will help your next migration feel more like soaring elegantly through the jet stream of North America versus swimming against the currents of the Columbia River.

What is considered a data migration? 

A data migration involves moving information from one place to another. As simple as that sounds, you wouldn’t be reading this unless you knew what we all know: migrations = headaches.

Much of this has to do with how we treat our tools and data today. Data makes up the DNA of every organization. The tools we use to build our products, secure our organization, and generally get work done become both a source of data as well as the dedicated storage systems. As teams grow or change shape, often the tools they once used no longer serve their purpose. The data, however, is almost always important to keep. 

The data needs to go somewhere. Whether it moves somewhere new as you swap out one system for another or gets stored in a dedicated, organized spot as you sunset systems altogether. 

Types of data migration 

There are many types of migrations but, chances are, any tech team with a few years or funding rounds behind them has gone through or is going through at least one of these three types of migration: 

1. Application migration

In an application migration, software moves from one system to another such as rehosting, refactoring, re-platforming, or retiring a software.

2. Cloud migration

A cloud migration is moving from a traditional on-site (aka on-premise) data center to a cloud provider or hybrid of on-prem and cloud storage environments.

3. Storage migration

Finally, a storage migration is moving from one system or strategy for storing information to another.

Challenges that come with migrating data

Migrations are hard! It always meant my team was sidelined from doing new feature work to rework something. While reworking or migrating, those new features and customer asks are not getting addressed. This creates so much pressure between dev teams, org levels, and roles within the organization. For example, the dev teams now have to move work around and plan out new features and how they will land with code that has yet been written. That’s not easy.

A migration implies a change from one thing to a different thing. Those differences lead to challenges that are often hard to predict without already going through the exact experience once or twice before. In an application migration, one system may call one field one thing while your new system calls it something else. While that’s a simple difference and might be straightforward enough to plan for and update, even something like that can create ripple effects down the road. Imagine what a more complex difference could do. 

With that in mind, here are a few common complications you can usually expect during a migration:

1. Lack of resources and investment (time and money)

Migrations are time-intensive which translates to overhead costs for labor that goes into running pretty much everything that follows in this list. In addition, there are often hard costs associated with a data migration. 

2. Strategy is hard with so many unknowns

Data migration projects that happen without proper planning have been known to go far past their deadlines, result in lost or incorrect data, cause extended downtimes, and cost companies far more than they budgeted for with time and resources. 

4. Security and compliance issues

Migrations have the potential to expose your data to vulnerabilities and environments that could pose a security or compliance risk. Depending on what types of data you’re dealing with and any regulations you’d need to adhere to, you’ll need to plan certain measures to keep your data secure in transit. This might include additional encryptions and extra monitoring to avoid a breach or exposure that threatens your organization, its people, and its customers either from a cost or reputation standpoint.

5. Downtime

There are often parts of a migration process that can’t move forward without a partial or full shutdown. Interruptions can range from inconvenient to downright chaos. In the best of cases, downtime is planned and communicated ahead of time, but unpredictable disruptions to a service can be costly with long-lasting impact on your brand and bottom line. 

6. Compatibility issues

A common problem with adding any new technology is that it can be difficult to know whether your old systems are compatible with it. When data moves between operating systems or different file formats things can change and break causing major confusion or even data loss. 

7. Lost/missing data 

Data loss during the data migration process is possible. While it might not be a big deal if repetitive data doesn’t make it through, or access changes, a complete loss of data could cause serious problems. 

This is why being detailed matters so much and also why you should make sure that your data migration plan includes a backup strategy.

8. Resistance 

Not everyone loves change. Especially not right away. Resistance will kill any sort of change from happening (or at least happening smoothly). The best practice for getting everyone on board with this type of change is to very clearly communicate the “why” or the purpose of a migration and continually reiterate how the change supports a mission that already has collective buy-in.

5(ish) step migration strategy

1. Pre-migration planning and preparation and, most importantly, discovery

While exploration takes place long before a decision is made to make a change, once you’ve decided to make a migration, you need to really dig in and build a comprehensive situational awareness. 

Tackling this with a discovery mentality is a good way to avoid developing blind spots. Use that time to build your checklists and take a copious amount of inventory to evaluate what goes and what stays and what special treatments are required.

All of that is before anyone starts coding. 

2. Executing: Migrating your data

Being in the thick of the migration is rough and stressful. You often have stricter rules on which features HAVE to be included and the timeframe it MUST be delivered in. If you fall behind, there are decisions around who can get pulled from which team. Which rarely ends up being a value add. 

The team has developed a rhythm at this point. Adding someone new slows everyone down because they need details about what has been done, what’s left, and why we made specific decisions. Not ALL decisions but some are more critical than others. 

2.5. A few “oh shit” moments

Typically, migrations involve a point of struggle at all org levels either all at once or different times. 

Directors get pressure from executives. 

Managers get pressure from contributors. 

Security is like “What are you doing? How is that going to change?” 

Ops is doing a lot of the same. 

Customer Success is worried about an influx of customer calls if we get something wrong. 

All of us are worried.

3. Pre-release collaboration

As you start to write code, there can be a high level of anxiety from teams inside and outside the SDLC. They want to and frankly need to understand all those same decisions with different degrees of specificity. 

Feature parity is generally the biggest concern. Undoubtedly there are some features that don’t seem relevant. (Don’t worry, that one customer who depends on it will let you know if you forget it.) 

All of this comes down to context though. Doers often don’t have all the context around why the decision was made, to what extent, and why that particular timeline? Executives don’t understand how MESSY that bit of code is and how pervasive. There isn’t a place for folks to see how everything is connected. What it looks like to rip one piece of code out and replace it. 

When it comes to this, I have used a few tips and tricks. Namely, a mini-con. (Which is just a dorky way of saying, a comprehensive planning session that includes all players.): 

The product manager, architect, and UX team present the plan, break down use cases, and ask everyone to participate in making things better. What this does is brings everyone together. Now I have a team who can talk to the plan. 

And if we missed something, we’ve ALL missed it. 

4. Communication and change management around migration impact

I love this phase, you are almost there. The migration is almost complete. 

Prepping the company and all the respective teams for the new code is a pretty exciting and nerve-wracking process. 

If you changed a workflow, where the features are on the page, the look and feel, etc. are all points of potential contention. It’s good but it requires you to have some really fantastic galvanizing skills. 

Last steps are up to the Release Engineers, Technical Writers, Support teams, Marketing, and Product Marketing. This is where we put shine on the fact that we took a lot of time to provide you with the same set of features that shine a bit brighter. All important work for scale, uptime, and performance but those aren’t things that make customers happy unless you aren’t doing them well before. 

5. Post-migration ongoing observation, support, and maintenance

As with any change you make, be sure to set aside time and resources to be retrospective. This can include monitoring protocols and manual check-ins as well as a check-in with your migration task force and any impacted parties to make sure you’re aware of all the lasting effects.

3 OpenContext features for your next migration project

When I look back on all of my memories of migration projects, I wish I had a tool like OpenContext. In fact, many of the challenges I’ve faced with migration projects and listed above provided major inspiration for what OpenContext has built. When done well, the discovery portion of a traditional migration process is a very manual way of doing what OpenContext does inherently saving you a big step and alleviating a lot of the doubt that goes along with it. 

While there are quite a few ways you might find OpenContext makes your migration process smoother, the features that stand out in my mind are the robust tagging capabilities, graph database-driven visualizations of relationships between tech and teams (Context Graphs), and ability to drill down on component details and associated legacy knowledge via Context Pages.

1. Tags

Once you have your migration scoped out, tags make your discovery phase as straightforward as can be. OpenContext users can add tags to keep track of details about components that you’d like to be able to search or see at-a-glance. Examples would be something like languages used, related tech debt, or other details. Then, you can sort OpenContext by virtually any tag (such as “Java,” for example) and see everything with that tag. 

In a migration that means you could search the application, system, etc. you are migrating away from and populate a thorough list of components to consider during the transfer process.

2. Context Graphs

Because OpenContext uses a graph database vs. relational database, the Context Graph, is quite powerful. In addition to showing you everything you have knowingly tagged, you can also use the Context Graph to see a broader picture of other connected components that would’ve gone undetected simply because no one was aware of the connection (or found it important information to document or share). 

3. Context Pages

Context Pages about your migration components are a reliable source for understanding everything about each platform and code components. As you go through the levels the immediate connecting points and associated teams, these are great to reference and make sure you understand the details of what this migration is doing.

BONUS: The Home Page

It’s hard to log in to OpenContext and NOT go down a discovery path of your current interest. And, as this is the starting point in our tool, it’s also the starting point to any migration. The OpenContext Home Page lets you pick any starting point and find the related code and infrastructure from the get-go.

Remember: A successful migration is a team effort.

From executive planning, to mini-con’s, to documents (Notion, DropBox, Google, MSFT) leveraging Miro/Figma or other design tools, Asana, Jira, Git, and lots and lots of spreadsheets. It’s a lot of coordination to get everyone on the same page and to understand the starting point in the same way. 

That context is hard won without a tool like OpenContext.

‍