The more complex the data migration project, the greater the margin for error. That’s why it helps to follow best practice to reduce the likelihood of data getting lost or corrupted during the migration, and to minimise the potential impact on the business.
Here are our tips for helping to reduce data loss in complex data migration projects:
1. Define the data that is required for the migration.
a) Do not migrate data that is no longer required.
i. Fields required in the source system may be superfluous to business needs and benefits.
ii. There may be fields of inaccurate data that are mandated but not required, and therefore do not need to be migrated. These fields are often filled with “.” or other irrelevant characters that were input to satisfy source system constraints when users entered the data.
b) Do not migrate data that is out of date, as this may affect marketing campaigns or statistical analysis within the new system.
c) Do migrate data to give business advantage – but make sure you define beforehand which data is business critical.
2. Use a tool to profile the data so you have a full picture of the current quality of the data, and where the gaps may be.
3. Cleanse the data where required. You can do this manually before the data migration starts, or you can do down the automated route prior or during migration.
4. Define the quality rules the data has to abide by early on in the process to give a robust framework for the migration. You can define these rules from system analysis, business analysis, and via gap analysis between the source and target systems.
5. Get sign-off on the rules. The complete set of quality rules needs to be reviewed and approved by the domain and system experts for both the source and target systems before you begin the data migration.
6. Verify the data against the rules. It is best practice in complex data migration projects to make this an ongoing process, where possible. The rules should continue to be defined, so you can maintain a constant view of the data and ensure that its quality is fit for purpose. That way, you can raise any issues early on, mitigate risks, and lower the chance of losing data during the migration.
7. Define a clear flow for data during migration, error reporting and rerun procedures.
8. Use a tool for data migration to professionalise and automate the process, thus further reducing the risk of error and data loss.