By Emmanuel Udeh, Subsurface Data Analyst at Shell.
Quality information and data are critical for effective performance of any business. It is essential that the strategic and operational decisions which ensure schedules and budgets are met are underpinned by quality business intelligence.
An operator’s ability to leverage Information and data as key business assets is therefore an important success factor in developing, executing and managing projects and facilities. This mitigates risk, ensures safety, and maintains compliance in a lower for longer regime.
Information technology, together with data management, plays a critical role in supporting an organization’s systems and tools. This helps personnel to access data from a multitude of sources. They can then effectively integrate information to drive decision making and collaborate with partners, both externally and internally.
The data migration challenge
A 2011 study by Geosoft Inc showed that system, database and service maintenance costs were the most important concern of Information Management (IM) and Information Technology (IT) departments when thinking about maintaining any data management solution or program.
IM/IT departments are constantly challenged with providing access to the quality information and data required to help energy businesses deliver value to their shareholders. There is an ever-increasing volume of data, both unstructured and structured, to be managed. It is very important that all companies update their systems, applications and platforms regularly, in order to give personnel access to the most up-to-date and best quality data they need for decision making.
Data migration is the transfer of data from an old system to a new system. This may be due to several factors such as technological advancements or a change in business processes. In order to ensure data quality post migration, with the minimal operational interruption, a clear strategy for migration of system software and data needs to be clearly outlined and followed.
Data migration strategies
While the general perception towards change management activities can make data migration seem problematic and difficult to apply, no one actually doubts in its necessity. Generally, there are two principal migration strategies that businesses planning a data migration can consider: either a big bang migration or a trickle migration. Big bang migrations involve completing the entire migration in a small, defined processing window. This approach can seem attractive, in that it completes the migration in the shortest possible time, but it carries several risks. These include significant downtime of core systems and erosion of data quality due to increased pressure on migration teams to deliver on schedule. Trickle migrations take an incremental approach to migrating data. Rather than aiming to complete the whole event in a short time frame, a trickle migration involves running the old and new systems in parallel and migrating data in phases. This method inherently provides minimal downtime to any core systems and applications constantly required by the business.
To choose the most suitable methodology, the IM/IT team needs to answer key questions, particularly how to limit the business impact of the data migration. The team generally makes their decision after considering factors such as:
- Time required to conclude migration activities
- Volume and complexity of data to be migrated
- Compatibility of both systems and their underlying technologies
- Finally, cost cannot be ignored.
Data migration process
All the factors listed above help the IM/IT team to decide which migration strategy to follow. It is then important that a migration team is set up. This team will comprise of functional and technical experts, both in-house and external. They will closely with the business so that all the migration requirements are properly captured and addressed.
A standard migration process usually follows these steps:
Figure 1: A standard migration process
- Project scoping and management engagement
It is important before any migration project starts to have the support of senior business leaders. The key is to align the migration project to overall business objectives and ensure that this is communicated properly. This stage will also clearly define what is achievable in terms of what the new systems will offer to the business. Realistic milestone-based timelines should be set in order to manage expectations.
- Migration planning and preparations
During the preparation and planning phase, analysis of legacy and new systems should be conducted in close consultation with the business users. Activities such as data mapping and systems topology analysis will be carried out to address fundamental questions such as:
- What data will be migrated? Include every data type and legal obligations
- How will this data be migrated? Manually by a team of super users or automated via specialist software?
Properly addressing these questions will help to assess the impact on the migration project of project costs and schedule. It will also avoid compliance issues by ensuring that the migration is in line with specific regional regulatory obligations.
- Training and migration
Most data migrations invariably involve moving from a legacy system to a new system, which business users will have to operate and manipulate to access the data stored in them. As such, training of end users and in-house support personnel are critical components of any data migration project. At this stage, actual data migration can now commence in line with the chosen strategy and delivery approach. Though a few projects require a big bang approach, trickle migrations generally tend to deliver better results, as it is easier to audit and test what has been transferred. This reduces business impact due to loss of data or system downtime. Data cleaning needs to be incorporated also, to prevent transfer of invalid or obsolete data types from the legacy system to the new system. However, no matter how well a migration goes, some clean-up is always needed during the validation and testing stage.
- Data validation and testing
As mentioned earlier, a trickle migration, due to its phased nature, makes it easier to carry out data validation and testing of migrated data. This helps to identify and rectify migration issues early in the migration cycle. A key activity at this stage is that the migration team and functional experts carry out tests on the new system. They should ensure that the data has been migrated correctly and that specified system transactions on the new platform are processed successfully.
- Post-migration support and project signoff
The ultimate aim of any migration activity is to improve business performance and deliver competitive advantage. It is therefore important that there is a dedicated team available to offer support in identifying and correcting system issues that may arise post-migration. This will also ensure a proper handshake between the end users and the various functional support staffs, who will need an adequate level of knowledge required to administer the new systems. After this, the project team validates that every mapped data has been successfully migrated and all post-migration questions has been answered before business signoff.
According to a 2007 survey by Bloor research, companies already spend at least five billion USD per year on migrations, yet 80 percent of them still go over time or over budget. This is because most businesses fail to recognize the fact that they need to apply proven methodologies and strategies. For any migration project to have the best chance of success, it needs to be geared towards maintaining and improving the quality of data. This is vital in increasing the value that companies can derive from information.
