Ian France, Technical Director

Virtualization
“Virtualization was a growing theme this year. It enables different systems to be accessed via a virtualization layer, rather than trying to maintain a centralised corporate data store. This is seen as providing a more affordable solution for organisations, and avoids problems of the central store being out-of-date. However, it is unclear how virtualization tackles the problem of multiple versions of the data, nor the problem of data stored in arbitrary spreadsheets.”

Data maturity and quality
“A big theme at the conference was managing data quality and assessing how much clean-up and processing has been conducted on it. A connected theme was how to achieve and maintain a canonical version of “the truth”, or as close as possible. It was discussed in the context of quality, trust and usage.”

Excel data storage
“I heard much concern about the complexity of developing loaders for Excel-based data. A big issue is that Excel documents can often be changed by the user, which requires changes to the loader mappings. If these mappings are hard-coded in Java, or C#, this can be an expensive undertaking. One simple way of solving this is to have a staged approach, where the complex loader (e.g. Excel to PPDM) uses a company-standard Excel spreadsheet or other format. This is the only loader mapped directly to the MDM, so common business rules, validation and so on are implemented only once. The user then has to copy or convert their data into the standard Excel first. Another approach is to have a flexible loader which can be swiftly changed to accept the new format.”

Richard Cook, E&P Technical Data Lead

Making data more widely available
“I saw increasing confidence in organisations’ data. In particular, there is a move towards making it available for use more widely and to more areas of the business, democratising data access and reducing the time taken to use the data. The conference shared examples of faster and better business decisions due to integrated high quality data. For instance, previous experience was used to avoid drilling problems, or reducing time to answer a business question from days to minutes.”

Data management continuity
“As raised at previous conferences, the data management industry faces a decline in knowledge as experienced data managers retire. In particular, the conference discussed the benefits of a professional qualification and the need to attract younger people.”

Karl Glenn, Business Development Director

Improving integration with Petrel
“There were many informal discussions around the need to improve loading and general integration with Petrel. A number of operators highlighted a requirement for a solution to building custom loaders, exports and integration between Petrel and many non-SIS sources and targets. I get the impression this need is growing.”

Popular data standards.
“WITSML, PRODML, RESQML and PPDM were among the standards discussed at our booth. PPDM and WITSML were very often mentioned as being good options for a canonical data store and data transfer respectively. Everyone seemed to recognise the problem that both standards are hardly ever used in a consistent way by companies. There is a definite need to load to and from them in a flexible and easily configurable manner.”

Chris Griffin, Senior Consultant

Risk management
“I saw a growing interest in identifying and managing risk. Of course, risk is inherent in data migrations across all industries, but is particularly relevant when looking at E&P technical data. I talked to a number of operators who are faced with large-scale migrations in the next twelve months and all recognised the value in a formal data migration methodology. This needs to be coupled with leveraging tools and expertise to deliver projects on time, to budget, and to business needs.”

Did we miss you at PNEC, or were you unable to attend? To find out more about our expertise in E&P data management, please get in touch.

The event, which is organised by Petroleum Network Education Conferences, takes place in Houston on 14-16 May. We will be conducting live demos of our E&P data connectors and loaders, and will be sharing real-life examples of how we solve complex E&P data management challenges.

Visit us at Booth 11 in the Century Room to see a demo and to learn how we can help manage even the most challenging E&P data. To reserve a specific time and date for a one-to-one demonstration, simply get in touch by contacting us at info@etlsolutions.com.

We look forward to seeing you in Houston.

Our Transformation Manager migration tool is used in a once-only ‘Prepare Phase’ to automatically create transforms that will be used during every project migration. These generic transforms will be run during the Extract, Verify and Update phases of the migration process (see the diagram below).

Clients’ standards are defined using two spreadsheets; this is the starting point for each migration of a legacy project.

The standard values spreadsheet (see below) contains a worksheet for each table that includes their standard values. These tables contain a number of verification code tables (_VC), a number of reference data tables (_R), and other tables, e.g. OW_DATA_SOURCE, that contain standards data. Each worksheet consists of a header row specifying the fields defined by the company’s standards, followed by rows which provide the set of permitted values.

The data dictionary spreadsheet, shown below, defines data to be entered into the OW Data_Dict Table to specify default values or data ranges covered by the company standards.

The data dictionary spreadsheet is used to set default values, which may be null, and a range of values that each field must conform to, expressed either as a numeric value range, e.g. for CASING_SIZE ‘0..10e+4’, or as a set of value strings.

Any project can be updated by running a command line script during an Extract phase to determine any non-standard values used by the project. This automatic process reviews the project tables, determining any values currently used which require updating to conform to the client’s standards. A set of mapping CSV files is created with values currently used which are not to standard.

A mapping file (.CSV) is created for each table and its natural key field, as in the spreadsheet below, which contain values that are not present in the client’s standards.

An absent mapping spreadsheet indicates that all current values are consistent with the standard values. A mapping file initially contains the set of existing values used in the project and forms the interface for a user to enter a standards value to be used instead. When mapping spreadsheets have already been created for a previous project, these can be provided and will be updated during the extract process. In this case, the spreadsheet will contain some new standard values as recommendations for the project migration.

One of the principal activities the user undertakes, referred to as the Review phase, is to check the mapping spreadsheets and provide any standard values that are required to be used instead of the non-standard presently used.  Once this has been completed, the user again runs a command line script to verify that once the client’s standard values are applied, the project will be consistent with the standards. Transformation Manager transforms are run to verify that this is the case.  The output is a set of reports; one of these as viewed by Excel is shown below.

During the Verify stage, the user will correct all non-standard values and data dictionary errors until the reports indicate that a completely consistent project will be produced if migration proceeds.

Up to this stage, the project data has been accessed in read only mode and the project has been available to all other users. It is not until the final Upgrade phase of the project migration that exclusive access to the project is required.

Once verification has been completed, the final Update phase is again run from the command line and completes the migration process.

The Extract, Review and Update phases implemented using Transformation Manager are completely automatic, and provide warnings if the migration process is not completely compatible with the client’s standards. The main user input is in the Review phase, where a user with knowledge of the project specifies which standard values will be used to replace any non-standard values presently used.

Throughout the migration phases, the user operates in a simple user environment, with minimal involvement of system or database administrators. The project remains online until the final Upgrade phase, which requires exclusive access to the project database for a relatively short duration.

In conclusion, if the business wants to leverage the benefits of corporate data which conforms to a defined set of standards, using metadata-driven standardization tools will enable the business to truly implement and maintain the standards. Also, such a programme can be delivered in a fraction of the effort, time and risk that often adversely affects even well supported initiatives.

Download and keep this information in our free white paper: Implementing corporate data standards in the field using metadata-centric tools.

Image via: freedigitalphotos.net.

The data is usually held in database tables – which themselves contain catalogue controlled values – and has often been extended or adapted for regional circumstances.  This causes two day-to-day issues:

1. Corporate reports assembled from such data can easily be misleading. An attempt to discover something as simple as the relative number of ‘on shore’ and ‘off shore’ wells may fail if one of the regional stalls has decided to classify wells as ‘tidal’, ‘shallow water’ and ‘deep water’ – or just as ‘on’ and ‘off’.
2. Engineers trained in one region, or working for one subcontractor, can find it difficult to work in another region where standard reference values are chosen from a different set.

While these are awkward issues, it is typically difficult to assign a direct cost to any accounting centre and hence calculate the benefit of fixing it.  A more pressing reason often arises when stores are to be upgraded to newer (and more consistent) versions, or when data held in legacy stores needs to be sufficiently cleaned so it can be migrated into corporate standard stores. Often, the day-to-day issues are tolerated until a migration is required by the business.

What makes this a tricky problem is the mismatch between the tools available to the average engineer – usually not much more than Excel and SQLplus – and the extreme complexity of a modern data store. PPDM 3.8, for example, contains over 1700 tables. A hand-tool approach to such a problem is going to be very expensive, tying up valuable staff for weeks on end and quite possibly failing.

It is in these circumstances that metadata tools in the right domain specialist’s hands come into their own. In next week’s post we will share our process for interpreting and manipulating the data.

Read more detail in our free white paper: Implementing corporate data standards in the field using metadata-centric tools.

Image via: freedigitalphotos.net.

The challenge

Snap-on Business Solutions’ Prism product provides sophisticated analysis and reporting services to automotive dealers and manufacturers.  The challenge was to develop an efficient process for extracting accurate service, repairs, counter sales and parts data daily from dealerships for use in Prism.  (Click here for ideas on how dealerships can use their data to boost profitability.)

The solution

Our DataHub is used to obtain extracts from range of Dealer Management Systems (DMS) in different markets.  For instance, we integrate data in France from dealers’ I’Car and DCSNet systems, and in the Netherlands using EVA systems.

Daily data extracts are securely collected overnight via the internet, with minimal processing required at the dealership.  The information collected includes parts stock, services, repairs and parts counter sales.  The data is transferred to our DataHub, a core that forms the central part of the dealership data network, with points that are data sources.  The software enables the core to share data bi-directionally with any number of sources.

The data is aggregated, replicated, converted and loaded into a target data store, before being transmitted to Prism.  Once established, the DataHub operates automatically through a central configuration management facility.  The software manages any changes in functionality, maintenance routines or software updates, and then deploys them automatically to each location or point.  This reduces the burden of maintenance on the dealer and maintains a full audit trail.

After the data is passed to Prism, further enhanced processing occurs and data can be viewed by the manufacturer and dealers.  The result is a system of accurate integration with minimal risk and increased transparency.  The process is now being expanded to 200 dealers in France, over 100 in Belgium and a similar number in the Netherlands, providing Snap-on with complete control of the dealerships’ data.

Click here to find out more about our DMS data integration services.

Image via Free Digital Photos.

Transformation Manager is a software suite capable of tackling even the most complex data movement projects.  With a ten-year track record in data movement, Transformation Manager is used by some of the world’s largest companies to manage their data.  It is also used by the Oil & Gas industry’s most successful software vendors to develop data connectors and loaders.  The software is specifically designed to enable users to connect, transform and integrate data from any source to any target system.

The new version, Transformation Manager 5.2, includes a host of useful features.  These include a debugger, making error detection a more straightforward process.  Developers can now also save time by running projects within the software’s Designer tool, helping to speed development and creating a faster test cycle.  The user interface has been enhanced, with a clearer indication of any data in a transformation that has not been mapped.  This enables a more precise transformation, with reduced potential for user error.

Transformation Manager 5.2 is available now.  For more information, or to request a demo, please contact Karl Glenn, Business Development Director at ETL Solutions, on +44 (0)1248 675070 or email kg@etlsolutions.com.

Stage 1: Configuration

At the start of any data migration, our consultants carry out a structural configuration to ensure that all aspects of the migration are functionally correct.  This will include:

• Ensuring the routes of communication are defined
• Ensuring storage and  versioning of project artefacts is available
• Ensuring hardware is available and accessible.

Click here to find out more about the planning stage of a data migration project.

Stage 2: Migration design

This stage covers the extraction and cleansing of the data, verification of the data and transformation of the data.  These core steps follow our bespoke data migration methodology to enable a smooth data flow during the migration.

Stage 3: Testing design

The testing design stage defines an overall test plan for all stages of the migration.

This includes how each stage will be tested at unit level, followed by how the entire migration will be tested from start to end to ensure that it is functionally correct across all components.

Stage 4: Migration development

The migration is developed in stages using an agile methodology.  This has proved a successful approach in migrations where a number of stakeholders are involved.  An agile approach, which is clearly visible across all teams, ensures that risks are mitigated early in the process.

This approach also enables those downstream, at the target end, to get actual data sooner for testing purposes.  Click here to see our detailed methodology for petroleum data migration.

Stage 5: Testing development

Testing is carried out within a test framework that can be provided or created for the purpose.  This framework allows unit tests to be run regularly to immediately highlight any issues that arise.

Stage 6: Execution

A number of dry runs will be defined to test the go-live strategy, allowing the go-live plan to be confirmed and refined as required.  The initial dry run may use only part of the sample data.  After an initial dry run, a test migration takes place.

This process helps ensure that the project is delivered successfully with minimum risk.

To find out how we can help with your data migration project, please get in touch.

This week we found a great infographic by American Internet Services (click here to view its source), which estimates that 2.5 quintillion bytes of data are created EVERY DAY.

Ten potential projects were identified by the meeting, including mapping the PPDM data model and WITSML.  The aim of these projects will be to ensure greater collaboration between alternative data standards in the Oil & Gas industry.  The meeting also reported growing interest in the PRODML and RESQML standards.

The Standards Leadership Council was formed by standards bodies in the Oil & Gas industry to promote greater synergy in E&P data management.  To find out more about the Council, visit oilandgasstandards.org.

Step 1: Defining the client’s requirements

Our clients’ needs can range from testing the suitability of the proposed software to sharing knowledge within the client’s organisation.   In our recent Oil & Gas proof of concept, the main aim was to demonstrate the effectiveness of our Transformation Manager data migration tool in meeting the client’s petroleum data management requirements.  Specifically, the proof of concept demonstrated that it is possible to use Transformation Manager to efficiently implement and deploy industry data standards to different types of petroleum data files.

In our example, there were two types of files  - LAS and DLIS – to migrate in the context of the PPDM and WITSML industry standards.  The proof of concept accordingly aimed to deliver three scenarios:

• Embed transforms within near real-time message handling, particularly WITSML
• Data migration converting to LAS files to and from the client’s PPDM-based system
• Data migration converting to DLIS files to and from the client’s PPDM-based system.

Step 2: Defining the client’s input

In any project, we clearly define the requirements from the client before starting work.  In this example, the items required before the proof of concept began included:

• Business rules to define the required mappings, such as details of field mappings, lookups and error handling
• Samples of the data to be extracted, such as well header and log curve data
• Any relevant information about the source and target models, such as local choices in the use of PPDM
• Sample files in the required format, including LAS 2.0 and 3.0.

Step 3: Action planning

Our action plan followed a similar path to our previous proofs of concept.  It was followed through to implementation and included:

• A review of the information received from the client:
  • Each source and target data model, including format, connection options and sample data
  • Validation rules
  • Mapping rules
• Design of the workflow of the data through the integration process:
  • Document flow of data
  • Document the decisions made
• Design the integration for implementation within our Transformation Manager software
• Implement the integration within Transformation Manager, including testing
• Demonstrate the results to the client (see Step 4 for more details)
• Describe the process and usage of Transformation Manager to the user
• Create a report describing the concepts, outcome and potential usage within future projects.

Step 4: Proof of concept delivery

In the petroleum data management example mentioned above, our team delivered:

• A final, tested, Transformation Manager deployment pack which delivered the data transformation to the client’s specification
• A licensed version of Transformation Manager for internal evaluation
• A proof of concept report as described above
• Knowledge transfer to the client, including the use of Transformation Manager, a review of its capabilities and an overview methodology
• A presentation and demo showing key processes, outcomes and future options.

The result was a software component which can efficiently deploy the client’s data to a range of the most popular petroleum data management file formats.

To find out how we can meet your own E&P data management requirements, please get in touch.

Image via Free Digital Photos.