Release Notes for TM v3.3

Release 3.3 (released August 2004) contained the following new features:

Database Performance Improvements

In this release, significant improvements have been made to improve the performance of writing to Relational Databases.

Three new techniques have been introduced:

  • Pattern batching
  • target plug-ins
  • source plug-ins.

Pattern batching is enabled using a simple call ALLOWBATCHONCONSTRAINTS which is now made in the ‘$document’ transform. Once pattern batching is enabled, TM no longer sends individual updates to the database but constructs patterned groups of items that both satisfy database constraints and yet take full advantage of the much superior performance of the JDBC connection in batch mode.

In general, pattern batching increases performance by a substantial margin - a factor of ten is not unusual. Pattern batching should normally be turned on - it can be used as a diagnostic aid, particularly when using the debugger. Further details may be found in the JDBC Guide.

Table handler plug-ins

Both Target and Source table handler Plug-ins are available. These are very powerful as they allow complete behavioral redefinition of the TM JDBC Adapter on a table by table basis using a table handler. Once installed, the table handler may field requests to find, update, create or retrieve a value for an object - the handler may also take special action on start up or shut down.

However a table handler is not required to necessarily field all these requests. One type of very useful table handler, for example, only fields requests to find the object. It uses its internal knowledge of the table population to do this far more efficiently than can ever be done through direct SQL JDC calls (typically by bulk reading the required identification information into memory in its connect method).

Table handlers may also intercept an operation and then return the object to the default table handler and allow the default table handler to complete the operation. A typical use of such a handler is to create some kind of system audit.

ETL supply three examples of table handlers as source code, namely:

  • samples.jdbcPlugIn.memEx.TMMemExSQLTableHandler
  • samples.jdbcPlugIn.quickFind.TMQuickFindSQLTableHandler
  • samples.jdbcPlugIn.srcSimple.TMSimpleSrcSQLTableHandler

Further details may be found in the JDBC Guide.

 

Configurable Flat File Readers

A set of Configurable Flat File Readers is provided by ETL to access Comma Separated Variable (CSV) and various Flat File formats. The set of adapters have been chosen to cover many of the most common formats in use and are easily configurable by non-Java consultants and data experts. Once an appropriate pre-defined adapter has been chosen the remaining configuration may be done easily by the data consultant.

TM also provides Open Interfaces which allows rapid production of meta model loaders and instance readers and writers to any source or sink of data through simple interfaces defined in Java (refer to Loader Open Interface Guide and Data Open Interface Guide with respect to this ).

A variety of pre-built configurable flat file samples are provided by ETL as follows:

samples.openinterfaces.csv.columnLoader.TmColLoader File and Column Loader

samples.openinterfaces.csv.hdrRead.TmCSVHdrReadAdapter Database Dump file Loader

samples.openinterfaces.csv.structuredLine.TmSLineLoader Structured Line Loader

net.etltm.tml.helper.TMExampleBlankModel Blank Model Loader

samples.openinterfaces.javaErr.TmJavaErrWriteAdapter Text Error Output Loader

The first three of these are described in the User Guide. The remaining two are not yet described in detail.

All code described in this document is also available as source code (source code and compiled code made available with the Transformation Manager distribution) and thus may be modified very easily to suit individual customer requirements.

 

Traceability (beta functionality)

A backward traceability (beta) feature is added in the release v3.3 of Transformation Manager (TM). In this context traceability is the discovery of what source attributes and transforms contribute to the creation of a target attribute (backward traceability), and which target attributes and transforms operate on a source attribute (forward traceability). This feature operates over a pipeline of one or more transformation projects and enables users to quickly identify the contributing factors or dependencies that exist when resolving data quality issues. A project that contributes values for another project is termed a feed project. This feature is designed primarily to work with text repositories, although it is also backwards compatible with relational repositories with functionality limited to a single project.

Traceability takes full advantage of the high level abstraction of the mapping language used in TM. In particular traceability operates across RDBMS, XML, Java, Generic models identically. It therefore enables the tracking of a pipeline of transforms, for example an XML file which transforms to database and Flat File report, and where the database transforms to Java Classes and another database, and then the second database transforms to XML and a generic target.

Traceability is described in more detail in the on-line help.