Release Notes for TM v3.2

Release 3.2 (released June 2004) contained the following new features:

 

Text Repository

The major improvement introduced with this version of TM is to allow the introduction of the text-based repository. The advantages of the text-based repository are that:

  • It provides a multi-user environment for development of transforms. The underlying repository can be stored directly in a version control system - for example CVS - and checked out in read or read/write modes.
  • Projects or models may be added or removed from a Transformation Manager text repository simply by copying the relevant files.

The Text Repository provides a much faster working environment for, e.g. loading of models, building projects etc. A recent performance test when loading a relatively complex model from an Oracle schema demonstrated that the Text Repository can be loaded 5 times faster than using a Relational Repository.

When creating a new repository users may select which type of repository to use. In addition, by selecting File | Convert Repository a user can convert an existing relational repository to a text-based repository.

From this release users are encouraged to user the text-based repository (refer below the restriction for users who use IPRs (Inter Project Relationship), although the relational repository is still supported.

The method for converting from a text-based repository to a relational repository is using the project import/export features.

Full instructions for using the text-based repository are included in the online help.

 

External UDFs

Users can access functions that they have developed externally to TM. There are three types of functions that can be imported:

  • Static Explicit (public static functions that have net.etltm.TMContext as their first parameter)
  • Static Implicit (public static functions)
  • Public Functions

The management of accessing external UDFs is done via the Project Preferences dialog on the UDF Management pane. Settings include:

  • The classes to be imported
  • Whether the imported functions are Type Maps
  • Abbreviations to use when importing the functions
  • Jar file name, if the functions are to be imported from a jar

The functions are read into TM when a project is opened. They appear in the User Defined Functions dialog and are used in the same manner.

 

Lookups Using JNDI

When lookup functions are generated using the Lookup Wizard, the lookup uses a connection alias, based on an underlying JNDI connection.

The connection alias refers to a logical database connection that will be used by the lookup. At runtime, a database connection for the alias will be requested by this name. The first request will obtain a physical connection, and subsequent requests of that name will return that same physical connection.

 

Improved Loader

Major performance improvements have been introduced when loading RDBMS models of reasonable complexity, particularly where one schema refers to another. Although Transformation Manager made relatively few queries to read the metadata, where one schema refers to another it was found that excessive database queries caused the loading to be unnecessarily slow. It was determined that the metadata should be read once and stored in memory for future use. In addition, the loader produces a text record of the metadata read which is available for analysis in case the metadata fails to load as expected.

The Loader Performance improvement has been measured and compared with TM performance. When loading a fairly complex model from an Oracle schema the time taken to load was measured for the following three scenarios:

  1. Previous TM version using Access repository - 1hr 2m
  2. Version 3.2 beta using Access repository - 5m 30s
  3. Version 3.2 beta using Text repository - 58s

Significant gains have been achieved when continuing to use an Access repository, and even better improvements using the Text-based repository.

 

Transform Priorities

Each transform in a project can be given a priority. The transform priority serves two purposes:

  1. It indicates the order that independent transforms should be run.
  2. It provides a mechanism for excluding certain transforms from generation.

Priority is set in the Transform Properties dialog. A value of between +- 100,000 can be set. The default value for each transform is 0. A setting in the project preferences sets the minimum priority which will be run (refer Online User Guide).

 

Debugger Plug-in

Transformation Manager Debugger is a separately priced plug-in. It is available with this release without additional license for potential customers to continue evaluation and testing.

The TM debugger can debug projects from within Transformation Builder, and can also be attached to deployed Java code.

Debug sessions are configured using the Debug Configuration Wizard. This allows users to retrieve previous configurations, and to create new ones. A project may have multiple debug configurations.

Breakpoints, and Conditional Breakpoints can be set in a project via the Transformation Manager User Interface. Debugging sessions are controlled using the Debug Controller. This allows users to step, pause, run to breakpoint, etc.

When execution is paused at an execution point a number of tabs present information to the user:

  • Console Output
  • Exceptions
  • History
  • Reader
  • Source
  • Stack
  • Target
  • Variables
  • Writer

Each of these provide detailed analysis of one aspect of TM.

 

Transform Navigator

The Transform Navigator dialog is provided to as an alternative method for discovering relationships between the transforms that make up a project.

The dialog contains a tree representation of the transforms. Each node in the tree represents a transform, and by looking at the children of any particular node, users are able to see the transforms that this one will call (cascade).

Each node has an icon which indicates whether the transform is independent, dependent, or contains an error.

Transforms can be selected from the tree to be displayed in the main Transform Manager editor.

 

Make Option

Make is a new menu option, which performs a smart-build. This results in a faster, optimised build and is useful when editing projects if a user wants to check for syntax errors without generating the Java transformation code.

 

Batching Improvement

Significant improvements have been made to improve the performance of writing to Relational Databases extending TM batching when writing to multiple tables. In addition, a new ‘%’ operator has been introduced. This operator may be interpreted as a find or create identification '*' operator or the cancel identification '-' according to a mode of update configuration parameter. This allows one SML transform definition to be used identically in an update or bulk load scenario.

A number of other batching improvement features have been added as beta features in release v3.2 and these are described in the JDBC Guide, new features section.