Release Notes for TM v3.4

Release 3.4 (released December 2004) contained the following new features:

 

Group Adapter

A Group Adapter allows a number of models, not necessarily of the same type, to be used together as the sources of a transform. A wizard will allow you to select a number of models to be grouped. During this process a unique 'prefix' for each model will be assigned. Once a group or composite model has been created it may be used as the source of any TM transform. See the TM Quick Start Guide for more information.

 

Logging Improvements

Transformation Manager provides functions that support limited logging to text streams. There are three distinct type of logging supported:

System Generated Logging

TM generates system messages as a matter of course, for example the activity times or full report of activities can be requested, where an activity is a stage of transformation processing. The extent of the system generated logging is controlled by setting a logging level and the default settings are set for normal usage.

Error Recording

It is possible to set up Transformation Manager so that all errors thrown by User Defined Functions (UDFs) and XSD violations are Recorded, rather than Thrown or Noted. This recording process means that a string representing the error is simply added to a list or 'recorded errors' maintained by Transformation Manager. It is then the transform designer's responsibility to then read query or otherwise items from this list.

Logging Functions

TM provides a number of logging functions compatible with Java logging mechanisms such as Log4J, Java 1.4 logging, etc. By default, TM will use Log4J logging, but alternative logging mechanisms can be plugged in at run-time through the Java API or by specifying a system property, the latter being useful for setting the logger implementation used by the test tool or the debugger.

 

Traceability

Two forms of traceability are provided, one that relates to meta-data and the other that relates to instance data.

Meta-data Traceability

Data changes can occur often as a result of a number of transforms which operate in sequence. One form of meta-data traceability is currently provided: data lineage. This provides a way of discovering quickly the contributing source attributes and transforms that affect the results.

This type of analysis is not limited to one project. A very important feature of the meta-data traceability is that a series of projects that transform data through a series of stages may be linked together in a pipeline. A typical use of meta-data traceability is to perform impact analysis - we can ask questions such as, "If this field input data were to change, which fields further down the pipeline could possibly be affected?”

Instance Traceability

Instance traceability permits the user to discover how individual items are being transformed. It allows the system to produce a very intelligent trace showing exactly how any particular instance item in the target has been created.

A particularly powerful way of using the instance traceability feature is in conjunction with the new regression test system, which is integrated into the Transform Design Tool itself. This regression system allows you to set up a series of source items, expected target items and transforms, and automatically to execute the transforms and compare the results.

TM Design Tool supports three different ways of using instance traceability:

  • Raw Instance Traceability via SML
  • Instance break points in SML debugger
  • Integrated Traceability and Regression test system

 

Regression Suite – Test Driven Development

The regression suite has now been added to TM Design Tool, to provide features that will allow equivalent transforms to be developed more quickly, by first highlighting the differences in the output and the TM statements that have caused the differences.

There are currently two tools in the Regression Suite:

  • Load External Difference File...
  • Run Tests Now...

Run Tests Now is the main entry point to the regression test system. You will be invited to choose a properties file describing the regression test. Once that has been selected, the test system is run immediately.

The regression suite may also be used to test performance. In this case, typically, the comparator will be turned off and each test may be run a number of times.

Once the regression test system has finished, you will be presented with a dialogue showing the results of the test run.

 

Additional Techniques for Sharing Transform Code

TM now supports three techniques for sharing transform code:

  • super/sub project
  • redirection
  • inclusion

Super / Sub Projects

The concept of the Super Project is similar to that of a super class in object oriented programming languages. Projects may inherit from one or more Super Projects. The Super Projects specify the base behaviour, while the Sub Projects refine that behaviour.

Note that the distinguishing feature of the super/sub project system is that this inheritance may take place within transform itself - i.e. a subtype transform may redefine or modify the behaviour of a super type transform between the same two points. In general, you will want to use the super project system only in situations where the super type transform is relatively complete itself and where the subtype transforms are small, and thus used merely to redefine behaviour in some specific areas. The archetypal situation would be for the super type transform to be developed at a single Engineering Centre and then shipped out to a number of world locations. In these world locations local variation is then handled by the addition of a very small number of subtype transforms.

Further information is provided on this topic in the ‘How to use Super Projects’ section of the online help.

Redirection

In comparison with the super /sub projects, redirection is a very simple concept. Redirection allows the text storage of any particular transform to point at another file location. The redirection feature thus allows a number of transform projects to share transforms between common points.

Inclusion

Inclusion is the simplest concept of all. Inclusion allows any transform to include any number of arbitrary blocks of text. In this sense, inclusion is extremely similar to the #include concept found in some computer languages.

 

WITH…END_WITH construct

To help keep the number of transforms down and also make the project more understandable, add a WITH <sourcepath> ... END_WITH construct, eg

transform A <- S;

a := b;

self := rT.rU;

transform A <- U;

a1 := c;

self := rV;

transform A <- V;

...

would become a single transform

transform A<-S;

a := b;

WITH rT.rU;

a1 := c;

WITH rV;

...

END_WITH;

END_WITH;

Hence, in one (or at least fewer) transform(s), you can see how A is being built.

 

Typed jObjects

A Typed Java Object (jObject) feature enables transforms to write to Java objects that have been created by the client architecture. The principal use of a typed jObject is that it is written to as a result of calling a user-defined function. It is also possible to manipulate the jObject as if it were an object created by TM.

Hence, for example, TM could retrieve an address instance from a user-defined function that has been supplied by the client architecture. It could then modify the address instance using values of the transform source, e.g. an XML instance. The address could then be returned as the transform target, or it could be related to another Java instance created by TM, e.g. a person parent object.

The Typed Java Object feature is useful when integrating Transformation Manager into an existing system that already creates part of the target. It is a very advanced feature and should be used with care when absolutely necessary. For example, its use complicates the understanding of the transform and in part hinders the use of TM as documentation tool.

 

Support for Java Constants

It is not uncommon that Java models contain static constants to represent those items that were described as enums and symbolic constants in more primitive languages. In Java, this is done by using the static and final modifiers, as appropriate. Take, for example, this small class:

package samples.UDFExamples;

public class XType

{

public final static String STRING_CONSTANT = "string constant";

public final static int INT_CONSTANT = 42;

public final static String[] STRING_CONSTANTS = new String []{"red", "blue", "green"};

public static String NOT_FINAL_STRING_CONSTANT = "non final

string constant";

public static String UNASSIGNED_STRING_CONSTANT ;

public String myData="data";

public java.sql.Timestamp myTimeStampData; }

Note that there is some degree of ambiguity in the Java language. Here the first three items are, without doubt, meant to be considered as symbolic constants. There is some possibility that the next two items are also to be considered as symbolic constants. The final two items, not being static, are never considered as symbolic constants.

To handle this ambiguity, TM Design Tool provides one flag in the 'Model Import Settings' ( see Global Preference Editor | Model Import | Java : treat all statics as constants.). This flag is set true by default. In this condition, all static values which are assigned an initial value are considered as potential constants and loaded into the model appropriately. Thus, in the above example 'NOT_FINAL_STRING_CONSTANT' will be loaded as a constant, while 'UNASSIGNED_STRING_CONSTANT' will be loaded as though it is a conventional class data member. If the 'treat all statics as constants' is false then all non final static values are treated as class data members - even if they have been assigned an initial value. TM ignores lexical conventions (i.e. the fact that UNASSIGNED_STRING_CONSTANT is in block capitals is not used).

 

 

Automatic Back-up

TM Design Tool contains an integrated back-up system. This system can only be used with text-based repositories. By default, this back-up system is turned off.

The back-up system is controlled by two values set in the Global Preferences Editor | Text Repository Storage pane. (The Global Preferences Editor may be reached by pressing Ctrl+D when no particular project is open). The two settings relevant to back-up are: Create Project Back-ups and Project Back-up Path.

The first setting must be set to true for automatic back-ups to be made. The second should be the name of the pre-existing path on which you wish to place your back-ups.

 

Deployment Tool

The TM Deployment Tool provides a stand-alone environment for scheduling and running transforms. A Deployment Task is a specific transform, with source and target defined, any parameters needed specified, and a schedule for running the transform.

The main dialog of the Deployment Tool lists the current tasks that have been created by the user and tasks that have been created previously will be read in automatically when the Deployment Tool is launched.

The following tasks are available:

  • Create New Task
  • Edit Selected Task
  • Delete Task
  • View Task History

 

The getTime Function

GETTIME([dateFormatterString]) returns the current date / time as a string.

dateFormatterString

This optional argument specifies the format of the returned string. This argument is passed internally to a java.text.SimpleDateFormat object. See the java API documentation for a full description of this class and the arguments to pass in.

Examples:

GETTIME() returns: Thu Sep 23 14:04:20 BST 2004

GETTIME('dd/MM/yyyy \'at\' hh:mm:ss z') returns: 23/09/2004 at 02:09:37 BST

 

Re-synchronize

This performs an evaluation of the current transformation code. Any errors in the transformation code are displayed in the error pane.

TM Design Tool analyses the transformation code in two stages. The first of these is a line by line analysis and, as such, is not affected by errors in the code, even those which are above the current insertion point. The second mechanism is an interline analysis, which is used by the intelligent insert within WITH and FOR_EACH blocks.

While Re-Synchronize will discover all the syntax errors it does not necessarily prove that the code is buildable. In particular, it does not identify missing or badly described functions, nor discover non-executable paths in the code. A full Build must be performed for this purpose.

 

Second read-only copy of TM Design Tool

TM now supports a second read-only copy of the Design Tool.

 

INCLUDE Feature Relative Paths

The INCLUDE feature may now be specified as a relative path so that it is no longer necessary to build knowing the Windows install location. This is achieved by using the $CURRENTREPOSITORY command. i.e. instead of writing:

INCLUDE('c:\Program Files\tm\repositories\TradeIntelligence#1\common\trade\fra_irs.txt');

you may now write

INCLUDE('$CURRENTREPOSITORY\common\trade\fra_irs.txt');

 

Java Case Switch Values

TM now supports Java case switch values. It also allows expressions as case “constants” provided they are bracketed.