Home- Release Notes for TM v3.6
Release Notes for TM v3.6
Release 3.6 (released June 2005) contained the following new features:
Import / Export
TM’s Import / Export feature allows projects and/or models to be exchanged between different text repositories. Any projects that are exported include any internal UDFs created using TM Design Tool. An Export Wizard allows selected projects and models to be exported from a text repository to a zip file. Similarly an Import Wizard is provided that allows selected projects and models from a zip file to be imported to a text repository.
Change Model
A user may change either the source or target data model within a transform project. During this process the user may preserve information previously added by the user (relationships and advanced relationships fields). A summary log may also be produced which documents the full effects of the model change process. This feature replaces the previous Model Clone feature of TM.
Please refer to the Change Model Guide.
Restore Backups
A backup of TM Text Repository may be automatically saved as each change occurs. Restore enables a user to select from a list of Backups (arranged by date) and restore to the current repository. Either a project or model can be selected for restore and it will overwrite if that selected already exists.
TM Deployment Tool Improvements
TM Deployment Tool can now sequence transforms and is also able to deploy transforms which have been built or can launch Java code by executing the 'Main' method. The TM Deployment Tool now records and displays in a tabbed pane the output generated by the transforms, which provides performance and execution details, similar to that available in the TM Test Tool.
New TM Functions
Over 50 new functions have been added to TM.
1. Functions pertaining to array lists
ADDARRAYLISTS
This function adds together the contents of two JObjects and returns a java.util.ArrayList. The first parameter passed into the function may also be modified. The first two parameters passed into the function must implement the interface java.util.Collection.
If AllowDups is true then the resulting array list may contain duplicate items. If AllowDups is false then duplicate items will not be added.
ADDTOARRAYLIST
The first parameter passed into the function must implement the interfacejava.util.Collection. The first parameter passed into the function may also be modified.
This function adds any object to an array list. If no position is specified then the object is appended to the end of the list.
SORTARRAYLIST
This function sorts all the items in the list using the Java method java.util.Collections.sort().
All items in the list must implement the java method comparable method. If this is not true then the list is not sorted (standard TM objects are sortable).
SUBTRACTARRAYLISTS
This function subtracts the contents of the second list from the first list and returns the resulting, possibly smaller, list.
The two parameters passed into the function must implement the interface java.util.Collection.
CREATEARRAYLISTFROMAGG
This function iterate all items on the dot separated path (which must be supplied in quotes) and adds them to the resulting array list.
If no instance position is supplied, then the path is assumed to start from the source entity. Otherwise it starts from the supplied instance.
This function will return a design time error should the path not be visible.
GETFROMARRAYLIST
This function returns the element at the index position in the java.utilList supplied.
The indexes are numbered from one.
A blank object will be returned if the object is out of range.
GETSIZEOFARRAYLIST
This function returns the size of the array list.
REMOVEFROMARRAYLIST
This function removes the element at the index position in the java.utilList supplied.
The indexes are numbered from one.
No change is made if the object is out of range.
REVERSEARRAYLIST
This function returns an array list with all the elements in the opposite order.
GETCOLASJLIST
This function extracts a column from an existing named Matrix as an ArrayList Jojbect.
GETROWASJLIST
This function extracts a row from an existing named Matrix as an ArrayList Jojbect.
GETNATIVEARRAY
This function create a Java native array of objects (or inbuilt simple types) from an input that must be a jObject implementing java.util.Collection . The resulting native array consists of objects each of which implement the type specified in ArrayTypeAsString.
If the value supplied in the Input though they cannot be converted then the DudObject is used - note that the DudObject object will be converted into a suitable primitive showed the array be a primitive.
For example: GETNATIVEARRAY(aListOfNumbers,'java.lang.int',-1);
JMAPADDKEYVALUE
This function adds a key/value pair to any JObject which implements java.util.Map. It returns null if the item is a newly added - otherwise it will return the old item which had the same key value.
JMAPCONTAINSKEY
This function returns true if the key is present in the existing JObject which implements java.util.Map.
JMAPCONTAINSVALUE
This function returns true if the value is present in the existing JObject which implements java.util.Map.
JMAPGETBYKEY
This function returns the jobject if the key is present in the existing JObject which implements
java.util.Map. Otherwise it returns a null JObject .
JMAPGETKEYSET
This function returns a java.lang.ArrayList containing all the keys.
JMAPGETSIZE
This function returns the size of the jobject which is input.
JMAPGETVALUESET
This function returns a java.lang.ArrayList containing all the values.
MAKEARRAYLISTFROMNATIVEARRAY
This function takes any jobject which is a native java array and returns java.lang.ArrayList containing the same items in series. Both object and primitive arrays are handled identically. If the parameter is not an array then a java.lang.ArrayList containing all item, the input item, is returned.
2. Database Specific Functions
IGNORENONEXISTENTCOLUMN
This function instructs the JDBC data base read adapter to ignore columns that do not exist in the source. All values read from these columns will be 'null'. If this functions is not called then by default the JDBC data base read adapter will throw an exception as it encounters the missing columns.
IGNORENONEXISTENTTABLE
This function instructs the JDBC data base read adapter to ignore tables that do not exist in the source. All values read from these tables will be 'null'. If this functions is not called then by default the JDBC data base read adapter will throw an exception as it encounters the missing tables.
SETAUTOCOMMIT
This only affects database systems. If set to true then the JDBC drivers are set to Auto commit.
SETEXECUTIONMODE
This call is only relevant to database targets. Using this call the entire system may be set into INSERT or UPDATE mode. In insert mode only insert operations are performed, in update mode only update operations are performed. Use of this call requires an expert understanding of how TM operates over databases.
SETGENERICDATABASE
This call is only relevant to database targets.
SETDELETEMODE
This call is only relevant to database targets. Using this call the entire system may be set into DELETE mode. In this mode Transformation Manager ignores all create operations and all update operations are turned into delete operations. The system may be placed into delete mode, or placed out of delete mode at any time. Note that delete mode has no connection with the operation of query propagation delete calls - these delete items from this source, whereas SETDELETEMODE is intended to delete items from the target.
SETBATCHONCONSTRAINTSOPTIMISTIC
This tuning setting is only relevant to database targets. Setting this flag to true may increase the speed of the system a target does not contain many constraints.
SETINVALIDATIONMODE
This method allows you to turn off all database updates. In place of actually performing the Updates the JDBC system simply compares what he finds in each identified row with the values that it would insert should it have performed the update. This facility this allows a user to rapidly verify that a given set of transforms would produce a particular population. It may be used in the pre-production stages of a transformation project to check that a new feed built using Transformation Manager matches and existing hand written feed.
SETRDBMSAUTOTUNING
This function turns on RDBMS auto tuning. Currently all parameters are ignored and the system is placed in a faster executing mode , all source and target to make items are cached and batching is enabled with a batch size of 4000. Individual behaviours may be over ridden using other RDBMS tuning facilities placed lower down in the code. This function may only be called in a $ Document transform.
In future calling this function will also enable true Auto tuning - the system will adjust cache sizes depending on the behaviour of the system.
SETORACLESEQUENCERFORIDS
This call must be made in the pre-Document transform. This call is only relevant to database targets. It tells the system to start using an oracle sequencer as the source of surrogate key ID's .
3. XML specific Functions
SETIDENTEDXML
This call is only relevant to XML targets.
SETIMPLIEDCASCADESTOSTRING
In XML to XML transforms many of the transforms are merely implied - in other words they simply copy the $content of otherwise unspecified nodes.
In general, these implied transforms are completely type aware - i.e. they will use any type maps that have been supplied by the user and will also perform whatever the default type conversions for the various XSD types are.
Calling this function will set the system to consider all such $content to $content copies as simple string operations. This is faster if a simple string copy is all that is required.
SETMAXIMUMORDERINGNUMBER
This call is relevant to XML targets only.
It controls the number of patterns that are considered when the target is trying to satisfy the structure as defined in the DTD or XML schema. Increasing this number is generally used in very complex DTD's only - it may have an adverse effect on execution speed.
4. Other Functions
GETALWAYS
This function is used in SML to undo the effect of skipping null inputs.
Any source attributes fetched with the GETALWAYS will always be fetched - even if they are null.
GETATTRBYNAME
This SML function allows source or target objects to be read from dynamically named attributes. If no source flag is specified then with the assumption is that the call is reading from the source.
GETELAPSEDTIME
This function returns the number of milliseconds which have elapsed since TM began running. This can be useful in progress meters, and also can be used in conjunction with the call terminate to terminate the transform which has run for a specific length of time.
GETTHAT
This function returns the current transform target as a Java object. It is usually useful only if a transform is in fact writing to Java. Under certain circumstances this method may return a null - for instance if you are not constructing the transform.
GETTHIS
This function returns the current transform target as a Java object. It is usually useful only if a transform is in fact reading to Java.
LOADMEMORYTABLE
This may be used to load a target or source table into memory according to a set of very complex criteria. Please contact ETL Solutions should you wish to use this function.
MEMBEROF
This function returns true if the value (considered as a string) is present in the comma separated list provided.
MODULO
This function returns the mathematical modulus of input over base.
REPORTFALLBACKS
If 'report' is true then a report is made each time a generated Java adaptor is unable to handle a call and instead uses introspection or reflection. This kind of logging will slow the system down but is the useful diagnostic should choose suspect that the generated adaptor is not working correctly.
POPPARAMETER
This may be used to retrieve up a parameter value that has been pushed.
POPRETURN
This may be used to retrieve up and turn value from a cascade call.
SETBASESEQNUM
This sets up the base sequence number used by the $UniqueKey and surrogate key functions. This is relevant to database systems, and to all other systems which produce derived relationships (i.e. includes a target XML the and generic systems).
SETATTRBYNAME
This allows the SML transform dynamically to set an attribute by name. All booked in adaptors will throw an exception of should this be called on a non-existent attribute to - user defined generic adaptors may allow this behaviour to be a silent fail.
SETGENERATEDJAVAADAPTERBEHAVIOUR
This allows the user to customise the behaviour of a generated Java adaptor on it either source or target side. If allowFallBacks is false and the generated code cannot handle setter or getter then the system will throw a run type exception rather than using introspection or reflection.
The setting of logFallBacks indicates if such occasions should be logged.
SETSHOWORDERINGERRORS
This setting is false by default.
Setting this setting to true will slow down XML target operations. It will, however, display far more information should there be difficulties in creating a target which truly satisfies the structure as defined in the DTD or XML schema.
SETSTRICTORDERING
By default this setting is true - it may be set to false to encourage the system to write an XML target despite some difficulties in satisfying the structure as defined in the DTD or XML schema.
SETTARGETHASCONSTRAINTS
By default Transformation Manager assumes that a relational target has enabled constraints - and that these constraints are reflected in the meta model displayed in the design tool. However, in some cases the constraints have been relaxed. You can use SETTARGETHASCONSTRAINTS(false) to indicate that there will be no constraints in the target data store and Transformation Manager will be able to write to the tables far more quickly.
Do not use this setting if there are constraints - the execution will almost certainly fail.
SETUSEEXACTPRECISIONONFLOAT
By default Transformation Manager internally supports more precision in float and double than the official definitions of these XSD types. Setting this setting on causes the system always to respect the true maximum precision of float and double. There is a slight execution cost to invoking this option.
SKIPNULL
This function can be used to selectively skip attributes should the value be null. It should be considered the opposite of GETALWAYS.
STRSUBSTITUTE
This function corresponds to the string substitute operation available in Oracle.
REMOVEQUOTES
This function takes two parameters 'Input' of any type and an optional boolean flag 'isList'.
Its simplest mode of operation is simply to remove leading and trailing quotes (either ''' or '"') from any string.
Note that the quotes must be balanced.
If the 'isList' parameter is set to true and the Input parameter does not implements java.util.List, then the function will remove all quotes from any string object that is supplied and replace them with space.
If the 'isList' parameter is set to true and the Input parameter implements java.util.List, then the function will remove leading and trailing quotes from the string representation of all items in the list.
PRINTLN
This function prints a string representation of its input parameter on the standard output console. Unlike ECHO and various logging functions the output is immediate and is not buffered.