Basic pattern
Because transforms are called on a instance by instance basis it is easy to implement the basic loop design pattern:
////either built-in adapter code
TMDHIterator anIt= getObjectsOfInterest(aTMSourceQuery);
////or generic user code -
TMDHIterator anIt= user code returning an iterator of TMReadElement's
while (true)
{
TMReadElement aInputDh = anIt.getNext();
if (null==aInputDh)
{
break;
}
TMWriteElement aOutputDh
= (TMWriteElement )myTMTransformExecutor.transform(aInputDh);
// further user processing such as:
myUserWrittenTarget.consumeHndle(aOutputDh); //consumeHndle is a user written
// function
}
This allows fine control of the transform process. Note that simple transforms will be stateless outside the call to "transform". However more complex transforms will require that part TMTransformExecutor retains "stalled" handles and also references to fkey primary key relationships.
Variations
Simplest Pattern
The easiest sort of generic source to target the loop pattern is:
while (true)
{
TMReadElement aInputDh
= (TMReadElement)anIt.getNext();
if (null==aInputDh)
{
break;
}
myTMTransformExecutor.transform(aInputDh);
}
// then some call to myWriteAdapter.'getData.'
This pattern is probably more common should the target be a non-generic adapter. Here the return value from the each call to 'transform' is simply ignored. We can do this because the write side handle is already 'included' in the relevant write adapter. For RDBMS systems this means that data will eventually be flushed to disc, while for other types of adapter some kind of getData method is available to getJava or SAX events after the loop has finished.
More Complex Patterns
However, it is also possible to use the return value from the transform within the group for increased flexibility. The principal reason you would wish use these options with the in-built Java or XML adapters is to allow the transformation process to scale beyond limitations of memory. For databases, on the other hand, this approach is more likely to be adopted because the user wishes for a more intimate involvement in transaction control and JDBC optimisations. (Note that by default the ETL database handler is fully scalable anyway).
Thus with the in-built Java adapter as target the loop could read:
while (true)
{
TMReadElement aInputDh = anIt.getNext();
if (null==aInputDh)
{
break;
}
JavaDH aDH = (JavaDH) anIt.transform(aInputDh);
// Now calling aDH.getData() will get us the objects in the userdefined classes.
//Itmaybe that we can 'consume' these objects in some way (i.e. send serialised forms
//of them via CORBA or other network protocols) then free up the object by cutting its
//reference to the TMJavaWriteAdapter using 'aDH.free()'. This allows such a transform
//to be totally scalable.
}
Thus with the in-built XML adapter as target the loop could read:
while (true)
{
TMReadElement aInputDh = anIt.getNext();
if (null==aInputDh)
{
break;
}
XMLDH aDH = (XMLDH)myTMTransformExecutor.transform(aInputDh);
// Now we have a a choice we can either call
//aDH.getTreeFragment();
// get the sax events for part of xml tree which is represented by this current XMLDH
//followed by
//aDH.free();
// to remove it from the write adaptor.
//OR we let the answers build up in the write adapter itself. Thus
// maybe at intervals, within the execution of the while loop we
// can call something like:
aTargetAdapter.getRootAndSerialise(); // sax events for whole adapter
aTargetAdapter.clear();
}
Thus with the in-built RDBMS adapter as target the loop could read:
while (true)
{
//maybe rdbms starting transactions/batching etc
TMReadElement aInputDh = anIt.getNext();
if (null==aInputDh)
{
break;
}
DBDH aDH = (DBDH )myTMTransformExecutor.transform(aInputDh);
// rdbms stopping/committing transactions/batching etc
}
Note that there may be many calls to getObjectsOfInterest followed by particular types of processing.
Controlling what is returned by ObjectsOfInterest
The only parameter to getObjectsOfInterest is a TMSourceQuery. This is, potentially, a very complex structure that allows the user to perform a very complex set of selections on the source. (In conjunction with TMQueryPropogator this selection can even be defined in target terms - see TMQueryPropogator). Simple uses include:
Getting One Domain
Using the static factory class method in TMQueryObjectsFactory to describe a selection of all objects from one domain.
TMSourceQuery aTMSourceQuery =
net.etltm.qp.TMQueryObjectsFactory.createSourceQuery(domain));
anIterator=adapter.getObjectsOfInterest(aTMSourceQuery);
Getting Several Domains
Using the static factory class method in TMQueryObjectsFactory to describe a selection of many objects (e.g. 'Account', 'Name', 'Transaction') from several domains.
Vector v = new Vector();
v.add("Account");
v.add("Name");
v.add("Transaction");
TMSourceQuery aTMSourceQuery =
net.etltm.qp.TMQueryObjectsFactory.createSourceQueryV(v));
anIterator=adapter.getObjectsOfInterest(aTMSourceQuery);
Selecting attributes
Limiting the selection to certain attributes on a domain (e.g. 'payor', 'payee' and 'date') on any particular class (e.g. 'transaction')
TMSourceQuery aTMSourceQuery =
net.etltm.qp.TMQueryObjectsFactory.createSourceQuery(transaction));
aTMSourceQuery.addAttribute("payor");
aTMSourceQuery.addAttribute("payee");
aTMSourceQuery.addAttribute ("date")
anIterator=adapter.getObjectsOfInterest(aTMSourceQuery);
Getting all domains
This can be done with:
anIterator=adapter.getObjectsOfInterest(null);
But note that some adapters cannot implement this call - the JDBC in-built adapter, for example, does not implement this call - you need to know what domains (i.e. tables) you are selecting.
Convenience Classes
There are number of convenience classes defined for this system - mostly in the package 'net.etltm.helper';
TMDHListIterator
The class TMDHListIterator found in net.etltm.helper is a simple implementation of a TMDHIterator based on a Java list. If your underlying architecture allows you to represent your iterated objects in this fashion then this class provides a convenient way of building the TMDHIterators required by the above methods direct from a Java list.
AbstractTMReadAdapter
Extending the AbstractTMReadAdapter is the easiest way of building a generic read adapter. This approach is suitable provided that you are building an adapter to handle instances of moderate size. Thus if you can reasonably expect the adapter to hold all input objects in memory this is the recommended choice.
AbstractTMWriteAdapter
Extending the AbstractTMWriteAdapter is the easiest way of building a generic write adapter. This approach is suitable provided that you are building an adapter to handle instances of moderate size. Thus if you can reasonably expect the adapter to hold all output objects in memory, before serialising them or sending them to another component, this is the recommended choice.
Notes on the Transform Methods
These basic transform calls are very simple - each takes a source item and returns a target item.
Both these calls return a DataHandle, i.e. single items of type TMDH (which can be cast to TMWriteElements or your own types as required).
If you have designed a transform so that it directly "unNests" then it may create a target array rather than a single TMDH. In this case the form to use is TMTransformExecutor : transformM(TMReadElement).
This will, potentially return a list of TMDH's. If you use the transform method and your transform does "unNest" then no exception will be thrown but you will only ever get the first element created.
Note the single parameter call runs any defined transform from TMReadElement's source domain. If there are two or more independent transforms from the same source domain (not a design pattern encouraged but nevertheless not excluded) then it will simply choose one of them randomly. (Future versions of the product may support mode and priority to a help in making this process more determinable).
The second method resolves this ambiguity and attempts to run the defined transform from TMReadElement's source domain to the specified list of target domains. The names in TargetDomainList should be in priority order. If "insist" is true then this is the only operation it will attempt - it there were no such defined transform it will simply throw an error. If "insist" is false and if it fails to find any successful transforms to the domains in the target Domain it will continue and perform any transform from TMReadElement's source domain.
If targetDomainList is null then 'insist' is ignored and the behaviour is as the single parameter call.
Using TM adapters outside of TM itself
All TM adapters, include the in-built adapters can be used out side TM itself within your own software architecture. They are simply re-usable components.