Home- Performance Examples
Performance Examples
Latest performance figures produced from a recent data migration project migrating corporate data from all of the regional data centres to a single corporate datastore for a global oil company:
When reading and writing to relational databases using a desktop PC with Pentium 4 processor and 512 megabytes of RAM typical speeds without the performance features described above can typically exceed 2000 reads or inserts per second for a database with medium complexity, and using some of the RDBMS advanced features can extend that to 4000 reads or inserts per second
Also transferring >8000 records and created >60,000 records in the references database in <2 mins
Another example was for an Investment banking customer and was running complex transformation logic:
The performance timings when transforming 50,000 CSV records into an Oracle database table are shown below.
- Database Inserts only (assumes no database updates are required) - no performance tuning: 182.31s
- Database Inserts only (assumes no database updates are required) - with database batching: 40.87s
- Database Inserts or Updates (TM checks whether a record already exists) – no performance tuning: 420.25s
- Database Inserts or Updates (TM checks whether a record already exists) – with database batching: 173.67s
The source is a flat file (A global car manufacturer HQ parts data, containing approx 350,000 records). Below are timings found when writing (inserts only) to these databases.
Records per minute:
MySQL 936
Access 18115
Oracle 14442
With Allowbatching() on, I then get the following:
Records per minute:
MySQL 1406
Access 22443
Oracle 58149
TM provides a number of advanced features to improve performance and scalability of transforms when reading and writing RDBMS, XML, and Java:
- Insert Only Mode
- JDBC Optimisations
- Multiple Tables Batching
- RDBMS Table Plug-ins
- Completely Scalable XML
- Model Specific Java Adapter