Next >> << Prev

10. Pre and Post Processes: Additional custom processing

Pre and Post Processes are a powerful way to add custom processing to exports and loads. They allow you to execute pretty much any kind of external process before and after the run of exports and loads.

These processes can be a combination of:

A use for pre and post processes can be to add reporting functionalities. For instance you can insert rows in a control table to report on exports or loads taking place, you can also send an email to inform people about it.

Another use can be to prepare a job by cleaning up data before a load or downloading the data to load from an FTP server (or uploading the exported data). You can also do complex processing by running exe, batch files or VBS scripts.

This gives you a lot of flexibility around your exports and loads.

Note: Pre and Post Processes are only available in DMT Enterprise Edition.

Adding and Editing Pre and Post Processes

Pre and Post Processes Link

Pre and Post Processes can be added to almost every type of groups and jobs.

Select the requested group or job and click on the Pre & Post Processes link. It opens the Pre and Post Processes form, where you can define the Pre and Post Processes to use. The check box indicates if Pre or Post Processes are present.

You can combine any of the 5 types of processes together (e.g. Run an SQL statement AND run an executable AND send an Email).

Pre and Post Processes Tabs

When you open the Pre and Post Processes form, the top 2 tabs are for Pre Processes or the Post Processes, and the tabs below fir the type of process to use. Select the ones that apply.

Pre and Post Processes Tabs
 

10.1. Run a DMT Export or Load


This process runs any DMT Export or Load

Export or Load Pre or Post Process
 

Load Name: Select the name of the Load you want to run.

Note: You can run Loads from a DMT Export, and Exports from a DMT Load.

 

10.2. Run an SQL Statement


This process executes an SQL statement against any data source.

SQL Pre or Post Process
 

Database Connection: Choose against which database the SQL statement is run.

You can define the ODBC connections in the Settings Tab. You can refresh the ODBC connections by right clicking on this field.

SQL Statement: Enter the SQL statement to run. You can enter any SQL statement understood by the ODBC driver you're using to access your data source (including stored procedures and package methods).

In the example above, the pre process clears the contact records that don't have a contactID.

 

10.3. Run an Executable


This process executes any executable process (.EXE, .BAT, .VBS, etc.).

Executable Pre or Post Process
 

Command Line: Enter the path to the executable file.
You can enter the file name manually or click on the button on the right ( > ) to open the file selection form.

Arguments: Enter the arguments to be passed to the executable when launched

Wait: Tick this box if you want the group or job run to wait until the process terminates. In the duration field, enter the number of minutes to wait until the process is killed if not completed.

In the example above, the pre process executes the "Prepare Data file.Bat" with the /DEBUG argument. It waits for the process to complete before starting the group or job run. If the process is not completed after 3 minute, it’s automatically killed.

 

10.4. Upload or Download a file via FTP


This process uploads or downloads a file via FTP.

FTP Pre or Post Process
 

FTP Connection: Choose which FTP connection to use for the transfer.

FTP connections are defined in the Settings Tab. You can refresh the FTP connections by right clicking on this field.

Timeout: Enter the timeout (in seconds) to use when accessing files through FTP.

Upload/Download: Choose here if you want to upload or download a file.

Source and Destination File: Choose the source and destination files and folder (on the FTP server and the local machine).

In the example on the left, the post process uploads the extracted data file "New Contacts.csv" to the FTP server in the Exported folder.

 

10.5. Send an Email


This process sends a notification Email.

Email Pre or Post Process
 

To: Recipient(s) of the email. You can enter multiple emails separated by comas, semi columns or spaces.

Subject: Subject of the email

Attach extracted Data File: Check this box to automatically attach the extracted data file to the email (in case of a load, it would automatically attach the loaded data file).

Compress Attachment (Zip): Compress the attachment in ZIP format to make it smaller to send (recommended).

Email Content: Body of the email i.e. the text of its content.

Note: You can use Run Variables in the Subject and the Content.

Action if Pre and/or Post Process(es) Fail

In case Pre and/or Post Processes fail, you can tell DMT to act upon it.

Pre and Post Processes Action if Fail

Stop the Run: An error is generated and the current export or Load is stopped.

Generate an Error: An error is generated but the current export or Load continues running (it might however stop in case it's set to not allow errors or the maximum number of allowed errors is reached).

Generate a Warning: A warning is generated and the current export or Load continues running