Skip to main content
Skip table of contents

Delta Load Combinations -- Upsert, Replace, Append, Load All

ETL+ delta load options are Load Upsert, Load Replace, and Load Append. Each load option has limitations but can be combined for better coverage. 🚧 Work in progress

image-20240603-202209.png

Load options on the ETL+ Extract, Transform and Load (ETL) Page

Delta Load Combination Scenarios

  • Load Upsert run daily with a Load All run on weekends.

Upsert Example: Column with Last Modified Value = LastModifiedDateTime

  • Load Replace run daily with a Load All run on weekends.

image-20231125-004433.png

Example 1: Load Replace with Target Value or Constant field configured to process records from the last 60 days.

Load Types Contrasted

Advantages & Uses

Limitations

Load Upsert

Fast, can be run many times a day. Updates existing records or inserts new records.
See Force Replace when Upsert option below.

Does not delete records from target/data warehouse table

Load Replace

Retains a history of changes.

Good for keeping a history of records from the source table.
See Force Replace when Upsert, default for others option below.

Can save several versions of source records on the target data warehouse table. Does not use a unique key to identify the target records to replace. (See example 1 below).

Load Append

Good for keeping a history of records from the source table.

May create duplicates on the target data warehouse table depending on the filter values.
Good for keeping a history of records from the source table.

Load All

Makes a %100 copy of the source table.

Doesn’t keep any history/older records on the target table.

How It Works

Load Upsert

Updates existing records on the target table and inserts new records.

Example 2: Column with Last Modified Value = D_LastModifiedDateTime

Load Replace

Replaces certain records – typically recently updated records.
Requires a last-modified column in the source table. Typically a modified date/date-time stamp.

Example 3: Load panel configured for Load Replace

Load Append

Appends (adds) all the records on the source table to the existing records on the target table.

Load All

Deletes the target table and loads all records from the source table to the target.
Tables configured with other load types can be temporarily converted to Load All with the Force Load All option on the Job page.

Force Replace when Upsert

Tables configured with Load Upsert are temporarily converted to Load Replace by the Force Replace when Upsert, default for others option from the Load type list, Job Properties panel, Job page.

Force Load All

All Tables are temporarily converted to Load All by the Force Load All option from the Load type list, Job Properties panel, Job page.

  • Target table – a table on the data warehouse. The table loaded by the ETL.

Load Type Options

image-20240603-192404.png

Load type option list found on the Job Properties panel of the Job page

JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.