Delta Load Combinations -- Upsert, Replace, Append, Load All

ETL+ delta load options are Load Upsert, Load Replace, and Load Append. Each load option has limitations but can be combined for better coverage. 🚧 Work in progress

Delta Load Combination Scenarios

  • Load Upsert run daily with a Load All run on weekends.

image-20220228-232848.png
Upsert Example: Column with Last Modified Value = LastModifiedDateTime
  • Load Replace run daily with a Load All run on weekends.

image-20231125-004433.png
Example 1: Load Replace with Target Value or Constant field configured to process records from the last 60 days.

Load Types Contrasted


Advantages & Uses


Limitations

Load Upsert

Fast, can be run many times a day. Updates existing records or inserts new records.
See Force Replace when Upsert option below.

Does not delete records from target/data warehouse table


Load Replace

Retains a history of changes.

Good for keeping a history of records from the source table.
See Force Replace when Upsert, default for others option below.

Can save several versions of source records on the target data warehouse table. Does not use a unique key to identify the target records to replace. (See example 1 below).



Load Append

Good for keeping a history of records from the source table.

May create duplicates on the target data warehouse table depending on the filter values.
Good for keeping a history of records from the source table.


Load All

Makes a %100 copy of the source table.

Doesn’t keep any history/older records on the target table.



How It Works

Load Upsert

Updates existing records on the target table and inserts new records.

image-20220228-232848.png
Example 2: Column with Last Modified Value = D_LastModifiedDateTime

Load Replace

Replaces certain records – typically recently updated records.
Requires a last-modified column in the source table. Typically a modified date/date-time stamp.


image-20231213-000513.png
Example 3: Load panel configured for Load Replace

Load Append

Appends (adds) all the records on the source table to the existing records on the target table.

Load All

Deletes the target table and loads all records from the source table to the target.
Tables configured with other load types can be temporarily converted to Load All with the Force Load All option on the Job page.

Force Replace when Upsert

Tables configured with Load Upsert are temporarily converted to Load Replace by the Force Replace when Upsert, default for others option from the Load type list, Job Properties panel, Job page.

Force Load All

All Tables are temporarily converted to Load All by the Force Load All option from the Load type list, Job Properties panel, Job page.

  • Target table – a table on the data warehouse. The table loaded by the ETL.


Load Type Options

image-20240603-192404.png
Load type option list found on the Job Properties panel of the Job page