DataSelf ETL+ Release Notes & History (Download Links)

Latest Release Download Links:

• Installer: DataSelf ETL Plus Setup.msi (1MB)

• Released: 05/03/2024

• Usually not required (standalone files):

  • DataSelf ETL+ 64-bit (v2023.10.1905.48): DataSelf ETL Plus.exe (16MB)

  • DataSelf ETL+ 32-bit (v2023.10.1905.48): DataSelf ETL Plus.exe (16MB)

  • DataSelf Agent Service (v2023.10.1602): DataSelfAgentService.exe (1MB)

Hotfix 73

Only use it when advised by DataSelf:

• DataSelf ETL+ 64-bit (v2023.10.1905.73): DataSelf ETL Plus.exe (16MB)

• DataSelf ETL+ 32-bit (v2023.10.1905.73): DataSelf ETL Plus.exe (16MB)

• Released: 10/28/2024

• Hotfix installation instructions are here.

New Features v2023.10.1905

• Released: 10/19/2023

1. Source metadata refresh via the Agent (Source Objects panel):
   Source metadata refresh via the Agent: New feature that can work with the local Agent or from remote servers.

2. Command line for Source metadata refresh via the Agent: New -rd <source PK> command line executes the source refresh defined by the PK.

3. Source metadata refresh via the Agent: New Refresh via Agent button icon (circular arrow with the letter A) to start the source metadata refresh via an Agent.

4. Source metadata refresh via the Agent: As a starting point, the Agent to run the source metadata refresh is the last one that successfully loaded a table from this source.

5. Source metadata refresh via the Agent: Users can change the Agent to run a source’s metadata refresh by using a new Source properties' icon on the top right.

6. Source metadata refresh via the Agent: When changing the Agent to run a source’s metadata refresh, uses can select any assigned Agent to their entity, plus the local computer’s Agent, or via the UI.

7. Source metadata refresh via the Agent: On the page above, the user can define whether the agent change to run the source metadata refresh will be on this ETL+ session only or permanently.

8. Source metadata refresh via the Agent: The assigned Agent to refresh a source’s metadata can be found on the popup when hovering over a source.

9. Source metadata refresh via the Agent: When selecting a source, its metadata’s Refresh icon will show if it’ll run via the UI or an Agent.

10. Source metadata refresh via the Agent: When right-clicking on a source, its metadata’s Refresh icon will show if it’ll run via the UI or with an Agent.

11. Source metadata refresh via the Agent: When right-clicking on a source, its metadata’s Refresh icon description shows if it’ll run via the UI or the Agent name.

12. Source metadata refresh via the Agent: The new icon shows the Agent name that will run this task.

13. Source metadata refresh via the Agent: The new icon shows None if no Agent is allocated to this source (no Agent has ever successfully loaded a table from this source).

14. Source metadata refresh via the Agent: When starting this process in an ETL session, the source’s icon changes to a Source Refresh icon.

15. Source metadata refresh via the Agent: This task works with sources and zData Warehouse.

16. Source metadata refresh via the Agent: When the metadata has been refreshed in the cloud, the user receives a prompt notifying it. Clicking Ok on the prompt and the source icon returns to normal reloading the metadata into the ETL session.

17. Source metadata refresh via the Agent: Triggering metadata refreshes are logged as User Actions.

18. Source metadata loading from SQL is significantly faster.

19. Upgrade install via the Agent: When clicking to upgrade installations via the Agent on the Entity page, it prompts for Yes/No.

20. Upgrade install via the Agent: It upgrades all the entity’s installs that have active Agents.

21. Upgrade install via the Agent: For Agents running, the upgrade takes less than a minute.

22. Upgrade install via the Agent: For Agents not running, the upgrade takes place when the Agent runs.

23. Upgrade install via the Agent: The upgrade happens behind the scenes, it won’t interfere with running ETL+ sessions such as running Jobs or open ETL+.

24. IT Debug mode ON: New debug mode using the command line interface (-d).

25. IT Debug mode ON: Logs the beginning of a Job run with the local .NET version.

26. IT Debug mode ON: Logs locally many OS, hardware, and internet access variables in a new log file.

27. IT Debug mode ON: Desc and SourcError log 5,000 characters (up from 500).

28. IT Debug mode ON: This debug mode also runs when Setting → Debug is set to Level 2.

29. IT Debug mode ON: When running a Job with this setting ON, the beginning of the table load takes a few extra seconds to start because of the logging of system environment variables locally.

30. Online metadata: ETL+ shows on the bottom a message when it’s unable to connect to its metadata.

31. Online metadata: When the metadata is disconnected, the user can continue to use ETL+ as long and it doesn’t need to exchange data with the metadata.

32. Online metadata: When the metadata is disconnected, the user receives a notification when it tries to exchange data with the metadata.

33. Online metadata: Once ETL+ re-connects to the metadata, it operates normally again.

34. User Interface (UI):
    UI: refined ETL page Action icons: cleaner and green for load actions; all others are black.

35. UI: refined Job page Action icons: cleaner and green for load actions; all others are black.

36. UI: refined Source page Action icons: cleaner and black.

37. UI: refined Log page Action icons: cleaner and black.

38. UI: Agent actions have 'A' on their icons (e.g.: Run Job via Agent).

39. UI: Minor corrections and misspellings.

40. UI: Source “Add Table” action relabeled to “Add Table to Target”.

41. UI: Hovering over an ETL Source now also shows its PK.

42. UI: When “Add Table to Target”, if the table already exists, the message now says: <TableName> already exists. Add it as <TableName>(n)?” E.g.: Abc already exists. Add it as Abc(n)?”

43. UI: Job Steps now show the same table icons as the ETL page, thus making it easy to view Load types (All, Replace, Upsert, Append, Freeze).

44. UI: Target icons for Expand and Collapse columns are now always available.

45. UI: Expand columns shortcut: Alt + Shift + Right Arrow.

46. UI: Collapse columns shortcut: Alt + Shift + Left Arrow.

47. UI: Remapped Help icons pointing to the new KB pages.

48. UI: Added page titles to Running Job, Table Loading, and Source Metadata Refresh.

49. UI: Job Refresh Email header rearranged for easier email preview reading.

50. UI: Windows Task Bar: The Job UI shows a new taskbar label: “ETL+ Job <JobID> <EntityID>”.

51. UI: Windows Task Bar: The Log UI shows a new taskbar label: “ETL+ Log <EntityID>”.

52. UI: When an entity is open, an easier-to-understand notification when another user saves it.

53. Job Load UI: Now shows the JobID as a title.

54. Job Load UI: Now with a black background.

55. Job Load UI: Now shows the ETL+ version.

56. Log UI: Now opens shifted to the right and bottom from the screen center (to avoid opening behind the main ETL page).

57. UI: Run Job and End Job icons: Always active regardless of Job status.

58. UI: Moved End Job icon under the Run Job menu.

59. UI: Product registration email now has updated links to KB.

60. UI: Entity page: Shows the EntityPK.

61. UI: Entity page: Seach now works across all columns.

62. UI: When prompting to upgrade/downgrade an entity with this version, clicking No goes back to the Entity page (no more prompt with link to the KB).

63. When running Jobs and a source’s captured metadata is NULL, ETL+ will no longer load the source metadata. This is an unnecessary step.

64. ETL+ downloads and installs Microsoft Access Database Engine 2016 Redistributable, when necessary.

65. When a source doesn’t have proper credentials, Jobs will skip its table loads.

66. Customized metadata: Standardized database name.

67. OData extraction: Tables with no Design’s Key+ configured will extract without pagination.

68. Cloud logging: Some Log Descriptions could have a huge amount of characters. Descriptions are now truncated at 500 characters.

69. Cloud logging: Some SourceError descriptions could have a huge amount of characters. Descriptions are now truncated at 500 characters.

70. .NET SQL commits/rollbacks apply to the ETL+ metadata.

71. Security: Security upgrades to DataSelf’s website (including new security tokens for file downloads). More secure ETL+ process to download files from DataSelf’s website. More secure ETL+ process to extract Excel files from SSL-based websites. More secure ETL+ process to extract Google Sheets from SSL-based websites. More secure ETL+ process to extract CSV files from SSL-based websites. More secure ETL+ process to extract OData from SSL-based websites. Stronger encryption of private data saved on the local HD. Stronger encryption of private data saved in the metadata. Stronger encryption of ETL+ private data. Stronger encryption of Agent private data. Stronger encryption of IP app private data. Stronger and more limited ETL+ metadata credentials. Embedded DataSelf’s Google Sheets JSON string in ETL+. Encrypted DataSelf’s Google Sheets JSON string. Creation of hidden folders following revised Microsoft best practices. New location and file name for ProgramData ETL+ configuration file. New location and file name for a user’s ETL+ configuration file. New working folders named “DataSelf Corp”. New ETL+ user passwords won’t work in versions older than 2023.07. Admin email alert about SQL injection attempt. ETL downloadable exe files' pubic key updated in the metadata.

Fixed Issues v2023.10.1905

1. Replace delta configuration for table joins only accepts MAX(FilterColumn).

2. Running Jobs from sources that don’t have their source metadata captured can hang the Job refresh indefinitely.

3. Remote ETL+ upgrade isn’t working in many installs.

4. Sometimes, the source metadata was deleted. For instance, opening an entity with a prior version and then with this one deletes the captured source metadata.

5. The first attempt to open ETL+ by a regular user usually fails (ETL+ just dies). It works normally after that. Fixed.

6. Upon upgrading an entity, connections to SQL might be set to Windows Authentication.

7. If adding a zDw SQL View to ETL+, it won’t recognize that the object name already exists in SQL and will allow saving it.

8. After adding a zDw SQL View to ETL+, trying to delete the ETL+ table will freeze the UI.

9. Trying to switch a data source from one type to another crashes ETL+.

10. When running Jobs, if ETL had a data warehouse connection error, it would switch into a failed load for all remaining job steps.

11. When running Jobs, a single failure to connect to the data warehouse errors out. The process wasn’t retrying 3 times.

12. In single-tenant data warehouses where sources are set to customized schemas, 0_UnionID columns might not bring in the correct value.

13. User actions to run a Job via de UI or Agent weren’t being logged.

14. Odata v3 extraction from GIs with spaces in their names might fail.

15. Regular users could open pre-saved 99 user UI in DataSelf domains.

16. Google Sheets: Issues extracting tables with space or line breaks in column names.

17. Google Sheets: Issues extracting some tables. You might need to re-load the sheet source metadata and remap it.

18. Importing a table that was already pre-mapped was duplicating the pre-mapped SQL Statement instead of adding the table from scratch again.

19. Adding new tables sometimes wouldn’t automatically create their Extract SQL Statment.

20. Excel mappings would not list all columns in the target metadata.

21. Excel extraction accepts commas in column names.

22. Excel mapping of files with special characters would fail.

23. CSV mappings would not list all columns in the target metadata.

24. Upon logging in, ETL+ does not remember the last open entity.

25. Email refresh report: Send on Error Only wasn’t working.

26. SELECTs in Design’s T-SQL weren’t working.

27. Log page shortcut Ctrl + G on the Job page wasn’t working.

28. Renaming Jobs #2 or higher wasn’t working.

29. The Add Job action wouldn’t work after deleting all Jobs.

30. Forgot Pw button on the login page was emailing the password from prior ETL+ versions.

31. Source icons on the menu were stretched and blurred.

32. Target Object icons on the right-click and Action were stretched and blurred.

33. Extraction of Excel with commas in column names errored out.

34. If the Extract SQL Statement had ‘0_UnionID’ as part of a column name, 0_UnionID new field was not being created.

35. Extraction of UNPIVOT from zData Warehouse was crashing the ETL+.

36. Reimport SQL Statements with constants with no value were converted to NULL. This could cause errors. Conversion removed.

37. Sometimes, adding an Excel table was putting double single quotes on the SQL Statement, which caused failed extractions.

38. OData DAC table names were always handled as case-sensitive based on what was in the captured source metadata. This could cause extraction errors. Now it’s only case-sensitive if Acumatica’s SQL database is set to case-sensitive.

39. Workaround for OData error: “Odata[0100] - The contents of table Note are not available for this user. The remote server returned an error: (400) Bad Request.“ → this is an intermittent Acumatica limitation when rendering large tables via OData without proper indexing in the OData request. Fix: Configure the ETL+ Design’s Key+ index to match with the source's table’s PK (columns list and order), plus configure a delta refresh for this table.

Hotfix v2023.10.1905.28

• Released: 12/04/2023

1. Design page: varbinary(x) data types are failing to save.

2. Design page: Changes to column data types are only executed during Load All.

3. Design page: User actions are not being logged.

4. MySQL Source: Unable to extract tables with data types: geometry and blob.

5. MySQL Source: Incorrect conversion of data types: tinyint, mediumint, datetime, year.

6. OData Source: Extractions are not retrying on the initial table load errors.

7. OData Source: Extractions are not retrying during table load errors.

8. IP whitelisting: In some installs, ETL+ cannot get out of the IP whitelisting process.

9. Regular users with multiple entities might be unable to open some entities.

10. Mapping a new source table already mapped is duplicating the existing target table (instead of adding a new one).

11. Customizing the ETL+ metadata on a new install from scratch might not save the hidden file with the custom settings, and the ETL+ metadata will default to dw3.

12. Resetting the password of admin users doesn’t prompt for creating a new one.

13. Admin users without any entity assignment aren’t able to log in.

14. Improved error message when a delta refresh filter column has a data type different than expected or is nonexistent.

15. If a source table has been mapped and customized, adding it again duplicates the target object (duplicates the first target table customizations instead of creating a new fresh mapped table).

16. If a source table has been mapped and customized, and then is deleted, adding the source table duplicates the previous target object customizations (instead of creating a new fresh mapped table).

17. Added <ETL username> to several user action log descriptions.

18. The mechanism that finds table name duplication is case-sensitive - which can cause table name conflicts in non-case-sensitive SQL DBs.

19. When running a Job and there’s a failure to connect to the data warehouse, the ETL+ cannot reconnect to the data warehouse for the remainder of the Job run.

20. If a user cannot log in to ETL+ with the error message “This user doesn’t have access to any entity”, 1) be sure the user is added to an entity; 2) if that doesn’t fix it, delete the hidden ProgramData DataSelf folders.

21. In certain conditions, Upsert & Replace only worked correctly after the first delta load in a given ETL+ session.

22. Force Replace when Upsert uses the filter value from the Upsert configuration (instead of Replace).

23. If a duplicated table is renamed, ETL+ won’t allow it to duplicate again.

24. The load all of OData tables with no indexes might fail.

25. Unable to auto-creation of ObjectTID column in zLog_ETL and zLog_TableStats data warehouse tables.

26. The insertion of log lines into the data warehouse zLog tables was partial and logged errors.

27. OData extractions were not logging connection error retrials.

28. Relabeled ETL Metadata page: title “Metadata Server” and “Default (dw3)”.

Hotfix v2023.10.1905.35

• Released: 12/16/2023

1. T-SQL Statements with CROSS JOIN crash when running via Jobs.

2. Extraction of OData with tables with over 100 pages is not using the Temp2 flow.

3. OData extraction of large tables with Upsert might switch the Upsert filter to Replace when hitting 100 extracted pages.

4. OData table extraction with no Design’s Key+ configured is still paginating.

Hotfix v2023.10.1905.42

• Released: 03/08/2023

1. Loading NetSuite source metadata sometimes fails. Fixed.

2. ETL+ can’t connect to its metadata on some computers. It then incorrectly starts the IP whitelisting process. Fixed.

3. ETL+ can’t list some tables from some Google Sheets. Fixed.

4. In some installs, sometimes running Jobs stall and never finish. Fixed.

Hotfix v2023.10.1905.48 * CURRENT VERSION *

• Released: 05/03/2024

1. For some entities, sometimes, Jobs get stuck or just skip all steps. Fixed.

2. Remote upgrades not working in most installs. Fixed.

3. zLog_ETL not populating StatementID column. Fixed.

4. The Agent app was visible in the ETL+ root directory after an upgrade. Fixed (hidden).

5. When requesting an entity's upgrade on the current computer, the upgrade process will cancel all Agent commands queued for that entity, close all ETL+ apps on that computer, and then execute the upgrade.

6. At the end of the upgrade, the upgrade re-starts the Agent in case it fails to start.

7. The remote upgrade works with ETL+ version 2023.10.1905 or newer.

8. The remote upgrade is 64/32 bit sensitive. Each ETL+ installation upgrades accordingly to its 64 or 32 bit version.

Hotfix v2023.10.1905.62

• Released: 09/16/2024

1. In some entities, the hotfix significantly speeds up saving the ETL page.

2. In some target tables, the hotfix significantly speeds up confirming changes to Extract SQL Statements when Parsing is OFF.

3. Some connections to PostgreSQL weren’t listing all available tables and views.

4. The extraction of some PostgreSQL numeric fields wasn’t properly mapped in ETL+.

5. Acumatica OData extractions: the extraction assumes Acumatica supports “row skip”. In prior versions, ETL+ always tested this feature before starting every table extraction.

6. In some entities, the Job’s Agent dropdown doesn't list all active Agents.

Hotfix v2023.10.1905.73

• Released: 10/28/2024

1. Significantly speeds up the saving of the ETL page in entities with a large number of target objects (opening such entities is still slow).

2. Fixes some entities experiencing obscure ODBC data extraction errors (bulk copy adjustments).

3. Fixes some issues when saving the Design page.

4. On the Design page, forcing a column to VarChar automatically keeps its Dw Data Type configuration and creates a TRY_CONVERT expression.

5. Assigning Agents to Sources wasn’t working on some installs.

6. Assigning Agents to Jobs wasn’t working on some installs.

Known Issues

1. Creating new sources and clicking Connect often doesn’t load the list of tables. Workaround: run Refresh after creating the source.

2. Download to CSV via Jobs only works with the default schema.

3. Sometimes, adding an Excel new source (or CSV), adding a table, and clicking Save will crash ETL+.

4. New sources with Excel from cloud URLs will only load after closing and re-opening the entity.

5. Replace is not working in GoogleSheet tables.

6. Changing the source’s database of a remote install won’t stick.

7. Excel mapping: Mapping table names with special characters might fail.

8. Excel mapping: Mapping column names with special characters might fail.

9. Settings - Job Notification Email: The dropdown for when to notify might not remember the selection unless the Email box is configured (issued identified with the No Notification option).

10. It doesn’t notify that it can’t accept SQL passwords with ; (semicolons) using SQL Authentication.

11. Running a Job via the Agent: if ETL+ can’t connect to the data warehouse, it will end the session without an error and it doesn’t log informing what the issue is.

12. Job Script: Clicking Properties of a Job Script will set the Job as modified (* on the Save button) even if the Job Script was not saved.

13. ETL+ might be unable to connect to a customized ETL metadata server. Workaround: run ETL+ via the command line with -md parameter, delete all files from C:\ProgramData\DataSelf Corp\DataSelf ETL Plus, enter the new ETL metadata server connection information → Save.

14. Replace with a character filter >= '' or ' ' will append rows instead of replace.

15. OData delta extraction cannot use GUID columns for delta filtering (Acumatica limitation?). When loading a table that has this field used in a delta filter, the ETL+ might retry the table load indefinitely.

16. Paginated OData extraction (indexed extraction): During a load of a large number of rows, there’s a risk of data duplication for rows inserted during the extraction with an index value lower than the ongoing extraction values. Workaround: Create an index with a guaranteed ascending value for the load (e.g.: CreatedDate & PK Columns).

17. OData Upsert: The engine includes the Upsert’s LastModifiedValue column as the first index column in the URL. In rare cases, this might cause row duplication.

18. User 99: ETL+ might not retain OData source metadata from some Acumatica installs. Workaround: Close the entity, then ETL+, then re-open the entity.

Released: 7/21/2023

New Features

1. Extraction Performance: ETL processes run dramatically faster when extracting from MS SQL sources that are hosted in the same SQL instance as the data warehouse.

2. This improvement also applies to zDataWarehouse re-imports.

3. ETL+ Configurable Metadata: Now the ETL+ metadata location is configurable. This allows the deployment of the complete ETL+ engine (server, client, and agent) behind a client’s firewall, as well as having an ETL+ solution for testing purposes (with no effect on the production stack).

4. ETL+ Metadata: Ability to configure the location of the metadata database for a given ETL+ installation.

5. ETL+ Metadata: Run ETL+ in a command line with -md parameter to configure a custom metadata location.

6. ETL+ Metadata: When configuring a custom metadata location, click the Default button to set the metadata back to its default location (dw3), and close the dialog.

7. Saving a Copy of Source Metadata: Now a copy of sources' metadata is saved in ETL+. This copy might contain a list of tables, columns, column formatting, and primary keys if available. This allows users to map new tables and columns more quickly and without the need to be connected to the source database.

8. Source Metadata: On the Source page, clicking Connect retrieves and stores the source database’s table and column metadata.

9. Source Metadata: On the Source panel, clicking Refresh retrieves and stores the source database’s table and column metadata.

10. Source Metadata: During ETL+ loads, ETL+ uses the stored source metadata to build and execute extraction queries. In prior ETL+ versions, ETL+ first read the column list and formatting, built the extraction query, and then executed it. This new approach reduces the time to extract tables, from a fraction of a second to several seconds per table depending on the speed of the source database.

11. Source Metadata: Since ETL+ can now store the source database’s metadata, users will be able to add tables and columns to their ETL processes without the need to connect live to source systems to read what’s available.

12. Source Metadata: On this release, ETL+ only uses the in-memory source metadata when adding tables as new objects. If the user edits the Extract SQL Statement, ETL+ will use other methods to resolve column formatting.

13. Source Metadata: Hovering the mouse over a Source on the Source Objects panel shows when its metadata was loaded to the cloud.

14. Source Metadata: Hovering the mouse over a Source on the Target Objects panel shows when its metadata was loaded to the cloud.

15. Source Metadata: When loading a table, ETL+ will load the source metadata to the cloud if that hasn’t been done before.

16. Saving Mapped Tables' Column Metadata: Now each mapped table has its source and target column metadata saved in ETL+. Users can quickly see that data on the redesigned Design page, this is helpful for maintenance and troubleshooting. Also, this saved metadata allows ETL+ to execute table loads more quickly (in prior versions, it had to read the source and target column formatting before executing a load).

17. Design page: Redesigned page from scratch.

18. Design page: The page and its column sizes can be manually adjusted.

19. Design page: Source PK column(s) added.

20. Design page: Source PK column(s) (when available) are now numbered based on the source’s PK definition.

21. Design page: Source Data Type added. It shows the data type received from the source database.

22. Design page: VarChar column with checkboxes added. This allows users to force the extraction of a source column into a varchar(max) data type in the temp table in MS SQL. This is helpful when MS SQL might have issues accepting the values from a given source data type. Example: a date column with invalid amounts in SQL. Or a source numeric column with dirty data (like strings in some rows).

23. Design page: DW PK renamed to Key+.

24. Design page: Key+ tooltip provides clearer information about this column.

25. Design page: Key+ is now numbered to set the sequence of multi-column unique keys and PK.

26. Design page: By default, when available, the Key+ comes preconfigured to match the source table’s PK from MS SQL Server.

27. Design page: By default, when available, the Key+ comes preconfigured to match the source table’s PK from Oracle.

28. Design page: By default, when available, the Key+ comes preconfigured to match the source table’s PK from OData.

29. Design page: Logged-in username added at the bottom-left corner.

30. Design page: Entity ID in session added at the bottom-left corner.

31. Design page: When changing the Dw Data Type, there’s a revised message to notify users about setting a new T-SQL expression: “Changed data type from <source type> to <target type>. Overwrite T-SQL to: <new T-SQL expression>. Confirm? Recommended: Yes)”.

32. Design page: When converting a char data type to date, ETL creates TRY_CONVERT. This converts invalid date values into NULL.

33. Design page: When converting a char data type to datetime, ETL creates TRY_CONVERT. This converts invalid datetime values into NULL.

34. Design page: Clicking Confirm no longer requires clicking Confirm on the ETL page.

35. Removed Target Objects → … → Load All Columns. This feature became obsolete.

36. Remote Upgrade: Now admin users can trigger the upgrade of ETL+ and its Agent remotely (without the need to be logged into the computer with ETL+). This requires an active Agent on the target computer.

37. Remote Upgrade: Ability to upgrade ETL+ and Agent remotely in Agent-active installations.

38. Remote Upgrade: The Agent has a new function to upgrade its ETL+ install.

39. Remote Upgrade: ETL+ has a new command line function to upgrade its Agent.

40. User Authentication: SQL Source passwords now have new actions: View/Hide Password, Lock/Unlock Password.

41. SQL Passwords: On an unlocked SQL password, users can view and hide the password.

42. SQL Passwords: Users can click the Lock Password icon to permanently lock it. Locked passwords cannot be viewed.

43. SQL Passwords: Unlocking a password will require password re-entry. From that point on, users can view and hide the password.

44. Data Warehouse SQL Authentication: Support to Azure Active Directory - Password.

45. Source MS SQL Authentication: Support to Azure Active Directory - Password.

46. Saved ETL+ user credentials are now encrypted and stored in C:\Users\<username>\AppData\Local\DataSelf_Corporation\ds_lg_info.ds

47. OData Refinements: Now ETL+ can map and extract PX_ tables and columns that before weren’t directly available via the ETL+ mapping wizard.

48. OData v4 Refinements: Redesign a part of the processing of OData v4 metadata.

49. OData v4 Refinements: Ability to read a broader range of OData v4 tables and columns.

50. OData v4 Refinements: Prior ETL+ versions couldn’t list some regular tables on the Source Panel. This is now fixed.

51. OData v4 Refinements: Prior ETL+ versions couldn’t automatically list many available columns from some tables during the table mapping. This is now fixed.

52. OData v4 Refinements: Ability to list and read PX_ tables.

53. The mapping of OData v3 and 4 tables mimics the source’s column formatting.

54. The extraction of OData v3 and 4 tables no longer truncates string fields to 255 characters.

55. Extraction of large OData tables now uses the data stored on the TableName_Temp2 to reset the OData filter to extract the next block of 100 pages. This addresses an Acumatica issue when rendering data from large tables via OData.

56. For OData with delta extractions based on date filters, the JSON/XML time zone is now ignored. This eliminates the need to read the time zone before the extraction starts (removed Currency table extraction that was previously used to read the default time zone of an Acumatica tenant).

57. When adding new OData sources, the default page size is 5,000 rows (up from 1,000).

58. Extraction Engine: The code has been refactored into classes for cleaner execution and maintenance.

59. The mapping of tables from non-structured sources (like Excel, CSV, and ODBC) mimics the source’s column formatting.

60. The extraction of tables from non-structured sources (like Excel, CSV, and ODBC) no longer truncates string fields to 255 characters.

61. ODBC Source Page: Added credentials section.

62. ODBC Source Page: Made it clearer that the list of available DSNs is 32 or 64 bits.

63. Now users can open and edit ODBC Sources even on computers that don’t have any DSN.

64. Replace Delta Refresh: Dedicated metadata filter column (no longer shared with Upsert).

65. Replace Delta Refresh: Updated the feature description “1st: Insert rows based on the condition. Ongoing: Replace rows based on the condition”.

66. Replace Delta Refresh: changed labels from “Field” to “Source Field”, and blank to “Target Value or Constant”.

67. Upsert Delta Refresh: Dedicated metadata filter column (no longer shared with Replace).

68. Upsert Delta Refresh: Renamed Primary Key to Unique Key.

69. Refinements/adjustments to extract Salesforce.com data via ODBC.

70. Refinements/adjustments to extract Snowflake databases via ODBC.

71. Refinements/adjustments to extract NetSuite data via ODBC.

72. Refinements/adjustments to extract Oracle data via ODBC.

73. Refinements/adjustments to extract Oracle data via native driver.

74. Google Sheets sources will automatically download the DataSelf JSON Google Sheets file if not available locally.

75. Now you can add other expressions outside of CTEs (ex.: a table union).

76. Extract SQL Statement Parsing: Initial Parsing release.

77. SQL Parsing: Tries to capture the target table’s column list and data types and stores them in the metadata.

78. SQL Parsing: A new toggle icon on the SQL Statement sets if parsing is on or off.

79. Parsing ON: Clicking Confirm on a changed SQL Statement will try to execute parsing.

80. Parsing OFF: Clicking Confirm on a changed SQL Statement will just save the statement to the metadata - no parsing.

81. Parsing ON: Clicking Confirm on the Design page will try to execute parsing.

82. Parsing OFF: Clicking Confirm on the Design page will just save the Design changes.

83. Freeze Table: Added a checkbox to the More Actions target object menu to freeze a table’s content.
    For more see ETL+ Freeze

84. Freeze Table Loads: Frozen tables can’t be changed via ETL+, unless the table is unfrozen. For more see ETL+ Freeze

85. Frozen tables retain their delta refresh configurations (if needed again).

86. Log will show “Skip table” for frozen tables that are part of refresh Jobs.

87. Hovering over a frozen table shows Load Type = Frozen.

88. Job Refinements: Fixing some scheduling bugs as well as incorporating weekly and monthly scheduling.

89. Job Scheduling: Next date/time to run a Job uses a more refined algorithm.

90. Job Scheduling: Next date/time to run a Job can use scheduling based on weekly parameters (configurable via ETL+ Web - beta release).

91. Job Scheduling: Next date/time to run a Job can use scheduling based on monthly parameters (configurable via ETL+ Web - beta release).

92. Job App: Simplified process to select an Agent app. Select it from the dropdown, and click Save.

93. Job Load Type: Changed to a dropdown list with a new option: Force Replace when Upsert, default for others.

94. UI Improvements: Source types now have their own icons on the ETL page.

95. Source Icon: Google Sheets dedicated icon.

96. Source Icon: MySQL dedicated icon.

97. Source Icon: MS Access dedicated icon.

98. Source Icon: MS Excel dedicated icon.

99. Source Icon: MS SQL Server dedicated icon.

100. Source Icon: OData dedicated icon.

101. Source Icon: ODBC dedicated icon.

102. Source Icon: Oracle dedicated icon.

103. Source Icon: PostgreSQL dedicated icon.

104. Source Icon: CSV dedicated icon.

105. ETL Page: Source types now have their own icons on the Target Object panel (smaller than source icons).

106. Target Object Source Grouping Icon: Google Sheets dedicated icon.

107. Target Object Source Grouping Icon: MySQL dedicated icon.

108. Target Object Source Grouping Icon: MS Access dedicated icon.

109. Target Object Source Grouping Icon: MS Excel dedicated icon.

110. Target Object Source Grouping Icon: MS SQL Server dedicated icon.

111. Target Object Source Grouping Icon: OData dedicated icon.

112. Target Object Source Grouping Icon: ODBC dedicated icon.

113. Target Object Source Grouping Icon: Oracle dedicated icon.

114. Target Object Source Grouping Icon: PostgreSQL dedicated icon.

115. Target Object Source Grouping Icon: CSV dedicated icon.

116. Source Properties Page: Source types now have their own icons.

117. The bottom left corner of ETL+ pages now shows the EntityKey value and metadata URL.

118. The Job page now shows the data warehouse information.

119. The data warehouse information under the ETL page title now has a tooltip with more details.

120. The data warehouse information under the Job page title now has a tooltip with more details.

121. ODBC Data Sources page: layout re-organized for clarity.

122. ETL Source panel was renamed to Source Objects.

123. ETL Objects panel was renamed to Target Objects.

124. ETL SQL Statement panel was renamed to Extract SQL Statement.

125. Target Object icons: Dedicated icon to represent tables with Replace.

126. Target Object icons: Dedicated icon to represent tables with Upsert.

127. Target Object icons: Dedicated icon to represent tables with Append.

128. Target Object icons: Dedicated icon to represent tables with Freeze.

129. Settings page: Layout reorganized.

130. When clicking actions, the mouse cursor goes to “waiting mode” until the action completes (ex.: on Save buttons).

131. Source Objects - New Source menu: Removed (Beta) tag from Google Sheets, MySQL, Oracle, and PostgreSQL.

132. When an entity is open, by default, all objects on the Target Objects are collapsed.

133. Client Information’s Cloud Script checkbox moved to a better position.

134. When importing a table from the zData Warehouse, by default, the table is renamed (to be sure it’s not overwritten in case the table hasn’t been mapped as a target object in ETL+).

135. It’s now possible open/edit existing ODBC sources even when there are no local DSNs.

136. Target Object icons: A small red dot notifies modified tables that haven’t been saved to the metadata.

137. User Action Logging and Debugging: Now many ETL+ user actions are logged. This should help collaboration and troubleshooting.

138. Logging: Many user actions are now logged on the zETL_Log table.

139. Logging: New logging type = 100 for user actions.

140. Logging: Saved data warehouse configuration.

141. Logging: Saved a table’s Extract SQL Statement panel.

142. Logging: Saved Job page.

143. Logging: Settings page saved.

144. Logging: Client Info page saved.

145. Logging: Table added.

146. Logging: Table deleted.

147. Logging: Table renamed.

148. Logging: Job added.

149. Logging: Job deleted.

150. Logging: Job script added.

151. Logging: Job script deleted.

152. Logging: Source added,

153. Logging: Source deleted.

154. Logging: Agent on a machine added.

155. Logging: Job App selected.

156. Logging: Entity duplicated.

157. Command Line -d (debug mode): Runs ETL+ in local debug mode on the local hard drive.

158. Command Line -d: There’s a new local file Log_Debug.txt showing information about the environment where ETL+ is running.

159. Command Line -d: Information logged: OS version.

160. Command Line -d: Information logged: OS local language.

161. Command Line -d: Information logged: .NET version.

162. Command Line -d: Information logged: 32/64-bit platform.

163. Command Line -d: Information logged: Test the https access to dataself.com.

164. Command Line -d: Information logged: Outbound port testing.

165. Command Line -d: Information logged: Available local hard drives total and available space.

166. Command Line -d: Information logged: Test access to ETL+ metadata - if fails, test access to a test database on the same SQL instance where ETL+ metadata is.

167. ETL+ Multi-session Mode: When there’s more than one ETL+ Desktop session running with the same entity, each session shows “Other session(s) open: <# of sessions>” on the ETL and Job pages. This feature notifies of other users on the same entity and prevents configuration overwrite conflicts.

168. Hovering the mouse over “Other session(s) open” message will show the other session’s usernames.

169. The message above is auto-refreshed every 15 seconds.

170. This feature monitors open sessions on pages: ETL, Job, Settings, Client Info, and Data Sources.

171. When one of the sessions saves a change to the metadata, the other sessions will receive a notification that they are outdated. Those sessions turn into a Runtime mode (no ability to change, just view and run loads and scripts), and the Save buttons become disabled.

172. When a session turns into a Runtime mode, close and re-open the entity to make changes to it.

173. ETL+ Runtime mode: ETL+ can only execute pre-configured templates, and configure the pre-configured data sources, and job schedules.

174. ETL+ Limited mode: ETL+ can only work and modify pre-configured templates.

175. ETL+ admin improvements: Admin users can configure many of the texts used in the ETL+ user interface such as error messages and log descriptions.

176. When duplicating an entity for Pro and Enterprise, the 99 user can now deploy the Tableau script when saving the Client Info page.

177. Double-clicking an entity on the Entity list will open it.

178. When the debug mode is on, the Job Refresh Start log line shows the .NET version installed on the host computer.

179. Data Warehouse admin refinements: When duplicating an entity, 99 users select the data warehouse prior to filling out the Client Info page.

180. Data Warehouse key=1 (multi-tenant) is no longer editable via the ETL+ UI (security reasons). 99 users can view it, but not edit it. IT users can edit its settings directly in the metadata database.

181. Regular users: they can now modify their single-tenant data warehouse settings.

182. Regular users: from the ETL page, click Settings → Data Warehouse → fill out the data warehouse settings → Save.

183. When creating a new entity (or duplicating a new one), there’s no need to close it. One can use it right off the bat.

184. ETL+ Watchdog: A new Linux-based engine to monitor ETL+ processes.

185. ETL+ WatchDog: Email notification of Job start failures is sent 5 mins from the scheduled start time.

186. ETL+ WatchDog: Email includes DeviceName, Agent status, Job status, and other metadata info.

187. ETL+ WatchDog: Email contains a link to open the ETL+ Web Job page with expanded logging.

188. ETL+ Installer: The new installer easily sets up the DataSelf ETL+ and Agent Service environment in Windows computers.

189. By default, the ETL+ is installed in the following folder (the user can change the folder):

     1. C:\Program Files\DataSelf ETL Plus

190. When applicable, the ETL+ saves its working files in the following Log and Working Files folder:

     1. C:\Users\Public\DataSelf

191. Settings Page: New entry to customize the location of the Log and Working Files folder.

192. Settings Page: Icon to configure the Log and Working Files folder location.

193. When applicable, these are the folders that are saved in the Log and Working Files folder:

     1. DS_Credentials

     2. DS_Downdows

     3. DS_SourceFiles

     4. OlderVersions

194. ETL page: Added Log and Working Files icon on the top right. Clicking it gives you ways to reach that folder as well as the folder where ETL+ is running from.

195. Job page: Added Log and Working Files icon on the top right. Clicking it gives you ways to reach that folder as well as the folder where ETL+ is running from.

196. The installer also installs and deploys the new DataSelf Agent Service.

197. DataSelf Agent Service: Now the Agent runs as a Windows Service.

198. The Agent is automatically installed and run as a Windows Service upon installing DataSelf ETl+.

199. By default, the Agent Service runs with the LOCAL SERVICE Windows account.

200. The Agent can be maintained (stop, start, pause, credentials change) in the Windows Services app.

201. The Agent local log file has been moved to the Log and Working Files folder.

202. The Agent now logs: when the service starts and its version.

203. The Agent now logs: when the service shuts down.

204. The Agent now logs: successful connection to the ETL+ metadata.

205. The Agent now logs: if using the default or a custom ETL+ metadata.

206. The Agent now logs: successful connection to the ETL+ metadata.

207. The Agent now logs: failure connecting to the ETL+ metadata.

208. The Agent now logs: date & time when a new task runs.

209. The Agent now logs: date & time when a new task ends.

210. ETL+ Web - Beta version: New web UI allows admin users to monitor and make changes to certain ETL+ processes anytime and anywhere.

211. ETL+ Web: Released for super admin users.

212. ETL+ Web: User authentication using Azure AD, SAML-based, Single Sign-on (SSO) enabled.

213. ETL+ Web: User authentication is locked to @dataself.com users.

214. ETL+ Web: Jobs page created mimicking the equivalent ETL+ Desktop features.

215. ETL+ Web: ETL Log page created mimicking and improving the equivalent ETL+ Desktop features.

216. ETL+ Web: Settings page created mimicking part of ETL+ Desktop features.

217. ETL+ Web: Client Info page created mimicking part of ETL+ Desktop features.

218. ETL+ Web: A Transfer page to easily exchange files.

219. ETL+ Web: Users page created to manage SSO.

220. ETL+ Web: Menu buttons to access ETL+ pages.

221. ETL+ Web: The menu button has a green background and indicates the ETL+ Page in session.

222. ETL+ Web: Top-left button shows the Master Entity filter.

223. ETL+ Web: Top-left shows the selected entity’s data warehouse information.

224. ETL+ Web: Top-right shows username logged in.

225. ETL+ Web: Top-right shows ETL+ Web version.

226. ETL+ Web: Top-right shows ETL+ Web's metadata's SQL instance name.

227. ETL+ Web - Master Entity filter: Shows a description of the selected entity(ies) and the number of entities when more than one is selected.

228. ETL+ Web - Master Entity filter: Allows searching.

229. ETL+ Web - Master Entity filter: Check/Uncheck All icons.

230. ETL+ Web - Master Entity filter: When a single entity shows on the list, press Enter to select it.

231. ETL+ Web - RESET FILTER button clears filters to their default values.

232. ETL+ Web - User page: Contains columns for email, actions, first and last name, token, user key, type, status, default entity, created at.

233. ETL+ Web - User page: One can sort the page by the page headers.

234. ETL+ Web - User page: Email search box.

235. ETL+ Web - Job page: Contains columns for Entity ID - Key, Job, Actions, Agent, Job Status, Job Name, Last Run, Next Run, Load, App.

236. ETL+ Web - Job page: One can sort the page by the page headers.

237. ETL+ Web - Job page: Search box for the name of the App running the Job schedule.

238. ETL+ Web - Job page: Agent column filter with values: Offline, Active, WTS.

239. ETL+ Web - Job page: Job Status column filter with values: Running, WTS Assigned, ETL+ Desktop Only, Scheduled, Unscheduled.

240. ETL+ Web - Job page: Load column filter with values: Default, Forced.

241. ETL+ Web - Job page: Action: Run Job.

242. ETL+ Web - Job page: Action: Clicking Run Job will prompt confirmation.

243. ETL+ Web - Job page: Action: Stop Job.

244. ETL+ Web - Job page: Action: Clicking Stop Job will prompt confirmation.

245. ETL+ Web - Job page: Action: Job Properties.

246. ETL+ Web - Job page: Action: Job Properties icon toggles the Properties panel on and off.

247. ETL+ Web - Job page: Action: Selecting a Job Properties highlights the Job with green.

248. ETL+ Web - Job page: Job Properties - Schedule toggle icon.

249. ETL+ Web - Job page: Job Properties - Dropdown for schedule frequency: Daily, Weekly, Monthly.

250. ETL+ Web - Job page: Job Properties - Weekly options provide checkboxes to run the Job on Sun, Mon, Tue, Wed, Thu, Fri, Sat.

251. ETL+ Web - Job page: Job Properties - Monthly options provide checkboxes to run the Job on each day of the month from the 1st to the 28th, Last day, Check All, Uncheck All.

252. ETL+ Web - Job page: Job Properties - Dialog for setting the Job’s Start’s Date/Time.

253. ETL+ Web - Job page: Job Properties - Repeat task every toggle icon.

254. ETL+ Web - Job page: Job Properties - Text box for min.

255. ETL+ Web - Job page: Job Properties - Text box for For a duration of X hours.

256. ETL+ Web - Job page: Job Properties - Load Type radio button.

257. ETL+ Web - Job page: Job Properties - Dropdown to select the App to run the Job schedule. Options: WTS and Agents assigned to this Job.

258. ETL+ Web - Job page: Job Properties - Save button.

259. ETL+ Web - Job page: Job Properties - Save button enabled only when changes are pending saving.

260. ETL+ Web - Job page: Job Properties - Closing properties without saving changes will prompt confirmation.

261. ETL+ Web - Job page: Job Properties - Saving a Job’s Properties will update the Job’s columns Agent, Job Status, Next Run, Load, and App.

262. ETL+ Web - Job page: Action: Job Steps.

263. ETL+ Web - Job page: Action: Job Steps icon opens the Steps panel on the right.

264. ETL+ Web - Job page: Action: Job Steps icon toggles the Steps panel on and off.

265. ETL+ Web - Job page: Job Steps - Close and Save buttons.

266. ETL+ Web - Job page: Job Steps - Save button enabled when there are changes to save.

267. ETL+ Web - Job page: Job Steps - Closing panel without saving prompts confirmation.

268. ETL+ Web - Job page: Job Steps - HIDE UNCHECKED icon. To hide all rows that are unchecked.

269. ETL+ Web - Job page: Job Steps - HIDE TABLE icon. To hide all Table rows.

270. ETL+ Web - Job page: Job Steps - HIDE TABLEAU icon. To hide all Tableau rows.

271. ETL+ Web - Job page: Job Steps - HIDE CLOUD SCRIPT icon. To hide all Cloud Script rows.

272. ETL+ Web - Job page: Job Steps - HIDE COMMAND LINE icon. To hide all Command Line rows.

273. ETL+ Web - Job page: Job Steps - HIDE CSV DOWNLOAD icon. To hide all CSV Download rows.

274. ETL+ Web - Job page: Job Steps - Move to Top icon. Select rows, and click this icon to move them to the top.

275. ETL+ Web - Job page: Job Steps - Move Up icon. Select rows, and click this icon to move them one row up.

276. ETL+ Web - Job page: Job Steps - Move Down icon. Select rows, and click this icon to move them one row down.

277. ETL+ Web - Job page: Job Steps - Move to Bottom icon. Select rows, and click this icon to move them to the bottom.

278. ETL+ Web - Job page: Job Steps - Selected Check/Uncheck icon. Select multiple steps and click this icon to check/uncheck their checkboxes in bulk.

279. ETL+ Web - Job page: Job Steps - Tables Check/Uncheck icon.

280. ETL+ Web - Job page: Job Steps - Scripts have the Properties icon enabled.

281. ETL+ Web - Job page: Job Steps - Scripts Properties icon toggles its settings panel on and off.

282. ETL+ Web - Job page: Job Steps - Clicking the Properties icon of a Script allows one to view and change the script settings.

283. ETL+ Web - Job page: Job Steps - Drag & Drop Steps position - Select one or multiple steps (use Ctrl or Shift), drag and drop them up or down on the Step list.

284. ETL+ Web - ETL Log page: Contains columns for Entity, DateTime, Desc, Source Error, Type, RunID, Device/User, Job.

285. ETL+ Web - ETL Log page: One can sort the page by the page headers.

286. ETL+ Web - ETL Log page: Search boxes for Desc, Source Error, Device/User.

287. ETL+ Web - ETL Log page: Center-top: 250 Rows Retrieved.

288. ETL+ Web - ETL Log page: Change the number of Rows Retrieved and press Enter. Prompt to confirm if the row count to retrieve is over 10K.

289. ETL+ Web - ETL Log page: By default, the log list is sorted by DateTime where the most recent log row is at the bottom of the list.

290. ETL+ Web - ETL Log page: Log Desc that is longer than 200 characters will show an icon on the far right. Click the icon to see the whole Desc.

291. ETL+ Web - ETL Log page: Log SourceError that is longer than 20 characters will show an icon on the far right. Click the icon to see the whole SourceError desc.

292. ETL+ Web - Settings page: Client Info - shows Entity, Name, Users, Source System, Phone, License dropdown.

293. ETL+ Web - Settings page: Has an Entity dropdown filter with the entities available from the Entity Master filter. The Setting’s Entity filter can only show one entity at a time. By default, it shows the first of the list.

294. ETL+ Web - URL parameters: Syntax: https://etl.dataself.com/<page>?parameter1&parameter2&etc

295. ETL+ Web - URL parameters: <page> can be logs, jobs, users, settings, clientinfo, transfer, and use cmd for commands (like run a Job).

296. ETL+ Web - URL parameters: ? separates URL from parameters

297. ETL+ Web - URL parameters: & parameter separator

298. ETL+ Web - URL parameters: e=<EntityKey> (e.g.: e=12)

299. ETL+ Web - URL parameters: Sort ascending sort=<ColumnName> (e.g.: e=12&sort=EntityID)

300. ETL+ Web - URL parameters: Sort descending sort=-<ColumnName> (e.g.: e=12&sort=-EntityID)

301. ETL+ Web - URL parameters: filter[<ColumnName>]=<Value>, requires e= (e.g.: e=12&filter[Job_Key]=1)

302. ETL+ Web - URL parameters: Expand job properties jxp=<JobID>, requires e= (e.g.: e=12&jxp=1)

303. ETL+ Web - URL command: Run Job jr=<JobID> requires cmd?e= (e.g.: cmd?e=12&jr=1)

304. ETL+ Web - URL command: Stop Job js=<JobID> requires cmd?e= (e.g.: cmd?e=12&js=1)

Fixed Issues

1. Regular users with multiple entities couldn’t select the bottom 1 or 2 entities. Fixed.

2. When login into ETL+, regular users with multiple entities had to select the entity 1 or 2 rows above the desired one. Fixed.

3. When duplication an ETL+ entity, EntityID words between single quotes weren't replaced with the new EntityID. Fixed.

4. When logged with a regular user, if the ETL+ couldn’t connect to the data warehouse, it would open the 99 user data warehouse configuration page.

5. Opening an entity that had its data warehouse offline was triggering a large number of log error lines. Now it only shows 3 log error lines.

6. Converting datetime to date type wasn’t working. Fixed.

7. Timestamp columns converted to char(23). Fixed.

8. When opening the Job page, Job ID Steps don't show until selecting another Job on the left. Fixed.

9. With Providex sources, when linking tables to use a filter from the parent table with Replace, ETL is logging failure because of incorrect column name for the filter name (ex.: H.InvoiceDate). Fixed.

10. Upon adding a new table from a NetSuite ODBC, the SQL syntax will have an extra ] at the end of the table name. Fixed.

11. ETL+ delta extractions issues with date filters fixed.

12. ETL+ sends email even when the feature is set to Cloud notification or No Notification.

13. Many SQL Statements for MongoDB ODBC will only load using SELECT *. Fixed.

14. Client Info page: The User’s Email registration button wasn’t working. Fixed.

Known Issues

1. Replace delta configuration for table joins only accepts MAX(FilterColumn).

2. Sometimes, the source metadata is deleted. For instance, opening an entity with a prior version and then with this one deletes the captured source metadata.

3. Upon upgrading an entity, connections to SQL might be set to Windows Authentication.

4. If adding a zDw SQL View to ETL+, it won’t recognize that the object name already exists in SQL and will allow saving it. If trying to delete the ETL+ table, the UI freezes. Kill ETL+, rename the SQL view, delete the table.

5. Trying to switch a data source from one type to another crashes ETL+.

6. When running Jobs, if ETL had a data warehouse connection error, it would switch into a failed load for all remaining job steps.

7. When running Jobs, a single failure to connect to the data warehouse errors out. The process wasn’t retrying 3 times.

8. In single-tenant data warehouses where sources are set to customized schemas, 0_UnionID columns might not bring in the correct value. For more on schemas See ETL+ Source Drivers | Schema-Checkbox

9. User actions to run a Job via de UI or Agent weren’t being logged.

10. Odata v3 extraction from GIs with spaces in their names might fail.

11. Regular users could open pre-saved 99 user UI in DataSelf domains.

12. Google Sheets: Issues extracting tables with space or line breaks in column names.

13. Google Sheets: Issues extracting some tables. You might need to re-load the sheet source metadata and remap it.

14. Importing a table that was already pre-mapped was duplicating the pre-mapped SQL Statement instead of adding the table from scratch again.

15. Adding new tables sometimes wouldn’t automatically create their Extract SQL Statment.

16. Command line -d: Not working as designed.

17. Running Jobs from Acumatica sources that don’t have their source metadata captured, and Acumatica doesn’t render the metadata, ETL+ Job refresh will hang indefinitely.

18. When running Jobs and a source’s captured metadata is NULL, ETL+ will load the source metadata.

19. Excel and CSV mappings would not list all columns in the target metadata.

20. Upon logging in, ETL+ is not remembering the last open entity.

21. Agent start is logging as connected to the default metadata even when customized.

22. Regular users: they can create/save a new data warehouse if their data warehouse is set to multi-tenant.

23. The Job Status column doesn’t necessarily reflect the status until leaving the Job page and coming back.

24. Debug OFF still creates a local log file.

25. When a source doesn’t have its metadata saved, and a table is loaded on the ETL UI, the source metadata load happens without visual feedback. It looks like it's frozen, but it’s not.

26. Target varchar(n) fields from OData sources are filling up with trailing spaces. This doesn’t occur for varchar(max).

27. F2 and Rename actions don’t work on Job ID 2 or greater (use double-click instead).

28. Many tooltips stop working after a source refresh.

29. Upon upgrading, some installs might incorrectly initiate IP whitelisting. Execute popular fix from ETL+ Installation Troubleshooting.

30. Sometimes, refreshing the zData Warehouse will not load the list of tables. Close ETL+ and re-open.

31. With Providex sources, remote ETL sessions can’t open Design nor show delta refresh filters for tables with table linking.

32. When you need to parse a SQL Statement to build its list of columns, but the Confirm button is not enabled, change the Statement, Confirm, change it back, Confirm.

Released: 08/29/2022

New Features

1. ETL page: title shows the connected “Data Warehouse: [<SqlServerName>].[<Database>]”

2. ETL page: Settings icon updated.

3. ETL Page - Find: Added Find feature to quickly highlight matches on sources, data warehouse tables, and SQL Statements.

4. ETL Page - Find: Click the Find icon to expose/collapse the search box.

5. ETL Page - Find: Find is case insensitive for source, table, and column names, and is case sensitive for ETL SQL Statements (to be improved in future releases).

6. ETL Page: click (Ctrl + F) to open the Find box.

7. ETL Page: click (Ctrl + G) to open the Log page.

8. ETL Page: click (F1) to go to Help.

9. ETL page, left panel renamed to “ETL Source”.

10. ETL Source - MS Access: Added MS Access as a new source type.

11. ETL Source - MS Access: Enter file path and password if required.

12. ETL Source - MS Access: Replace delta refresh available.

13. ETL Source - MS Access: Upsert delta refresh available.

14. ETL Source - PostgreSQL: Added PostgreSQL as a new source type.

15. ETL Source - PostgreSQL: Enter server address, port, credentials, and database name.

16. ETL Source - PostgreSQL: Replace delta refresh available.

17. ETL Source - PostgreSQL: Upsert delta refresh available.

18. ETL Source - BigQuery ODBC: Support and adjustments to BigQuery ODBC.

19. ETL Source - Google Sheet (beta): Support to extract data from Google Sheets. 64bit beta version.

20. ETL Source - Google Sheet (beta): Enter Spreadsheet ID, Target (optional). More instructions to be provided.

21. ETL Source - Google Sheet (beta): Date fields are automatically converted to varchar(max) to prevent failures with invalid dates in MS SQL. Users can enter SQL Expressions on the Design page to convert invalid dates to valid values.

22. ETL Source - MySQL (beta): Support to extract data from MySQL. 64bit beta version.

23. ETL Source - MySQL (beta): Enter server name, port, credentials, and database name.

24. ETL Source - Oracle (beta): Support to extract data from Oracle. 64bit beta version.

25. ETL Source - Oracle (beta): Enter Default or TNS, host name, port, credentials, and service name.

26. ETL Source: Deleting ETL+ Sources deletes its mapped tables on the center panel and now also the physical data warehouse tables. User is prompted to confirm the physical table deletion.

27. ETL Source: select a source → (Ctrl + i) opens its Properties.

28. ETL Source: double-click a table to add/map it to the load process.

29. ETL Source: (F5) to refresh a source.

30. ETL Page, center panel: Renamed to ETL Objects. More descriptive of what the panel is about. Objects currently include source headings (to group sources' tables together), SQL schemas, tables, and table columns.

31. ETL Objects: Renaming tables on the center panel renames the physical data warehouse tables.

32. ETL Objects: Duplicating tables on the center panel duplicates the physical data warehouse tables. This is helpful when users want to perform tests on a table without affecting the official one.

33. ETL Objects: Deleting tables on the center panel deletes the physical data warehouse tables. User is prompted to confirm the physical deletion.

34. ETL Objects: zData Warehouse heading renamed to zData Warehouse (Reimports).

35. ETL Objects: Job icon updated.

36. ETL Objects: select a table → (Ctrl + Del) key to delete.

37. ETL Objects: select a table → (Ctrl + 2) key to duplicate.

38. ETL Objects: select a table → (F2) key to rename.

39. ETL Objects: select a table → (Ctrl + D) key to open Design.

40. ETL Objects: select a table → (Ctrl + O) key to re-load and show Columns and data types.

41. ETL Objects: select a table → (Ctrl + P) key to preview.

42. ETL Objects: select a table → (Ctrl + L) key to Load Now.

43. ETL Objects: click (Ctrl + J) to open the Job page.

44. ETL Objects: Added ... icon to the panel header for additional non-popular actions.

45. ETL Objects: select a table → ... → Show Columns.

46. ETL Objects: select a table → ... → Download to CSV.

47. ETL Objects: ... → Load All Columns.

48. ETL Objects: ... → Expand All / Collapse All for expanding/collapsing the source headers.

49. ETL Objects - CSV: Added Export to CSV tables that have been loaded to the data warehouse.

50. ETL Objects - CSV: Right-click an ETL Object table → More Actions → Download to CSV downloads data in CSV format into the folder: ETL+ folder/DS_Downloads/<EntityID>.<TableName>.csv (ex.: abcinc.TName1.csv).

51. ETL page: click (Ctrl + S) to Save.

52. ETL page: click (Ctrl + O) inside of ETL SQL Statement to Confirm.

53. Job page: “Refresh Batch” renamed to “Job” - this feature can run scripts that are not “refreshes”.

54. Design page: When changing the Data Type of a column, ETL+ posts the SQL expressions automatically applied. Users can edit the expression.

55. Design page: After a SQL expression is posted to a column, if a user changes the column’s Data Type again, ETL+ will prompt if it should update the SQL expression.

56. Design page: Loading of MS SQL Timestamp columns automatically converted to char(23).

57. DataSelf Agent: Greatly anticipated ETL+ auxiliary application to execute scheduled and remote tasks. Long-term vision: a user runs ETL+ on her laptop at a hotel and requests a data refresh from her on-premises ETL+ install, and orders a margarita (drink orders not included in ETL+ lol). Their on-prem DS Agent receives and executes the request on the on-prem ETL+. And then it notifies the user when the refresh is complete. Magic happens while the margarita buzz kicks in!

58. Agent: A locally deployed Agent can run scheduled jobs.

59. Agent: A locally deployed Agent can run on-demand jobs.

60. Agent: Because of the Agent framework, users can now stop a running job no matter if it’s running via ETL+ UI or via an Agent.

61. Agent: An ETL+ Job can be associated with one Agent or to WTS.

62. Agent: ETL+ Production installs must have direct access to source systems and have the Agent locally deployed and running with a local Windows Admin user.

63. Agent: ETL+ will capture in the metadata the Windows Device name to associate the ETL+ Entity and Job IDs to run by the Agent.

64. Agent: When assigning a computer to run the Agent for the first time, ETL+ assigns the username and password saved on the Job → Properties → Agent credentials. This user must have local Windows Admin Rights.

65. Agent: Runs as a Windows service (in the background) and checks the cloud metadata every 10 secs for instructions to execute.

66. Agent: Checks the cloud metadata if there are Jobs to run now. The rule is: (Jobs associated to run by this Agent) & Enabled & (Next Run Time <= now).

67. Agent: Checks the cloud metadata if there are on-demand tasks to execute. Currently, it can execute two tasks: run a Job now, and kill the execution of a running Job.

68. Agent: DateTime values related to the Agent are fixed in PST. Next release will add flexibility to this setting.

69. Agent Auto Start: When assigning a computer to run the Agent for the first time, ETL+ creates a WTS task to automatically run the Agent.

70. Agent Auto Start: On a given computer, only one Agent runs at a time. Attempts to run a second session in parallel will only log the attempt.

71. Agent Auto Start: The WTS that runs the Agent is scheduled to run every hour. This can be manually changed to run on Windows Startup, and change Setting to stop the app after 3 days.

72. Agent Protocol: The communication between the Agent and the cloud is via a single metadata table.

73. Agent Protocol: The Agent and ETL+ use the metadata table to monitor actions such as terminating a running Job.

74. Agent Scheduling: Once per day schedules.

75. Agent Scheduling: Repeat task every X days.

76. Agent Scheduling: Repeat task every X minutes within Y hours.

77. Agent Scheduling: Future schedules (ex.: run it next year).

78. Agent app: Right-click on the Agent on Windows Explorer → Properties → Details, see the Agent version on File version.

79. Agent app - Agent Log.txt: Creates Agent Log.txt log file on the same directory where the Agent is. This file contains high-level Agent debugging and monitoring logs.

80. Agent app - Agent Log.txt: Logs the Agent version upon its start.

81. Agent app - Agent Log.txt: Logs failed attempts to run the Agent (it’s usually ok since only one Agent can run on each computer).

82. Agent app - Agent Log.txt: Logs the ETL+ command line running a Job.

83. Agent app - Agent Log.txt: Logs the end of a Job run.

84. Job: Added Status column on the job list to show which app runs the Job. Options: WTS (Windows Task Scheduler) to DataSelf Agent.

85. Job: Status column = WTS Enabled: The job is assigned to run via Windows Task Scheduler like in prior ETL+ versions.

86. Job: Status column = WTS Disabled: The job is assigned to run via Windows Task Scheduler like in prior ETL+ versions. However, ETL+ will not run the Job because it’s disabled even if/when WTS tries to run it.

87. Job: Status = Agent Offline: The Job is assigned to an Agent that is either not running, or is unable to access DataSelf cloud.

88. Job: Status = Agent Ready: The Job is assigned to an Agent and is ready to run scheduled or on-demand Jobs.

89. Job: Status = Agent Disabled: The Job is assigned to an Agent but it won’t run the Job.

90. Job: Status = Agent Running: The Job is running.

91. Job: Status = Agent Stopping: The Job is/was running, but the user just clicked End Job in ETL+.

92. Job: Added Load column showing Default or Forced Load All.

93. Job: Added Start at column showing a summary of the schedule description.

94. Job: Moved Run Job icon to the left panel header.

95. Job: Added Job Properties icon on the left panel header.

96. Job: Added ... (More Actions) icon.

97. Job: Added ... → Run Job via Agent icon. This sends a message to the Agent to run the Job on demand.

98. Job: Added ... → End Job icon.

99. Job: Moved Rename Job icon under the ... icon.

100. Job: Moved Delete Job icon under the ... icon.

101. Job: Moved Create Win Shortcut icon under the ... icon.

102. Job - Right-click: Same changes as the icons above.

103. Job - Pop up: Shows Job name

104. Job - Pop up: Shows Agent to run the Job. Computer Name if DataSelf Agent will run the Job.

105. Job - Pop up: Shows Job Schedule.

106. Job - Pop up: Shows Load Type.

107. Job - Pop up: Shows Last Run Time.

108. Job - Pop up: Shows Next Run Time.

109. Job: Removed Windows Task Scheduler (WTS) section.

110. Job: Removed WTS Settings.

111. Job: Clicking the Job Properties icon shows a section for Schedule, Load Type, and App.

112. Job: Settings icon updated.

113. Job - Find: Added Find feature to quickly highlight matches on Jobs and Job Steps.

114. Job: click (Ctrl + F) to open the Find box.

115. Job: click (Ctrl + G) to open the Log page.

116. Job: click (F1) to go to Help.

117. Job: Standardized user interaction as main ETL page so right-clicking on panel objects pops up the available features. Ex.: right-click a job → Run Job.

118. Job: Added Refresh icon that updates Job status.

119. Job: Added Expand icon to show additional scheduling columns.

120. Job: Expand additional column: Last Run.

121. Job: Expand additional column: Next Run.

122. Job: Expand additional column: Server.

123. Job: The Refresh feature enters in a 10-sec auto-refresh mode when the columns are expanded.

124. Job Steps: now shows the Job <ID> on the header.

125. Job Steps: Added Move to Top icon/feature.

126. Job Steps: Added Move to Bottom icon/feature.

127. Job Steps: select a script → (Ctrl + i) to open its properties.

128. Job Steps: select a script → (Ctrl + Del) key to delete.

129. Job Steps: select a row → (Ctrl + D) key to push it down one row.

130. Job Steps: select a row → press and hold the (Ctrl + D) key to push it down.

131. Job Steps: select a row → (Ctrl + U) key to push it up one row.

132. Job Steps: select a row → press and hold the (Ctrl + U) key to push it up.

133. Job Steps: (Ctrl + T) to create a new Tableau Extract Refresh script.

134. Job Steps: New script Download to CSV to run on a schedule. Select a Job → + on the Steps panel → Download to CSV → select a table -> Save.

135. Job Steps: Right-click on a Job Step shows a popup with the icons/actions above.

136. Job Properties - Schedule: Layout re-arranged.

137. Job Properties - Schedule: Added Enabled checkbox. Unchecking it will prevent the Job from running via the command line (typically, this is done by the Agent and WTS).

138. Job Properties - Schedule: The Start time zone shows PST when the Agent is selected.

139. Job Properties - Schedule: The Start doesn’t show a time zone when WTS is selected. This assumes it’ll be run based on the time zone of the WTS running the Job.

140. Job Start at column shows PST when the Agent is selected. It doesn’t show a time zone when WTS is selected (this assumes it’ll be run based on the time zone of the WTS running the Job).

141. Job Last Run column shows PST when the Agent is selected. It doesn’t show a time zone when WTS is selected (this assumes it’ll be run based on the time zone of the WTS running the Job).

142. Job Next Run column shows PST when the Agent is selected. It doesn’t show a time zone when WTS is selected (this assumes it’ll be run based on the time zone of the WTS running the Job).

143. Job App: New section to manage the application running Jobs.

144. Job App: In a new installation, the App dropdown has the following options:

     1. Deploy the Agent on this device (<DeviceName>)

     2. WTS.

145. Job App: To deploy the Agent on a device, select Deploy the Agent on this device (<DeviceName>) on the dropdown, enter credentials with local Windows Admin rights on the right panel, check Assign the Agent on this device for all Jobs if applicable, press Apply. If the Agent hasn’t been downloaded yet to the ETL+ folder, this will download the Agent to the ETL+ folder, create a WTS task to run the Agent automatically during reboots, and run the Agent as a service.

146. Job App: Repeat the step above to re-deploy the Agent - in case the Agent is running as expected.

147. Job App: To upgrade the Agent, delete DataSelf Agent.exe from the ETL+ folder and deploy the Agent again.

148. Job App: Once an Agent is assigned to a Job, its App dropdown shows Agent on <DeviceName> or Agent on <DeviceName> (this device). This informs the Agent is assigned to a Job and the Windows Computer name running that Agent.

149. Job App: If there are multiple Agents available to run a Job, if required, use the App dropdown to assign a different Agent to a Job. For instance, the production server will be down for a few days, and the user assigns a Job to a backup server to run the Job during that period.

150. Job App: ETL+ and Job refresh will work the same way as in prior versions when the App dropdown shows WTS.

151. Job App: When the App = WTS, the right panels shows the credentials for the local Windows Admin user and the Apply button (similar to prior versions).

152. Job App: When the App = WTS, users can disable the scheduled refresh of a Job using the Schedule Enable checkbox. This doesn’t affect the WTS activities per se, but it’ll prevent ETL+ from actively running such disabled Jobs.

153. Load Engine: Depending on the source system, loading of Date and DateTime columns automatically converts invalid values to NULL.

154. Load Engine: The delta filter column now creates a physical index in the data warehouse.

155. Load Engine: PK columns used in delta loads now create physical indexes in the data warehouse.

156. Load Engine: Adjustments when recalculating MS SQL indexes for the filter column used in tables with Replace. Resulting in significant performance improvement when processing deltas in large tables.

157. Load Engine: Adjustments when recalculating MS SQL indexes for the filter and PK columns used in tables with Upsert. Resulting in significant performance improvement when processing deltas in large tables.

158. Command Line: parameter -e <EntityID>. Ex.: -e abccorp

159. Command Line: parameter -j <JobID>. Ex.: -j 1

160. Command Line: New command syntax: -e <EntityID> -j <JobID> -ui <Page> <action>.

161. Command Line: parameter -e <EntityID> -ui load -j <JobID>. Ex.: this runs JobID on a new Load session. This automatically closes once the job completes the refresh.

162. Command Line: parameter -e <EntityID> -ui log. Ex.: this opens the Log page on a new ETL+ session. This session stays open until the user closes it. Click the Refresh icon to view the latest log rows.

163. Command Line: parameter -e <EntityID> -ui agent. Ex.: opens the Agent page from an ETL+ session.

164. Command Line: ETL+ uses the latest locally saved encrypted credentials when running command lines.

165. Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, it’ll show the message “Another ETL+ is loading this table. Waiting for table release.”

166. Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, every 15 secs it’ll show the message “Checking table release…”

167. Load Engine: If an ETL+ entity tries to load a table that is currently being loaded by another entity, but the other entity isn’t active for over 90 seconds, the new entity takes over the table load.

168. Load Engine: The mechanism to check for “Another ETL+ is loading this table. Waiting for table release.” checks for which Statement.RefreshRunID is loading the statement and when was the last date/time the loading was active on Statement.LastRefreshBg.

169. Load Engine: Checking if Log tables exist only during the opening of an entity (instead of during the load of every table).

170. Load Engine: Shows when the load started (metadata date/time).

171. Job Loading Page: Runs on its own window (runs via command line).

172. Agent Monitor Page: Opens Agent page on its own window (runs via command line).

173. Security: New passwords require at least 12 complex characters (up from 8).

174. Security: New passwords now accept # @ as part of special characters.

175. Security: When ETL+ can’t connect to its metadata: "Unable to connect to cloud metadata via https and SSL SQL. ETL+ needs to whitelist your public IP, or your firewall or antimalware is preventing ETL+ connection to the cloud. Start ETL+ IP Whitelisting?"

176. Security: ETL Page - ETL SQL Statement: For multi-tenant data warehouses, clicking Confirm validates that only dbo and entityID schemas can be read from zData Warehouse.

177. Log Page: Runs on its own window (runs via command line). This allows users to keep it open for monitoring the results of load tests.

178. Log Page: It will remain open even if the user closes the main ETL+ session. This can be useful for using the Log page for monitoring ETL+ loads exclusively.

179. Log Page: Click Close to close it.

180. Log Page: Added Rows Retrieved parameter. Enter a new value and press Refresh icon.

181. Log Page: (F5) shortcut to refresh Log page.

182. Log Page: Titles and radio buttons changed to ETL Log and Table Stats.

183. Log Page: Table Stats layout rearranged and some labels renamed for easier viewing.

184. Log Page: Default sorting of Log pages is the Index column.

185. Log Page: User Admin users have access to an additional column that shows the (unmasked) Source Error message returned by source system.

186. Log - Table Stats: Table row counting uses SQL metadata (instead of count(*)). Faster performance.

187. Logging: Checking if Log tables exist only during the opening of an entity (instead of during a load of every table).

188. Settings - Refresh Email Notification: Removed email attachments. No longer necessary with cloud logging.

189. Settings: Debug Level dropdown shows a summary of what the log level does.

190. Settings: Debug Off now removes the log file on the local HD. For saving disk space and avoiding requiring ETL+ to Run as Admin when running from Windows-protected folders

191. Settings: Debug Off no longer counts and records the number of rows before and after a table load. Performance improvement.

192. Settings: New Debug Level 0 adds the log file on the local HD plus counts and records the total number of rows before and after a table load.

193. Settings: Debug Level 1 adds Level 1 log plus Level 0 features.

194. Settings: Debug Level 2 adds Level 2 log plus Level 0 features.

195. Settings - Restore ETL SQL Statement As Of: New dropdown list on Settings allows users to retrieve prior ETL SQL Server Statements. Useful to recover deleted and changed statements from the past.

196. Login: Forgot Password? button. It’ll email the password when initiated from whitelisted IPs with a valid username (email address).

197. ETL+ Plain Deployment: New framework for an on-premises data warehouse that easily loads the zDateUntreated table from a cloud Excel file.

198. Cloud Scripts: Increased to up to 7 parallel processes.

199. Metadata: Redesigned metadata for improved security, reporting, and indexing. This makes prior ETL+ versions incompatible and separated with the new version.

200. Metadata AWS platform: Doubled the CPU capacity.

201. Metadata AWS platform: Doubled the RAM capacity.

202. Login within the same month release: Starting on this version, users will be able to upgrade and downgrade ETL+ seamlessly within the same ETL+ month release.

203. Login within the same month release: Removed prompt about opening an entity with a different metadata version within the same ETL+ month release.

204. Login with a newer month’s release: Opening an entity with a newer ETL+ version than the last opened release is seamless.

205. Login with an older month’s release: ETL+ will suggest the user to upgrade ETL+ and opens the ETL+ Upgrade page.

206. Creation of .ink files in \ProgramData\DataSelf\ETLplus removed.

207. zEntity: Process to guarantee that there’s no duplication of EntityID values.

208. Admin user: Remembers the last admin credentials in DataSelf protected computers.

209. Admin user: Renamed “Tableau Deploy” to “Cloud Script” on the Client Information page.

210. Admin user: Added Unlock Editing button to release an entity locked by a user.

211. Support to Windows Server 2022.

212. UI Pages: Data warehouse was removed from the bottom left corner since it’s prominently listed on the ETL main page.

Fixed Issues

1. Tables with delta refresh no longer log the misleading message: “Error : Column name '<PK_column1,PK_column2,PK_column3etc>' does not exist in the target table or view.”

2. Regular users: When login in to ETL+, the list of available entities used to show deleted ones as well.

3. When manually loading a table, it will retry 3 times in case of failure (instead of 12 times).

4. Tables with delta refresh no longer run into the error: ‘Invalid column name ‘0_UnionID', but ETL+ still processes the delta refresh correctly.

5. When super users switched to a new entity within the same ETL+ session could create zEntity and zLog tables in the new entity’s data warehouse with the schema from the prior entity.

6. When super users switched to a new entity within the same ETL+ session could send refresh emails with the prior entity’s refresh email subject.

7. Deleting all Jobs no longer requires going to the ETL main page and back.

8. Deleting an entity and recreating it with the same ID would add a new row to zEntity table.

9. Clicking the Log icon after certain tasks (such as running a Job) no longer presents an "Unhandled exception" error.

10. Regular users: Mousing over Sources is now showing the popup with details.

11. Regular users: When login into ETL+, ETL+ will correctly open the selected entity from the dropdown list.

Known Issues

1. Replace with filters such as getdate()-30 are duplicating rows on day = getdate()-30.

2. ETL+ is not automatically converting Timestamp columns to char(23).

3. Mechanism alerting users to upgrade their ETL+ is not friendly.

4. ETL+ is trimming leading spaces in column names from Excel sources.

5. ETL+ dies when opening it for the very first time on a machine. Fix: run it again.

6. The ODBC delimiters don’t seem to be working.

7. When importing tables from SAP Hana ODBC, ETL+ is not creating SQL Statements that are case sensitive.

8. Stopping a Job via Agent sometimes doesn’t log the action.

9. When configuring a Job schedule with Repeat every 2 or more days and the Start Date is in the past, the calculation of the Next Run might be incorrect.

10. Job → Agent Monitor page: no longer needed.

11. Sometimes still errors out with: “Two or more ETL+ tried to refresh the EntityID TableName data warehouse table at the same time.”

12. Unable to rename Jobs 2 and onwards (?).

13. When opening the Job page, Job <ID> Steps don't show until selecting another Job on the left.

14. With Providex sources, when linking tables to use a filter from the parent table with Replace, ETL is logging failure because of incorrect column name for the filter name (ex.: H.InvoiceDate).

15. When Log Level is set to Off, it still creates empty local files when running Jobs via the command line.

16. ETL+ to create new dws with Containment type = Partial.

17. Creation of SQL tables many times don’t sort the columns in the same order as in the SQL statement.

18. Job Status column might not represent its status. Closing and reopening the Job page usually fixes the status column.

19. Stopping a Job via the Agent sometimes doesn’t log the action.

20. Upon adding a new table from a NetSuite ODBC, the SQL syntax will have an extra ] at the end of the table name. To fix it: remove ] and save.

21. The creation of SQL Statement must be case sensitive for ODBC to SAP Hana. The default delimiters are double-quotes “ (?).

22. Sometimes, trying to configure a Job’s app = WTS right off the bat doesn’t work.

23. Many SQL Statements for MongoDB ODBC will only load using SELECT *.

24. Some Acumatica OData v4 tables might show the error “The contents of table TableName are not available for this user“ and fail to import. This is an Acumatica security issue and we’ve requested how to fix it. While we wait for the Acumatica fix, use OData v3 to import such tables.

25. Adding tables from Sage Intacct might not work.

26. Job: If a table fails to refresh, the ETL+ might unexpectedly load several tables multiple times.

27. Sometimes ETL+ will mistakenly flag that the metadata has been changed and you need to exit and re-enter the entity. Just exit and re-enter.

28. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

29. The icon color and type for the column formats on the center panel aren’t 100% consistent yet.

30. Opening an entity with tables mapped to the zData Warehouse with an older ETL+ version will push such tables to other source databases on the center panel. DataSelf can fix that manually.

31. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

32. A few installs are hitting the error “There is insufficient system memory in resource pool 'internal' to run this query.” once in a while. It happened to only one table in a job, and a different on another job. Load All. First report started in v2021.09.

33. Sometimes ETL+ is closing unexpectedly after a data load.

34. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

35. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

36. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

37. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

Download files:
64 bits: DataSelf ETL Plus 64bits - 2022.02.0505.exe
32 bits: DataSelf ETL Plus 32bits - 2022.02.0505.exe

Released: 02/05/2022

New Features

1. ETL Log: Log icon now retrieves cloud log. Users can retrieve the log from anywhere.

2. ETL Log: By default, the Log shows the latest logs at the top of the page (sorted <DateTime> Desc).

3. ETL Log: Sort the Log page by clicking the Log column headers.

4. ETL Log: Click Sequence header to sort by <RefreshRunID> Desc & <DateTime> Asc.

5. ETL Log: Refresh icon re-loads the latest log entries.

6. ETL Log: Logged dates now show the time zone (currently, PST only).

7. ETL Log: Option to view Stats or Summary logs.

8. ETL Log: Stats show row stats, processing time, and refresh criteria by loaded table.

9. ETL Log: Summary shows loaded tables, rows, error messages, refresh batch headers (admin user: machine username).

10. ETL Log: Creates and populates the Log Summary in the data warehouse table <dw schema>.<zRefreshLog_Summary>.

11. ETL Log: Creates and populates the Log Stats in the data warehouse table <dw schema>.<zRefreshLog_Stats>.

12. ETL Log: Log columns available to Super User only: PK_Entity, MachineUser.

13. Table labeling: Improved labeling to uniquely identify tables (MS SQL standard):

    1. <Source Name> - <SQL Schema>.<Table Name>

    2. Ex.: ExcelFile - dbo.BudgetTable

14. Table labeling: In multi-tenant data warehouses (such as in Advanced Analytics), the schema doesn’t show on the center panel since all loaded tables belong to the same schema (<EntityID>).

15. Table labeling: Refresh Batch Steps don’t show schemas to simplify the table list. Schemas can be seen on a pop up by mousing over tables.

16. Source - zData Warehouse: The data warehouse automatically shows as a data source for additional data preparations and transformations.

17. Source - zData Warehouse: “zData ” source is listed at the bottom of the panels.

18. Source - zData Warehouse: Uses a green source icon to tell it apart from regular sources.

19. Source - zData Warehouse: This source provides access to all data warehouse tables and views for single-tenant data warehouses.

20. Source - zData Warehouse: This source provides access to data warehouse tables and views where <SQL schema> = <EntityID> or <SQL schema> = <dbo> in multi-tenant data warehouses (DataSelf Analytics Advanced Edition).

21. Source - zData Warehouse: Source’s Properties disabled.

22. Source - zData Warehouse: Source 'sDelete disabled.

23. Source - zData Warehouse: Use Settings -> Data Warehouse for its configuration.

24. Source Properties: Renamed Dw Schema to Schema. Sets Alias as the source tables' schema.

25. Data Warehouse Panel - zData Warehouse: When importing zData Warehouse tables, they’ll be grouped under a green grouping icon.

26. Data Warehouse Panel: Renaming a table only applies to its name, not the schema.

27. Data Warehouse Panel: Tables imported from the data warehouse keep the same SQL schema as the data warehouse default schema.

28. Data Warehouse Panel - Columns: Double-click a data warehouse table to show:

    1. Column list

    2. Column formatting

    3. Primary key

29. Data Warehouse Panel - Columns: Click + / - by a table name to expand/collapse its column formatting.

30. Data Warehouse Panel - Columns: :ColumnBlack: icon for regular columns.

31. Data Warehouse Panel - Columns: :ColumnGreen: icon for transformed columns.

32. Data Warehouse Panel - Columns: :PK: icon for primary key columns.

33. Data Warehouse Panel - Columns: :PK_Green: icon for primary key transformed columns.

34. Data Warehouse Panel - Design: added Design icon to access table Design page.

35. Data Warehouse Panel - Design: For many sources, the list of columns shows even when the SQL Statement involves transformations, table links, table unions and CTEs.

36. Design: New page for table design:

    1. Configure primary key columns

    2. Change column formats

    3. Apply column transformations

37. Design: Many sources don’t allow transformations during the extraction, or extraction transformations are limited and/or slow. The data is extracted and loaded into a temporary data warehouse table, and then transformed into to official table. These transformations happen in the data warehouse engine, therefore they are fast, efficient and provide a broad range of MS SQL transformation functions.

38. Design: Select a data warehouse table → Design to open the Design page.

39. Design: New page shows PK (primary key), Column Name, Dw Data Type, Transform SQL Expression.

40. Design: List shows gray and white banded lines.

41. Design: To change a Column’s format, use its Dw Data Type dropdown menu, or double-click the dropdown and type in the new format.

42. Design: Column MS SQL transformations available on Transform MS SQL Expression. Ex.: LEFT(ColumnName,5).

43. Design: Click the column header for Column Name or Source Data Type to sort the list.

44. Design: Press & hold & drag with the right button of a mouse on the border of a column to change its width/height.

45. Design: ETL+ will execute the Design changes when user clicks Confirm.

46. Design: If issues are found during Design changes execution, ETL+ will report which columns were successfully changed and which ones failed.

47. Design - PK: PK check boxes define a table’s primary key columns. Currently:

    1. PKs are necessary to process Upsert delta refreshes.

    2. No validation if the PK columns are the correct primary key.

    3. No creation of the physical PKs in the data warehouse.

    4. If physical PKs are needed, please create them directly in SQL after the first ETL+ import.

48. Design - PK: When importing a new MS SQL or Odata table that has PK in the source, ETL+ will automatically recognize and pre-configure its PK.

49. Design - PK: Check the PK check boxes to change/define the table’s primary key columns.

50. Design - PK: For text-based sources like OData, varchar(max) columns are automatically changed to varchar(50). We recommend reviewing the varchar size.

51. Design - PK: ETL+ logs the following when MS SQL can’t create/recalculate a PK: "Primary Key failure. PK requires rows with unique values and no NULLs. Review the PK fields.". This means the selected PK columns are not the table’s correct primary key - review/fix the PK configuration.

52. Design - PK: When upgrading an ETL+ entity, Primary Index columns automatically become Primary Key columns.

53. Column Formats: On the data warehous panel (center panel), double-click or right-click on a table name -> Show Columns to enable the expansion of this table’s column names and formats.

54. Column Formats: Click +/- by a column name to expand/collapse its column names and formats.

55. Column Formats: To reload column names and formats, Double-click again or right-click on a table name -> Show Columns again.

56. ETL Page - Load Type: Load Type section now only uses vertical real estate needed.

57. ETL Page - Load Type: Replace All has been renamed to Load All.

58. ETL Page - Load Type: Replace with Control has been renamed to Replace.

59. ETL Page - Load Type: Update has been renamed to Upsert.

60. ETL Page - Load Type: Hovering over the Load Type options shows its function.

61. ETL Page - Load Type: Hover over Load All: 1st: Load all. Ongoing: Reload all.

62. ETL Page - Load Type: Hover over Replace: 1st: Load all. Ongoing: Replace filtered rows.

63. ETL Page - Load Type: Hover over Upsert: 1st: Load all. Ongoing: Update modified, insert new rows.

64. ETL Page - Load Type: Hover over Append: 1st: Load all. Ongoing: Append all.

65. ETL Page - Load Type: Selecting Replace or Upsert expands its settings section.

66. ETL Page - Replace: Filter column now is a drop down.

67. ETL Page - Replace: The filter expression is automatically set to Max(FilterColumn).

68. ETL Page - Replace: Users can customize the filter variable. For instance:

    1. Constant: 1000

    2. Date constant: ‘02/15/2021’ (use quotes around the MM/DD/YYYY date format)

    3. SQL Expressions: for instance, getdate()-30

69. ETL Page - Upsert: Renamed Primary Index to Primary Key.

70. ETL Page - Upsert: Primary Key lists the PK columns (from the Design page).

71. ETL Page - Upsert: Primary Key lists the PK columns in the same order they appear in the SQL Statement.

72. ETL Page - Upsert: To edit the PK columns, click the Edit icon by the Primary Key or the Design icon on the table.

73. ETL Page - Upsert: Use the dropdown to select the Column with Last Modified Value.

74. ETL Page - Upsert: The Update filter is automatically set to Max(LastModifiedValueColumn).

75. ETL Page - SQL Statement: Support to MS SQL Server CTE (Common Table Expressions).

76. ETL Page - SQL Statement: Support to SQL statement comments. Syntax /* comment text */. This can only be used after an initial SELECT statement.

77. ETL Version Control: When opening an entity with a different ETL version than the last one used for the entity, ETL will notify and prompt what to do next.

78. ETL Version Control: When using a newer ETL+ version than the last one used on an entity, users will be prompted to upgrade or cancel.

79. ETL Version Control: When using an older ETL+ version than the last one used on an entity, users will be prompted to download the older version (by going to our download page) or cancel.

80. Load Engine: Ability to apply transformations after the load (previously, only available during extraction when the source SQL syntax allowed).

81. Load Engine: Source loading adds a 0_UnionID column with content = table schema. This is helpful for multi-company table linking. This new column will be further refined when more multi-company features become available.

82. Load Engine: Settings 0_Entity now creates 0_<EntityID> column (instead of 0_<schema>).

83. Load Engine: If the source table already has a 0_<EntityID> column, load engine will import this column (instead of creating it).

84. Load Engine: If the source table already has a [0_UnionID] column, load engine will import this column (instead of creating it).

85. Load Engine: ETL+ retries 3 times upon running into a load error in the UI, and then proceeds. In a multi table refresh, use the Cancel button to stop the refresh (Cancel applies before the next table load).

86. Load Engine - zEntity: Creates a <dw schema>.zEntity table with the following columns: PK_Entity, EntityID, EntityName, CYE_Value, DT_LatestLoadStart, FYrStart_Int2, PnLType, ReportingDate, 0_UnionID, 0_<EntityID>.

87. Load Engine - zEntity: If <dw schema>.zEntity already exists, ETL+ doesn’t overwrite it.

88. Load Engine - zEntity Columns: PK_Entity, EntityID and EntityName defining the entity.

89. Load Engine - zEntity.CYE_Value: It’s populated by Settings page -> CYE Value. This is used by many Financials templates to set up a description or a GL number to Current Year Earnings.

90. Load Engine - zEntity.DT_LatestLoadStart: At the start of a new load (individual or group of tables), ETL+ writes the current date & time on this column. Use case: to populate report DataAsOf fields.

91. Load Engine - zEntity.FYrStart_Int2: It’s populated by Settings page -> FYear Start: Mmm. A number between 0 and 12that defines the 1st calendar month of the first fiscal year (assumes period starts on the 1st of the month). Ex.: 2 for Feb 1st. 0 means a custom fiscal year. Use case: this field controls part of the logic on DataSelf’s 5_Date and 6_DatePeriod date tables.

92. Load Engine - zEntity.PnLType: It’s populated by Settings page -> P&L: Type. This column controls how DataSelf P&reports work: S → Standard P&L or E → EBIDTA type.

93. Load Engine - zEntity.ReportingDate: For customers that want to customize the actual reporting date. For instance, show this fiscal period metrics based on the end of last month’s date. It’s populated by Settings page -> Reporting Date. Options: 0 -> Today. 1 -> Yesterday, 2 -> Last month's end.

94. Load Engine - Load All: The load happens on a temporary table to allow users to query the official table with minimum interruption. Once the load completes, the temporary table becomes the official one. The temporary table is named Schema.TableName_TEMP.

95. Load Engine - Load All: The transition between temporary to official table happens very quickly when there are no transformations performed (delete official, rename temporary to official, create <0_EntityID> when applicable).

96. Load Engine - Load All: The transition between temporary to official table happens quickly when there are transformations performed (delete official, insert into official from temp table applying transformations).

97. Load Engine - Replace and Upsert: When user changes the mappings of a delta refresh table, and the table already has data, user is promped with:

    1. Column mappings modified. The delta refresh needs one "Load All" data load to sync up OR manual adjustments to the physical table.
       Force one "Load All" in the next load (manual or scheduled)?
       Yes/No/Cancel

98. Load Engine - Replace and Upsert: When user clicks Yes to the prompt above, ETL+ will force a Load All in the next table load (manual or scheduled), and then resumes the delta refresh.

99. Refresh Batch: Rename icon added for Refresh Batch Names.

100. Refresh Batch: Removed Run Refresh Batch icon from header (cleaner UI).

101. Refresh Batch: Removed Delete Refresh Batch icon from header (cleaner UI).

102. Refresh Batch - Steps: Select a step and Click & Hold the Up or Down icon to quickly move it.

103. Refresh Batch - Steps: Hover over Up and Down icons to show quick help pop up.

104. Refresh Batch - WTS: Repeat Task checkbox unchecked by default.

105. Refresh Batch - Steps: Pop up when hovering steps has been cleaned up and re-organized the layout to make it clearer what the step is about.

106. Data prep: If the source data brings in a Date and or DateTime value that is incorrect for SQL (common in text based data sources like Excel or OData, and Providex), apply transformations to fix the issue. Examples:

     1. Invalid date values (ex.: Feb 30): Convert the field to a character field.

     2. A date/time column has more characters than accepted by SQL. Remove the extra characters: CAST(LEFT(DateColumn),19) as DATETIME).

107. OData Extraction - Load All: ETL+ process the data extraction sorted based on the PK (usually available) or a key available from the OData metadata.

108. OData Extraction - Replace and Upsert: ETL+ process the data extraction sorted based on the delta refresh filter (ex.: InvoiceDate) plus the PK (usually available) or a key available from the OData metadata.

109. OData Extraction - Replace and Upsert: When starting a new load and there’s no dw.TableName_Temp2, ETL+ checks the delta refresh filter value from dw.TableName, applies the filter value into the source data extraction process, and loads the data into TableName_Temp. If the end of the extraction is reached before 100 pages, ETL+ deletes TableName, and renames TableName_Temp to TableName_Temp.

110. OData Extraction - Replace and Upsert: Starting an extraction like listed in the prior feature above, but if 100 pages are loaded without the end of the extraction, ETL+ moves/append TableName_Temp onto TableName_Temp2, deletes TableName_Temp, logs off Acumatica, logs in Acumatica again, and resumes the load into TableName_Temp. This process repeats until the end of the extraction, then TableName_Temp is appended onto TableName_Temp2, TableName is deleted, and TableName_Temp2 is renamed into TableName.

111. OData Extraction - Replace and Upsert: When starting a new load and there’s a TableName_Temp2 on the data warehouse, ETL+ starts the delta load from TableName_Temp2 (instead of from TableName) and processes the load as per the prior feature.

112. OData Extraction - Upsert: Be sure to configure the order of PK columns in the SQL statement to guarantee an ascending extraction. Failing to do this might cause delta data extraction inconsistencies. We highy recommend adding a date (like CreateDate) as the first column of a PK controlling an Upsert.

113. OData Extraction - Upsert: When executing OData extractions, the order of the PK columns in the OData extraction URL following the order of the columns the SQL statement (instead of alphabetical).

114. ODBC Configuration: Added an explanation about adding credentials to an ODBC driver.

115. ODBC Configuration: Added an explanation about potential reason for missing ODBCs is because of mismatch of ETL+ and ODBC 32/64-bit builds.

116. ODBC Extraction: Support to Pervasive ODBC.

117. Installation: For every entity, usually only one ETL+ installation requires setting up the connection to data sources. Therefore, we removed the installation step to connect every ETL+ to its data sources.

118. Installation: We removed the Windows Task Schedule step from the installation. Cleaner install.

119. All Users: On the user credentials screen, the Login button is preselected. Once credentials are entered, press Enter to login or click the Login button.

120. All Users - ETL+ Version Control: ETL+ flags its metadata with the ETL+ version used to maintain it.

121. All Users - ETL+ Version Control: When a user opens an entity with a different version than what’s on the metadata, ETL+ will prompt the user if he/she wants to proceed with the different version or Cancel.

122. All Users - ETL+ Version Control: If a user wants to open an entity with a newer or older ETL+ version than what’s in the metadata, the ETL+ will proceed and update the metadata ETL+ version information.

123. All Users - ETL+ Version Control: If a user doesn’t want to open an entity with a newer or older ETL+ version than what’s in the metadata, the ETL+ will prompt if the user wants to open ETL+ Release History webpage to download other versions.

124. Admin User: ETL+ admin user page now shows the ETL+ metadata version.

125. All Users: The new ETL+ installation flow no longer has a WTS step. This was confusing for many users.

126. Client Info: ETL+ license is now part of the metadata.

127. All Users: When opening an entity, the center panel cursor goes to the top of the page.

128. All Users: Added Send button on Client Info page to re-send product registration email.

129. All Users: Added Send button on Client Info page to re-send product registration email that now copies sales@ and support@ dataself emails.

130. Admin User: metadata.EntityAttribute new table to host customer and tech support information.

131. Admin User: Entities page has a search box - enter characters to find and select matching EntityIDs and Names.

132. Admin User: When landing on the Entities page, the cursor goes to the search box.

133. Admin User: When leaving the ETL page back to the Entities page, the prior Entity will pre-selected in the new Search box.

134. Admin User: When a single entity shows on the Entities page, just click Enter to Edit it (or click Edit with mouse).

135. Admin User: Tableau Script Deployment removed from Refresh Batch page (it was confusing).

136. Admin User- Duplicate Entity: Upon duplication of an Entity, ETL+ finds & replaces <OriginalEntityID> with <NewEntityID> in all SQL Statements. This is needed for many out of the box data warehouse table data manipulations. Ex.: for properly processing zEntity and zCalendarTable tables.

137. Admin User: ETL+ Customer Portal with easy access to consolidated metadata information.

138. Admin User: ETL+ Customer Portal can only be accessed from behind DataSelf’s firewall.

139. Admin User: Deployment of Advanced, Pro and Enterprise share the same foundation (ETL+ templates, Tableau tds, Tableau main and QA twb workbooks).

140. Admin User: Deployment of Pro and Enteprise replaces EntityID in SQL Statements with dbo (which is the default schema for such deployments).

141. Admin User: For Pro and Enterprise deployments, ETL+ will automatically create a SYSTEM_Date table in the data warehouse. Required for certain internal analytics processes.

142. Admin User: For Pro and Enterprise deployments, ETL+ will automatically create the zDateUntreated table for date manipulations.

143. Admin User: For Pro and Enterprise deployment, ETL+ will automatically create auxiliary tables for source database specific data manipulations (such as document types and GL Financials data sets).

144. Admin User: ETL+ License field controls the cloud template deployment script.

145. Admin User: Client Info page allows the configuration of the ETL+ License type: DataSelf ETL+, VAR, DIY, Advanced, Pro, Enterprise.

146. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = VAR + clicking Save will deploy a DataSelf Advanced Analytics NFR system on https://bi1.dataself.com.

147. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = Advanced + clicking Save will deploy a DataSelf Advanced Analytics system on https://dataselfbi.com.

148. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = Pro + clicking Save will deploy a DataSelf Pro Analytics system on https://dataselfbi.com.

149. Admin User: On the Client Info page, checking the Tableau Deploy checkbox + License = Enterprise + clicking Save will deploy a DataSelf Enterprise Analytics system on https://dataselfbi.com.

150. Admin User: Analytics Settings is now part of the ETL+ metadata. It contains information such as Analytics type (Tableau, soon Power BI as well) and other settings to automate maintenance tasks such as the deployment of their templates and extract refresh.

151. Admin User: New Stored Procedure to automate the maintenance of analytics tasks such as the deployment of Analytics templates and extract refresh.

152. Admin User: The Analytics Server URL from Analytics Settings shows at the bottom of the Tableau Extract Refresh page.

153. Upgraded Microsoft Code Signing with RSA4096 encryption.

154. Compatible with Windows 11.

Fixed Issues

1. Product registration email missed the token information. Fixed.

2. Adding a source table to the data warehouse was disabling the center panel’s icons for that table. The user had to click another table and go back to the new table to enable the icons. Fixed.

3. WTS was automatically pre-checking the Repeat task every feature. Fixed.

4. WTS was not setting Run whether user is logged on or not. This is actually a security issue, be sure to enter a local Windows admin username and password in the ETL+ WTS page to create the WTS task with the setting above.

5. Upon installing ETL+ on a new computer, if it doesn’t start, right-click ETL+ → Run as administrator. This is a required security feature to kick off the IP whitelisting process.

6. Mapping of Acumatica OData columns with spaces in their names in GIs will come without spaces in OData. Ex.: “Geo Customer” in GI comes as “GeoCustomer” in OData.

Known Issues

1. Tables with delta refresh will present an error: ‘Invalid column name ‘0_UnionID', but ETL+ still processes the delta refresh correctly. To prevent the error message from happening, manually create a new column 0_UnionID varchar(100) on tables with that error message.

2. Clicking the Log icon after certain tasks (such as running a Refresh Batch) might present a "Unhandled exception" error. Click Quit and re-open. It mostly only happens once.

3. Upon adding a new table from a NetSuite ODBC, the SQL syntax will have an extra ] at the end of the table name. To fix it: remove ] and save.

4. Some Acumatica OData v4 tables might show error “The contents of table TableName are not available for this user“ and fail to import. This is an Acumatica security issue and we’ve requested how to fix it. While we wait for the Acumatica fix, use OData v3 to import such tables.

5. When manually loading a table, it will retry 12 times in case of failure.

6. Adding tables from Sage Intacct might not work.

7. Deleting all Refresh Batches requires going to ETL main page and back to get a new one created.

8. Opening an entity from an older metadata version with the new one using a regular user (non Admin) will not update the metadata version in the cloud. Work around: open it with an Admin user.

9. Refresh Batch: If a table fails to refresh, the ETL+ might unexpectedly load several tables multiple times.

10. Sometimes ETL+ will mistakenly flag that the metadata has been changed and you need to exit and re-enter the entity. Just exit and re-enter.

11. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

12. Regular users: Mousing over Sources doesn’t show the pop up with details.

13. Regular users: When login to ETL+, in the first attempt, ETL+ will only open the first entity even when one uses another entity on the dropdown list.

14. The icon color and type for the column formats on the center panel isn’t 100% consistent yet.

15. Opening an entity with tables mapped to the zData Warehouse with an older ETL+ version will push such tables to other source databased on the center panel. DataSelf can fix that manually.

16. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

17. Sometimes ETL+ is closing unexpectedly after a data load.

18. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

19. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

20. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

21. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

22. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

23. Cloud WatchDog not working reliably. It’s been disabled.

Released: 09/13/2021

New Features

1. ETL UI: Added '?' help icon on main pages.

2. ETL UI: User Guide available at DataSelf ETL+ User Guide .

3. ETL UI: Many buttons have been replaced by icons.

4. ETL UI: Hover over icons to view their functions.

5. ETL UI: Hover over data structures such as data sources and tables to view metadata.

6. ETL UI: Contextual icons - they are grayed out when they don’t apply to selected objects (ex.: selecting a table on the Source panel disables Delete Data Source).

7. ETL page: Right clicking on the left two panels pops up the icon functions. The pop ups show the icons and their function names.

8. ETL page: Clean up of the ETL+ main page (some functions have been moved to other pages).

9. ETL page: ETL SQL Statement’s title shows the selected TableName (or Source.TableName when multiple sources are mapped).

10. ETL page: ‘Log File’ icon to open local log file in Notepad.

11. ETL page: ‘Refresh Schedule’ button changed/renamed to ‘Refresh Batch’ icon.

12. ETL page: ‘Refresh Batch Now’ function moved to the 'Refresh Batch' page.

13. Multiple data sources: When opening ETL+, the data source(s) will not list its table names by default. This enables a fast opening of ETL+ since it doesn’t have to wait to list source system table names dynamically. Click a data source and the ‘Refresh’ icon to list its table names - this process can take several seconds if the source system is slow to render its table name list.

14. Multiple data sources: Imports data from multiple data sources within the same ETL+ entity.

15. Multiple data sources: Source panel shows all data sources created.

16. Multiple data sources: Data sources are represented by a cylinder icon, and their table name list can be expanded or collapsed with the +/- icon by the data source icon. Tables are represented by a square window icon.

17. Multiple data sources: Select a table from a data source and click the ‘Add Table’ icon/function to map it to the ETL process.

18. Multiple data sources: Click the ‘Add Data Source’ icon/function, select a data source type (CSV, MS Excel, MS SQL Server, OData, ODBC), configure the connection information, click Connect (or Save).

19. Multiple data sources: Configuration of data sources now includes an editable Alias (DSN Alias).

20. Multiple data sources: Added ‘Save’ button (saves and closes without trying to connect to source).

21. Multiple data sources: When creating a new DSN, the Alias is auto populated as following:

    1. MS SQL: Database name.

    2. ODBC: ODBC name.

    3. Excel: File name.

    4. CSV: Windows Folder name.

    5. OData: Last word of the OData URL.

22. Multiple data sources: Configuration of data sources now includes a ‘Dw Schema’ checkbox. When checked, tables from this data source will be stored in the data warehouse using a SQL schema = DSN Alias. When unchecked, tables will be stored in the default data warehouse SQL schema.

23. Multiple data sources: In the DSN configuration page, use the ‘Change Data Source’ icon to replace it to another type (ex.: from OData to MS SQL).

24. Multiple data sources: The DSN pages have a Save button (helpful when doing remote DSN maintenance without access to the DSN).

25. Multiple data sources: Select a data source and click the ‘Delete Data Source’ icon/function to delete.

26. Multiple data sources: Select a data source and click the ‘Refresh’ icon/function to reconnect to the data source and list its tables.

27. Multiple data sources: Select a data source and click the ‘Properties’ icon/function to configure its settings.

28. Multiple data sources: Mousing over the left panel’s data source icons shows its metadata.

29. Multiple data sources: Mousing over the left panel’s tables from a data source shows its metadata.

30. Multiple data sources: The center panel organizes the data warehouse tables by data source alias.

31. Multiple data sources: The center panel data source alias icon is a square with 3 objects in it. You can expand/collapse the list of data warehouse tables with the +/- icon by this icon.

32. Multiple data sources: Mousing over the center panel’s data source alias icons shows its metadata.

33. Multiple data sources: Mousing over the center panel’s tables shows its metadata.

34. Multiple data sources: When a data source has its table names listed on the left panel, and the user clicks that same table on the center panel, the cursor will select the table on the source system list.

35. Multiple data sources: Clicking on a center panel’s data source alias icon and the ‘Load Now’ icon will load all of its mapped tables to the data warehouse.

36. Multiple data sources: Ability to import data from the data warehouse for further data prep.

37. Multiple data sources: When adding a table from a source, the cursor selects it on the center panel.

38. Load UI: The load UI page is now integrated in the ETL+ application (not a detached layer).

39. Load UI: Cancel button stops the load process when the table currently loading ends.

40. Load UI: Cancel button stops the load process between OData extraction blocks and tables.

41. Load UI: Shows how many rows have been loaded from the table being extracted now.

42. Load UI: Shows how many rows per table have been loaded: (X rows <action>).

43. Load UI: Shows the type of load: the <action> variable in (X rows <action>):

    1. Replace All: loaded

    2. Replace with Control: replaced

    3. Update: updated

    4. Append: appended

44. Load UI: Refreshing a long list of steps scrolls up showing the latest steps at the bottom of the list.

45. Load UI: Shows at the bottom of the table list when waiting (sleep mode) versus extracting data.

46. Load UI: Shows when refreshing each Tableau Extract.

47. Load UI: New background and font colors provide better contrast.

48. Load UI: Shows Source.TableName when multiple sources are available, otherwise only TableName.

49. Load UI: Creates MS SQL indexes for the filter column used in tables with Replace.

50. Load UI: Creates MS SQL indexes for the filter and PK columns used in tables with Upsert.

51. Cloud Watchdog: Monitors ETL+ scheduled refresh cloud logging and sends email notifications.

52. Cloud Watchdog: Notifies “Missed execution?” when a scheduled Refresh Batch hasn’t started 120 seconds after the scheduled time.

53. Cloud Watchdog: Notifies “Taking too long?” when a scheduled Refresh Batch takes longer than (10 mins + 1.5 x Last Refresh Time) to complete. This might be an indication that the refresh will not complete.

54. Cloud Watchdog: Always notifies when ETL+ “Notification Email” is set to “Cloud - Always Notify”.

55. Cloud Watchdog: Notifies on error when ETL+ “Notification Email” is set to “Cloud - Notify on Error”.

56. Settings: Creation of the Settings page and added its icon on ETL+ main page.

57. Settings: ETL version posted.

58. Settings: Page provides access to ‘Client Info’, notification about ‘3 secs between error retries', ‘Debug’ drop down, ‘0_EntityID Column’, ‘Data Warehouse’ configuration, ‘About’, 'Notification Email’.

59. Settings: ‘Refresh Batch Notification Email’ includes dropdown for:

    1. Always Notify: ETL+ always sends notification email.

    2. Notify on Error: ETL+ only sends email on error.

    3. Cloud - Always Notify: Cloud Watchdog always sends notification email.

    4. Cloud - Notify on Error: Cloud Watchdog only sends email on error.

    5. No Notification: No notifications from ETL+ nor Watch Watch Dog.

60. Settings: “Create Now” button added to “New Data Warehouse” page.

61. Refresh Batch: Renaming ‘Refresh Schedule’ to ‘Refresh Batch’.

62. Refresh Batch: Re-layout of the page.

63. Refresh Batch: Bottom left section shows WTS summary configuration.

64. Refresh Batch: ‘WTS Settings’ button provides access to WTS settings and Apply button.

65. Refresh Batch: To run a Batch either click the ‘Refresh Now' icon on a batch name, or select the Batch and click 'Refresh Now’ icon on the panel header.

66. Refresh Batch: Click the 'Delete' icon on a batch name to delete it.

67. Refresh Batch: ‘Add’ and ‘Create Shortcut’ became icons on the batch name list.

68. Refresh Batch: List of Refresh Batch Steps shows twice as much vertical real estate.

69. Refresh Batch: Clearer description of ‘All Data Warehouse Tables (include new ones)’ checkbox.

70. Refresh Batch: Click a refresh step and ‘Move Up’ & ‘Move Down’ icons to re-order the refresh steps.

71. Refresh Batch: New step types for ‘Tableau Extract Refresh', ‘Cloud Script’, 'Command Line’.

72. Refresh Batch: Click ‘+' icon ('Add Script’), and select a script type to create.

73. Refresh Batch: Select ‘Tableau Extract Refresh’ to configure an extract to be refreshed. These extracts must be hosted at https://dataselfbi.com and pull data from DataSelf's cloud data warehouses.

74. Refresh Batch: ‘Tableau Extract Refresh’ script requires the extract name.

75. Refresh Batch: ‘Tableau Extract Refresh’ default values for other fields:

    1. Project = <EntityID>

    2. Parent Project = NULL

    3. Tableau Site = <EntityID>

76. Refresh Batch: Select ‘Cloud Script’ to provide its name. Consult DataSelf to discuss your cloud script options (such as running other cloud ETL+ extractions, DataSelf Automatic Distribution, and Python custom applications).

77. Refresh Batch: Select ‘Command Line’ to provide its name and parameters. This will run the configured command line locally. Example of use cases: running locally Tableau extract refreshes, or on-premises DataSelf Automatic Distribution.

78. Refresh Batch: Select a script step, and click icons ‘Delete Script’, ‘Configure Script', or 'Run Script’ as needed.

79. Refresh Batch: Removed ‘Refresh Type' since this can now be done directly in 'Refresh Batch Steps’.

80. Refresh Batch: Moved ‘Debug’ drop down to Settings page.

81. Refresh Batch: Removed ‘Enabled’ from batch list (not functional yet).

82. Refresh Batch: Other UI buttons/icons become disabled when editing scripts.

83. Refresh Batch: Icons on the Steps panel differentiate objects between tables and scripts.

84. Refresh Batch: Mousing over icons provide their functions and name and/or metadata.

85. Refresh Batch: Mousing over a table pops up its metadata (DSN and refresh settings).

86. Refresh Batch: Removed ‘On Demand Data Refresh’ UI with delay start - no longer needed.

87. Refresh Batch: Shows ETL+ version on the top right.

88. Refresh Batch: With multiple data sources, Batch Steps show DataSourceName.TableName.

89. Refresh Batch: Added ‘Tableau Template Deployment’ icon - available to admin users only.

90. Refresh Batch: ‘Log File’ icon to open local log file in Notepad.

91. Engine: Notifies when ETL+ doesn’t find the required .NET installed.

92. Engine: When an extraction errors out, retry 3 times (3 seconds apart).

93. Engine: When manually running extractions, connection errors prompt for Retry, Cancel or Skip.

94. OData Extraction: For OData tables without built-in indexes, ETL+ will only extract data in blocks when indexes have been configured in ETL+. Without indexes, such extractions will be continuous (not in blocks).

95. OData Extraction: Character fields are loaded into varchar(max) in the data warehouse and are limited to the first 250 characters (next release will give users control of target column format).

96. OData Extraction: When running an RWC or Update extraction from large datasets, ETL+ logs to Acumatica, loads in sets of up to 100 extraction blocks into a temp table, loads the temp table to the target table, logs off, and then repeats this process for the next set of extraction blocks. This helps Acumatica rendering speed, and provides intermediary starting points in case of issues in the middle of the extraction process.

97. OData Extraction: When processing delta refreshes, the page size will automatically expand when the delta filter cannot get over the next page of data. The expanded page size only applies to the table being loaded. Message: The delta extraction filter requires an Extraction Pagination expansion from <current> to <new> rows.

98. OData Extraction: Ability to insert texts that include SQL special characters (such as ' \* *\).

99. OData Extraction: Debug Log file includes OData URLs executed when Debug is set to Level 2.

100. Acumatica OData v3 delta refreshes: Acumatica does NOT accept GI formula fields (ex.: =[GLHistoryByPeriod.FinPeriodID]) as filter for RWC and Update.

101. Acumatica OData v4 delta refreshes: Acumatica requires that filters for RWC and Update must not have NULLs and ETL+ extraction blocks must be bigger than the max number of records for each block of records with the same filter value.

102. OData v4 Extraction: Changed extraction process for large tables when OData v4 refused to render data.

103. OData v4 Extraction: When working with RWC or Update and using date filters, ETL+ filtering adapts to the date format of the source Acumatica.

104. OData v4 Extraction: If a table doesn’t have a native OData index, ETL+ errors out and logs “Extraction time out. Table without native OData index. Please create one via Replace with Control or Update.”

105. ODBC Extraction: Sage Intacct ODBC support.

106. ODBC Extraction: Quickbooks Online ODBC support.

107. OBDC Extraction: MongoDB ODBC support (non-relational database).

108. OBDC Extraction: Using native MondoDB ' (single quote) or ANSI [] (brackets) around reserved words and nested collections.

109. ODBC Extraction: Support to views from source system (previously, only tables).

110. Logging: ETL+ logs description and time for new functions/scripts (Tableau Extract Refresh, etc).

111. Logging: When launching ETL+ via command line, add the 3rd parameter “1” and ETL+ will log locally “Start ETL+” even before loading the cloud metadata.

112. Logging: Posts Source.TableName when multiple sources are available, otherwise only TableName.

113. Logging: Posts the number of rows loaded.

114. Logging: Posts the type of load: the <action> variable in (X rows <action>):

     1. Replace All: loaded

     2. Replace with Control: replaced

     3. Update: updated

     4. Append: appended

115. Logging: For large OData tables with RWC or Update, logs when 100 extraction blocks are loaded.

116. Logging: Posts 32 or 64bit ETL+ version on log refresh headers.

117. Client Information: ETL version posted.

118. Client Information: The following fields were moved to more relevant pages: Data source, Settings, 0_EntityID Column, Notification Email.

119. Client Information: Creation of a Dropdown for DataSelf License type (DataSelf ETL+, VAR, DIY, Advanced, Pro, Enterprise). Only available for super admin.

120. Client Information: Addition of checkbox to deploy out-of-the-box Tableau Templates for Advanced Analytics.

121. User Logging Page: ETL version posted.

122. Error handling: Notifies and logs incorrect Column names used in 'Replace with Control' or 'Update'.

123. Error Handling: Improved message description when ETL+ can’t connect to its data warehouse.

124. Error Handling: Added message describing that MS SQL doesn’t accept insert in timestamp columns.

125. Error Handling: For certain extraction errors, retry the extraction 3 times.

126. New Deployments: Removed step to load data now (it caused confusion).

127. New Deployments: ‘Deploy Tableau Script’ runs on dw3 cloud server the following command line: “C:\DataSelf\<SourceSystem>\Deploy.Bat <EntityID> <DwServer> <DwDbName>”

128. Super admin: ‘Client Info’ and ‘Refresh Batch’ pages allow the deployment of out-of-the-box ‘Tableau Templates’ (site, tdsx, twb) for Advanced Analytics and QA templates for Pro and Enterprise.

129. Super admin: Entity creation populates EntityID in Refresh Batch Email Notification’s Subject.

130. Super admin: Entity creation populates user email in Refresh Batch Email Notification’s Email.

Resolved Issues

1. Counter of Load UI page fixed (it was always counting 1 extra item).

2. Handling of OData v4 having more than one table with the same name.

3. Creation of EntityID now requires a unique ID.

4. Errors out in case of an attempt to create a duplicated data warehouse Schema.TableName.

5. Tagging of the NextRefreshTime metadata field wasn’t populating correctly.

6. More than 4 index fields in Update was looping in some OData extractions.

7. OData XML extraction of non-standard characters fixed.

Known Issues

1. Importing large XML content into varchar(max) might run into error: server stack limit has been reached.

2. Deleting all Refresh Batches requires going to ETL main page and back to get a new one created.

3. Changing the data warehouse of an entity won’t show on the bottom left corner until the entity is closed and re-opened.

4. Sometimes ETL+ is closing unexpectedly after a data load.

5. Sometimes ETL+ runs into an “Unhandled exception” error. Close and re-open ETL+.

6. Locking the ETL+ editing when someone else is editing it doesn’t work in some situations.

7. ETL+ still tries to send email even when set to Cloud notification, or No Notification.

8. ETL+ delta extraction from NetSuite ODBC might not work when using dates for filtering.

9. When opening an entity created in prior ETL+ versions, the data warehouse panel will show the source header blank. To fix this, open the entity in the prior ETL+ version, make any modification and save it. Now open the entity in the latest ETL+ and the source header appears.

10. Importing tables from the data warehouse works correctly but it might incorrectly log an error message.

Release date: 03/31/2021

New Features

1. Integrated IP Whitelisting. If ETL+ can’t connect to DataSelf cloud servers, it whitelists the new IP once the user enters the correct product registration information.

2. Friendlier installation. ETL+ is now an executable-only application. One just needs to download and run it. No installer, no changes to the Windows environment, no need to Run as Admin (unless in some systems when creating Windows Task Schedule tasks).

3. ETL+ automatically sorts columns alphabetically when adding new source system tables.

4. Enable SELECT * from MS SQL and OData data sources. Preview only works with MS SQL sources.

5. OData extraction ignores duplicated column names (avoiding OData → SQL extraction errors).

6. Refresh Schedule page: “All Tables” checkbox to include new tables added to the ETL process.

7. Refresh Schedule page: “Tableau Subset” checkbox runs a dedicated Tableau refresh batch for the selected Refresh Schedule. Example: the default Refresh Schedule refreshes all Tableau extracts overnight, and a 2nd one refreshes a subset of Tableau extracts hourly. The creation of Tableau refresh batches continues to be a manual process done by DataSelf.

8. Refresh Schedule page: “Create Shortcut” checkbox to create ETL+ auto refresh shortcuts.

9. Refresh Schedule page set to 800x600 resolution.

10. Refresh Schedule page lists Schedule ID (3rd parameter when running ETL+ via command line).

11. Windows Task Scheduler: “Repeat tasks every” X minutes (ex.: run every 5 mins starting at 8am).

12. Windows Task Scheduler: “For a duration of” X hours (ex.: run every 5 mins starting at 8am for 12 hours). To configure indefinite tasks, set duration to 24 hours.

13. Windows Task Scheduler: Enable configuration of “Recurs every” X days (ex.: once a week).

14. Windows Task Scheduler: Schedules tasks into the future (to prevent users from mistakenly configuring tasks in the past).

15. Windows Task Scheduler: UI re-layout for better clarity and more real estate for configurations.

16. Speed: “Update” delta refresh creates data warehouse indexes for the filter and index columns.

17. Speed: “Replace with Control” delta refresh creates a data warehouse index for the filter column.

18. SQL Integrity: “Update” delta refresh turns varchar(max) filter and index columns into char(50).

19. SQL Integrity: “Replace with Control” delta refresh turns a varchar(max) filter column into char(50).

20. When running ETL+ for the 1st time, the DataSelf cloud servers connection only happens after the user has accepted the EULA.

21. If ETL+ can’t download the EULA: it’ll show the EULA page blank (probably an internet connection, firewall or antivirus issue).

22. When ETL+ can’t connect to DataSelf cloud servers: shows error about IP whitelisting and/or firewall and antivirus adjustments. And then tries to whitelist IP (see feature 1 above).

23. Super admin: Settings page to select, modify or new data warehouse and its SQL schema. Blank schema = EntityID. Enter the new name to change the schema. Ex.: dbo.

24. Super admin: Data warehouse configuration page has a Multi-Tenant checkbox. Check it for Advanced Analytics data warehouses, and uncheck it for Enterprise Analytics.

25. Super admin: Client Info page: “0_EntityID Column” checkbox to create this column.

26. Super admin: ETL+ logging in the cloud.

27. Super admin: ETL+ cloud auto refresh log includes EntityID, TableName, Begin Date/Time, End Date/Time, Rows Before, Rows Deleted, Rows Updated, Rows Kept, Rows Inserted, Rows After, Refresh Criteria, Result Type, ETL+ User, Refresh Schedule ID, Refresh Schedule Run ID, Windows Computer Name, Windows Username.

28. Super admin: “Entity” box converts all characters to lower case.

29. Super admin: Creation of new OData entities sets the OData extraction blocks to 1000 records and 100ms pause (instead of 0 recs and 0ms pause which causes issues).

30. Super admin: Wider Entity ID column on the Entities page.

Resolved Issues

1. Super admin: Issues when creating new entities.

2. The 32-bit version wasn’t saving ODBC credentials.

Known Issues

1. When configuring Acumatica OData delta refreshes, in the case that OData doesn’t render the indexes of a table (Acumatica bug?), you might get an “invalid type for an index” or “invalid for use as a key column” error message. To fix it, keep the delta configuration as is, change the “Load Type” to “Replace All”, do a table refresh, and switch back to delta refresh.

2. Failing to entering correct credentials 5 times in a row closes ETL+ but allows immediate 5 new re-attempts. The correct process is to force a 5 minute wait after 5 failed attempts.

Release date: 02/28/2021

New Features

1. Move Windows User variables to the Windows ProgramData area.

2. Simplifies running ETL+ and its auto refresh processes regardless of Windows users.

3. Collaboration: When a user is editing an ETL+ entity, other users can only log, view or copy its settings. To release editing to other users, Save the entity or close ETL+.

   Other users trying to edit it will receive the message: “username is currently editing this entity. You can only view it. To re-load the modified entity, exit and re-enter ETL.”

4. Collaboration: When a user tries to edit ETL+ but the metadata has changed by another user, ETL+ gives the message: ETL+ metadata has changed. In order to edit your ETL+, please exit and re-enter to re-load your metadata.

5. Allowing users to open more than one ETL+ at the same time.

6. When running Refresh Schedules in parallel, log each of them on their own block (easier to read log file).

7. Table Load Types in Refresh Schedules: Refresh Schedules can run the default table load type (Refresh All, Replace with Control, Update, Append), or force a "Replace All". This helpful when using delta refreshes on a frequent schedule, and then forcing a replace all on a less frequent schedule to capture old records that have been deleted or modified.

8. ETL+ lets the same Refresh Schedule run in parallel, but might issue the error message: [DS ERROR : 0902] ETL+ cloud log conflict. Two or more ETL+ are running the same Refresh Schedule at the same time. The data refresh might be ok.

9. ETL+ allows the refreshing of the same data warehouse table in parallel. Users need to understand the implications of using this feature since it can lead to data consistency issues. Ex.: it’s Ok to append different data sets in parallel, but a full refresh in parallel is likely not desirable.

10. When editing ETL+, add an * to the Save button to indicate changes haven’t been saved yet. Clicking Save removes the * flag.

11. Support to OData v4. First release. Test it before using it for production.

12. Resized ETL+ UI to fit small resolutions down to 800x600.

13. ETL+ UI can be switched between default size, full screen mode and minimized.

14. New redesigned ETL+ metadata for better auto documentation, faster performance and reporting.

15. Upgrade from prior ETL+ versions requires the re-creation of entities, users and statements.

16. When refreshing tables on ETL+ UI, the refresh progress indicator shows the Refresh Schedule name at the top, the table counter and % progress inside of the circle, and the table being imported at the bottom.

17. Show ETL version and 32/64bits on the main page.

18. Ability to define MS SQL Server data warehouse’s database schema. The default schema = ETL+ EntityID. ETL+ admin users can set the schema to any other value (such as dbo). Multi-tenant data warehouses' ID=1 will always have schema = ETL+ EntityID.

19. RWC and Update will do the first refresh in full refresh mode.

20. Removed dozens of locally saved variables that are no longer used (they were used before metadata moved to the cloud).

21. "Leave without saving?" when a user tries to close ETL+ without saving changes.

22. For users with access to multiple entities, ETL+ remembers the last entity opened.

23. Show ETL version and 32/64bits on the top right of main pages.

24. Strong encryption of ProgramData info.

25. Log changes to the source system’s connection string.

26. Left and right trim to all user input fields (avoids spaces in usernames and passwords).

27. ETL+ captures who saved an ETL SQL Statement version. Currently, only available in the ETL+ metadata.

28. More informative error message when ETL+ cannot connect to its metadata or data warehouse.

29. If two or more ETL try to refresh the same table at the same time, if the table has RWC or Update, it might give the error message: [DS ERROR : 0910] Two or more ETL+ tried to refresh the XYZ data warehouse table at the same time. This can cause data consistency issues. Please run a full refresh for this table again.

30. ETL+ version number set to YYYY.MM.DDNN (NN is sequential starting on 01).

31. ETL+ EULA and About pages inform the latest ETL+ version and URL for download (in case users are installing an older version accidentally).

Resolved Issues

1. New features number 1 and 2 resolve several auto refresh friction points.

2. Fixed data auto refresh not running from some servers via command line.

3. Fixed a testing feature that prevented ETL+ from connecting to the cloud from some servers.

4. Fixed the auto email of new product registrations.

5. Fixed ETL+ Excel and CSV data sources to include headers on row 1.

6. Fixed Update and Replace with Control refresh from ODBC sources.

7. Fixed error message showing data warehouse structure names.

Release date: 01/13/2021

New Features

1. Multiple refresh schedules.

2. Add Debug Mode dropdown to Batch page. Levels: Debug Off, Debug L1, Debug L2. The debug applies to each DSBI site, and all of its refreshes and users.

3. Allow ETL+ to run multiple refreshes of the same Entity in parallel.

4. Make UI 99 to be sorted by EntityID (not Entity Name).

5. Add the dw name to the bottom left corner of the ETL+ UI.

6. Improve UI titles and descriptions for clarity and self documentation.

Resolved Issues

1. Test "Communications" and "User" tables with all fields from Sage CRM source.

2. Scheduler isn't remembering the saved schedule settings.

3. ETL+ was hanging in WTS. Issue with the feature that puts notifications on the system tray.

4. When trying to upload 2 tables now, on the dw table list: clicking on a table, pressing and holding Crtl and clicking on a 2nd table crashes the ETL.

5. Log when refresh runs via shortcut shows [DS ERROR: 9997] - Thread was being aborted.

Release date: 12/14/2020

New Features

1. Create button on the Source Tables column for "Include".

2. When creating the ETL+ password, re-label "Password" to "New Password".

3. Selecting DBs in the MS SQL credentials page: Refresh Button to reload the list of DBs.

4. Delete Data Warehouse Table now has a "Confirm Deletion?" prompt.

5. Password cannot have # as the only special character. Message: Password cannot have # as the only special character.

6. For CSV Data Sources: "The first row has a Column names" should be checked and grayed out and non-editable.

7. For CSV Data Sources: "The first row has a Column names" Delimiter should have comma ',' and grayedout and non-editable.

8. Replace with Control: Order By field box removed. It automatically creates the index based on the Control field.

9. Debug Mode creates a log file: Log entityID_debug.txt.

10. Support for NetSuite ODBC.

11. Download CSV and Excel files with URLs with strings after the file name.

12. Ability to use subqueries in SQL Statements.

13. 0_EntityID to all tables.

14. Replace with Control delta refresh.

15. Update delta refresh.

16. Download CSV and Excel files from URLs into DS_SourceFiles folder where the ETL+ is, and then extract data to data warehouse.

Resolved Issues

1. Refresh and Upload buttons fail with THROW Error.

2. The ETL is not accepting edits in Client Information and Refresh Schedule. DS ERROR 04000 - Transaction Failed.

3. When saving a data warehouse table, ETL shows: "Do you want to cancel all changes made in this section?".

4. In the Microsoft Excel data source, "the first row has a Column names" should be checked.

5. It takes too long to enable other functions after saving a SQL Statement.

6. Fixed OData extract: DSID: nfr_wac_acumatica1, Dw Table: 'TST jn - Prod Order Detail - In Process Released', Upload fails.

7. It wasn’t running the Tbl refresh at the end of the auto refresh. 

8. The data refresh email should only be sent when running the full refresh. 

9. Mapping of Odata fields failed.

10. When adding New Table, the cursor will allow to change this new table name. However, it can only save the new name if all other table names listed are above "New Target Table".

11. The ODBC Data Source box had two Connect buttons.

12. Using Preview in OData on some tables causing closing, freezing or error message.

Release date: 10/27/2020, First Release

Released Features

1. Minimum system requirements: 1 CPU, 2GB of RAM, 5GB of free diskspace, Windows 10, Windows Server 2012 R2 or newer, 32 and 64bit.

2. Installation process: EULA, user credentials and token, IP whitelisting, connection to data source, scheduling the data auto refresh via Windows Task Scheduler (WTS), refreshing data now, and optionally maintaining ETL+ metadata.

3. Microsoft certified application.

4. Creation of Windows Task Scheduler (WTS) task to run the data auto refresh, including schedule and user to run WTS service.

5. Cloud-based ETL+ metadata allows remote maintenance.

6. Cloud uptime of 99.9%.

7. Data source built-in connectors:

   1. ODBC (32 and 64 bits)

   2. MS SQL Server

   3. Odata version 3

   4. Excel (will prompt to install MS Access driver if required)

   5. CSV (will prompt to install MS Access driver if required)

8. Refresh of tables and subsequent Tableau data sources via encoded batch files.

9. Auto data refresh email notification.

10. Data load type: Replace all.

11. Create evolution bar for the data extraction when running via ETL+ UI.

12. Add refresh evolution process in system tray when running as a service.

13. Stores data and metadata in SSL secured multi-tenant AWS MS SQL Server data warehouse.

14. Upload data to AWS cloud servers via SSL secured IP-to-IP connection.

15. Locally encrypt user credentials to ETL+ and on-promises systems.

16. Credentials using user token, and requiring user to create an user with their email and complex password (minimum of 8 characters with lower/upper case, numbers and special characters).

17. If a users fails to enter correct credentials 5 times in a row, block access for 5 minutes.

18. ETL+ UI panes: source tables, data warehouse tables, SQL Statements.

19. About page to show EULA and product release.

20. Data warehouse table functions: new, duplicate, rename, delete, preview, upload now.

21. Refresh Now with options to Refresh Full, DW only, Tableau only.

22. Refresh Now with counter of 9 seconds (configurable), and buttons for Cancel, Pause, Run Now.

23. ETL SQL Statement with Edit Mode and Confirm button.

24. Locally saved ETL+ log file with refresh steps.

25. Email refresh process log to maintenance users.

26. Creates shortcut on Desktop to run ETL+ auto refresh processes.