Running and managing Data reader protocols

This topic describes how to run a Data reader and various common tasks that administrators can perform on Data readers.

 

In the Data reader home page, you can see all existing Data readers and their main information: the table is sortable and searchable using the interactive header fields. You can also show or hide columns to your liking, by clicking the Column chooser button in the upper right corner of the table.

If a data source in the "Source" column is shown in red, that source is not currently available. This usually happens when a Data model is pushed from a development environment to a production environment using the Transporter tool: when this happens, connection strings may become invalid or may be missing altogether, because the Transporter package does not include connection strings.

 

The "Log" column allows you to enable the creation of a log file of the rejected records for each Data reader. If no record is discarded, then no log file is created at all. However, the general Data model log file always contains a log line related to each Data reader execution.

Each log line includes the following information:

  • Rejected element records
  • Reason for rejection
  • Entity and reader column number

The log file with the rejected records is created in the following locations:

  • For Text file Data readers, in the same path where the source text file is located
  • For SQL Data readers, in the "Other logs" folder under the "LOGS" section in the Cloud Administration Portal. The log file name includes the Data reader name, the Data reader ID and the timestamp. For example, SourceData_025_202208.log
  • For SAP Data readers, in the "Other logs" folder under the "LOGS" section in the Cloud Administration Portal. The log file name includes the Data model name, the Data reader ID and the timestamp. For example, Production_9958_202208.log

 

The "Stop on error" column, when enabled, stops each protocol in case a record is rejected.

 

Running a Data reader protocol

A Data reader protocol can be launched either from the Data reader home page or from the Capsule environment using a Procedure. See Procedures for more details.

To launch a Data reader protocol from the Data reader home page, select it and click the "RUN" button on top of the table.

 

To launch a Text file Data reader, select the protocol, select a source file from the drop down list available in the "File" column, and then click the "RUN" button.

You can also select the "All" option to have the Data reader process all files in the configured path. In this case, the Data reader will cycle through all files and will throw an exception for each file that contains an incorrect data type. All exceptions are logged in specific log files in the same directory, as described in the previous paragraph.

Then, based on the mapping configuration, the following scenarios may occur:

  • If the mapping is set to "Replace time slice", the Data reader will clear the target Cube's time period that corresponds to the incoming data and then save it in the Cube. This process is repeated for each compatible file in the configured path
  • If the mapping is set to "Add", the Data reader will add the incoming data to the target Cube. In this case, if the target Cube is numeric, the incoming value from the data source is added to the Cube. If the Cube cell already contains a value, the new value is summed to the existing one. If target Cube is not a numeric Cube (i.e. a date Cube, a text Cube, etc.) the incoming value always overwrites the existing one. This process is repeated for each compatible file in the configured path

When the Data reader has finished running, the dropdown menu will display the last file processed.

 

If a Data Reader exceeds the Max Item number when loading Entity members, a warning message appears and the Data reader will discard all excess members from that point on. If the Data reader log has been enabled from the Data reader table, a log file of the rejected records for each Data reader will be created. Lear more about Data reader log in the previous paragraph.

 

 

Managing Data reader protocols

You can perform different actions on one or more Data readers by selecting them and by clicking on the different buttons above the table.

The available actions are described below.

  • To edit the configuration of an existing Data reader, select it and click on the "EDIT" button that appears on top of the table.
    Once in the configuration area, you can edit the connection settings, mapping and the formulas and validation rules set in the ETL section using the menu in the upper right corner of the page.
  • To copy an existing Data reader protocol, select it and click the "SAVE AS" button on top of the table, then enter the name of the new protocol
  • To delete a Data reader, select it and click on the "DELETE" button that appears on top of the table. This action will delete the Data reader and cannot be undone.