Input record adapters

The record adapters load the source data.

To start, here is a quick review of how sample data included with the deployment template is processed. The sample wine application includes a sample dataset in the [appdir]/test_data/baseline directory. When processing the sample wine data, the load_baseline_test_data.bat or load_baseline_test_data.sh scripts are used to copy the contents of this directory into the [appdir]/data/incoming/ directory, as well as to set a flag in the EAC. This flag, named baseline_data_ready, indicates to the deployment template scripts that the data extraction process is complete and data is ready for processing. Once that has occurred, the deployment template's baseline update process copies these files into the [appdir]/data/processing/ directory before running the primary Forge process.

When using a default deployment template application, it is therefore necessary for all input record adapters to look in the [appdir]/data/processing/ directory for incoming data extracts. The deployment template handles this automatically by specifying the ‑‑inputDir flag when running the primary forge process. This flag overrides any absolute path specified for specific input adapters with the proper deployment template path: [appdir]/data/processing/. However, the --inputDir flag respects relative paths, resolving them relative to the path specified as the input directory.

The URL property of any record adapter component therefore only needs to specify the relative path to a specific file or subdirectory within the [appdir]/data/incoming/ directory. (Remember that files and subdirectories in the incoming directory are copied to the processing directory by the deployment template before Forge is run.)

For example, if a single extract file called data.txt is copied into the [appdir]/data/incoming/ directory before running a baseline, the URL property of that data's input record adapter should specify a URL of data.txt.

For a more complex deployment where, for instance, multiple text extract files are copied into the [appdir]/data/incoming/extracted_data/ directory before running a baseline update, the URL property of a single input record adapter configured to read these files should be set to extracted_data/*.txt.