Design a batch processing solution using the Data Source Connector
See a list of rulebase features not supported by the Data Source Connector
Understand alternatives to using the Data Source Connector
The Oracle Determinations Engine provides a generic API for embedded high-performance inferencing inside product components such as Oracle Determinations Server and Oracle Web Determinations. These components have their own interfaces and are well suited to large enterprise architectures and solutions, but if the intention is to apply a known set of data to a known rulebase (or set of rulebases), then they may prove to be more complex than is necessary.
The key requirements for a batch processing solution using the Data Source Connector are that:
The major contributing factors to performance bottlenecks in the Data Source Connector are a lack of available memory, the format of the data in the files being processed and the way in which the files are processed. The following are suggested as a means of addressing these performance issues:
Memory:
The solution for the memory issue is fairly obvious in that you must have plenty of memory available to the Data Source Connector (DSC).
Data format:
In order to maximize performance, it is essential to carefully consider the format of your csv files before they are loaded into DSC. The data in the csv files needs to be in a particular format, so by planning for this, you are reducing the amount of work that needs to be performed by DSC and therefore reducing performance overheads.
See the topic Create a .csv file descriptor for more information.
Processing method:
Performance is affected more by the size of files rather than the number of files, so if possible, have smaller files and process them in batches.
Increase the number of threads:
Performance can also be enhanced by increasing the number of simultaneous execution threads to be used by DSC, in the configuration file. By default this is set to 4, but the appropriate value to use for each configuration is something that can only be identified through performance testing.
The Data Source Connector supports the core engine features, such as custom functions and custom formatters. Since there is no user interactivity, the Data Source Connector does not handle inferencing events.
Essentially, there are two viable alternatives to using DSC:
Of these alternatives, the Determinations Server is probably the preferred option as it is scaleable by nature.