The sample CAS crawl pipeline also supports partial updates, as an alternative way to apply incremental CAS crawl updates to a Dgraph.
The partial pipeline looks as follows:
The LoadIncrementalCrawls record adapter reads in the incremental crawl data.
The main work is done by the UpdateManipulator (a record
manipulator), which uses expressions that evaluate the value of the
Endeca.Action property on a record:
- If the value of the Endeca.Action property is "UPSERT", then use an UPDATE_RECORD expression with an action of ADD_OR_REPLACE in order to add or update the record.
- If the value of the Endeca.Action property is "DELETE", then use an UPDATE_RECORD expression with an action of DELETE_OR_IGNORE in order to remove the record.
A partial update can only be run after a full crawl has been run and processed with a baseline update. In addition, if partial updates are being used, the baseline pipeline should be run only after a full crawl and not after incremental crawls.
Note:
For completeness, when using partial updates,
the sample baseline pipeline should be updated so that it no longer
reads and joins incremental crawl files (i.e., the delta
functionality should be removed from the pipeline), since the
incremental files will be processed and applied as partial updates
instead.
