# Coordinating CAS crawls and baseline or partial updates

After running a CAS crawl, you run a baseline or partial update that incorporates the records from a Record Store instance. How you configure the coordination between your CAS crawls and baseline or partial updates depends upon how complicated your environment is. This topic describes several scenarios.

## A single CAS crawl and then a Forge update

There is a simple case where the Deployment Template runs a full CAS crawl and then runs a baseline update that incorporates the records from the crawl. To create this sequential workflow of CAS crawl and then baseline update, you can doc the following:

- Remove the default `Forge.isDataReady` check from the baseline update script. This call checks whether a flag was set that indicates the incoming data files are ready for Forge. Removing this call means that the lock manager does not check the flag or wait on the flag to be cleared before running a CAS crawl.
- add a call to `runBaselineCasCrawl()` to run the full CAS crawl.

For example, this baseline update script calls `CAS.runBaselineCasCrawl("MyCrawl")` which runs a full CAS crawl named `MyCrawl`. Then the script continues with baseline update processing.

    <!--
        ########################################################################
        # Baseline update script
        #
      -->
      <script id="BaselineUpdate">
        <log-dir>./logs/provisioned_scripts</log-dir>
        <provisioned-script-command>./control/baseline_update.bat</provisioned-script-command>
        <bean-shell-script>
          <![CDATA[
        log.info("Starting baseline update script.");
        // obtain lock
        if (LockManager.acquireLock("update_lock")) {

            // call the baseline crawl script to run a full CAS
            // crawl.
            CAS.runBaselineCasCrawl("MyCrawl");

            if (ConfigManager.isWebStudioEnabled()) {
              // get Web Studio config, merge with Dev Studio config
              ConfigManager.downloadWsConfig();
              ConfigManager.fetchMergedConfig();
            } else {
              ConfigManager.fetchDsConfig();
            }

            // clean directories
            Forge.cleanDirs();
            PartialForge.cleanCumulativePartials();
            Dgidx.cleanDirs();

            // fetch extracted data files to forge input
            Forge.getIncomingData();
            LockManager.removeFlag("baseline_data_ready");

            // fetch config files to forge input
            Forge.getConfig();

            // archive logs and run ITL
            Forge.archiveLogDir();
            Forge.run();
            Dgidx.archiveLogDir();
            Dgidx.run();

            // distributed index, update Dgraphs
            DistributeIndexAndApply.run();

            // if Web Studio is integrated, update Web Studio with latest
            // dimension values
            if (ConfigManager.isWebStudioEnabled()) {
              ConfigManager.cleanDirs();
              Forge.getPostForgeDimensions();
              ConfigManager.updateWsDimensions();
            }

            // archive state files, index
            Forge.archiveState();
            Dgidx.archiveIndex();

            // (start or) cycle the LogServer
            LogServer.cycle();

          // release lock
          LockManager.releaseLock("update_lock");
          log.info("Baseline update script finished.");
          } else {
                log.warning("Failed to obtain lock.");
          }
          ]]>
        </bean-shell-script>
      </script>

You run the baseline update by running `baseline_update` in the apps/[appDir]/control directory.

For example:

    C:\Endeca\apps\DocApp\control>baseline_update.bat

There is also the very similar case where the Deployment Template runs an incremental CAS crawl and then runs a partial update that incorporates the records from the crawl. To create this sequential workflow of incremental CAS crawl and then partial update, you can do the following:

- Remove the default `PartialForge.isPartialDataReady` check from the partial update script. This call checks whether a flag was set that indicates the incoming data files are ready for Forge. Removing this call means that the lock manager does not check the flag or wait on the flag to be cleared before running a CAS crawl.
- Add a call `runIncrementalCasCrawl()` to run the incremental CAS crawl.
- Remove the call to `PartialForge.getPartialIncomingData()` that fetches extracted data files.

For example, this partial update script calls `CAS.runIncrementalCasCrawl("MyCrawl")` which runs an incremental CAS crawl named `MyCrawl`. Then the script continues with partial update processing.

      <!--
        ########################################################################
        # Partial update script
        #
      -->
      <script id="PartialUpdate">
        <log-dir>./logs/provisioned_scripts</log-dir>
        <provisioned-script-command>./control/partial_update.bat</provisioned-script-command>
        <bean-shell-script>
          <![CDATA[
        log.info("Starting partial update script.");

        // obtain lock
        if (LockManager.acquireLock("update_lock")) {

           // call the partial crawl script to run an incremental
           // CAS crawl.
           CAS.runIncrementalCasCrawl("MyCrawl");

           // archive logs
            PartialForge.archiveLogDir();

            // clean directories
            PartialForge.cleanDirs();

            // fetch config files to forge input
            PartialForge.getConfig();

            // run ITL
            PartialForge.run();

            // timestamp partial, save to cumulative partials dir
            PartialForge.timestampPartials();
            PartialForge.fetchPartialsToCumulativeDir();

            // distribute partial update, update Dgraphs
            DgraphCluster.cleanLocalPartialsDirs();
            DgraphCluster.copyPartialUpdateToDgraphServers();
            DgraphCluster.applyPartialUpdates();

            // archive partials
            PartialForge.archiveCumulativePartials();

          // release lock
          LockManager.releaseLock("update_lock");
          log.info("Partial update script finished.");
        } else {
          log.warning("Failed to obtain lock.");
        }
          ]]>
        </bean-shell-script>
      </script>

You run the partial update by running `partial_update` in the apps/[appDir]/control directory.

For example:

    C:\Endeca\apps\DocApp\control>partial_update.bat

## Multiple CAS crawls and multiple Forge updates

There is a more complicated case where multiple CAS crawls are running on their own schedules, and updates are running on their own schedules. To coordinate this asynchronous workflow of CAS crawls and baseline or partial updates, you add code that calls methods in `ContentAcquisitionServerComponent`.

For details about `ContentAcquisitionServerComponent`, see EAC Component API Reference for CAS Server (Javadoc) installed in <Endeca installation path>\CAS\<version>\doc\cas-dt-javadoc.

In your AppConfig.xml code, the main coordination task is one of determining how you time running CAS crawls and how you time running baseline or partial updates that consume records from those crawls. For example, suppose you have an application that runs three full CAS crawls and those records are consumed by a single baseline update. In that scenario, each of the three full crawls has its own full crawl script in AppConfig.xml that runs on a nightly schedule. And the AppConfig.xml file contains a baseline update that runs nightly to consume the latest generation of records from each of the three crawls. The `Forge.isDataReady` check is not required in the baseline update script because the source data is not locked.

Here is an example script for one of the full CAS crawls named `endeca`.

    <!--
        ########################################################################
        # full crawl script
        #
     -->

      <script id="endeca_fullCasCrawldoc">
        <log-dir>./logs/provisioned_scripts</log-dir>
        <provisioned-script-command>./control/runcommand.bat endeca_fullCasCrawldoc</provisioned-script-command>
        <bean-shell-script>
          <![CDATA[
        crawlName = "endeca";

        log.info("Starting full CAS crawl '" + crawlName + "'.");

        // obtain lock
        if (LockManager.acquireLock("crawl_lock_" + crawlName)) {

          CAS.runBaselineCasCrawl(crawlName);

          LockManager.releaseLock("crawl_lock_" + crawlName);
        }
        else {
          log.warning("Failed to obtain lock.");
        }

        log.info("Finished full CAS crawl '" + crawlName + "'.");
          ]]>
        </bean-shell-script>
      </script>

Here is an example script for a Forge baseline update that processes records from the three CAS crawls mentioned above (The record adapter configuration in the Developer Studio project specifies that Forge reads from multiple Record Store instances).

    <!--
        ########################################################################
        # Baseline update script
        #
      -->
      <script id="BaselineUpdate">
        <log-dir>./logs/provisioned_scripts</log-dir>
        <provisioned-script-command>./control/baseline_update.bat</provisioned-script-command>
        <bean-shell-script>
          <![CDATA[
        log.info("Starting baseline update script.");
        // obtain lock
        if (LockManager.acquireLock("update_lock")) {

            if (ConfigManager.isWebStudioEnabled()) {
              // get Web Studio config, merge with Dev Studio config
              ConfigManager.downloadWsConfig();
              ConfigManager.fetchMergedConfig();
            } else {
              ConfigManager.fetchDsConfig();
            }

            // clean directories
            Forge.cleanDirs();
            PartialForge.cleanCumulativePartials();
            Dgidx.cleanDirs();

            // fetch extracted data files to forge input
            Forge.getIncomingData();
            LockManager.removeFlag("baseline_data_ready");

            // fetch config files to forge input
            Forge.getConfig();

            // archive logs and run ITL
            Forge.archiveLogDir();
            Forge.run();
            Dgidx.archiveLogDir();
            Dgidx.run();

            // distributed index, update Dgraphs
            DistributeIndexAndApply.run();

            // if Web Studio is integrated, update Web Studio with latest
            // dimension values
            if (ConfigManager.isWebStudioEnabled()) {
              ConfigManager.cleanDirs();
              Forge.getPostForgeDimensions();
              ConfigManager.updateWsDimensions();
            }

            // archive state files, index
            Forge.archiveState();
            Dgidx.archiveIndex();

            // (start or) cycle the LogServer
            LogServer.cycle();

          // release lock
          LockManager.releaseLock("update_lock");
          log.info("Baseline update script finished.");
          } else {
                log.warning("Failed to obtain lock.");
          }
          ]]>
        </bean-shell-script>
      </script>