Crawler Plug-in Restrictions

Antonio Garcia, one of the major users of Oracle Secure Enterprise Search, wants the crawler to reject files that are very large. He points out the fact that, plain text files, such as log files grow very large. He expects the crawler to reject files that it does not want to index, based on it's type or size, such as zip files.

Timothy tries to resolve this issue. But, finally he finds out that the crawler cannot handle rejections on it's own. The crawler plug-in has to be coded to specifically check for and handle different mimetypes and file sizes. The crawler reads HTML and plain text into the memory. Therefore, if the files are very large, it could run out of memory. The crawler relies on the crawler plug-in to handle the various mimetypes and limit the file size.

This is one of the restrictions of Oracle Secure Enterprise Search crawler.