Multi-Threaded File Search and Performance Tuning Options
VX Search is optimized for modern multi-core/multi-CPU systems and powerful disk RAID arrays and is capable of performing file search operations using a number of processing threads. In order to enable parallel scanning of input disks and directories, select the 'Performance' tab on the file search options dialog and specify how many directory scanning and file search threads and to use.
If the search operation is performed on a multi-core/multi-CPU system, set the total number of processing threads to the number of physical CPUs or CPU cores available in the computer system. The performance improvement may be especially significant when searching files in multiple physical disks or network shares using complex search queries.
For example, on a quad-core server with a test data set stored on SSD disks, file search performance reaches up to 35,000 Files/Sec when searching files using 1 processing thread, 58,000 Files/Sec when searching files in the same data set using 2 parallel processing threads and 72,000 Files/Sec when performing the same file search operation using 4 parallel processing threads.
When searching files over the network, the network latency significantly impacts the file search performance and using multiple parallel directory scanning threads and multiple file search threads is even more important. For example, the performance of a file search operation over the network reaches up to 5,000 Files/Sec when searching files using one processing thread, up to 9,800 Files/Sec when performing the same file search operation using 2 parallel processing threads and up to 19,600 when performing the same file search operation using 4 parallel processing threads.
VX Search provides the ability show files user names in file search results, but this option significantly reduces the file search performance and it should be used only when required. The operation of obtaining a user name for a file is a relatively slow operation, which causes VX Search to delay processing files while waiting for the operating system to return user names for file search results. In order to mitigate the performance degradation, it is highly recommended to configure the file search operation to use a number of parallel file search threads, which will be inquiring user names for multiple files simultaneously.
For example, when searching files stored on SSD disks with the option to show files user names enabled, the file search operation reaches up to 18,000 Files/Sec with 1 file search thread, up to 33,200 Files/Sec with 2 parallel file search threads, up to 46,000 Files/Sec with 4 parallel file search threads and up to 52,000 Files/Sec with 8 parallel file search threads.
In order to reach a high file search performance when searching files over the network with the option to show files user names enabled, the file search operation should be configured to use 4-8 parallel file search threads. For example, a file search operation over the network with the option to show file user names enabled reaches up to 2,000 Files/Sec when using 1 file search threads and scales very well up to 8,800 Files/Sec when performing the same file search operating using 8 parallel file search threads.