MOVEit DMZ includes several applications which run periodically on the server, collectively called the Scheduled Tasks. These applications take care of maintaining a MOVEit DMZ system, and executing time-based actions such as delayed notifications and password expirations. The applications are run by two tasks in the Windows Scheduled Tasks list, which are added automatically during the MOVEit DMZ installation.
There are two groups of Scheduled Tasks, with some applications being in both groups. The first group is the DayTime group, which, by default, runs every 5 minutes of every day between 2AM and 12AM. Applications in this group are those which need to run throughout the day, such as for issuing delayed notifications, and cleaning up cached entries in the database. The Windows Scheduled Tasks entry that runs this group is called MOVEitDayTimeTask.
The second group is the Nightly group, which, by default, runs every night at 1AM. Applications in this group are mostly responsible for checking the consistency of the MOVEit DMZ system, archiving logs and secure messages, and expiring data. The Windows Scheduled Tasks entry that runs this group is called MOVEitNightlyTask.
In version 3.4 a batch file called RunOneTask was also introduced to allow administrators to selectively run individual tasks. More information about this utility can be found below.
Below is a list of the applications in each group.
DayTime Tasks
DeleteParmFiles
Cleans up stale parameter files needed for communication between the MOVEit DMZ application and the MOVEitISAPI file transfer module.
EmailNotify
Sends delayed notifications to both senders and recipients.
TableCleanup
Deletes stale sessions and folder permissions granted to various sessions.
Reenables users and IP addresses locked out for signon violations.
Nightly Tasks
ArchiveLog
Archives and then deletes audit log entries from the database once they are older then the configured retention period. Stores the archives in the /Archive/Logs folder on the MOVEit DMZ filesystem. This task also resets tamper-evident hash chains to reflect the new start of each organization's audit logs after logs are archived/deleted.
ArchiveMessages
Archives (optional) and then deletes package messages and attachments once they are older then the configured retention period. Stores the archives in the /Archive/Packages folder on the MOVEit DMZ filesystem.
ConsistencyCheck
Makes sure records in various database tables are consistent. For example, it may check that a file, the folder that contains the file, and the user who uploaded the file all belong to the same organization.
Makes sure database records match file system data and vice versa. In this check, each file and folder record in the database is matched up with a file and folder on the Windows file system.
If the ConsistencyCheck task finds errors, it will send a notification to the configured administrative e-mail address recommending that a DBFixup be performed to reconcile the errors. Run DBFixup from on the MOVEit DMZ console through the Start | Programs | MOVEit DMZ menu and it will step you through several prompts. You will need the password of the MySQL user that is used by MOVEit DMZ to access the database. This is a password that is set up at installation. For each database error, the DBFixup will either delete the offending table entry or modify it to make it consistent. For file system errors, DBFixup will delete un-matched files and folders.
Note: If MOVEit DMZ is running in a web farm environment, DBFixup can be run on any node, but only by a user who has access to the shared filesystem on the NAS backend. If DBFixup is not run by a user with the appropriate access, an error message will be displayed noting the requirement.
Note: DBFixup should be run as administrator.
In some cases, especially if there is a heavy overnight load while the consistency checker is running, it is possible to report false positives. The checker may find files that were in the process of being created, deleted, or moved at the exact time the checker ran. These errors will usually disappear the next day that the ConsistencyCheck task runs. If there is any doubt about a reported inconsistency, there is no harm in just waiting another day. The information in the notification email can also be used to research a file or folder ID through the Web interface.
CreateReports
Executes any scheduled reports that should be run and saves the report contents to their configured locations.
DeleteParmFiles
See above.
DeletePendingUsers
Removes deleted users from the database once they are no longer referenced by other elements.
Expires temporary users and issues warnings about pending temporary user expirations.
EmailNotify
See above.
GarbageCollection
Deletes old files from folders with the file cleanup option enabled.
Marks new files as not new once they are older then the folder's configured NewTime setting.
Deletes partial files older than 12 hours.
Deletes old and empty subfolders from folders with the subfolder cleanup option enabled.
Marks new secure messages as not new once they are older then the organization's configured NewTime setting.
Deletes old, unassigned SSH keys and SSL certs from the holding tank.
PasswordAgeUsers
Suspends users whose passwords are older than the configured password aging settings.
Sends warning notifications to users whose passwords are within the configured password age warning settings.
Sends notifications to interested administrators and GroupAdmins about password expirations and password warning notifications.
SyncLDAP
Synchronizes the properties of advanced-LDAP-authenticated users on the MOVEit DMZ server with the associated user records on the LDAP server.
Optionally adds user records for users that are found by the LDAP authentication source but do not exist yet on MOVEit DMZ.
SysCheck
Checks on the first day of every month to see if any of the system debug levels are at or above the Some Debug level. If so, sends an email message to the Send ErrorsTo email address(es), noting that high debug levels can impact performance.
Checks the remaining space available on each local system drive. If any drive has less than a configurable number of megabytes remaining, sends an email message to the Send Errors To email address(es), noting which drives are low on space.
By default, the minimum drive space remaining value is 1024MB (1GB). This value can be changed through the MOVEit DMZ Config utility.
Checks every few minutes that the machine and ISAPI URLs are still valid. The main reason that these URLs suddenly become invalid is that someone makes a security change to the IIS server and neglects to run the MOVEit DMZ Check utility to see whether their change is fatal or not. If either of these URLs time out, return a bad message or otherwise appear unhealthy, SysCheck sends an email message to the Send Errors To email address(es) with some suggestions to resolve the problem.
If Content Scanning (Anti-virus and/or Data Loss Prevention) is enabled, checks every few minutes that the scan engine is available. If the scan engine is not available, sends an email message to the Send Errors To email address and warns that the MOVEit DMZ server will not be able to transfer files until this situation is addressed. When the scan engine becomes available again, sends an email that states that scanning is now working.
TamperCheck
Goes through all log entries (by organization) and ensures that each organization's chain of cryptographic hashes remains intact. If any tampering is detected, a notification with an explanation and logs are sent to the Send Errors To email address(es).
A MOVEit DMZ Log Tamper Check link that manually starts TamperCheck (and displays running results in a command-line window) is available from the Start menu under the MOVEit DMZ program group. Any TamperCheck that ends with the phrase Completed with errors should be considered a failed TamperCheck; the exact reason for the failure will be explained in the log and in the notification email messages.
Logging and Error Handling
Each one of these applications writes its own log file to the common MOVEit Logs folder. (Typically, this is something like D:\moveitdmz\logs). Each run of generates a new log (subsequent runs do not append to an existing log), so old logs are automatically grandfathered so that up to 5 old copies of each scheduled application's log file are available.
Each scheduler application also writes out a log file specifically for errors that it encounters. These files will end in a .err extension, and will also be automatically grandfathered, just as the normal log files are. When no errors occur, the error log file will be empty. Otherwise, the specifics of the error encountered will be written out in the file. Also, if an error does occur, an email message will be sent to the Send Errors To email address configured for the system informing the recipient which host the error occurred on and which application encountered the error. The contents of the error log file will be included in the body of the email, and the appropriate normal log file for the application will also be attached to the email.
Manually Running a Single Task (RunOneTask)
The daily and nightly sets of scheduled tasks may be run manually as a set at any time through the Windows Scheduled Task interface. However, to run individual tasks within a set, you must use a batch file utility called RunOneTask, also located in your Scheduler directory.
D:\MOVEitDMZ\Scheduler>runonetask
Usage: RunOneTask NonWebDir moveitDSN TaskToRun
If you have installed MOVEit DMZ into a folder whose name uses spaces, you should use the 8.3 version of any foldernames provided to ensure RunOneTask is run properly. For example, if I have installed MOVEit DMZ into the D:\m i\mi dmz folder (instead of the usual D:\moveitdmz\midmz) then my RunOneTask command should resemble the following example.
In addition to the two groups of Scheduled Tasks, there is one more application which, while not directly scheduled, does run periodically. This is the SysStat service, which is responsible for periodically recording several performance statistics in a table in the MOVEit DMZ database. These values can be used to get a good overall picture of the health of the server.
By default, the SysStat service wakes up every 323 seconds (roughly five minutes) and records samples of the various performance statistics that it keeps track of. The default 323 second sleep period is chosen to make sure the service remains offset from the more even 5 minute schedule of the daytime scheduled tasks. Every 72 cycles, the service also does a complete check of disk utilization for each of the various MOVEit DMZ components. This operation is performed only periodically because the disk utilization values typically change far more slowly than other system performance statistics, and because the disk check takes longer to accomplish and requires more system resources than the sampling of the other statistics. Also for this reason, the SysStat service will not do a full disk utilization check on the first run after it starts up. Instead, the first disk check will be performed at a random future run, and then periodically after that.
The above configuration values are customizable, as is the length of time which the service keeps statistics in the database for (the default is 30 days). The values can be changed on the Miscellaneous tab of the MOVEit DMZ Configuration Utility.
The SysStat service stores its statistics samples in the sysstats table in the MOVEit DMZ database. The fields available in the table are listed below, along with descriptions of their contents. Fields prefixed below with an asterisk indicate those fields which are only populated every 72 cycles by default. During off-cycles, these fields are set to 0.
ID - Simple auto-incrementing ID number for each entry
StatTime - Date/Time stamp indicating when the samples in the entry were recorded
ResilNode - If the system is in a web farm DMZ cluster, this will contain the node number the samples in the entry were recorded on.
FilesDriveRootPath - Root drive path of the MOVEit DMZ encrypted file store.
FilesDriveSpaceFree - Number of bytes available on the drive containing the MOVEit DMZ encrypted file store.
FilesDriveSpaceUsed - Number of bytes used on the drive containing the MOVEit DMZ encrypted file store.
*FilesSpaceUsed - Number of bytes used by solely the MOVEit DMZ encrypted file store.
DBDriveRootPath - Root drive path of the MOVEit DMZ MySQL database.
DBDriveSpaceFree - Number of bytes available on the drive containing the MOVEit DMZ MySQL database.
DBDriveSpaceUsed - Number of bytes used on the drive containing the MOVEit DMZ MySQL database.
*DBSpaceUsed - Number of bytes used by solely the MOVEit DMZ MySQL database.
LogsDriveRootPath - Root drive path of the MOVEit DMZ debug log store.
LogsDriveSpaceFree - Number of bytes available on the drive containing the MOVEit DMZ debug log store.
LogsDriveSpaceUsed - Number of bytes used on the drive containing the MOVEit DMZ debug log store.
*LogsSpaceUsed - Number of bytes used by solely the MOVEit DMZ debug log store.
FilesTotalDB - Number of files on the MOVEit DMZ system according to the database records.
FilesSizeTotalDB - Total bytecount of files on the MOVEit DMZ system according to the database records.
CPUUsagePercentTotal - Total percentage of CPU usage by all running processes.
CPUUsagePercentDMZ - Percentage of CPU usage by the MOVEit DMZ web application (aspnet_wp).
CPUUsagePercentISAPI - Percentage of CPU usage by the MOVEit ISAPI module. As an ISAPI module, MOVEit ISAPI is run under the DLLHOST.EXE application. The SysStats service automatically determines which running DLLHOST process is responsible for the MOVEit ISAPI module, and determines the CPU usage percentage of that process.
CPUUsagePercentIIS - Percentage of CPU usage by the IIS webserver (inetinfo).
CPUUsagePercentDB - Percentage of CPU usage by the MySQL database server (mysqld-nt).
CPUUsagePercentDMZFTP - Percentage of CPU usage by the MOVEit DMZ FTP server (MIFTPSrv).
CPUUsagePercentDMZSSH - Percentage of CPU usage by the MOVEit DMZ SSH server (MIDMZSSHSrv).
CPUUsagePercentSched - Percentage of CPU usage by the various MOVEit DMZ scheduler applications (GarbageCollecti, EmailNotify, ArchiveLog, DeletePendingUs, PasswordAgeUser, ArchiveMessages, ConsistencyChec, DeleteParmFiles, SyncLDAP).
CPUUsagePercentCentral - Percentage of CPU usage by the MOVEit Central application, if running (MICentral).
MemUsedTotal - Total bytecount of memory used by all running processes.
MemFreeTotal - Bytecount of available memory.
MemUsedDMZ - Bytecount of memory used by the MOVEit DMZ web application.
MemUsedISAPI - Bytecount of memory used by the MOVEit ISAPI module.
MemUsedIIS - Bytecount of memory used by the IIS webserver.
MemUsedDB - Bytecount of memory used by the MySQL database server.
MemUsedDMZFTP - Bytecount of memory used by the MOVEit DMZ FTP server.
MemUsedDMZSSH - Bytecount of memory used by the MOVEit DMZ SSH server.
MemUsedSched - Bytecount of memory used by the various MOVEit DMZ scheduler applications.
MemUsedCentral - Bytecount of memory used by the MOVEit Central application, if installed.
VMSizeDMZ - Bytecount of virtual memory used by the MOVEit DMZ web application.
VMSizeISAPI - Bytecount of virtual memory used by the MOVEit ISAPI module.
VMSizeIIS - Bytecount of virtual memory used by the IIS webserver.
VMSizeDB - Bytecount of virtual memory used by the MySQL database server.
VMSizeDMZFTP - Bytecount of virtual memory used by the MOVEit DMZ FTP server.
VMSizeDMZSSH - Bytecount of virtual memory used by the MOVEit DMZ SSH server.
VMSizeSched - Bytecount of virtual memory used by the various MOVEit DMZ scheduler applications.
VMSizeCentral - Bytecount of virtual memory used by the MOVEit Central application, if installed.
HandlesTotal - Total count of handles open by all running processes.
HandlesDMZ - Count of handles open by the MOVEit DMZ web application.
HandlesISAPI - Count of handles open by the MOVEit ISAPI module.
HandlesIIS - Count of handles open by the IIS webserver.
HandlesDB - Count of handles open by the MySQL database server.
HandlesDMZFTP - Count of handles open by the MOVEit DMZ FTP server.
HandlesDMZSSH - Count of handles open by the MOVEit DMZ SSH server.
HandlesSched - Count of handles open by the various MOVEit DMZ scheduler applications.
HandlesCentral - Count of handles open by the MOVEit Central application, if installed.
ProcessesTotal - Total count of running processes.
ThreadsTotal - Total count of running threads by all running processes.
ThreadsDMZ - Count of running threads owned by the MOVEit DMZ web application.
ThreadsISAPI - Count of running threads owned by the MOVEit ISAPI module.
ThreadsIIS - Count of running threads owned by the IIS webserver.
ThreadsDB - Count of running threads owned by the MySQL database server.
ThreadsDMZFTP - Count of running threads owned by the MOVEit DMZ FTP server.
ThreadsDMZSSH - Count of running threads owned by the MOVEit DMZ SSH server.
ThreadsSched - Count of running threads owned by the various MOVEit DMZ scheduler applications.
ThreadsCentral - Count of running threads owned by the MOVEit Central application, if installed.
SessionsTotal - Total ASP.NET sessions registered with MOVEit DMZ.
SessionsActive - Total active (touched within last 5 minutes) ASP.NET sessions registered with MOVEit DMZ.
If serious errors occur during a statistics gathering cycle, SysStat will report them by sending an email message to the Send Errors To email address configured on the system, as well as by logging the errors in the Windows Application Event Log. In most cases, SysStat will record the information it was able to gather and continue its work. If SysStat is unable to gather information on a particular field, it will typically record a value of 0 or -1 for that field. The value -1 indicates an unknown error occurred. The value 0 is used in the case of a known error occurring (most often a process that SysStat is gathering information on is not running). Note that a value of 0 for a field does NOT always imply that an error occurred (CPUUsagePercent is often 0 for processes that are running but not doing anything).
A value of -1 is also recorded in certain database- and filesystem-related fields when the database and/or filesystem is remote. This is usually the case when MOVEit DMZ is in a webfarm configuration. Additionally, database performance statistics will only be recorded for a local MySQL database; local and remote SQL Server databases will not be queried for performance statistics.
MOVEit DMZ Helper Service
The MOVEit DMZ Helper service performs a number of utility functions for other MOVEit DMZ services.
CA and Client Certificate Management - The web interface services (IIS, etc.) do not run with sufficient privileges to directly alter the Microsoft Certificate Store. The MOVEit DMZ Helper service allows the web interface to indirectly create, import and delete certificates from the store.
