Advanced Topics - System Internals - Scheduled Tasks
Overview
MOVEit DMZ includes several applications which run periodically on the server, collectively called the
Scheduled Tasks. These applications take care of maintaining a MOVEit DMZ system, and executing time-based
actions such as delayed notifications and password expirations. The applications are run by two tasks in
the Windows Scheduled Tasks list, which are added automatically during the MOVEit DMZ installation.
There are two groups of Scheduled Tasks, with some applications being in both groups. The first group
is the "DayTime" group, which, by default, runs every 5 minutes of every day between 2AM and 12AM. Applications
in this group are those which need to run throughout the day, such as for issuing delayed notifications,
and cleaning up cached entries in the database. The Windows Scheduled Tasks entry that runs this group is
called "MOVEitDayTimeTask".
The second group is the "Nightly" group, which, by default, runs every night at 1AM. Applications in
this group are mostly responsible for checking the consistency of the MOVEit DMZ system, archiving logs
and secure messages, and expiring data. The Windows Scheduled Tasks entry that runs this group is called
"MOVEitNightlyTask".
In version 3.4 a batch file called "RunOneTask" was also introduced to allow
administrators to selectively run individual tasks. More information about this utility
can be found below.
Below is a list of the applications in each group.
DayTime Tasks
DeleteParmFiles
- Cleans up stale parameter files needed for communication between the MOVEit DMZ application
and the MOVEitISAPI file transfer module.
EmailNotify
- Sends delayed notifications to both senders and recipients.
TableCleanup
- Deletes stale sessions and folder permissions granted to various sessions.
- Reenables users and IP addresses locked out for signon violations.
"Nightly" Tasks
ArchiveLog
- Archives and then deletes audit log entries from the database once they are older then
the configured retention period. Stores the archives in the /Archive/Logs folder on the MOVEit
DMZ filesystem.
This task also resets tamper-evident hash chains to reflect the new start of each organization's
audit logs after logs are archived/deleted.
ArchiveMessages
- Archives (optional) and then deletes secure messages and attachments once they are older
then the configured retention period. Stores the archives in the /Archive/Secure Messages
folder on the MOVEit DMZ filesystem.
ConsistencyCheck
- Makes sure records in various database tables are consistent. For example, it may check
that a file, the folder that contains the file, and the user who uploaded the file all belong
to the same organization.
- Makes sure database records match file system data and vice versa. In this check, each
file and folder record in the database is matched up with a file and folder on the Windows file
system.
If the ConsistencyCheck task finds errors, it will send a notification to the
configured administrative e-mail address recommending that a "DBFixup" be performed to
reconcile the errors. Run DBFixup from on the MOVEit DMZ console through the "Start |
Programs | MOVEit DMZ", menu and it will step you through several prompts. You will need the password of the MySQL user that is used by
MOVEit DMZ to access the database. This is a password that is set up at installation. For each
database error, the DBFixup will either delete the offending table entry or modify it to
make it consistent. For file system errors, DBFixup will delete un-matched files and
folders.
Note: If MOVEit DMZ is running in a web farm environment, DBFixup can be run on any node, but only by a user who has access to the shared filesystem on the NAS backend. If DBFixup is not run by a user with the appropriate access, an error message will be displayed noting the requirement.
Note: DBFixup should be run as administrator.
In some cases, especially if there is a heavy overnight load while the consistency
checker is running, it is possible to report "false positives". The checker may find files
that were in the process of being created, deleted, or moved at the exact time the
checker ran. These errors will usually disappear the next day that the ConsistencyCheck
task runs. If there is any doubt about a reported inconsistency, there is no harm in just
waiting another day. The information in the notification email can also be used to research
a file or folder ID through the Web interface.
CreateReports
- Executes any scheduled reports that should be run and saves the report contents to their
configured locations.
DeleteParmFiles
See above.
DeletePendingUsers
- Removes deleted users from the database once they are no longer referenced by other elements.
- Expires temporary users and issues warnings about pending temporary user expirations.
EmailNotify
See above.
GarbageCollection
- Deletes old files from folders with the file cleanup option enabled.
- Marks new files as not new once they are older then the folder's configured NewTime setting.
- Deletes partial files older than 12 hours.
- Deletes old and empty subfolders from folders with the subfolder cleanup option enabled.
- Marks new secure messages as not new once they are older then the organization's configured
NewTime setting.
- Deletes old, unassigned SSH keys and SSL certs from the holding tank.
- Deletes stale sessions and folder permissions granted to various sessions.
PasswordAgeUsers
- Suspends users whose passwords are older than the configured password aging settings.
- Sends warning notifications to users whose passwords are within the configured password
age warning settings.
- Sends notifications to interested administrators and GroupAdmins about password expirations
and password warning notifications.
SyncLDAP
- Synchronizes the properties of advanced-LDAP-authenticated users on the MOVEit DMZ server with
the associated user records on the LDAP server.
- Optionally adds user records for users that are found by the LDAP authentication source but do
not exist yet on MOVEit DMZ.
SysCheck
TamperCheck
- Goes through all log entries (by organization) and ensures that each organization's
chain of cryptographic hashes remains intact. If any tampering is detected, a notification
with an explanation
and logs are sent to the "Send Errors To" email address(es).
A "MOVEit DMZ Log Tamper Check" link that manually starts
TamperCheck (and displays running results in a command-line window) is available
from the Start menu under the MOVEit DMZ program group. Any TamperCheck that ends with
the phrase "Completed with errors" should be considered a failed TamperCheck; the exact
reason for the failure will be explained in the log and in the notification email messages.
Logging and Error Handling
Each one of these applications writes its own log file to the common MOVEit "Logs" folder.
(Typically, this is something like "D:\moveitdmz\logs").
Each run of generates a new log (subsequent runs do not append to an existing log),
so old logs are automatically "grandfathered" so that up to 5 old copies of each
scheduled application's log file are available.
Each scheduler application also writes out a log file specifically for errors that it encounters.
These files will end in a ".err" extension, and will also be automatically grandfathered, just as
the normal log files are. When no errors occur, the error log file will be empty. Otherwise, the
specifics of the error encountered will be written out in the file. Also, if an error does occur,
an email message will be sent to the "Send Errors To" email address configured for the system informing
the recipient which host the error occurred on and which application encountered the error. The contents
of the error log file will be included in the body of the email, and the appropriate normal log file
for the application will also be attached to the email.
The "daily" and "nightly" sets of scheduled tasks may be run manually as a set
at any time through the Windows Scheduled Task interface. However, to run individual
tasks within a set, you must use a batch file utility called "RunOneTask", also located
in your "Scheduler" directory.
D:\MOVEitDMZ\Scheduler>runonetask
Usage: RunOneTask NonWebDir moveitDSN TaskToRun
If you have installed MOVEit DMZ into a folder whose name uses spaces, you
should use the "8.3" version of any foldernames provided to ensure RunOneTask
is run properly. For example, if I have installed MOVEit DMZ into the "D:\m i\mi dmz" folder (instead of the usual "D:\moveitdmz\midmz")
then my RunOneTask command should resemble the following example.
D:\m i\mi dmz\Scheduler>runonetask d:\mi09f8~1\midmz~1 moveitdmz syscheck
NonWebDir=d:\mi09f8~1\midmz~1
SchedLogDir=d:\mi09f8~1\midmz~1\logs
BEGIN One Task Run of syscheck
1 file(s) moved.
1 file(s) moved.
END One Task Run of syscheck
In addition to the two groups of Scheduled Tasks, there is one more application which, while not directly
scheduled, does run periodically. This is the SysStat service, which is responsible for periodically recording
several performance statistics in a table in the MOVEit DMZ database. These values can be used to get a good
overall picture of the health of the server.
By default, the SysStat service "wakes up" every 323 seconds (roughly five minutes) and records samples of
the various performance statistics that it keeps track of. The default 323 second sleep period is chosen to
make sure the service remains offset from the more even 5 minute schedule of the daytime scheduled tasks. Every
72 cycles, the service also does a complete check of disk utilization for each of the various MOVEit DMZ components.
This operation is performed only periodically because the disk utilization values typically change far more slowly
than other system performance statistics, and because the disk check takes longer to accomplish and requires more
system resources than the sampling of the other statistics. Also for this reason, the SysStat service will not do
a full disk utilization check on the first run after it starts up. Instead, the first disk check will be performed
at a random future run, and then periodically after that.
The above configuration values are customizable, as is the length of time which the service keeps statistics in
the database for (the default is 30 days). The values can be changed on the Miscellaneous tab of the
MOVEit DMZ Configuration Utility.
The SysStat service stores its statistics samples in the "sysstats" table in the MOVEit DMZ database. The fields
available in the table are listed below, along with descriptions of their contents. Fields prefixed below with an
asterisk indicate those fields which are only populated every 72 cycles by default. During off-cycles, these fields
are set to 0.
- ID - Simple auto-incrementing ID number for each entry
- StatTime - Date/Time stamp indicating when the samples in the entry were recorded
- ResilNode - If the system is in a web farm DMZ cluster, this will contain the node number the samples
in the entry were recorded on.
- FilesDriveRootPath - Root drive path of the MOVEit DMZ encrypted file store.
- FilesDriveSpaceFree - Number of bytes available on the drive containing the MOVEit DMZ encrypted
file store.
- FilesDriveSpaceUsed - Number of bytes used on the drive containing the MOVEit DMZ encrypted file
store.
- *FilesSpaceUsed - Number of bytes used by solely the MOVEit DMZ encrypted file store.
- DBDriveRootPath - Root drive path of the MOVEit DMZ MySQL database.
- DBDriveSpaceFree - Number of bytes available on the drive containing the MOVEit DMZ MySQL database.
- DBDriveSpaceUsed - Number of bytes used on the drive containing the MOVEit DMZ MySQL database.
- *DBSpaceUsed - Number of bytes used by solely the MOVEit DMZ MySQL database.
- LogsDriveRootPath - Root drive path of the MOVEit DMZ debug log store.
- LogsDriveSpaceFree - Number of bytes available on the drive containing the MOVEit DMZ debug log
store.
- LogsDriveSpaceUsed - Number of bytes used on the drive containing the MOVEit DMZ debug log store.
- *LogsSpaceUsed - Number of bytes used by solely the MOVEit DMZ debug log store.
- FilesTotalDB - Number of files on the MOVEit DMZ system according to the database records.
- FilesSizeTotalDB - Total bytecount of files on the MOVEit DMZ system according to the database records.
- CPUUsagePercentTotal - Total percentage of CPU usage by all running processes.
- CPUUsagePercentDMZ - Percentage of CPU usage by the MOVEit DMZ web application (aspnet_wp).
- CPUUsagePercentISAPI - Percentage of CPU usage by the MOVEit ISAPI module. As an ISAPI module, MOVEit
ISAPI is run under the DLLHOST.EXE application. The SysStats service automatically determines which running DLLHOST
process is responsible for the MOVEit ISAPI module, and determines the CPU usage percentage of that process.
- CPUUsagePercentIIS - Percentage of CPU usage by the IIS webserver (inetinfo).
- CPUUsagePercentDB - Percentage of CPU usage by the MySQL database server (mysqld-nt).
- CPUUsagePercentDMZFTP - Percentage of CPU usage by the MOVEit DMZ FTP server (MIFTPSrv).
- CPUUsagePercentDMZSSH - Percentage of CPU usage by the MOVEit DMZ SSH server (MIDMZSSHSrv).
- CPUUsagePercentSched - Percentage of CPU usage by the various MOVEit DMZ scheduler applications
(GarbageCollecti, EmailNotify, ArchiveLog, DeletePendingUs, PasswordAgeUser, ArchiveMessages, ConsistencyChec,
DeleteParmFiles, SyncLDAP).
- CPUUsagePercentCentral - Percentage of CPU usage by the MOVEit Central application, if running (MICentral).
- MemUsedTotal - Total bytecount of memory used by all running processes.
- MemFreeTotal - Bytecount of available memory.
- MemUsedDMZ - Bytecount of memory used by the MOVEit DMZ web application.
- MemUsedISAPI - Bytecount of memory used by the MOVEit ISAPI module.
- MemUsedIIS - Bytecount of memory used by the IIS webserver.
- MemUsedDB - Bytecount of memory used by the MySQL database server.
- MemUsedDMZFTP - Bytecount of memory used by the MOVEit DMZ FTP server.
- MemUsedDMZSSH - Bytecount of memory used by the MOVEit DMZ SSH server.
- MemUsedSched - Bytecount of memory used by the various MOVEit DMZ scheduler applications.
- MemUsedCentral - Bytecount of memory used by the MOVEit Central application, if installed.
- VMSizeDMZ - Bytecount of virtual memory used by the MOVEit DMZ web application.
- VMSizeISAPI - Bytecount of virtual memory used by the MOVEit ISAPI module.
- VMSizeIIS - Bytecount of virtual memory used by the IIS webserver.
- VMSizeDB - Bytecount of virtual memory used by the MySQL database server.
- VMSizeDMZFTP - Bytecount of virtual memory used by the MOVEit DMZ FTP server.
- VMSizeDMZSSH - Bytecount of virtual memory used by the MOVEit DMZ SSH server.
- VMSizeSched - Bytecount of virtual memory used by the various MOVEit DMZ scheduler applications.
- VMSizeCentral - Bytecount of virtual memory used by the MOVEit Central application, if installed.
- HandlesTotal - Total count of handles open by all running processes.
- HandlesDMZ - Count of handles open by the MOVEit DMZ web application.
- HandlesISAPI - Count of handles open by the MOVEit ISAPI module.
- HandlesIIS - Count of handles open by the IIS webserver.
- HandlesDB - Count of handles open by the MySQL database server.
- HandlesDMZFTP - Count of handles open by the MOVEit DMZ FTP server.
- HandlesDMZSSH - Count of handles open by the MOVEit DMZ SSH server.
- HandlesSched - Count of handles open by the various MOVEit DMZ scheduler applications.
- HandlesCentral - Count of handles open by the MOVEit Central application, if installed.
- ProcessesTotal - Total count of running processes.
- ThreadsTotal - Total count of running threads by all running processes.
- ThreadsDMZ - Count of running threads owned by the MOVEit DMZ web application.
- ThreadsISAPI - Count of running threads owned by the MOVEit ISAPI module.
- ThreadsIIS - Count of running threads owned by the IIS webserver.
- ThreadsDB - Count of running threads owned by the MySQL database server.
- ThreadsDMZFTP - Count of running threads owned by the MOVEit DMZ FTP server.
- ThreadsDMZSSH - Count of running threads owned by the MOVEit DMZ SSH server.
- ThreadsSched - Count of running threads owned by the various MOVEit DMZ scheduler applications.
- ThreadsCentral - Count of running threads owned by the MOVEit Central application, if installed.
- SessionsTotal - Total ASP.NET sessions registered with MOVEit DMZ.
- SessionsActive - Total active (touched within last 5 minutes) ASP.NET sessions registered with MOVEit DMZ.
If serious errors occur during a statistics gathering cycle, SysStat will report them by sending an email message to the Send Errors To email address configured on the system, as well as by logging the errors in the Windows Application Event Log. In most cases, SysStat will record the information it was able to gather and continue its work. If SysStat is unable to gather information on a particular field, it will typically record a value of 0 or -1 for that field. The value -1 indicates an unknown error occurred. The value 0 is used in the case of a known error occurring (most often a process that SysStat is gathering information on is not running). Note that a value of 0 for a field does NOT always imply that an error occurred (CPUUsagePercent is often 0 for processes that are running but not doing anything).
A value of -1 is also recorded in certain database- and filesystem-related fields when the database and/or filesystem is
remote. This is usually the case when MOVEit DMZ is in a webfarm configuration. Additionally, database performance
statistics will only be recorded for a local MySQL database; local and remote SQL Server databases will not be queried for
performance statistics.
The MOVEit DMZ Helper service performs a number of utility functions for other MOVEit DMZ services.
- CA and Client Certificate Management - The web interface services (IIS, etc.) do not run with sufficient privileges to directly alter the Microsoft Certificate Store. The MOVEit DMZ Helper service allows the web interface to indirectly create, import and delete certificates from the store.