Source

A source is a reference to a host which defines a single location from which files are obtained for use in a task. A task may be configured with an unlimited number of sources.

Common Source Options:

• Folder(s) - Indicates the folder name or path in which MOVEit Central should look for files on the remote host. Note that to reference a remote filesystem share, you must have selected the MOVEit Central host that corresponds to that filesystem share. You may use Macros in this field.

  For filesystem, MOVEitDMZ, FTP, and SSH hosts, you may use wildcards here. For example, "/home/jal/data*/reports" will match any folder named "reports" which is a direct subfolder of a folder whose name starts with "data" and which is a subfolder of /home/jal. For example, both /home/jal/data/reports and /home/jal/dataaccounting/reports match, but /home/jal/reports does not.

  Additionally, a special MOVEit Central-specific "**" wildcard operator is available. "**" matches any number of intermediate subfolders. For instance, "/home/jal/**/reports" matches any folder named reports which is a direct or indirect subfolder of jal. The folders /home/jal/data1/2005/07/reports, /home/jal/2005/reports, and /home/jal/reports all match this folder specification, but /home/jal/2005/report1 and /home/jal/2005/reports/mydata do not. (However, that last folder would match if "Include Subdirectories" were checked.) Note that a trailing ** means the same thing as checking "Include Subdirectories"; e.g. /home/jal/2005/** means the same thing as /home/jal/2005 with "Include Subdirectories" checked.

  Full UNC paths cannot be used here when using the local filesystem host. In order to use a remote filesystem by UNC or mounted drive letter, the host must be added as a Share to the hosts list. If a UNC is entered, MOVEit Central Admin will attempt to find a matching Share host to use instead, and prompt the user to use that host.

• Ignore Folder(s) - When this option is checked, one or more folder masks may be entered. (See "Folders" above for exact syntax.) When MOVEit Central finds a folder name that matches one of the entered masks while searching subfolders recursively, it will be ignored. Note that the folder mask(s) match against the names of sub-folders in the current folder being searched, therefore the folder mask(s) should reflect folder names and NOT full or relative folder paths. You may use Macros in this field.
• File(s) - The filename (i.e. "readme.txt") or filemask (i.e. "*.txt") MOVEit Central should use to select files on the remote host. You may use Macros in this field.

  There are two special wildcard characters:

  *, which matches zero or more characters at that position in the filename

  ?, which matches exactly one character at that position in the filename

  You may use multiple wildcard characters in a single mask.

  For example, a*.rpt matches a.rpt, a1.rpt, and apple.rpt, but not apple.rp or lemon.rpt.
  a?.rpt matches a1.rpt and aQ.rpt, but not a.rpt, a12.rpt, or a1.rp

  You may also use multiple filenames or filemasks, separated by semi-colons (";"). The semi-colons act as an "Or" operator. For instance, the filemask "fred*.*;*.zip" will match fred7.txt and will also match sally.zip.

  The ampersand operator ("&") acts as an "And" operator unless it is preceded by $. ("$&" means to match an ampersand literally.) For instance, given a filemask of "*.txt&ready.dat&final.log", MOVEit Central will not process any files on the source location unless "ready.txt", "final.log" and at least one "*.txt" file are available. A semi-colon (";") has a higher precedence than "&", so "*.txt;*.doc&*.zip" means "either *.txt or *.doc, AND ALSO *.zip".

  If you want the task to wait until all filetypes in the mask are available from the source, also choose Retry If No Files Found.

  NOTE: If the filemask on an existing source is changed, and the Collect Only New Files option is enabled, a confirmation message will be displayed warning the user that changing the filemask while collecting only new files could cause old files to be downloaded from the source host. This is because file collection timestamps are stored by filemask internally in MOVEit Central. Changing the filemask negates existing timestamps.

• Ignore File(s) - When this option is checked, one or more file masks may be entered. (See "Files" above for exact syntax.) When MOVEit Central finds a file that matches one of the entered masks, it will be ignored. Note that the ampersand operator ("&") has no context in this field and is therefore always treated as a literal. You may use Macros in this field.
• Special Filter(s) - When this option is checked, you may specify filters that allow MOVEit Central to select files based on their size and date/time of last modification.

  Filters are built in a pop-up window accessed through the nearby "Edit" button. Supported size filters are expressed in your choice of bytes, KB, MB or GB. Supported date/time filters are expressed the difference between now and some other date in hours, days, months or years. A "match all/any" selection controls whether files must match all or just one of the filter criteria.
  

• "Browse" Button - The Browse button provides an easy way to select a folder on the remote host, which will automatically fill the Folder Name field. Files on the remote host can also be selected, which will automatically fill the Filemask field. If your source simply needs to download a specific file from a specific directory on the host, the Browse button is the easiest configuration method to use. If you need a more complex mask, however, the Filemask and Folder Name fields can still be edited after being filled by the Browse button.

  NOTE: If the source is using an FTP or SSH host with the Blind Downloads option enabled, the Browse button will not be available. Blind Downloads are used when the client is not allowed to do directory listings, or directory listings will not appear correctly. The Browse button usually will not function correctly in these situations, so it is disabled.

  NOTE: browsing will not work if the username or password fields contain references to certain types of macros, such as non-global task parameters.

• Collect Only New Files - When checked, picks only those files that are "new" off the remote server. For filesystem, FTP, and SSH hosts, MOVEit Central maintains a database of most recent timestamps by host, directory, and filemask. See Host Operations and Editing Task Failure Entries for how to edit this database. For MOVEit DMZ hosts, MOVEit Central versions prior to 3.2 relied upon MOVEit DMZ's interpretation of which files are new. Starting in version 3.2, MOVEit Central relies upon MOVEit DMZ's definition of newness only until the first new file is downloaded for any given folder. After that, MOVEit Central will use the same new file definition used by the other source types. In the case of filesystem files, MOVEit Central will use the Last Modified stamp when determining newness.
• After Successful Transfer - Indicates what MOVEit Central should do with an original file if all process and destination steps are completed without problems. The selected action (if any) will be performed on a per-file basis as soon as MOVEit Central has delivered it or processed it through all steps successfully; there is no "mass delete" or "mass rename" step performed against source files. The options are:
  • Do Nothing - this is the default.
  • Delete original(s) - Delete the original file from the source. (This is an effective way to prevent "double-posts".)
  • Rename original(s) to: - Renames the original files to the given name. The name you specify may contain macros, such as [OrigName]. Some hosts (especially FTP hosts) will permit you to move source files across folders using this option and properly formatted paths.

  NOTE: due to limitations in task level Transfer Exceptions, the "After successful transfer..." options should NOT be used in tasks that don't contain any destinations (e.g. a task that uses the built-in script "HTTP Put" instead of a standard destination step). This can potentially cause files on the source(s) to be deleted or renamed before being successfully processed by a task run.

  See POP3 Sources for the special case of deleting POP3 messages.

• Include Subdirectories - When checked, MOVEit Central will search subdirectories of the given folder for files matching the given file mask, in addition to the given folder itself.
• Expand compressed (zip) file(s) - When checked, tells MOVEit Central to attempt to uncompress any files ending in a .zip extension.
• Retry If No Files Found - If checked, times out and retries a directory listing if the previous listing was successful but resulted in no matching files. The usual retry count and retry timeout are used. This feature can be used to "wait" for a file to appear during the running of a task. If the maximum number of retries is reached and still no matching files are found, the task continues with the next step with no error flagged.

  If unchecked, a directory listing returning no files is not retried, and as a result, no files are downloaded from this source.

• Sign On With Default Username - If checked, this task will use the usual username and password for this host. If unchecked, operators may specific alternate credentials to use for this transfer.
• Use Default Connect Timeout - Gives the option to override the Connect Timeout setting for the specified host and set a different value. If this option is not checked, MOVEit Central will use the value entered in the Connect Timeout field. The Connect Timeout setting specifies how many seconds MOVEit Central will wait when attempting to connect to the host.
• Use Default Data Timeout - Gives the option to override the Data Timeout setting for the specified host and set a different value. If this option is not checked, MOVEit Central will use the value entered in the Data Timeout field. The Data Timeout setting specifies how many seconds MOVEit Central will wait when sending data to or receiving data from the host.
• Use Default Retry Count - Gives the option to override the Retry Count setting for the specified host and set a different value. If this option is not checked, MOVEit Central will use the value entered in the Retry Count field. The Retry Count setting specifies how many times MOVEit Central will attempt a step if it fails.
• Use Default Retry Timeout - Gives the option to override the Retry Timeout setting for the specified host and set a different value. If this option is not checked, MOVEit Central will use the value entered in the Retry Timeout field. The Retry Timeout setting specifies how many seconds MOVEit Central will wait between retry attempts of a step.

Specialized Source Options:

• Sign On with Default Username (MOVEit DMZ, FTP, and SSH servers only) - If checked, tells the program to sign on with the username and password configured in the host. If not checked, tells the program to use the username and password specified in the Username and Password boxes. In the case of FTP servers, this also applies to the little-used Account field. These fields may contain macro references.
• Use Default Server Transfer Type (FTP servers only) - If checked, causes MOVEit Central to use the transfer type configured in the host. If not checked, causes MOVEit Central to use the type (ASCII vs. Binary) selected.
• Use Default FTP Transfer Mode (FTP servers only) - If checked, causes MOVEit Central to use the transfer mode configured in the host. If not checked, causes MOVEit Central to use the mode (Active vs. Passive) selected. Passive is generally preferred, but not all servers support passive mode.
• Use Default XSHA1 Setting (FTP servers only) - If checked, causes MOVEit Central to use the XSHA1 setting configured in the host. If not checked, causes MOVEit Central to use the settings value selected.
• Additional commands to execute before transfer (FTP servers only) - "quote" commands to send to the server before each file is downloaded. This is in addition to the per-connection before-each-file "quote" commands specified in the host configuration. Macros may be used in this field.
• Additional commands to execute after transfer (FTP servers only) - "quote" commands to send to the server after each file is downloaded. This is in addition to the per-connection after-each-file "quote" commands specified in the host configuration. Macros may be used in this field.
• Use Default Blind Download (FTP and SSH servers only) - If checked, causes MOVEit Central to use the "blind downloads" setting configured in the host. If not checked, causes MOVEit Central to use the mode selected. This rarely-used setting is intended for unusual servers that have problematic implementations of commands like CWD and LIST commands.
• Sign On with Default Client Key (SSH servers only) - If checked, causes the public key defined in the host configuration to be used for authentication. (If no public key is configured, then public key authentication will not be used.) If unchecked, operators may choose an alternate public key using the "..." button. Most SSH servers require either a password or a client key; a few require both.
• Sign On with Default Client Cert (MOVEit DMZ and FTP servers only) - If checked, causes the SSL client certificate defined in the host to be used when connecting to the host. (If no client certificate is configured, then none will be used.) If unchecked, operators may choose an alternate certificate using the "..." button. See SSL Client Certificates.
• Rescan before Xfer (Filesystem, FTP and SSH servers only) - This option helps prevent MOVEit Central from downloading "incomplete" files from servers that make such files available for download before they are closed. It specifies that once one or more files matching the download criteria have been identified, MOVEit Central should rescan the directory, looking for changes in the files' size and date. In the case of files on a filesystem source, MOVEit Central also checks to make sure it can obtain read access to the file--that is, it checks that the file is not locked by another program. If a file has changed or is locked, it will be removed from the list of files to download. The purpose of "Rescan before Xfer" is to detect when another application is currently changing the files, so that MOVEit Central will not download a partial file. The value is the number of seconds to wait between scans. The default is 0, which deactivates the feature. A reasonable value for when you do need the feature might be 60 or 120 seconds.
• MD5 Checking (FTP and SSH servers only) - Specifies whether MOVEit Central should look for an MD5 file, containing MD5 hashes of source files on the FTP server, and what that MD5 file should be called (see the FTP Source Integrity page for more information). The following settings are available for checking for an MD5 file:
  • Never - Central will not look for an MD5 file.
  • If Present - Central will look for an MD5 file. If the file contains a hash for a source file, Central will check the file against the hash. If the file matches the hash, Central will allow the file to continue, otherwise it will generate an error. If Central does not find a hash for the source file, it will allow the file to continue.
  • Required - Central will look for an MD5 file and generate an error if it is not found. All files downloaded will be checked against the MD5 file. If a file does not match its hash, or a hash does not exist for the file, Central will generate an error.
• Download Limits (MOVEit DMZ, FTP, SSH, and filesystem only) - Limits the number of files that Central will download from this source in a single task run. The limit can be by file count or by byte count. In the case of a limit by byte count, Central will stop downloading files after the first file that causes the number of downloaded bytes to exceed the configured limit. A value of 0 means no limit. If Use Default is checked, the defaults configured for the associated host are used; otherwise, the values specified in the source are used.

  If the Re-Run Task Automatically option is enabled, Central will automatically re-run the task if the configured limits are exceeded, allowing Central to pick up the remaining source files without waiting for the next scheduled run of the task. Note that if this option is enabled and neither the Collect Only New Files or Delete/Rename After Successful Download options are enabled, the task could potentially end up looping indefinitely. For this reason, MOVEit Central Admin will display a warning message if it detects this condition while a source is being edited.

• Use Default UNC Transfer Type (Share hosts only) – If checked, causes MOVEit Central to use the Use Windows CopyFile API setting configured in the host. If not checked, causes MOVEit Central to use the selected Use Windows CopyFile API setting. By default, MOVEit Central uses the Windows CopyFileEx function to download files from UNC shares. Under certain conditions, the performance of this function can potentially suffer, especially when transferring large files. If bad performance is being experienced when downloading files from this UNC host, deselecting the Use Windows CopyFile API for UNC Transfers setting may help remedy the situation.

See also AS1, AS2, AS3.