Source

A source is a reference to a host that defines a single location from which files are obtained for use in a task. A task can be configured with an unlimited number of sources.

Common Source Options:

• Folder(s) - Indicates the folder name or path in which MOVEit Automation looks for files on the remote host. Note that to reference a remote filesystem share, you must have selected the MOVEit Automation host that corresponds to that filesystem share. You can use Macros in this field.

  For filesystem, MOVEit Transfer, FTP, and SSH hosts, you may use wildcards here. For example, "/home/jal/data*/reports" will match any folder named "reports" which is a direct subfolder of a folder whose name starts with "data" and which is a subfolder of /home/jal. For example, both /home/jal/data/reports and /home/jal/dataaccounting/reports match, but /home/jal/reports does not.

  Additionally, a special MOVEit Automation-specific "**" wildcard operator is available. "**" matches any number of intermediate subfolders. For instance, "/home/jal/**/reports" matches any folder named reports which is a direct or indirect subfolder of jal. The folders /home/jal/data1/2005/07/reports, /home/jal/2005/reports, and /home/jal/reports all match this folder specification, but /home/jal/2005/report1 and /home/jal/2005/reports/mydata do not. (However, that last folder would match if "Include Subdirectories" were checked.) Note that a trailing ** means the same thing as checking "Include Subdirectories"; e.g. /home/jal/2005/** means the same thing as /home/jal/2005 with "Include Subdirectories" checked.

  Full UNC paths cannot be used here when using the local filesystem host. In order to use a remote filesystem by UNC or mounted drive letter, the host must be added as a Share to the hosts list. If a UNC is entered, MOVEit Automation Admin will attempt to find a matching Share host to use instead, and prompt the user to use that host.

• Ignore Folder(s) - When this option is checked, one or more folder masks may be entered. (See "Folders" above for exact syntax.) When MOVEit Automation finds a folder name that matches one of the entered masks while searching subfolders recursively, it will be ignored. Note that the folder mask(s) match against the names of sub-folders in the current folder being searched, therefore the folder mask(s) should reflect folder names and NOT full or relative folder paths. You may use Macros in this field.
• File(s) - The filename (i.e. "readme.txt") or filemask (i.e. "*.txt") MOVEit Automation should use to select files on the remote host. You may use Macros in this field.

  There are two special wildcard characters:

  *, which matches zero or more characters at that position in the filename

  ?, which matches exactly one character at that position in the filename

  You may use multiple wildcard characters in a single mask.

  For example, a*.rpt matches a.rpt, a1.rpt, and apple.rpt, but not apple.rp or lemon.rpt.
  a?.rpt matches a1.rpt and aQ.rpt, but not a.rpt, a12.rpt, or a1.rp

  You may also use multiple filenames or filemasks, separated by semi-colons (";"). The semi-colons act as an "Or" operator. For instance, the filemask "fred*.*;*.zip" will match fred7.txt and will also match sally.zip.

  The ampersand operator ("&") acts as an "And" operator unless it is preceded by $. ("$&" means to match an ampersand literally.) For instance, given a filemask of "*.txt&ready.dat&final.log", MOVEit Automation will not process any files on the source location unless "ready.txt", "final.log" and at least one "*.txt" file are available. A semi-colon (";") has a higher precedence than "&", so "*.txt;*.doc&*.zip" means "either *.txt or *.doc, AND ALSO *.zip".

  If you want the task to wait until all filetypes in the mask are available from the source, also choose Retry If No Files Found.

  Note: If the filemask on an existing source is changed, and the Collect Only New Files option is enabled, a confirmation message will be displayed warning the user that changing the filemask while collecting only new files could cause old files to be downloaded from the source host. This is because file collection timestamps are stored by filemask internally in MOVEit Automation. Changing the filemask negates existing timestamps.

• Ignore File(s) - When this option is checked, one or more file masks may be entered. (See "Files" above for exact syntax.) When MOVEit Automation finds a file that matches one of the entered masks, it will be ignored. Note that the ampersand operator ("&") has no context in this field and is therefore always treated as a literal. You may use Macros in this field.
• Special Filter(s) - When this option is checked, you may specify filters that allow MOVEit Automation to select files based on their size and date/time of last modification.

  Filters are built in a pop-up window accessed through the nearby "Edit" button. Supported size filters are expressed in your choice of bytes, KB, MB or GB. Supported date/time filters are expressed the difference between now and some other date in hours, days, months or years. A "match all/any" selection controls whether files must match all or just one of the filter criteria.
  

• "Browse" Button - The Browse button provides an easy way to select a folder on the remote host, which will automatically fill the Folder Name field. Files on the remote host can also be selected, which will automatically fill the Filemask field. If your source simply needs to download a specific file from a specific directory on the host, the Browse button is the easiest configuration method to use. If you need a more complex mask, however, the Filemask and Folder Name fields can still be edited after being filled by the Browse button.

  NOTE: If the source is using an FTP or SSH host with the Blind Downloads option enabled, the Browse button will not be available. Blind Downloads are used when the client is not allowed to do directory listings, or directory listings will not appear correctly. The Browse button usually will not function correctly in these situations, so it is disabled.

  NOTE: browsing will not work if the username or password fields contain references to certain types of macros, such as non-global task parameters.

• Collect Only New Files - When checked, picks only those files that are "new" off the remote server. For filesystem, FTP, and SSH hosts, MOVEit Automation maintains a database of most recent timestamps by host, directory, and filemask. See Editing Source Timestamps for how to edit this database. MOVEit Automation relies upon the MOVEit Transfer definition of newness only until the first new file is downloaded for any given folder. After that, MOVEit Automation uses the same new file definition used by the other source types.For filesystem files, MOVEit Automation uses the Last Modified stamp when determining newness.

  Note: State information for the "Collect Only New Files" option is saved on a per-Task basis, not on a per-Host bais. If the task has run previously and sent files, the files will be resent initially after turning on the option, if they still exist on the source.

• After Successful Transfer - Indicates what MOVEit Automation should do with an original file if all process and destination steps are completed without problems. The selected action (if any) will be performed on a per-file basis as soon as MOVEit Automation has delivered it or processed it through all steps successfully; there is no "mass delete" or "mass rename" step performed against source files. The options are:
  • Do Nothing - this is the default.
  • Delete original(s) - Delete the original file from the source. (This is an effective way to prevent "double-posts".)
  • Rename original(s) to: - Renames the original files to the given name. The name you specify may contain macros, such as [OrigName]. Some hosts (especially FTP hosts) will permit you to move source files across folders using this option and properly formatted paths.

  See POP3 Sources for the special case of deleting POP3 messages.

• Include Subdirectories - When checked, MOVEit Automation will search subdirectories of the given folder for files matching the given file mask, in addition to the given folder itself.
• Expand compressed (zip) file(s) - When checked, tells MOVEit Automation to attempt to uncompress any files ending in a .zip extension.
• Retry If No Files Found - If checked, times out and retries a directory listing if the previous listing was successful but resulted in no matching files. The usual retry count and retry timeout are used. This feature can be used to "wait" for a file to appear during the running of a task. If the maximum number of retries is reached and still no matching files are found, the task continues with the next step with no error flagged.

  If unchecked, a directory listing returning no files is not retried, and as a result, no files are downloaded from this source.

• Sign On With Default Username - If checked, this task will use the usual username and password for this host. If unchecked, operators may specify alternate credentials to use for this transfer.
• Use Default Connect Timeout - Gives the option to override the Connect Timeout setting for the specified host and set a different value. If this option is not checked, MOVEit Automation will use the value entered in the Connect Timeout field. The Connect Timeout setting specifies how many seconds MOVEit Automation will wait when attempting to connect to the host.
• Use Default Data Timeout - Gives the option to override the Data Timeout setting for the specified host and set a different value. If this option is not checked, MOVEit Automation will use the value entered in the Data Timeout field. The Data Timeout setting specifies how many seconds MOVEit Automation will wait when sending data to or receiving data from the host.
• Use Default Retry Count - Gives the option to override the Retry Count setting for the specified host and set a different value. If this option is not checked, MOVEit Automation will use the value entered in the Retry Count field. The Retry Count setting specifies how many times MOVEit Automation will attempt a step if it fails.
• Use Default Retry Timeout - Gives the option to override the Retry Timeout setting for the specified host and set a different value. If this option is not checked, MOVEit Automation will use the value entered in the Retry Timeout field. The Retry Timeout setting specifies how many seconds MOVEit Automation will wait between retry attempts of a step.

Specialized Source Options:

• Sign On with Default Username (MOVEit Transfer, FTP, and SSH servers only) - If checked, tells the program to sign on with the username and password configured in the host. If not checked, tells the program to use the username and password specified in the Username and Password boxes. In the case of FTP servers, this also applies to the little-used Account field. These fields may contain macro references.
• Use Default Server Transfer Type (FTP servers only) - If checked, causes MOVEit Automation to use the transfer type configured in the host. If not checked, causes MOVEit Automation to use the type (ASCII vs. Binary) selected.
• Use Default FTP Transfer Mode (FTP servers only) - If checked, causes MOVEit Automation to use the transfer mode configured in the host. If not checked, causes MOVEit Automation to use the mode (Active vs. Passive) selected. Passive is generally preferred, but not all servers support passive mode.
• Transfer - Reuse SSL Session for Data Connections (FTP SSL servers) - If checked, forces data connections to use the same SSL session as the existing control connection, which overrides the default unchecked setting on the Hosts Advanced options configurations. This allows you to comply with partner server settings that require reuse of an SSL session for data connections.
• Use Default XSHA1 Setting (FTP servers only) - If checked, causes MOVEit Automation to use the XSHA1 setting configured in the host. If not checked, causes MOVEit Automation to use the settings value selected.
• Additional commands to execute before transfer (FTP servers only) - "quote" commands to send to the server before each file is downloaded. This is in addition to the per-connection before-each-file "quote" commands specified in the host configuration. Macros may be used in this field.
• Additional commands to execute after transfer (FTP servers only) - "quote" commands to send to the server after each file is downloaded. This is in addition to the per-connection after-each-file "quote" commands specified in the host configuration. Macros may be used in this field.
• Use Default Blind Download (FTP and SSH servers only) - If checked, causes MOVEit Automation to use the "blind downloads" setting configured in the host. If not checked, causes MOVEit Automation to use the mode selected. This rarely-used setting is intended for unusual servers that have problematic implementations of commands like CWD and LIST commands.
• Sign On with Default Client Key (SSH servers only) - If checked, causes the public key defined in the host configuration to be used for authentication. (If no public key is configured, then public key authentication will not be used.) If unchecked, operators may choose an alternate public key using the "..." button. Most SSH servers require either a password or a client key; a few require both.
• Sign On with Default Client Cert (MOVEit Transfer and FTP servers only) - If checked, causes the SSL client certificate defined in the host to be used when connecting to the host. (If no client certificate is configured, then none will be used.) If unchecked, operators may choose an alternate certificate using the "..." button. See SSL Client Certificates.
• Rescan before Xfer (Filesystem, FTP and SSH servers only) - This option helps prevent MOVEit Automation from downloading "incomplete" files from servers that make such files available for download before they are closed. It specifies that once one or more files matching the download criteria have been identified, MOVEit Automation should rescan the directory, looking for changes in the files' size and date. If a file has changed, the behavior of the task will be different depending on the task type:
  • Traditional/Advanced tasks: The file will be removed from the list of files to download. The purpose of this feature is to detect when another application is currently changing the files, so that MOVEit Automation will not download a partial file. The value is the number of seconds to wait between scans.
  • Sync tasks: The task will continue rescanning until the files have have not changed in successive scans. If files are continuously changing, this option should not be used, or the task may get stuck. The purpose of this feature is to detect when the source has "settled down," so that the task does not need to run multiple times to capture and sync all of the changes in a directory.

    The value is the number of seconds to wait between scans. The default is 0, which deactivates the feature .A reasonable value for when you do need the feature might be 5 or 10 seconds. (This setting will be overriddent if specified in any source related to this host.)

• MD5 Checking (FTP and SSH servers only) - Specifies whether MOVEit Automation should look for an MD5 file, containing MD5 hashes of source files on the FTP server, and what that MD5 file should be called (see the FTP Source Integrity page for more information). The following settings are available for checking for an MD5 file:
  • Never - Central will not look for an MD5 file.
  • If Present - Central will look for an MD5 file. If the file contains a hash for a source file, Central will check the file against the hash. If the file matches the hash, Central will allow the file to continue, otherwise it will generate an error. If Central does not find a hash for the source file, it will allow the file to continue.
  • Required - Central will look for an MD5 file and generate an error if it is not found. All files downloaded will be checked against the MD5 file. If a file does not match its hash, or a hash does not exist for the file, Central will generate an error.
• Download Limits (MOVEit Transfer, FTP, SSH, and filesystem only) - Limits the number of files that Central will download from this source in a single task run. The limit can be by file count or by byte count. In the case of a limit by byte count, Central will stop downloading files after the first file that causes the number of downloaded bytes to exceed the configured limit. A value of 0 means no limit. If Use Default is checked, the defaults configured for the associated host are used; otherwise, the values specified in the source are used.

  If the Re-Run Task Automatically option is enabled, Central will automatically re-run the task if the configured limits are exceeded, allowing Central to pick up the remaining source files without waiting for the next scheduled run of the task. Note that if this option is enabled and neither the Collect Only New Files or Delete/Rename After Successful Download options are enabled, the task could potentially end up looping indefinitely. For this reason, MOVEit Automation Admin will display a warning message if it detects this condition while a source is being edited.

• Use Default UNC Transfer Type (Share hosts only) – If checked, causes MOVEit Automation to use the Use Windows CopyFile API setting configured in the host. If not checked, causes MOVEit Automation to use the selected Use Windows CopyFile API setting. By default, MOVEit Automation uses the Windows CopyFileEx function to download files from UNC shares. Under certain conditions, the performance of this function can potentially suffer, especially when transferring large files. If bad performance is being experienced when downloading files from this UNC host, deselecting the Use Windows CopyFile API for UNC Transfers setting may help remedy the situation.

See also AS1, AS2, AS3.