Advanced Topics - Custom Directory Parsing

MOVEit Central provides a high degree of flexibility for downloading from unusual FTP and SSH servers.

When downloading from an FTP server, an automated FTP client must request a directory listing and "parse" the results in order to determine the names, date stamps, sizes, and other information about the files on the server. A problem faced by any automated FTP client is that there is no official standard format for this information. Each FTP server vendor chooses its own format for this information. As a result, there are scores of different and incompatible formats. FTP clients must make sense of these many different types of directory listings. SSH clients face a similar situation, although the problem is not as severe.

MOVEit Central provides a great deal of flexibility in dealing with this unfortunate situation by providing several different options:

• Automatic recognition of common servers. By default, MOVEit Central will automatically recognize and parse directory listings from the most common types of FTP and SSH servers, including Windows, UNIX, Novell, MVS, Unisys, and Van Dyke.
• "Blind" downloads. If the name of the file is known in advance, you can instruct MOVEit Central to download the file without requesting a directory listing, bypassing the problem altogether.
• Column-based custom parsing. If the directory listing has a simple tabular format that can be described by rows and columns, you can specify these parameters in the host definition.
• Directory parsing script. For the ultimate in flexibility, you can write a script in VBScript to parse a directory listing. Literally any format can be handled this way.

The options below are configured in the host's Advanced Options dialog, under Custom Parsing.

Column-based Custom Parsing

• Filename start column. This is the column number (with 1 being the first column) in which the filename starts. The filename ends at the first space, or end-of-line.
• Date start column. If non-zero, this specifies the column at which the file's date stamp starts. The date/time format must match one of the formats described in Date and Time Formats below. It is permissible to leave this field zero, but then the "Only New Files" option will not be available.
• Skip Lines Top. Specifies the number of lines of header information that should be skipped at the beginning of the listing. Typically this is 0.
• Skip Lines Bottom. Specifies the number of lines of trailer information that should be skipped at the end of the listing. Typically this is 0.

For the following fictitious 5-line FTP directory listing:

XYZZY FTP Server Directory Listing, prepared at 14:10:55

         267 2006-02-15 15:27:39 trnreport.txt
      537401 2005-11-29 21:12:47 xyz2005.rpt
*** END OF FILE LIST.

the settings would be:

Directory Parsing Script

• sDirListing = MIDirGetListing()

  Returns the entire verbatim listing from the FTP or SSH server. This typically contains lines separated by CR and LF (ASCII 13 and 10).

• MIDirAddEntry FilenameToMatch, Date, Size, bIsDir, FilenameForGet, FilenameOriginal

  Adds an entry to the FTP or SSH directory listing being parsed.

  FilenameToMatch	is the filename against which MOVEit Central should match when doing filename wildcard matches.
  Date	is a string in one of the date formats specified below. This can be "", but in that case, the "Only New Files" feature will not work.
  Size	is the size in bytes. Set this to 0 if the size is unknown.
  bIsDir	is a boolean variable that should be True if this entry is a directory rather than a file.
  FilenameForGet	is the filename to send to the server when requesting a download of the file. It is usually the same as FilenameToMatch
  FilenameOriginal	is the filename to return as the original filename, in such contexts as the [OrigName] macro. It is usually the same as FilenameToMatch.

  The last two parameters, FilenameForGet and FilenameOriginal, are for use with FTP servers running on operating systems with unusual filesystems. Normally all three filenames should be set the same.

Most other MOVEit Central MIxxx functions are not available in a directory parsing script.

To explain the differences between the three versions of the filename, consider a hypothetical FTP server that allows multiple numbered versions of a file, with the version following the filename in a directory listing. Suppose the directory listing looks like this:

MYFILE.DAT;22     45321  2006-05-06 08:11:56
MYFILE.DAT;21     44090  2006-05-05 17:20:40
README.TXT;3       8192  2005-12-30 21:38:27

In this directory listing, there are two versions of MYFILE.DAT, with version 22 being the more recent.

Ordinarily, the user will not know in advance which numeric version is desired; the user will only know that they want the most recent version, or perhaps the next-to-most-recent version, etc. Therefore, it would be unreasonable to configure a MOVEit Central source with a filemask that refers to a specific version number.

For the purposes of this FTP server, we can invent a filemask syntax in which the most recent version is referred to as MYFILE.DAT(0), the next most recent version as MYFILE.DAT(-1), etc. However, when transferring the file to the destination, we don't care about the version number at all, because most destination servers will not recognize file versions. We want to name the file simply MYFILE.DAT.

Thus, there are three versions of the name

• The name we use in the file mask in the source to select the file, e.g., MYFILE.DAT(0)
• The name that MOVEit Central must use when retrieving the file, e.g. MYFILE.DAT;22
• The name used in the destination as the Original Name, e.g., MYFILE.DAT

So, a script parsing this directory listing would do the equivalent of:

FilenameToMatch = "MYFILE.DAT(0)"
MyDate = "2006-05-06 08:11:56"
MySize = 45321
bIsDir = False
FilenameForGet = "MYFILE.DAT;22"
FilenameOriginal = "MYFILE.DAT"
MIDirAddEntry FilenameToMatch, MyDate, MySize, bIsDir, FilenameForGet, FilenameOriginal

Date and Time Formats

Both of MOVEit Central's custom directory parsing options recognize several datestamp formats. If the date format used by an FTP server does not match one of the following, you will not be able to use column-based parsing. Instead, you will have to use a script to massage the date before sending it to MIDirAddEntry.

A file date/time stamp is assumed to consist of a date, followed by one or more spaces, followed by a time. If the time is not recognized, it will be treated as midnight (00:00:00).

The date formats are:

• YYYY-MM-DD - The preferred format.
• YY-MM-DD - This uses a Y2K-like "pivot year" of 1970. Values from 0 to 70 are assumed to be 2000 through 2070. Values 71-99 are assumed to be 1971-1999.
• MM/DD/YY - Uses the same pivot year.
• MM/DD - Assumes that the date is within the last 12 months. If the date is today or prior to the current day of the year, it's assumed to be in this year, else it's assumed to be in the previous year.

The time formats are:

• hh:mm - Time in 24-hour format.
• hh:mm:ss - Time in 24-hour format.
• hh:mmAMPM - Time in 12-hour format. AMPM must be AM, am, PM, or pm.
• hh:mm:ssAMPM - Time in 12-hour format, as above

Sample Script

The following script demonstrates custom directory parsing.

' This script parses a directory listing from a Windows FTP server.
' In real life, this script would not be necessary, because 
' MOVEit Central is able to natively recognize and parse directory
' listings on Windows FTP servers.
'
' A Windows FTP server returns a directory list like:
' 05-02-06  04:22PM                  734 ModsNotes.txt
' 10-13-05  09:13AM       <DIR>          Incoming
' 123456789a123456789b123456789c123456789d123456789
Option Explicit
Sub Main()
   Dim sDirListing, aryLines, MyLine, TestForDir, MyName, j
   Dim MyYear, MyMonthDay, MyTime, MyDate, MySize, bIsDir, NFiles, NDirs
   sDirListing = MIDirGetListing()
   ' Break apart the listing into an array of lines.
   aryLines = Split(sDirListing, vbCrLf)
   NFiles = 0
   NDirs = 0
   For j = LBound(aryLines) To UBound(aryLines)
      MyLine = aryLines(j)
      ' Heuristic to ignore any unreasonably short lines.
      If Len(MyLine) > 39 Then
         ' MOVEit Central doesn't understand mm-dd-yy format,
         ' so change a date like 10-13-05 to 05-10-13.
         MyMonthDay = Mid(MyLine, 1, 5)
         MyYear = Mid(MyLine, 7, 2)
         MyTime = Mid(MyLine, 11, 7)
         MyDate = MyYear & "-" & MyMonthDay & " " & MyTime
         MyName = Mid(MyLine, 40)
         TestForDir = Mid(MyLine, 25, 5)
         If TestForDir = "<DIR>" Then
            bIsDir = True
            MySize = 0
            NDirs = NDirs + 1
         Else
            bIsDir = False
            MySize = Mid(MyLine, 19, 38-19+1)
            NFiles = NFiles + 1
         End If
         MIDirAddEntry MyName, MyDate, MySize, bIsDir, MyName, MyName
      End If
   Next
   If MIGetDebugLevel() >= 50 Then
      MILogMsg "Found " & NFiles & " files and " & NDirs & " dirs"
   End If
End Sub

Main