Integrated File Transfer still required for Automation

This post is part of the "Automation-Orchestration" architecture series. Posts of this series together comprise a whitepaper on Automation and Orchestration for Innovative IT-aaS Architectures.

There is a wide range of sophistication when it comes to the IT systems that businesses operate. Some came on line around 2000, but others have been in use for much longer. Some have constructed systems in order to maintain high quality services for years to come. Still others are constantly adapting their systems to take advantage of the latest technology.

Because of this wide disparity, an automation solution must be able to handle current and future innovations in integration, orchestration and performance. It must also be backwards compatible so it can support legacy technologies.

Saying this, one of the technologies that an automation solution must support is file transfer between systems. Along with this, it must also support elaboration, interpretation, and transformation of file content to create new levels of automation integration for enterprise IT.

Experiences with multiple customers show that replacing legacy file transfer applications with state-of-the-art APIs is sometimes simply too time consuming and costly. However, it is crucial that these legacy system capabilities are provided for an automated and integrated IT landscape. Strangely enough, therefore, being able to address, process, interpret, and transfer files with the demands and challenges of an automated IT environment is still a must-have criteria for an enterprise automation solution.

Support of multiple different file transfer protocols

FTP (file transfer protocols – see a list here[1]) not equals FTP: Following are the most common FTPs still in use which must be supported by the selected automation solution:

• FTP: This is the standard protocol definition for transferring files in an insecure manner. When operating behind a firewall, using FTP for transporting files is convenient and needs to be an integrated feature of your enterprise automation solution.
• FTPS: adds support for “Transport Layer Security” (TLS) and the “Secure Socket Layer” (SSL) encryption protocols to FTP. Many enterprises rely on this type of protocol for security reasons, especially when it comes to moving beyond the network.
• FTPES: This differs from FTPS only in terms of the timing of the encryption and transferring of login information. It adds an additional safety control to FTPS-based file transfers
• SFTP: has been added to the Secure Shell protocol (SSH) by the Internet Engineering Task Force (IETF)[2] in order to allow for access, transfer and management of files through any reliable (SSH) data stream.

In addition to supporting all of the above protocols, an automation solution can enhance FT integration in automation scenarios by offering a direct endpoint-to-endpoint file transfer – based on a proprietary protocol. Providing this protocol eases the need for a central management engine implementation solely to transport files from one system to another.

Standard FT protocols

The most convenient way to allow connecting FTP capable remote systems based on the protocols listed above is through a graphical UI that allows defining the transfer much the way it is done with standard FTP clients. The actual transfer itself is normally executed by means of a dedicated adapter only initiated by the centrally managed and executed automation workflows. To comply with security requirements limiting login information to only top-level administrators, sensitive information such as username, password, or certificates are stored in separate objects. At the same time, file transfers are integrated into automation flows by specialists who do not have access to the detailed login information but can still make use of the prepared security objects.

Endpoint-to-endpoint File Transfer

In addition to supporting the standard FTP protocols, the automation solution’s ecosystem should offer a direct secure file transfer between two endpoints within the IT landscape.

In this case the automation solution issues the establishment of a direct, encrypted connection between the affected endpoints – normally using a proprietary internal protocol. This type of mechanism eliminates the need for additional tools and increases the performance of file transfers significantly.

Other features the solution should support include:

• Multiple code translation (e.g. from ASCII to EBCDIC)
• Data compression
• Wildcard transfer
• Regular checkpoint log (in order to re-issue aborted transfers from the last checkpoint recorded)
• Checksum verification based on hashing algorithms (like e.g. MD5)
• Industry standard transfer encryption (e.g. AES-128, AES-192 or AES-256)

Process integration

Finally, the key to offering enterprise ready integrated file transfer through any protocol is to allow seamless integration into existing automation workflows while leveraging all the automation functionality without additional re-coding or re-interpretation of transferred files. This includes:

• Using file transfer results and file content in other automation objects.
• Including file transfer invocation, execution, and result processing in the scripting environment.
• Using files within pre or post conditions of action or workflow execution or augmenting pre/post conditions by making use of results from transferred files.
• Bundling file transfers to groups of endpoints executing similar – but not necessarily identical – file transfer processes.

This allows the integration of legacy file transfers into innovative business processes without losing transparency and flexibility.