WO2012005609A1

WO2012005609A1 - File transfer system

Info

Publication number: WO2012005609A1
Application number: PCT/NZ2011/000126
Authority: WO
Inventors: Valentine Boiarkine
Original assignee: Blade Limited
Priority date: 2010-07-08
Filing date: 2011-07-05
Publication date: 2012-01-12

Abstract

A transfer process suitable for large files (> 100Mb to many Gb) from a sender's computer to a recipient's computer over a network, involves reading the file and storing chunks of the file in a memory buffer, then transfer agent on the client computer spawns four chunk transfer processes simultaneously, each chunk starting with consecutive positions on transferred the file. The default chunk size being 1.5MB of file data. If a chunk transfer succeeds on the first try, the chunk size used will increase by a predetermined value. Upon further consecutive successes the chunk size used will continue to grow by the predetermined value until the maximum chunk size is reached typically 5MB. If a chunk transfer fails more times than a predetermined value, the next chunk size used is reduced by a predetermined value until a minimum chunk size (preferably 0.1 MB) is reached.

Description

FILE TRANSFER SYSTEM

Field of the Invention

The invention relates to a file transfer system. More particularly it relates to a file transfer system for transferring files over a network. Background of Invention

File transfer systems allow files of various sizes to be transferred from one location to another. These systems usually involve transferring files electronically over a network to a server utilizing file transfer protocols. A common prior art method of transferring files is utilizing an internet, WAN or LAN server where files are uploaded to a server by a sender and the uploaded files are downloaded by recipients when required. Dedicated software such as internet download manager software and most internet browsers allow files to be uploaded to and downloaded from servers in this manner. The reliability and speed of the download/upload of files change depending on the network and server type/settings and also the transfer conditions. Hence various methods have been used in the prior art to transfer files with increasing speed and reliability.

One such is shown in US patent specification 6061733 in which the client may choose to download a file in portions of a size chosen by the client. Another is the DIME (Direct Internet Message Encapasulation) protocol for SOAP.

In order to improve the speed and reliability of a file transfer, it has been known to segment a file into smaller chunks and transfer the chunks sequentially or simultaneously over multiple download/upload streams. This would ensure that the maximum bandwidth of the connection is used and also allows for the resumption of an interrupted download from the point that the transfer was interrupted. There are configurable limitations in this method since most servers

oc define a maximum number of streams that can be uploaded/downloaded at a given time. Furthermore the chunk size is usually predetermined by software or by a user and does not allow for any flexibility in chunk size if the transfer conditions become altered or failed during a transfer of a large file. Large files present particular problems because the file transfer time is often so long that transfer conditions will alter during the transfer, and any initial setup conditions may not be optimum at the later stages of the transfer.

Another problem with some existing file transfer methods is that the transfer may fail or data becomes corrupt after the transfer. Some common causes of transfer failure and corruption include packet loss, high latency network connectivity, low speed network connectivity, high network congestion, network interruption including cable pull, loss of intermediate device such as switch or router, loss of firewall, network swap e.g.: start upload over wireless, swap to Ethernet network/IP scheme 1, swap to Ethernet network/IP scheme 2, etc, user defined pause and resume at any time during the transfer or graceful restart or instant power loss of server/client computers. It has been difficult to overcome failure and file corruption caused by these factors using prior art file transfer methods

A further a problem with some prior art methods is that files cannot be transferred securely to a destination. Some file transfer protocols that are used in the transfer of files such as FTP, Skype and Bit torrent protocols have no method of data encryption and the data can be obtained by third parties. Therefore these methods cause a serious threat to the privacy of the transferred files and can also cause data to become corrupt or be infected by viruses.

Object of the Invention

It is an object of the invention to provide a file transfer system that ameliorates some of the disadvantages and limitations of the known art or at least provide the public with a useful choice.

Summary of Invention

In a first aspect the invention resides in a method of transferring one or more files from a sender's computer to a recipient's computer over a network, the method comprising the steps of:

- identifying a file on the sender's computer,

loc - reading the file and storing chunks of the file in a memory buffer,

- sending one or more chunks to the recipient's computer as web service calls

wherein each chunk is attached to a web service call as a binary attachment and each web service call having a data envelope containing information on the size, position and/or global unique ID of the attached chunk,

- receiving the one or more chunks at the recipient's computer,

- opening the envelope of the web service call corresponding to each chunk, and reading the information pertaining to the size, position and/or global unique ID of each attached chunk,

and if;

- the chunk(s) have been successfully received by the recipient's computer,

- sending a message from the recipient's computer to the sender's computer that the transfer of that chunk has been successful and,

- incrementing the size of the next chunk(s) sent from the sender's computer, or if;

- the chunk(s) have not been successfully received after one or more transfer attempts and/or only a part of the chunk has been received by the recipient's computer,

- sending a message to the sender's computer on the size of the chunk part received, wherein the sender's computer will resend the chunk starting from the point which sending stopped on the previous attempt,

- decrementing the size of the next chunk(s) sent from the sender's computer, until a chunk is successfully sent to the recipient's computer.

Preferably the network is the internet, WAN or LAN.

Preferably the web service calls are XML web service calls that are sent over HTTPS protocol.

Preferably the sender's computer is a client computer and the receiver's computer is a server computer in an upload transfer and the sender's computer is a server computer and the receiver's computer is a client computer in a download transfer.

loc More preferably the client computer runs a 'client transfer agent' and 'client launcher software' and the server computer runs a 'server transfer service' and optional 'server monitor software'. Preferably two or more chunks are sent to recipient's computer simultaneously.

More preferably four sequential chunks are sent to the recipient's computer concurrently.

Preferably if a transfer of a chunk(s) is interrupted or fails due to transfer failure causes defined before, the resending of the chunk(s) is initiated automatically.

Preferably the chunks have size and CRC checks embedded.

Preferably the chunks change size in accordance with transfer conditions.

Preferably the incrementing of chunk sizes sent from the sender's computer occurs until a predefined maximum chunk size is reached wherein the size of chunks are not incremented when this limit is reached. Preferably the decrementing of chunk sizes sent from the sender's computer occurs until a successful transfer of a chunk is achieved or until a predefined minimum chunk size is reached wherein the size of chunks are not decremented when this limit is reached.

Preferably after the transfer of a file(s), the initial file(s) on the sender's computer and the transferred file(s) on the recipient's computer are binary identical where the transfer method allows for zero file corruption tolerance.

In another aspect the invention resides in a file transfer system for transferring at least one file between a client computer(s) and a server computer(s) over a network, the file transfer system comprising:

- a client transfer agent running on the client computer and,

- a server transfer service running on the server computer - wherein the client transfer agent sends messages to the server transfer service, the messages being either file upload requests or file download requests

wherein, if a file upload request is sent from the client computer to the server computer

- the client transfer agent separates a file into one or more chunks and transfers the one or more chunks of the file to the server computer as web service calls,

wherein each chunk is attached to a web service call as a binary attachment and each web service call also having information on the size, position and/or global unique ID of the attached chunk and,

- the server transfer service receives the web service calls from the client transfer agent, retrieves the chunks of each web service call and assembles the chunks to form an identical copy of the file(s) at the server computer using information on the size, position and/or global unique ID of each chunk,

and if a file download request is sent from the client computer to the server computer

- the server transfer service separates the file into one or more chunks and transfers the one or more chunks of the file to the client computer as web service calls, wherein each chunk is attached to a web service call as a binary attachment and each web service call also having information on the size, position and/or global unique ID of the attached chunk and,

- the client transfer agent receives the web service calls from the server transfer service, retrieves the chunks of each web service call and assembles the chunks to form an identical copy of the file(s) at the client computer using information on the size, position and/or global unique ID of each chunk,

wherein after the transfer of each web service call, if the attached chunk of the web service call has been successfully transferred,

- indicating on the client computer that the transfer of the chunk has been successful and,

- incrementing the size of the next chunk(s) sent,

or if the attached chunk of the web service call has not been successfully transferred after one or more transfer attempts and/or only a part of the chunk has been received - sending a message to the server computer for the chunk to be transferred again starting from the point which transfer stopped on the previous attempt,

- decrementing the size of the next chunk(s) sent until a chunk is successfully transferred. Preferably the server transfer service is an XML web service of the server which is accessible using HTTPS over the internet, LAN or WAN. Preferably the web service calls are XML web service calls.

Preferably the client computer includes client launcher software in the form of a graphical user interface, a windows service, a continuous replication service and/or a command line utility for initiating a file transfer from the client transfer agent and/or displaying, logging or relaying messages from the client transfer agent to a user.

Preferably the server computer includes server monitoring software for providing services such as email notification, logging progress and other transfer metadata to a database, monitoring scheduled transfers and/or reporting on activity and usage.

This invention may also broadly be said to consist in the parts, elements and features referred to or indicated in the specification of the application, individually or collectively, and any or all collectively of any two or more of the parts, elements or features, and where specific integers are mentioned herein which have known equivalents such equivalents are deemed to be incorporated herein as if individually set forth.

Brief Description

The invention will now be described, by way of example only, by reference to the accompanying drawings:

Figure 1 is a block diagram showing an initialisation phase of a file upload in accordance with a first preferred embodiment of the invention.

Figure la-c are screenshots of a graphic user interface showing the initialisation of a file transfer.

Figure 2 is a block diagram showing a transfer phase of a file upload.

Figure 2a is a screenshot of a graphic user interface showing the progress of a file transfer. Figure 3 is a block diagram showing a completion phase of a file upload. Figure 3a is a screenshot of a graphic user interface showing the completion of a file transfer.

Figure 4 is a block diagram showing an initialisation phase of a file download.

Figure 5 is a block diagram showing a transfer phase of a file download.

Figure 6 is a block diagram showing a completion phase of a file download.

Figure 7 is a block diagram showing the transfer phase of a file upload illustrating the adaptive chunk size feature of the invention.

Description of Drawings

The following description will describe the invention in relation to preferred embodiments of the invention, namely a file transfer system. The invention is in no way limited to these preferred embodiments as they are purely to exemplify the invention only and that possible variations and modifications would be readily apparent without departing from the scope of the invention.

The transfer system includes four main software components which are the client launcher software 101, the client transfer agent 103, the server transfer service 105 and the server monitoring software 107 as shown in figures 1-6. While only the client transfer agent and the server transfer service are core elements of this invention, the client launcher and server monitor components are expected to be present in some form when the file transfer system is in use. All four components are described in more detail below.

• Client Launcher Software: The purpose of this software is to call the client transfer agent and instruct it to transfer files. This software will also display, log, or otherwise relay messages from the transfer agent to the end user. Some examples of suitable launcher software are (a) a graphical user interface (GUI) giving users the ability to upload and download files to a server (figure la-c) (b) a Windows service that replicates files between a local data store and the server (c) a command line utility that reconciles a local data store with a directory on the server (d) a continuous replication service that continuously transfers changes made to a log file to the server, to keep the contents of the log file in near real-time synchronization.

• Client Transfer Agent: The client transfer agent controls the transfer progress. It is responsible for sending and receiving messages to the server transfer service and controlling the logic around fault tolerance and resume. The client transfer agent communicates with the client launcher software via event messages. The client transfer agent communicates with the server transfer service via XML web service calls.

• Server Transfer Service: The server transfer service is an XML web service running on the server computer and accessible using HTTPS over the internet, LAN or WAN. The server transfer service is responsible for receiving data from the client and writing it to the file on the server during upload. During download, the server sends chunks of data to the client transfer agent as specified by the client transfer agent. The server saves files and retrieves files to and from the designated data store directory located locally on the server hosting server transfer service or on a UNC share. The server transfer service communicates with the server monitoring software via event messages.

• Server Monitoring Software: This is the software providing additional functionality for the server. This functionality may include, but is not limited to, e-mail notification, logging progress and other transfer metadata to a database, monitoring scheduled transfers, reporting on activity and usage.

The file transfers (download and upload) are performed in three phases. The three phases are an initialisation phase, transfer phase and a completion phase. Each phase exchanges one or more messages via XML web service calls between the client computer and the server as explained in more detail later. Each message carries envelope data and an optional binary attachment/payload for transfer of chunks of files.

Figure 1 shows a flowchart of the initialisation phase of a file upload transfer. The initialization phase starts when client launcher software 101 (e.g.: a GUI as shown in figure la-c) signals to the client transfer agent 103 that a new transfer should be performed 109. The client launcher software requires the following information to initiate an upload transfer:

• URL of the transfer web service on the server computer (as shown at 130 in figure la)

• Credentials to authenticate to the web service (username, password, domain) or instruction to perform anonymous authentication (as shown at 131 in figure la) • Full path of each file, located locally or on a network share, to be transferred (as shown in figure lb)

• Optional information such as folder on the server the file should be placed in, notification settings, etc (as shown at 133 in figure lc).

Once the client launcher software initiates the start of transfer to the client transfer agent 109, the transfer agent running on the client computer creates an initialization request message 111 as shown in figure 1. This XML message is populated with (1) indication whether this transfer is an upload or a download (2) name of file to be transferred (3) expected size of file (4) whether this is an overwrite file request or a continuous replication request.

The server transfer service 105 authenticates the client computer and either accepts or rejects the connection. If the server accepts the connection, a new user impersonation context will be created, impersonating the user entity based on the authentication credentials that the client computer provided at transfer initiation. This is essential as when the files are created or accessed, this user context is used, thus the correct files are shown to the user based on permissions given to the user and when files are created they are owned by this user. Finally, any activity performed by a user is audited in accordance with Microsoft Windows auditing methodology.

When the server 105 receives the initialization request message 111, it creates a unique transfer ID at 113 to allow logging and tracking, allocates space 114 in either a temporary file (if this is an overwrite file request), or an existing file, if this is a continuous replication request. Furthermore the server monitor 107 logs all transfer details such as file name, time, file size, etc 115.

The server 105 will send back to the transfer agent 103 at 116 an 'Initialization Response' XML response received at 117 and containing: (1) Error information (if any); (2) the unique ID for this transfer - this unique ID is the name of the temporary file or name of the server file being continuously replicated by continuous replication; (3) the maximum chunk size supported by this server. Examples of a Transfer ID are "d40dfd3e-8b5c-4528-af9e- dl6596729693" - randomly allocated globally unique ID (GUID) for a new transfer - this is the name of the temporary file the server holds while receiving the file to prevent an incomplete file being visible on the server. For a continuous replication transfer, an example of a Transfer ID would be "Replicating_File.log" - the name of the file being continuously replicated. The maximum chunk size information allows the server 105 to dictate the maximum size of a chunk it can receive. As chunk data is buffered in server memory, servers that expect many concurrent users can limit the maximum chunk size for performance reasons. The default maximum chunk size on the server is unlimited i.e.: accepts any chunk size the client computer may send.

The transfer agent 103 running on the client computer signals to the client launcher software at 118 that the transfer has been initialized 119, or was unable to be initialized because of an error e.g.: "the server was out of disk space", "the server certificate was un-trusted", etc. Normally the initialization succeeds at 118 the transfer agent persists at 120 and moves on to the transfer phase at 121. If the transfer agent 103 does not receive a response from the server 105 preferably within a 90 second timeout period, or is unable to find the server e.g.: network error, the transfer agent queues the transfer, and will continue re-trying to initialize the transfer preferably every 90 seconds, until cancelled by client launcher software 101 or initialization is successful.

Should the client launching process at 119 fail with a transient error at 122 the transfer may be queued at 124 for later initialization and resumption.

When the upload transfer has initialized successfully, the transfer phase begins (figure 2). During transfer phase the actual file data is transferred (uploaded).

The transfer agent 103 (figure 2) on the client computer spawns preferably four chunk transfer processes 201 simultaneously, each chunk starting with consecutive positions on transferred the file. However any other number of chunk transfer processes can be spawned depending on the server limitations. These positions will be determined by the starting chunk size configured on the client computer, with the default chunk size being 1.0MB (1024KB) of file data. If a chunk transfer succeeds on the first try, the chunk size used will increase by a predetermined value (e.g.: 0.5MB). Upon further consecutive successes the chunk size used will continue to grow by that value (e.g.: 1.5MB -> 2MB -> 2.5MB and so on) until the maximum chunk size is reached (preferably 5MB). If a chunk transfer fails more times than a predetermined value (preferably 4 times), the next chunk size used is reduced by a predetermined value (e.g.: 0.5MB or less) until a minimum chunk size (preferably 0.1MB) is reached.

The default chunk sizes, chunk growth and chunk shrinkage values, minimum and maximum chunk sizes and all timeout values are configurable and can be dictated by the client launcher software 101. It is expected that these values will change continuously, as network connectivity is expected to improve in the future. They can also be tuned for a specific situation e.g.: satellite, slow WAN, congested WAN, etc.

Each chunk transfer process 201 first reads the current chunk size from the launcher at 202 then asynchronously sends an upload data request message at 203, as shown in figure 2. At the server an upload data request message is received at 205, the data written to a temporary file at 206 and an upload data response message sent at 207.. At the transfer agent an incoming response is continually monitored for at 208 and if received is passed to a response type sensor 209. If no response is received a check is made at 210 for an upload timeout (typically at 8 minutes) and if this time is exceeded the chunk is resent. If a fatal error is received at 209 then the process is aborted at 212 and signalled to the client launcher at 215. Where the received response indicates at 211 that the end of a chunk has been reached the file transferred position for that file chunk is updated at 214, a signal sent to the launcher process and another chunk is started.

The XML message 203 envelope contains metadata i.e.: (1) Transfer ID (2) chunk start position; and has a binary attachment which is the actual chunk data. The binary attachment infrastructure carries an expected length and a cyclic redundancy check (CRC) value of the chunk. If the server does not receive the entire chunk, or the chunk received by the server does not match the CRC value, the server will respond with an error, causing the client to resubmit the chunk again. The server, when it receives the message 203, checks the chunk for integrity 205. If no integrity errors are found, the server writes the binary data contained in the attachment to the position specified by the envelope message, in the file dictated by the Transfer ID contained in the envelope at 206. Furthermore the server monitor software 107 logs all activity 217 as in the initialisation phase.

If the server was successfully able to write the binary data contained in the attachment to the correct position, the server responds with an upload data response message 207 containing no error details i.e.: acknowledgement of success. If the server encounters an error, the error is classified as fatal (e.g.: access denied) or transient (e.g.: the chunk CRC did not compute meaning the chunk was corrupt on reception). The error details are returned to the client in the upload data response message 207.

The client transfer agent 103 monitors for a response at 208, looping through timeout monitor 210 until a response is received or the transfer times out, when a resend of the same chunk is instigated.. If it receives a response with a fatal error at 209 the transfer process is aborted at 212 and signalled to the client launcher at 215. The launcher software will show these details on the screen (as shown in Figure 2a) or log them to a log file, or perform some other action. A transient error, or receipt of a network error at 211 also causes the transfer agent to reinstigate transfer of the same chunk. The transfer agent re-submits the same chunk, with the same metadata again. The transfer agent will continue re-trying for a number of times (preferably 20), and then wait a specified amount of time (preferably 20 seconds) until continuing to retry. The delay is added to ensure the re-try process is not tying up too many resources. The client computer will continue re-trying until the client launcher software instructs the transfer agent to stop, or a successful transfer is obtained.

If the client transfer agent 103 receives a response containing a fatal error 209, the transfer agent will signal the client launcher software 215 that there was a fatal error, and pass the error details to be displayed on screen, logged to event log, etc. The client transfer agent will then abort the transfer and exit.

If the transfer agent receives a response message with an end of file notification at 214 the completion of that chunk is notified to the signal process at 215 and the process stops. Other response messages signify the receipt of a message with no errors and the chunk position is updated at 216 and returns to start the transfer of another chunk. Information stored relating to a chunk transfer must be sufficient for the transfer to resume from this point at which it was persisted without requiring any further information. The information persisted include transfer ID, upload or download, full path to folder, file name, server URL, size of file and current position. The transfer agent signals the progress to the client launcher software 215 to display on a progress bar 135 (figure 2a), log to a log file or performs some other action with this information.

If the last file in the sequence is reached, the completion phase will begin. If not, the transfer agent will move onto the next file. If the end of the file is not reached, the transfer agent will spawn additional chunk transfer processes 201 to maintain multiple concurrently running processes (preferably 4).

At any point during the upload transfer the client software can cause the transfer agent to interrupt itself i.e.: pause, and then resume. When paused, the transfer agent will abandon any chunks being currently transferred, and exit. When resumed, the transfer agent will read progress data from the last persistence point (as persisted on disk) and resume transfer (starting from the current point persisted). This allows the transfer to be resumed even after the client computer loses power. Once the upload transfer phase is complete (i.e. the last chunk of the file was successfully transferred and acknowledged by the server), the client will finalize the transfer in the upload completion phase as shown in figure 3.

In the completion phase, the client computer sends a completion request message 301 (without any attachments) to the server. The XML completion request message contains information such as transfer ID, file name and destination folder.

Upon receipt of the completion request message, the server 105 moves the temporary file to its final destination 303 (i.e.: appropriate folder, and renames it to the correct name). If the transfer was a part of a continuous replication, the server does not need to move or rename the file. The server 105 then signals completion to the server monitor software 107 where the data is logged in a file 305 or database or displayed on screen and may send a notification of the completion, by email for instance. Next the server sends a completion response message at 306 which is received at 307 carrying error details (if any) to the client transfer agent. When the transfer agent receives a server error at 311 the client computer will signal completion to the client launcher software 309 to be shown on screen as shown in Figure 3 a. Receipt of a success message (no error) at 308 from the server allows the success of the transfer to be persisted to the client disk at 310 before the success is shown on screen,

In the case where the upload was not a continuous replication scenario, the transfer agent clears the persistence point data from disk, so that any new transfers of this file starts from the beginning of the file. If it is indeed a continuous replication scenario, persistence point data stays intact, so that any subsequent transfers of the file will begin from the point where the transfer ended (i.e. only changes are replicated).

Figure 4-6 show the three phases of a file download transfer using the file transfer system of this invention. Similar to the upload phases, messages are exchanged in the form of XML web service calls between the client computer and the server and each message carries envelope data and an optional binary attachment payload for transferring chunks of files.

The download initialization phase begins when the client launcher software 101 signals to the transfer agent 103 that a new transfer should be performed 401 (figure 4). The client launcher software requires the following information to initiate a download transfer:

• URL of the transfer web service on the server computer

• Credentials to authenticate to the web service (username, password, domain) or instruction to perform anonymous authentication

• Relative path of each file, located on the server - this path is relative to the server's data store directory (i.e.: the directory the server is exposing to the internet)

• Local folder or network share to which the file will be downloaded As shown in figure 4, once the client launcher initiates the start of transfer 401, the transfer agent running on the client computer creates an initialization request message 403. This XML message is populated with (1) indication whether this transfer is an upload or a download; (2) Name of file to be transferred; (3) Expected size of file; (4) whether this is an overwrite file request or a continuous replication request.

Next the server authenticates the client computer and either accepts or rejects the client connection, similar to the file upload scenario described before. If the server accepts the connection, a new user impersonation context is created, impersonating the user entity based on the authentication credentials that the client computer provided at transfer initiation. This is essential as when the files are created or accessed, this user context is used, thus the correct files are shown to the user, based on permissions given to the user and when files are created they are owned by this user. Finally, any activity performed by a user is audited in accordance with Microsoft Windows auditing methodology.

When the server 105 receives the initialization message 403, it checks for existence of the file(s) specified for download, and whether the user context has permissions to read the file 405. As mentioned before, the server monitor 107 logs all transfer details such as file name, time, file size, etc 407.

The server returns to the client transfer agent a response 409 containing: (1) Error information (if any); (2) Unique ID for this transfer - this unique ID is the name of the temporary file or name of the server file being continuously replicated by continuous replication; (3) Maximum chunk size supported by this server.

An example of a transfer ID is "d40dfd3e-8b5c-4528-af9e-d 16596729693" - randomly allocated GUID for a new transfer. In the download scenario the transfer ID is the same for continuous replication transfer and basic file transfer.

The maximum chunk size information allows the server to dictate the maximum size of the chunk it can send. As the chunk data is buffered in server memory, servers that expect many concurrent users may wish to limit the maximum chunk size for performance reasons. The default maximum chunk size on the server is unlimited i.e.: sends any chunk size length requested by the client transfer agent.

The transfer agent 103 running on the client computer will allocate space in a temporary file located in the folder specified during download initialization and signal to the client launcher software 101 that the transfer has been initialized 411, or was unable to be initialized because of an error e.g.: "the computer was out of disk space", "the server certificate was un-trusted", etc. If at 414 the transfer agent does not receive a response from the server preferably within a 90 second timeout period, or is unable to find the server (e.g. network error), the transfer agent queues the transfer at 415, and will continue re-trying to initialize the transfer preferably every 90 seconds, until cancelled by the client launcher software or initialization is successful.

When the file transfer has initialized successfully as at 410 the data is persisted to disk at 412 and, the download transfer phase begins at 413 (figure 5). During the transfer phase the actual file data is transferred (downloaded).

In the download transfer phase shown in figure 5, the transfer agent of the client computer spawns preferably 4 chunk transfer processes 501 simultaneously, each chunk starting with consecutive positions on transferred the file. However any other number of chunk transfer processes can be spawned depending on server limitations. These positions will be determined by the starting chunk size configured on the client computer, the default size being 1.0MB (1024KB) of data. If a chunk transfer succeeds on the first try, the chunk size used will increase by a predetermined value (e.g.: 0.5MB). Upon further consecutive successes the chunk size used will continue to grow by that value (e.g.: 1.5MB -> 2MB -> 2.5MB and so on) until the maximum chunk size is reached (preferably 5MB). If a chunk transfer fails more times than a predetermined value (preferably 4), the next chunk size used reduced by a predetermined value (e.g.: 0.5MB) or less until a minimum chunk size (preferably 0.1MB) is reached.

The default chunk sizes, chunk growth and chunk shrinkage values, minimum and maximum chunk sizes and all timeout values are configurable and can be dictated by the client launcher software 101. It is expected that these values will change continuously, as network connectivit is expected to improve in the future. They can also be tuned for a specific situation e.g.: satellite, slow WAN, congested WAN, etc.

Each chunk transfer process 501 asynchronously sends a download data request message 503, as shown in figure 5 and waits for a server response 507, an error (e.g. server could not be reached) or a timeout with a defined value (default 8 minutes). The message envelope contains the following metadata: (1) Transfer ID and (2) Chunk start position. As mentioned before, transfer progress and errors are logged by server monitor software 517. When the server, receives a download data request message 505, it reads a chunk from the required position in a file at 506 and stores it in memory. It then sends a download data response message 507, which has a binary attachment containing the actual chunk data. The binary attachment infrastructure carries an expected length and a cyclic redundancy check (CRC) value of the chunk. If the transfer agent does not receive the entire chunk, or the chunk received by the transfer agent does not match the CRC value, the transfer agent will re-submit the download data request 509. If no integrity errors are found, the transfer agent writes the binary data contained in the attachment to the position specified by the envelope message 505, in the file dictated by the temporary file name. If the server encounters an error, the error is classified as fatal (e.g. access denied) 509 or transient 511 (e.g. lost network connection). The error details are returned to the transfer agent in the download data response message 507.

If the transfer agent receives a transient error 511, receives a network error, the transfer agent will signal the client launcher software that a transmission error was experienced 515. The launcher software will show these details on the screen or log them to a log file, or perform some other appropriate action. The transfer agent will also send another download data request message asking for the same chunk, at the same position, with the same metadata again 503. The transfer agent continues re-trying for a number of times (preferably 20), and then wait a specified amount of time (preferably 20 seconds) until continuing to retry. The delay is added to ensure the re-try process is not tying up unnecessary resources. The transfer agent will continue re-trying until the client launcher software instructs the transfer agent to stop, or a successful transfer is obtained. If the transfer agent receives a response containing a fatal error 509, the transfer agent signals the client launcher software 515 that there was a fatal error, and passes the error details to be displayed on screen, logged to event log, etc. The transfer agent will then abort the transfer and exit at 512.

If the transfer agent writes the attachment data to file without error at 516 (i.e. success), the transfer agent persists its position on disk. Information persisted must be sufficient for the transfer to resume from this point without requiring any further information. The information persisted include transfer ID, upload or download, full path to folder, file name, server URL, size of file and current position. The transfer agent signals the progress to the client launcher software to display on a progress bar, log to a log file or to perform some other action with this information. Once the progress was persisted successfully, the chunk transfer process exits and the transfer agent checks whether the end of file is reached at 514. If so, file completion is signalled to the client launcher software 515. If the last file in the sequence is reached, the completion phase will begin. If not, the transfer of the next file will begin. If the end of the file is not reached, the transfer agent spawns additional chunk transfer processes 501 to maintain multiple concurrently running processes (preferably 4).

At any point during the download transfer the client software can cause the transfer agent to interrupt itself i.e.: pause, and then resume. When paused, the transfer agent will abandon any chunks being currently transferred, and exit. When resumed, the transfer agent will read progress data from the last persistence point (as persisted on disk) and resume transfer (starting from the current point persisted). This allows the transfer to be resumed even after the client computer loses power.

Once the download transfer phase is complete (i.e. the last chunk of the file was successfully written to the file), the transfer agent finalizes the transfer in the download completion phase. In the download scenario, the transfer agent 103 does not need to send any data to the server 105 to finalize the transfer as shown in figure 6. The transfer agent renames the temporary file 601 (e.g.: download_file.zip.tmp) to the correct name (i.e.: download.zip). If the transfer was a part of the continuous replication, the transfer agent does not need to rename the file. The transfer agent also signals completion to the client software 603 so that it is displayed on screen, logged to file, etc.

If the download was not a continuous replication scenario, the transfer agent clears the persistence point data from disk at 602, so that any new transfers of this file starts from the beginning of the file. If the download was in fact a continuous replication scenario, persistence point data stays intact, so that any subsequent transfers of this file begin from the point where this transfer ended (i.e. only changes are replicated).

Figure 7 shows a file upload transfer process as described in Figure 2 in which the chunk size file adjustment process is identified. As the file transfer proceeds the chunk size is adjusted to take advantage of good transfer conditions or compensate for poor transfer conditions as explained previously.

As in Figure 2 an upload data request message 203 is sent as an Upload Data message at 704, received at 205 and is either correctly received as at 705, when the data is written to the indicated position 707 and a "success" response sent at708 or the receive fails at 706 and a "failure" response is sent at 713.

Where a failure response is received at 210, or the time for the expected upload times out, the response is determined as fatal or otherwise at 209 on the basis of the type of failure or the number of previous failure responses. If fatal the process is aborted at 211 and an eventual restart may be attempted. Where the failure is judged non-fatal at 209 the chunk size last used is decreased at 714 as explained previously and a read of that chunk carried out at 203 for transfer.

Where the transfer was a success 708 the state of the file transfer is checked at 709 and transferred to the progress log of the launcher 215 before the chunk size is increased at 712 and the read of the next chunk initiated 203. Where the transfer is finished the client launcher software 101 is advised 715 and the transfer terminates.

The size of the chunk step increase or decrease will depend on the number of failures and the data rate, their being a calculable maximum achievable transfer rate based on the number and size of failed uploads.

The chunk size adjustment process for a file download transfer (not illustrated) is similar to the download transfer phase shown in figure 5 where the chunk size is incremented upon a successful transfer 513 and the chunk size is decremented on subsequent attempts 509 upon a transfer failure.

a) The file transfer system disclosed in this specification has the ability to change the chunk size of a transferred file depending on the transfer conditions during a transfer. As explained previously, the chunk size increases with each successful chunk transfer until chunk size reaches a predefined maximum size and decreases each time the chunk transfer fails until a minimum chunk size is reached. This allows the file transfer system to adapt efficiently to the transfer conditions so that bigger chunks are transferred in a strong server connection that does not often fail and smaller chunks are transferred if the server connection often fails. Further, when conditions change over the time of an extended transfer, the chunk size will adapt to the changed conditions by either increasing or decreasing the chunk size transferred.

b) The file transfer system of this specification is immune to transfer failure and corruption caused by a number of problems that can occur during file transfer. More specifically, no progress is lost and transfer recovers and continues automatically in the case of packet loss, high latency network connectivity, low speed network connectivity, high network congestion, network interruption including cable pull, loss of intermediate device such as switch or router, loss of firewall, network swap e.g.: start upload over wireless, swap to Ethernet network/IP scheme 1, swap to Ethernet network/IP scheme 2, etc, user defined pause and resume at any time during the transfer or graceful restart or instant power loss of server/client computers. In the case of power loss or restart of the client computer, the transfer resumes automatically or manually depending on the settings of the client launcher software. Furthermore CRC checks are performed on transferred files to ensure that the files are identical copies of the original files and are free from corruption as explained previously.

. c) All messages are exchanged using the Hypertext Transfer Protocol Secure (HTTPS) in this invention. HTTPS is a file transfer protocol which is widely accepted as the most secure and accessible protocol currently available in the market. The use of HTTPS also provides the industry standard Secure Socket Layer (SSL) data encryption. Due to its high level of security SSL is also used for online banking services, online credit card payments and government data encryption. Therefore the privacy of the transferred files is ensured and the risk of access by third parties is minimised.

d) The file transfer system has the ability to transfer unlimited amount of data and the software does not have any limitations in terms of maximum transfer size. In testing the system, individual files of up to 100 GB in size have been transferred. The data transfer limit may only be restricted by the limitations of the client and server computers and the server connection.

Variations

Even though the file transfer system of this invention uses the HTTPS protocol to transfer files to a server using XML web service calls, other protocols and other forms of digital messaging can be used. It is expected that file transfer protocols and messaging will become more efficient and increasingly secure in future. Therefore the system can be adapted for use with those new or already existing technologies currently under development.

All file sizes, minimum and maximum chunk sizes and timeout values are configurable and can be dictated by the client launcher software as mentioned previously. The values disclosed in this specification are for example only and can be varied depending on the individual network and conditions. It is expected that these values will change continuously, as network connectivity is expected to improve in the future. They can also be tuned for a specific situation e.g.: satellite, slow WAN, congested WAN, etc. Throughout the description of this specification, the word "comprise" and variations of that word such as "comprising" and "comprises", are not intended to exclude other additives, components, integers or steps. It will of course be realised that while the foregoing has been given by way of illustrative example of this invention, all such and other modifications and variations thereto as would be apparent to persons skilled in the art are deemed to fall within the broad scope and ambit of this invention as is hereinbefore described.

Claims

A method of transferring one or more files from a sender's computer to a recipient's computer over a network, the method comprising the steps of:

identifying a file on the sender's computer,

reading the file and storing chunks of the file in a memory buffer, sending one or more chunks to the recipient's computer as web service calls, wherein each chunk is attached to a web service call as a binary attachment, each web service call having a data envelope containing information on the size, position and/or global unique ID of the attached chunk,

receiving the one or more chunks at the recipient's computer, opening the envelope of the web service call corresponding to each chunk, and reading the information pertaining to the size, position and/or global unique ID of each attached chunk,

and where;

the chunk(s) have been successfully received by the recipient's computer, sending a message from the recipient's computer to the sender's computer that the transfer of that chunk has been successful and,

incrementing the size of the next chunk(s) sent from the sender's computer, or where;

the chunk(s) have not been successfully received after one or more transfer attempts and/or only a part of the chunk has been received by the recipient's computer,

sending a message to the sender's computer on the size of the chunk part received,

wherein the sender's computer will resend the chunk starting from the point which sending stopped on the previous attempt,

decrementing the size of the next chunk(s) sent from the sender's computer, until a chunk is successfully sent to the recipient's computer. 2. A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 1 wherein the web service calls are XML web service calls that are sent over HTTPS protocol. A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 1 wherein two or more chunks are sent to recipient's computer simultaneously.

A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 3 wherein four sequential chunks are sent to the recipient's computer concurrently.

A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 4 wherein where a transfer of a chunk(s) is interrupted or fails due to transfer failure causes defined before, the resending of the chunk(s) is initiated automatically.

A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 1 wherein the chunks change size in accordance with transfer conditions.

A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 1 wherein the incrementing of chunk sizes sent from the sender's computer occurs until a predefined maximum chunk size is reached wherein the size of chunks are not incremented when this limit is reached.

A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 1 wherein the decrementing of chunk sizes sent from the sender's computer occurs until a successful transfer of a chunk is achieved or until a predefined minimum chunk size is reached wherein the size of chunks are not decremented when this limit is reached.

A method of transferring one or more files from a sender's computer to a recipient's computer over a network as claimed in claim 1 wherein after the transfer of a file(s), the initial file(s) on the sender's computer and the transferred file(s) on the recipient's computer are binary identical where the transfer method allows for zero file corruption tolerance.