Large-scale data science hinges on the availability of large volumes of data. Very often, these can only be processed at select data centres and transfers among such centres becomes a challenge. Whereas the general network bandwidth between centres is good and sufficient to support transfers on the order of Terabytes to even Petabytes, some centres impose restrictions in terms of service offering and security measures. Because of this, JSC offers various services and client programs that you can use depending on the partner site or intended direction of the transfer.
UFTP (UNICORE FTP) is a file transfer tool similar to Unix’ FTP. Its main features include high-performance file transfers from client to server (and vice versa), list directories, make/remove files or directories, sync files and data sharing. In addition, users can easily share their data even with users who do not have Unix-level access to the data. UFTP is available on JSC's data access server JUDAC.
Another option is GridFTP, which is installed on our JUDAC system (as client and server) allowing transfers to and from different target sites and data centers where the service is also available.
Please review our LIST OF ALREADY AVAILABLE DATA SETS before you start transferring large amounts of data to JSC. If it is a commonly used data set, as is frequently the case in the Machine Learning or Artificial Intelligence domains, the data may already reside at JSC.
