| Table of Contents |
|---|
There are several tools available to transfer data from external source to HPC supported resources. All except one tool is available on Sol and Hawk and can be accessed from the command line. Some of these tools can be used to download data from web servers, cloud providers and remote *nix systems
...
For more information, visit https://curl.se/
...
SCP
Secure copy protocol (SCP) is a means of securely transferring data between a local host and a remote host or between two remote hosts. It uses Secure Shell (SSH) for data transfer and uses the same mechanisms for authentication, thereby ensuring the authenticity and confidentiality of the data in transit. A client can send (upload) files to a server, optionally including their basic attributes (permissions, timestamps). Clients can also request files or directories from a server (download).
scp SCP is installed system wide and does not require a module to be loaded.
...
For more information, view the scp man page by typing man scp on the command-line or visit https://linux.die.net/man/1/scp
...
SFTP
sftpSFTP is a command-line interface client program to transfer files using the SSH File Transfer Protocol (SFTP), which runs inside the encrypted Secure Shell connection. It provides an interactive interface similar to that of traditional command-line FTP clients.
sftp SFTP is installed system wide and does not require a module to be loaded.
...
At the sftp command prompt, use linux commands to navigate the file system and the get/put command to transfer from/to the remote system. Some standard command-line sftp commands
| Command | Description |
|---|---|
| get | Copy a file from the remote host to the local computer. |
| put | Copy a file from the local computer to the remote host |
| help (or ?) | Get help on the use of SFTP commands |
| exit (or quit) | Close the connection to the remote host, and exit SFTP |
rsync
rsync is installed system wide and does not require a module to be loaded.
Wget
GNU Wget or Wget is a free software package for retrieving files using HTTP, HTTPS, FTP and FTPS, the most widely used Internet protocols. It is a non-interactive command line tool, so it may easily be called from scripts, cron jobs, terminals without X-Windows support, etc.
Wget is installed system wide and does not require a module to be loaded.
| Code Block | ||||
|---|---|---|---|---|
| ||||
wget www.example.com |
For more information, visit https://www.gnu.org/software/wget/
aria
axel
Rclone
Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. More details available on the Rclone page.
Globus
Globus is a third party web based tool to manage transfer of data between two gridftp endpoints. Sol and Hawk do not have a gridftp endpoint so you cannot transfer data from external sources directly to your home directory. Instead, you need to first transfer data either to your local system or to Lehigh's Data Transfer Node (DTN) and then transfer to Sol using either scp, sftp or rsync as described above. More details are available on the Globus page.
...