Globus

Globus is the recommended tools for transferring large amounts of data, to and from Lehigh, via the Science DMZ.

What is Globus (formerly known as Globus Online)?

Globus is software-as-a-service for file transfer and sharing. Designed specifically for researchers, Globus provides fast, reliable, and secure file transfer among XSEDE resources or between an XSEDE resource and another machine (such as a campus cluster, lab server, or personal computer). Globus is core campus bridging technology that enables researchers to scale their computational research from the desktop, across campus, and to national cyberinfrastructure. Beyond file transfer, Globus allows researchers to securely share data with collaborators directly from existing systems, without investing in additional campus or cloud storage just for the purposes of sharing.

How do I get started with using globus to transfer data to and from my laptop/desktop/workstation at Lehigh University?

Install Globus Connect Personal (you only need to do this once)

Effective Feb. 12, 2016, you do not need to create a globus account, you can login using your Lehigh University credentials.
Install Globus Connect Personal on the computer that is a source or destination for your data transfer. If you do not have admin rights on your computer, you need to open footprints ticket requesting this software to be installed or contact your Computing Consultant.
1. https://www.globus.org/globus-connect-personal. There are version for Windows, Mac and Linux. This version is only for single user accounts. If the computer is a multiple user computer, please do not install Globus.
2. Globus is installed on the ScienceDMZ Data Transfer Node.
To transfer data, you need to create a globus endpoint on the computer where Globus Connect is installed. Login to your globus account and click on the Manage Data tab
1. Click on Manage Endpoints > Add a Globus Connect Personal link
2. Give a name to your endpoint. It should be in a format of yourusername#endpointname (the yourusername# is already set) and click on generate setup key
3. Enter this key in the pop window for the Globus Connect Personal install
4. You should now see your local system as one of the endpoint in your online account.

Transfer Data

To transfer data, click on the the Quick Links > Transfer File link on the top right corner.
1. You should see a window similar to WINSCP or FTP client transfer tools.
On one of the windows (right or left doesn’t matter which) enter your globus endpoint as yourusername#endpointname and in the other window add the endpoint of the computer that you want to transfer data to/from. For example, all xsede systems have an endpoint xsede#systemname as in xsede#stampede. Lehigh DTN is the endpoint for the ScienceDMZ
1. You should refer to the user guides in case the remote HPC system has a different authentication.
  1. For example, on xsede systems, you get redirected to xsede’s oauth login page.
  2. On some systems, you need to generate a short term certificate that you may need to authenticate against.
  3. Lehigh DTN uses Lehigh Single Sign-On and you will be directed to sso.cc.lehigh.edu to enter your Lehigh Credentials.
After you enter your password/passphrase, you should be able to navigate the file system on both endpoints. If you have used FTP/SCP/SFTP client tools, you should be familiar with the setup.
To transfer files, click on one of the large left or right triangles between the endpoint names. The base of the triangle corresponds to data transfer source and the vertex to the destination.
You will receive an email when the transfer is complete. You can log out from the online session while the transfer is taking place. DO NOT SHUTDOWN THE COMPUTER ON WHICH YOU ARE TRANSFERRING DATA TO/FROM. If you do, globus will try to restart the transfer from where it left off when the computer is back online.

FAQ from Globus Website

How is Globus Used?

Globus is used for moving data between any two resources, whether it is a small number of very large (even terabyte-sized) files or a very large number of small files. Note: A resource is represented in Globus as an endpoint, identified by a unique name (e.g., the endpoint for TACC's Stampede is "xsede#stampede", and NICS's Kraken is "xsede#kraken").

Globus can be used to move data:

Between two XSEDE resources
Between an XSEDE resource and your personal laptop or desktop
Between an XSEDE resource and another HPC system on your campus (not available at Lehigh yet)
Between any two non-XSEDE systems

Globus also provides an easy way to share data with collaborators directly from an existing storage system. There is no need to move data to a different system to public cloud service just for the purposes of sharing.

How do I get started with using Globus?

In order to use Globus you must have a Globus account, that you can create by going to https://www.globus.org/signup. After signing up you will receive an e-mail request to confirm your e-mail address; click on the link to confirm and you will then be able to access the file transfer and sharing service.

Note: You may also, optionally, link your Globus account to your XSEDE User Portal (XUP) identity – this will enable you to access the Globus service using your XSEDE user name and password.

How do I move data between XSEDE resources?

Globus can be used to move data between any two XSEDE resources. All XSEDE resources are already configured as Globus endpoints.

How do I move data between XSEDE and another HPC system?

Any multi-user HPC or shared storage system can be configured as a Globus endpoint using Globus Connect Server. On most servers, installing Globus Connect Server requires just a few commands. Once the software is installed, any user with a local account on that system can move and share files between it and an XSEDE resource. Information on Globus Connect Server is available on the Globus support web site.

How do I move data between two non-XSEDE machines?

Globus can also be used to move data between any two systems such as a campus cluster, a non-XSEDE computing facility, or a machine that captures data from an instrument – even if the machine is behind a firewall or NAT. You can easily turn any system into a Globus endpoint using Globus Connect.

How do I move data between XSEDE and a local machine?

It's easy to set up a Globus endpoint on a user's machine and upload or download files between this machine and an XSEDE resource. Globus makes it possible to easily transfer files to and from any machine – even if behind a firewall or NAT without any administrative privileges – using Globus Connect Personal.

What is Globus Connect?

Globus Connect enables your system to use the Globus file transfer and sharing service. It makes it simple to create a Globus endpoint on practically any system, from a personal laptop to a national supercomputer. Globus Connect is free to install and use for users at non-profit research and education institutions.

Which Globus Connect Versions should I use?

Globus Connect comes in two flavors: Globus Connect Personal is designed for use by a single user on a personal machine; Globus Connect Server is designed to be installed by a system administrator on multi-user computing and storage resources.

If you want to install Globus Connect on your personal or work computer (Windows, Mac or Linux) for your data transfer needs, please download and install Globus Connect Personal .

How do I share data from my Globus endpoint?

You can share files from an XSEDE resource or any other Globus endpoint that has been enabled for sharing. Your collaborators need only a Globus account to access files that you have shared with them. Globus ensures that access controls on the local system are enforced while allowing other Globus users to easily access and download the data. A Globus endpoint can be enabled for sharing if your institution has subscribed to a Globus Provider plan (for multi-user servers).

Lehigh University has a Globus subscription plan as of Sep 30, 2021. A policy for getting upgraded to Globus Plus User for sharing data will be provided soon.