Rclone is a command line program to manage files on cloud storage. It is a feature rich alternative to cloud vendors' web storage interfaces. Over 40 cloud storage products support rclone including S3 object stores, business & consumer file storage services, as well as standard transfer protocols.
Usage
Version | module name |
---|
1.53.3 | rclone/1.53.3 |
Configuration
You need to configure rclone for each storage system that you want to transfer data to/from. Please see detailed instructions at https://rclone.org/docs/. The following are instructions for connecting to Google Drive and Dropbox . This requires an access to a browser to authenticate to dropbox. Carry out the configuration steps in the CentOS 8 Desktop app on Open OnDemand or create a SSH tunnel using instructions from Jupyter Notebooks.
Expand |
---|
title | Google Drive Configuration |
---|
|
Expand |
---|
title | Obtaining a Google Client API ID (optional) |
---|
| RClone provides a client_id that is shared by all users. Considering Google limits the number of transactions per second, you may receive the following errors message. googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded At least one user encountered this error while creating a google drive remote. If you plan on transferring data back and forth between Sol and Google Drive, you might want to get your own Client ID and use it below. The following is description provided at https://rclone.org/drive/#making-your-own-client-id Log into the Google API Console with your Google account. It doesn't matter what Google account you use. (It need not be the same account as the Google Drive you want to access) Select a project or create a new project. Under "ENABLE APIS AND SERVICES" search for "Drive", and enable the "Google Drive API". Click "Credentials" in the left-side panel (not "Create credentials", which opens the wizard), then "Create credentials" If you already configured an "Oauth Consent Screen", then skip to the next step; if not, click on "CONFIGURE CONSENT SCREEN" button (near the top right corner of the right panel), then select "External" and click on "CREATE"; on the next screen, enter an "Application name" ("rclone" is OK) then click on "Save" (all other data is optional). Click again on "Credentials" on the left panel to go back to the "Credentials" screen. Click on the "+ CREATE CREDENTIALS" button at the top of the screen, then select "OAuth client ID". Choose an application type of "Desktop app" if you using a Google account or "Other" if you using a GSuite account and click "Create". (the default name is fine) It will show you a client ID and client secret. Use these values in rclone config to add a new remote or edit an existing remote.
|
Code Block |
---|
language | bash |
---|
title | Run rclone config command |
---|
| [alp514.sol](1006): rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q> |
Code Block |
---|
language | bash |
---|
title | Enter n for New remote. You will be prompted to name the remote. |
---|
| n/s/q> n
name> lugdrive |
Code Block |
---|
language | bash |
---|
title | Enter 13 at the prompt to configure Google Drive |
---|
| Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Citrix Sharefile
\ "sharefile"
9 / Dropbox
\ "dropbox"
10 / Encrypt/Decrypt a remote
\ "crypt"
11 / FTP Connection
\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
13 / Google Drive
\ "drive"
14 / Google Photos
\ "google photos"
15 / Hubic
\ "hubic"
16 / In memory object storage system.
\ "memory"
17 / Jottacloud
\ "jottacloud"
18 / Koofr
\ "koofr"
19 / Local Disk
\ "local"
20 / Mail.ru Cloud
\ "mailru"
21 / Mega
\ "mega"
22 / Microsoft Azure Blob Storage
\ "azureblob"
23 / Microsoft OneDrive
\ "onedrive"
24 / OpenDrive
\ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
26 / Pcloud
\ "pcloud"
27 / Put.io
\ "putio"
28 / QingCloud Object Storage
\ "qingstor"
29 / SSH/SFTP Connection
\ "sftp"
30 / Sugarsync
\ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
\ "tardigrade"
32 / Transparently chunk/split large files
\ "chunker"
33 / Union merges the contents of several upstream fs
\ "union"
34 / Webdav
\ "webdav"
35 / Yandex Disk
\ "yandex"
36 / http Connection
\ "http"
37 / premiumize.me
\ "premiumizeme"
38 / seafile
\ "seafile"
Storage> 13** See help for drive backend at: https://rclone.org/drive/ ** |
Code Block |
---|
language | bash |
---|
title | Hit Enter to use default options. See rclone documentation for option advanced configuration options |
---|
| Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Scope that rclone should use when requesting access from drive.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / Full access all files, excluding Application Data Folder.
\ "drive"
2 / Read-only access to file metadata and file contents.
\ "drive.readonly"
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ "drive.file"
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ "drive.appfolder"
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ "drive.metadata.readonly"
scope> 1
ID of the root folder
Leave blank normally.
Fill in to access "Computers" folders (see docs), or for rclone to use
a non root folder as its starting point.
Enter a string value. Press Enter for the default ("").
root_folder_id>
Service Account Credentials JSON file path
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a string value. Press Enter for the default ("").
service_account_file>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n>
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=<secret key>
Log in and authorize rclone for access
|
Code Block |
---|
language | bash |
---|
title | Open a web browser, login to google and grant access to rclone. |
---|
| Waiting for code...
Got code
Configure this as a team drive?
y) Yes
n) No (default)
y/n>
--------------------
[lugdrive]
type = drive
scope = drive
token = {"access_token":"<secret>","token_type":"Bearer","refresh_token":"<secret>","expiry":"2021-04-09T15:26:48.766637414-04:00"}
--------------------
|
Code Block |
---|
language | bash |
---|
title | Enter Y to accept the configuration |
---|
| y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y |
Code Block |
---|
language | bash |
---|
title | Enter q to quit configuration |
---|
| Current remotes:
Name Type
==== ====
lugdrive drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q |
|
Expand |
---|
title | Dropbox Configuration |
---|
|
Code Block |
---|
language | bash |
---|
title | Run rclone config command |
---|
| [alp514.sol](1010): rclone config
Current remotes:
Name Type
==== ====
lugdrive drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> |
Code Block |
---|
language | bash |
---|
title | Enter n for New remote. You will be prompted to name the remote. |
---|
| e/n/d/r/c/s/q> n
name> dropbox |
Code Block |
---|
language | bash |
---|
title | Enter 9 at the prompt to configure dropbox |
---|
| Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
1 / 1Fichier
\ "fichier"
2 / Alias for an existing remote
\ "alias"
3 / Amazon Drive
\ "amazon cloud drive"
4 / Amazon S3 Compliant Storage Provider (AWS, Alibaba, Ceph, Digital Ocean, Dreamhost, IBM COS, Minio, Tencent COS, etc)
\ "s3"
5 / Backblaze B2
\ "b2"
6 / Box
\ "box"
7 / Cache a remote
\ "cache"
8 / Citrix Sharefile
\ "sharefile"
9 / Dropbox
\ "dropbox"
10 / Encrypt/Decrypt a remote
\ "crypt"
11 / FTP Connection
\ "ftp"
12 / Google Cloud Storage (this is not Google Drive)
\ "google cloud storage"
13 / Google Drive
\ "drive"
14 / Google Photos
\ "google photos"
15 / Hubic
\ "hubic"
16 / In memory object storage system.
\ "memory"
17 / Jottacloud
\ "jottacloud"
18 / Koofr
\ "koofr"
19 / Local Disk
\ "local"
20 / Mail.ru Cloud
\ "mailru"
21 / Mega
\ "mega"
22 / Microsoft Azure Blob Storage
\ "azureblob"
23 / Microsoft OneDrive
\ "onedrive"
24 / OpenDrive
\ "opendrive"
25 / OpenStack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
\ "swift"
26 / Pcloud
\ "pcloud"
27 / Put.io
\ "putio"
28 / QingCloud Object Storage
\ "qingstor"
29 / SSH/SFTP Connection
\ "sftp"
30 / Sugarsync
\ "sugarsync"
31 / Tardigrade Decentralized Cloud Storage
\ "tardigrade"
32 / Transparently chunk/split large files
\ "chunker"
33 / Union merges the contents of several upstream fs
\ "union"
34 / Webdav
\ "webdav"
35 / Yandex Disk
\ "yandex"
36 / http Connection
\ "http"
37 / premiumize.me
\ "premiumizeme"
38 / seafile
\ "seafile"
Storage> 9
** See help for dropbox backend at: https://rclone.org/dropbox/ ** |
Code Block |
---|
language | bash |
---|
title | Hit Enter to use default options. See rclone documentation for option advanced configuration options |
---|
| OAuth Client Id
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_id>
OAuth Client Secret
Leave blank normally.
Enter a string value. Press Enter for the default ("").
client_secret>
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n>
Remote config
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n>
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=<secret key>
Log in and authorize rclone for access |
Code Block |
---|
language | bash |
---|
title | Open a web browser, login to dropbox and grant access to rclone |
---|
| Waiting for code...
Got code
--------------------
[dropbox]
type = dropbox
token = {"access_token":"<my secret token>","token_type":"bearer","expiry":"0001-01-01T00:00:00Z"}
-------------------- |
Code Block |
---|
language | bash |
---|
title | Enter Y to accept the configuration |
---|
| y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y |
Code Block |
---|
language | bash |
---|
title | Enter q to quit configuration |
---|
| Current remotes:
Name Type
==== ====
dropbox dropbox
lugdrive drive
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q |
|
Examples
Some commonly used command are:
Code Block |
---|
language | bash |
---|
title | Get list of configured remotes |
---|
|
rclone listremotes |
Code Block |
---|
language | bash |
---|
title | List contents of remote directory |
---|
|
rclone ls remote:directory |
Code Block |
---|
language | bash |
---|
title | create a remote directory (including intermediate directories if they do not exist) |
---|
|
rclone mkdir remote:directory/new_directory |
Code Block |
---|
language | bash |
---|
title | copy files to remote (source can also be another remote) |
---|
|
rclone copy source:path remote:directory/new_directory |
Code Block |
---|
language | bash |
---|
title | delete a remote file or directory |
---|
|
rclone delete remote:directory/filename |
Code Block |
---|
language | bash |
---|
title | delete a directory |
---|
|
rclone rmdir remote:directory/new_directory
# use rmdirs to delete empty directories under new_directory |
Code Block |
---|
language | bash |
---|
title | Is your transfer speed to dropbox around 1MB/s? |
---|
|
rclone --bwlimit 10M --dropbox-chunk-size 128M copy source:path remote:directory/new_directory |
Links
Please visit the rclone website for usage documentation.
Image Added