Rclone.org - Rclone Tips

Rclone is an open-source command-line data transfer tool that works with numerous cloud storage vendors.  Configuring it for use involves creating 'remotes' in which account names. passwords and tokens can be stored.  In general, the program works like rsync, with internal commands and the specification of which 'remotes' to apply the command to.

• The Research Computing team has some notes on Rclone also.
• The documentation provided by the vendor is good, but highly detailed -- read carefully.  There are also significant additional pages on each cloud service.
• Since this tool is capable of 'exfiltrating' a large amount of data in an automated way, it can cause our information security software (CrowdStrike) to act, and shut down the process.  You may need to work with the Information Security team to craft an exception.
• General syntax of an rclone command:
  • rclone <command> <source remote name>:<path/to/directory or file>  <destination remote name>:<path/to/directory> <options>
• First step is to use the 'rclone config' command to configure a cloud service with which to exchange files

rclone config

• The stepwise process is documented nicely for the Google Drive provider in https://rclone.org/drive/.  Note that when configuring that type of remote, you can specify a particular 'Shared Drive' that your Google account can access as the root of that remote configuration.
• Once configured, rclone can transfer files from cloud services to local storage or vice versa, or cloud service to cloud service.
• Before doing any transfer, it's highly recommended to use the listing commands 'ls' (show all files and folders) and 'lsd' (show just directories) to make sure that you have the correct syntax for the source and destination directories for any anticipated transfer.  Generally, once you're sure of the syntax for the directories you wish to transfer, one simply has to change the command, and copy the directory specification strings.

rclone ls MyGoogleDrive:
rclone lsd MyGoogleDrive:
rclone ls MyGoogleDrive:DropboxFiles/ProjectX

rclone ls "BrashlerGoogleDrive:CC Docs/Error Screenshots"
    73434 Attempting-Win11_install.JPG
   360846 CannotConnectPrinter.JPG
  1314880 Remote Desktop Settings.PNG

• Rclone can employ filters on it's commands to include or exclude files based on rules:

rclone copy remote:dir /path/to/dir --include "*.jpg"

• It's good to create a scratch text file with the text of the commands you use to make it simple to re-use them, or parts of them.
• Rclone has nice logging features (--log-file c:\users\dab406\rclonelogs\recenttransfer) to enable trapping any issues or errors with transfers.
• Rclone has a nice '--progress' switch that outputs overall statistics on it's progress, which is very nice for large transfers.
• Rclone's 'copy' command is, by default, recursive, and will look through all directories to transfer files.  It is also 'version aware' comparing files and directories in the source and destination by modification date to determine if they need to be transferred.
• For use with Google Drive, rclone has switches to point transfers at 'Shared with Me' files, and also to ignore native Google docs (so they can be left in Google).

rclone ls "BrashlerGoogleDrive:DROPBOX" --drive-shared-with-me --drive-skip-gdocs

• By default, rclone executes 'download' conversions for native Google document types to their corresponding Microsoft Office document types (Sheets become .xlsx files, etc.) when transferring out of Google Drive

rclone copy "BrashlerGoogleDrive:CC Docs/MyGoogleSheet" c:\TransferredSheet
dir
Transferredsheet\MyGoogleSheet.xlsx

• By default, rclone only creates 4 parallel processes each for checking and transferring files, but larger numbers are often useful and helpful, and can speed up your transfer.

rclone copy "BrashlerGoogleDrive:CC Docs/Error Screenshots" BrashlerDropbox:GoogleFiles --transfers=45 --checkers=45

• You can tell rclone to be unconcerned with the size of the backlog of files to transfer with the '--max-backlog=999999' option and increase the amount of memory used per checker/transfer process (--buffer-size=75M).
• You can tell rclone, when using Google Drive, to use the '--fast-list' feature which reduces API calls in exchange for using more memory.  You can also use the --drive-chunk-size option to increase the size of file chunks that are checked when moving large files (512M worked fine for me).
• By default, rclone uses it's own 'client ID' with a given cloud storage service, but to maximize bandwidth with a provider for large transfers, it's recommended to create your own client ID, to use when creating your 'remotes'.  The process is naturally unique to each cloud service.
  
  • Google Drive
  • Dropbox
• Once you've done a transfer, Rclone also has a nice comparison command ('check') to enable a file-by-file comparison of two cloud directories.