Rclone on rclone - rsync for cloud storage

Sia

Nick Craig-Wood — Wed, 02 Oct 2019 00:00:00 UTC

Sia

Sia (sia.tech) is a decentralized cloud storage platform based on the blockchain technology. With rclone you can use it like any other remote filesystem or mount Sia folders locally. The technology behind it involves a number of new concepts such as Siacoins and Wallet, Blockchain and Consensus, Renting and Hosting, and so on. If you are new to it, you'd better first familiarize yourself using their excellent support documentation.

Introduction

Before you can use rclone with Sia, you will need to have a running copy of Sia-UI or siad (the Sia daemon) locally on your computer or on local network (e.g. a NAS). Please follow the Get started guide and install one.

rclone interacts with Sia network by talking to the Sia daemon via HTTP API which is usually available on port 9980. By default you will run the daemon locally on the same computer so it's safe to leave the API password blank (the API URL will be http://127.0.0.1:9980 making external access impossible).

However, if you want to access Sia daemon running on another node, for example due to memory constraints or because you want to share single daemon between several rclone and Sia-UI instances, you'll need to make a few more provisions:

Ensure you have Sia daemon installed directly or in a docker container because Sia-UI does not support this mode natively.
Run it on externally accessible port, for example provide --api-addr :9980 and --disable-api-security arguments on the daemon command line.
Enforce API password for the siad daemon via environment variable SIA_API_PASSWORD or text file named apipassword in the daemon directory.
Set rclone backend option api_password taking it from above locations.

Notes:

If your wallet is locked, rclone cannot unlock it automatically. You should either unlock it in advance by using Sia-UI or via command line siac wallet unlock. Alternatively you can make siad unlock your wallet automatically upon startup by running it with environment variable SIA_WALLET_PASSWORD.
If siad cannot find the SIA_API_PASSWORD variable or the apipassword file in the SIA_DIR directory, it will generate a random password and store in the text file named apipassword under YOUR_HOME/.sia/ directory on Unix or C:\Users\YOUR_HOME\AppData\Local\Sia\apipassword on Windows. Remember this when you configure password in rclone.
The only way to use siad without API password is to run it on localhost with command line argument --authorize-api=false, but this is insecure and strongly discouraged.

Configuration

Here is an example of how to make a sia remote called mySia. First, run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> mySia
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
...
29 / Sia Decentralized Cloud
   \ "sia"
...
Storage> sia
Sia daemon API URL, like http://sia.daemon.host:9980.
Note that siad must run with --disable-api-security to open API port for other hosts (not recommended).
Keep default if Sia daemon runs on localhost.
Enter a string value. Press Enter for the default ("http://127.0.0.1:9980").
api_url> http://127.0.0.1:9980
Sia Daemon API Password.
Can be found in the apipassword file located in HOME/.sia/ or in the daemon directory.
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[mySia]
type = sia
api_url = http://127.0.0.1:9980
api_password = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Once configured, you can then use rclone like this:

List directories in top level of your Sia storage

rclone lsd mySia:

List all the files in your Sia storage

rclone ls mySia:

Upload a local directory to the Sia directory called backup

rclone copy /home/source mySia:backup

Standard options

Here are the Standard options specific to sia (Sia Decentralized Cloud).

--sia-api-url

Sia daemon API URL, like http://sia.daemon.host:9980.

Note that siad must run with --disable-api-security to open API port for other hosts (not recommended). Keep default if Sia daemon runs on localhost.

Properties:

Config: api_url
Env Var: RCLONE_SIA_API_URL
Type: string
Default: "http://127.0.0.1:9980"

--sia-api-password

Sia Daemon API Password.

Can be found in the apipassword file located in HOME/.sia/ or in the daemon directory.

NB Input to this must be obscured - see rclone obscure.

Properties:

Config: api_password
Env Var: RCLONE_SIA_API_PASSWORD
Type: string
Required: false

Advanced options

Here are the Advanced options specific to sia (Sia Decentralized Cloud).

--sia-user-agent

Siad User Agent

Sia daemon requires the 'Sia-Agent' user agent by default for security

Properties:

Config: user_agent
Env Var: RCLONE_SIA_USER_AGENT
Type: string
Default: "Sia-Agent"

--sia-encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Properties:

Config: encoding
Env Var: RCLONE_SIA_ENCODING
Type: Encoding
Default: Slash,Question,Hash,Percent,Del,Ctl,InvalidUtf8,Dot

--sia-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_SIA_DESCRIPTION
Type: string
Required: false

Limitations

Modification times not supported
Checksums not supported
rclone about not supported
rclone can work only with Siad or Sia-UI at the moment, the SkyNet daemon is not supported yet.
Sia does not allow control characters or symbols like question and pound signs in file names. rclone will transparently encode them for you, but you'd better be aware

1Fichier

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

1Fichier

This is a backend for the 1fichier cloud storage service. Note that a Premium subscription is required to use the API.

Paths are specified as remote:path

Paths may be as deep as required, e.g. remote:directory/subdirectory.

Configuration

The initial setup for 1Fichier involves getting the API key from the website which you need to do in your browser.

Here is an example of how to make a remote called remote. First run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[snip]
XX / 1Fichier
   \ "fichier"
[snip]
Storage> fichier
** See help for fichier backend at: https://rclone.org/fichier/ **

Your API Key, get it from https://1fichier.com/console/params.pl
Enter a string value. Press Enter for the default ("").
api_key> example_key

Edit advanced config? (y/n)
y) Yes
n) No
y/n> 
Remote config
Configuration complete.
Options:
- type: fichier
- api_key: example_key
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Once configured you can then use rclone like this,

List directories in top level of your 1Fichier account

rclone lsd remote:

List all the files in your 1Fichier account

rclone ls remote:

To copy a local directory to a 1Fichier directory called backup

rclone copy /home/source remote:backup

Modification times and hashes

1Fichier does not support modification times. It supports the Whirlpool hash algorithm.

Duplicated files

1Fichier can have two files with exactly the same name and path (unlike a normal file system).

Duplicated files cause problems with the syncing and you will see messages in the log about duplicates.

Restricted filename characters

In addition to the default restricted characters set the following characters are also replaced:

Character	Value	Replacement
\	0x5C	＼
<	0x3C	＜
>	0x3E	＞
"	0x22	＂
$	0x24	＄
`	0x60	｀
'	0x27	＇

File names can also not start or end with the following characters. These only get replaced if they are the first or last character in the name:

Character	Value	Replacement
SP	0x20	␠

Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

Standard options

Here are the Standard options specific to fichier (1Fichier).

--fichier-api-key

Your API Key, get it from https://1fichier.com/console/params.pl.

Properties:

Config: api_key
Env Var: RCLONE_FICHIER_API_KEY
Type: string
Required: false

Advanced options

Here are the Advanced options specific to fichier (1Fichier).

--fichier-shared-folder

If you want to download a shared folder, add this parameter.

Properties:

Config: shared_folder
Env Var: RCLONE_FICHIER_SHARED_FOLDER
Type: string
Required: false

--fichier-file-password

If you want to download a shared file that is password protected, add this parameter.

NB Input to this must be obscured - see rclone obscure.

Properties:

Config: file_password
Env Var: RCLONE_FICHIER_FILE_PASSWORD
Type: string
Required: false

--fichier-folder-password

If you want to list the files in a shared folder that is password protected, add this parameter.

NB Input to this must be obscured - see rclone obscure.

Properties:

Config: folder_password
Env Var: RCLONE_FICHIER_FOLDER_PASSWORD
Type: string
Required: false

--fichier-cdn

Set if you wish to use CDN download links.

Properties:

Config: cdn
Env Var: RCLONE_FICHIER_CDN
Type: bool
Default: false

--fichier-encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Properties:

Config: encoding
Env Var: RCLONE_FICHIER_ENCODING
Type: Encoding
Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot

--fichier-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_FICHIER_DESCRIPTION
Type: string
Required: false

Limitations

rclone about is not supported by the 1Fichier backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

See List of backends that do not support rclone about and rclone about

Akamai Netstorage

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Akamai NetStorage

Paths are specified as remote: You may put subdirectories in too, e.g. remote:/path/to/dir. If you have a CP code you can use that as the folder after the domain such as <domain>/<cpcode>/<internal directories within cpcode>.

For example, this is commonly configured with or without a CP code:

With a CP code. [your-domain-prefix]-nsu.akamaihd.net/123456/subdirectory/
Without a CP code. [your-domain-prefix]-nsu.akamaihd.net

See all buckets rclone lsd remote: The initial setup for Netstorage involves getting an account and secret. Use rclone config to walk you through the setup process.

Configuration

Here's an example of how to make a remote called ns1.

To begin the interactive configuration process, enter this command:

rclone config

Type n to create a new remote.

n) New remote
d) Delete remote
q) Quit config
e/n/d/q> n

For this example, enter ns1 when you reach the name> prompt.

name> ns1

Enter netstorage as the type of storage to configure.

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
XX / NetStorage
   \ "netstorage"
Storage> netstorage

Select between the HTTP or HTTPS protocol. Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes.

Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / HTTP protocol
   \ "http"
 2 / HTTPS protocol
   \ "https"
protocol> 1

Specify your NetStorage host, CP code, and any necessary content paths using this format: <domain>/<cpcode>/<content>/

Enter a string value. Press Enter for the default ("").
host> baseball-nsu.akamaihd.net/123456/content/

Set the netstorage account name

Enter a string value. Press Enter for the default ("").
account> username

Set the Netstorage account secret/G2O key which will be used for authentication purposes. Select the y option to set your own password then enter your secret. Note: The secret is stored in the rclone.conf file with hex-encoded encryption.

y) Yes type in my own password
g) Generate random password
y/g> y
Enter the password:
password:
Confirm the password:
password:

View the summary and confirm your remote configuration.

[ns1]
type = netstorage
protocol = http
host = baseball-nsu.akamaihd.net/123456/content/
account = username
secret = *** ENCRYPTED ***
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

This remote is called ns1 and can now be used.

Example operations

Get started with rclone and NetStorage with these examples. For additional rclone commands, visit https://rclone.org/commands/.

See contents of a directory in your project

rclone lsd ns1:/974012/testing/

Sync the contents local with remote

rclone sync . ns1:/974012/testing/

Upload local content to remote

rclone copy notes.txt ns1:/974012/testing/

Delete content on remote

rclone delete ns1:/974012/testing/notes.txt

Move or copy content between CP codes.

Your credentials must have access to two CP codes on the same remote. You can't perform operations between different remotes.

rclone move ns1:/974012/testing/notes.txt ns1:/974450/testing2/

Features

Symlink Support

The Netstorage backend changes the rclone --links, -l behavior. When uploading, instead of creating the .rclonelink file, use the "symlink" API in order to create the corresponding symlink on the remote. The .rclonelink file will not be created, the upload will be intercepted and only the symlink file that matches the source file name with no suffix will be created on the remote.

This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks. Due to internal rclone limitations, it is not possible to upload an individual symlink file to any remote backend. You can always use the "backend symlink" command to create a symlink on the NetStorage server, refer to "symlink" section below.

Individual symlink files on the remote can be used with the commands like "cat" to print the destination name, or "delete" to delete symlink, or copy, copy/to and move/moveto to download from the remote to local. Note: individual symlink files on the remote should be specified including the suffix .rclonelink.

Note: No file with the suffix .rclonelink should ever exist on the server since it is not possible to actually upload/create a file with .rclonelink suffix with rclone, it can only exist if it is manually created through a non-rclone method on the remote.

Implicit vs. Explicit Directories

With NetStorage, directories can exist in one of two forms:

Explicit Directory. This is an actual, physical directory that you have created in a storage group.
Implicit Directory. This refers to a directory within a path that has not been physically created. For example, during upload of a file, nonexistent subdirectories can be specified in the target path. NetStorage creates these as "implicit." While the directories aren't physically created, they exist implicitly and the noted path is connected with the uploaded file.

Rclone will intercept all file uploads and mkdir commands for the NetStorage remote and will explicitly issue the mkdir command for each directory in the uploading path. This will help with the interoperability with the other Akamai services such as SFTP and the Content Management Shell (CMShell). Rclone will not guarantee correctness of operations with implicit directories which might have been created as a result of using an upload API directly.

`--fast-list` / ListR support

NetStorage remote supports the ListR feature by using the "list" NetStorage API action to return a lexicographical list of all objects within the specified CP code, recursing into subdirectories as they're encountered.

Rclone will use the ListR method for some commands by default. Commands such as lsf -R will use ListR by default. To disable this, include the --disable listR option to use the non-recursive method of listing objects.
Rclone will not use the ListR method for some commands. Commands such as sync don't use ListR by default. To force using the ListR method, include the --fast-list option.

There are pros and cons of using the ListR method, refer to rclone documentation. In general, the sync command over an existing deep tree on the remote will run faster with the "--fast-list" flag but with extra memory usage as a side effect. It might also result in higher CPU utilization but the whole task can be completed faster.

Note: There is a known limitation that "lsf -R" will display number of files in the directory and directory size as -1 when ListR method is used. The workaround is to pass "--disable listR" flag if these numbers are important in the output.

Purge

NetStorage remote supports the purge feature by using the "quick-delete" NetStorage API action. The quick-delete action is disabled by default for security reasons and can be enabled for the account through the Akamai portal. Rclone will first try to use quick-delete action for the purge command and if this functionality is disabled then will fall back to a standard delete method.

Note: Read the NetStorage Usage API for considerations when using "quick-delete". In general, using quick-delete method will not delete the tree immediately and objects targeted for quick-delete may still be accessible.

Standard options

Here are the Standard options specific to netstorage (Akamai NetStorage).

--netstorage-host

Domain+path of NetStorage host to connect to.

Format should be <domain>/<internal folders>

Properties:

Config: host
Env Var: RCLONE_NETSTORAGE_HOST
Type: string
Required: true

--netstorage-account

Set the NetStorage account name

Properties:

Config: account
Env Var: RCLONE_NETSTORAGE_ACCOUNT
Type: string
Required: true

--netstorage-secret

Set the NetStorage account secret/G2O key for authentication.

Please choose the 'y' option to set your own password then enter your secret.

NB Input to this must be obscured - see rclone obscure.

Properties:

Config: secret
Env Var: RCLONE_NETSTORAGE_SECRET
Type: string
Required: true

Advanced options

Here are the Advanced options specific to netstorage (Akamai NetStorage).

--netstorage-protocol

Select between HTTP or HTTPS protocol.

Most users should choose HTTPS, which is the default. HTTP is provided primarily for debugging purposes.

Properties:

Config: protocol
Env Var: RCLONE_NETSTORAGE_PROTOCOL
Type: string
Default: "https"
Examples:
- "http"
  - HTTP protocol
- "https"
  - HTTPS protocol

--netstorage-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_NETSTORAGE_DESCRIPTION
Type: string
Required: false

Backend commands

Here are the commands specific to the netstorage backend.

Run them with

rclone backend COMMAND remote:

The help below will explain what arguments each command takes.

See the backend command for more info on how to pass options and arguments.

These can be run on a running backend using the rc command backend/command.

du

Return disk usage information for a specified directory

rclone backend du remote: [options] [<arguments>+]

The usage information returned, includes the targeted directory as well as all files stored in any sub-directories that may exist.

symlink

You can create a symbolic link in ObjectStore with the symlink action.

rclone backend symlink remote: [options] [<arguments>+]

The desired path location (including applicable sub-directories) ending in the object that will be the target of the symlink (for example, /links/mylink). Include the file extension for the object, if applicable. rclone backend symlink <src> <path>

Alias

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Alias

The alias remote provides a new name for another remote.

Paths may be as deep as required or a local path, e.g. remote:directory/subdirectory or /directory/subdirectory.

During the initial setup with rclone config you will specify the target remote. The target remote can either be a local path or another remote.

Subfolders can be used in target remote. Assume an alias remote named backup with the target mydrive:private/backup. Invoking rclone mkdir backup:desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/desktop.

There will be no special handling of paths containing .. segments. Invoking rclone mkdir backup:../desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/../desktop. The empty path is not allowed as a remote. To alias the current directory use . instead.

The target remote can also be a connection string. This can be used to modify the config of a remote for different uses, e.g. the alias myDriveTrash with the target remote myDrive,trashed_only: can be used to only show the trashed files in myDrive.

Configuration

Here is an example of how to make an alias called remote for local folder. First run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Alias for an existing remote
   \ "alias"
[snip]
Storage> alias
Remote or path to alias.
Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".
remote> /mnt/storage/backup
Remote config
Configuration complete.
Options:
- type: alias
- remote: /mnt/storage/backup
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
remote               alias

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

Once configured you can then use rclone like this,

List directories in top level in /mnt/storage/backup

rclone lsd remote:

List all the files in /mnt/storage/backup

rclone ls remote:

Copy another local directory to the alias directory called source

rclone copy /home/source remote:source

Standard options

Here are the Standard options specific to alias (Alias for an existing remote).

--alias-remote

Remote or path to alias.

Can be "myremote:path/to/dir", "myremote:bucket", "myremote:" or "/local/path".

Properties:

Config: remote
Env Var: RCLONE_ALIAS_REMOTE
Type: string
Required: true

Advanced options

Here are the Advanced options specific to alias (Alias for an existing remote).

--alias-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_ALIAS_DESCRIPTION
Type: string
Required: false

Amazon Drive

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Amazon Drive

Amazon Drive, formerly known as Amazon Cloud Drive, was a cloud storage service run by Amazon for consumers.

From 2023-12-31, Amazon Drive has been discontinued by Amazon so the Amazon Drive backend has been removed.

You can still use Amazon Photos to access your photos and videos.

Amazon S3

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Amazon S3 Storage Providers

The S3 backend can be used with a number of different providers:

AWS S3 Home Config
Alibaba Cloud (Aliyun) Object Storage System (OSS) Home Config
Ceph Home Config
China Mobile Ecloud Elastic Object Storage (EOS) Home Config
Cloudflare R2 Home Config
Arvan Cloud Object Storage (AOS) Home Config
DigitalOcean Spaces Home Config
Dreamhost Home Config
GCS Home Config
Huawei OBS Home Config
IBM COS S3 Home Config
IDrive e2 Home Config
IONOS Cloud Home Config
Leviia Object Storage Home Config
Liara Object Storage Home Config
Linode Object Storage Home Config
Magalu Object Storage Home Config
Minio Home Config
Petabox Home Config
Qiniu Cloud Object Storage (Kodo) Home Config
RackCorp Object Storage Home Config
Rclone Serve S3 Home Config
Scaleway Home Config
Seagate Lyve Cloud Home Config
SeaweedFS Home Config
StackPath Home Config
Storj Home Config
Synology C2 Object Storage Home Config
Tencent Cloud Object Storage (COS) Home Config
Wasabi Home Config

Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:bucket/path/to/dir.

Once you have made a remote (see the provider specific section above) you can use it like this:

See all buckets

rclone lsd remote:

Make a new bucket

rclone mkdir remote:bucket

List the contents of a bucket

rclone ls remote:bucket

Sync /home/local/directory to the remote bucket, deleting any excess files in the bucket.

rclone sync --interactive /home/local/directory remote:bucket

Configuration

Here is an example of making an s3 configuration for the AWS S3 provider. Most applies to the other providers as well, any differences are described below.

First run

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ "s3"
[snip]
Storage> s3
Choose your S3 provider.
Choose a number from below, or type in your own value
 1 / Amazon Web Services (AWS) S3
   \ "AWS"
 2 / Ceph Object Storage
   \ "Ceph"
 3 / DigitalOcean Spaces
   \ "DigitalOcean"
 4 / Dreamhost DreamObjects
   \ "Dreamhost"
 5 / IBM COS S3
   \ "IBMCOS"
 6 / Minio Object Storage
   \ "Minio"
 7 / Wasabi Object Storage
   \ "Wasabi"
 8 / Any other S3 compatible provider
   \ "Other"
provider> 1
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> XXX
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> YYY
Region to connect to.
Choose a number from below, or type in your own value
   / The default endpoint - a good choice if you are unsure.
 1 | US Region, Northern Virginia, or Pacific Northwest.
   | Leave location constraint empty.
   \ "us-east-1"
   / US East (Ohio) Region
 2 | Needs location constraint us-east-2.
   \ "us-east-2"
   / US West (Oregon) Region
 3 | Needs location constraint us-west-2.
   \ "us-west-2"
   / US West (Northern California) Region
 4 | Needs location constraint us-west-1.
   \ "us-west-1"
   / Canada (Central) Region
 5 | Needs location constraint ca-central-1.
   \ "ca-central-1"
   / EU (Ireland) Region
 6 | Needs location constraint EU or eu-west-1.
   \ "eu-west-1"
   / EU (London) Region
 7 | Needs location constraint eu-west-2.
   \ "eu-west-2"
   / EU (Frankfurt) Region
 8 | Needs location constraint eu-central-1.
   \ "eu-central-1"
   / Asia Pacific (Singapore) Region
 9 | Needs location constraint ap-southeast-1.
   \ "ap-southeast-1"
   / Asia Pacific (Sydney) Region
10 | Needs location constraint ap-southeast-2.
   \ "ap-southeast-2"
   / Asia Pacific (Tokyo) Region
11 | Needs location constraint ap-northeast-1.
   \ "ap-northeast-1"
   / Asia Pacific (Seoul)
12 | Needs location constraint ap-northeast-2.
   \ "ap-northeast-2"
   / Asia Pacific (Mumbai)
13 | Needs location constraint ap-south-1.
   \ "ap-south-1"
   / Asia Pacific (Hong Kong) Region
14 | Needs location constraint ap-east-1.
   \ "ap-east-1"
   / South America (Sao Paulo) Region
15 | Needs location constraint sa-east-1.
   \ "sa-east-1"
region> 1
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
endpoint>
Location constraint - must be set to match the Region. Used when creating buckets only.
Choose a number from below, or type in your own value
 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
   \ ""
 2 / US East (Ohio) Region.
   \ "us-east-2"
 3 / US West (Oregon) Region.
   \ "us-west-2"
 4 / US West (Northern California) Region.
   \ "us-west-1"
 5 / Canada (Central) Region.
   \ "ca-central-1"
 6 / EU (Ireland) Region.
   \ "eu-west-1"
 7 / EU (London) Region.
   \ "eu-west-2"
 8 / EU Region.
   \ "EU"
 9 / Asia Pacific (Singapore) Region.
   \ "ap-southeast-1"
10 / Asia Pacific (Sydney) Region.
   \ "ap-southeast-2"
11 / Asia Pacific (Tokyo) Region.
   \ "ap-northeast-1"
12 / Asia Pacific (Seoul)
   \ "ap-northeast-2"
13 / Asia Pacific (Mumbai)
   \ "ap-south-1"
14 / Asia Pacific (Hong Kong)
   \ "ap-east-1"
15 / South America (Sao Paulo) Region.
   \ "sa-east-1"
location_constraint> 1
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
   \ "public-read"
   / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
 3 | Granting this on a bucket is generally not recommended.
   \ "public-read-write"
 4 / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access.
   \ "authenticated-read"
   / Object owner gets FULL_CONTROL. Bucket owner gets READ access.
 5 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-read"
   / Both the object owner and the bucket owner get FULL_CONTROL over the object.
 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ "bucket-owner-full-control"
acl> 1
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
server_side_encryption> 1
The storage class to use when storing objects in S3.
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
 3 / Reduced redundancy storage class
   \ "REDUCED_REDUNDANCY"
 4 / Standard Infrequent Access storage class
   \ "STANDARD_IA"
 5 / One Zone Infrequent Access storage class
   \ "ONEZONE_IA"
 6 / Glacier storage class
   \ "GLACIER"
 7 / Glacier Deep Archive storage class
   \ "DEEP_ARCHIVE"
 8 / Intelligent-Tiering storage class
   \ "INTELLIGENT_TIERING"
 9 / Glacier Instant Retrieval storage class
   \ "GLACIER_IR"
storage_class> 1
Remote config
Configuration complete.
Options:
- type: s3
- provider: AWS
- env_auth: false
- access_key_id: XXX
- secret_access_key: YYY
- region: us-east-1
- endpoint:
- location_constraint:
- acl: private
- server_side_encryption:
- storage_class:
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d>

Modification times and hashes

Modification times

The modified time is stored as metadata on the object as X-Amz-Meta-Mtime as floating point since the epoch, accurate to 1 ns.

If the modification time needs to be updated rclone will attempt to perform a server side copy to update the modification if the object can be copied in a single part. In the case the object is larger than 5Gb or is in Glacier or Glacier Deep Archive storage the object will be uploaded rather than copied.

Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.

Hashes

For small objects which weren't uploaded as multipart uploads (objects sized below --s3-upload-cutoff if uploaded with rclone) rclone uses the ETag: header as an MD5 checksum.

However for objects which were uploaded as multipart uploads or with server side encryption (SSE-AWS or SSE-C) the ETag header is no longer the MD5 sum of the data, so rclone adds an additional piece of metadata X-Amz-Meta-Md5chksum which is a base64 encoded MD5 hash (in the same format as is required for Content-MD5). You can use base64 -d and hexdump to check this value manually:

echo 'VWTGdNx3LyXQDfA0e2Edxw==' | base64 -d | hexdump

or you can use rclone check to verify the hashes are OK.

For large objects, calculating this hash can take some time so the addition of this hash can be disabled with --s3-disable-checksum. This will mean that these objects do not have an MD5 checksum.

Note that reading this from the object takes an additional HEAD request as the metadata isn't returned in object listings.

Reducing costs

Avoiding HEAD requests to read the modification time

By default, rclone will use the modification time of objects stored in S3 for syncing. This is stored in object metadata which unfortunately takes an extra HEAD request to read which can be expensive (in time and money).

The modification time is used by default for all operations that require checking the time a file was last updated. It allows rclone to treat the remote more like a true filesystem, but it is inefficient on S3 because it requires an extra API call to retrieve the metadata.

The extra API calls can be avoided when syncing (using rclone sync or rclone copy) in a few different ways, each with its own tradeoffs.

--size-only
- Only checks the size of files.
- Uses no extra transactions.
- If the file doesn't change size then rclone won't detect it has changed.
- rclone sync --size-only /path/to/source s3:bucket
--checksum
- Checks the size and MD5 checksum of files.
- Uses no extra transactions.
- The most accurate detection of changes possible.
- Will cause the source to read an MD5 checksum which, if it is a local disk, will cause lots of disk activity.
- If the source and destination are both S3 this is the recommended flag to use for maximum efficiency.
- rclone sync --checksum /path/to/source s3:bucket
--update --use-server-modtime
- Uses no extra transactions.
- Modification time becomes the time the object was uploaded.
- For many operations this is sufficient to determine if it needs uploading.
- Using --update along with --use-server-modtime, avoids the extra API call and uploads files whose local modification time is newer than the time it was last uploaded.
- Files created with timestamps in the past will be missed by the sync.
- rclone sync --update --use-server-modtime /path/to/source s3:bucket

These flags can and should be used in combination with --fast-list - see below.

If using rclone mount or any command using the VFS (eg rclone serve) commands then you might want to consider using the VFS flag --no-modtime which will stop rclone reading the modification time for every object. You could also use --use-server-modtime if you are happy with the modification times of the objects being the time of upload.

Avoiding GET requests to read directory listings

Rclone's default directory traversal is to process each directory individually. This takes one API call per directory. Using the --fast-list flag will read all info about the objects into memory first using a smaller number of API calls (one per 1000 objects). See the rclone docs for more details.

rclone sync --fast-list --checksum /path/to/source s3:bucket

--fast-list trades off API transactions for memory use. As a rough guide rclone uses 1k of memory per object stored, so using --fast-list on a sync of a million objects will use roughly 1 GiB of RAM.

If you are only copying a small number of files into a big repository then using --no-traverse is a good idea. This finds objects directly instead of through directory listings. You can do a "top-up" sync very cheaply by using --max-age and --no-traverse to copy only recent files, eg

rclone copy --max-age 24h --no-traverse /path/to/source s3:bucket

You'd then do a full rclone sync less often.

Note that --fast-list isn't required in the top-up sync.

Avoiding HEAD requests after PUT

By default, rclone will HEAD every object it uploads. It does this to check the object got uploaded correctly.

You can disable this with the --s3-no-head option - see there for more details.

Setting this flag increases the chance for undetected upload failures.

Increasing performance

Using server-side copy

If you are copying objects between S3 buckets in the same region, you should use server-side copy. This is much faster than downloading and re-uploading the objects, as no data is transferred.

For rclone to use server-side copy, you must use the same remote for the source and destination.

rclone copy s3:source-bucket s3:destination-bucket

When using server-side copy, the performance is limited by the rate at which rclone issues API requests to S3. See below for how to increase the number of API requests rclone makes.

Increasing the rate of API requests

You can increase the rate of API requests to S3 by increasing the parallelism using --transfers and --checkers options.

Rclone uses a very conservative defaults for these settings, as not all providers support high rates of requests. Depending on your provider, you can increase significantly the number of transfers and checkers.

For example, with AWS S3, if you can increase the number of checkers to values like 200. If you are doing a server-side copy, you can also increase the number of transfers to 200.

rclone sync --transfers 200 --checkers 200 --checksum s3:source-bucket s3:destination-bucket

You will need to experiment with these values to find the optimal settings for your setup.

Versions

When bucket versioning is enabled (this can be done with rclone with the rclone backend versioning command) when rclone uploads a new version of a file it creates a new version of it Likewise when you delete a file, the old version will be marked hidden and still be available.

Old versions of files, where available, are visible using the --s3-versions flag.

It is also possible to view a bucket as it was at a certain point in time, using the --s3-version-at flag. This will show the file versions as they were at that time, showing files that have been deleted afterwards, and hiding files that were created since.

If you wish to remove all the old versions then you can use the rclone backend cleanup-hidden remote:bucket command which will delete all the old hidden versions of files, leaving the current ones intact. You can also supply a path and only old versions under that path will be deleted, e.g. rclone backend cleanup-hidden remote:bucket/path/to/stuff.

When you purge a bucket, the current and the old versions will be deleted then the bucket will be deleted.

However delete will cause the current versions of the files to become hidden old versions.

Here is a session showing the listing and retrieval of an old version followed by a cleanup of the old versions.

Show current version and all the versions with --s3-versions flag.

$ rclone -q ls s3:cleanup-test
        9 one.txt

$ rclone -q --s3-versions ls s3:cleanup-test
        9 one.txt
        8 one-v2016-07-04-141032-000.txt
       16 one-v2016-07-04-141003-000.txt
       15 one-v2016-07-02-155621-000.txt

Retrieve an old version

$ rclone -q --s3-versions copy s3:cleanup-test/one-v2016-07-04-141003-000.txt /tmp

$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul  2 17:46 /tmp/one-v2016-07-04-141003-000.txt

Clean up all the old versions and show that they've gone.

$ rclone -q backend cleanup-hidden s3:cleanup-test

$ rclone -q ls s3:cleanup-test
        9 one.txt

$ rclone -q --s3-versions ls s3:cleanup-test
        9 one.txt

Versions naming caveat

When using --s3-versions flag rclone is relying on the file name to work out whether the objects are versions or not. Versions' names are created by inserting timestamp between file name and its extension.

        9 file.txt
        8 file-v2023-07-17-161032-000.txt
       16 file-v2023-06-15-141003-000.txt

If there are real files present with the same names as versions, then behaviour of --s3-versions can be unpredictable.

Cleanup

If you run rclone cleanup s3:bucket then it will remove all pending multipart uploads older than 24 hours. You can use the --interactive/i or --dry-run flag to see exactly what it will do. If you want more control over the expiry date then run rclone backend cleanup s3:bucket -o max-age=1h to expire all uploads older than one hour. You can use rclone backend list-multipart-uploads s3:bucket to see the pending multipart uploads.

Restricted filename characters

S3 allows any valid UTF-8 string as a key.

Invalid UTF-8 bytes will be replaced, as they can't be used in XML.

The following characters are replaced since these are problematic when dealing with the REST API:

Character	Value	Replacement
NUL	0x00	␀
/	0x2F	／

The encoding will also encode these file names as they don't seem to work with the SDK properly:

File name	Replacement
.	．
..	．．

Multipart uploads

rclone supports multipart uploads with S3 which means that it can upload files bigger than 5 GiB.

Note that files uploaded both with multipart upload and through crypt remotes do not have MD5 sums.

rclone switches from single part uploads to multipart uploads at the point specified by --s3-upload-cutoff. This can be a maximum of 5 GiB and a minimum of 0 (ie always upload multipart files).

The chunk sizes used in the multipart upload are specified by --s3-chunk-size and the number of chunks uploaded concurrently is specified by --s3-upload-concurrency.

Multipart uploads will use --transfers * --s3-upload-concurrency * --s3-chunk-size extra memory. Single part uploads to not use extra memory.

Single part transfers can be faster than multipart transfers or slower depending on your latency from S3 - the more latency, the more likely single part transfers will be faster.

Increasing --s3-upload-concurrency will increase throughput (8 would be a sensible value) and increasing --s3-chunk-size also increases throughput (16M would be sensible). Increasing either of these will use more memory. The default values are high enough to gain most of the possible performance without using too much memory.

Buckets and Regions

With Amazon S3 you can list buckets (rclone lsd) using any region, but you can only access the content of a bucket from the region it was created in. If you attempt to access a bucket from the wrong region, you will get an error, incorrect region, the bucket is not in 'XXX' region.

Authentication

There are a number of ways to supply rclone with a set of AWS credentials, with and without using the environment.

The different authentication methods are tried in this order:

Directly in the rclone configuration file (env_auth = false in the config file):
- access_key_id and secret_access_key are required.
- session_token can be optionally set when using AWS STS.
Runtime configuration (env_auth = true in the config file):
- Export the following environment variables before running rclone:
  - Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
  - Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
  - Session Token: AWS_SESSION_TOKEN (optional)
- Or, use a named profile:
  - Profile files are standard files used by AWS CLI tools
  - By default it will use the profile in your home directory (e.g. ~/.aws/credentials on unix based systems) file and the "default" profile, to change set these environment variables or config keys:
    - AWS_SHARED_CREDENTIALS_FILE to control which file or the shared_credentials_file config key.
    - AWS_PROFILE to control which profile to use or the profile config key.
- Or, run rclone in an ECS task with an IAM role (AWS only).
- Or, run rclone on an EC2 instance with an IAM role (AWS only).
- Or, run rclone in an EKS pod with an IAM role that is associated with a service account (AWS only).
- Or, use process credentials to read config from an external program.

With env_auth = true rclone (which uses the SDK for Go v2) should support all authentication methods that the aws CLI tool does and the other AWS SDKs.

If none of these option actually end up providing rclone with AWS credentials then S3 interaction will be non-authenticated (see the anonymous access section for more info).

S3 Permissions

When using the sync subcommand of rclone the following minimum permissions are required to be available on the bucket being written to:

ListBucket
DeleteObject
GetObject
PutObject
PutObjectACL
CreateBucket (unless using s3-no-check-bucket)

When using the lsd subcommand, the ListAllMyBuckets permission is required.

Example policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::USER_SID:user/USER_NAME"
            },
            "Action": [
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
              "arn:aws:s3:::BUCKET_NAME/*",
              "arn:aws:s3:::BUCKET_NAME"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "arn:aws:s3:::*"
        }
    ]
}

Notes on above:

This is a policy that can be used when creating bucket. It assumes that USER_NAME has been created.
The Resource entry must include both resource ARNs, as one implies the bucket and the other implies the bucket's objects.
When using s3-no-check-bucket and the bucket already exsits, the "arn:aws:s3:::BUCKET_NAME" doesn't have to be included.

For reference, here's an Ansible script that will generate one or more buckets that will work with rclone sync.

Key Management System (KMS)

If you are using server-side encryption with KMS then you must make sure rclone is configured with server_side_encryption = aws:kms otherwise you will find you can't transfer small objects - these will create checksum errors.

Glacier and Glacier Deep Archive

You can upload objects using the glacier storage class or transition them to glacier using a lifecycle policy. The bucket can still be synced or copied into normally, but if rclone tries to access data from the glacier storage class you will see an error like below.

2017/09/11 19:07:43 Failed to sync: failed to open source object: Object in GLACIER, restore first: path/to/file

In this case you need to restore the object(s) in question before using rclone.

Note that rclone only speaks the S3 API it does not speak the Glacier Vault API, so rclone cannot directly access Glacier Vaults.

Object-lock enabled S3 bucket

According to AWS's documentation on S3 Object Lock:

If you configure a default retention period on a bucket, requests to upload objects in such a bucket must include the Content-MD5 header.

As mentioned in the Modification times and hashes section, small files that are not uploaded as multipart, use a different tag, causing the upload to fail. A simple solution is to set the --s3-upload-cutoff 0 and force all the files to be uploaded as multipart.

Standard options

Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others).

--s3-provider

Choose your S3 provider.

Properties:

Config: provider
Env Var: RCLONE_S3_PROVIDER
Type: string
Required: false
Examples:
- "AWS"
  - Amazon Web Services (AWS) S3
- "Alibaba"
  - Alibaba Cloud Object Storage System (OSS) formerly Aliyun
- "ArvanCloud"
  - Arvan Cloud Object Storage (AOS)
- "Ceph"
  - Ceph Object Storage
- "ChinaMobile"
  - China Mobile Ecloud Elastic Object Storage (EOS)
- "Cloudflare"
  - Cloudflare R2 Storage
- "DigitalOcean"
  - DigitalOcean Spaces
- "Dreamhost"
  - Dreamhost DreamObjects
- "GCS"
  - Google Cloud Storage
- "HuaweiOBS"
  - Huawei Object Storage Service
- "IBMCOS"
  - IBM COS S3
- "IDrive"
  - IDrive e2
- "IONOS"
  - IONOS Cloud
- "LyveCloud"
  - Seagate Lyve Cloud
- "Leviia"
  - Leviia Object Storage
- "Liara"
  - Liara Object Storage
- "Linode"
  - Linode Object Storage
- "Magalu"
  - Magalu Object Storage
- "Minio"
  - Minio Object Storage
- "Netease"
  - Netease Object Storage (NOS)
- "Petabox"
  - Petabox Object Storage
- "RackCorp"
  - RackCorp Object Storage
- "Rclone"
  - Rclone S3 Server
- "Scaleway"
  - Scaleway Object Storage
- "SeaweedFS"
  - SeaweedFS S3
- "StackPath"
  - StackPath Object Storage
- "Storj"
  - Storj (S3 Compatible Gateway)
- "Synology"
  - Synology C2 Object Storage
- "TencentCOS"
  - Tencent Cloud Object Storage (COS)
- "Wasabi"
  - Wasabi Object Storage
- "Qiniu"
  - Qiniu Object Storage (Kodo)
- "Other"
  - Any other S3 compatible provider

--s3-env-auth

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).

Only applies if access_key_id and secret_access_key is blank.

Properties:

Config: env_auth
Env Var: RCLONE_S3_ENV_AUTH
Type: bool
Default: false
Examples:
- "false"
  - Enter AWS credentials in the next step.
- "true"
  - Get AWS credentials from the environment (env vars or IAM).

--s3-access-key-id

AWS Access Key ID.

Leave blank for anonymous access or runtime credentials.

Properties:

Config: access_key_id
Env Var: RCLONE_S3_ACCESS_KEY_ID
Type: string
Required: false

--s3-secret-access-key

AWS Secret Access Key (password).

Leave blank for anonymous access or runtime credentials.

Properties:

Config: secret_access_key
Env Var: RCLONE_S3_SECRET_ACCESS_KEY
Type: string
Required: false

--s3-region

Region to connect to.

Properties:

Config: region
Env Var: RCLONE_S3_REGION
Provider: AWS
Type: string
Required: false
Examples:
- "us-east-1"
  - The default endpoint - a good choice if you are unsure.
  - US Region, Northern Virginia, or Pacific Northwest.
  - Leave location constraint empty.
- "us-east-2"
  - US East (Ohio) Region.
  - Needs location constraint us-east-2.
- "us-west-1"
  - US West (Northern California) Region.
  - Needs location constraint us-west-1.
- "us-west-2"
  - US West (Oregon) Region.
  - Needs location constraint us-west-2.
- "ca-central-1"
  - Canada (Central) Region.
  - Needs location constraint ca-central-1.
- "eu-west-1"
  - EU (Ireland) Region.
  - Needs location constraint EU or eu-west-1.
- "eu-west-2"
  - EU (London) Region.
  - Needs location constraint eu-west-2.
- "eu-west-3"
  - EU (Paris) Region.
  - Needs location constraint eu-west-3.
- "eu-north-1"
  - EU (Stockholm) Region.
  - Needs location constraint eu-north-1.
- "eu-south-1"
  - EU (Milan) Region.
  - Needs location constraint eu-south-1.
- "eu-central-1"
  - EU (Frankfurt) Region.
  - Needs location constraint eu-central-1.
- "ap-southeast-1"
  - Asia Pacific (Singapore) Region.
  - Needs location constraint ap-southeast-1.
- "ap-southeast-2"
  - Asia Pacific (Sydney) Region.
  - Needs location constraint ap-southeast-2.
- "ap-northeast-1"
  - Asia Pacific (Tokyo) Region.
  - Needs location constraint ap-northeast-1.
- "ap-northeast-2"
  - Asia Pacific (Seoul).
  - Needs location constraint ap-northeast-2.
- "ap-northeast-3"
  - Asia Pacific (Osaka-Local).
  - Needs location constraint ap-northeast-3.
- "ap-south-1"
  - Asia Pacific (Mumbai).
  - Needs location constraint ap-south-1.
- "ap-east-1"
  - Asia Pacific (Hong Kong) Region.
  - Needs location constraint ap-east-1.
- "sa-east-1"
  - South America (Sao Paulo) Region.
  - Needs location constraint sa-east-1.
- "il-central-1"
  - Israel (Tel Aviv) Region.
  - Needs location constraint il-central-1.
- "me-south-1"
  - Middle East (Bahrain) Region.
  - Needs location constraint me-south-1.
- "af-south-1"
  - Africa (Cape Town) Region.
  - Needs location constraint af-south-1.
- "cn-north-1"
  - China (Beijing) Region.
  - Needs location constraint cn-north-1.
- "cn-northwest-1"
  - China (Ningxia) Region.
  - Needs location constraint cn-northwest-1.
- "us-gov-east-1"
  - AWS GovCloud (US-East) Region.
  - Needs location constraint us-gov-east-1.
- "us-gov-west-1"
  - AWS GovCloud (US) Region.
  - Needs location constraint us-gov-west-1.

--s3-endpoint

Endpoint for S3 API.

Leave blank if using AWS to use the default endpoint for the region.

Properties:

Config: endpoint
Env Var: RCLONE_S3_ENDPOINT
Provider: AWS
Type: string
Required: false

--s3-location-constraint

Location constraint - must be set to match the Region.

Used when creating buckets only.

Properties:

Config: location_constraint
Env Var: RCLONE_S3_LOCATION_CONSTRAINT
Provider: AWS
Type: string
Required: false
Examples:
- ""
  - Empty for US Region, Northern Virginia, or Pacific Northwest
- "us-east-2"
  - US East (Ohio) Region
- "us-west-1"
  - US West (Northern California) Region
- "us-west-2"
  - US West (Oregon) Region
- "ca-central-1"
  - Canada (Central) Region
- "eu-west-1"
  - EU (Ireland) Region
- "eu-west-2"
  - EU (London) Region
- "eu-west-3"
  - EU (Paris) Region
- "eu-north-1"
  - EU (Stockholm) Region
- "eu-south-1"
  - EU (Milan) Region
- "EU"
  - EU Region
- "ap-southeast-1"
  - Asia Pacific (Singapore) Region
- "ap-southeast-2"
  - Asia Pacific (Sydney) Region
- "ap-northeast-1"
  - Asia Pacific (Tokyo) Region
- "ap-northeast-2"
  - Asia Pacific (Seoul) Region
- "ap-northeast-3"
  - Asia Pacific (Osaka-Local) Region
- "ap-south-1"
  - Asia Pacific (Mumbai) Region
- "ap-east-1"
  - Asia Pacific (Hong Kong) Region
- "sa-east-1"
  - South America (Sao Paulo) Region
- "il-central-1"
  - Israel (Tel Aviv) Region
- "me-south-1"
  - Middle East (Bahrain) Region
- "af-south-1"
  - Africa (Cape Town) Region
- "cn-north-1"
  - China (Beijing) Region
- "cn-northwest-1"
  - China (Ningxia) Region
- "us-gov-east-1"
  - AWS GovCloud (US-East) Region
- "us-gov-west-1"
  - AWS GovCloud (US) Region

--s3-acl

Canned ACL used when creating buckets and storing or copying objects.

This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.

For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl

Note that this ACL is applied when server-side copying objects as S3 doesn't copy the ACL from the source but rather writes a fresh one.

If the acl is an empty string then no X-Amz-Acl: header is added and the default (private) will be used.

Properties:

Config: acl
Env Var: RCLONE_S3_ACL
Provider: !Storj,Synology,Cloudflare
Type: string
Required: false
Examples:
- "default"
  - Owner gets Full_CONTROL.
  - No one else has access rights (default).
- "private"
  - Owner gets FULL_CONTROL.
  - No one else has access rights (default).
- "public-read"
  - Owner gets FULL_CONTROL.
  - The AllUsers group gets READ access.
- "public-read-write"
  - Owner gets FULL_CONTROL.
  - The AllUsers group gets READ and WRITE access.
  - Granting this on a bucket is generally not recommended.
- "authenticated-read"
  - Owner gets FULL_CONTROL.
  - The AuthenticatedUsers group gets READ access.
- "bucket-owner-read"
  - Object owner gets FULL_CONTROL.
  - Bucket owner gets READ access.
  - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- "bucket-owner-full-control"
  - Both the object owner and the bucket owner get FULL_CONTROL over the object.
  - If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
- "private"
  - Owner gets FULL_CONTROL.
  - No one else has access rights (default).
  - This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
- "public-read"
  - Owner gets FULL_CONTROL.
  - The AllUsers group gets READ access.
  - This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
- "public-read-write"
  - Owner gets FULL_CONTROL.
  - The AllUsers group gets READ and WRITE access.
  - This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
- "authenticated-read"
  - Owner gets FULL_CONTROL.
  - The AuthenticatedUsers group gets READ access.
  - Not supported on Buckets.
  - This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.

--s3-server-side-encryption

The server-side encryption algorithm used when storing this object in S3.

Properties:

Config: server_side_encryption
Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
Provider: AWS,Ceph,ChinaMobile,Minio
Type: string
Required: false
Examples:
- ""
  - None
- "AES256"
  - AES256
- "aws:kms"
  - aws:kms

--s3-sse-kms-key-id

If using KMS ID you must provide the ARN of Key.

Properties:

Config: sse_kms_key_id
Env Var: RCLONE_S3_SSE_KMS_KEY_ID
Provider: AWS,Ceph,Minio
Type: string
Required: false
Examples:
- ""
  - None
- "arn:aws:kms:us-east-1:*"
  - arn:aws:kms:*

--s3-storage-class

The storage class to use when storing new objects in S3.

Properties:

Config: storage_class
Env Var: RCLONE_S3_STORAGE_CLASS
Provider: AWS
Type: string
Required: false
Examples:
- ""
  - Default
- "STANDARD"
  - Standard storage class
- "REDUCED_REDUNDANCY"
  - Reduced redundancy storage class
- "STANDARD_IA"
  - Standard Infrequent Access storage class
- "ONEZONE_IA"
  - One Zone Infrequent Access storage class
- "GLACIER"
  - Glacier storage class
- "DEEP_ARCHIVE"
  - Glacier Deep Archive storage class
- "INTELLIGENT_TIERING"
  - Intelligent-Tiering storage class
- "GLACIER_IR"
  - Glacier Instant Retrieval storage class

Advanced options

Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, ArvanCloud, Ceph, ChinaMobile, Cloudflare, DigitalOcean, Dreamhost, GCS, HuaweiOBS, IBMCOS, IDrive, IONOS, LyveCloud, Leviia, Liara, Linode, Magalu, Minio, Netease, Petabox, RackCorp, Rclone, Scaleway, SeaweedFS, StackPath, Storj, Synology, TencentCOS, Wasabi, Qiniu and others).

--s3-bucket-acl

Canned ACL used when creating buckets.

For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl

Note that this ACL is applied when only when creating buckets. If it isn't set then "acl" is used instead.

If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl: header is added and the default (private) will be used.

Properties:

Config: bucket_acl
Env Var: RCLONE_S3_BUCKET_ACL
Type: string
Required: false
Examples:
- "private"
  - Owner gets FULL_CONTROL.
  - No one else has access rights (default).
- "public-read"
  - Owner gets FULL_CONTROL.
  - The AllUsers group gets READ access.
- "public-read-write"
  - Owner gets FULL_CONTROL.
  - The AllUsers group gets READ and WRITE access.
  - Granting this on a bucket is generally not recommended.
- "authenticated-read"
  - Owner gets FULL_CONTROL.
  - The AuthenticatedUsers group gets READ access.

--s3-requester-pays

Enables requester pays option when interacting with S3 bucket.

Properties:

Config: requester_pays
Env Var: RCLONE_S3_REQUESTER_PAYS
Provider: AWS
Type: bool
Default: false

--s3-sse-customer-algorithm

If using SSE-C, the server-side encryption algorithm used when storing this object in S3.

Properties:

Config: sse_customer_algorithm
Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
Provider: AWS,Ceph,ChinaMobile,Minio
Type: string
Required: false
Examples:
- ""
  - None
- "AES256"
  - AES256

--s3-sse-customer-key

To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.

Alternatively you can provide --sse-customer-key-base64.

Properties:

Config: sse_customer_key
Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
Provider: AWS,Ceph,ChinaMobile,Minio
Type: string
Required: false
Examples:
- ""
  - None

--s3-sse-customer-key-base64

If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.

Alternatively you can provide --sse-customer-key.

Properties:

Config: sse_customer_key_base64
Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_BASE64
Provider: AWS,Ceph,ChinaMobile,Minio
Type: string
Required: false
Examples:
- ""
  - None

--s3-sse-customer-key-md5

If using SSE-C you may provide the secret encryption key MD5 checksum (optional).

If you leave it blank, this is calculated automatically from the sse_customer_key provided.

Properties:

Config: sse_customer_key_md5
Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
Provider: AWS,Ceph,ChinaMobile,Minio
Type: string
Required: false
Examples:
- ""
  - None

--s3-upload-cutoff

Cutoff for switching to chunked upload.

Any files larger than this will be uploaded in chunks of chunk_size. The minimum is 0 and the maximum is 5 GiB.

Properties:

Config: upload_cutoff
Env Var: RCLONE_S3_UPLOAD_CUTOFF
Type: SizeSuffix
Default: 200Mi

--s3-chunk-size

Chunk size to use for uploading.

When uploading files larger than upload_cutoff or files with unknown size (e.g. from "rclone rcat" or uploaded with "rclone mount" or google photos or google docs) they will be uploaded as multipart uploads using this chunk size.

Note that "--s3-upload-concurrency" chunks of this size are buffered in memory per transfer.

If you are transferring large files over high-speed links and you have enough memory, then increasing this will speed up the transfers.

Rclone will automatically increase the chunk size when uploading a large file of known size to stay below the 10,000 chunks limit.

Files of unknown size are uploaded with the configured chunk_size. Since the default chunk size is 5 MiB and there can be at most 10,000 chunks, this means that by default the maximum size of a file you can stream upload is 48 GiB. If you wish to stream upload larger files then you will need to increase chunk_size.

Increasing the chunk size decreases the accuracy of the progress statistics displayed with "-P" flag. Rclone treats chunk as sent when it's buffered by the AWS SDK, when in fact it may still be uploading. A bigger chunk size means a bigger AWS SDK buffer and progress reporting more deviating from the truth.

Properties:

Config: chunk_size
Env Var: RCLONE_S3_CHUNK_SIZE
Type: SizeSuffix
Default: 5Mi

--s3-max-upload-parts

Maximum number of parts in a multipart upload.

This option defines the maximum number of multipart chunks to use when doing a multipart upload.

This can be useful if a service does not support the AWS S3 specification of 10,000 chunks.

Rclone will automatically increase the chunk size when uploading a large file of a known size to stay below this number of chunks limit.

Properties:

Config: max_upload_parts
Env Var: RCLONE_S3_MAX_UPLOAD_PARTS
Type: int
Default: 10000

--s3-copy-cutoff

Cutoff for switching to multipart copy.

Any files larger than this that need to be server-side copied will be copied in chunks of this size.

The minimum is 0 and the maximum is 5 GiB.

Properties:

Config: copy_cutoff
Env Var: RCLONE_S3_COPY_CUTOFF
Type: SizeSuffix
Default: 4.656Gi

--s3-disable-checksum

Don't store MD5 checksum with object metadata.

Normally rclone will calculate the MD5 checksum of the input before uploading it so it can add it to metadata on the object. This is great for data integrity checking but can cause long delays for large files to start uploading.

Properties:

Config: disable_checksum
Env Var: RCLONE_S3_DISABLE_CHECKSUM
Type: bool
Default: false

--s3-shared-credentials-file

Path to the shared credentials file.

If env_auth = true then rclone can use a shared credentials file.

If this variable is empty rclone will look for the "AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty it will default to the current user's home directory.

Linux/OSX: "$HOME/.aws/credentials"
Windows:   "%USERPROFILE%\.aws\credentials"

Properties:

Config: shared_credentials_file
Env Var: RCLONE_S3_SHARED_CREDENTIALS_FILE
Type: string
Required: false

--s3-profile

Profile to use in the shared credentials file.

If env_auth = true then rclone can use a shared credentials file. This variable controls which profile is used in that file.

If empty it will default to the environment variable "AWS_PROFILE" or "default" if that environment variable is also not set.

Properties:

Config: profile
Env Var: RCLONE_S3_PROFILE
Type: string
Required: false

--s3-session-token

An AWS session token.

Properties:

Config: session_token
Env Var: RCLONE_S3_SESSION_TOKEN
Type: string
Required: false

--s3-upload-concurrency

Concurrency for multipart uploads and copies.

This is the number of chunks of the same file that are uploaded concurrently for multipart uploads and copies.

If you are uploading small numbers of large files over high-speed links and these uploads do not fully utilize your bandwidth, then increasing this may help to speed up the transfers.

Properties:

Config: upload_concurrency
Env Var: RCLONE_S3_UPLOAD_CONCURRENCY
Type: int
Default: 4

--s3-force-path-style

If true use path style access if false use virtual hosted style.

If this is true (the default) then rclone will use path style access, if false then rclone will use virtual path style. See the AWS S3 docs for more info.

Some providers (e.g. AWS, Aliyun OSS, Netease COS, or Tencent COS) require this set to false - rclone will do this automatically based on the provider setting.

Note that if your bucket isn't a valid DNS name, i.e. has '.' or '_' in, you'll need to set this to true.

Properties:

Config: force_path_style
Env Var: RCLONE_S3_FORCE_PATH_STYLE
Type: bool
Default: true

--s3-v2-auth

If true use v2 authentication.

If this is false (the default) then rclone will use v4 authentication. If it is set then rclone will use v2 authentication.

Use this only if v4 signatures don't work, e.g. pre Jewel/v10 CEPH.

Properties:

Config: v2_auth
Env Var: RCLONE_S3_V2_AUTH
Type: bool
Default: false

--s3-use-dual-stack

If true use AWS S3 dual-stack endpoint (IPv6 support).

See AWS Docs on Dualstack Endpoints

Properties:

Config: use_dual_stack
Env Var: RCLONE_S3_USE_DUAL_STACK
Type: bool
Default: false

--s3-use-accelerate-endpoint

If true use the AWS S3 accelerated endpoint.

See: AWS S3 Transfer acceleration

Properties:

Config: use_accelerate_endpoint
Env Var: RCLONE_S3_USE_ACCELERATE_ENDPOINT
Provider: AWS
Type: bool
Default: false

--s3-leave-parts-on-error

If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.

It should be set to true for resuming uploads across different sessions.

WARNING: Storing parts of an incomplete multipart upload counts towards space usage on S3 and will add additional costs if not cleaned up.

Properties:

Config: leave_parts_on_error
Env Var: RCLONE_S3_LEAVE_PARTS_ON_ERROR
Provider: AWS
Type: bool
Default: false

--s3-list-chunk

Size of listing chunk (response list for each ListObject S3 request).

This option is also known as "MaxKeys", "max-items", or "page-size" from the AWS S3 specification. Most services truncate the response list to 1000 objects even if requested more than that. In AWS S3 this is a global maximum and cannot be changed, see AWS S3. In Ceph, this can be increased with the "rgw list buckets max chunk" option.

Properties:

Config: list_chunk
Env Var: RCLONE_S3_LIST_CHUNK
Type: int
Default: 1000

--s3-list-version

Version of ListObjects to use: 1,2 or 0 for auto.

When S3 originally launched it only provided the ListObjects call to enumerate objects in a bucket.

However in May 2016 the ListObjectsV2 call was introduced. This is much higher performance and should be used if at all possible.

If set to the default, 0, rclone will guess according to the provider set which list objects method to call. If it guesses wrong, then it may be set manually here.

Properties:

Config: list_version
Env Var: RCLONE_S3_LIST_VERSION
Type: int
Default: 0

--s3-list-url-encode

Whether to url encode listings: true/false/unset

Some providers support URL encoding listings and where this is available this is more reliable when using control characters in file names. If this is set to unset (the default) then rclone will choose according to the provider setting what to apply, but you can override rclone's choice here.

Properties:

Config: list_url_encode
Env Var: RCLONE_S3_LIST_URL_ENCODE
Type: Tristate
Default: unset

--s3-no-check-bucket

If set, don't attempt to check the bucket exists or create it.

This can be useful when trying to minimise the number of transactions rclone does if you know the bucket exists already.

It can also be needed if the user you are using does not have bucket creation permissions. Before v1.52.0 this would have passed silently due to a bug.

Properties:

Config: no_check_bucket
Env Var: RCLONE_S3_NO_CHECK_BUCKET
Type: bool
Default: false

--s3-no-head

If set, don't HEAD uploaded objects to check integrity.

This can be useful when trying to minimise the number of transactions rclone does.

Setting it means that if rclone receives a 200 OK message after uploading an object with PUT then it will assume that it got uploaded properly.

In particular it will assume:

the metadata, including modtime, storage class and content type was as uploaded
the size was as uploaded

It reads the following items from the response for a single part PUT:

the MD5SUM
The uploaded date

For multipart uploads these items aren't read.

If an source object of unknown length is uploaded then rclone will do a HEAD request.

Setting this flag increases the chance for undetected upload failures, in particular an incorrect size, so it isn't recommended for normal operation. In practice the chance of an undetected upload failure is very small even with this flag.

Properties:

Config: no_head
Env Var: RCLONE_S3_NO_HEAD
Type: bool
Default: false

--s3-no-head-object

If set, do not do HEAD before GET when getting objects.

Properties:

Config: no_head_object
Env Var: RCLONE_S3_NO_HEAD_OBJECT
Type: bool
Default: false

--s3-encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Properties:

Config: encoding
Env Var: RCLONE_S3_ENCODING
Type: Encoding
Default: Slash,InvalidUtf8,Dot

--s3-memory-pool-flush-time

How often internal memory buffer pools will be flushed. (no longer used)

Properties:

Config: memory_pool_flush_time
Env Var: RCLONE_S3_MEMORY_POOL_FLUSH_TIME
Type: Duration
Default: 1m0s

--s3-memory-pool-use-mmap

Whether to use mmap buffers in internal memory pool. (no longer used)

Properties:

Config: memory_pool_use_mmap
Env Var: RCLONE_S3_MEMORY_POOL_USE_MMAP
Type: bool
Default: false

--s3-disable-http2

Disable usage of http2 for S3 backends.

There is currently an unsolved issue with the s3 (specifically minio) backend and HTTP/2. HTTP/2 is enabled by default for the s3 backend but can be disabled here. When the issue is solved this flag will be removed.

See: https://github.com/rclone/rclone/issues/4673, https://github.com/rclone/rclone/issues/3631

Properties:

Config: disable_http2
Env Var: RCLONE_S3_DISABLE_HTTP2
Type: bool
Default: false

--s3-download-url

Custom endpoint for downloads. This is usually set to a CloudFront CDN URL as AWS S3 offers cheaper egress for data downloaded through the CloudFront network.

Properties:

Config: download_url
Env Var: RCLONE_S3_DOWNLOAD_URL
Type: string
Required: false

--s3-directory-markers

Upload an empty object with a trailing slash when a new directory is created

Empty folders are unsupported for bucket based remotes, this option creates an empty object ending with "/", to persist the folder.

Properties:

Config: directory_markers
Env Var: RCLONE_S3_DIRECTORY_MARKERS
Type: bool
Default: false

--s3-use-multipart-etag

Whether to use ETag in multipart uploads for verification

This should be true, false or left unset to use the default for the provider.

Properties:

Config: use_multipart_etag
Env Var: RCLONE_S3_USE_MULTIPART_ETAG
Type: Tristate
Default: unset

--s3-use-unsigned-payload

Whether to use an unsigned payload in PutObject

Rclone has to avoid the AWS SDK seeking the body when calling PutObject. The AWS provider can add checksums in the trailer to avoid seeking but other providers can't.

This should be true, false or left unset to use the default for the provider.

Properties:

Config: use_unsigned_payload
Env Var: RCLONE_S3_USE_UNSIGNED_PAYLOAD
Type: Tristate
Default: unset

--s3-use-presigned-request

Whether to use a presigned request or PutObject for single part uploads

If this is false rclone will use PutObject from the AWS SDK to upload an object.

Versions of rclone < 1.59 use presigned requests to upload a single part object and setting this flag to true will re-enable that functionality. This shouldn't be necessary except in exceptional circumstances or for testing.

Properties:

Config: use_presigned_request
Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST
Type: bool
Default: false

--s3-versions

Include old versions in directory listings.

Properties:

Config: versions
Env Var: RCLONE_S3_VERSIONS
Type: bool
Default: false

--s3-version-at

Show file versions as they were at the specified time.

The parameter should be a date, "2006-01-02", datetime "2006-01-02 15:04:05" or a duration for that long ago, eg "100d" or "1h".

Note that when using this no file write operations are permitted, so you can't upload files or delete them.

See the time option docs for valid formats.

Properties:

Config: version_at
Env Var: RCLONE_S3_VERSION_AT
Type: Time
Default: off

--s3-version-deleted

Show deleted file markers when using versions.

This shows deleted file markers in the listing when using versions. These will appear as 0 size files. The only operation which can be performed on them is deletion.

Deleting a delete marker will reveal the previous version.

Deleted files will always show with a timestamp.

Properties:

Config: version_deleted
Env Var: RCLONE_S3_VERSION_DELETED
Type: bool
Default: false

--s3-decompress

If set this will decompress gzip encoded objects.

It is possible to upload objects to S3 with "Content-Encoding: gzip" set. Normally rclone will download these files as compressed objects.

If this flag is set then rclone will decompress these files with "Content-Encoding: gzip" as they are received. This means that rclone can't check the size and hash but the file contents will be decompressed.

Properties:

Config: decompress
Env Var: RCLONE_S3_DECOMPRESS
Type: bool
Default: false

--s3-might-gzip

Set this if the backend might gzip objects.

Normally providers will not alter objects when they are downloaded. If an object was not uploaded with Content-Encoding: gzip then it won't be set on download.

However some providers may gzip objects even if they weren't uploaded with Content-Encoding: gzip (eg Cloudflare).

A symptom of this would be receiving errors like

ERROR corrupted on transfer: sizes differ NNN vs MMM

If you set this flag and rclone downloads an object with Content-Encoding: gzip set and chunked transfer encoding, then rclone will decompress the object on the fly.

If this is set to unset (the default) then rclone will choose according to the provider setting what to apply, but you can override rclone's choice here.

Properties:

Config: might_gzip
Env Var: RCLONE_S3_MIGHT_GZIP
Type: Tristate
Default: unset

--s3-use-accept-encoding-gzip

Whether to send Accept-Encoding: gzip header.

By default, rclone will append Accept-Encoding: gzip to the request to download compressed objects whenever possible.

However some providers such as Google Cloud Storage may alter the HTTP headers, breaking the signature of the request.

A symptom of this would be receiving errors like

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided.

In this case, you might want to try disabling this option.

Properties:

Config: use_accept_encoding_gzip
Env Var: RCLONE_S3_USE_ACCEPT_ENCODING_GZIP
Type: Tristate
Default: unset

--s3-no-system-metadata

Suppress setting and reading of system metadata

Properties:

Config: no_system_metadata
Env Var: RCLONE_S3_NO_SYSTEM_METADATA
Type: bool
Default: false

--s3-sts-endpoint

Endpoint for STS (deprecated).

Leave blank if using AWS to use the default endpoint for the region.

Properties:

Config: sts_endpoint
Env Var: RCLONE_S3_STS_ENDPOINT
Provider: AWS
Type: string
Required: false

--s3-use-already-exists

Set if rclone should report BucketAlreadyExists errors on bucket creation.

At some point during the evolution of the s3 protocol, AWS started returning an AlreadyOwnedByYou error when attempting to create a bucket that the user already owned, rather than a BucketAlreadyExists error.

Unfortunately exactly what has been implemented by s3 clones is a little inconsistent, some return AlreadyOwnedByYou, some return BucketAlreadyExists and some return no error at all.

This is important to rclone because it ensures the bucket exists by creating it on quite a lot of operations (unless --s3-no-check-bucket is used).

If rclone knows the provider can return AlreadyOwnedByYou or returns no error then it can report BucketAlreadyExists errors when the user attempts to create a bucket not owned by them. Otherwise rclone ignores the BucketAlreadyExists error which can lead to confusion.

This should be automatically set correctly for all providers rclone knows about - please make a bug report if not.

Properties:

Config: use_already_exists
Env Var: RCLONE_S3_USE_ALREADY_EXISTS
Type: Tristate
Default: unset

--s3-use-multipart-uploads

Set if rclone should use multipart uploads.

You can change this if you want to disable the use of multipart uploads. This shouldn't be necessary in normal operation.

This should be automatically set correctly for all providers rclone knows about - please make a bug report if not.

Properties:

Config: use_multipart_uploads
Env Var: RCLONE_S3_USE_MULTIPART_UPLOADS
Type: Tristate
Default: unset

--s3-sdk-log-mode

Set to debug the SDK

This can be set to a comma separated list of the following functions:

Signing
Retries
Request
RequestWithBody
Response
ResponseWithBody
DeprecatedUsage
RequestEventMessage
ResponseEventMessage

Use Off to disable and All to set all log levels. You will need to use -vv to see the debug level logs.

Properties:

Config: sdk_log_mode
Env Var: RCLONE_S3_SDK_LOG_MODE
Type: Bits
Default: Off

--s3-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_S3_DESCRIPTION
Type: string
Required: false

Metadata

User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.

Here are the possible system metadata items for the s3 backend.

Name	Help	Type	Example	Read Only
btime	Time of file birth (creation) read from Last-Modified header	RFC 3339	2006-01-02T15:04:05.999999999Z07:00	Y
cache-control	Cache-Control header	string	no-cache	N
content-disposition	Content-Disposition header	string	inline	N
content-encoding	Content-Encoding header	string	gzip	N
content-language	Content-Language header	string	en-US	N
content-type	Content-Type header	string	text/plain	N
mtime	Time of last modification, read from rclone metadata	RFC 3339	2006-01-02T15:04:05.999999999Z07:00	N
tier	Tier of the object	string	GLACIER	Y

See the metadata docs for more info.

Backend commands

Here are the commands specific to the s3 backend.

Run them with

rclone backend COMMAND remote:

The help below will explain what arguments each command takes.

See the backend command for more info on how to pass options and arguments.

These can be run on a running backend using the rc command backend/command.

restore

Restore objects from GLACIER or INTELLIGENT-TIERING archive tier

rclone backend restore remote: [options] [<arguments>+]

This command can be used to restore one or more objects from GLACIER to normal storage or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier.

Usage Examples:

rclone backend restore s3:bucket/path/to/object -o priority=PRIORITY -o lifetime=DAYS
rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY -o lifetime=DAYS
rclone backend restore s3:bucket -o priority=PRIORITY -o lifetime=DAYS
rclone backend restore s3:bucket/path/to/directory -o priority=PRIORITY

This flag also obeys the filters. Test first with --interactive/-i or --dry-run flags

rclone --interactive backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1

All the objects shown will be marked for restore, then

rclone backend restore --include "*.txt" s3:bucket/path -o priority=Standard -o lifetime=1

It returns a list of status dictionaries with Remote and Status keys. The Status will be OK if it was successful or an error message if not.

[
    {
        "Status": "OK",
        "Remote": "test.txt"
    },
    {
        "Status": "OK",
        "Remote": "test/file4.txt"
    }
]

Options:

"description": The optional description for the job.
"lifetime": Lifetime of the active copy in days, ignored for INTELLIGENT-TIERING storage
"priority": Priority of restore: Standard|Expedited|Bulk

restore-status

Show the restore status for objects being restored from GLACIER or INTELLIGENT-TIERING storage

rclone backend restore-status remote: [options] [<arguments>+]

This command can be used to show the status for objects being restored from GLACIER to normal storage or from INTELLIGENT-TIERING Archive Access / Deep Archive Access tier to the Frequent Access tier.

Usage Examples:

rclone backend restore-status s3:bucket/path/to/object
rclone backend restore-status s3:bucket/path/to/directory
rclone backend restore-status -o all s3:bucket/path/to/directory

This command does not obey the filters.

It returns a list of status dictionaries.

[
    {
        "Remote": "file.txt",
        "VersionID": null,
        "RestoreStatus": {
            "IsRestoreInProgress": true,
            "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
        },
        "StorageClass": "GLACIER"
    },
    {
        "Remote": "test.pdf",
        "VersionID": null,
        "RestoreStatus": {
            "IsRestoreInProgress": false,
            "RestoreExpiryDate": "2023-09-06T12:29:19+01:00"
        },
        "StorageClass": "DEEP_ARCHIVE"
    },
    {
        "Remote": "test.gz",
        "VersionID": null,
        "RestoreStatus": {
            "IsRestoreInProgress": true,
            "RestoreExpiryDate": "null"
        },
        "StorageClass": "INTELLIGENT_TIERING"
    }
]

Options:

"all": if set then show all objects, not just ones with restore status

list-multipart-uploads

List the unfinished multipart uploads

rclone backend list-multipart-uploads remote: [options] [<arguments>+]

This command lists the unfinished multipart uploads in JSON format.

rclone backend list-multipart s3:bucket/path/to/object

It returns a dictionary of buckets with values as lists of unfinished multipart uploads.

You can call it with no bucket in which case it lists all bucket, with a bucket or with a bucket and path.

{
  "rclone": [
    {
      "Initiated": "2020-06-26T14:20:36Z",
      "Initiator": {
        "DisplayName": "XXX",
        "ID": "arn:aws:iam::XXX:user/XXX"
      },
      "Key": "KEY",
      "Owner": {
        "DisplayName": null,
        "ID": "XXX"
      },
      "StorageClass": "STANDARD",
      "UploadId": "XXX"
    }
  ],
  "rclone-1000files": [],
  "rclone-dst": []
}

cleanup

Remove unfinished multipart uploads.

rclone backend cleanup remote: [options] [<arguments>+]

This command removes unfinished multipart uploads of age greater than max-age which defaults to 24 hours.

Note that you can use --interactive/-i or --dry-run with this command to see what it would do.

rclone backend cleanup s3:bucket/path/to/object
rclone backend cleanup -o max-age=7w s3:bucket/path/to/object

Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.

Options:

"max-age": Max age of upload to delete

cleanup-hidden

Remove old versions of files.

rclone backend cleanup-hidden remote: [options] [<arguments>+]

This command removes any old hidden versions of files on a versions enabled bucket.

Note that you can use --interactive/-i or --dry-run with this command to see what it would do.

rclone backend cleanup-hidden s3:bucket/path/to/dir

versioning

Set/get versioning support for a bucket.

rclone backend versioning remote: [options] [<arguments>+]

This command sets versioning support if a parameter is passed and then returns the current versioning status for the bucket supplied.

rclone backend versioning s3:bucket # read status only
rclone backend versioning s3:bucket Enabled
rclone backend versioning s3:bucket Suspended

It may return "Enabled", "Suspended" or "Unversioned". Note that once versioning has been enabled the status can't be set back to "Unversioned".

set

Set command for updating the config parameters.

rclone backend set remote: [options] [<arguments>+]

This set command can be used to update the config parameters for a running s3 backend.

Usage Examples:

rclone backend set s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=s3: [-o opt_name=opt_value] [-o opt_name2=opt_value2]
rclone rc backend/command command=set fs=s3: -o session_token=X -o access_key_id=X -o secret_access_key=X

The option keys are named as they are in the config file.

This rebuilds the connection to the s3 backend when it is called with the new parameters. Only new parameters need be passed as the values will default to those currently in use.

It doesn't return anything.

Anonymous access to public buckets

If you want to use rclone to access a public bucket, configure with a blank access_key_id and secret_access_key. Your config should end up looking like this:

[anons3]
type = s3
provider = AWS

Then use it as normal with the name of the public bucket, e.g.

rclone lsd anons3:1000genomes

You will be able to list and copy data but not upload it.

You can also do this entirely on the command line

rclone lsd :s3,provider=AWS:1000genomes

Providers

AWS S3

This is the provider used as main example and described in the configuration section above.

AWS Snowball Edge

AWS Snowball is a hardware appliance used for transferring bulk data back to AWS. Its main software interface is S3 object storage.

To use rclone with AWS Snowball Edge devices, configure as standard for an 'S3 Compatible Service'.

If using rclone pre v1.59 be sure to set upload_cutoff = 0 otherwise you will run into authentication header issues as the snowball device does not support query parameter based authentication.

With rclone v1.59 or later setting upload_cutoff should not be necessary.

eg.

[snowball]
type = s3
provider = Other
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
endpoint = http://[IP of Snowball]:8080
upload_cutoff = 0

Ceph

Ceph is an open-source, unified, distributed storage system designed for excellent performance, reliability and scalability. It has an S3 compatible object storage interface.

To use rclone with Ceph, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:

[ceph]
type = s3
provider = Ceph
env_auth = false
access_key_id = XXX
secret_access_key = YYY
region =
endpoint = https://ceph.endpoint.example.com
location_constraint =
acl =
server_side_encryption =
storage_class =

If you are using an older version of CEPH (e.g. 10.2.x Jewel) and a version of rclone before v1.59 then you may need to supply the parameter --s3-upload-cutoff 0 or put this in the config file as upload_cutoff 0 to work around a bug which causes uploading of small files to fail.

Note also that Ceph sometimes puts / in the passwords it gives users. If you read the secret access key using the command line tools you will get a JSON blob with the / escaped as \/. Make sure you only write / in the secret access key.

Eg the dump from Ceph looks something like this (irrelevant keys removed).

{
    "user_id": "xxx",
    "display_name": "xxxx",
    "keys": [
        {
            "user": "xxx",
            "access_key": "xxxxxx",
            "secret_key": "xxxxxx\/xxxx"
        }
    ],
}

Because this is a json dump, it is encoding the / as \/, so if you use the secret key as xxxxxx/xxxx it will work fine.

Cloudflare R2

Cloudflare R2 Storage allows developers to store large amounts of unstructured data without the costly egress bandwidth fees associated with typical cloud storage services.

Here is an example of making a Cloudflare R2 configuration. First run:

rclone config

This will guide you through an interactive setup process.

Note that all buckets are private, and all are stored in the same "auto" region. It is necessary to use Cloudflare workers to share the content of a bucket publicly.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> r2
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
...
XX / Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, DigitalOcean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, Magalu, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Synology, Tencent COS and Wasabi
   \ (s3)
...
Storage> s3
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
...
XX / Cloudflare R2 Storage
   \ (Cloudflare)
...
provider> Cloudflare
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> ACCESS_KEY
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> SECRET_ACCESS_KEY
Option region.
Region to connect to.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
   \ (auto)
region> 1
Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a value. Press Enter to leave empty.
endpoint> https://ACCOUNT_ID.r2.cloudflarestorage.com
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave your config looking something like:

[r2]
type = s3
provider = Cloudflare
access_key_id = ACCESS_KEY
secret_access_key = SECRET_ACCESS_KEY
region = auto
endpoint = https://ACCOUNT_ID.r2.cloudflarestorage.com
acl = private

Now run rclone lsf r2: to see your buckets and rclone lsf r2:bucket to look within a bucket.

For R2 tokens with the "Object Read & Write" permission, you may also need to add no_check_bucket = true for object uploads to work correctly.

Note that Cloudflare decompresses files uploaded with Content-Encoding: gzip by default which is a deviation from what AWS does. If this is causing a problem then upload the files with --header-upload "Cache-Control: no-transform"

Dreamhost

Dreamhost DreamObjects is an object storage system based on CEPH.

To use rclone with Dreamhost, configure as above but leave the region blank and set the endpoint. You should end up with something like this in your config:

[dreamobjects]
type = s3
provider = DreamHost
env_auth = false
access_key_id = your_access_key
secret_access_key = your_secret_key
region =
endpoint = objects-us-west-1.dream.io
location_constraint =
acl = private
server_side_encryption =
storage_class =

Google Cloud Storage

GoogleCloudStorage is an S3-interoperable object storage service from Google Cloud Platform.

To connect to Google Cloud Storage you will need an access key and secret key. These can be retrieved by creating an HMAC key.

[gs]
type = s3
provider = GCS
access_key_id = your_access_key
secret_access_key = your_secret_key
endpoint = https://storage.googleapis.com

Note that --s3-versions does not work with GCS when it needs to do directory paging. Rclone will return the error:

s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker

This is Google bug #312292516.

DigitalOcean Spaces

Spaces is an S3-interoperable object storage service from cloud provider DigitalOcean.

To connect to DigitalOcean Spaces you will need an access key and secret key. These can be retrieved on the "Applications & API" page of the DigitalOcean control panel. They will be needed when prompted by rclone config for your access_key_id and secret_access_key.

When prompted for a region or location_constraint, press enter to use the default value. The region must be included in the endpoint setting (e.g. nyc3.digitaloceanspaces.com). The default values can be used for other settings.

Going through the whole process of creating a new remote by running rclone config, each prompt should be answered as shown below:

Storage> s3
env_auth> 1
access_key_id> YOUR_ACCESS_KEY
secret_access_key> YOUR_SECRET_KEY
region>
endpoint> nyc3.digitaloceanspaces.com
location_constraint>
acl>
storage_class>

The resulting configuration file should look like:

[spaces]
type = s3
provider = DigitalOcean
env_auth = false
access_key_id = YOUR_ACCESS_KEY
secret_access_key = YOUR_SECRET_KEY
region =
endpoint = nyc3.digitaloceanspaces.com
location_constraint =
acl =
server_side_encryption =
storage_class =

Once configured, you can create a new Space and begin copying files. For example:

rclone mkdir spaces:my-new-space
rclone copy /path/to/files spaces:my-new-space

Huawei OBS

Object Storage Service (OBS) provides stable, secure, efficient, and easy-to-use cloud storage that lets you store virtually any volume of unstructured data in any format and access it from anywhere.

OBS provides an S3 interface, you can copy and modify the following configuration and add it to your rclone configuration file.

[obs]
type = s3
provider = HuaweiOBS
access_key_id = your-access-key-id
secret_access_key = your-secret-access-key
region = af-south-1
endpoint = obs.af-south-1.myhuaweicloud.com
acl = private

Or you can also configure via the interactive command line:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> obs
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
[snip]
Storage> s3
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
 9 / Huawei Object Storage Service
   \ (HuaweiOBS)
[snip]
provider> 9
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> your-access-key-id
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> your-secret-access-key
Option region.
Region to connect to.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / AF-Johannesburg
   \ (af-south-1)
 2 / AP-Bangkok
   \ (ap-southeast-2)
[snip]
region> 1
Option endpoint.
Endpoint for OBS API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / AF-Johannesburg
   \ (obs.af-south-1.myhuaweicloud.com)
 2 / AP-Bangkok
   \ (obs.ap-southeast-2.myhuaweicloud.com)
[snip]
endpoint> 1
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
[snip]
acl> 1
Edit advanced config?
y) Yes
n) No (default)
y/n>
--------------------
[obs]
type = s3
provider = HuaweiOBS
access_key_id = your-access-key-id
secret_access_key = your-secret-access-key
region = af-south-1
endpoint = obs.af-south-1.myhuaweicloud.com
acl = private
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
obs                  s3

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> q

IBM COS (S3)

Information stored with IBM Cloud Object Storage is encrypted and dispersed across multiple geographic locations, and accessed through an implementation of the S3 API. This service makes use of the distributed storage technologies provided by IBM’s Cloud Object Storage System (formerly Cleversafe). For more information visit: (http://www.ibm.com/cloud/object-storage)

To configure access to IBM COS S3, follow the steps below:

Run rclone config and select n for a new remote.

	2018/02/14 14:13:11 NOTICE: Config file "C:\\Users\\a\\.config\\rclone\\rclone.conf" not found - using defaults
	No remotes found, make a new one?
	n) New remote
	s) Set configuration password
	q) Quit config
	n/s/q> n

Enter the name for the configuration

	name> <YOUR NAME>

Select "s3" storage.

Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ "s3"
[snip]
Storage> s3

Select IBM COS as the S3 Storage Provider.

Choose the S3 provider.
Choose a number from below, or type in your own value
	 1 / Choose this option to configure Storage to AWS S3
	   \ "AWS"
 	 2 / Choose this option to configure Storage to Ceph Systems
  	 \ "Ceph"
	 3 /  Choose this option to configure Storage to Dreamhost
     \ "Dreamhost"
   4 / Choose this option to the configure Storage to IBM COS S3
   	 \ "IBMCOS"
 	 5 / Choose this option to the configure Storage to Minio
     \ "Minio"
	 Provider>4

Enter the Access Key and Secret.

	AWS Access Key ID - leave blank for anonymous access or runtime credentials.
	access_key_id> <>
	AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
	secret_access_key> <>

Specify the endpoint for IBM COS. For Public IBM COS, choose from the option below. For On Premise IBM COS, enter an endpoint address.

	Endpoint for IBM COS S3 API.
	Specify if using an IBM COS On Premise.
	Choose a number from below, or type in your own value
	 1 / US Cross Region Endpoint
   	   \ "s3-api.us-geo.objectstorage.softlayer.net"
	 2 / US Cross Region Dallas Endpoint
   	   \ "s3-api.dal.us-geo.objectstorage.softlayer.net"
 	 3 / US Cross Region Washington DC Endpoint
   	   \ "s3-api.wdc-us-geo.objectstorage.softlayer.net"
	 4 / US Cross Region San Jose Endpoint
	   \ "s3-api.sjc-us-geo.objectstorage.softlayer.net"
	 5 / US Cross Region Private Endpoint
	   \ "s3-api.us-geo.objectstorage.service.networklayer.com"
	 6 / US Cross Region Dallas Private Endpoint
	   \ "s3-api.dal-us-geo.objectstorage.service.networklayer.com"
	 7 / US Cross Region Washington DC Private Endpoint
	   \ "s3-api.wdc-us-geo.objectstorage.service.networklayer.com"
	 8 / US Cross Region San Jose Private Endpoint
	   \ "s3-api.sjc-us-geo.objectstorage.service.networklayer.com"
	 9 / US Region East Endpoint
	   \ "s3.us-east.objectstorage.softlayer.net"
	10 / US Region East Private Endpoint
	   \ "s3.us-east.objectstorage.service.networklayer.com"
	11 / US Region South Endpoint
[snip]
	34 / Toronto Single Site Private Endpoint
	   \ "s3.tor01.objectstorage.service.networklayer.com"
	endpoint>1

Specify a IBM COS Location Constraint. The location constraint must match endpoint when using IBM Cloud Public. For on-prem COS, do not make a selection from this list, hit enter

	 1 / US Cross Region Standard
	   \ "us-standard"
	 2 / US Cross Region Vault
	   \ "us-vault"
	 3 / US Cross Region Cold
	   \ "us-cold"
	 4 / US Cross Region Flex
	   \ "us-flex"
	 5 / US East Region Standard
	   \ "us-east-standard"
	 6 / US East Region Vault
	   \ "us-east-vault"
	 7 / US East Region Cold
	   \ "us-east-cold"
	 8 / US East Region Flex
	   \ "us-east-flex"
	 9 / US South Region Standard
	   \ "us-south-standard"
	10 / US South Region Vault
	   \ "us-south-vault"
[snip]
	32 / Toronto Flex
	   \ "tor01-flex"
location_constraint>1

Specify a canned ACL. IBM Cloud (Storage) supports "public-read" and "private". IBM Cloud(Infra) supports all the canned ACLs. On-Premise COS supports all the canned ACLs.

Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
      1 / Owner gets FULL_CONTROL. No one else has access rights (default). This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS
      \ "private"
      2  / Owner gets FULL_CONTROL. The AllUsers group gets READ access. This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS
      \ "public-read"
      3 / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access. This acl is available on IBM Cloud (Infra), On-Premise IBM COS
      \ "public-read-write"
      4  / Owner gets FULL_CONTROL. The AuthenticatedUsers group gets READ access. Not supported on Buckets. This acl is available on IBM Cloud (Infra) and On-Premise IBM COS
      \ "authenticated-read"
acl> 1

Review the displayed configuration and accept to save the "remote" then quit. The config file should look like this

	[xxx]
	type = s3
	Provider = IBMCOS
	access_key_id = xxx
	secret_access_key = yyy
	endpoint = s3-api.us-geo.objectstorage.softlayer.net
	location_constraint = us-standard
	acl = private

Execute rclone commands

	1)	Create a bucket.
		rclone mkdir IBM-COS-XREGION:newbucket
	2)	List available buckets.
		rclone lsd IBM-COS-XREGION:
		-1 2017-11-08 21:16:22        -1 test
		-1 2018-02-14 20:16:39        -1 newbucket
	3)	List contents of a bucket.
		rclone ls IBM-COS-XREGION:newbucket
		18685952 test.exe
	4)	Copy a file from local to remote.
		rclone copy /Users/file.txt IBM-COS-XREGION:newbucket
	5)	Copy a file from remote to local.
		rclone copy IBM-COS-XREGION:newbucket/file.txt .
	6)	Delete a file on remote.
		rclone delete IBM-COS-XREGION:newbucket/file.txt

IDrive e2

Here is an example of making an IDrive e2 configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> e2

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
[snip]
Storage> s3

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
XX / IDrive e2
   \ (IDrive)
[snip]
provider> IDrive

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> YOUR_ACCESS_KEY

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> YOUR_SECRET_KEY

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
   / Owner gets FULL_CONTROL.
 2 | The AllUsers group gets READ access.
   \ (public-read)
   / Owner gets FULL_CONTROL.
 3 | The AllUsers group gets READ and WRITE access.
   | Granting this on a bucket is generally not recommended.
   \ (public-read-write)
   / Owner gets FULL_CONTROL.
 4 | The AuthenticatedUsers group gets READ access.
   \ (authenticated-read)
   / Object owner gets FULL_CONTROL.
 5 | Bucket owner gets READ access.
   | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ (bucket-owner-read)
   / Both the object owner and the bucket owner get FULL_CONTROL over the object.
 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ (bucket-owner-full-control)
acl> 

Edit advanced config?
y) Yes
n) No (default)
y/n> 

Configuration complete.
Options:
- type: s3
- provider: IDrive
- access_key_id: YOUR_ACCESS_KEY
- secret_access_key: YOUR_SECRET_KEY
- endpoint: q9d9.la12.idrivee2-5.com
Keep this "e2" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

IONOS Cloud

IONOS S3 Object Storage is a service offered by IONOS for storing and accessing unstructured data. To connect to the service, you will need an access key and a secret key. These can be found in the Data Center Designer, by selecting Manager resources > Object Storage Key Manager.

Here is an example of a configuration. First, run rclone config. This will walk you through an interactive setup process. Type n to add the new remote, and then enter a name:

Enter name for new remote.
name> ionos-fra

Type s3 to choose the connection type:

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
[snip]
Storage> s3

Type IONOS:

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
XX / IONOS Cloud
   \ (IONOS)
[snip]
provider> IONOS

Press Enter to choose the default option Enter AWS credentials in the next step:

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth>

Enter your Access Key and Secret key. These can be retrieved in the Data Center Designer, click on the menu “Manager resources” / "Object Storage Key Manager".

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> YOUR_ACCESS_KEY

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> YOUR_SECRET_KEY

Choose the region where your bucket is located:

Option region.
Region where your bucket will be created and your data stored.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Frankfurt, Germany
   \ (de)
 2 / Berlin, Germany
   \ (eu-central-2)
 3 / Logrono, Spain
   \ (eu-south-2)
region> 2

Choose the endpoint from the same region:

Option endpoint.
Endpoint for IONOS S3 Object Storage.
Specify the endpoint from the same region.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Frankfurt, Germany
   \ (s3-eu-central-1.ionoscloud.com)
 2 / Berlin, Germany
   \ (s3-eu-central-2.ionoscloud.com)
 3 / Logrono, Spain
   \ (s3-eu-south-2.ionoscloud.com)
endpoint> 1

Press Enter to choose the default option or choose the desired ACL setting:

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
   / Owner gets FULL_CONTROL.
[snip]
acl>

Press Enter to skip the advanced config:

Edit advanced config?
y) Yes
n) No (default)
y/n>

Press Enter to save the configuration, and then q to quit the configuration process:

Configuration complete.
Options:
- type: s3
- provider: IONOS
- access_key_id: YOUR_ACCESS_KEY
- secret_access_key: YOUR_SECRET_KEY
- endpoint: s3-eu-central-1.ionoscloud.com
Keep this "ionos-fra" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Done! Now you can try some commands (for macOS, use ./rclone instead of rclone).

Create a bucket (the name must be unique within the whole IONOS S3)

rclone mkdir ionos-fra:my-bucket

List available buckets

rclone lsd ionos-fra:

Copy a file from local to remote

rclone copy /Users/file.txt ionos-fra:my-bucket

List contents of a bucket

rclone ls ionos-fra:my-bucket

Copy a file from remote to local

rclone copy ionos-fra:my-bucket/file.txt

Minio

Minio is an object storage server built for cloud application developers and devops.

It is very easy to install and provides an S3 compatible server which can be used by rclone.

To use it, install Minio following the instructions here.

When it configures itself Minio will print something like this

Endpoint:  http://192.168.1.106:9000  http://172.23.0.1:9000
AccessKey: USWUXHGYZQYFYFFIT3RE
SecretKey: MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
Region:    us-east-1
SQS ARNs:  arn:minio:sqs:us-east-1:1:redis arn:minio:sqs:us-east-1:2:redis

Browser Access:
   http://192.168.1.106:9000  http://172.23.0.1:9000

Command-line Access: https://docs.minio.io/docs/minio-client-quickstart-guide
   $ mc config host add myminio http://192.168.1.106:9000 USWUXHGYZQYFYFFIT3RE MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03

Object API (Amazon S3 compatible):
   Go:         https://docs.minio.io/docs/golang-client-quickstart-guide
   Java:       https://docs.minio.io/docs/java-client-quickstart-guide
   Python:     https://docs.minio.io/docs/python-client-quickstart-guide
   JavaScript: https://docs.minio.io/docs/javascript-client-quickstart-guide
   .NET:       https://docs.minio.io/docs/dotnet-client-quickstart-guide

Drive Capacity: 26 GiB Free, 165 GiB Total

These details need to go into rclone config like this. Note that it is important to put the region in as stated above.

env_auth> 1
access_key_id> USWUXHGYZQYFYFFIT3RE
secret_access_key> MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region> us-east-1
endpoint> http://192.168.1.106:9000
location_constraint>
server_side_encryption>

Which makes the config file look like this

[minio]
type = s3
provider = Minio
env_auth = false
access_key_id = USWUXHGYZQYFYFFIT3RE
secret_access_key = MOJRH0mkL1IPauahWITSVvyDrQbEEIwljvmxdq03
region = us-east-1
endpoint = http://192.168.1.106:9000
location_constraint =
server_side_encryption =

So once set up, for example, to copy files into a bucket

rclone copy /path/to/files minio:bucket

Qiniu Cloud Object Storage (Kodo)

Qiniu Cloud Object Storage (Kodo), a completely independent-researched core technology which is proven by repeated customer experience has occupied absolute leading market leader position. Kodo can be widely applied to mass data management.

To configure access to Qiniu Kodo, follow the steps below:

Run rclone config and select n for a new remote.

rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Give the name of the configuration. For example, name it 'qiniu'.

name> qiniu

Select s3 storage.

Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
[snip]
Storage> s3

Select Qiniu provider.

Choose a number from below, or type in your own value
1 / Amazon Web Services (AWS) S3
   \ "AWS"
[snip]
22 / Qiniu Object Storage (Kodo)
   \ (Qiniu)
[snip]
provider> Qiniu

Enter your SecretId and SecretKey of Qiniu Kodo.

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> AKIDxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxx

Select endpoint for Qiniu Kodo. This is the standard endpoint for different region.

   / The default endpoint - a good choice if you are unsure.
 1 | East China Region 1.
   | Needs location constraint cn-east-1.
   \ (cn-east-1)
   / East China Region 2.
 2 | Needs location constraint cn-east-2.
   \ (cn-east-2)
   / North China Region 1.
 3 | Needs location constraint cn-north-1.
   \ (cn-north-1)
   / South China Region 1.
 4 | Needs location constraint cn-south-1.
   \ (cn-south-1)
   / North America Region.
 5 | Needs location constraint us-north-1.
   \ (us-north-1)
   / Southeast Asia Region 1.
 6 | Needs location constraint ap-southeast-1.
   \ (ap-southeast-1)
   / Northeast Asia Region 1.
 7 | Needs location constraint ap-northeast-1.
   \ (ap-northeast-1)
[snip]
endpoint> 1

Option endpoint.
Endpoint for Qiniu Object Storage.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / East China Endpoint 1
   \ (s3-cn-east-1.qiniucs.com)
 2 / East China Endpoint 2
   \ (s3-cn-east-2.qiniucs.com)
 3 / North China Endpoint 1
   \ (s3-cn-north-1.qiniucs.com)
 4 / South China Endpoint 1
   \ (s3-cn-south-1.qiniucs.com)
 5 / North America Endpoint 1
   \ (s3-us-north-1.qiniucs.com)
 6 / Southeast Asia Endpoint 1
   \ (s3-ap-southeast-1.qiniucs.com)
 7 / Northeast Asia Endpoint 1
   \ (s3-ap-northeast-1.qiniucs.com)
endpoint> 1

Option location_constraint.
Location constraint - must be set to match the Region.
Used when creating buckets only.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / East China Region 1
   \ (cn-east-1)
 2 / East China Region 2
   \ (cn-east-2)
 3 / North China Region 1
   \ (cn-north-1)
 4 / South China Region 1
   \ (cn-south-1)
 5 / North America Region 1
   \ (us-north-1)
 6 / Southeast Asia Region 1
   \ (ap-southeast-1)
 7 / Northeast Asia Region 1
   \ (ap-northeast-1)
location_constraint> 1

Choose acl and storage class.

Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
   / Owner gets FULL_CONTROL.
 2 | The AllUsers group gets READ access.
   \ (public-read)
[snip]
acl> 2
The storage class to use when storing new objects in Tencent COS.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Standard storage class
   \ (STANDARD)
 2 / Infrequent access storage mode
   \ (LINE)
 3 / Archive storage mode
   \ (GLACIER)
 4 / Deep archive storage mode
   \ (DEEP_ARCHIVE)
[snip]
storage_class> 1
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[qiniu]
- type: s3
- provider: Qiniu
- access_key_id: xxx
- secret_access_key: xxx
- region: cn-east-1
- endpoint: s3-cn-east-1.qiniucs.com
- location_constraint: cn-east-1
- acl: public-read
- storage_class: STANDARD
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
qiniu                s3

RackCorp

RackCorp Object Storage is an S3 compatible object storage platform from your friendly cloud provider RackCorp. The service is fast, reliable, well priced and located in many strategic locations unserviced by others, to ensure you can maintain data sovereignty.

Before you can use RackCorp Object Storage, you'll need to "sign up" for an account on our "portal". Next you can create an access key, a secret key and buckets, in your location of choice with ease. These details are required for the next steps of configuration, when rclone config asks for your access_key_id and secret_access_key.

Your config should end up looking a bit like this:

[RCS3-demo-config]
type = s3
provider = RackCorp
env_auth = true
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region = au-nsw
endpoint = s3.rackcorp.com
location_constraint = au-nsw

Rclone Serve S3

Rclone can serve any remote over the S3 protocol. For details see the rclone serve s3 documentation.

For example, to serve remote:path over s3, run the server like this:

rclone serve s3 --auth-key ACCESS_KEY_ID,SECRET_ACCESS_KEY remote:path

This will be compatible with an rclone remote which is defined like this:

[serves3]
type = s3
provider = Rclone
endpoint = http://127.0.0.1:8080/
access_key_id = ACCESS_KEY_ID
secret_access_key = SECRET_ACCESS_KEY
use_multipart_uploads = false

Note that setting disable_multipart_uploads = true is to work around a bug which will be fixed in due course.

Scaleway

Scaleway The Object Storage platform allows you to store anything from backups, logs and web assets to documents and photos. Files can be dropped from the Scaleway console or transferred through our API and CLI or using any S3-compatible tool.

Scaleway provides an S3 interface which can be configured for use with rclone like this:

[scaleway]
type = s3
provider = Scaleway
env_auth = false
endpoint = s3.nl-ams.scw.cloud
access_key_id = SCWXXXXXXXXXXXXXX
secret_access_key = 1111111-2222-3333-44444-55555555555555
region = nl-ams
location_constraint = nl-ams
acl = private
upload_cutoff = 5M
chunk_size = 5M
copy_cutoff = 5M

Scaleway Glacier is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" storage_class. So you can configure your remote with the storage_class = GLACIER option to upload directly to Scaleway Glacier. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)

Seagate Lyve Cloud

Seagate Lyve Cloud is an S3 compatible object storage platform from Seagate intended for enterprise use.

Here is a config run through for a remote called remote - you may choose a different name of course. Note that to create an access key and secret key you will need to create a service account first.

$ rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote

Choose s3 backend

Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
[snip]
Storage> s3

Choose LyveCloud as S3 provider

Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
XX / Seagate Lyve Cloud
   \ (LyveCloud)
[snip]
provider> LyveCloud

Take the default (just press enter) to enter access key and secret in the config file.

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth>

AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> XXX

AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> YYY

Leave region blank

Region to connect to.
Leave blank if you are using an S3 clone and you don't have a region.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Use this if unsure.
 1 | Will use v4 signatures and an empty region.
   \ ()
   / Use this only if v4 signatures don't work.
 2 | E.g. pre Jewel/v10 CEPH.
   \ (other-v2-signature)
region>

Choose an endpoint from the list

Endpoint for S3 API.
Required when using an S3 clone.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Seagate Lyve Cloud US East 1 (Virginia)
   \ (s3.us-east-1.lyvecloud.seagate.com)
 2 / Seagate Lyve Cloud US West 1 (California)
   \ (s3.us-west-1.lyvecloud.seagate.com)
 3 / Seagate Lyve Cloud AP Southeast 1 (Singapore)
   \ (s3.ap-southeast-1.lyvecloud.seagate.com)
endpoint> 1

Leave location constraint blank

Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint>

Choose default ACL (private).

Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
[snip]
acl>

And the config file should end up looking like this:

[remote]
type = s3
provider = LyveCloud
access_key_id = XXX
secret_access_key = YYY
endpoint = s3.us-east-1.lyvecloud.seagate.com

SeaweedFS

SeaweedFS is a distributed storage system for blobs, objects, files, and data lake, with O(1) disk seek and a scalable file metadata store. It has an S3 compatible object storage interface. SeaweedFS can also act as a gateway to remote S3 compatible object store to cache data and metadata with asynchronous write back, for fast local speed and minimize access cost.

Assuming the SeaweedFS are configured with weed shell as such:

> s3.bucket.create -name foo
> s3.configure -access_key=any -secret_key=any -buckets=foo -user=me -actions=Read,Write,List,Tagging,Admin -apply
{
  "identities": [
    {
      "name": "me",
      "credentials": [
        {
          "accessKey": "any",
          "secretKey": "any"
        }
      ],
      "actions": [
        "Read:foo",
        "Write:foo",
        "List:foo",
        "Tagging:foo",
        "Admin:foo"
      ]
    }
  ]
}

To use rclone with SeaweedFS, above configuration should end up with something like this in your config:

[seaweedfs_s3]
type = s3
provider = SeaweedFS
access_key_id = any
secret_access_key = any
endpoint = localhost:8333

So once set up, for example to copy files into a bucket

rclone copy /path/to/files seaweedfs_s3:foo

Wasabi

Wasabi is a cloud-based object storage service for a broad range of applications and use cases. Wasabi is designed for individuals and organizations that require a high-performance, reliable, and secure data storage infrastructure at minimal cost.

Wasabi provides an S3 interface which can be configured for use with rclone like this.

No remotes found, make a new one?
n) New remote
s) Set configuration password
n/s> n
name> wasabi
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Minio, Liara)
   \ "s3"
[snip]
Storage> s3
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> YOURACCESSKEY
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> YOURSECRETACCESSKEY
Region to connect to.
Choose a number from below, or type in your own value
   / The default endpoint - a good choice if you are unsure.
 1 | US Region, Northern Virginia, or Pacific Northwest.
   | Leave location constraint empty.
   \ "us-east-1"
[snip]
region> us-east-1
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Specify if using an S3 clone such as Ceph.
endpoint> s3.wasabisys.com
Location constraint - must be set to match the Region. Used when creating buckets only.
Choose a number from below, or type in your own value
 1 / Empty for US Region, Northern Virginia, or Pacific Northwest.
   \ ""
[snip]
location_constraint>
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
[snip]
acl>
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
server_side_encryption>
The storage class to use when storing objects in S3.
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
 3 / Reduced redundancy storage class
   \ "REDUCED_REDUNDANCY"
 4 / Standard Infrequent Access storage class
   \ "STANDARD_IA"
storage_class>
Remote config
--------------------
[wasabi]
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region = us-east-1
endpoint = s3.wasabisys.com
location_constraint =
acl =
server_side_encryption =
storage_class =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave the config file looking like this.

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region =
endpoint = s3.wasabisys.com
location_constraint =
acl =
server_side_encryption =
storage_class =

Alibaba OSS

Here is an example of making an Alibaba Cloud (Aliyun) OSS configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> oss
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ "s3"
[snip]
Storage> s3
Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Amazon Web Services (AWS) S3
   \ "AWS"
 2 / Alibaba Cloud Object Storage System (OSS) formerly Aliyun
   \ "Alibaba"
 3 / Ceph Object Storage
   \ "Ceph"
[snip]
provider> Alibaba
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> accesskeyid
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> secretaccesskey
Endpoint for OSS API.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / East China 1 (Hangzhou)
   \ "oss-cn-hangzhou.aliyuncs.com"
 2 / East China 2 (Shanghai)
   \ "oss-cn-shanghai.aliyuncs.com"
 3 / North China 1 (Qingdao)
   \ "oss-cn-qingdao.aliyuncs.com"
[snip]
endpoint> 1
Canned ACL used when creating buckets and storing or copying objects.

Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
 2 / Owner gets FULL_CONTROL. The AllUsers group gets READ access.
   \ "public-read"
   / Owner gets FULL_CONTROL. The AllUsers group gets READ and WRITE access.
[snip]
acl> 1
The storage class to use when storing new objects in OSS.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
 3 / Archive storage mode.
   \ "GLACIER"
 4 / Infrequent access storage mode.
   \ "STANDARD_IA"
storage_class> 1
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[oss]
type = s3
provider = Alibaba
env_auth = false
access_key_id = accesskeyid
secret_access_key = secretaccesskey
endpoint = oss-cn-hangzhou.aliyuncs.com
acl = private
storage_class = Standard
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

China Mobile Ecloud Elastic Object Storage (EOS)

Here is an example of making an China Mobile Ecloud Elastic Object Storage (EOS) configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> ChinaMobile
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
 ...
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
 ...
Storage> s3
Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 ...
 4 / China Mobile Ecloud Elastic Object Storage (EOS)
   \ (ChinaMobile)
 ...
provider> ChinaMobile
Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth>
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> accesskeyid
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> secretaccesskey
Option endpoint.
Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / The default endpoint - a good choice if you are unsure.
 1 | East China (Suzhou)
   \ (eos-wuxi-1.cmecloud.cn)
 2 / East China (Jinan)
   \ (eos-jinan-1.cmecloud.cn)
 3 / East China (Hangzhou)
   \ (eos-ningbo-1.cmecloud.cn)
 4 / East China (Shanghai-1)
   \ (eos-shanghai-1.cmecloud.cn)
 5 / Central China (Zhengzhou)
   \ (eos-zhengzhou-1.cmecloud.cn)
 6 / Central China (Changsha-1)
   \ (eos-hunan-1.cmecloud.cn)
 7 / Central China (Changsha-2)
   \ (eos-zhuzhou-1.cmecloud.cn)
 8 / South China (Guangzhou-2)
   \ (eos-guangzhou-1.cmecloud.cn)
 9 / South China (Guangzhou-3)
   \ (eos-dongguan-1.cmecloud.cn)
10 / North China (Beijing-1)
   \ (eos-beijing-1.cmecloud.cn)
11 / North China (Beijing-2)
   \ (eos-beijing-2.cmecloud.cn)
12 / North China (Beijing-3)
   \ (eos-beijing-4.cmecloud.cn)
13 / North China (Huhehaote)
   \ (eos-huhehaote-1.cmecloud.cn)
14 / Southwest China (Chengdu)
   \ (eos-chengdu-1.cmecloud.cn)
15 / Southwest China (Chongqing)
   \ (eos-chongqing-1.cmecloud.cn)
16 / Southwest China (Guiyang)
   \ (eos-guiyang-1.cmecloud.cn)
17 / Nouthwest China (Xian)
   \ (eos-xian-1.cmecloud.cn)
18 / Yunnan China (Kunming)
   \ (eos-yunnan.cmecloud.cn)
19 / Yunnan China (Kunming-2)
   \ (eos-yunnan-2.cmecloud.cn)
20 / Tianjin China (Tianjin)
   \ (eos-tianjin-1.cmecloud.cn)
21 / Jilin China (Changchun)
   \ (eos-jilin-1.cmecloud.cn)
22 / Hubei China (Xiangyan)
   \ (eos-hubei-1.cmecloud.cn)
23 / Jiangxi China (Nanchang)
   \ (eos-jiangxi-1.cmecloud.cn)
24 / Gansu China (Lanzhou)
   \ (eos-gansu-1.cmecloud.cn)
25 / Shanxi China (Taiyuan)
   \ (eos-shanxi-1.cmecloud.cn)
26 / Liaoning China (Shenyang)
   \ (eos-liaoning-1.cmecloud.cn)
27 / Hebei China (Shijiazhuang)
   \ (eos-hebei-1.cmecloud.cn)
28 / Fujian China (Xiamen)
   \ (eos-fujian-1.cmecloud.cn)
29 / Guangxi China (Nanning)
   \ (eos-guangxi-1.cmecloud.cn)
30 / Anhui China (Huainan)
   \ (eos-anhui-1.cmecloud.cn)
endpoint> 1
Option location_constraint.
Location constraint - must match endpoint.
Used when creating buckets only.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / East China (Suzhou)
   \ (wuxi1)
 2 / East China (Jinan)
   \ (jinan1)
 3 / East China (Hangzhou)
   \ (ningbo1)
 4 / East China (Shanghai-1)
   \ (shanghai1)
 5 / Central China (Zhengzhou)
   \ (zhengzhou1)
 6 / Central China (Changsha-1)
   \ (hunan1)
 7 / Central China (Changsha-2)
   \ (zhuzhou1)
 8 / South China (Guangzhou-2)
   \ (guangzhou1)
 9 / South China (Guangzhou-3)
   \ (dongguan1)
10 / North China (Beijing-1)
   \ (beijing1)
11 / North China (Beijing-2)
   \ (beijing2)
12 / North China (Beijing-3)
   \ (beijing4)
13 / North China (Huhehaote)
   \ (huhehaote1)
14 / Southwest China (Chengdu)
   \ (chengdu1)
15 / Southwest China (Chongqing)
   \ (chongqing1)
16 / Southwest China (Guiyang)
   \ (guiyang1)
17 / Nouthwest China (Xian)
   \ (xian1)
18 / Yunnan China (Kunming)
   \ (yunnan)
19 / Yunnan China (Kunming-2)
   \ (yunnan2)
20 / Tianjin China (Tianjin)
   \ (tianjin1)
21 / Jilin China (Changchun)
   \ (jilin1)
22 / Hubei China (Xiangyan)
   \ (hubei1)
23 / Jiangxi China (Nanchang)
   \ (jiangxi1)
24 / Gansu China (Lanzhou)
   \ (gansu1)
25 / Shanxi China (Taiyuan)
   \ (shanxi1)
26 / Liaoning China (Shenyang)
   \ (liaoning1)
27 / Hebei China (Shijiazhuang)
   \ (hebei1)
28 / Fujian China (Xiamen)
   \ (fujian1)
29 / Guangxi China (Nanning)
   \ (guangxi1)
30 / Anhui China (Huainan)
   \ (anhui1)
location_constraint> 1
Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
   / Owner gets FULL_CONTROL.
 2 | The AllUsers group gets READ access.
   \ (public-read)
   / Owner gets FULL_CONTROL.
 3 | The AllUsers group gets READ and WRITE access.
   | Granting this on a bucket is generally not recommended.
   \ (public-read-write)
   / Owner gets FULL_CONTROL.
 4 | The AuthenticatedUsers group gets READ access.
   \ (authenticated-read)
   / Object owner gets FULL_CONTROL.
acl> private
Option server_side_encryption.
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / None
   \ ()
 2 / AES256
   \ (AES256)
server_side_encryption>
Option storage_class.
The storage class to use when storing new objects in ChinaMobile.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Default
   \ ()
 2 / Standard storage class
   \ (STANDARD)
 3 / Archive storage mode
   \ (GLACIER)
 4 / Infrequent access storage mode
   \ (STANDARD_IA)
storage_class>
Edit advanced config?
y) Yes
n) No (default)
y/n> n
--------------------
[ChinaMobile]
type = s3
provider = ChinaMobile
access_key_id = accesskeyid
secret_access_key = secretaccesskey
endpoint = eos-wuxi-1.cmecloud.cn
location_constraint = wuxi1
acl = private
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Leviia Cloud Object Storage

Leviia Object Storage, backup and secure your data in a 100% French cloud, independent of GAFAM..

To configure access to Leviia, follow the steps below:

Run rclone config and select n for a new remote.

rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Give the name of the configuration. For example, name it 'leviia'.

name> leviia

Select s3 storage.

Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ (s3)
[snip]
Storage> s3

Select Leviia provider.

Choose a number from below, or type in your own value
1 / Amazon Web Services (AWS) S3
   \ "AWS"
[snip]
15 / Leviia Object Storage
   \ (Leviia)
[snip]
provider> Leviia

Enter your SecretId and SecretKey of Leviia.

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> ZnIx.xxxxxxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxx

Select endpoint for Leviia.

   / The default endpoint
 1 | Leviia.
   \ (s3.leviia.com)
[snip]
endpoint> 1

Choose acl.

Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
   / Owner gets FULL_CONTROL.
 2 | The AllUsers group gets READ access.
   \ (public-read)
[snip]
acl> 1
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[leviia]
- type: s3
- provider: Leviia
- access_key_id: ZnIx.xxxxxxx
- secret_access_key: xxxxxxxx
- endpoint: s3.leviia.com
- acl: private
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
leviia                s3

Liara

Here is an example of making a Liara Object Storage configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
n/s> n
name> Liara
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
   \ "s3"
[snip]
Storage> s3
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> YOURACCESSKEY
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> YOURSECRETACCESSKEY
Region to connect to.
Choose a number from below, or type in your own value
   / The default endpoint
 1 | US Region, Northern Virginia, or Pacific Northwest.
   | Leave location constraint empty.
   \ "us-east-1"
[snip]
region>
Endpoint for S3 API.
Leave blank if using Liara to use the default endpoint for the region.
Specify if using an S3 clone such as Ceph.
endpoint> storage.iran.liara.space
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
[snip]
acl>
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
server_side_encryption>
The storage class to use when storing objects in S3.
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
storage_class>
Remote config
--------------------
[Liara]
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
endpoint = storage.iran.liara.space
location_constraint =
acl =
server_side_encryption =
storage_class =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave the config file looking like this.

[Liara]
type = s3
provider = Liara
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region =
endpoint = storage.iran.liara.space
location_constraint =
acl =
server_side_encryption =
storage_class =

Linode

Here is an example of making a Linode Object Storage configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> linode

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...Linode, ...and others
   \ (s3)
[snip]
Storage> s3

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
XX / Linode Object Storage
   \ (Linode)
[snip]
provider> Linode

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> ACCESS_KEY

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> SECRET_ACCESS_KEY

Option endpoint.
Endpoint for Linode Object Storage API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Atlanta, GA (USA), us-southeast-1
   \ (us-southeast-1.linodeobjects.com)
 2 / Chicago, IL (USA), us-ord-1
   \ (us-ord-1.linodeobjects.com)
 3 / Frankfurt (Germany), eu-central-1
   \ (eu-central-1.linodeobjects.com)
 4 / Milan (Italy), it-mil-1
   \ (it-mil-1.linodeobjects.com)
 5 / Newark, NJ (USA), us-east-1
   \ (us-east-1.linodeobjects.com)
 6 / Paris (France), fr-par-1
   \ (fr-par-1.linodeobjects.com)
 7 / Seattle, WA (USA), us-sea-1
   \ (us-sea-1.linodeobjects.com)
 8 / Singapore ap-south-1
   \ (ap-south-1.linodeobjects.com)
 9 / Stockholm (Sweden), se-sto-1
   \ (se-sto-1.linodeobjects.com)
10 / Washington, DC, (USA), us-iad-1
   \ (us-iad-1.linodeobjects.com)
endpoint> 3

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
[snip]
acl> 

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:
- type: s3
- provider: Linode
- access_key_id: ACCESS_KEY
- secret_access_key: SECRET_ACCESS_KEY
- endpoint: eu-central-1.linodeobjects.com
Keep this "linode" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave the config file looking like this.

[linode]
type = s3
provider = Linode
access_key_id = ACCESS_KEY
secret_access_key = SECRET_ACCESS_KEY
endpoint = eu-central-1.linodeobjects.com

Magalu

Here is an example of making a Magalu Object Storage configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Enter name for new remote.
name> magalu

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...Magalu, ...and others
   \ (s3)
[snip]
Storage> s3

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
XX / Magalu Object Storage
   \ (Magalu)
[snip]
provider> Magalu

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> ACCESS_KEY

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> SECRET_ACCESS_KEY

Option endpoint.
Endpoint for Magalu Object Storage API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / São Paulo, SP (BR), br-se1
   \ (br-se1.magaluobjects.com)
 2 / Fortaleza, CE (BR), br-ne1
   \ (br-ne1.magaluobjects.com)
endpoint> 2

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
[snip]
acl> 

Edit advanced config?
y) Yes
n) No (default)
y/n> n

Configuration complete.
Options:
- type: s3
- provider: magalu
- access_key_id: ACCESS_KEY
- secret_access_key: SECRET_ACCESS_KEY
- endpoint: br-ne1.magaluobjects.com
Keep this "magalu" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave the config file looking like this.

[magalu]
type = s3
provider = Magalu
access_key_id = ACCESS_KEY
secret_access_key = SECRET_ACCESS_KEY
endpoint = br-ne1.magaluobjects.com

ArvanCloud

ArvanCloud ArvanCloud Object Storage goes beyond the limited traditional file storage. It gives you access to backup and archived files and allows sharing. Files like profile image in the app, images sent by users or scanned documents can be stored securely and easily in our Object Storage service.

ArvanCloud provides an S3 interface which can be configured for use with rclone like this.

No remotes found, make a new one?
n) New remote
s) Set configuration password
n/s> n
name> ArvanCloud
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 (also Dreamhost, Ceph, ChinaMobile, ArvanCloud, Liara, Minio)
   \ "s3"
[snip]
Storage> s3
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars). Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID - leave blank for anonymous access or runtime credentials.
access_key_id> YOURACCESSKEY
AWS Secret Access Key (password) - leave blank for anonymous access or runtime credentials.
secret_access_key> YOURSECRETACCESSKEY
Region to connect to.
Choose a number from below, or type in your own value
   / The default endpoint - a good choice if you are unsure.
 1 | US Region, Northern Virginia, or Pacific Northwest.
   | Leave location constraint empty.
   \ "us-east-1"
[snip]
region> 
Endpoint for S3 API.
Leave blank if using ArvanCloud to use the default endpoint for the region.
Specify if using an S3 clone such as Ceph.
endpoint> s3.arvanstorage.com
Location constraint - must be set to match the Region. Used when creating buckets only.
Choose a number from below, or type in your own value
 1 / Empty for Iran-Tehran Region.
   \ ""
[snip]
location_constraint>
Canned ACL used when creating buckets and/or storing objects in S3.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Choose a number from below, or type in your own value
 1 / Owner gets FULL_CONTROL. No one else has access rights (default).
   \ "private"
[snip]
acl>
The server-side encryption algorithm used when storing this object in S3.
Choose a number from below, or type in your own value
 1 / None
   \ ""
 2 / AES256
   \ "AES256"
server_side_encryption>
The storage class to use when storing objects in S3.
Choose a number from below, or type in your own value
 1 / Default
   \ ""
 2 / Standard storage class
   \ "STANDARD"
storage_class>
Remote config
--------------------
[ArvanCloud]
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region = ir-thr-at1
endpoint = s3.arvanstorage.com
location_constraint =
acl =
server_side_encryption =
storage_class =
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave the config file looking like this.

[ArvanCloud]
type = s3
provider = ArvanCloud
env_auth = false
access_key_id = YOURACCESSKEY
secret_access_key = YOURSECRETACCESSKEY
region =
endpoint = s3.arvanstorage.com
location_constraint =
acl =
server_side_encryption =
storage_class =

Tencent COS

Tencent Cloud Object Storage (COS) is a distributed storage service offered by Tencent Cloud for unstructured data. It is secure, stable, massive, convenient, low-delay and low-cost.

To configure access to Tencent COS, follow the steps below:

Run rclone config and select n for a new remote.

rclone config
No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n

Give the name of the configuration. For example, name it 'cos'.

name> cos

Select s3 storage.

Choose a number from below, or type in your own value
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ "s3"
[snip]
Storage> s3

Select TencentCOS provider.

Choose a number from below, or type in your own value
1 / Amazon Web Services (AWS) S3
   \ "AWS"
[snip]
11 / Tencent Cloud Object Storage (COS)
   \ "TencentCOS"
[snip]
provider> TencentCOS

Enter your SecretId and SecretKey of Tencent Cloud.

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"
env_auth> 1
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
access_key_id> AKIDxxxxxxxxxx
AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").
secret_access_key> xxxxxxxxxxx

Select endpoint for Tencent COS. This is the standard endpoint for different region.

 1 / Beijing Region.
   \ "cos.ap-beijing.myqcloud.com"
 2 / Nanjing Region.
   \ "cos.ap-nanjing.myqcloud.com"
 3 / Shanghai Region.
   \ "cos.ap-shanghai.myqcloud.com"
 4 / Guangzhou Region.
   \ "cos.ap-guangzhou.myqcloud.com"
[snip]
endpoint> 4

Choose acl and storage class.

Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Owner gets Full_CONTROL. No one else has access rights (default).
   \ "default"
[snip]
acl> 1
The storage class to use when storing new objects in Tencent COS.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Default
   \ ""
[snip]
storage_class> 1
Edit advanced config? (y/n)
y) Yes
n) No (default)
y/n> n
Remote config
--------------------
[cos]
type = s3
provider = TencentCOS
env_auth = false
access_key_id = xxx
secret_access_key = xxx
endpoint = cos.ap-guangzhou.myqcloud.com
acl = default
--------------------
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
Current remotes:

Name                 Type
====                 ====
cos                  s3

Netease NOS

For Netease NOS configure as per the configurator rclone config setting the provider Netease. This will automatically set force_path_style = false which is necessary for it to run properly.

Petabox

Here is an example of making a Petabox configuration. First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
n/s> n

Enter name for new remote.
name> My Petabox Storage

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[snip]
XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ "s3"
[snip]
Storage> s3

Option provider.
Choose your S3 provider.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
[snip]
XX / Petabox Object Storage
   \ (Petabox)
[snip]
provider> Petabox

Option env_auth.
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1

Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> YOUR_ACCESS_KEY_ID

Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> YOUR_SECRET_ACCESS_KEY

Option region.
Region where your bucket will be created and your data stored.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / US East (N. Virginia)
   \ (us-east-1)
 2 / Europe (Frankfurt)
   \ (eu-central-1)
 3 / Asia Pacific (Singapore)
   \ (ap-southeast-1)
 4 / Middle East (Bahrain)
   \ (me-south-1)
 5 / South America (São Paulo)
   \ (sa-east-1)
region> 1

Option endpoint.
Endpoint for Petabox S3 Object Storage.
Specify the endpoint from the same region.
Choose a number from below, or type in your own value.
 1 / US East (N. Virginia)
   \ (s3.petabox.io)
 2 / US East (N. Virginia)
   \ (s3.us-east-1.petabox.io)
 3 / Europe (Frankfurt)
   \ (s3.eu-central-1.petabox.io)
 4 / Asia Pacific (Singapore)
   \ (s3.ap-southeast-1.petabox.io)
 5 / Middle East (Bahrain)
   \ (s3.me-south-1.petabox.io)
 6 / South America (São Paulo)
   \ (s3.sa-east-1.petabox.io)
endpoint> 1

Option acl.
Canned ACL used when creating buckets and storing or copying objects.
This ACL is used for creating objects and if bucket_acl isn't set, for creating buckets too.
For more info visit https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl
Note that this ACL is applied when server-side copying objects as S3
doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / Owner gets FULL_CONTROL.
 1 | No one else has access rights (default).
   \ (private)
   / Owner gets FULL_CONTROL.
 2 | The AllUsers group gets READ access.
   \ (public-read)
   / Owner gets FULL_CONTROL.
 3 | The AllUsers group gets READ and WRITE access.
   | Granting this on a bucket is generally not recommended.
   \ (public-read-write)
   / Owner gets FULL_CONTROL.
 4 | The AuthenticatedUsers group gets READ access.
   \ (authenticated-read)
   / Object owner gets FULL_CONTROL.
 5 | Bucket owner gets READ access.
   | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ (bucket-owner-read)
   / Both the object owner and the bucket owner get FULL_CONTROL over the object.
 6 | If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
   \ (bucket-owner-full-control)
acl> 1

Edit advanced config?
y) Yes
n) No (default)
y/n> No

Configuration complete.
Options:
- type: s3
- provider: Petabox
- access_key_id: YOUR_ACCESS_KEY_ID
- secret_access_key: YOUR_SECRET_ACCESS_KEY
- region: us-east-1
- endpoint: s3.petabox.io
Keep this "My Petabox Storage" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

This will leave the config file looking like this.

[My Petabox Storage]
type = s3
provider = Petabox
access_key_id = YOUR_ACCESS_KEY_ID
secret_access_key = YOUR_SECRET_ACCESS_KEY
region = us-east-1
endpoint = s3.petabox.io

Storj

Storj is a decentralized cloud storage which can be used through its native protocol or an S3 compatible gateway.

The S3 compatible gateway is configured using rclone config with a type of s3 and with a provider name of Storj. Here is an example run of the configurator.

Type of storage to configure.
Storage> s3
Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Choose a number from below, or type in your own boolean value (true or false).
Press Enter for the default (false).
 1 / Enter AWS credentials in the next step.
   \ (false)
 2 / Get AWS credentials from the environment (env vars or IAM).
   \ (true)
env_auth> 1
Option access_key_id.
AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
access_key_id> XXXX (as shown when creating the access grant)
Option secret_access_key.
AWS Secret Access Key (password).
Leave blank for anonymous access or runtime credentials.
Enter a value. Press Enter to leave empty.
secret_access_key> XXXX (as shown when creating the access grant)
Option endpoint.
Endpoint of the Shared Gateway.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / EU1 Shared Gateway
   \ (gateway.eu1.storjshare.io)
 2 / US1 Shared Gateway
   \ (gateway.us1.storjshare.io)
 3 / Asia-Pacific Shared Gateway
   \ (gateway.ap1.storjshare.io)
endpoint> 1 (as shown when creating the access grant)
Edit advanced config?
y) Yes
n) No (default)
y/n> n

Note that s3 credentials are generated when you create an access grant.

Backend quirks

--chunk-size is forced to be 64 MiB or greater. This will use more memory than the default of 5 MiB.
Server side copy is disabled as it isn't currently supported in the gateway.
GetTier and SetTier are not supported.

Backend bugs

Due to issue #39 uploading multipart files via the S3 gateway causes them to lose their metadata. For rclone's purpose this means that the modification time is not stored, nor is any MD5SUM (if one is available from the source).

This has the following consequences:

Using rclone rcat will fail as the medatada doesn't match after upload
Uploading files with rclone mount will fail for the same reason
- This can worked around by using --vfs-cache-mode writes or --vfs-cache-mode full or setting --s3-upload-cutoff large
Files uploaded via a multipart upload won't have their modtimes
- This will mean that rclone sync will likely keep trying to upload files bigger than --s3-upload-cutoff
- This can be worked around with --checksum or --size-only or setting --s3-upload-cutoff large
- The maximum value for --s3-upload-cutoff is 5GiB though

One general purpose workaround is to set --s3-upload-cutoff 5G. This means that rclone will upload files smaller than 5GiB as single parts. Note that this can be set in the config file with upload_cutoff = 5G or configured in the advanced settings. If you regularly transfer files larger than 5G then using --checksum or --size-only in rclone sync is the recommended workaround.

Comparison with the native protocol

Use the the native protocol to take advantage of client-side encryption as well as to achieve the best possible download performance. Uploads will be erasure-coded locally, thus a 1gb upload will result in 2.68gb of data being uploaded to storage nodes across the network.

Use this backend and the S3 compatible Hosted Gateway to increase upload performance and reduce the load on your systems and network. Uploads will be encrypted and erasure-coded server-side, thus a 1GB upload will result in only in 1GB of data being uploaded to storage nodes across the network.

For more detailed comparison please check the documentation of the storj backend.

Memory usage {memory}

The most common cause of rclone using lots of memory is a single directory with millions of files in. Despite s3 not really having the concepts of directories, rclone does the sync on a directory by directory basis to be compatible with normal filing systems.

Rclone loads each directory into memory as rclone objects. Each rclone object takes 0.5k-1k of memory, so approximately 1GB per 1,000,000 files, and the sync for that directory does not begin until it is entirely loaded in memory. So the sync can take a long time to start for large directories.

To sync a directory with 100,000,000 files in you would need approximately 100 GB of memory. At some point the amount of memory becomes difficult to provide so there is a workaround for this which involves a bit of scripting.

At some point rclone will gain a sync mode which is effectively this workaround but built in to rclone.

Limitations

rclone about is not supported by the S3 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

See List of backends that do not support rclone about and rclone about

Synology C2 Object Storage

Synology C2 Object Storage provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.

The S3 compatible gateway is configured using rclone config with a type of s3 and with a provider name of Synology. Here is an example run of the configurator.

First run:

rclone config

This will guide you through an interactive setup process.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config

n/s/q> n

Enter name for new remote.1
name> syno

Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value

XX / Amazon S3 Compliant Storage Providers including AWS, ...
   \ "s3"

Storage> s3

Choose your S3 provider.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 24 / Synology C2 Object Storage
   \ (Synology)

provider> Synology

Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).
Only applies if access_key_id and secret_access_key is blank.
Enter a boolean value (true or false). Press Enter for the default ("false").
Choose a number from below, or type in your own value
 1 / Enter AWS credentials in the next step
   \ "false"
 2 / Get AWS credentials from the environment (env vars or IAM)
   \ "true"

env_auth> 1

AWS Access Key ID.
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").

access_key_id> accesskeyid

AWS Secret Access Key (password)
Leave blank for anonymous access or runtime credentials.
Enter a string value. Press Enter for the default ("").

secret_access_key> secretaccesskey

Region where your data stored.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / Europe Region 1
   \ (eu-001)
 2 / Europe Region 2
   \ (eu-002)
 3 / US Region 1
   \ (us-001)
 4 / US Region 2
   \ (us-002)
 5 / Asia (Taiwan)
   \ (tw-001)

region > 1

Option endpoint.
Endpoint for Synology C2 Object Storage API.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
 1 / EU Endpoint 1
   \ (eu-001.s3.synologyc2.net)
 2 / US Endpoint 1
   \ (us-001.s3.synologyc2.net)
 3 / TW Endpoint 1
   \ (tw-001.s3.synologyc2.net)

endpoint> 1

Option location_constraint.
Location constraint - must be set to match the Region.
Leave blank if not sure. Used when creating buckets only.
Enter a value. Press Enter to leave empty.
location_constraint>

Edit advanced config? (y/n)
y) Yes
n) No
y/n> y

Option no_check_bucket.
If set, don't attempt to check the bucket exists or create it.
This can be useful when trying to minimise the number of transactions
rclone does if you know the bucket exists already.
It can also be needed if the user you are using does not have bucket
creation permissions. Before v1.52.0 this would have passed silently
due to a bug.
Enter a boolean value (true or false). Press Enter for the default (true).

no_check_bucket> true

Configuration complete.
Options:
- type: s3
- provider: Synology
- region: eu-001
- endpoint: eu-001.s3.synologyc2.net
- no_check_bucket: true
Keep this "syno" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote

y/e/d> y

Authors

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Authors and contributors

Authors

Nick Craig-Wood nick@craig-wood.com

Contributors

Alex Couper amcouper@gmail.com
Leonid Shalupov leonid@shalupov.com shalupov@diverse.org.ru
Shimon Doodkin helpmepro1@gmail.com
Colin Nicholson colin@colinn.com
Klaus Post klauspost@gmail.com
Sergey Tolmachev tolsi.ru@gmail.com
Adriano Aurélio Meirelles adriano@atinge.com
C. Bess cbess@users.noreply.github.com
Dmitry Burdeev dibu28@gmail.com
Joseph Spurrier github@josephspurrier.com
Björn Harrtell bjorn@wololo.org
Xavier Lucas xavier.lucas@corp.ovh.com
Werner Beroux werner@beroux.com
Brian Stengaard brian@stengaard.eu
Jakub Gedeon jgedeon@sofi.com
Jim Tittsler jwt@onjapan.net
Michal Witkowski michal@improbable.io
Fabian Ruff fabian.ruff@sap.com
Leigh Klotz klotz@quixey.com
Romain Lapray lapray.romain@gmail.com
Justin R. Wilson jrw972@gmail.com
Antonio Messina antonio.s.messina@gmail.com
Stefan G. Weichinger office@oops.co.at
Per Cederberg cederberg@gmail.com
Radek Šenfeld rush@logic.cz
Fredrik Fornwall fredrik@fornwall.net
Asko Tamm asko@deekit.net
xor-zz xor@gstocco.com
Tomasz Mazur tmazur90@gmail.com
Marco Paganini paganini@paganini.net
Felix Bünemann buenemann@louis.info
Durval Menezes jmrclone@durval.com
Luiz Carlos Rumbelsperger Viana maxd13_luiz_carlos@hotmail.com
Stefan Breunig stefan-github@yrden.de
Alishan Ladhani ali-l@users.noreply.github.com
0xJAKE 0xJAKE@users.noreply.github.com
Thibault Molleman thibaultmol@users.noreply.github.com
Scott McGillivray scott.mcgillivray@gmail.com
Bjørn Erik Pedersen bjorn.erik.pedersen@gmail.com
Lukas Loesche lukas@mesosphere.io
emyarod emyarod@users.noreply.github.com
T.C. Ferguson tcf909@gmail.com
Brandur brandur@mutelight.org
Dario Giovannetti dev@dariogiovannetti.net
Károly Oláh okaresz@aol.com
Jon Yergatian jon@macfanatic.ca
Jack Schmidt github@mowsey.org
Dedsec1 Dedsec1@users.noreply.github.com
Hisham Zarka hzarka@gmail.com
Jérôme Vizcaino jerome.vizcaino@gmail.com
Mike Tesch mjt6129@rit.edu
Marvin Watson marvwatson@users.noreply.github.com
Danny Tsai danny8376@gmail.com
Yoni Jah yonjah+git@gmail.com yonjah+github@gmail.com
Stephen Harris github@spuddy.org sweharris@users.noreply.github.com
Ihor Dvoretskyi ihor.dvoretskyi@gmail.com
Jon Craton jncraton@gmail.com
Hraban Luyat hraban@0brg.net
Michael Ledin mledin89@gmail.com
Martin Kristensen me@azgul.com
Too Much IO toomuchio@users.noreply.github.com
Anisse Astier anisse@astier.eu
Zahiar Ahmed zahiar@live.com
Igor Kharin igorkharin@gmail.com
Bill Zissimopoulos billziss@navimatics.com
Bob Potter bobby.potter@gmail.com
Steven Lu tacticalazn@gmail.com
Sjur Fredriksen sjurtf@ifi.uio.no
Ruwbin hubus12345@gmail.com
Fabian Möller fabianm88@gmail.com f.moeller@nynex.de
Edward Q. Bridges github@eqbridges.com
Vasiliy Tolstov v.tolstov@selfip.ru
Harshavardhana harsha@minio.io
sainaen sainaen@gmail.com
gdm85 gdm85@users.noreply.github.com
Yaroslav Halchenko debian@onerussian.com
John Papandriopoulos jpap@users.noreply.github.com
Zhiming Wang zmwangx@gmail.com
Andy Pilate cubox@cubox.me
Oliver Heyme olihey@googlemail.com olihey@users.noreply.github.com de8olihe@lego.com
wuyu wuyu@yunify.com
Andrei Dragomir adragomi@adobe.com
Christian Brüggemann mail@cbruegg.com
Alex McGrath Kraak amkdude@gmail.com
bpicode bjoern.pirnay@googlemail.com
Daniel Jagszent daniel@jagszent.de
Josiah White thegenius2009@gmail.com
Ishuah Kariuki kariuki@ishuah.com ishuah91@gmail.com
Jan Varho jan@varho.org
Girish Ramakrishnan girish@cloudron.io
LingMan LingMan@users.noreply.github.com
Jacob McNamee jacobmcnamee@gmail.com
jersou jertux@gmail.com
thierry thierry@substantiel.fr
Simon Leinen simon.leinen@gmail.com ubuntu@s3-test.novalocal
Dan Dascalescu ddascalescu+github@gmail.com
Jason Rose jason@jro.io
Andrew Starr-Bochicchio a.starr.b@gmail.com
John Leach john@johnleach.co.uk
Corban Raun craun@instructure.com
Pierre Carlson mpcarl@us.ibm.com
Ernest Borowski er.borowski@gmail.com
Remus Bunduc remus.bunduc@gmail.com
Iakov Davydov iakov.davydov@unil.ch dav05.gith@myths.ru
Jakub Tasiemski tasiemski@gmail.com
David Minor dminor@saymedia.com
Tim Cooijmans cooijmans.tim@gmail.com
Laurence liuxy6@gmail.com
Giovanni Pizzi gio.piz@gmail.com
Filip Bartodziej filipbartodziej@gmail.com
Jon Fautley jon@dead.li
lewapm 32110057+lewapm@users.noreply.github.com
Yassine Imounachen yassine256@gmail.com
Chris Redekop chris-redekop@users.noreply.github.com chris.redekop@gmail.com
Jon Fautley jon@adenoid.appstal.co.uk
Will Gunn WillGunn@users.noreply.github.com
Lucas Bremgartner lucas@bremis.ch
Jody Frankowski jody.frankowski@gmail.com
Andreas Roussos arouss1980@gmail.com
nbuchanan nbuchanan@utah.gov
Durval Menezes rclone@durval.com
Victor vb-github@viblo.se
Mateusz pabian.mateusz@gmail.com
Daniel Loader spicypixel@gmail.com
David0rk davidork@gmail.com
Alexander Neumann alexander@bumpern.de
Giri Badanahatti <gbadanahatti@us.ibm.com@Giris-MacBook-Pro.local>
Leo R. Lundgren leo@finalresort.org
wolfv wolfv6@users.noreply.github.com
Dave Pedu dave@davepedu.com
Stefan Lindblom lindblom@spotify.com
seuffert oliver@seuffert.biz
gbadanahatti 37121690+gbadanahatti@users.noreply.github.com
Keith Goldfarb barkofdelight@gmail.com
Steve Kriss steve@heptio.com
Chih-Hsuan Yen yan12125@gmail.com
Alexander Neumann fd0@users.noreply.github.com
Matt Holt mholt@users.noreply.github.com
Eri Bastos bastos.eri@gmail.com
Michael P. Dubner pywebmail@list.ru
Antoine GIRARD sapk@users.noreply.github.com
Mateusz Piotrowski mpp302@gmail.com
Animosity022 animosity22@users.noreply.github.com earl.texter@gmail.com
Peter Baumgartner pete@lincolnloop.com
Craig Rachel craig@craigrachel.com
Michael G. Noll miguno@users.noreply.github.com
hensur me@hensur.de
Oliver Heyme de8olihe@lego.com
Richard Yang richard@yenforyang.com
Piotr Oleszczyk piotr.oleszczyk@gmail.com
Rodrigo rodarima@gmail.com
NoLooseEnds NoLooseEnds@users.noreply.github.com
Jakub Karlicek jakub@karlicek.me
John Clayton john@codemonkeylabs.com
Kasper Byrdal Nielsen byrdal76@gmail.com
Benjamin Joseph Dag bjdag1234@users.noreply.github.com
themylogin themylogin@gmail.com
Onno Zweers onno.zweers@surfsara.nl
Jasper Lievisse Adriaanse jasper@humppa.nl
sandeepkru sandeep.ummadi@gmail.com sandeepkru@users.noreply.github.com
HerrH atomtigerzoo@users.noreply.github.com
Andrew 4030760+sparkyman215@users.noreply.github.com
dan smith XX1011@gmail.com
Oleg Kovalov iamolegkovalov@gmail.com
Ruben Vandamme github-com-00ff86@vandamme.email
Cnly minecnly@gmail.com
Andres Alvarez 1671935+kir4h@users.noreply.github.com
reddi1 xreddi@gmail.com
Matt Tucker matthewtckr@gmail.com
Sebastian Bünger buengese@gmail.com buengese@protonmail.com
Martin Polden mpolden@mpolden.no
Alex Chen Cnly@users.noreply.github.com
Denis deniskovpen@gmail.com
bsteiss 35940619+bsteiss@users.noreply.github.com
Cédric Connes cedric.connes@gmail.com
Dr. Tobias Quathamer toddy15@users.noreply.github.com
dcpu 42736967+dcpu@users.noreply.github.com
Sheldon Rupp me@shel.io
albertony 12441419+albertony@users.noreply.github.com
cron410 cron410@gmail.com
Anagh Kumar Baranwal 6824881+darthShadow@users.noreply.github.com
Felix Brucker felix@felixbrucker.com
Santiago Rodríguez scollazo@users.noreply.github.com
Craig Miskell craig.miskell@fluxfederation.com
Antoine GIRARD sapk@sapk.fr
Joanna Marek joanna.marek@u2i.com
frenos frenos@users.noreply.github.com
ssaqua ssaqua@users.noreply.github.com
xnaas me@xnaas.info
Frantisek Fuka fuka@fuxoft.cz
Paul Kohout pauljkohout@yahoo.com
dcpu 43330287+dcpu@users.noreply.github.com
jackyzy823 jackyzy823@gmail.com
David Haguenauer ml@kurokatta.org
teresy hi.teresy@gmail.com
buergi patbuergi@gmx.de
Florian Gamboeck mail@floga.de
Ralf Hemberger 10364191+rhemberger@users.noreply.github.com
Scott Edlund sedlund@users.noreply.github.com
Erik Swanson erik@retailnext.net
Jake Coggiano jake@stripe.com
brused27 brused27@noemailaddress
Peter Kaminski kaminski@istori.com
Henry Ptasinski henry@logout.com
Alexander kharkovalexander@gmail.com
Garry McNulty garrmcnu@gmail.com
Mathieu Carbou mathieu.carbou@gmail.com
Mark Otway mark@otway.com
William Cocker 37018962+WilliamCocker@users.noreply.github.com
François Leurent 131.js@cloudyks.org
Arkadius Stefanski arkste@gmail.com
Jay dev@jaygoel.com
andrea rota a@xelera.eu
nicolov nicolov@users.noreply.github.com
Matt Joiner anacrolix@gmail.com
Dario Guzik dario@guzik.com.ar
qip qip@users.noreply.github.com
yair@unicorn yair@unicorn
Matt Robinson brimstone@the.narro.ws
kayrus kay.diam@gmail.com
Rémy Léone remy.leone@gmail.com
Wojciech Smigielski wojciech.hieronim.smigielski@gmail.com
weetmuts oehrstroem@gmail.com
Jonathan vanillajonathan@users.noreply.github.com
James Carpenter orbsmiv@users.noreply.github.com
Vince vince0villamora@gmail.com
Nestar47 47841759+Nestar47@users.noreply.github.com
Six brbsix@gmail.com
Alexandru Bumbacea alexandru.bumbacea@booking.com
calisro robert.calistri@gmail.com
Dr.Rx david.rey@nventive.com
marcintustin marcintustin@users.noreply.github.com
jaKa Močnik jaka@koofr.net
Fionera fionera@fionera.de
Dan Walters dan@walters.io
Danil Semelenov sgtpep@users.noreply.github.com
xopez 28950736+xopez@users.noreply.github.com
Ben Boeckel mathstuf@gmail.com
Manu manu@snapdragon.cc
Kyle E. Mitchell kyle@kemitchell.com
Gary Kim gary@garykim.dev
Jon jonathn@github.com
Jeff Quinn jeffrey.quinn@bluevoyant.com
Peter Berbec peter@berbec.com
didil 1284255+didil@users.noreply.github.com
id01 gaviniboom@gmail.com
Robert Marko robimarko@gmail.com
Philip Harvey 32467456+pharveybattelle@users.noreply.github.com
JorisE JorisE@users.noreply.github.com
garry415 garry.415@gmail.com
forgems forgems@gmail.com
Florian Apolloner florian@apolloner.eu
Aleksandar Janković office@ajankovic.com ajankovic@users.noreply.github.com
Maran maran@protonmail.com
nguyenhuuluan434 nguyenhuuluan434@gmail.com
Laura Hausmann zotan@zotan.pw laura@hausmann.dev
yparitcher y@paritcher.com
AbelThar abela.tharen@gmail.com
Matti Niemenmaa matti.niemenmaa+git@iki.fi
Russell Davis russelldavis@users.noreply.github.com
Yi FU yi.fu@tink.se
Paul Millar paul.millar@desy.de
justinalin justinalin@qnap.com
EliEron subanimehd@gmail.com
justina777 chiahuei.lin@gmail.com
Chaitanya Bankanhal bchaitanya15@gmail.com
Michał Matczuk michal@scylladb.com
Macavirus macavirus@zoho.com
Abhinav Sharma abhi18av@outlook.com
ginvine 34869051+ginvine@users.noreply.github.com
Patrick Wang mail6543210@yahoo.com.tw
Cenk Alti cenkalti@gmail.com
Andreas Chlupka andy@chlupka.com
Alfonso Montero amontero@tinet.org
Ivan Andreev ivandeex@gmail.com
David Baumgold david@davidbaumgold.com
Lars Lehtonen lars.lehtonen@gmail.com
Matei David matei.david@gmail.com
David david.bramwell@endemolshine.com
Anthony Rusdi 33247310+antrusd@users.noreply.github.com
Richard Patel me@terorie.dev
庄天翼 zty0826@gmail.com
SwitchJS dev@switchjs.com
Raphael PowershellNinja@users.noreply.github.com
Sezal Agrawal sezalagrawal@gmail.com
Tyler TylerNakamura@users.noreply.github.com
Brett Dutro brett.dutro@gmail.com
Vighnesh SK booterror99@gmail.com
Arijit Biswas dibbyo456@gmail.com
Michele Caci michele.caci@gmail.com
AlexandrBoltris ua2fgb@gmail.com
Bryce Larson blarson@saltstack.com
Carlos Ferreyra crypticmind@gmail.com
Saksham Khanna sakshamkhanna@outlook.com
dausruddin 5763466+dausruddin@users.noreply.github.com
zero-24 zero-24@users.noreply.github.com
Xiaoxing Ye ye@xiaoxing.us
Barry Muldrey barry@muldrey.net
Sebastian Brandt sebastian.brandt@friday.de
Marco Molteni marco.molteni@mailbox.org
Ankur Gupta 7876747+ankur0493@users.noreply.github.com
Maciej Zimnoch maciej@scylladb.com
anuar45 serdaliyev.anuar@gmail.com
Fernando ferferga@users.noreply.github.com
David Cole david.cole@sohonet.com
Wei He git@weispot.com
Outvi V 19144373+outloudvi@users.noreply.github.com
Thomas Kriechbaumer thomas@kriechbaumer.name
Tennix tennix@users.noreply.github.com
Ole Schütt ole@schuett.name
Kuang-che Wu kcwu@csie.org
Thomas Eales wingsuit@users.noreply.github.com
Paul Tinsley paul.tinsley@vitalsource.com
Felix Hungenberg git@shiftgeist.com
Benjamin Richter github@dev.telepath.de
landall cst_zf@qq.com
thestigma thestigma@gmail.com
jtagcat 38327267+jtagcat@users.noreply.github.com
Damon Permezel permezel@me.com
boosh boosh@users.noreply.github.com
unbelauscht 58393353+unbelauscht@users.noreply.github.com
Motonori IWAMURO vmi@nifty.com
Benjapol Worakan benwrk@live.com
Dave Koston dave.koston@stackpath.com
Durval Menezes DurvalMenezes@users.noreply.github.com
Tim Gallant me@timgallant.us
Frederick Zhang frederick888@tsundere.moe
valery1707 valery1707@gmail.com
Yves G theYinYeti@yalis.fr
Shing Kit Chan chanshingkit@gmail.com
Franklyn Tackitt franklyn@tackitt.net
Robert-André Mauchin zebob.m@gmail.com
evileye 48332831+ibiruai@users.noreply.github.com
Joachim Brandon LeBlanc brandon@leblanc.codes
Patryk Jakuszew patryk.jakuszew@gmail.com
fishbullet shindu666@gmail.com
greatroar <@>
Bernd Schoolmann mail@quexten.com
Elan Ruusamäe glen@pld-linux.org
Max Sum max@lolyculture.com
Mark Spieth mspieth@users.noreply.github.com
harry me@harry.plus
Samantha McVey samantham@posteo.net
Jack Anderson jack.anderson@metaswitch.com
Michael G draget@speciesm.net
Brandon Philips brandon@ifup.org
Daven dooven@users.noreply.github.com
Martin Stone martin@d7415.co.uk
David Bramwell 13053834+dbramwell@users.noreply.github.com
Sunil Patra snl_su@live.com
Adam Stroud adam.stroud@gmail.com
Kush kushsharma@users.noreply.github.com
Matan Rosenberg matan129@gmail.com
gitch1 63495046+gitch1@users.noreply.github.com
ElonH elonhhuang@gmail.com
Fred fred@creativeprojects.tech
Sébastien Gross renard@users.noreply.github.com
Maxime Suret 11944422+msuret@users.noreply.github.com
Caleb Case caleb@storj.io calebcase@gmail.com
Ben Zenker imbenzenker@gmail.com
Martin Michlmayr tbm@cyrius.com
Brandon McNama bmcnama@pagerduty.com
Daniel Slyman github@skylayer.eu
Alex Guerrero guerrero@users.noreply.github.com
Matteo Pietro Dazzi matteopietro.dazzi@gft.com
edwardxml 56691903+edwardxml@users.noreply.github.com
Roman Kredentser shareed2k@gmail.com
Kamil Trzciński ayufan@ayufan.eu
Zac Rubin z-0@users.noreply.github.com
Vincent Feltz
Heiko Bornholdt bornholdt@informatik.uni-hamburg.de
Matteo Pietro Dazzi matteopietro.dazzi@gmail.com
jtagcat gitlab@c7.ee
Petri Salminen petri@salminen.dev
Tim Burke tim.burke@gmail.com
Kai Lüke kai@kinvolk.io
Garrett Squire github@garrettsquire.com
Evan Harris eharris@puremagic.com
Kevin keyam@microsoft.com
Morten Linderud morten@linderud.pw
Dmitry Ustalov dmitry.ustalov@gmail.com
Jack 196648+jdeng@users.noreply.github.com
kcris cristian.tarsoaga@gmail.com
tyhuber1 68970760+tyhuber1@users.noreply.github.com
David Ibarra david.ibarra@realty.com
Tim Gallant tim@lilt.com
Kaloyan Raev kaloyan@storj.io
Jay McEntire jay.mcentire@gmail.com
Leo Luan leoluan@us.ibm.com
aus 549081+aus@users.noreply.github.com
Aaron Gokaslan agokaslan@fb.com
Egor Margineanu egmar@users.noreply.github.com
Lucas Kanashiro lucas.kanashiro@canonical.com
WarpedPixel WarpedPixel@users.noreply.github.com
Sam Edwards sam@samedwards.ca
wjielai gouki0123@gmail.com
Muffin King jinxz_k@live.com
Christopher Stewart 6573710+1f47a@users.noreply.github.com
Russell Cattelan cattelan@digitalelves.com
gyutw 30371241+gyutw@users.noreply.github.com
Hekmon edouardhur@gmail.com
LaSombra lasombra@users.noreply.github.com
Dov Murik dov.murik@gmail.com
Ameer Dawood ameer1234567890@gmail.com
Dan Hipschman dan.hipschman@opendoor.com
Josh Soref jsoref@users.noreply.github.com
David david@staron.nl
Ingo ingo@hoffmann.cx
Adam Plánský adamplansky@users.noreply.github.com adamplansky@gmail.com
Manish Gupta manishgupta.ait@gmail.com
Deepak Sah sah.sslpu@gmail.com
Marcin Zelent marcin@zelent.net
zhucan zhucan.k8s@gmail.com
James Lim james.lim@samsara.com
Laurens Janssen BD69BM@insim.biz
Bob Bagwill bobbagwill@gmail.com
Nathan Collins colli372@msu.edu
lostheli
kelv kelvin@acks.org
Milly milly.ca@gmail.com
gtorelly gtorelly@gmail.com
Brad Ackerman brad@facefault.org
Mitsuo Heijo mitsuo.heijo@gmail.com
Claudio Bantaloukas rockdreamer@gmail.com
Benjamin Gustin gustin.ben@gmail.com
Ingo Weiss ingo@redhat.com
Kerry Su me@sshockwave.net
Ilyess Bachiri ilyess.bachiri@sonder.com
Yury Stankevich urykhy@gmail.com
kice wslikerqs@gmail.com
Denis Neuling denisneuling@gmail.com
Janne Johansson icepic.dz@gmail.com
Patrik Nordlén patriki@gmail.com
CokeMine aptx4561@gmail.com
Sơn Trần-Nguyễn github@sntran.com
lluuaapp 266615+lluuaapp@users.noreply.github.com
Zach Kipp kipp.zach@gmail.com
Riccardo Iaconelli riccardo@kde.org
Sakuragawa Misty gyc990326@gmail.com
Nicolas Rueff nicolas@rueff.fr
Pau Rodriguez-Estivill prodrigestivill@gmail.com
Bob Pusateri BobPusateri@users.noreply.github.com
Alex JOST 25005220+dimejo@users.noreply.github.com
Alexey Tabakman samosad.ru@gmail.com
David Sze sze.david@gmail.com
cynthia kwok cynthia.m.kwok@gmail.com
Miron Veryanskiy MironVeryanskiy@gmail.com
K265 k.265@qq.com
Vesnyx Vesnyx@users.noreply.github.com
Dmitry Chepurovskiy me@dm3ch.net
Rauno Ots rauno.ots@cgi.com
Georg Neugschwandtner georg.neugschwandtner@gmx.net
pvalls polvallsrue@gmail.com
Robert Thomas 31854736+wolveix@users.noreply.github.com
Romeo Kienzler romeo.kienzler@gmail.com
tYYGH tYYGH@users.noreply.github.com
georne 77802995+georne@users.noreply.github.com
Maxwell Calman mcalman@MacBook-Pro.local
Naveen Honest Raj naveendurai19@gmail.com
Lucas Messenger lmesseng@cisco.com
Manish Kumar krmanish260@gmail.com
x0b x0bdev@gmail.com
CERN through the CS3MESH4EOSC Project
Nick Gaya nicholasgaya+github@gmail.com
Ashok Gelal 401055+ashokgelal@users.noreply.github.com
Dominik Mydlil dominik.mydlil@outlook.com
Nazar Mishturak nazarmx@gmail.com
Ansh Mittal iamAnshMittal@gmail.com
noabody noabody@yahoo.com
OleFrost 82263101+olefrost@users.noreply.github.com
Kenny Parsons kennyparsons93@gmail.com
Jeffrey Tolar tolar.jeffrey@gmail.com
jtagcat git-514635f7@jtag.cat
Tatsuya Noyori 63089076+public-tatsuya-noyori@users.noreply.github.com
lewisxy lewisxy@users.noreply.github.com
Nolan Woods nolan_w@sfu.ca
Gautam Kumar 25435568+gautamajay52@users.noreply.github.com
Chris Macklin chris.macklin@10xgenomics.com
Antoon Prins antoon.prins@surfsara.nl
Alexey Ivanov rbtz@dropbox.com
Serge Pouliquen sp31415@free.fr
acsfer carlos@reendex.com
Tom tom@tom-fitzhenry.me.uk
Tyson Moore tyson@tyson.me
database64128 free122448@hotmail.com
Chris Lu chrislusf@users.noreply.github.com
Reid Buzby reid@rethink.software
darrenrhs darrenrhs@gmail.com
Florian Penzkofer fp@nullptr.de
Xuanchen Wu 117010292@link.cuhk.edu.cn
partev petrosyan@gmail.com
Dmitry Sitnikov fo2@inbox.ru
Haochen Tong i@hexchain.org
Michael Hanselmann public@hansmi.ch
Chuan Zh zhchuan7@gmail.com
Antoine GIRARD antoine.girard@sapk.fr
Justin Winokur (Jwink3101) Jwink3101@users.noreply.github.com
Mariano Absatz (git) scm@baby.com.ar
Greg Sadetsky lepetitg@gmail.com
yedamo logindaveye@gmail.com
hota lindwurm.q@gmail.com
vinibali vinibali1@gmail.com
Ken Enrique Morel ken.morel.santana@gmail.com
Justin Hellings justin.hellings@gmail.com
Parth Shukla pparth@pparth.net
wzl wangzl31@outlook.com
HNGamingUK connor@earnshawhome.co.uk
Jonta 359397+Jonta@users.noreply.github.com
YenForYang YenForYang@users.noreply.github.com
SimJoSt / Joda Stößer git@simjo.st
Logeshwaran waranlogesh@gmail.com
Rajat Goel rajat@dropbox.com
r0kk3rz r0kk3rz@gmail.com
Matthew Sevey mjsevey@gmail.com
Filip Rysavy fil@siasky.net
Ian Levesque ian@ianlevesque.org
Thomas Stachl thomas@stachl.me
Dmitry Bogatov git#v1@kaction.cc
thomae 4493560+thomae@users.noreply.github.com
trevyn trevyn-git@protonmail.com
David Liu david.yx.liu@oracle.com
Chris Nelson stuff@cjnaz.com
Felix Bünemann felix.buenemann@gmail.com
Atílio Antônio atiliodadalto@hotmail.com
Carlo Mion mion00@gmail.com
Chris Lu chris.lu@gmail.com
Vitor Arruda vitor.pimenta.arruda@gmail.com
bbabich bbabich@datamossa.com
David dp.davide.palma@gmail.com
Borna Butkovic borna@favicode.net
Fredric Arklid fredric.arklid@consid.se
Andy Jackson Andrew.Jackson@bl.uk
Sinan Tan i@tinytangent.com
deinferno 14363193+deinferno@users.noreply.github.com
rsapkf rsapkfff@pm.me
Will Holtz wholtz@gmail.com
GGG KILLER gggkiller2@gmail.com
Logeshwaran Murugesan logeshwaran@testpress.in
Lu Wang coolwanglu@gmail.com
Bumsu Hyeon ksitht@gmail.com
Shmz Ozggrn 98463324+ShmzOzggrn@users.noreply.github.com
Kim kim@jotta.no
Niels van de Weem n.van.de.weem@smile.nl
Koopa codingkoopa@gmail.com
Yunhai Luo yunhai-luo@hotmail.com
Charlie Jiang w@chariri.moe
Alain Nussbaumer alain.nussbaumer@alleluia.ch
Vanessasaurus 814322+vsoch@users.noreply.github.com
Isaac Levy isaac.r.levy@gmail.com
Gourav T workflowautomation@protonmail.com
Paulo Martins paulo.pontes.m@gmail.com
viveknathani viveknathani2402@gmail.com
Eng Zer Jun engzerjun@gmail.com
Abhiraj abhiraj.official15@gmail.com
Márton Elek elek@apache.org elek@users.noreply.github.com
Vincent Murphy vdm@vdm.ie
ctrl-q 34975747+ctrl-q@users.noreply.github.com
Nil Alexandrov nalexand@akamai.com
GuoXingbin 101376330+guoxingbin@users.noreply.github.com
Berkan Teber berkan@berkanteber.com
Tobias Klauser tklauser@distanz.ch
KARBOWSKI Piotr piotr.karbowski@gmail.com
GH geeklihui@foxmail.com
rafma0 int.main@gmail.com
Adrien Rey-Jarthon jobs@adrienjarthon.com
Nick Gooding 73336146+nickgooding@users.noreply.github.com
Leroy van Logchem lr.vanlogchem@gmail.com
Zsolt Ero zsolt.ero@gmail.com
Lesmiscore nao20010128@gmail.com
ehsantdy ehsan.tadayon@arvancloud.com ehsantadayon85@gmail.com
SwazRGB 65694696+swazrgb@users.noreply.github.com
Mateusz Puczyński mati6095@gmail.com
Michael C Tiernan - MIT-Research Computing Project mtiernan@mit.edu
Kaspian 34658474+KaspianDev@users.noreply.github.com
Werner EvilOlaf@users.noreply.github.com
Hugal31 hugo.laloge@gmail.com
Christian Galo 36752715+cgalo5758@users.noreply.github.com
Erik van Velzen erik@evanv.nl
Derek Battams derek@battams.ca
Paul devnoname120@gmail.com
SimonLiu simonliu009@users.noreply.github.com
Hugo Laloge hla@lescompanions.com
Mr-Kanister 68117355+Mr-Kanister@users.noreply.github.com
Rob Pickerill r.pickerill@gmail.com
Andrey to.merge@gmail.com
Eric Wolf 19wolf@gmail.com
Nick nick.naumann@mailbox.tu-dresden.de
Jason Zheng jszheng17@gmail.com
Matthew Vernon mvernon@wikimedia.org
Noah Hsu i@nn.ci
m00594701 mengpengbo@huawei.com
Art M. Gallagher artmg50@gmail.com
Sven Gerber 49589423+svengerber@users.noreply.github.com
CrossR r.cross@lancaster.ac.uk
Maciej Radzikowski maciej@radzikowski.com.pl
Scott Grimes scott.grimes@spaciq.com
Phil Shackleton 71221528+philshacks@users.noreply.github.com
eNV25 env252525@gmail.com
Caleb inventor96@users.noreply.github.com
J-P Treen jp@wraptious.com
Martin Czygan 53705+miku@users.noreply.github.com
buda sandrojijavadze@protonmail.com
mirekphd 36706320+mirekphd@users.noreply.github.com
vyloy vyloy@qq.com
Anthrazz 25553648+Anthrazz@users.noreply.github.com
zzr93 34027824+zzr93@users.noreply.github.com
Paul Norman penorman@mac.com
Lorenzo Maiorfi maiorfi@gmail.com
Claudio Maradonna penguyman@stronzi.org
Ovidiu Victor Tatar ovi.tatar@googlemail.com
Evan Spensley epspensley@gmail.com
Yen Hu 61753151+0x59656e@users.noreply.github.com
Steve Kowalik steven@wedontsleep.org
Jordi Gonzalez Muñoz jordigonzm@gmail.com
Joram Schrijver i@joram.io
Mark Trolley marktrolley@gmail.com
João Henrique Franco joaohenrique.franco@gmail.com
anonion aman207@users.noreply.github.com
Ryan Morey 4590343+rmorey@users.noreply.github.com
Simon Bos simonbos9@gmail.com
YFdyh000 yfdyh000@gmail.com * Josh Soref 2119212+jsoref@users.noreply.github.com
Øyvind Heddeland Instefjord instefjord@outlook.com
Dmitry Deniskin 110819396+ddeniskin@users.noreply.github.com
Alexander Knorr 106825+opexxx@users.noreply.github.com
Richard Bateman richard@batemansr.us
Dimitri Papadopoulos Orfanos 3234522+DimitriPapadopoulos@users.noreply.github.com
Lorenzo Milesi lorenzo.milesi@yetopen.com
Isaac Aymerich isaac.aymerich@gmail.com
YanceyChiew 35898533+YanceyChiew@users.noreply.github.com
Manoj Ghosh msays2000@gmail.com
Bachue Zhou bachue.shu@gmail.com
Manoj Ghosh manoj.ghosh@oracle.com
Tom Mombourquette tom@devnode.com
Robert Newson rnewson@apache.org
Samuel Johnson esamueljohnson@gmail.com
coultonluke luke@luke.org.uk
Anthony Pessy anthony@cogniteev.com
Philip Harvey pharvey@battelleecology.org
dgouju dgouju@users.noreply.github.com
Clément Notin clement.notin@gmail.com
x3-apptech 66947598+x3-apptech@users.noreply.github.com
Arnie97 arnie97@gmail.com
Roel Arents 2691308+roelarents@users.noreply.github.com
Aaron Gokaslan aaronGokaslan@gmail.com
techknowlogick matti@mdranta.net
rkettelerij richard@mindloops.nl
Kamui fin-kamui@pm.me
asdffdsazqqq 90116442+asdffdsazqqq@users.noreply.github.com
Nathaniel Wesley Filardo nfilardo@microsoft.com
ycdtosa ycdtosa@users.noreply.github.com
Erik Agterdenbos agterdenbos@users.noreply.github.com
Kevin Verstaen 48050031+kverstae@users.noreply.github.com
MohammadReza mrvashian@gmail.com
vanplus 60313789+vanplus@users.noreply.github.com
Jack 16779171+jkpe@users.noreply.github.com
Abdullah Saglam abdullah.saglam@stonebranch.com
Marks Polakovs github@markspolakovs.me
piyushgarg piyushgarg80@gmail.com
Kaloyan Raev kaloyan-raev@users.noreply.github.com
IMTheNachoMan imthenachoman@gmail.com
alankrit alankrit@google.com
Bryan Kaplan #@bryankaplan.com
LXY 767763591@qq.com
Simmon Li (he/him) li.simmon@gmail.com
happyxhw 44490504+happyxhw@users.noreply.github.com
Simmon Li (he/him) hello@crespire.dev
Matthias Baur baurmatt@users.noreply.github.com
Hunter Wittenborn hunter@hunterwittenborn.com
logopk peter@kreuser.name
Gerard Bosch 30733556+gerardbosch@users.noreply.github.com
ToBeFree github@tfrei.de
NodudeWasTaken 75137537+NodudeWasTaken@users.noreply.github.com
Peter Brunner peter@lugoues.net
Ninh Pham dongian.rapclubkhtn@gmail.com
Ryan Caezar Itang sitiom@proton.me
Peter Brunner peter@psykhe.com
Leandro Sacchet leandro.sacchet@animati.com.br
dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
cycneuramus 56681631+cycneuramus@users.noreply.github.com
Arnavion me@arnavion.dev
Christopher Merry christopher.merry@mlb.com
Thibault Coupin thibault.coupin@gmail.com
Richard Tweed RichardoC@users.noreply.github.com
Zach Kipp Zacho2@users.noreply.github.com
yuudi 26199752+yuudi@users.noreply.github.com
NickIAm NickIAm@users.noreply.github.com
Juang, Yi-Lin frankyjuang@gmail.com
jumbi77 jumbi77@users.noreply.github.com
Aditya Basu ab.aditya.basu@gmail.com
ed s@ocv.me
Drew Parsons dparsons@emerall.com
Joel joelnb@users.noreply.github.com
wiserain mail275@gmail.com
Roel Arents roel.arents@kadaster.nl
Shyim github@shyim.de
Rintze Zelle 78232505+rzelle-lallemand@users.noreply.github.com
Damo damoclark@users.noreply.github.com
WeidiDeng weidi_deng@icloud.com
Brian Starkey stark3y@gmail.com
jladbrook jhladbrook@gmail.com
Loren Gordon lorengordon@users.noreply.github.com
dlitster davidlitster@gmail.com
Tobias Gion tobias@gion.io
Jānis Bebrītis janis.bebritis@wunder.io
Adam K github.com@ak.tidy.email
Andrei Smirnov smirnov.captain@gmail.com
Janne Hellsten jjhellst@gmail.com
cc 12904584+shvc@users.noreply.github.com
Tareq Sharafy tareq.sha@gmail.com
kapitainsky dariuszb@me.com
douchen playgoobug@gmail.com
Sam Lai 70988+slai@users.noreply.github.com
URenko 18209292+URenko@users.noreply.github.com
Stanislav Gromov kullfar@gmail.com
Paulo Schreiner paulo.schreiner@delivion.de
Mariusz Suchodolski mariusz@suchodol.ski
danielkrajnik dan94kra@gmail.com
Peter Fern github@0xc0dedbad.com
zzq i@zhangzqs.cn
mac-15 usman.ilamdin@phpstudios.com
Sawada Tsunayoshi 34431649+TsunayoshiSawada@users.noreply.github.com
Dean Attali daattali@gmail.com
Fjodor42 molgaard@gmail.com
BakaWang wa11579@hotmail.com
Mahad 56235065+Mahad-lab@users.noreply.github.com
Vladislav Vorobev x.miere@gmail.com
darix darix@users.noreply.github.com
Benjamin 36415086+bbenjamin-sys@users.noreply.github.com
Chun-Hung Tseng henrybear327@users.noreply.github.com
Ricardo D'O. Albanus rdalbanus@users.noreply.github.com
gabriel-suela gscsuela@gmail.com
Tiago Boeing contato@tiagoboeing.com
Edwin Mackenzie-Owen edwin.mowen@gmail.com
Niklas Hambüchen mail@nh2.me
yuudi yuudi@users.noreply.github.com
Zach github@prozach.org
nielash 31582349+nielash@users.noreply.github.com
Julian Lepinski lepinsk@users.noreply.github.com
Raymond Berger RayBB@users.noreply.github.com
Nihaal Sangha nihaal.git@gmail.com
Masamune3210 1053504+Masamune3210@users.noreply.github.com
James Braza jamesbraza@gmail.com
antoinetran antoinetran@users.noreply.github.com
alexia me@alexia.lol
nielash nielronash@gmail.com
Vitor Gomes vitor.gomes@delivion.de mail@vitorgomes.com
Jacob Hands jacob@gogit.io
hideo aoyama 100831251+boukendesho@users.noreply.github.com
Roberto Ricci io@r-ricci.it
Bjørn Smith bjornsmith@gmail.com
Alishan Ladhani 8869764+aladh@users.noreply.github.com
zjx20 zhoujianxiong2@gmail.com
Oksana 142890647+oks-maytech@users.noreply.github.com
Volodymyr Kit v.kit@maytech.net
David Pedersen limero@me.com
Drew Stinnett drew@drewlink.com
Pat Patterson pat@backblaze.com
Herby Gillot herby.gillot@gmail.com
Nikita Shoshin shoshin_nikita@fastmail.com
rinsuki 428rinsuki+git@gmail.com
Beyond Meat 51850644+beyondmeat@users.noreply.github.com
Saleh Dindar salh@fb.com
Volodymyr 142890760+vkit-maytech@users.noreply.github.com
Gabriel Espinoza 31670639+gspinoza@users.noreply.github.com
Keigo Imai keigo.imai@gmail.com
Ivan Yanitra iyanitra@tesla-consulting.com
alfish2000 alfish2000@gmail.com
wuxingzhong qq330332812@gmail.com
Adithya Kumar akumar42@protonmail.com
Tayo-pasedaRJ 138471223+Tayo-pasedaRJ@users.noreply.github.com
Peter Kreuser logo@kreuser.name
Piyush
fotile96 fotile96@users.noreply.github.com
Luc Ritchie luc.ritchie@gmail.com
cynful cynful@users.noreply.github.com
wjielai wjielai@tencent.com
Jack Deng jackdeng@gmail.com
Mikubill 31246794+Mikubill@users.noreply.github.com
Artur Neumann artur@jankaritech.com
Saw-jan saw.jan.grg3e@gmail.com
Oksana Zhykina o.zhykina@maytech.net
karan karan.gupta92@gmail.com
viktor viktor@yakovchuk.net
moongdal moongdal@tutanota.com
Mina Galić freebsd@igalic.co
Alen Šiljak dev@alensiljak.eu.org
你知道未来吗 rkonfj@gmail.com
Abhinav Dhiman 8640877+ahnv@users.noreply.github.com
halms 7513146+halms@users.noreply.github.com
ben-ba benjamin.brauner@gmx.de
Eli Orzitzer e_orz@yahoo.com
Anthony Metzidis anthony.metzidis@gmail.com
emyarod afw5059@gmail.com
keongalvin keongalvin@gmail.com
rarspace01 rarspace01@users.noreply.github.com
Paul Stern paulstern45@gmail.com
Nikhil Ahuja nikhilahuja@live.com
Harshit Budhraja 52413945+harshit-budhraja@users.noreply.github.com
Tera 24725862+teraa@users.noreply.github.com
Kyle Reynolds kylereynoldsdev@gmail.com
Michael Eischer michael.eischer@gmx.de
Thomas Müller 1005065+DeepDiver1975@users.noreply.github.com
DanielEgbers 27849724+DanielEgbers@users.noreply.github.com
Jack Provance 49460795+njprov@users.noreply.github.com
Gabriel Ramos 109390599+gabrielramos02@users.noreply.github.com
Dan McArdle d@nmcardle.com
Joe Cai joe.cai@bigcommerce.com
Anders Swanson anders.swanson@oracle.com
huajin tong 137764712+thirdkeyword@users.noreply.github.com
John-Paul Smith john-paulsmith@users.noreply.github.com
racerole 148756161+racerole@users.noreply.github.com
Gachoud Philippe ph.gachoud@gmail.com
YukiUnHappy saberhana@yandex.com
Kyle Reynolds kyle.reynolds@bridgerphotonics.com
Lewis Hook lewis@hook.im
hoyho luohaihao@gmail.com
Vitaly 9034218+gvitali@users.noreply.github.com
iotmaestro iotmaestro@proton.me
psychopatt 66741203+psychopatt@users.noreply.github.com
Alex Garel alex@garel.org
Warrentheo warrentheo@hotmail.com
Alexandre Lavigne lavigne958@gmail.com
yoelvini 134453420+yoelvini@users.noreply.github.com
Erisa A erisa@cloudflare.com
Pieter van Oostrum pieter@vanoostrum.org
jakzoe 155812065+jakzoe@users.noreply.github.com
guangwu guoguangwu@magic-shield.com
static-moonlight 107991124+static-moonlight@users.noreply.github.com
yudrywet yudeyao@yeah.net
Butanediol git@xnh.app
Dave Nicolson david.nicolson@gmail.com
Katia Esposito katia@linux.com
pawsey-kbuckley 36438302+pawsey-kbuckley@users.noreply.github.com
hidewrong 167099254+hidewrong@users.noreply.github.com
Michael Terry mike@mterry.name
Sunny 25066078+LoSunny@users.noreply.github.com
overallteach cricis@foxmail.com
JT Olio jt@olio.lol
Evan McBeth 64177332+AtomicRobotMan0101@users.noreply.github.com
Dominik Joe Pantůček dominik.pantucek@trustica.cz
yumeiyin 155420652+yumeiyin@users.noreply.github.com
Bruno Fernandes 54373093+folkzb@users.noreply.github.com
Thomas Schneider tspam.github@brainfuck.space
Charles Hamilton 52973156+chamilton-ccn@users.noreply.github.com
Tomasz Melcer tomasz@melcer.pl
Michał Dzienisiewicz michal.piotr.dz@gmail.com
Florian Klink flokli@flokli.de
Bill Fraser bill@wfraser.dev
Thearas thearas850@gmail.com
Filipe Herculano fifo_@live.com
Russ Bubley russ.bubley@googlemail.com
Paul Collins paul.collins@canonical.com
Tomasz Melcer liori@exroot.org
itsHenry 2671230065@qq.com
Ke Wang me@ke.wang
AThePeanut4 49614525+AThePeanut4@users.noreply.github.com
Tobias Markus tobbi.bugs@googlemail.com
Ernie Hershey github@ernie.org
Will Miles wmiles@sgl.com
David Seifert 16636962+SoapGentoo@users.noreply.github.com
Fornax wimbrand96@gmail.com
Sam Harrison sam.harrison@files.com
Péter Bozsó 3806723+peterbozso@users.noreply.github.com
Georg Welzel gwelzel@mailbox.org
John Oxley john.oxley@gmail.com joxley@meta.com
Pawel Palucha pawel.palucha@aetion.com
crystalstall crystalruby@qq.com
nipil nipil@users.noreply.github.com
yuval-cloudinary 46710068+yuval-cloudinary@users.noreply.github.com
Mathieu Moreau mrx23dot@users.noreply.github.com
fsantagostinobietti 6057026+fsantagostinobietti@users.noreply.github.com
Oleg Kunitsyn 114359669+hiddenmarten@users.noreply.github.com

B2

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Backblaze B2

B2 is Backblaze's cloud storage system.

Paths are specified as remote:bucket (or remote: for the lsd command.) You may put subdirectories in too, e.g. remote:bucket/path/to/dir.

Configuration

Here is an example of making a b2 configuration. First run

rclone config

This will guide you through an interactive setup process. To authenticate you will either need your Account ID (a short hex number) and Master Application Key (a long hex number) OR an Application Key, which is the recommended method. See below for further details on generating and using an Application Key.

No remotes found, make a new one?
n) New remote
q) Quit config
n/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Backblaze B2
   \ "b2"
[snip]
Storage> b2
Account ID or Application Key ID
account> 123456789abc
Application Key
key> 0123456789abcdef0123456789abcdef0123456789
Endpoint for the service - leave blank normally.
endpoint>
Remote config
Configuration complete.
Options:
- type: b2
- account: 123456789abc
- key: 0123456789abcdef0123456789abcdef0123456789
- endpoint:
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

This remote is called remote and can now be used like this

See all buckets

rclone lsd remote:

Create a new bucket

rclone mkdir remote:bucket

List the contents of a bucket

rclone ls remote:bucket

Sync /home/local/directory to the remote bucket, deleting any excess files in the bucket.

rclone sync --interactive /home/local/directory remote:bucket

Application Keys

B2 supports multiple Application Keys for different access permission to B2 Buckets.

You can use these with rclone too; you will need to use rclone version 1.43 or later.

Follow Backblaze's docs to create an Application Key with the required permission and add the applicationKeyId as the account and the Application Key itself as the key.

Note that you must put the applicationKeyId as the account – you can't use the master Account ID. If you try then B2 will return 401 errors.

--fast-list

This remote supports --fast-list which allows you to use fewer transactions in exchange for more memory. See the rclone docs for more details.

Modification times

The modification time is stored as metadata on the object as X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in the Backblaze standard. Other tools should be able to use this as a modified time.

Modified times are used in syncing and are fully supported. Note that if a modification time needs to be updated on an object then it will create a new version of the object.

Restricted filename characters

In addition to the default restricted characters set the following characters are also replaced:

Character	Value	Replacement
\	0x5C	＼

Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

Note that in 2020-05 Backblaze started allowing \ characters in file names. Rclone hasn't changed its encoding as this could cause syncs to re-transfer files. If you want rclone not to replace \ then see the --b2-encoding flag below and remove the BackSlash from the string. This can be set in the config.

SHA1 checksums

The SHA1 checksums of the files are checked on upload and download and will be used in the syncing process.

Large files (bigger than the limit in --b2-upload-cutoff) which are uploaded in chunks will store their SHA1 on the object as X-Bz-Info-large_file_sha1 as recommended by Backblaze.

For a large file to be uploaded with an SHA1 checksum, the source needs to support SHA1 checksums. The local disk supports SHA1 checksums so large file transfers from local disk will have an SHA1. See the overview for exactly which remotes support SHA1.

Sources which don't support SHA1, in particular crypt will upload large files without SHA1 checksums. This may be fixed in the future (see #1767).

Files sizes below --b2-upload-cutoff will always have an SHA1 regardless of the source.

Transfers

Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from my SSD equipped laptop the optimum setting is about --transfers 32 though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of --transfers 4 is definitely too low for Backblaze B2 though.

Note that uploading big files (bigger than 200 MiB by default) will use a 96 MiB RAM buffer by default. There can be at most --transfers of these in use at any moment, so this sets the upper limit on the memory used.

Versions

The default setting of B2 is to keep old versions of files. This means when rclone uploads a new version of a file it creates a new version of it. Likewise when you delete a file, the old version will be marked hidden and still be available.

Whether B2 keeps old versions of files or not can be adjusted on a per bucket basis using the "Lifecycle settings" on the B2 control panel or when creating the bucket using the --b2-lifecycle flag or after creation using the rclone backend lifecycle command.

You may opt in to a "hard delete" of files with the --b2-hard-delete flag which permanently removes files on deletion instead of hiding them.

Old versions of files, where available, are visible using the --b2-versions flag.

It is also possible to view a bucket as it was at a certain point in time, using the --b2-version-at flag. This will show the file versions as they were at that time, showing files that have been deleted afterwards, and hiding files that were created since.

If you wish to remove all the old versions, and unfinished large file uploads, then you can use the rclone cleanup remote:bucket command which will delete all the old versions of files, leaving the current ones intact. You can also supply a path and only old versions under that path will be deleted, e.g. rclone cleanup remote:bucket/path/to/stuff.

Note that cleanup will remove partially uploaded files from the bucket if they are more than a day old. If you want more control over the expiry date then run rclone backend cleanup b2:bucket -o max-age=1h to remove all unfinished large file uploads older than one hour, leaving old versions intact.

If you wish to remove all the old versions, leaving current files and unfinished large files intact, then you can use the rclone backend cleanup-hidden remote:bucket command. You can also supply a path and only old versions under that path will be deleted, e.g. rclone backend cleanup-hidden remote:bucket/path/to/stuff.

When you purge a bucket, the current and the old versions will be deleted then the bucket will be deleted.

However delete will cause the current versions of the files to become hidden old versions.

Here is a session showing the listing and retrieval of an old version followed by a cleanup of the old versions.

Show current version and all the versions with --b2-versions flag.

$ rclone -q ls b2:cleanup-test
        9 one.txt

$ rclone -q --b2-versions ls b2:cleanup-test
        9 one.txt
        8 one-v2016-07-04-141032-000.txt
       16 one-v2016-07-04-141003-000.txt
       15 one-v2016-07-02-155621-000.txt

Retrieve an old version

$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp

$ ls -l /tmp/one-v2016-07-04-141003-000.txt
-rw-rw-r-- 1 ncw ncw 16 Jul  2 17:46 /tmp/one-v2016-07-04-141003-000.txt

Clean up all the old versions and show that they've gone.

$ rclone -q cleanup b2:cleanup-test

$ rclone -q ls b2:cleanup-test
        9 one.txt

$ rclone -q --b2-versions ls b2:cleanup-test
        9 one.txt

Versions naming caveat

When using --b2-versions flag rclone is relying on the file name to work out whether the objects are versions or not. Versions' names are created by inserting timestamp between file name and its extension.

        9 file.txt
        8 file-v2023-07-17-161032-000.txt
       16 file-v2023-06-15-141003-000.txt

If there are real files present with the same names as versions, then behaviour of --b2-versions can be unpredictable.

Data usage

It is useful to know how many requests are sent to the server in different scenarios.

All copy commands send the following 4 requests:

/b2api/v1/b2_authorize_account
/b2api/v1/b2_create_bucket
/b2api/v1/b2_list_buckets
/b2api/v1/b2_list_file_names

The b2_list_file_names request will be sent once for every 1k files in the remote path, providing the checksum and modification time of the listed files. As of version 1.33 issue #818 causes extra requests to be sent when using B2 with Crypt. When a copy operation does not require any files to be uploaded, no more requests will be sent.

Uploading files that do not require chunking, will send 2 requests per file upload:

/b2api/v1/b2_get_upload_url
/b2api/v1/b2_upload_file/

Uploading files requiring chunking, will send 2 requests (one each to start and finish the upload) and another 2 requests for each chunk:

/b2api/v1/b2_start_large_file
/b2api/v1/b2_get_upload_part_url
/b2api/v1/b2_upload_part/
/b2api/v1/b2_finish_large_file

Versions

Versions can be viewed with the --b2-versions flag. When it is set rclone will show and act on older versions of files. For example

Listing without --b2-versions

$ rclone -q ls b2:cleanup-test
        9 one.txt

And with

$ rclone -q --b2-versions ls b2:cleanup-test
        9 one.txt
        8 one-v2016-07-04-141032-000.txt
       16 one-v2016-07-04-141003-000.txt
       15 one-v2016-07-02-155621-000.txt

Showing that the current version is unchanged but older versions can be seen. These have the UTC date that they were uploaded to the server to the nearest millisecond appended to them.

Note that when using --b2-versions no file write operations are permitted, so you can't upload files or delete them.

B2 and rclone link

Rclone supports generating file share links for private B2 buckets. They can either be for a file for example:

./rclone link B2:bucket/path/to/file.txt
https://f002.backblazeb2.com/file/bucket/path/to/file.txt?Authorization=xxxxxxxx

or if run on a directory you will get:

./rclone link B2:bucket/path
https://f002.backblazeb2.com/file/bucket/path?Authorization=xxxxxxxx

you can then use the authorization token (the part of the url from the ?Authorization= on) on any file path under that directory. For example:

https://f002.backblazeb2.com/file/bucket/path/to/file1?Authorization=xxxxxxxx
https://f002.backblazeb2.com/file/bucket/path/file2?Authorization=xxxxxxxx
https://f002.backblazeb2.com/file/bucket/path/folder/file3?Authorization=xxxxxxxx

Standard options

Here are the Standard options specific to b2 (Backblaze B2).

--b2-account

Account ID or Application Key ID.

Properties:

Config: account
Env Var: RCLONE_B2_ACCOUNT
Type: string
Required: true

--b2-key

Application Key.

Properties:

Config: key
Env Var: RCLONE_B2_KEY
Type: string
Required: true

--b2-hard-delete

Permanently delete files on remote removal, otherwise hide files.

Properties:

Config: hard_delete
Env Var: RCLONE_B2_HARD_DELETE
Type: bool
Default: false

Advanced options

Here are the Advanced options specific to b2 (Backblaze B2).

--b2-endpoint

Endpoint for the service.

Leave blank normally.

Properties:

Config: endpoint
Env Var: RCLONE_B2_ENDPOINT
Type: string
Required: false

--b2-test-mode

A flag string for X-Bz-Test-Mode header for debugging.

This is for debugging purposes only. Setting it to one of the strings below will cause b2 to return specific errors:

"fail_some_uploads"
"expire_some_account_authorization_tokens"
"force_cap_exceeded"

These will be set in the "X-Bz-Test-Mode" header which is documented in the b2 integrations checklist.

Properties:

Config: test_mode
Env Var: RCLONE_B2_TEST_MODE
Type: string
Required: false

--b2-versions

Include old versions in directory listings.

Note that when using this no file write operations are permitted, so you can't upload files or delete them.

Properties:

Config: versions
Env Var: RCLONE_B2_VERSIONS
Type: bool
Default: false

--b2-version-at

Show file versions as they were at the specified time.

Note that when using this no file write operations are permitted, so you can't upload files or delete them.

Properties:

Config: version_at
Env Var: RCLONE_B2_VERSION_AT
Type: Time
Default: off

--b2-upload-cutoff

Cutoff for switching to chunked upload.

Files above this size will be uploaded in chunks of "--b2-chunk-size".

This value should be set no larger than 4.657 GiB (== 5 GB).

Properties:

Config: upload_cutoff
Env Var: RCLONE_B2_UPLOAD_CUTOFF
Type: SizeSuffix
Default: 200Mi

--b2-copy-cutoff

Cutoff for switching to multipart copy.

Any files larger than this that need to be server-side copied will be copied in chunks of this size.

The minimum is 0 and the maximum is 4.6 GiB.

Properties:

Config: copy_cutoff
Env Var: RCLONE_B2_COPY_CUTOFF
Type: SizeSuffix
Default: 4Gi

--b2-chunk-size

Upload chunk size.

When uploading large files, chunk the file into this size.

Must fit in memory. These chunks are buffered in memory and there might a maximum of "--transfers" chunks in progress at once.

5,000,000 Bytes is the minimum size.

Properties:

Config: chunk_size
Env Var: RCLONE_B2_CHUNK_SIZE
Type: SizeSuffix
Default: 96Mi

--b2-upload-concurrency

Concurrency for multipart uploads.

This is the number of chunks of the same file that are uploaded concurrently.

Note that chunks are stored in memory and there may be up to "--transfers" * "--b2-upload-concurrency" chunks stored at once in memory.

Properties:

Config: upload_concurrency
Env Var: RCLONE_B2_UPLOAD_CONCURRENCY
Type: int
Default: 4

--b2-disable-checksum

Disable checksums for large (> upload cutoff) files.

Normally rclone will calculate the SHA1 checksum of the input before uploading it so it can add it to metadata on the object. This is great for data integrity checking but can cause long delays for large files to start uploading.

Properties:

Config: disable_checksum
Env Var: RCLONE_B2_DISABLE_CHECKSUM
Type: bool
Default: false

--b2-download-url

Custom endpoint for downloads.

This is usually set to a Cloudflare CDN URL as Backblaze offers free egress for data downloaded through the Cloudflare network. Rclone works with private buckets by sending an "Authorization" header. If the custom endpoint rewrites the requests for authentication, e.g., in Cloudflare Workers, this header needs to be handled properly. Leave blank if you want to use the endpoint provided by Backblaze.

The URL provided here SHOULD have the protocol and SHOULD NOT have a trailing slash or specify the /file/bucket subpath as rclone will request files with "{download_url}/file/{bucket_name}/{path}".

Example:

https://mysubdomain.mydomain.tld (No trailing "/", "file" or "bucket")

Properties:

Config: download_url
Env Var: RCLONE_B2_DOWNLOAD_URL
Type: string
Required: false

--b2-download-auth-duration

Time before the public link authorization token will expire in s or suffix ms|s|m|h|d.

This is used in combination with "rclone link" for making files accessible to the public and sets the duration before the download authorization token will expire.

The minimum value is 1 second. The maximum value is one week.

Properties:

Config: download_auth_duration
Env Var: RCLONE_B2_DOWNLOAD_AUTH_DURATION
Type: Duration
Default: 1w

--b2-memory-pool-flush-time

How often internal memory buffer pools will be flushed. (no longer used)

Properties:

Config: memory_pool_flush_time
Env Var: RCLONE_B2_MEMORY_POOL_FLUSH_TIME
Type: Duration
Default: 1m0s

--b2-memory-pool-use-mmap

Whether to use mmap buffers in internal memory pool. (no longer used)

Properties:

Config: memory_pool_use_mmap
Env Var: RCLONE_B2_MEMORY_POOL_USE_MMAP
Type: bool
Default: false

--b2-lifecycle

Set the number of days deleted files should be kept when creating a bucket.

On bucket creation, this parameter is used to create a lifecycle rule for the entire bucket.

If lifecycle is 0 (the default) it does not create a lifecycle rule so the default B2 behaviour applies. This is to create versions of files on delete and overwrite and to keep them indefinitely.

If lifecycle is >0 then it creates a single rule setting the number of days before a file that is deleted or overwritten is deleted permanently. This is known as daysFromHidingToDeleting in the b2 docs.

The minimum value for this parameter is 1 day.

You can also enable hard_delete in the config also which will mean deletions won't cause versions but overwrites will still cause versions to be made.

See: rclone backend lifecycle for setting lifecycles after bucket creation.

Properties:

Config: lifecycle
Env Var: RCLONE_B2_LIFECYCLE
Type: int
Default: 0

--b2-encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Properties:

Config: encoding
Env Var: RCLONE_B2_ENCODING
Type: Encoding
Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot

--b2-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_B2_DESCRIPTION
Type: string
Required: false

Backend commands

Here are the commands specific to the b2 backend.

Run them with

rclone backend COMMAND remote:

The help below will explain what arguments each command takes.

See the backend command for more info on how to pass options and arguments.

These can be run on a running backend using the rc command backend/command.

lifecycle

Read or set the lifecycle for a bucket

rclone backend lifecycle remote: [options] [<arguments>+]

This command can be used to read or set the lifecycle for a bucket.

Usage Examples:

To show the current lifecycle rules:

rclone backend lifecycle b2:bucket

This will dump something like this showing the lifecycle rules.

[
    {
        "daysFromHidingToDeleting": 1,
        "daysFromUploadingToHiding": null,
        "fileNamePrefix": ""
    }
]

If there are no lifecycle rules (the default) then it will just return [].

To reset the current lifecycle rules:

rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=30
rclone backend lifecycle b2:bucket -o daysFromUploadingToHiding=5 -o daysFromHidingToDeleting=1

This will run and then print the new lifecycle rules as above.

Rclone only lets you set lifecycles for the whole bucket with the fileNamePrefix = "".

You can't disable versioning with B2. The best you can do is to set the daysFromHidingToDeleting to 1 day. You can enable hard_delete in the config also which will mean deletions won't cause versions but overwrites will still cause versions to be made.

rclone backend lifecycle b2:bucket -o daysFromHidingToDeleting=1

See: https://www.backblaze.com/docs/cloud-storage-lifecycle-rules

Options:

"daysFromHidingToDeleting": After a file has been hidden for this many days it is deleted. 0 is off.
"daysFromUploadingToHiding": This many days after uploading a file is hidden

cleanup

Remove unfinished large file uploads.

rclone backend cleanup remote: [options] [<arguments>+]

This command removes unfinished large file uploads of age greater than max-age, which defaults to 24 hours.

Note that you can use --interactive/-i or --dry-run with this command to see what it would do.

rclone backend cleanup b2:bucket/path/to/object
rclone backend cleanup -o max-age=7w b2:bucket/path/to/object

Durations are parsed as per the rest of rclone, 2h, 7d, 7w etc.

Options:

"max-age": Max age of upload to delete

cleanup-hidden

Remove old versions of files.

rclone backend cleanup-hidden remote: [options] [<arguments>+]

This command removes any old hidden versions of files.

Note that you can use --interactive/-i or --dry-run with this command to see what it would do.

rclone backend cleanup-hidden b2:bucket/path/to/dir

Limitations

rclone about is not supported by the B2 backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

See List of backends that do not support rclone about and rclone about

Bisync

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Bisync

bisync is in beta and is considered an advanced command, so use with care. Make sure you have read and understood the entire manual (especially the Limitations section) before using, or data loss can result. Questions can be asked in the Rclone Forum.

Getting started

Install rclone and setup your remotes.
Bisync will create its working directory at ~/.cache/rclone/bisync on Linux, /Users/yourusername/Library/Caches/rclone/bisync on Mac, or C:\Users\MyLogin\AppData\Local\rclone\bisync on Windows. Make sure that this location is writable.
Run bisync with the --resync flag, specifying the paths to the local and remote sync directory roots.
For successive sync runs, leave off the --resync flag. (Important!)
Consider using a filters file for excluding unnecessary files and directories from the sync.
Consider setting up the --check-access feature for safety.
On Linux or Mac, consider setting up a crontab entry. bisync can safely run in concurrent cron jobs thanks to lock files it maintains.

For example, your first command might look like this:

rclone bisync remote1:path1 remote2:path2 --create-empty-src-dirs --compare size,modtime,checksum --slow-hash-sync-only --resilient -MvP --drive-skip-gdocs --fix-case --resync --dry-run

If all looks good, run it again without --dry-run. After that, remove --resync as well.

Here is a typical run log (with timestamps removed for clarity):

rclone bisync /testdir/path1/ /testdir/path2/ --verbose
INFO  : Synching Path1 "/testdir/path1/" with Path2 "/testdir/path2/"
INFO  : Path1 checking for diffs
INFO  : - Path1    File is new                         - file11.txt
INFO  : - Path1    File is newer                       - file2.txt
INFO  : - Path1    File is newer                       - file5.txt
INFO  : - Path1    File is newer                       - file7.txt
INFO  : - Path1    File was deleted                    - file4.txt
INFO  : - Path1    File was deleted                    - file6.txt
INFO  : - Path1    File was deleted                    - file8.txt
INFO  : Path1:    7 changes:    1 new,    3 newer,    0 older,    3 deleted
INFO  : Path2 checking for diffs
INFO  : - Path2    File is new                         - file10.txt
INFO  : - Path2    File is newer                       - file1.txt
INFO  : - Path2    File is newer                       - file5.txt
INFO  : - Path2    File is newer                       - file6.txt
INFO  : - Path2    File was deleted                    - file3.txt
INFO  : - Path2    File was deleted                    - file7.txt
INFO  : - Path2    File was deleted                    - file8.txt
INFO  : Path2:    7 changes:    1 new,    3 newer,    0 older,    3 deleted
INFO  : Applying changes
INFO  : - Path1    Queue copy to Path2                 - /testdir/path2/file11.txt
INFO  : - Path1    Queue copy to Path2                 - /testdir/path2/file2.txt
INFO  : - Path2    Queue delete                        - /testdir/path2/file4.txt
NOTICE: - WARNING  New or changed in both paths        - file5.txt
NOTICE: - Path1    Renaming Path1 copy                 - /testdir/path1/file5.txt..path1
NOTICE: - Path1    Queue copy to Path2                 - /testdir/path2/file5.txt..path1
NOTICE: - Path2    Renaming Path2 copy                 - /testdir/path2/file5.txt..path2
NOTICE: - Path2    Queue copy to Path1                 - /testdir/path1/file5.txt..path2
INFO  : - Path2    Queue copy to Path1                 - /testdir/path1/file6.txt
INFO  : - Path1    Queue copy to Path2                 - /testdir/path2/file7.txt
INFO  : - Path2    Queue copy to Path1                 - /testdir/path1/file1.txt
INFO  : - Path2    Queue copy to Path1                 - /testdir/path1/file10.txt
INFO  : - Path1    Queue delete                        - /testdir/path1/file3.txt
INFO  : - Path2    Do queued copies to                 - Path1
INFO  : - Path1    Do queued copies to                 - Path2
INFO  : -          Do queued deletes on                - Path1
INFO  : -          Do queued deletes on                - Path2
INFO  : Updating listings
INFO  : Validating listings for Path1 "/testdir/path1/" vs Path2 "/testdir/path2/"
INFO  : Bisync successful

Command line syntax

$ rclone bisync --help
Usage:
  rclone bisync remote1:path1 remote2:path2 [flags]

Positional arguments:
  Path1, Path2  Local path, or remote storage with ':' plus optional path.
                Type 'rclone listremotes' for list of configured remotes.

Optional Flags:
      --backup-dir1 string                   --backup-dir for Path1. Must be a non-overlapping path on the same remote.
      --backup-dir2 string                   --backup-dir for Path2. Must be a non-overlapping path on the same remote.
      --check-access                         Ensure expected RCLONE_TEST files are found on both Path1 and Path2 filesystems, else abort.
      --check-filename string                Filename for --check-access (default: RCLONE_TEST)
      --check-sync string                    Controls comparison of final listings: true|false|only (default: true) (default "true")
      --compare string                       Comma-separated list of bisync-specific compare options ex. 'size,modtime,checksum' (default: 'size,modtime')
      --conflict-loser ConflictLoserAction   Action to take on the loser of a sync conflict (when there is a winner) or on both files (when there is no winner): , num, pathname, delete (default: num)
      --conflict-resolve string              Automatically resolve conflicts by preferring the version that is: none, path1, path2, newer, older, larger, smaller (default: none) (default "none")
      --conflict-suffix string               Suffix to use when renaming a --conflict-loser. Can be either one string or two comma-separated strings to assign different suffixes to Path1/Path2. (default: 'conflict')
      --create-empty-src-dirs                Sync creation and deletion of empty directories. (Not compatible with --remove-empty-dirs)
      --download-hash                        Compute hash by downloading when otherwise unavailable. (warning: may be slow and use lots of data!)
      --filters-file string                  Read filtering patterns from a file
      --force                                Bypass --max-delete safety check and run the sync. Consider using with --verbose
  -h, --help                                 help for bisync
      --ignore-listing-checksum              Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks)
      --max-lock Duration                    Consider lock files older than this to be expired (default: 0 (never expire)) (minimum: 2m) (default 0s)
      --no-cleanup                           Retain working files (useful for troubleshooting and testing).
      --no-slow-hash                         Ignore listing checksums only on backends where they are slow
      --recover                              Automatically recover from interruptions without requiring --resync.
      --remove-empty-dirs                    Remove ALL empty directories at the final cleanup step.
      --resilient                            Allow future runs to retry after certain less-serious errors, instead of requiring --resync. Use at your own risk!
  -1, --resync                               Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first.
      --resync-mode string                   During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.) (default "none")
      --retries int                          Retry operations this many times if they fail (requires --resilient). (default 3)
      --retries-sleep Duration               Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
      --slow-hash-sync-only                  Ignore slow checksums for listings and deltas, but still consider them during sync calls.
      --workdir string                       Use custom working dir - useful for testing. (default: {WORKDIR})
      --max-delete PERCENT                   Safety check on maximum percentage of deleted files allowed. If exceeded, the bisync run will abort. (default: 50%)
  -n, --dry-run                              Go through the motions - No files are copied/deleted.
  -v, --verbose                              Increases logging verbosity. May be specified more than once for more details.

Arbitrary rclone flags may be specified on the bisync command line, for example rclone bisync ./testdir/path1/ gdrive:testdir/path2/ --drive-skip-gdocs -v -v --timeout 10s Note that interactions of various rclone flags with bisync process flow has not been fully tested yet.

Paths

Path1 and Path2 arguments may be references to any mix of local directory paths (absolute or relative), UNC paths (//server/share/path), Windows drive paths (with a drive letter and :) or configured remotes with optional subdirectory paths. Cloud references are distinguished by having a : in the argument (see Windows support below).

Path1 and Path2 are treated equally, in that neither has priority for file changes (except during --resync), and access efficiency does not change whether a remote is on Path1 or Path2.

The listings in bisync working directory (default: ~/.cache/rclone/bisync) are named based on the Path1 and Path2 arguments so that separate syncs to individual directories within the tree may be set up, e.g.: path_to_local_tree..dropbox_subdir.lst.

Any empty directories after the sync on both the Path1 and Path2 filesystems are not deleted by default, unless --create-empty-src-dirs is specified. If the --remove-empty-dirs flag is specified, then both paths will have ALL empty directories purged as the last step in the process.

Command-line flags

--resync

This will effectively make both Path1 and Path2 filesystems contain a matching superset of all files. By default, Path2 files that do not exist in Path1 will be copied to Path1, and the process will then copy the Path1 tree to Path2.

The --resync sequence is roughly equivalent to the following (but see --resync-mode for other options):

rclone copy Path2 Path1 --ignore-existing [--create-empty-src-dirs]
rclone copy Path1 Path2 [--create-empty-src-dirs]

The base directories on both Path1 and Path2 filesystems must exist or bisync will fail. This is required for safety - that bisync can verify that both paths are valid.

When using --resync, a newer version of a file on the Path2 filesystem will (by default) be overwritten by the Path1 filesystem version. (Note that this is NOT entirely symmetrical, and more symmetrical options can be specified with the --resync-mode flag.) Carefully evaluate deltas using --dry-run.

For a resync run, one of the paths may be empty (no files in the path tree). The resync run should result in files on both paths, else a normal non-resync run will fail.

For a non-resync run, either path being empty (no files in the tree) fails with Empty current PathN listing. Cannot sync to an empty directory: X.pathN.lst This is a safety check that an unexpected empty path does not result in deleting everything in the other path.

Note that --resync implies --resync-mode path1 unless a different --resync-mode is explicitly specified. It is not necessary to use both the --resync and --resync-mode flags -- either one is sufficient without the other.

Note: --resync (including --resync-mode) should only be used under three specific (rare) circumstances:

It is your first bisync run (between these two paths)
You've just made changes to your bisync settings (such as editing the contents of your --filters-file)
There was an error on the prior run, and as a result, bisync now requires --resync to recover

The rest of the time, you should omit --resync. The reason is because --resync will only copy (not sync) each side to the other. Therefore, if you included --resync for every bisync run, it would never be possible to delete a file -- the deleted file would always keep reappearing at the end of every run (because it's being copied from the other side where it still exists). Similarly, renaming a file would always result in a duplicate copy (both old and new name) on both sides.

If you find that frequent interruptions from #3 are an issue, rather than automatically running --resync, the recommended alternative is to use the --resilient, --recover, and --conflict-resolve flags, (along with Graceful Shutdown mode, when needed) for a very robust "set-it-and-forget-it" bisync setup that can automatically bounce back from almost any interruption it might encounter. Consider adding something like the following:

--resilient --recover --max-lock 2m --conflict-resolve newer

--resync-mode CHOICE

In the event that a file differs on both sides during a --resync, --resync-mode controls which version will overwrite the other. The supported options are similar to --conflict-resolve. For all of the following options, the version that is kept is referred to as the "winner", and the version that is overwritten (deleted) is referred to as the "loser". The options are named after the "winner":

path1 - (the default) - the version from Path1 is unconditionally considered the winner (regardless of modtime and size, if any). This can be useful if one side is more trusted or up-to-date than the other, at the time of the --resync.
path2 - same as path1, except the path2 version is considered the winner.
newer - the newer file (by modtime) is considered the winner, regardless of which side it came from. This may result in having a mix of some winners from Path1, and some winners from Path2. (The implementation is analogous to running rclone copy --update in both directions.)
older - same as newer, except the older file is considered the winner, and the newer file is considered the loser.
larger - the larger file (by size) is considered the winner (regardless of modtime, if any). This can be a useful option for remotes without modtime support, or with the kinds of files (such as logs) that tend to grow but not shrink, over time.
smaller - the smaller file (by size) is considered the winner (regardless of modtime, if any).

For all of the above options, note the following:

If either of the underlying remotes lacks support for the chosen method, it will be ignored and will fall back to the default of path1. (For example, if --resync-mode newer is set, but one of the paths uses a remote that doesn't support modtime.)
If a winner can't be determined because the chosen method's attribute is missing or equal, it will be ignored, and bisync will instead try to determine whether the files differ by looking at the other --compare methods in effect. (For example, if --resync-mode newer is set, but the Path1 and Path2 modtimes are identical, bisync will compare the sizes.) If bisync concludes that they differ, preference is given to whichever is the "source" at that moment. (In practice, this gives a slight advantage to Path2, as the 2to1 copy comes before the 1to2 copy.) If the files do not differ, nothing is copied (as both sides are already correct).
These options apply only to files that exist on both sides (with the same name and relative path). Files that exist only on one side and not the other are always copied to the other, during --resync (this is one of the main differences between resync and non-resync runs.).
--conflict-resolve, --conflict-loser, and --conflict-suffix do not apply during --resync, and unlike these flags, nothing is renamed during --resync. When a file differs on both sides during --resync, one version always overwrites the other (much like in rclone copy.) (Consider using --backup-dir to retain a backup of the losing version.)
Unlike for --conflict-resolve, --resync-mode none is not a valid option (or rather, it will be interpreted as "no resync", unless --resync has also been specified, in which case it will be ignored.)
Winners and losers are decided at the individual file-level only (there is not currently an option to pick an entire winning directory atomically, although the path1 and path2 options typically produce a similar result.)
To maintain backward-compatibility, the --resync flag implies --resync-mode path1 unless a different --resync-mode is explicitly specified. Similarly, all --resync-mode options (except none) imply --resync, so it is not necessary to use both the --resync and --resync-mode flags simultaneously -- either one is sufficient without the other.

--check-access

Access check files are an additional safety measure against data loss. bisync will ensure it can find matching RCLONE_TEST files in the same places in the Path1 and Path2 filesystems. RCLONE_TEST files are not generated automatically. For --check-access to succeed, you must first either: A) Place one or more RCLONE_TEST files in both systems, or B) Set --check-filename to a filename already in use in various locations throughout your sync'd fileset. Recommended methods for A) include:

rclone touch Path1/RCLONE_TEST (create a new file)
rclone copyto Path1/RCLONE_TEST Path2/RCLONE_TEST (copy an existing file)
rclone copy Path1/RCLONE_TEST Path2/RCLONE_TEST --include "RCLONE_TEST" (copy multiple files at once, recursively)
create the files manually (outside of rclone)
run bisync once without --check-access to set matching files on both filesystems will also work, but is not preferred, due to potential for user error (you are temporarily disabling the safety feature).

Note that --check-access is still enforced on --resync, so bisync --resync --check-access will not work as a method of initially setting the files (this is to ensure that bisync can't inadvertently circumvent its own safety switch.)

Time stamps and file contents for RCLONE_TEST files are not important, just the names and locations. If you have symbolic links in your sync tree it is recommended to place RCLONE_TEST files in the linked-to directory tree to protect against bisync assuming a bunch of deleted files if the linked-to tree should not be accessible. See also the --check-filename flag.

--check-filename

Name of the file(s) used in access health validation. The default --check-filename is RCLONE_TEST. One or more files having this filename must exist, synchronized between your source and destination filesets, in order for --check-access to succeed. See --check-access for additional details.

--compare

As of v1.66, bisync fully supports comparing based on any combination of size, modtime, and checksum (lifting the prior restriction on backends without modtime support.)

By default (without the --compare flag), bisync inherits the same comparison options as sync (that is: size and modtime by default, unless modified with flags such as --checksum or --size-only.)

If the --compare flag is set, it will override these defaults. This can be useful if you wish to compare based on combinations not currently supported in sync, such as comparing all three of size AND modtime AND checksum simultaneously (or just modtime AND checksum).

--compare takes a comma-separated list, with the currently supported values being size, modtime, and checksum. For example, if you want to compare size and checksum, but not modtime, you would do:

--compare size,checksum

Or if you want to compare all three:

--compare size,modtime,checksum

--compare overrides any conflicting flags. For example, if you set the conflicting flags --compare checksum --size-only, --size-only will be ignored, and bisync will compare checksum and not size. To avoid confusion, it is recommended to use either --compare or the normal sync flags, but not both.

If --compare includes checksum and both remotes support checksums but have no hash types in common with each other, checksums will be considered only for comparisons within the same side (to determine what has changed since the prior sync), but not for comparisons against the opposite side. If one side supports checksums and the other does not, checksums will only be considered on the side that supports them.

When comparing with checksum and/or size without modtime, bisync cannot determine whether a file is newer or older -- only whether it is changed or unchanged. (If it is changed on both sides, bisync still does the standard equality-check to avoid declaring a sync conflict unless it absolutely has to.)

It is recommended to do a --resync when changing --compare settings, as otherwise your prior listing files may not contain the attributes you wish to compare (for example, they will not have stored checksums if you were not previously comparing checksums.)

--ignore-listing-checksum

When --checksum or --compare checksum is set, bisync will retrieve (or generate) checksums (for backends that support them) when creating the listings for both paths, and store the checksums in the listing files. --ignore-listing-checksum will disable this behavior, which may speed things up considerably, especially on backends (such as local) where hashes must be computed on the fly instead of retrieved. Please note the following:

As of v1.66, --ignore-listing-checksum is now automatically set when neither --checksum nor --compare checksum are in use (as the checksums would not be used for anything.)
--ignore-listing-checksum is NOT the same as --ignore-checksum, and you may wish to use one or the other, or both. In a nutshell: --ignore-listing-checksum controls whether checksums are considered when scanning for diffs, while --ignore-checksum controls whether checksums are considered during the copy/sync operations that follow, if there ARE diffs.
Unless --ignore-listing-checksum is passed, bisync currently computes hashes for one path even when there's no common hash with the other path (for example, a crypt remote.) This can still be beneficial, as the hashes will still be used to detect changes within the same side (if --checksum or --compare checksum is set), even if they can't be used to compare against the opposite side.
If you wish to ignore listing checksums only on remotes where they are slow to compute, consider using --no-slow-hash (or --slow-hash-sync-only) instead of --ignore-listing-checksum.
If --ignore-listing-checksum is used simultaneously with --compare checksum (or --checksum), checksums will be ignored for bisync deltas, but still considered during the sync operations that follow (if deltas are detected based on modtime and/or size.)

--no-slow-hash

On some remotes (notably local), checksums can dramatically slow down a bisync run, because hashes cannot be stored and need to be computed in real-time when they are requested. On other remotes (such as drive), they add practically no time at all. The --no-slow-hash flag will automatically skip checksums on remotes where they are slow, while still comparing them on others (assuming --compare includes checksum.) This can be useful when one of your bisync paths is slow but you still want to check checksums on the other, for a more robust sync.

--slow-hash-sync-only

Same as --no-slow-hash, except slow hashes are still considered during sync calls. They are still NOT considered for determining deltas, nor or they included in listings. They are also skipped during --resync. The main use case for this flag is when you have a large number of files, but relatively few of them change from run to run -- so you don't want to check your entire tree every time (it would take too long), but you still want to consider checksums for the smaller group of files for which a modtime or size change was detected. Keep in mind that this speed savings comes with a safety trade-off: if a file's content were to change without a change to its modtime or size, bisync would not detect it, and it would not be synced.

--slow-hash-sync-only is only useful if both remotes share a common hash type (if they don't, bisync will automatically fall back to --no-slow-hash.) Both --no-slow-hash and --slow-hash-sync-only have no effect without --compare checksum (or --checksum).

--download-hash

If --download-hash is set, bisync will use best efforts to obtain an MD5 checksum by downloading and computing on-the-fly, when checksums are not otherwise available (for example, a remote that doesn't support them.) Note that since rclone has to download the entire file, this may dramatically slow down your bisync runs, and is also likely to use a lot of data, so it is probably not practical for bisync paths with a large total file size. However, it can be a good option for syncing small-but-important files with maximum accuracy (for example, a source code repo on a crypt remote.) An additional advantage over methods like cryptcheck is that the original file is not required for comparison (for example, --download-hash can be used to bisync two different crypt remotes with different passwords.)

When --download-hash is set, bisync still looks for more efficient checksums first, and falls back to downloading only when none are found. It takes priority over conflicting flags such as --no-slow-hash. --download-hash is not suitable for Google Docs and other files of unknown size, as their checksums would change from run to run (due to small variances in the internals of the generated export file.) Therefore, bisync automatically skips --download-hash for files with a size less than 0.

See also: Hasher backend, cryptcheck command, rclone check --download option, md5sum command

--max-delete

As a safety check, if greater than the --max-delete percent of files were deleted on either the Path1 or Path2 filesystem, then bisync will abort with a warning message, without making any changes. The default --max-delete is 50%. One way to trigger this limit is to rename a directory that contains more than half of your files. This will appear to bisync as a bunch of deleted files and a bunch of new files. This safety check is intended to block bisync from deleting all of the files on both filesystems due to a temporary network access issue, or if the user had inadvertently deleted the files on one side or the other. To force the sync, either set a different delete percentage limit, e.g. --max-delete 75 (allows up to 75% deletion), or use --force to bypass the check.

Also see the all files changed check.

--filters-file

By using rclone filter features you can exclude file types or directory sub-trees from the sync. See the bisync filters section and generic --filter-from documentation. An example filters file contains filters for non-allowed files for synching with Dropbox.

If you make changes to your filters file then bisync requires a run with --resync. This is a safety feature, which prevents existing files on the Path1 and/or Path2 side from seeming to disappear from view (since they are excluded in the new listings), which would fool bisync into seeing them as deleted (as compared to the prior run listings), and then bisync would proceed to delete them for real.

To block this from happening, bisync calculates an MD5 hash of the filters file and stores the hash in a .md5 file in the same place as your filters file. On the next run with --filters-file set, bisync re-calculates the MD5 hash of the current filters file and compares it to the hash stored in the .md5 file. If they don't match, the run aborts with a critical error and thus forces you to do a --resync, likely avoiding a disaster.

--conflict-resolve CHOICE

In bisync, a "conflict" is a file that is new or changed on both sides (relative to the prior run) AND is not currently identical on both sides. --conflict-resolve controls how bisync handles such a scenario. The currently supported options are:

none - (the default) - do not attempt to pick a winner, keep and rename both files according to --conflict-loser and --conflict-suffix settings. For example, with the default settings, file.txt on Path1 is renamed file.txt.conflict1 and file.txt on Path2 is renamed file.txt.conflict2. Both are copied to the opposite path during the run, so both sides end up with a copy of both files. (As none is the default, it is not necessary to specify --conflict-resolve none -- you can just omit the flag.)
newer - the newer file (by modtime) is considered the winner and is copied without renaming. The older file (the "loser") is handled according to --conflict-loser and --conflict-suffix settings (either renamed or deleted.) For example, if file.txt on Path1 is newer than file.txt on Path2, the result on both sides (with other default settings) will be file.txt (winner from Path1) and file.txt.conflict1 (loser from Path2).
older - same as newer, except the older file is considered the winner, and the newer file is considered the loser.
larger - the larger file (by size) is considered the winner (regardless of modtime, if any).
smaller - the smaller file (by size) is considered the winner (regardless of modtime, if any).
path1 - the version from Path1 is unconditionally considered the winner (regardless of modtime and size, if any). This can be useful if one side is usually more trusted or up-to-date than the other.
path2 - same as path1, except the path2 version is considered the winner.

For all of the above options, note the following:

If either of the underlying remotes lacks support for the chosen method, it will be ignored and fall back to none. (For example, if --conflict-resolve newer is set, but one of the paths uses a remote that doesn't support modtime.)
If a winner can't be determined because the chosen method's attribute is missing or equal, it will be ignored and fall back to none. (For example, if --conflict-resolve newer is set, but the Path1 and Path2 modtimes are identical, even if the sizes may differ.)
If the file's content is currently identical on both sides, it is not considered a "conflict", even if new or changed on both sides since the prior sync. (For example, if you made a change on one side and then synced it to the other side by other means.) Therefore, none of the conflict resolution flags apply in this scenario.
The conflict resolution flags do not apply during a --resync, as there is no "prior run" to speak of (but see --resync-mode for similar options.)

--conflict-loser CHOICE

--conflict-loser determines what happens to the "loser" of a sync conflict (when --conflict-resolve determines a winner) or to both files (when there is no winner.) The currently supported options are:

num - (the default) - auto-number the conflicts by automatically appending the next available number to the --conflict-suffix, in chronological order. For example, with the default settings, the first conflict for file.txt will be renamed file.txt.conflict1. If file.txt.conflict1 already exists, file.txt.conflict2 will be used instead (etc., up to a maximum of 9223372036854775807 conflicts.)
pathname - rename the conflicts according to which side they came from, which was the default behavior prior to v1.66. For example, with --conflict-suffix path, file.txt from Path1 will be renamed file.txt.path1, and file.txt from Path2 will be renamed file.txt.path2. If two non-identical suffixes are provided (ex. --conflict-suffix cloud,local), the trailing digit is omitted. Importantly, note that with pathname, there is no auto-numbering beyond 2, so if file.txt.path2 somehow already exists, it will be overwritten. Using a dynamic date variable in your --conflict-suffix (see below) is one possible way to avoid this. Note also that conflicts-of-conflicts are possible, if the original conflict is not manually resolved -- for example, if for some reason you edited file.txt.path1 on both sides, and those edits were different, the result would be file.txt.path1.path1 and file.txt.path1.path2 (in addition to file.txt.path2.)
delete - keep the winner only and delete the loser, instead of renaming it. If a winner cannot be determined (see --conflict-resolve for details on how this could happen), delete is ignored and the default num is used instead (i.e. both versions are kept and renamed, and neither is deleted.) delete is inherently the most destructive option, so use it only with care.

For all of the above options, note that if a winner cannot be determined (see --conflict-resolve for details on how this could happen), or if --conflict-resolve is not in use, both files will be renamed.

--conflict-suffix STRING[,STRING]

--conflict-suffix controls the suffix that is appended when bisync renames a --conflict-loser (default: conflict). --conflict-suffix will accept either one string or two comma-separated strings to assign different suffixes to Path1 vs. Path2. This may be helpful later in identifying the source of the conflict. (For example, --conflict-suffix dropboxconflict,laptopconflict)

With --conflict-loser num, a number is always appended to the suffix. With --conflict-loser pathname, a number is appended only when one suffix is specified (or when two identical suffixes are specified.) i.e. with --conflict-loser pathname, all of the following would produce exactly the same result:

--conflict-suffix path
--conflict-suffix path,path
--conflict-suffix path1,path2

Suffixes may be as short as 1 character. By default, the suffix is appended after any other extensions (ex. file.jpg.conflict1), however, this can be changed with the --suffix-keep-extension flag (i.e. to instead result in file.conflict1.jpg).

--conflict-suffix supports several dynamic date variables when enclosed in curly braces as globs. This can be helpful to track the date and/or time that each conflict was handled by bisync. For example:

--conflict-suffix {DateOnly}-conflict
// result: myfile.txt.2006-01-02-conflict1

All of the formats described here and here are supported, but take care to ensure that your chosen format does not use any characters that are illegal on your remotes (for example, macOS does not allow colons in filenames, and slashes are also best avoided as they are often interpreted as directory separators.) To address this particular issue, an additional {MacFriendlyTime} (or just {mac}) option is supported, which results in 2006-01-02 0304PM.

Note that --conflict-suffix is entirely separate from rclone's main --sufix flag. This is intentional, as users may wish to use both flags simultaneously, if also using --backup-dir.

Finally, note that the default in bisync prior to v1.66 was to rename conflicts with ..path1 and ..path2 (with two periods, and path instead of conflict.) Bisync now defaults to a single dot instead of a double dot, but additional dots can be added by including them in the specified suffix string. For example, for behavior equivalent to the previous default, use:

[--conflict-resolve none] --conflict-loser pathname --conflict-suffix .path

--check-sync

Enabled by default, the check-sync function checks that all of the same files exist in both the Path1 and Path2 history listings. This check-sync integrity check is performed at the end of the sync run by default. Any untrapped failing copy/deletes between the two paths might result in differences between the two listings and in the untracked file content differences between the two paths. A resync run would correct the error.

Note that the default-enabled integrity check locally executes a load of both the final Path1 and Path2 listings, and thus adds to the run time of a sync. Using --check-sync=false will disable it and may significantly reduce the sync run times for very large numbers of files.

The check may be run manually with --check-sync=only. It runs only the integrity check and terminates without actually synching.

Note that currently, --check-sync only checks listing snapshots and NOT the actual files on the remotes. Note also that the listing snapshots will not know about any changes that happened during or after the latest bisync run, as those will be discovered on the next run. Therefore, while listings should always match each other at the end of a bisync run, it is expected that they will not match the underlying remotes, nor will the remotes match each other, if there were changes during or after the run. This is normal, and any differences will be detected and synced on the next run.

For a robust integrity check of the current state of the remotes (as opposed to just their listing snapshots), consider using check (or cryptcheck, if at least one path is a crypt remote) instead of --check-sync, keeping in mind that differences are expected if files changed during or after your last bisync run.

For example, a possible sequence could look like this:

Normally scheduled bisync run:

rclone bisync Path1 Path2 -MPc --check-access --max-delete 10 --filters-file /path/to/filters.txt -v --no-cleanup --ignore-listing-checksum --disable ListR --checkers=16 --drive-pacer-min-sleep=10ms --create-empty-src-dirs --resilient

Periodic independent integrity check (perhaps scheduled nightly or weekly):

rclone check -MvPc Path1 Path2 --filter-from /path/to/filters.txt

If diffs are found, you have some choices to correct them. If one side is more up-to-date and you want to make the other side match it, you could run:

rclone sync Path1 Path2 --filter-from /path/to/filters.txt --create-empty-src-dirs -MPc -v

(or switch Path1 and Path2 to make Path2 the source-of-truth)

Or, if neither side is totally up-to-date, you could run a --resync to bring them back into agreement (but remember that this could cause deleted files to re-appear.)

*Note also that rclone check does not currently include empty directories, so if you want to know if any empty directories are out of sync, consider alternatively running the above rclone sync command with --dry-run added.

--resilient

Caution: this is an experimental feature. Use at your own risk!

By default, most errors or interruptions will cause bisync to abort and require --resync to recover. This is a safety feature, to prevent bisync from running again until a user checks things out. However, in some cases, bisync can go too far and enforce a lockout when one isn't actually necessary, like for certain less-serious errors that might resolve themselves on the next run. When --resilient is specified, bisync tries its best to recover and self-correct, and only requires --resync as a last resort when a human's involvement is absolutely necessary. The intended use case is for running bisync as a background process (such as via scheduled cron).

When using --resilient mode, bisync will still report the error and abort, however it will not lock out future runs -- allowing the possibility of retrying at the next normally scheduled time, without requiring a --resync first. Examples of such retryable errors include access test failures, missing listing files, and filter change detections. These safety features will still prevent the current run from proceeding -- the difference is that if conditions have improved by the time of the next run, that next run will be allowed to proceed. Certain more serious errors will still enforce a --resync lockout, even in --resilient mode, to prevent data loss.

Behavior of --resilient may change in a future version. (See also: --recover, --max-lock, Graceful Shutdown)

--recover

If --recover is set, in the event of a sudden interruption or other un-graceful shutdown, bisync will attempt to automatically recover on the next run, instead of requiring --resync. Bisync is able to recover robustly by keeping one "backup" listing at all times, representing the state of both paths after the last known successful sync. Bisync can then compare the current state with this snapshot to determine which changes it needs to retry. Changes that were synced after this snapshot (during the run that was later interrupted) will appear to bisync as if they are "new or changed on both sides", but in most cases this is not a problem, as bisync will simply do its usual "equality check" and learn that no action needs to be taken on these files, since they are already identical on both sides.

In the rare event that a file is synced successfully during a run that later aborts, and then that same file changes AGAIN before the next run, bisync will think it is a sync conflict, and handle it accordingly. (From bisync's perspective, the file has changed on both sides since the last trusted sync, and the files on either side are not currently identical.) Therefore, --recover carries with it a slightly increased chance of having conflicts -- though in practice this is pretty rare, as the conditions required to cause it are quite specific. This risk can be reduced by using bisync's "Graceful Shutdown" mode (triggered by sending SIGINT or Ctrl+C), when you have the choice, instead of forcing a sudden termination.

--recover and --resilient are similar, but distinct -- the main difference is that --resilient is about retrying, while --recover is about recovering. Most users will probably want both. --resilient allows retrying when bisync has chosen to abort itself due to safety features such as failing --check-access or detecting a filter change. --resilient does not cover external interruptions such as a user shutting down their computer in the middle of a sync -- that is what --recover is for.

--max-lock

Bisync uses lock files as a safety feature to prevent interference from other bisync runs while it is running. Bisync normally removes these lock files at the end of a run, but if bisync is abruptly interrupted, these files will be left behind. By default, they will lock out all future runs, until the user has a chance to manually check things out and remove the lock. As an alternative, --max-lock can be used to make them automatically expire after a certain period of time, so that future runs are not locked out forever, and auto-recovery is possible. --max-lock can be any duration 2m or greater (or 0 to disable). If set, lock files older than this will be considered "expired", and future runs will be allowed to disregard them and proceed. (Note that the --max-lock duration must be set by the process that left the lock file -- not the later one interpreting it.)

If set, bisync will also "renew" these lock files every --max-lock minus one minute throughout a run, for extra safety. (For example, with --max-lock 5m, bisync would renew the lock file (for another 5 minutes) every 4 minutes until the run has completed.) In other words, it should not be possible for a lock file to pass its expiration time while the process that created it is still running -- and you can therefore be reasonably sure that any expired lock file you may find was left there by an interrupted run, not one that is still running and just taking awhile.

If --max-lock is 0 or not set, the default is that lock files will never expire, and will block future runs (of these same two bisync paths) indefinitely.

For maximum resilience from disruptions, consider setting a relatively short duration like --max-lock 2m along with --resilient and --recover, and a relatively frequent cron schedule. The result will be a very robust "set-it-and-forget-it" bisync run that can automatically bounce back from almost any interruption it might encounter, without requiring the user to get involved and run a --resync. (See also: Graceful Shutdown mode)

--backup-dir1 and --backup-dir2

As of v1.66, --backup-dir is supported in bisync. Because --backup-dir must be a non-overlapping path on the same remote, Bisync has introduced new --backup-dir1 and --backup-dir2 flags to support separate backup-dirs for Path1 and Path2 (bisyncing between different remotes with --backup-dir would not otherwise be possible.) --backup-dir1 and --backup-dir2 can use different remotes from each other, but --backup-dir1 must use the same remote as Path1, and --backup-dir2 must use the same remote as Path2. Each backup directory must not overlap its respective bisync Path without being excluded by a filter rule.

The standard --backup-dir will also work, if both paths use the same remote (but note that deleted files from both paths would be mixed together in the same dir). If either --backup-dir1 and --backup-dir2 are set, they will override --backup-dir.

Example:

rclone bisync /Users/someuser/some/local/path/Bisync gdrive:Bisync --backup-dir1 /Users/someuser/some/local/path/BackupDir --backup-dir2 gdrive:BackupDir --suffix -2023-08-26 --suffix-keep-extension --check-access --max-delete 10 --filters-file /Users/someuser/some/local/path/bisync_filters.txt --no-cleanup --ignore-listing-checksum --checkers=16 --drive-pacer-min-sleep=10ms --create-empty-src-dirs --resilient -MvP --drive-skip-gdocs --fix-case

In this example, if the user deletes a file in /Users/someuser/some/local/path/Bisync, bisync will propagate the delete to the other side by moving the corresponding file from gdrive:Bisync to gdrive:BackupDir. If the user deletes a file from gdrive:Bisync, bisync moves it from /Users/someuser/some/local/path/Bisync to /Users/someuser/some/local/path/BackupDir.

In the event of a rename due to a sync conflict, the rename is not considered a delete, unless a previous conflict with the same name already exists and would get overwritten.

Operation

Runtime flow details

bisync retains the listings of the Path1 and Path2 filesystems from the prior run. On each successive run it will:

list files on path1 and path2, and check for changes on each side. Changes include New, Newer, Older, and Deleted files.
Propagate changes on path1 to path2, and vice-versa.

Safety measures

Lock file prevents multiple simultaneous runs when taking a while. This can be particularly useful if bisync is run by cron scheduler.
Handle change conflicts non-destructively by creating .conflict1, .conflict2, etc. file versions, according to --conflict-resolve, --conflict-loser, and --conflict-suffix settings.
File system access health check using RCLONE_TEST files (see the --check-access flag).
Abort on excessive deletes - protects against a failed listing being interpreted as all the files were deleted. See the --max-delete and --force flags.
If something evil happens, bisync goes into a safe state to block damage by later runs. (See Error Handling)

Normal sync checks

Type	Description	Result	Implementation
Path2 new	File is new on Path2, does not exist on Path1	Path2 version survives	`rclone copy` Path2 to Path1
Path2 newer	File is newer on Path2, unchanged on Path1	Path2 version survives	`rclone copy` Path2 to Path1
Path2 deleted	File is deleted on Path2, unchanged on Path1	File is deleted	`rclone delete` Path1
Path1 new	File is new on Path1, does not exist on Path2	Path1 version survives	`rclone copy` Path1 to Path2
Path1 newer	File is newer on Path1, unchanged on Path2	Path1 version survives	`rclone copy` Path1 to Path2
Path1 older	File is older on Path1, unchanged on Path2	Path1 version survives	`rclone copy` Path1 to Path2
Path2 older	File is older on Path2, unchanged on Path1	Path2 version survives	`rclone copy` Path2 to Path1
Path1 deleted	File no longer exists on Path1	File is deleted	`rclone delete` Path2

Unusual sync checks

Type	Description	Result	Implementation
Path1 new/changed AND Path2 new/changed AND Path1 == Path2	File is new/changed on Path1 AND new/changed on Path2 AND Path1 version is currently identical to Path2	No change	None
Path1 new AND Path2 new	File is new on Path1 AND new on Path2 (and Path1 version is NOT identical to Path2)	Conflicts handled according to `--conflict-resolve` & `--conflict-loser` settings	default: `rclone copy` renamed `Path2.conflict2` file to Path1, `rclone copy` renamed `Path1.conflict1` file to Path2
Path2 newer AND Path1 changed	File is newer on Path2 AND also changed (newer/older/size) on Path1 (and Path1 version is NOT identical to Path2)	Conflicts handled according to `--conflict-resolve` & `--conflict-loser` settings	default: `rclone copy` renamed `Path2.conflict2` file to Path1, `rclone copy` renamed `Path1.conflict1` file to Path2
Path2 newer AND Path1 deleted	File is newer on Path2 AND also deleted on Path1	Path2 version survives	`rclone copy` Path2 to Path1
Path2 deleted AND Path1 changed	File is deleted on Path2 AND changed (newer/older/size) on Path1	Path1 version survives	`rclone copy` Path1 to Path2
Path1 deleted AND Path2 changed	File is deleted on Path1 AND changed (newer/older/size) on Path2	Path2 version survives	`rclone copy` Path2 to Path1

As of rclone v1.64, bisync is now better at detecting false positive sync conflicts, which would previously have resulted in unnecessary renames and duplicates. Now, when bisync comes to a file that it wants to rename (because it is new/changed on both sides), it first checks whether the Path1 and Path2 versions are currently identical (using the same underlying function as check.) If bisync concludes that the files are identical, it will skip them and move on. Otherwise, it will create renamed duplicates, as before. This behavior also improves the experience of renaming directories, as a --resync is no longer required, so long as the same change has been made on both sides.

All files changed check

If all prior existing files on either of the filesystems have changed (e.g. timestamps have changed due to changing the system's timezone) then bisync will abort without making any changes. Any new files are not considered for this check. You could use --force to force the sync (whichever side has the changed timestamp files wins). Alternately, a --resync may be used (Path1 versions will be pushed to Path2). Consider the situation carefully and perhaps use --dry-run before you commit to the changes.

Modification times

By default, bisync compares files by modification time and size. If you or your application should change the content of a file without changing the modification time and size, then bisync will not notice the change, and thus will not copy it to the other side. As an alternative, consider comparing by checksum (if your remotes support it). See --compare for details.

Error handling

Certain bisync critical errors, such as file copy/move failing, will result in a bisync lockout of following runs. The lockout is asserted because the sync status and history of the Path1 and Path2 filesystems cannot be trusted, so it is safer to block any further changes until someone checks things out. The recovery is to do a --resync again.

It is recommended to use --resync --dry-run --verbose initially and carefully review what changes will be made before running the --resync without --dry-run.

Most of these events come up due to an error status from an internal call. On such a critical error the {...}.path1.lst and {...}.path2.lst listing files are renamed to extension .lst-err, which blocks any future bisync runs (since the normal .lst files are not found). Bisync keeps them under bisync subdirectory of the rclone cache directory, typically at ${HOME}/.cache/rclone/bisync/ on Linux.

Some errors are considered temporary and re-running the bisync is not blocked. The critical return blocks further bisync runs.

Lock file

When bisync is running, a lock file is created in the bisync working directory, typically at ~/.cache/rclone/bisync/PATH1..PATH2.lck on Linux. If bisync should crash or hang, the lock file will remain in place and block any further runs of bisync for the same paths. Delete the lock file as part of debugging the situation. The lock file effectively blocks follow-on (e.g., scheduled by cron) runs when the prior invocation is taking a long time. The lock file contains PID of the blocking process, which may help in debug. Lock files can be set to automatically expire after a certain amount of time, using the --max-lock flag.

Note that while concurrent bisync runs are allowed, be very cautious that there is no overlap in the trees being synched between concurrent runs, lest there be replicated files, deleted files and general mayhem.

Return codes

rclone bisync returns the following codes to calling program:

0 on a successful run,
1 for a non-critical failing run (a rerun may be successful),
2 for a critically aborted run (requires a --resync to recover).

Graceful Shutdown

Bisync has a "Graceful Shutdown" mode which is activated by sending SIGINT or pressing Ctrl+C during a run. Once triggered, bisync will use best efforts to exit cleanly before the timer runs out. If bisync is in the middle of transferring files, it will attempt to cleanly empty its queue by finishing what it has started but not taking more. If it cannot do so within 30 seconds, it will cancel the in-progress transfers at that point and then give itself a maximum of 60 seconds to wrap up, save its state for next time, and exit. With the -vP flags you will see constant status updates and a final confirmation of whether or not the graceful shutdown was successful.

At any point during the "Graceful Shutdown" sequence, a second SIGINT or Ctrl+C will trigger an immediate, un-graceful exit, which will leave things in a messier state. Usually a robust recovery will still be possible if using --recover mode, otherwise you will need to do a --resync.

If you plan to use Graceful Shutdown mode, it is recommended to use --resilient and --recover, and it is important to NOT use --inplace, otherwise you risk leaving partially-written files on one side, which may be confused for real files on the next run. Note also that in the event of an abrupt interruption, a lock file will be left behind to block concurrent runs. You will need to delete it before you can proceed with the next run (or wait for it to expire on its own, if using --max-lock.)

Limitations

Supported backends

Bisync is considered BETA and has been tested with the following backends:

Local filesystem
Google Drive
Dropbox
OneDrive
S3
SFTP
Yandex Disk
Crypt

It has not been fully tested with other services yet. If it works, or sorta works, please let us know and we'll update the list. Run the test suite to check for proper operation as described below.

The first release of rclone bisync required both underlying backends to support modification times, and refused to run otherwise. This limitation has been lifted as of v1.66, as bisync now supports comparing checksum and/or size instead of (or in addition to) modtime. See --compare for details.

Concurrent modifications

When using Local, FTP or SFTP remotes with --inplace, rclone does not create temporary files at the destination when copying, and thus if the connection is lost the created file may be corrupt, which will likely propagate back to the original path on the next sync, resulting in data loss. It is therefore recommended to omit --inplace.

Files that change during a bisync run may result in data loss. Prior to rclone v1.66, this was commonly seen in highly dynamic environments, where the filesystem was getting hammered by running processes during the sync. As of rclone v1.66, bisync was redesigned to use a "snapshot" model, greatly reducing the risks from changes during a sync. Changes that are not detected during the current sync will now be detected during the following sync, and will no longer cause the entire run to throw a critical error. There is additionally a mechanism to mark files as needing to be internally rechecked next time, for added safety. It should therefore no longer be necessary to sync only at quiet times -- however, note that an error can still occur if a file happens to change at the exact moment it's being read/written by bisync (same as would happen in rclone sync.) (See also: --ignore-checksum, --local-no-check-updated)

Empty directories

By default, new/deleted empty directories on one path are not propagated to the other side. This is because bisync (and rclone) natively works on files, not directories. However, this can be changed with the --create-empty-src-dirs flag, which works in much the same way as in sync and copy. When used, empty directories created or deleted on one side will also be created or deleted on the other side. The following should be noted:

--create-empty-src-dirs is not compatible with --remove-empty-dirs. Use only one or the other (or neither).
It is not recommended to switch back and forth between --create-empty-src-dirs and the default (no --create-empty-src-dirs) without running --resync. This is because it may appear as though all directories (not just the empty ones) were created/deleted, when actually you've just toggled between making them visible/invisible to bisync. It looks scarier than it is, but it's still probably best to stick to one or the other, and use --resync when you need to switch.

Renamed directories

By default, renaming a folder on the Path1 side results in deleting all files on the Path2 side and then copying all files again from Path1 to Path2. Bisync sees this as all files in the old directory name as deleted and all files in the new directory name as new.

A recommended solution is to use --track-renames, which is now supported in bisync as of rclone v1.66. Note that --track-renames is not available during --resync, as --resync does not delete anything (--track-renames only supports sync, not copy.)

Otherwise, the most effective and efficient method of renaming a directory is to rename it to the same name on both sides. (As of rclone v1.64, a --resync is no longer required after doing so, as bisync will automatically detect that Path1 and Path2 are in agreement.)

`--fast-list` used by default

Unlike most other rclone commands, bisync uses --fast-list by default, for backends that support it. In many cases this is desirable, however, there are some scenarios in which bisync could be faster without --fast-list, and there is also a known issue concerning Google Drive users with many empty directories. For now, the recommended way to avoid using --fast-list is to add --disable ListR to all bisync commands. The default behavior may change in a future version.

Case (and unicode) sensitivity

As of v1.66, case and unicode form differences no longer cause critical errors, and normalization (when comparing between filesystems) is handled according to the same flags and defaults as rclone sync. See the following options (all of which are supported by bisync) to control this behavior more granularly:

--fix-case
--ignore-case-sync
--no-unicode-normalization
--local-unicode-normalization and --local-case-sensitive (caution: these are normally not what you want.)

Note that in the (probably rare) event that --fix-case is used AND a file is new/changed on both sides AND the checksums match AND the filename case does not match, the Path1 filename is considered the winner, for the purposes of --fix-case (Path2 will be renamed to match it).

Windows support

Bisync has been tested on Windows 8.1, Windows 10 Pro 64-bit and on Windows GitHub runners.

Drive letters are allowed, including drive letters mapped to network drives (rclone bisync J:\localsync GDrive:). If a drive letter is omitted, the shell current drive is the default. Drive letters are a single character follows by :, so cloud names must be more than one character long.

Absolute paths (with or without a drive letter), and relative paths (with or without a drive letter) are supported.

Working directory is created at C:\Users\MyLogin\AppData\Local\rclone\bisync.

Note that bisync output may show a mix of forward / and back \ slashes.

Be careful of case independent directory and file naming on Windows vs. case dependent Linux

Filtering

See filtering documentation for how filter rules are written and interpreted.

Bisync's --filters-file flag slightly extends the rclone's --filter-from filtering mechanism. For a given bisync run you may provide only one --filters-file. The --include*, --exclude*, and --filter flags are also supported.

How to filter directories

Filtering portions of the directory tree is a critical feature for synching.

Examples of directory trees (always beneath the Path1/Path2 root level) you may want to exclude from your sync:

Directory trees containing only software build intermediate files.
Directory trees containing application temporary files and data such as the Windows C:\Users\MyLogin\AppData\ tree.
Directory trees containing files that are large, less important, or are getting thrashed continuously by ongoing processes.

On the other hand, there may be only select directories that you actually want to sync, and exclude all others. See the Example include-style filters for Windows user directories below.

Filters file writing guidelines

Begin with excluding directory trees:
- e.g. - /AppData/
- ** on the end is not necessary. Once a given directory level is excluded then everything beneath it won't be looked at by rclone.
- Exclude such directories that are unneeded, are big, dynamically thrashed, or where there may be access permission issues.
- Excluding such dirs first will make rclone operations (much) faster.
- Specific files may also be excluded, as with the Dropbox exclusions example below.
Decide if it's easier (or cleaner) to:
- Include select directories and therefore exclude everything else -- or --
- Exclude select directories and therefore include everything else
Include select directories:
- Add lines like: + /Documents/PersonalFiles/** to select which directories to include in the sync.
- ** on the end specifies to include the full depth of the specified tree.
- With Include-style filters, files at the Path1/Path2 root are not included. They may be included with + /*.
- Place RCLONE_TEST files within these included directory trees. They will only be looked for in these directory trees.
- Finish by excluding everything else by adding - ** at the end of the filters file.
- Disregard step 4.
Exclude select directories:
- Add more lines like in step 1. For example: -/Desktop/tempfiles/, or - /testdir/. Again, a ** on the end is not necessary.
- Do not add a - ** in the file. Without this line, everything will be included that has not been explicitly excluded.
- Disregard step 3.

A few rules for the syntax of a filter file expanding on filtering documentation:

Lines may start with spaces and tabs - rclone strips leading whitespace.
If the first non-whitespace character is a # then the line is a comment and will be ignored.
Blank lines are ignored.
The first non-whitespace character on a filter line must be a + or -.
Exactly 1 space is allowed between the +/- and the path term.
Only forward slashes (/) are used in path terms, even on Windows.
The rest of the line is taken as the path term. Trailing whitespace is taken literally, and probably is an error.

Example include-style filters for Windows user directories

This Windows include-style example is based on the sync root (Path1) set to C:\Users\MyLogin. The strategy is to select specific directories to be synched with a network drive (Path2).

- /AppData/ excludes an entire tree of Windows stored stuff that need not be synched. In my case, AppData has >11 GB of stuff I don't care about, and there are some subdirectories beneath AppData that are not accessible to my user login, resulting in bisync critical aborts.
Windows creates cache files starting with both upper and lowercase NTUSER at C:\Users\MyLogin. These files may be dynamic, locked, and are generally don't care.
There are just a few directories with my data that I do want synched, in the form of + /<path>. By selecting only the directory trees I want to avoid the dozen plus directories that various apps make at C:\Users\MyLogin\Documents.
Include files in the root of the sync point, C:\Users\MyLogin, by adding the + /* line.
This is an Include-style filters file, therefore it ends with - ** which excludes everything not explicitly included.

- /AppData/
- NTUSER*
- ntuser*
+ /Documents/Family/**
+ /Documents/Sketchup/**
+ /Documents/Microcapture_Photo/**
+ /Documents/Microcapture_Video/**
+ /Desktop/**
+ /Pictures/**
+ /*
- **

Note also that Windows implements several "library" links such as C:\Users\MyLogin\My Documents\My Music pointing to C:\Users\MyLogin\Music. rclone sees these as links, so you must add --links to the bisync command line if you which to follow these links. I find that I get permission errors in trying to follow the links, so I don't include the rclone --links flag, but then you get lots of Can't follow symlink… noise from rclone about not following the links. This noise can be quashed by adding --quiet to the bisync command line.

Example exclude-style filters files for use with Dropbox

Dropbox disallows synching the listed temporary and configuration/data files. The - <filename> filters exclude these files where ever they may occur in the sync tree. Consider adding similar exclusions for file types you don't need to sync, such as core dump and software build files.
bisync testing creates /testdir/ at the top level of the sync tree, and usually deletes the tree after the test. If a normal sync should run while the /testdir/ tree exists the --check-access phase may fail due to unbalanced RCLONE_TEST files. The - /testdir/ filter blocks this tree from being synched. You don't need this exclusion if you are not doing bisync development testing.
Everything else beneath the Path1/Path2 root will be synched.
RCLONE_TEST files may be placed anywhere within the tree, including the root.

Example filters file for Dropbox

# Filter file for use with bisync
# See https://rclone.org/filtering/ for filtering rules
# NOTICE: If you make changes to this file you MUST do a --resync run.
#         Run with --dry-run to see what changes will be made.

# Dropbox won't sync some files so filter them away here.
# See https://help.dropbox.com/installs-integrations/sync-uploads/files-not-syncing
- .dropbox.attr
- ~*.tmp
- ~$*
- .~*
- desktop.ini
- .dropbox

# Used for bisync testing, so excluded from normal runs
- /testdir/

# Other example filters
#- /TiBU/
#- /Photos/

How --check-access handles filters

At the start of a bisync run, listings are gathered for Path1 and Path2 while using the user's --filters-file. During the check access phase, bisync scans these listings for RCLONE_TEST files. Any RCLONE_TEST files hidden by the --filters-file are not in the listings and thus not checked during the check access phase.

Troubleshooting

Reading bisync logs

Here are two normal runs. The first one has a newer file on the remote. The second has no deltas between local and remote.

2021/05/16 00:24:38 INFO  : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/"
2021/05/16 00:24:38 INFO  : Path1 checking for diffs
2021/05/16 00:24:38 INFO  : - Path1    File is new                         - file.txt
2021/05/16 00:24:38 INFO  : Path1:    1 changes:    1 new,    0 newer,    0 older,    0 deleted
2021/05/16 00:24:38 INFO  : Path2 checking for diffs
2021/05/16 00:24:38 INFO  : Applying changes
2021/05/16 00:24:38 INFO  : - Path1    Queue copy to Path2                 - dropbox:/file.txt
2021/05/16 00:24:38 INFO  : - Path1    Do queued copies to                 - Path2
2021/05/16 00:24:38 INFO  : Updating listings
2021/05/16 00:24:38 INFO  : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/"
2021/05/16 00:24:38 INFO  : Bisync successful

2021/05/16 00:36:52 INFO  : Synching Path1 "/path/to/local/tree/" with Path2 "dropbox:/"
2021/05/16 00:36:52 INFO  : Path1 checking for diffs
2021/05/16 00:36:52 INFO  : Path2 checking for diffs
2021/05/16 00:36:52 INFO  : No changes found
2021/05/16 00:36:52 INFO  : Updating listings
2021/05/16 00:36:52 INFO  : Validating listings for Path1 "/path/to/local/tree/" vs Path2 "dropbox:/"
2021/05/16 00:36:52 INFO  : Bisync successful

Dry run oddity

The --dry-run messages may indicate that it would try to delete some files. For example, if a file is new on Path2 and does not exist on Path1 then it would normally be copied to Path1, but with --dry-run enabled those copies don't happen, which leads to the attempted delete on Path2, blocked again by --dry-run: ... Not deleting as --dry-run.

This whole confusing situation is an artifact of the --dry-run flag. Scrutinize the proposed deletes carefully, and if the files would have been copied to Path1 then the threatened deletes on Path2 may be disregarded.

Retries

Rclone has built-in retries. If you run with --verbose you'll see error and retry messages such as shown below. This is usually not a bug. If at the end of the run, you see Bisync successful and not Bisync critical error or Bisync aborted then the run was successful, and you can ignore the error messages.

The following run shows an intermittent fail. Lines 5 and _6- are low-level messages. Line 6 is a bubbled-up warning message, conveying the error. Rclone normally retries failing commands, so there may be numerous such messages in the log.

Since there are no final error/warning messages on line 7, rclone has recovered from failure after a retry, and the overall sync was successful.

1: 2021/05/14 00:44:12 INFO  : Synching Path1 "/path/to/local/tree" with Path2 "dropbox:"
2: 2021/05/14 00:44:12 INFO  : Path1 checking for diffs
3: 2021/05/14 00:44:12 INFO  : Path2 checking for diffs
4: 2021/05/14 00:44:12 INFO  : Path2:  113 changes:   22 new,    0 newer,    0 older,   91 deleted
5: 2021/05/14 00:44:12 ERROR : /path/to/local/tree/objects/af: error listing: unexpected end of JSON input
6: 2021/05/14 00:44:12 NOTICE: WARNING  listing try 1 failed.                 - dropbox:
7: 2021/05/14 00:44:12 INFO  : Bisync successful

This log shows a Critical failure which requires a --resync to recover from. See the Runtime Error Handling section.

2021/05/12 00:49:40 INFO  : Google drive root '': Waiting for checks to finish
2021/05/12 00:49:40 INFO  : Google drive root '': Waiting for transfers to finish
2021/05/12 00:49:40 INFO  : Google drive root '': not deleting files as there were IO errors
2021/05/12 00:49:40 ERROR : Attempt 3/3 failed with 3 errors and: not deleting files as there were IO errors
2021/05/12 00:49:40 ERROR : Failed to sync: not deleting files as there were IO errors
2021/05/12 00:49:40 NOTICE: WARNING  rclone sync try 3 failed.           - /path/to/local/tree/
2021/05/12 00:49:40 ERROR : Bisync aborted. Must run --resync to recover.

Denied downloads of "infected" or "abusive" files

Google Drive has a filter for certain file types (.exe, .apk, et cetera) that by default cannot be copied from Google Drive to the local filesystem. If you are having problems, run with --verbose to see specifically which files are generating complaints. If the error is This file has been identified as malware or spam and cannot be downloaded, consider using the flag --drive-acknowledge-abuse.

Google Docs (and other files of unknown size)

As of v1.66, Google Docs (including Google Sheets, Slides, etc.) are now supported in bisync, subject to the same options, defaults, and limitations as in rclone sync. When bisyncing drive with non-drive backends, the drive -> non-drive direction is controlled by --drive-export-formats (default "docx,xlsx,pptx,svg") and the non-drive -> drive direction is controlled by --drive-import-formats (default none.)

For example, with the default export/import formats, a Google Sheet on the drive side will be synced to an .xlsx file on the non-drive side. In the reverse direction, .xlsx files with filenames that match an existing Google Sheet will be synced to that Google Sheet, while .xlsx files that do NOT match an existing Google Sheet will be copied to drive as normal .xlsx files (without conversion to Sheets, although the Google Drive web browser UI may still give you the option to open it as one.)

If --drive-import-formats is set (it's not, by default), then all of the specified formats will be converted to Google Docs, if there is no existing Google Doc with a matching name. Caution: such conversion can be quite lossy, and in most cases it's probably not what you want!

To bisync Google Docs as URL shortcut links (in a manner similar to "Drive for Desktop"), use: --drive-export-formats url (or alternatives.)

Note that these link files cannot be edited on the non-drive side -- you will get errors if you try to sync an edited link file back to drive. They CAN be deleted (it will result in deleting the corresponding Google Doc.) If you create a .url file on the non-drive side that does not match an existing Google Doc, bisyncing it will just result in copying the literal .url file over to drive (no Google Doc will be created.) So, as a general rule of thumb, think of them as read-only placeholders on the non-drive side, and make all your changes on the drive side.

Likewise, even with other export-formats, it is best to only move/rename Google Docs on the drive side. This is because otherwise, bisync will interpret this as a file deleted and another created, and accordingly, it will delete the Google Doc and create a new file at the new path. (Whether or not that new file is a Google Doc depends on --drive-import-formats.)

Lastly, take note that all Google Docs on the drive side have a size of -1 and no checksum. Therefore, they cannot be reliably synced with the --checksum or --size-only flags. (To be exact: they will still get created/deleted, and bisync's delta engine will notice changes and queue them for syncing, but the underlying sync function will consider them identical and skip them.) To work around this, use the default (modtime and size) instead of --checksum or --size-only.

To ignore Google Docs entirely, use --drive-skip-gdocs.

Usage examples

Cron

Rclone does not yet have a built-in capability to monitor the local file system for changes and must be blindly run periodically. On Windows this can be done using a Task Scheduler, on Linux you can use Cron which is described below.

The 1st example runs a sync every 5 minutes between a local directory and an OwnCloud server, with output logged to a runlog file:

# Minute (0-59)
#      Hour (0-23)
#           Day of Month (1-31)
#                Month (1-12 or Jan-Dec)
#                     Day of Week (0-6 or Sun-Sat)
#                         Command
  */5  *    *    *    *   /path/to/rclone bisync /local/files MyCloud: --check-access --filters-file /path/to/bysync-filters.txt --log-file /path/to//bisync.log

See crontab syntax for the details of crontab time interval expressions.

If you run rclone bisync as a cron job, redirect stdout/stderr to a file. The 2nd example runs a sync to Dropbox every hour and logs all stdout (via the >>) and stderr (via 2>&1) to a log file.

0 * * * * /path/to/rclone bisync /path/to/local/dropbox Dropbox: --check-access --filters-file /home/user/filters.txt >> /path/to/logs/dropbox-run.log 2>&1

bisync can keep a local folder in sync with a cloud service, but what if you have some highly sensitive files to be synched?

Usage of a cloud service is for exchanging both routine and sensitive personal files between one's home network, one's personal notebook when on the road, and with one's work computer. The routine data is not sensitive. For the sensitive data, configure an rclone crypt remote to point to a subdirectory within the local disk tree that is bisync'd to Dropbox, and then set up an bisync for this local crypt directory to a directory outside of the main sync tree.

Linux server setup

/path/to/DBoxroot is the root of my local sync tree. There are numerous subdirectories.
/path/to/DBoxroot/crypt is the root subdirectory for files that are encrypted. This local directory target is setup as an rclone crypt remote named Dropcrypt:. See rclone.conf snippet below.
/path/to/my/unencrypted/files is the root of my sensitive files - not encrypted, not within the tree synched to Dropbox.
To sync my local unencrypted files with the encrypted Dropbox versions I manually run bisync /path/to/my/unencrypted/files DropCrypt:. This step could be bundled into a script to run before and after the full Dropbox tree sync in the last step, thus actively keeping the sensitive files in sync.
bisync /path/to/DBoxroot Dropbox: runs periodically via cron, keeping my full local sync tree in sync with Dropbox.

Windows notebook setup

The Dropbox client runs keeping the local tree C:\Users\MyLogin\Dropbox always in sync with Dropbox. I could have used rclone bisync instead.
A separate directory tree at C:\Users\MyLogin\Documents\DropLocal hosts the tree of unencrypted files/folders.
To sync my local unencrypted files with the encrypted Dropbox versions I manually run the following command: rclone bisync C:\Users\MyLogin\Documents\DropLocal Dropcrypt:.
The Dropbox client then syncs the changes with Dropbox.

rclone.conf snippet

[Dropbox]
type = dropbox
...

[Dropcrypt]
type = crypt
remote = /path/to/DBoxroot/crypt          # on the Linux server
remote = C:\Users\MyLogin\Dropbox\crypt   # on the Windows notebook
filename_encryption = standard
directory_name_encryption = true
password = ...
...

Testing

You should read this section only if you are developing for rclone. You need to have rclone source code locally to work with bisync tests.

Bisync has a dedicated test framework implemented in the bisync_test.go file located in the rclone source tree. The test suite is based on the go test command. Series of tests are stored in subdirectories below the cmd/bisync/testdata directory. Individual tests can be invoked by their directory name, e.g. go test . -case basic -remote local -remote2 gdrive: -v

Tests will make a temporary folder on remote and purge it afterwards. If during test run there are intermittent errors and rclone retries, these errors will be captured and flagged as invalid MISCOMPAREs. Rerunning the test will let it pass. Consider such failures as noise.

Test command syntax

usage: go test ./cmd/bisync [options...]

Options:
  -case NAME        Name(s) of the test case(s) to run. Multiple names should
                    be separated by commas. You can remove the `test_` prefix
                    and replace `_` by `-` in test name for convenience.
                    If not `all`, the name(s) should map to a directory under
                    `./cmd/bisync/testdata`.
                    Use `all` to run all tests (default: all)
  -remote PATH1     `local` or name of cloud service with `:` (default: local)
  -remote2 PATH2    `local` or name of cloud service with `:` (default: local)
  -no-compare       Disable comparing test results with the golden directory
                    (default: compare)
  -no-cleanup       Disable cleanup of Path1 and Path2 testdirs.
                    Useful for troubleshooting. (default: cleanup)
  -golden           Store results in the golden directory (default: false)
                    This flag can be used with multiple tests.
  -debug            Print debug messages
  -stop-at NUM      Stop test after given step number. (default: run to the end)
                    Implies `-no-compare` and `-no-cleanup`, if the test really
                    ends prematurely. Only meaningful for a single test case.
  -refresh-times    Force refreshing the target modtime, useful for Dropbox
                    (default: false)
  -verbose          Run tests verbosely

Note: unlike rclone flags which must be prefixed by double dash (--), the test command flags can be equally prefixed by a single - or double dash.

Running tests

go test . -case basic -remote local -remote2 local runs the test_basic test case using only the local filesystem, synching one local directory with another local directory. Test script output is to the console, while commands within scenario.txt have their output sent to the .../workdir/test.log file, which is finally compared to the golden copy.
The first argument after go test should be a relative name of the directory containing bisync source code. If you run tests right from there, the argument will be . (current directory) as in most examples below. If you run bisync tests from the rclone source directory, the command should be go test ./cmd/bisync ....
The test engine will mangle rclone output to ensure comparability with golden listings and logs.
Test scenarios are located in ./cmd/bisync/testdata. The test -case argument should match the full name of a subdirectory under that directory. Every test subdirectory name on disk must start with test_, this prefix can be omitted on command line for brevity. Also, underscores in the name can be replaced by dashes for convenience.
go test . -remote local -remote2 local -case all runs all tests.
Path1 and Path2 may either be the keyword local or may be names of configured cloud services. go test . -remote gdrive: -remote2 dropbox: -case basic will run the test between these two services, without transferring any files to the local filesystem.
Test run stdout and stderr console output may be directed to a file, e.g. go test . -remote gdrive: -remote2 local -case all > runlog.txt 2>&1

Test execution flow

The base setup in the initial directory of the testcase is applied on the Path1 and Path2 filesystems (via rclone copy the initial directory to Path1, then rclone sync Path1 to Path2).
The commands in the scenario.txt file are applied, with output directed to the test.log file in the test working directory. Typically, the first actual command in the scenario.txt file is to do a --resync, which establishes the baseline {...}.path1.lst and {...}.path2.lst files in the test working directory (.../workdir/ relative to the temporary test directory). Various commands and listing snapshots are done within the test.
Finally, the contents of the test working directory are compared to the contents of the testcase's golden directory.

Notes about testing

Test cases are in individual directories beneath ./cmd/bisync/testdata. A command line reference to a test is understood to reference a directory beneath testdata. For example, go test ./cmd/bisync -case dry-run -remote gdrive: -remote2 local refers to the test case in ./cmd/bisync/testdata/test_dry_run.
The test working directory is located at .../workdir relative to a temporary test directory, usually under /tmp on Linux.
The local test sync tree is created at a temporary directory named like bisync.XXX under system temporary directory.
The remote test sync tree is located at a temporary directory under <remote:>/bisync.XXX/.
path1 and/or path2 subdirectories are created in a temporary directory under the respective local or cloud test remote.
By default, the Path1 and Path2 test dirs and workdir will be deleted after each test run. The -no-cleanup flag disables purging these directories when validating and debugging a given test. These directories will be flushed before running another test, independent of the -no-cleanup usage.
You will likely want to add - /testdir/ to your normal bisync --filters-file so that normal syncs do not attempt to sync the test temporary directories, which may have RCLONE_TEST miscompares in some testcases which would otherwise trip the --check-access system. The --check-access mechanism is hard-coded to ignore RCLONE_TEST files beneath bisync/testdata, so the test cases may reside on the synched tree even if there are check file mismatches in the test tree.
Some Dropbox tests can fail, notably printing the following message: src and dst identical but can't set mod time without deleting and re-uploading This is expected and happens due to the way Dropbox handles modification times. You should use the -refresh-times test flag to make up for this.
If Dropbox tests hit request limit for you and print error message too_many_requests/...: Too many requests or write operations. then follow the Dropbox App ID instructions.

Updating golden results

Sometimes even a slight change in the bisync source can cause little changes spread around many log files. Updating them manually would be a nightmare.

The -golden flag will store the test.log and *.lst listings from each test case into respective golden directories. Golden results will automatically contain generic strings instead of local or cloud paths which means that they should match when run with a different cloud service.

Your normal workflow might be as follows:

Git-clone the rclone sources locally
Modify bisync source and check that it builds
Run the whole test suite go test ./cmd/bisync -remote local
If some tests show log difference, recheck them individually, e.g.: go test ./cmd/bisync -remote local -case basic
If you are convinced with the difference, goldenize all tests at once: go test ./cmd/bisync -remote local -golden
Use word diff: git diff --word-diff ./cmd/bisync/testdata/. Please note that normal line-level diff is generally useless here.
Check the difference carefully!
Commit the change (git commit) only if you are sure. If unsure, save your code changes then wipe the log diffs from git: git reset [--hard].

Structure of test scenarios

<testname>/initial/ contains a tree of files that will be set as the initial condition on both Path1 and Path2 testdirs.
<testname>/modfiles/ contains files that will be used to modify the Path1 and/or Path2 filesystems.
<testname>/golden/ contains the expected content of the test working directory (workdir) at the completion of the testcase.
<testname>/scenario.txt contains the body of the test, in the form of various commands to modify files, run bisync, and snapshot listings. Output from these commands is captured to .../workdir/test.log for comparison to the golden files.

Supported test commands

test <some message> Print the line to the console and to the test.log: test sync is working correctly with options x, y, z
copy-listings <prefix> Save a copy of all .lst listings in the test working directory with the specified prefix: save-listings exclude-pass-run
move-listings <prefix> Similar to copy-listings but removes the source
purge-children <dir> This will delete all child files and purge all child subdirs under given directory but keep the parent intact. This behavior is important for tests with Google Drive because removing and re-creating the parent would change its ID.
delete-file <file> Delete a single file.
delete-glob <dir> <pattern> Delete a group of files located one level deep in the given directory with names matching a given glob pattern.
touch-glob YYYY-MM-DD <dir> <pattern> Change modification time on a group of files.
touch-copy YYYY-MM-DD <source-file> <dest-dir> Change file modification time then copy it to destination.
copy-file <source-file> <dest-dir> Copy a single file to given directory.
copy-as <source-file> <dest-file> Similar to above but destination must include both directory and the new file name at destination.
copy-dir <src> <dst> and sync-dir <src> <dst> Copy/sync a directory. Equivalent of rclone copy and rclone sync.
list-dirs <dir> Equivalent to rclone lsf -R --dirs-only <dir>
bisync [options] Runs bisync against -remote and -remote2.

Supported substitution terms

{testdir/} - the root dir of the testcase
{datadir/} - the modfiles dir under the testcase root
{workdir/} - the temporary test working directory
{path1/} - the root of the Path1 test directory tree
{path2/} - the root of the Path2 test directory tree
{session} - base name of the test listings
{/} - OS-specific path separator
{spc}, {tab}, {eol} - whitespace
{chr:HH} - raw byte with given hexadecimal code

Substitution results of the terms named like {dir/} will end with / (or backslash on Windows), so it is not necessary to include slash in the usage, for example delete-file {path1/}file1.txt.

Benchmarks

This section is work in progress.

Here are a few data points for scale, execution times, and memory usage.

The first set of data was taken between a local disk to Dropbox. The speedtest.net download speed was ~170 Mbps, and upload speed was ~10 Mbps. 500 files (~9.5 MB each) had been already synched. 50 files were added in a new directory, each ~9.5 MB, ~475 MB total.

Change	Operations and times	Overall run time
500 files synched (nothing to move)	1x listings for Path1 & Path2	1.5 sec
500 files synched with --check-access	1x listings for Path1 & Path2	1.5 sec
50 new files on remote	Queued 50 copies down: 27 sec	29 sec
Moved local dir	Queued 50 copies up: 410 sec, 50 deletes up: 9 sec	421 sec
Moved remote dir	Queued 50 copies down: 31 sec, 50 deletes down: <1 sec	33 sec
Delete local dir	Queued 50 deletes up: 9 sec	13 sec

This next data is from a user's application. They had ~400GB of data over 1.96 million files being sync'ed between a Windows local disk and some remote cloud. The file full path length was on average 35 characters (which factors into load time and RAM required).

Loading the prior listing into memory (1.96 million files, listing file size 140 MB) took ~30 sec and occupied about 1 GB of RAM.
Getting a fresh listing of the local file system (producing the 140 MB output file) took about XXX sec.
Getting a fresh listing of the remote file system (producing the 140 MB output file) took about XXX sec. The network download speed was measured at XXX Mb/s.
Once the prior and current Path1 and Path2 listings were loaded (a total of four to be loaded, two at a time), determining the deltas was pretty quick (a few seconds for this test case), and the transfer time for any files to be copied was dominated by the network bandwidth.

References

rclone's bisync implementation was derived from the rclonesync-V2 project, including documentation and test mechanisms, with @cjnaz's full support and encouragement.

rclone bisync is similar in nature to a range of other projects:

Bisync adopts the differential synchronization technique, which is based on keeping history of changes performed by both synchronizing sides. See the Dual Shadow Method section in Neil Fraser's article.

Also note a number of academic publications by Benjamin Pierce about Unison and synchronization in general.

Changelog

`v1.68`

Fixed an issue affecting backends that round modtimes to a lower precision.

`v1.67`

Added integration tests against all backends.

`v1.66`

Copies and deletes are now handled in one operation instead of two
--track-renames and --backup-dir are now supported
Partial uploads known issue on local/ftp/sftp has been resolved (unless using --inplace)
Final listings are now generated from sync results, to avoid needing to re-list
Bisync is now much more resilient to changes that happen during a bisync run, and far less prone to critical errors / undetected changes
Bisync is now capable of rolling a file listing back in cases of uncertainty, essentially marking the file as needing to be rechecked next time.
A few basic terminal colors are now supported, controllable with --color (AUTO|NEVER|ALWAYS)
Initial listing snapshots of Path1 and Path2 are now generated concurrently, using the same "march" infrastructure as check and sync, for performance improvements and less risk of error.
Fixed handling of unicode normalization and case insensitivity, support for --fix-case, --ignore-case-sync, --no-unicode-normalization
--resync is now much more efficient (especially for users of --create-empty-src-dirs)
Google Docs (and other files of unknown size) are now supported (with the same options as in sync)
Equality checks before a sync conflict rename now fall back to cryptcheck (when possible) or --download, instead of of --size-only, when check is not available.
Bisync no longer fails to find the correct listing file when configs are overridden with backend-specific flags.
Bisync now fully supports comparing based on any combination of size, modtime, and checksum, lifting the prior restriction on backends without modtime support.
Bisync now supports a "Graceful Shutdown" mode to cleanly cancel a run early without requiring --resync.
New --recover flag allows robust recovery in the event of interruptions, without requiring --resync.
A new --max-lock setting allows lock files to automatically renew and expire, for better automatic recovery when a run is interrupted.
Bisync now supports auto-resolving sync conflicts and customizing rename behavior with new --conflict-resolve, --conflict-loser, and --conflict-suffix flags.
A new --resync-mode flag allows more control over which version of a file gets kept during a --resync.
Bisync now supports --retries and --retries-sleep (when --resilient is set.)

`v1.64`

Fixed an issue causing dry runs to inadvertently commit filter changes
Fixed an issue causing --resync to erroneously delete empty folders and duplicate files unique to Path2
--check-access is now enforced during --resync, preventing data loss in certain user error scenarios
Fixed an issue causing bisync to consider more files than necessary due to overbroad filters during delete operations
Improved detection of false positive change conflicts (identical files are now left alone instead of renamed)
Added support for --create-empty-src-dirs
Added experimental --resilient mode to allow recovery from self-correctable errors
Added new --ignore-listing-checksum flag to distinguish from --ignore-checksum
Performance improvements for large remotes
Documentation and testing improvements

Box

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Box

Paths are specified as remote:path

Paths may be as deep as required, e.g. remote:directory/subdirectory.

The initial setup for Box involves getting a token from Box which you can do either in your browser, or with a config.json downloaded from Box to use JWT authentication. rclone config walks you through it.

Configuration

Here is an example of how to make a remote called remote. First run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Box
   \ "box"
[snip]
Storage> box
Box App Client Id - leave blank normally.
client_id> 
Box App Client Secret - leave blank normally.
client_secret>
Box App config.json location
Leave blank normally.
Enter a string value. Press Enter for the default ("").
box_config_file>
Box App Primary Access Token
Leave blank normally.
Enter a string value. Press Enter for the default ("").
access_token>

Enter a string value. Press Enter for the default ("user").
Choose a number from below, or type in your own value
 1 / Rclone should act on behalf of a user
   \ "user"
 2 / Rclone should act on behalf of a service account
   \ "enterprise"
box_sub_type>
Remote config
Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
Configuration complete.
Options:
- type: box
- client_id:
- client_secret:
- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"XXX"}
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

See the remote setup docs for how to set it up on a machine with no Internet browser available.

Note that rclone runs a webserver on your local machine to collect the token as returned from Box. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this it may require you to unblock it temporarily if you are running a host firewall.

Once configured you can then use rclone like this,

List directories in top level of your Box

rclone lsd remote:

List all the files in your Box

rclone ls remote:

To copy a local directory to an Box directory called backup

rclone copy /home/source remote:backup

Using rclone with an Enterprise account with SSO

If you have an "Enterprise" account type with Box with single sign on (SSO), you need to create a password to use Box with rclone. This can be done at your Enterprise Box account by going to Settings, "Account" Tab, and then set the password in the "Authentication" field.

Once you have done this, you can setup your Enterprise Box account using the same procedure detailed above in the, using the password you have just set.

Invalid refresh token

According to the box docs:

Each refresh_token is valid for one use in 60 days.

This means that if you

Don't use the box remote for 60 days
Copy the config file with a box refresh token in and use it in two places
Get an error on a token refresh

then rclone will return an error which includes the text Invalid refresh token.

To fix this you will need to use oauth2 again to update the refresh token. You can use the methods in the remote setup docs, bearing in mind that if you use the copy the config file method, you should not use that remote on the computer you did the authentication on.

Here is how to do it.

$ rclone config
Current remotes:

Name                 Type
====                 ====
remote               box

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> e
Choose a number from below, or type in an existing value
 1 > remote
remote> remote
Configuration complete.
Options:
- type: box
- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2017-07-08T23:40:08.059167677+01:00"}
Keep this "remote" remote?
Edit remote
Value "client_id" = ""
Edit? (y/n)>
y) Yes
n) No
y/n> n
Value "client_secret" = ""
Edit? (y/n)>
y) Yes
n) No
y/n> n
Remote config
Already have a token - refresh?
y) Yes
n) No
y/n> y
Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
Log in and authorize rclone for access
Waiting for code...
Got code
Configuration complete.
Options:
- type: box
- token: {"access_token":"YYY","token_type":"bearer","refresh_token":"YYY","expiry":"2017-07-23T12:22:29.259137901+01:00"}
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Modification times and hashes

Box allows modification times to be set on objects accurate to 1 second. These will be used to detect whether objects need syncing or not.

Box supports SHA1 type hashes, so you can use the --checksum flag.

Restricted filename characters

In addition to the default restricted characters set the following characters are also replaced:

Character	Value	Replacement
\	0x5C	＼

File names can also not end with the following characters. These only get replaced if they are the last character in the name:

Character	Value	Replacement
SP	0x20	␠

Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

Transfers

For files above 50 MiB rclone will use a chunked transfer. Rclone will upload up to --transfers chunks at the same time (shared among all the multipart uploads). Chunks are buffered in memory and are normally 8 MiB so increasing --transfers will increase memory use.

Deleting files

Depending on the enterprise settings for your user, the item will either be actually deleted from Box or moved to the trash.

Emptying the trash is supported via the rclone however cleanup command however this deletes every trashed file and folder individually so it may take a very long time. Emptying the trash via the WebUI does not have this limitation so it is advised to empty the trash via the WebUI.

Root folder ID

You can set the root_folder_id for rclone. This is the directory (identified by its Folder ID) that rclone considers to be the root of your Box drive.

Normally you will leave this blank and rclone will determine the correct root to use itself.

However you can set this to restrict rclone to a specific folder hierarchy.

In order to do this you will have to find the Folder ID of the directory you wish rclone to display. This will be the last segment of the URL when you open the relevant folder in the Box web interface.

So if the folder you want rclone to use has a URL which looks like https://app.box.com/folder/11xxxxxxxxx8 in the browser, then you use 11xxxxxxxxx8 as the root_folder_id in the config.

Standard options

Here are the Standard options specific to box (Box).

--box-client-id

OAuth Client Id.

Leave blank normally.

Properties:

Config: client_id
Env Var: RCLONE_BOX_CLIENT_ID
Type: string
Required: false

--box-client-secret

OAuth Client Secret.

Leave blank normally.

Properties:

Config: client_secret
Env Var: RCLONE_BOX_CLIENT_SECRET
Type: string
Required: false

--box-box-config-file

Box App config.json location

Leave blank normally.

Leading ~ will be expanded in the file name as will environment variables such as ${RCLONE_CONFIG_DIR}.

Properties:

Config: box_config_file
Env Var: RCLONE_BOX_BOX_CONFIG_FILE
Type: string
Required: false

--box-access-token

Box App Primary Access Token

Leave blank normally.

Properties:

Config: access_token
Env Var: RCLONE_BOX_ACCESS_TOKEN
Type: string
Required: false

--box-box-sub-type

Properties:

Config: box_sub_type
Env Var: RCLONE_BOX_BOX_SUB_TYPE
Type: string
Default: "user"
Examples:
- "user"
  - Rclone should act on behalf of a user.
- "enterprise"
  - Rclone should act on behalf of a service account.

Advanced options

Here are the Advanced options specific to box (Box).

--box-token

OAuth Access Token as a JSON blob.

Properties:

Config: token
Env Var: RCLONE_BOX_TOKEN
Type: string
Required: false

--box-auth-url

Auth server URL.

Leave blank to use the provider defaults.

Properties:

Config: auth_url
Env Var: RCLONE_BOX_AUTH_URL
Type: string
Required: false

--box-token-url

Token server url.

Leave blank to use the provider defaults.

Properties:

Config: token_url
Env Var: RCLONE_BOX_TOKEN_URL
Type: string
Required: false

--box-root-folder-id

Fill in for rclone to use a non root folder as its starting point.

Properties:

Config: root_folder_id
Env Var: RCLONE_BOX_ROOT_FOLDER_ID
Type: string
Default: "0"

--box-upload-cutoff

Cutoff for switching to multipart upload (>= 50 MiB).

Properties:

Config: upload_cutoff
Env Var: RCLONE_BOX_UPLOAD_CUTOFF
Type: SizeSuffix
Default: 50Mi

--box-commit-retries

Max number of times to try committing a multipart file.

Properties:

Config: commit_retries
Env Var: RCLONE_BOX_COMMIT_RETRIES
Type: int
Default: 100

--box-list-chunk

Size of listing chunk 1-1000.

Properties:

Config: list_chunk
Env Var: RCLONE_BOX_LIST_CHUNK
Type: int
Default: 1000

--box-owned-by

Only show items owned by the login (email address) passed in.

Properties:

Config: owned_by
Env Var: RCLONE_BOX_OWNED_BY
Type: string
Required: false

--box-impersonate

Impersonate this user ID when using a service account.

Setting this flag allows rclone, when using a JWT service account, to act on behalf of another user by setting the as-user header.

The user ID is the Box identifier for a user. User IDs can found for any user via the GET /users endpoint, which is only available to admins, or by calling the GET /users/me endpoint with an authenticated user session.

See: https://developer.box.com/guides/authentication/jwt/as-user/

Properties:

Config: impersonate
Env Var: RCLONE_BOX_IMPERSONATE
Type: string
Required: false

--box-encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Properties:

Config: encoding
Env Var: RCLONE_BOX_ENCODING
Type: Encoding
Default: Slash,BackSlash,Del,Ctl,RightSpace,InvalidUtf8,Dot

--box-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_BOX_DESCRIPTION
Type: string
Required: false

Limitations

Note that Box is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

Box file names can't have the \ character in. rclone maps this to and from an identical looking unicode equivalent ＼ (U+FF3C Fullwidth Reverse Solidus).

Box only supports filenames up to 255 characters in length.

Box has API rate limits that sometimes reduce the speed of rclone.

rclone about is not supported by the Box backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

See List of backends that do not support rclone about and rclone about

Get your own Box App ID

Here is how to create your own Box App ID for rclone:

Go to the Box Developer Console and login, then click My Apps on the sidebar. Click Create New App and select Custom App.
In the first screen on the box that pops up, you can pretty much enter whatever you want. The App Name can be whatever. For Purpose choose automation to avoid having to fill out anything else. Click Next.
In the second screen of the creation screen, select User Authentication (OAuth 2.0). Then click Create App.
You should now be on the Configuration tab of your new app. If not, click on it at the top of the webpage. Copy down Client ID and Client Secret, you'll need those for rclone.
Under "OAuth 2.0 Redirect URI", add http://127.0.0.1:53682/
For Application Scopes, select Read all files and folders stored in Box and Write all files and folders stored in box (assuming you want to do both). Leave others unchecked. Click Save Changes at the top right.

Bugs

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Bugs and Limitations

Limitations

Directory timestamps aren't preserved on some backends

As of v1.66, rclone supports syncing directory modtimes, if the backend supports it. Some backends do not support it -- see overview for a complete list. Additionally, note that empty directories are not synced by default (this can be enabled with --create-empty-src-dirs.)

Rclone struggles with millions of files in a directory/bucket

Currently rclone loads each directory/bucket entirely into memory before using it. Since each rclone object takes 0.5k-1k of memory this can take a very long time and use a large amount of memory.

Millions of files in a directory tends to occur on bucket-based remotes (e.g. S3 buckets) since those remotes do not segregate subdirectories within the bucket.

Bucket-based remotes and folders

Bucket-based remotes (e.g. S3/GCS/Swift/B2) do not have a concept of directories. Rclone therefore cannot create directories in them which means that empty directories on a bucket-based remote will tend to disappear.

Some software creates empty keys ending in / as directory markers. Rclone doesn't do this as it potentially creates more objects and costs more. This ability may be added in the future (probably via a flag/option).

Bugs

Bugs are stored in rclone's GitHub project:

Cache

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Cache

The cache remote wraps another existing remote and stores file structure and its data for long running tasks like rclone mount.

Status

The cache backend code is working but it currently doesn't have a maintainer so there are outstanding bugs which aren't getting fixed.

The cache backend is due to be phased out in favour of the VFS caching layer eventually which is more tightly integrated into rclone.

Until this happens we recommend only using the cache backend if you find you can't work without it. There are many docs online describing the use of the cache backend to minimize API hits and by-and-large these are out of date and the cache backend isn't needed in those scenarios any more.

Configuration

To get started you just need to have an existing remote which can be configured with cache.

Here is an example of how to make a remote called test-cache. First run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
n/r/c/s/q> n
name> test-cache
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Cache a remote
   \ "cache"
[snip]
Storage> cache
Remote to cache.
Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
remote> local:/test
Optional: The URL of the Plex server
plex_url> http://127.0.0.1:32400
Optional: The username of the Plex user
plex_username> dummyusername
Optional: The password of the Plex user
y) Yes type in my own password
g) Generate random password
n) No leave this optional password blank
y/g/n> y
Enter the password:
password:
Confirm the password:
password:
The size of a chunk. Lower value good for slow connections but can affect seamless reading.
Default: 5M
Choose a number from below, or type in your own value
 1 / 1 MiB
   \ "1M"
 2 / 5 MiB
   \ "5M"
 3 / 10 MiB
   \ "10M"
chunk_size> 2
How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache.
Accepted units are: "s", "m", "h".
Default: 5m
Choose a number from below, or type in your own value
 1 / 1 hour
   \ "1h"
 2 / 24 hours
   \ "24h"
 3 / 24 hours
   \ "48h"
info_age> 2
The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted.
Default: 10G
Choose a number from below, or type in your own value
 1 / 500 MiB
   \ "500M"
 2 / 1 GiB
   \ "1G"
 3 / 10 GiB
   \ "10G"
chunk_total_size> 3
Remote config
--------------------
[test-cache]
remote = local:/test
plex_url = http://127.0.0.1:32400
plex_username = dummyusername
plex_password = *** ENCRYPTED ***
chunk_size = 5M
info_age = 48h
chunk_total_size = 10G

You can then use it like this,

List directories in top level of your drive

rclone lsd test-cache:

List all the files in your drive

rclone ls test-cache:

To start a cached mount

rclone mount --allow-other test-cache: /var/tmp/test-cache

Write Features

Offline uploading

In an effort to make writing through cache more reliable, the backend now supports this feature which can be activated by specifying a cache-tmp-upload-path.

A files goes through these states when using this feature:

An upload is started (usually by copying a file on the cache remote)
When the copy to the temporary location is complete the file is part of the cached remote and looks and behaves like any other file (reading included)
After cache-tmp-wait-time passes and the file is next in line, rclone move is used to move the file to the cloud provider
Reading the file still works during the upload but most modifications on it will be prohibited
Once the move is complete the file is unlocked for modifications as it becomes as any other regular file
If the file is being read through cache when it's actually deleted from the temporary path then cache will simply swap the source to the cloud provider without interrupting the reading (small blip can happen though)

Files are uploaded in sequence and only one file is uploaded at a time. Uploads will be stored in a queue and be processed based on the order they were added. The queue and the temporary storage is persistent across restarts but can be cleared on startup with the --cache-db-purge flag.

Write Support

Writes are supported through cache. One caveat is that a mounted cache remote does not add any retry or fallback mechanism to the upload operation. This will depend on the implementation of the wrapped remote. Consider using Offline uploading for reliable writes.

One special case is covered with cache-writes which will cache the file data at the same time as the upload when it is enabled making it available from the cache store immediately once the upload is finished.

Read Features

Multiple connections

To counter the high latency between a local PC where rclone is running and cloud providers, the cache remote can split multiple requests to the cloud provider for smaller file chunks and combines them together locally where they can be available almost immediately before the reader usually needs them.

This is similar to buffering when media files are played online. Rclone will stay around the current marker but always try its best to stay ahead and prepare the data before.

Plex Integration

There is a direct integration with Plex which allows cache to detect during reading if the file is in playback or not. This helps cache to adapt how it queries the cloud provider depending on what is needed for.

Scans will have a minimum amount of workers (1) while in a confirmed playback cache will deploy the configured number of workers.

This integration opens the doorway to additional performance improvements which will be explored in the near future.

Note: If Plex options are not configured, cache will function with its configured options without adapting any of its settings.

How to enable? Run rclone config and add all the Plex options (endpoint, username and password) in your remote and it will be automatically enabled.

Affected settings:

cache-workers: Configured value during confirmed playback or 1 all the other times

Certificate Validation

When the Plex server is configured to only accept secure connections, it is possible to use .plex.direct URLs to ensure certificate validation succeeds. These URLs are used by Plex internally to connect to the Plex server securely.

The format for these URLs is the following:

https://ip-with-dots-replaced.server-hash.plex.direct:32400/

The ip-with-dots-replaced part can be any IPv4 address, where the dots have been replaced with dashes, e.g. 127.0.0.1 becomes 127-0-0-1.

To get the server-hash part, the easiest way is to visit

https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token

This page will list all the available Plex servers for your account with at least one .plex.direct link for each. Copy one URL and replace the IP address with the desired address. This can be used as the plex_url value.

Known issues

Mount and --dir-cache-time

--dir-cache-time controls the first layer of directory caching which works at the mount layer. Being an independent caching mechanism from the cache backend, it will manage its own entries based on the configured time.

To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct one, try to set --dir-cache-time to a lower time than --cache-info-age. Default values are already configured in this way.

Windows support - Experimental

There are a couple of issues with Windows mount functionality that still require some investigations. It should be considered as experimental thus far as fixes come in for this OS.

Most of the issues seem to be related to the difference between filesystems on Linux flavors and Windows as cache is heavily dependent on them.

Any reports or feedback on how cache behaves on this OS is greatly appreciated.

Risk of throttling

Future iterations of the cache backend will make use of the pooling functionality of the cloud provider to synchronize and at the same time make writing through it more tolerant to failures.

There are a couple of enhancements in track to add these but in the meantime there is a valid concern that the expiring cache listings can lead to cloud provider throttles or bans due to repeated queries on it for very large mounts.

Some recommendations:

don't use a very small interval for entry information (--cache-info-age)
while writes aren't yet optimised, you can still write through cache which gives you the advantage of adding the file in the cache at the same time if configured to do so.

Future enhancements:

cache and crypt

One common scenario is to keep your data encrypted in the cloud provider using the crypt remote. crypt uses a similar technique to wrap around an existing remote and handles this translation in a seamless way.

There is an issue with wrapping the remotes in this order: **cloud remote** -> **crypt** -> **cache**

During testing, I experienced a lot of bans with the remotes in this order. I suspect it might be related to how crypt opens files on the cloud provider which makes it think we're downloading the full file instead of small chunks. Organizing the remotes in this order yields better results: **cloud remote** -> **cache** -> **crypt**

absolute remote paths

cache can not differentiate between relative and absolute paths for the wrapped remote. Any path given in the remote config setting and on the command line will be passed to the wrapped remote as is, but for storing the chunks on disk the path will be made relative by removing any leading / character.

This behavior is irrelevant for most backend types, but there are backends where a leading / changes the effective directory, e.g. in the sftp backend paths starting with a / are relative to the root of the SSH server and paths without are relative to the user home directory. As a result sftp:bin and sftp:/bin will share the same cache folder, even if they represent a different directory on the SSH server.

Cache and Remote Control (--rc)

Cache supports the new --rc mode in rclone and can be remote controlled through the following end points: By default, the listener is disabled if you do not add the flag.

rc cache/expire

Purge a remote from the cache backend. Supports either a directory or a file. It supports both encrypted and unencrypted file names if cache is wrapped by crypt.

Params:

remote = path to remote (required)
withData = true/false to delete cached data (chunks) as well (optional, false by default)

Standard options

Here are the Standard options specific to cache (Cache a remote).

--cache-remote

Remote to cache.

Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended).

Properties:

Config: remote
Env Var: RCLONE_CACHE_REMOTE
Type: string
Required: true

--cache-plex-url

The URL of the Plex server.

Properties:

Config: plex_url
Env Var: RCLONE_CACHE_PLEX_URL
Type: string
Required: false

--cache-plex-username

The username of the Plex user.

Properties:

Config: plex_username
Env Var: RCLONE_CACHE_PLEX_USERNAME
Type: string
Required: false

--cache-plex-password

The password of the Plex user.

NB Input to this must be obscured - see rclone obscure.

Properties:

Config: plex_password
Env Var: RCLONE_CACHE_PLEX_PASSWORD
Type: string
Required: false

--cache-chunk-size

The size of a chunk (partial file data).

Use lower numbers for slower connections. If the chunk size is changed, any downloaded chunks will be invalid and cache-chunk-path will need to be cleared or unexpected EOF errors will occur.

Properties:

Config: chunk_size
Env Var: RCLONE_CACHE_CHUNK_SIZE
Type: SizeSuffix
Default: 5Mi
Examples:
- "1M"
  - 1 MiB
- "5M"
  - 5 MiB
- "10M"
  - 10 MiB

--cache-info-age

How long to cache file structure information (directory listings, file size, times, etc.). If all write operations are done through the cache then you can safely make this value very large as the cache store will also be updated in real time.

Properties:

Config: info_age
Env Var: RCLONE_CACHE_INFO_AGE
Type: Duration
Default: 6h0m0s
Examples:
- "1h"
  - 1 hour
- "24h"
  - 24 hours
- "48h"
  - 48 hours

--cache-chunk-total-size

The total size that the chunks can take up on the local disk.

If the cache exceeds this value then it will start to delete the oldest chunks until it goes under this value.

Properties:

Config: chunk_total_size
Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
Type: SizeSuffix
Default: 10Gi
Examples:
- "500M"
  - 500 MiB
- "1G"
  - 1 GiB
- "10G"
  - 10 GiB

Advanced options

Here are the Advanced options specific to cache (Cache a remote).

--cache-plex-token

The plex token for authentication - auto set normally.

Properties:

Config: plex_token
Env Var: RCLONE_CACHE_PLEX_TOKEN
Type: string
Required: false

--cache-plex-insecure

Skip all certificate verification when connecting to the Plex server.

Properties:

Config: plex_insecure
Env Var: RCLONE_CACHE_PLEX_INSECURE
Type: string
Required: false

--cache-db-path

Directory to store file structure metadata DB.

The remote name is used as the DB file name.

Properties:

Config: db_path
Env Var: RCLONE_CACHE_DB_PATH
Type: string
Default: "$HOME/.cache/rclone/cache-backend"

--cache-chunk-path

Directory to cache chunk files.

Path to where partial file data (chunks) are stored locally. The remote name is appended to the final path.

This config follows the "--cache-db-path". If you specify a custom location for "--cache-db-path" and don't specify one for "--cache-chunk-path" then "--cache-chunk-path" will use the same path as "--cache-db-path".

Properties:

Config: chunk_path
Env Var: RCLONE_CACHE_CHUNK_PATH
Type: string
Default: "$HOME/.cache/rclone/cache-backend"

--cache-db-purge

Clear all the cached data for this remote on start.

Properties:

Config: db_purge
Env Var: RCLONE_CACHE_DB_PURGE
Type: bool
Default: false

--cache-chunk-clean-interval

How often should the cache perform cleanups of the chunk storage.

The default value should be ok for most people. If you find that the cache goes over "cache-chunk-total-size" too often then try to lower this value to force it to perform cleanups more often.

Properties:

Config: chunk_clean_interval
Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
Type: Duration
Default: 1m0s

--cache-read-retries

How many times to retry a read from a cache storage.

Since reading from a cache stream is independent from downloading file data, readers can get to a point where there's no more data in the cache. Most of the times this can indicate a connectivity issue if cache isn't able to provide file data anymore.

For really slow connections, increase this to a point where the stream is able to provide data but your experience will be very stuttering.

Properties:

Config: read_retries
Env Var: RCLONE_CACHE_READ_RETRIES
Type: int
Default: 10

--cache-workers

How many workers should run in parallel to download chunks.

Higher values will mean more parallel processing (better CPU needed) and more concurrent requests on the cloud provider. This impacts several aspects like the cloud provider API limits, more stress on the hardware that rclone runs on but it also means that streams will be more fluid and data will be available much more faster to readers.

Note: If the optional Plex integration is enabled then this setting will adapt to the type of reading performed and the value specified here will be used as a maximum number of workers to use.

Properties:

Config: workers
Env Var: RCLONE_CACHE_WORKERS
Type: int
Default: 4

--cache-chunk-no-memory

Disable the in-memory cache for storing chunks during streaming.

By default, cache will keep file data during streaming in RAM as well to provide it to readers as fast as possible.

This transient data is evicted as soon as it is read and the number of chunks stored doesn't exceed the number of workers. However, depending on other settings like "cache-chunk-size" and "cache-workers" this footprint can increase if there are parallel streams too (multiple files being read at the same time).

If the hardware permits it, use this feature to provide an overall better performance during streaming but it can also be disabled if RAM is not available on the local machine.

Properties:

Config: chunk_no_memory
Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
Type: bool
Default: false

--cache-rps

Limits the number of requests per second to the source FS (-1 to disable).

This setting places a hard limit on the number of requests per second that cache will be doing to the cloud provider remote and try to respect that value by setting waits between reads.

If you find that you're getting banned or limited on the cloud provider through cache and know that a smaller number of requests per second will allow you to work with it then you can use this setting for that.

A good balance of all the other settings should make this setting useless but it is available to set for more special cases.

NOTE: This will limit the number of requests during streams but other API calls to the cloud provider like directory listings will still pass.

Properties:

Config: rps
Env Var: RCLONE_CACHE_RPS
Type: int
Default: -1

--cache-writes

Cache file data on writes through the FS.

If you need to read files immediately after you upload them through cache you can enable this flag to have their data stored in the cache store at the same time during upload.

Properties:

Config: writes
Env Var: RCLONE_CACHE_WRITES
Type: bool
Default: false

--cache-tmp-upload-path

Directory to keep temporary files until they are uploaded.

This is the path where cache will use as a temporary storage for new files that need to be uploaded to the cloud provider.

Specifying a value will enable this feature. Without it, it is completely disabled and files will be uploaded directly to the cloud provider

Properties:

Config: tmp_upload_path
Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
Type: string
Required: false

--cache-tmp-wait-time

How long should files be stored in local cache before being uploaded.

This is the duration that a file must wait in the temporary location cache-tmp-upload-path before it is selected for upload.

Note that only one file is uploaded at a time and it can take longer to start the upload if a queue formed for this purpose.

Properties:

Config: tmp_wait_time
Env Var: RCLONE_CACHE_TMP_WAIT_TIME
Type: Duration
Default: 15s

--cache-db-wait-time

How long to wait for the DB to be available - 0 is unlimited.

Only one process can have the DB open at any one time, so rclone waits for this duration for the DB to become available before it gives an error.

If you set it to 0 then it will wait forever.

Properties:

Config: db_wait_time
Env Var: RCLONE_CACHE_DB_WAIT_TIME
Type: Duration
Default: 1s

--cache-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_CACHE_DESCRIPTION
Type: string
Required: false

Backend commands

Here are the commands specific to the cache backend.

Run them with

rclone backend COMMAND remote:

The help below will explain what arguments each command takes.

See the backend command for more info on how to pass options and arguments.

These can be run on a running backend using the rc command backend/command.

stats

Print stats on the cache backend in JSON format.

rclone backend stats remote: [options] [<arguments>+]

Chunker

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Chunker

The chunker overlay transparently splits large files into smaller chunks during upload to wrapped remote and transparently assembles them back when the file is downloaded. This allows to effectively overcome size limits imposed by storage providers.

Configuration

To use it, first set up the underlying remote following the configuration instructions for that remote. You can also use a local pathname instead of a remote.

First check your chosen remote is working - we'll call it remote:path here. Note that anything inside remote:path will be chunked and anything outside won't. This means that if you are using a bucket-based remote (e.g. S3, B2, swift) then you should probably put the bucket in the remote s3:bucket.

Now configure chunker using rclone config. We will call this one overlay to separate it from the remote itself.

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> overlay
Type of storage to configure.
Choose a number from below, or type in your own value
[snip]
XX / Transparently chunk/split large files
   \ "chunker"
[snip]
Storage> chunker
Remote to chunk/unchunk.
Normally should contain a ':' and a path, e.g. "myremote:path/to/dir",
"myremote:bucket" or maybe "myremote:" (not recommended).
Enter a string value. Press Enter for the default ("").
remote> remote:path
Files larger than chunk size will be split in chunks.
Enter a size with suffix K,M,G,T. Press Enter for the default ("2G").
chunk_size> 100M
Choose how chunker handles hash sums. All modes but "none" require metadata.
Enter a string value. Press Enter for the default ("md5").
Choose a number from below, or type in your own value
 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise
   \ "none"
 2 / MD5 for composite files
   \ "md5"
 3 / SHA1 for composite files
   \ "sha1"
 4 / MD5 for all files
   \ "md5all"
 5 / SHA1 for all files
   \ "sha1all"
 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported
   \ "md5quick"
 7 / Similar to "md5quick" but prefers SHA1 over MD5
   \ "sha1quick"
hash_type> md5
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
--------------------
[overlay]
type = chunker
remote = remote:bucket
chunk_size = 100M
hash_type = md5
--------------------
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

Specifying the remote

In normal use, make sure the remote has a : in. If you specify the remote without a : then rclone will use a local directory of that name. So if you use a remote of /path/to/secret/files then rclone will chunk stuff in that directory. If you use a remote of name then rclone will put files in a directory called name in the current directory.

Chunking

When rclone starts a file upload, chunker checks the file size. If it doesn't exceed the configured chunk size, chunker will just pass the file to the wrapped remote (however, see caveat below). If a file is large, chunker will transparently cut data in pieces with temporary names and stream them one by one, on the fly. Each data chunk will contain the specified number of bytes, except for the last one which may have less data. If file size is unknown in advance (this is called a streaming upload), chunker will internally create a temporary copy, record its size and repeat the above process.

When upload completes, temporary chunk files are finally renamed. This scheme guarantees that operations can be run in parallel and look from outside as atomic. A similar method with hidden temporary chunks is used for other operations (copy/move/rename, etc.). If an operation fails, hidden chunks are normally destroyed, and the target composite file stays intact.

When a composite file download is requested, chunker transparently assembles it by concatenating data chunks in order. As the split is trivial one could even manually concatenate data chunks together to obtain the original content.

When the list rclone command scans a directory on wrapped remote, the potential chunk files are accounted for, grouped and assembled into composite directory entries. Any temporary chunks are hidden.

List and other commands can sometimes come across composite files with missing or invalid chunks, e.g. shadowed by like-named directory or another file. This usually means that wrapped file system has been directly tampered with or damaged. If chunker detects a missing chunk it will by default print warning, skip the whole incomplete group of chunks but proceed with current command. You can set the --chunker-fail-hard flag to have commands abort with error message in such cases.

Caveat: As it is now, chunker will always create a temporary file in the backend and then rename it, even if the file is below the chunk threshold. This will result in unnecessary API calls and can severely restrict throughput when handling transfers primarily composed of small files on some backends (e.g. Box). A workaround to this issue is to use chunker only for files above the chunk threshold via --min-size and then perform a separate call without chunker on the remaining files.

Chunk names

The default chunk name format is *.rclone_chunk.###, hence by default chunk names are BIG_FILE_NAME.rclone_chunk.001, BIG_FILE_NAME.rclone_chunk.002 etc. You can configure another name format using the name_format configuration file option. The format uses asterisk * as a placeholder for the base file name and one or more consecutive hash characters # as a placeholder for sequential chunk number. There must be one and only one asterisk. The number of consecutive hash characters defines the minimum length of a string representing a chunk number. If decimal chunk number has less digits than the number of hashes, it is left-padded by zeros. If the decimal string is longer, it is left intact. By default numbering starts from 1 but there is another option that allows user to start from 0, e.g. for compatibility with legacy software.

For example, if name format is big_*-##.part and original file name is data.txt and numbering starts from 0, then the first chunk will be named big_data.txt-00.part, the 99th chunk will be big_data.txt-98.part and the 302nd chunk will become big_data.txt-301.part.

Note that list assembles composite directory entries only when chunk names match the configured format and treats non-conforming file names as normal non-chunked files.

When using norename transactions, chunk names will additionally have a unique file version suffix. For example, BIG_FILE_NAME.rclone_chunk.001_bp562k.

Metadata

Besides data chunks chunker will by default create metadata object for a composite file. The object is named after the original file. Chunker allows user to disable metadata completely (the none format). Note that metadata is normally not created for files smaller than the configured chunk size. This may change in future rclone releases.

Simple JSON metadata format

This is the default format. It supports hash sums and chunk validation for composite files. Meta objects carry the following fields:

ver - version of format, currently 1
size - total size of composite file
nchunks - number of data chunks in file
md5 - MD5 hashsum of composite file (if present)
sha1 - SHA1 hashsum (if present)
txn - identifies current version of the file

There is no field for composite file name as it's simply equal to the name of meta object on the wrapped remote. Please refer to respective sections for details on hashsums and modified time handling.

No metadata

You can disable meta objects by setting the meta format option to none. In this mode chunker will scan directory for all files that follow configured chunk name format, group them by detecting chunks with the same base name and show group names as virtual composite files. This method is more prone to missing chunk errors (especially missing last chunk) than format with metadata enabled.

Hashsums

Chunker supports hashsums only when a compatible metadata is present. Hence, if you choose metadata format of none, chunker will report hashsum as UNSUPPORTED.

Please note that by default metadata is stored only for composite files. If a file is smaller than configured chunk size, chunker will transparently redirect hash requests to wrapped remote, so support depends on that. You will see the empty string as a hashsum of requested type for small files if the wrapped remote doesn't support it.

Many storage backends support MD5 and SHA1 hash types, so does chunker. With chunker you can choose one or another but not both. MD5 is set by default as the most supported type. Since chunker keeps hashes for composite files and falls back to the wrapped remote hash for non-chunked ones, we advise you to choose the same hash type as supported by wrapped remote so that your file listings look coherent.

If your storage backend does not support MD5 or SHA1 but you need consistent file hashing, configure chunker with md5all or sha1all. These two modes guarantee given hash for all files. If wrapped remote doesn't support it, chunker will then add metadata to all files, even small. However, this can double the amount of small files in storage and incur additional service charges. You can even use chunker to force md5/sha1 support in any other remote at expense of sidecar meta objects by setting e.g. hash_type=sha1all to force hashsums and chunk_size=1P to effectively disable chunking.

Normally, when a file is copied to chunker controlled remote, chunker will ask the file source for compatible file hash and revert to on-the-fly calculation if none is found. This involves some CPU overhead but provides a guarantee that given hashsum is available. Also, chunker will reject a server-side copy or move operation if source and destination hashsum types are different resulting in the extra network bandwidth, too. In some rare cases this may be undesired, so chunker provides two optional choices: sha1quick and md5quick. If the source does not support primary hash type and the quick mode is enabled, chunker will try to fall back to the secondary type. This will save CPU and bandwidth but can result in empty hashsums at destination. Beware of consequences: the sync command will revert (sometimes silently) to time/size comparison if compatible hashsums between source and target are not found.

Modification times

Chunker stores modification times using the wrapped remote so support depends on that. For a small non-chunked file the chunker overlay simply manipulates modification time of the wrapped remote file. For a composite file with metadata chunker will get and set modification time of the metadata object on the wrapped remote. If file is chunked but metadata format is none then chunker will use modification time of the first data chunk.

Migrations

The idiomatic way to migrate to a different chunk size, hash type, transaction style or chunk naming scheme is to:

Collect all your chunked files under a directory and have your chunker remote point to it.
Create another directory (most probably on the same cloud storage) and configure a new remote with desired metadata format, hash type, chunk naming etc.
Now run rclone sync --interactive oldchunks: newchunks: and all your data will be transparently converted in transfer. This may take some time, yet chunker will try server-side copy if possible.
After checking data integrity you may remove configuration section of the old remote.

If rclone gets killed during a long operation on a big composite file, hidden temporary chunks may stay in the directory. They will not be shown by the list command but will eat up your account quota. Please note that the deletefile command deletes only active chunks of a file. As a workaround, you can use remote of the wrapped file system to see them. An easy way to get rid of hidden garbage is to copy littered directory somewhere using the chunker remote and purge the original directory. The copy command will copy only active chunks while the purge will remove everything including garbage.

Caveats and Limitations

Chunker requires wrapped remote to support server-side move (or copy + delete) operations, otherwise it will explicitly refuse to start. This is because it internally renames temporary chunk files to their final names when an operation completes successfully.

Chunker encodes chunk number in file name, so with default name_format setting it adds 17 characters. Also chunker adds 7 characters of temporary suffix during operations. Many file systems limit base file name without path by 255 characters. Using rclone's crypt remote as a base file system limits file name by 143 characters. Thus, maximum name length is 231 for most files and 119 for chunker-over-crypt. A user in need can change name format to e.g. *.rcc## and save 10 characters (provided at most 99 chunks per file).

Note that a move implemented using the copy-and-delete method may incur double charging with some cloud storage providers.

Chunker will not automatically rename existing chunks when you run rclone config on a live remote and change the chunk name format. Beware that in result of this some files which have been treated as chunks before the change can pop up in directory listings as normal files and vice versa. The same warning holds for the chunk size. If you desperately need to change critical chunking settings, you should run data migration as described above.

If wrapped remote is case insensitive, the chunker overlay will inherit that property (so you can't have a file called "Hello.doc" and "hello.doc" in the same directory).

Chunker included in rclone releases up to v1.54 can sometimes fail to detect metadata produced by recent versions of rclone. We recommend users to keep rclone up-to-date to avoid data corruption.

Changing transactions is dangerous and requires explicit migration.

Standard options

Here are the Standard options specific to chunker (Transparently chunk/split large files).

--chunker-remote

Remote to chunk/unchunk.

Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended).

Properties:

Config: remote
Env Var: RCLONE_CHUNKER_REMOTE
Type: string
Required: true

--chunker-chunk-size

Files larger than chunk size will be split in chunks.

Properties:

Config: chunk_size
Env Var: RCLONE_CHUNKER_CHUNK_SIZE
Type: SizeSuffix
Default: 2Gi

--chunker-hash-type

Choose how chunker handles hash sums.

All modes but "none" require metadata.

Properties:

Config: hash_type
Env Var: RCLONE_CHUNKER_HASH_TYPE
Type: string
Default: "md5"
Examples:
- "none"
  - Pass any hash supported by wrapped remote for non-chunked files.
  - Return nothing otherwise.
- "md5"
  - MD5 for composite files.
- "sha1"
  - SHA1 for composite files.
- "md5all"
  - MD5 for all files.
- "sha1all"
  - SHA1 for all files.
- "md5quick"
  - Copying a file to chunker will request MD5 from the source.
  - Falling back to SHA1 if unsupported.
- "sha1quick"
  - Similar to "md5quick" but prefers SHA1 over MD5.

Advanced options

Here are the Advanced options specific to chunker (Transparently chunk/split large files).

--chunker-name-format

String format of chunk file names.

The two placeholders are: base file name (*) and chunk number (#...). There must be one and only one asterisk and one or more consecutive hash characters. If chunk number has less digits than the number of hashes, it is left-padded by zeros. If there are more digits in the number, they are left as is. Possible chunk files are ignored if their name does not match given format.

Properties:

Config: name_format
Env Var: RCLONE_CHUNKER_NAME_FORMAT
Type: string
Default: "*.rclone_chunk.###"

--chunker-start-from

Minimum valid chunk number. Usually 0 or 1.

By default chunk numbers start from 1.

Properties:

Config: start_from
Env Var: RCLONE_CHUNKER_START_FROM
Type: int
Default: 1

--chunker-meta-format

Format of the metadata object or "none".

By default "simplejson". Metadata is a small JSON file named after the composite file.

Properties:

Config: meta_format
Env Var: RCLONE_CHUNKER_META_FORMAT
Type: string
Default: "simplejson"
Examples:
- "none"
  - Do not use metadata files at all.
  - Requires hash type "none".
- "simplejson"
  - Simple JSON supports hash sums and chunk validation.
  - It has the following fields: ver, size, nchunks, md5, sha1.

--chunker-fail-hard

Choose how chunker should handle files with missing or invalid chunks.

Properties:

Config: fail_hard
Env Var: RCLONE_CHUNKER_FAIL_HARD
Type: bool
Default: false
Examples:
- "true"
  - Report errors and abort current command.
- "false"
  - Warn user, skip incomplete file and proceed.

--chunker-transactions

Choose how chunker should handle temporary files during transactions.

Properties:

Config: transactions
Env Var: RCLONE_CHUNKER_TRANSACTIONS
Type: string
Default: "rename"
Examples:
- "rename"
  - Rename temporary files after a successful transaction.
- "norename"
  - Leave temporary file names and write transaction ID to metadata file.
  - Metadata is required for no rename transactions (meta format cannot be "none").
  - If you are using norename transactions you should be careful not to downgrade Rclone
  - as older versions of Rclone don't support this transaction style and will misinterpret
  - files manipulated by norename transactions.
  - This method is EXPERIMENTAL, don't use on production systems.
- "auto"
  - Rename or norename will be used depending on capabilities of the backend.
  - If meta format is set to "none", rename transactions will always be used.
  - This method is EXPERIMENTAL, don't use on production systems.

--chunker-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_CHUNKER_DESCRIPTION
Type: string
Required: false

Citrix ShareFile

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Citrix ShareFile

Citrix ShareFile is a secure file sharing and transfer service aimed as business.

Configuration

The initial setup for Citrix ShareFile involves getting a token from Citrix ShareFile which you can in your browser. rclone config walks you through it.

Here is an example of how to make a remote called remote. First run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Type of storage to configure.
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
XX / Citrix Sharefile
   \ "sharefile"
Storage> sharefile
** See help for sharefile backend at: https://rclone.org/sharefile/ **

ID of the root folder

Leave blank to access "Personal Folders".  You can use one of the
standard values here or any folder ID (long hex number ID).
Enter a string value. Press Enter for the default ("").
Choose a number from below, or type in your own value
 1 / Access the Personal Folders. (Default)
   \ ""
 2 / Access the Favorites folder.
   \ "favorites"
 3 / Access all the shared folders.
   \ "allshared"
 4 / Access all the individual connectors.
   \ "connectors"
 5 / Access the home, favorites, and shared folders as well as the connectors.
   \ "top"
root_folder_id> 
Edit advanced config? (y/n)
y) Yes
n) No
y/n> n
Remote config
Use web browser to automatically authenticate rclone with remote?
 * Say Y if the machine running rclone has a web browser you can use
 * Say N if running rclone on a (remote) machine without web browser access
If not sure try Y. If Y failed, try N.
y) Yes
n) No
y/n> y
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX
Log in and authorize rclone for access
Waiting for code...
Got code
Configuration complete.
Options:
- type: sharefile
- endpoint: https://XXX.sharefile.com
- token: {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"}
Keep this "remote" remote?
y) Yes this is OK
e) Edit this remote
d) Delete this remote
y/e/d> y

See the remote setup docs for how to set it up on a machine with no Internet browser available.

Note that rclone runs a webserver on your local machine to collect the token as returned from Citrix ShareFile. This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this it may require you to unblock it temporarily if you are running a host firewall.

Once configured you can then use rclone like this,

List directories in top level of your ShareFile

rclone lsd remote:

List all the files in your ShareFile

rclone ls remote:

To copy a local directory to an ShareFile directory called backup

rclone copy /home/source remote:backup

Paths may be as deep as required, e.g. remote:directory/subdirectory.

Modification times and hashes

ShareFile allows modification times to be set on objects accurate to 1 second. These will be used to detect whether objects need syncing or not.

ShareFile supports MD5 type hashes, so you can use the --checksum flag.

Transfers

For files above 128 MiB rclone will use a chunked transfer. Rclone will upload up to --transfers chunks at the same time (shared among all the multipart uploads). Chunks are buffered in memory and are normally 64 MiB so increasing --transfers will increase memory use.

Restricted filename characters

In addition to the default restricted characters set the following characters are also replaced:

Character	Value	Replacement
\	0x5C	＼
*	0x2A	＊
<	0x3C	＜
>	0x3E	＞
?	0x3F	？
:	0x3A	：
\|	0x7C	｜
"	0x22	＂

File names can also not start or end with the following characters. These only get replaced if they are the first or last character in the name:

Character	Value	Replacement
SP	0x20	␠
.	0x2E	．

Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

Standard options

Here are the Standard options specific to sharefile (Citrix Sharefile).

--sharefile-client-id

OAuth Client Id.

Leave blank normally.

Properties:

Config: client_id
Env Var: RCLONE_SHAREFILE_CLIENT_ID
Type: string
Required: false

--sharefile-client-secret

OAuth Client Secret.

Leave blank normally.

Properties:

Config: client_secret
Env Var: RCLONE_SHAREFILE_CLIENT_SECRET
Type: string
Required: false

--sharefile-root-folder-id

ID of the root folder.

Leave blank to access "Personal Folders". You can use one of the standard values here or any folder ID (long hex number ID).

Properties:

Config: root_folder_id
Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
Type: string
Required: false
Examples:
- ""
  - Access the Personal Folders (default).
- "favorites"
  - Access the Favorites folder.
- "allshared"
  - Access all the shared folders.
- "connectors"
  - Access all the individual connectors.
- "top"
  - Access the home, favorites, and shared folders as well as the connectors.

Advanced options

Here are the Advanced options specific to sharefile (Citrix Sharefile).

--sharefile-token

OAuth Access Token as a JSON blob.

Properties:

Config: token
Env Var: RCLONE_SHAREFILE_TOKEN
Type: string
Required: false

--sharefile-auth-url

Auth server URL.

Leave blank to use the provider defaults.

Properties:

Config: auth_url
Env Var: RCLONE_SHAREFILE_AUTH_URL
Type: string
Required: false

--sharefile-token-url

Token server url.

Leave blank to use the provider defaults.

Properties:

Config: token_url
Env Var: RCLONE_SHAREFILE_TOKEN_URL
Type: string
Required: false

--sharefile-upload-cutoff

Cutoff for switching to multipart upload.

Properties:

Config: upload_cutoff
Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
Type: SizeSuffix
Default: 128Mi

--sharefile-chunk-size

Upload chunk size.

Must a power of 2 >= 256k.

Making this larger will improve performance, but note that each chunk is buffered in memory one per transfer.

Reducing this will reduce memory usage but decrease performance.

Properties:

Config: chunk_size
Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
Type: SizeSuffix
Default: 64Mi

--sharefile-endpoint

Endpoint for API calls.

This is usually auto discovered as part of the oauth process, but can be set manually to something like: https://XXX.sharefile.com

Properties:

Config: endpoint
Env Var: RCLONE_SHAREFILE_ENDPOINT
Type: string
Required: false

--sharefile-encoding

The encoding for the backend.

See the encoding section in the overview for more info.

Properties:

Config: encoding
Env Var: RCLONE_SHAREFILE_ENCODING
Type: Encoding
Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot

--sharefile-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_SHAREFILE_DESCRIPTION
Type: string
Required: false

Limitations

Note that ShareFile is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".

ShareFile only supports filenames up to 256 characters in length.

rclone about is not supported by the Citrix ShareFile backend. Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.

See List of backends that do not support rclone about and rclone about

Combine

Nick Craig-Wood — Mon, 01 Jan 0001 00:00:00 UTC

Combine

The combine backend joins remotes together into a single directory tree.

For example you might have a remote for images on one provider:

$ rclone tree s3:imagesbucket
/
├── image1.jpg
└── image2.jpg

And a remote for files on another:

$ rclone tree drive:important/files
/
├── file1.txt
└── file2.txt

The combine backend can join these together into a synthetic directory structure like this:

$ rclone tree combined:
/
├── files
│   ├── file1.txt
│   └── file2.txt
└── images
    ├── image1.jpg
    └── image2.jpg

You'd do this by specifying an upstreams parameter in the config like this

upstreams = images=s3:imagesbucket files=drive:important/files

During the initial setup with rclone config you will specify the upstreams remotes as a space separated list. The upstream remotes can either be a local paths or other remotes.

Configuration

Here is an example of how to make a combine called remote for the example above. First run:

 rclone config

This will guide you through an interactive setup process:

No remotes found, make a new one?
n) New remote
s) Set configuration password
q) Quit config
n/s/q> n
name> remote
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
...
XX / Combine several remotes into one
   \ (combine)
...
Storage> combine
Option upstreams.
Upstreams for combining
These should be in the form
    dir=remote:path dir2=remote2:path
Where before the = is specified the root directory and after is the remote to
put there.
Embedded spaces can be added using quotes
    "dir=remote:path with space" "dir2=remote2:path with space"
Enter a fs.SpaceSepList value.
upstreams> images=s3:imagesbucket files=drive:important/files
Configuration complete.
Options:
- type: combine
- upstreams: images=s3:imagesbucket files=drive:important/files
Keep this "remote" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y

Configuring for Google Drive Shared Drives

Rclone has a convenience feature for making a combine backend for all the shared drives you have access to.

Assuming your main (non shared drive) Google drive remote is called drive: you would run

rclone backend -o config drives drive:

This would produce something like this:

[My Drive]
type = alias
remote = drive,team_drive=0ABCDEF-01234567890,root_folder_id=:

[Test Drive]
type = alias
remote = drive,team_drive=0ABCDEFabcdefghijkl,root_folder_id=:

[AllDrives]
type = combine
upstreams = "My Drive=My Drive:" "Test Drive=Test Drive:"

If you then add that config to your config file (find it with rclone config file) then you can access all the shared drives in one place with the AllDrives: remote.

See the Google Drive docs for full info.

Standard options

Here are the Standard options specific to combine (Combine several remotes into one).

--combine-upstreams

Upstreams for combining

These should be in the form

dir=remote:path dir2=remote2:path

Where before the = is specified the root directory and after is the remote to put there.

Embedded spaces can be added using quotes

"dir=remote:path with space" "dir2=remote2:path with space"

Properties:

Config: upstreams
Env Var: RCLONE_COMBINE_UPSTREAMS
Type: SpaceSepList
Default:

Advanced options

Here are the Advanced options specific to combine (Combine several remotes into one).

--combine-description

Description of the remote.

Properties:

Config: description
Env Var: RCLONE_COMBINE_DESCRIPTION
Type: string
Required: false

Metadata

Any metadata supported by the underlying remote is read and written.

See the metadata docs for more info.

Rclone on rclone - rsync for cloud storage

Sia

Sia

Introduction

Configuration

Standard options

--sia-api-url

--sia-api-password

Advanced options

--sia-user-agent

--sia-encoding

--sia-description

Limitations

1Fichier

1Fichier

Configuration

Modification times and hashes

Duplicated files

Restricted filename characters

Standard options

--fichier-api-key

Advanced options

--fichier-shared-folder

--fichier-file-password

--fichier-folder-password

--fichier-cdn

--fichier-encoding

--fichier-description

Limitations

Akamai Netstorage

Akamai NetStorage

Configuration

Example operations

See contents of a directory in your project

Sync the contents local with remote

Upload local content to remote

Delete content on remote

Move or copy content between CP codes.

Features

Symlink Support

Implicit vs. Explicit Directories

--fast-list / ListR support

Purge

Standard options

--netstorage-host

--netstorage-account

--netstorage-secret

Advanced options

--netstorage-protocol

--netstorage-description

Backend commands

du

symlink

Alias

Alias

Configuration

Standard options

--alias-remote

Advanced options

--alias-description

Amazon Drive

Amazon Drive

Amazon S3

Amazon S3 Storage Providers

Configuration

Modification times and hashes

Modification times

Hashes

Reducing costs

Avoiding HEAD requests to read the modification time

Avoiding GET requests to read directory listings

Avoiding HEAD requests after PUT

Increasing performance

Using server-side copy

Increasing the rate of API requests

Versions

Versions naming caveat

Cleanup

Restricted filename characters

Multipart uploads

`--fast-list` / ListR support