Swift CLI
In order to use all the commands provided from Swift, it is recommended that you have the following python package installed:
pip install python-swiftclient
Note
See Using OpenStack Command-line Interface for how to set up the Openstack command line client. This is the preferred client, but in some cases the Swift client provides coverage that is not yet supported.
To test whether the Swift client have been installed successfully and we can access Swift commands, we can try to list containers in our current project:
swift list
This should return nothing if there are no containers in your project, or a list of containers similar to below:
CONTAINER_1
CONTAINER_2
CONTAINER_3
We can also try to list containers using the Openstack client, to test that this has been installed successfully and that we can access Swift commands in the preferred manner:
openstack container list
This should return an empty line if there are no containers in your project, or a table containing the names of containers similar to the table below:
+-------------+
| Name |
+-------------+
| CONTAINER_1 |
| CONTAINER_2 |
| CONTAINER_3 |
+-------------+
Commands
We are now able to run commands for working with containers and objects. Below is a list of some of the commands that can be used:
# Openstack client commands are of the form:
openstack container <commands>
openstack object <commands>
# Containers
container create <container> # Create a new container
container delete <container> # Delete a container
container list # List containers
container save <container> # Save a container's contents locally
container set --property <key>=<value> <container> # Set a container's properties
container show <container> # Display a container's details
container unset --property <key> <container> # Unset a container's properties
# Objects
object create <container> <object> # Upload an object to a container
object delete <container> <object> # Delete an object from a container
object list <container> # List objects in a container
object save <container> <object> # Save an object locally
object set --property <key>=<value> <container> <object> # Set an object's properties
object show <container> <object> # Display an object's details
object unset --property <key> <container> <object> # Unset an object's properties
# Object store
object store account set --property <key>=<value> # Set an account's properties
object store account show # Display an account's details
object store account unset --property <key> # Unset an account's properties
# Swift client commands are of the form:
swift <commands>
# Containers
delete <container> # Delete a container, including all objects within
download <container> # Save a container's contents locally
list # List containers
post <container> # Update a container's details, or create a container if it does not exist
stat <container> # Display a container's details
# Objects
copy <container> <object> # Updates an object's metadata, or copies an object to a new destination
delete <container> <object> # Delete an object from a container
download <container> <object> # Save an object locally
list <container> # List objects in a container
post <container> <object> # Update an object's details
stat <container> <file> # Display's an object's details
upload <container> <object> # Upload an object to a container
# Other
auth # Display authentication variables
capabilities # Displays cluster capabilities for an object
delete --all # Deletes everything in the account
download --all # Downloads everything in the account
post # Update an account's details
stat # Display information for the account
tempurl <method> <time> <path> <key> # Generates a temporary URL for an object
Warning
Equivalent commands using the Openstack and Swift clients may differ in behaviour. For example, openstack container delete <container> will fail unless the container is empty or -r is used, whereas swift delete <container> will delete the contents of the container, as well as the container itself.
Further details and options can be seen using openstack container <command> --help, openstack object <command> --help and swift <command> --help.
Listing Files
To list objects in CONTAINER_1:
openstack object list CONTAINER_1
# This should return a table similar to:
+---------------------+
| Name |
+---------------------+
| FILE_1.txt |
| FILE_2.txt |
| FOLDER_1/FILE_1.txt |
+---------------------+
Getting Information About Your Account, Containers and Files
To print information about your account:
swift stat
# This should return text similar to:
Account: v1
Containers: 3
Objects: 9
Bytes: 21
Objects in policy "default-placement-bytes": 0
Bytes in policy "default-placement-bytes": 0
Containers in policy "default-placement": 3
Objects in policy "default-placement": 9
Bytes in policy "default-placement": 21
Meta Temp-Url-Key: MYKEY
Meta Key: value
X-Timestamp: 1654612569.67835
X-Account-Bytes-Used-Actual: 20480
X-Trans-Id: tx000000000000008f227e6-00629f6259-218cd2a5a-default
X-Openstack-Request-Id: tx000000000000008f227e6-00629f6259-218cd2a5a-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
To print basic information about CONTAINER_1:
openstack container show CONTAINER_1
# This should return a table similar to:
+--------------+-------------+
| Field | Value |
+--------------+-------------+
| account | v1 |
| bytes_used | 18 |
| container | CONTAINER_1 |
| object_count | 3 |
+--------------+-------------+
To print more detailed information about CONTAINER_1:
swift stat CONTAINER_1
# This should return text similar to:
Account: v1
Container: CONTAINER_1
Objects: 3
Bytes: 18
Read ACL:
Write ACL:
Sync To:
Sync Key:
X-Timestamp: 1652205950.32071
X-Container-Bytes-Used-Actual: 12288
X-Storage-Policy: default-placement
X-Storage-Class: STANDARD
Last-Modified: Fri, 13 May 2022 20:04:16 GMT
X-Trans-Id: tx0000000000000058f1bbc-006283a793-21529d22a-default
X-Openstack-Request-Id: tx0000000000000058f1bbc-006283a793-21529d22a-default
Accept-Ranges: bytes
Content-Type: text/plain; charset=utf-8
To print basic information about FILE_1.txt in CONTAINER_1:
openstack object show CONTAINER_1 FILE_1.txt
# This should return a table similar to:
+----------------+----------------------------------+
| Field | Value |
+----------------+----------------------------------+
| account | v1 |
| container | CONTAINER_1 |
| content-length | 4 |
| content-type | text/plain |
| etag | 0cbc6611f5540bd0809a388dc95a615b |
| last-modified | Tue, 17 May 2022 13:50:29 GMT |
| object | FILE_1.txt |
| properties | Orig-Filename='FILE_1.txt' |
+----------------+----------------------------------+
To print further information about FILE_1.txt in CONTAINER_1:
swift stat CONTAINER_1 FILE_1.txt
# This should return text similar to:
Account: v1
Container: CONTAINER_1
Object: FILE_1.txt
Content Type: text/plain
Content Length: 4
Last Modified: Tue, 17 May 2022 13:50:29 GMT
ETag: 0cbc6611f5540bd0809a388dc95a615b
Meta Orig-Filename: FILE_1.txt
Accept-Ranges: bytes
X-Timestamp: 1652795429.54108
X-Trans-Id: tx000000000000004ef007b-006283a908-218cd2a5a-default
X-Openstack-Request-Id: tx000000000000004ef007b-006283a908-218cd2a5a-default
Creating Containers
Containers can be created using:
openstack container create [-h] [-f {csv,json,table,value,yaml}] [-c COLUMN]
[--quote {all,minimal,none,nonnumeric}] [--noindent] [--max-width <integer>]
[--fit-width] [--print-empty] [--sort-column SORT_COLUMN]
[--sort-ascending | --sort-descending]
<container-name> [<container-name> ...]
Create new container
positional arguments:
<container-name>
New container name(s)
optional arguments:
-h, --help show this help message and exit
output formatters:
output formatter options
-f {csv,json,table,value,yaml}, --format {csv,json,table,value,yaml}
the output format, defaults to table
-c COLUMN, --column COLUMN
specify the column(s) to include, can be repeated to show multiple columns
--sort-column SORT_COLUMN
specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are
ignored), can be repeated
--sort-ascending sort the column(s) in ascending order
--sort-descending sort the column(s) in descending order
For example:
openstack container create CONTAINER_1
# This should return a table similar to:
+---------+-------------+------------------------------------------------------+
| account | container | x-trans-id |
+---------+-------------+------------------------------------------------------+
| v1 | CONTAINER_1 | tx00000000000000384233c-006273aa93-21531bd94-default |
+---------+-------------+------------------------------------------------------+
Note
Containers created using the Openstack client will not be publicly accessible. This can be changed via the GUI, or by updating the container’s metadata.
Uploading Files
Objects can be uploaded into containers using:
openstack object create [-h] [-f {csv,json,table,value,yaml}] [-c COLUMN]
[--quote {all,minimal,none,nonnumeric}] [--noindent] [--max-width <integer>]
[--fit-width] [--print-empty] [--sort-column SORT_COLUMN] [--sort-ascending | --sort-descending]
[--name <name>]
<container> <filename> [<filename> ...]
Upload object to container
positional arguments:
<container> Container for new object
<filename> Local filename(s) to upload
optional arguments:
-h, --help show this help message and exit
--name <name>
Upload a file and rename it. Can only be used when uploading a single object
output formatters:
output formatter options
-f {csv,json,table,value,yaml}, --format {csv,json,table,value,yaml}
the output format, defaults to table
-c COLUMN, --column COLUMN
specify the column(s) to include, can be repeated to show multiple columns
--sort-column SORT_COLUMN
specify the column(s) to sort the data (columns specified first have a priority, non-existing columns are
ignored), can be repeated
--sort-ascending sort the column(s) in ascending order
--sort-descending sort the column(s) in descending order
CSV Formatter:
--quote {all,minimal,none,nonnumeric}
when to include quotes, defaults to nonnumeric
json formatter:
--noindent whether to disable indenting the JSON
table formatter:
--max-width <integer>
Maximum display width, <1 to disable. You can also use the CLIFF_MAX_TERM_WIDTH environment variable, but the
parameter takes precedence.
--fit-width Fit the table to the display width. Implied if --max-width greater than 0. Set the environment variable
CLIFF_FIT_WIDTH=1 to always enable
--print-empty Print empty table if there is no data to show.
Multiple files may be uploaded simultaneously by listing then after the container name:
openstack object create CONTAINER_1 FILE_1.txt FILE_2.txt
# This should return a table similar to:
+------------+-------------+----------------------------------+
| object | container | etag |
+------------+-------------+----------------------------------+
| FILE_1.txt | CONTAINER_1 | ff22941336956098ae9a564289d1bf1b |
| FILE_2.txt | CONTAINER_1 | 9c8c1df0ae41d9a418d596e7ddfefb3b |
+------------+-------------+----------------------------------+
Note
The name of the object uploaded will include its relative local path, unless otherwise specified using the --name option. For example, if ./FOLDER_1/FILE_1.txt is uploaded, it will be named FOLDER_1/FILE_1.txt in the container by default.
Creating Folders
Folders can be created when uploading a file. For example, FOLDER_1 and FOLDER_2 can be created with the following:
openstack object create CONTAINER_1 FOLDER_1/FOLDER_2/FILE_1.txt
# This should return a table similar to:
+------------------------------+-------------+----------------------------------+
| object | container | etag |
+------------------------------+-------------+----------------------------------+
| FOLDER_1/FOLDER_2/FILE_1.txt | CONTAINER_1 | 2205e48de5f93c784733ffcca841d2b5 |
+------------------------------+-------------+----------------------------------+
To create an empty folder in a container, a local empty folder can be uploaded using the Swift client:
swift upload CONTAINER_1 FOLDER_1/
# This should return the name of the created folder:
FOLDER_1/
Note
The trailing / must be included to create a folder in this way.
Saving Containers
The full contents of a container can be saved using the Openstack client. For example:
openstack container save CONTAINER_1
All files will be downloaded to your current directory, with directories implied by object names being created as necessary to recreate the structure.
For example, saving the following container would save FILE_1.txt in ./FOLDER_1, which will be created if it does not exist:
openstack object list CONTAINER_1
+---------------------+
| Name |
+---------------------+
| FOLDER_1/FILE_1.txt |
+---------------------+
Warning
Local files will be overwritten if files with the same name are downloaded.
However, this command will fail if folders exist as unique objects in the container. For example, FOLDER_1/ in the following:
openstack object list CONTAINER_1
+---------------------+
| Name |
+---------------------+
| FOLDER_1/ |
| FOLDER_1/FILE_1.txt |
+---------------------+
Note
This occurs if folders have been created using the Create Folder screen, rather than the Upload File screen with the GUI, or if swift upload <container> <empty folder> has been used.
The choice of folder creation mechanism should not affect the file structure when downloading containers/files or viewing files through the GUI.
In this case, the Swift client must be used to save containers:
swift download CONTAINER_1
By default, this will save all files to the current directory, and, as before, any directories that do not exist will be created.
Deleting Files
Multiple objects can be deleted using:
openstack object delete CONTAINER_1 FILE_1.txt FILE_2.txt
This will return nothing if successful.
Saving Files
Individual files can be saved using the Openstack client. For example:
openstack object save CONTAINER_1 FILE_1.txt
Multiple files can be saved using the Swift client. For example:
swift download CONTAINER_1 FILE_1.txt FILE_2.txt
Saving Folders
The full contents of a folder can be saved by using the Swift client to download each file. For example:
swift download CONTAINER_1 FOLDER_1/FILE_1.txt FOLDER_1/FILE_2.txt
Deleting Folders
If a folder is not a unique object, but exists through file names, it can be deleted by deleting all files within the folder. For example, if FILE_1.txt and FILE_2.txt are the only files in FOLDER_1, the following will delete the folder:
openstack object delete CONTAINER_1 FOLDER_1/FILE_1.txt FOLDER_1/FILE_2.txt
If a folder is stored as a unique object, this can be deleted in the same way as a file:
openstack object delete CONTAINER_1 FOLDER_1/
However, this will not delete any files within the folder. To delete the folder, both the folder object and the folder contents must be deleted:
openstack object delete CONTAINER_1 FOLDER_1/ FOLDER_1/FILE_1.txt FOLDER_1/FILE_2.txt
Deleting Containers
A container can be deleted using:
openstack container delete CONTAINER_1
This will return nothing if successful. An error will be thrown if the container is not empty, unless the -r or --recursive options are used to delete all objects within the container at the same time.
Large Files
Swift does not allow objects larger than 5GiB, so larger files must be segmented. This must be done using the Swift client:
swift upload <container> <object> --segment-size <size>
# <size> is the maximum segment size in Bytes. For example, to upload segments no larger than 1GiB:
swift upload CONTAINER_1 FILE_1.txt --segment-size 1G
Warning
Attempts to upload large files through the GUI or the Openstack client will fail. This may not occur until after an attempt has been made to upload the file, which may take a significant length of time.
This will upload the segments into a separate container, by default named <container>_segments, and create a “manifest” file describing the entire object in <container>.
Note
A Dynamic Large Object is created by default, but if --use-slo is included with segment-size, a Static Large Object will be created instead. This still allows concurrent upload of segments and downloads via a single object, but it does not rely on eventually consistent container listings.
The entire object can be downloaded via the manifest file as if it were any other file, through the GUI or using the Openstack client:
openstack object save CONTAINER_1 FILE_1.txt
To delete the entire object, the Swift client must be used. For example:
swift delete CONTAINER_1 FILE_1.txt
If successful, this will output the manifest file name, as well as the file names of each segment, all of which will have been deleted. The segments container must be deleted separately.
Warning
Attempting to delete a segmented file using openstack object delete will delete the manifest file, but not the segments.
In this case, the folder containing the segments must be deleted manually, as described in the first example of Deleting Folders.
Copying Files
Multiple files can be copied within a container, or between containers, using the Swift client.
For example, copying FILE_1.txt and FILE_2.txt from CONTAINER_1 to CONTAINER_2:
swift copy --destination /CONTAINER_2 CONTAINER_1 FILE_1.txt FILE_2.txt
Warning
This will overwrite any destination files sharing the same name.
If successful, this will create any containers and folders specified that do not exist, and output <file> copied to <destination> for each file.
Note
The output will also include created container <container>, even for containers that already exist.
Editing Container Metadata
Multiple custom properties of a container can be added or overwritten simultaneously through repeated use of the --property option:
openstack container set --property KEY_1=VALUE_1 --property KEY_2=VALUE_2 CONTAINER_1
Note
Underscores (‘_’) in the property key will be converted to dashes (‘-‘).
These properties will be listed when printing information about the container:
openstack container show CONTAINER_1
+--------------+----------------------------------+
| Field | Value |
+--------------+----------------------------------+
| account | v1 |
| bytes_used | 8 |
| container | CONTAINER_1 |
| object_count | 3 |
| properties | Key-1='VALUE_1', Key-2='VALUE_2' |
+--------------+----------------------------------+
Custom properties may also be removed from a container:
openstack container unset --property KEY_1 --property KEY_2 CONTAINER_1
Setting system metadata, such making a container public, can be done through the Swift client using:
swift post <container> --read-acl ".r:*,.rlistings"
Warning
The above command will allow the contents of a container to be viewed by anyone, with no authentication required.
Access can instead be shared with specific groups. For example: <project-id>:<user-id>, <project-id>:*, *:<user-id> *:* or <role_name>.
User IDs, rather than names, should be used, as names are not globally unique.
Similarly, containers can be made private using:
swift post <container> --read-acl ""
Note
Container ACLs are stored in the X-Container-Write and X-Container-Read metadata, but are set by --write-acl and --read-acl respectively.
Write access grants the ability to perform PUT, POST and DELETE operations on objects within a container, but not POST or DELETE operations on the container itself.
Read access grants the ability to perform GET and HEAD operations on objects within a container, but access to privileged metadata such as X-Container-Sync-Key is not granted.
Editing Object Metadata
As for containers, multiple custom properties for object may be set simultaneously:
openstack object set --property KEY_1=VALUE_1 --property KEY_2=VALUE_2 CONTAINER_1 FILE_1.txt
Warning
Any existing object properties that are not listed will be removed, including the default Orig_Filename property.
Note
Underscores (‘_’) in the property key will be converted to dashes (‘-‘).
These properties will be listed when printing information about the object:
openstack object show CONTAINER_1 FILE_1.txt
+----------------+----------------------------------+
| Field | Value |
+----------------+----------------------------------+
| account | v1 |
| container | CONTAINER_1 |
| content-length | 4 |
| content-type | text/plain |
| etag | 0cbc6611f5540bd0809a388dc95a615b |
| last-modified | Tue, 17 May 2022 17:35:29 GMT |
| object | FILE_1.txt |
| properties | Key-1='VALUE_1', Key-2='VALUE_2' |
+----------------+----------------------------------+
Custom properties may also be removed from objects:
openstack object unset --property KEY_1 --property KEY_2 CONTAINER_1 FILE_1.txt
Creating a Temporary URL
Secret keys used in the cryptographic signature for temporary URLs can be created using the Swift client. For example, to set the secret key to MYKEY:
swift post -m "Temp-URL-Key:MYKEY"
Note
Two secret key values per account, and two per container can be stored.
A temporary URL for a Swift object can then be generated using the Swift client, typically to allow GET or PUT access. For example:
swift tempurl GET 1000 https://s3.echo.stfc.ac.uk/swift/v1/CONTAINER_1/FILE_1.txt MYKEY
# This should return a URL similar to:
https://s3.echo.stfc.ac.uk/swift/v1/CONTAINER_1/FILE_1.txt?temp_url_sig=?temp_url_sig=da39a3ee5e6b4b0d3255bfef95601890afd80709&temp_url_expires=1323479485
Note
The URL returned includes an ampersand, so must be enclosed in quotation marks in a command shell.
References
https://docs.openstack.org/python-openstackclient/train/cli/command-objects/container.html?
https://docs.openstack.org/python-openstackclient/train/cli/command-objects/object.html
https://docs.openstack.org/python-openstackclient/train/cli/decoder.html#swift-cli
https://docs.openstack.org/python-swiftclient/train/cli/index.html
https://docs.openstack.org/swift/train/overview_large_objects.html
https://docs.openstack.org/swift/train/overview_acl.html
https://docs.openstack.org/swift/train/api/temporary_url_middleware.html#secret-keys