Object Storage using the Python SDK

Note

See Python SDK for how to set up using the Python SDK.

Containers

Containers for an account can be interacted with using a Connection instance, conn, and the Object Store service. Some examples are given below.

Note

Containers can typically be passed into functions using either a Container instance or the container name as a string.

Listing containers for an account:

for cont in conn.object_store.containers():
    print(cont)

Creating a new container:

cont = conn.object_store.create_container("CONTAINER_1")

Deleting a container using its name:

conn.object_store.delete_container("CONTAINER_1")

Note

The container must be empty before deletion. This command will not raise an error if the container specified is not found unless ignore_missing is set to False.

Getting metadata for a container by passing a Container instance:

cont = conn.object_store.get_container_metadata(cont)
print(cont)

Setting system metadata for a container, such as the read and write access, as well as an example custom property, by passing a Container instance:

conn.object_store.set_container_metadata(cont, write_ACL="example_project:example_user", read_ACL=".r:*,.rlistings", example_key="example_value")

Note

System metadata is set using system keys: content_type, is_content_type_detected, versions_location, read_ACL, write_ACL, sync_to, sync_key.

Deleting metadata for a container by passing a Container instance:

conn.object_store.delete_container_metadata(cont, ["example_key", "write_ACL", "read_ACL"])

Note

Alternatively, set_container_metadata can be used with values set to "" to delete custom and system metadata.

Note

When setting and deleting custom metadata, capitalisation is ignored, and when deleting custom metadata, ‘-’ and ‘_’ are treated equivalently. This is not the case for system metadata.

Objects

Objects in a container can be interacted with using a Connection instance, conn, and the Object Store service. Some examples are given below.

Note

Similarly to interacting with containers, objects can typically be specified using either an Object instance or the object and container names as strings.

Listing objects in a container by passing the container name:

objs = conn.object_store.objects("CONTAINER_1")
for obj in objs:
    print(obj)

In the example above, objs is a generator object. Specific Object instances can be obtained from this in a number of ways, such as list comprehension:

obj_1 = [obj for obj in objs if obj.name=="FILE_1.txt"][0]

Objects can also be accessed directly using the container name and file name to return an Object instance:

obj_2 = conn.object_store.get_object("FILE_1.txt", "CONTAINER_1")

Equivalently:

obj_2 = conn.object_store.get_object_metadata("FILE_1.txt", "CONTAINER_1")

Note

The Object instance returned by the two examples above (obj_2) differs slightly to that obtained using conn.object_store.objects() (obj_1).

For example, the file name can be obtained via the name or id attributes of obj_1, but only the id attribute of obj_2. However, obj_2 includes metadata not included in obj_1, such as accept-ranges and x-timestamp.

Specific objects can also be accessed via a Connection instance by passing the container name and file name. This will return a tuple, containing (headers, body) for the object specified:

obj_tuple = conn.get_object('CONTAINER_1', 'FILE_1.txt')

Similarly, using a Connection instance, container name and file name, a Response object can be returned, which stores the object headers and content as attributes:

response = conn.get_object_raw('CONTAINER_1', 'FILE_1.txt')

Getting metadata for a container using an Object instance (in the form of either obj_1 or obj_2):

obj = conn.object_store.get_object_metadata(obj)
print(obj)

Note

If an object in the form of obj_1 is passed to get_object_metadata, the object returned will include all the attributes of both obj_1 and obj_2.

Downloading an object’s contents using an Object instance (in the form of either obj_1 or obj_2):

file_1 = conn.object_store.download_object(obj)

Alternatively, downloading contents using the Response object from above:

file_1 = response.content

In the two examples above, file_1 will store the file contents as a bytes object. This can be written out in a number of ways, such as:

with open("SAVED_FILE_1.txt", "wb") as binary_file:
    binary_file.write(file_1)

Saving contents directly, without storing an intermediate Object or Response object:

conn.get_object('CONTAINER_1', 'FILE_1.txt', outfile="SAVED_FILE_1.txt")

Uploading a new object:

new_obj = conn.object_store.upload_object(container="CONTAINER_1",
                                        name="FILE_1.txt",
                                        data="Hello, world!")

Deleting an object using the container and file names:

conn.object_store.delete_object("FILE_1.txt", container="CONTAINER_1")

Note

An error will not be raised if the object specified is not found unless ignore_missing is set to False.

Setting system and custom metadata for an object by passing an Object instance:

conn.object_store.set_object_metadata(obj, delete_after="3000", example_key="example_value")

Note

System metadata is set using system keys: content_type, content_encoding, content_disposition, delete_after, delete_at, is_content_type_detected.

delete_at is also set automatically by setting delete_after.

Deleting custom metadata for an object using the file and container names:

conn.object_store.delete_object_metadata("FILE_1.txt", "CONTAINER_4", ["example-key"])

Note

Alternatively, set_object_metadata can be used with values set to "" to delete custom metadata. However, neither option will delete system metadata.

Note

When deleting custom metadata, the key should be in lower case, and underscores, ‘_’, in the original key name should be replaced with dashes, ‘-‘.

swiftclient

An alternative to openstacksdk is swiftclient, which comprises a command line tool (see Swift CLI) and two separate APIs, SwiftService and Connection, for accessing swift programmatically.

This can be installed using:

pip install python-swiftclient

SwiftService

Below are two examples to illustrate the use of the swiftclient.SwiftService API.

Note

There are several authentication mechanisms available, including setting environment variables that are automatically passed to your SwiftService instance. See Using OpenStack Command-line Interface for how to set these up.

Listing containers for your account:

import logging
from swiftclient.service import SwiftService, SwiftError

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

with SwiftService() as swift:
    try:
        list_parts_gen = swift.list()
        for page in list_parts_gen:
            if page["success"]:
                for item in page["listing"]:
                    i_name = item["name"]
                    i_size = int(item["bytes"])
                    i_count = int(item["count"])
                    print(f"{i_name} [size: {i_size}] [count: {i_count}]")
    except SwiftError as e:
        logger.error(e.value)

Listing and downloading all text files in a container:

import logging
from swiftclient.service import SwiftService, SwiftError

logging.basicConfig(level=logging.ERROR)
logging.getLogger("requests").setLevel(logging.CRITICAL)
logging.getLogger("swiftclient").setLevel(logging.CRITICAL)
logger = logging.getLogger(__name__)

def is_txt(obj):
    return (
        obj["name"].lower().endswith('.txt') or
        obj["content_type"] == 'text/plain'
    )

container = "CONTAINER_1"

with SwiftService() as swift:
    try:
        list_options = {"prefix": "archive_2016-01-01/"}
        list_parts_gen = swift.list(container=container)
        for page in list_parts_gen:
            if page["success"]:
                objects = [
                    obj["name"] for obj in page["listing"] if is_txt(obj)
                ]
                for down_res in swift.download(
                        container=container,
                        objects=objects):
                    if down_res['success']:
                        print(f"'{down_res['object']}' downloaded")
                    else:
                        print(f"'{down_res['object']}' download failed")
            else:
                raise page["error"]
    except SwiftError as e:
        logger.error(e.value)

Warning

The content_type object key may not exist for objects, such as placeholders, which can lead to errors if the above code is run.

Connection

Below are two examples to illustrate the use of the swiftclient.Connection API.

Note

There are a number of kwarg combinations that can be used when creating a Connection instance for authentication.

Listing the response headers and containers for your account:

from swiftclient.client import Connection
from keystoneauth1 import session
from keystoneauth1.identity import v3

# Create a password auth plugin
auth = v3.Password(auth_url='https://openstack.stfc.ac.uk:5000/v3',
                    username='example_user',
                    password='example_password',
                    project_id='example_id',
                    user_domain_name="example_domain_name")

# Create session
keystone_session = session.Session(auth=auth)

# Create swiftclient Connection
conn = Connection(session=keystone_session)

resp_headers, containers = conn.get_account()
print(f"Response headers: {resp_headers}")
for container in containers:
    c_name = container["name"]
    c_count = int(container["count"])
    c_size = int(container["bytes"])
    print(f"{c_name} [size: {c_size}] [count: {c_count}]")

Deleting an object:

from swiftclient.client import Connection, ClientException
from keystoneauth1 import session
from keystoneauth1.identity import v3

# Create a password auth plugin
auth = v3.Password(auth_url='https://openstack.stfc.ac.uk:5000/v3',
                    username='example_user',
                    password='example_password',
                    project_id='example_id',
                    user_domain_name="example_domain_name")

# Create session
keystone_session = session.Session(auth=auth)

# Create swiftclient Connection
conn = Connection(session=keystone_session)

obj = 'FILE_1.txt'
container = 'CONTAINER_1'
try:
    conn.delete_object(container, obj)
    print("Successfully deleted the object")
except ClientException as e:
    print(f"Failed to delete the object with error: {e}")

References

https://docs.openstack.org/openstacksdk/train/

https://docs.openstack.org/openstacksdk/train/user/resources/object_store/v1/container.html

https://docs.openstack.org/openstacksdk/train/user/resources/object_store/v1/obj.html

https://docs.openstack.org/openstacksdk/train/user/proxies/object_store.html

https://docs.openstack.org/openstacksdk/train/user/connection.html

https://docs.openstack.org/openstacksdk/train/user/guides/object_store.html

https://docs.openstack.org/python-swiftclient/train/introduction.html