Ruan Bekker's Blog

From a Curious mind to Posts on Github

Using Python Boto3 and DreamHosts DreamObjects to Interact With Their Object Storage Offering

In this post I will demonstrate how to interact with Dreamhost’s Object Storage Service Offering called DreamObjects using Python Boto3 library. Dreamhost offers Object Storage at great pricing, for more information have a look at their Documentation

Whats on the Menu:

We will do the following:

  • List Buckets
  • List Objects
  • Put Object
  • Get Object
  • Upload Object
  • Download Object
  • Delete Object(s)

Configuration

First we need to configure credentials, by providing the access key and access secret key, that is provided by DreamHost:

1
2

$ pip install awscli
$ aws configure --profile dreamhost

After your credentials is set to your profile, we will need to import boto3 and instantiate the s3 client with our profile name, region name and endpoint url:

1
2
3

>>> import boto3
>>> session = boto3.Session(region_name='us-west-2', profile_name='dreamhost')
>>> s3 = session.client('s3', endpoint_url='https://objects-us-west-1.dream.io')

List Buckets:

To list our Buckets:

1
2
3
4
5
6
7
8

>>> response = s3.list_buckets()
>>> print(response)
{u'Owner': {u'DisplayName': 'foobar', u'ID': 'foobar'}, u'Buckets': [{u'CreationDate': datetime.datetime(2017, 4, 15, 21, 51, 3, 921000, tzinfo=tzutc()), u'Name': 'ruanbucket'}], 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': 'tx00000000000000003cd88-005ac361f5-foobar-default', 'HTTPHeaders': {'date': 'Tue, 03 Apr 2018 11:13:57 GMT', 'content-length': '306', 'x-amz-request-id': 'tx00000000000000003cd88-005ac361f5-foobar-default', 'content-type': 'application/xml'}}}

>>> for bucket in response['Buckets']:
...     print(bucket['Name'])
...
ruanbucket

List Objects:

List all the Objects, after the given prefix:

1
2
3
4
5
6
7

>>> response = s3.list_objects(Bucket='ruanbucket', Prefix='logs/sysadmins.co.za/access/')
>>> for obj in response['Contents']:
...     print obj['Key']
...
logs/sysadmins.co.za/access/access.log-2017-10-10.gz
logs/sysadmins.co.za/access/access.log-2017-10-11.gz
logs/sysadmins.co.za/access/access.log-2017-10-12.gz

Put Object:

Write text as the body to the destination key on the Bucket:

1
2
3

>>> response = s3.put_object(Bucket='ruanbucket', Body='My Name is Ruan\n', Key='uploads/docs/file.txt')
>>> print(response)
{u'Body': <botocore.response.StreamingBody object at 0x13cde10>, u'AcceptRanges': 'bytes', u'ContentType': 'binary/octet-stream', 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': 'tx0000000000000000053f2-005ac3e0db-foobar-default', 'HTTPHeaders': {'content-length': '16', 'accept-ranges': 'bytes', 'last-modified': 'Tue, 03 Apr 2018 20:14:54 GMT', 'etag': '"292edceea84d1234465f725c3921fc2a"', 'x-amz-request-id': 'tx0000000000000000053f2-005ac3e0db-foobar-default', 'date': 'Tue, 03 Apr 2018 20:15:23 GMT', 'content-type': 'binary/octet-stream'}}, u'LastModified': datetime.datetime(2018, 4, 3, 20, 14, 54, tzinfo=tzutc()), u'ContentLength': 16, u'ETag': '"292edceea84d1234465f725c3921fc2a"', u'Metadata': {}}

List the Object that we have created in the Bucket::

1
2
3
4
5

>>> response = s3.list_objects(Bucket='ruanbucket', Prefix='uploads/')
>>> for obj in response['Contents']:
...     print obj['Key']
...
uploads/docs/file.txt

Get Object:

Read the value from the key that was uploaded:

1
2
3

>>> response = s3.get_object(Bucket='ruanbucket', Key='uploads/docs/file.txt')
>>> print(response['Body'].read())
My Name is Ruan

Upload Files:

Upload the file from disk to the Bucket:

1
2
3

>>> with open('myfile.txt', 'rb') as data:
...     s3.upload_fileobj(Fileobj=data, Bucket='ruanbucket', Key='uploads/docs/uploadobj.txt')
...

Read the contents from the uploaded file:

1
2
3

>>> response = s3.get_object(Bucket='ruanbucket', Key='uploads/docs/uploadobj.txt')
>>> print(response['Body'].read())
This is some text

Download File:

Download the file from the Bucket to the local disk:

1
2
3

>>> with open('downloaded.txt', 'wb') as data:
...     s3.download_fileobj(Bucket='ruanbucket', Key='uploads/docs/uploadobj.txt', Fileobj=data)
...

Read the file’s content from disk:

1
2

>>> print(open('downloaded.txt').read())
This is some text

Delete Object:

Delete one object:

1
2
3

>>> response = s3.delete_object(Bucket='ruanbucket', Key='uploads/docs/uploadobj.txt')
>>> print(response)
{'ResponseMetadata': {'HTTPStatusCode': 204, 'RetryAttempts': 0, 'HostId': '', 'RequestId': 'tx00000000000000000be5a-005ac3e61a-foobar-default', 'HTTPHeaders': {'date': 'Tue, 03 Apr 2018 20:37:46 GMT', 'x-amz-request-id': 'tx00000000000000000be5a-005ac3e61a-foobar-default'}}}

Delete Objects:

Delete more than one object with a single API call:

1
2
3

>>> response = s3.delete_objects(Bucket='ruanbucket', Delete={'Objects': [{'Key': 'uploads/docs/file.txt'}, {'Key': 'uploads/docs/file2.txt'}, {'Key': 'uploads/docs/file3.txt'}]})
>>> print(response)
{u'Deleted': [{u'Key': 'uploads/docs/file.txt'}, {u'Key': 'uploads/docs/file2.txt'}, {u'Key': 'uploads/docs/file3.txt'}], 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': 'tx000000000000000011008-005ac3e951-foobar-default', 'HTTPHeaders': {'date': 'Tue, 03 Apr 2018 20:51:29 GMT', 'content-length': '270', 'x-amz-request-id': 'tx000000000000000011008-005ac3e951-217c0ac5-default', 'content-type': 'application/xml'}}}

For more information on the above, have a look at Boto’s Documentation and DreamHost’s Website