Delete Old Items With Amazons DynamoDB TTL Feature
As you may know a DynamoDB Table’s Partition Splits on 2 factors, Read/Write Capacity Units and when Storage goes over 10GB.
Automatically Deleting Old Data in DynamoDB:
With the TTL Feature in DynamoDB, we can enable TTL on a Attribute on our Table, the attributes value needs to have an epoc time value, more specifically, when the current time is the same as the value of on of the items attribute value, that item will be expired, which will be deleted.
What we will be doing:
Use Boto3 in Python
Create DynamoDB Table: ‘session-table’
Set TTL Attribute on ‘ExpirationTime’, so whenever the epoch time is equals to the AttributeValue it will delete the item
Do one PUT Item with 48 Hours expiry Date from the Write
Do 240 PUT Items with 24 Hours expiry Date from the Write
Verify after 24 hours if only one item is in our table.
Pre-Requisites:
Install the AWS CLI, Boto3 and configure your credentials, so that boto3 can read from your credential provider:
1234567
$ pip install awscli
$ pip install boto3
$ aws configure
AWS Access Key ID [****************XYZ]:
AWS Secret Access Key [****************xyz]:
Default region name [eu-west-1]:
Default output format [json]:
From the Console, enable TTL and set the TTL Attribute on ExpirationTime
Write Data to DynamoDB
We have 2 functions that will write the current epoch time to the CreationTime attribute and ExpirationTime will have the current time plus the 24 hours in seconds, which will be used for the 240 items that will be written using the for loop and the other function with the 48 hours of seconds, which will be a single write item.
Then we will just write random data to the session data attribute:
importboto3importtimeimportrandomfromuuidimportuuid4names=['james','john','steve','peter','frank','steven','jonathan','stephen','will','adam','william']retailer=['shoprite','edgars','pnp','bestbuy','ok','grocer-a','amazon','seveneleven','shop-a']session=boto3.Session(region_name='eu-west-1',profile_name='dev')ddb=session.resource('dynamodb')client=ddb.Table('session-table')defcurrent_time():int(time.time())defcurrent_time():returnint(time.time())defexpiration_time():returnint(time.time())+86400def48h_expiration_time():returnint(time.time())+172800# expiry on 48 hoursclient.put_item(Item={'sessionid':str(uuid4()),'CreationTime':current_time(),'ExpirationTime':48h_expiration_time(),'SessionData':{'Name':random.choice(names),'Retailer':random.choice(retailer),'TimeOfTransaction':current_time(),'Amount':random.randint(100,9000)}})# expiry on 24 hoursforxinxrange(240):time.sleep(1)client.put_item(Item={'sessionid':str(uuid4()),'CreationTime':current_time(),'ExpirationTime':expiration_time(),'SessionData':{'Name':random.choice(names),'Retailer':random.choice(retailer),'TimeOfTransaction':current_time(),'Amount':random.randint(100,9000)}})
Verify:
Verify after 24 hours if the item with the 48 hour expiration time is still in our table: