Zwei Lifecycle-Richtlinien, die jeder S3-Bucket haben sollte
07.03.2024
•
Jonathan Merlevede
Abgebrochene Mehrteil-Uploads und abgelaufene Löschmarker: was sind sie und warum Sie sich wegen der schlechten AWS-Standarde darum kümmern müssen.
There may be items in your buckets that you do not see but that adversely impact S3 costs and performance. This post explains what these invisible objects are and what you can do to remove them, as, for reasons of backward compatibility and possibly also because of perverse incentives, AWS does not remove them for you by default.
TL;DR version: The objects in question are parts of abandoned multipart uploads and expired object deletion markers. I think every bucket should have a lifecycle policy to remove them. If you do not know what these objects are and are interested in knowing, read on.
Aborted, incomplete multipart uploads
What are multipart uploads?
Uploading small objects to AWS S3 is possible using just a singlePutObject operation. For larger objects, we use multipart uploads, and the flow is more involved:
Perform the
CreateMultiPartUploadoperation. You specify an object key; AWS returns you an upload ID.Perform
UploadPartoperations, one for every part. You present AWS with the object key, upload ID, a “part number”, and part of the file you want to upload.Perform the
CompleteMultiPartUploadoperation. You specify the object key and upload ID; AWS then creates your object as the concatenation of all the parts you uploaded.
All of this is usually handled by your upload tool or library. For example, if you use the AWS CLI to upload files (using aws s3 cp), it will use multipart uploads by default for files larger than 8MiB.
What is problematic about multipart uploads?
If you never complete an upload, associated uploaded parts remain in your bucket forever. These parts are stored in your bucket; you pay to keep them while you do not see them.
To illustrate this, let’s create a 5GiB test file and commence uploading it to the cloud using aws s3 cp:
After leaving the upload running for a while, kill the upload abruptly:
You can see that the multipart upload still exists:
You can list associated parts using list-parts :
What can I do about abandoned multipart uploads?
If you know that an upload is abandoned and should be aborted, you can remove uploaded parts by aborting the upload using the AbortMultipartUload operation:
You can verify that the upload ID, and therefore also associated parts, are removed:
What can I really do about abandoned multipart uploads?
The above is highly impractical, as it requires you to monitor your bucket for abandoned uploads and manually abort them. Instead, tell AWS to abort multipart uploads automatically after a certain period expires using an (expiration) object lifecycle rule.
You can do this using the CLI or IaC tools like Terraform, but also from the AWS Console. While you’re there, tick the box “Delete expired object delete markers” too; we will explain what this does in the next section.
Are there any disadvantages to this?
Object expiration through lifecycle policies is free.The only thing to account for is that the number of days you set in the policy limits how long a multipart upload can take. Keep this in mind if you are uploading 5TiB objects from a slow connection. Choose something ridiculously long for a single upload but not so long that it impacts your storage bill, like 14 days.
Expired delete markers
Other invisible objects that tend to linger around in buckets are called “expired object delete markers.” These objects only exist in versioned buckets.
What are object delete markers?
When you enable bucket versioning, every prefix becomes associated with a stack of versioned items. The most recent item is the “latest” or “current” one. There are two types of versioned items: object versions and delete markers. Writing data to a location pushes an object version onto the stack; deleting an object pushes a delete marker. If the current item is an object version, this object version is visible as an object in your bucket; current delete markers remain invisible.
We can easily illustrate this using a couple of aws CLI commands.
Before we upload an object, no version exists:
After uploading an object, we see the object and a single object version:
Deletion of the file adds a deletion marker:
Listing objects using aws s3 ls returns an empty result. Removal of a delete marker would restore the object. You could do this as follows (but let’s not for now):
What are current and noncurrent object delete markers?
If a delete marker is the latest or current item on the version stack, we refer to it as a current object delete marker. Otherwise, we refer to it as a noncurrent object delete marker.In the example above, the single delete marker at the prefix tmp/testfile is a current delete marker. Uploading another object to the same location creates a new object version:
At this point, the delete marker with version ID kNig7WIYhADWCr47u_nRrQ8QYdeW4eIj is still there but has become “noncurrent”, as indicated by its property IsLatest with value false.
What are expired object delete markers?
Expired object delete markers are delete markers at a prefix with no noncurrent object versions.We can turn the delete marker from our example into an expired object delete marker by removing the object versions at the same location:
The delete marker with version IDkNig7WIYhADWCr47u_nRrQ8QYdeW4eIj is now an expired object delete marker.
Why are expired object delete markers bad?
The problem is that expired object delete markers can be current and can remain in your bucket forever unless you do something about them. Lingering markers can slow list requests and result in redundant results when listing object versions.
When enabling bucket versioning, you typically implement an expiration policy for noncurrent items, as not doing so often quickly becomes prohibitively expensive. With such a policy, all delete markers eventually become current, expired ones. Your expiration policy will not remove these from your bucket. In our example, the delete marker at tmp/testfile will never be automatically removed by a policy expiring noncurrent versions.
How can I remove expired object delete markers?
You can remove delete markers manually:
id=kNig7WIYhADWCr47u_nRrQ8QYdeW4eIj
aws s3api delete-object --bucket $bucket --key $key --version-id $id
How can I really remove expired object delete markers?
As for multipart uploads, the best way to remove expired object delete markers is through an explicit (expiration) lifecycle policy targeting expired object delete markers. One way to do this is by ticking a box in the Console (see the screenshot above). (You should probably use an IaC tool, though.)
Are there any disadvantages to this?
In the improbable event that you would like to have an account of which objects existed in the past and when they were removed but do not need the ability to restore said objects, consider not removing expired object delete markers.As for aborted multipart uploads, removing delete markers through lifecycle policies is free. Delete markers only exist in versioned buckets, but having a policy to remove them is never harmful.
Conclusion
We have seen what incomplete multipart uploads are and why you should abort the abandoned ones. We have also seen what expired object delete markers are, that they can be current and that you should remove them.AWS does not abort or remove anything for you by default. Therefore, whenever you create a bucket, create an expiration lifecycle policy that aborts multipart uploads after some days and removes expired object delete markers.
👏 If you liked this article, don’t forget to clap🗣️ Share your insights in the comments; I’ll try to respond🗞️ Follow me and subscribe to datamindedbe for more posts on cloud, platform-, data-, and software engineering👀 For more about Data Minded, visit our website
Latest
Why not to build your own data platform
A round-table discussion summary on imec’s approach to their data platform
Securely use Snowflake from VS Code in the browser
A primary activity among our users involves utilizing dbt within the IDE environment.
The benefits of a data platform team
For years, organizations have been building and using data platforms to get value out of data.