Resolving UnauthorizedOperation with AWS EBS CSI Driver When Deleting Volumes

Discover how to fix the "UnauthorizedOperation" error with AWS EBS CSI driver volume deletion in your EKS cluster by modifying IAM policies and tags.
---
This video is based on the question https://stackoverflow.com/q/73576780/ asked by the user 'DmitrySemenov' ( https://stackoverflow.com/u/1233751/ ) and on the answer https://stackoverflow.com/a/73793515/ provided by the user 'DmitrySemenov' ( https://stackoverflow.com/u/1233751/ ) at 'Stack Overflow' website. Thanks to these great users and Stackexchange community for their contributions.

Visit these links for original content and any more details, such as alternate solutions, latest updates/developments on topic, comments, revision history etc. For example, the original title of the Question was: AWS EBS CSI Driver: Could not delete volume ID "vol-XXX": DeleteDisk could not delete volume: UnauthorizedOperation

Also, Content (except music) licensed under CC BY-SA https://meta.stackexchange.com/help/licensing
The original Question post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/by-sa/4.0/ ) license, and the original Answer post is licensed under the 'CC BY-SA 4.0' ( https://creativecommons.org/licenses/by-sa/4.0/ ) license.

If anything seems off to you, please feel free to write me at vlogize [AT] gmail [DOT] com.
---
Introduction

If you’re running an AWS EBS CSI driver within an EKS (Elastic Kubernetes Service) cluster and encounter the error message Could not delete volume ID "vol-XXX": DeleteDisk could not delete volume: UnauthorizedOperation, you’re not alone. This issue occurs when deleting Persistent Volumes (PVs) after their respective Persistent Volume Claims (PVCs) are deleted. The error signifies a permissions issue that prevents the EBS CSI driver from automatically managing volume deletions.

Understanding the Issue

Common Symptoms

Persistent Volumes not being deleted after the associated PVC is deleted.

Frequent error messages in the logs of ebs-csi-controller, indicating permissions issues.

Error Breakdown

From the error logs, you might encounter messages similar to:

[[See Video to Reveal this Text or Code Snippet]]

This error indicates that the IAM policy assigned to your EBS CSI driver lacks the necessary permissions to delete the volume.

The Solution

Modifying IAM Policy

To resolve this issue, you need to ensure that the IAM role used by your AWS EC2 instances has the appropriate permissions to perform the delete operations. The key steps are outlined below:

Add Required Tags:
To trigger the conditions set in the IAM policy for EBS operations, you need to add specific tags to your EBS volumes. The required tags are:

CSIVolumeName = UNIQUE_NAME (where UNIQUE_NAME corresponds to the PVC name)

ebs.csi.aws.com/cluster = true

Automate Tagging:
You can run a bash script to add these tags for all your volumes dynamically:

[[See Video to Reveal this Text or Code Snippet]]

Ensure IAM Role Policies:
Make sure that the IAM role associated with the EC2 instance running your ebs-csi-controller pod has these permissions included in its policy:

ec2:DeleteVolume with conditions for the required tags.

Check the IAM policy attached to your nodes. It typically looks like this:

[[See Video to Reveal this Text or Code Snippet]]

Verifying Roles

To ensure that the correct IAM role is attached to your worker nodes:

Identify the node running the ebs-csi-controller:

[[See Video to Reveal this Text or Code Snippet]]

Check the instance profile for the node:

[[See Video to Reveal this Text or Code Snippet]]

List the roles associated with that instance profile:

[[See Video to Reveal this Text or Code Snippet]]

Conclusion

When you delete a PVC linked to an EBS volume and encounter an UnauthorizedOperation error, it is often due to insufficient IAM permissions. By ensuring that you properly tag volumes and configure IAM policies, you can resolve this issue efficiently. This ensures the Kubernetes EBS CSI driver can manage your volumes without facing authorization hurdles.

By following these steps, you will enhance the automatically managed lifecycle of AWS EBS volumes in your EKS cluster, streamlining your cloud operations.

Summary

Add necessary tags to EBS volumes.

Verify IAM policy permissions for volume deletion.

Utilize automation scripts to manage tagging dynamically.

Now you can have your EBS volumes cleanly detached and deleted without running into permission-related obstacles!

Видео Resolving UnauthorizedOperation with AWS EBS CSI Driver When Deleting Volumes канала vlogize