AWS Bucket and File Permission and workaround

In AWS, Bucket Policy and CORS configuration are mechanisms used to control access to resources stored in Amazon S3 (Simple Storage Service).

Bucket Policy:

A Bucket Policy is a JSON-based access control policy that defines permissions for users and services interacting with an S3 bucket. It specifies who can access the bucket (principals), what actions they can perform (like `GetObject`, `PutObject`), and under what conditions. Policies can grant public access, restrict access to specific IP ranges, or enforce encryption. For example, you can allow public read access to objects in the bucket by permitting the `s3:GetObject` action to all users (`Principal: "*"`). Bucket policies ensure fine-grained control over data access and security.

CORS (Cross-Origin Resource Sharing):

CORS configuration is used when you need to allow web applications hosted on different domains to access your S3 bucket. By default, web browsers block cross-origin requests due to security restrictions. A CORS configuration in S3 allows you to specify the allowed origins, HTTP methods (`GET`, `POST`, etc.), and headers that can be used in cross-origin requests. This is essential for scenarios like loading images, PDFs, or other resources from S3 into web applications hosted on a different domain.

Together, Bucket Policies and CORS ensure secure, controlled, and flexible access to S3 resources.

Let's say that you have configured the below Bucket Polict and CORS configuration and still getting the below error:

Block all public access - Off

Bucket Policy - { "Version": "2012-10-17", "Statement": [ { "Sid": "PublicReadGetObject", "Effect": "Allow", "Principal": "*", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::aws-sam-cli-managed-default-samclisourcebucket-ws8uc0ok5o2c/*" } ] }

Cross-origin resource sharing (CORS) [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "PUT", "POST", "DELETE", "GET" ], "AllowedOrigins": [ "*" ], "ExposeHeaders": [] } ]

And the link - https://aws-sam-cli-managed-default-samclisourcebucket-ws8uc0ok5o2c.s3.us-east-1.amazonaws.com/uploads/172818838784704-valid.pdf

gives the below error.

<Error> <script/> <script/> <Code>InvalidArgument</Code> <Message>Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4.</Message> <ArgumentName>Authorization</ArgumentName> <ArgumentValue>null</ArgumentValue> <RequestId>EZX12YVCPJ5ZY4ZQ</RequestId> <HostId>E6tayS6prVQevLWT/gdlnKIvb7g/9igJv0wD4lbof//GAVn9OScmU52eY4DMKWYDIT0GMycSF5g=</HostId> </Error>

Based on the settings shared, S3 bucket seems to be configured for public access, and have the necessary permissions and CORS configuration in place. However, it still isn't working as expected. Let’s break down the potential issues and provide suggestions to help troubleshoot the problem.

The bucket policy allows public read access (`s3:GetObject`) for all objects under the bucket `"aws-sam-cli-managed-default-samclisourcebucket-ws8uc0ok5o2c"`. This should theoretically allow any public user to access the objects.

However, there could be a few other factors preventing access:

- Object Permissions: Even if the bucket policy allows public access, individual objects within the bucket might still have restrictive permissions. Make sure that the objects themselves don't have any restrictive ACLs (Access Control Lists). You can either explicitly make the objects public or rely on the bucket policy to grant access.

Solution: Check the object permissions and ensure that they are not set to private.

- S3 Block Public Access Settings: Block all public access is off. Double-check if this setting applies to both the bucket and the account level. Sometimes, public access may still be blocked at the account level if this setting is enabled.

Solution: Ensure that public access is fully enabled at both the bucket and account level.

2. CORS Configuration

CORS configuration allows all methods (`GET`, `POST`, `PUT`, etc.) and all origins (`*`). However, there could still be a few issues related to CORS.

- Browser Caching: Sometimes, the browser caches previous responses. If you’ve recently changed the bucket policy or CORS configuration, the browser might still be using an old, cached policy.

Solution: Clear your browser cache or test in incognito mode to see if the issue persists.

- CORS Errors in Console: Check the browser console for CORS-related errors. Sometimes, detailed CORS errors are logged there, which can help identify specific issues.

3. PDF Viewer

`PDFViewer` code uses Google Docs Viewer to load the PDF file via an iframe. There are some limitations to this approach:

- Google Docs Viewer Limitations: Google Docs Viewer may fail to render certain files, particularly large files or files hosted on services that require special authentication (even though the bucket is public, Google might face issues in accessing it).

Solution: Try directly opening the PDF URL in the browser (without embedding it in Google Docs Viewer) to see if the file loads successfully. If it does, then the issue may be related to Google Docs Viewer, and you may need to switch to a different PDF viewer solution (e.g., `react-pdf` or `pdf.js`).

4. Pre-signed URL Misconfiguration

If you were using pre-signed URLs for private access and they weren’t working, it could have been due to:

- KMS Encryption Requirement: The error you saw earlier ("Requests specifying Server Side Encryption with AWS KMS managed keys require AWS Signature Version 4") suggests that KMS encryption was enabled on the bucket or objects. In such cases, a pre-signed URL would be required, as public access wouldn’t work.

Solution: Double-check if you’re using any encryption settings (such as KMS) for the objects. If so, you must use pre-signed URLs.

5. Next.js Server-side Restrictions

Depending on your server-side settings and file structure, application might not be correctly forwarding the PDF URL or headers to allow access to the resource.

Solution: Log the actual URL being generated for the `pdf_url` in the `PDFViewer` component to make sure it's correctly formatted and publicly accessible. Try opening the URL directly in the browser to verify.

To summarize:

- Ensure that object permissions are not overriding the public access allowed by the bucket policy.

- Double-check that public access is allowed at both the bucket and account level.

- Clear browser cache to avoid any cached policy or CORS issues.

- Test the PDF URL directly in a browser to confirm if the issue is related to the viewer.

- If using encryption (KMS), ensure the correct configuration of pre-signed URLs.

If the above doesn't solve the issues, theer could be an issue with Bucket Encryption Settings.

The error we see indicates that the S3 object you are trying to access is encrypted with AWS KMS (Key Management Service) and requires AWS Signature Version 4 for authorization. Essentially, even though bucket policy allows public access, the specific object you are trying to access has server-side encryption enabled using AWS KMS keys. This encryption requires an authenticated request with proper credentials, which public access does not provide.

Steps to Resolve the Issue

1. Disable KMS Encryption for Public Objects:

- If you want the objects to be publicly accessible without the need for a signed URL or authentication, you need to disable KMS encryption for those objects.

Steps:

- Go to your S3 bucket in the AWS Management Console.

- Find the object (in your case, `uploads/172818838784704-valid.pdf`).

- Check the "Properties" tab for the object and see if "Server-side encryption settings" is enabled.

- If it is encrypted with AWS KMS, you will need to either:

- Re-upload the object without server-side encryption.

- Or change the encryption setting to S3-managed encryption (SSE-S3) instead of KMS, which does not require signed requests.

2. Use Pre-signed URLs (if encryption must be maintained):

- If you need to keep the object encrypted using KMS for security reasons, you must generate a pre-signed URL for each request to access the file.

- A pre-signed URL gives time-limited access to the object and is signed with the necessary credentials.

Steps to generate pre-signed URLs:

- You can generate a pre-signed URL programmatically using the AWS SDK (e.g., using `aws-sdk` in Node.js) or the AWS CLI.

Example in AWS CLI:

```bash

aws s3 presign s3://aws-sam-cli-managed-default-samclisourcebucket-ws8uc0ok5o2c/uploads/172818838784704-valid.pdf --expires-in 3600

```

This will generate a pre-signed URL that is valid for 1 hour (`3600` seconds).

3. Check Bucket Encryption Settings:

- It's also worth checking if bucket-level encryption is automatically applying KMS encryption to all objects when uploaded.

Steps:

- In the S3 Console, go to your bucket.

- Under the Properties tab, look for Default encryption.

- If it's set to use AWS KMS, either change it to use S3-managed keys (SSE-S3) for public objects, or ensure that pre-signed URLs are generated for all such objects.

If you're comfortable disabling KMS encryption for the objects in question, that should resolve the public access issue. However, if security requirements mandate KMS encryption, you'll need to work with pre-signed URLs to provide authenticated access.