Day 5: Serverless File Upload with AWS S3

Overview:
Handling file uploads is a common requirement for many web applications. However, managing file storage and scalability can be tricky. In this blog, we’ll build a serverless file upload system using AWS S3 and AWS Lambda. This approach allows you to store files in the cloud without having to manage servers, with the added benefit of scaling automatically.

We will create a simple API that uploads files to an S3 bucket using a pre-signed URL generated by AWS Lambda. This will allow users to upload files directly to S3 without sending the file data through the backend, improving performance and security.

Prerequisites:

• AWS Account.
• Installed AWS CLI (instructions here).
• Basic knowledge of AWS Lambda, S3, and Node.js.

Step 1: Set Up an S3 Bucket

First, we need to create an S3 bucket to store the uploaded files.

1. Go to the AWS S3 Console:
   Open the Amazon S3 Console.
2. Create a Bucket:

• Click Create bucket.
• Enter a unique Bucket name (e.g., my-file-upload-bucket).
• Choose the appropriate Region.
• Disable Block all public access (if you want files to be publicly accessible).
• Leave other settings as default and click Create bucket.

Now, you have an S3 bucket ready to store your files.

Step 2: Create a Lambda Function to Generate Pre-signed URLs

AWS Lambda will generate the pre-signed URL that clients will use to upload files directly to S3.

1. Go to the AWS Lambda Console:
   Open the Lambda Console.
2. Create a Lambda Function:

• Click Create Function.
• Choose Author from scratch.
• Set Function name to generateUploadUrl.
• Set Runtime to Node.js 18.x (or any version you prefer).
• Choose Create a new role with basic Lambda permissions. Click Create Function.

1. Add the Lambda Code:
   Scroll down to the Code section and replace the code with the following:

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

exports.handler = async (event) => {
    const bucketName = 'my-file-upload-bucket';  // Replace with your bucket name
    const fileName = event.queryStringParameters.fileName; // Get fileName from the query parameter
    const fileType = event.queryStringParameters.fileType; // Get fileType from the query parameter

    const params = {
        Bucket: bucketName,
        Key: fileName,
        Expires: 60, // Pre-signed URL expiration time (in seconds)
        ContentType: fileType
    };

    try {
        const uploadUrl = await s3.getSignedUrlPromise('putObject', params);
        return {
            statusCode: 200,
            body: JSON.stringify({ uploadUrl })
        };
    } catch (error) {
        console.error(error);
        return {
            statusCode: 500,
            body: JSON.stringify({ error: 'Error generating pre-signed URL' })
        };
    }
};

This Lambda function generates a pre-signed URL using the getSignedUrlPromise method. The pre-signed URL allows clients to upload files directly to S3, bypassing the server.

4. Set Environment Variables (Optional):
   If you want to avoid hardcoding the S3 bucket name, you can set it as an environment variable:

• Go to Configuration -> Environment variables.
• Add a variable BUCKET_NAME with your bucket name as the value.
• Update the code to reference the environment variable:

const bucketName = process.env.BUCKET_NAME;

5. Save and Deploy the Lambda function.

Step 3: Set Up API Gateway

We need to expose the Lambda function through an API, which we’ll accomplish using API Gateway.

1. Go to API Gateway Console:
   Open the API Gateway Console.
2. Create an API:

• Click Create API.
• Choose HTTP API and click Build.

1. Add a Route:

• Click Add integration.
• Choose Lambda function and select your Lambda function (generateUploadUrl).
• Choose the ANY method.
• Set Resource path to /upload-url.
• Click Next.

1. Deploy the API:

• Click Deploy.
• Copy the Invoke URL provided by API Gateway. This is your endpoint for requesting pre-signed URLs.

Step 4: Testing the Pre-signed URL API

Now that the API is set up, let’s test generating a pre-signed URL using a curl request or Postman.

1. Test using Curl:

curl "https://your-api-id.execute-api.region.amazonaws.com/upload-url?fileName=testfile.txt&fileType=text/plain"

This will return a JSON response with the pre-signed URL:

{
  "uploadUrl": "https://my-file-upload-bucket.s3.amazonaws.com/testfile.txt?AWSAccessKeyId=..."
}

2. Upload a File using the Pre-signed URL:

You can now use this URL to upload a file directly to S3. Use curl to upload a file:

curl -X PUT -T "path/to/your/file.txt" "https://my-file-upload-bucket.s3.amazonaws.com/testfile.txt?AWSAccessKeyId=..."

If everything is set up correctly, the file should now be in your S3 bucket!

Step 5: Adding CORS to the S3 Bucket

If you’re building a frontend that interacts with this API, you’ll need to enable Cross-Origin Resource Sharing (CORS) on your S3 bucket to allow uploads from the browser.

1. Go to the S3 Console.
2. Open your bucket, and go to the Permissions tab.
3. Scroll down to the CORS configuration section, and click Edit.
4. Add the following CORS configuration:

[
    {
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["GET", "PUT"],
        "AllowedOrigins": ["*"],
        "ExposeHeaders": []
    }
]

This allows any origin to upload files via PUT requests and retrieve them via GET requests. You can restrict this further depending on your application needs.

Step 6: Integrating with a Frontend (Optional)

Here’s a basic example of how you could integrate the S3 file upload flow in a frontend using JavaScript.

<!DOCTYPE html>
<html>
<body>
    <input type="file" id="fileInput" />
    <button id="uploadButton">Upload</button>

    <script>
        document.getElementById('uploadButton').onclick = async function () {
            const file = document.getElementById('fileInput').files[0];
            const fileName = file.name;
            const fileType = file.type;

            // Request a pre-signed URL from the backend
            const response = await fetch(`https://your-api-id.execute-api.region.amazonaws.com/upload-url?fileName=${fileName}&fileType=${fileType}`);
            const data = await response.json();
            const uploadUrl = data.uploadUrl;

            // Upload the file to S3 using the pre-signed URL
            await fetch(uploadUrl, {
                method: 'PUT',
                headers: {
                    'Content-Type': fileType
                },
                body: file
            });

            alert('File uploaded successfully!');
        };
    </script>
</body>
</html>

This basic HTML page allows users to upload a file directly to S3 via a pre-signed URL.

Step 7: Cleaning Up Resources

Once you’re done testing, it’s a good practice to clean up the resources to avoid unnecessary costs.

1. Delete the S3 bucket (if you no longer need it).
2. Delete the Lambda function:

• Go to the Lambda Console, select your function, and click Delete.

1. Delete the API Gateway:

• Go to API Gateway Console, select your API, and click Delete API.

Conclusion

In this tutorial, we built a serverless file upload system using AWS S3 and AWS Lambda. We created a Lambda function to generate pre-signed URLs, which allowed users to upload files directly to an S3 bucket without passing the file through a backend server. This method is efficient, secure, and scalable.

Feel free to copy the code and configurations to build your own serverless file upload system! You can extend this further by adding file validation, limiting upload size, or connecting the system to a database to store metadata about uploaded files.