API with NestJS #11. Managing private files with Amazon S3

AWS JavaScript NestJS TypeScript

This entry is part 11 of 177 in the API with NestJS

There is quite a bit more to Amazon S3 than storing public files. In this article, we look into how we can manage private files. To do so, we learn how to set up a proper private Amazon S3 bucket and how to upload and access files. We use streams and generate presigned URLs with an expiration time.

You can find the code from this series in this repository.

Setting up Amazon S3

The first thing to do is to create a new bucket.

Amazon Private Bucket

This time, we intend to restrict access to the files we upload. Every time we want our users to be able to access a file, they will need to do it through our API.

Amazon Private Bucket

The IAM user that we’ve created in the previous part of this series has access to all our buckets. Therefore, all we need to do to start using it is to add the name of the bucket to our environment variables.

.env

/src/app.module.ts

Managing files through the API

Once we have the above set up, we can start uploading files to our private bucket. When doing so, we want to save them in a similar way wheKolejny rok n dealing with public files. This time we won’t save the URL of the file, though.

Let’s allow our users to manage some files. To do that, let’s create the entity of a private file. It needs to contain the id of the user.

/src/privateFiles/privateFile.entity.ts

Now we need to add information about the other side of the relationship.

/src/users/user.entity.ts

If you want to know more about defining relationships with Postgres and TypeORM, check out API with NestJS #7. Creating relationships with Postgres and TypeORM

We need to save the key so that we can access or delete our private files. Let’s create a separate service to manage them.

/src/files/privateFiles.service.ts

/src/files/privateFiles.module.ts

Once that’s done, we can use all of the above to upload private files for our users.

/src/users/users.service.ts

/src/users/users.controller.ts

After doing all of the above, our users can start uploading private files.

Postman files upload

You can also implement deleting the files in a very similar way as in the previous part of this series. It might be worth checking if the currently logged in user is an owner of the file, though.

Accessing private files

Since the files we upload above are private, we can’t access them by simply entering a URL. Trying to do so will result in getting an error.

There is more than one way to approach this issue. Let’s start with the most straightforward one.

Fetching the file from Amazon S3 as a stream

The first solution to the above issue is to send the file through our API. The most fitting way to do that is to pipe a readable stream that we can get from the AWS SDK to our response. Thanks to working directly with streams, we don’t have to download the file into the memory in our server.

If you want to know more about piping streams in Node check out Node.js TypeScript #5. Writable streams, pipes, and the process streams

The first thing to do is to get a readable stream of data from our Amazon S3 bucket.

/src/privateFiles/privateFiles.service.ts

Now we need to make sure if the users should be able to download the file.

/src/users/users.service.ts

The interesting thing happens in the controller. Since we are working with streams directly, we need to access the Response object that NestJS uses under the hood.

/src/users/users.controller.ts

Above, we use the FindOneParams DTO for the purpose of validation. If you want to know more, check out API with NestJS #4. Error handling and data validation

Generating presigned URLs

Responding with the data of the file is not the only solution to provide our users with their files. Another way is generating presigned URLs that allow access for a specific expiration time.

We can generate URLs for different actions. To create one for getting a resource, we need to use the   operation name:

/src/files/privateFiles.service.ts

The default expiration time of a presigned URL is 15 minutes. We could change it by adding an   parameter.

Let’s provide the user with an array of all of the uploaded files.

/src/users/users.service.ts

/src/users/users.controller.ts

Now, the user can access all of the files in a very straightforward way.

Postman files list

Summary

In this article, we’ve broadened our knowledge about Amazon S3. This time, we’ve learned how to manage private files. It included fetching them with the use of streams and generating presigned URLs. All of the above knowledge covers a variety of cases. Therefore, it allows us to manage files with Amazon S3 properly.

Series Navigation<< API with NestJS #10. Uploading public files to Amazon S3API with NestJS #12. Introduction to Elasticsearch >>
Subscribe
Notify of
guest
5 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Navninder
2 years ago

Is there a way to change the downloaded file name with this presigned url? I am looking for a solution for the frontend where I can serve the files with their file type. Right now the link only downloads a file without any extension or filename.

Navninder
2 years ago
Reply to  Navninder

I found out that to download files with their file types. You’ll need to define the ContentType and ContentDisposition attributes when uploading to the bucket.

serhii
serhii
2 years ago

You should add region for S3 object init otherwise you might get an error like that
The authorization mechanism you have provided is not supported.

import { S3Client } from “@aws-sdk/client-s3”;
// Set the AWS Region.
const REGION = “REGION”; //e.g. “us-east-1”
// Create an Amazon S3 service client object.
const s3Client = new S3Client({ region: REGION });
export { s3Client };

That happened to me because my aws-region was:
AWS Region
EU (Frankfurt) eu-central-1

read more here:

https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-started-nodejs.html

Last edited 2 years ago by serhii
Ngoc Anh
Ngoc Anh
1 year ago

Anyone have an idea on a secure implementation of video blocking with nestjs, if there is a sample on github that would be better

Nick
Nick
8 months ago

I have a question. If i want to write unit test private-file.service.spec.ts, what should i do ? I tried with below code but it didn’t work