API with NestJS #10. Uploading public files to Amazon S3

AWS JavaScript NestJS TypeScript

This entry is part 10 of 168 in the API with NestJS

While storing files directly in the database is doable, it might not be the best approach. Files can take a lot of space, and it might impact the performance of the application. Also, it increases the size of the database and, therefore, makes backups bigger and slower. A good alternative is storing files separately using an external provider, such as Google Cloud, Azure, or Amazon AWS.

In this article, we look into uploading files to Amazon Simple Storage Service, also referred to as S3. You can find all the code from this series in this repository.

Connecting to Amazon S3

Amazon S3 provides storage that we can use with any type of file. We organize files into buckets and manage them in our API through an SDK.

Once we create the AWS account, we can log in as a root user. Even though we might authorize as root to use the S3 through our API, this is not the best approach.

Setting up a user

Let’s create a new user that has a restricted set of permissions. To do so, we need to open the Identity and Access Management (IAM) panel and create a user:

Amazon S3 interface

Since we want this user to be able to manage everything connected to S3, let’s set up proper access.

Amazon S3 interface

After doing that, we are presented with Access key ID and Secret access key. We need those to connect to AWS through our API. We also need to choose one of the available regions.

Let’s add them to our   file:

.env

Also, let’s add it to our environment variables validation schema in AppModule:

Connecting to AWS through SDK

Once we have the necessary variables, we can connect to AWS using the official SDK for Node. Let’s install it first.

Since we’ve got everything that we need to configure the SDK, let’s use it. One of the ways to do so is to use   straight in our   file.

main.ts

 Creating our first bucket

In Amazon S3 data is organized in buckets. We can have multiple buckets with different settings.

Let’s open the Amazon S3 panel and create a bucket. Please note that the name of the bucket must be unique.

Amazon S3 interface

We can set up our bucket to contain public files. All files that we upload to this bucket will be publicly available. We might use it to manage files such as avatars.

Amazon S3 interface

The last step here is to add the name of the bucket to our environment variables.

.env

src/app.module.ts

Uploading images through our API

Since we’ve got the AWS connection set up, we can proceed with uploading our files. For starters, let’s create a   entity.

src/files/publicFile.entity.ts

By saving the URL directly in the database, we can access the public file very quickly. The   property uniquely identifies the file in the bucket. We need it to access the file, for example, if we want to delete it.

The next step is creating a service that uploads files to the bucket and saves the data about the file to our Postgres database. Since we want keys to be unique, we use the uuid library:

src/files/files.service.ts

The   method expects a buffer. It is a chunk of memory that keeps a binary representation of our file. If you want to know more about it, check out Node.js TypeScript #3. Explaining the Buffer.

Creating an endpoint for uploading files

Now we need to create an endpoint for the user to upload the avatar. To link the files with users, we need to modify the   by adding the   column.

src/users/user.entity.ts

If you wan to know more about relationships with Postgres and TypeORM, check out API with NestJS #7. Creating relationships with Postgres and TypeORM

Let’s add a method to the   that uploads files and links them to the user.

src/users/users.service.ts

This might be a fitting place to include some functionalities like checking the size of the image or compressing it.

The last piece is adding an endpoint the users can send avatars to. To do that, we follow the NestJS documentation and use the   that utilizes multer under the hood.

src/users/users.controller.ts

The file above has quite a few useful properties such as the mimetype. You can use it if you want some additional validation and disallow certain types of files.

Postman interface

Deleting existing files

Aside from uploading files, we also need a way to remove them. To keep our database consistent with the Amazon S3 storage, we remove the files from both places. First, let’s add the method to the .

src/files/files.service.ts

Now, we need to use it in our . Important addition is that when a user uploads an avatar while already having one, we delete the old one.

src/users/users.service.ts

Summary

In this article, we’ve learned the basics of how Amazon S3 works and how to use it in our API. To do that, we’ve provided the necessary credentials to AWS SDK. Thanks to that, we were able to upload and delete files to AWS. We’ve also kept our database in sync with Amazon S3, to track our files. To upload files through API we’ve used the FileInterceptor, which uses Multer under the hood.

Since there is more to Amazon S3 than handling public files, there is still quite a bit to cover here, and you might expect it in this series.

Series Navigation<< API with NestJS #9. Testing services and controllers with integration testsAPI with NestJS #11. Managing private files with Amazon S3 >>
Subscribe
Notify of
guest
10 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
psz
psz
3 years ago

does this solution upload files directly to s3 from frontend? or multer saves the file on disk first and then it’s transfered to s3?

wsdydeni
wsdydeni
3 years ago
Reply to  psz

upload files directly to s3 from frontend

SpaceCowboy74
SpaceCowboy74
3 years ago

I made a slight adjustment for code reuse.

I refactored deleteAvatar to be a wrapper to deleteUserAvatar. The second function takes a user object. deleteAvatar now looks up a user and passes it to deleteUserAvatar and addAvatar now passes a user object to deleteUserAvatar. It wasn’t much, but was some code duplication in the existing functionality.

Anton
Anton
3 years ago

cant update user have this errors: 
No entity column “posts” was found
It happens because

return user with this properties:
{

posts: undefined
}
what is wrong ?

Cybertech
Cybertech
2 years ago
Reply to  Anton

info is not complete to help u sort, where is id coming from? are u sure it’s not meant to be await this.usersRepository.findOne({ id: userId });

Anton
Anton
3 years ago

I figure out the main reason of it, it is because I had typescript config esnext but in tutorial it is es2019, so how can I avoid this error? should I delete all undefined properties before update ?

David
3 years ago

Don’t forget to specify the suitable access control list (ACL), also the folder if you want to organize the files.

Last edited 3 years ago by David
Bhargav Gohil
Bhargav Gohil
2 years ago

There is an UPDATE in S3. After creating a bucket,

Go to Permissions > Object Ownership > select ACL’s enabled > and select Bucket owner preferred

Otherwise you will get an error for ACL related.

Last edited 2 years ago by Bhargav Gohil
helios
helios
1 year ago

I’m getting this error when trying to run the application:

Dagwal
4 months ago
Reply to  helios

Are you sure with your env secret files