Canonical: https://molo17.com/blog/couchbase-server-integrate-couchbase-to-your-serverless-backend/ Markdown mirror: https://molo17.com/blog/couchbase-server-integrate-couchbase-to-your-serverless-backend/index.md Title: #6 Couchbase Server: integrate Couchbase to your serverless backend Description: Do you need to use Couchbase as a database on your serverless backend? Or do you simply need a Proof-of-Concept? Discover Couchbase Server and AWS Lambda. [← All articles](/blog/) Article # #6 Couchbase Server: integrate Couchbase to your serverless backend 1 Apr 2020 [Knowledge sharing](/blog/?category=knowledge-sharing) 7 min read Do you need to use Couchbase as a database on your serverless backend? Or do you simply need a Proof-of-Concept (POC)? Discover Couchbase Server and how to use AWS \[…\]  **Do you need to use Couchbase as a database on your serverless backend? Or do you simply need a Proof-of-Concept (POC)? Discover Couchbase Server and how to use **AWS Lambda**** **with [MOLO17](\"https://molo17.com/\").** ## **Couchbase Server – Introduction** In this article we will integrate Couchbase in a serverless backend, using [serverless framework](\"https://serverless.com/\") with [Node.js](\"https://nodejs.org/\"), [docker](\"https://www.docker.com/\") and [AWS Lambda Layer](\"https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html\"). As a test project we will create some CRUD REST APIs which allows the user to perform various operations on the contact entity objects. A contact will have the following attributes: - Name – mandatory, max length 64 chars - Surname -mandatory, max length 64 chars - Email – optional, max length 64 chars and must be a valid email address The example project is obviously available on [GitHub](\"https://github.com/MOLO17/couchbase-serverless-backend/\"). ## **Couchbase SDK** For this project we will be using the [2.6.11](\"https://www.npmjs.com/package/couchbase/v/2.6.11\") version of the Node.js SDK.The SDK is not fully javascript written, in fact the core is C written. This is a critical point for our project, AWS Lambda runs on a Linux distribution powered by AWS.  I’m using a MacBook and if I try to use the same Couchbase dependencies used locally, also in Lambda, the last will throw a `/var/task/node_modules/couchbase/build/Release/couchbase_impl.node: invalid ELF header` exception. To avoid this issue we need to install the Couchbase SDK dependency in runtime similar to that supported by Lambdas. Here we can use Docker! No worries, in the [repository](\"https://github.com/MOLO17/couchbase-serverless-backend\") you will find some scripts and utilities needed to minimize your operations. In the following [Dockerfile](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/master/layers/couchbase/Dockerfile\"): FROM ideavate/amazonlinux-node:12 WORKDIR /couchbase COPY package.json ./ COPY package-lock.json ./ VOLUME /couchbase/node\_modules CMD \[ \\"npm\\", \\"install\\" \] we’re using the \\”_amazonlinux_\\” base image with Node.js 12. We’re going to copy the “_package.json_” and the “_package-lock.json_” files, and declare a volume for the “_node\_modules_”. At the container launch we’re going to run the `npm install` command. Once the image has been successfully built, we can launch the `docker run` command. Remember to mount the volume specifying “_nodejs/node\_modules_” as the source folder and “_/couchbase/node\_modules_” as the destination one. In this way in the “_nodejs/node\_modules_” we’ll have all the dependencies of SDK that will be later packed up in a Lambda Layer. Note: you can speed up the process by running the `npm run-script build` and `npm run-script install-couchbase` commands inside the “_layers/couchbase_” project folder. These will respectively build the image and run the container with the right folder mount as volume. ## **Serverless Framework** As introduced before, this project will use the serverless framework, which allows us to package and deploy our lambdas easily and fastly. Once globally installed with `npm install --global serverless`, we can configure the AWS credentials with the `sls config credentials --provider aws --key access_key --secret secret_access_key` command. a_ccess\_key_ and _secret\_access\_key_ are the programmatic credentials. For more information on how to generate credentials, you can use the following tutorial. Tutorial – How to generate AWS credentials ## **Lambda Layer** Now that we have the _“node\_modules_” with the Couchbase SDK ready to use, we can create the Lambda Layer. First of all, what are Lambda Layers? Why do we use them in this project? At the end of 2018, AWS announced this new service as a part of AWS Lambda. It allows to create and deploy packages separated from the Lambdas, so that we can share libraries, custom runtimes or other dependencies between lambdas on multiple accounts and/or globally. In our case it helps to limit and unify the operations needed to install the Couchbase SDK. Using serverless framework, the Lambda Layer definition results really simple and fast. We can just add the following on the [serverless.yml](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/layers/serverless.yaml#L9-L13\"). \[...\] layers: couchbase-node-sdk-2-6-11: path: couchbase compatibleRuntimes: - nodejs12.x \[...\] To refer our layer on multiple [CloudFormation](\"https://aws.amazon.com/it/cloudformation/\") stacks, we can add the following [definition](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/layers/serverless.yaml#L15-L21\"). \[...\] resources: Outputs: CouchbaseNodeSdk2611LayerExport: Value: Ref: CouchbaseDashnodeDashsdkDash2Dash6Dash11LambdaLayer Export: Name: couchbase-node-sdk-2-6-11-layer This allows us to use the `couchbase-node-sdk-2-6-11-layer` variable to obtain the layer resource identifier. Every Lambda Layer will be mount in the “_/opt_” folder of each lambda container. In the previous steps we created the “_nodejs_” folder with “_node\_modules_” inside. This choice wasn’t a random one. In this way our functions can use the Couchbase SDK dependency normally, as it was a part of the deployment package. AWS will automatically extends the node dependency path with “_/opt/nodejs/node\_modules_“. ## **CRUD APIs** Let\\’s move on to defining our CRUD APIs: - **POST /**: to create a new contact - **GET /{contactId}**: to retrieve a contact - **PUT /{contactId}**: to update a contact - **DELETE /{contactId}**: to delete a contact ![\\"Developer](\"https://molo17.com/wp-content/uploads/2020/03/Couchbase-for-the-Web-1024x1024.jpeg\") ### Use of **Lambda Layer** In order to retrieve the layer previously created we can use the import function in our [serverless.yml](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/serverless.yml#L21-L23\") as follows. \[...\] custom: couchbase-sdk-layer: Fn::ImportValue: couchbase-node-sdk-2-6-11-layer \[...\] Now that we have our layer identifier we can add it to a Lambda as follows. \[...\] functions: create: handler: src/create.default layers: - ${self:custom.couchbase-sdk-layer} events: \[...\] ### Contact creation in Couchbase Server Proceeding with the contact creation definition, we need to define the endpoint on the [serverless.yml](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/serverless.yml#L26-L34\") as follows. \[...\] functions: create: handler: src/create.default layers: - ${self:custom.couchbase-sdk-layer} events: - http: method: post path: / cors: true \[...\] Finally we can start to implement the [create](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/master/src/create.js\") function code. In the example function we’re going to: - Connect to the Couchbase cluster - Open the bucket - Validate the payload - Insert the contact document into Couchbase The cluster connection and  bucket opening operations will be done during the function initialization phase and not on the execution phase. Establish a connection with Couchbase involves several TCP/HTTP message exchanges, executing these operations for each request will make them heavy. \[...\] cluster.authenticate(CLUSTER\_USERNAME, CLUSTER\_PASSWORD); const bucket = cluster.openBucket(BUCKET); bucket.on(\\"error\\", error => { console.log(\\"Error from bucket:\\", JSON.stringify(error, null, 2)); }); \[...\] Note that the first line of the function implementation sets the **callbackWaitsForEmptyEventLoop context** property to **false**. \[...\] const createHandler = async (event, context) => { context.callbackWaitsForEmptyEventLoop = false; \[...\] This is needed to return immediately the result instead of waiting that the Node event loop empty. In our case some of the bucket connection will make the function timeout. Payload validations are carried out immediately. If the result is negative, the function will return a **400 Bad Request** status code indicating the invalid field. Once all the validations have been passed, the document to insert will be prepared adding the `Contact` type property. Furthermore, the document id will be calculated concatenating the type with the **`::`** separator and a generated **UUID**. These two operations are a Couchbase modeling best practices. They allow both to perform specific queries by type and to improve the readability of these. \[...\]const document = { id, name, surname, email, type: \\"Contact”}; const documentId = \`contact::${id}\`; \[...\] At the end, to insert the document into Couchbase we are going to use the **insert** method of the SDK as following. const documentInserted = await new Promise((resolve, reject) => { bucket.insert(documentId, document, (error, result) => { if (error) { reject(error); } else { resolve({ ...result, ...document }); } }); }); ### Retrieve a contact in Couchbase Server The contact retrieve function follows the contact creation logics. We will find here: the [endpoint definition](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/serverless.yml#L36-L48\"), the [Couchbase and bucket connection](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/read.js#L30-L36\") and the waiting event loop [disabling](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/read.js#L39\"). To [retrieve the document](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/read.js#L48-L56\") we will be using the **get** SDK method. \[...\]const document = await new Promise((resolve, reject) => { bucket.get(documentId, (error, result) => { if (error) { reject(error); } else { resolve(result.value); } }); });\[...\] If the document doesn’t exists, the sdk will throw a code 13 error, so we can return a **404 Not Found** status code to the client. ### **Update** a contact in Couchbase Server Same process for the contact update function. Here we will find: the [endpoint definition](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/serverless.yml#L50-L62\"), the [Couchbase and bucket connection](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/update.js#L32-L38\") and the waiting event loop [disabling](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/update.js#L41\"). To [update the document](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/update.js#L81-L89\") we will be using the **upsert** SDK method. \[...\]const documentUpdated = await new Promise((resolve, reject) => { bucket.upsert(documentId, document, (error, result) => { if (error) { reject(error); } else { resolve({ ...result, ...document }); } }); });\[...\] If the document doesn’t exists it will be created, the **upsert** method in fact inserts or update a document. ### **Delete** a contact in Couchbase Server Finally, we will also follow the same procedure for the contact deletion function. Here we will find: the [endpoint definition](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/serverless.yml#L36-L48\"), the [Couchbase and bucket connection](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/read.js#L30-L36\") and the waiting event loop [disabling](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/read.js#L39\"). To [delete the document](\"https://github.com/MOLO17/couchbase-serverless-backend/blob/8e8926caffaa48b1ce6b5d1389aac4bbb99e6e34/src/delete.js#L49-L57\") we will be using the **remove** SDK method. \[...\] const document = await new Promise((resolve, reject) => { bucket.remove(documentId, (error, result) => { if (error) { reject(error); } else { resolve({ contactId, ...result }); } }); }); \[...\] As for retrieve, if the document doesn’t exists, the SDK will throw a 13 error, so that we will return a **404 Not Found** status code. ## **Conclusions** In this article we have seen how to integrate Couchbase in a serverless backend, simplifying operations during development usings the [framework serverless](\"https://serverless.com/\") and [AWS Lambda Layer](\"https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html\"). Did you like this article? Read also the other articles of our [Discover Couchbase](\"https://molo17.com/category/knowledge-sharing/discover-couchbase/\") series. [← Older article Smart working & Network Failover during COVID-19](/blog/smart-working-during-covid-19/) [Newer article → How to develop Telemedicine solutions with Couchbase](/blog/how-to-develop-telemedicine-solutions-with-couchbase/) ## Keep reading 1. [Knowledge sharing Accelerating Couchbase NoSQL database adoption in SMBs with MOLO17 MOLO17, company focused on data mobilization thanks to its capabilities to unleash data potential connecting, through open standards and proprietary solutions, mobile users need with legacy and or closed \[…\] 6 Aug 2021](/blog/accelerating-couchbase-nosql-database-adoption-in-smbs-with-molo17/) 2. [Knowledge sharing #4 Discover Couchbase: your first app with Couchbase Lite In the previous articles we had a first look at Couchbase Lite and we discovered how to integrate it into a mobile app. Moreover, we got aware of Couchbase \[…\] 7 Feb 2020](/blog/your-first-app-with-couchbase-lite/) 3. [Knowledge sharing #3 Discover Couchbase: introduction to Couchbase Sync Gateway When talking about mobile apps, making available offline data to the users is a critical point. Enterprise apps are good candidates for such a need. Apps like ones meant \[…\] 28 Nov 2019](/blog/3-discover-couchbase-introduction-to-couchbase-sync-gateway/) [Back to all articles](/blog/) [More in Knowledge sharing →](/blog/?category=knowledge-sharing)