Do you need to use Couchbase as a database on your serverless backend? Or do you simply need a Proof-of-Concept (POC)? Discover Couchbase Server and how to use AWS Lambda with MOLO17.
Couchbase Server – Introduction
In this article we will integrate Couchbase in a serverless backend, using serverless framework with Node.js, docker and AWS Lambda Layer. As a test project we will create some CRUD REST APIs which allows the user to perform various operations on the contact entity objects.
A contact will have the following attributes:
- Name – mandatory, max length 64 chars
- Surname -mandatory, max length 64 chars
- Email – optional, max length 64 chars and must be a valid email address
The example project is obviously available on GitHub.
Couchbase SDK
For this project we will be using the 2.6.11 version of the Node.js SDK.The SDK is not fully javascript written, in fact the core is C written. This is a critical point for our project, AWS Lambda runs on a Linux distribution powered by AWS.
I’m using a MacBook and if I try to use the same Couchbase dependencies used locally, also in Lambda, the last will throw a /var/task/node_modules/couchbase/build/Release/couchbase_impl.node: invalid ELF header exception.
To avoid this issue we need to install the Couchbase SDK dependency in runtime similar to that supported by Lambdas. Here we can use Docker!
No worries, in the repository you will find some scripts and utilities needed to minimize your operations.
In the following Dockerfile:
FROM ideavate/amazonlinux-node:12 WORKDIR /couchbase COPY package.json ./ COPY package-lock.json ./ VOLUME /couchbase/node_modules CMD [ "npm", "install" ]
we’re using the “amazonlinux” base image with Node.js 12. We’re going to copy the “package.json” and the “package-lock.json” files, and declare a volume for the “node_modules”. At the container launch we’re going to run the npm install command.
Once the image has been successfully built, we can launch the docker run command. Remember to mount the volume specifying “nodejs/node_modules” as the source folder and “/couchbase/node_modules” as the destination one. In this way in the “nodejs/node_modules” we’ll have all the dependencies of SDK that will be later packed up in a Lambda Layer.
Note: you can speed up the process by running the npm run-script build and npm run-script install-couchbase commands inside the “layers/couchbase” project folder. These will respectively build the image and run the container with the right folder mount as volume.
Serverless Framework
As introduced before, this project will use the serverless framework, which allows us to package and deploy our lambdas easily and fastly.
Once globally installed with npm install --global serverless, we can configure the AWS credentials with the sls config credentials --provider aws --key access_key --secret secret_access_key command. access_key and secret_access_key are the programmatic credentials. For more information on how to generate credentials, you can use the following tutorial.
Lambda Layer
Now that we have the “node_modules” with the Couchbase SDK ready to use, we can create the Lambda Layer.
First of all, what are Lambda Layers? Why do we use them in this project?
At the end of 2018, AWS announced this new service as a part of AWS Lambda. It allows to create and deploy packages separated from the Lambdas, so that we can share libraries, custom runtimes or other dependencies between lambdas on multiple accounts and/or globally.
In our case it helps to limit and unify the operations needed to install the Couchbase SDK.
Using serverless framework, the Lambda Layer definition results really simple and fast. We can just add the following on the serverless.yml.
[...]
layers:
couchbase-node-sdk-2-6-11:
path: couchbase
compatibleRuntimes:
- nodejs12.x
[...]To refer our layer on multiple CloudFormation stacks, we can add the following definition.
[...]
resources:
Outputs:
CouchbaseNodeSdk2611LayerExport:
Value:
Ref: CouchbaseDashnodeDashsdkDash2Dash6Dash11LambdaLayer
Export:
Name: couchbase-node-sdk-2-6-11-layerThis allows us to use the couchbase-node-sdk-2-6-11-layer variable to obtain the layer resource identifier.
Every Lambda Layer will be mount in the “/opt” folder of each lambda container.
In the previous steps we created the “nodejs” folder with “node_modules” inside. This choice wasn’t a random one. In this way our functions can use the Couchbase SDK dependency normally, as it was a part of the deployment package. AWS will automatically extends the node dependency path with “/opt/nodejs/node_modules“.
CRUD APIs
Let’s move on to defining our CRUD APIs:
- POST /: to create a new contact
- GET /{contactId}: to retrieve a contact
- PUT /{contactId}: to update a contact
- DELETE /{contactId}: to delete a contact
Use of Lambda Layer
In order to retrieve the layer previously created we can use the import function in our serverless.yml as follows.
[...]
custom:
couchbase-sdk-layer:
Fn::ImportValue: couchbase-node-sdk-2-6-11-layer
[...]Now that we have our layer identifier we can add it to a Lambda as follows.
[...]
functions:
create:
handler: src/create.default
layers:
- ${self:custom.couchbase-sdk-layer}
events:
[...]Contact creation in Couchbase Server
Proceeding with the contact creation definition, we need to define the endpoint on the serverless.yml as follows.
[...]
functions:
create:
handler: src/create.default
layers:
- ${self:custom.couchbase-sdk-layer}
events:
- http:
method: post
path: /
cors: true
[...]Finally we can start to implement the create function code.
In the example function we’re going to:
- Connect to the Couchbase cluster
- Open the bucket
- Validate the payload
- Insert the contact document into Couchbase
The cluster connection and bucket opening operations will be done during the function initialization phase and not on the execution phase. Establish a connection with Couchbase involves several TCP/HTTP message exchanges, executing these operations for each request will make them heavy.
[...]
cluster.authenticate(CLUSTER_USERNAME, CLUSTER_PASSWORD);
const bucket = cluster.openBucket(BUCKET);
bucket.on("error", error => {
console.log("Error from bucket:", JSON.stringify(error, null, 2));
});
[...]Note that the first line of the function implementation sets the callbackWaitsForEmptyEventLoop context property to false.
[...]
const createHandler = async (event, context) => {
context.callbackWaitsForEmptyEventLoop = false;
[...]This is needed to return immediately the result instead of waiting that the Node event loop empty. In our case some of the bucket connection will make the function timeout.
Payload validations are carried out immediately. If the result is negative, the function will return a 400 Bad Request status code indicating the invalid field.
Once all the validations have been passed, the document to insert will be prepared adding the Contact type property. Furthermore, the document id will be calculated concatenating the type with the :: separator and a generated UUID. These two operations are a Couchbase modeling best practices. They allow both to perform specific queries by type and to improve the readability of these.
[...]const document = {
id,
name,
surname,
email,
type: "Contact”};
const documentId = `contact::${id}`;
[...]At the end, to insert the document into Couchbase we are going to use the insert method of the SDK as following.
const documentInserted = await new Promise((resolve, reject) => {
bucket.insert(documentId, document, (error, result) => {
if (error) {
reject(error);
} else {
resolve({ ...result, ...document });
}
});
});Retrieve a contact in Couchbase Server
The contact retrieve function follows the contact creation logics. We will find here: the endpoint definition, the Couchbase and bucket connection and the waiting event loop disabling.
To retrieve the document we will be using the get SDK method.
[...]const document = await new Promise((resolve, reject) => {
bucket.get(documentId, (error, result) => {
if (error) {
reject(error);
} else {
resolve(result.value);
}
});
});[...]If the document doesn’t exists, the sdk will throw a code 13 error, so we can return a 404 Not Found status code to the client.
Update a contact in Couchbase Server
Same process for the contact update function. Here we will find: the endpoint definition, the Couchbase and bucket connection and the waiting event loop disabling.
To update the document we will be using the upsert SDK method.
[...]const documentUpdated = await new Promise((resolve, reject) => {
bucket.upsert(documentId, document, (error, result) => {
if (error) {
reject(error);
} else {
resolve({ ...result, ...document });
}
});
});[...]If the document doesn’t exists it will be created, the upsert method in fact inserts or update a document.
Delete a contact in Couchbase Server
Finally, we will also follow the same procedure for the contact deletion function. Here we will find: the endpoint definition, the Couchbase and bucket connection and the waiting event loop disabling.
To delete the document we will be using the remove SDK method.
[...]
const document = await new Promise((resolve, reject) => {
bucket.remove(documentId, (error, result) => {
if (error) {
reject(error);
} else {
resolve({ contactId, ...result });
}
});
});
[...]As for retrieve, if the document doesn’t exists, the SDK will throw a 13 error, so that we will return a 404 Not Found status code.
Conclusions
In this article we have seen how to integrate Couchbase in a serverless backend, simplifying operations during development usings the framework serverless and AWS Lambda Layer.
Did you like this article?
Read also the other articles of our Discover Couchbase series.
