Introduction

Neptune for Couchbase is our Database-as-a-Service sporting the best performance, availability and scalability ever. It pairs well with our support plans for further assistance from our qualified technicians.

You might want to migrate away from a Community Edition when your project evolves out of the prototyping stage, when you are faced with availability issues or data size limitations or you simply want to focus your resources elsewhere instead of managing a cluster by yourself. Due to Neptune for Couchbase being a managed service, all of these issues are taken care of by MOLO17.

Make sure to read this document to the end to assess the procedure beforehand, and once you feel confident, proceed with the instructions. Feel free to contact us if you have any questions or for further assistance.

Applicability and Limitations

This guide instructs you on how to move your data and indexes from Couchbase Server Community Edition versions 6.6, 7.0 and 7.1 to our Neptune for Couchbase.

Although Neptune for Couchbase supports Eventing functions, Full-Text Search and Analytics, migrating data for these features is not described in this document.

Prerequisites

You are supposed to have:

  • an existing Couchbase Server, Community Edition (CE) installation with data you want to migrate
  • some degree of familiarity with Couchbase Server
  • familiarity with running commands at a command-line interface
  • an existing account with Neptune for Couchbase. This can be obtained through our sales representatives following this link: sales@molo17.com.

Tools

Up to version 6.6 of Couchbase Server, Community Edition the tool that must be used is cbbackup, together with its restore tool cbrestore.

Starting from version 7.0 of Couchbase Server, Community Edition the recommended tool is cbbackupmgr.

Step-by-step

1. Prepare for migration

This requires producing a list of information that you will need during the migration process, to speed up operations and reduce downtime.

Log in to your Couchbase Server, Community Edition web console, and check the status of your nodes and buckets.

If you don’t already have a Neptune for Couchbase account, and are gathering data to contact our representatives, make sure to make a list of:

  • number of your cluster nodes
  • memory and storage available to each node
  • bucket list
  • RAM quota for each bucket
  • Conflict resolution settings for each bucket

To successfully verify the migration, later, you will need to record:

  • the number of documents (“Items” column)  in your source buckets
  • a list of the indices in your source cluster

Also, it will be useful to have a sample N1QL query that your applications use to access data on the buckets.

2. Activate an account on Neptune for Couchbase

Our sales representatives will assist you in properly sizing the Neptune for Couchbase service to fit your needs.

At the end of the activation, you will receive access credentials to your new buckets. You can also access the web console to check on your data.

3a. Migrate using cbbkpmgr (version 7 and above)

This tool works by setting up a backup repository and performing backups there; this also allows for policies to be applied to the repository. Each repository can contain data about multiple buckets.

3a.1. Locate the tool

You can find the cbbackupmgr tool in the root directory of the Couchbase Server installation, that is placed by default in these locations:

  • on linux systems, in /opt/couchbase/bin/
  • on Windows systems, in C:\Program Files\Couchbase\Server\bin\
  • on macOS, in /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin

If you cannot directly access the root directory of the Couchbase Server installation you can download and install it on your system, to access the tool.

3a.2. Create a new local backup repository

To create a new local repository you need a new empty directory (in the following examples, /backups/) and then run the following command:

cbbackupmgr config –archive /backups/ –repo [repository-name] –disable-eventing –disable-analytics –disable-cluster-analytics –disable-bucket-query –disable-cluster-query –include-data [buckets-list]

Where [repository-name] is the name you’d like to give to your repository and [buckets-list] is the list of buckets to operate on, separated by a comma.

To verify the repository contents (and check that it has been successfully created) run:

cbbackupmgr info -a /backups -all

3a.3. Backup the buckets

To actually perform the backup, run the following command:

cbbackupmgr backup --archive /backups --repo [repository-name] --cluster couchbases://[couchbase-url] --username [admin-username] --password [admin-password] --cacert [path-to-ca-certificate] --full-backup

Again you can verify repository contents with

cbbackupmgr info -a /backups -all

to ensure that the operation completed successfully.

3a.4. Restoring on Neptune for Couchbase

Once you backed up data from your original server you can restore the data into Neptune with the following command:

cbbackupmgr restore --archive /backups --repo [repository-name] --cluster couchbases://[neptune-url] --username [username] --password [password] --cacert [path-to-neptune-ca-certificate]

Once the restore is complete you should see the message “Restore completed successfully” in the terminal.

3b. Migrate using cbbackup/cbrestore (version 6.6)

These tools work by dumping and restoring a single bucket at a time. The following procedure deals with a single bucket; to migrate more than one bucket, just repeat the steps.

3b.1. Locate the tools

You can find the cbbackup and cbrestore tools in the root directory of the Couchbase Server installation, that is placed by default in these locations:

  • on linux systems, in /opt/couchbase/bin/
  • on Windows systems, in C:\Program Files\Couchbase\Server\bin\
  • on macOS, in /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/bin

If you cannot directly access the root directory of the Couchbase Server installation you can download and install it on your system, to access the tools.

3b.2. Backup the bucket

Once you have an empty directory to use for the backup, run the following command:

cbbackup -u [admin-username] -p [admin-password] couchbase://[couchbase-url] /backups -b [bucket-name]

Once the operation completes, you can inspect the directory contents to ensure that you have the correct backup.

3b.3. Restoring on Neptune for Couchbase

Once you backed up data from your source bucket you can restore the data into Neptune with the following command:

cbrestore /backups https://[neptune-url] --bucket-source=[bucket-name] --bucket-destination=[destination-bucket-name] --from-date=[backup-date] --to-date=[backup-date-after] --username=[username] --password=[password] --ssl --cacert=[path-to-neptune-ca-certificate]

Due to the way cbrestore works, you need to specify both the date of when you performed the backup and the day after that (backup-date-after). Once the restore is complete you should see the message “Done” in the terminal.

4. Rebuild Indexes

You can leverage N1QL to automatically produce the query string that will build your indexes that require deferred build. Also, if you feel like your indices will take too much time to build, you can split the query in batches.

To obtain the query that will build your indexes, run this query from the “Query” section in the web console:

SELECT RAW CONCAT("BUILD INDEX ON ", k ,
"(['", CONCAT2 ("','", idnames), "']);")
FROM system:indexes AS s
LET bid = CONCAT("`",s.bucket_id, "`"),
    sid = CONCAT("`", s.scope_id, "`"),
    kid = CONCAT("`", s.keyspace_id, "`"),
    k = NVL2(bid, CONCAT2(".", bid, sid, kid), kid)
WHERE s.namespace_id = "default"
GROUP BY k
LETTING idnames = ARRAY_AGG(s.name)
HAVING ARRAY_LENGTH(idnames) > 0;

This query will compose a string like this:

"BUILD INDEX ON `bucket`(['def_index1','def_index2']);"

that can be used as a query itself to actually perform the index rebuilding operation.

This query will include both deferred and non-deferred build indices; if you prefer you can manually exclude non-deferred indices from the list before running the query.

5. Verify the migration

Using the web console, access your bucket list on Neptune for Couchbase. Verify that the number in the “Items” column matches with the number in the original bucket.

Verify that all the Indices that were on your source cluster are present in the Neptune for Couchbase web console, and that they have all been successfully built.

Last, try to perform a sample query on your data. It is more useful if this is a typical usage query your application uses to access your data.

Contact us

You can contact our sales representatives that will be happy to assist you in assessing your needs, gathering data about your current installation, and size an appropriate Neptune for Couchbase service for you.