Using DynamoDB with GraphQL Dataloader

Cover Image for Using DynamoDB with GraphQL Dataloader

Dataloaders are a powerful tool to use when resolving GraphQL queries. They address the N+1 issue that occur naturally when sending requests from resolvers inside list results. They do this by asking you to send batch requests to your backend.

If you are using DynamoDB as either a cache layer or as your main database, pairing it with Dataloaders is a powerful pattern. Especially to save costs in low traffic applications, to handle huge amounts of traffic in high traffic applications, or to avoid the headache of persistent connections in lambas. This article will show how to do that using JavaScript/TypeScript and the AWS SDK.

Setting up your connection

First you need to get programmatic access to your table. Start by finding the table ARN. You can do this by going to DynamoDB part of the AWS Console, going to Tables, <Your table>, and looking under "Additional info" in the Overview tab.

Next, go to the IAM part of the AWS Console, and create a new policy using the JSON tab. Use this policy (but replace TABLE_ARN with the ARN of your table):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "dynamodb:BatchGetItem",
        "dynamodb:BatchWriteItem",
        "dynamodb:ConditionCheckItem",
        "dynamodb:DeleteItem",
        "dynamodb:GetItem",
        "dynamodb:PutItem",
        "dynamodb:Query",
        "dynamodb:Scan",
        "dynamodb:UpdateItem"
      ],
      "Resource": ["TABLE_ARN", "TABLE_ARN/index/*"],
      "Effect": "Allow"
    }
  ]
}

Stay in IAM and create a new user with programmatic access. When setting permissions, select "Attach existing policies directly" and use the one you just created. Take note of the key id and secret access key.

In your node project, make sure you have the AWS SDK installed.

Set the following environment variables:

AWS_ACCESS_KEY_ID=THE_KEY_FROM_YOUR_CREATED_USER
AWS_SECRET_ACCESS_KEY=THE_SECRET_KEY_FROM_YOUR_CREATED_USER

Create a new module where you will set up your connection and expose the DynamoDB interface:

import AWS from "aws-sdk";

AWS.config.update({
  region: "us-east-1", // Or whatever region you are using
});

const TableName = "YOUR_TABLE";
export const docClient = new AWS.DynamoDB.DocumentClient();

export const batchGet100Items = <T>(
  ids: { pk: string; sk: string }[] // Change to your key structure
): Promise<T[]> => {
  return new Promise((resolve, reject) => {
    docClient.batchGet(
      {
        RequestItems: {
          [TableName]: {
            Keys: ids,
          },
        },
      },
      (err, data) => {
        if (err) {
          return reject(err);
        }
        resolve((data.Responses[TableName] || []) as T[]);
      }
    );
  });
};

Now you should be ready to use it in your dataloaders!

Creating your Dataloader

I assume you are already familiar with using Dataloaders, and I will only bring up the three DynamoDB specific parts. The DynamoDB BatchGetItem API will only return 100 items at a time. Composite keys needs a cache key function.

  1. The API will return at most 16MB, which I will not provide a fix for. If you have large items, use the "UnprocessedKeys" return value to fire off more requests.
  2. To address the first point, you can either programmatically keep sending DynamoDB requests to fetch all items if you have batches more than 100 items. However, my recommendation is to use the dataloader maxBatchSize option and set it to 100.
  3. To address the second point. I recommend you to use the cacheKeyFn option to something like `key =>``${key.pk}:${key.sk}```.

A complete dataloader would look something like:

const myLoader = new DataLoader((keys) => batchGet100Items(keys), {
  cacheKeyFn: (key) => key.pk + ":" + key.sk,
  maxBatchSize: 100,
});

And now you should be able to use it in your resolver functions like so:

myLoader.load({ pk: "123", sk: "2022-04-11" });
Peter Nycander
Peter Nycander