Pull consumers

A pull consumer asks for messages instead of receiving requests, so it can run anywhere with network access: a long-running worker, a cron job, or a script on your laptop. A queue without a consumer path is a pull queue.

Process messages in a loop

consume long-polls the queue, runs your function on each message, and settles it for you. It returns when signal aborts, or once the queue is empty with stopWhenEmpty.

import { GigadriveClient, NonRetryableError } from '@gigadrive/sdk';

// clientId and clientSecret fall back to GIGADRIVE_CLIENT_ID and GIGADRIVE_CLIENT_SECRET
const client = new GigadriveClient({ applicationId: '0197b2f1-2f4a-7a0b-8a2d-222222222222' });
const thumbnails = client.queues.queue<{ objectKey: string }>('thumbnails', { environment: 'production' });

const controller = new AbortController();
process.on('SIGTERM', () => controller.abort());

await thumbnails.consume(
  async (job) => {
    if (!job.objectKey.endsWith('.jpg')) throw new NonRetryableError('Unsupported file type');
    await renderThumbnail(job.objectKey);
  },
  { maxMessages: 10, visibilityTimeout: '5m', signal: controller.signal }
);

The token needs the network:queues:consume scope. A worker outside Gigadrive Network uses an API key you create, as described in API keys.

Your function's outcome settles each message the same way it does for a push consumer: returning acknowledges it, RetryLaterError defers it without spending an attempt, NonRetryableError dead-letters it, and any other error retries it with backoff. consume acknowledges messages that finish within 25 milliseconds of each other in one batch request, which costs fewer operations than acknowledging them one by one. A batch acknowledgement that fails in transit, or with a 5xx or 429 response, is retried twice. A settlement that still fails goes to the onError callback, and the message is delivered again after its lease ends.

Leases

Receiving a message leases it to you for the visibility timeout, the queue's setting unless you pass one, and up to 12 hours. Nobody else receives it during the lease. A message you have not settled when the lease ends becomes available again and counts as a failed attempt.

consume waits up to 20 seconds for messages by default and returns as soon as one is ready, so an idle worker makes about three requests a minute rather than spinning, without adding delay when a message arrives. receive returns at once unless you pass wait, at most 20 seconds; with it, receive also returns as soon as a message is ready. A paused queue returns no messages.

Settle messages yourself

receive returns leased messages without processing them, for when you need control over each step:

const [message] = await thumbnails.receive({ maxMessages: 1, wait: '20s' });

if (message) {
  await message.extendLease('10m'); // the work takes longer than expected
  await renderThumbnail(message.payload.objectKey);
  await message.ack();
}

Each call needs the message's receipt, which the SDK keeps for you. A receipt from an expired lease answers 409 receipt_mismatch, because the message may already belong to another worker.

Ordering groups

Pull consumers honor ordering groups too: while one message of a group is leased, the next one is not handed out.