Comprehensive Guide for Creating and Managing Cron Jobs in Nodejs / Adonisjs Applications

Comprehensive Guide for Creating and Managing Cron Jobs in Nodejs / Adonisjs Applications

Ndianabasi Udonkang's photo
Ndianabasi Udonkang
·

15 min read

Table of contents

In this article, we will discuss in great details the strategies which you can adopt for creating and managing cron jobs for your Nodejs applications. Some of the examples will be demonstrated with the Adonisjs framework.

What are Cron Jobs?

Traditionally, cron jobs are regular shell scripts which are setup/scheduled to run via the operating system cron at predefined intervals. The interval could be daily, twice a day, once a month, one a week, etc.

While the operating system cron can be used to schedule jobs via the crontab file, there now exists JavaScript/Nodejs libraries which allows one to execute scripts at predefined intervals without the dependence on the operating system cron. Such libraries are installed within an application, so that the jobs are executed within the process of the currently-running Nodejs application.

For the sake of differentiation, in this article, we will call jobs set up via the operating system cron, System Cron Jobs, and jobs set up within an application, In-process Cron Jobs. So, the two strategies we will discuss for creating and scheduling cron job within our Nodejs/Adonisjs application are:

  1. System Cron Jobs, and
  2. In-Process Cron Job.

System Cron Jobs

Our Cron job discussion will be focused on the Linux operating system.

System cron jobs are scheduled with the operating system Crontab file. Let's have a look at some basics of the System cron.

Crontab file

A Crontab file is just a simple text file where instructions are defined for the Cron daemon. The Cron daemon is the persistent background service which runs your cron instructions. Each user of a Linux machine has a separate Crontab file.

Creating or Editing the Crontab file

You can maintain the Crontab file of the current (logged in) user with the command:

crontab -e

If you are accessing for the first time, a new crontab file will be created and opened with your default text editor such as nano or vim.

If you are a privileged user (with sudo access), you can edit the Crontab file of other users with the command:

sudo crontab -u other-user -e

Displaying the Crontab file

If you need to simply display the content of the Crontab file over standard output, use the command:

crontab -l

For other users, do:

sudo crontab -u other-user -l

Examples of Cron instructions

# Define the variable `APP_DIR`
APP_DIR=/var/www/my_app

# Delete temporary files at five minutes after midnight, every day
# Uses a Bash script
5 0 * * *       $APP_DIR/jobs/delete_tmp_files.sh > $APP_DIR/logs/cron.log 2>&1

# Generate monthly report for the application at 2:00pm on the first of every month
# Uses Adonisjs Ace command
0 14 1 * *     cd $APP_DIR && node ace generate:monthly_report > $APP_DIR/logs/cron.log 2>&1

# Send newsletters at 8 am on weekdays
# Uses a JavaScript script
0 8 * * 1-5    $APP_DIR/jobs/send_newsletters.js > $APP_DIR/logs/cron.log 2>&1

You can use crontab.guru, do generate Cron schedules

Logging the Output of a Cron job

In the Cron instructions below:

0 8 * * 1-5    $APP_DIR/jobs/send_newsletters.js > $APP_DIR/logs/cron.log 2>&1

You will notice the output redirection: > $APP_DIR/logs/cron.log 2>&1

Let's break it down:

  1. The redirection operator > sends the standard output from the script or command on the left-hand side into the log file on the right-hand side. In a typical shell script, contents for the standard output are generated with either the echo or printf commands. In a typical JavaScript script, contents for the standard output are generated with the console.log method. The standard output is typically the console.

  2. The second redirection 2>&1 is used to redirect the standard error (indicated as the file descriptor 2 into the standard output (indicated as the file descriptor 1). In a typical shell script, contents for the standard error are generated when a script exits with an error. In a typical JavaScript script, contents for the standard error are generated with the console.error method.

    The & is used to signify the the 1 on the right-side of the redirection operator > is a file descriptor and not a file name. The & is not necessary when a file descriptor is used on the left-side of a redirection operator. Also notice that there are no space in the redirection instruction 2>&1 between two file descriptors. Read more that the shell redirections and Bash One-Liners Redirections.

    Bonus: A File Descriptor is a reference to an open file within the filesystem of an operating system. When you open a file for reading or writing using any command or function provided by the language you are using, the opened file is known to the operating system via its file descriptor. File descriptors are basically numerical pointers such as 1, 2, 3, etc. For any operating system, three files are always opened by default and assigned the file descriptors: 1, 2, and 3. There are: the standard input (with file descriptor 0), standard output (with file descriptor 1), and standard error (with file descriptor 2).

The redirection discussed above can be simplified with:

0 8 * * 1-5    $APP_DIR/jobs/send_newsletters.js &>$APP_DIR/logs/cron.log

Again read this excellent article on Bash One-Liners Redirections

Advantages of Using the System Cron

  1. Since System Cron jobs are scheduled and ran with the Cron daemon, the jobs will run even when the application is not running.
  2. You can easily pause or discontinue a system cron job by commenting out the instructions on the Crontab file.
  3. Since the system cron jobs are not running within the process of the application, you can scale the application without worrying about duplication of the jobs.

Disadvantages of Using the System Cron

  1. Cannot be used in edge deployments which do not provide access to the operation system Crontab.
  2. Difficult to use in some serverless deployments unless the deployment service provides Cron scheduling interface.
  3. Requires tampering with the system crontab
  4. Crontab file are not automatically moved when you migrate your application to another server. You will need to copy your Crontab file manually to the new server.

Security Considerations When Using the System Cron

It is important to adopt security/safety precautions when working with system cron jobs since the jobs are executed by the system shell.

  1. Use the least privileged user. If your application root is located at /var/www/my_app and the /var/www directory and subdirectories are owned by the www-data user (the default web user for Ubuntu, you should create the Crontab instructions for the application with thewww-data` user by executing:

    sudo crontab -u www-data -e

    When the cron jobs for the www-data user are executed, the privileges of the scripts will be limited to the privileges of the www-data user. On very rare reasons should you create cron jobs with your own user (log in) account or the root user account.

  2. Thoroughly test your scripts and understand every bit of what it does before creating cron jobs with them.

Strategies for Creating System Cron Jobs

We will discuss three (3) strategies which can be adopted for Creating System Cron Job:

  1. Use of Bash shell scripts
  2. Use of Nodejs shell scripts
  3. Use of Adonis Ace commands

Use of Bash shell scripts for Cron Jobs

Learn how to write Bash shell scripts.

The following steps should typically be followed when working with a Bash shell script:

  1. Bash shell scripts typically begin with a shebang. A shebang for the Bash shell looks like:

    #!/bin/bash

    The shebang tells the shell which interpreter to use for interpreting the file. The shebang is respected by all programming languages and will be ignored when it appears on the first line of the script.

  2. It is common practice to save Bash shell script files with the extension .sh. E.g. my_backup_script.sh.

  3. After the saving the file and as a security practice, set the owner, group, and permissions for the file. It is important to make the file executable. See the commands below:

APP_DIR=/var/www/my_app

# Equivalent of `cd /var/www/my_app`
cd $APP_DIR

# Set the user and group for the script
sudo chown www-data:www-data jobs/delete_tmp_files.sh

# Set the permissions for the script
# Here we want only the user and group to have `execution` (`x`) permission
sudo chmod ug+x jobs/delete_tmp_files.sh

The commands above are also applicable when working with JavaScript shell scripts.

Use of Nodejs Shell Scripts for Cron Jobs

As a JavaScript developer, you might not be as proficient in Bash as you are with JavaScript. Instead of struggling to learn Bash, you could use JavaScript combined with standard Nodejs API to write your shell scripts. Also, by writing your shell script with JavaScript, you have full access to all NPM modules including JavaScript SDKs for 3rd party platforms you might want to interact with within your cron job.

The following steps should typically be followed when working with a Nodejs shell script:

  1. Nodejs shell scripts typically begin with a shebang and the shebang for the Nodejs looks like:

    #!/usr/bin/env node

  2. As you already know, you would save Nodejs file with the extension .js.

  3. Follow the commands in the previous section to set the user and group for the script and also set the execution permissions.

Example of a Nodejs Shell Script

The script below is a working script for backing up a PostgreSQL database

#!/usr/bin/env node

"use strict";

require("dotenv").config();
const path = require("path");
const { DateTime } = require("luxon");
const { exec } = require("child_process");
const { existsSync, mkdirSync } = require("fs");

class DbBackupHandler {
  constructor() {
    this.DB_DATABASE = process.env.PG_DB_DATABASE;
    this.DB_USER = process.env.PG_BACKUP_USER;
    this.DB_PASSWORD = process.env.PG_BACKUP_USER_PASSWORD;

    if (!this.DB_DATABASE || !this.DB_USER || !this.DB_PASSWORD) {
      throw new Error("Invalid credentials");
    }
  }

  get dbCredentials() {
    return {
      db: this.DB_DATABASE,
      user: this.DB_USER,
      password: this.DB_PASSWORD
    };
  }

  get now() {
    return DateTime.now().toFormat("yyyy-LL-dd HH:mm:ss");
  }

  run() {
    return new Promise(async (resolve, reject) => {
      try {
        console.info(`DbBackupHandler: DB Backup started at: ${this.now}`);

        const fileName = `${this.dbCredentials.db}-${DateTime.now().toFormat(
          "yyyy-LL-dd-HH-mm-ss"
        )}.gz`;
        const relativeDirName = "backups";
        const fullDirName = `${path.join(process.cwd(), relativeDirName)}`;
        const fullFilePath = `${fullDirName}/${fileName}`;

        // Create the backup directory if not exists
        if (!existsSync(relativeDirName)) {
          mkdirSync(relativeDirName);
        }

        exec(
          `PGPASSWORD=${this.dbCredentials.password} pg_dump -U ${this.dbCredentials.user} -Fc -w ${this.dbCredentials.db} | gzip > ${fullFilePath}`,
          async (error, _stdout, stderr) => {
            if (stderr) {
              console.error(`DbBackupHandler: exec stderr:`, stderr);
            }
            if (error) {
              console.error(`DbBackupHandler: exec error:`, error);
              reject(error);
            } else {
              console.info(`DbBackupHandler: Local backup created.`);
              return resolve(
                "DbBackupHandler: Backup completed at: " + this.now
              );
            }
          }
        );
      } catch (error) {
        reject(error);
      }
    });
  }
}

async function main() {
  return new DbBackupHandler().run();
}

main().then(console.log).catch(console.error);

module.exports = DbBackupHandler;

The script above can be scheduled to run every midnight with the Cron instruction:

APP_DIR=/var/www/my_app

0 0 * * *       $APP_DIR/jobs/db_backup_script.js > $APP_DIR/logs/cron.log 2>&1

Use of the Adonisjs Ace Commands for Cron Jobs

The Adonisjs Framework comes with an in-built CLI call Ace. With the Ace CLI, you can run in-built commands or create your own custom commands. The custom commands are written in the familiar JavaScript language. The Ace CLI allows you to define command description, arguments, flags, prompts, and craft beautiful command UI for feedback while your command is running.

To use the Adonisjs Ace CLI for cron jobs, follow the steps below:

  1. Within the directory of your Adonisjs application, create a new command by running:

    node ace make:command DbBackup

    This should create a new command file named: commands/DbBackup.ts.

  2. Regenerate the Ace command manifest file:

    node ace generate:manifest

  3. Now, if you run node ace, you will see the new command listed among the available commands for your application as db:backup. This means that you can run the command with node ace db:backup

    Ace command list

  4. Go ahead and develop the command script located in commands/DbBackup.ts. See the example below for a working Ace command.

  5. Then schedule the command to run with the Cron instruction:

APP_DIR=/var/www/my_app

# Generate daily backup for the application at 01:00am
0 1 * * *     cd $APP_DIR && node ace db:backup &>$APP_DIR/logs/cron.log

Example of a Working Adonisjs Ace Command for Cron Job

Below is an example of a working Ace command for PostgreSQL backup to local disk. Ensure that you run: node ace generate:manifest to regenerate the manifest file. Afterwards, node ace will show the update listing of the command. In the script below, I avoided loading the Adonisjs application and relying on the IoC container which requires starting up the application.

import('dotenv').then((file) => file.config())

import path from 'path'
import { DateTime } from 'luxon'
import { exec } from 'child_process'
import { existsSync, mkdirSync } from 'fs'
import { BaseCommand } from '@adonisjs/core/build/standalone'

export default class DbBackup extends BaseCommand {
  /**
   * Command name is used to run the command
   */
  public static commandName = 'db:backup'

  /**
   * Command description is displayed in the "help" output
   */
  public static description = 'Simple backup script for your PostgreSQL database'

  public static settings = {
    /**
     * We do want to load the Adonisjs application
     * when this command is ran
     */
    loadApp: false,

    /**
     * We want the process within which the command is run to exit
     * immediately the command completes execution
     */
    stayAlive: false,
  }

  public async run() {
    const NOW = DateTime.now().toFormat('yyyy-LL-dd HH:mm:ss')

    const DB_CREDENTIALS = {
      db: process.env.PG_DB_NAME,
      user: process.env.PG_USER,
      password: process.env.PG_PASSWORD,
    }

    if (!DB_CREDENTIALS.db || !DB_CREDENTIALS.user || !DB_CREDENTIALS.password) {
      throw new Error('Invalid credentials')
    }

    try {
      console.info(`DbBackupHandler: DB Backup started at: ${NOW}`)

      const fileName = `${DB_CREDENTIALS.db}-${DateTime.now().toFormat('yyyy-LL-dd-HH-mm-ss')}.gz`
      const relativeDirName = 'backups'
      const fullDirName = `${path.join(process.cwd(), relativeDirName)}`
      const fullFilePath = `${fullDirName}/${fileName}`

      // Create the backup directory if not exists
      if (!existsSync(relativeDirName)) {
        mkdirSync(relativeDirName)
      }

      exec(
        `PGPASSWORD=${DB_CREDENTIALS.password} pg_dump -U ${DB_CREDENTIALS.user} -Fc -w ${DB_CREDENTIALS.db} | gzip > ${fullFilePath}`,
        async (error, _stdout, stderr) => {
          if (stderr) {
            console.error(`DbBackupHandler: exec stderr:`, stderr)
          }
          if (error) {
            throw error
          } else {
            console.info(`DbBackupHandler: Local backup created.`)
            console.info('DbBackupHandler: Backup completed at: ' + NOW)
          }
        }
      )
    } catch (error) {
      console.error('error: %o', error)
    }
  }
}

In-Process Cron Jobs

In-Process Cron Jobs are created by using special packages to create schedules within the same process as the currently-running Nodejs application. A handler function or method is then attached to each schedule so they are executed when the scheduled time arrives.

One example of such packages which can be used to create In-Process Cron Jobs is node-schedule.

It is possible to execute Ace commands within In-Process Cron Jobs via packages like execa or Nodejs Child Process

Advantages of In-Process Cron Jobs

  • Simple. In-process cron jobs run within the same Nodejs process as the application server so does not require special setup. Great for applications which won't be scaled horizontally.
  • Fast. Since in-process cron jobs do not boot-up the application server. Instead they execute as a regular functions or methods would within the application.
  • Portable. Because the schedules are created within the source code of the application. So, the schedules go wherever the source code goes.
  • Can be used on edge deployments which do not provide access to the operation system Crontab.

Disadvantages of In-Process Cron Jobs

  • Job Duplication. If multiple instances of the application are running (like when deploying the application with process managers like PM2 in cluster mode), each instance of the application will execute the In-Process Cron Job.
  • Not reliable. In-process cron jobs will only run when the application is running. Therefore, the cannot be used for mission-critical jobs.
  • Not easily pause-able. It might be difficult to pause the execution of in-process cron jobs unless you make use of environment variables, in which case, you will have to restart the application each time you toggle the variables.

Example of In-Process Cron Jobs with AdonisJs

In the example below, we are going to implement an In-Process Cron Job for database backup.

Step 1: Clone the repository and install dependencies

git clone https://github.com/ndianabasi/google-contacts.git

# We focus on the API server
cd api
yarn install

Step 2: Create the .env file

cp .env.example .env

Add an entrance variable to the .env file.

MYSQL_USER=lucid
MYSQL_PASSWORD=
MYSQL_DB_NAME=lucid
+ ENABLE_DB_BACKUPS=true

Step 3: Create strong type for the ENABLE_DB_BACKUPS variable

Open api/env.ts file and add:

  MYSQL_PASSWORD: Env.schema.string.optional(),
  MYSQL_DB_NAME: Env.schema.string(),
+ ENABLE_DB_BACKUPS: Env.schema.boolean(),

Step 4: Setup the database

  1. Create a MySQL database for the application.
  2. Replace the value of the database environment variables in .env with the real database credentials. Most especially: MYSQL_USER, MYSQL_PASSWORD, and MYSQL_DB_NAME.
  3. Migrate the database with: node ace migration:run.
  4. Seed the database with: node ace db:seed

Step 5: Improve .gitignore file

We want to ignore the database backup files which will be stored in api/backups folder.

Open api/.gitignore file and add:

.env
tmp
+ backups

Step 6: Install node-schedule package

yarn add node-schedule && yarn add @types/node-schedule -D

Step 7: Create the Backup Handler

Create a new file api/app/Cron/Handlers/DailyDbBackupHandler.ts with the following contents:

import path from 'path'
import { DateTime } from 'luxon'
import { exec } from 'child_process'
import Env from '@ioc:Adonis/Core/Env'
import { existsSync, mkdirSync } from 'fs'
import Logger from '@ioc:Adonis/Core/Logger'

export default class DailyDbBackupHandler {
  private logger: typeof Logger

  constructor() {
    this.logger = Logger
  }

  public run() {
    return new Promise(async (resolve, reject) => {
      try {
        const NOW = DateTime.now().toFormat('yyyy-LL-dd HH:mm:ss')

        const DB_CREDENTIALS = {
          db: Env.get('MYSQL_DB_NAME'),
          user: Env.get('MYSQL_USER'),
          password: Env.get('MYSQL_PASSWORD'),
        }

        if (!DB_CREDENTIALS.db || !DB_CREDENTIALS.user) {
          throw new Error('Invalid credentials')
        }

        this.logger.info('DbBackupHandler: DB Backup started at: %s', NOW)

        const fileName = `${DB_CREDENTIALS.db}-${DateTime.now().toFormat('yyyy-LL-dd-HH-mm-ss')}.gz`
        const relativeDirName = 'backups'
        const fullDirName = `${path.join(process.cwd(), relativeDirName)}`
        const fullFilePath = `${fullDirName}/${fileName}`

        // Create the backup directory if not exists
        if (!existsSync(relativeDirName)) {
          mkdirSync(relativeDirName)
        }

        exec(
          `mysqldump -u${DB_CREDENTIALS.user} -p${DB_CREDENTIALS.password} --compact ${DB_CREDENTIALS.db} | gzip > ${fullFilePath}`,
          async (error, _stdout, stderr) => {
            if (stderr) {
              this.logger.info('DbBackupHandler: %s', stderr)
            }
            if (error) {
              return reject(error)
            }
            this.logger.info('DbBackupHandler: Local backup created at: %s', NOW)
            this.logger.info('DbBackupHandler: Backup completed at: %s', NOW)
            resolve('done')
          }
        )
      } catch (error) {
        reject(error)
      }
    })
  }
}

Step 7: Create the api/app/Cron/index.ts scheduling file with the following contents:

import scheduler from 'node-schedule'
import Env from '@ioc:Adonis/Core/Env'
import Logger from '@ioc:Adonis/Core/Logger'
import DailyDbBackupHandler from './Handlers/DailyDbBackupHandler'

/**
 * Runs every 12 hours
 */
scheduler.scheduleJob('0 */12 * * *', async function () {
  const isDbBackupsEnabled = Env.get('ENABLE_DB_BACKUPS')

  if (isDbBackupsEnabled) {
    await new DailyDbBackupHandler()
      .run()
      .catch((error) => Logger.error('DailyDbBackupHandler: %o', error))
  }
})

Logger.info('In-process Cron Jobs Registered!!!')

Here you can change how frequent you want to backup to be made. For testing purposes, you might want to change the schedule to every minute:

scheduler.scheduleJob('* * * * *', async function () {
  const isDbBackupsEnabled = Env.get('ENABLE_DB_BACKUPS')

  if (isDbBackupsEnabled) {
    await new DailyDbBackupHandler()
      .run()
      .catch((error) => Logger.error('DailyDbBackupHandler: %o', error))
  }
})

Step 8: Run the schedule

Import the schedule entry file into api/providers/AppProvider.ts and execute the schedules when the application is ready

  public async boot() {
    // IoC container is ready
  }

  public async ready() {
   // App is ready
+  import('App/Cron/index')
  }

  public async shutdown() {
    // Cleanup, since app is going down
  }

When the server restarts, you should see a log in the console:

[1661263670963] INFO (google-contacts-clone-api/55793 on xxxx): In-process Cron Jobs Registered!!!
[1661263670967] INFO (google-contacts-clone-api/55793 on xxxx): started server on 0.0.0.0:3333

Congratulations. You have setup an In-process Cron Job for database backup.

See all relevant files here: github.com/ndianabasi/google-contacts/compa..

Cover image background from Freepik.

croncronjobNode.jsAdonisJS