Serverless Website Monitoring

I recently have been playing around with some tools for monitoring websites but with these tools they can be an unnecessary expense for a start-up. So I went and built my own monitoring stack in AWS CDK to monitor if my website was running and alert me of any issues.

I’m going to assume you know a bit about CDK and jump right into the stack code and what it is made up of.

The stack is going to contain the following services:

  • Dynamo DB – this is used to store a history of the website status and the current status.
  • SNS – used to notify the subscribed users that the website has gone down or has come back up (by text or email)
  • Event Bridge – the scheduler.
  • Lambda – to run the check, write the results and publish to SNS.

First of the imports I have used, I like to shorten the import names so the code is a bit easier to read.

import { Construct } from 'constructs';
import {
  aws_lambda as lambda,
  aws_lambda_nodejs as lambdaNode,
  aws_sns as sns,
  aws_sns_subscriptions as subscriptions,
  aws_dynamodb as dynamodb,
  aws_events as events,
  aws_events_targets as targets,
  aws_iam as iam,
  Stack,
  StackProps,
  Duration,
  RemovalPolicy,
} from 'aws-cdk-lib';
require('custom-env').env(process.env.NODE_ENV);

Next the Variables are setup, I like to do this mostly at the top so its nice and clear, I am also using a .env file in this case to declare all the values.

if(process.env.USE_MONITORING_FEATURE === 'true'){

    // Some Checks to make sure the environment variables are setup correctly.
    if (process.env.ZONE_NAME === undefined) {
      throw new Error(( 'Zone undefined'));
    }
    if (process.env.MONITORING_STRING === undefined) {
      throw new Error(( 'MONITORING_STRING undefined'));
    }
    if (process.env.MONITORING_EMAIL === undefined) {
      throw new Error(( 'MONITORING_EMAIL undefined'));
    }
    if (process.env.MONITORING_RATE_MINS === undefined) {
      throw new Error(( 'MONITORING_RATE_MINS undefined'));
    }

    const monitorAppURL = "https://" + process.env.ZONE_NAME;
    const monitorAppString = process.env.MONITORING_STRING;
    const monitorServerFrom = process.env.ZONE_NAME;
    const monitoringEmail = process.env.MONITORING_EMAIL;
    const scheduleRuleName = `ScheduleMonitorRule-${monitorServerFrom}`;
    const mainScheduleRate = parseInt(process.env.MONITORING_RATE_MINS);

The Variables in the .env

USE_MONITORING_FEATURE=true
MONITORING_STRING=Per Hour
MONITORING_EMAIL=greenchapeljohn@gmail.com
MONITORING_RATE_MINS=10

First we are going to create the DynamoDB tables, in CDK this is nice and easy. We are going to use billing mode as PAY_PER_REQUEST and to DESTROY the table on stack removal.

We need two tables one to store the historic data and one to store the latest values.

    const ddbResultsTable = new dynamodb.Table(this, 'MonitorResultsTable', {
      partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: RemovalPolicy.DESTROY
    });
    const ddbLatestResultsTable = new dynamodb.Table(this, 'MonitorLatestResultsTable', {
      partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: RemovalPolicy.DESTROY
    });

Next we create the SNS topic, to create the topic is simple and we can also easily add some subscriptions to this topic. Below we are adding an SMS subscription and an Email subscription. When an event fires on the topic the subscribed users will be notified with our simple message.

    const monitorTopic = new sns.Topic(this, 'NotifySNS');
    monitorTopic.addSubscription(new subscriptions.SmsSubscription('+440000000000'));
    monitorTopic.addSubscription(new subscriptions.EmailSubscription(monitoringEmail));

Next we have the Scheduler, for this we use EventBridge and in event bridge we create a schedule rule, this can be done with rate or a cron expression, below we have used rate.

    // Schedular
    const cronEvent = new events.Rule(this, 'MonitorAppScheduleRule', {
      ruleName: scheduleRuleName,
      schedule: events.Schedule.rate(Duration.minutes(mainScheduleRate))
    });

Now we have most of the surrounding infrastructure setup we need to create the Lambda function for running the check. I like to write all my code in typescript so I’m using a Lambda Nodejs Function. For the http request in the lambda I also import ‘got’ and pass in all the needed environment variables to the lambda.

// Function called by schedular to Monitor the site
    const runMonitorCheckFn = new lambdaNode.NodejsFunction(this, 'MonitorCheckHandler', {
      runtime: lambda.Runtime.NODEJS_14_X,
      entry: 'lambda/monitoring/monitorSites.ts',
      handler: 'main',
      timeout: Duration.seconds(30),
      bundling: {
        externalModules: [
        ],
        nodeModules: [
          'got'
        ],
        minify: true
      },
      environment: {
        "monitorTopicArn": monitorTopic.topicArn,
        "resultsHistoricTable": ddbResultsTable.tableName,
        "resultsLatestTable": ddbLatestResultsTable.tableName,
        "monitorUrl": monitorAppURL,
        "monitorTitle": monitorAppString,
        "monitorServer": monitorServerFrom,
        "eventSchedularName": scheduleRuleName,
        "eventSchedularRate": mainScheduleRate.toString(),
        region: process.env.CDK_DEFAULT_REGION!
      }
    });

We will jump into the actual lambda code later, before we do we must setup the permissions. First we will give the lambda we just created Publish permissions on the monitoring SNS topic. Then the two DynamoDB tables, the Historic Table one needing just write and the Latest Values Table needing read / write.

Next we need to create our own inline policy, this is used to allow the lambda to adjust the rate that the EventBridge Schedule rule is triggered.

    monitorTopic.grantPublish(runMonitorCheckFn);
    ddbResultsTable.grantWriteData(runMonitorCheckFn);
    ddbLatestResultsTable.grantReadWriteData(runMonitorCheckFn);

    const changeCronRatePolicy = new iam.PolicyStatement({
      actions: ['events:DescribeRule', 'events:PutRule'],
      resources: [cronEvent.ruleArn],
    });

    runMonitorCheckFn.role?.attachInlinePolicy(new iam.Policy(this, 'Monitoring-Change-Cron-Rate-Policy', {
      statements:[changeCronRatePolicy]
    }));

Finally we need to add the lambda as the target for the EventBridge Schedule Rule.

    cronEvent.addTarget(new targets.LambdaFunction(runMonitorCheckFn));

The Full CDK Code

import { Construct } from 'constructs';
import {
  aws_lambda as lambda,
  aws_lambda_nodejs as lambdaNode,
  aws_sns as sns,
  aws_sns_subscriptions as subscriptions,
  aws_dynamodb as dynamodb,
  aws_events as events,
  aws_events_targets as targets,
  aws_iam as iam,
  Stack,
  StackProps,
  Duration,
  RemovalPolicy,
} from 'aws-cdk-lib';
require('custom-env').env(process.env.NODE_ENV);

export class CDKMonitoringStack extends Stack {
  constructor(scope: Construct, id: string, props?: StackProps) {
    super(scope, id, props);

  if(process.env.USE_MONITORING_FEATURE === 'true'){

    // Some Checks to make sure the environment variables are setup correctly.
    if (process.env.ZONE_NAME === undefined) {
      throw new Error(( 'Zone undefined'));
    }
    if (process.env.MONITORING_STRING === undefined) {
      throw new Error(( 'MONITORING_STRING undefined'));
    }
    if (process.env.MONITORING_EMAIL === undefined) {
      throw new Error(( 'MONITORING_EMAIL undefined'));
    }
    if (process.env.MONITORING_RATE_MINS === undefined) {
      throw new Error(( 'MONITORING_RATE_MINS undefined'));
    }

    const monitorAppURL = "https://" + process.env.ZONE_NAME;
    const monitorAppString = process.env.MONITORING_STRING;
    const monitorServerFrom = process.env.ZONE_NAME;
    const monitoringEmail = process.env.MONITORING_EMAIL;
    const scheduleRuleName = `ScheduleMonitorRule-${monitorServerFrom}`;
    const mainScheduleRate = parseInt(process.env.MONITORING_RATE_MINS);



    const ddbResultsTable = new dynamodb.Table(this, 'MonitorResultsTable', {
      partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: RemovalPolicy.DESTROY
    });
    const ddbLatestResultsTable = new dynamodb.Table(this, 'MonitorLatestResultsTable', {
      partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
      billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
      removalPolicy: RemovalPolicy.DESTROY
    });

    const monitorTopic = new sns.Topic(this, 'NotifySNS');
    // monitorTopic.addSubscription(new subscriptions.SmsSubscription('+440000000000'));
    monitorTopic.addSubscription(new subscriptions.EmailSubscription(monitoringEmail));


    // Schedular
    const cronEvent = new events.Rule(this, 'MonitorAppScheduleRule', {
      ruleName: scheduleRuleName,
      schedule: events.Schedule.rate(Duration.minutes(mainScheduleRate))
    });

    // Function called by schedular to Monitor the site
    const runMonitorCheckFn = new lambdaNode.NodejsFunction(this, 'MonitorCheckHandler', {
      runtime: lambda.Runtime.NODEJS_14_X,
      entry: 'lambda/monitoring/monitorSites.ts',
      handler: 'main',
      timeout: Duration.seconds(30),
      bundling: {
        externalModules: [
        ],
        nodeModules: [
          'got'
        ],
        minify: true
      },
      environment: {
        "monitorTopicArn": monitorTopic.topicArn,
        "resultsHistoricTable": ddbResultsTable.tableName,
        "resultsLatestTable": ddbLatestResultsTable.tableName,
        "monitorUrl": monitorAppURL,
        "monitorTitle": monitorAppString,
        "monitorServer": monitorServerFrom,
        "eventSchedularName": scheduleRuleName,
        "eventSchedularRate": mainScheduleRate.toString(),
        region: process.env.CDK_DEFAULT_REGION!
      }
    });
    monitorTopic.grantPublish(runMonitorCheckFn);
    ddbResultsTable.grantWriteData(runMonitorCheckFn);
    ddbLatestResultsTable.grantReadWriteData(runMonitorCheckFn);
    const changeCronRatePolicy = new iam.PolicyStatement({
      actions: ['events:DescribeRule', 'events:PutRule'],
      resources: [cronEvent.ruleArn],
    });
    runMonitorCheckFn.role?.attachInlinePolicy(new iam.Policy(this, 'Monitoring-Change-Cron-Rate-Policy', {
      statements:[changeCronRatePolicy]
    }));

    cronEvent.addTarget(new targets.LambdaFunction(runMonitorCheckFn));

  }
  }


}

Next we have the Lambda code itself to run the website check.

The website check is made up of two parts checking if we do a http get on the website we get anything and then looking for a specific string on the returned page.

First of we have the monitoring class, this is used by the handler. You are likely to create your own but this class will do the following.

  • Go and get the website data with a GET request
  • Check the String we are looking for is in the page
  • Write the results to the historic Dynamo table
  • Get the latest values from the DynamoDB table
  • Write the Latest Values to the Current values table
  • Publish the SNS Event
  • Change the EventBridge polling rate
const aws = require('aws-sdk');
const got = require('got');
import {DynamoDB} from 'aws-sdk';


export class MonitoringClass {

  // Go and Get the Website data with a GET request
    async PollWebsite(url: string){
        // Post request to get the token
        try {
          const token = await got.get( url);
          return {ok: true, response: token.body};
        } catch (error) {
          console.log('error', error);
          return {ok: false, response: error};
        }

    }

  // Check the String we are looking for is in the page
    CheckForString(str: string, title: string) {
        return str.includes(title);
    }

  // Write the results to the historic Dynamo table
    async WriteToHistoricDB(urlStr: string, titleStr: string, statusB: boolean, infoStr: string) {
        try{
            const documentClient = new DynamoDB.DocumentClient();
            const params: DynamoDB.DocumentClient.PutItemInput = {
                TableName : process.env.resultsHistoricTable!,
                Item: {
                    ID: Date.now().toString(),
                    url: urlStr,
                    title: titleStr,
                    status: statusB,
                    info: infoStr
                }
            }
            const data = await documentClient.put(params).promise();
        }
        catch(e) {
            throw new Error('Failed to write to DB');
        }
    }

    // Get the latest values from the DynamoDB table
      async GetLastValues(url: string) {
          const documentClient = new DynamoDB.DocumentClient();
          const params: DynamoDB.DocumentClient.GetItemInput = {
              TableName: process.env.resultsLatestTable!,
              Key: {
                  ID: url
              }
          };
          try{
              const result = await documentClient.get(params).promise();
              return result;
          } catch (e) {
              throw new Error('Failed to get last values');
          }
      }

      // Write the Latest Values to the Current values table
      async WriteLatestValue(urlStr: string, statusB: boolean, infoStr: string) {
          try{
              const documentClient = new DynamoDB.DocumentClient();
              const params: DynamoDB.DocumentClient.PutItemInput = {
                  TableName : process.env.resultsLatestTable!,
                  Item: {
                      ID: urlStr,
                      time: Date.now().toString(),
                      status: statusB,
                      info: infoStr
                  }
              }
              const data = await documentClient.put(params).promise();
              console.log('Added ', data);
          }
          catch(e) {
              throw new Error('Failed to write to DB');
          }
      }

    // Publish the SNS Event
    async SendEventSNS(message: string, subject: string, topicArn: string, from: string){
      // Note for Text messages sender ID must be less that 11 chars
        if (from.length > 11) {
            throw new Error('SenderID > 11 char');
        }
        try{
            var sns = new aws.SNS();
            var params = {
                Message: message,
                Subject: subject,
                TopicArn: topicArn,
                MessageAttributes: {
                "AWS.SNS.SMS.SenderID": {
                    "DataType":"String",
                    "StringValue": from
                }
                }
            };
            const pub = await sns.publish(params).promise();
        } catch (e) {
            throw new Error('Failed to write to DB');
        }
    }


    // Change the EventBridge polling rate
    async UpdatePollRate(rate: number) {
        const eventBridge = new aws.EventBridge();

        const params = {
            Name: process.env.eventSchedularName!
        };

        const getRule = await eventBridge.describeRule(params).promise();

        let expression = `rate(${rate} minutes)`
        if (rate === 1) {
            expression = `rate(${rate} minute)`
        }

        const newParams = {
            Name: getRule.Name,
            ScheduleExpression: expression
        };
        const updateRule = await eventBridge.putRule(newParams).promise();
    }

}

With the class in place we can then create the Lambda handler. The following lambda runs as follows. You will need the packages aws-lambda, @types/aws-lambda and got installed.

  • Setup the constants passed into the process as env variables
  • Get the previous test results
  • Run the GET request on the URL
    • If this passes, Check for string
      • If string found then log and set polling rate back to pre defined rate
      • If not found, Send notification log to Dynamo and set the polling rate to 1 minute.
    • If this fails, Send notification log to Dynamo and set the polling rate to 1 minute.
import { Context, Callback } from 'aws-lambda';
import { MonitoringClass } from './MonitoringClass';

export async function main(event: any, _context: Context, callback: Callback) {

  // const evBody = JSON.parse(event.body);
  const monitorTopic = process.env.monitorTopicArn!;

  const Monitoring = new MonitoringClass();
  const url = process.env.monitorUrl!;
  const title = process.env.monitorTitle!;
  const server = process.env.monitorServer!;
  const eventSchedularRate: number = parseInt(process.env.eventSchedularRate!);

  try {

    // Get the previous test results
    const getLastValues = await Monitoring.GetLastValues(url);

    // Run the GET request
    const poll = await Monitoring.PollWebsite(url);
    if (poll.ok) {
      // Find the String we are looking for ( usually page title)
      const containsString = await Monitoring.CheckForString(poll.response, title);
      if (containsString) {

        // Site Found
        await Monitoring.WriteToHistoricDB(url, title, true, 'ok');

        // IF changes alsos send SNS and update the poll rate back to the predefined one
        if (getLastValues!.Item!.status === false) {
          await Monitoring.WriteLatestValue(url, true, 'ok');
          await Monitoring.SendEventSNS(`${url} Restored`, `${url} Restored`, monitorTopic, server);
          await Monitoring.UpdatePollRate(eventSchedularRate);
        }

        return {
          statusCode: 200,
          headers: { "Content-Type": "application/json" },
          body: JSON.stringify({ message: "Complete" })
        };

      } else {
        // No Title found
        await Monitoring.WriteToHistoricDB(url, title, false, 'No Title');

        //IF changes also send SNS and update the poll rate to run every 1 minute
        if (getLastValues!.Item!.status === true) {
          await Monitoring.WriteLatestValue(url, false, 'Website Down');
          await Monitoring.SendEventSNS(`${url} is not reachable`, `${url} is not reachable`, monitorTopic, server);
          await Monitoring.UpdatePollRate(1);
        }
        return {
          statusCode: 400,
          headers: { "Content-Type": "application/json" },
          body: "Title Not Found"
        };
      }
    } else {
      // Website Down
      await Monitoring.WriteToHistoricDB(url, title, false, 'URL Not Found');

      //IF changes also send SNS and update the poll rate to run every 1 minute
      if (getLastValues!.Item!.status === true) {
        await Monitoring.WriteLatestValue(url, false, 'Website Down');
        await Monitoring.SendEventSNS(`${url} is not reachable`, `${url} is not reachable`, monitorTopic, server);
        await Monitoring.UpdatePollRate(1);
      }

      return {
        statusCode: 400,
        headers: { "Content-Type": "application/json" },
        body: 'Website Down'
      };
    }



  } catch (e) {

    const error: any = e;
    console.log('error ', error);
    return {
      statusCode: 400,
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify(error)
    };
  }
}




Categories: AWS

Tags: , , , , , , , , ,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: