I recently have been playing around with some tools for monitoring websites but with these tools they can be an unnecessary expense for a start-up. So, I went and built my own monitoring stack in AWS CDK to monitor if my website was running and alert me of any issues.
I’m going to assume you know a bit about CDK and jump right into the stack code and what it is made up of.
The stack is going to contain the following services:
- Dynamo DB – this is used to store a history of the website status and the current status.
- SNS – used to notify the subscribed users that the website has gone down or has come back up (by text or email)
- Event Bridge – the scheduler.
- Lambda – to run the check, write the results and publish to SNS.
First of the imports I have used, I like to shorten the import names so the code is a bit easier to read.
import { Construct } from 'constructs';
import {
aws_lambda as lambda,
aws_lambda_nodejs as lambdaNode,
aws_sns as sns,
aws_sns_subscriptions as subscriptions,
aws_dynamodb as dynamodb,
aws_events as events,
aws_events_targets as targets,
aws_iam as iam,
Stack,
StackProps,
Duration,
RemovalPolicy,
} from 'aws-cdk-lib';
require('custom-env').env(process.env.NODE_ENV);
Next the Variables are setup, I like to do this mostly at the top so its nice and clear, I am also using a .env file in this case to declare all the values.
if(process.env.USE_MONITORING_FEATURE === 'true'){
// Some Checks to make sure the environment variables are setup correctly.
if (process.env.ZONE_NAME === undefined) {
throw new Error(( 'Zone undefined'));
}
if (process.env.MONITORING_STRING === undefined) {
throw new Error(( 'MONITORING_STRING undefined'));
}
if (process.env.MONITORING_EMAIL === undefined) {
throw new Error(( 'MONITORING_EMAIL undefined'));
}
if (process.env.MONITORING_RATE_MINS === undefined) {
throw new Error(( 'MONITORING_RATE_MINS undefined'));
}
const monitorAppURL = "https://" + process.env.ZONE_NAME;
const monitorAppString = process.env.MONITORING_STRING;
const monitorServerFrom = process.env.ZONE_NAME;
const monitoringEmail = process.env.MONITORING_EMAIL;
const scheduleRuleName = `ScheduleMonitorRule-${monitorServerFrom}`;
const mainScheduleRate = parseInt(process.env.MONITORING_RATE_MINS);
The Variables in the .env
USE_MONITORING_FEATURE=true
MONITORING_STRING=Per Hour
MONITORING_EMAIL=greenchapeljohn@gmail.com
MONITORING_RATE_MINS=10
First we are going to create the DynamoDB tables, in CDK this is nice and easy. We are going to use billing mode as PAY_PER_REQUEST and to DESTROY the table on stack removal.
We need two tables one to store the historic data and one to store the latest values.
const ddbResultsTable = new dynamodb.Table(this, 'MonitorResultsTable', {
partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
removalPolicy: RemovalPolicy.DESTROY
});
const ddbLatestResultsTable = new dynamodb.Table(this, 'MonitorLatestResultsTable', {
partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
removalPolicy: RemovalPolicy.DESTROY
});
Next we create the SNS topic, to create the topic is simple and we can also easily add some subscriptions to this topic. Below we are adding an SMS subscription and an Email subscription. When an event fires on the topic the subscribed users will be notified with our simple message.
const monitorTopic = new sns.Topic(this, 'NotifySNS');
monitorTopic.addSubscription(new subscriptions.SmsSubscription('+440000000000'));
monitorTopic.addSubscription(new subscriptions.EmailSubscription(monitoringEmail));
Next we have the Scheduler, for this we use EventBridge and in event bridge we create a schedule rule, this can be done with rate or a cron expression, below we have used rate.
// Schedular
const cronEvent = new events.Rule(this, 'MonitorAppScheduleRule', {
ruleName: scheduleRuleName,
schedule: events.Schedule.rate(Duration.minutes(mainScheduleRate))
});
Now we have most of the surrounding infrastructure setup we need to create the Lambda function for running the check. I like to write all my code in typescript so I’m using a Lambda Nodejs Function. For the http request in the lambda I also import ‘got’ and pass in all the needed environment variables to the lambda.
// Function called by schedular to Monitor the site
const runMonitorCheckFn = new lambdaNode.NodejsFunction(this, 'MonitorCheckHandler', {
runtime: lambda.Runtime.NODEJS_14_X,
entry: 'lambda/monitoring/monitorSites.ts',
handler: 'main',
timeout: Duration.seconds(30),
bundling: {
externalModules: [
],
nodeModules: [
'got'
],
minify: true
},
environment: {
"monitorTopicArn": monitorTopic.topicArn,
"resultsHistoricTable": ddbResultsTable.tableName,
"resultsLatestTable": ddbLatestResultsTable.tableName,
"monitorUrl": monitorAppURL,
"monitorTitle": monitorAppString,
"monitorServer": monitorServerFrom,
"eventSchedularName": scheduleRuleName,
"eventSchedularRate": mainScheduleRate.toString(),
region: process.env.CDK_DEFAULT_REGION!
}
});
We will jump into the actual lambda code later, before we do we must setup the permissions. First we will give the lambda we just created Publish permissions on the monitoring SNS topic. Then the two DynamoDB tables, the Historic Table one needing just write and the Latest Values Table needing read / write.
Next we need to create our own inline policy, this is used to allow the lambda to adjust the rate that the EventBridge Schedule rule is triggered.
monitorTopic.grantPublish(runMonitorCheckFn);
ddbResultsTable.grantWriteData(runMonitorCheckFn);
ddbLatestResultsTable.grantReadWriteData(runMonitorCheckFn);
const changeCronRatePolicy = new iam.PolicyStatement({
actions: ['events:DescribeRule', 'events:PutRule'],
resources: [cronEvent.ruleArn],
});
runMonitorCheckFn.role?.attachInlinePolicy(new iam.Policy(this, 'Monitoring-Change-Cron-Rate-Policy', {
statements:[changeCronRatePolicy]
}));
Finally we need to add the lambda as the target for the EventBridge Schedule Rule.
cronEvent.addTarget(new targets.LambdaFunction(runMonitorCheckFn));
The Full CDK Code
import { Construct } from 'constructs';
import {
aws_lambda as lambda,
aws_lambda_nodejs as lambdaNode,
aws_sns as sns,
aws_sns_subscriptions as subscriptions,
aws_dynamodb as dynamodb,
aws_events as events,
aws_events_targets as targets,
aws_iam as iam,
Stack,
StackProps,
Duration,
RemovalPolicy,
} from 'aws-cdk-lib';
require('custom-env').env(process.env.NODE_ENV);
export class CDKMonitoringStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);
if(process.env.USE_MONITORING_FEATURE === 'true'){
// Some Checks to make sure the environment variables are setup correctly.
if (process.env.ZONE_NAME === undefined) {
throw new Error(( 'Zone undefined'));
}
if (process.env.MONITORING_STRING === undefined) {
throw new Error(( 'MONITORING_STRING undefined'));
}
if (process.env.MONITORING_EMAIL === undefined) {
throw new Error(( 'MONITORING_EMAIL undefined'));
}
if (process.env.MONITORING_RATE_MINS === undefined) {
throw new Error(( 'MONITORING_RATE_MINS undefined'));
}
const monitorAppURL = "https://" + process.env.ZONE_NAME;
const monitorAppString = process.env.MONITORING_STRING;
const monitorServerFrom = process.env.ZONE_NAME;
const monitoringEmail = process.env.MONITORING_EMAIL;
const scheduleRuleName = `ScheduleMonitorRule-${monitorServerFrom}`;
const mainScheduleRate = parseInt(process.env.MONITORING_RATE_MINS);
const ddbResultsTable = new dynamodb.Table(this, 'MonitorResultsTable', {
partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
removalPolicy: RemovalPolicy.DESTROY
});
const ddbLatestResultsTable = new dynamodb.Table(this, 'MonitorLatestResultsTable', {
partitionKey: {name:'ID', type: dynamodb.AttributeType.STRING},
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
removalPolicy: RemovalPolicy.DESTROY
});
const monitorTopic = new sns.Topic(this, 'NotifySNS');
// monitorTopic.addSubscription(new subscriptions.SmsSubscription('+440000000000'));
monitorTopic.addSubscription(new subscriptions.EmailSubscription(monitoringEmail));
// Schedular
const cronEvent = new events.Rule(this, 'MonitorAppScheduleRule', {
ruleName: scheduleRuleName,
schedule: events.Schedule.rate(Duration.minutes(mainScheduleRate))
});
// Function called by schedular to Monitor the site
const runMonitorCheckFn = new lambdaNode.NodejsFunction(this, 'MonitorCheckHandler', {
runtime: lambda.Runtime.NODEJS_14_X,
entry: 'lambda/monitoring/monitorSites.ts',
handler: 'main',
timeout: Duration.seconds(30),
bundling: {
externalModules: [
],
nodeModules: [
'got'
],
minify: true
},
environment: {
"monitorTopicArn": monitorTopic.topicArn,
"resultsHistoricTable": ddbResultsTable.tableName,
"resultsLatestTable": ddbLatestResultsTable.tableName,
"monitorUrl": monitorAppURL,
"monitorTitle": monitorAppString,
"monitorServer": monitorServerFrom,
"eventSchedularName": scheduleRuleName,
"eventSchedularRate": mainScheduleRate.toString(),
region: process.env.CDK_DEFAULT_REGION!
}
});
monitorTopic.grantPublish(runMonitorCheckFn);
ddbResultsTable.grantWriteData(runMonitorCheckFn);
ddbLatestResultsTable.grantReadWriteData(runMonitorCheckFn);
const changeCronRatePolicy = new iam.PolicyStatement({
actions: ['events:DescribeRule', 'events:PutRule'],
resources: [cronEvent.ruleArn],
});
runMonitorCheckFn.role?.attachInlinePolicy(new iam.Policy(this, 'Monitoring-Change-Cron-Rate-Policy', {
statements:[changeCronRatePolicy]
}));
cronEvent.addTarget(new targets.LambdaFunction(runMonitorCheckFn));
}
}
}
Next we have the Lambda code itself to run the website check.
The website check is made up of two parts checking if we do a http get on the website we get anything and then looking for a specific string on the returned page.
First of we have the monitoring class, this is used by the handler. You are likely to create your own but this class will do the following.
- Go and get the website data with a GET request
- Check the String we are looking for is in the page
- Write the results to the historic Dynamo table
- Get the latest values from the DynamoDB table
- Write the Latest Values to the Current values table
- Publish the SNS Event
- Change the EventBridge polling rate
const aws = require('aws-sdk');
const got = require('got');
import {DynamoDB} from 'aws-sdk';
export class MonitoringClass {
// Go and Get the Website data with a GET request
async PollWebsite(url: string){
// Post request to get the token
try {
const token = await got.get( url);
return {ok: true, response: token.body};
} catch (error) {
console.log('error', error);
return {ok: false, response: error};
}
}
// Check the String we are looking for is in the page
CheckForString(str: string, title: string) {
return str.includes(title);
}
// Write the results to the historic Dynamo table
async WriteToHistoricDB(urlStr: string, titleStr: string, statusB: boolean, infoStr: string) {
try{
const documentClient = new DynamoDB.DocumentClient();
const params: DynamoDB.DocumentClient.PutItemInput = {
TableName : process.env.resultsHistoricTable!,
Item: {
ID: Date.now().toString(),
url: urlStr,
title: titleStr,
status: statusB,
info: infoStr
}
}
const data = await documentClient.put(params).promise();
}
catch(e) {
throw new Error('Failed to write to DB');
}
}
// Get the latest values from the DynamoDB table
async GetLastValues(url: string) {
const documentClient = new DynamoDB.DocumentClient();
const params: DynamoDB.DocumentClient.GetItemInput = {
TableName: process.env.resultsLatestTable!,
Key: {
ID: url
}
};
try{
const result = await documentClient.get(params).promise();
return result;
} catch (e) {
throw new Error('Failed to get last values');
}
}
// Write the Latest Values to the Current values table
async WriteLatestValue(urlStr: string, statusB: boolean, infoStr: string) {
try{
const documentClient = new DynamoDB.DocumentClient();
const params: DynamoDB.DocumentClient.PutItemInput = {
TableName : process.env.resultsLatestTable!,
Item: {
ID: urlStr,
time: Date.now().toString(),
status: statusB,
info: infoStr
}
}
const data = await documentClient.put(params).promise();
console.log('Added ', data);
}
catch(e) {
throw new Error('Failed to write to DB');
}
}
// Publish the SNS Event
async SendEventSNS(message: string, subject: string, topicArn: string, from: string){
// Note for Text messages sender ID must be less that 11 chars
if (from.length > 11) {
throw new Error('SenderID > 11 char');
}
try{
var sns = new aws.SNS();
var params = {
Message: message,
Subject: subject,
TopicArn: topicArn,
MessageAttributes: {
"AWS.SNS.SMS.SenderID": {
"DataType":"String",
"StringValue": from
}
}
};
const pub = await sns.publish(params).promise();
} catch (e) {
throw new Error('Failed to write to DB');
}
}
// Change the EventBridge polling rate
async UpdatePollRate(rate: number) {
const eventBridge = new aws.EventBridge();
const params = {
Name: process.env.eventSchedularName!
};
const getRule = await eventBridge.describeRule(params).promise();
let expression = `rate(${rate} minutes)`
if (rate === 1) {
expression = `rate(${rate} minute)`
}
const newParams = {
Name: getRule.Name,
ScheduleExpression: expression
};
const updateRule = await eventBridge.putRule(newParams).promise();
}
}
With the class in place we can then create the Lambda handler. The following lambda runs as follows. You will need the packages aws-lambda, @types/aws-lambda and got installed.
- Setup the constants passed into the process as env variables
- Get the previous test results
- Run the GET request on the URL
- If this passes, Check for string
- If string found then log and set polling rate back to pre defined rate
- If not found, Send notification log to Dynamo and set the polling rate to 1 minute.
- If this fails, Send notification log to Dynamo and set the polling rate to 1 minute.
- If this passes, Check for string
import { Context, Callback } from 'aws-lambda';
import { MonitoringClass } from './MonitoringClass';
export async function main(event: any, _context: Context, callback: Callback) {
// const evBody = JSON.parse(event.body);
const monitorTopic = process.env.monitorTopicArn!;
const Monitoring = new MonitoringClass();
const url = process.env.monitorUrl!;
const title = process.env.monitorTitle!;
const server = process.env.monitorServer!;
const eventSchedularRate: number = parseInt(process.env.eventSchedularRate!);
try {
// Get the previous test results
const getLastValues = await Monitoring.GetLastValues(url);
// Run the GET request
const poll = await Monitoring.PollWebsite(url);
if (poll.ok) {
// Find the String we are looking for ( usually page title)
const containsString = await Monitoring.CheckForString(poll.response, title);
if (containsString) {
// Site Found
await Monitoring.WriteToHistoricDB(url, title, true, 'ok');
// IF changes alsos send SNS and update the poll rate back to the predefined one
if (getLastValues!.Item!.status === false) {
await Monitoring.WriteLatestValue(url, true, 'ok');
await Monitoring.SendEventSNS(`${url} Restored`, `${url} Restored`, monitorTopic, server);
await Monitoring.UpdatePollRate(eventSchedularRate);
}
return {
statusCode: 200,
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ message: "Complete" })
};
} else {
// No Title found
await Monitoring.WriteToHistoricDB(url, title, false, 'No Title');
//IF changes also send SNS and update the poll rate to run every 1 minute
if (getLastValues!.Item!.status === true) {
await Monitoring.WriteLatestValue(url, false, 'Website Down');
await Monitoring.SendEventSNS(`${url} is not reachable`, `${url} is not reachable`, monitorTopic, server);
await Monitoring.UpdatePollRate(1);
}
return {
statusCode: 400,
headers: { "Content-Type": "application/json" },
body: "Title Not Found"
};
}
} else {
// Website Down
await Monitoring.WriteToHistoricDB(url, title, false, 'URL Not Found');
//IF changes also send SNS and update the poll rate to run every 1 minute
if (getLastValues!.Item!.status === true) {
await Monitoring.WriteLatestValue(url, false, 'Website Down');
await Monitoring.SendEventSNS(`${url} is not reachable`, `${url} is not reachable`, monitorTopic, server);
await Monitoring.UpdatePollRate(1);
}
return {
statusCode: 400,
headers: { "Content-Type": "application/json" },
body: 'Website Down'
};
}
} catch (e) {
const error: any = e;
console.log('error ', error);
return {
statusCode: 400,
headers: { "Content-Type": "application/json" },
body: JSON.stringify(error)
};
}
}
Check it out on GitHub
Categories: AWS
greenchapel.dev 
Leave a Reply