Configuring Prometheus AM Executor for automation

with every prometheus alert comes great responsibility!

In this quick write-up We will go through on configuring Prometheus AM Executor with a custom script to automate tasks according to the Prometheus alert label.

Image 1.0 — Totally unnecessary Image I added as a ‘story preview’ is needed.

Totally Legally valid Disclaimer:

In here I won’t be going through setting up the Prometheus Server, Node exporters or Alert Manager, But I will show the alert manager configurations in order to route the alerts to the prometheus AM Executor.

Action Plan:

In my current setup when there is a Prometheus Alert, that alert notification is routed to one of our Slack channels using Alert manager configurations. What we are going to do is using this alert to run a script which will automate a task. In order to do that we will be using Prometheus AM Executor and will implement a scenario where a custom Disk cleanup script is executed when a Disk_Low alert is fired from the Prometheus Server. The connection between the Prometheus Server and the AM executor will happen via the Alert Manager service.

TLDR:

The Prometheus Server will run at the port 9090. The Alert Manger service will capture the Prometheus alerts and route them to a Slack Channel and to the AM Executor service which will be running on port 8088. Once the AM Executor receives an alert (In the Json format) it will execute a pre written script. In this script you can write any logic to achieve a task using the already shipped inbuilt Variables.

Action:

1. Configuring the Prometheus AM Executor.

Prerequisite: Go language [Go figure this out from your own!]

Download the zip file from the official GH repo to the server where you need to run the Executor service. I will be using the same server where both the Prometheus Server and the AlertManager is running.

Unzip the .zip file and go inside the downloaded directory and run below commands.

go test -count 1 -v ./…
go build

This will create an executable file named prometheus-am-executor.

If you simply run this executable (ie: ./prometheus-am-executor) it will start running on the default port (Port 8080). We need to run this on a specific port and we need to trigger a custom script when there is an alert.

We can simple do that by executing below.

./prometheus-am-executor -l “localhost:8088” -v ./sample-script.sh
l — Specifying a custom port
-v — Verbose Output

2. Let’s configure the custom script.

We can use the inbuilt variables shipped with the Executor Service. I have used only the below values.

• AMX_RECEIVER: name of receiver in the AM triggering the alert — We will talk about this when we configure the Alertmanager.
• AMX_STATUS: alert status, Firing or Resolved.
• AMX_EXTERNAL_URL: URL to reach alertmanager — Not needed in our case, but if our executor service handles notifications from more than one alertmanager this will come handy!
• AMX_LABEL_<label>: alert label pairs — <label> can be found from the Prometheus Server. In this example I will using the ‘alertname’ label and the ‘instance’ label which is already configured in the Prometheus side. These are key value pairs and the label is associated with the current status/source instance of the alert.

ie:

AMX_LABEL_alertname=Low_Disk_Space or 
AMX_LABEL_alertname=Instance_Down
AMX_LABEL_instance=172.10.3.105 #The instance which is firing the alert

Sample Script

3. Testing the Executor Service

We can feed a dummy alert and see whether the script is triggered. We will be starting the Executor service with nohup and then we will use the given example json.

nohup ./prometheus-am-executor -l “localhost:8088” -v ./sample-script.sh &

Then we can edit the file inside examples directory and change the json key pair values to match our code logics and edge cases .

vim examples/alert_payload.json
#replicate a dummy alert using the alert_payload.json
curl --include -H 'Content-Type: application/json' --data-binary "@examples/alert_payload.json" -X GET 'http://localhost:8088/'

Above curl should invoke the AM executor service and we should be able to see the relevant logs in the nohup.out as well.

Now We are sure that if an alert is received by the executor (Port 8088) the custom script is triggered. Then we need to point real Prometheus Alerts to be routed to this service.

4. Routing the Prometheus Alerts to the AM Executor service.

First we need to find the Configuration file used with the Alertmanager service.

ps -aux | grep alertmanager
alertma+ 1095 0.1 0.0 33816 66348 ? SLsl Jun02 242:08 /usr/local/bin/alertmanager — config.file=/etc/alertmanager/alertmanager.yml — web.external-url http://localhost:9093

According to the output we can identify that the Configuration file is /etc/alertmanager/alertmanager.yml.

The current configuration looks like this.

We need to add another receiver for AM Executor. This is the receiver we mentioned in the AMX_RECEIVER label. Only after adding a new receiver which forwards alert notification to the AM Executor service, AM Executor can trigger the script.

Find the new Alertmanager.yml which forwards alert notifications to both Slack and the AM Executor.

Now the connectivity is made, Alertmanager will send alert notification json to the AM Executor service and once a JSON payload is received it will trigger the custom script. Using the inbuilt variable values we can run a script to automate tasks.

Heck, this is long!

Find me on LinkedIn and bash me with your thoughts!