Pretty AlertManager Alerts in Slack

Prometheus alerting through AlertManager can be configured to send messages to Slack.

Alerts sent to Slack, by default, are not very rich in detail:

The default configuration for Slack in the AlertManager documentation is set under receivers:

receivers:
- name: slack_webhook
  slack_configs:
  - send_resolved: false
    api_url: https://something.slack.com/APIURLSECRET
    channel: alerts
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "slack.default.title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ template "slack.default.pretext" . }}'
    text: '{{ template "slack.default.text" . }}'
    fallback: '{{ template "slack.default.fallback" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: '{{ template "slack.default.iconurl" . }}'
templates: []

To get more details around alerts as displayed in a Slack channel, make use of the label annotations in AlertManager. Here is an updated, better receivers block to use when sending alerts to Slack.

receivers:
- name: slack_webhook
  slack_configs:
  - send_resolved: false
    api_url: https://something.slack.com/APIURLSECRET
    channel: alerts
    username: '{{ template "slack.default.username" . }}'
    color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
    title: '{{ template "slack.default.title" . }}'
    title_link: '{{ template "slack.default.titlelink" . }}'
    pretext: '{{ .CommonAnnotations.summary }}'
    text: |-
      {{ range .Alerts }}
         *Alert:* {{ .Annotations.summary }} - `{{ .Labels.severity }}`
        *Description:* {{ .Annotations.description }}
        *Details:*
        {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
        {{ end }}
      {{ end }}
    fallback: '{{ template "slack.default.fallback" . }}'
    icon_emoji: '{{ template "slack.default.iconemoji" . }}'
    icon_url: '{{ template "slack.default.iconurl" . }}'
templates: '/etc/alertmanager/config/*.tmpl'

This assumes your alerts are grouped. Notice the use of {{ range .Alerts }} which will show detailed information on all alerts that are firing.

Assuming this is for CoreOS’s Tectonic, this AlertManager template will need to be added to the AlertManager Kubernetes secret (as default.tmpl):

{{ define "__alertmanager" }}Cluster: CLUSTER_NAME{{ end }}
{{ define "__alertmanagerURL" }}{{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}{{ end }}

{{ define "__subject" }}[{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .GroupLabels.SortedPairs.Values | join " " }} {{ if gt (len .CommonLabels) (len .GroupLabels) }}({{ with .CommonLabels.Remove .GroupLabels.Names }}{{ .Values | join " " }}{{ end }}){{ end }}{{ end }}
{{ define "__description" }}{{ end }}

{{ define "__text_alert_list" }}{{ range . }}Labels:
{{ range .Labels.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Annotations:
{{ range .Annotations.SortedPairs }} - {{ .Name }} = {{ .Value }}
{{ end }}Source: {{ .GeneratorURL }}
{{ end }}{{ end }}


{{ define "slack.default.title" }}{{ template "__subject" . }}{{ end }}
{{ define "slack.default.username" }}{{ template "__alertmanager" . }}{{ end }}
{{ define "slack.default.fallback" }}{{ template "slack.default.title" . }} | {{ template "slack.default.titlelink" . }}{{ end }}
{{ define "slack.default.pretext" }}{{ end }}
{{ define "slack.default.titlelink" }}{{ template "__alertmanagerURL" . }}{{ end }}
{{ define "slack.default.iconemoji" }}{{ end }}
{{ define "slack.default.iconurl" }}{{ end }}
{{ define "slack.default.text" }}{{ end }}
{{ define "slack.default.footer" }}{{ end }}

Multiple Clusters

Some will have multiple Kubernetes clusters sending alerts to the same Slack channel. For example: a platform team managing clusters for different internal app teams. In this case, set an external label in the configuration in each cluster, as documented here under PrometheusK8sConfig:

externalLabels:
  [ - cluster: production ]

The extra label will be added to alerts sent by AlertManager to Slack in this block in receivers:

{{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
{{ end }}

Finished Product

Alerts sent to Slack will now look closer to this:

Special Thanks

This post is cobbled together from a few places. This post from InfinityWorks and this GitHub issue were very helpful.