Tag Cloud
Concourse CI/CD devops prometheus kubernetes monitoring modbus kepware c-programming IoT golang telegram bot python cli urwid elasticsearch aws ecs apache spark scala AWS EMR hadoop webhooks ssl nginx digital-ocean emr apache pig datapipeline found.io elastic-cloud rails try capybara docker capistrano heka bigquery kafka protobuf vim iterm javascript emberjs git scripting dnsmasq bem frontend meteorjs meteorite heroku

Monitoring Stack in Kubernetes, with Prometheus

For the past year or so, I’ve been working with DevOps in Tulip. It’s a fairly big change in direction but quite frankly, it’s been a refreshing experience!

One of the first projects was to build a monitoring system for a number of different components in our kubernetes cluster: various microservices, main monolith application, our ingress controller, and the health of the cluster itself. I thought Prometheus fit fairly well with what we wanted, so we went ahead with that. I would say that it has been served us pretty well!


Some context about Prometheus; it is a pull-based monitoring system that is sometimes compared to Nagios. It’s not event-based, so applications do not report each individual event to Prometheus as it happens (like SegmentIO). Also, since its pull-based, we have to define all its targets (to scrape) in advance. Contrary to certain arguments, I actually think this is a plus. It is more tedious and harder to set up, but it is also harder to get into a situation where it becomes a blackbox and you have no idea what you’re dumping into it. It’s also easier to detect if a target is actually down vs a push-based system.

Prometheus Operator

Prometheus Operator is an open-source tool that makes deploying a Prometheus stack (AlertManager, Grafana, Prometheus) so, much easier than hand-crafting the entire stack. It helps generate a whole lot of boiler plates and pretty much reduces the entire deployment down to native kubernetes declarations and YAML.

If you’re familiar with Kubernetes, then you’ve probably heard of custom resource definitions, or CRDs in short. Think of them as definitions of objects like pods, deployments or daemonsets that the cluster can understand and act on if needed. For the purpose of deploying a monitoring stack, Prometheus Operator introduces 3 new CRDs - Prometheus, AlertManager and ServiceMonitor, and a controller that is in-charge of deploying and connfiguring the respective services into the Kubernetes Cluster.

For example: if a prometheus CRD like the one below is present in the cluster, the prometheus-operator controller would create a matching deployment of Prometheus into the kubernetes cluster, that in this case would also link up with the alertmanager by that name in the monitoring namespace.

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
  name: clustermon-prometheus-oper-prometheus
  namespace: monitoring
    - name: clustermon-prometheus-oper-alertmanager
      namespace: monitoring
      pathPrefix: /
      port: web
  baseImage: quay.io/prometheus/prometheus


kube-prometheus used to be a set of contrib helm charts that utilizes the capabilities of the Prometheus Operator to deploy an entire monitoring stack (with some assumptions and defaults ofc). It has since been absorbed into the main helm charts and moved to the official stable chart repository.

There are various exporters included such as: kube-dns, kube-state-metrics, node-exporter and many others that are necessary to monitor the health of a Kubernetes cluster (and more). You can find the full list here. It also has a simple set of kubernetes-mixins for Grafana as well (if you choose to install that).


This section gives a general idea of the components involved.

An important implementation decision that I’ll like to point out is that Grafana is (mostly) stateless. Any new dashboards, or changes would need to be commited to code; in general I think this conforms better to the infrastructure-as-code kind of idealogy which makes it much easier to replicate the same infrastructure across multiple clouds / regions.

Monitoring Stack

Custom Helm Chart

You can find a stripped down version fo the things I’ll talk about in this repository: k8s-prometheus-operator-helm-example

Note: The contents are all based on prometheus-operator helm chart 5.10.5.

# requirements.yaml
  - name: prometheus-operator
    version: 5.10.5
    repository: https://kubernetes-charts.storage.googleapis.com/

This part of the chart is responsible for loading *.json dashboard configurations exported from Grafana and creating them as individual configmaps in Kubernetes. The config-reloaders in Grafana would then read them and reconfigure itself.

# templates/dashboards-configmap.yaml
{{- $files := .Files.Glob "dashboards/*.json" }}
apiVersion: v1
kind: ConfigMapList
{{- range $path, $fileContents := $files }}
{{- $dashboardName := regexReplaceAll "(^.*/)(.*)\\.json$" $path "${2}" }}
- apiVersion: v1
  kind: ConfigMap
    name: {{ printf "%s-%s" (include "prometheus-operator.fullname" $) $dashboardName | trunc 63 | trimSuffix "-" }}
      grafana_dashboard: "1"
      app: {{ template "prometheus-operator.name" $ }}-grafana
{{ include "prometheus-operator.labels" $ | indent 6 }}
    {{ $dashboardName }}.json: |-
{{ $.Files.Get $path | indent 6}}
{{- end }}


Prometheus rocks a TSDB for data storage so the instance that the pod runs on needs to have a huge volume attached to it. In my setup, I’ve chosen to run Prometheus on a node, by itself, with no other pods scheduled on it. I do this by setting up taints on a particular node and having Prometheus selectively schedule to that node and tolerate those taints. Normal pods without that toleration would then just refuse to schedule on it

(This is slightly different in the example app above)

# values.yaml

    # So that Prometheus can schedule onto the node with this taint
    # Other pods will not have this toleration and won't schedule on it
    - key: "dedicated"
      operator: "Exists"
      effect: "NoSchedule"
    - key: "dedicated"
      operator: "Exists"
      effect: "NoExecute"

    # The Prometheus node
      node: prometheus

    # This PV is named in my case, but you can also just do a dynamic claim template like here:
    # https://github.com/aranair/k8s-prometheus-operator-helm-example/blob/master/clustermon/values.yaml#L181-L189
          volumeName: prometheus-pv
              app: prometheus-pv
              storage: 1500Gi
      selector: {}

If you take a look at the prometheus-operator’s default [values.yaml][helm-chart-values] file, you will find just about any configuration you can think of.

Monitoring Custom Services

The ServiceMonitor CRD from the prometheus-operators is used to describe the set of targets to be monitored by Prometheus; the controller would automatically generate the Prometheus config needed.

For example, a ServiceMonitor for monitoring Traefik, our ingress controller would look something like this: additionalServiceMonitors: - name: traefik-monitor namespace: monitoring selector: matchLabels: app: traefik # this should be the selector for the Service namespaceSelector: matchNames: - kube-system # Which namespace to look for the Service in endpoints: - basicAuth: # Take creds from secret named traefik-monitor-metrics-auth password: name: traefik-monitor-metrics-auth key: password username: name: traefik-monitor-metrics-auth key: user port: metrics interval: 10s

These would show up as targets in the prometheus deployment, e.g.

Traefik Targets Prometheus

You can then use PromQL to query things.. like average number of open connections per second looking back at 5min windows, (then extrapolate to 5mins by multiplying by 300)

Traefik Avg backend open connections

Charting isnt the best in Prometheus but to be fair, that’s not really the primary function of Prometheus. It can get you what you need eventually, but it just takes way more effort than it should.


Grafana fills that gap; with this setup, a Grafana instance is automatically setup with Prometheus targeted as a data source. So generally what I’ll do is experiement in Prometheus with PromQL, then port over to a Grafana dashboard with proper variables and timeframes then export in json and check that in into our git repository. Overtime, we have developed quite a number of dashboards that monitor many of the services in our cluster (as well as many good default mixins provided out of the box).

Grafana Dashboards

One example is shown below, where it displays the total CPU/RAM usage; we can also click to drill down to each individual pod.

Traefik CPU/RAM

This next one is a dashboard that I built to monitor the health of Traefik, looking at the number of times its had to hot-reload configurations, and latencies and other useful metrics. We also track the Apdex for example for both entrypoints and backends.

Traefik Custom

Prometheus Rules

Prometheus rules can be defined in PromQL; these are primarily alerts that you might want the system to flag. There are many built-in rules that come along with the default installation.

Like when the kube api pods’ error rate is high: alert: KubeAPIErrorsHigh expr: sum(rate(apiserver_request_count{code=~"^(?:5..)$",job="apiserver"}[5m])) / sum(rate(apiserver_request_count{job="apiserver"}[5m])) * 100 > 3 for: 10m labels: severity: critical annotations: message: API server is returning errors for {{ $value }}% of requests. runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubeapierrorshigh

Or like when there are CLBO pods:

alert: KubePodCrashLooping
expr: rate(kube_pod_container_status_restarts_total{job="kube-state-metrics"}[15m])
  * 60 * 5 > 0
for: 1h
  severity: critical
  message: Pod {{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }})
    is restarting {{ printf "%.2f" $value }} times / 5 minutes.
  runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodcrashlooping

But you can also define your own custom ones; we have quite a number of custom rules. As an example, there is an alert that will fire when there are > 10 etcd failed proposals in the past 10 mins, which might indicate some stability issues with the etcd cluster.

  - name: custom-alerts
    - name: generic.rules
      - alert: EtcdFailedProposals
        expr: increase(etcd_server_proposal_failed_total[10m]) > 10
          severity: warning
          group: tulip
          summary: "etcd failed proposals"
          description: "{{ $labels.instance }} failed etcd proposals over the past 10 minutes has increased. May signal etcd cluster instability"

Or when a specific pod has restarted X number of times:

    - name: generic.rules
        - alert: TraefikPodCrashLooping
          expr: round(increase(kube_pod_container_status_restarts_total{pod=~"traefik-.*"}[5m])) > 5
            severity: critical
            group: tulip
            summary: "Traefik pod is restarting frequently"
            description: "Traefik pod {{$labels.pod}} has restarted {{$value}} times in the last 5 mins"

When these alerts fire, you can see them in Prometheus directly; they are also sent off to the AlertManger if one is linked up with Prometheus.


AlertManager can be configured to send to Slack, VictorOps, PagerDuty, or various other sorts of alerting systems.

      smtp_auth_username: ''
      smtp_auth_password: ''
      victorops_api_key: ''
      victorops_api_url: ''

In our setup, I configured it to post to Slack whenever there is a Warning level alert, and to VictorOps when there is a critical level alert.

  group_by: ['job']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'null'

  # This can be used to route specific specific type of alerts to specific teams.
  - match:
      alertname: DeadMansSwitch
    receiver: 'null'
  - match:
      alertname: TargetDown
    receiver: 'null'
  - match:
      severity: warning
      group: custom
    group_by: ['namespace']
    receiver: 'slack'
  - match:
      severity: critical
      group: custom
    group_by: ['namespace']
    receiver: 'victorops'

  - name: 'null'
  - name: 'sysadmins-email'
      - to: 'sysadmin@example.com'

  - name: 'slack'
      - username: 'Prometheus'
        send_resolved: true
        api_url: ''
        title: '[{{ .Status | toUpper }}] Warning Alert'
        text: >-
          {{ template "slack.techops.text" . }}

  - name: 'victorops'
      - routing_key: 'routing_key'
        message_type: '{{ .CommonLabels.severity }}'
        entity_display_name: '{{ .CommonAnnotations.summary }}'
        state_message: >-
          {{ template "slack.techops.text" . }}
        api_url: ''
        api_key: ''

Generally speaking, warning alerts could indicate some level of degraded service but might self-recover, such as when a node is down and pods auto-reschedule; or they could also be non time-critical situations that does not need immediate intervention. And critical alerts are reserved for mission-critical services or infrastructure, that can cause a bunch of issues if not recovered quickly. These would page someone and be resolved as quickly as we can.

Example of an alert that has gone off in AlertManager:

Example AlertManager Alert

Slack Alert:

Example Slack Alert

From here, you can have inhibitions that, when present, other alerts will not fire; or silences that would silence alerts based on their tags.


Together, I think the 3 components form a rather well-rounded monitoring stack for k8s infrastructure and services’ metrics. I think, down the road, the next big extension would be to have it spin up federated clusters to monitor different AWS regions and/or clusters.

PS: Here’s the repo that has a simplified version of everything I’ve talked about above: k8s-prometheus-operator-helm-example. And feel free to let me know in the comments section below if you have any questions or run into any issues playing with that example.

comments powered by Disqus