• Example Spring application with logging
• Deploy a Spring Boot Application on Kubernetes
• Deploy a Grok Exporter as a sidecar on Kubernetes
  • Log capture using grok exporter in Kubernetes
• Deploy a Prometheus on Kubernetes
  • Prometheus ConfigMap
  • Prometheus Deployment
  • Prometheus Service
• Summary

In previous post, you have looked into Prometheus basics. In this part, it is time to create Prometheus service on the Kubernetes which can scrape metrics from additional metric exporter. It’s the perfect solution when you can’t export metrics from the application source code and your application logs the information you want to expose as metrics.

Example Spring application with logging

To understand how an grok exporter works, we must first create an application that logs some data to files.

I have prepared a very simple application in Spring Boot that has two functionalities:

1. Welcomes users entering the home page. If the user does not provide his name in @RequestParam - the application will answer: Hello Guest

   

   HelloController source code: Github

2. Every 10 seconds creates a log in the form: DATE TIME SERVICE_NAME CONNECTION_TIME.

Service names are randomly selected from 3 defined names. Response times are also random. The logs are supposed to resemble real logs of the application connecting to other services and measuring the time of connection. The source code is on the Github

Logs are saved to the logs/applog.log file. Log4j configuration: log4j.xml code: Github

Application code is also available on Github, docker image is on dockerHub.

Deploy a Spring Boot Application on Kubernetes

Firstly, we have to build docker image with application. I have already done it - image is pushed to the DockerHub. You can just import the image in the deployment definition (image: docker.io/michf/spring-logging-app:latest) for Kubernetes to deploy application.

If you want to create an image of your own application you can create a Dockerfile based on mine and push to the Dockerhub.

Now, we have to prepare Kubernetes deployment configuration. Let’s start from the Deployment config:

 
          apiVersion: apps/v1
          kind: Deployment
          metadata:
            name: spring-logging-app
            labels:
              app: spring-logging-app
          spec:
            replicas: 1
            selector:
              matchLabels:
                app: spring-logging-app
            template:
              metadata:
                labels:
                  app: spring-logging-app
              spec:
                containers:
                  - name: spring-logging-app
                    image: docker.io/michf/spring-logging-app:latest
                    ports:
                      - containerPort: 8080

Service definition:

 
apiVersion: v1
kind: Service
metadata:
  name: spring-logging-app
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      targetPort: 8080
      port: 8080
    selector:
        sidecar: spring-logging-app

That’s all. You can deploy logging application on Kubernetes.

Deploy a Grok Exporter as a sidecar on Kubernetes

We will be collecting data from our logs using a sidecar container. A sidecar container is a secondary container which is run within the same Pod. In our case grok exporter will be a sidecar container.

To configure sidecar pattern, we will create a volume (with /logs path) for our Pod to be shared by all containers in the Pod. Then we will configure grok exporter to read logs from shared location.

Firstly, we have to create grok service. Service will be a target in Prometheus:

 
apiVersion: v1
kind: Service
metadata:
  name: grok-exporter
  labels:
    sidecar: grok-exporter
spec:
  type: ClusterIP
  ports:
    - protocol: TCP
      targetPort: 9144
      port: 9144
    selector:
        sidecar: grok-exporter

9144 is a default grok exporter port.

Log capture using grok exporter in Kubernetes

To capture logs we need to create a configuration file. For this purpose we will use Kubernetes ConfigMap:

 
kind: ConfigMap
metadata:
  name: grok-exporter
apiVersion: v1
data:
  config.yml: |-
    global:
      config_version: 2
    input:
      type: file
      fail_on_missing_logfile: false
      path: /logs/applog.log
    grok:
      patterns_dir: ./patterns
    metrics:
      - type: histogram
        name: service_call
        help: Services call times
        match: '%{DATE} %{TIME} %{WORD:service} connection time= %{INT:val}'
        value: ''
  buckets: [100, 1000, 3000]
    labels:
      service    :'{{.service}}'
    - type: counter
      name: hello_requests_total
      help: Hello controller requests counter
      match: '%{DATE} %{TIME} HelloController request from: %{USER:user}'
      labels:
        user    : '{{.user}}'
    server:
      port: 9144

Configmap will create and mount the config.yml file (configuration for grok exporter). In config.yml we set the path to the mounted volume (/logs/applog.log) and two types of metrics. The histogram will inform you about how many connections to external services took less than 100 ms, how many calls took more than 100, but less than 1000, and how many calls lasted more than 3000 ms. Counter will inform you how many http requests has served our application for a given user.

The match: property is an expression pattern. You can find all grok patterns on the Github. Then you can adjust the pattern to catch your logs.

Let’s add volume and grok exporter image to the deployment definition from previous point:

 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-logging-app
  labels:
    app: spring-logging-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spring-logging-app
      sidecar: grok-exporter
  template:
    metadata:
      labels:
        app: spring-logging-app
        sidecar: grok-exporter
    spec:
      containers:
        - name: spring-logging-app
          image: docker.io/michf/spring-logging-app:latest
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: logs
              mountPath: /logs/
                - name: grok
          image: palobo/grok_exporter
          imagePullPolicy: Always
          ports:
            - containerPort: 9144
              protocol: TCP
          volumeMounts:
            - name: grok-config-volume
              mountPath: /etc/grok_exporter
            - name: logs
              mountPath: /logs
            volumes:
                - name: grok-config-volume
          configMap:
                name: grok-exporter
                    - name: logs
          emptyDir: {}

Now, we can deploy above Deployment. After entering the pod via Kubernetes Dashboard, you should see information about two containers (from grok and from the logging application):

Let’s check if grok expese endpoint wiht metrics. I used kubectl port-forward pod-name port:port to forward a grok port to localhost

 

kubectl port-forward spring-logging-app-79bb56876-c4h29 9144:9144     

---

You can go to 127.0.0.1:9144/metrics and check the application metrics based on the logs:.

Counter metrics:

Histogram metrics:

Deploy a Prometheus on Kubernetes

Prometheus ConfigMap

Firstly, create a configmap with the Prometheus scrape config:

 
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus
data:
  prometheus.yml: |-
    global:
      scrape_interval: 15s
    scrape_configs:
    - job_name: 'grok'
      static_configs:
      - targets: ['grok-exporter:9144']

Prometheus will collect information from the grok-exporter service every 15 seconds.

Prometheus Deployment

In Prometheus deployment config We have to mount created a moment ago config map as a file inside /etc/prometheus:

 
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
spec:
  selector:
    matchLabels:
      app: prometheus
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.12.0
          ports:
            - name: web
              containerPort: 9090
          volumeMounts:
            - name: prometheus-config
              mountPath: /etc/prometheus
            - name: prometheus-data
              mountPath: /prometheus
            volumes:
                - name: prometheus-data
          emptyDir: {}
                    - name: prometheus-config
          configMap:
                  name: prometheus

Prometheus Service

To acces the Prometheus GUI over an IP/DNS, we need to expose it as a service:

 
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  labels:
    app: prometheus
spec:
  selector:
    app: prometheus
  type: NodePort
  ports:
    - protocol: TCP
      name: web
      nodePort: 31199
      port: 9090
      targetPort: 9090

Summary

Now we can access the Prometheus dashboard using kubectl port-forward 9090:9090. You should see the Prometheus dashboard with scraped data:

Thank you for reading! I hope you enjoyed the post.