Ingress Nginx (Prometheus)¶

Collect Metrics, Traces, and Logs related to Ingress Nginx (Prometheus)

Configuration¶

Prerequisites¶

DataKit has been deployed. Please refer to Kubernetes cluster <Install Datakit>
K8s environment has been deployed
Ingress-nginx version: 1.9.6

Configure Ingress-nginx Service ConfigMap Resource¶

In the YAML file for installing Ingress-nginx, add the following configuration:

apiVersion: v1
data:
  allow-snippet-annotations: "false"
  enable-opentracing: "true"
  opentracing-trust-incoming-span: "true"
  datadog-collector-host: "datakit-service.datakit.svc.cluster.local"
  datadog-collector-port: "9529"
  datadog-service-name: "ingress-nginx"
  datadog-environment: "testing"
  datadog-operation-name-override: "HTTP $request_method $service_name $uri $opentelemetry_trace_id"
  datadog-priority-sampling: "false"
  datadog-sample-rate: "1.0"
kind: ConfigMap
metadata:
  labels:
    app.Kubernetes.io/component: controller
    app.Kubernetes.io/instance: ingress-nginx
    app.Kubernetes.io/name: ingress-nginx
    app.Kubernetes.io/part-of: ingress-nginx
    app.Kubernetes.io/version: 1.9.6
  name: ingress-nginx-controller
  namespace: ingress-nginx

Execute the following command to apply the configuration

kuebctl apply -f ingress-nginx.yaml

Collector Configuration¶

Metrics Collection¶

Enable the KubernetesPrometheus collector by ConfigMap method injection

Add kubernetesprometheus.conf under ConfigMap in the datakit.yaml file

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
    Kubernetesprometheus.conf: |-
      [inputs.Kubernetesprometheus]
        [[inputs.Kubernetesprometheus.instances]]
          role       = "pod"
          #namespaces = ["ingress-nginx"]
          scheme     = "http"
          #selector   = "app=ingress-nginx"
          port       = "10254"
          path       = "/metrics"
          node_local = "true"
          [inputs.Kubernetesprometheus.instances.custom]
            measurement = "ingress-nginx"
            [inputs.Kubernetesprometheus.instances.custom.tags]
              instance         = "__Kubernetes_mate_instance"
              host             = "__Kubernetes_mate_host"
              pod_name         = "__Kubernetes_pod_name"
              pod_namespace    = "__Kubernetes_pod_namespace"

Mount kubernetesprometheus.conf to /usr/local/datakit/conf.d/kubernetesprometheus/kubernetesprometheus.conf in DataKit

        - mountPath: /usr/local/datakit/conf.d/Kubernetesprometheus/Kubernetesprometheus.conf
          name: datakit-conf
          subPath: Kubernetesprometheus.conf

Execute the following command to restart DataKit

kubectl delete -f datakit.yaml
kubectl apply -f datakit.yaml

In the Ingress-nginx.yaml file for installing Ingress-nginx-controller, set the Service type to NodePort and expose port 10254

spec:
  type: NodePort
......
    - name: prometheus
      port: 10254
      targetPort: prometheus

Execute the following command to restart the Ingress-nginx service

kuebctl apply -f ingress-nginx.yaml

Trace Collection¶

Enable DDTrace collector

Modify the datakit.yaml file and append DDtrace to the default enabled collector configuration

 - name: ENV_DEFAULT_ENABLED_INPUTS
   value: cpu,disk,diskio,mem,swap,system,hostobject,net,host_processes,container,ddtrace

Execute the following command to restart DataKit

kubectl delete -f datakit.yaml
kubectl apply -f datakit.yaml

Download the DataKit Operator YAML file

wget https://truewatch.com/datakit-operator/datakit-operator.yaml

Execute the installation command

kubectl apply -f datakit-operator.yaml

Ingress Access Test

The Ingress-nginx-controller Service type is NodePort, mapping port 80 to port 30796 (Note: 192.168.0.5 is the node IP running the Ingress-nginx Pod)

curl -H "Host:service.com" http://192.168.0.5:30796/service-a/api/a

Service-A received: Hello from Service-B

In the TrueWatch APM section, you can see that the Ingress-nginx trace data is reported normally

Log Collection¶

DataKit collects container standard output logs by default, no additional configuration is required.

Metrics¶

Ingress Nginx metrics are located under the ingress-nginx measurement. Here we mainly introduce the description of Ingress Nginx related metrics

Metric	Description	Data Type	Unit
nginx_ingress_controller_requests	The total number of client requests	int	count
nginx_ingress_controller_nginx_process_connections	current number of client connections with state	int	count
nginx_ingress_controller_success	Cumulative number of Ingress controller reload operations	int	count
nginx_ingress_controller_config_last_reload_successful	Whether the last configuration reload attempt was successful	int	count
nginx_ingress_controller_nginx_process_resident_memory_bytes	number of bytes of memory in use	float	B
nginx_ingress_controller_nginx_process_cpu_seconds_total	Cpu usage in seconds	float	B
nginx_process_resident_memory_bytes	number of bytes of memory in use	int	B
nginx_ingress_controller_request_duration_seconds_bucket	The request processing time in milliseconds	int	count
nginx_ingress_controller_request_size_sum	The request length (including request line, header, and request body)	int	count
nginx_ingress_controller_response_size_sum	The response length (including request line, header, and request body)	int	count
nginx_ingress_controller_ssl_expire_time_seconds	Number of seconds since 1970 to the SSL Certificate expire	int	count