跳转至

在 Kubernetes 上部署 Prometheus

前面我们已经了解了 Prometheus 的基本使用方式,主要是使用的二进制方式进行部署的,在实际生产环境来说,Prometheus 更适合用来部署在 Kubernetes 集群中,本节我们将介绍如何用手动方式在 Kubernetes 集群上部署 Prometheus,关于 Kubernetes 本身的使用可以参考我们的另外课程《Kubernetes 进阶训练营》

安装

由于我们这里是要运行在 Kubernetes 系统中,所以我们直接用 Docker 镜像的方式运行。这里我们使用的实验环境是基于 Kubernetes v1.22 版本,一共 3 个节点:

☸ ➜ kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
master1   Ready    control-plane,master   49d   v1.22.2
node1     Ready    <none>                 49d   v1.22.2
node2     Ready    <none>                 49d   v1.22.2

为了方便管理,我们将监控相关的所有资源对象都安装在 kube-mon 这个 namespace 下面,没有的话可以提前创建:

☸ ➜ kubectl create ns kube-mon

为了能够方便的管理配置文件,我们这里将 prometheus.yml 配置文件用 ConfigMap 的形式进行管理:

# config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: kube-mon
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      scrape_timeout: 15s
    scrape_configs:
    - job_name: 'prometheus'
      static_configs:
      - targets: ['localhost:9090']

我们这里暂时只配置了对 prometheus 本身的监控,直接创建该资源对象:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/config-1.yaml
configmap "prometheus-config" created

配置文件创建完成了,以后如果我们有新的资源需要被监控,我们只需要将上面的 ConfigMap 对象更新即可。现在我们来创建 prometheus 的 Pod 资源:

# prometheus-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: kube-mon
  labels:
    app: prometheus
spec:
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - image: prom/prometheus:v2.31.1
          name: prometheus
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus" # 指定tsdb数据路径
            - "--storage.tsdb.retention.time=24h"
            - "--web.enable-admin-api" # 控制对admin HTTP API的访问,其中包括删除时间序列等功能
            - "--web.enable-lifecycle" # 支持热更新,直接执行localhost:9090/-/reload立即生效
          ports:
            - containerPort: 9090
              name: http
          volumeMounts:
            - mountPath: "/etc/prometheus"
              name: config-volume
            - mountPath: "/prometheus"
              name: data
          resources:
            requests:
              cpu: 200m
              memory: 1024Mi
            limits:
              cpu: 200m
              memory: 1024Mi
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: prometheus-data
        - configMap:
            name: prometheus-config
          name: config-volume

持久化

另外为了 prometheus 的性能和数据持久化我们这里是直接将通过一个 LocalPV 来进行数据持久化的,注意一定不能使用 nfs 来持久化数据,通过 --storage.tsdb.path=/prometheus 指定数据目录,创建如下所示的一个 PVC 资源对象,注意是一个 LocalPV,和 node2 节点具有亲和性:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: prometheus-local
  labels:
    app: prometheus
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 20Gi
  storageClassName: local-storage
  local:
    path: /data/k8s/prometheus
  persistentVolumeReclaimPolicy: Retain
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values:
                - node2
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: prometheus-data
  namespace: kube-mon
spec:
  selector:
    matchLabels:
      app: prometheus
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: local-storage

由于 prometheus 可以访问 Kubernetes 的一些资源对象,所以需要配置 rbac 相关认证,这里我们使用了一个名为 prometheus 的 serviceAccount 对象:

# rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-mon
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
      - services
      - endpoints
      - pods
      - nodes/proxy
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - configmaps
      - nodes/metrics
    verbs:
      - get
  - nonResourceURLs:
      - /metrics
    verbs:
      - get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: kube-mon

由于我们要获取的资源信息,在每一个 namespace 下面都有可能存在,所以我们这里使用的是 ClusterRole 的资源对象,值得一提的是我们这里的权限规则声明中有一个 nonResourceURLs 的属性,是用来对非资源型 metrics 进行操作的权限声明,这个在以前我们很少遇到过,然后直接创建上面的资源对象即可:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/rbac.yaml
serviceaccount "prometheus" created
clusterrole.rbac.authorization.k8s.io "prometheus" created
clusterrolebinding.rbac.authorization.k8s.io "prometheus" created

现在我们就可以添加 promethues 的资源对象了:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/deploy.yaml
deployment.apps/prometheus created
☸ ➜ kubectl get pods -n kube-mon
NAME                         READY   STATUS             RESTARTS   AGE
prometheus-df4f47d95-vksmc   0/1     CrashLoopBackOff   3          98s
☸ ➜ kubectl logs -f prometheus-df4f47d95-vksmc -n kube-mon
level=info ts=2019-12-12T03:08:49.424Z caller=main.go:332 msg="Starting Prometheus" version="(version=2.14.0, branch=HEAD, revision=edeb7a44cbf745f1d8be4ea6f215e79e651bfe19)"
level=info ts=2019-12-12T03:08:49.424Z caller=main.go:333 build_context="(go=go1.13.4, user=root@df2327081015, date=20191111-14:27:12)"
level=info ts=2019-12-12T03:08:49.425Z caller=main.go:334 host_details="(Linux 3.10.0-1062.4.1.el7.x86_64 #1 SMP Fri Oct 18 17:15:30 UTC 2019 x86_64 prometheus-df4f47d95-vksmc (none))"
level=info ts=2019-12-12T03:08:49.425Z caller=main.go:335 fd_limits="(soft=1048576, hard=1048576)"
level=info ts=2019-12-12T03:08:49.425Z caller=main.go:336 vm_limits="(soft=unlimited, hard=unlimited)"
level=error ts=2019-12-12T03:08:49.425Z caller=query_logger.go:85 component=activeQueryTracker msg="Error opening query log file" file=/prometheus/queries.active err="open /prometheus/queries.active: permission denied"
panic: Unable to create mmap-ed active query log

goroutine 1 [running]:
github.com/prometheus/prometheus/promql.NewActiveQueryTracker(0x7ffd8cf6ec5d, 0xb, 0x14, 0x2b4f400, 0xc0006f33b0, 0x2b4f400)
        /app/promql/query_logger.go:115 +0x48c
main.main()
        /app/cmd/prometheus/main.go:364 +0x5229

权限

创建 Pod 后,我们可以看到并没有成功运行,出现了 open /prometheus/queries.active: permission denied 这样的错误信息,这是因为我们的 prometheus 的镜像中是使用的 nobody 这个用户,然后现在我们通过 LocalPV 挂载到宿主机上面的目录的 ownership 却是 root

☸ ➜ ls -la /data/k8s
total 36
drwxr-xr-x   6 root root  4096 Dec 12 11:07 .
dr-xr-xr-x. 19 root root  4096 Nov  9 23:19 ..
drwxr-xr-x   2 root root  4096 Dec 12 11:07 prometheus

所以当然会出现操作权限问题了,这个时候我们就可以通过 securityContext 来为 Pod 设置下 volumes 的权限,通过设置 runAsUser=0 指定运行的用户为 root,也可以通过设置一个 initContainer 来修改数据目录权限:

......
initContainers:
- name: fix-permissions
  image: busybox
  command: [chown, -R, "nobody:nobody", /prometheus]
  volumeMounts:
  - name: data
    mountPath: /prometheus

这个时候我们重新更新下 prometheus:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/deploy-fixed.yaml
deployment.apps/prometheus configured
☸ ➜ kubectl get pods -n kube-mon
NAME                          READY   STATUS    RESTARTS      AGE
prometheus-649968556c-t4prd   1/1     Running   1 (14m ago)   27h
☸ ➜ kubectl logs -f prometheus-649968556c-t4prd -n kube-mon
ts=2021-12-15T06:55:25.752Z caller=main.go:444 level=info msg="Starting Prometheus" version="(version=2.31.1, branch=HEAD, revision=411021ada9ab41095923b8d2df9365b632fd40c3)"
ts=2021-12-15T06:55:25.752Z caller=main.go:449 level=info build_context="(go=go1.17.3, user=root@9419c9c2d4e0, date=20211105-20:35:02)"
ts=2021-12-15T06:55:25.752Z caller=main.go:450 level=info host_details="(Linux 3.10.0-1160.25.1.el7.x86_64 #1 SMP Wed Apr 28 21:49:45 UTC 2021 x86_64 prometheus-649968556c-t4prd (none))"
ts=2021-12-15T06:55:25.752Z caller=main.go:451 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2021-12-15T06:55:25.752Z caller=main.go:452 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2021-12-15T06:55:25.756Z caller=web.go:542 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2021-12-15T06:55:26.150Z caller=main.go:839 level=info msg="Starting TSDB ..."
......
ts=2021-12-15T06:55:27.048Z caller=main.go:869 level=info msg="TSDB started"
ts=2021-12-15T06:55:27.048Z caller=main.go:996 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
ts=2021-12-15T06:55:27.050Z caller=main.go:1033 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=1.555486ms db_storage=754ns remote_storage=38.847µs web_handler=433ns query_engine=852ns scrape=1.030952ms scrape_sd=73.933µs notify=894ns notify_sd=2.504µs rules=19.359µs
ts=2021-12-15T06:55:27.050Z caller=main.go:811 level=info msg="Server is ready to receive web requests."

Pod 创建成功后,为了能够在外部访问到 prometheus 的 webui 服务,我们还需要创建一个 Service 对象:

# prometheus-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: kube-mon
  labels:
    app: prometheus
spec:
  selector:
    app: prometheus
  type: NodePort
  ports:
    - name: web
      port: 9090
      targetPort: http

为了方便测试,我们这里创建一个 NodePort 类型的服务,当然我们可以创建一个 Ingress对象,通过域名来进行访问:

☸ ➜ kubectl apply -f https://p8s.io/docs/k8s/manifests/prometheus/svc.yaml
service "prometheus" created
☸ ➜ kubectl get svc -n kube-mon
NAME         TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
prometheus   NodePort   10.111.160.152   <none>        9090:30407/TCP   4m33s

现在我们就可以通过 http://任意节点IP:30407 访问 prometheus 的 webui 服务了:

prometheus webui

现在我们可以查看当前监控系统中的一些监控目标(Status -> Targets):

prometheus webui targets

由于我们现在还没有配置任何的报警信息,所以 Alerts 菜单下面现在没有任何数据,隔一会儿,我们可以去 Graph 菜单下面查看我们抓取的 prometheus 本身的一些监控数据了:

prometheus webui metrics

比如我们这里就选择 scrape_duration_seconds 这个指标,然后点击 Execute,就可以看到类似于下面的图表数据了:

prometheus webui query