Prometheus监控MySQL

docker-compose配置

通过docker-compose.yaml文件配置启动MySQL和mysqld-exporter:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
version: '3.1'
services:
    db:
        image: mysql
        restart: always
        container_name: mysql
        environment:
            TZ: Asia/Shanghai
            LANG: en_US.UTF-8
            MYSQL_ROOT_PASSWORD: 123456
        command:
            --default-authentication-plugin=mysql_native_password
            --character-set-server=utf8mb4
            --collation-server=utf8mb4_general_ci
            --lower_case_table_names=1
            --performance_schema=1
            --sql-mode=""
            --skip-log-bin
        volumes:
            - /data/mysql/conf:/etc/mysql/conf.d
            - /data/mysql/data:/var/lib/mysql
        ports:
            - 3306:3306
    
    mysqld-exporter:
        image: prom/mysqld-exporter
        container_name: mysqld-exporter
        restart: always
        command:
          - '--collect.info_schema.processlist'
          - '--collect.info_schema.innodb_metrics'
          - '--collect.info_schema.tablestats'
          - '--collect.info_schema.tables'
          - '--collect.info_schema.userstats'
          - '--collect.engine_innodb_status'
        environment:
            - DATA_SOURCE_NAME=exporter:password@(192.168.18.69:3306)/
        ports:
            - 9104:9104
        depends_on:
            - db

启动

1
docker-compose up -d

Prometheus配置

主要组件:

  • Prometheus server: 用于收集和存储时间序列数据
  • exporter: 客户端生成监控指标
  • Alertmanager: 处理警报
  • Grafana: 数据可视化和输出
  • Pushgateway:主动推送数据给Prometheus server

配置prometheus去采集mysql_exporter的监控样本数据

配置文件/etc/prometheus/prometheus.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
# 全局配置
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  # scrape_timeout is set to the global default (10s).
# 告警配置
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['192.168.1.200:9093']
# 加载一次规则,并根据全局“评估间隔”定期评估它们。
rule_files:
  - "/etc/prometheus/rules.yml"
# 控制Prometheus监视哪些资源
# 默认配置中,有一个名为prometheus的作业,它会收集Prometheus服务器公开的时间序列数据。
scrape_configs:
  # 作业名称将作为标签“job=<job_name>`添加到此配置中获取的任何数据。
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          env: dev
          role: docker
  - job_name: 'mysqld_exporter'
    static_configs:
      - targets: ['192.168.18.69:9104']
        labels:
          env: dev
          role: docker
        instance: mysql服务器
  • job_name: 'node' 是用node_export采集主机信息的节点

  • job_name: 'mysqld_exporter' 是用mysqld_exporter采集MySQL实例信息的节点

安装主机监控node_export

node_export用于采集主机信息,本质是一个采用http的协议的api

RedHat家族的操作系统可以采用yum进行安装

yum 安装方法: https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/

1
2
3
4
curl -Lo /etc/yum.repos.d/_copr_ibotty-prometheus-exporters.repo https://copr.fedorainfracloud.org/coprs/ibotty/prometheus-exporters/repo/epel-7/ibotty-prometheus-exporters-epel-7.repo
yum -y install node_exporter
systemctl start node_exporter
systemctl enable node_exporter.service

二进制文件安装 官网下载地址(https://prometheus.io/download/)

1
2
tar -zxvf node_exporter-1.0.0-rc.1.linux-amd64.tar.gz
 ./node_exporter --web.listen-address=:9100 

访问地址: http://localhost:9100/metrics

编辑告警和MySQL监控规则文件

/etc/prometheus/rules.yml内容:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
groups:
- name: example
  rules: # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 1m
    labels:
      serverity: page
    annotations:
      summary: "Instance {{ $labels.instance }} down"
      description: "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes."

- name: MySQL
    rules:
    - alert: MysqlDown
      expr: mysql_up == 0
      for: 30s
      labels:
        severity: critical
      annotations:
        summary: "MySQL Down,实例:{{ $labels.instance }}"
        description: "MySQL_exporter连不上MySQL了,当前状态为:{{ $value }}"
    - alert: MysqlSlowQueries
      expr: increase(mysql_global_status_slow_queries[2m]) > 0
      for: 2m
      labels:
        severity: warning
      annotations:
        summary: "Mysql慢日志告警,实例:{{ $labels.instance }}"
        description: "MySQL在过去2分钟有新的{{ $value }}条慢查询"

编辑告警配置文件

/etc/alertmanager/alertmanager.yml内容:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
global:
  resolve_timeout: 5m
  smtp_smarthost: 'xxx@xxx:587'
  smtp_from: 'xxx@xxx'
  smtp_auth_username: 'xxx@xxx'
  smtp_auth_password: 'xxxx'
  smtp_require_tls: true
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'test-mails'
receivers:
- name: 'test-mails'
  email_configs:
  - to: 'xxx@qq.com'

docker-compose启动Prometheus

/docker-compose/prometheus/docker-compose.yml内容

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
services:
  prometheus:
   image: prom/prometheus
   volumes:
     - /etc/prometheus/:/etc/prometheus/
     - prometheus_data:/prometheus
   command:
     - '--config.file=/etc/prometheus/prometheus.yml'
     - '--storage.tsdb.path=/prometheus'
     - '--web.console.libraries=/usr/share/prometheus/console_libraries'
     - '--web.console.templates=/usr/share/prometheus/consoles'
     - '--web.external-url=http://192.168.1.200:9090/'
     - '--web.enable-lifecycle'
     - '--storage.tsdb.retention=15d'
   ports:
     - 9090:9090
   links:
     - alertmanager:alertmanager
   restart: always
  alertmanager:
   image: prom/alertmanager
   ports:
     - 9093:9093
   volumes:
     - /etc/alertmanager/:/etc/alertmanager/
     - alertmanager_data:/alertmanager
   command:
     - '--config.file=/etc/alertmanager/alertmanager.yml'
     - '--storage.path=/alertmanager'
   restart: always
  grafana:
   image: grafana/grafana
   ports:
     - 3000:3000
   volumes:
     - /etc/grafana/:/etc/grafana/provisioning/
     - grafana_data:/var/lib/grafana
   environment:
     - GF_INSTALL_PLUGINS=camptocamp-prometheus-alertmanager-datasource
   links:
     - prometheus:prometheus
     - alertmanager:alertmanager
   restart: always

volumes:
  prometheus_data: {}
  grafana_data: {}
  alertmanager_data: {}

启动 docker-compose up -d

访问端点

  • http://localhost:9090 Prometheus server主页
  • http://localhost:9090/metrics Prometheus server自身指标
  • http://192.168.0.80:3000 Grafana

参考