Prometheus and Grafana

Prometheus

Testing metric recall

Use curl to test the recall of metrics. The AUTHTOKEN must be configured in the app:

curl \
  -k \
  -X GET \
  -H "Authorization: Bearer <AUTHTOKEN>" \
  https://<APPPLATFORM_HOSTNAME_OR_IP>/<DOMAIN>/<INSTANCE>/prometheus/metrics

Scrape Job

A scrape job has to be installed for Prometheus to recall the metrics. This can be done in various ways. For instance for a job in the configuration file prometheus.yml:

- job_name: 'innovaphone_<INSTANCE>'
  metrics_path: '/<DOMAIN>/<INSTANCE>/prometheus/metrics'
  authorization:
    type: Bearer
    credentials: <AUTHTOKEN>
  tls_config:
    insecure_skip_verify: false
  static_configs:
    - targets:
      - "<APPPLATFORM_HOSTNAME_OR_IP>"

Metrics

Overview of recorded metrics

innovaphone_waitingqueue_status_sum

Total number of status messages of the RCC-Api of calls:

# HELP innovaphone_waitingqueue_status_sum Sum of call state messages
# TYPE innovaphone_waitingqueue_status_sum counter
innovaphone_waitingqueue_status_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="r-setup"} 22
innovaphone_waitingqueue_status_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="x-alert"} 22
innovaphone_waitingqueue_status_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="x-conn"} 22
innovaphone_waitingqueue_status_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="r-rel"} 22
innovaphone_waitingqueue_status_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="x-rel"} 22
innovaphone_waitingqueue_status_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="del"} 22

innovaphone_waitingqueue_duration_sum

Amount from the beginning of the call in seconds, in which status messages have occured:

# HELP innovaphone_waitingqueue_duration_sum Sum of duration since call start by states in seconds
# TYPE innovaphone_waitingqueue_duration_sum counter
innovaphone_waitingqueue_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="r-setup"} 0
innovaphone_waitingqueue_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="x-alert"} 0
innovaphone_waitingqueue_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="x-conn"} 330
innovaphone_waitingqueue_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="r-rel"} 2689
innovaphone_waitingqueue_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="x-rel"} 2689
innovaphone_waitingqueue_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0",msg="del"} 2689

innovaphone_waitingqueue_calls_active

Latest total number of current calls:

# HELP innovaphone_waitingqueue_calls_active Sum of active calls
# TYPE innovaphone_waitingqueue_calls_active gauge
innovaphone_waitingqueue_calls_active{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0"} 0

innovaphone_waitingqueue_calls_duration_sum

The amount of call duration of current calls:

# HELP innovaphone_waitingqueue_calls_duration_sum Gauge sum of age of active calls in seconds
# TYPE innovaphone_waitingqueue_calls_duration_sum gauge
innovaphone_waitingqueue_calls_duration_sum{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",cn="WQ0"} 0

innovaphone_app_uptime

The time in seconds since the app instance has started, aka the up-time of the app instance:

# HELP innovaphone_app_uptime Total number of seconds since app start in seconds
# TYPE innovaphone_app_uptime counter
innovaphone_app_uptime{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com"} 2789

innovaphone_app_info

Information of app version, which is the build number:

# HELP innovaphone_app_info App Infos
# TYPE innovaphone_app_info gauge
innovaphone_app_info{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",version="7"} 7

innovaphone_app_httpserver_requests_total

The app provides a web server that recalls the metrics. This metric counts all incoming calls:

# HELP innovaphone_app_httpserver_requests_total Total number of handled, incoming http requests since app start
# TYPE innovaphone_app_httpserver_requests_total counter
innovaphone_app_httpserver_requests_total{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com"} 1498

innovaphone_app_httpserver_requestmethod_total

The number of incoming HTTP requests on the webserver of the app broken down by the HTTP method:

# HELP innovaphone_app_httpserver_requestmethod_total Total number of handled, incoming http requests by request method since app start
# TYPE innovaphone_app_httpserver_requestmethod_total counter
innovaphone_app_httpserver_requestmethod_total{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",method="GET"} 1498

innovaphone_app_httpserver_requestmethod_total

The number of incoming HTTP requests on the webserver of the app broken down by path and HTTP response code:

# HELP innovaphone_app_httpserver_responsecode_total Total number of responses by requestpath and statuscode
# TYPE innovaphone_app_httpserver_responsecode_total counter
innovaphone_app_httpserver_responsecode_total{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",path="/metrics",statuscode="200"} 1494

innovaphone_app_connections_users

Number of websocket connections of app users in the app service:

# HELP innovaphone_app_connections_users Total number of connected user sockets
# TYPE innovaphone_app_connections_users gauge
innovaphone_app_connections_users{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com"} 1

innovaphone_app_connections_admins

Number of websocket connections of app admins in the app service:

# HELP innovaphone_app_connections_admins Total number of connected admin sockets
# TYPE innovaphone_app_connections_admins gauge
innovaphone_app_connections_admins{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com"} 1

innovaphone_app_connected_apis

Number of websocket connections of PBX APIs in the app service broken down by domain, PBX und API:

# HELP innovaphone_app_connected_apis Number of connected api sockets
# TYPE innovaphone_app_connected_apis gauge
innovaphone_app_connected_apis{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",pbxdns="pbx.company.com",api="RCC"} 1

innovaphone_group_member_states

Number of group members or number of members with a specific group status:

# HELP innovaphone_group_member_states State of the members of a group
# TYPE innovaphone_group_member_states gauge
innovaphone_group_member_states{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",group="Support",state="all"} 11
innovaphone_group_member_states{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",group="Support",state="in"} 6
innovaphone_group_member_states{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",group="Support",state="out"} 4
innovaphone_group_member_states{serviceid="trizwo-wqstatus",servicedns="apps.company.com",instance="wqstatuskundensupport",domain="company.com",pbx="pbx-main",group="Support",state="static"} 1

Grafana

Dashboard

A example dashboard can be imported.