Skip to content

InfluxDB

·

The InfluxDB collector is used to collect the data of the InfluxDB.

Configuration

Preconditions

The influxdb collector is only applicable to influxdb v1.x, and the prom collector is required for influxdb v2.x.

Already tested version:

  • 1.8.10

Collector Configuration

Go to the conf.d/samples directory under the DataKit installation directory, copy influxdb.conf.sample and name it influxdb.conf. Examples are as follows:

[[inputs.influxdb]]
  url = "http://localhost:8086/debug/vars"

  ## (optional) collect interval, default is 10 seconds
  interval = '10s'

  ## Username and password to send using HTTP Basic Authentication.
  # username = ""
  # password = ""

  ## http request & header timeout
  timeout = "5s"

  ## Set true to enable election
  election = true

  ## TLS config
  # [inputs.influxdb.tlsconf]
    # insecure_skip_verify = true
    ## Following ca_certs/cert/cert_key are optional, if insecure_skip_verify = true.
    # ca_certs = ["/opt/tls/ca.crt"]
    # cert = "/opt/tls/client.root.crt"
    # cert_key = "/opt/tls/client.root.key"
    ## we can encode these file content in base64 format:
    # ca_certs_base64 = ["LONG_BASE64_STRING......"]
    # cert_base64 = "LONG_BASE64_STRING......"
    # cert_key_base64 = "LONG_BASE64_STRING......"
    # server_name = "your-SNI-name"

  # [inputs.influxdb.log]
  # files = []
  # #grok pipeline script path
  # pipeline = "influxdb.p"

  [inputs.influxdb.tags]
    # some_tag = "some_value"
    # more_tag = "some_other_value"

Once configured, restart DataKit.

The collector can now be turned on by ConfigMap injection collector configuration.

InfluxDB v2.x

[[inputs.prom]]
  ## Exporter address
  url = "http://127.0.0.1:8086/metrics"

  metric_types = ["counter", "gauge"]

  interval = "10s"

  ## TLS configuration.
  tls_open = false
  # tls_ca = "/tmp/ca.crt"
  # tls_cert = "/tmp/peer.crt"
  # tls_key = "/tmp/peer.key"

  [[inputs.prom.measurements]]
    prefix = "boltdb_"
    name = "influxdb_v2_boltdb"

  [[inputs.prom.measurements]]
    prefix = "go_"
    name = "influxdb_v2_go"

  ## Histogram type.
  # [[inputs.prom.measurements]]
  #   prefix = "http_api_request_"
  #   name = "influxdb_v2_http_request"

  [[inputs.prom.measurements]]
    prefix = "influxdb_"
    name = "influxdb_v2"

  [[inputs.prom.measurements]]
    prefix = "service_"
    name = "influxdb_v2_service"

  [[inputs.prom.measurements]]
    prefix = "task_"
    name = "influxdb_v2_task" 

  ## Customize tags.
  [inputs.prom.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"

Metric

For all of the following data collections, the global election tags will added automatically, we can add extra tags in [inputs.influxdb.tags] if needed:

 [inputs.influxdb.tags]
  # some_tag = "some_value"
  # more_tag = "some_other_value"
  # ...

influxdb_cq

Tags & Fields Description
host
(tag)		 System hostname.
query_fail The total number of continuous queries that executed but failed.
Type: float | (count)
Unit: count
query_ok The total number of continuous queries that executed successfully.
Type: float | (count)
Unit: count

influxdb_database

Tags & Fields Description
database
(tag)		 Database name.
host
(tag)		 System hostname.
num_measurements The current number of measurements in the specified database.
Type: float | (gauge)
Unit: count
num_series The current series cardinality of the specified database.
Type: float | (gauge)
Unit: count

influxdb_httpd

Tags & Fields Description
bind
(tag)		 Bind port.
host
(tag)		 System hostname.
auth_fail The number of HTTP requests that were aborted due to authentication being required, but not supplied or incorrect.
Type: float | (count)
Unit: count
client_error The number of HTTP responses due to client errors, with a 4XX HTTP status code.
Type: float | (count)
Unit: count
flux_query_req The number of Flux query requests served.
Type: float | (count)
Unit: count
flux_query_req_duration_ns The duration (wall-time), in nanoseconds, spent executing Flux query requests.
Type: float | (count)
Unit: time,ns
ping_req The number of times InfluxDB HTTP server served the /ping HTTP endpoint.
Type: float | (count)
Unit: count
points_written_dropped The number of points dropped by the storage engine.
Type: float | (count)
Unit: count
points_written_fail The number of points accepted by the HTTP /write endpoint, but unable to be persisted.
Type: float | (count)
Unit: count
points_written_ok The number of points successfully accepted and persisted by the HTTP /write endpoint.
Type: float | (count)
Unit: count
prom_read_req The number of read requests to the Prometheus /read endpoint.
Type: float | (count)
Unit: count
prom_write_req The number of write requests to the Prometheus /write endpoint.
Type: float | (count)
Unit: count
query_req The number of query requests.
Type: float | (count)
Unit: count
query_req_duration_ns The total query request duration, in nanosecond (ns).
Type: float | (count)
Unit: time,ns
query_resp_bytes The total number of bytes returned in query responses.
Type: float | (count)
Unit: digital,B
recovered_panics The total number of panics recovered by the HTTP handler.
Type: float | (count)
Unit: count
req The total number of HTTP requests served.
Type: float | (count)
Unit: count
req_active The number of currently active requests.
Type: float | (gauge)
Unit: count
req_duration_ns The duration (wall time), in nanoseconds, spent inside HTTP requests.
Type: float | (count)
Unit: time,ns
server_error The number of HTTP responses due to server errors.
Type: float | (count)
Unit: count
status_req The number of status requests served using the HTTP /status endpoint.
Type: float | (count)
Unit: count
values_written_ok The number of values (fields) successfully accepted and persisted by the HTTP /write endpoint.
Type: float | (count)
Unit: count
write_req The number of write requests served using the HTTP /write endpoint.
Type: float | (count)
Unit: count
write_req_active The number of currently active write requests.
Type: float | (gauge)
Unit: count
write_req_bytes The total number of bytes of line protocol data received by write requests, using the HTTP /write endpoint.
Type: float | (count)
Unit: digital,B
write_req_duration_ns The duration (wall time), in nanoseconds, of write requests served using the /write HTTP endpoint.
Type: float | (count)
Unit: time,ns

influxdb_memstats

Tags & Fields Description
host
(tag)		 System hostname.
alloc The currently allocated number of bytes of heap objects.
Type: float | (gauge)
Unit: digital,B
buck_hash_sys The bytes of memory in profiling bucket hash tables.
Type: float | (gauge)
Unit: digital,B
frees The cumulative number of freed (live) heap objects.
Type: float | (count)
Unit: count
gc_cpu_fraction The fraction of CPU time used by the garbage collection cycle.
Type: float | (gauge)
Unit: percent,percent_decimal
gc_sys The bytes of memory in garbage collection metadata.
Type: float | (gauge)
Unit: digital,B
heap_alloc The size, in bytes, of all heap objects.
Type: float | (gauge)
Unit: digital,B
heap_idle The number of bytes of idle heap objects.
Type: float | (gauge)
Unit: digital,B
heap_inuse The number of bytes in in-use spans.
Type: float | (gauge)
Unit: digital,B
heap_objects The number of allocated heap objects.
Type: float | (gauge)
Unit: count
heap_released The number of bytes of physical memory returned to the OS.
Type: float | (gauge)
Unit: digital,B
heap_sys The number of bytes of heap memory obtained from the OS.
Type: float | (gauge)
Unit: digital,B
last_gc Time the last garbage collection finished, as nanoseconds since 1970 (the UNIX epoch).
Type: float | (gauge)
Unit: timeStamp,nsec
lookups The number of pointer lookups performed by the runtime.
Type: float | (count)
Unit: count
mallocs The total number of heap objects allocated.
Type: float | (count)
Unit: count
mcache_inuse The bytes of allocated mcache structures.
Type: float | (gauge)
Unit: digital,B
mcache_sys The bytes of memory obtained from the OS for mcache structures.
Type: float | (gauge)
Unit: digital,B
mspan_inuse The bytes of allocated mcache structures.
Type: float | (gauge)
Unit: digital,B
mspan_sys The bytes of memory obtained from the OS for mspan.
Type: float | (gauge)
Unit: digital,B
next_gc The target heap size of the next garbage collection cycle.
Type: float | (gauge)
Unit: digital,B
num_forced_gc The number of GC cycles that were forced by the application calling the GC function.
Type: float | (count)
Unit: count
num_gc The number of completed garbage collection cycles.
Type: float | (count)
Unit: count
other_sys The number of bytes of memory used other than heap_sys/stacks_sys/mspan_sys/mcache_sys/buckhash_sys/gc_sys.
Type: float | (gauge)
Unit: digital,B
pause_ns The time garbage collection cycles are paused in nanoseconds.
Type: float | (gauge)
Unit: time,ns
pause_total_ns The total time garbage collection cycles are paused in nanoseconds.
Type: float | (count)
Unit: time,ns
stack_inuse The number of bytes in in-use stacks.
Type: float | (gauge)
Unit: digital,B
stack_sys The total number of bytes of memory obtained from the stack in use.
Type: float | (gauge)
Unit: digital,B
sys The total bytes of memory obtained from the OS by the Go runtime.
Type: float | (gauge)
Unit: digital,B
total_alloc The cumulative bytes allocated for heap objects.
Type: float | (count)
Unit: digital,B

influxdb_queryExecutor

Tags & Fields Description
host
(tag)		 System hostname.
queries_active The number of active queries currently being handled.
Type: float | (gauge)
Unit: count
queries_executed The number of queries executed (started).
Type: float | (count)
Unit: count
queries_finished The number of queries that have finished executing.
Type: float | (count)
Unit: count
query_duration_ns The cumulative duration (wall time), in nanoseconds, of executed queries.
Type: float | (count)
Unit: time,ns
recovered_panics The number of panics recovered by the Query Executor.
Type: float | (count)
Unit: count

influxdb_runtime

Tags & Fields Description
host
(tag)		 System hostname.
alloc The currently allocated number of bytes of heap objects.
Type: float | (gauge)
Unit: digital,B
frees The cumulative number of freed (live) heap objects.
Type: float | (count)
Unit: count
heap_alloc The size, in bytes, of all heap objects.
Type: float | (gauge)
Unit: digital,B
heap_idle The number of bytes of idle heap objects.
Type: float | (gauge)
Unit: digital,B
heap_inuse The number of bytes in in-use spans.
Type: float | (gauge)
Unit: digital,B
heap_objects The number of allocated heap objects.
Type: float | (gauge)
Unit: count
heap_released The number of bytes of physical memory returned to the OS.
Type: float | (gauge)
Unit: digital,B
heap_sys The number of bytes of heap memory obtained from the OS.
Type: float | (gauge)
Unit: digital,B
lookups The number of pointer lookups performed by the runtime.
Type: float | (count)
Unit: count
mallocs The total number of heap objects allocated.
Type: float | (count)
Unit: count
num_gc The number of completed garbage collection cycles.
Type: float | (count)
Unit: count
num_goroutine The total number of Go routines.
Type: float | (gauge)
Unit: count
pause_total_ns The total time garbage collection cycles are paused in nanoseconds.
Type: float | (count)
Unit: time,ns
sys The total bytes of memory obtained from the OS by the Go runtime.
Type: float | (gauge)
Unit: digital,B
total_alloc The cumulative bytes allocated for heap objects.
Type: float | (count)
Unit: digital,B

influxdb_shard

Tags & Fields Description
database
(tag)		 Database name.
engine
(tag)		 Engine.
host
(tag)		 System hostname.
id
(tag)		 ID.
index_type
(tag)		 Index type.
path
(tag)		 Path.
retention_policy
(tag)		 Retention policy.
wal_path
(tag)		 Wal path.
disk_bytes The size, in bytes, of the shard, including the size of the data directory and the WAL directory.
Type: float | (gauge)
Unit: digital,B
fields_create The number of fields created.
Type: float | (count)
Unit: count
series_create The number of series created.
Type: float | (count)
Unit: count
write_bytes The number of bytes written to the shard.
Type: float | (count)
Unit: digital,B
write_points_dropped The number of requests to write points that were dropped from a write.
Type: float | (count)
Unit: count
write_points_err The number of requests to write points that failed to be written due to errors.
Type: float | (count)
Unit: count
write_points_ok The number of points written successfully.
Type: float | (count)
Unit: count
write_req The total number of write requests.
Type: float | (count)
Unit: count
write_req_err The total number of write requests that failed due to errors.
Type: float | (count)
Unit: count
write_req_ok The total number of successful write requests.
Type: float | (count)
Unit: count
write_values_ok The number of write values written successfully.
Type: float | (count)
Unit: count

influxdb_subscriber

Tags & Fields Description
host
(tag)		 System hostname.
create_failures The number of subscriptions that failed to be created.
Type: float | (count)
Unit: count
points_written The total number of points that were successfully written to subscribers.
Type: float | (count)
Unit: count
write_failures The total number of batches that failed to be written to subscribers.
Type: float | (count)
Unit: count

influxdb_tsm1_cache

Tags & Fields Description
database
(tag)		 Database name.
engine
(tag)		 Engine.
host
(tag)		 System hostname.
id
(tag)		 ID.
index_type
(tag)		 Index type.
path
(tag)		 Path.
retention_policy
(tag)		 Retention policy.
wal_path
(tag)		 Wal path.
cache_age_ms The duration, in milliseconds, since the cache was last snapshotted at sample time.
Type: float | (gauge)
Unit: time,ms
cached_bytes The total number of bytes that have been written into snapshots.
Type: float | (count)
Unit: digital,B
disk_bytes The size, in bytes, of on-disk snapshots.
Type: float | (gauge)
Unit: digital,B
mem_bytes The size, in bytes, of in-memory cache.
Type: float | (gauge)
Unit: digital,B
snapshot_count The current level (number) of active snapshots.
Type: float | (gauge)
Unit: count
wal_compaction_time_ms The duration, in milliseconds, that the commit lock is held while compacting snapshots.
Type: float | (count)
Unit: time,ms
write_dropped The total number of writes dropped due to timeouts.
Type: float | (count)
Unit: count
write_err The total number of writes that failed.
Type: float | (count)
Unit: count
write_ok The total number of successful writes.
Type: float | (count)
Unit: count

influxdb_tsm1_engine

Tags & Fields Description
database
(tag)		 Database name.
engine
(tag)		 Engine.
host
(tag)		 System hostname.
id
(tag)		 ID.
index_type
(tag)		 Index type.
path
(tag)		 Path.
retention_policy
(tag)		 Retention policy.
wal_path
(tag)		 Wal path.
cache_compaction_duration The duration (wall time), in nanoseconds, spent in cache compactions.
Type: float | (count)
Unit: time,ns
cache_compaction_err The number of cache compactions that have failed due to errors.
Type: float | (count)
Unit: count
cache_compactions The total number of cache compactions that have ever run.
Type: float | (count)
Unit: count
cache_compactions_active The number of cache compactions that are currently running.
Type: float | (gauge)
Unit: count
tsm_full_compaction_duration The duration (wall time), in nanoseconds, spent in full compactions.
Type: float | (count)
Unit: time,ns
tsm_full_compaction_err The total number of TSM full compactions that have failed due to errors.
Type: float | (count)
Unit: count
tsm_full_compaction_queue The current number of pending TSM full compactions.
Type: float | (gauge)
Unit: count
tsm_full_compactions The total number of TSM full compactions that have ever run.
Type: float | (count)
Unit: count
tsm_full_compactions_active The number of TSM full compactions currently running.
Type: float | (gauge)
Unit: count
tsm_level1_compaction_duration The duration (wall time), in nanoseconds, spent in TSM level 1 compactions.
Type: float | (count)
Unit: time,ns
tsm_level1_compaction_err The total number of TSM level 1 compactions that have failed due to errors.
Type: float | (count)
Unit: count
tsm_level1_compaction_queue The current number of pending TSM level 1 compactions.
Type: float | (gauge)
Unit: count
tsm_level1_compactions The total number of TSM level 1 compactions that have ever run.
Type: float | (count)
Unit: count
tsm_level1_compactions_active The number of TSM level 1 compactions that are currently running.
Type: float | (gauge)
Unit: count
tsm_level2_compaction_duration The duration (wall time), in nanoseconds, spent in TSM level 2 compactions.
Type: float | (count)
Unit: time,ns
tsm_level2_compaction_err The number of TSM level 2 compactions that have failed due to errors.
Type: float | (count)
Unit: count
tsm_level2_compaction_queue The current number of pending TSM level 2 compactions.
Type: float | (gauge)
Unit: count
tsm_level2_compactions The total number of TSM level 2 compactions that have ever run.
Type: float | (count)
Unit: count
tsm_level2_compactions_active The number of TSM level 2 compactions that are currently running.
Type: float | (gauge)
Unit: count
tsm_level3_compaction_duration The duration (wall time), in nanoseconds, spent in TSM level 3 compactions.
Type: float | (count)
Unit: time,ns
tsm_level3_compaction_err The number of TSM level 3 compactions that have failed due to errors.
Type: float | (count)
Unit: count
tsm_level3_compaction_queue The current number of pending TSM level 3 compactions.
Type: float | (gauge)
Unit: count
tsm_level3_compactions The total number of TSM level 3 compactions that have ever run.
Type: float | (count)
Unit: count
tsm_level3_compactions_active The number of TSM level 3 compactions that are currently running.
Type: float | (gauge)
Unit: count
tsm_optimize_compaction_duration The duration (wall time), in nanoseconds, spent during TSM optimize compactions.
Type: float | (count)
Unit: time,ns
tsm_optimize_compaction_err The total number of TSM optimize compactions that have failed due to errors.
Type: float | (count)
Unit: count
tsm_optimize_compaction_queue The current number of pending TSM optimize compactions.
Type: float | (gauge)
Unit: count
tsm_optimize_compactions The total number of TSM optimize compactions that have ever run.
Type: float | (count)
Unit: count
tsm_optimize_compactions_active The number of TSM optimize compactions that are currently running.
Type: float | (gauge)
Unit: count

influxdb_tsm1_filestore

Tags & Fields Description
database
(tag)		 Database name.
engine
(tag)		 Engine.
host
(tag)		 System hostname.
id
(tag)		 ID.
index_type
(tag)		 Index type.
path
(tag)		 Path.
retention_policy
(tag)		 Retention policy.
wal_path
(tag)		 Wal path.
disk_bytes The size, in bytes, of disk usage by the TSM file store.
Type: float | (gauge)
Unit: digital,B
num_files The current number of files in the TSM file store.
Type: float | (gauge)
Unit: count

influxdb_tsm1_wal

Tags & Fields Description
database
(tag)		 Database name.
engine
(tag)		 Engine.
host
(tag)		 System hostname.
id
(tag)		 ID.
index_type
(tag)		 Index type.
path
(tag)		 Path.
retention_policy
(tag)		 Retention policy.
wal_path
(tag)		 Wal path.
current_segment_disk_bytes The current size, in bytes, of the segment disk.
Type: float | (gauge)
Unit: digital,B
old_segments_disk_bytes The size, in bytes, of old WAL segments on disk.
Type: float | (gauge)
Unit: digital,B
write_err The number of writes that failed due to errors.
Type: float | (count)
Unit: count
write_ok The number of writes that succeeded.
Type: float | (count)
Unit: count

influxdb_write

Tags & Fields Description
host
(tag)		 System hostname.
point_req The total number of every point requested to be written to this data node.
Type: float | (count)
Unit: count
point_req_local The total number of point requests that have been attempted to be written into a shard on the same (local) node.
Type: float | (count)
Unit: count
req The total number of batches of points requested to be written to this node.
Type: float | (count)
Unit: count
sub_write_drop The total number of batches of points that failed to be sent to the subscription dispatcher.
Type: float | (count)
Unit: count
sub_write_ok The total number of batches of points that were successfully sent to the subscription dispatcher.
Type: float | (count)
Unit: count
write_drop The total number of write requests for points that have been dropped due to timestamps not matching any existing retention policies.
Type: float | (count)
Unit: count
write_error The total number of batches of points that were not successfully written, due to a failure to write to a local or remote shard.
Type: float | (count)
Unit: count
write_ok The total number of batches of points written at the requested consistency level.
Type: float | (count)
Unit: count
write_timeout The total number of write requests that failed to complete within the default write timeout duration.
Type: float | (count)
Unit: count

Log Collection

To collect the InfluxDB log, open files in influxdb.conf and write to the absolute path of the InfluxDB log file. For example:

[inputs.influxdb.log]
    # Fill in the absolute path
    files = ["/path/to/demo.log"] 
    ## grok pipeline script path
    pipeline = "influxdb.p"