./sysutils/beats, Data Shippers for Elasticsearch

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 6.0.0, Package name: beats-6.0.0, Maintainer: filip

The Beats are lightweight processes, written in Go, that you install
on your servers to capture all sorts of operational data like logs,
operating system metrics or network packet data, and to send it to
Elasticsearch, either directly or via Logstash, so it can be
visualized with Kibana.


Required to build:
[lang/go] [pkgtools/cwrappers]

Master sites:

SHA1: b9e80f854c7042b6f5110393d9a09733026f4ad5
RMD160: 0eaee5bf8e8ff764e1e09bb3be8c52ecc06ef8f3
Filesize: 14764.394 KB

Version history: (Expand)


CVS history: (Expand)


   2017-11-20 16:55:16 by Filip Hajny | Files touched by this commit (3) | Package updated
Log message:
Update sysutils/beats to 6.0.0.

=== Beats version 6.0.0

==== Breaking changes

Affecting all Beats

- The log directory (`path.log`) for Windows services is now set to
  `C:\ProgramData\[beatname]\logs`.
- The _all field is disabled in Elasticsearch 6.0.
- Fail if removed setting output.X.flush_interval is explicitly
  configured.
- Rename the `/usr/bin/beatname.sh` script (e.g. `metricbeat.sh`) to
  `/usr/bin/beatname`.
- Beat does not start if elasticsearch index pattern was modified but
  not the template name and pattern.
- Fail if removed setting output.X.flush_interval is explicitly
  configured.
- Rename `kubernetes` processor to `add_kubernetes_metadata`.
- Rename `.full.yml` config files to `*.reference.yml`.
- The `scripts/import_dashboards` is removed from packages. Use the
  `setup` command instead.
- Change format of the saved kibana dashboards to have a single JSON
  file for each dashboard
- Rename `configtest` command to `test config`.
- Remove setting `queue_size` and `bulk_queue_size`.
- Remove setting `dashboard.snapshot` and `dashboard.snapshot_url`.
- Beats can no longer be launched from Windows Explorer (GUI), command
  line is required.

Filebeat

- Rename `input_type` field to `prospector.type`
- The `@metadata.type` field, added by the Logstash output, is now
  hardcoded to `doc` and will be removed in future versions.

Metricbeat

- Change all `system.cpu..pct` metrics to be scaled by the number of
  CPU cores.
- Remove filters setting from metricbeat modules.
- Added `type` field to filesystem metrics.

Heartbeat

- Renamed the heartbeat RPM/DEB name to `heartbeat-elastic`.

Packetbeat

- Remove not-working `runoptions.uid` and `runoptions.gid` options in
  Packetbeat.
- Remove the already unsupported `pf_ring` sniffer option.

Auditbeat

- Changed file metricset config to make `file.paths` a list instead of
  a dictionary.

==== Bugfixes

Affecting all Beats

- Fix data race accessing watched containers.
- Do not require template if index change and template disabled
- Fix missing ACK in redis output.
- Fix the `/usr/bin/beatname` script to accept `-d ""` as a parameter.
- Combine `fields.yml` properties when they are defined in different
  sources.
- Keep Docker & Kubernetes pod metadata after container dies while
  they are needed by processors.
- Fix `fields.yml` lookup when using `export template` with a custom
  `path.config` param.
- Remove runner creation from every reload check
- Fix add_kubernetes_metadata matcher registry lookup.
- Register kubernetes `field_format` matcher and remove logger in
  `Encode` API
- Fix go plugins not loaded when beat starts
- Add support for `initContainers` in `add_kubernetes_metadata`
  processor.
- Eliminate deprecated _default_ mapping in 6.x
- Fix pod name indexer to use both namespace, pod name to frame index
  key
- Don't stop with error loading the ES template if the ES output is
  not enabled.
- Fix race condition in internal logging rotator.
- Normalize all times to UTC to ensure proper index naming.
- Fix issue with loading dashboards to ES 6.0 when .kibana index did
  not already exist.
- Fix importing the dashboards when the limit for max open files is
  too low.
- Fix configuration documentation for kubernetes processor
- Fix misspelling in `add_locale` configuration option for
  abbreviation.

Filebeat

- Fix machine learning jobs setup for dynamic modules.
- Fix default paths for redis 4.0.1 logs on macOS
- Fix Filebeat not starting if command line and modules configs are
  used together.
- Fix double `@timestamp` field when JSON decoding was used.
- Fix issue where the `fileset.module` could have the wrong value.
- Fix race condition on harvester stopping with reloading enabled.
- Fix recursive glob config parsing and resolution across restarts.
- Allow string characters in user agent patch version (NGINX and
  Apache)
- Fix grok pattern in filebeat module system/auth without hostname.

Winlogbeat

- Removed validation of top-level config keys.

Metricbeat

- Use `beat.name` instead of `beat.hostname` in the Host Overview
  dashboard.
- Fix the loading of 5.x dashboards.
- Fix a memory allocation issue where more memory was allocated than
  needed in the windows-perfmon metricset.
- Don't start metricbeat if external modules config is wrong and
  reload is disabled
- The MongoDB module now connects on each fetch, to avoid stopping the
  whole Metricbeat instance if MongoDB is not up when starting.
- Fix kubernetes events module to be able to index time fields
  properly.
- Fixed `cmd_set` and `cmd_get` being mixed in the Memcache module.
- Added missing mongodb configuration file to the `modules.d` folder.
- Fix wrong MySQL CRUD queries timelion visualization
- Add new metrics to CPU metricsset
- Fix issue affecting Windows services timing out at startup.
- Fix incorrect docker.diskio.total metric calculation.
- Vsphere module: used memory field corrected.
- Set correct format for percent fields in memory module.
- Fix a debug statement that said a module wrapper had stopped when it
  hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.
- Add filtering to system filesystem metricset to remove relative
  mountpoints like those from Linux network namespaces.
- Remove unnecessary print statement in schema apis.
- Fix type of field `haproxy.stat.check.health.last`.

Heartbeat

- Fix monitor.name being empty by default.
- Fix wrong event timestamps.

Packetbeat

- Fix missing length check in the PostgreSQL module.
- Fix panic in ACK handler if event is dropped on blocked queue
- Update flow timestamp on each packet being received.
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.
- Enable memcache filtering only if a port is specified in the config
  file.

Auditbeat

- Fix `file.max_file_size` config option for the audit file metricset.

==== Added

Affecting all Beats

- Enable flush timeout by default.
- Add @metadata.version to events send to Logstash.
- Add setting to enable/disable the slow start in logstash output.
- Update init scripts to use the `test config` subcommand instead of
  the deprecated `-configtest` flag.
- Get by default the credentials for connecting to Kibana from the
  Elasticsearch output configuration.
- Added `cloud.id` and `cloud.auth` settings, for simplifying using
  Beats with the Elastic Cloud.
- Add lz4 compression support to kafka output.
- Add newer kafka versions to kafka output.
- Configure the index name when loading the dashboards and the index
  pattern.
- New cli subcommands interface.
- Allow source path matching in `add_docker_metadata` processor.
- Add support for analyzers and multifields in fields.yml.
- Add support for JSON logging.
- Add `test output` command, to test Elasticsearch and Logstash output
  settings.
- Introduce configurable event queue settings: queue.mem.events,
  queue.mem.flush.min_events and queue.mem.flush.timeout.
- Enable pipelining in Logstash output by default.
- Added 'result' field to Elasticsearch QueryResult struct for
  compatibility with 6.x Index and Delete API responses.
- The sample dashboards are now included in the Beats packages.
- Add `pattern` option to be used in the fields.yml to specify the
  pattern for a number field.
- Upgraded to Golang 1.8.3.
- Added the possibility to set Elasticsearch mapping template settings
  from the Beat configuration file.
- Add a variable to the SysV init scripts to make it easier to change
  the user.
- Add the option to write the generated Elasticsearch mapping template
  into a file.
- Add `instance_name` in GCE add_cloud_metadata processor.
- Add `add_docker_metadata` processor.
- Add `logging.files` `permissions` option.

Filebeat

- Add Kubernetes manifests to deploy Filebeat.
- Changed the number of shards in the default configuration to 3.
- Don't start filebeat if external modules/prospectors config is wrong
  and reload is disabled
- Add `filebeat.registry_flush` setting, to delay the registry
  updates.
- Add experimental Redis module.
- Nginx module: use the first not-private IP address as the remote_ip.
- Load Ingest Node pipelines when the Elasticsearch connection is
  established, instead of only once at startup.
- Add support for loading Xpack Machine Learning configurations from
  the modules, and added sample configurations for the Nginx module.
- Add udp prospector type.
- Enabled Cgo which means libc is dynamically compiled.
- Add Beta module config reloading mechanism
- Remove spooler and publisher components and settings.
- Added ability to sort harvested files.
- Add experimental Redis slow log prospector type.

Winlogbeat

- Changed the number of shards in the default configuration to 3.
- Add the ability to use LevelRaw if Level isn't populated in the
  event XML.

Metricbeat

- Add Kubernetes manifests to deploy Metricbeat.
- Auto-select a hostname (based on the host on which the Beat is
  running) in the Host Overview dashboard.
- Add `filesystem.ignore_types` to system module for ignoring
  filesystem types.
- Add support to exclude labels from kubernetes pod metadata.
- Add random startup delay to each metricset to avoid the thundering
  herd problem.
- Add the ability to configure audit rules to the kernel module.
- Add the ability to configure kernel's audit failure mode.
- Add experimental Aerospike module.
- Vsphere module: collect custom fields from virtual machines.
- Add `test modules` command, to test modules expected output.
- Add `processors` setting to metricbeat modules.
- Support `npipe` protocol (Windows) in Docker module.
- Add macOS implementation of the system diskio metricset.
- Add process_summary metricset that records high level metrics about
  processes.
- Add `kube-state-metrics` based metrics to `kubernetes` module
- Add debug logging to Jolokia JMX metricset.
- Add events metricset for kubernetes metricbeat module
- Change Metricbeat default configuration file to be better optimized
  for most users.
- Add experimental RabbitMQ module.
- Add Kibana dashboard for the Kubernetes modules.

Heartbeat

- Changed the number of shards in the default configuration to 1.
- Enabled Cgo which means libc is dynamically compiled.

Packetbeat

- Changed the number of shards in the default configuration to 3.

Auditbeat

- Changed the number of shards in the default configuration to 3.
- Add support for receiving audit events using a multicast socket.
- Added `file.hash_types` config option for controlling the hash
  types.
- Added the ability to specify byte unit suffixes to
  `file.max_file_size`.
- Add file integrity metricset to the audit module.

==== Deprecated

Affecting all Beats

- The `@metadata.type` field, added by the Logstash output, is
  deprecated, hardcoded to `doc` and will be removed in future
versions.

Filebeat

- The `filebeat.config_dir` option is deprecated. Use
  `filebeat.config.prospector` options instead.
- Deprecate `input_type` prospector config. Use `type` config option
  instead.

=== Beats version 5.6.4

- Fix race condition in internal logging rotator.
- Add support for enabling TLS renegotiation.
- Add setting to enable/disable the slow start in logstash output.
- Packetbeat: Fix missing length check in the PostgreSQL module.
   2017-11-02 16:30:50 by Filip Hajny | Files touched by this commit (1)
Log message:
sysutils/beats: Provide a better hint to where libpcap is. May fix
the build in some situations.
   2017-10-17 13:31:36 by Filip Hajny | Files touched by this commit (2) | Package updated
Log message:
Update sysutils/beats to 5.6.3.

No changes - sync version with databases/elasticsearch.
   2017-09-27 14:40:44 by Filip Hajny | Files touched by this commit (2) | Package updated
Log message:
Update sysutils/beats to 5.6.2.

=== Beats version 5.6.0

==== Breaking changes

Affecting all Beats
- The _all.norms setting in the Elasticsearch template is no longer
  disabled.

==== Bugfixes

Filebeat
- Fix issue where the `fileset.module` could have the wrong value.

Packetbeat
- Update flow timestamp on each packet being received.

Metricbeat
- Fix a debug statement that said a module wrapper had stopped when it
  hadn't.
- Use MemAvailable value from /proc/meminfo on Linux 3.14.
- Fix panic when events were dropped by filters.

==== Added

Affecting all Beats
- Add option to the import_dashboards script to load the dashboards via
  Kibana API.

Filebeat
- Add support for loading Xpack Machine Learning configurations from the
  modules, and added sample configurations for the Nginx module.
- Add ability to parse nginx logs exposing the X-Forwarded-For header
  instead of the remote address.

Metricbeat
- Add `filesystem.ignore_types` to system module for ignoring filesystem
  types.

==== Deprecated

Affecting all Beats
- Loading more than one output is deprecated and will be removed in 6.0.
   2017-09-22 14:25:44 by Tobias Nygren | Files touched by this commit (2)
Log message:
beats: regen distinfo, bump rev

Upstream moved the release tag after a documentation fix.
   2017-08-18 11:08:54 by Filip Hajny | Files touched by this commit (2)
Log message:
=== Beats version 5.5.2

No changes in this release.

=== Beats version 5.5.1

==== Bugfixes

Affecting all Beats
- Normalize all times to UTC to ensure proper index naming.

=== Beats version 5.5.0

==== Breaking changes

Affecting all Beats
- Usage of field `_type` is now ignored and hardcoded to `doc`.

Metricbeat
- Change all `system.cpu.*.pct` metrics to be scaled by the number of
  CPU cores.

==== Bugfixes

Affecting all Beats
- Fix console output.

Filebeat
- Allow string characters in user agent patch version (NGINX and Apache)

Metricbeat
- Fix type of field `haproxy.stat.check.health.last`.

Packetbeat
- Fix `packetbeat.interface` options that contain underscores (e.g.
  `with_vlans` or `bpf_filter`).
- Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing.

==== Deprecated

Filebeat
- Deprecate `document_type` prospector config option as _type is removed
  in elasticsearch 6.0. Use fields instead.

Winlogbeat
- Deprecated metrics endpoint. It is superseded by a libbeat feature
  that can serve metrics on an HTTP endpoint.
   2017-07-23 09:20:42 by Thomas Klausner | Files touched by this commit (3)
Log message:
More RELRO check disabling for go packages.
   2017-07-04 16:22:48 by Filip Hajny | Files touched by this commit (2) | Package updated
Log message:
Update sysutils/beats to 5.4.3.

Doc changes only.