./www/logswan, Fast Web log analyzer using probabilistic data structures

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]

Branch: CURRENT, Version: 2.0.3, Package name: logswan-2.0.3, Maintainer: fred

Logswan is a fast Web log analyzer using probabilistic data structures. It is
targeted at very large log files, typically APIs logs. It has constant memory
usage regardless of the log file size, and takes approximatively 4MB of RAM.

Unique visitors counting is performed using two HyperLogLog counters (one for
IPv4, and another one for IPv6), providing a relative accuracy of 0.10%.

Project design goals include : speed, memory-usage efficiency, and keeping the
code as simple as possible.

Required to run:
[textproc/jansson] [geography/libmaxminddb]

Required to build:

Master sites:

SHA1: 6b80763f07b1231f6f2eaf80db2ea6eadffa7ca6
RMD160: 7a95859cac7d661af273938319f0298899a304ff
Filesize: 21.288 KB

Version history: (Expand)

CVS history: (Expand)

   2019-01-17 22:56:09 by Kamil Rytarowski | Files touched by this commit (2) | Package updated
Log message:
logswan: Update to 2.0.3

Update submitted by Frederic Cambus, the upstream developer of this package.

author's notes

Notable changes:

- Logswan now links against geography/libmaxminddb instead of net/GeoIP
- Take back MAINTAINER, as I'm upstream for this package

upstream changelog

Logswan 2.0.3 (2018-10-15)

- Use -std=c11, Logswan requires a C11 compiler for libmaxminddb
- Enable support for parsing HTTP/2.0 requests, for real this time
- Revert back to using INT64_MAX for strtonum() maxval, as maxval is long long
  and using UINT64_MAX caused bandwidth parsing to always fail (Thanks James Loh
  for reporting the issue)
- Move maps of countries and continents to separate files

Logswan 2.0.2 (2018-08-05)

- Use UINT64_MAX for strtonum() maxval
- Add missing headers and reorder includes
- Add support for HTTP/2.0

Logswan 2.0.1 (2018-06-27)

- Do not use -Werror by default
- Do not always call inet_pton two times per log line, this speeds things up
- Use bool types for isIPv4 and isIPv6
- Use the monotonic clock to determine runtime
- Move conditional includes for 'pledge' and 'strtonum' in compat.h

Logswan 2.0.0 (2018-03-16)

- Use type off_t for results struct member fileSize
- Reflect OpenBSD's pledge() changes
- Switch to using libmaxminddb and GeoIP2/GeoLite2 databases
- Add Antarctica to the list of continents
   2017-08-16 22:21:18 by Thomas Klausner | Files touched by this commit (180)
Log message:
Follow some http redirects.
   2017-02-15 21:51:57 by Maya Rashish | Files touched by this commit (2) | Package updated
Log message:
logswan: update to 1.07. Add simple smoketest.

Changes since 1.06:
    Harmonize arrays names
    Remove array of months, it's currently unused and will likely remain so
    Simplify internal JSON array and object names
    Use OpenBSD style(9) for function prototypes and declarations
    Revert back to using strtok, at least for now
    Do not use EXIT_SUCCESS and EXIT_FAILURE macros anymore
    Fix implicit function declaration rrror on NetBSD (Thanks Maya Rashish)
    Remove now useless variables initialization and unnecessary includes
    Do not add an extra new line when displaying usage or version
   2017-01-12 18:53:01 by Maya Rashish | Files touched by this commit (1) | Package updated
Log message:
Add note to drop hack next update, it was fixed upstream.
   2016-12-18 19:49:39 by Maya Rashish | Files touched by this commit (2) | Package updated
Log message:
logswan: update to 1.06.

Changed license to 2-clause BSD (from 3-clause)
Minor code shuffling.

From fcambus (author)
   2016-09-11 17:59:29 by Kamil Rytarowski | Files touched by this commit (3)
Log message:
Reset maintainer to pkgsrc-users@NetBSD.org

Change requested by Frederic Cambus.
   2016-07-26 19:15:37 by Kamil Rytarowski | Files touched by this commit (3) | Package updated
Log message:
Update www/logswan to 1.05

Package changes:

- Trimmed package description
- Update MAINTAINER + added missing trailing slash for HOMEPAGE

Upstream ChangeLog:
Logswan 1.05 (2016-02-25)

- Documentation update (notes on measuring Logswan memory usage)
- Add additional include directories for compat functions + dependencies
  to avoid using relative path in includes
- Check that *lineBuffer is not NUL before attempting to parse log line
- Perform GeoIP lookup and HLL add in the same if block
- Increment IPv4 and IPv6 hits counters individually and conditionally
- Use CMake to check if the system has OpenBSD's pledge available and
  link pledge conditionally using a null implementation when compiled
  on non OpenBSD systems
- Adding an array of months, for the upcoming split log functionalities

Patch provided by the upstream developer Frederic Cambus
   2016-01-11 08:11:20 by Kamil Rytarowski | Files touched by this commit (3) | Package updated
Log message:
Update www/logswan to 1.04

Package changes:
- follow upstream with the rule what's installed of the files
- use newly added -DGEOIPDIR in the place of older string replace function
- stop installing an example output as a configuration file

Upstream changelog:
Logswan 1.04 (2016-01-10)

- Moving global variables into main
- Using 'size_t' instead of 'int' for array indexes in for loops
- Using 'uint32_t' for all non 'uint64_t' integers
- Do not increment hits and processed lines counter for each parsed line,
  compute total only once when everything is parsed
- Setting 'CMAKE_BUILD_TYPE' to 'Release' and formatting fixes
- Sanitize CMake script to build under NetBSD (Thanks Kamil Rytarowski)
- Initializing some uninitialized variables
- Renaming 'DATADIR' variables to 'GEOIPDIR'

Logswan 1.03 (2016-01-01)

- Remove header display and do not print name of processed file
- Print results to stderr instead of stdout
- Output JSON data to stdout instead of creating a new file
- Define GeoIP databases path in CMakeLists.txt
- Adding log file name in the JSON output
- Removing some hardcoded values and replacing them with constants
  defined in config.h
- Breaking the loop when a match is found in the request parser
- Using enumeration constants instead of macros
- Process GeoIP continent information
- Re-ordering protocols and methods with more common occurences on top of
  the list, allowing to break earlier when iterating through the array
- Adding support for reading logs from standard input
- Renaming 'definitions' files to 'config'
- Increasing countries array size, as an attempt to be future-proof
- Initial support for using pledge() on OpenBSD
- Documentation updates (HLL precision, Features list, GeoIP databases)
- Updated JSON output example
- Added a manual page

Logswan 1.02 (2015-11-02)

- Renaming 'resource' variable to 'request' in the 'logLine' struct
- Do not attempt to parse empty date tokens
- Do not attempt to parse empty request tokens (Thanks Brian Carpenter for
reporting the issue)