./mail/rspamd, Fast, free and open-source spam filtering system

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 2.1, Package name: rspamd-2.1, Maintainer: bartosz.kuzma

Rspamd is fast, modular and lightweight spam filter. It is designed to work
with big ammount of mail and can be easily extended with own filters written in
lua.


Required to run:
[textproc/icu] [security/gnupg2] [mail/gmime] [devel/fann] [devel/glib2] [devel/ragel] [devel/lua-lpeg] [security/libsodium] [databases/hiredis] [lang/lua53]

Required to build:
[pkgtools/cwrappers]

Master sites:

SHA1: 17e7e7025c66bf7c15313a58adc1bd9ee0fac4b0
RMD160: 223eadaa92fc6bf9ca0c25f83dccd2d5a9acb2cc
Filesize: 4133.103 KB

Version history: (Expand)


CVS history: (Expand)


   2019-11-17 12:11:21 by Thomas Klausner | Files touched by this commit (1)
Log message:
rspamd: guess correction for weird BUILDLINK_FNAME_TRANSFORM
   2019-11-05 20:16:32 by Roland Illig | Files touched by this commit (2)
Log message:
mail/rspamd: make cfgfiles.mk conform to the preferred code style
   2019-11-03 19:19:32 by =?UTF-8?B?RnLDqWTDqXJpYyBGYXViZXJ0ZWF1?= | Files touched by this commit (4) | Package updated
Log message:
rspamd: update to 2.1

pkgsrc changes:
---------------
  * Change BUILDLINK_TRANSFORM to BUILDLINK_FNAME_TRANSFORM to appease
    pkglint.
  * Add c++ to USE_LANGUAGES because the configure step failed.

upstream changes:
-----------------
2.1: 28 Oct 2019
  * [Conf] Update neural.conf
  * [CritFix] Fix dkim verification for multiple headers listed
  * [Feature] Add support of uudecode
  * [Feature] Allow to explicitly set events backend
  * [Feature] Implement configurable limits for SPF lookups
  * [Feature] Lua_scanners: Use lua magic for inclusion/exclusion logic
  * [Feature] Multimap: Do not check files in office archives
  * [Feature] Neural: Add sampling when storing training vectors
  * [Feature] SPF: Allow to disable AAAA checks in configuration
  * [Feature] Spf: Add limits configuration support
  * [Feature] Store etag in cached HTTP maps + better logging
  * [Feature] Support segwit BTC addresses, fix LTC verification
  * [Feature] Support uuencoding
  * [Fix] Add configurable number of threads for OpenBLAS
  * [Fix] Add workaround for ragel 7 in hyperscan related maps code
  * [Fix] Another fix for numeric urls parsing
  * [Fix] Correct EMA time calculations
  * [Fix] Do not treat archives as text
  * [Fix] Do not use strdup on data extracted from lua
  * [Fix] Fix a failure calcuating URL reputation.
  * [Fix] Fix crash due to constructors init order
  * [Fix] Fix crash on parts with no cd
  * [Fix] Fix empty prefilters that require mime structures
  * [Fix] Fix event loop creation
  * [Fix] Fix issues sending DMARC reports.
  * [Fix] Fix misprint
  * [Fix] Fix saving of the file maps
  * [Fix] Fix size calculations when converting from utf16
  * [Fix] Fix support of disable_monitoring in rbl
  * [Fix] Fix use-after-free
  * [Fix] Fix zip files check to relax requirements
  * [Fix] Important hiredis fixes
  * [Fix] Lot's of fixes in maps check logic
  * [Fix] Lua_tcp: Deal with temporary fails on write
  * [Fix] Lua_tcp: Make write errors fatal and rework error handlers
  * [Fix] Meta: Filter some more values
  * [Fix] Neural: Add protection agains infinities
  * [Fix] Oops, fix math.huge invocation
  * [Fix] Plug memory leak
  * [Fix] Sigh, another email to string fix
  * [Fix] Try to fix another ownership race in ssl connection
  * [Fix] Uuencode: Fix parsing of corrupted uuencode
  * [Fix] lua_scanners - razor rename need_check function
  * [Rework] Require CMake 3.9 to work, remove manual lto crap

2.0: 11 Oct 2019
  * [Conf] Add BROKEN_HEADERS_MAILLIST composite
  * [Conf] Add path to greylist-whitelist-domains.inc
  * [Conf] Clarify documentation in the config files
  * [Conf] Introduce maps.d directories
  * [Conf] Log settings id by default
  * [Conf] Make LEAKED_PASSWORD_SCAM a composite rule again
  * [Conf] Move all surbl/emails rules to rbl
  * [Conf] Register new Spamhaus codes
  * [Conf] Remove configs for deleted modules
  * [Conf] Remove surbl parts, fix hash_format attribute
  * [Conf] Show autolearn sample
  * [Conf] Slashing: Change default stats backend to Redis
  * [Conf] Surbl: Utilise new `check_emails` option
  * [Conf] Update header
  * [Conf] Use multi-prefixes RBLs in the default config
  * [CritFix] Deal with case-sensivity in Content-Disposition parser
  * [CritFix] Eliminate old endpoint
  * [CritFix] Fix case sensivity when parsing Content-Type
  * [CritFix] Fix loading of DKIM public keys
  * [CritFix] Fix procesing of urls
  * [CritFix] Fix whitelisting when both spf and dkim are required to be valid
  * [CritFix] Langdet: Fix language detection where no stop words found
  * [Feature] Add description to the groups
  * [Feature] Add limit for number of URLs in Lua
  * [Feature] Add logging of groups to the log_format
  * [Feature] Add lua_smtp library
  * [Feature] Add maps cache and type refinement
  * [Feature] Add p0f scanner
  * [Feature] Adopt emails module to use lua_maps
  * [Feature] Allow options matching in composites
  * [Feature] Allow selectors in rbl module
  * [Feature] Allow to output group results
  * [Feature] Asn: Allow to use bgpdump when NET::MRT is broken
  * [Feature] Calculate tokens occurrences distribution
  * [Feature] Clickhouse: Add authenticated user and settings id columns
  * [Feature] Clickhouse: Store groups data
  * [Feature] Clickhouse: Utilise LowCardinality feature
  * [Feature] Implement Redis prefixes registration logic
  * [Feature] Implement settings id propagation between deps
  * [Feature] Improve AV results caching
  * [Feature] Improve autolearning
  * [Feature] Improve logging locking logic (remove it actually)
  * [Feature] Improve settings processing
  * [Feature] Langdet: Limit number of stop words to be checked
  * [Feature] Libucl: Allow to sort keys in ucl objects
  * [Feature] Lua_config: Extend get symbols method
  * [Feature] Lua_maps: Allow static maps for key-value pairs
  * [Feature] Lua_mimepart: Add function filter_words
  * [Feature] Lua_selectors: Add `words` selector
  * [Feature] Lua_selectors: Add sort and uniq transform functions
  * [Feature] Lua_selectors: Allow table arguments for selectors
  * [Feature] Lua_tcp: Add preliminary support of SSL connections
  * [Feature] Lua_trie: More flexible API
  * [Feature] Lua_util: Add filter_specific_url function
  * [Feature] Lua_util: table_digest can now recursively traverse tables
  * [Feature] Maillist: Improve detection
  * [Feature] Maps: Allow caching for complex maps
  * [Feature] Monitored: Support random lookups
  * [Feature] Multimap: Add combined maps prototype
  * [Feature] Multimap: Add dependend maps via redis keys selectors
  * [Feature] Multimap: Allow multiple email addresses matches
  * [Feature] Multimap: Also check detected charset when do filename checks
  * [Feature] Output number of messages processed to proctitle
  * [Feature] Perform clean SSL shutdown
  * [Feature] Performance: Do not use base64 SIMD version for bad inputs
  * [Feature] RBL: Support bit results in replies
  * [Feature] RBL: Support type specific prefixes
  * [Feature] Ratelimit: Consider number of SMTP recipients
  * [Feature] Rbl: Add ability to check urls
  * [Feature] Rbl: Add resolve_ip based RBLs
  * [Feature] Rbl: Make config checks much more strict
  * [Feature] Rbl: Support per-rule whitelists
  * [Feature] Rbl: Support process script
  * [Feature] Rbl: Support replyto addresses
  * [Feature] SURBL: Allow to check email domains
  * [Feature] Selectors: Add `list` generator
  * [Feature] Selectors: Add `specific_urls` extractor
  * [Feature] Selectors: Add flatten function
  * [Feature] Selectors: Support filter_map and apply_map functions
  * [Feature] Store Clickhouse data outside of lua alloc
  * [Feature] Support caching for encrypted files and macros
  * [Feature] Support images when extracting urls
  * [Feature] Support more hyperscan flags
  * [Feature] Support protocol flags
  * [Feature] URL: Apply stringprep to hostnames to filter garbage
  * [Feature] Upstreams: Add lazy resolving logic to all upstreams
  * [Feature] Upstreams: Set noresolve flag on numeric upstreams
  * [Feature] Use `scores` in apply section
  * [Feature] Use maps logic from lua_maps for multimap
  * [Feature] Use random monitored in rbl module
  * [Feature] lua_scanners - add Razor support
  * [Fix] Add another safe-guard in urls processing
  * [Fix] Add debug to ssl, fixed write hangs
  * [Fix] Add missing groups to C callback symbols
  * [Fix] Add more checks for ghosts symbols
  * [Fix] Allow to enable or add new actions via settings
  * [Fix] Allow to set 0 size for spf/dkim caches
  * [Fix] Another bunch of fixes towards protocol mess
  * [Fix] Another fix to deal with bad URLs
  * [Fix] Arc: Another bunch of fixes for arc signing
  * [Fix] Arc: More arc signing fixes
  * [Fix] Avoid another overflow in fpconv
  * [Fix] Clickhouse: Fix quoting
  * [Fix] Clickhouse: Fix retention query quoting
  * [Fix] Distinguish empty and non-empty prefilters
  * [Fix] Distinguish remote and local addrs parsing
  * [Fix] Do not assert if length of sig is bad, just fail verification
  * [Fix] Do not assert if we have broken mime boundary in the headers
  * [Fix] Do not call implicit strlen to avoid issues
  * [Fix] Do not count images urls when checking url regexps for compatibility
  * [Fix] Do not output rbl suffix in symbol option
  * [Fix] Do not use config pool to avoid issues with double reload
  * [Fix] Do not use ephemeral string
  * [Fix] Do not use lightuserdata for traceback
  * [Fix] Do not use priority in metric registration
  * [Fix] Emails: Check email sanity before testing on BL
  * [Fix] Emails: Fix misprint in key name
  * [Fix] Escape utf in regexp to dodge ragel/hyperscan issue
  * [Fix] Extend task_timeout to postfilters stage
  * [Fix] Fix ARC signing after fixing another bug in it...
  * [Fix] Fix AV scan logic
  * [Fix] Fix DMARC_NA behaviour in case of no valid policies
  * [Fix] Fix LRU hash iteration logic
  * [Fix] Fix alignment mess
  * [Fix] Fix configuring symbols without scores
  * [Fix] Fix disabling of the actions
  * [Fix] Fix dkim signing exceptions
  * [Fix] Fix embedded images linking logic
  * [Fix] Fix events leak
  * [Fix] Fix eviction corner case
  * [Fix] Fix fuzzy image score calculation #2962
  * [Fix] Fix hang in fuzzy_learn when explicit rotation is set
  * [Fix] Fix headers propagation logic
  * [Fix] Fix hearbeats restart issue
  * [Fix] Fix history reset
  * [Fix] Fix log parameter
  * [Fix] Fix lua_ip_equal logic
  * [Fix] Fix more issues with nested messages + tests
  * [Fix] Fix normalization of non-alphabet based languages
  * [Fix] Fix offsets when parsing message/rfc822 in multipart
  * [Fix] Fix options in rbl symbols
  * [Fix] Fix out of bound access in lua logger
  * [Fix] Fix out-of-bound read in qp decode
  * [Fix] Fix parent CTE propagation
  * [Fix] Fix parsing of the received headers with empty part
  * [Fix] Fix pending checks for events
  * [Fix] Fix printing of NULL pointer with fixed length
  * [Fix] Fix race condition in watcher handler
  * [Fix] Fix read-after-end in quoted printable decoding
  * [Fix] Fix redis sentinel support
  * [Fix] Fix registry leak in case of DNS errors
  * [Fix] Fix reload logic
  * [Fix] Fix sending of large entries via HTTPS
  * [Fix] Fix settings reload
  * [Fix] Fix some more corner cases for fpconv
  * [Fix] Fix trie code when there are regexps and Hyperscan is absent
  * [Fix] Further fixes to printing of the FP numbers
  * [Fix] Fuzzy_check: Fix timeouts
  * [Fix] Grrr, fix empty ip case
  * [Fix] Html: Fix processing of fjlig entity
  * [Fix] Lang_det: Try better to distinguish Chinese and Japanese
  * [Fix] Lua_mime: Fix reversed extensions map
  * [Fix] Lua_task: Fix message-less API
  * [Fix] Lua_tcp: Report connection failures
  * [Fix] Lua_tcp: Various fixes and debugging improvements
  * [Fix] Metadata_exporter: This plugin is idempotent not a postfilter
  * [Fix] More fixes to extract_specific_urls
  * [Fix] More stages fixes
  * [Fix] Neural: Another bunch of fixes
  * [Fix] Neural: use version in ANN key profile
  * [Fix] Postpone lua state destruction to allow lua dtors to be used
  * [Fix] Prefer surbl/emails rule on rbl to preserve compatibility
  * [Fix] RBL: Fix behaviour of emails_domainonly
  * [Fix] Ratelimit: Fix dynamic score
  * [Fix] Rbl: Fix emailbl functions
  * [Fix] Really fix hyperscan workaround
  * [Fix] Set sanity limits for pcre2
  * [Fix] Settings: Fix settings check flags
  * [Fix] Sort keys when getting data from Lua when filling rules
  * [Fix] Statistics: Do not query Redis tokens when there are no learns
  * [Fix] Stop IO event on write finished in http connection
  * [Fix] Use heuristically detected text parts data
  * [Fix] Various fixes to QP encoding algorithm
  * [Fix] Various fixes to SSL state machine handler
  * [Fix] Various fixes to asn module
  * [Fix] Workaround for empty charset in rfc2231 encoding
  * [Project] Switch from torch to KANN
  * [Project] Add heartbeat events
  * [Project] Add preliminary support of the Kaspersky Scan Engine
  * [Project] Add preliminary version of maps expressions
  * [Project] Add preprocessed settings to the config structure
  * [Project] Add simple forward propagation function
  * [Project] Add small helpers for migration simplifications
  * [Project] Allow to replace body in milter
  * [Project] Bundle libev
  * [Project] First refactoring step libevent->libev
  * [Project] Implement syntax highlighting for Lua
  * [Project] Lua_magic: Adopt lua_magic stuff in mime_types
  * [Project] Remove libfann, gd and other unsupported stuff
  * [Project] Remove torch
  * [Project] Rework upstreams
  * [Rework] Allow execution of async events when hs compiles regexps
  * [Rework] Bayes expiry: eliminate `default` expiration mode
  * [Rework] Dkim: Remove signing code
  * [Rework] Dkim_signing: Move sign condition to dkim_signing
  * [Rework] Do not lowercase all data send to ClickHouse
  * [Rework] Drop url tags
  * [Rework] Eliminate lua_squeeze as it has shown no improvements
  * [Rework] Eliminate virtual scan time as it is useless
  * [Rework] Lua core: Use lightuserdata to index classes
  * [Rework] Lua_util: Another rework for extract_specific_urls
  * [Rework] Migrate from ip_score to reputation
  * [Rework] Move mime modification functions to lua_mime library
  * [Rework] Rbl: Major whitelisting logic rework
  * [Rework] Remove deprecated plugins
  * [Rework] Remove log helper worker
  * [Rework] Remove rspamd.classifiers.lua
  * [Rework] Rename filter.h to a more sane name
  * [Rework] Reorganise selectors implementation
  * [Rework] Replace linenoise with replxx
  * [Rework] Reputation: Remove ipnet from the ip reputation
  * [Rework] Reputation: Slashing - change name of symbols
  * [Rework] Rework children operations
  * [Rework] Rework config reload
  * [Rework] Rework expression API
  * [Rework] Rework image urls processing
  * [Rework] Rework initialisation to reduce static leaks count
  * [Rework] Rework request headers processing
  * [Rework] Slashing: Change versioning schema - move to 2.0
  * [Rework] Slashing: Turn off postfilters when passthrough result is set
  * [Rework] Start moving to replxx
  * [Rework] Stop support of signed HTTP maps to simplify code
  * [Rework] Store ASN as UInt32 in ClickHouse
  * [Rework] Url_redirector: Rewrite plugin
  * [Rework] Use a dedicated library for autolearn
  * [Rework] Use libsodium instead of hand crafted crypto implementations
  * [Rework] Use opaque structure to store a table of mime headers
  * [Rules] Add dedicated bitcoin addresses filter rule
  * [Rules] Add more detection to LEAKED_PASSWORD_SCAM
  * [Rules] Catch LTC addresses
  * [Rules] Reduce weight of RSPAMD_EMAILBL
  * [Rules] Rework LEAKED_PASSWORD_SCAM rule one more time
   2019-11-02 17:25:29 by Roland Illig | Files touched by this commit (112)
Log message:
mail: align variable assignments

pkglint -Wall -F --only aligned -r

No manual corrections.
   2019-08-11 15:25:21 by Thomas Klausner | Files touched by this commit (3557) | Package updated
Log message:
Bump PKGREVISIONs for perl 5.30.0
   2019-06-21 17:02:54 by Jonathan Perkin | Files touched by this commit (4) | Package removed
Log message:
rspamd: Commit patch changes missed in previous update.
   2019-06-21 17:00:12 by Jonathan Perkin | Files touched by this commit (4) | Package updated
Log message:
rspamd: Update to 1.9.4.

1.9.4: 23 May 2019
  * [CritFix] Fix case sensitivity when parsing Content-Type
  * [Fix] Arc: Another bunch of fixes for arc signing
  * [Fix] Arc: More arc signing fixes
  * [Fix] Avoid another overflow in fpconv
  * [Fix] Fix ARC signing after fixing another bug in it...
  * [Fix] Fix dkim signing exceptions
  * [Fix] Fix some more corner cases for fpconv
  * [Fix] Further fixes to printing of the FP numbers
  * [Fix] Ratelimit: Fix dynamic score

1.9.3: 13 May 2019
  * [Conf] Add IP_SCORE_FREEMAIL composite rule
  * [Feature] Add cryptobox method to generate dkim keypairs
  * [Feature] Add fast hashes to lua cryptobox hash
  * [Feature] Add least passthrough results
  * [Feature] Allow oversign if exists mode
  * [Feature] Clickhouse: Modernise table initial schema
  * [Feature] Implement IUF interface for specific fast hashes
  * [Feature] Lua_util: Allow to obfuscate different fields
  * [Feature] Tune memory management in Rspamd and Lua
  * [Fix] Avoid buffer overflow when printing long lua strings
  * [Fix] Change the default oversigning headers to a more sane list
  * [Fix] Clickhouse: Do not store digest as it is not needed now
  * [Fix] Clickhouse: Fix lots of storage issues
  * [Fix] Clickhouse: Support custom actions
  * [Fix] Deny URLs where hostname is bogus
  * [Fix] Do not blacklist mail by SPF/DMARC for local/authed users
  * [Fix] Fix DoS caused by bug in glib
  * [Fix] Fix UCL parsing of the multiline strings
  * [Fix] Fix buffer overflow when printing small floats
  * [Fix] Fix init code for servers keypairs cache
  * [Fix] Fix issue with urls with no tld (e.g. IP)
  * [Fix] Fix memory in arc signing logic
  * [Fix] Fix memory leak in language detector during reloads
  * [Fix] Fix mixed case content type processing
  * [Fix] Fix processing of the ip urls in file
  * [Fix] Fix use after free
  * [Fix] HTML: Fix `size` attribute processing
  * [Fix] Hum, it seems that 99ff1c8 was not correct
  * [Fix] Lua_task: Fix task:get_from method
  * [Fix] Preserve fd when mapping file to scan
  * [Fix] Re-use milter_headers settings when doing arc signing
  * [Fix] Set dmarc force action as least action
  * [Fix] Switch to GMT
  * [Fix] allow PKCS7 signatures to be text/plain, too
  * [Project] Add initial version of the vault management tool
  * [Project] Add vault support for DKIM and ARC signing
  * [Project] Implement keys rotation in the vault
  * [Project] Improve dkim keys generation for vault
  * [Project] Improve keys creation in rspamadm vault
  * [Rework] Move lua_worker to a dedicated unit
  * [WebUI] Add URL fragments (#) support
  * [WebUI] Fix AJAX request URL

1.9.2: 16 Apr 2019
  * [Conf] Allow to load users plugins from plugins.d
  * [Conf] oversign openpgp and autocrypt headers
  * [Feature] Add SPF FFI library for Lua
  * [Feature] Add more verbosity for SPF caching
  * [Feature] Antivirus: Handle encrypted files specially
  * [Feature] Clickhouse: Slashing - add new fields to CH
  * [Feature] Dkim_signing: Add OpenDKIM like signing_table and key_table
  * [Feature] Dkim_signing: Allow to use new options as maps
  * [Feature] Import fpconv library
  * [Feature] Lua_maps: Allow static regexp and glob maps
  * [Feature] Parse ical files
  * [Feature] Rspamadm: Add dns_tool utility
  * [Feature] Store SPF records digests
  * [Feature] Use fpconv girsu2 implementation for printing floats
  * [Fix] Clickhouse: Use integer seconds when inserting rows
  * [Fix] Fix floating point printing
  * [Fix] Fix processing of embedded urls
  * [Fix] Lua_clickhouse: Fix CH errors processing
  * [Fix] Make spf digest stable
  * [Fix] Properly detect encrypted files in zip archives
  * [Fix] Slashing: Store times in GMT timezone in ClickHouse
  * [Rules] Add additional conditions to perform BTC checks
  * [Rules] Fix pay-to-hash addresses validation

1.9.1: 5 Apr 2019
  * [Conf] Add vendor groups for symbols
  * [Feature] Add `rspamadm template` command
  * [Feature] Allow to add messages from settings
  * [Feature] Allow unconnected DNS servers operations
  * [Feature] Check limits after being set, migrate to uint64
  * [Feature] Greylist: Allow to disable greylisting depending on symbols
  * [Feature] Improve lua binary strings output
  * [Feature] Mime_types: Implement user configurable extension filters
  * [Feature] Mime_types: When no extension defined, detect it by content
  * [Feature] Preprocess config files using jinja templates
  * [Feature] Replies: Filter replies sender to limit whitelisting to direct messages
  * [Feature] Treat all tags with HREF as a potential hyperlinks
  * [Feature] Validate BTC addresses in LEAKED_PASSWORD_SCAM
  * [Fix] Add crash safety for HTTP async routines
  * [Fix] Another fix for Redis sentinel
  * [Fix] Clickhouse: Fix table schema upload
  * [Fix] Core: Fix squeezed dependencies handling for virtual symbols
  * [Fix] Finally fix default parameters parsing in actions section
  * [Fix] Fix ES sending logic (restore from coroutines mess)
  * [Fix] Fix finishing script for clickhouse collection
  * [Fix] Fix priority for regexp symbols registriation
  * [Fix] Fix various issues found by PVS Studio
  * [Fix] Initialize lua debugging earlier
  * [Fix] Neural: Fix training
  * [Fix] Rework cached Redis logic to avoid sentinels breaking
  * [Fix] SURBL: Fix regression in surbl module
  * [Fix] Fix double signing in the milter
  * [Project] Add support of HTTP proxy in requests
  * [Rework] Change lua global variables registration
  * [Rework] Rework HTML content urls extraction
  * [Rework] Start rework of aliasing in Rspamd
  * [WebUI] Combine Scan and Learning into one tab
  * [WebUI] Fix symbol score input type
  * [WebUI] Show grayed out pie
  * [WebUI] Update Throughput summary values dynamically

1.9.0: 12 Mar 2019
  * [Conf] Add missing includes
  * [Conf] Move to options
  * [Conf] Rbl: DWL is actually special whitelist
  * [Conf] Relax some uribl rules
  * [Conf] Remove abuse.ch
  * [CritFix] Html: Entities are not valid within tag params values
  * [Feature] Add `rspamadm mime sign` tool
  * [Feature] Add configgraph utility
  * [Feature] Add dedicated ZW spaces detection for URLs
  * [Feature] Add flag to url object when visible part is url_like
  * [Feature] Add method task:lookup_words
  * [Feature] Add pyzor support (by crosenberg)
  * [Feature] Allow to add upstream watchers to Lua API
  * [Feature] Allow to set rewrite subject pattern from settings
  * [Feature] Better escaping of unicode
  * [Feature] Clickhouse: Allow to store subject in Clickhouse
  * [Feature] Core: Add QP encoding utility
  * [Feature] Core: Add libmagic detection for all parts
  * [Feature] Core: Add support for gzip archives
  * [Feature] Core: Allow to construct scan tasks from raw data
  * [Feature] Core: Detect charset in archived files
  * [Feature] Core: Ignore and mark invisible spaces
  * [Feature] Core: Normalise zero-width spaces in urls
  * [Feature] Core: Process data urls for images
  * [Feature] Core: Relax quoted-printable encoding
  * [Feature] Core: Support RFC2231 encoding in headers
  * [Feature] Core: Support telephone URLs
  * [Feature] Core: allow to emit soft reject on task timeout
  * [Feature] DCC: Add bulkness and reputation checks to dcc
  * [Feature] Elastic: Modernize plugin
  * [Feature] Export visible part of url to lua
  * [Feature] Fuzzy_storage: add preliminary support of rate limits
  * [Feature] HTML: Specially treat data urls in HTML
  * [Feature] Implement event watchers for upstreams
  * [Feature] Implement includes tracing in Lua
  * [Feature] Improve dkim part in configwizard
  * [Feature] Lua_scanners: Add VadeSecure engine support
  * [Feature] Lua_task: Add flexible method to get specific urls
  * [Feature] Mime_types: Add MIME_BAD_UNICODE rule
  * [Feature] Mime_types: Use detected content type as well
  * [Feature] Plugins: Add preliminary version of the external services plugin
  * [Feature] Query sentinel on master errors
  * [Feature] Regexp: Allow local lua functions in Rspamd regexp module
  * [Feature] Rspamadm: Allow to append footers to plain messages
  * [Feature] Rspamadm: Allow to rewrite headers in messages
  * [Feature] Selectors: Add `ipmask` processor
  * [Feature] Settings: Allow hostname match
  * [Feature] Settings: Allow local when selecting settings
  * [Feature] Settings: Allow multiple selectors
  * [Feature] Settings: Allow to inverse conditions
  * [Feature] Support User-Agent in HTTP requests
  * [Feature] Support ed25519 dkim keys generation
  * [Feature] Try to filter bad unicode types during normalisation
  * [Feature] external_services - oletools (olefy) support
  * [Feature] lua_scanners - icap protocol support
  * [Feature] lua_scanners - spamassassin spam scanner
  * [Fix] Add filter for absurdic URLs
  * [Fix] Add some more cases for Received header
  * [Fix] Allow to disable/enable composite symbols
  * [Fix] Arc: Use a separated list of headers for arc signing
  * [Fix] Archive: Final fixes for 7z archives
  * [Fix] Clickhouse: Fix database usage
  * [Fix] Controller: Make save stats timer persistent
  * [Fix] Core: Detect encrypted rarv5 archives
  * [Fix] Core: Don't detect language twice
  * [Fix] Core: Fix address rotation bug
  * [Fix] Core: Fix content calculations for message parts
  * [Fix] Core: Fix emails comments parsing and other issues
  * [Fix] Core: Fix etags support
  * [Fix] Core: Fix headers folding on the last token
  * [Fix] Core: Fix iso-8859-16 encoding
  * [Fix] Core: Fix log_urls flag (and encrypted logging)
  * [Fix] Core: Fix part length when dealing with boundaries
  * [Fix] Core: Fix parts distance calculations
  * [Fix] Core: Fix processing of NDNs of certain type
  * [Fix] Core: Implement logic to find some bad characters in URLs
  * [Fix] Core: treat nodes with ttl properly in lru cache
  * [Fix] Fix Content-Type parsing
  * [Fix] Fix HTTP headers signing case
  * [Fix] Fix control interface
  * [Fix] Fix deletion of the duplicate headers
  * [Fix] Fix emails filtering in emails module
  * [Fix] Fix greylisting log message and logic
  * [Fix] Fix issues with storing of the accepted addr in rspamd control
  * [Fix] Fix maps object update race condition
  * [Fix] Fix memor leaks and whitespace processing
  * [Fix] Fix processing of null bytes in headers
  * [Fix] Fix rcpt_mime and from_mime in user settings
  * [Fix] Fix rfc2047 decoding for CD headers
  * [Fix] Fix rfc2231 for Content-Disposition header
  * [Fix] Fix setting of the subject pattern in config
  * [Fix] Greylist: fix records checking
  * [Fix] HTML: Another HTML comments exception fix
  * [Fix] HTML: Another entities decoding logic fix
  * [Fix] HTML: Fix HTML comments with many dashes
  * [Fix] HTML: Fix entities in HTML attributes
  * [Fix] HTML: Fix some more SGML tags issues
  * [Fix] Ignore whitespaces at the end of value in DKIM records
  * [Fix] MID module: Fix DKIM domain matching
  * [Fix] Milter_headers: Fix remove_upstream_spam_flag and modernise config
  * [Fix] Mime_parser: Fix issue with parsing of the trailing garbadge
  * [Fix] Mime_parser: Fix parsing of mime parts without closing boundary
  * [Fix] Multimap: Fix operating with userdata
  * [Fix] Process orphaned `symbols` section
  * [Fix] Rdns: Fix multiple replies in fake replies
  * [Fix] Rework groups scores definitions
  * [Fix] Set proper element when reading data from Sentinel
  * [Fix] Set rspamd user to initialise supplementary groups on reload
  * [Fix] Settings: Fix selectors usage
  * [Fix] Sort data received from Sentinel to avoid constant replacing
  * [Fix] groups.conf - filename typo
  * [Fix] lua_scanner - oletools typos, logging
  * [Fix] lua_scanners - actions and symbol_fail
  * [Fix] lua_scanners - fix luacheck
  * [Fix] lua_scanners - kaspersky - response with fname
  * [Fix] lua_scanners - savapi redis prefix
  * [Fix] tests - antivirus - fprot symbols
  * [Project] Add concept of flexible actions
  * [Project] Add heuristical from parser to received parser
  * [Project] Add new flags to clickhouse, redis and elastic exporters
  * [Project] Attach new received parser
  * [Project] Fallback to callbacks from coroutines
  * [Project] Implement keep-alive support in lua_http
  * [Project] Lua_udp: Implement fully functional client
  * [Project] Plug keepalive knobs into http connection handling
  * [Project] Rspamadm: Add `modify` tool
  * [Rework] Convert rspamd-server to a shared library
  * [Rework] Dcc: Rework DCC plugin
  * [Rework] Enable explicit coroutines symbols
  * [Rework] Rework telephone urls parsing logic
  * [Rework] Rewrite RBL module
  * [Rework] Settings: Rework settings check
  * [Rework] Slashing: Distinguish lualibdir, pluginsdir and sharedir
  * [Rework] Unify task_timeout
  * [Rework] Use VEX instructions in assembly, relocate
  * [WebUI] Notify user if uploaded data was not learned
  * [WebUI] Remove redundant condition

1.8.3: 03 Dec 2018
  * [CritFix] Make flags mutually exclusive for mime parts
  * [CritFix] Strictly deny unencoded bad utf8 sequences in headers
  * [Feature] Add Kaspersky antivirus support
  * [Feature] Add method to get dkim results
  * [Feature] Add more words regexp classes
  * [Feature] Allow to choose words format in `rspamadm mime`
  * [Feature] Allow to get all types of words from Lua
  * [Feature] Allow to get task flags in C expressions
  * [Feature] Allow to require encryption when accepting connections
  * [Feature] Ignore bogus whitespaces in the words
  * [Feature] Implement more strict configuration tests
  * [Feature] Improve SPF results in Authentication-Results
  * [Feature] Support ClickHouse database
  * [Fix] Add failsafety for utf8 regexps
  * [Fix] Do not trigger BROKEN_CONTENT_TYPE on innocent text parts
  * [Fix] Emit error if connection has been terminated with no stop pattern
  * [Fix] Fix boundaries checks in embedded messages
  * [Fix] Fix double free
  * [Fix] Perform policy downgrade on sample out, add tests
  * [Fix] Properly escape utf8 regexps in hyperscan mode
  * [Fix] Selectors - attachments args condition
  * [Fix] Some fixes for raw parts
  * [Fix] Treat learning errors as non-fatal
  * [Fix] Use tld when looking for DKIM domains
  * [Project] Words unicode structure rework
  * [Project] Add preliminary Redis Sentinel support
  * [Project] Improve Authentication-Results header
  * [Project] Rework DKIM checks results
  * [Project] Use more generalised API to produce meta words

1.8.2: 19 Nov 2018
  * [Conf] Add DWL support in the default configuration
  * [Conf] Disable rspamd_update by default (again)
  * [Conf] Fix configuration sample for ratelimit
  * [CritFix] Disable broken url tags by default
  * [CritFix] Fix \0 processing when doing RSA sign
  * [CritFix] Fix adding symbols to their primary groups
  * [Feature] Add `rspamadm cookie` utility
  * [Feature] Add specialised functions for generating encrypted cookies
  * [Feature] Add support of cookies in replies module
  * [Feature] Add support of words regexps
  * [Feature] Allow to add 3rd party clang plugins
  * [Feature] Allow to create lua regexps from glob or plain patterns
  * [Feature] Allow to set custom limits for upstream lists
  * [Feature] Detect orphaned parts and attach them to message
  * [Feature] Filter tokens in bayes
  * [Feature] Fold b= value when doing arc sealing
  * [Feature] Ignore cookies in the future and too old in the past
  * [Feature] Skip stop words in statistics
  * [Feature] Store stop words and allow to query them
  * [Feature] Support query arguments in controller's custom commands
  * [Feature] Tune upstream limits in Rspamd proxy
  * [Feature] Use different callback symbols for different uribls
  * [Feature] Write DKIM selector in dkim allow/reject symbols
  * [Fix] Add obs_fws state support to eoh state machine
  * [Fix] Add sanity check when applying mime boundaries heuristic
  * [Fix] Antivirus - virus names with 0 were recognized as tables
  * [Fix] Disable headernames in bayes temporarily
  * [Fix] Do not allow syntax errors in include files...
  * [Fix] Do not allow to merge an object with an array (or vice versa)
  * [Fix] Don't perform forged recipients check for missing recipients
  * [Fix] Fix DKIM based RBLs
  * [Fix] Fix actrie implementation (sync from upstream), fixed OOB read
  * [Fix] Fix explicit methods call in selectors
  * [Fix] Fix extraction of additional parts
  * [Fix] Fix finalization for internal plugins
  * [Fix] Fix override_defaults function
  * [Fix] Fix squeezed symbols when using settings
  * [Fix] Fix urls insertion in Clickhouse module
  * [Fix] Furhter fixes to ratelimits logic
  * [Fix] Ignore signatures when looking for boundaries
  * [Fix] Properly set learned count
  * [Fix] Really fix ratelimits configuration and work
  * [Fix] Remove ambigious format flag from printf
  * [Fix] Restore URLs exporting in ClickHouse plugin
  * [Fix] Rework bayes calculations...
  * [Fix] Switch from chi-square to naive for large Fisher value
  * [Fix] Treat normal password as enable password if there is no enable password
  * [Fix] Use proper syntax for making DNS requests
  * [Fix] Various fixes in embedded plugins
  * [Project] Change fuzzy check selection logic to lua_fuzzy library
  * [Project] Rework async events and symbols
  * [Project] Move all metatokens in Bayes to lua_stat from C
  * [WebUI] Add history rows per page control

1.8.1: 16 Oct 2018
  * [CritFix] Fix options insertion
  * [CritFix] Fix words decay one more time (affects long messages)
  * [CritFix] Increase default words_decay
  * [CritFix] Plug memory leak in redis pool
  * [Feature] Add `check_violation` feature to DKIM/ARC signing
  * [Feature] Add only unique elements to Clickhouse url arrays
  * [Feature] Allow `g+:` and `g-:` composite atoms
  * [Feature] Allow dkim domains check in surbl
  * [Feature] Allow maps with HTTP auth
  * [Feature] Allow to disable actions by users settings
  * [Feature] Extend whitelisting options
  * [Feature] Store url object in images
  * [Feature] Use verdict instead of the plain action in plugins
  * [Fix] Allow to call fstring append with NULL string
  * [Fix] DCC - luacheck
  * [Fix] Do not load torch on each rspamadm invocation
  * [Fix] Fix boundaries detection and rework stop words algorithm
  * [Fix] Fix dependencies for DNS_SIGNED symbol
  * [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit
  * [Fix] Fix groups mess
  * [Fix] Fix groups mess
  * [Fix] Fix parsing address with comments
  * [Fix] Fix resolving in DMARC reports
  * [Fix] Fix various issues with parsing of the received headers
  * [Fix] Fix watchers issue in lua_tcp when doing no resolving
  * [Fix] Plug memory leak in language detector (affects reloads)
  * [Fix] Remove one letter stop words
  * [Fix] Slashing: backport chunk logic from libucl
  * [Fix] Stop libevent from using cached time in rspamadm
  * [Fix] Try to fix watchers chaining
  * [Fix] Various fixes in redis sync interface
  * [Fix] ip_score - respect check_authed and check_local settings from config
  * [Project] Rework passthrough actions
  * [Project] Clustering module
  * [Rework] Always create result for a task
  * [Rework] Completely rewrite DMARC checks logic
  * [Rework] Rework and fix whitelist plugin
  * [WebUI] Add symbols sorting buttons
  * [WebUI] Change symbols order without updating history
  * [WebUI] Colorize symbols
  * [WebUI] Do not display password form when secure_ip is set
  * [WebUI] Fix symbol description tooltips display
  * [WebUI] History: add sorting by symbol score value
   2019-04-03 02:33:20 by Ryo ONODERA | Files touched by this commit (748)
Log message:
Recursive revbump from textproc/icu