./mail/spamdyke, Filters incoming SMTP connections to qmail

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]

Branch: CURRENT, Version: 5.0.1nb2, Package name: spamdyke-5.0.1nb2, Maintainer: schmonz

spamdyke monitors incoming traffic, acting as a middleman between
qmail and the remote server. It catches the sender and recipient
addresses as they go by and logs them to syslog. If it sees something
it doesn't like (e.g. a blacklisted sender), it cuts the connection,
closes qmail and fakes the rest of the SMTP transaction with the
remote server. qmail thinks the remote server disconnected normally.
The remote server thinks qmail is rejecting the message. It's the
best of both worlds.

spamdyke can optionally reject the connection if the remote server's
reverse DNS entry does not exist, does not resolve, contains its
IP address and either contains a prohibited keyword (like "dynamic")
or ends in a country code; if the IP address, reverse DNS entry,
or envelope sender is listed in a blacklist; or if data is sent
before the SMTP greeting banner is displayed. spamdyke can also
limit recipients per connection, greylist for some or all domains,
and close connections that go idle or take too long.

Required to build:

Master sites:

SHA1: 4de76158a98b3ee33e7002160446ae36de7c96d2
RMD160: 7237108ebf0ff2a5c02b7e305985a11e93720f8d
Filesize: 374.727 KB

Version history: (Expand)

CVS history: (Expand)

   2018-12-25 17:24:26 by Amitai Schleier | Files touched by this commit (2)
Log message:
Sprinkle static on an inline. Fixes SmartOS and Ubuntu bulk builds (that
somehow don't have any -On in CFLAGS).
   2018-07-29 10:41:18 by Amitai Schleier | Files touched by this commit (6) | Package updated
Log message:
Apply patch from joerg@ to build with clang (by removing anonymous inner
functions). Bump PKGREVISION.
   2017-07-09 18:16:32 by Amitai Schleier | Files touched by this commit (5)
Log message:
Fix build with PKGSRC_USE_SSP, and support PKGSRC_USE_FORTIFY.
While here, remove reference to LOCALBASE.
   2016-03-05 12:29:49 by Jonathan Perkin | Files touched by this commit (1813) | Package updated
Log message:
Bump PKGREVISION for security/openssl ABI bump.
   2015-11-04 00:27:24 by Alistair G. Crooks | Files touched by this commit (312)
Log message:
Add SHA512 digests for distfiles for mail category

Problems found locating distfiles:
	Package mutt: missing distfile patch-1.5.24.rr.compressed.gz
	Package p5-Email-Valid: missing distfile Email-Valid-1.198.tar.gz
	Package pine: missing distfile fancy.patch.gz
	Package postgrey: missing distfile targrey-0.31-postgrey-1.34.patch
	Package qmail: missing distfile badrcptto.patch
	Package qmail: missing distfile outgoingip.patch
	Package qmail: missing distfile qmail-1.03-realrcptto-2006.12.10.patch
	Package qmail: missing distfile qmail-smtpd-viruscan-1.3.patch
	Package thunderbird24: missing distfile enigmail-1.7.2.tar.gz
	Package thunderbird31: missing distfile enigmail-1.7.2.tar.gz

Otherwise, existing SHA1 digests verified and found to be the same on
the machine holding the existing distfiles (morden).  All existing
SHA1 digests retained for now as an audit trail.
   2015-05-03 16:22:08 by Amitai Schlair | Files touched by this commit (3) | Package updated
Log message:
Update to 5.0.1. From the changelog:

Fixed a typo in the README file for the OpenSSL "dhparam" command.  \ 
Thanks to
  Eric Shubert for reporting this one.
Removed unused variables from the dns* commands in the utils folder to fix
  compiler warnings.
Fixed a bug in read_file() that returned uninitialized pointers if a file
  contained blank lines or comments at the top, causing segfaults when they
  were free()d.  Thanks to Jeffrey Gordon and Quinn Comendant for reporting
  this one.
Changed the directory naming scheme in the "generator" program to \ 
include the
  flowchart step numbers in the name.  The old pattern was just too hard to
  follow visually and far too difficult to search for a specific test.
Added more steps to the recipient validation flowchart and spamdyke-qrv's
  recipient validation filter to correctly handle addresses that are forwarded
  to an external address.  Thanks to Stephen Marley for reporting this one.
Changed search_file() in spamdyke-qrv to return a "not found" result \ 
when the
  file does not exist, instead of an error.
Added a delay loop to exec_command_argv() in spamdyke and spamdyke-qrv to work
  around a race condition -- sometimes the child process will close its pipes
  in preparation for exiting and the parent's waitpid() will fire before the
  child has fully exited. This leads to erroneous returns showing the child
  has not exited when it really only needed another timeslice or two.  This is
  fixed by looping with nanosleep() to wait a few tenths of a second after
  seeing this return code.
Added a way to stop a test script run by creating a file named "stop". \ 
  allows it to be stopped without killing the process and potentially leaving
  the test platform in a partially (mis)configured state.
Fixed the accessor function for the header-blacklist-entry and
  header-blacklist-file options to find their data in the filter_settings
  object instead of the option_set object.  This is because the data is moved
  from the option_set immediately after it is set so the blacklist effect is
  cumulative when set from configuration directories.  Reading from the wrong
  location meant the config-test feature was never testing those options at
  all.  Thanks to Stefan for reporting this one.
Fixed a pair of bugs in process_config_file(): one that would add empty values
  to the end of a list of blacklist/whitelist files if a directive was
  followed by a blank line and a commented-out directive (causing errors when
  the values are used), the other that would throw errors if a line in a
  configuration file contained only one space.  Thanks to Les Fenison for
  reporting these.
Fixed a bug in middleman() that would return an improper greeting when
  injecting both AUTH and STARTTLS banners into the EHLO response.  Clients
  seeing this improper greeting would hang forever and eventually timeout.
  Thanks to Elliot Denk for reporting this one and sending a patch!
Fixed a major thinko in smtp_filter that was carrying over the rejection data
  between recipients, even if a recipient had a configuration directory file
  that altered the overall configuration.  This was leading to some
  recipients being incorrectly rejected under very specific (and likely very
  rare) conditions, which just happened to be met on my own server.
Fixed a bug in copy_base_options that was not copying the "reason" \ 
data from
  the last rejection.
Fixed an infinite loop in dnsdummy when priorities over 0 are used.
Fixed a typo in dnsdummy that was truncating data when the verbose flags were
  used (weird, yes).
Changed dnsdummy to fork a child process to return each query.  This was the
  easiest solution to implement to allow new queries to be processed while
  waiting n seconds to send answers to previous queries.  This is a fragile
  and wasteful solution -- if dnsdummy were intended for production use, a
  queue would be a much better solution.
Changed all of the "verbose"-level error messages to include the name \ 
of the
  function, file and line that generated it.  Every other message prefixed
  with "ERROR" already did this, so this makes things more consistent.
Renamed all of the "FILTER" messages and added a new logging macro to print
  them named SPAMDYKE_LOG_FILTER().  This way they can continue to be output
  without function, file and line information.
Renamed the SPAMDYKE_LOG_CONFIG_TEST() macro to
  instead of having a special LOG_LEVEL_CONFIG_TEST setting.  This way the
  config-test messages can be changed to emit file, function and lines if
  needed (or not).
Changed dnsdummy to encode multiple answers in the same response, if its
  config file contains multiple matches for the same query.
Fixed a bug in dnsdummy that was adding extra bytes to the end of each
  answer.  This turned out to be covering a matching (compensating) bug in
  spamdyke's DNS parsing code.  I really hate it when that happens!
Fixed a serious bug in nihdns_expand() that was causing spamdyke to
  incorrectly parse DNS responses with multiple answers; it would use the
  first answer, then skip the wrong number of bytes, causing it to conclude
  any subsequent answers were corrupted.
Changed nihdns_expand() to return separate values for the number of bytes in
  the decoded string and the number of bytes the string occupies in the DNS
  packet.  Due to packet compression, the numbers can be very different.
Changed generator to add records to the named configuration so domains will
  resolve correctly during testing.  Since using port numbers in resolv.conf
  is not allowed, there is no easy way to use dnsdummy for these tests.
Discovered qmail-send does not check the percenthack or virtualdomains files
  when resolving forward addresses, only locals and assign.  Updated
  spamdyke-qrv to behave the same way.
Refined the success/failure detection in generator after learning more about
  qmail's behavior.  If only it had some kind of accurate documentation...
Extended the tests created by generator to also test conditions where
  spamdyke-qrv calls vpopmail to look up addresses.  This increased the
  number of spamdyke-qrv tests more than tenfold!
Added a "diagnostic output" flag to spamdyke-qrv to print the decision path
  it used to evaluate the address.  Also added a test to the test generator
  to compare the diagnostic output with the expected decision path, to
  catch tests that may be producing the desired effect for the wrong reason.
Fixed a bug in set_config_value() to make it possible to set
  CONFIG_TYPE_NAME_MULTIPLE options to "none" or unset specific values.
  Thanks to Konstantin for reporting this one.
Added flags to smtpdummy to advertise STARTTLS support in response to EHLO.
  It doesn't actually do TLS, it just advertises it.
Fixed smtp_filter() to block a client's STARTTLS command if tls-level is
  "none".  Thanks to Les Fenison for reporting this one.
Added a flag to the configure script for both spamdyke and spamdyke-qrv to
  compile with the address sanitizer library to catch memory access errors.
  Adjusted the version string to show when the sanitizer is in use.
Changed the test scripts to always compile spamdyke with the address
  sanitizer (if available) when testing.  The tests run a lot slower, but
  the sanitizer is too awesome to not use.
Fixed a buffer underrun in examine_entry that was causing segfaults when
  searching files where wildcards are allowed at the beginning of the lines.
  Thanks to Dirk Kannapinn for reporting this one.
Discovered a horrible problem with snprintf()'s %n format -- it returns the
  number of bytes it _would_have_ written *if* there were infinite space, not
  the number of bytes *actually* written as the man page states.  So using %n
  at the end of the format string as a substitute for immediately calling
  strlen() is not safe.  Good thing I don't ever do that, right? ...wait,
  I use that feature EVERYWHERE! (grrrrr)  Thanks to the Google Address
  Sanitizer team for finding this one.  Whoever implemented the %n feature
  in glibc can report to me any time for a free punch in the throat.
  I want my weekend back.
Reverted the (apparently) useless change from 4.3.0 to use %n in snprintf()
  instead of the return value and replaced snprintf() with a macro named
  SNPRINTF() that explicitly compares the return value with the size of the
  buffer and returns the number of bytes ACTUALLY written.
Fixed a harmless buffer overrun in sub_examine_tcprules_entry() that could
  have overwritten one byte of another variable on the stack with a null
  byte.  Since the address was valid and that other variable is set just
  after the overwrite anyway, it wasn't actually a problem.  But fixing it
  makes the address sanitizer happy, so it's fixed.
Fixed a pair of huge buffer overruns in config_test_file_read() and
  config_test_file_read_write() that could load 63K of file contents past
  the end of the buffer (on the stack).  Fortunately, these functions are
  only used by the config-test feature, never during normal operation.
Fixed a buffer overflow in find_address() that would overwrite a single byte
  in the caller's stack with a null byte when parsing BATV addresses.  From
  what I can tell, the effect of this bug would be to either truncate the
  parsed address or cause a segfault.
Added undo_softlimit() to try to increase the "soft" limits on address \ 
  stack size and memory size to maximum if they are less than infinite (and
  squawk if they cannot be reset to maximum).  This will (hopefully) prevent
  problems caused by DJB's "softlimit" program, which is a useless piece of
  trash many qmail install guides *still* recommend using.
Fixed a bug in the logging code of tls_read() that was using an "error"
  message to log at "verbose" level.  The error message had more printf()
  format specifiers than the verbose logger was providing, which was leading
  to segfaults when the message was printed.  Many thanks to Konstanin for
  a lot of help tracking this one down.
   2014-02-13 00:18:57 by Matthias Scheler | Files touched by this commit (1568)
Log message:
Recursive PKGREVISION bump for OpenSSL API version bump.
   2014-01-29 21:22:20 by Amitai Schlair | Files touched by this commit (3) | Package updated
Log message:
Update to 5.0.0. From the changelog:

Rearranged the test scripts to put them in folders by category.  This just
  makes the directory listing a little more manageable.
Corrected some typos in the README file.  Thanks to John Mendoza for reporting
Fixed a very obscure bug in spamdyke_log(): on Linux systems (possibly only
  64-bit systems), vsyslog() occasionally will not print all the variable
  arguments.  One way was found to trigger this behavior -- when the
  rdns-blacklist-dir filter is activated from a configuration directory.
Fixed a bug in find_domain() that could cause segfaults when parsing certain
  invalid formats.  Thanks to Gary Gendel for reporting this one.
Added a backup/restore feature to the "run" script in the \ 
"tests" folder to
  save a copy of the most critical system and qmail files before running any
  scripts.  This is needed because some of the scripts alter those files and,
  if they don't run correctly or are cancelled, the originals are lost.
Added a "-skipcompile" flag to the "run" script in the \ 
"tests" folder to skip
  reconfiguring and recompiling all of the binaries when the script is run.
Changed the "run" script in the "tests" folder to empty \ 
qmail's queue before
  and after the tests are run.
Changed the "run" script in the "tests" folder to compare \ 
the current system
  and qmail configuration files to the latest backup after every script
  finishes.  If they don't match, the latest backup is restored.  If they
  still don't match, the script stops with an error.
Changed nihdns_query() to accept an optional "preferred" type of \ 
response.  If
  multiple types are queried, it will wait for at least one timeout period for
  an answer of that type to arrive instead of always accepting the first
  answer to arrive.  It will accept a saved answer before resending the
  queries, however.
NOT BACKWARDS COMPATIBLE: Changed nihdns_mx() to prefer an MX record over an A
  record, if both exist.  Given the choice, the MX record will be checked for
  validity and the A record will be ignored.  Thanks to Bruce Schreiber for
  suggesting this one.
Fixed filter_level() and smtp_filter() to disregard whitelisting and require
  authentication if the "filter-level" option is set to \ 
"require-auth", as the
  documentation says it should.  Thanks to Arne for reporting this one.
Changed nihdns_create_packet() to strip trailing dots from names before using
  them in DNS queries.  A trailing dot is the traditional way to tell libc's
  resolver not to append the local domain name and many sysadmins expect to
  have to use it.  Since spamdyke never appends the local domain and doesn't
  use libc's resolver, it isn't necessary and causes lookups to fail.  Thanks
  to Dossy Shiobara for reporting this one.
Changed middleman() to always send a "STARTTLS" response to \ 
"EHLO" as a
  continuation, never as the last line (only when spamdyke is inserting
  "STARTTLS").  This works around a bug in the Android mail client, \ 
which only
  looks for "STARTTLS" as a continuation.  Thanks to Jonas Pasche for \ 
  about how to work around this bug on his blog.
NOT BACKWARDS COMPATIBLE: Changed the meaning of "whitelisted" to only \ 
  the connection from spamdyke's spam filters; whitelisting no longer allows
  the connection to relay mail.  This means spamdyke will now only set the
  RELAYCLIENT environment variable if the "relay-level" option is set to
  "allow-all".  Relaying must now be controlled through tcpserver or \ 
  Many thanks to Eric Shubert for suggesting and debating this with me.
NOT BACKWARDS COMPATIBLE: Removed the "access-file" and
  "rejection-text-access-denied" options because they were only needed for
  controlling relaying.  Also removed the test scripts that exercised them and
  modified many other test scripts that used them.
NOT BACKWARDS COMPATIBLE: Removed the "no-check" value from the \ 
  option and changed the meaning of the "normal" value to use the logic
  previously assigned to "no-check".
Added the option "reject-sender" to take multiple values.  If the value
  "not-local" is given, the sender will be rejected if the domain name \ 
is not
  hosted locally.  If the value "authentication-mismatch" is given, \ 
the sender
  will be rejected if the sender address does not exactly match the username
  given during authentication (or if the authentication username is not an
  email address, the sender username must match the authentication username).
  If the value "authentication-domain-mismatch" is given, the sender \ 
will be
  rejected if the domain name is not part of the username given during
  authentication.  Thanks to Mark Frater for suggesting this one.
Added the options "rejection-text-sender-not-local" and
  "rejection-text-sender-authentication-mismatch" to set the rejection text
  given when the "reject-sender" option's filters are triggered.
NOT BACKWARDS COMPATIBLE: Removed the option "reject-missing-sender-mx" and
  folded its filter into the "reject-sender" filter's \ 
"no-mx" option.
  "rejection-text-missing-sender-mx" to \ 
  "reject-identical-sender-recipient" to "reject-recipient" \ 
with the value
  "same-as-sender".  The functionality remains the same.
  "rejection-text-identical-sender-recipient" to
NOT BACKWARDS COMPATIBLE: Renamed the option "local-domains-file" to
  "qmail-rcpthosts-file".  The naming has always been confusing, since \ 
  distinguishes between domains that should be accepted by qmail-smtpd during
  SMTP (rcpthosts) and domains that are actually hosted locally with mailboxes
  on the local filesystem (locals).  These options have always meant the
  former, but now that spamdyke needs to know both lists of domains, it's time
  to rename them.  This option is also now allowed in configuration
NOT BACKWARDS COMPATIBLE: Removed the option "local-domains-entry" because
  supplying domains that can be accepted during SMTP to spamdyke only (but
  not qmail) will cause inconsistent results during recipient validation.
  If a domain is to be accepted during SMTP, it should be added to the control
  files used by both spamdyke and qmail.
Added CDB searching code in cdb.[ch] to read DJB's "constant database" \ 
  during recipient validation.  The format of these files is claimed (by DJB)
  to be fast and efficient.  Don't believe the hype...
Added the option "qmail-morercpthosts-cdb" to allow CDB files to be \ 
  that contain lists of domains for which mail should be accepted during SMTP.
  Does anyone actually use this qmail "feature"?
Poured over qmail's documentation and source code to figure out exactly how
  it determines where to deliver a message.  The documentation is frequently
  in error and extensive testing was required to discover the truth.  The
  resulting procedure is encapsulated in a flowchart in the documentation
Added the "generator" program to create test scripts to check every \ 
  path through the recipient validation flowchart, both with spamdyke in place
  and without (to check the flowchart is correct).  A program to generate the
  scripts was required, since there are nearly 250K possible paths to test.
Added the value "invalid" to the option "reject-recipient" \ 
to check if a local
  recipient address exists before accepting a message.  This validation
  process uses the same logic as qmail when deciding whether/where to deliver
  a message, so no extra steps are needed to make this work (e.g. maintaining
  a list of valid addresses in a separate file).  If this process determines
  a local address is valid, delivery is guaranteed.  This option should
  eliminate qmail's habit of sending backscatter spam.
Added the value "unavailable" to the option \ 
"reject-recipient" to check if a
  local recipient is accepting mail at the moment.  Probably as a holdover
  from the elder days when people actually edited .qmail files by hand, qmail
  checks file permissions on files and folders before delivering a message.
  If they are set to certain values, qmail will queue the message until the
  permissions are fixed or bounce the message if is queued too long.  In these
  enlightened times, such permissions are more likely to be due to an error or
  oversight than deliberate intent.
Added the options "qmail-assign-cdb", \ 
  "qmail-envnoathost-file", "qmail-locals-file", \ 
  "qmail-percenthack-file" and "qmail-virtualdomains-file" \ 
to allow spamdyke
  to use different control files than qmail.  It's very unlikely anyone will
  ever need these options (and it would be unwise to use them), but they're
  available just in case.
Added the option "rejection-text-recipient-invalid" to set the \ 
rejection text
  when the "invalid" filter on "reject-recipient" is triggered.
Added the option "rejection-text-recipient-unavailable" to set the \ 
  text when the "unavailable" filter on "reject-recipient" \ 
is triggered.
Removed the function filter_recipient_local() and moved its logic into
Removed the function filter_recipient_relay() and moved its logic into
Changed the "help" option to just show a listing of available options \ 
  help text.
Added the "more-help" option to show the full listing of options with \ 
all help
Added the options "ip-relay-entry", "ip-relay-file", \ 
"rdns-relay-entry" and
  "rdns-relay-file" to allow relaying from specific IPs and/or rDNS names,
  since whitelisting no longer implies the ability to relay.  If any of these
  options are matched, the RELAYCLIENT variable will be set before qmail is
Created the "create_cdb" program to generate CDB files of arbitrary size,
  filled with random data, for testing spamdyke's CDB validation routines.
  create_cdb also has the ability to corrupt the generated CDB in seven ways;
  this makes for more specific testing than simply using a file of random
Removed all uses of the TESTSD_* environment variables from the test scripts
  and replaced them with appropriate invocations of dnsdummy.  This allows the
  test scripts to run without potential interference from external DNS
  changes and without needing a running spamdyke server to find example
Fixed smtp_filter() and middleman() to clear the list of saved recipient
  addresses after printing the log messages.  This prevents duplicate log
  messages when multiple email messages are delivered in the same connection.
  Thanks to Teodor Milkov and David Davidov for reporting this one.
Added the "-skippatched" and "-skipunpatched" flags to the \ 
"run" scripts to
  skip any tests that require a patched or unpatched version of qmail,
Fixed a minor bug in find_username() that would truncate the last character
  of the username when no domain is given.  This hasn't been a problem since
  spamdyke rejects recipient addresses without domain names anyway, but one
  of the recipient validation test scripts found it.
Added the option "tls-dhparams-file" option to read DH params from a file
  for creating ephemeral keys during SSL/TLS key negotiation.  Thanks to
  Marc Gregel for suggesting this one.
Changed all error messages to output the filename, function name and line
  number that generated them, just like the debug and excessive messages.
Added a new log level, LOG_LEVEL_CONFIG_TEST, for config-test error messages.
  The level is treated much the same as LOG_LEVEL_ERROR except the filename,
  function name and line numbers are not printed.
Added a new decision level, FILTER_DECISION_AUTHENTICATED for authenticated
  connections.  The filter routines use this level to distinguish between
  connections that should be unfiltered due to authentication versus
Added a new config option type: CONFIG_TYPE_ALIAS.  Options of this type are
  aliases for other options.  This eliminates the duplication of values and
  potential for oversights in the graylist/greylist options.
Added some code to the "run" script in the "tests" directory \ 
to try to detect
  core dumps.  Some of the tests will declare success even if spamdyke
  segfaults and cuts off the output prematurely.
Removed the unused functions reset_rejection() and skip_cfws().
Discovered spamdyke cannot read all the files it needs for recipient
  validation during normal operation because they are owned by different users
  with restrictive permissions and spamdyke does not run as root.  I'm not
  sure how I missed that, but it completely moots more than a year of work.
Moved all the recipient valiation code into an external program named
  "spamdyke-qrv".  This program is meant to only perform recipient \ 
  and nothing else, so it should be safe to run as root (at least safer than
  running spamdyke as root).
Removed the options "qmail-assign-cdb", \ 
  "qmail-envnoathost-file", "qmail-locals-file", \ 
"qmail-me-file" and
  "qmail-percenthack-file" from spamdyke, since the recipient \ 
validation code
  is gone.
Added the option "recipient-validation-command" for passing the path to
  spamdyke-qrv, which will be called when recipient validation is needed.