Path to this page:
Subject: CVS commit: pkgsrc/mail/spamdyke
From: Amitai Schlair
Date: 2014-01-29 21:22:20
Message id: 20140129202220.150A796@cvs.netbsd.org
Log Message:
Update to 5.0.0. From the changelog:
Rearranged the test scripts to put them in folders by category. This just
makes the directory listing a little more manageable.
Corrected some typos in the README file. Thanks to John Mendoza for reporting
those.
Fixed a very obscure bug in spamdyke_log(): on Linux systems (possibly only
64-bit systems), vsyslog() occasionally will not print all the variable
arguments. One way was found to trigger this behavior -- when the
rdns-blacklist-dir filter is activated from a configuration directory.
Fixed a bug in find_domain() that could cause segfaults when parsing certain
invalid formats. Thanks to Gary Gendel for reporting this one.
Added a backup/restore feature to the "run" script in the \
"tests" folder to
save a copy of the most critical system and qmail files before running any
scripts. This is needed because some of the scripts alter those files and,
if they don't run correctly or are cancelled, the originals are lost.
Added a "-skipcompile" flag to the "run" script in the \
"tests" folder to skip
reconfiguring and recompiling all of the binaries when the script is run.
Changed the "run" script in the "tests" folder to empty \
qmail's queue before
and after the tests are run.
Changed the "run" script in the "tests" folder to compare \
the current system
and qmail configuration files to the latest backup after every script
finishes. If they don't match, the latest backup is restored. If they
still don't match, the script stops with an error.
Changed nihdns_query() to accept an optional "preferred" type of \
response. If
multiple types are queried, it will wait for at least one timeout period for
an answer of that type to arrive instead of always accepting the first
answer to arrive. It will accept a saved answer before resending the
queries, however.
NOT BACKWARDS COMPATIBLE: Changed nihdns_mx() to prefer an MX record over an A
record, if both exist. Given the choice, the MX record will be checked for
validity and the A record will be ignored. Thanks to Bruce Schreiber for
suggesting this one.
Fixed filter_level() and smtp_filter() to disregard whitelisting and require
authentication if the "filter-level" option is set to \
"require-auth", as the
documentation says it should. Thanks to Arne for reporting this one.
Changed nihdns_create_packet() to strip trailing dots from names before using
them in DNS queries. A trailing dot is the traditional way to tell libc's
resolver not to append the local domain name and many sysadmins expect to
have to use it. Since spamdyke never appends the local domain and doesn't
use libc's resolver, it isn't necessary and causes lookups to fail. Thanks
to Dossy Shiobara for reporting this one.
Changed middleman() to always send a "STARTTLS" response to \
"EHLO" as a
continuation, never as the last line (only when spamdyke is inserting
"STARTTLS"). This works around a bug in the Android mail client, \
which only
looks for "STARTTLS" as a continuation. Thanks to Jonas Pasche for \
writing
about how to work around this bug on his blog.
NOT BACKWARDS COMPATIBLE: Changed the meaning of "whitelisted" to only \
exempt
the connection from spamdyke's spam filters; whitelisting no longer allows
the connection to relay mail. This means spamdyke will now only set the
RELAYCLIENT environment variable if the "relay-level" option is set to
"allow-all". Relaying must now be controlled through tcpserver or \
xinetd.
Many thanks to Eric Shubert for suggesting and debating this with me.
NOT BACKWARDS COMPATIBLE: Removed the "access-file" and
"rejection-text-access-denied" options because they were only needed for
controlling relaying. Also removed the test scripts that exercised them and
modified many other test scripts that used them.
NOT BACKWARDS COMPATIBLE: Removed the "no-check" value from the \
"relay-level"
option and changed the meaning of the "normal" value to use the logic
previously assigned to "no-check".
Added the option "reject-sender" to take multiple values. If the value
"not-local" is given, the sender will be rejected if the domain name \
is not
hosted locally. If the value "authentication-mismatch" is given, \
the sender
will be rejected if the sender address does not exactly match the username
given during authentication (or if the authentication username is not an
email address, the sender username must match the authentication username).
If the value "authentication-domain-mismatch" is given, the sender \
will be
rejected if the domain name is not part of the username given during
authentication. Thanks to Mark Frater for suggesting this one.
Added the options "rejection-text-sender-not-local" and
"rejection-text-sender-authentication-mismatch" to set the rejection text
given when the "reject-sender" option's filters are triggered.
NOT BACKWARDS COMPATIBLE: Removed the option "reject-missing-sender-mx" and
folded its filter into the "reject-sender" filter's \
"no-mx" option.
NOT BACKWARDS COMPATIBLE: Renamed the option
"rejection-text-missing-sender-mx" to \
"rejection-text-sender-no-mx".
NOT BACKWARDS COMPATIBLE: Renamed the option
"reject-identical-sender-recipient" to "reject-recipient" \
with the value
"same-as-sender". The functionality remains the same.
NOT BACKWARDS COMPATIBLE: Renamed the option
"rejection-text-identical-sender-recipient" to
"rejection-text-recipient-same-as-sender".
NOT BACKWARDS COMPATIBLE: Renamed the option "local-domains-file" to
"qmail-rcpthosts-file". The naming has always been confusing, since \
qmail
distinguishes between domains that should be accepted by qmail-smtpd during
SMTP (rcpthosts) and domains that are actually hosted locally with mailboxes
on the local filesystem (locals). These options have always meant the
former, but now that spamdyke needs to know both lists of domains, it's time
to rename them. This option is also now allowed in configuration
directories.
NOT BACKWARDS COMPATIBLE: Removed the option "local-domains-entry" because
supplying domains that can be accepted during SMTP to spamdyke only (but
not qmail) will cause inconsistent results during recipient validation.
If a domain is to be accepted during SMTP, it should be added to the control
files used by both spamdyke and qmail.
Added CDB searching code in cdb.[ch] to read DJB's "constant database" \
files
during recipient validation. The format of these files is claimed (by DJB)
to be fast and efficient. Don't believe the hype...
Added the option "qmail-morercpthosts-cdb" to allow CDB files to be \
provided
that contain lists of domains for which mail should be accepted during SMTP.
Does anyone actually use this qmail "feature"?
Poured over qmail's documentation and source code to figure out exactly how
it determines where to deliver a message. The documentation is frequently
in error and extensive testing was required to discover the truth. The
resulting procedure is encapsulated in a flowchart in the documentation
folder.
Added the "generator" program to create test scripts to check every \
possible
path through the recipient validation flowchart, both with spamdyke in place
and without (to check the flowchart is correct). A program to generate the
scripts was required, since there are nearly 250K possible paths to test.
Added the value "invalid" to the option "reject-recipient" \
to check if a local
recipient address exists before accepting a message. This validation
process uses the same logic as qmail when deciding whether/where to deliver
a message, so no extra steps are needed to make this work (e.g. maintaining
a list of valid addresses in a separate file). If this process determines
a local address is valid, delivery is guaranteed. This option should
eliminate qmail's habit of sending backscatter spam.
Added the value "unavailable" to the option \
"reject-recipient" to check if a
local recipient is accepting mail at the moment. Probably as a holdover
from the elder days when people actually edited .qmail files by hand, qmail
checks file permissions on files and folders before delivering a message.
If they are set to certain values, qmail will queue the message until the
permissions are fixed or bounce the message if is queued too long. In these
enlightened times, such permissions are more likely to be due to an error or
oversight than deliberate intent.
Added the options "qmail-assign-cdb", \
"qmail-defaultdelivery-file",
"qmail-envnoathost-file", "qmail-locals-file", \
"qmail-me-file",
"qmail-percenthack-file" and "qmail-virtualdomains-file" \
to allow spamdyke
to use different control files than qmail. It's very unlikely anyone will
ever need these options (and it would be unwise to use them), but they're
available just in case.
Added the option "rejection-text-recipient-invalid" to set the \
rejection text
when the "invalid" filter on "reject-recipient" is triggered.
Added the option "rejection-text-recipient-unavailable" to set the \
rejection
text when the "unavailable" filter on "reject-recipient" \
is triggered.
Removed the function filter_recipient_local() and moved its logic into
filter_recipient_valid().
Removed the function filter_recipient_relay() and moved its logic into
filter_recipient_valid().
Changed the "help" option to just show a listing of available options \
without
help text.
Added the "more-help" option to show the full listing of options with \
all help
text.
Added the options "ip-relay-entry", "ip-relay-file", \
"rdns-relay-entry" and
"rdns-relay-file" to allow relaying from specific IPs and/or rDNS names,
since whitelisting no longer implies the ability to relay. If any of these
options are matched, the RELAYCLIENT variable will be set before qmail is
started.
Created the "create_cdb" program to generate CDB files of arbitrary size,
filled with random data, for testing spamdyke's CDB validation routines.
create_cdb also has the ability to corrupt the generated CDB in seven ways;
this makes for more specific testing than simply using a file of random
garbage.
Removed all uses of the TESTSD_* environment variables from the test scripts
and replaced them with appropriate invocations of dnsdummy. This allows the
test scripts to run without potential interference from external DNS
changes and without needing a running spamdyke server to find example
values.
Fixed smtp_filter() and middleman() to clear the list of saved recipient
addresses after printing the log messages. This prevents duplicate log
messages when multiple email messages are delivered in the same connection.
Thanks to Teodor Milkov and David Davidov for reporting this one.
Added the "-skippatched" and "-skipunpatched" flags to the \
"run" scripts to
skip any tests that require a patched or unpatched version of qmail,
respectively.
Fixed a minor bug in find_username() that would truncate the last character
of the username when no domain is given. This hasn't been a problem since
spamdyke rejects recipient addresses without domain names anyway, but one
of the recipient validation test scripts found it.
Added the option "tls-dhparams-file" option to read DH params from a file
for creating ephemeral keys during SSL/TLS key negotiation. Thanks to
Marc Gregel for suggesting this one.
Changed all error messages to output the filename, function name and line
number that generated them, just like the debug and excessive messages.
Added a new log level, LOG_LEVEL_CONFIG_TEST, for config-test error messages.
The level is treated much the same as LOG_LEVEL_ERROR except the filename,
function name and line numbers are not printed.
Added a new decision level, FILTER_DECISION_AUTHENTICATED for authenticated
connections. The filter routines use this level to distinguish between
connections that should be unfiltered due to authentication versus
whitelisting.
Added a new config option type: CONFIG_TYPE_ALIAS. Options of this type are
aliases for other options. This eliminates the duplication of values and
potential for oversights in the graylist/greylist options.
Added some code to the "run" script in the "tests" directory \
to try to detect
core dumps. Some of the tests will declare success even if spamdyke
segfaults and cuts off the output prematurely.
Removed the unused functions reset_rejection() and skip_cfws().
Discovered spamdyke cannot read all the files it needs for recipient
validation during normal operation because they are owned by different users
with restrictive permissions and spamdyke does not run as root. I'm not
sure how I missed that, but it completely moots more than a year of work.
Moved all the recipient valiation code into an external program named
"spamdyke-qrv". This program is meant to only perform recipient \
validation
and nothing else, so it should be safe to run as root (at least safer than
running spamdyke as root).
Removed the options "qmail-assign-cdb", \
"qmail-defaultdelivery-file",
"qmail-envnoathost-file", "qmail-locals-file", \
"qmail-me-file" and
"qmail-percenthack-file" from spamdyke, since the recipient \
validation code
is gone.
Added the option "recipient-validation-command" for passing the path to
spamdyke-qrv, which will be called when recipient validation is needed.
Files: