Log message:
Update to 5.0.1. From the changelog:
Fixed a typo in the README file for the OpenSSL "dhparam" command. \
Thanks to
Eric Shubert for reporting this one.
Removed unused variables from the dns* commands in the utils folder to fix
compiler warnings.
Fixed a bug in read_file() that returned uninitialized pointers if a file
contained blank lines or comments at the top, causing segfaults when they
were free()d. Thanks to Jeffrey Gordon and Quinn Comendant for reporting
this one.
Changed the directory naming scheme in the "generator" program to \
include the
flowchart step numbers in the name. The old pattern was just too hard to
follow visually and far too difficult to search for a specific test.
Added more steps to the recipient validation flowchart and spamdyke-qrv's
recipient validation filter to correctly handle addresses that are forwarded
to an external address. Thanks to Stephen Marley for reporting this one.
Changed search_file() in spamdyke-qrv to return a "not found" result \
when the
file does not exist, instead of an error.
Added a delay loop to exec_command_argv() in spamdyke and spamdyke-qrv to work
around a race condition -- sometimes the child process will close its pipes
in preparation for exiting and the parent's waitpid() will fire before the
child has fully exited. This leads to erroneous returns showing the child
has not exited when it really only needed another timeslice or two. This is
fixed by looping with nanosleep() to wait a few tenths of a second after
seeing this return code.
Added a way to stop a test script run by creating a file named "stop". \
This
allows it to be stopped without killing the process and potentially leaving
the test platform in a partially (mis)configured state.
Fixed the accessor function for the header-blacklist-entry and
header-blacklist-file options to find their data in the filter_settings
object instead of the option_set object. This is because the data is moved
from the option_set immediately after it is set so the blacklist effect is
cumulative when set from configuration directories. Reading from the wrong
location meant the config-test feature was never testing those options at
all. Thanks to Stefan for reporting this one.
Fixed a pair of bugs in process_config_file(): one that would add empty values
to the end of a list of blacklist/whitelist files if a directive was
followed by a blank line and a commented-out directive (causing errors when
the values are used), the other that would throw errors if a line in a
configuration file contained only one space. Thanks to Les Fenison for
reporting these.
Fixed a bug in middleman() that would return an improper greeting when
injecting both AUTH and STARTTLS banners into the EHLO response. Clients
seeing this improper greeting would hang forever and eventually timeout.
Thanks to Elliot Denk for reporting this one and sending a patch!
Fixed a major thinko in smtp_filter that was carrying over the rejection data
between recipients, even if a recipient had a configuration directory file
that altered the overall configuration. This was leading to some
recipients being incorrectly rejected under very specific (and likely very
rare) conditions, which just happened to be met on my own server.
Fixed a bug in copy_base_options that was not copying the "reason" \
data from
the last rejection.
Fixed an infinite loop in dnsdummy when priorities over 0 are used.
Fixed a typo in dnsdummy that was truncating data when the verbose flags were
used (weird, yes).
Changed dnsdummy to fork a child process to return each query. This was the
easiest solution to implement to allow new queries to be processed while
waiting n seconds to send answers to previous queries. This is a fragile
and wasteful solution -- if dnsdummy were intended for production use, a
queue would be a much better solution.
Changed all of the "verbose"-level error messages to include the name \
of the
function, file and line that generated it. Every other message prefixed
with "ERROR" already did this, so this makes things more consistent.
Renamed all of the "FILTER" messages and added a new logging macro to print
them named SPAMDYKE_LOG_FILTER(). This way they can continue to be output
without function, file and line information.
Renamed the SPAMDYKE_LOG_CONFIG_TEST() macro to
SPAMDYKE_LOG_CONFIG_TEST_ERROR() and changed it to use LOG_LEVEL_ERROR
instead of having a special LOG_LEVEL_CONFIG_TEST setting. This way the
config-test messages can be changed to emit file, function and lines if
needed (or not).
Added SPAMDYKE_LOG_CONFIG_TEST_INFO() and SPAMDYKE_LOG_CONFIG_TEST_VERBOSE()
as analogs to SPAMDYKE_LOG_CONFIG_TEST_ERROR().
Changed dnsdummy to encode multiple answers in the same response, if its
config file contains multiple matches for the same query.
Fixed a bug in dnsdummy that was adding extra bytes to the end of each
answer. This turned out to be covering a matching (compensating) bug in
spamdyke's DNS parsing code. I really hate it when that happens!
Fixed a serious bug in nihdns_expand() that was causing spamdyke to
incorrectly parse DNS responses with multiple answers; it would use the
first answer, then skip the wrong number of bytes, causing it to conclude
any subsequent answers were corrupted.
Changed nihdns_expand() to return separate values for the number of bytes in
the decoded string and the number of bytes the string occupies in the DNS
packet. Due to packet compression, the numbers can be very different.
Changed generator to add records to the named configuration so domains will
resolve correctly during testing. Since using port numbers in resolv.conf
is not allowed, there is no easy way to use dnsdummy for these tests.
Discovered qmail-send does not check the percenthack or virtualdomains files
when resolving forward addresses, only locals and assign. Updated
spamdyke-qrv to behave the same way.
Refined the success/failure detection in generator after learning more about
qmail's behavior. If only it had some kind of accurate documentation...
Extended the tests created by generator to also test conditions where
spamdyke-qrv calls vpopmail to look up addresses. This increased the
number of spamdyke-qrv tests more than tenfold!
Added a "diagnostic output" flag to spamdyke-qrv to print the decision path
it used to evaluate the address. Also added a test to the test generator
to compare the diagnostic output with the expected decision path, to
catch tests that may be producing the desired effect for the wrong reason.
Fixed a bug in set_config_value() to make it possible to set
CONFIG_TYPE_NAME_MULTIPLE options to "none" or unset specific values.
Thanks to Konstantin for reporting this one.
Added flags to smtpdummy to advertise STARTTLS support in response to EHLO.
It doesn't actually do TLS, it just advertises it.
Fixed smtp_filter() to block a client's STARTTLS command if tls-level is
"none". Thanks to Les Fenison for reporting this one.
Added a flag to the configure script for both spamdyke and spamdyke-qrv to
compile with the address sanitizer library to catch memory access errors.
Adjusted the version string to show when the sanitizer is in use.
Changed the test scripts to always compile spamdyke with the address
sanitizer (if available) when testing. The tests run a lot slower, but
the sanitizer is too awesome to not use.
Fixed a buffer underrun in examine_entry that was causing segfaults when
searching files where wildcards are allowed at the beginning of the lines.
Thanks to Dirk Kannapinn for reporting this one.
Discovered a horrible problem with snprintf()'s %n format -- it returns the
number of bytes it _would_have_ written *if* there were infinite space, not
the number of bytes *actually* written as the man page states. So using %n
at the end of the format string as a substitute for immediately calling
strlen() is not safe. Good thing I don't ever do that, right? ...wait,
I use that feature EVERYWHERE! (grrrrr) Thanks to the Google Address
Sanitizer team for finding this one. Whoever implemented the %n feature
in glibc can report to me any time for a free punch in the throat.
I want my weekend back.
Reverted the (apparently) useless change from 4.3.0 to use %n in snprintf()
instead of the return value and replaced snprintf() with a macro named
SNPRINTF() that explicitly compares the return value with the size of the
buffer and returns the number of bytes ACTUALLY written.
Fixed a harmless buffer overrun in sub_examine_tcprules_entry() that could
have overwritten one byte of another variable on the stack with a null
byte. Since the address was valid and that other variable is set just
after the overwrite anyway, it wasn't actually a problem. But fixing it
makes the address sanitizer happy, so it's fixed.
Fixed a pair of huge buffer overruns in config_test_file_read() and
config_test_file_read_write() that could load 63K of file contents past
the end of the buffer (on the stack). Fortunately, these functions are
only used by the config-test feature, never during normal operation.
Fixed a buffer overflow in find_address() that would overwrite a single byte
in the caller's stack with a null byte when parsing BATV addresses. From
what I can tell, the effect of this bug would be to either truncate the
parsed address or cause a segfault.
Added undo_softlimit() to try to increase the "soft" limits on address \
space,
stack size and memory size to maximum if they are less than infinite (and
squawk if they cannot be reset to maximum). This will (hopefully) prevent
problems caused by DJB's "softlimit" program, which is a useless piece of
trash many qmail install guides *still* recommend using.
Fixed a bug in the logging code of tls_read() that was using an "error"
message to log at "verbose" level. The error message had more printf()
format specifiers than the verbose logger was providing, which was leading
to segfaults when the message was printed. Many thanks to Konstanin for
a lot of help tracking this one down.
|
Log message:
Update to 5.0.0. From the changelog:
Rearranged the test scripts to put them in folders by category. This just
makes the directory listing a little more manageable.
Corrected some typos in the README file. Thanks to John Mendoza for reporting
those.
Fixed a very obscure bug in spamdyke_log(): on Linux systems (possibly only
64-bit systems), vsyslog() occasionally will not print all the variable
arguments. One way was found to trigger this behavior -- when the
rdns-blacklist-dir filter is activated from a configuration directory.
Fixed a bug in find_domain() that could cause segfaults when parsing certain
invalid formats. Thanks to Gary Gendel for reporting this one.
Added a backup/restore feature to the "run" script in the \
"tests" folder to
save a copy of the most critical system and qmail files before running any
scripts. This is needed because some of the scripts alter those files and,
if they don't run correctly or are cancelled, the originals are lost.
Added a "-skipcompile" flag to the "run" script in the \
"tests" folder to skip
reconfiguring and recompiling all of the binaries when the script is run.
Changed the "run" script in the "tests" folder to empty \
qmail's queue before
and after the tests are run.
Changed the "run" script in the "tests" folder to compare \
the current system
and qmail configuration files to the latest backup after every script
finishes. If they don't match, the latest backup is restored. If they
still don't match, the script stops with an error.
Changed nihdns_query() to accept an optional "preferred" type of \
response. If
multiple types are queried, it will wait for at least one timeout period for
an answer of that type to arrive instead of always accepting the first
answer to arrive. It will accept a saved answer before resending the
queries, however.
NOT BACKWARDS COMPATIBLE: Changed nihdns_mx() to prefer an MX record over an A
record, if both exist. Given the choice, the MX record will be checked for
validity and the A record will be ignored. Thanks to Bruce Schreiber for
suggesting this one.
Fixed filter_level() and smtp_filter() to disregard whitelisting and require
authentication if the "filter-level" option is set to \
"require-auth", as the
documentation says it should. Thanks to Arne for reporting this one.
Changed nihdns_create_packet() to strip trailing dots from names before using
them in DNS queries. A trailing dot is the traditional way to tell libc's
resolver not to append the local domain name and many sysadmins expect to
have to use it. Since spamdyke never appends the local domain and doesn't
use libc's resolver, it isn't necessary and causes lookups to fail. Thanks
to Dossy Shiobara for reporting this one.
Changed middleman() to always send a "STARTTLS" response to \
"EHLO" as a
continuation, never as the last line (only when spamdyke is inserting
"STARTTLS"). This works around a bug in the Android mail client, \
which only
looks for "STARTTLS" as a continuation. Thanks to Jonas Pasche for \
writing
about how to work around this bug on his blog.
NOT BACKWARDS COMPATIBLE: Changed the meaning of "whitelisted" to only \
exempt
the connection from spamdyke's spam filters; whitelisting no longer allows
the connection to relay mail. This means spamdyke will now only set the
RELAYCLIENT environment variable if the "relay-level" option is set to
"allow-all". Relaying must now be controlled through tcpserver or \
xinetd.
Many thanks to Eric Shubert for suggesting and debating this with me.
NOT BACKWARDS COMPATIBLE: Removed the "access-file" and
"rejection-text-access-denied" options because they were only needed for
controlling relaying. Also removed the test scripts that exercised them and
modified many other test scripts that used them.
NOT BACKWARDS COMPATIBLE: Removed the "no-check" value from the \
"relay-level"
option and changed the meaning of the "normal" value to use the logic
previously assigned to "no-check".
Added the option "reject-sender" to take multiple values. If the value
"not-local" is given, the sender will be rejected if the domain name \
is not
hosted locally. If the value "authentication-mismatch" is given, \
the sender
will be rejected if the sender address does not exactly match the username
given during authentication (or if the authentication username is not an
email address, the sender username must match the authentication username).
If the value "authentication-domain-mismatch" is given, the sender \
will be
rejected if the domain name is not part of the username given during
authentication. Thanks to Mark Frater for suggesting this one.
Added the options "rejection-text-sender-not-local" and
"rejection-text-sender-authentication-mismatch" to set the rejection text
given when the "reject-sender" option's filters are triggered.
NOT BACKWARDS COMPATIBLE: Removed the option "reject-missing-sender-mx" and
folded its filter into the "reject-sender" filter's \
"no-mx" option.
NOT BACKWARDS COMPATIBLE: Renamed the option
"rejection-text-missing-sender-mx" to \
"rejection-text-sender-no-mx".
NOT BACKWARDS COMPATIBLE: Renamed the option
"reject-identical-sender-recipient" to "reject-recipient" \
with the value
"same-as-sender". The functionality remains the same.
NOT BACKWARDS COMPATIBLE: Renamed the option
"rejection-text-identical-sender-recipient" to
"rejection-text-recipient-same-as-sender".
NOT BACKWARDS COMPATIBLE: Renamed the option "local-domains-file" to
"qmail-rcpthosts-file". The naming has always been confusing, since \
qmail
distinguishes between domains that should be accepted by qmail-smtpd during
SMTP (rcpthosts) and domains that are actually hosted locally with mailboxes
on the local filesystem (locals). These options have always meant the
former, but now that spamdyke needs to know both lists of domains, it's time
to rename them. This option is also now allowed in configuration
directories.
NOT BACKWARDS COMPATIBLE: Removed the option "local-domains-entry" because
supplying domains that can be accepted during SMTP to spamdyke only (but
not qmail) will cause inconsistent results during recipient validation.
If a domain is to be accepted during SMTP, it should be added to the control
files used by both spamdyke and qmail.
Added CDB searching code in cdb.[ch] to read DJB's "constant database" \
files
during recipient validation. The format of these files is claimed (by DJB)
to be fast and efficient. Don't believe the hype...
Added the option "qmail-morercpthosts-cdb" to allow CDB files to be \
provided
that contain lists of domains for which mail should be accepted during SMTP.
Does anyone actually use this qmail "feature"?
Poured over qmail's documentation and source code to figure out exactly how
it determines where to deliver a message. The documentation is frequently
in error and extensive testing was required to discover the truth. The
resulting procedure is encapsulated in a flowchart in the documentation
folder.
Added the "generator" program to create test scripts to check every \
possible
path through the recipient validation flowchart, both with spamdyke in place
and without (to check the flowchart is correct). A program to generate the
scripts was required, since there are nearly 250K possible paths to test.
Added the value "invalid" to the option "reject-recipient" \
to check if a local
recipient address exists before accepting a message. This validation
process uses the same logic as qmail when deciding whether/where to deliver
a message, so no extra steps are needed to make this work (e.g. maintaining
a list of valid addresses in a separate file). If this process determines
a local address is valid, delivery is guaranteed. This option should
eliminate qmail's habit of sending backscatter spam.
Added the value "unavailable" to the option \
"reject-recipient" to check if a
local recipient is accepting mail at the moment. Probably as a holdover
from the elder days when people actually edited .qmail files by hand, qmail
checks file permissions on files and folders before delivering a message.
If they are set to certain values, qmail will queue the message until the
permissions are fixed or bounce the message if is queued too long. In these
enlightened times, such permissions are more likely to be due to an error or
oversight than deliberate intent.
Added the options "qmail-assign-cdb", \
"qmail-defaultdelivery-file",
"qmail-envnoathost-file", "qmail-locals-file", \
"qmail-me-file",
"qmail-percenthack-file" and "qmail-virtualdomains-file" \
to allow spamdyke
to use different control files than qmail. It's very unlikely anyone will
ever need these options (and it would be unwise to use them), but they're
available just in case.
Added the option "rejection-text-recipient-invalid" to set the \
rejection text
when the "invalid" filter on "reject-recipient" is triggered.
Added the option "rejection-text-recipient-unavailable" to set the \
rejection
text when the "unavailable" filter on "reject-recipient" \
is triggered.
Removed the function filter_recipient_local() and moved its logic into
filter_recipient_valid().
Removed the function filter_recipient_relay() and moved its logic into
filter_recipient_valid().
Changed the "help" option to just show a listing of available options \
without
help text.
Added the "more-help" option to show the full listing of options with \
all help
text.
Added the options "ip-relay-entry", "ip-relay-file", \
"rdns-relay-entry" and
"rdns-relay-file" to allow relaying from specific IPs and/or rDNS names,
since whitelisting no longer implies the ability to relay. If any of these
options are matched, the RELAYCLIENT variable will be set before qmail is
started.
Created the "create_cdb" program to generate CDB files of arbitrary size,
filled with random data, for testing spamdyke's CDB validation routines.
create_cdb also has the ability to corrupt the generated CDB in seven ways;
this makes for more specific testing than simply using a file of random
garbage.
Removed all uses of the TESTSD_* environment variables from the test scripts
and replaced them with appropriate invocations of dnsdummy. This allows the
test scripts to run without potential interference from external DNS
changes and without needing a running spamdyke server to find example
values.
Fixed smtp_filter() and middleman() to clear the list of saved recipient
addresses after printing the log messages. This prevents duplicate log
messages when multiple email messages are delivered in the same connection.
Thanks to Teodor Milkov and David Davidov for reporting this one.
Added the "-skippatched" and "-skipunpatched" flags to the \
"run" scripts to
skip any tests that require a patched or unpatched version of qmail,
respectively.
Fixed a minor bug in find_username() that would truncate the last character
of the username when no domain is given. This hasn't been a problem since
spamdyke rejects recipient addresses without domain names anyway, but one
of the recipient validation test scripts found it.
Added the option "tls-dhparams-file" option to read DH params from a file
for creating ephemeral keys during SSL/TLS key negotiation. Thanks to
Marc Gregel for suggesting this one.
Changed all error messages to output the filename, function name and line
number that generated them, just like the debug and excessive messages.
Added a new log level, LOG_LEVEL_CONFIG_TEST, for config-test error messages.
The level is treated much the same as LOG_LEVEL_ERROR except the filename,
function name and line numbers are not printed.
Added a new decision level, FILTER_DECISION_AUTHENTICATED for authenticated
connections. The filter routines use this level to distinguish between
connections that should be unfiltered due to authentication versus
whitelisting.
Added a new config option type: CONFIG_TYPE_ALIAS. Options of this type are
aliases for other options. This eliminates the duplication of values and
potential for oversights in the graylist/greylist options.
Added some code to the "run" script in the "tests" directory \
to try to detect
core dumps. Some of the tests will declare success even if spamdyke
segfaults and cuts off the output prematurely.
Removed the unused functions reset_rejection() and skip_cfws().
Discovered spamdyke cannot read all the files it needs for recipient
validation during normal operation because they are owned by different users
with restrictive permissions and spamdyke does not run as root. I'm not
sure how I missed that, but it completely moots more than a year of work.
Moved all the recipient valiation code into an external program named
"spamdyke-qrv". This program is meant to only perform recipient \
validation
and nothing else, so it should be safe to run as root (at least safer than
running spamdyke as root).
Removed the options "qmail-assign-cdb", \
"qmail-defaultdelivery-file",
"qmail-envnoathost-file", "qmail-locals-file", \
"qmail-me-file" and
"qmail-percenthack-file" from spamdyke, since the recipient \
validation code
is gone.
Added the option "recipient-validation-command" for passing the path to
spamdyke-qrv, which will be called when recipient validation is needed.
|