GPL. It's written in C++, with bindings to allow use from Perl,
applications. It supports the Probabilistic Information Retrieval
model and also supports a rich set of boolean query operators.
Xapian. Unlike most other website search solutions, Xapian's
2024-03-08 20:00:54 by Amitai Schleier | Files touched by this commit (3) | |
Log message:
xapian: update to 1.4.25. Changes:
API:
* MSet::get_eset(): Don't fetch the collection frequency for each term unless
we're using the Bo1EWeight expansion scheme which actually needs it. In a
simple test this reduced the time taken to do a search and generate expand
terms by a third. Partly addresses #264.
* QueryParser::parse_query(): Fix parse error when using FLAG_CJK_NGRAM (aka
FLAG_NGRAMS) with a query string which has non-CJK followed by whitespace,
CJK, and more non-CJK. Patch from Robert Stepanek
(https://github.com/xapian/xapian/pulls/331).
testsuite:
* unittest: Improve sparse file detection by using SEEK_HOLE, which is
specified by POSIX and seems to be widely supported. On platforms without it
or on an FS with a > 128K block size we will skip the tests involving a 4GB
file, but that's acceptable. On ZFS st_blocks reports the number of blocks
after compression and also lags behind when data has only been committed to
the journal, which means our previous check based on st_blocks couldn't be
made to work without potentially falsely detecting sparse file support.
Fixes #823, reported by someplaceguy.
* apitest: Enable adddoc2 and adddoc5 testcases for sharded databases. We
now just skip the TermIterator::get_termfreq() checks in this case.
glass backend:
* Check Btree level value from disk is in range, which avoids potential out of
range access on corrupt database. Fixes #824, reported by group13.
* Reject invalid blocksize read from corrupted version file. Throw
DatabaseCorruptError if value is out of range or not a power of two.
* Optimise allterms iteration. Most terms don't contain any zero bytes, and
for such terms the key for the first chunk in the termlist table is just the
termname so no decoding is needed when advancing the iterator. This optimisation
is 8.4% faster in a simple test of iterating allterms via xapian-delve.
* Compaction of an empty non-optional table now gives an empty output, whereas
previous it was one block in size (8K by default). This isn't important in
general as the non-optional tables are not likely to be empty in a real
database, but it's helpful for making small test database and it seems weird
that compaction would make a database much larger in percentage terms in this
edge case.
chert backend:
* Check Btree level value from disk is in range, which avoids potential out of
range access on corrupt database. Fixes #824, reported by group13.
build system:
* configure: DragonflyBSD automatically pulls in library dependencies, so set
link_all_deplibs_CXX=no there.
documentation:
* Document allterms_begin() and termlist_begin() iteration order. Thanks to
Eric Wong for querying this.
* Document TermIterator::get_termfreq() quirk. In the case of a TermIterator
from termlist_begin() on a Document from a sharded database, you get term
frequencies from just the shard. Fixes #423
portability:
* Support building on platforms without AI_NUMERICSERV (e.g. macOS 10.5).
Patch from Sergey Fedorov.
|
2023-07-10 17:07:26 by Amitai Schleier | Files touched by this commit (8) |
Log message:
Update to 1.4.23. From the changelog:
API:
* Add `FLAG_NGRAMS` as a preferred new alias for `FLAG_CJK_NGRAM` and
`SNIPPET_NGRAMS` as a preferred new alias for `SNIPPET_CJK_NGRAM`.
In the next release series these features have been expanded to cover many
more languages so the "CJK" in the name has become inaccurate as it \
stands
for "Chinese, Japanese and Korean").
* Database::check(): Improve the exception message for an empty filename from
"Failed to rewind file descriptor -1 (Bad file descriptor)" to \
"Couldn't find
Xapian database or table to check (No such file or directory)".
testsuite:
* The test harness now supports generated databases for all backends and
all testcases which can use generated databases now do, which together
improve test coverage as we now run many testcases under more backends.
Generated databases can also be cached, which helps speed up the test
suite.
* Remove some testcases which are now redundant with other testcases or
no longer useful for other reasons.
* Add regression test for #781, reported by Germán M. Bravo. This bug was
fixed 4 years in 1.4.12 but a clean regression test only became possible
thanks to changes in this release.
* The test harness would attempt to throw a Xapian::Database (rather than
Xapian::DatabaseError) if it failed to rename a temporary database to its
final name. This would fail because there wasn't a database called "rename
failed". Reported by stark3y on #xapian.
* apitest: Fix exceed32bitcombineddb1 testcase (which only runs with
--enable-64bit-docid).
* apitest: Fix nomoredocids1 when configured with --enable-64bit-docid.
* Add testcase for removal of positions for replaced doc (regression test for
bug in git master not present in 1.4.x).
* unittest: Unit test block file functions
* Stop using std::endl in tests since this seems to be C++ best practice
as it causes a flush of the stream, which is rarely actually wanted. Also
often the replacement \n can be combined with a string literal.
* Use Xapian::docid for document ids in testcases. Using `int` or `size_t`
works, but is less correct and can trigger compiler warnings because not all
values are representable.
matcher:
* BM25PlusWeight: With some parameter combinations we were requesting stats we
didn't actually need. Confirmed with Vivek Pal on IRC.
glass backend:
* Avoid unnecessary copying of data when adding an item to a B-tree table which
should speed up indexing a bit.
* xapian-check: If the first chunk for a term is a continuation chunk, the
affected term was missing from the error message.
* xapian-check: We now report the document id as context in many more error
messages.
chert backend:
* Fix wdf upper bounds used when search a modified chert-format
WritableDatabase object. A bug introduced in 1.4.19 meant we were using the
wdf upper bounds from the last committed version of the database. This bug
could cause assertion failures when configured with --enable-assertions.
remote backend:
* Fix thread-safety issue starting remote prog server. Previously we did some
memory allocation in the child process after fork() but before we exec-ed the
specified program, but in a multi-threaded program (which libxapian might be
used in) it's only safe to call async-signal-safe functions in the child
process after fork() until exec, and malloc, etc aren't async-signal-safe.
* If we failed to open /dev/null in the child process while starting a remote
prog server, we would try to throw an exception. That's not going to work
well so now we just call `_exit(-1)`.
inmemory backend:
* Throw exception on docid overflow like we do in other backends.
build system:
* configure: When probing for socklen_t or equivalent include the same headers
as we do in the code to reduce the risk of configure deciding to use
socklen_t but it not getting defined in the code.
* configure: When probing if a particular compiler or linker option is
supported, we were relying on the compiler exit status but some compilers
only warn about unknown options. We now inspect the compiler's stderr
output to try to detect such cases.
* configure: Avoid compiler warning during GCC version check when compiler
needs an option to enable C++11 support.
* Avoid running pwd since the directories we need are available in automake
variables.
documentation:
* Suggest protocol buffers for structured document data. Fixes #53.
* Clarify documentation for release() methods.
* Fix typo "shared database" to "sharded database" in API docs.
* Document that transactions aren't atomic across shards.
tools:
* xapian-progsrv,xapian-tcpsrv: Support multiple DBs with --writable.
* xapian-tcpsrv: Fix default timeouts in --help output which have been reported
as the name of the constant rather than its value since 1.3.3.
* xapian-tcpsrv: When --one-shot is specified, don't do the usual test open of
the specified database(s) as it doesn't really seem useful in this case. The
test harness uses --one-shot so this reduces overhead when running remote
tests.
* Stop using std::endl in tools since this seems to be C++ best practice
as it causes a flush of the stream, which is rarely actually wanted. Also
often the replacement \n can be combined with a string literal.
portability:
* Don't pass mode to open when it's unused as this triggers "missing O_CREAT or
O_TMPFILE?" warnings when compiling for Android.
* Stop using INFINITY macro. If the implementation supports floating point
infinities then HUGE_VAL gives us infinity as a double directly. If not,
then it's the maximum finite value of a double.
* Don't auto-enable _FORTIFY_SOURCE on mingw or mingw-w64. Enabling
_FORTIFY_SOURCE on newer mingw-w64 requires linking with -lssp so we
attempted to stop automatically enabling it there in 1.4.19 but this fix
didn't actually work. Trying to get this to work automatically has proved
difficult and I couldn't find evidence that _FORTIFY_SOURCE was actually
supported on mingw. If it is, enabling manually will still work.
* Fix mingw32 build with C++17 compiler, which ends up with std::byte
conflicting with byte typedefs in system headers, due to us having `using
namespace std;` in some of our internal headers. Switch the ones which
are causing problems to more targetted `using std::string;`, etc instead.
* Fix WIN32 build with --disable-backend-remote which was failing because we
were always trying to build common/socket_utils.cc which fails because
SOCKLEN_T hasn't been probed. Fixes #821, reported by mgautier.
* soaktest: Use C++11 <random> which is more portable than random().
* In WIN32 builds, pass the program pathname separately to CreateProcess()
which is apparently more robust if the program pathname contains spaces.
* Stop trying to set Microsoft-specific SO_EXCLUSIVEADDRUSE option on our
listening sockets. It's not possible to set both SO_REUSEADDR and
SO_EXCLUSIVEADDRUSE so this call will always fail with WSAEINVAL, but we
were ignoring these errors because SO_EXCLUSIVEADDRUSE required admin
privileges in older OS versions.
* Suppress MSVC deprecation warning for GetVersionEx since none of the
suggested replacements seems to actually provide the functionality we are
using from it.
* Fix some warnings from MSVC in the fallback code for overflow-checked
arithmetic.
* Support Enquire::set_time_limit() on GNU Hurd since Hurd now implements
timer_create().
* Remove lingering traces of IRIX support as it's been dead for many years.
debug code:
* Fix build failure with --enable-assertions due to incorrect variable name in
assertion. Patch from Alexei Kharchev in
https://github.com/xapian/xapian/pull/327.
* Fix GCC warning with --enable-log.
* Fix debug logging for 3 GlassDatabase methods to log their parameter.
* Add a mechanism to support debug logging templated return types containing
commas. This was added to fix a build failure on master with --enable-log
reported by ttyS3.
Bindings:
Documentation:
* Where we document how to build the bindings, link to the download page so
the reader can easily find the source to download. Reported by John G. Heim.
* Update lists of debian packages to install for bindings development to
show be for the latest stable release and the one before (bookworm and
bullseye).
* HACKING: Fix typos in deprecation warning section
* HACKING: Update link to Pike bindings
* HACKING: Document issues caused by macOS SIP. Fixes #732.
General:
* Add `FLAG_NGRAMS` as a preferred new alias for `FLAG_CJK_NGRAM` and
`SNIPPET_NGRAMS` as a preferred new alias for `SNIPPET_CJK_NGRAM`.
In the next release series these features have been expanded to cover many
more languages so the "CJK" in the name has become inaccurate as it \
stands
for "Chinese, Japanese and Korean").
* Avoid running pwd in makefiles since the directories we need are available in
automake variables.
CSharp:
* SmokeTest.cs: Fix some incorrect messages if testcases fail.
* Smoketest.cs: Test stemmer description.
Lua:
* Update docs for supported Lua versions.
Perl:
* Generate HTML docs for each class.
* Document all QueryParser FLAG_* constants.
* Update one place which didn't correctly indicate the current status of the
Perl bindings. Fixes #523.
Python:
* Fix bug in wrapper for xapian.Query when constructing OP_WILDCARD queries.
A workaround for this bug which works with existing releases is to pass
0 for the third and fourth optional parameters like so:
xapian.Query(xapian.Query.OP_WILDCARD, "xyz", 0, 0)
Python3:
* Fix bug in wrapper for xapian.Query when constructing OP_WILDCARD queries.
A workaround for this bug which works with existing releases is to pass
0 for the third and fourth optional parameters like so:
xapian.Query(xapian.Query.OP_WILDCARD, "xyz", 0, 0)
Ruby:
* Improve documentation about __call__ methods.
|
2022-12-25 00:10:04 by Thomas Klausner | Files touched by this commit (1) |
Log message:
xapian: remove comment about csharp-xapian
|
2022-12-21 16:29:10 by Thomas Klausner | Files touched by this commit (1) |
Log message:
xapian: note that updating the version WILL break csharp-xapian
|
2022-09-25 14:24:54 by Amitai Schleier | Files touched by this commit (3) |
Log message:
Update to 1.4.21. From the changelog:
glass backend:
* Avoid throwing InvalidArgument when searching for overlong terms in some
cases. Such terms can't match, but are valid to query for. Patch from
Robert Stepanek in https://github.com/xapian/xapian/pull/313.
chert backend:
* Avoid throwing InvalidArgument when searching for overlong terms in some
cases. Such terms can't match, but are valid to query for.
build system:
* Clean up cygwin and mingw configure checks. When we check $host_os, always
anchor at the start (e.g. cygwin* not *cygwin*), and check for msys* as well
since that's a derivative of cygwin and behaves similarly for the things
we're checking here.
* Update to use AX_CXX_COMPILE_STDCXX which is a replacement for
AX_CXX_COMPILE_STDCXX_11 (which we were using) which also supports newer C++
standards versions which will be useful. For C++11 the only difference seems
to be that the macro now checks for attribute support - we use C++11
attributes so that seems a good thing.
documentation:
* INSTALL: Restructure MSVC section for clarity.
* INSTALL: Mention -D_FORTIFY_SOURCE=3 too (fairly new - requires GCC 12).
* Consistently say "macOS" not "Mac OS X", "OS X", etc.
* admin_notes.rst:
+ Update locking section to cover Open File Description locks
+ Add some discussion of block sizes (fixes #819, reported by mgautier)
+ Fix cut-and-paste error - we were suggesting that the docdata table only
exists if there's positional data. Noted by Gaurav Arora.
+ Improve markup
examples:
* Stop using std::endl in examples since this seems to be C++ best practice
as it causes a flush of the stream, which is rarely actually wanted. Also
often the replacement \n can be combined with a string literal.
portability:
* Stop trying to check for incompatible C++ ABI between the compiler used to
build xapian-core and the compiler used to build code using xapian-core.
This check was helpful in the GCC 3 days, but ABI versions 2 and up are
compatible aside from obscure corner cases, and GCC now defaults to using the
latest ABI version it supports. The result is that this check is no longer
useful enough to justify the noise.
We still check for incompatible _GLIBCXX_DEBUG between the library and
application builds, since that will cause things not to work, and the normal
error message doesn't make it clear what's wrong.
Reported by David Bremner.
* Fix new warnings from GCC 12.
* Avoid undefined value use when unpacking a key in a corrupted glass docdata
table. We now skip further checks on the entry in this case.
* Merge allocations in MSVC directory reading compatibility code so we can
allocate in a single malloc() call.
* Add accept() wrapper which checks an assumption that Microsoft's SOCKET type
only actually holds 32 bit values even in 64 bit platforms and throws an
exception if violated.
* Eliminate a use of sprintf.
* Squash some unhelpful MSVC deprecation warnings.
* Declare dummy invalid parameter handler noexcept to fix a warning from MSVC.
* Include <stdlib.h> in configure check for sys_errlist as that's where it is
with mingw and MSVC.
debug code:
* Fix debug logging for Xapian::Internal::intrusive_ptr<const T>.
GCC 12.2 warned about infinite recursion in the old version, and this seems
to be the case. This code has been there a long time, but is only used when
debug logging is enabled. It does seem to be used, so presumably nobody's
tried to log something which triggered it.
Updating during the freeze for the bug and portability fixes.
|
2022-01-08 17:30:35 by Amitai Schleier | Files touched by this commit (1) |
Log message:
Oops, missed updating the language bindings in previous. Spotted by wiz@.
|
2022-01-02 10:31:20 by Amitai Schleier | Files touched by this commit (2) |
Log message:
Update to 1.4.19. From the changelog:
API:
* New QueryParser::FLAG_NO_POSITIONS flag. With this flag enabled, any query
operations which would use positional information are replaced by the nearest
equivalent which doesn't (so phrase searches, NEAR and ADJ will result in
OP_AND). This is intended to replace the automatic conversion of OP_PHRASE,
etc to OP_AND when a database has no positional information, which will no
longer happen in the release series after 1.4.
* Give a compile error for code which adds a Database to WritableDatabase.
Prior to 1.4.19, this compiled and effectively created a \
"black-hole" shard
which quietly discarded any changes made to it.
In 1.4.19 it's still possible to perform this operation by assigning the
WritableDatabase to a Database first, which is harder to fix. This case
throws an exception on git master where it's easier to address.
Reported by David Bremner on #xapian.
* Fix TermIterator::skip_to() with sharded databases which sometimes was
failing to advance all the way to the requested term. Uncovered while
addressing warning from GCC's -Wduplicated-cond, reported by dcb in #816.
* Clamp edit distance to one less than the length of the word we've been asked
to correct, which makes the algorithm we use more efficient. We already
require suggestion to have at least one character in common, so the only
change to suggestions is we'll no longer suggest corrections which are
twice as long or longer even if the edit distance would allow it, which
seems like an improvement in itself.
* Minor optimisation expanding wildcards.
* PostingIterator::get_description(): For an all-docs iterator on a glass
database, get_description() would call get_docid() which isn't valid to
do once the iterator has reached the end.
testsuite:
* Expand allterms test coverage.
matcher:
* Fetch wdf upper bound from postlist which avoids an extra postlist table
cursor seek per weighted query term, and also means we now use a per-shard
wdf upper bound for local shards which will in typically give a tighter
weight upper bound which will tend to make various other matcher
optimisations more effective. Eric Wong reported this speeds up a
particularly slow case from ~2 minutes to ~3 seconds.
With this change, OP_ELITE_SET can now select a different subset of terms for
each shard regardless of shard type (previously this only happened for remote
shards).
* Avoid triggering a pointless maximum weight recalculation if an unweighted
child of a MultiAndPostList prunes.
* Only check if the database has positional information when the query
uses positional information. This should help improve notmuch delete
performance. Thanks to andreas on #notmuch for analysis of the problem.
glass backend:
* Optimise Glass::Inverter::has_positions(). Use const auto& instead of just
auto for the loop variables. Reported to be faster by andreas on #notmuch.
* Cache result of Glass::Inverter::has_positions() since calculating it is
potentially very expensive, while maintaining a cached answer is very cheap.
remote backend:
* Add missing closing parenthesis to reported remote prog context, which has
been missing since this code was first added over 20 years ago! Spotted by
Gaurav Arora.
build system:
* Enable compiler option -fno-semantic-interposition if supported.
This GCC option allows the compiler to optimise essentially assuming
that functions/variables aren't replaced at dynamic link time.
Such replacement is not something that it's useful to do for Xapian
symbols, and we already turn on -Bsymbolic-functions by default which
prevents such replacement anyway by resolving references within the
library at build time.
Reduces the size of the stripped library on x86-64 Debian unstable by
~1%, and likely makes it faster too.
* Avoid bogus deprecation warning when compiling with GCC without optimisation.
In this situation, GCC emits a deprecation warning for code in the definition
of QueryParser::add_valuerangeprocessor() which is provided for backwards
API compatibility even if this method is never used anywhere.
This isn't helpful, especially if the user is using -Werror, so disable the
-Wdeprecated-deprecations warning for this code.
Reported by starmad on #xapian.
* Fix GCC -Wmaybe-uninitialized warning. The warning seems bogus as it's about
the this pointer being passed to a method which doesn't reference the object,
but we can just make the method static to avoid the warning, and that's
arguably cleaner for a method called from the object initialiser list.
* Automatically enable GCC warnings -Wduplicated-cond and -Wduplicated-branches
if using a GCC version new enough to support them. The usefulness of
-Wduplicated-cond was highlighted by dcb in #816.
* Replace uses of obsolete autoconf macros, fixing warnings if configure is
regenerated with a recent release of autoconf.
* Simplify configure probe for sigsetjmp and siglongjmp. Just probe
individually with AC_CHECK_DECLS and then check that both exist with a
preprocessor check.
* Update XO_LIB_XAPIAN to fix warning that AC_ERROR is obsolete with modern
autoconf.
* Support linking against static libxapian with cmake. Patch from Anonymous
Maarten in https://github.com/xapian/xapian/pull/317
* Clean up handling of libs we link libxapian with - previously any libraries
explicitly specified to configure by the user via LIBS=... as well as -lm
(if configure determined it was needed) could get added to XAPIAN_LIBS
multiple times, as well as also getting added to the libxapian link command
anyway by automake/libtool standard handling.
Specifying a library more than once on the link line is not a problem on
common platforms, but may be an issue somewhere (and it's on less common
platforms where the user is more likely to have to specify LIBS to configure
and/or where -lm may be needed).
documentation:
* configure: Add missing AC_ARG_VAR for all programs so that they are
documented in --help output, and so that autoconf knows they are \
"precious"
and preserves them if configure is rerun even when they're specified via an
environment variable.
* Don't use x^2 to mean x squared in API docs. This is potentially confusing
since in C/C++ (and some other languages), ^ means exclusive-or. Write x²
instead, which should be clear to all readers.
* Improve docs for Xapian::Stopper and SimpleStopper.
* docs/intro_ir.rst: Fixed an incorrect term index. Patch from Jaak Ristioja
in https://github.com/xapian/xapian/pull/321.
* Update for the IRC channel move from freenode to libera.chat.
examples:
* quest: Don't enable spelling correction by default. It was really only on by
default because the spelling correction support in quest was added before
--flags. It seems more helpful for the default to match the
Xapian::QueryParser API, and also this fixes the weird situation that
`--flags default` isn't the default you get without any `--flags` option.
* quest: Multiple `--flags` options now get combined - previously only the last
was used.
portability:
* Don't automatically use _FORTIFY_SOURCE on mingw-w64. Recent mingw-w64
versions require -lssp to be linked when _FORTIFY_SOURCE is enabled, so just
skip the automatic enabling. Users who want to enable it can specify it
explicitly.
Fixes #808, reported by xpbxf4.
* Workaround NFS issue in test harness function for deleting test databases.
On NFS, rmdir() can fail with EEXIST or ENOTEMPTY (POSIX allows either)
due to .nfs* files which are used by NFS clients to implement the Unix
semantics of a deleted but open file continuing to exist. We now sleep
and retry a few times in this situation to give the NFS client a chance
to process the closing of the open handle. Problem mentioned in #631.
* configure: Drop -lm special case for Sun C++ as this no longer seems to
be required. Tested with Sun C++ 5.13, which is the oldest version we
now support due to us now requiring C++11.
* Use strerrordesc_np() if available. This is a GNU-specific replacement for
sys_errlist and sys_nerr. It was added in glibc 2.32 since which sys_errlist
and sys_nerr are no longer declared in the headers.
* Update debug logging to use std::uncaught_exceptions() under C++17 and later
since this allows the debug logging to detect a function without RETURN()
annotation which exits normally while there's an uncaught exception
(previously the debug logging would think the stack was being unwound through
the function). This also avoids deprecation warnings - the old
std::uncaught_exception() (note: singular) function was deprecated by
C++17 and removed in C++20.
* Increase size of buffer passed to strerror_r() from 128 to 1024 bytes, which
is the size recommended by the man page on Linux.
* Fix -Wdeprecated-copy warning from clang 13.
|
2021-10-26 13:23:42 by Nia Alarie | Files touched by this commit (1161) |
Log message:
textproc: Replace RMD160 checksums with BLAKE2s checksums
All checksums have been double-checked against existing RMD160 and
SHA512 hashes
Unfetchable distfiles (fetched conditionally?):
./textproc/convertlit/distinfo clit18src.zip
|