./textproc/xapian, Probabilistic Information Retrieval search engine

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 1.4.12, Package name: xapian-1.4.12, Maintainer: schmonz

Xapian is an Open Source Search Engine Library, released under the
GPL. It's written in C++, with bindings to allow use from Perl,
Python, PHP, Java, Tcl, C# and Ruby (so far!)

Xapian is a highly adaptable toolkit which allows developers to
easily add advanced indexing and search facilities to their own
applications. It supports the Probabilistic Information Retrieval
model and also supports a rich set of boolean query operators.

If you're after a packaged search engine for your website, you
should take a look at Omega: an application we supply built upon
Xapian. Unlike most other website search solutions, Xapian's
versatility allows you to extend Omega to meet your needs as they
grow.


Required to run:
[devel/libuuid]

Required to build:
[pkgtools/cwrappers]

Master sites:

SHA1: 4db3d2f14b6eedfdae60069f4dccc252e2944fc9
RMD160: 0a25d7792062f597cf26946ce4e1af56f2e6c5e9
Filesize: 2909.23 KB

Version history: (Expand)


CVS history: (Expand)


   2019-08-31 21:31:04 by Amitai Schleier | Files touched by this commit (1)
Log message:
Fix checksum on new patch.
   2019-08-29 15:41:15 by Amitai Schleier | Files touched by this commit (1)
Log message:
Apply previous fix to PHP5 as well.
   2019-08-28 21:23:39 by Amitai Schleier | Files touched by this commit (1)
Log message:
Reorder includes in PHP bindings to fix SmartOS bulk build error:
"/usr/include/locale.h:53:12: error: 'lconv' is already declared in this \ 
scope"
<https://trac.xapian.org/ticket/793>
   2019-08-02 23:28:12 by Amitai Schleier | Files touched by this commit (5) | Package updated
Log message:
Update to 1.4.12. From the changelog:

API:

* Xapian::PostingSource: When a PostingSource without a clone() method is used
  with a Database containing multiple shards, the documented behaviour has
  always been that Xapian::InvalidOperationError is thrown.  However, since at
  least 1.4.0, this exception hasn't been thrown, but instead a single
  PostingSource object would get used for all the shards, typically leading to
  incorrect results.  The actual behaviour now matches what was documented.

* Xapian::Database: Add size() method which reports the number of shards.

* Xapian::Database::check(): You can now pass a stub database which will check
  all the databases listed in it (or throw Xapian::UnimplementError for
  backends which don't support checking).

* Xapian::Document: When updating a document use a emplace_hint() to make the
  bulk insertion O(n) instead of O(n·log(n)), and use std::move() to avoid
  copying OmDocumentTerm objects.

* Xapian::Query: Add missing get_unique_terms_end() method.

* Xapian::iterator_valid(): Implement for Utf8Iterator

testsuite:

* Fix keepalive1 failures on some platforms.  On some platforms a timeout
  gives NetworkTimeoutError and on others NetworkError - since 1.4.10 changed
  to checking the exact exception type, keepalive1 has been failing on the
  former set of platforms.  We now just check for NetworkError or a subclass
  here (since NetworkTimeoutError is a subclass of NetworkError).

* Run cursordelbug1 testcase with multi databases too.

matcher:

* Ownership of PostingSource objects during the match now makes use of the
  optional reference-counting mechanism rather than a separate flag.

remote backend:

* Fix remote protocol design bug.  Previously some messages didn't send a reply
  but could result in an exception being sent over the link.  That exception
  would then get read as a response to the next message instead of its actual
  response so we'd be out of step.  Fixes #783, reported by Germán M. Bravo.
  This fix necessitated a minor version bump in the remote protocol (to 39.1).
  If you are upgrading a live system which uses the remote backend, upgrade the
  servers before the clients.

* Fix socket leaks on errors during opening a database.  Fixes
  https://github.com/xapian/xapian/pull/237 and #781, reported by Germán M.
  Bravo.

* Don't close remote DB socket on receiving EOF as the levels above won't
  know it's been closed and may try to perform operations on it, which would be
  problematic if that fd gets reused in the meantime.  Leaving it open means
  any further operations will also get EOF.  Reported by Germán M. Bravo.

* We add a wrapper around the libc socket() function which deals with the
  corner case where SOCK_CLOEXEC is defined but socket() fails if it is
  specified (which can happen with a newer libc and older kernel).
  Unfortunately, this wrapper wasn't checking the returned value from socket()
  correctly, so when SOCK_CLOEXEC was specified and non-zero it would create
  the socket() with SOCK_CLOEXEC, then leak that one and create it again
  without SOCK_CLOEXEC.  We now check the return value properly.

* Fix potential infinite loop in ValueCountMatchSpy::merge_results() if passed
  serialised results with extra data appended (which shouldn't happen in normal
  use).

build system:

* Current versions of valgrind result in false positives on current versions of
  macOS, so on this platform configure now only enables use of valgrind if it's
  specified explicitly.  Fixes #713, reported by Germán M. Bravo.

* Refactor macros to probe for compiler flags so they automatically cache
  their results and consistently report success/failure.

* Rename our custom TYPE_SOCKLEN_T macro to XAPIAN_TYPE_SOCKLEN_T.  The
  AX_TYPE_SOCKLEN_T macro defines an alias of TYPE_SOCKLEN_T for itself which
  means it can get used instead in some situations, but it isn't compatible
  with our macro.  We can't just switch to AX_TYPE_SOCKLEN_T as it doesn't
  handle cases we need, so just rename our macro to avoid potential problems.

documentation:

* Improve API documentation for Xapian::Query class.  Add missing doc
  comments and improve some of the existing ones.  Problems highlighted by
  Дилян Палаузов in #790.

* Add Unicode consortium names and codes for categories from Chapter 4, Version
  11 of the Unicode standard.  Patch from David Bremner.

* Improve configure --help output - drop "[default=no]" for --enable-*
  options which default off.  Fixes #791, reported by and patch from Дилян
  Палаузов.

* Fix API documentation typo - Query::op (the type) not op_ (a parameter name).

* Note which version Document::remove_postings() was added in.

* In the remote protocol documentation, MSG_REPLACEDOCUMENTTERM was documented
  as not having a reply, but actually REPLY_ADDDOCUMENT is sent.

* Update list of <xapian/iterator.h> users.

tools:

* copydatabase: A change in 1.4.6 which added support for \ as directory
  separator on platforms where that's the norm broke the code in copydatabase
  which removes a trailing slash from input databases.  Bug reported and
  culprit commit identified by Eric Wong.

portability:

* Resolve crash on Windows when using clang-cl and MSVC.  Reported by Christian
  Mollekopf in https://github.com/xapian/xapian/pull/256.

* Add missing '#include <cstring>'.  Patch from Tanmay Sachan.

* Fix str() helper function when converting the most negative value
  of a signed integer type.

* Avoid calling close() on fd we know must actually be a WIN32 SOCKET.

* Include <ios> not <iomanip> for std::boolalpha.

* Rework setenv() compatibility handling.  Now that Solaris 9 is dead we can
  assume setenv() is provided by Unix-like platforms (POSIX requires it).  For
  other platforms, provide a compatibility implementation of setenv() which
  so the compatibility code is encapsulated in one place rather than replicated
  at every use.

* Fix maintainer-mode builds on Solaris where /bin/tr is not POSIX compliant.
  We now use the simple workaround suggested by the autoconf manual.

* Improve support for Sun C++ (see #631):

  + Suppress unhelpful warning for lambda with multiple return statements.

  + Enable reporting the tags corresponding to warnings, which we need
    to know in order to suppress any new unhelpful warnings.

  + Adjust our workaround for bug with this compiler's <cmath> header to avoid
    a compiler warning.

  + Use -xldscope=symbolic for Sun C++.  This flag is roughly equivalent to
    -Bsymbolic-functions which we've probed for with GNU ld since Xapian 1.0.0.

And from the changelog for the language bindings:

Documentation:

* Update bindings HACKING document.  Reported as out of date by Niwesh Gupta.

CSharp:

* Work around mono terminfo parsing bug - older cli-sn fails with e.g.
  TERM=xterm-256color due to: https://github.com/mono/mono/issues/6752
  Encountered on Kubuntu 18.10 and debugged by Tejasvi Tomar.  Seems to be
  fixed in the mono version in Debian buster.

Perl:

* Suppress warnings from older Perl headers due to use of constructs which look
  like C++11 user-defined literals.  They're fixed in newer versions so they're
  just noise in our build.  We were working around these in the CI build, so
  drop that workaround as we want to make the build warning-clean for users
  too.  Reported by daniel93 on #xapian.

Python3:

* Fix build for changes in Sphinx 2.0 (which drops support for sphinx.main()).
  Fixes #778, reported by karolyi.  Also reported by Gaurav Arora.

* We now throw UnicodeEncodeError for bad Unicode string input.  Previously
  cases such as a lone surrogate would be handled by quietly skipping the bad
  codepoints when converting to UTF-8 to pass to Xapian.

* We no longer use the deprecated old-style Py_UNICODE API, which currently
  gives deprecation warnings and is slated to be removed in Python 4.0.

Ruby:

* Add support for block iteration.  All the iterator methods in the Ruby API
  now accept an optional block.  If no block is given an array is returned so
  existing code will still work.  Partly based on a patch in
  https://github.com/xapian/xapian/pull/232 from Cong Ding.

* Add missing wrappers for all the C++ methods returning iterators which
  weren't wrapped for Ruby.  Fixes #777, reported by do.

* Suppress warnings from Ruby 2.3 headers due to use of register and
  constructs which look like C++11 user-defined literals.  They're fixed in
  newer versions so they're just noise in our build.  We were working around
  these in the CI build, so drop that workaround as we want to make the build
  warning-clean for users too.  Reported by daniel93 on #xapian.

* smoketest.rb: Don't leave temporary databases behind in /tmp.
   2019-06-07 15:37:39 by Ryo ONODERA | Files touched by this commit (3) | Package updated
Log message:
Add rpath to an output of xapian-config --libs, bump PKGREVISION
   2019-03-04 02:38:00 by Amitai Schleier | Files touched by this commit (3) | Package updated
Log message:
Update to 1.4.11. From the changelog:

API:

* MSet::SNIPPET_CJK_NGRAM - new flag for use with MSet::snippet() to enable
  support for selecting and highlighting snippets which works with the
  QueryParser and TermGenerator FLAG_CJK_NGRAM flags.  This mode can also be
  enabled by setting environment variable XAPIAN_CJK_NGRAM to a non-empty
  value.  (There was nominally already support for XAPIAN_CJK_NGRAM in
  MSet::snippet(), but it didn't work usefully - the highlighting added was all
  empty start/end pairs at the end of the span of CJK characters containing the
  CJK ngram terms, which to the user would typically look like it was selecting
  the end of the text and not highlighting anything).

* Deprecate XAPIAN_CJK_NGRAM environment variable.  There are now flags which
  can be used instead in all cases, and there's sadly no portable thread-safe
  way to read an environment variable so checking environment variables is
  problematic in library code that may be used in multithreaded programs.

* Query::OP_ELITE_SET currently incorrectly recursively flattens any OP_OR (or
  OP_OR-like) subqueries into the list of subqueries it selects from - until
  that's fixed, we now select from the full exploded list rather than the last
  n (where n is the number of direct subqueries of the OP_ELITE_SET).

testsuite:

* Testcases which need a generated database now get run with a sharded
  database.

* Avoid using strerror() in the testsuite which removes an obstacle to running
  tests in parallel in separate threads.

matcher:

* Extend detection of cases of OP_SYNONYM with disjoint subqueries (which means
  we don't need document length) which was added in 1.4.8 - we now detect when
  all subqueries are different terms, or when all subqueries are
  non-overlapping wildcards.  The second case is what QueryParser produces for
  a wildcard or partial query with a query prefix which maps to more than one
  term prefix.

glass backend:

* Handle an empty value slot lower bound gracefully.  This shouldn't happen for
  a non-empty slot, but has been reported by a notmuch user so it seems there
  is (or perhaps was as the database was several years old) a way it can come
  about.  We now check for this situation and set the smallest possible valid
  lower bound instead, so other code assuming a valid lower bound will work
  correctly.  Reported by jb55.

chert backend:

* Handle an empty value slot lower bound gracefully, equivalent to the change
  made for glass.

documentation:

* HACKING: We no longer use auto_ptr<>.

* NEWS: Correct factual error in old entry - the 0.4.1 release was Open Muscat
  not OmSee (the OmSee name was only applied after that final release was made,
  and only used internally to BrightStation).

portability:

* Suppress more clang -Wself-assign-overloaded warnings in testcases which are
  deliberately testing handling of self-assignment.

* Add missing includes of <cerrno>.  Fixes #776, reported by Matthieu Gautier.

debug code:

* When configured with --enable-log, the O_SYNC flag was always specified when
  opening the logfile, with the intention that the most recent log entries
  wouldn't get lost if there was a crash, but O_SYNC can incur a significant
  performance overhead and most debugging is not of such crashes.  So we no
  longer specify O_SYNC by default, but you can now request synchronous logging
  by including %! anywhere in the filename specified with XAPIAN_DEBUG_LOG
  (the %! is replaced with the empty string).  We also now use O_DSYNC if
  available in preference to O_SYNC, since the mtime of the log file isn't
  important.
   2019-02-12 20:21:18 by Amitai Schleier | Files touched by this commit (4) | Package updated
Log message:
Xapian-core 1.4.10 (2019-02-12):

API:

* DatabaseClosedError: New exception class thrown instead of DatabaseError when
  an operation is attempted which can't be completed because it involves a
  database which close() was previously called on.  DatabaseClosedError is a
  subclass of DatabaseError so existing code catching DatabaseError will still
  work as before.  Fixes #772, reported by Germán M. Bravo.  Patch from
  Vaibhav Kansagara.

* DatabaseNotFoundError: New exception class thrown instead of
  DatabaseOpeningError when the problem is the problem is "file not \ 
found" or
  similar.  DatabaseNotFoundError is a subclass of DatabaseOpeningError so
  existing code catching DatabaseOpeningError will still work as before.  Fixes
  #773, reported by Germán M. Bravo.  Patch from Vaibhav Kansagara.

* Query: Make &=, |= and ^= on Query objects opportunistically append to
  an existing query with a matching query operator which has a reference
  count of 1.  This provides an easy way to incrementally build flatter query
  trees.

* Query: Support `query &= ~query2` better - this now is handled exactly
  equivalent to `query = query & ~query2` and gives `query AND_NOT query2`
  instead of `query AND (<alldocuments> AND_NOT query2)`.

* QueryParser: Now uses &=, |= and ^= to produce flatter query trees.  This
  fixes problems with running out of stack space when handling Query object
  trees built by abusing QueryParser to parse very large machine-generated
  queries.

* Stopper: Fix incorrect accents in Hungarian stopword list.  Patch from David
  Corbett.

testsuite:

* Test MSet::snippet() with small and zero lengths.  Fixes #759.  Patch from
  Vaibhav Kansagara.

* Fix testcase stubdb4 annotations - this testcase doesn't need a backend.

* Add PATH annotation for testcases needing get_database_path() to avoid having
  to repeatedly list the backends where this is supported in testcase
  annotations.

* TEST_EXCEPTION helper macro now checks that the exact specified exception
  type is thrown.  Previously it would allow a subclass of the specified
  exception type, but in testcases we really want to be able to test for an
  exact type.  Issue noted by Vaibhav Kansagara on IRC.

matcher:

* Map OP_VALUE_GE/OP_VALUE_LE on an empty slot to EmptyPostList.  We already do
  this for OP_VALUE_RANGE, and it's a little more efficient than creating a
  postlist object which checks the empty value slot.

glass backend:

* We no longer flush all pending positional changes when a postlist, termlist
  or all-terms is opened on a modified WritableDatabase.  Doing so was
  incurring a significant performance cost, and the first of these happens
  internally when `replace_document(term, doc)` is used, which is the usual way
  to support non-numeric unique ids.  We now only flush pending positional
  changes when committing.  Reported and diagnosed by Germán M. Bravo.

remote backend:

* Use poll() where available instead of select().  poll() is specified by
  POSIX.1-2001 so should be widely available by now, and it allows watching any
  fd (select() is limited to watching fds < FD_SETSIZE).  For any platforms
  which still lack poll() we now workaround this select() limitation when a
  high numbered fd needs to be watched (for example, by trying a non-blocking
  read or write and on EAGAIN sleeping for a bit before retrying).

* Stop watching fds for "exceptional conditions" - none of these are \ 
relevant
  to our usage.

* Remove 0.1s timeout in ready_to_read().  The comment says this is to avoid a
  busy loop, but that's out of date - the matcher first checks which remotes
  are ready to read and then does a second pass to handle those which weren't
  with a blocking read.

build system:

* Stop probing for header sys/errno.h which is no longer used - it was only
  needed for Compaq C++, support for which was dropped in 1.4.8.

documentation:

* docs/valueranges.html: Update to document RangeProcessor instead of
  ValueRangeProcessor - the latter is deprecated and will be gone in the next
  release series.

* Document RangeProcessor::operator()() returns OP_INVALID to signal it doesn't
  recognise a range.

* Update some URLs for pages which have moved.

* Use https for URLs where available.

* HACKING: Update "empty()" section for changes in C++11.

portability:

* Suppress clang warnings for self-assignment tests.  Some testcases trigger
  this new-ish clang warning while testing that self-assignment works, which
  seems a useful thing to be testing - at least one of these is a regression
  test.

* Add std::move to fix clang -Wreturn-std-move warning (which is enabled by
  -Wall).

* Add casts to fix ubsan warnings.  These cases aren't undefined behaviour, but
  are reported by ubsan extra checks implicit-integer-truncation and/or
  implicit-conversion which it is useful to be able to enable to catch
  potential bugs.

* Fix check for when to use _byteswap_ulong() - in practice this would only
  have caused a problem if a platform provided _byteswap_ushort() but not
  _byteswap_ulong(), but we're not aware of any which do.

* Fix return values of do_bswap() helpers to match parameter types (previously
  we always returned int and only supported swapping types up to 32 bits, so
  this probably doesn't result in any behavioural changes).

* Only include <intrin.h> if we'll use it instead of always including it when
  it exists.  Including <intrin.h> can result in warnings about duplicate
  declarations of builtin functions under mingw.

* Remove call to close()/closesocket() when the argument is always -1 (since
  the change to use getaddrinfo() in 1.3.3).
   2018-11-05 06:42:46 by Amitai Schleier | Files touched by this commit (3) | Package updated
Log message:
Update to 1.4.9. From the changelog:

API:

* Document::add_posting(): Fix bugs with the change in 1.4.8 to more
  efficiently handle insertion of a batch of extra positions in ascending
  order.  These could lead to missing positions and corrupted encoded
  positional data.

remote backend:

* Avoid hang if remote connection shutdown fails by not waiting for the
  connection to close in this situation.  Seems to fix occasional hangs seen on
  macOS.  Patch from Germán M. Bravo.