./textproc/xapian, Probabilistic Information Retrieval search engine

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 1.4.9, Package name: xapian-1.4.9, Maintainer: schmonz

Xapian is an Open Source Search Engine Library, released under the
GPL. It's written in C++, with bindings to allow use from Perl,
Python, PHP, Java, Tcl, C# and Ruby (so far!)

Xapian is a highly adaptable toolkit which allows developers to
easily add advanced indexing and search facilities to their own
applications. It supports the Probabilistic Information Retrieval
model and also supports a rich set of boolean query operators.

If you're after a packaged search engine for your website, you
should take a look at Omega: an application we supply built upon
Xapian. Unlike most other website search solutions, Xapian's
versatility allows you to extend Omega to meet your needs as they
grow.


Required to run:
[devel/libuuid]

Required to build:
[pkgtools/cwrappers]

Master sites:

SHA1: 118a47b76b1002303f6761ba395eb42cb36f3b0e
RMD160: 891d756046a46851b98d14c05a09f516092b0a01
Filesize: 2812.523 KB

Version history: (Expand)


CVS history: (Expand)


   2018-11-05 06:42:46 by Amitai Schleier | Files touched by this commit (3) | Package updated
Log message:
Update to 1.4.9. From the changelog:

API:

* Document::add_posting(): Fix bugs with the change in 1.4.8 to more
  efficiently handle insertion of a batch of extra positions in ascending
  order.  These could lead to missing positions and corrupted encoded
  positional data.

remote backend:

* Avoid hang if remote connection shutdown fails by not waiting for the
  connection to close in this situation.  Seems to fix occasional hangs seen on
  macOS.  Patch from Germán M. Bravo.
   2018-10-28 04:43:17 by Amitai Schleier | Files touched by this commit (4) | Package updated
Log message:
Update to 1.4.8. From the changelog:

API:

* QueryParser,TermGenerator: Add new stemming mode STEM_SOME_FULL_POS.
  This stores positional information for both stemmed and unstemmed terms,
  allowing NEAR and ADJ to work with stemmed terms.  The extra positional
  information is likely to take up a significant amount of extra disk space so
  the default STEM_SOME is likely to be a better choice for most users.

* Database::check(): Fetch and decompress the document data to catch problems
  with the splitting of large data into multiple entries, corruption of the
  compressed data, etc.  Also check that empty document data isn't explicitly
  stored for glass.

* Fix an incorrect type being used for term positions in the TermGenerator API.
  These were Xapian::termcount but should be Xapian::termpos.  Both are
  typedefs for the same 32-bit unsigned integer type by default (almost always
  "unsigned int") so this change is entirely compatible, except that if you
  were configuring 1.4.7 or earlier with --enable-64bit-termcount you need to
  also use the new --enable-64bit-termpos configure option with 1.4.8 and up or
  rebuild your applications.  This change was necessary to make
  --enable-64bit-termpos actually useful.

* Add Document::remove_postings() method which removes all postings in a
  specified term position range much more efficiently than by calling
  remove_posting() repeatedly.  It returns the number of postings removed.

* Fix bugs with handling term positions >= 0x80000000.  Reported by Gaurav
  Arora.

* Document::add_posting(): More efficiently handle insertion of a batch of
  extra positions in ascending order.

* Query: Simplify OP_SYNONYM with single OP_WILDCARD subquery by converting to
  OP_WILDCARD with combiner OP_SYNONYM, which means such cases can take
  advantage of the new matcher optimisation in this release to avoid needing
  document length for OP_WILDCARD with combiner OP_SYNONYM.

matcher:

* Avoid needing document length for an OP_WILDCARD with combiner OP_SYNONYM.
  We know that we can't get any duplicate terms in the expansion of a wildcard
  so the sum of the wdf from them can't possibly exceed the document length.

* OP_SYNONYM: No longer tries to initialise weights for its subquery, which
  should reduce the time taken to set up a large wildcard query.

* OP_SYNONYM: Fix frequency estimates when OP_SYNONYM is used with a
  subquery containing OP_XOR or OP_MAX - in such cases the frequency
  estimates for the first subquery of the OP_XOR/OP_MAX were used for
  all its subqueries.  Also the estimated collection frequency is
  now rounded to the nearest integer rather than always being rounded
  down.

glass backend:

* Revert change made in 1.4.6:

    Enable glass's "open_nearby_postlist" optimisation (which \ 
especially helps
    large wildcard queries) for writable databases without any uncommitted
    changes as well.

  The amended check isn't conservative enough as there may be postlist changes
  in the inverter while the table is unmodified.  This breaks testcase
  T150-tagging.sh in notmuch's testsuite, reported by David Bremner.

* When indexing a document without any terms we now avoid some unnecessary work
  when storing its termlist.

tools:

* xapian-delve: Test for all docs empty using get_total_length() which is
  slightly simpler internally than get_avlength(), and avoids an exact floating
  point equality check.

examples:

* quest: Support --weight=coord.

* xapian-pos: New tool to show term position info to help debugging when using
  positional information in more complex ways.

portability:

* Fix undefined behaviour from C++ ODR violation due to using the same name
  two different non-static inline functions.  It seems that with current GCC
  versions the desired function always ends up being used, but with current
  clang the other function is sometimes used, resulting in database corruption
  when using value slots in docid 16384 or higher with the default glass
  backend.  Patch from Germán M. Bravo.

* Suppress alignment cast warning on sparc Linux.  The pointer being cast is to
  a record returned by getdirentries(), so it should be suitable aligned.

* Drop special handling for Compaq C++.  We never actually achieved a working
  build using it, and I can find no evidence that this compiler still exists,
  let alone that it was updated for C++11 which we now require.

* Create new database directories in race-free way.

* Avoid throwing and handling an exception in replace_document() when
  adding a document with a specified docid which is <= last_docid but currently
  unused.

* Use our portable code for handling UUIDs on all platforms, and only use
  platform-specific code for generating a new UUID.  This fixes a bug with
  converting UUIDs to and from string representation on FreeBSD, NetBSD and
  OpenBSD on little-endian platforms which resulted in reversed byte order in
  the first three components, so the same database would report a different
  UUID on these platforms compared to other platforms.  With this fix, the
  UUIDs of existing databases will appear to change on these platforms
  (except in rare "palindronic" cases).  Reported by Germán M. Bravo.

* Fix to build with a C++17 compiler.  Previously we used a "byte" type
  internally which clashed with "std::byte" in source files which use
  "using namespace std;".  Fixes #768, reported by Laurent Stacul.

* Adjust apitest testcase stubdb2 to allow for NetBSD oddity: NetBSD's
  getaddrinfo() in IPv4 mode seems to resolve ::1 to an IPv4 address on the
  local network.

* Avoid timer_create() on OpenBSD and NetBSD.  On OpenBSD it always fails with
  ENOSYS (and there's no prototype in the libc headers), while on NetBSD it
  seems to work, but the timer never seems to fire, so it's useless to us (see
  #770).

* Use SOCK_NONBLOCK if available to avoid a call to fcntl().  It's supported by
  at least Linux, FreeBSD, NetBSD and OpenBSD.

* Use O_NOINHERIT for O_CLOEXEC on Windows.  This flag has essentially the same
  effect, and it's common in other codebases to do this.

* On AIX O_CLOEXEC may be a 64-bit constant which won't fit in an int.  To
  workaround this stupidity we now call the non-standard open64x() instead
  of open() when the flags don't fit in an int.

* Add functions to add/multiply with overflow check.  These are implemented
  with compiler builtins or equivalent where possible, so the overflow check
  will typically just require a check of the processor's overflow or carry
  flag.
   2018-08-26 15:25:33 by Amitai Schleier | Files touched by this commit (6) | Package updated
Log message:
Update to 1.4.7. From the changelog:

API:

* Database::check(): Fix bogus error reports for documents with length zero
  due to a new check added in 1.4.6 that the doclength was between the stored
  upper and lower bounds, which failed to allow for the lower bound ignoring
  documents with length zero (since documents indexed only by boolean terms
  aren't involved in weighted searches).  Reported by David Bremner.

* Query: Use of Query::MatchAll in multithreaded code causes problems because
  the reference counting gets messed up by concurrent updates.  Document that
  Query(string()) should be used instead of MatchAll in multithreaded code, and
  avoid using it in library code.  Reported by Germán M. Bravo.

* Stem:

  + Stemming algorithms added for Irish, Lithuanian, Nepali and Tamil.

  + Merge Snowball compiler changes which improve code generation.

  + Merge optimisations to the Arabic and Turkish stemmers.

glass backend:

* A long-lived cursor on a table in a WritableDatabase could get into
  an invalid state, which typically resulted in a DatabaseCorruptError
  being thrown with the message:

      Db block overwritten - are there multiple writers?

  But in fact the on-disk database is not corrupted - it's just that
  the cursor in memory has got into an inconsistent state.  It looks
  like we'll always detect the inconsistency before it can cause on-disk
  corruption but it's hard to be completely certain.

  The bug is in code to rebuild the cursor when the underlying table
  changes in ways which require that, which is a fairly rare occurrence
  to start with, and only triggers when a block in the cursor has been
  released, reallocated, and we tried to load it in the cursor at the
  same level - the cursor wrongly assumes it has the current version
  of the block.

  Reported with a reproducer by Sylvain Taverne.  Confirmed by David
  Bremner as also fixing a problem in notmuch for which he hadn't managed
  to find a reduced reproducer.
   2018-08-22 11:48:07 by Thomas Klausner | Files touched by this commit (3558)
Log message:
Recursive bump for perl5-5.28.0
   2018-07-06 18:21:57 by Amitai Schleier | Files touched by this commit (5) | Package updated
Log message:
Update to 1.4.6. From the changelog:

API:

* API classes now support C++11 move semantics when using a compiler which
  we are confident supports them (currently compilers which define
  __cplusplus >= 201103 plus a special check for MSVC 2015 or later).
  C++11 move semantics provide a clean and efficient way for threaded code to
  hand-off Xapian objects to worker threads, but in this case it's very
  unhelpful for availability of these semantics to vary by compiler as it
  quietly leads to a build with non-threadsafe behaviour.  To address this,
  user code can #define XAPIAN_MOVE_SEMANTICS before #include <xapian.h> to
  force this on, and will then get a compilation failure if the compiler lacks
  suitable support.

* MSet::snippet():

  + We were only escaping output for HTML/XML in some cases, which would
    potentially allow HTML to be injected into output (this has been assigned
    CVE-2018-0499).

  + Include certain leading non-word characters in snippets.  Previously we
    started the snippet at the start of the first actual word, but there are
    various cases where including non-word characters in front of the actual
    word adds useful context or otherwise aids comprehension.  Reported by
    Robert Stepanek in https://github.com/xapian/xapian/pull/180

* Add MSetIterator::get_sort_key() method.  The sort key has always been
  available internally, but wasn't exposed via the public API before, which
  seems like an oversight as the collapse key has long been available.
  Reported by 张少华 on xapian-discuss.

* Database::compact():

  + Allow Compactor::resolve_duplicate_metadata() implementations to delete
    entries.  Previously if an implementation returned an empty string this
    would result in a user meta-data entry with an empty value, which isn't
    normally achievable (empty meta-data values aren't stored), and so will
    cause odd behaviour.  We now handle an empty returned value by interpreting
    it in the natural way - it means that the merged result is to not set a
    value for that key in the output database.

  + Since 1.3.5 compacting a WritableDatabase with uncommitted changes throws
    Xapian::InvalidOperationError when compacting to a single-file glass
    database.  This release adds similar checks for chert and when compacting
    to a multiple-file glass database.

  + In the unlikely event that the total number of documents or the total
    length of all documents overflow when trying to compact a multi-database,
    we throw an exception.  This is now a DatabaseError exception instead of a
    const char* exception (a hang-over from before this code was turned into a
    public API in the library).

* Document::remove_term(): Handle removing term at current TermIterator
  position - previously the underlying iterator was invalidated, leading to
  undefined behaviour (typically a segmentation fault).  Reported by Gaurav
  Arora.

* TermIterator::get_termfreq() now always returns an exact answer.  Previously
  for multi-databases we approximated the result, which is probably either a
  hang-over from when this method was used during Enquire::get_eset(), or else
  due to a thinking that this method would be used in that situation (it
  certainly is not now).  If the user creates a TermIterator object and asks it
  for term frequencies then we really should give them the correct answer - it
  isn't hugely costly and the documentation doesn't warn that it might be
  approximated.

* QueryParser::parse_query():

  + Now adds a colon after the prefix when prefixing a boolean term which
    starts with a colon.  This means the mapping is reversible, and matches
    what omega actually does in this case when it tries to reverse the mapping.
    Thanks to Andy Chilton for pointing out this corner case.

  + The parser now makes use of newer features in the lemon parser generator to
    make parsing faster and use less memory.

* Enquire::get_mset(): Fix bug with get_mset(0, 0, X) when X > 0 which was
  causing an attempt to access an element in an empty vector.  Reported by
  sielicki in #xapian.

* Stem:

  + Add Indonesian stemming algorithm.

  + Small optimisations to almost all stemming algorithms.

* Stopper:

  + Add Indonesian stopword list.

  + The installed version of the Finnish stopword list now has one word per
    line.  Previously it had several space-separated words on some lines, which
    works with C++'s std::istream_iterator but may be inconvenient for use from
    some other languages.

  + The installed versions of stopword lists are now sorted in byte order
    rather than whatever collation order is specified by LC_COLLATE or similar
    at build time.  This makes the build more reproducible, and also may be
    more efficient for loading into some data structures.

* WritableDatabase::replace_document(term, doc): Check for last_docid wrapping
  when used on a sharded database.

* Database::locked(): Consistently throw FeatureUnavailableError on platforms
  where we can't test for a database lock without trying to take it.
  Previously GNU Hurd threw DatabaseLockError while platforms where we don't
  use fcntl() locking at all threw UnimplementedError.

* Database and WritableDatabase constructors: Fix handling of entries for
  disabled backends in stub database files to throw FeatureUnavailableError
  instead of DatabaseError.

* Database::get_value_lower_bound() now works correctly for sharded databases.
  Previously it returned the empty string if any shard had no values in the
  specified slot.

* PostingIterator was failing to keep an internal reference to the parent
  Database object for sharded databases.

* ValueIterator::skip_to() and check() had an off-by-one error in their docid
  calculations in some cases with sharded databases.

* Add Database::get_total_length() method.  Previously you had to calculate
  this from get_avlength() and get_doccount(), taking into account rounding
  issues.  But even then you couldn't reliably get the exact value when total
  length is large since a double's mantissa has more limited precision than an
  unsigned long long.

* Add Xapian::iterator_rewound() for bidirectional iterators, to test if the
  iterator is at the start (useful for testing whether we're done when
  iterating backwards).

* DatabaseOpeningError exceptions now provide errno via get_error_string()
  rather than turning it into a string and including it in the exception
  message.

* WritableDatabase::replace_document(): when passed a Document object which
  came from a database and has unmodified values, we used to always read
  those values into a memory structure.  Now we only do this if the document
  is being replaced to the same document ID which it came from, which should
  make other cases a bit more efficient.

* Enquire::get_eset(): When approximating term frequencies we now round to the
  nearest integer - previously we always rounded down.
matcher:

* OP_VALUE_*: When a value slot's lower and upper bound are equal, we know
  that exactly how many documents the subquery can match (either 0 or those
  bounds).  This also avoids a division by zero which previously happened
  when trying to calculate the estimate.

* Speed up sorting by keys.  Use string::compare() to avoid having to call
  operator< if operator> returns false.

* Fix clamping of maxitems argument to get_mset() - it was being clamped
  to db.get_doccount(), now it's clamped to db.get_doccount() - first.  In
  practice this doesn't actually seem to cause any issues.

* If a match time limit is in effect, when it expires we now clamp
  check_at_least to first + maxitems instead of to maxitems.  In practice this
  also doesn't seem to actually cause any issues (at least we've failed to
  construct a testcase where it actually makes an observable difference).

* Fix percentages when only some shards have positions.  If the final shard
  didn't have positions this would lead to under-counting the total number leaf
  of subqueries which would lead to incorrect positional calculations (and a
  division by zero if the top level of the query was positional.  This bug was
  introduced in 1.4.3.

* OP_NEAR: Fix "phantom positions", where OP_NEAR would think a term \ 
without
  positional information occurred at position 1 if it had the lowest term
  frequency amongst the OP_NEAR's subqueries.

* Fix termfreq used in weight calculations for a term occurring more than once
  in the query.  Previously the termfreq for such terms was multiplied by the
  number of different query positions they appeared at.

* OP_SYNONYM: We use the doclength upper bound for the wdf upper bound of a
  synonym - now we avoid fetching it twice when the doclength upper bound is
  explicitly needed.

* Short-cut init() when factor is 0 in most Weight subclasses.  This indicates
  the object is for the term-independent weight contribution, which is always 0
  for most schemes, so there's no point fetching any stats or doing any
  calculations.  This fixes a divide by zero for TfIdfWeight, detected by
  UBSan.

* OP_OR: Fix bug which caused orcheck1 to fail once hooked up to run with the
  inmemory backend.

* Iterating of positions has been sped up, which means phrase matching is now
  faster (by a little over 5% in some simple tests).

* Fix use after free of QueryOptimiser hint in certain cases involving
  multiple databases only some of which have positional information.
  This bug was introduced by changes in xapian-core 1.4.3.  Fixes #752,
  reported and analysed by Robert Stepanek.

* An unweighted OP_AND_MAYBE is now optimised to just its left branch - the
  other branch or branches only contribute weight, so can be completely ignored
  when the operator is unweighted.

glass backend:

* Fix glass freelist bug when changes to a new database which didn't modify the
  termlist table were committed.  In this corner case, a block which had been
  allocated to be the root block in the termlist table was leaked.  This was
  largely harmless, except that it was detected by Database::check() and caused
  it to report an error.  Reported by Antoine Beaupré and David Bremner.

* Fix glass freelist bug with cancel_transaction().  The freelist wasn't
  reset to how it was before the transaction, resulting in leaked blocks.
  This was largely harmless, except that it was detected by Database::check()
  and caused it to report an error.

* Improve the per-term wdf upper bound.  Previously we used min(cf(term),
  wdf_upper_bound(db)) which is tight for any terms which attain that
  upper bound, and also for terms with termfreq == 1 (the latter are common
  in the database (e.g. 66% for a database of wikipedia), but probably
  much less common in searches).  When termfreq > 1 we now use
  max(first_wdf(term), cf(term) - first_wdf(term)), which means terms with
  termfreq == 2 will also attain their bound (another 11% for the same
  database) while terms with higher termfreq but below the global bound will
  get a tighter bound.

* Fix Database::locked() on single-file glass db to just return false (such
  databases can't be opened as a WritableDatabase so there can't be a write
  lock).  Previously this failed with: "DatabaseLockError: Unable to get write
  lock on /flintlock: Testing lock"

* Fix compaction when both the input and output are specified as a file
  descriptor.  Previously this threw an exception due to an overeager check
  that destination != source.

* Use O_TRUNC when compacting to single file.  If the output already exists but
  is larger than our output we don't want to just overwrite the start of it.
  This case also used to result in confusing compaction percentages.

* Enable glass's "open_nearby_postlist" optimisation (which especially \ 
helps
  large wildcard queries) for writable databases without any uncommitted
  changes as well.

* Make get_unique_terms() more efficient for glass.  We approximate
  get_unique_terms() by the length of the termlist (which counts boolean terms
  too) but clamp this to be no larger than the document length.  Since we need
  to open the termlist to get its length, it makes more sense to get the
  document length from that termlist for no extra cost rather than looking it
  up in the postlist table.

* Database::check() now checks document lengths against the stored document
  length lower and upper bounds.  Patch from Uppinder Chugh.  Fixes
  https://trac.xapian.org/ticket/617.

* Fix bogus handling of most-recently-read value slot statistics.  It seems
  that we get lucky and this can't actually cause a problem in practice due
  to another layer of caching above, but if nothing else it's a bug waiting to
  happen.

* If we fail to create the directory for a new database because the path
  already exists, the exception now reports EEXIST as the errno value rather
  than whatever errno value happened to be set from an earlier library call.

remote backend:

* xapian-tcpsrv --one-shot no longer forks.  We need fork to handle multiple
  concurrent connections, but when handling a single connection forking just
  adds overhead and potentially complicates process management for our caller.
  This aligns with the behaviour under __WIN32__ where we use threads instead
  of forking, and service the connection from the main thread with --one-shot.

* Fix repeat call to ValueIterator::check() on the same docid to not always
  set valid to true for remote backend.

inmemory backend:

* Fix repeat call to ValueIterator::check() on the same docid to not always
  set valid to true for inmemory backend.

* Use binary chop instead of linear search in all places where we're searching
  for a term or document - we weren't taking advantage of the sorted order
  everywhere.

tools:

* xapian-delve:

  + Document values can contain binary data, so escape them by default for
    output.  Other options now supported are to decode as a packed integer
    (like omindex uses for last modified), decode using
    Xapian::sortable_unserialise(), and to show the raw form (which was the
    previous behaviour).

  + Report current database revision.

* xapian-inspect:

  + Report entry count when opening table

  + Support inspecting single file DBs via a new --table option (which can also
    be used with a non-single-file DB instead of specifying the path to the
    table).

  + Add "first" and "last" commands which jump to the \ 
first/last entry in the
    current table respectively.

  + "until" now counts and reports the number of entries advanced by.

  + Document "until" with no arguments - this advances to the end of \ 
the table,
    but wasn't mentioned in the help.

  + Commands "goto" and "until" which take a key as an \ 
argument now expect the
    key in the same escaped form that's used for display.  This makes it much
    simpler to interact with tables with binary keys.

  + Fix to expect .glass not .DB extension of glass tables.
   2017-09-06 11:03:07 by Thomas Klausner | Files touched by this commit (86)
Log message:
Follow some redirects.
   2017-08-21 20:49:39 by Amitai Schleier | Files touched by this commit (1)
Log message:
Add patchsum for previous.
   2017-08-07 22:01:40 by Amitai Schleier | Files touched by this commit (3) | Package updated
Log message:
Allow installing multiple versions of Lua bindings. Bump PKGREVISION.