./devel/librdkafka, Apache Kafka C/C++ library

[ CVSweb ] [ Homepage ] [ RSS ] [ Required by ] [ Add to tracker ]


Branch: CURRENT, Version: 1.3.0, Package name: librdkafka-1.3.0, Maintainer: pkgsrc-users

librdkafka is a C library implementation of the Apache Kafka protocol,
containing both Producer and Consumer support.


Required to run:
[security/openssl] [archivers/lz4] [archivers/zstd]

Required to build:
[pkgtools/cwrappers] [lang/python37]

Master sites:

SHA1: 20c4ddb2437fc875ba92777a3906fb3a375d7e7f
RMD160: bbb790083dbeb87dcb757d1d9f88e14cf2e28244
Filesize: 2565.457 KB

Version history: (Expand)


CVS history: (Expand)


   2020-05-27 21:37:44 by Thomas Klausner | Files touched by this commit (60)
Log message:
*: reset MAINTAINER for fhajny on his request
   2020-02-01 21:49:43 by Adam Ciarcinski | Files touched by this commit (3) | Package updated
Log message:
librdkafka: updated to 1.3.0

librdkafka v1.3.0 release
This is a feature release adding support for KIP-392 Fetch from follower, \ 
allowing a consumer to fetch messages from the closest replica to increase \ 
throughput and reduce cost.

Features
KIP-392 - Fetch messages from closest replica / follower (by @mhowlett)
Added experimental (subject to change or removal) mock broker to make \ 
application and librdkafka development testing easier.

Fixes
Fix consumer_lag in stats when consuming from broker versions <0.11.0.0 \ 
(regression in librdkafka v1.2.0).

librdkafka v1.2.2 release
v1.2.2 fixes the producer performance regression introduced in v1.2.1 which may \ 
affect high-throughput producer applications.

Fixes
Fix producer insert msgq regression in v1.2.1.
Upgrade builtin lz4 to 1.9.2.
Don't trigger error when broker hostname changes.
Less strict message.max.bytes check for individual messages.
Don't call timespec_get() on OSX (since it was removed in recent XCode) by \ 
@maparent .
configure: add --runstatedir for compatibility with autoconf.
LZ4 is available from ProduceRequest 0, not 3.
Address 12 code issues identified by Coverity static code analysis.

Enhancements
Add warnings for inconsistent security configuration.
Optimizations to hdr histogram (stats) rollover.
Reorganized examples and added a cleaner consumer example, added minimal C++ \ 
producer example.
Print compression type per message-set when debug=msg

librdkafka v1.2.1 release
Warning: v1.2.1 has a producer performance regression which may affect \ 
high-throughput producer applications. We recommend such users to stay on \ 
v1.2.0.

v1.2.1 is a maintenance release:
Properly handle new Kafka-framed SASL GSSAPI frame semantics on Windows.
This bug was introduced in v1.2.0 and broke GSSAPI authentication on Windows.
Fix msgq (re)insertion code to avoid O(N^2) insert sort operations on retry
The msgq insert code now properly handles interleaved and overlapping
message range inserts, which may occur during Producer retries for
high-throughput applications.
configure: added --disable-c11threads to avoid using libc-provided C11 threads.
configure: added more autoconf compatibility options to ignore

librdkafka v1.2.0 release

WARNING: There is an issue with SASL GSSAPI authentication on Windows with this \ 
release. Upgrade directly to v1.2.1 which fixes the issue.

v1.2.0 is a feature release making the consumer transaction aware.

Transaction aware consumer (isolation.level=read_committed) implemented by @mhowlett.
Sub-millisecond buffering (linger.ms) on the producer.
Improved authentication errors (KIP-152)

Consumer-side transaction support
This release adds consumer-side support for transactions.
In previous releases, the consumer always delivered all messages to the \ 
application, even those in aborted or not yet committed transactions. In this \ 
release, the consumer will by default skip messages in aborted transactions.
This is controlled through the new isolation.level configuration property which
defaults to read_committed (only read committed messages, filter out aborted and \ 
not-yet committed transactions), to consume all messages, including for aborted \ 
transactions, you may set this property to read_uncommitted to get the behaviour \ 
of previous releases.
For consumers in read_committed mode, the end of a partition is now defined to \ 
be the offset of the last message of a successfully committed transaction \ 
(referred to as the 'Last Stable Offset').
For non-transactional messages there is no change from previous releases, they \ 
will always be read, but a consumer will not advance into a not yet committed \ 
transaction on the partition.

Upgrade considerations
linger.ms default was changed from 0 to 0.5 ms to promote some level of batching \ 
even with default settings.

New configuration properties
Consumer property isolation.level=read_committed ensures the consumer will only \ 
read messages from successfully committed producer transactions. Default is \ 
read_committed. To get the previous behaviour, set the property to \ 
read_uncommitted, which will read all messages produced to a topic, regardless \ 
if the message was part of an aborted or not yet committed transaction.
Enhancements
Offset commit metadata (arbitrary application-specified data) is now returned by \ 
rd_kafka_committed() and rd_kafka_offsets_for_times()
C++: Added Conf::c_ptr*() to retrieve the underlying C config object.
Added on_thread_start() and on_thread_exit() interceptors.
Increase queue.buffering.max.kbytes max to INT_MAX.
Optimize varint decoding, increasing consume performance by ~15%.

Fixes
General:
Rate limit IO-based queue wakeups to linger.ms, this reduces CPU load and lock \ 
contention for high throughput producer applications.
Reduce memory allocations done by rd_kafka_topic_partition_list_new().
Fix socket recv error handling on MSVC (by Jinsu Lee).
Avoid 1s stalls in some scenarios when broker wakeup-fd is triggered.
SSL: Use only hostname (not port) when valid broker hostname (by Hunter Jacksson)
SSL: Ignore OpenSSL cert verification results if \ 
enable.ssl.certificate.verification=false
rdkafka_example_cpp: fix metadata listing mode (@njzcx)
SASL Kerberos/GSSAPI: don't treat kinit ECHILD errors as errors
Fix compare overflows
configure: Add option to disable automagic dependency on zstd (by Thomas Deutschmann)
Documentation updates and fixes by Cedric Cellier and @ngrandem
Set thread name on MacOS X (by Nikhil Benesch)
C++: Fix memory leak in Headers (by Vladimir Sakharuk)
Fix UBSan (undefined behaviour errors)
CONFIGURATION.md: escape || inside markdown table
Refresh broker list metadata even if no topics to refresh

Consumer:
Make rd_kafka_pause|resume_partitions() synchronous, making sure that a \ 
subsequent consumer_poll() will not return messages for the paused partitions.
Fix incorrect toppar destroy in OffsetRequest
Fix message version 1 offset calculation
Defer commit in transport error to avoid consumer_close hang.

Producer:
Messages were not timed out for leader-less partitions.
Improve message timeout granularity to millisecond precision (the smallest \ 
ffective message timeout will still be 1000ms).
message.timeout.ms=0 is now accepted even if linger.ms > 0 (by Jeff Snyder)
Don't track max.poll.interval.ms unless in Consumer mode, this saves quite a few \ 
memory barries for high-performance Producers.
Optimization: avoid atomic fatal error code check when idempotence is disabled.
   2020-01-18 22:51:16 by Jonathan Perkin | Files touched by this commit (1836)
Log message:
*: Recursive revision bump for openssl 1.1.1.
   2019-07-10 03:31:21 by Min Sik Kim | Files touched by this commit (3) | Package updated
Log message:
devel/librdkafka: Update to 1.0.1

Notable changes in 1.0:
- Idempotent producer - guaranteed ordering, exactly-once producing.
- Sparse/on-demand connections - connections are no longer maintained
  to all brokers in the cluster.
- KIP-62 - max.poll.interval.ms for high-level consumers.
   2019-04-25 09:33:32 by Maya Rashish | Files touched by this commit (620)
Log message:
PKGREVISION bump for anything using python without a PYPKGPREFIX.

This is a semi-manual PKGREVISION bump.
   2018-12-15 22:12:25 by Thomas Klausner | Files touched by this commit (67) | Package updated
Log message:
*: update email for fhajny
   2018-04-05 11:01:16 by Filip Hajny | Files touched by this commit (3) | Package updated
Log message:
devel/librdkafka: Update to 0.11.4.

Default changes

- socket.max.fails changed to 1 to provide same functionality (fail
  request immediately on error) now when retries are working properly
  again.
- fetch.max.bytes (new config property) is automatically adjusted to
  be >= message.max.bytes, and receive.message.max.bytes is
  automatically adjusted to be > fetch.max.bytes.

New features

- Message Headers support
- Java-compatible Murmur2 partitioners
- Add PKCS#12 Keystore support - ssl.keystore.location

Noteworthy fixes

- Formalise and fix Producer retries and retry-ordering
- Ordering is now retained despite retries if max.in.flight=1.
- Behaviour is now documented
- Fix timeouts for retried requests and properly handle retries for
  all request types
- Add and use fetch.max.bytes to limit total Fetch response size.
  Fixes "Invalid response size" issues.

Enhancements

- Added sasl.mechanism and compression.type configuration property
  aliases for conformance with Java client.
- Improved Producer performance
- C++: add c_ptr() to Handle,Topic,Message classes to expose
  underlying librdkafka object
- Honour per-message partition in produce_batch() if MSG_F_PARTITION
  set
- Added on_request_sent() interceptor
- Added experimental flexible producer queuing.strategy=fifo|lifo
- Broker address DNS record round-robin: try to maintain round-robin
  position across resolve calls.
- Set system thread name for internal librdkafka threads
- Added more concise and user-friendly 'consumer' debug context
- Add partitioner (string) topic configuration property to set the
  builtin partitioners
- Generate rdkafka-static.pc (pkg-config) for static linking

Fixes

- Fix producer memory leak on <0.11 brokers when compressed messageset
  is below copy threshold
- CRC32C - fix unaligned access on ARM
- Fix read after free in buf_write_seek
- Fix broker wake up
- Fix consumer hang when rebalancing during commit
- CMake fixes for Windows
- LeaveGroup was not sent on close when doing final offset commits
- Fix for consumer slowdown/stall on compacted topics where actual
  last offset < MsgSet.LastOffset
- Fix global->topic conf fallthru in C++ API
- Fix infinite loop on LeaveGroup failure
- Fix possible crash on OffsetFetch retry
- Incorporate compressed message count when deciding on fetch backoff
- Fix debug-only crash on Solaris (%s NULL)
- Drain broker ops queue on termination to avoid hang
- cmake: Allow build static library
- Don't store invalid offset as next one when pausing
- use #if instead of #ifdef / defined() for atomics
- fixed .lib paths in nuget packaging
- Fixes strerror_r crash on alpine
- Allow arbitrary lengthed (>255) SASL PLAIN user/pass
- Trigger ApiVersionRequest on reconnect if broker.version.fallback
  supports it
- Read Fetch MsgAttributes as int8
- Portability: stop using typeof in rdkafka_transport.c
- Portability: replace use of #pragma once with header guards
- mklove: add LIBS in reverse order to maintain dependency order
- Fix build when python is not available #1358
   2017-12-09 17:57:04 by Filip Hajny | Files touched by this commit (3) | Package updated
Log message:
Update devel/librdkafka to 0.11.3.

Default changes
- Change default queue.buffering.max.kbytes and
  queued.max.message.kbytes to 1GB
- win32: Use sasl.kerberos.service.name for broker principal, not
  sasl.kerberos.principal

Enhancements
- Default producer message offsets to OFFSET_INVALID rather than 0
- new nuget package layout + debian9 librdkafka build
- Allow for calling rd_kafka_queue_io_event_enable() from the C++
  world
- rdkafka_performance: allow testing latency with different size
  messages

Fixes
- Improved stability on termination
- offsets_for_times() return ERR__TIMED_OUT if brokers did not respond
  in time
- Let list_groups() return ERR__PARTIAL with a partial group list
- Properly handle infinite
- Fix offsets_store() return value when at least one valid partition
- portability: rdendian: add le64toh() alias for older glibc
- Add MIPS build and fix CRC32 to work on big endian CPUs
- osx: fix endian checking for software crc32c
- Fix comparison in rd_list_remove_cmp
- stop calling cnd_timedwait() with a timeout of 0h
- Fix DNS cache logic broker.address.ttl
- Fix broker thread "hang" in CONNECT state
- Reset rkb_blocking_max_ms on broker DOWN to avoid busy-loop during
  CONNECT
- Fix memory leak when producev() fails
- Raise cmake minimum version to 3.2
- Do not assume LZ4 worst
- Fix ALL_BROKERS_DOWN re-generation
- rdkafka-performance: busy wait to wait short periods of time