Log Message:
Update devel/kafka to 0.11.0.0.

New Feature
- KIP-146: Support per-connector/per-task classloaders in Connect
- Add Record Headers
- Add purgeDataBefore() API in AdminClient
- Add KStream.peek(ForeachAction)
- Add a tool to Reset Consumer Group Offsets
- Add internal leave.group.on.close config to consumer
- Add Exactly-Once Semantics to Streams
- Implement Transactional Coordinator
- Range Scan for Windowed State Stores

Improvement
- Extend wire protocol to allow CRC32C
- Cluster collection returning methods should never return null
- Add Prompt to kafka-console-producer
- Allow users greater access to register custom streams metrics
- Exponential backoff for broker reconnect attempts (KIP-144)
- Add JMH module for Benchmarks
- Split the ProducerBatch and resend when received
  RecordTooLargeException
- Allow per stream/table timestamp extractor
- support throttling on request rate
- Replication Quota Tests: Ensure ZK updated before tests start
- REST configuration not visible in connector properties config files
- TopicCommand --describe shows topics marked for deletion as
  under-replicated and unavailable (KIP-137)
- RocksDB checkpoint files lost on kill -9
- Treat null-key records the same way for joins and aggreations
- Per listener security setting overrides (KIP-103)
- Kafka does not offer kafka-configs.bat on Windows box
- Parametrize streams benchmarks to run at scale
- Error message from Struct.validate() should include the name of the
  offending field.
- Add application.id to StreamThread name
- Improve Streams Reset Tool console output
- Add Float serializer, deserializer, serde
- Exploit #peek to implement #print() and other methods
- The Kafka build should run findbugs
- Inner classes which don't need a reference to the outer class should
  be static
- Fix findbugs warnings in kafka-tools
- throw NoOffsetForPartitionException once for all assigned partitions
  from poll
- Stream round-robin scheduler is inneficient
- Fix several FindBugs warnings in Clients and Connect
- SCRAM secret's should be better protected with Zookeeper ACLs
- set internal.leave.group.on.close to false in KafkaStreams
- Add listener tag to socket-server-metrics.connection-... metrics
- Followups from KIP-101
- We shouldn't pass the underlying exception to
  RetriableCommitFailedException when an async offset commit fails.
- Optionally print out metrics after running the perf tests
- ReassignPartitionsCommand should protect against empty replica list
  assignment
- Censor SCRAM config change logging
- DumpLogSegments should not open index files with `rw`
- Add support for reading PID snapshot files to DumpLogSegments
- Improve internal Task APIs
- Trunk compatibility tests should test against 0.10.2
- Improve message for Kafka failed startup with non-Kafka data in
  data.dirs
- Controlled shutdown timeout message improvement
- reassign-partitions to check if broker of ID exists in cluster
- Add option "dry run" to Streams application reset tool
- AdminClient: add controller and clusterId methods to
  DescribeClusterResults
- KIP-153: Include only client traffic in BytesOutPerSec metric
- Application Reset Tool does not need to seek for internal topics
- New Short serializer, deserializer, serde
- Change Default unclean.leader.election.enabled from True to False
- Sticky Assignor should not cache the calculated assignment (KIP-54
  follow-up)
- Support ExtendedDeserializer in Kafka Streams
- Remove Broker ACL resource type
- Request log should log throttle time
- Generate javadoc for AdminClient and show configs in documentation
- Annoying braces showed on log.error using streams

Bug
- Implement generation/term per leader to reconcile messages correctly
- Hold the produce request with ack > 1 in purgatory until replicas'
  HW has larger than the produce offset (KIP-101)
- Delete consumer offsets from kafka once the topic is deleted
- Clean up Controller Object on forced Resignation
- ConsumerGroupCommand throws GroupCoordinatorNotAvailableException
  when describing a non-existent group before the offset topic is
  created
- SASL unit tests dont work with IBM JDK
- Follower Broker cannot start if offsets are already out of range
- Remove deprecated producer configs.
- Kafka default -Xloggc settings should include GC log rotation flags
- Streams is creating two ProducerRecords for each send via
  RecordCollector
- Broker faills to start after ungraceful shutdown due to
  non-monotonically incrementing offsets in logs
- Log should check the return value of dir.mkdirs()
- KIP-115: __consumer_offsets wrong number of replicas at startup
- Issue with processing order of consumer properties in console
  consumer
- Exit Strategy: using exceptions instead of inline invocation of
  exit/halt
- Documentation still refers to AsyncProducer and SyncProducer
- KIP-151: Connect REST API should expose whether each connector is a
  source or sink
- KTable doesn't work with ProcessorTopologyTestDriver in Kafka
  0.10.1.0
- rat license check not running in Jenkins
- When using ProcessorTopologyTestDriver, the combination of map and
  .groupByKey does not produce any result
- Set more conservative default values on RocksDB for memory usage
- Significant startup delays in KStreams app
- Kafka should not require SSL trust store password
- When the destination brokers are down or misconfigured in config,
  Streams should fail fast
- Connect Producer and Consumer ignore ssl parameters configured for
  worker
- Annotate integration tests and provide gradle build targets to run
  subsets of tests
- Controller send thread can't stop when broker change listener event
  trigger for dead brokers
- the argument of shell in doc wrong and command parsed error
- Kafka Streams allows you to provide strings with illegal characters
  for internal topic names
- gradle-generated core eclipse project has incorrect source folder
  structure
- Refresh consumer metadata more frequently for unknown subscribed
  topics
- KIP-154: Connect should create internal topics
- Fix Streams window retention policy
- KIP-100 api changes break Java 8 lambda expressions in some cases
- Python VerifiableConsumer service has thread-safety bug for
  event_handlers
- Avoid unnecessary task movement across threads during rebalance
- OffsetValidationTest fails validation with "Current position greater
  than the total number of consumed records"
- StreamsKafkaClient drops security configs
- Fix Transient failure in
  BrokerApiVersionsCommandTest.checkBrokerApiVersionCommandOutput
- Implement remaining KIP-66 SMTs
- Fix logic for re-checking if internal topic is ready
- connect jars are missing LICENSE/NOTICE files
- Timed out waiting for consumption in
  OffsetValidationTest.test_broker_failure
- Clean up of state directories can possibly remove stores that are
  about to be used by another thread
- KafkaConsumer#commitSync should clone its input
- Fix deadlock issue during MM shutdown
- Remove generic type of class ClientState
- Memory leak in RecordAccumulator.append
- Streams_bounce test failing occassionally
- fix join-time-max and sync-time-max MeasurableStat type
- Streams Simple Benchmark MB/sec calculation is not correct for Join
  operations
- SimpleBenchmark test fails for streams
- Improve NetworkClient trace logging of request details
- Connect WorkerSinkTask is missing checks for NO_TIMESTAMP
- \ 
org.apache.kafka.clients.producer.KafkaProducerTest#testConstructorFailureCloseR \ 
esource
  and Similar Tests are Failing on some Systems (127.0.53.53 Collision
  Warning)
- Implement graceful handling for improperly formed compressed message
  sets
- Kafka client Heartbeat thread use all the cpu.
- Failure in
  kafka/tests/kafkatest/tests/core/security_rolling_upgrade_test.py
- KIP-128: Blackbox or pass through converter or ByteArrayConverter
  for connect
- Broker level configuration 'log.segment.bytes' not used when
  'segment.bytes' not configured per topic.
- ProcessorTopologyTestDriver does not forward extracted timestamps to
  internal topics
- Kafka cannot recover after a disk full
- Kafka Streams - unable to add state stores when using wildcard
  topics on the source
- Fix some findbugs warnings in Kafka Java client
- Streams State transition ASCII diagrams need fixing and polishing
- KafkaConsumer: ConsumerConfig gets logged twice.
- docker/run_tests.sh should set up /opt/kafka-dev to be the source
  directory
- SchemaBuilder should be more lax about checking that fields are
  unset if they are being set to the same value
- ReplicaFetchThread may fail to create due to existing metric
- ZookeeperLeaderElector not respecting zookeeper.set.acl
- Fix some findbugs warnings in Kafka Streams
- ProcessorTopologyTestDriver does not work when using .through()
- Config validation in Connector plugins need to compare against both
  canonical and simple class names
- NetworkClient should only consider a connection to be fail after
  attempt to connect
- Stream thread getting into deadlock state while trying to get
  rocksdb lock in retryWithBackoff
- SessionStore.fetch(key) is a performance bottleneck
- Struct SchemaBuilder should not allow duplicate fields.
- Querying window store may return unwanted keys
- Kafka Secure Migrator tool doesn't secure all the nodes
- Kafka console consumer property is ignored
- Kafka Connect does not log connector configuration errors
- processstreamwithcachedstatestore and other streams benchmarks fail
  occasionally
- kafka.request.logger TRACE regression
- Fix findbugs "default character set in use" warnings
- Fix findbugs "format string should use %n rather than \n" in tools
- Fix findbugs warnings in kafka-core
- Make ProduceRequest thread-safe
- Utils#delete should correctly handle I/O errors and symlinks
- Remove unused code for reading Shell command stdout and add unit
  test
- Add streams tests with brokers failing
- Document that stores must not be closed when Processors are closed
- Fix findbugs warnings in Kafka-Connect-API
- Add a configurable delay to the initial consumer group rebalance
- KStreamsTestDriver fails with NPE when KStream.to() sinks are used
- Transformation Key/Value type references should be to class name(),
  not canonicalName()
- Batch resetting offsets in Streams' StoreChangelogReader
- Kafka Connect: Offset committing times out before expected
- Fix an "unread field" findbugs warning in streams examples
- Suppress findbugs warnings about machine-generated code in
  jmh-benchmarks
- remove controller concurrent access to non-threadsafe NetworkClient,
  Selector, and SSLEngine
- Delete the kafka to prefix the name of the keystore and truststore
  file will be more suitable
- kafka-connect: fix findbugs issues in connect/runtime
- testReprocessingFromScratch unit test failure
- Fix findbugs warnings in kafka-clients
- Fix remaining findbugs warnings in Kafka Streams
- StreamThread should catch InvalidTopicException
- JoinIntegrationTest fails occasionally
- Fail the build when findbugs fails
- SSL Channel not ready but tcp is established and the server is hung
  will not sending metadata
- running multiple kafka streams instances causes one or more instance
  to get into file contention
- Increase number of Streams producer retries from the default of 0
- InFlightRequests#isEmpty() always returns false
- Add FindCoordinatorRequest RPC stub and update InitPidRequest for
  KIP-98
- NullPointerException while using GlobalKTable in KafkaStreams
- Chroot check should be done for each ZkUtils instance
- Avoid DNS reverse lookup in security-critical TLS code path
- Kafka Streams skipped-records-rate sensor producing nonzero values
  even when FailOnInvalidTimestamp is used as extractor
- Kafka Streams stuck rebalancing after exception thrown in rebalance
  listener
- Defer exception to the next pollOnce() if consumer's fetch position
  has already increased
- PartitionRecords.fetchRecords(...) should defer exception to the
  next call if iterator has already moved across any valid record
- two versions of jackson-annotations-xxx.jar in distribution tgz
- Update topic expiry time in Metadata every time the topic metadata
  is requested
- some spelling error in code comment
- Kafka Streams SessionStore.findSessions javadoc broken
- ThreadCacheTest.cacheOverheadsSmallValues fails intermittently
- Replica Deletion Regression from KIP-101
- ProducerPerformanceService failing due to parsing error
- Remove KafkaController's incrementControllerEpoch method parameter
- Transient test failure
  SocketServerTest.testMetricCollectionAfterShutdown
- Controller Health Metrics (KIP-143)
- Flaky ResetIntegrationTest
- Windows platform does not offer kafka-broker-api-versions.bat
- MinTimestampTracker uses confusing variable names
- LZ4 decompression is 4-5x slower than Snappy on small batches /
  messages
- SetSchemaMetadata does not replace the schemas in structs correctly
- streams task gets stuck after re-balance due to LockException
- KafkaConsumer.close should be idempotent
- KafkaAdminClientIntegration test should wait until metadata is
  propagated to all brokers
- CachingSessionStore doesn't fetchPrevious correctly.
- SASL tests failing with Could not find a 'KafkaServer' or
  'sasl_plaintext.KafkaServer' entry in the JAAS configuration
- RocksDb might stall in environments with 1 core only
- RocksDbStore#openIterators should be synchronized, since it is
  accessed from multiple threads
- Percentilles are calculated incorrectly
- CachingSessionStore doesn't use the default keySerde.
- RocksDBSessionStore doesn't use default aggSerde.
- KafkaConsumer should not skip a corrupted record after throwing an
  exception.
- IllegalStateException in ensureOpenForRecordAppend
- Small JavaDoc fix for AdminClient#describeTopics
- Cached Session/Window store may return error on
  iterator.peekNextKey()
- NullPointerException (NPE) in
  SourceNodeRecordDeserializer.deserialize
- Reflections logs excessive warnings when scanning classpaths
- Recommended values for Connect transformations contain the wrong
  class name
- Kafka broker fails to start if a topic containing dot in its name is
  marked for delete but hasn't been deleted during previous uptime
- Regression in on-disk log size when using Snappy compression with
  0.8.2 log message format
- GlobalKTable does not checkpoint offsets after restoring state
- Tests which delete singleton metrics break subsequent metrics tests
- handleFetchRequest should do down conversion after throttling
- kakfa-clients consume 100% CPU with manual partition assignment when
  network connection is lost
- Move ACLs, Config, NodeVersions classes into org.apache.kafka.common
- Follow-up improvements for consumer offset reset tool (KIP-122)
- kafka-console-consumer: `--value-deserializer` is not working but
  `--property value.deserializer` does
- One StopReplicaRequest will caused two Responses
- Do not apply exponential backoff if users have overridden
  reconnect.backoff.ms
- PlainSaslServerFactory should allow a null Map in getMechanismNames
- FetchRequest doesn't implement toString
- Missing logging information in ReplicaFetcher
- Stores not queryable after one thread died
- Log cleaning can increase message size and cause cleaner to crash
  with buffer overflow
- MemoryRecords.filterTo can return corrupt data if output buffer is
  not large enough
- AdminClient: add close with timeout, fix some timeout bugs
- Connection Lose during Kafka Kerberos Renewal process
- Replica list in the metadata cache on the broker may have different
  order from zookeeper
- rocksdb.config.setter must be a class instance, not a class name
- ListGroup requires Describe on Cluster, but the command-line
  AclCommand tool does not allow this to be set
- There is a Misspell in ResetIntegrationTest
- Change message.timestamp.difference.max.ms back to Long.MaxValue
- Some socket connections not closed after restart of Kafka Streams
- KafkaConsumer occasionally hits IllegalStateException
- Modify Unstable annotations in Streams API
- baseTimestamp should always have a create timestamp
- MirrorMaker not preserving headers
- Down-converted uncompressed batches should respect fetch offset
- Add EOS integration tests for Streams API
- Add EOS system tests for Streams API
- Kafka Streams skipped-records-rate sensor produces nonzero values
  when the timestamps are valid
- ConsoleConsumer prints out object addresses rather than what is
  expected
- AdminClient gets "server returned information about unknown
  correlation ID" when communicating with older brokers
- Transactions: Concurrent transactional consumer loses messages when
  there are broker bounces
- Transient test failure:
  KafkaConsumerTest.testChangingRegexSubscription
- Log recovery can fail if topic names contain one of the index
  suffixes
- Transactional Producer allows batches to expire and commits
  transactions regardless
- KafkaAdminClient#timeoutCallsInFlight does not work as expected
- Distributed Herder Deadlocks on Shutdown
- Add more AdminClient checks to ClientCompatibilityTest
- Log cleaner fails due to large offset in segment file
- Console consumer offset commit regression
- TransactionCoordinator doesn't complete transition to PrepareCommit
  state
- TransactionCoordinator doesn't complete transition to CompleteCommit
- ZkUtils.getAllPartitions() may fail if a topic is marked for
  deletion
- Multiple produce request failures causes invalid state transition in
  TransactionManager
- Transient test failure:
  SaslPlainSslEndToEndAuthorizationTest.testNoProduceWithDescribeAcl
- Streams producer `client.id` are not unique for EOS
- TimestampConverter's "type" config conflicts with the basic
  Transformation "type" config
- Flaky test
  TransactionsTest.testReadCommittedConsumerShouldNotSeeUndecidedData
- Scripts to startup Connect in system tests have too short a timeout
- Update java docs for consumer and producer to be up to date for EOS
- Producer fails with NPE if compressed V0 or V1 record is larger than
  batch size
- MemoryRecordsBuilder.hasRoomfor doesn't take into account the
  headers while computing available space
- Controller incorrectly logs rack information when new brokers are
  added
- Connector validate REST endpoint returning duplicate entries in
  "groups"
- Connector config validation REST API endpoint not including fields
  for transformations
- TransactionalProducer sleeps unnecessarily long during back to back
  transactions
- org.apache.kafka logging should go to server.log
- The ProducerPerformance tool should support transactions
- Connect validation API stops returning recommendations for some
  fields after the right sequence of requests

Task
- Remove deprecated APIs for 0.11.0.0
- Drop support for Scala 2.10 (KIP-119)
- Remove unstable markers from Connect APIs
- KTable materialization and improved semantics

Test
- Transient failure in
  ZooKeeperSecurityUpgradeTest.test_zk_security_upgrade with
  security_protocol = SASL_PLAINTEXT, SSL
- Test with two SASL_SSL listeners with different JAAS contexts
- Implement KIP-98 transactional methods in the MockProducer
- Add cases for concurrent transactional reads and writes in system
  tests
- SyncProducerTest.testReachableServer has become flaky

Sub-task
- KIP-54: Add rebalance with a minimal number of reassignments to
  server-defined strategy list
- ProducerSendTest.testCloseWithZeroTimeoutFromCallerThread Transient
  Failure
- Transient Failure in kafka.api.PlaintextProducerSendTest.testFlush
- Mark the old Scala consumer and related classes as deprecated
- Create Java Admin Client
- Implement KIP-140 RPCs and APIs for creating, altering, and listing
  ACLs
- Describe/Alter Configs protocol, server and client (KIP-133)
- Transient test failure:
  kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime
- Unstable test
  KStreamRepartitionJoinTest.shouldCorrectlyRepartitionOnJoinOperations
- Transient test failure:
  ConsumerBounceTest.testConsumptionWithBrokerFailures
- Transient failure in QueryableStateIntegrationTest.queryOnRebalance
- Address 2.12 eta-expansion warnings
- Update release process so that Scala 2.12 artifacts are published
- Run tests on travis-ci using docker
- Transient failure in
  \ 
org.apache.kafka.clients.consumer.KafkaConsumerTest.testWakeupWithFetchDataAvail \ 
able
- QueryableStateIntegrationTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable
  is occasionally failing on jenkins
- Improve test coverage of ProcessorStateManager
- Improve test coverage of StreamsPartitionAssignor
- Improve test coverage of ProcessorTopology
- Improve test coverage AbstractProcessorContext
- Improve test coverage of GlobalStreamThread
- Improve test coverage StreamTask
- Improve test coverage GlobalStateManagerImpl
- Improve test coverage KStreamBuilder
- Improve test coverage MemoryLRUCache
- Improve test coverage of CompositeReadOnlyWindowStore
- Improve test coverage KafkaStreams
- Improve test coverage TopologyBuilder
- Transient test failure:
  ConsumerCoordinatorTest.testHeartbeatThreadClose
- Message format changes for idempotent/transactional producer
- Implement idempotent producer
- Implement transactional clients
- Transient test failure:
  org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion
  (again)
- Consider disabling record level CRC checks for message format V2
- Failure in
  \ 
kafka.admin.DescribeConsumerGroupTest.testDescribeExistingGroupWithNoMembersWith \ 
NewConsumer
- Implementation and unit tests for request handler thread utilization
  quota
- Add network handler thread utilization to request quota calculation
- Add documentation for request quotas
- Transient failure of AdminClientTest.testDeleteRecordsWithException
- Add producer per task support
- Add request/response classes for transactions (KIP-98)
- KeyValueStore.put may throw exception unrelated to the current put
  attempt
- Exactly-once upgrade notes
- Update Message Delivery Semantics section to take into account
  KIP-98
- convert kafka controller to a single-threaded event queue model
- Additional validation in validateMessagesAndAssignOffsets
- Reconsider default retries for idempotent producer
- add controller integration tests
- Load only batch header when rebuilding producer ID map
- Refactor AdminUtils to use zkUtils methods instad of
  zkUtils.zkClient
- remove preferred replica election state from ControllerContext
- Implement transaction index for KIP-98
- shouldInnerLeftJoin unit test fails
- TransactionCoordinator - Check inter broker protocol and message
  format and raise errors if incompatible
- TransactionCoordinator - Add ACL check for each request
- Change InterBrokerSendThread to use a Queue per broker
- WriteTxnMarkers and complete commit/abort on partition immigration
- Abort long running transactions
- Move coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry
- KafkaProducer's TransactionManager needs a review on synchronization
- Refactor TransactionCoordinator in-memory structure and error
  handling logic
- KIP-98 : broker side handling for the TxnOffsetCommitRequest
- Add a reference to AdminClient to docs/api.html
- TC should not accept empty string transactional id
- Transient failure:
  ControllerIntegrationTest.testPreferredReplicaLeaderElection
- Log connection termination during authentication
- Transient failure:
  ControllerIntegrationTest.testControllerMoveIncrementsControllerEpoch
- Transient failure: RequestQuotaTest.testResponseThrottleTime
- Avoid expensive initialization of producer state when upgrading
- Add Integration tests for transactional producer
- Autogenerate Consumer Fetcher metrics
- LogCleaner should be transaction-aware
- Handle topic deletion for ongoing transactions
- SaslScramSslEndToEndAuthorizationTest.testNoConsumeWithoutDescribeAclViaSubscribe
- TransactinoCoordinator does not bump epoch when aborting open
  transactions
- Consumer GroupCoordinator should continue to materialize committed
  offsets in offset order even for transactional offset commits
- Remove retention time from TxnOffsetCommit RPC
- Transaction index recovery does not snapshot properly
- Producer should drop queued sends when transaction is aborted
- Fix flaky test LogCleanerTest.testCommitMarkerRemoval
- move all partition and replica state transition rules into their
  states
- TransactionalId authorization should imply ProducerId authorization
- Producer should not send AbortTxn unless transaction has actually
  begun
- TransactionsBounceTest occasionally sees INVALID_TXN_STATE errors
- TransactionBounceTest occasionally fails due to partition errors
- KafkaConsumer.committed() should get latest committed offsets from
  the server
- Review and improve AdminClient Javadoc for the first release
  (KIP-117)
- Review and potentially tweak AdminClient API for the initial release
  (KIP-117)
- TransactionCoordinator must expire transactionalIds
- Protect concurrent access to the cached transaction status
- System tests for KIP-98 / transactions
- Transactions integration test: Use factory methods to keep track of
  open producers and consumers and close them all on tearDown
- Update clients and server code to make sure that epoch and sequence
  numbers wrap around
- AdminClient should not trigger auto creation of topics
- Fix authorization checks in AdminClient
- TC should handle UNSUPPORTED_FOR_MESSAGE_FORMAT in WriteTxnMarker
  response
- reset ControllerContext during resignation
- Update KIP-98 to reflect changes during implementation.
- Update produce/fetch throttle time metrics for any request throttle
- Resolve AddPartitions response error code inconsistency
- Transactions system test with hard broker bounces fails sporadically
- Add additional test cases for batch splitting to ensure
  idempotent/transactional metadata is preserved
- Broker clean bounce test puts the broker into a
  'CONCURRENT_TRANSACTIONS' state permanently
- Broker returns messages beyond "latest stable offset" to
  transactional consumer in read_committed mode
- StackOverFlow error in transaction coordinator
- Producer attempts to send transactional messages before adding
  partitions to transaction
- Fix regression in compressed message iteration affecting magic v0
  and v1
- Transactions: Concurrent transactional consumer reads aborted
  messages
- Last Stable Offset not returned in Fetch request
- Transactional producer cannot find coordinator when trying to abort
  transaction after error
- Transactional producer aborts batches incorrectly in abortable error
  state
- Producer IllegalStateException: Batch has already been completed
- Produce state lost if no snapshot retained
- TransactionalMessageCopier should be force killed on test shutdown
- UnsupportedOperationException in WriteTxnMarkers handler
- Consumer should use last offset from batch to set next fetch offset
- read current brokers from zookeeper upon processing broker change