org.apache.flume.source.kafka

Class KafkaSource

java.lang.Object

org.apache.flume.source.BasicSourceSemantics

org.apache.flume.source.AbstractPollableSource

org.apache.flume.source.kafka.KafkaSource

All Implemented Interfaces:

BatchSizeSupported, Configurable, LifecycleAware, NamedComponent, PollableSource, Source

public class KafkaSource
extends AbstractPollableSource
implements Configurable, BatchSizeSupported

A Source for Kafka which reads messages from kafka topics. kafka.bootstrap.servers: A comma separated list of host:port pairs to use for establishing the initial connection to the Kafka cluster. For example host1:port1,host2:port2,... Required for kafka.

kafka.consumer.group.id: the group ID of consumer group. Required

kafka.topics: the topic list separated by commas to consume messages from. Required

maxBatchSize: Maximum number of messages written to Channel in one batch. Default: 1000

maxBatchDurationMillis: Maximum number of milliseconds before a batch (of any size) will be written to a channel. Default: 1000

kafka.consumer.*: Any property starting with "kafka.consumer" will be passed to the kafka consumer So you can use any configuration supported by Kafka 0.9.0.X useFlumeEventFormat: Reads events from Kafka Topic as an Avro FlumeEvent. Used in conjunction with useFlumeEventFormat (Kafka Sink) or parseAsFlumeEvent (Kafka Channel)

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	KafkaSource.Subscriber<T> This class is a helper to subscribe for topics by using different strategies

Nested classes/interfaces inherited from interface org.apache.flume.PollableSource

PollableSource.Status

Constructor Summary

Constructors

Constructor and Description
KafkaSource()

Method Summary

Modifier and Type	Method and Description
protected void	doConfigure(Context context) We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1.
protected PollableSource.Status	doProcess()
protected void	doStart()
protected void	doStop()
long	getBatchSize() Returns the batch size

Methods inherited from class org.apache.flume.source.AbstractPollableSource

configure, getBackOffSleepIncrement, getMaxBackOffSleepInterval, process

Methods inherited from class org.apache.flume.source.BasicSourceSemantics

getChannelProcessor, getLifecycleState, getName, getStartException, isStarted, setChannelProcessor, setLifecycleState, setName, start, stop, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flume.conf.Configurable

configure

Methods inherited from interface org.apache.flume.Source

getChannelProcessor, setChannelProcessor

Methods inherited from interface org.apache.flume.lifecycle.LifecycleAware

getLifecycleState, start, stop

Methods inherited from interface org.apache.flume.NamedComponent

getName, setName

Constructor Detail

KafkaSource

public KafkaSource()

Method Detail

getBatchSize

public long getBatchSize()

Description copied from interface: BatchSizeSupported

Returns the batch size

Specified by:

getBatchSize in interface BatchSizeSupported

doProcess

protected PollableSource.Status doProcess()
                                   throws EventDeliveryException

Specified by:

doProcess in class AbstractPollableSource

Throws:

EventDeliveryException

doConfigure

protected void doConfigure(Context context)
                    throws FlumeException

We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1. Generate a properties object with some static defaults that can be overridden if corresponding properties are specified 2. We add the configuration users added for Kafka (parameters starting with kafka.consumer and must be valid Kafka Consumer properties 3. Add source level properties (with no prefix)

Specified by:

doConfigure in class BasicSourceSemantics

Parameters:

context -

Throws:

FlumeException

doStart

protected void doStart()
                throws FlumeException

Specified by:

doStart in class BasicSourceSemantics

Throws:

FlumeException

doStop

protected void doStop()
               throws FlumeException

Specified by:

doStop in class BasicSourceSemantics

Throws:

FlumeException