org.apache.flume.source.kafka
Class KafkaSource

java.lang.Object
  org.apache.flume.source.AbstractSource
      org.apache.flume.source.kafka.KafkaSource

All Implemented Interfaces:

Configurable, LifecycleAware, NamedComponent, PollableSource, Source

public class KafkaSource
extends AbstractSource
implements Configurable, PollableSource

A Source for Kafka which reads messages from a kafka topic. zookeeperConnect: Kafka's zookeeper connection string. Required

groupId: the group ID of consumer group. Required

topic: the topic to consume messages from. Required

maxBatchSize: Maximum number of messages written to Channel in one batch. Default: 1000

maxBatchDurationMillis: Maximum number of milliseconds before a batch (of any size) will be written to a channel. Default: 1000

kafka.auto.commit.enable: If true, commit automatically every time period. if false, commit on each batch. Default: false

kafka.consumer.timeout.ms: Polling interval for new data for batch. Low value means more CPU usage. High value means the time.upper.limit may be missed. Default: 10 Any property starting with "kafka" will be passed to the kafka consumer So you can use any configuration supported by Kafka 0.8.1.1

Nested Class Summary

Nested classes/interfaces inherited from interface org.apache.flume.PollableSource

PollableSource.Status

Constructor Summary

KafkaSource()

Method Summary

void	configure(Context context) We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1.
PollableSource.Status	process() Attempt to pull an item from the source, sending it to the channel.
void	start() Starts a service or component.
void	stop() Stops a service or component.

Methods inherited from class org.apache.flume.source.AbstractSource

getChannelProcessor, getLifecycleState, getName, setChannelProcessor, setName, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface org.apache.flume.Source

getChannelProcessor, setChannelProcessor

Methods inherited from interface org.apache.flume.lifecycle.LifecycleAware

getLifecycleState

Methods inherited from interface org.apache.flume.NamedComponent

getName, setName

Constructor Detail

KafkaSource

public KafkaSource()

Method Detail

process

public PollableSource.Status process()
                              throws EventDeliveryException

Description copied from interface: PollableSource

Attempt to pull an item from the source, sending it to the channel.

When driven by an EventDrivenSourceRunner process is guaranteed to be called only by a single thread at a time, with no concurrency. Any other mechanism driving a pollable source must follow the same semantics.

Specified by:

process in interface PollableSource

Returns:

READY if one or more events were created from the source. BACKOFF if no events could be created from the source.

Throws:

EventDeliveryException - If there was a failure in delivering to the attached channel, or if a failure occurred in acquiring data from the source.

configure

public void configure(Context context)

We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1. Generate a properties object with some static defaults that can be overridden by Source configuration 2. We add the configuration users added for Kafka (parameters starting with kafka. and must be valid Kafka Consumer properties 3. We add the source documented parameters which can override other properties

Specified by:

configure in interface Configurable

Parameters:

context -

start

public void start()

Description copied from interface: LifecycleAware

Starts a service or component.

Implementations should determine the result of any start logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

start in interface LifecycleAware

Overrides:

start in class AbstractSource

stop

public void stop()

Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

stop in interface LifecycleAware

Overrides:

stop in class AbstractSource