org.apache.flume.source.kafka
Class KafkaSource

java.lang.Object
  extended by org.apache.flume.source.AbstractSource
      extended by org.apache.flume.source.kafka.KafkaSource
All Implemented Interfaces:
Configurable, LifecycleAware, NamedComponent, PollableSource, Source

public class KafkaSource
extends AbstractSource
implements Configurable, PollableSource

A Source for Kafka which reads messages from a kafka topic. zookeeperConnect: Kafka's zookeeper connection string. Required

groupId: the group ID of consumer group. Required

topic: the topic to consume messages from. Required

maxBatchSize: Maximum number of messages written to Channel in one batch. Default: 1000

maxBatchDurationMillis: Maximum number of milliseconds before a batch (of any size) will be written to a channel. Default: 1000

kafka.auto.commit.enable: If true, commit automatically every time period. if false, commit on each batch. Default: false

kafka.consumer.timeout.ms: Polling interval for new data for batch. Low value means more CPU usage. High value means the time.upper.limit may be missed. Default: 10 Any property starting with "kafka" will be passed to the kafka consumer So you can use any configuration supported by Kafka 0.8.1.1


Nested Class Summary
 
Nested classes/interfaces inherited from interface org.apache.flume.PollableSource
PollableSource.Status
 
Constructor Summary
KafkaSource()
           
 
Method Summary
 void configure(Context context)
          We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1.
 PollableSource.Status process()
           Attempt to pull an item from the source, sending it to the channel.
 void start()
           Starts a service or component.
 void stop()
           Stops a service or component.
 
Methods inherited from class org.apache.flume.source.AbstractSource
getChannelProcessor, getLifecycleState, getName, setChannelProcessor, setName, toString
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 
Methods inherited from interface org.apache.flume.Source
getChannelProcessor, setChannelProcessor
 
Methods inherited from interface org.apache.flume.lifecycle.LifecycleAware
getLifecycleState
 
Methods inherited from interface org.apache.flume.NamedComponent
getName, setName
 

Constructor Detail

KafkaSource

public KafkaSource()
Method Detail

process

public PollableSource.Status process()
                              throws EventDeliveryException
Description copied from interface: PollableSource

Attempt to pull an item from the source, sending it to the channel.

When driven by an EventDrivenSourceRunner process is guaranteed to be called only by a single thread at a time, with no concurrency. Any other mechanism driving a pollable source must follow the same semantics.

Specified by:
process in interface PollableSource
Returns:
READY if one or more events were created from the source. BACKOFF if no events could be created from the source.
Throws:
EventDeliveryException - If there was a failure in delivering to the attached channel, or if a failure occurred in acquiring data from the source.

configure

public void configure(Context context)
We configure the source and generate properties for the Kafka Consumer Kafka Consumer properties are generated as follows: 1. Generate a properties object with some static defaults that can be overridden by Source configuration 2. We add the configuration users added for Kafka (parameters starting with kafka. and must be valid Kafka Consumer properties 3. We add the source documented parameters which can override other properties

Specified by:
configure in interface Configurable
Parameters:
context -

start

public void start()
Description copied from interface: LifecycleAware

Starts a service or component.

Implementations should determine the result of any start logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:
start in interface LifecycleAware
Overrides:
start in class AbstractSource

stop

public void stop()
Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:
stop in interface LifecycleAware
Overrides:
stop in class AbstractSource


Copyright © 2009-2015 Apache Software Foundation. All Rights Reserved.