|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.apache.flume.sink.AbstractSink
org.apache.flume.sink.kafka.KafkaSink
public class KafkaSink
A Flume Sink that can publish messages to Kafka. This is a general implementation that can be used with any Flume agent and a channel. The message can be any event and the key is a string that we read from the header For use of partitioning, use an interceptor to generate a header with the partition key
Mandatory properties are: brokerList -- can be a partial list, but at least 2 are recommended for HA however, any property starting with "kafka." will be passed along to the Kafka producer Read the Kafka producer documentation to see which configurations can be used Optional properties topic - there's a default, and also - this can be in the event header if you need to support events with different topics batchSize - how many messages to process in one batch. Larger batches improve throughput while adding latency. requiredAcks -- 0 (unsafe), 1 (accepted by at least one broker, default), -1 (accepted by all brokers) header properties (per event): topic key
Nested Class Summary |
---|
Nested classes/interfaces inherited from interface org.apache.flume.Sink |
---|
Sink.Status |
Field Summary | |
---|---|
static String |
KEY_HDR
|
static String |
TOPIC_HDR
|
Constructor Summary | |
---|---|
KafkaSink()
|
Method Summary | |
---|---|
void |
configure(Context context)
We configure the sink and generate properties for the Kafka Producer Kafka producer properties is generated as follows: 1. |
Sink.Status |
process()
Requests the sink to attempt to consume data from attached channel |
void |
start()
Starts a service or component. |
void |
stop()
Stops a service or component. |
Methods inherited from class org.apache.flume.sink.AbstractSink |
---|
getChannel, getLifecycleState, getName, setChannel, setName, toString |
Methods inherited from class java.lang.Object |
---|
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait |
Field Detail |
---|
public static final String KEY_HDR
public static final String TOPIC_HDR
Constructor Detail |
---|
public KafkaSink()
Method Detail |
---|
public Sink.Status process() throws EventDeliveryException
Sink
Requests the sink to attempt to consume data from attached channel
Note: This method should be consuming from the channel within the bounds of a Transaction. On successful delivery, the transaction should be committed, and on failure it should be rolled back.
process
in interface Sink
EventDeliveryException
- In case of any kind of failure to
deliver data to the next hop destination.public void start()
LifecycleAware
Starts a service or component.
Implementations should determine the result of any start logic and effect
the return value of LifecycleAware.getLifecycleState()
accordingly.
start
in interface LifecycleAware
start
in class AbstractSink
public void stop()
LifecycleAware
Stops a service or component.
Implementations should determine the result of any stop logic and effect
the return value of LifecycleAware.getLifecycleState()
accordingly.
stop
in interface LifecycleAware
stop
in class AbstractSink
public void configure(Context context)
configure
in interface Configurable
context
-
|
||||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |