org.apache.flume.sink
Class LoadBalancingSinkProcessor

java.lang.Object
  org.apache.flume.sink.AbstractSinkProcessor
      org.apache.flume.sink.LoadBalancingSinkProcessor

All Implemented Interfaces:

Configurable, LifecycleAware, SinkProcessor

public class LoadBalancingSinkProcessor
extends AbstractSinkProcessor

Provides the ability to load-balance flow over multiple sinks.

The LoadBalancingSinkProcessor maintains an indexed list of active sinks on which the load must be distributed. This implementation supports distributing load using either via ROUND_ROBIN or via RANDOM selection mechanism. The choice of selection mechanism defaults to ROUND_ROBIN type, but can be overridden via configuration.

When invoked, this selector picks the next sink using its configured selection mechanism and invokes it. In case the selected sink fails with an exception, the processor picks the next available sink via its configured selection mechanism. This implementation does not blacklist the failing sink and instead continues to optimistically attempt every available sink. If all sinks invocations result in failure, the selector propagates the failure to the sink runner.

Sample configuration:

  host1.sinkgroups.group1.sinks = sink1 sink2
  host1.sinkgroups.group1.processor.type = load_balance
  host1.sinkgroups.group1.processor.selector = 
  host1.sinkgroups.group1.processor.selector.selector_property = 
  

The value of processor.selector could be either round_robin for round-robin scheme of load-balancing or random for random selection. Alternatively you can specify your own implementation of the selection algorithm by implementing the LoadBalancingSelector interface. If no selector mechanism is specified, the round-robin selector is used by default.

This implementation is not thread safe at this time

See Also:

FailoverSinkProcessor, LoadBalancingSinkProcessor.SinkSelector

Nested Class Summary

static interface

LoadBalancingSinkProcessor.SinkSelector An interface that allows the LoadBalancingSinkProcessor to use a load-balancing strategy such as round-robin, random distribution etc.

Field Summary

static String	CONFIG_BACKOFF
static String	CONFIG_SELECTOR
static String	CONFIG_SELECTOR_PREFIX
static String	SELECTOR_NAME_RANDOM
static String	SELECTOR_NAME_RANDOM_BACKOFF
static String	SELECTOR_NAME_ROUND_ROBIN
static String	SELECTOR_NAME_ROUND_ROBIN_BACKOFF

Constructor Summary

LoadBalancingSinkProcessor()

Method Summary

void	configure(Context context) Request the implementing class to (re)configure itself.
Sink.Status	process() Handle a request to poll the owned sinks.
void	start() Starts a service or component.
void	stop() Stops a service or component.

Methods inherited from class org.apache.flume.sink.AbstractSinkProcessor

getLifecycleState, getSinks, setSinks

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

CONFIG_SELECTOR

public static final String CONFIG_SELECTOR

See Also:

Constant Field Values

CONFIG_SELECTOR_PREFIX

public static final String CONFIG_SELECTOR_PREFIX

See Also:

Constant Field Values

CONFIG_BACKOFF

public static final String CONFIG_BACKOFF

See Also:

Constant Field Values

SELECTOR_NAME_ROUND_ROBIN

public static final String SELECTOR_NAME_ROUND_ROBIN

See Also:

Constant Field Values

SELECTOR_NAME_RANDOM

public static final String SELECTOR_NAME_RANDOM

See Also:

Constant Field Values

SELECTOR_NAME_ROUND_ROBIN_BACKOFF

public static final String SELECTOR_NAME_ROUND_ROBIN_BACKOFF

See Also:

Constant Field Values

SELECTOR_NAME_RANDOM_BACKOFF

public static final String SELECTOR_NAME_RANDOM_BACKOFF

See Also:

Constant Field Values

Constructor Detail

LoadBalancingSinkProcessor

public LoadBalancingSinkProcessor()

Method Detail

configure

public void configure(Context context)

Description copied from interface: Configurable

Request the implementing class to (re)configure itself.

When configuration parameters are changed, they must be reflected by the component asap.

There are no thread safety guarrantees on when configure might be called.

start

public void start()

Description copied from interface: LifecycleAware

Starts a service or component.

Implementations should determine the result of any start logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

start in interface LifecycleAware

Overrides:

start in class AbstractSinkProcessor

stop

public void stop()

Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

stop in interface LifecycleAware

Overrides:

stop in class AbstractSinkProcessor

process

public Sink.Status process()
                    throws EventDeliveryException

Description copied from interface: SinkProcessor

Handle a request to poll the owned sinks.

The processor is expected to call Sink.process() on whatever sink(s) appropriate, handling failures as appropriate and throwing EventDeliveryException when there is a failure to deliver any events according to the delivery policy defined by the sink processor implementation. See specific implementations of this interface for delivery behavior and policies.

Returns:

Returns READY if events were successfully consumed, or BACKOFF if no events were available in the channel to consume.

Throws:

EventDeliveryException - if the behavior guaranteed by the processor couldn't be carried out.