org.apache.flume.sink
Class LoadBalancingSinkProcessor

java.lang.Object
  extended by org.apache.flume.sink.AbstractSinkProcessor
      extended by org.apache.flume.sink.LoadBalancingSinkProcessor
All Implemented Interfaces:
Configurable, LifecycleAware, SinkProcessor

public class LoadBalancingSinkProcessor
extends AbstractSinkProcessor

Provides the ability to load-balance flow over multiple sinks.

The LoadBalancingSinkProcessor maintains an indexed list of active sinks on which the load must be distributed. This implementation supports distributing load using either via ROUND_ROBIN or via RANDOM selection mechanism. The choice of selection mechanism defaults to ROUND_ROBIN type, but can be overridden via configuration.

When invoked, this selector picks the next sink using its configured selection mechanism and invokes it. In case the selected sink fails with an exception, the processor picks the next available sink via its configured selection mechanism. This implementation does not blacklist the failing sink and instead continues to optimistically attempt every available sink. If all sinks invocations result in failure, the selector propagates the failure to the sink runner.

Sample configuration:

  host1.sinkgroups.group1.sinks = sink1 sink2
  host1.sinkgroups.group1.processor.type = load_balance
  host1.sinkgroups.group1.processor.selector = 
  host1.sinkgroups.group1.processor.selector.selector_property = 
  
The value of processor.selector could be either round_robin for round-robin scheme of load-balancing or random for random selection. Alternatively you can specify your own implementation of the selection algorithm by implementing the LoadBalancingSelector interface. If no selector mechanism is specified, the round-robin selector is used by default.

This implementation is not thread safe at this time

See Also:
FailoverSinkProcessor, LoadBalancingSinkProcessor.SinkSelector

Nested Class Summary
static interface LoadBalancingSinkProcessor.SinkSelector
           An interface that allows the LoadBalancingSinkProcessor to use a load-balancing strategy such as round-robin, random distribution etc.
 
Field Summary
static String CONFIG_BACKOFF
           
static String CONFIG_SELECTOR
           
static String CONFIG_SELECTOR_PREFIX
           
static String SELECTOR_NAME_RANDOM
           
static String SELECTOR_NAME_RANDOM_BACKOFF
           
static String SELECTOR_NAME_ROUND_ROBIN
           
static String SELECTOR_NAME_ROUND_ROBIN_BACKOFF
           
 
Constructor Summary
LoadBalancingSinkProcessor()
           
 
Method Summary
 void configure(Context context)
           Request the implementing class to (re)configure itself.
 Sink.Status process()
          Handle a request to poll the owned sinks.
 void start()
           Starts a service or component.
 void stop()
           Stops a service or component.
 
Methods inherited from class org.apache.flume.sink.AbstractSinkProcessor
getLifecycleState, getSinks, setSinks
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

CONFIG_SELECTOR

public static final String CONFIG_SELECTOR
See Also:
Constant Field Values

CONFIG_SELECTOR_PREFIX

public static final String CONFIG_SELECTOR_PREFIX
See Also:
Constant Field Values

CONFIG_BACKOFF

public static final String CONFIG_BACKOFF
See Also:
Constant Field Values

SELECTOR_NAME_ROUND_ROBIN

public static final String SELECTOR_NAME_ROUND_ROBIN
See Also:
Constant Field Values

SELECTOR_NAME_RANDOM

public static final String SELECTOR_NAME_RANDOM
See Also:
Constant Field Values

SELECTOR_NAME_ROUND_ROBIN_BACKOFF

public static final String SELECTOR_NAME_ROUND_ROBIN_BACKOFF
See Also:
Constant Field Values

SELECTOR_NAME_RANDOM_BACKOFF

public static final String SELECTOR_NAME_RANDOM_BACKOFF
See Also:
Constant Field Values
Constructor Detail

LoadBalancingSinkProcessor

public LoadBalancingSinkProcessor()
Method Detail

configure

public void configure(Context context)
Description copied from interface: Configurable

Request the implementing class to (re)configure itself.

When configuration parameters are changed, they must be reflected by the component asap.

There are no thread safety guarrantees on when configure might be called.


start

public void start()
Description copied from interface: LifecycleAware

Starts a service or component.

Implementations should determine the result of any start logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:
start in interface LifecycleAware
Overrides:
start in class AbstractSinkProcessor

stop

public void stop()
Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:
stop in interface LifecycleAware
Overrides:
stop in class AbstractSinkProcessor

process

public Sink.Status process()
                    throws EventDeliveryException
Description copied from interface: SinkProcessor

Handle a request to poll the owned sinks.

The processor is expected to call Sink.process() on whatever sink(s) appropriate, handling failures as appropriate and throwing EventDeliveryException when there is a failure to deliver any events according to the delivery policy defined by the sink processor implementation. See specific implementations of this interface for delivery behavior and policies.

Returns:
Returns READY if events were successfully consumed, or BACKOFF if no events were available in the channel to consume.
Throws:
EventDeliveryException - if the behavior guaranteed by the processor couldn't be carried out.


Copyright © 2009-2012 Apache Software Foundation. All Rights Reserved.