org.apache.flume.channel

Class SpillableMemoryChannel

java.lang.Object

org.apache.flume.channel.AbstractChannel

org.apache.flume.channel.BasicChannelSemantics

org.apache.flume.channel.file.FileChannel

org.apache.flume.channel.SpillableMemoryChannel

All Implemented Interfaces:

Channel, Configurable, LifecycleAware, NamedComponent

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class SpillableMemoryChannel
extends FileChannel

SpillableMemoryChannel will use main memory for buffering events until it has reached capacity. Thereafter file channel will be used as overflow.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	SpillableMemoryChannel.DrainOrderQueue

Field Summary

Fields

Modifier and Type	Field and Description
static String	AVG_EVENT_SIZE Estimated average size of events expected to be in the channel
static String	BYTE_CAPACITY max number of bytes used for all events in the queue.
static String	BYTE_CAPACITY_BUFFER_PERCENTAGE percent of buffer between byteCapacity and the estimated event size.
static String	CAPACITY file channel capacity overridden by Spillable Channel
protected ChannelCounter	channelCounter
static int	defaultMemoryCapacity
static int	defaultOverflowCapacity
static int	defaultOverflowDeactivationThreshold
static int	defaultOverflowTimeout
SpillableMemoryChannel.DrainOrderQueue	drainOrder
static String	KEEP_ALIVE file channel setting that is overriden by Spillable Channel
static String	MEMORY_CAPACITY Max number of events to be stored in memory
ArrayDeque<Event>	memQueue
static String	OVERFLOW_CAPACITY max number of events in overflow.
static String	OVERFLOW_DEACTIVATION_THRESHOLD Internal use only.
static String	OVERFLOW_TIMEOUT Seconds to wait before enabling disk overflow when memory fills up

Fields inherited from class org.apache.flume.channel.file.FileChannel

transactionCapacity

Constructor Summary

Constructors

Constructor and Description
SpillableMemoryChannel()

Method Summary

Modifier and Type	Method and Description
void	configure(Context context) Read parameters from context memoryCapacity = total number of events allowed at one time in the memory queue.
protected BasicTransactionSemantics	createTransaction() Called to create new Transaction objects, which must extend BasicTransactionSemantics.
int	getMaxMemQueueSize()
int	getMemoryCapacity()
protected Integer	getOverflowCapacity()
int	getOverflowTimeout()
protected int	getTotalStored()
protected boolean	isOverflowDisabled()
int	queueSize()
void	start() Starts a service or component.
void	stop() Stops a service or component.

Methods inherited from class org.apache.flume.channel.file.FileChannel

getDepth, isOpen, setName, toString

Methods inherited from class org.apache.flume.channel.BasicChannelSemantics

getTransaction, initialize, put, take

Methods inherited from class org.apache.flume.channel.AbstractChannel

getLifecycleState, getName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

MEMORY_CAPACITY

public static final String MEMORY_CAPACITY

Max number of events to be stored in memory

See Also:

Constant Field Values

OVERFLOW_TIMEOUT

public static final String OVERFLOW_TIMEOUT

Seconds to wait before enabling disk overflow when memory fills up

See Also:

Constant Field Values

OVERFLOW_DEACTIVATION_THRESHOLD

public static final String OVERFLOW_DEACTIVATION_THRESHOLD

Internal use only. To remain undocumented in User guide. Determines the percent free space available in mem queue when we stop spilling to overflow

See Also:

Constant Field Values

BYTE_CAPACITY_BUFFER_PERCENTAGE

public static final String BYTE_CAPACITY_BUFFER_PERCENTAGE

percent of buffer between byteCapacity and the estimated event size.

See Also:

Constant Field Values

BYTE_CAPACITY

public static final String BYTE_CAPACITY

max number of bytes used for all events in the queue.

See Also:

Constant Field Values

OVERFLOW_CAPACITY

public static final String OVERFLOW_CAPACITY

max number of events in overflow.

See Also:

Constant Field Values

KEEP_ALIVE

public static final String KEEP_ALIVE

file channel setting that is overriden by Spillable Channel

See Also:

Constant Field Values

CAPACITY

public static final String CAPACITY

file channel capacity overridden by Spillable Channel

See Also:

Constant Field Values

AVG_EVENT_SIZE

public static final String AVG_EVENT_SIZE

Estimated average size of events expected to be in the channel

See Also:

Constant Field Values

defaultMemoryCapacity

public static final int defaultMemoryCapacity

See Also:

Constant Field Values

defaultOverflowCapacity

public static final int defaultOverflowCapacity

See Also:

Constant Field Values

defaultOverflowTimeout

public static final int defaultOverflowTimeout

See Also:

Constant Field Values

defaultOverflowDeactivationThreshold

public static final int defaultOverflowDeactivationThreshold

See Also:

Constant Field Values

memQueue

public ArrayDeque<Event> memQueue

channelCounter

protected ChannelCounter channelCounter

drainOrder

public final SpillableMemoryChannel.DrainOrderQueue drainOrder

Constructor Detail

SpillableMemoryChannel

public SpillableMemoryChannel()

Method Detail

getTotalStored

protected int getTotalStored()

getMemoryCapacity

public int getMemoryCapacity()

getOverflowTimeout

public int getOverflowTimeout()

getMaxMemQueueSize

public int getMaxMemQueueSize()

getOverflowCapacity

protected Integer getOverflowCapacity()

isOverflowDisabled

protected boolean isOverflowDisabled()

queueSize

public int queueSize()

configure

public void configure(Context context)

Read parameters from context

memoryCapacity = total number of events allowed at one time in the memory queue. overflowCapacity = total number of events allowed at one time in the overflow file channel. byteCapacity = the max number of bytes used for events in the memory queue. byteCapacityBufferPercentage = type int. Defines the percent of buffer between byteCapacity and the estimated event size. overflowTimeout = type int. Number of seconds to wait on a full memory before deciding to enable overflow

Specified by:

configure in interface Configurable

Overrides:

configure in class FileChannel

start

public void start()

Description copied from interface: LifecycleAware

Starts a service or component.

Implementations should determine the result of any start logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

start in interface LifecycleAware

Overrides:

start in class FileChannel

stop

public void stop()

Description copied from interface: LifecycleAware

Stops a service or component.

Implementations should determine the result of any stop logic and effect the return value of LifecycleAware.getLifecycleState() accordingly.

Specified by:

stop in interface LifecycleAware

Overrides:

stop in class FileChannel

createTransaction

protected BasicTransactionSemantics createTransaction()

Description copied from class: BasicChannelSemantics

Called to create new Transaction objects, which must extend BasicTransactionSemantics. Each object is used for only one transaction, but is stored in a thread-local and retrieved by getTransaction for the duration of that transaction.

Overrides:

createTransaction in class FileChannel