org.apache.flume.channel.recoverable.memory.wal
Class WAL<T extends org.apache.hadoop.io.Writable>

java.lang.Object
  org.apache.flume.channel.recoverable.memory.wal.WAL<T>

All Implemented Interfaces:

Closeable

public class WAL<T extends org.apache.hadoop.io.Writable>
extends Object
implements Closeable

Provides Write Ahead Log functionality for a generic Writable. All entries stored in the WAL must be assigned a unique increasing sequence id. WAL files will be removed when the following condition holds (defaults): At least 512MB of WAL's exist, the file in question is greater than five minutes old and the largest committed sequence id is greater than the largest sequence id in the file.

  WAL wal = new WAL(path, Writable.class);
  wal.writeEvent(event, 1);
  wal.writeEvent(event, 2);
  wal.writeSequenceID(1);
  wal.writeEvent(event, 3);

  System crashes or shuts down...

  WAL wal = new WAL(path, Writable.class);
  [Event 2, Event 3]  = wal.replay();
 

WAL files will be created in the specified data directory. They will be rolled at 64MB and deleted five minutes after they are no longer needed. that is the current sequence id) is greater than the greatest sequence id in the file. The only synchronization this class does is around rolling log files. When a roll of the log file is required, the thread which discovers this will execute the roll. Any threads calling a write*() method during the roll will block until the roll is complete.

Field Summary

static long	DEFAULT_MAX_LOGS_SIZE Number of bytes, to keep before we start pruning logs.
static long	DEFAULT_MIN_LOG_RETENTION_PERIOD Minimum number of ms to keep a log file.
static long	DEFAULT_ROLL_SIZE Number of bytes before we roll the file.
static long	DEFAULT_WORKER_INTERVAL How often in ms the background worker runs

Constructor Summary

WAL(File path, Class<T> clazz, long rollSize, long maxLogsSize, long minLogRentionPeriod, long workerInterval)
Creates a wal object with no defaults, using the specified parameters in the constructor for operation.

Method Summary

void	close()
static void	main(String[] args) Reads in a WAL and writes out a new WAL.
WALReplayResult<T>	replay()
void	setMaxLogsSize(long maxLogsSize)
void	setMinLogRetentionPeriod(long minLogRetentionPeriod)
void	setRollSize(long rollSize)
void	setWorkerInterval(long workerInterval)
void	writeEntries(List<WALEntry<T>> entries)
void	writeEntry(WALEntry<T> entry)
void	writeSequenceID(long sequenceID)
void	writeSequenceIDs(List<Long> sequenceIDs)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

DEFAULT_ROLL_SIZE

public static final long DEFAULT_ROLL_SIZE

Number of bytes before we roll the file.

See Also:

Constant Field Values

DEFAULT_MAX_LOGS_SIZE

public static final long DEFAULT_MAX_LOGS_SIZE

Number of bytes, to keep before we start pruning logs.

See Also:

Constant Field Values

DEFAULT_MIN_LOG_RETENTION_PERIOD

public static final long DEFAULT_MIN_LOG_RETENTION_PERIOD

Minimum number of ms to keep a log file.

See Also:

Constant Field Values

DEFAULT_WORKER_INTERVAL

public static final long DEFAULT_WORKER_INTERVAL

How often in ms the background worker runs

See Also:

Constant Field Values

Constructor Detail

WAL

public WAL(File path,
           Class<T> clazz,
           long rollSize,
           long maxLogsSize,
           long minLogRentionPeriod,
           long workerInterval)
    throws IOException

Creates a wal object with no defaults, using the specified parameters in the constructor for operation.

Parameters:

path -

clazz -

rollSize - bytes - max size of a single file before we roll

maxLogsSize - bytes - total amount of logs to keep excluding the current log

minLogRentionPeriod - ms - minimum amount of time to keep a log

workerInterval - ms - how often the background worker checks for old logs

Throws:

IOException

Method Detail

replay

public WALReplayResult<T> replay()
                                                                throws IOException

Throws:

IOException

writeEntries

public void writeEntries(List<WALEntry<T>> entries)
                  throws IOException

Throws:

IOException

writeEntry

public void writeEntry(WALEntry<T> entry)
                throws IOException

Throws:

IOException

writeSequenceID

public void writeSequenceID(long sequenceID)
                     throws IOException

Throws:

IOException

writeSequenceIDs

public void writeSequenceIDs(List<Long> sequenceIDs)
                      throws IOException

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface Closeable

Throws:

IOException

setRollSize

public void setRollSize(long rollSize)

setMaxLogsSize

public void setMaxLogsSize(long maxLogsSize)

setMinLogRetentionPeriod

public void setMinLogRetentionPeriod(long minLogRetentionPeriod)

setWorkerInterval

public void setWorkerInterval(long workerInterval)

main

public static void main(String[] args)
                 throws IOException,
                        ClassNotFoundException

Reads in a WAL and writes out a new WAL. Used if for some reason a replay cannot occur due to the size of the WAL or assumptions about the number of sequenceids.

Throws:

IOException

ClassNotFoundException