org.apache.flume.channel.recoverable.memory.wal
Class WAL<T extends org.apache.hadoop.io.Writable>

java.lang.Object
  extended by org.apache.flume.channel.recoverable.memory.wal.WAL<T>
All Implemented Interfaces:
Closeable

public class WAL<T extends org.apache.hadoop.io.Writable>
extends Object
implements Closeable

Provides Write Ahead Log functionality for a generic Writable. All entries stored in the WAL must be assigned a unique increasing sequence id. WAL files will be removed when the following condition holds (defaults): At least 512MB of WAL's exist, the file in question is greater than five minutes old and the largest committed sequence id is greater than the largest sequence id in the file.

  WAL wal = new WAL(path, Writable.class);
  wal.writeEvent(event, 1);
  wal.writeEvent(event, 2);
  wal.writeSequenceID(1);
  wal.writeEvent(event, 3);

  System crashes or shuts down...

  WAL wal = new WAL(path, Writable.class);
  [Event 2, Event 3]  = wal.replay();
 
WAL files will be created in the specified data directory. They will be rolled at 64MB and deleted five minutes after they are no longer needed. that is the current sequence id) is greater than the greatest sequence id in the file. The only synchronization this class does is around rolling log files. When a roll of the log file is required, the thread which discovers this will execute the roll. Any threads calling a write*() method during the roll will block until the roll is complete.


Field Summary
static long DEFAULT_MAX_LOGS_SIZE
          Number of bytes, to keep before we start pruning logs.
static long DEFAULT_MIN_LOG_RETENTION_PERIOD
          Minimum number of ms to keep a log file.
static long DEFAULT_ROLL_SIZE
          Number of bytes before we roll the file.
static long DEFAULT_WORKER_INTERVAL
          How often in ms the background worker runs
 
Constructor Summary
WAL(File path, Class<T> clazz, long rollSize, long maxLogsSize, long minLogRentionPeriod, long workerInterval)
          Creates a wal object with no defaults, using the specified parameters in the constructor for operation.
 
Method Summary
 void close()
           
static void main(String[] args)
          Reads in a WAL and writes out a new WAL.
 WALReplayResult<T> replay()
           
 void setMaxLogsSize(long maxLogsSize)
           
 void setMinLogRetentionPeriod(long minLogRetentionPeriod)
           
 void setRollSize(long rollSize)
           
 void setWorkerInterval(long workerInterval)
           
 void writeEntries(List<WALEntry<T>> entries)
           
 void writeEntry(WALEntry<T> entry)
           
 void writeSequenceID(long sequenceID)
           
 void writeSequenceIDs(List<Long> sequenceIDs)
           
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

DEFAULT_ROLL_SIZE

public static final long DEFAULT_ROLL_SIZE
Number of bytes before we roll the file.

See Also:
Constant Field Values

DEFAULT_MAX_LOGS_SIZE

public static final long DEFAULT_MAX_LOGS_SIZE
Number of bytes, to keep before we start pruning logs.

See Also:
Constant Field Values

DEFAULT_MIN_LOG_RETENTION_PERIOD

public static final long DEFAULT_MIN_LOG_RETENTION_PERIOD
Minimum number of ms to keep a log file.

See Also:
Constant Field Values

DEFAULT_WORKER_INTERVAL

public static final long DEFAULT_WORKER_INTERVAL
How often in ms the background worker runs

See Also:
Constant Field Values
Constructor Detail

WAL

public WAL(File path,
           Class<T> clazz,
           long rollSize,
           long maxLogsSize,
           long minLogRentionPeriod,
           long workerInterval)
    throws IOException
Creates a wal object with no defaults, using the specified parameters in the constructor for operation.

Parameters:
path -
clazz -
rollSize - bytes - max size of a single file before we roll
maxLogsSize - bytes - total amount of logs to keep excluding the current log
minLogRentionPeriod - ms - minimum amount of time to keep a log
workerInterval - ms - how often the background worker checks for old logs
Throws:
IOException
Method Detail

replay

public WALReplayResult<T> replay()
                                                                throws IOException
Throws:
IOException

writeEntries

public void writeEntries(List<WALEntry<T>> entries)
                  throws IOException
Throws:
IOException

writeEntry

public void writeEntry(WALEntry<T> entry)
                throws IOException
Throws:
IOException

writeSequenceID

public void writeSequenceID(long sequenceID)
                     throws IOException
Throws:
IOException

writeSequenceIDs

public void writeSequenceIDs(List<Long> sequenceIDs)
                      throws IOException
Throws:
IOException

close

public void close()
           throws IOException
Specified by:
close in interface Closeable
Throws:
IOException

setRollSize

public void setRollSize(long rollSize)

setMaxLogsSize

public void setMaxLogsSize(long maxLogsSize)

setMinLogRetentionPeriod

public void setMinLogRetentionPeriod(long minLogRetentionPeriod)

setWorkerInterval

public void setWorkerInterval(long workerInterval)

main

public static void main(String[] args)
                 throws IOException,
                        ClassNotFoundException
Reads in a WAL and writes out a new WAL. Used if for some reason a replay cannot occur due to the size of the WAL or assumptions about the number of sequenceids.

Throws:
IOException
ClassNotFoundException


Copyright © 2009-2012 Apache Software Foundation. All Rights Reserved.