org.apache.flume.client.avro
Class SpoolingFileLineReader

java.lang.Object
  org.apache.flume.client.avro.SpoolingFileLineReader

All Implemented Interfaces:

LineReader

public class SpoolingFileLineReader
extends Object
implements LineReader

A LineReader which reads log data from files stored in a spooling directory and renames each file once all of its data has been read (through readLine() calls). The user must commit() each read, to indicate that the lines have been fully processed. Read calls will return no data if there are no files left to read. This class, in general, is not thread safe. This reader assumes that files with unique file names are left in the spooling directory and not modified once they are placed there. Any user behavior which violates these assumptions, when detected, will result in a FlumeException being thrown. This class makes the following guarantees, if above assumptions are met: 1) Once a log file has been renamed with the completedSuffix, all of its records have been read through the readLine() function and commit()'d exactly once. 2) All files in the spooling directory will eventually be opened and delivered to a readLine() caller. NOTE: This class is for internal Flume components, this is not an extension point for developers customizing Flume.

Field Summary

String

completedSuffix

Constructor Summary

SpoolingFileLineReader(File directory, String completedSuffix, int bufferMaxLines, int bufferMaxLineLength)
Create a SpoolingFileLineReader to watch the given directory.

Method Summary

void	close() Clean-up any state associated with this reader.
void	commit() Commit the last lines which were read.
String	getLastFileRead() Return the filename which generated the data from the last successful #readLines() call.
String	readLine() Get the next line associated with the input stream.
List<String>	readLines(int n) Get up to n lines associated with the input stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

completedSuffix

public String completedSuffix

Constructor Detail

SpoolingFileLineReader

public SpoolingFileLineReader(File directory,
                              String completedSuffix,
                              int bufferMaxLines,
                              int bufferMaxLineLength)

Create a SpoolingFileLineReader to watch the given directory. Lines are buffered between when they are read and when they are committed. The buffer has a fixed size. Its size is determined by (maxLinestoBuffer * bufferSizePerLine).

Parameters:

directory - The directory to watch

completedSuffix - The suffix to append to completed files

bufferMaxLines - The maximum number of lines to keep in a pre-commit buffer

bufferMaxLineLength - The maximum line length for lines in the pre-commit buffer, in characters

Method Detail

getLastFileRead

public String getLastFileRead()

Return the filename which generated the data from the last successful #readLines() call. Returns null if called before any file contents are read.

readLine

public String readLine()
                throws IOException

Description copied from interface: LineReader

Get the next line associated with the input stream. If this returns null, the input underlying input source is considered finished. Note that this is allowed to block for indefinite amounts of time waiting to generate a new line.

Specified by:

readLine in interface LineReader

Throws:

IOException

commit

public void commit()
            throws IOException

Commit the last lines which were read.

Throws:

IOException

readLines

public List<String> readLines(int n)
                       throws IOException

Description copied from interface: LineReader

Get up to n lines associated with the input stream. If this returns less than n lines, the input underlying input source is considered finished. Note that this is allowed to block for indefinite amounts of time waiting to generate a new line.

Specified by:

readLines in interface LineReader

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: LineReader

Clean-up any state associated with this reader.

Specified by:

close in interface LineReader

Throws:

IOException