org.apache.flume.client.avro

Class ReliableSpoolingFileEventReader

java.lang.Object

org.apache.flume.client.avro.ReliableSpoolingFileEventReader

All Implemented Interfaces:

Closeable, AutoCloseable, EventReader, ReliableEventReader

@InterfaceAudience.Private
 @InterfaceStability.Evolving
public class ReliableSpoolingFileEventReader
extends Object
implements ReliableEventReader

A ReliableEventReader which reads log data from files stored in a spooling directory and renames each file once all of its data has been read (through EventDeserializer.readEvent() calls). The user must commit() each read, to indicate that the lines have been fully processed.

Read calls will return no data if there are no files left to read. This class, in general, is not thread safe.

This reader assumes that files with unique file names are left in the spooling directory and not modified once they are placed there. Any user behavior which violates these assumptions, when detected, will result in a FlumeException being thrown.

This class makes the following guarantees, if above assumptions are met:

• Once a log file has been renamed with the completedSuffix, all of its records have been read through the EventDeserializer.readEvent() function and commit()ed at least once.
• All files in the spooling directory will eventually be opened and delivered to a readEvents(int) caller.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	ReliableSpoolingFileEventReader.Builder Special builder class for ReliableSpoolingFileEventReader
static class	ReliableSpoolingFileEventReader.TrackingPolicy

Method Summary

Modifier and Type	Method and Description
void	close() Clean-up any state associated with this reader.
void	commit() Commit the last lines which were read.
String	getLastFileRead() Return the filename which generated the data from the last successful readEvents(int) call.
Event	readEvent() Get the next line associated with the input stream.
List<Event>	readEvents(int numEvents) Get up to n lines associated with the input stream.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Method Detail

getLastFileRead

public String getLastFileRead()

Return the filename which generated the data from the last successful readEvents(int) call. Returns null if called before any file contents are read.

readEvent

public Event readEvent()
                throws IOException

Description copied from interface: EventReader

Get the next line associated with the input stream. If this returns null, the input underlying input source is considered finished. Note that this is allowed to block for indefinite amounts of time waiting to generate a new line.

Specified by:

readEvent in interface EventReader

Throws:

IOException

readEvents

public List<Event> readEvents(int numEvents)
                       throws IOException

Description copied from interface: EventReader

Get up to n lines associated with the input stream. If this returns less than n lines, the input underlying input source is considered finished. Note that this is allowed to block for indefinite amounts of time waiting to generate a new line.

Specified by:

readEvents in interface EventReader

Throws:

IOException

close

public void close()
           throws IOException

Description copied from interface: EventReader

Clean-up any state associated with this reader.

Specified by:

close in interface Closeable

Specified by:

close in interface AutoCloseable

Specified by:

close in interface EventReader

Throws:

IOException

commit

public void commit()
            throws IOException

Commit the last lines which were read.

Specified by:

commit in interface ReliableEventReader

Throws:

IOException