org.apache.flume.sink.hdfs
Class AvroEventSerializer

java.lang.Object
  extended by org.apache.flume.sink.hdfs.AvroEventSerializer
All Implemented Interfaces:
Configurable, EventSerializer

public class AvroEventSerializer
extends Object
implements EventSerializer, Configurable

This class serializes Flume events into Avro data files. The Flume event body is read as an Avro datum, and is then written to the EventSerializer's output stream in Avro data file format.

The Avro schema is determined by reading a Flume event header. The schema may be specified either as a literal, by setting AVRO_SCHEMA_LITERAL_HEADER (not recommended, since the full schema must be transmitted in every event), or as a URL which the schema may be read from, by setting AVRO_SCHEMA_URL_HEADER. Schemas read from URLs are cached by instances of this class so that the overhead of retrieval is minimized.


Nested Class Summary
static class AvroEventSerializer.Builder
           
 
Field Summary
static String AVRO_SCHEMA_LITERAL_HEADER
           
static String AVRO_SCHEMA_URL_HEADER
           
 
Fields inherited from interface org.apache.flume.serialization.EventSerializer
CTX_PREFIX
 
Method Summary
 void afterCreate()
          Hook to write a header after file is opened for the first time.
 void afterReopen()
          Hook to handle any framing needed when file is re-opened (for write).
Could have been named afterOpenForAppend().
 void beforeClose()
          Hook to write a trailer before the stream is closed.
 void configure(Context context)
           Request the implementing class to (re)configure itself.
 void flush()
          Hook to flush any internal write buffers to the underlying stream.
 boolean supportsReopen()
          Specify whether this output format supports reopening files for append.
 void write(Event event)
          Serialize and write the given event.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

AVRO_SCHEMA_LITERAL_HEADER

public static final String AVRO_SCHEMA_LITERAL_HEADER
See Also:
Constant Field Values

AVRO_SCHEMA_URL_HEADER

public static final String AVRO_SCHEMA_URL_HEADER
See Also:
Constant Field Values
Method Detail

configure

public void configure(Context context)
Description copied from interface: Configurable

Request the implementing class to (re)configure itself.

When configuration parameters are changed, they must be reflected by the component asap.

There are no thread safety guarrantees on when configure might be called.

Specified by:
configure in interface Configurable

afterCreate

public void afterCreate()
                 throws IOException
Description copied from interface: EventSerializer
Hook to write a header after file is opened for the first time.

Specified by:
afterCreate in interface EventSerializer
Throws:
IOException

afterReopen

public void afterReopen()
                 throws IOException
Description copied from interface: EventSerializer
Hook to handle any framing needed when file is re-opened (for write).
Could have been named afterOpenForAppend().

Specified by:
afterReopen in interface EventSerializer
Throws:
IOException

write

public void write(Event event)
           throws IOException
Description copied from interface: EventSerializer
Serialize and write the given event.

Specified by:
write in interface EventSerializer
Parameters:
event - Event to write to the underlying stream.
Throws:
IOException

flush

public void flush()
           throws IOException
Description copied from interface: EventSerializer
Hook to flush any internal write buffers to the underlying stream. It is NOT necessary for an implementation to then call flush() / sync() on the underlying stream itself, since those semantics would be provided by the driver that calls this API.

Specified by:
flush in interface EventSerializer
Throws:
IOException

beforeClose

public void beforeClose()
                 throws IOException
Description copied from interface: EventSerializer
Hook to write a trailer before the stream is closed. Implementations must not buffer data in this call since EventSerializer.flush() is not guaranteed to be called after beforeClose().

Specified by:
beforeClose in interface EventSerializer
Throws:
IOException

supportsReopen

public boolean supportsReopen()
Description copied from interface: EventSerializer
Specify whether this output format supports reopening files for append. For example, this method should return false if EventSerializer.beforeClose() writes a trailer that "finalizes" the file (this type of behavior is file format-specific).
Could have been named supportsAppend().

Specified by:
supportsReopen in interface EventSerializer


Copyright © 2009-2015 Apache Software Foundation. All Rights Reserved.