package avro
Main package for Avro APIs. Import all.
import com.spotify.scio.avro._- Alphabetic
- By Inheritance
- avro
- Syntax
- SCollectionSyntax
- ScioContextSyntax
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Type Members
- sealed trait AvroIO[T] extends ScioIO[T]
- final case class AvroTaps(self: Taps) extends Product with Serializable
- final case class GenericRecordIO(path: String, schema: Schema) extends AvroIO[GenericRecord] with Product with Serializable
- final case class GenericRecordParseIO[T](path: String, parseFn: (GenericRecord) => T)(implicit coder: Coder[T]) extends AvroIO[T] with Product with Serializable
Given a parseFn, read GenericRecord and apply a function mapping => T before producing output.
Given a parseFn, read GenericRecord and apply a function mapping => T before producing output. This IO applies the function at the time of de-serializing Avro GenericRecords.
This IO doesn't define write, and should not be used to write Avro GenericRecords.
- final case class GenericRecordParseTap[T](path: String, parseFn: (GenericRecord) => T)(implicit evidence$3: Coder[T]) extends Tap[T] with Product with Serializable
Tap for reading GenericRecord Avro files and applying a parseFn to parse it to the given type T
- final case class GenericRecordTap(path: String, schema: Schema) extends Tap[GenericRecord] with Product with Serializable
Tap for GenericRecord Avro files.
- final case class ObjectFileIO[T](path: String)(implicit evidence$1: Coder[T]) extends ScioIO[T] with Product with Serializable
- case class ObjectFileTap[T](path: String)(implicit evidence$4: Coder[T]) extends Tap[T] with Product with Serializable
Tap for object files.
Tap for object files. Note that serialization is not guaranteed to be compatible across Scio releases.
- final case class ProtobufIO[T <: Message](path: String)(implicit evidence$2: ClassTag[T]) extends ScioIO[T] with Product with Serializable
- final case class SpecificRecordIO[T <: SpecificRecord](path: String)(implicit evidence$3: ClassTag[T], evidence$4: Coder[T]) extends AvroIO[T] with Product with Serializable
- final case class SpecificRecordTap[T <: SpecificRecord](path: String)(implicit evidence$1: ClassTag[T], evidence$2: Coder[T]) extends Tap[T] with Product with Serializable
Tap for SpecificRecord Avro files.
- type doc = avro.types.doc
Annotation for Avro field and record documentation.
Value Members
- val AvroType: avro.types.AvroType.type
Typed Avro annotations and converters.
- implicit def avroGenericRecordSCollectionOps(c: SCollection[GenericRecord]): GenericRecordSCollectionOps
- Definition Classes
- SCollectionSyntax
- implicit def avroObjectFileSCollectionOps[T](c: SCollection[T]): ObjectFileSCollectionOps[T]
- Definition Classes
- SCollectionSyntax
- implicit def avroProtobufSCollectionOps[T <: Message](c: SCollection[T]): ProtobufSCollectionOps[T]
- Definition Classes
- SCollectionSyntax
- implicit def avroScioContextOps(c: ScioContext): ScioContextOps
- Definition Classes
- ScioContextSyntax
- implicit def avroSpecificRecordSCollectionOps[T <: SpecificRecord](c: SCollection[T]): SpecificRecordSCollectionOps[T]
- Definition Classes
- SCollectionSyntax
- implicit def avroTypedAvroSCollectionOps[T <: HasAvroAnnotation](c: SCollection[T]): TypedAvroSCollectionOps[T]
- Definition Classes
- SCollectionSyntax
- object AvroIO
- object AvroSysProps
- Annotations
- @registerSysProps()
- object AvroTyped
- object ObjectFileIO extends Serializable
- object ProtobufIO extends Serializable