Packages

package avro

Main package for Avro APIs. Import all.

import com.spotify.scio.avro._
Source
package.scala
Linear Supertypes
AvroCoders, CoderGrammar, Syntax, SCollectionSyntax, ScioContextSyntax, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. avro
  2. AvroCoders
  3. CoderGrammar
  4. Syntax
  5. SCollectionSyntax
  6. ScioContextSyntax
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Package Members

  1. package dynamic

    IO package for avro dynamic destinations.

    IO package for avro dynamic destinations. Import All.

    import com.spotify.scio.avro.dynamic._
  2. package io
  3. package schemas
  4. package syntax
  5. package types

Type Members

  1. sealed trait AvroIO[T <: IndexedRecord] extends ScioIO[T]
  2. final case class AvroMagnolifyTypedIO[T](path: String)(implicit evidence$5: AvroType[T], evidence$6: Coder[T]) extends ScioIO[T] with Product with Serializable
  3. final case class AvroTaps(self: Taps) extends Product with Serializable
  4. final case class AvroTypedIO[T <: HasAvroAnnotation](path: String)(implicit evidence$1: scala.reflect.api.JavaUniverse.TypeTag[T], evidence$2: Coder[T]) extends ScioIO[T] with Product with Serializable
  5. final case class GenericRecordIO(path: String, schema: Schema) extends AvroIO[GenericRecord] with Product with Serializable
  6. final case class GenericRecordParseIO[T](path: String, parseFn: (GenericRecord) => T)(implicit evidence$2: Coder[T]) extends ScioIO[T] with Product with Serializable

    Given a parseFn, read GenericRecord and apply a function mapping => T before producing output.

    Given a parseFn, read GenericRecord and apply a function mapping => T before producing output. This IO applies the function at the time of de-serializing Avro GenericRecords.

    This IO doesn't define write, and should not be used to write Avro GenericRecords.

  7. final case class GenericRecordTap(path: String, schema: Schema, params: ReadParam) extends Tap[GenericRecord] with Product with Serializable

    Tap for GenericRecord Avro files.

  8. final case class ObjectFileIO[T](path: String)(implicit evidence$1: Coder[T]) extends ScioIO[T] with Product with Serializable
  9. sealed trait ProtobufIO[T] extends ScioIO[T]
  10. final case class ProtobufObjectFileIO[T <: Message](path: String)(implicit evidence$1: ClassTag[T]) extends ProtobufIO[T] with Product with Serializable
  11. final case class ProtobufTypedObjectFileIO[T, U <: Message](path: String)(implicit evidence$2: Coder[T], evidence$3: ClassTag[U], pt: ProtobufType[T, U]) extends ProtobufIO[T] with Product with Serializable
  12. class ReflectRecordDatumFactory[T] extends AvroDatumFactory[T]

    AvroDatumFactory for reflect records, i.e.

    AvroDatumFactory for reflect records, i.e. classes avro maps to a schema by reflection.

    Like Beam's AvroDatumFactory.ReflectDatumFactory, with avro's BigDecimalConversion added when it exists (avro 1.11+). Since avro 1.12.1, ReflectData.get() maps java.math.BigDecimal to the big-decimal logical type, which can't be written or read without that conversion, so Beam's factory fails on schemas derived with it.

  13. final case class SpecificRecordIO[T <: SpecificRecord](path: String)(implicit evidence$1: ClassTag[T]) extends AvroIO[T] with Product with Serializable
  14. final case class SpecificRecordTap[T <: SpecificRecord](path: String, params: ReadParam[T])(implicit evidence$1: ClassTag[T]) extends Tap[T] with Product with Serializable

    Tap for SpecificRecord Avro files.

  15. type doc = avro.types.doc

    Annotation for Avro field and record documentation.

Value Members

  1. val AvroType: avro.types.AvroType.type

    Typed Avro annotations and converters.

  2. def aggregate[T](implicit arg0: Coder[T]): Coder[Iterable[T]]
    Definition Classes
    CoderGrammar
  3. def avroCoder[T <: IndexedRecord](factory: AvroDatumFactory[T], schema: Schema): Coder[T]
    Definition Classes
    AvroCoders
  4. implicit def avroFilesSCollectionOps[T](c: SCollection[T])(implicit ev: <:<[T, String]): FilesSCollectionOps
    Definition Classes
    SCollectionSyntax
  5. def avroGenericRecordCoder: Coder[GenericRecord]
    Definition Classes
    AvroCoders
  6. def avroGenericRecordCoder(schema: Schema): Coder[GenericRecord]

    Create a Coder for Avro GenericRecord given the schema of the GenericRecord.

    Create a Coder for Avro GenericRecord given the schema of the GenericRecord.

    schema

    AvroSchema for the Coder.

    returns

    Coder[GenericRecord]

    Definition Classes
    AvroCoders
  7. implicit def avroGenericRecordSCollectionOps(c: SCollection[GenericRecord]): GenericRecordSCollectionOps
    Definition Classes
    SCollectionSyntax
  8. implicit def avroObjectFileSCollectionOps[T](c: SCollection[T]): ObjectFileSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  9. implicit def avroProtobufSCollectionOps[T <: Message](c: SCollection[T]): ProtobufSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  10. def avroReflectCoder[T](implicit arg0: ClassTag[T]): Coder[T]

    Create a Coder for a class avro maps to a schema by reflection, e.g.

    Create a Coder for a class avro maps to a schema by reflection, e.g. a POJO. Supports java.math.BigDecimal fields and trusts the class and the classes of its fields on avro 1.12.1+, see ReflectRecordDatumFactory. Not implicit: classes without an implicit coder use the kryo fallback.

    Definition Classes
    AvroCoders
  11. implicit def avroScioContextOps(c: ScioContext): ScioContextOps
    Definition Classes
    ScioContextSyntax
  12. implicit def avroSpecificFixedCoder[T <: SpecificFixed](implicit arg0: ClassTag[T]): Coder[T]
    Definition Classes
    AvroCoders
  13. implicit def avroSpecificRecordCoder[T <: SpecificRecord](implicit arg0: ClassTag[T]): Coder[T]
    Definition Classes
    AvroCoders
  14. implicit def avroSpecificRecordSCollectionOps[T <: SpecificRecord](c: SCollection[T]): SpecificRecordSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  15. implicit def avroTypedAvroSCollectionOps[T <: HasAvroAnnotation](c: SCollection[T]): TypedAvroSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  16. def beam[T](beam: Coder[T]): Coder[T]
    Definition Classes
    CoderGrammar
  17. def disjunction[T, Id](typeName: String, coder: Map[Id, Coder[T]])(id: (T) => Id)(implicit arg0: Coder[Id]): Coder[T]
    Definition Classes
    CoderGrammar
  18. def kryo[T](implicit ct: ClassTag[T]): Coder[T]

    Create an instance of Kryo Coder for a given Type.

    Create an instance of Kryo Coder for a given Type.

    Eg: A kryo Coder for org.joda.time.Interval would look like:

    implicit def jiKryo: Coder[Interval] = Coder.kryo[Interval]
    Definition Classes
    CoderGrammar
  19. def kv[K, V](koder: Coder[K], voder: Coder[V]): Coder[KV[K, V]]
    Definition Classes
    CoderGrammar
  20. def raw[T](beam: Coder[T]): Coder[T]

    Create a ScioCoder from a Beam Coder

    Create a ScioCoder from a Beam Coder

    Definition Classes
    CoderGrammar
  21. def transform[U, T](c: Coder[U])(f: (Coder[U]) => Coder[T])(implicit ct: ClassTag[T]): Coder[T]
    Definition Classes
    CoderGrammar
  22. implicit def typedAvroProtobufSCollectionOps[T](c: SCollection[T]): TypedMagnolifyProtobufSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  23. implicit def typedMagnolifyAvroSCollectionOps[T](c: SCollection[T]): TypedMagnolifyAvroSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  24. def xmap[U, T](c: Coder[U])(f: (U) => T, t: (T) => U)(implicit ct: ClassTag[T]): Coder[T]

    Given a Coder[A], create a Coder[B] by defining two functions A => B and B => A.

    Given a Coder[A], create a Coder[B] by defining two functions A => B and B => A. The Coder[A] can be resolved implicitly by calling Coder[A]

    Eg: Coder for org.joda.time.Interval can be defined by having the following implicit in scope. Without this implicit in scope Coder derivation falls back to Kryo.

    implicit def jiCoder: Coder[Interval] =
  Coder.xmap(Coder[(Long, Long)])(t => new Interval(t._1, t._2),
      i => (i.getStartMillis, i.getEndMillis))

    In the above example we implicitly derive Coder[(Long, Long)] and we define two functions, one to convert a tuple (Long, Long) to Interval, and a second one to convert an Interval to a tuple of (Long, Long)

    Definition Classes
    CoderGrammar
  25. object AvroCompat

    Compatibility with avro runtime versions newer than the one scio compiles against.

    Compatibility with avro runtime versions newer than the one scio compiles against. Avro APIs that don't exist in all supported versions are accessed by reflection, and skipped when missing.

  26. object AvroIO
  27. object AvroMagnolifyTypedIO extends Serializable
  28. object AvroSysProps extends SysProps
  29. object AvroTypedIO extends Serializable
  30. object GenericRecordIO extends Serializable
  31. object GenericRecordParseIO extends Serializable
  32. object ObjectFileIO extends Serializable
  33. object ObjectFileTap
  34. object ProtobufFileTap
  35. object ProtobufIO
  36. object ProtobufObjectFileIO extends Serializable
  37. object ProtobufTypedObjectFileIO extends Serializable
  38. object ReflectRecordDatumFactory extends Serializable
  39. object SpecificRecordIO extends Serializable

Deprecated Value Members

  1. object AvroTyped
    Annotations
    @deprecated
    Deprecated

    (Since version 0.14.0) Use AvroTypedIO instead

Inherited from AvroCoders

Inherited from CoderGrammar

Inherited from Syntax

Inherited from SCollectionSyntax

Inherited from ScioContextSyntax

Inherited from AnyRef

Inherited from Any

Ungrouped