Packages

package avro

Main package for Avro APIs. Import all.

import com.spotify.scio.avro._
Source
package.scala
Linear Supertypes
AvroCoders, CoderGrammar, Syntax, SCollectionSyntax, ScioContextSyntax, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. avro
  2. AvroCoders
  3. CoderGrammar
  4. Syntax
  5. SCollectionSyntax
  6. ScioContextSyntax
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Package Members

  1. package dynamic

    IO package for avro dynamic destinations.

    IO package for avro dynamic destinations. Import All.

    import com.spotify.scio.avro.dynamic._
  2. package io
  3. package schemas
  4. package syntax
  5. package types

Type Members

  1. sealed trait AvroIO[T <: IndexedRecord] extends ScioIO[T]
  2. final case class AvroTaps(self: Taps) extends Product with Serializable
  3. final case class AvroTypedIO[T <: HasAvroAnnotation](path: String)(implicit evidence$1: scala.reflect.api.JavaUniverse.TypeTag[T], evidence$2: Coder[T]) extends ScioIO[T] with Product with Serializable
  4. final case class GenericRecordIO(path: String, schema: Schema) extends AvroIO[GenericRecord] with Product with Serializable
  5. final case class GenericRecordParseIO[T](path: String, parseFn: (GenericRecord) => T)(implicit evidence$2: Coder[T]) extends ScioIO[T] with Product with Serializable

    Given a parseFn, read GenericRecord and apply a function mapping => T before producing output.

    Given a parseFn, read GenericRecord and apply a function mapping => T before producing output. This IO applies the function at the time of de-serializing Avro GenericRecords.

    This IO doesn't define write, and should not be used to write Avro GenericRecords.

  6. final case class GenericRecordTap(path: String, schema: Schema, params: ReadParam) extends Tap[GenericRecord] with Product with Serializable

    Tap for GenericRecord Avro files.

  7. final case class ObjectFileIO[T](path: String)(implicit evidence$1: Coder[T]) extends ScioIO[T] with Product with Serializable
  8. final case class ProtobufIO[T <: Message](path: String)(implicit evidence$1: ClassTag[T]) extends ScioIO[T] with Product with Serializable
  9. final case class SpecificRecordIO[T <: SpecificRecord](path: String)(implicit evidence$1: ClassTag[T]) extends AvroIO[T] with Product with Serializable
  10. final case class SpecificRecordTap[T <: SpecificRecord](path: String, params: ReadParam[T])(implicit evidence$1: ClassTag[T]) extends Tap[T] with Product with Serializable

    Tap for SpecificRecord Avro files.

  11. type doc = avro.types.doc

    Annotation for Avro field and record documentation.

Value Members

  1. val AvroType: avro.types.AvroType.type

    Typed Avro annotations and converters.

  2. def aggregate[T](implicit arg0: Coder[T]): Coder[Iterable[T]]
    Definition Classes
    CoderGrammar
  3. def avroCoder[T <: IndexedRecord](factory: AvroDatumFactory[T], schema: Schema): Coder[T]
    Definition Classes
    AvroCoders
  4. implicit def avroFilesSCollectionOps[T](c: SCollection[T])(implicit ev: <:<[T, String]): FilesSCollectionOps
    Definition Classes
    SCollectionSyntax
  5. def avroGenericRecordCoder: Coder[GenericRecord]
    Definition Classes
    AvroCoders
  6. def avroGenericRecordCoder(schema: Schema): Coder[GenericRecord]

    Create a Coder for Avro GenericRecord given the schema of the GenericRecord.

    Create a Coder for Avro GenericRecord given the schema of the GenericRecord.

    schema

    AvroSchema for the Coder.

    returns

    Coder[GenericRecord]

    Definition Classes
    AvroCoders
  7. implicit def avroGenericRecordSCollectionOps(c: SCollection[GenericRecord]): GenericRecordSCollectionOps
    Definition Classes
    SCollectionSyntax
  8. implicit def avroObjectFileSCollectionOps[T](c: SCollection[T]): ObjectFileSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  9. implicit def avroProtobufSCollectionOps[T <: Message](c: SCollection[T]): ProtobufSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  10. implicit def avroScioContextOps(c: ScioContext): ScioContextOps
    Definition Classes
    ScioContextSyntax
  11. implicit def avroSpecificFixedCoder[T <: SpecificFixed](implicit arg0: ClassTag[T]): Coder[T]
    Definition Classes
    AvroCoders
  12. implicit def avroSpecificRecordCoder[T <: SpecificRecord](implicit arg0: ClassTag[T]): Coder[T]
    Definition Classes
    AvroCoders
  13. implicit def avroSpecificRecordSCollectionOps[T <: SpecificRecord](c: SCollection[T]): SpecificRecordSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  14. implicit def avroTypedAvroSCollectionOps[T <: HasAvroAnnotation](c: SCollection[T]): TypedAvroSCollectionOps[T]
    Definition Classes
    SCollectionSyntax
  15. def beam[T](beam: Coder[T]): Coder[T]
    Definition Classes
    CoderGrammar
  16. def disjunction[T, Id](typeName: String, coder: Map[Id, Coder[T]])(id: (T) => Id)(implicit arg0: Coder[Id]): Coder[T]
    Definition Classes
    CoderGrammar
  17. def kryo[T](implicit ct: ClassTag[T]): Coder[T]

    Create an instance of Kryo Coder for a given Type.

    Create an instance of Kryo Coder for a given Type.

    Eg: A kryo Coder for org.joda.time.Interval would look like:

    implicit def jiKryo: Coder[Interval] = Coder.kryo[Interval]
    Definition Classes
    CoderGrammar
  18. def kv[K, V](koder: Coder[K], voder: Coder[V]): Coder[KV[K, V]]
    Definition Classes
    CoderGrammar
  19. def raw[T](beam: Coder[T]): Coder[T]

    Create a ScioCoder from a Beam Coder

    Create a ScioCoder from a Beam Coder

    Definition Classes
    CoderGrammar
  20. def transform[U, T](c: Coder[U])(f: (Coder[U]) => Coder[T])(implicit ct: ClassTag[T]): Coder[T]
    Definition Classes
    CoderGrammar
  21. def xmap[U, T](c: Coder[U])(f: (U) => T, t: (T) => U)(implicit ct: ClassTag[T]): Coder[T]

    Given a Coder[A], create a Coder[B] by defining two functions A => B and B => A.

    Given a Coder[A], create a Coder[B] by defining two functions A => B and B => A. The Coder[A] can be resolved implicitly by calling Coder[A]

    Eg: Coder for org.joda.time.Interval can be defined by having the following implicit in scope. Without this implicit in scope Coder derivation falls back to Kryo.

    implicit def jiCoder: Coder[Interval] =
  Coder.xmap(Coder[(Long, Long)])(t => new Interval(t._1, t._2),
      i => (i.getStartMillis, i.getEndMillis))

    In the above example we implicitly derive Coder[(Long, Long)] and we define two functions, one to convert a tuple (Long, Long) to Interval, and a second one to convert an Interval to a tuple of (Long, Long)

    Definition Classes
    CoderGrammar
  22. object AvroIO
  23. object AvroSysProps extends SysProps
  24. object AvroTypedIO extends Serializable
  25. object GenericRecordIO extends Serializable
  26. object GenericRecordParseIO extends Serializable
  27. object ObjectFileIO extends Serializable
  28. object ObjectFileTap
  29. object ProtobufFileTap
  30. object ProtobufIO extends Serializable
  31. object SpecificRecordIO extends Serializable

Deprecated Value Members

  1. object AvroTyped
    Annotations
    @deprecated
    Deprecated

    (Since version 0.14.0) Use AvroTypedIO instead

Inherited from AvroCoders

Inherited from CoderGrammar

Inherited from Syntax

Inherited from SCollectionSyntax

Inherited from ScioContextSyntax

Inherited from AnyRef

Inherited from Any

Ungrouped