Packages

final case class GenericRecordParseIO[T](path: String, parseFn: (GenericRecord) => T)(implicit evidence$2: Coder[T]) extends ScioIO[T] with Product with Serializable

Given a parseFn, read GenericRecord and apply a function mapping => T before producing output. This IO applies the function at the time of de-serializing Avro GenericRecords.

This IO doesn't define write, and should not be used to write Avro GenericRecords.

Source
AvroIO.scala
Linear Supertypes
Serializable, Product, Equals, ScioIO[T], AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. GenericRecordParseIO
  2. Serializable
  3. Product
  4. Equals
  5. ScioIO
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new GenericRecordParseIO(path: String, parseFn: (GenericRecord) => T)(implicit arg0: Coder[T])

Type Members

  1. type ReadP = ReadParam[GenericRecord]

    Read parameter type

    Read parameter type

    Definition Classes
    GenericRecordParseIO → ScioIO
  2. type WriteP = Nothing

    Write parameter type

    Write parameter type

    Definition Classes
    GenericRecordParseIO → ScioIO

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  8. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  9. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  10. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  11. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  12. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  13. val parseFn: (GenericRecord) => T
  14. val path: String
  15. def productElementNames: Iterator[String]
    Definition Classes
    Product
  16. def read(sc: ScioContext, params: ReadP): SCollection[T]

    Get an SCollection[T] by applying the parseFn on GenericRecord from an Avro file.

    Get an SCollection[T] by applying the parseFn on GenericRecord from an Avro file.

    Attributes
    protected
    Definition Classes
    GenericRecordParseIO → ScioIO
  17. def readTest(sc: ScioContext, params: ReadP): SCollection[T]

    Called only in a JobTest.

    Called only in a JobTest. Return test data for this testId as an SCollection

    Attributes
    protected
    Definition Classes
    ScioIO
  18. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  19. def tap(read: ReadP): Tap[Nothing]

    Write options also return a ClosedTap.

    Write options also return a ClosedTap. Once the job completes you can open the Tap. Tap abstracts away the logic of reading the dataset directly as an Iterator[T] or re-opening it in another ScioContext. The Future is complete once the job finishes. This can be used to do light weight pipeline orchestration e.g. WordCountOrchestration.scala.

    Definition Classes
    GenericRecordParseIO → ScioIO
  20. val tapT: Aux[T, Nothing]

    Output tap type.

    Output tap type.

    This _must_ be a stable value (a val, not a def) in every implementation, otherwise the return type of write cannot be inferred.

    Definition Classes
    GenericRecordParseIO → ScioIO
  21. def testId: String

    Identifier for JobTest IO matching

    Identifier for JobTest IO matching

    Definition Classes
    GenericRecordParseIO → ScioIO
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  23. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  24. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  25. def write(data: SCollection[T], params: Nothing): Tap[Nothing]

    Writes are undefined for GenericRecordParseIO since it is used only for reading.

    Writes are undefined for GenericRecordParseIO since it is used only for reading.

    Attributes
    protected
    Definition Classes
    GenericRecordParseIO → ScioIO
  26. def writeTest(data: SCollection[T], params: WriteP): Tap[(tapT)#T]

    Called only in a JobTest.

    Called only in a JobTest. Write data to TestDataManager output and return the Tap type

    Attributes
    protected
    Definition Classes
    ScioIO

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from ScioIO[T]

Inherited from AnyRef

Inherited from Any

Ungrouped