Packages

trait ScioIO[T] extends AnyRef

Base trait for all Read/Write IO classes. Every IO connector must implement this. This trait has two abstract implicit methods #read, #write that need to be implemented in every subtype. Look at the com.spotify.scio.io.TextIO subclass for a reference implementation. IO connectors can choose to override #readTest and #writeTest if custom test logic is necessary.

Source
ScioIO.scala
Linear Supertypes
AnyRef, Any
Known Subclasses
AvroIO, AvroTypedIO, GenericRecordIO, GenericRecordParseIO, ObjectFileIO, ProtobufIO, SpecificRecordIO, BigQueryIO, BigQuerySelect, BigQueryStorage, BigQueryStorageSelect, Select, Storage, StorageQuery, Table, BigQueryTypedSelect, BigQueryTypedTable, TableRowJsonIO, BigtableIO, BigtableRead, BigtableWrite, CassandraIO, DatastoreIO, ElasticsearchIO, Read, ReadWrite, Write, JsonIO, BinaryIO, CustomIO, ReadIO, TestIO, TextIO, ZstdDictIO, JdbcIO, JdbcSelect, JdbcWrite, JdbcShardedSelect, Neo4jIO, ParquetAvroIO, ParquetExampleIO, ParquetTypeIO, PubsubIO, RedisIO, RedisRead, RedisWrite, SmbIO, SpannerIO, SpannerRead, SpannerWrite, TFExampleIO, TFRecordIO, TFSequenceExampleIO
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScioIO
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Type Members

  1. abstract type ReadP

    Read parameter type

  2. abstract type WriteP

    Write parameter type

Abstract Value Members

  1. abstract def read(sc: ScioContext, params: ReadP): SCollection[T]

    Read data according to the read configuration provided in params as an SCollection.

    Read data according to the read configuration provided in params as an SCollection.

    Attributes
    protected
  2. abstract def tap(read: ReadP): Tap[T]

    Write options also return a ClosedTap.

    Write options also return a ClosedTap. Once the job completes you can open the Tap. Tap abstracts away the logic of reading the dataset directly as an Iterator[T] or re-opening it in another ScioContext. The Future is complete once the job finishes. This can be used to do light weight pipeline orchestration e.g. WordCountOrchestration.scala.

  3. abstract val tapT: TapT[T]

    Output tap type.

    Output tap type.

    This _must_ be a stable value (a val, not a def) in every implementation, otherwise the return type of write cannot be inferred.

  4. abstract def write(data: SCollection[T], params: WriteP): Tap[T]

    Write data out according to write configuration provided in params, returning the Tap type.

    Write data out according to write configuration provided in params, returning the Tap type.

    Attributes
    protected

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. def readTest(sc: ScioContext, params: ReadP): SCollection[T]

    Called only in a JobTest.

    Called only in a JobTest. Return test data for this testId as an SCollection

    Attributes
    protected
  16. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  17. def testId: String

    Identifier for JobTest IO matching

  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  22. def writeTest(data: SCollection[T], params: WriteP): Tap[T]

    Called only in a JobTest.

    Called only in a JobTest. Write data to TestDataManager output and return the Tap type

    Attributes
    protected

Inherited from AnyRef

Inherited from Any

Ungrouped