Packages

class ScioContext extends TransformNameable

Main entry point for Scio functionality. A ScioContext represents a pipeline and can be used to create SCollections and distributed caches on that cluster.

Source
ScioContext.scala
Linear Supertypes
TransformNameable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScioContext
  2. TransformNameable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def binaryFile(path: String, reader: BinaryFileReader, compression: Compression = BinaryIO.ReadParam.DefaultCompression, emptyMatchTreatment: EmptyMatchTreatment = TextIO.ReadParam.DefaultEmptyMatchTreatment, suffix: String = BinaryIO.ReadParam.DefaultSuffix): SCollection[Array[Byte]]

    Get an SCollection of Array[Byte] from a binary file.

    Get an SCollection of Array[Byte] from a binary file.

    reader

    An instance of BinaryFileReader for the specific binary format used by the input file.

  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. def customInput[T, I >: PBegin <: PInput](name: String, transform: PTransform[I, PCollection[T]]): SCollection[T]

    Get an SCollection with a custom input transform.

    Get an SCollection with a custom input transform. The transform should have a unique name.

  8. def empty[T]()(implicit arg0: Coder[T]): SCollection[T]

    Form an empty SCollection.

  9. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  10. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  11. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  12. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  14. def initCounter(counters: Counter*): Seq[Counter]

    Initialize a given Counter metric.

  15. def initCounter(namespace: String, name: String): Counter

    Initialize a new Counter metric from namespace and name.

  16. def initCounter[T](name: String)(implicit arg0: ClassTag[T]): Counter

    Initialize a new Counter metric using T as namespace.

    Initialize a new Counter metric using T as namespace. Default is "com.spotify.scio.ScioMetrics" if T is not specified.

  17. def isClosed: Boolean

    Whether the context is closed.

  18. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  19. def isTest: Boolean

    Whether this is a test context.

  20. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  21. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  23. val options: PipelineOptions
  24. def optionsAs[T <: PipelineOptions](implicit arg0: ClassTag[T]): T

    Get PipelineOptions as a more specific sub-type.

  25. def parallelize[K, V](elems: Map[K, V])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Distribute a local Scala Map to form an SCollection.

  26. def parallelize[T](elems: Iterable[T])(implicit arg0: Coder[T]): SCollection[T]

    Distribute a local Scala Iterable to form an SCollection.

  27. def parallelizeTimestamped[T](elems: Iterable[T], timestamps: Iterable[Instant])(implicit arg0: Coder[T]): SCollection[T]

    Distribute a local Scala Iterable with timestamps to form an SCollection.

  28. def parallelizeTimestamped[T](elems: Iterable[(T, Instant)])(implicit arg0: Coder[T]): SCollection[T]

    Distribute a local Scala Iterable with timestamps to form an SCollection.

  29. def pipeline: Pipeline

    Underlying pipeline.

  30. def read[T](io: ScioIO[T] { type ReadP = Unit }): SCollection[T]
  31. def read[T](io: ScioIO[T])(params: ReadP): SCollection[T]

    Generic read method for all ScioIO[T] implementations, which will invoke the provided IO's com.spotify.scio.io.ScioIO[T]#readWithContext method along with read configurations passed in.

    Generic read method for all ScioIO[T] implementations, which will invoke the provided IO's com.spotify.scio.io.ScioIO[T]#readWithContext method along with read configurations passed in. The IO class can delegate test-specific behavior if necessary.

    io

    an implementation of ScioIO[T] trait

    params

    configurations need to pass to perform underline read implementation

  32. def run(): ScioExecutionContext

    Runs the underlying pipeline.

    Runs the underlying pipeline.

    Running closes the context and no further transformations can be applied to the pipeline once the context is closed.

    returns

    the ScioExecutionContext for the underlying job execution.

  33. def setAppName(name: String): Unit

    Set application name for the context.

  34. def setJobName(name: String): Unit

    Set job name for the context.

  35. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  36. def tempLocation: String
  37. def textFile(path: String, compression: Compression = TextIO.ReadParam.DefaultCompression, emptyMatchTreatment: EmptyMatchTreatment = TextIO.ReadParam.DefaultEmptyMatchTreatment, suffix: String = TextIO.ReadParam.DefaultSuffix): SCollection[String]

    Get an SCollection for a text file.

  38. def toString(): String
    Definition Classes
    AnyRef → Any
  39. def transform[U](name: String)(f: (ScioContext) => SCollection[U]): SCollection[U]
  40. def transform[U](f: (ScioContext) => SCollection[U]): SCollection[U]
  41. def unionAll[T](scs: => Iterable[SCollection[T]])(implicit arg0: Coder[T]): SCollection[T]

    Create a union of multiple SCollections.

    Create a union of multiple SCollections. Supports empty lists.

  42. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  43. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  44. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  45. def withName(name: String): ScioContext.this.type

    Set a custom name for the next transform to be applied.

    Set a custom name for the next transform to be applied.

    Definition Classes
    TransformNameable
  46. def wrap[T](p: PCollection[T]): SCollection[T]

    Wrap a PCollection.

Inherited from TransformNameable

Inherited from AnyRef

Inherited from Any

In-memory Collections

Input Sources

Other Members