Packages

class ScioContext extends TransformNameable

Main entry point for Scio functionality. A ScioContext represents a pipeline and can be used to create SCollections and distributed caches on that cluster.

Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScioContext
  2. TransformNameable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. def customInput[T, I >: PBegin <: PInput](name: String, transform: PTransform[I, PCollection[T]]): SCollection[T]

    Get an SCollection with a custom input transform.

    Get an SCollection with a custom input transform. The transform should have a unique name.

  7. def empty[T]()(implicit arg0: Coder[T]): SCollection[T]

    Form an empty SCollection.

  8. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  9. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  10. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  11. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. def initCounter(counters: Counter*): Seq[Counter]

    Initialize a given Counter metric.

  14. def initCounter(namespace: String, name: String): Counter

    Initialize a new Counter metric from namespace and name.

  15. def initCounter[T](name: String)(implicit arg0: ClassTag[T]): Counter

    Initialize a new Counter metric using T as namespace.

    Initialize a new Counter metric using T as namespace. Default is "com.spotify.scio.ScioMetrics" if T is not specified.

  16. def isClosed: Boolean

    Whether the context is closed.

  17. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  18. def isTest: Boolean

    Whether this is a test context.

  19. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  20. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  21. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  22. val options: PipelineOptions
  23. def optionsAs[T <: PipelineOptions](implicit arg0: ClassTag[T]): T

    Get PipelineOptions as a more specific sub-type.

  24. def parallelize[K, V](elems: Map[K, V])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Distribute a local Scala Map to form an SCollection.

  25. def parallelize[T](elems: Iterable[T])(implicit arg0: Coder[T]): SCollection[T]

    Distribute a local Scala Iterable to form an SCollection.

  26. def parallelizeTimestamped[T](elems: Iterable[T], timestamps: Iterable[Instant])(implicit arg0: Coder[T]): SCollection[T]

    Distribute a local Scala Iterable with timestamps to form an SCollection.

  27. def parallelizeTimestamped[T](elems: Iterable[(T, Instant)])(implicit arg0: Coder[T]): SCollection[T]

    Distribute a local Scala Iterable with timestamps to form an SCollection.

  28. def pipeline: Pipeline

    Underlying pipeline.

  29. def read[T](io: ScioIO[T] { type ReadP = Unit }): SCollection[T]
  30. def read[T](io: ScioIO[T])(params: ReadP): SCollection[T]

    Generic read method for all ScioIO[T] implementations, which will invoke the provided IO's com.spotify.scio.io.ScioIO[T]#readWithContext method along with read configurations passed in.

    Generic read method for all ScioIO[T] implementations, which will invoke the provided IO's com.spotify.scio.io.ScioIO[T]#readWithContext method along with read configurations passed in. The IO class can delegate test-specific behavior if necessary.

    io

    an implementation of ScioIO[T] trait

    params

    configurations need to pass to perform underline read implementation

  31. def run(): ScioExecutionContext

    Runs the underlying pipeline.

    Runs the underlying pipeline.

    Running closes the context and no further transformations can be applied to the pipeline once the context is closed.

    returns

    the ScioExecutionContext for the underlying job execution.

  32. def setAppName(name: String): Unit

    Set application name for the context.

  33. def setJobName(name: String): Unit

    Set job name for the context.

  34. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  35. def tempLocation: String
  36. def textFile(path: String, compression: Compression = beam.Compression.AUTO): SCollection[String]

    Get an SCollection for a text file.

  37. def toString(): String
    Definition Classes
    AnyRef → Any
  38. def unionAll[T](scs: => Iterable[SCollection[T]])(implicit arg0: Coder[T]): SCollection[T]

    Create a union of multiple SCollections.

    Create a union of multiple SCollections. Supports empty lists.

  39. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  40. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  41. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()
  42. def withName(name: String): ScioContext.this.type

    Set a custom name for the next transform to be applied.

    Set a custom name for the next transform to be applied.

    Definition Classes
    TransformNameable
  43. def wrap[T](p: PCollection[T]): SCollection[T]

    Wrap a PCollection.

Inherited from TransformNameable

Inherited from AnyRef

Inherited from Any

In-memory Collections

Input Sources

Other Members