sealed trait SCollection[T] extends PCollectionWrapper[T]

A Scala wrapper for PCollection. Represents an immutable, partitioned collection of elements that can be operated on in parallel. This class contains the basic operations available on all SCollections, such as map, filter, and sum. In addition, PairSCollectionFunctions contains operations available only on SCollections of key-value pairs, such as groupByKey and join; DoubleSCollectionFunctions contains operations available only on SCollections of Doubles.

Self Type
SCollection[T]
Linear Supertypes
PCollectionWrapper[T], TransformNameable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SCollection
  2. PCollectionWrapper
  3. TransformNameable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Abstract Value Members

  1. abstract val context: ScioContext

    The ScioContext associated with this PCollection.

    The ScioContext associated with this PCollection.

    Definition Classes
    PCollectionWrapper
  2. abstract val internal: PCollection[T]

    The PCollection being wrapped internally.

    The PCollection being wrapped internally.

    Definition Classes
    PCollectionWrapper

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. def ++(that: SCollection[T]): SCollection[T]

    Return the union of this SCollection and another one.

    Return the union of this SCollection and another one. Any identical elements will appear multiple times (use distinct to eliminate them).

  4. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  5. def aggregate[A, U](aggregator: Aggregator[T, A, U])(implicit arg0: Coder[A], arg1: Coder[U], coder: Coder[T]): SCollection[U]

    Aggregate with Aggregator.

    Aggregate with Aggregator. First each item T is mapped to A, then we reduce with a Semigroup of A, then finally we present the results as U. This could be more powerful and better optimized in some cases.

  6. def aggregate[U](zeroValue: U)(seqOp: (U, T) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: Coder[U], coder: Coder[T]): SCollection[U]

    Aggregate the elements using given combine functions and a neutral "zero value".

    Aggregate the elements using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of this SCollection, T. Thus, we need one operation for merging a T into an U and one operation for merging two U's. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.

  7. def applyKvTransform[K, V](transform: PTransform[_ >: PCollection[T], PCollection[KV[K, V]]])(implicit koder: Coder[K], voder: Coder[V]): SCollection[KV[K, V]]

    Apply a PTransform and wrap the output in an SCollection.

    Apply a PTransform and wrap the output in an SCollection. This is a special case of applyTransform for transforms with KV output.

  8. def applyTransform[U](transform: PTransform[_ >: PCollection[T], PCollection[U]])(implicit arg0: Coder[U]): SCollection[U]

    Apply a PTransform and wrap the output in an SCollection.

  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def asIterableSideInput: SideInput[Iterable[T]]

    Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with withSideInputs.

    Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with withSideInputs.

    The values of the Iterable for a window are not required to fit in memory, but they may also not be effectively cached. If it is known that every window fits in memory, and stronger caching is desired, use asListSideInput.

  11. def asListSideInput: SideInput[Seq[T]]

    Convert this SCollection to a SideInput, mapping each window to a Seq, to be used with withSideInputs.

    Convert this SCollection to a SideInput, mapping each window to a Seq, to be used with withSideInputs.

    The resulting Seq is required to fit in memory.

  12. def asSingletonSideInput(defaultValue: T): SideInput[T]

    Convert this SCollection of a single value per window to a SideInput with a default value, to be used with withSideInputs.

  13. def asSingletonSideInput: SideInput[T]

    Convert this SCollection of a single value per window to a SideInput, to be used with withSideInputs.

  14. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  15. def collect[U](pfn: PartialFunction[T, U])(implicit arg0: Coder[U]): SCollection[U]

    Filter the elements for which the given PartialFunction is defined, and then map.

  16. def combine[C](createCombiner: (T) ⇒ C)(mergeValue: (C, T) ⇒ C)(mergeCombiners: (C, C) ⇒ C)(implicit arg0: Coder[C], coder: Coder[T]): SCollection[C]

    Generic function to combine the elements using a custom set of aggregation functions.

    Generic function to combine the elements using a custom set of aggregation functions. Turns an SCollection[T] into a result of type SCollection[C], for a "combined type" C. Note that T and C can be different -- for example, one might combine an SCollection of type Int into an SCollection of type Seq[Int]. Users provide three functions:

    - createCombiner, which turns a T into a C (e.g., creates a one-element list)

    - mergeValue, to merge a T into a C (e.g., adds it to the end of a list)

    - mergeCombiners, to combine two C's into a single one.

  17. def count: SCollection[Long]

    Count the number of elements in the SCollection.

    Count the number of elements in the SCollection.

    returns

    a new SCollection with the count

  18. def countApproxDistinct(maximumEstimationError: Double = 0.02): SCollection[Long]

    Count approximate number of distinct elements in the SCollection.

    Count approximate number of distinct elements in the SCollection.

    maximumEstimationError

    the maximum estimation error, which should be in the range [0.01, 0.5]

  19. def countApproxDistinct(sampleSize: Int): SCollection[Long]

    Count approximate number of distinct elements in the SCollection.

    Count approximate number of distinct elements in the SCollection.

    sampleSize

    the number of entries in the statistical sample; the higher this number, the more accurate the estimate will be; should be >= 16

  20. def countByValue(implicit coder: Coder[(T, Long)]): SCollection[(T, Long)]

    Count of each unique value in this SCollection as an SCollection of (value, count) pairs.

  21. def cross[U](that: SCollection[U])(implicit arg0: Coder[U], tcoder: Coder[T]): SCollection[(T, U)]

    Return the cross product with another SCollection by replicating that to all workers.

    Return the cross product with another SCollection by replicating that to all workers. The right side should be tiny and fit in memory.

  22. def debug(out: () ⇒ PrintStream = () => Console.out, prefix: String = "", enabled: Boolean = true): SCollection[T]

    Print content of an SCollection to out().

    Print content of an SCollection to out().

    out

    where to write the debug information. Default: stdout

    prefix

    prefix for each logged entry. Default: empty string

    enabled

    if debugging is enabled or not. Default: true. It can be useful to set this to sc.isTest to avoid debugging when running in production.

  23. def distinct: SCollection[T]

    Return a new SCollection containing the distinct elements in this SCollection.

  24. def distinctBy[U](f: (T) ⇒ U)(implicit toder: Coder[T], uoder: Coder[U]): SCollection[T]

    Returns a new SCollection with distinct elements using given function to obtain a representative value for each input element.

    Returns a new SCollection with distinct elements using given function to obtain a representative value for each input element.

    U

    The type of representative values used to dedup.

    f

    The function to use to get representative values.

  25. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  26. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  27. def filter(f: (T) ⇒ Boolean): SCollection[T]

    Return a new SCollection containing only the elements that satisfy a predicate.

  28. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  29. def flatMap[U](f: (T) ⇒ TraversableOnce[U])(implicit arg0: Coder[U]): SCollection[U]

    Return a new SCollection by first applying a function to all elements of this SCollection, and then flattening the results.

  30. def flatten[U](implicit ev: (T) ⇒ TraversableOnce[U], coder: Coder[U]): SCollection[U]

    Return a new SCollection[U] by flattening each element of an SCollection[Traversable[U]].

  31. def fold(implicit mon: Monoid[T], coder: Coder[T]): SCollection[T]

    Fold with Monoid, which defines the associative function and "zero value" for T.

    Fold with Monoid, which defines the associative function and "zero value" for T. This could be more powerful and better optimized in some cases.

  32. def fold(zeroValue: T)(op: (T, T) ⇒ T)(implicit coder: Coder[T]): SCollection[T]

    Aggregate the elements using a given associative function and a neutral "zero value".

    Aggregate the elements using a given associative function and a neutral "zero value". The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.

  33. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  34. def groupBy[K](f: (T) ⇒ K)(implicit kcoder: Coder[K], vcoder: Coder[T]): SCollection[(K, Iterable[T])]

    Return an SCollection of grouped items.

    Return an SCollection of grouped items. Each group consists of a key and a sequence of elements mapping to that key. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.

    Note: This operation may be very expensive. If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairSCollectionFunctions.aggregateByKey or PairSCollectionFunctions.reduceByKey will provide much better performance.

  35. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  36. def hashFilter(that: SideSet[T])(implicit coder: Coder[T]): SCollection[T]

    Return a new SCollection containing only the elements that also exist in the SideSet.

  37. def hashLookup[V](that: SCollection[(T, V)])(implicit arg0: Coder[V], coder: Coder[T]): SCollection[(T, Iterable[V])]

    Look up values in an SCollection[(T, V)] for each element T in this SCollection by replicating that to all workers.

    Look up values in an SCollection[(T, V)] for each element T in this SCollection by replicating that to all workers. The right side should be tiny and fit in memory.

  38. def hashPartition(numPartitions: Int): Seq[SCollection[T]]

    Partition this SCollection using Object.hashCode() into n partitions

    Partition this SCollection using Object.hashCode() into n partitions

    numPartitions

    number of output partitions

    returns

    partitioned SCollections in a Seq

  39. def intersection(that: SCollection[T])(implicit coder: Coder[T]): SCollection[T]

    Return the intersection of this SCollection and another one.

    Return the intersection of this SCollection and another one. The output will not contain any duplicate elements, even if the input SCollections did.

    Note that this method performs a shuffle internally.

  40. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  41. def keyBy[K](f: (T) ⇒ K)(implicit coder: Coder[(K, T)]): SCollection[(K, T)]

    Create tuples of the elements in this SCollection by applying f.

  42. def map[U](f: (T) ⇒ U)(implicit arg0: Coder[U]): SCollection[U]

    Return a new SCollection by applying a function to all elements of this SCollection.

  43. def materialize(implicit coder: Coder[T]): ClosedTap[T]

    Extract data from this SCollection as a Future.

    Extract data from this SCollection as a Future. The Future will be completed once the pipeline completes successfully.

  44. def max(ord: Ordering[T])(implicit coder: Coder[T]): SCollection[T]
  45. def max(implicit ord: Ordering[T], coder: Coder[T], dummy: DummyImplicit): SCollection[T]

    Return the max of this SCollection as defined by the implicit Ordering[T].

    Return the max of this SCollection as defined by the implicit Ordering[T].

    returns

    a new SCollection with the maximum element

  46. def mean(implicit ev: Numeric[T]): SCollection[Double]

    Return the mean of this SCollection as defined by the implicit Numeric[T].

    Return the mean of this SCollection as defined by the implicit Numeric[T].

    returns

    a new SCollection with the mean of elements

  47. def min(ord: Ordering[T])(implicit coder: Coder[T]): SCollection[T]
  48. def min(implicit ord: Ordering[T], coder: Coder[T], dummy: DummyImplicit): SCollection[T]

    Return the min of this SCollection as defined by the implicit Ordering[T].

    Return the min of this SCollection as defined by the implicit Ordering[T].

    returns

    a new SCollection with the minimum element

  49. def name: String

    A friendly name for this SCollection.

  50. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  51. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  52. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  53. def pApply[U](transform: PTransform[_ >: PCollection[T], PCollection[U]]): SCollection[U]
    Attributes
    protected
    Definition Classes
    PCollectionWrapper
  54. def partition(p: (T) ⇒ Boolean): (SCollection[T], SCollection[T])

    Partition this SCollection into a pair of SCollections according to a predicate.

    Partition this SCollection into a pair of SCollections according to a predicate.

    p

    predicate on which to partition

    returns

    a pair of SCollections: the first SCollection consists of all elements that satisfy the predicate p and the second consists of all element that do not.

  55. def partition(numPartitions: Int, f: (T) ⇒ Int): Seq[SCollection[T]]

    Partition this SCollection with the provided function.

    Partition this SCollection with the provided function.

    numPartitions

    number of output partitions

    f

    function that assigns an output partition to each element, should be in the range [0, numPartitions - 1]

    returns

    partitioned SCollections in a Seq

  56. def partitionByKey[U](partitionKeys: Set[U])(f: (T) ⇒ U)(implicit arg0: Coder[U]): Map[U, SCollection[T]]

    Partition this SCollection into a map from possible key values to an SCollection of corresponding elements based on the provided function .

    Partition this SCollection into a map from possible key values to an SCollection of corresponding elements based on the provided function .

    partitionKeys

    The keys for the output partitions

    f

    function that assigns an output partition to each element, should be in the range of partitionKeys

    returns

    partitioned SCollections in a Map

  57. def quantilesApprox(numQuantiles: Int, ord: Ordering[T])(implicit coder: Coder[T]): SCollection[Iterable[T]]
  58. def quantilesApprox(numQuantiles: Int)(implicit ord: Ordering[T], coder: Coder[T], dummy: DummyImplicit): SCollection[Iterable[T]]

    Compute the SCollection's data distribution using approximate N-tiles.

    Compute the SCollection's data distribution using approximate N-tiles.

    returns

    a new SCollection whose single value is an Iterable of the approximate N-tiles of the elements

  59. def randomSplit(weightA: Double, weightB: Double)(implicit coder: Coder[T], ct: ClassTag[T]): (SCollection[T], SCollection[T], SCollection[T])

    Randomly splits this SCollection into three parts.

    Randomly splits this SCollection into three parts. Note: 0 < weightA + weightB < 1

    weightA

    weight for first SCollection, should be in the range (0, 1)

    weightB

    weight for second SCollection, should be in the range (0, 1)

    returns

    split SCollections in a Tuple3

  60. def randomSplit(weight: Double)(implicit coder: Coder[T], ct: ClassTag[T]): (SCollection[T], SCollection[T])

    Randomly splits this SCollection into two parts.

    Randomly splits this SCollection into two parts.

    weight

    weight for left hand side SCollection, should be in the range (0, 1)

    returns

    split SCollections in a Tuple2

  61. def randomSplit(weights: Array[Double])(implicit coder: Coder[T], ct: ClassTag[T]): Array[SCollection[T]]

    Randomly splits this SCollection with the provided weights.

    Randomly splits this SCollection with the provided weights.

    weights

    weights for splits, will be normalized if they don't sum to 1

    returns

    split SCollections in an array

  62. def readAll[U](read: PTransform[PCollection[String], PCollection[U]])(implicit arg0: Coder[U], ev: <:<[T, String]): SCollection[U]

    Read files represented by elements of this SCollection as file patterns.

    Read files represented by elements of this SCollection as file patterns.

    sc.parallelize("a.txt").readAll(TextIO.readAll())
  63. def readAllBytes(implicit ev: <:<[T, String]): SCollection[Array[Byte]]

    Read files as byte arrays represented by elements of this SCollection as file patterns.

  64. def reduce(op: (T, T) ⇒ T)(implicit coder: Coder[T]): SCollection[T]

    Reduce the elements of this SCollection using the specified commutative and associative binary operator.

  65. def sample(withReplacement: Boolean, fraction: Double)(implicit coder: Coder[T]): SCollection[T]

    Return a sampled subset of this SCollection.

  66. def sample(sampleSize: Int)(implicit coder: Coder[T]): SCollection[Iterable[T]]

    Return a sampled subset of this SCollection.

    Return a sampled subset of this SCollection.

    returns

    a new SCollection whose single value is an Iterable of the samples

  67. def saveAsBinaryFile(path: String, numShards: Int = ..., suffix: String = BinaryIO.WriteParam.DefaultSuffix, compression: Compression = ..., header: Array[Byte] = BinaryIO.WriteParam.DefaultHeader, footer: Array[Byte] = BinaryIO.WriteParam.DefaultFooter, framePrefix: (Array[Byte]) ⇒ Array[Byte] = ..., frameSuffix: (Array[Byte]) ⇒ Array[Byte] = ...)(implicit ev: <:<[T, Array[Byte]]): ClosedTap[Nothing]

    Save this SCollection as raw bytes.

    Save this SCollection as raw bytes. Note that elements must be of type Array[Byte].

  68. def saveAsCustomOutput[O <: POutput](name: String, transform: PTransform[PCollection[T], O]): ClosedTap[Nothing]

    Save this SCollection with a custom output transform.

    Save this SCollection with a custom output transform. The transform should have a unique name.

  69. def saveAsDatastore(projectId: String)(implicit ev: <:<[T, Entity]): ClosedTap[Nothing]

    Save this SCollection as a Datastore dataset.

    Save this SCollection as a Datastore dataset. Note that elements must be of type Entity.

  70. def saveAsPubsub(topic: String, idAttribute: String = null, timestampAttribute: String = null, maxBatchSize: Option[Int] = None, maxBatchBytesSize: Option[Int] = None)(implicit ct: ClassTag[T], coder: Coder[T]): ClosedTap[Nothing]

    Save this SCollection as a Pub/Sub topic.

  71. def saveAsPubsubWithAttributes[V](topic: String, idAttribute: String = null, timestampAttribute: String = null, maxBatchSize: Option[Int] = None, maxBatchBytesSize: Option[Int] = None)(implicit arg0: ClassTag[V], arg1: Coder[V], ev: <:<[T, (V, Map[String, String])]): ClosedTap[Nothing]

    Save this SCollection as a Pub/Sub topic using the given map as message attributes.

  72. def saveAsTextFile(path: String, numShards: Int = 0, suffix: String = ".txt", compression: Compression = Compression.UNCOMPRESSED)(implicit ct: ClassTag[T]): ClosedTap[String]

    Save this SCollection as a text file.

    Save this SCollection as a text file. Note that elements must be of type String.

  73. def setCoder(coder: Coder[T]): SCollection[T]

    Assign a Coder to this SCollection.

  74. def setSchema(schema: Schema[T]): SCollection[T]
  75. def subtract(that: SCollection[T])(implicit coder: Coder[T]): SCollection[T]

    Return an SCollection with the elements from this that are not in other.

  76. def sum(implicit sg: Semigroup[T], coder: Coder[T]): SCollection[T]

    Reduce with Semigroup.

    Reduce with Semigroup. This could be more powerful and better optimized than reduce in some cases.

  77. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  78. def take(num: Long): SCollection[T]

    Return a sampled subset of any num elements of the SCollection.

  79. def timestampBy(f: (T) ⇒ Instant, allowedTimestampSkew: Duration = Duration.ZERO)(implicit coder: Coder[T]): SCollection[T]

    Assign timestamps to values.

    Assign timestamps to values. With a optional skew

  80. def to[U](to: To[T, U]): SCollection[U]
  81. def toSideSet(implicit coder: Coder[T]): SideSet[T]
  82. def toString(): String
    Definition Classes
    AnyRef → Any
  83. def toWindowed(implicit coder: Coder[T]): WindowedSCollection[T]

    Convert this SCollection to an WindowedSCollection.

  84. def top(num: Int, ord: Ordering[T])(implicit coder: Coder[T]): SCollection[Iterable[T]]
  85. def top(num: Int)(implicit ord: Ordering[T], coder: Coder[T], d: DummyImplicit): SCollection[Iterable[T]]

    Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].

    Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].

    returns

    a new SCollection whose single value is an Iterable of the top k

  86. def transform[U](name: String)(f: (SCollection[T]) ⇒ SCollection[U]): SCollection[U]
    Annotations
    @experimental()
  87. def transform[U](f: (SCollection[T]) ⇒ SCollection[U]): SCollection[U]

    Apply a transform.

    Apply a transform.

    Annotations
    @experimental()
  88. def union(that: SCollection[T]): SCollection[T]

    Return the union of this SCollection and another one.

    Return the union of this SCollection and another one. Any identical elements will appear multiple times (use distinct to eliminate them).

  89. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  90. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  91. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  92. def windowByDays(number: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values into by days.

  93. def windowByMonths(number: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values into by months.

  94. def windowByWeeks(number: Int, startDayOfWeek: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values into by weeks.

  95. def windowByYears(number: Int, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values into by years.

  96. def withFanout(fanout: Int)(implicit coder: Coder[T]): SCollectionWithFanout[T]

    Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.

    Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.

    fanout

    the number of intermediate keys that will be used

  97. def withFixedWindows(duration: Duration, offset: Duration = Duration.ZERO, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values into fixed windows.

  98. def withGlobalWindow(options: WindowOptions = WindowOptions()): SCollection[T]

    Group values in to a single global window.

  99. def withName(name: String): SCollection.this.type

    Set a custom name for the next transform to be applied.

    Set a custom name for the next transform to be applied.

    Definition Classes
    TransformNameable
  100. def withPaneInfo(implicit coder: Coder[(T, PaneInfo)]): SCollection[(T, PaneInfo)]

    Convert values into pairs of (value, window).

  101. def withSessionWindows(gapDuration: Duration, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values based on sessions.

  102. def withSideInputs(sides: SideInput[_]*)(implicit coder: Coder[T]): SCollectionWithSideInput[T]

    Convert this SCollection to an SCollectionWithSideInput with one or more SideInputs, similar to Spark broadcast variables.

    Convert this SCollection to an SCollectionWithSideInput with one or more SideInputs, similar to Spark broadcast variables. Call SCollectionWithSideInput.toSCollection when done with side inputs.

    val s1: SCollection[Int] = // ...
    val s2: SCollection[String] = // ...
    val s3: SCollection[(String, Double)] = // ...
    
    // Prepare side inputs
    val side1 = s1.asSingletonSideInput
    val side2 = s2.asIterableSideInput
    val side3 = s3.asMapSideInput
    
    val p: SCollection[MyRecord] = // ...
    p.withSideInputs(side1, side2, side3).map { (x, s) =>
      // Extract side inputs from context
      val s1: Int = s(side1)
      val s2: Iterable[String] = s(side2)
      val s3: Map[String, Iterable[Double]] = s(side3)
      // ...
    }
  103. def withSideOutputs(sides: SideOutput[_]*): SCollectionWithSideOutput[T]

    Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutputs, so that a single transform can write to multiple destinations.

    Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutputs, so that a single transform can write to multiple destinations.

    // Prepare side inputs
    val side1 = SideOutput[String]()
    val side2 = SideOutput[Int]()
    
    val p: SCollection[MyRecord] = // ...
    p.withSideOutputs(side1, side2).map { (x, s) =>
      // Write to side outputs via context
      s.output(side1, "word").output(side2, 1)
      // ...
    }
  104. def withSlidingWindows(size: Duration, period: Duration = null, offset: Duration = Duration.ZERO, options: WindowOptions = WindowOptions()): SCollection[T]

    Window values into sliding windows.

  105. def withTimestamp(implicit coder: Coder[(T, Instant)]): SCollection[(T, Instant)]

    Convert values into pairs of (value, timestamp).

  106. def withWindow[W <: BoundedWindow](implicit tcoder: Coder[T]): SCollection[(T, W)]

    Convert values into pairs of (value, window).

    Convert values into pairs of (value, window).

    W

    window type, must be BoundedWindow or one of it's sub-types, e.g. GlobalWindow if this SCollection is not windowed or IntervalWindow if it is windowed.

  107. def withWindowFn[W <: BoundedWindow](fn: WindowFn[AnyRef, W], options: WindowOptions = WindowOptions()): SCollection[T]

    Window values with the given function.

  108. def write(io: ScioIO[T] { type WriteP = Unit })(implicit coder: Coder[T]): ClosedTap[T]
  109. def write(io: ScioIO[T])(params: WriteP)(implicit coder: Coder[T]): ClosedTap[T]

    Generic write method for all ScioIO[T] implementations, if it is test pipeline this will evaluate pre-registered output IO implementation which match for the passing ScioIO[T] implementation.

    Generic write method for all ScioIO[T] implementations, if it is test pipeline this will evaluate pre-registered output IO implementation which match for the passing ScioIO[T] implementation. if not this will invoke com.spotify.scio.io.ScioIO[T]#write method along with write configurations passed by.

    io

    an implementation of ScioIO[T] trait

    params

    configurations need to pass to perform underline write implementation

Inherited from PCollectionWrapper[T]

Inherited from TransformNameable

Inherited from AnyRef

Inherited from Any

Collection Operations

debug

Hash Operations

Output Sinks

Side Input and Output Operations

Transformations

Windowing Operations

Ungrouped