The ScioContext associated with this PCollection.

The PCollection being wrapped internally.

Return the union of this SCollection and another one. Any identical elements will appear multiple times (use distinct to eliminate them).

Aggregate with MonoidAggregator. First each item T is mapped to A, then we reduce with a Monoid of A, then finally we present the results as U. This could be more powerful and better optimized in some cases.

Aggregate with Aggregator. First each item T is mapped to A, then we reduce with a Semigroup of A, then finally we present the results as U. This could be more powerful and better optimized in some cases.

Aggregate the elements using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of this SCollection, T. Thus, we need one operation for merging a T into an U and one operation for merging two U's. Both of these functions are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.

Apply a PTransform and wrap the output in an SCollection. This is a special case of applyTransform for transforms with KV output.

Apply a PTransform and wrap the output in an SCollection. This is a special case of applyTransform for transforms with KV output.

Apply a PTransform and wrap the output in an SCollection.

Convert this SCollection to a SideInput, mapping each window to an Iterable, to be used with withSideInputs.

The values of the Iterable for a window are not required to fit in memory, but they may also not be effectively cached. If it is known that every window fits in memory, and stronger caching is desired, use asListSideInput.

Convert this SCollection to a SideInput, mapping each window to a Seq, to be used with withSideInputs.

The resulting Seq is required to fit in memory.

Convert this SCollection to a SideInput, mapping each window to a Set[T], to be used with withSideInputs.

The resulting SideInput is a one element singleton which is a Set of all elements in the SCollection for the given window. The complete Set must fit in memory of the worker.

Batches elements for amortized processing. Elements are batched per-window and batches emitted in the window corresponding to its contents.

Batches are emitted even if the maximum size is not reached when bundle finishes or when there are too many live windows.

Batches elements for amortized processing. Elements are batched per-window and batches emitted in the window corresponding to its contents.

Batches are emitted even if the maximum size is not reached when bundle finishes or when there are too many live windows.

Batches elements for amortized processing. Elements are batched per-window and batches emitted in the window corresponding to its contents.

Batches are emitted even if the maximum size is not reached when bundle finishes or when there are too many live windows.

Generic function to combine the elements using a custom set of aggregation functions. Turns an SCollection[T] into a result of type SCollection[C], for a "combined type" C. Note that T and C can be different -- for example, one might combine an SCollection of type Int into an SCollection of type Seq[Int]. Users provide three functions:

• createCombiner, which turns a T into a C (e.g., creates a one-element list)
• mergeValue, to merge a T into a C (e.g., adds it to the end of a list)
• mergeCombiners, to combine two C's into a single one.

Both mergeValue and mergeCombiners are allowed to modify and return their first argument instead of creating a new U to avoid memory allocation.

Count the number of elements in the SCollection.

Returns a single valued SCollection with estimated distinct count. Correctness is depends on the ApproxDistinctCounter estimator.

Count approximate number of distinct elements in the SCollection.

Count approximate number of distinct elements in the SCollection.

Return the cross product with another SCollection by replicating that to all workers. The right side should be tiny and fit in memory.

Print content of an SCollection to out().

Returns a new SCollection with distinct elements using given function to obtain a representative value for each input element.

Fold with Monoid, which defines the associative function and "zero value" for T. This could be more powerful and better optimized in some cases.

Aggregate the elements using a given associative function and a neutral "zero value". The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object allocation; however, it should not modify t2.

Return an SCollection of grouped items. Each group consists of a key and a sequence of elements mapping to that key. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.

Note: This operation may be very expensive. If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairSCollectionFunctions.aggregateByKey or PairSCollectionFunctions.reduceByKey will provide much better performance.

Return an SCollection of grouped items. Each group consists of a key and a sequence of elements transformed into a value of type U. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.

It is equivalent to groupBy(key).mapValues(_.map(f)), but more efficient.

Return an SCollection of grouped items. Each group consists of a key and the result of an associative reduce function. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.

The associative function is performed locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce

Look up values in an SCollection[(T, V)] for each element T in this SCollection by replicating that to all workers. The right side should be tiny and fit in memory.

Partition this SCollection using T.## into n partitions

Return the intersection of this SCollection and another one. The output will not contain any duplicate elements, even if the input SCollections did.

Note that this method performs a shuffle internally.

Extract data from this SCollection as a closed Tap. The Tap will be available once the pipeline completes successfully. .materialize() must be called before the ScioContext is run, as its implementation modifies the current pipeline graph.

val closedTap = sc.parallelize(1 to 10).materialize
sc.run().waitUntilDone().tap(closedTap)

Return the max of this SCollection as defined by the implicit Ordering[T].

Return the mean of this SCollection as defined by the implicit Numeric[T].

Return the min of this SCollection as defined by the implicit Ordering[T].

Partition this SCollection into a pair of SCollections according to a predicate.

Partition this SCollection with the provided function.

Partition this SCollection into a map from possible key values to an SCollection of corresponding elements based on the provided function .

Compute the SCollection's data distribution using approximate N-tiles.

Randomly splits this SCollection into three parts. Note: 0 < weightA + weightB < 1

Randomly splits this SCollection into two parts.

Randomly splits this SCollection with the provided weights.

Reads each file, represented as a pattern, in this SCollection.

Reads each file, represented as a pattern, in this SCollection. Files are split into multiple offset ranges and read with the FileBasedSource.

Reads each file, represented as a pattern, in this SCollection.

Reads each file, represented as a pattern, in this SCollection.

Reads each file, represented as a pattern, in this SCollection.

Reads each file, represented as a pattern, in this SCollection.

Reads each file, represented as a pattern, in this SCollection. Files are split into multiple offset ranges and read with the FileBasedSource.

Reads each file, represented as a pattern, in this SCollection.

Reads each file, represented as a pattern, in this SCollection. Files are split into multiple offset ranges and read with the FileBasedSource.

Pairs each element with the value of the provided SideInput in the element's window.

Reify as List:

val other: SCollection[Int] = sc.parallelize(Seq(1))
val coll: SCollection[(Int, Seq[Int])] =
  sc.parallelize(Seq(1, 2))
    .reifySideInputAsValues(other.asListSideInput)

Reify as Iterable:

val other: SCollection[Int] = sc.parallelize(Seq(1))
val coll: SCollection[(Int, Iterable[Int])] =
  sc.parallelize(Seq(1, 2))
    .reifySideInputAsValues(other.asIterableSideInput)

Reify as Map:

val other: SCollection[(Int, Int)] = sc.parallelize(Seq((1, 1)))
val coll: SCollection[(Int, Map[Int, Int])] =
  sc.parallelize(Seq(1, 2))
    .reifySideInputAsValues(other.asMapSideInput)

Reify as Multimap:

val other: SCollection[(Int, Int)]  = sc.parallelize(Seq((1, 1)))
val coll: SCollection[(Int, Map[Int, Iterable[Int]])]  =
  sc.parallelize(Seq(1, 2))
    .reifySideInputAsValues(other.asMultiMapSideInput)

Return a sampled subset of this SCollection. Does not trigger shuffling.

Return a sampled subset of this SCollection containing exactly sampleSize items. Involves combine operation resulting in shuffling. All the elements of the output should fit into main memory of a single worker machine.

Save this SCollection as raw bytes. Note that elements must be of type Array[Byte].

Save this SCollection with a custom output transform. The transform should have a unique name.

Save this SCollection as a text file. Note that elements must be of type String.

Creates a Zstd dictionary based on this SCollection targeting a dictionary of size zstdDictSizeBytes to be trained with approximately trainingBytesTarget bytes. The exact training size is determined by estimating the average element size with numElementsForSizeEstimation encoded elements and sampling this SCollection at an appropriate rate.

Reduce with Semigroup. This could be more powerful and better optimized than reduce in some cases.

Assign timestamps to values. With a optional skew

Return the top k (largest) elements from this SCollection as defined by the specified implicit Ordering[T].

Return the union of this SCollection and another one. Any identical elements will appear multiple times (use distinct to eliminate them).

Convert this SCollection to an SCollectionWithFanout that uses an intermediate node to combine parts of the data to reduce load on the final global combine step.

Set a custom name for the next transform to be applied.

Convert this SCollection to an SCollectionWithSideInput with one or more SideInput s, similar to Spark broadcast variables. Call SCollectionWithSideInput.toSCollection when done with side inputs.

val s1: SCollection[Int] = // ...
val s2: SCollection[String] = // ...
val s3: SCollection[(String, Double)] = // ...

// Prepare side inputs
val side1 = s1.asSingletonSideInput
val side2 = s2.asIterableSideInput
val side3 = s3.asMapSideInput
val side4 = s4.asMultiMapSideInput

val p: SCollection[MyRecord] = // ...
p.withSideInputs(side1, side2, side3).map { (x, s) =>
  // Extract side inputs from context
  val s1: Int = s(side1)
  val s2: Iterable[String] = s(side2)
  val s3: Map[String, Double] = s(side3)
  val s4: Map[String, Iterable[Double]] = s(side4)
  // ...
}

Convert this SCollection to an SCollectionWithSideOutput with one or more SideOutput s, so that a single transform can write to multiple destinations.

// Prepare side inputs
val side1 = SideOutput[String]()
val side2 = SideOutput[Int]()

val p: SCollection[MyRecord] = // ...
p.withSideOutputs(side1, side2).map { (x, s) =>
  // Write to side outputs via context
  s.output(side1, "word").output(side2, 1)
  // ...
}

Convert values into pairs of (value, window).

Generic write method for all ScioIO[T] implementations, if it is test pipeline this will evaluate pre-registered output IO implementation which match for the passing ScioIO[T] implementation. if not this will invoke com.spotify.scio.io.ScioIO[T]#write method along with write configurations passed by.