Packages

c

com.spotify.scio.smb.syntax

SortedBucketScioContext

final class SortedBucketScioContext extends Serializable

Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SortedBucketScioContext
  2. Serializable
  3. AnyRef
  4. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new SortedBucketScioContext(self: ScioContext)

Deprecated Type Members

  1. type SortMergeTransformReadBuilder[KeyType, K1, K2, R] = ReadBuilder[KeyType, K1, K2, R]
    Annotations
    @deprecated
    Deprecated

    (Since version 0.14.0) Use SortMergeTransform.ReadBuilder instead

  2. type SortMergeTransformWithSideInputsWriteBuilder[KeyType, R, W] = WithSideInputsWriteBuilder[KeyType, R, W]
    Annotations
    @deprecated
    Deprecated

    (Since version 0.14.0) Use SortMergeTransform.WithSideInputsWriteBuilder instead

  3. type SortMergeTransformWriteBuilder[KeyType, R, W] = WriteBuilder[KeyType, R, W]
    Annotations
    @deprecated
    Deprecated

    (Since version 0.14.0) Use SortMergeTransform.WriteBuilder instead

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  15. def sortMergeCoGroup[K1, K2, A, B, C, D](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], c: Read[C], d: Read[D])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B], arg4: Coder[C], arg5: Coder[D]): SCollection[((K1, K2), (Iterable[A], Iterable[B], Iterable[C], Iterable[D]))]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  16. def sortMergeCoGroup[K1, K2, A, B, C, D](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], c: Read[C], d: Read[D], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B], arg4: Coder[C], arg5: Coder[D]): SCollection[((K1, K2), (Iterable[A], Iterable[B], Iterable[C], Iterable[D]))]

    Secondary keyed variant

    Secondary keyed variant

    Annotations
    @experimental()
  17. def sortMergeCoGroup[K, A, B, C, D](keyClass: Class[K], a: Read[A], b: Read[B], c: Read[C], d: Read[D])(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B], arg3: Coder[C], arg4: Coder[D]): SCollection[(K, (Iterable[A], Iterable[B], Iterable[C], Iterable[D]))]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  18. def sortMergeCoGroup[K, A, B, C, D](keyClass: Class[K], a: Read[A], b: Read[B], c: Read[C], d: Read[D], targetParallelism: TargetParallelism)(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B], arg3: Coder[C], arg4: Coder[D]): SCollection[(K, (Iterable[A], Iterable[B], Iterable[C], Iterable[D]))]

    For each key K in a or b or c or d, return a resulting SCollection that contains a tuple with the list of values for that key in a, b, c and d.

    For each key K in a or b or c or d, return a resulting SCollection that contains a tuple with the list of values for that key in a, b, c and d.

    See note on SortedBucketScioContext.sortMergeJoin for information on how an SMB cogroup differs from a regular org.apache.beam.sdk.transforms.join.CoGroupByKey operation.

    keyClass

    cogroup key class. Must have a Coder in Beam's default org.apache.beam.sdk.coders.CoderRegistry as custom key coders are not supported yet.

    targetParallelism

    the desired parallelism of the job. See org.apache.beam.sdk.extensions.smb.TargetParallelism for more information.

    Annotations
    @experimental()
  19. def sortMergeCoGroup[K1, K2, A, B, C](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], c: Read[C])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B], arg4: Coder[C]): SCollection[((K1, K2), (Iterable[A], Iterable[B], Iterable[C]))]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  20. def sortMergeCoGroup[K1, K2, A, B, C](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], c: Read[C], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B], arg4: Coder[C]): SCollection[((K1, K2), (Iterable[A], Iterable[B], Iterable[C]))]

    Secondary keyed variant

    Secondary keyed variant

    Annotations
    @experimental()
  21. def sortMergeCoGroup[K, A, B, C](keyClass: Class[K], a: Read[A], b: Read[B], c: Read[C])(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B], arg3: Coder[C]): SCollection[(K, (Iterable[A], Iterable[B], Iterable[C]))]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  22. def sortMergeCoGroup[K, A, B, C](keyClass: Class[K], a: Read[A], b: Read[B], c: Read[C], targetParallelism: TargetParallelism)(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B], arg3: Coder[C]): SCollection[(K, (Iterable[A], Iterable[B], Iterable[C]))]

    For each key K in a or b or c, return a resulting SCollection that contains a tuple with the list of values for that key in a, b and c.

    For each key K in a or b or c, return a resulting SCollection that contains a tuple with the list of values for that key in a, b and c.

    See note on SortedBucketScioContext.sortMergeJoin for information on how an SMB cogroup differs from a regular org.apache.beam.sdk.transforms.join.CoGroupByKey operation.

    keyClass

    cogroup key class. Must have a Coder in Beam's default org.apache.beam.sdk.coders.CoderRegistry as custom key coders are not supported yet.

    targetParallelism

    the desired parallelism of the job. See org.apache.beam.sdk.extensions.smb.TargetParallelism for more information.

    Annotations
    @experimental()
  23. def sortMergeCoGroup[K1, K2, A, B](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B]): SCollection[((K1, K2), (Iterable[A], Iterable[B]))]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  24. def sortMergeCoGroup[K1, K2, A, B](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B]): SCollection[((K1, K2), (Iterable[A], Iterable[B]))]

    Secondary keyed variant.

    Secondary keyed variant.

    Annotations
    @experimental()
  25. def sortMergeCoGroup[K, A, B](keyClass: Class[K], a: Read[A], b: Read[B])(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B]): SCollection[(K, (Iterable[A], Iterable[B]))]

    targetParallelism defaults to TargetParallelism.auto()

  26. def sortMergeCoGroup[K, A, B](keyClass: Class[K], a: Read[A], b: Read[B], targetParallelism: TargetParallelism = TargetParallelism.auto())(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B]): SCollection[(K, (Iterable[A], Iterable[B]))]

    For each key K in a or b return a resulting SCollection that contains a tuple with the list of values for that key in a, and b.

    For each key K in a or b return a resulting SCollection that contains a tuple with the list of values for that key in a, and b.

    See note on SortedBucketScioContext.sortMergeJoin for information on how an SMB cogroup differs from a regular org.apache.beam.sdk.transforms.join.CoGroupByKey operation.

    keyClass

    cogroup key class. Must have a Coder in Beam's default org.apache.beam.sdk.coders.CoderRegistry as custom key coders are not supported yet.

    targetParallelism

    the desired parallelism of the job. See org.apache.beam.sdk.extensions.smb.TargetParallelism for more information.

    Annotations
    @experimental()
  27. def sortMergeGroupByKey[K1, K2, V](keyClass: Class[K1], keyClassSecondary: Class[K2], read: Read[V])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[V]): SCollection[((K1, K2), Iterable[V])]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  28. def sortMergeGroupByKey[K1, K2, V](keyClass: Class[K1], keyClassSecondary: Class[K2], read: Read[V], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[V]): SCollection[((K1, K2), Iterable[V])]

    Secondary keyed variant.

    Secondary keyed variant.

    Annotations
    @experimental()
  29. def sortMergeGroupByKey[K, V](keyClass: Class[K], read: Read[V], targetParallelism: TargetParallelism = TargetParallelism.auto())(implicit arg0: Coder[K], arg1: Coder[V]): SCollection[(K, Iterable[V])]

    For each key K in read return a resulting SCollection that contains a tuple with the list of values for that key in read.

    For each key K in read return a resulting SCollection that contains a tuple with the list of values for that key in read.

    See note on SortedBucketScioContext.sortMergeJoin for information on how an SMB group differs from a regular org.apache.beam.sdk.transforms.GroupByKey operation.

    keyClass

    cogroup key class. Must have a Coder in Beam's default org.apache.beam.sdk.coders.CoderRegistry as custom key coders are not supported yet.

    targetParallelism

    the desired parallelism of the job. See org.apache.beam.sdk.extensions.smb.TargetParallelism for more information.

    Annotations
    @experimental()
  30. def sortMergeJoin[K1, K2, L, R](keyClass: Class[K1], keyClassSecondary: Class[K2], lhs: Read[L], rhs: Read[R])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[L], arg3: Coder[R]): SCollection[((K1, K2), (L, R))]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  31. def sortMergeJoin[K1, K2, L, R](keyClass: Class[K1], keyClassSecondary: Class[K2], lhs: Read[L], rhs: Read[R], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[L], arg3: Coder[R]): SCollection[((K1, K2), (L, R))]

    Secondary keyed variant.

    Secondary keyed variant.

    Annotations
    @experimental()
  32. def sortMergeJoin[K, L, R](keyClass: Class[K], lhs: Read[L], rhs: Read[R], targetParallelism: TargetParallelism = TargetParallelism.auto())(implicit arg0: Coder[K], arg1: Coder[L], arg2: Coder[R]): SCollection[(K, (L, R))]

    Return an SCollection containing all pairs of elements with matching keys in lhs and rhs.

    Return an SCollection containing all pairs of elements with matching keys in lhs and rhs. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in lhs and (k, v2) is in rhs.

    Unlike a regular PairSCollectionFunctions.join, the key information (namely, how to extract a comparable K from L and R) is remotely encoded in a org.apache.beam.sdk.extensions.smb.BucketMetadata file in the same directory as the input records. This transform requires a filesystem lookup to ensure that the metadata for each source are compatible. In return for reading pre-sorted data, the shuffle step in a typical org.apache.beam.sdk.transforms.GroupByKey operation can be eliminated.

    keyClass

    join key class. Must have a Coder in Beam's default org.apache.beam.sdk.coders.CoderRegistry as custom key coders are not supported yet.

    targetParallelism

    the desired parallelism of the job. See org.apache.beam.sdk.extensions.smb.TargetParallelism for more information.

    Annotations
    @experimental()
  33. def sortMergeTransform[K1, K2, A, B, C](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], c: Read[C])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B], arg4: Coder[C]): ReadBuilder[KV[K1, K2], K1, K2, (Iterable[A], Iterable[B], Iterable[C])]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  34. def sortMergeTransform[K1, K2, A, B, C](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], c: Read[C], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B], arg4: Coder[C]): ReadBuilder[KV[K1, K2], K1, K2, (Iterable[A], Iterable[B], Iterable[C])]

    Secondary keyed variant

    Secondary keyed variant

    Annotations
    @experimental()
  35. def sortMergeTransform[K, A, B, C](keyClass: Class[K], a: Read[A], b: Read[B], c: Read[C])(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B], arg3: Coder[C]): ReadBuilder[K, K, Void, (Iterable[A], Iterable[B], Iterable[C])]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  36. def sortMergeTransform[K, A, B, C](keyClass: Class[K], a: Read[A], b: Read[B], c: Read[C], targetParallelism: TargetParallelism)(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B], arg3: Coder[C]): ReadBuilder[K, K, Void, (Iterable[A], Iterable[B], Iterable[C])]

    Perform a 3-way SortedBucketScioContext.sortMergeCoGroup operation, then immediately apply a transformation function to the merged cogroups and re-write using the same bucketing key and hashing scheme.

    Perform a 3-way SortedBucketScioContext.sortMergeCoGroup operation, then immediately apply a transformation function to the merged cogroups and re-write using the same bucketing key and hashing scheme. By applying the write, transform, and write in the same transform, an extra shuffle step can be avoided.

    Annotations
    @experimental()
  37. def sortMergeTransform[K1, K2, A, B](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B])(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B]): ReadBuilder[KV[K1, K2], K1, K2, (Iterable[A], Iterable[B])]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  38. def sortMergeTransform[K1, K2, A, B](keyClass: Class[K1], keyClassSecondary: Class[K2], a: Read[A], b: Read[B], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[A], arg3: Coder[B]): ReadBuilder[KV[K1, K2], K1, K2, (Iterable[A], Iterable[B])]

    Secondary keyed variant

    Secondary keyed variant

    Annotations
    @experimental()
  39. def sortMergeTransform[K, A, B](keyClass: Class[K], a: Read[A], b: Read[B])(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B]): ReadBuilder[K, K, Void, (Iterable[A], Iterable[B])]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  40. def sortMergeTransform[K, A, B](keyClass: Class[K], a: Read[A], b: Read[B], targetParallelism: TargetParallelism)(implicit arg0: Coder[K], arg1: Coder[A], arg2: Coder[B]): ReadBuilder[K, K, Void, (Iterable[A], Iterable[B])]

    Perform a 2-way SortedBucketScioContext.sortMergeCoGroup operation, then immediately apply a transformation function to the merged cogroups and re-write using the same bucketing key and hashing scheme.

    Perform a 2-way SortedBucketScioContext.sortMergeCoGroup operation, then immediately apply a transformation function to the merged cogroups and re-write using the same bucketing key and hashing scheme. By applying the write, transform, and write in the same transform, an extra shuffle step can be avoided.

    Annotations
    @experimental()
  41. def sortMergeTransform[K1, K2, R](keyClass: Class[K1], keyClassSecondary: Class[K2], read: Read[R], targetParallelism: TargetParallelism)(implicit arg0: Coder[K1], arg1: Coder[K2], arg2: Coder[R]): ReadBuilder[KV[K1, K2], K1, K2, Iterable[R]]

    Secondary keyed variant

    Secondary keyed variant

    Annotations
    @experimental()
  42. def sortMergeTransform[K, R](keyClass: Class[K], read: Read[R]): ReadBuilder[K, K, Void, Iterable[R]]

    targetParallelism defaults to TargetParallelism.auto()

    targetParallelism defaults to TargetParallelism.auto()

    Annotations
    @experimental()
  43. def sortMergeTransform[K, R](keyClass: Class[K], read: Read[R], targetParallelism: TargetParallelism): ReadBuilder[K, K, Void, Iterable[R]]

    Perform a SortedBucketScioContext.sortMergeGroupByKey operation, then immediately apply a transformation function to the merged groups and re-write using the same bucketing key and hashing scheme.

    Perform a SortedBucketScioContext.sortMergeGroupByKey operation, then immediately apply a transformation function to the merged groups and re-write using the same bucketing key and hashing scheme. By applying the write, transform, and write in the same transform, an extra shuffle step can be avoided.

    Annotations
    @experimental()
  44. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  45. def toString(): String
    Definition Classes
    AnyRef → Any
  46. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  47. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  48. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

cogroup

join

per_key

Ungrouped