c

com.spotify.scio.values

PairSCollectionFunctions

class PairSCollectionFunctions[K, V] extends AnyRef

Extra functions available on SCollections of (key, value) pairs through an implicit conversion.

Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PairSCollectionFunctions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new PairSCollectionFunctions(self: SCollection[(K, V)])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def aggregateByKey[A, U](aggregator: Aggregator[V, A, U])(implicit arg0: Coder[A], arg1: Coder[U], koder: Coder[K], voder: Coder[V]): SCollection[(K, U)]

    Aggregate the values of each key with Aggregator.

    Aggregate the values of each key with Aggregator. First each value V is mapped to A, then we reduce with a Semigroup of A, then finally we present the results as U. This could be more powerful and better optimized in some cases.

  5. def aggregateByKey[U](zeroValue: U)(seqOp: (U, V) ⇒ U, combOp: (U, U) ⇒ U)(implicit arg0: Coder[U], koder: Coder[K], voder: Coder[V]): SCollection[(K, U)]

    Aggregate the values of each key, using given combine functions and a neutral "zero value".

    Aggregate the values of each key, using given combine functions and a neutral "zero value". This function can return a different result type, U, than the type of the values in this SCollection, V. Thus, we need one operation for merging a V into a U and one operation for merging two U's. To avoid memory allocation, both of these functions are allowed to modify and return their first argument instead of creating a new U.

  6. def applyPerKeyDoFn[U](t: DoFn[KV[K, V], KV[K, U]])(implicit arg0: Coder[U], koder: Coder[K], vcoder: Coder[V]): SCollection[(K, U)]

    Apply a DoFn that processes KVs and wrap the output in an SCollection.

  7. def approxQuantilesByKey(numQuantiles: Int)(implicit ord: Ordering[V], koder: Coder[K], voder: Coder[V], dummy: DummyImplicit): SCollection[(K, Iterable[V])]
  8. def approxQuantilesByKey(numQuantiles: Int, ord: Ordering[V])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Iterable[V])]

    For each key, compute the values' data distribution using approximate N-tiles.

    For each key, compute the values' data distribution using approximate N-tiles.

    returns

    a new SCollection whose values are Iterables of the approximate N-tiles of the elements.

  9. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  10. def asMapSideInput(implicit koder: Coder[K], voder: Coder[V]): SideInput[Map[K, V]]

    Convert this SCollection to a SideInput, mapping key-value pairs of each window to a Map[key, value], to be used with SCollection.withSideInputs.

    Convert this SCollection to a SideInput, mapping key-value pairs of each window to a Map[key, value], to be used with SCollection.withSideInputs. It is required that each key of the input be associated with a single value.

    Currently, the resulting map is required to fit into memory.

  11. def asMultiMapSideInput(implicit koder: Coder[K], voder: Coder[V]): SideInput[Map[K, Iterable[V]]]

    Convert this SCollection to a SideInput, mapping key-value pairs of each window to a Map[key, Iterable[value]], to be used with SCollection.withSideInputs.

    Convert this SCollection to a SideInput, mapping key-value pairs of each window to a Map[key, Iterable[value]], to be used with SCollection.withSideInputs. In contrast to asMapSideInput, it is not required that the keys in the input collection be unique.

    Currently, the resulting map is required to fit into memory.

  12. def batchByKey(batchSize: Long)(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Iterable[V])]

    Batches inputs to a desired batch size.

    Batches inputs to a desired batch size. Batches will contain only elements of a single key.

    Elements are buffered until there are batchSize elements buffered, at which point they are outputed to the output SCollection.

    Windows are preserved (batches contain elements from the same window). Batches may contain elements from more than one bundle.

  13. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  14. def cogroup[W1, W2, W3](that1: SCollection[(K, W1)], that2: SCollection[(K, W2)], that3: SCollection[(K, W3)])(implicit arg0: Coder[W1], arg1: Coder[W2], arg2: Coder[W3], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Iterable[V], Iterable[W1], Iterable[W2], Iterable[W3]))]

    For each key k in this or that1 or that2 or that3, return a resulting SCollection that contains a tuple with the list of values for that key in this, that1, that2 and that3.

  15. def cogroup[W1, W2](that1: SCollection[(K, W1)], that2: SCollection[(K, W2)])(implicit arg0: Coder[W1], arg1: Coder[W2], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Iterable[V], Iterable[W1], Iterable[W2]))]

    For each key k in this or that1 or that2, return a resulting SCollection that contains a tuple with the list of values for that key in this, that1 and that2.

  16. def cogroup[W](that: SCollection[(K, W)])(implicit arg0: Coder[W], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Iterable[V], Iterable[W]))]

    For each key k in this or that, return a resulting SCollection that contains a tuple with the list of values for that key in this as well as that.

  17. def combineByKey[C](createCombiner: (V) ⇒ C)(mergeValue: (C, V) ⇒ C)(mergeCombiners: (C, C) ⇒ C)(implicit arg0: Coder[C], koder: Coder[K], voder: Coder[V]): SCollection[(K, C)]

    Generic function to combine the elements for each key using a custom set of aggregation functions.

    Generic function to combine the elements for each key using a custom set of aggregation functions. Turns an SCollection[(K, V)] into a result of type SCollection[(K, C)], for a "combined type" C Note that V and C can be different -- for example, one might group an SCollection of type (Int, Int) into an SCollection of type (Int, Seq[Int]). Users provide three functions:

    - createCombiner, which turns a V into a C (e.g., creates a one-element list)

    - mergeValue, to merge a V into a C (e.g., adds it to the end of a list)

    - mergeCombiners, to combine two C's into a single one.

  18. def countApproxDistinctByKey(maximumEstimationError: Double = 0.02)(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Long)]

    Count approximate number of distinct values for each key in the SCollection.

    Count approximate number of distinct values for each key in the SCollection.

    maximumEstimationError

    the maximum estimation error, which should be in the range [0.01, 0.5].

  19. def countApproxDistinctByKey(sampleSize: Int)(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Long)]

    Count approximate number of distinct values for each key in the SCollection.

    Count approximate number of distinct values for each key in the SCollection.

    sampleSize

    the number of entries in the statistical sample; the higher this number, the more accurate the estimate will be; should be >= 16.

  20. def countByKey(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Long)]

    Count the number of elements for each key.

    Count the number of elements for each key.

    returns

    a new SCollection of (key, count) pairs

  21. def distinctByKey(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Return a new SCollection of (key, value) pairs without duplicates based on the keys.

    Return a new SCollection of (key, value) pairs without duplicates based on the keys. The value is taken randomly for each key.

    returns

    a new SCollection of (key, value) pairs

  22. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  24. def filterValues(f: (V) ⇒ Boolean)(implicit koder: Coder[K]): SCollection[(K, V)]

    Pass each value in the key-value pair SCollection through a filter function without changing the keys.

  25. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  26. def flatMapValues[U](f: (V) ⇒ TraversableOnce[U])(implicit arg0: Coder[U], koder: Coder[K]): SCollection[(K, U)]

    Pass each value in the key-value pair SCollection through a flatMap function without changing the keys.

  27. def flattenValues[U](implicit arg0: Coder[U], ev: <:<[V, TraversableOnce[U]], koder: Coder[K]): SCollection[(K, U)]

    Return an SCollection having its values flattened.

  28. def foldByKey(implicit mon: Monoid[V], koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Fold by key with Monoid, which defines the associative function and "zero value" for V.

    Fold by key with Monoid, which defines the associative function and "zero value" for V. This could be more powerful and better optimized in some cases.

  29. def foldByKey(zeroValue: V)(op: (V, V) ⇒ V)(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Merge the values for each key using an associative function and a neutral "zero value" which may be added to the result an arbitrary number of times, and must not change the result (e.g., Nil for list concatenation, 0 for addition, or 1 for multiplication.).

  30. def fullOuterJoin[W](that: SCollection[(K, W)])(implicit arg0: Coder[W], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Option[V], Option[W]))]

    Perform a full outer join of this and that.

    Perform a full outer join of this and that. For each element (k, v) in this, the resulting SCollection will either contain all pairs (k, (Some(v), Some(w))) for w in that, or the pair (k, (Some(v), None)) if no elements in that have key k. Similarly, for each element (k, w) in that, the resulting SCollection will either contain all pairs (k, (Some(v), Some(w))) for v in this, or the pair (k, (None, Some(w))) if no elements in this have key k.

  31. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  32. def groupByKey(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Iterable[V])]

    Group the values for each key in the SCollection into a single sequence.

    Group the values for each key in the SCollection into a single sequence. The ordering of elements within each group is not guaranteed, and may even differ each time the resulting SCollection is evaluated.

    Note: This operation may be very expensive. If you are grouping in order to perform an aggregation (such as a sum or average) over each key, using PairSCollectionFunctions.aggregateByKey or PairSCollectionFunctions.reduceByKey will provide much better performance.

    Note: As currently implemented, groupByKey must be able to hold all the key-value pairs for any key in memory. If a key has too many values, it can result in an OutOfMemoryError.

  33. def groupWith[W1, W2, W3](that1: SCollection[(K, W1)], that2: SCollection[(K, W2)], that3: SCollection[(K, W3)])(implicit arg0: Coder[W1], arg1: Coder[W2], arg2: Coder[W3], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Iterable[V], Iterable[W1], Iterable[W2], Iterable[W3]))]

    Alias for cogroup.

  34. def groupWith[W1, W2](that1: SCollection[(K, W1)], that2: SCollection[(K, W2)])(implicit arg0: Coder[W1], arg1: Coder[W2], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Iterable[V], Iterable[W1], Iterable[W2]))]

    Alias for cogroup.

  35. def groupWith[W](that: SCollection[(K, W)])(implicit arg0: Coder[W], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Iterable[V], Iterable[W]))]

    Alias for cogroup.

  36. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  37. def hashPartitionByKey(numPartitions: Int): Seq[SCollection[(K, V)]]

    Partition this SCollection using K.hashCode() into n partitions

    Partition this SCollection using K.hashCode() into n partitions

    numPartitions

    number of output partitions

    returns

    partitioned SCollections in a Seq

  38. def intersectByKey(that: SCollection[K])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are in that.

    Return an SCollection with the pairs from this whose keys are in that.

    Unlike SCollection.intersection this preserves duplicates in this.

  39. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  40. def join[W](that: SCollection[(K, W)])(implicit arg0: Coder[W], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, W))]

    Return an SCollection containing all pairs of elements with matching keys in this and that.

    Return an SCollection containing all pairs of elements with matching keys in this and that. Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in this and (k, v2) is in that.

  41. def keys(implicit koder: Coder[K], voder: Coder[V]): SCollection[K]

    Return an SCollection with the keys of each tuple.

  42. def leftOuterJoin[W](that: SCollection[(K, W)])(implicit arg0: Coder[W], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, Option[W]))]

    Perform a left outer join of this and that.

    Perform a left outer join of this and that. For each element (k, v) in this, the resulting SCollection will either contain all pairs (k, (v, Some(w))) for w in that, or the pair (k, (v, None)) if no elements in that have key k.

  43. def mapValues[U](f: (V) ⇒ U)(implicit arg0: Coder[U], koder: Coder[K]): SCollection[(K, U)]

    Pass each value in the key-value pair SCollection through a map function without changing the keys.

  44. def maxByKey(ord: Ordering[V])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]
  45. def maxByKey(implicit ord: Ordering[V], koder: Coder[K], voder: Coder[V], dummy: DummyImplicit): SCollection[(K, V)]

    Return the max of values for each key as defined by the implicit Ordering[T].

    Return the max of values for each key as defined by the implicit Ordering[T].

    returns

    a new SCollection of (key, maximum value) pairs

  46. def minByKey(ord: Ordering[V])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]
  47. def minByKey(implicit koder: Coder[K], voder: Coder[V], ord: Ordering[V]): SCollection[(K, V)]

    Return the min of values for each key as defined by the implicit Ordering[T].

    Return the min of values for each key as defined by the implicit Ordering[T].

    returns

    a new SCollection of (key, minimum value) pairs

  48. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  49. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  50. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  51. def reduceByKey(op: (V, V) ⇒ V)(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Merge the values for each key using an associative reduce function.

    Merge the values for each key using an associative reduce function. This will also perform the merging locally on each mapper before sending results to a reducer, similarly to a "combiner" in MapReduce.

  52. def rightOuterJoin[W](that: SCollection[(K, W)])(implicit arg0: Coder[W], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Option[V], W))]

    Perform a right outer join of this and that.

    Perform a right outer join of this and that. For each element (k, w) in that, the resulting SCollection will either contain all pairs (k, (Some(v), w)) for v in this, or the pair (k, (None, w)) if no elements in this have key k.

  53. def sampleByKey(withReplacement: Boolean, fractions: Map[K, Double])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Return a subset of this SCollection sampled by key (via stratified sampling).

    Return a subset of this SCollection sampled by key (via stratified sampling).

    Create a sample of this SCollection using variable sampling rates for different keys as specified by fractions, a key to sampling rate map, via simple random sampling with one pass over the SCollection, to produce a sample of size that's approximately equal to the sum of math.ceil(numItems * samplingRate) over all key values.

    withReplacement

    whether to sample with or without replacement

    fractions

    map of specific keys to sampling rates

    returns

    SCollection containing the sampled subset

  54. def sampleByKey(sampleSize: Int)(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Iterable[V])]

    Return a sampled subset of values for each key of this SCollection.

    Return a sampled subset of values for each key of this SCollection.

    returns

    a new SCollection of (key, sampled values) pairs

  55. val self: SCollection[(K, V)]
  56. def sparseIntersectByKey(that: SCollection[K], thatNumKeys: Long, computeExact: Boolean = false, fpProb: Double = 0.01)(implicit koder: Coder[K], voder: Coder[V], hash: Hash128[K]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are in that when the cardinality of this >> that, but neither can fit in memory (see PairHashSCollectionFunctions.hashIntersectByKey).

    Return an SCollection with the pairs from this whose keys are in that when the cardinality of this >> that, but neither can fit in memory (see PairHashSCollectionFunctions.hashIntersectByKey).

    Unlike SCollection.intersection this preserves duplicates in this.

    thatNumKeys

    An estimate of the number of keys in that. This estimate is used to find the size and number of BloomFilters that Scio would use to pre-filter this in a "map" step before any join. Having a value close to the actual number improves the false positives in output. When computeExact is set to true, a more accurate estimate of the number of keys in that would mean less shuffle when finding the exact value.

    computeExact

    Whether or not to directly pass through bloom filter results (with a small false positive rate) or perform an additional inner join to confirm exact result set. By default this is set to false.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability for this transform. By default when computeExact is set to false, this reflects the probability that an output element is an incorrect intersect (meaning it may not be present in that) When computeExact is set to true, this fraction is used to find the acceptable false positive in the intermediate step before computing exact. Note: having fpProb = 0 doesn't mean an exact computation. This value along with thatNumKeys is used for creating a BloomFilter.

  57. def sparseJoin[W](that: SCollection[(K, W)], thatNumKeys: Long, fpProb: Double = 0.01)(implicit arg0: Coder[W], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, W))]

    Inner join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e.

    Inner join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e. when the intersection of keys is sparse in this. A Bloom Filter of keys in that is used to split this into 2 partitions. Only those with keys in the filter go through the join and the rest are filtered out before the join. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thatNumKeys

    An estimate of the number of keys in that. This estimate is used to find the size and number of BloomFilters that Scio would use to split this into overlap and intersection in a "map" step before an exact join. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability when computing the overlap. Note: having fpProb = 0 doesn't mean that Scio would calculate an exact overlap.

  58. def sparseLeftOuterJoin[W](that: SCollection[(K, W)], thatNumKeys: Long, fpProb: Double = 0.01)(implicit arg0: Coder[W], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, Option[W]))]

    Left outer join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e.

    Left outer join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e. when the intersection of keys is sparse in this. A Bloom Filter of keys in that is used to split this into 2 partitions. Only those with keys in the filter go through the join and the rest are concatenated. This is useful for joining historical aggregates with incremental updates. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thatNumKeys

    An estimate of the number of keys in that. This estimate is used to find the size and number of BloomFilters that Scio would use to split this into overlap and intersection in a "map" step before an exact join. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability when computing the overlap. Note: having fpProb = 0 doesn't mean that Scio would calculate an exact overlap.

  59. def sparseLookup[A, B](that1: SCollection[(K, A)], that2: SCollection[(K, B)], thisNumKeys: Long)(implicit arg0: Coder[A], arg1: Coder[B], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, Iterable[A], Iterable[B]))]

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that.

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that. A Bloom Filter of keys in this is used to filter out irrelevant keys in that. This is useful when searching for a limited number of values from one or more very large tables. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thisNumKeys

    An estimate of the number of keys in this. This estimate is used to find the size and number of BloomFilters that Scio would use to pre-filter that before doing a co-group. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

  60. def sparseLookup[A, B](that1: SCollection[(K, A)], that2: SCollection[(K, B)], thisNumKeys: Long, fpProb: Double)(implicit arg0: Coder[A], arg1: Coder[B], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, Iterable[A], Iterable[B]))]

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that.

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that. A Bloom Filter of keys in this is used to filter out irrelevant keys in that. This is useful when searching for a limited number of values from one or more very large tables. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thisNumKeys

    An estimate of the number of keys in this. This estimate is used to find the size and number of BloomFilters that Scio would use to pre-filter that1 and that2 before doing a co-group. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability when discarding elements of that1 and that2 in the pre-filter step.

  61. def sparseLookup[A](that: SCollection[(K, A)], thisNumKeys: Long)(implicit arg0: Coder[A], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, Iterable[A]))]

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that.

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that. A Bloom Filter of keys in this is used to filter out irrelevant keys in that. This is useful when searching for a limited number of values from one or more very large tables. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thisNumKeys

    An estimate of the number of keys in this. This estimate is used to find the size and number of BloomFilters that Scio would use to pre-filter that before doing a co-group. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

  62. def sparseLookup[A](that: SCollection[(K, A)], thisNumKeys: Long, fpProb: Double)(implicit arg0: Coder[A], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (V, Iterable[A]))]

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that.

    Look up values from that where that is much larger and keys from this wont fit in memory, and is sparse in that. A Bloom Filter of keys in this is used to filter out irrelevant keys in that. This is useful when searching for a limited number of values from one or more very large tables. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thisNumKeys

    An estimate of the number of keys in this. This estimate is used to find the size and number of BloomFilters that Scio would use to pre-filter that before doing a co-group. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability when discarding elements of that in the pre-filter step.

  63. def sparseOuterJoin[W](that: SCollection[(K, W)], thatNumKeys: Long, fpProb: Double = 0.01)(implicit arg0: Coder[W], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Option[V], Option[W]))]

    Full outer join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e.

    Full outer join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e. when the intersection of keys is sparse in this. A Bloom Filter of keys in that is used to split this into 2 partitions. Only those with keys in the filter go through the join and the rest are concatenated. This is useful for joining historical aggregates with incremental updates. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thatNumKeys

    An estimate of the number of keys in that. This estimate is used to find the size and number of BloomFilters that Scio would use to split this into overlap and intersection in a "map" step before an exact join. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability when computing the overlap. Note: having fpProb = 0 doesn't mean that Scio would calculate an exact overlap.

  64. def sparseRightOuterJoin[W](that: SCollection[(K, W)], thatNumKeys: Long, fpProb: Double = 0.01)(implicit arg0: Coder[W], hash: Hash128[K], koder: Coder[K], voder: Coder[V]): SCollection[(K, (Option[V], W))]

    Right outer join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e.

    Right outer join for cases when this is much larger than that which cannot fit in memory, but contains a mostly overlapping set of keys as this, i.e. when the intersection of keys is sparse in this. A Bloom Filter of keys in that is used to split this into 2 partitions. Only those with keys in the filter go through the join and the rest are concatenated. This is useful for joining historical aggregates with incremental updates. Read more about Bloom Filter: com.twitter.algebird.BloomFilter.

    thatNumKeys

    An estimate of the number of keys in that. This estimate is used to find the size and number of BloomFilters that Scio would use to split this into overlap and intersection in a "map" step before an exact join. Having a value close to the actual number improves the false positives in intermediate steps which means less shuffle.

    fpProb

    A fraction in range (0, 1) which would be the accepted false positive probability when computing the overlap. Note: having fpProb = 0 doesn't mean that Scio would calculate an exact overlap.

  65. def subtractByKey(that: SCollection[K])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are not in that.

  66. def sumByKey(sg: Semigroup[V])(implicit koder: Coder[K], voder: Coder[V], d: DummyImplicit): SCollection[(K, V)]
  67. def sumByKey(implicit sg: Semigroup[V], koder: Coder[K], voder: Coder[V]): SCollection[(K, V)]

    Reduce by key with Semigroup.

    Reduce by key with Semigroup. This could be more powerful and better optimized than reduceByKey in some cases.

  68. def swap(implicit koder: Coder[K], voder: Coder[V]): SCollection[(V, K)]

    Swap the keys with the values.

  69. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  70. def toString(): String
    Definition Classes
    AnyRef → Any
  71. def topByKey(num: Int, ord: Ordering[V])(implicit koder: Coder[K], voder: Coder[V]): SCollection[(K, Iterable[V])]
  72. def topByKey(num: Int)(implicit ord: Ordering[V], koder: Coder[K], voder: Coder[V], dummy: DummyImplicit): SCollection[(K, Iterable[V])]

    Return the top k (largest) values for each key from this SCollection as defined by the specified implicit Ordering[T].

    Return the top k (largest) values for each key from this SCollection as defined by the specified implicit Ordering[T].

    returns

    a new SCollection of (key, top k) pairs

  73. def values(implicit voder: Coder[V]): SCollection[V]

    Return an SCollection with the values of each tuple.

  74. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  75. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  76. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  77. def withHotKeyFanout(hotKeyFanout: Int)(implicit koder: Coder[K], voder: Coder[V]): SCollectionWithHotKeyFanout[K, V]

    Convert this SCollection to an SCollectionWithHotKeyFanout that uses an intermediate node to combine "hot" keys partially before performing the full combine.

    Convert this SCollection to an SCollectionWithHotKeyFanout that uses an intermediate node to combine "hot" keys partially before performing the full combine.

    hotKeyFanout

    constant value for every key

  78. def withHotKeyFanout(hotKeyFanout: (K) ⇒ Int)(implicit koder: Coder[K], voder: Coder[V]): SCollectionWithHotKeyFanout[K, V]

    Convert this SCollection to an SCollectionWithHotKeyFanout that uses an intermediate node to combine "hot" keys partially before performing the full combine.

    Convert this SCollection to an SCollectionWithHotKeyFanout that uses an intermediate node to combine "hot" keys partially before performing the full combine.

    hotKeyFanout

    a function from keys to an integer N, where the key will be spread among N intermediate nodes for partial combining. If N is less than or equal to 1, this key will not be sent through an intermediate node.

Inherited from AnyRef

Inherited from Any

CoGroup Operations

collection

Join Operations

per key

Per Key Aggregations

Transformations

Ungrouped