c

com.spotify.scio.values

PairHashSCollectionFunctions

class PairHashSCollectionFunctions[K, V] extends AnyRef

Extra functions available on SCollections of (key, value) pairs for hash based joins through an implicit conversion.

Source
PairHashSCollectionFunctions.scala
Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. PairHashSCollectionFunctions
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new PairHashSCollectionFunctions(self: SCollection[(K, V)])

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  6. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  7. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  8. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  9. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  10. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  11. def hashFullOuterJoin[W](sideInput: SideInput[Map[K, Iterable[W]]])(implicit arg0: Coder[W]): SCollection[(K, (Option[V], Option[W]))]

    Perform a full outer join with a SideInput[Map[K, Iterable[W]]].

    Perform a full outer join with a SideInput[Map[K, Iterable[W]]].

    Example:
    1. val si = pairSCollRight.asMultiMapSingletonSideInput
      val joined1 = pairSColl1Left.hashFullOuterJoin(si)
      val joined2 = pairSColl2Left.hashFullOuterJoin(si)
  12. def hashFullOuterJoin[W](rhs: SCollection[(K, W)]): SCollection[(K, (Option[V], Option[W]))]

    Perform a full outer join by replicating rhs to all workers.

    Perform a full outer join by replicating rhs to all workers. The right side should be tiny and fit in memory.

  13. def hashIntersectByKey(sideInput: SideInput[Set[K]]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are in the SideSet rhs.

    Return an SCollection with the pairs from this whose keys are in the SideSet rhs.

    Unlike SCollection.intersection this preserves duplicates in this.

  14. def hashIntersectByKey(rhs: SCollection[K]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are in rhs given rhs is small enough to fit in memory.

    Return an SCollection with the pairs from this whose keys are in rhs given rhs is small enough to fit in memory.

    Unlike SCollection.intersection this preserves duplicates in this.

  15. def hashJoin[W](sideInput: SideInput[Map[K, Iterable[W]]])(implicit arg0: Coder[W]): SCollection[(K, (V, W))]

    Perform an inner join with a MultiMap SideInput[Map[K, Iterable[V]]

    Perform an inner join with a MultiMap SideInput[Map[K, Iterable[V]]

    The right side is tiny and fits in memory. The SideInput can be used reused for multiple joins.

    Example:
    1. val si = pairSCollRight.asMultiMapSingletonSideInput
      val joined1 = pairSColl1Left.hashJoin(si)
      val joined2 = pairSColl2Left.hashJoin(si)
  16. def hashJoin[W](rhs: SCollection[(K, W)]): SCollection[(K, (V, W))]

    Perform an inner join by replicating rhs to all workers.

    Perform an inner join by replicating rhs to all workers. The right side should be tiny and fit in memory.

  17. def hashLeftOuterJoin[W](sideInput: SideInput[Map[K, Iterable[W]]])(implicit arg0: Coder[W]): SCollection[(K, (V, Option[W]))]

    Perform a left outer join with a MultiMap SideInput[Map[K, Iterable[V]]

    Perform a left outer join with a MultiMap SideInput[Map[K, Iterable[V]]

    Example:
    1. val si = pairSCollRight.asMultiMapSingletonSideInput
      val joined1 = pairSColl1Left.hashLeftOuterJoin(si)
      val joined2 = pairSColl2Left.hashLeftOuterJoin(si)
  18. def hashLeftOuterJoin[W](rhs: SCollection[(K, W)]): SCollection[(K, (V, Option[W]))]

    Perform a left outer join by replicating rhs to all workers.

    Perform a left outer join by replicating rhs to all workers. The right side should be tiny and fit in memory.

    rhs

    The tiny SCollection[(K, W)] treated as right side of the join.

    Example:
    1. val si = pairSCollRight  // Should be tiny
      val joined = pairSColl1Left.hashLeftOuterJoin(pairSCollRight)
  19. def hashSubtractByKey(rhs: SCollection[K]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are not in SCollection[V] rhs.

    Return an SCollection with the pairs from this whose keys are not in SCollection[V] rhs.

    Rhs must be small enough to fit into memory.

  20. def hashSubtractByKey(sideInput: SideInput[Set[K]]): SCollection[(K, V)]

    Return an SCollection with the pairs from this whose keys are not in SideInput[Set] rhs.

  21. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  22. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  23. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  24. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. val self: SCollection[(K, V)]
  26. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  27. def toString(): String
    Definition Classes
    AnyRef → Any
  28. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  29. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  30. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Join Operations

per key

Ungrouped