class PairHashSCollectionFunctions[K, V] extends AnyRef
Extra functions available on SCollections of (key, value) pairs for hash based joins through an implicit conversion.
- Alphabetic
- By Inheritance
- PairHashSCollectionFunctions
- AnyRef
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new PairHashSCollectionFunctions(self: SCollection[(K, V)])
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def ##: Int
- Definition Classes
- AnyRef → Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.CloneNotSupportedException]) @native()
- final def eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- def equals(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef → Any
- def finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.Throwable])
- final def getClass(): Class[_ <: AnyRef]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
- def hashFullOuterJoin[W](sideInput: SideInput[Map[K, Iterable[W]]])(implicit arg0: Coder[W]): SCollection[(K, (Option[V], Option[W]))]
Perform a full outer join with a
SideInput[Map[K, Iterable[W]]]
.Perform a full outer join with a
SideInput[Map[K, Iterable[W]]]
.val si = pairSCollRight.asMultiMapSingletonSideInput val joined1 = pairSColl1Left.hashFullOuterJoin(si) val joined2 = pairSColl2Left.hashFullOuterJoin(si)
Example: - def hashFullOuterJoin[W](rhs: SCollection[(K, W)]): SCollection[(K, (Option[V], Option[W]))]
Perform a full outer join by replicating
rhs
to all workers.Perform a full outer join by replicating
rhs
to all workers. The right side should be tiny and fit in memory. - def hashIntersectByKey(sideInput: SideInput[Set[K]]): SCollection[(K, V)]
Return an SCollection with the pairs from
this
whose keys are in the SideSetrhs
.Return an SCollection with the pairs from
this
whose keys are in the SideSetrhs
.Unlike SCollection.intersection this preserves duplicates in
this
. - def hashIntersectByKey(rhs: SCollection[K]): SCollection[(K, V)]
Return an SCollection with the pairs from
this
whose keys are inrhs
givenrhs
is small enough to fit in memory.Return an SCollection with the pairs from
this
whose keys are inrhs
givenrhs
is small enough to fit in memory.Unlike SCollection.intersection this preserves duplicates in
this
. - def hashJoin[W](sideInput: SideInput[Map[K, Iterable[W]]])(implicit arg0: Coder[W]): SCollection[(K, (V, W))]
Perform an inner join with a MultiMap
SideInput[Map[K, Iterable[V]]
Perform an inner join with a MultiMap
SideInput[Map[K, Iterable[V]]
The right side is tiny and fits in memory. The SideInput can be used reused for multiple joins.
val si = pairSCollRight.asMultiMapSingletonSideInput val joined1 = pairSColl1Left.hashJoin(si) val joined2 = pairSColl2Left.hashJoin(si)
Example: - def hashJoin[W](rhs: SCollection[(K, W)]): SCollection[(K, (V, W))]
Perform an inner join by replicating
rhs
to all workers.Perform an inner join by replicating
rhs
to all workers. The right side should be tiny and fit in memory. - def hashLeftOuterJoin[W](sideInput: SideInput[Map[K, Iterable[W]]])(implicit arg0: Coder[W]): SCollection[(K, (V, Option[W]))]
Perform a left outer join with a MultiMap
SideInput[Map[K, Iterable[V]]
Perform a left outer join with a MultiMap
SideInput[Map[K, Iterable[V]]
val si = pairSCollRight.asMultiMapSingletonSideInput val joined1 = pairSColl1Left.hashLeftOuterJoin(si) val joined2 = pairSColl2Left.hashLeftOuterJoin(si)
Example: - def hashLeftOuterJoin[W](rhs: SCollection[(K, W)]): SCollection[(K, (V, Option[W]))]
Perform a left outer join by replicating
rhs
to all workers.Perform a left outer join by replicating
rhs
to all workers. The right side should be tiny and fit in memory.- rhs
The tiny SCollection[(K, W)] treated as right side of the join.
val si = pairSCollRight // Should be tiny val joined = pairSColl1Left.hashLeftOuterJoin(pairSCollRight)
Example: - def hashSubtractByKey(rhs: SCollection[K]): SCollection[(K, V)]
Return an SCollection with the pairs from
this
whose keys are not in SCollection[V]rhs
.Return an SCollection with the pairs from
this
whose keys are not in SCollection[V]rhs
.Rhs must be small enough to fit into memory.
- def hashSubtractByKey(sideInput: SideInput[Set[K]]): SCollection[(K, V)]
Return an SCollection with the pairs from
this
whose keys are not in SideInput[Set]rhs
. - final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- final def ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
- final def notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- final def notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
- val self: SCollection[(K, V)]
- final def synchronized[T0](arg0: => T0): T0
- Definition Classes
- AnyRef
- def toString(): String
- Definition Classes
- AnyRef → Any
- final def wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException])
- final def wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws(classOf[java.lang.InterruptedException]) @native()