Type Members

1.  abstract class BaseAsyncBatchLookupDoFn[Input, BatchRequest, BatchResponse, Output, ClientType, FutureType, TryWrapper] extends DoFnWithResource[Input, KV[Input, TryWrapper], Pair[ClientType, Cache[String, Output]]] with Base[FutureType, BatchResponse]

   A DoFn that performs asynchronous lookup using the provided client.

   A DoFn that performs asynchronous lookup using the provided client. Lookup requests may be deduplicated.

2.  abstract class BaseAsyncDoFn[InputT, OutputT, ResourceT, FutureT] extends DoFnWithResource[InputT, OutputT, ResourceT] with Base[FutureT, OutputT]

   A DoFn that handles asynchronous requests to an external service.

3.  abstract class BaseAsyncLookupDoFn[A, B, C, F, T] extends DoFnWithResource[A, KV[A, T], Pair[C, Cache[A, B]]] with Base[F, B]

   A DoFn that performs asynchronous lookup using the provided client.

   A DoFn that performs asynchronous lookup using the provided client. Lookup requests may be deduplicated.

4.  class BatchDoFn[InputT] extends DoFn[InputT, Iterable[InputT]]

   Batches input into a desired batch size.

   Batches input into a desired batch size.

   Elements are buffered until there are enough elements for a batch, at which point they are emitted to the output PCollection

   Windows are preserved (batches contain elements from the same window). Batches are not spanning over bundles. Once a bundle is finished, the batch is emitted even if not full. This function can only batch 10 parallel windows. If new element comes from an 11th window, the bigger batch will be emitted to give room for this new element.

5.  class CollectFnWithResource[T, U, R] extends DoFnWithResource[T, U, R]
6.  abstract class DoFnWithResource[InputT, OutputT, ResourceT] extends DoFn[InputT, OutputT]

   A DoFn that manages an external resource.

7.  class FileDownloadDoFn[OutputT] extends DoFn[URI, OutputT]

   A DoFn that downloads URI elements and processes them as local Paths.

8.  class FilterFnWithResource[T, R] extends DoFnWithResource[T, T, R]
9.  class FlatMapFnWithResource[T, U, R] extends DoFnWithResource[T, U, R]
10.  class FutureHandlers extends AnyRef

    Utility to abstract away Guava, Java 8 and Scala future handling.

11.  abstract class GuavaAsyncBatchLookupDoFn[Input, BatchRequest, BatchResponse, Output, ClientType] extends BaseAsyncBatchLookupDoFn[Input, BatchRequest, BatchResponse, Output, ClientType, ListenableFuture[BatchResponse], Try[Output]] with Guava[BatchResponse]

12.  abstract class GuavaAsyncDoFn[InputT, OutputT, ResourceT] extends BaseAsyncDoFn[InputT, OutputT, ResourceT, ListenableFuture[OutputT]] with Guava[OutputT]

    A DoFn that handles asynchronous requests to an external service that returns Guava ListenableFutures.

13.  abstract class GuavaAsyncLookupDoFn[A, B, C] extends BaseAsyncLookupDoFn[A, B, C, ListenableFuture[B], Try[B]] with Guava[B]

    A DoFn that performs asynchronous lookup using the provided client for Guava ListenableFuture.

14.  abstract class JavaAsyncBatchLookupDoFn[Input, BatchRequest, BatchResponse, Output, ClientType] extends BaseAsyncBatchLookupDoFn[Input, BatchRequest, BatchResponse, Output, ClientType, CompletableFuture[BatchResponse], Try[Output]] with Java[BatchResponse]

15.  abstract class JavaAsyncDoFn[InputT, OutputT, ResourceT] extends BaseAsyncDoFn[InputT, OutputT, ResourceT, CompletableFuture[OutputT]] with Java[OutputT]

    A DoFn that handles asynchronous requests to an external service that returns Java 8 CompletableFutures.

16.  abstract class JavaAsyncLookupDoFn[A, B, C] extends BaseAsyncLookupDoFn[A, B, C, CompletableFuture[B], Try[B]] with Java[B]

    A DoFn that performs asynchronous lookup using the provided client for Java 8 CompletableFuture.

17.  class MapFnWithResource[T, U, R] extends DoFnWithResource[T, U, R]
18.  class ParallelCollectFn[T, U] extends ParallelLimitedFn[T, U]
19.  class ParallelFilterFn[T] extends ParallelLimitedFn[T, T]
20.  class ParallelFlatMapFn[T, U] extends ParallelLimitedFn[T, U]
21.  class ParallelMapFn[T, U] extends ParallelLimitedFn[T, U]
22.  class PipeDoFn extends DoFn[String, String]

    A DoFn that pipes elements through an external command via StdIn & StdOut.

23.  class RateLimiterDoFn[InputT] extends DoFnWithResource[InputT, InputT, RateLimiter]

    DoFn which will rate limit the number of elements processed per second.

    DoFn which will rate limit the number of elements processed per second.

    Used to rate limit throughput for a job writing to a database or making calls to external services. The limit is applied per worker and should be used with a fixed/max num workers. Having RateLimiterDoFn(1000) and 20 workers means your total rate will be 20000.

24.  abstract class ScalaAsyncDoFn[I, O, R] extends BaseAsyncDoFn[I, O, R, Future[O]] with ScalaFutureHandlers[O]

    A DoFn that handles asynchronous requests to an external service that returns Scala Future s.

25.  abstract class ScalaAsyncLookupDoFn[A, B, C] extends BaseAsyncLookupDoFn[A, B, C, Future[B], Try[B]] with ScalaFutureHandlers[B]

    A DoFn that performs asynchronous lookup using the provided client for Scala Future.

    A DoFn that performs asynchronous lookup using the provided client for Scala Future.

    A

    input element type.

    B

    client lookup value type.

    C

    client type.

26.  trait ScalaFutureHandlers[T] extends Base[Future[T], T]

    A FutureHandlers.Base implementation for Scala Future.

27.  implicit class FileDownloadSCollection extends AnyRef

    Enhanced version of SCollection with URI methods.

    Enhanced version of SCollection with URI methods.

    Definition Classes

    SCollectionFileDownloadSyntax

28.  implicit class CustomParallelismSCollection[T] extends AnyRef

    Enhanced version of SCollection with custom parallelism, where parallelism is the number of concurrent DoFn threads per worker (default to number of CPU cores).

    Enhanced version of SCollection with custom parallelism, where parallelism is the number of concurrent DoFn threads per worker (default to number of CPU cores).

    Definition Classes

    SCollectionParallelismSyntax

29.  implicit class PipeSCollection extends AnyRef

    Enhanced version of SCollection with pipe methods.

    Enhanced version of SCollection with pipe methods.

    Definition Classes

    SCollectionPipeSyntax

30.  implicit class SafeFlatMapSCollection[T] extends AnyRef

    Enhanced version of SCollection with specialized versions of flatMap.

    Enhanced version of SCollection with specialized versions of flatMap.

    Definition Classes

    SCollectionSafeSyntax

31.  implicit class SCollectionWithResourceFunctions[T] extends AnyRef

    Definition Classes

    SCollectionWithResourceSyntax