Packages

p

com.spotify.scio

transforms

package transforms

Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. transforms
  2. AllSyntax
  3. SCollectionSafeSyntax
  4. SCollectionPipeSyntax
  5. SCollectionParallelismSyntax
  6. SCollectionFileDownloadSyntax
  7. SCollectionWithResourceSyntax
  8. AnyRef
  9. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Package Members

  1. package syntax

Type Members

  1. abstract class BaseAsyncDoFn[InputT, OutputT, ResourceT, FutureT] extends DoFnWithResource[InputT, OutputT, ResourceT] with Base[FutureT, OutputT]

    A DoFn that handles asynchronous requests to an external service.

  2. abstract class BaseAsyncLookupDoFn[A, B, C, F, T] extends DoFn[A, KV[A, T]] with Base[F, B]

    A DoFn that performs asynchronous lookup using the provided client.

  3. class CollectFnWithResource[T, U, R] extends DoFnWithResource[T, U, R]
  4. abstract class DoFnWithResource[InputT, OutputT, ResourceT] extends DoFn[InputT, OutputT]

    A DoFn that manages an external resource.

  5. class FileDownloadDoFn[OutputT] extends DoFn[URI, OutputT]

    A DoFn that downloads URI elements and processes them as local Paths.

  6. class FilterFnWithResource[T, R] extends DoFnWithResource[T, T, R]
  7. class FlatMapFnWithResource[T, U, R] extends DoFnWithResource[T, U, R]
  8. class FutureHandlers extends AnyRef

    Utility to abstract away Guava, Java 8 and Scala future handling.

  9. abstract class GuavaAsyncDoFn[InputT, OutputT, ResourceT] extends BaseAsyncDoFn[InputT, OutputT, ResourceT, ListenableFuture[OutputT]] with Guava[OutputT]

    A DoFn that handles asynchronous requests to an external service that returns Guava ListenableFutures.

  10. abstract class GuavaAsyncLookupDoFn[A, B, C] extends BaseAsyncLookupDoFn[A, B, C, ListenableFuture[B], Try[B]] with Guava[B]

    A DoFn that performs asynchronous lookup using the provided client for Guava ListenableFuture.

  11. abstract class JavaAsyncDoFn[InputT, OutputT, ResourceT] extends BaseAsyncDoFn[InputT, OutputT, ResourceT, CompletableFuture[OutputT]] with Java[OutputT]

    A DoFn that handles asynchronous requests to an external service that returns Java 8 CompletableFutures.

  12. abstract class JavaAsyncLookupDoFn[A, B, C] extends BaseAsyncLookupDoFn[A, B, C, CompletableFuture[B], Try[B]] with Java[B]

    A DoFn that performs asynchronous lookup using the provided client for Java 8 CompletableFuture.

  13. class MapFnWithResource[T, U, R] extends DoFnWithResource[T, U, R]
  14. class ParallelCollectFn[T, U] extends ParallelLimitedFn[T, U]
  15. class ParallelFilterFn[T] extends ParallelLimitedFn[T, T]
  16. class ParallelFlatMapFn[T, U] extends ParallelLimitedFn[T, U]
  17. class ParallelMapFn[T, U] extends ParallelLimitedFn[T, U]
  18. class PipeDoFn extends DoFn[String, String]

    A DoFn that pipes elements through an external command via StdIn & StdOut.

  19. class RateLimiterDoFn[InputT] extends DoFnWithResource[InputT, InputT, RateLimiter]

    DoFn which will rate limit the number of elements processed per second.

    DoFn which will rate limit the number of elements processed per second.

    Used to rate limit throughput for a job writing to a database or making calls to external services. The limit is applied per worker and should be used with a fixed/max num workers. Having RateLimiterDoFn(1000) and 20 workers means your total rate will be 20000.

  20. abstract class ScalaAsyncDoFn[I, O, R] extends BaseAsyncDoFn[I, O, R, Future[O]] with ScalaFutureHandlers[O]

    A DoFn that handles asynchronous requests to an external service that returns Scala Futures.

  21. abstract class ScalaAsyncLookupDoFn[A, B, C] extends BaseAsyncLookupDoFn[A, B, C, Future[B], Try[B]] with ScalaFutureHandlers[B]

    A DoFn that performs asynchronous lookup using the provided client for Scala Future.

    A DoFn that performs asynchronous lookup using the provided client for Scala Future.

    A

    input element type.

    B

    client lookup value type.

    C

    client type.

  22. trait ScalaFutureHandlers[T] extends Base[Future[T], T]

    A FutureHandlers.Base implementation for Scala Future.

  23. implicit class FileDownloadSCollection extends AnyRef

    Enhanced version of SCollection with URI methods.

    Enhanced version of SCollection with URI methods.

    Definition Classes
    SCollectionFileDownloadSyntax
  24. implicit class CustomParallelismSCollection[T] extends AnyRef

    Enhanced version of SCollection with custom parallelism, where parallelism is the number of concurrent DoFn threads per worker (default to number of CPU cores).

    Enhanced version of SCollection with custom parallelism, where parallelism is the number of concurrent DoFn threads per worker (default to number of CPU cores).

    Definition Classes
    SCollectionParallelismSyntax
  25. implicit class PipeSCollection extends AnyRef

    Enhanced version of SCollection with pipe methods.

    Enhanced version of SCollection with pipe methods.

    Definition Classes
    SCollectionPipeSyntax
  26. implicit class SpecializedFlatMapSCollection[T] extends AnyRef

    Enhanced version of SCollection with specialized versions of flatMap.

    Enhanced version of SCollection with specialized versions of flatMap.

    Definition Classes
    SCollectionSafeSyntax
  27. implicit class SCollectionWithResourceFunctions[T] extends AnyRef

Value Members

  1. object JavaAsyncConverters

Inherited from AllSyntax

Inherited from SCollectionSafeSyntax

Inherited from SCollectionPipeSyntax

Inherited from AnyRef

Inherited from Any

Ungrouped