Packages

package sharded

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. Protected

Type Members

  1. final case class JdbcShardedReadOptions[T, S](connectionOptions: JdbcConnectionOptions, tableName: String, shardColumn: String, shard: Shard[S], rowMapper: (ResultSet) => T, fetchSize: Int = JdbcShardedReadOptions.DefaultFetchSize, numShards: Int = JdbcShardedReadOptions.DefaultNumShards) extends Product with Serializable

    A bag of options for the JDBC sharded read.

    A bag of options for the JDBC sharded read.

    connectionOptions

    Connection options

    tableName

    Name of a table or materialized view to read from

    shardColumn

    Column to shard by. Should ideally have evenly distributed values. Column type must have a corresponding com.spotify.scio.jdbc.sharded.Shard implementation.

    shard

    An implementation of the com.spotify.scio.jdbc.sharded.Shard trait which knows how to shard a column of a type S. Example of sharding by a column of type Long:

    sc.jdbcShardedSelect(getShardedReadOptions(opts), Shard.range[Long])
    rowMapper

    Function to map from a SQL java.sql.ResultSet to T

    fetchSize

    Amount of rows fetched per java.sql.ResultSet. Default value is 100000. To apply an unbounded fetch size set this parameter to -1

    numShards

    Number of shards to split the table into for reading. There is no guarantee that Beam will actually execute reads in parallel. It is up to Beam auto scaler to decide the level of parallelism to use (number of workers and threads per worker). But the behavior could be controlled with maxNumWorkers and numberOfWorkerHarnessThreads parameters (see more details about these parameters here). Defaults to 4

  2. final case class JdbcShardedSelect[T, S](readOptions: JdbcShardedReadOptions[T, S])(implicit evidence$1: Coder[T]) extends ScioIO[T] with Product with Serializable
  3. final class NumericRangeShard[T] extends RangeShard[T]
  4. final class PrefixShard[T] extends Shard[T]
  5. final case class PrefixShardQuery[T](prefix: T) extends ShardQuery with Product with Serializable
  6. final case class Range[A](lowerBound: A, upperBound: A) extends Product with Serializable
  7. trait RangeShard[T] extends Shard[T]
  8. final case class RangeShardQuery[T](range: Range[T], upperBoundInclusive: Boolean, quoteValues: Boolean) extends ShardQuery with Product with Serializable
  9. trait RangeShardStringCodec[T <: ShardString] extends Serializable
  10. final class RangeStringShard[T <: ShardString] extends RangeShard[T]
  11. trait Shard[T] extends Serializable
  12. sealed trait ShardQuery extends Serializable
  13. sealed trait ShardString extends Serializable

Value Members

  1. object JdbcShardedReadOptions extends Serializable
  2. object JdbcShardedSelect extends Serializable
  3. object NumericRangeShard extends Serializable
  4. object RangeShardStringCodec extends Serializable
  5. object Shard extends Serializable
  6. object ShardQuery extends Serializable
  7. object ShardString extends Serializable

Ungrouped