Packages

c

com.spotify.scio.jdbc.syntax

JdbcScioContextOps

final class JdbcScioContextOps extends AnyVal

Enhanced version of ScioContext with JDBC methods.

Source
ScioContextSyntax.scala
Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. JdbcScioContextOps
  2. AnyVal
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new JdbcScioContextOps(self: ScioContext)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    Any
  2. final def ##: Int
    Definition Classes
    Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def getClass(): Class[_ <: AnyVal]
    Definition Classes
    AnyVal → Any
  6. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  7. def jdbcSelect[T](connectionOptions: JdbcConnectionOptions, query: String, statementPreparator: (PreparedStatement) => Unit = ReadParam.DefaultStatementPreparator, fetchSize: Int = ReadParam.BeamDefaultFetchSize, outputParallelization: Boolean = ReadParam.DefaultOutputParallelization, dataSourceProviderFn: () => DataSource = ReadParam.DefaultDataSourceProviderFn, configOverride: (Read[T]) => Read[T] = ReadParam.defaultConfigOverride[T])(rowMapper: (ResultSet) => T)(implicit arg0: Coder[T]): SCollection[T]

    Get an SCollection for a JDBC query.

    Get an SCollection for a JDBC query.

    connectionOptions

    connection options

    query

    query string

    statementPreparator

    function to prepare a java.sql.PreparedStatement

    fetchSize

    use apache beam default fetch size if the value is -1

    outputParallelization

    reshuffle result to distribute it to all workers. Default to true.

    dataSourceProviderFn

    function to provide a custom javax.sql.DataSource

    configOverride

    function to override or replace a Read transform before applying it

    rowMapper

    function to map from a SQL java.sql.ResultSet to T

  8. def jdbcShardedSelect[T, S](readOptions: JdbcShardedReadOptions[T, S])(implicit arg0: Coder[T]): SCollection[T]

    Sharded JDBC read from a table or materialized view.

    Sharded JDBC read from a table or materialized view.

    readOptions

    The following parameters in the options class could be specified: shardColumn: the column to shard by. Must be of integer/long type ideally with evenly distributed values. numShards: number of shards to split the table into for reading. There is no guarantee that Beam will actually execute reads in parallel. It is up to Beam auto scaler to decide the level of parallelism to use (number of workers and threads per worker). But the behavior could be controlled with maxNumWorkers and numberOfWorkerHarnessThreads parameters (see more details about these parameters here). Defaults to 4. tableName: name of a table or materialized view to read from fetchSize: number of records to read from the JDBC source per one call to a database. Default value is 100,000. Set to -1 to make it unbounded. shard: An implementation of the com.spotify.scio.jdbc.sharded.Shard trait which knows how to shard a column of a type S. Example of sharding by a column of type Long:

    sc.jdbcShardedSelect(getShardedReadOptions(opts), Shard.range[Long])
  9. def toString(): String
    Definition Classes
    Any

Inherited from AnyVal

Inherited from Any

Ungrouped