final class JdbcScioContextOps extends AnyVal
- Alphabetic
- By Inheritance
- JdbcScioContextOps
- AnyVal
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new JdbcScioContextOps(self: ScioContext)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- Any
- final def ##: Int
- Definition Classes
- Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def getClass(): Class[_ <: AnyVal]
- Definition Classes
- AnyVal → Any
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def jdbcSelect[T](connectionOptions: JdbcConnectionOptions, query: String, statementPreparator: (PreparedStatement) => Unit = ReadParam.DefaultStatementPreparator, fetchSize: Int = ReadParam.BeamDefaultFetchSize, outputParallelization: Boolean = ReadParam.DefaultOutputParallelization, dataSourceProviderFn: () => DataSource = ReadParam.DefaultDataSourceProviderFn, configOverride: (Read[T]) => Read[T] = ReadParam.defaultConfigOverride[T])(rowMapper: (ResultSet) => T)(implicit arg0: Coder[T]): SCollection[T]
Get an SCollection for a JDBC query.
Get an SCollection for a JDBC query.
- connectionOptions
connection options
- query
query string
- statementPreparator
function to prepare a java.sql.PreparedStatement
- fetchSize
use apache beam default fetch size if the value is -1
- outputParallelization
reshuffle result to distribute it to all workers. Default to true.
- dataSourceProviderFn
function to provide a custom javax.sql.DataSource
- configOverride
function to override or replace a Read transform before applying it
- rowMapper
function to map from a SQL java.sql.ResultSet to
T
- def jdbcShardedSelect[T, S](readOptions: JdbcShardedReadOptions[T, S])(implicit arg0: Coder[T]): SCollection[T]
Sharded JDBC read from a table or materialized view.
Sharded JDBC read from a table or materialized view.
- readOptions
The following parameters in the options class could be specified: shardColumn: the column to shard by. Must be of integer/long type ideally with evenly distributed values. numShards: number of shards to split the table into for reading. There is no guarantee that Beam will actually execute reads in parallel. It is up to Beam auto scaler to decide the level of parallelism to use (number of workers and threads per worker). But the behavior could be controlled with maxNumWorkers and numberOfWorkerHarnessThreads parameters (see more details about these parameters here). Defaults to 4. tableName: name of a table or materialized view to read from fetchSize: number of records to read from the JDBC source per one call to a database. Default value is 100,000. Set to -1 to make it unbounded. shard: An implementation of the com.spotify.scio.jdbc.sharded.Shard trait which knows how to shard a column of a type S. Example of sharding by a column of type Long:
sc.jdbcShardedSelect(getShardedReadOptions(opts), Shard.range[Long])
- def toString(): String
- Definition Classes
- Any