final class ScioContextOps extends AnyVal
Enhanced version of ScioContext with Bigtable methods.
- Source
- ScioContextSyntax.scala
- Alphabetic
- By Inheritance
- ScioContextOps
- AnyVal
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ScioContextOps(self: ScioContext)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- Any
- final def ##: Int
- Definition Classes
- Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRanges: Seq[ByteKeyRange], rowFilter: RowFilter, maxBufferElementCount: Option[Int]): SCollection[Row]
Get an SCollection for a Bigtable table.
- def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRanges: Seq[ByteKeyRange], rowFilter: RowFilter): SCollection[Row]
Get an SCollection for a Bigtable table.
- def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter, maxBufferElementCount: Option[Int]): SCollection[Row]
Get an SCollection for a Bigtable table.
- def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter): SCollection[Row]
Get an SCollection for a Bigtable table.
- def bigtable(projectId: String, instanceId: String, tableId: String, keyRanges: Seq[ByteKeyRange] = BigtableRead.ReadParam.DefaultKeyRanges, rowFilter: RowFilter = BigtableRead.ReadParam.DefaultRowFilter, maxBufferElementCount: Option[Int] = BigtableRead.ReadParam.DefaultMaxBufferElementCount): SCollection[Row]
Get an SCollection for a Bigtable table.
- def bigtable(projectId: String, instanceId: String, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter, maxBufferElementCount: Option[Int]): SCollection[Row]
Get an SCollection for a Bigtable table.
- def bigtable(projectId: String, instanceId: String, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter): SCollection[Row]
Get an SCollection for a Bigtable table.
- def ensureTables(bigtableOptions: BigtableOptions, tablesAndColumnFamilies: Map[String, Iterable[String]]): Unit
- def ensureTables(bigtableOptions: BigtableOptions, tablesAndColumnFamilies: Map[String, Iterable[String]], createDisposition: CreateDisposition): Unit
Ensure that tables and column families exist.
Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.
- tablesAndColumnFamilies
A map of tables and column families. Keys are table names. Values are a list of column family names.
- def ensureTables(projectId: String, instanceId: String, tablesAndColumnFamilies: Map[String, Iterable[String]]): Unit
- def ensureTables(projectId: String, instanceId: String, tablesAndColumnFamilies: Map[String, Iterable[String]], createDisposition: CreateDisposition): Unit
Ensure that tables and column families exist.
Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.
- tablesAndColumnFamilies
A map of tables and column families. Keys are table names. Values are a list of column family names.
- def ensureTablesWithExpiration(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]]): Unit
- def ensureTablesWithExpiration(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]], createDisposition: CreateDisposition): Unit
Ensure that tables and column families exist.
Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.
- tablesAndColumnFamiliesWithExpiration
A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired cell expiration. Cell expiration is the duration before which garbage collection of a cell may occur. Note: minimum granularity is second.
- def ensureTablesWithExpiration(projectId: String, instanceId: String, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]]): Unit
- def ensureTablesWithExpiration(projectId: String, instanceId: String, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]], createDisposition: CreateDisposition): Unit
Ensure that tables and column families exist.
Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.
- tablesAndColumnFamiliesWithExpiration
A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired cell expiration. Cell expiration is the duration before which garbage collection of a cell may occur. Note: minimum granularity is second.
- def ensureTablesWithGcRules(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithGcRule: Map[String, Iterable[(String, Option[GcRule])]]): Unit
- def ensureTablesWithGcRules(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithGcRule: Map[String, Iterable[(String, Option[GcRule])]], createDisposition: CreateDisposition): Unit
Ensure that tables and column families exist.
Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.
- tablesAndColumnFamiliesWithGcRule
A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired cell expiration. Cell expiration is the duration before which garbage collection of a cell may occur. Note: minimum granularity is second.
- def ensureTablesWithGcRules(projectId: String, instanceId: String, tablesAndColumnFamiliesWithGcRules: Map[String, Iterable[(String, Option[GcRule])]]): Unit
- def ensureTablesWithGcRules(projectId: String, instanceId: String, tablesAndColumnFamiliesWithGcRules: Map[String, Iterable[(String, Option[GcRule])]], createDisposition: CreateDisposition): Unit
Ensure that tables and column families exist.
Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.
- tablesAndColumnFamiliesWithGcRules
A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired GcRule.
- def getBigtableClusterSizes(projectId: String, instanceId: String): Map[String, Int]
Get size of all clusters for specified Bigtable instance.
Get size of all clusters for specified Bigtable instance.
- returns
map of clusterId to its number of nodes
- def getClass(): Class[_ <: AnyVal]
- Definition Classes
- AnyVal → Any
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def toString(): String
- Definition Classes
- Any
- def updateNumberOfBigtableNodes(bigtableOptions: BigtableOptions, numberOfNodes: Int, clusterNames: Set[String], sleepDuration: Duration): Unit
Updates given clusters within the specified Bigtable instance to a specified number of nodes.
Updates given clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.
- clusterNames
Names of clusters to be updated, all if empty
- sleepDuration
How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional
- def updateNumberOfBigtableNodes(bigtableOptions: BigtableOptions, numberOfNodes: Int, sleepDuration: Duration): Unit
Updates all clusters within the specified Bigtable instance to a specified number of nodes.
Updates all clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.
- sleepDuration
How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional
- def updateNumberOfBigtableNodes(projectId: String, instanceId: String, numberOfNodes: Int, clusterNames: Set[String], sleepDuration: Duration): Unit
Updates given clusters within the specified Bigtable instance to a specified number of nodes.
Updates given clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.
- clusterNames
Names of clusters to be updated, all if empty
- sleepDuration
How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional
- def updateNumberOfBigtableNodes(projectId: String, instanceId: String, numberOfNodes: Int, sleepDuration: Duration = DefaultSleepDuration): Unit
Updates all clusters within the specified Bigtable instance to a specified number of nodes.
Updates all clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.
- sleepDuration
How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional