Packages

final class ScioContextOps extends AnyVal

Enhanced version of ScioContext with Bigtable methods.

Source
ScioContextSyntax.scala
Linear Supertypes
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. ScioContextOps
  2. AnyVal
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new ScioContextOps(self: ScioContext)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    Any
  2. final def ##: Int
    Definition Classes
    Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRanges: Seq[ByteKeyRange], rowFilter: RowFilter, maxBufferElementCount: Option[Int]): SCollection[Row]

    Get an SCollection for a Bigtable table.

  6. def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRanges: Seq[ByteKeyRange], rowFilter: RowFilter): SCollection[Row]

    Get an SCollection for a Bigtable table.

  7. def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter, maxBufferElementCount: Option[Int]): SCollection[Row]

    Get an SCollection for a Bigtable table.

  8. def bigtable(bigtableOptions: BigtableOptions, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter): SCollection[Row]

    Get an SCollection for a Bigtable table.

  9. def bigtable(projectId: String, instanceId: String, tableId: String, keyRanges: Seq[ByteKeyRange] = BigtableRead.ReadParam.DefaultKeyRanges, rowFilter: RowFilter = BigtableRead.ReadParam.DefaultRowFilter, maxBufferElementCount: Option[Int] = BigtableRead.ReadParam.DefaultMaxBufferElementCount): SCollection[Row]

    Get an SCollection for a Bigtable table.

  10. def bigtable(projectId: String, instanceId: String, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter, maxBufferElementCount: Option[Int]): SCollection[Row]

    Get an SCollection for a Bigtable table.

  11. def bigtable(projectId: String, instanceId: String, tableId: String, keyRange: ByteKeyRange, rowFilter: RowFilter): SCollection[Row]

    Get an SCollection for a Bigtable table.

  12. def ensureTables(bigtableOptions: BigtableOptions, tablesAndColumnFamilies: Map[String, Iterable[String]]): Unit
  13. def ensureTables(bigtableOptions: BigtableOptions, tablesAndColumnFamilies: Map[String, Iterable[String]], createDisposition: CreateDisposition): Unit

    Ensure that tables and column families exist.

    Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.

    tablesAndColumnFamilies

    A map of tables and column families. Keys are table names. Values are a list of column family names.

  14. def ensureTables(projectId: String, instanceId: String, tablesAndColumnFamilies: Map[String, Iterable[String]]): Unit
  15. def ensureTables(projectId: String, instanceId: String, tablesAndColumnFamilies: Map[String, Iterable[String]], createDisposition: CreateDisposition): Unit

    Ensure that tables and column families exist.

    Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.

    tablesAndColumnFamilies

    A map of tables and column families. Keys are table names. Values are a list of column family names.

  16. def ensureTablesWithExpiration(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]]): Unit
  17. def ensureTablesWithExpiration(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]], createDisposition: CreateDisposition): Unit

    Ensure that tables and column families exist.

    Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.

    tablesAndColumnFamiliesWithExpiration

    A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired cell expiration. Cell expiration is the duration before which garbage collection of a cell may occur. Note: minimum granularity is second.

  18. def ensureTablesWithExpiration(projectId: String, instanceId: String, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]]): Unit
  19. def ensureTablesWithExpiration(projectId: String, instanceId: String, tablesAndColumnFamiliesWithExpiration: Map[String, Iterable[(String, Option[Duration])]], createDisposition: CreateDisposition): Unit

    Ensure that tables and column families exist.

    Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.

    tablesAndColumnFamiliesWithExpiration

    A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired cell expiration. Cell expiration is the duration before which garbage collection of a cell may occur. Note: minimum granularity is second.

  20. def ensureTablesWithGcRules(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithGcRule: Map[String, Iterable[(String, Option[GcRule])]]): Unit
  21. def ensureTablesWithGcRules(bigtableOptions: BigtableOptions, tablesAndColumnFamiliesWithGcRule: Map[String, Iterable[(String, Option[GcRule])]], createDisposition: CreateDisposition): Unit

    Ensure that tables and column families exist.

    Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.

    tablesAndColumnFamiliesWithGcRule

    A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired cell expiration. Cell expiration is the duration before which garbage collection of a cell may occur. Note: minimum granularity is second.

  22. def ensureTablesWithGcRules(projectId: String, instanceId: String, tablesAndColumnFamiliesWithGcRules: Map[String, Iterable[(String, Option[GcRule])]]): Unit
  23. def ensureTablesWithGcRules(projectId: String, instanceId: String, tablesAndColumnFamiliesWithGcRules: Map[String, Iterable[(String, Option[GcRule])]], createDisposition: CreateDisposition): Unit

    Ensure that tables and column families exist.

    Ensure that tables and column families exist. Checks for existence of tables or creates them if they do not exist. Also checks for existence of column families within each table and creates them if they do not exist.

    tablesAndColumnFamiliesWithGcRules

    A map of tables and column families. Keys are table names. Values are a list of column family names along with the desired GcRule.

  24. def getBigtableClusterSizes(projectId: String, instanceId: String): Map[String, Int]

    Get size of all clusters for specified Bigtable instance.

    Get size of all clusters for specified Bigtable instance.

    returns

    map of clusterId to its number of nodes

  25. def getClass(): Class[_ <: AnyVal]
    Definition Classes
    AnyVal → Any
  26. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  27. def toString(): String
    Definition Classes
    Any
  28. def updateNumberOfBigtableNodes(bigtableOptions: BigtableOptions, numberOfNodes: Int, clusterNames: Set[String], sleepDuration: Duration): Unit

    Updates given clusters within the specified Bigtable instance to a specified number of nodes.

    Updates given clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.

    clusterNames

    Names of clusters to be updated, all if empty

    sleepDuration

    How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional

  29. def updateNumberOfBigtableNodes(bigtableOptions: BigtableOptions, numberOfNodes: Int, sleepDuration: Duration): Unit

    Updates all clusters within the specified Bigtable instance to a specified number of nodes.

    Updates all clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.

    sleepDuration

    How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional

  30. def updateNumberOfBigtableNodes(projectId: String, instanceId: String, numberOfNodes: Int, clusterNames: Set[String], sleepDuration: Duration): Unit

    Updates given clusters within the specified Bigtable instance to a specified number of nodes.

    Updates given clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.

    clusterNames

    Names of clusters to be updated, all if empty

    sleepDuration

    How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional

  31. def updateNumberOfBigtableNodes(projectId: String, instanceId: String, numberOfNodes: Int, sleepDuration: Duration = DefaultSleepDuration): Unit

    Updates all clusters within the specified Bigtable instance to a specified number of nodes.

    Updates all clusters within the specified Bigtable instance to a specified number of nodes. Useful for increasing the number of nodes at the beginning of a job and decreasing it at the end to lower costs yet still get high throughput during bulk ingests/dumps.

    sleepDuration

    How long to sleep after updating the number of nodes. Google recommends at least 20 minutes before the new nodes are fully functional

Inherited from AnyVal

Inherited from Any

Ungrouped