Packages

package bigquery

Main package for BigQuery APIs. Import all.

import com.spotify.scio.bigquery._

There are two BigQuery dialects, legacy and standard. APIs that take a BigQuery query string as argument, e.g. com.spotify.scio.bigquery.client.BigQuery.query.rows, com.spotify.scio.bigquery.client.BigQuery.query.schema, com.spotify.scio.bigquery.client.BigQuery.getTypedRows and BigQueryType.fromQuery, automatically detects the query's dialect. To override this, start the query with either #legacysql or #standardsql comment line.

Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. bigquery
  2. TableReferenceSyntax
  3. TableRowSyntax
  4. SCollectionSyntax
  5. ScioContextSyntax
  6. AnyRef
  7. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Package Members

  1. package client
  2. package dynamic

    BigQuery package for dynamic table destinations.

    BigQuery package for dynamic table destinations. Import All.

    import com.spotify.scio.bigquery.dynamic._
  3. package syntax
  4. package types
  5. package validation

Type Members

  1. sealed trait BigQueryIO[T] extends ScioIO[T]
  2. final case class BigQuerySelect(sqlQuery: Query) extends BigQueryIO[TableRow] with Product with Serializable

    Get an SCollection for a BigQuery SELECT query.

    Get an SCollection for a BigQuery SELECT query. Both Legacy SQL and Standard SQL dialects are supported. By default the query dialect will be automatically detected. To override this behavior, start the query string with #legacysql or #standardsql.

  3. final case class BigQueryStorage(table: Table, selectedFields: List[String], rowRestriction: Option[String]) extends BigQueryIO[TableRow] with Product with Serializable

    Get an IO for a BigQuery table using the storage API.

  4. final case class BigQueryStorageSelect(sqlQuery: Query) extends BigQueryIO[TableRow] with Product with Serializable
  5. final case class BigQueryStorageTap(table: Table, readOptions: TableReadOptions) extends Tap[TableRow] with Product with Serializable

    Tap for BigQuery tables using storage api.

  6. type BigQueryTag = bigquery.types.BigQueryTag

    BigQuery tag for macro generated classes/fields.

  7. final case class BigQueryTap(table: TableReference) extends Tap[TableRow] with Product with Serializable

    Tap for BigQuery tables.

  8. final case class BigQueryTaps(self: Taps) extends Product with Serializable
  9. final case class BigQueryTypedSelect[T](reader: TypedRead[T], sqlQuery: Query, fromTableRow: (TableRow) => T)(implicit evidence$2: Coder[T]) extends BigQueryIO[T] with Product with Serializable
  10. final case class BigQueryTypedTable[T](reader: TypedRead[T], writer: Write[T], table: Table, fn: (GenericRecord, TableSchema) => T)(implicit evidence$6: Coder[T]) extends BigQueryIO[T] with Product with Serializable
  11. final case class BigQueryTypedTap[T](table: Table, fn: (GenericRecord, TableSchema) => T)(implicit evidence$1: Coder[T]) extends Tap[T] with Product with Serializable
  12. sealed trait ExtendedErrorInfo extends AnyRef
  13. class MockBigQuery extends AnyRef

    Mock BigQuery environment for integration test.

    Mock BigQuery environment for integration test.

    Use mockTable to feed data into live BigQuery service and queryResult to query them.

  14. class MockTable extends AnyRef

    A BigQuery table being mocked for test.

  15. final case class Query(underlying: String) extends Source with Product with Serializable

    A wrapper type Query which wraps a SQL String.

  16. sealed trait Source extends AnyRef
  17. sealed trait Table extends Source

    Table abstracts the multiple ways of referencing Bigquery tables.

    Table abstracts the multiple ways of referencing Bigquery tables. Tables can be referenced by a table spec String or by a table reference GTableReference.

    Example:

    val table = Table.Spec("bigquery-public-data:samples.shakespeare")
    sc.bigQueryTable(table)
      .filter(r => "hamlet".equals(r.getString("corpus")) && "Polonius".equals(r.getString("word")))
      .saveAsTextFile("./output.txt")
    sc.run()

    Or create a Table from a GTableReference:

    val tableReference = new TableReference
    tableReference.setProjectId("bigquery-public-data")
    tableReference.setDatasetId("samples")
    tableReference.setTableId("shakespeare")
    val table = Table.Ref(tableReference)

    A helper method is provided to replace the "$LATEST" placeholder in the table name to the latest common partition.

    val table = Table.Spec("some_project:some_data.some_table_$LATEST").latest()
  18. type TableRow = google.api.services.bigquery.model.TableRow

    Alias for BigQuery TableRow.

  19. final case class TableRowJsonIO(path: String) extends ScioIO[TableRow] with Product with Serializable

    Get an IO for a BigQuery TableRow JSON file.

  20. final case class TableRowJsonTap(path: String) extends Tap[TableRow] with Product with Serializable

    Tap for BigQuery TableRow JSON files.

  21. case class TimePartitioning(type: String, field: String = null, expirationMs: Long = 0, requirePartitionFilter: Boolean = false) extends Product with Serializable

    Scala wrapper for com.google.api.services.bigquery.model.TimePartitioning.

  22. type description = bigquery.types.description

    Annotation for BigQuery field description.

Deprecated Type Members

  1. final case class BigQueryTable(table: Table) extends BigQueryIO[TableRow] with Product with Serializable

    Get an IO for a BigQuery table.

    Get an IO for a BigQuery table.

    Annotations
    @deprecated
    Deprecated

    (Since version 0.9.2) use BigQueryTypedTable(table, Format.TableRow) instead

Value Members

  1. val BigQueryType: bigquery.types.BigQueryType.type

    Typed BigQuery annotations and converters.

  2. val CREATE_IF_NEEDED: CreateDisposition

    Alias for BigQuery CreateDisposition.

  3. val CREATE_NEVER: CreateDisposition

    Alias for BigQuery CreateDisposition.

  4. val WRITE_APPEND: WriteDisposition

    Alias for BigQuery WriteDisposition.

  5. val WRITE_EMPTY: WriteDisposition

    Alias for BigQuery WriteDisposition.

  6. val WRITE_TRUNCATE: WriteDisposition

    Alias for BigQuery WriteDisposition.

  7. implicit def bigQuerySCollectionBeamSchemaOps[T](sc: SCollection[T])(implicit arg0: ClassTag[T]): SCollectionBeamSchemaOps[T]
    Definition Classes
    SCollectionSyntax
  8. implicit def bigQuerySCollectionGenericRecordOps[T <: GenericRecord](sc: SCollection[T]): SCollectionGenericRecordOps[T]
    Definition Classes
    SCollectionSyntax
  9. implicit def bigQuerySCollectionTableRowOps[T <: TableRow](sc: SCollection[T]): SCollectionTableRowOps[T]
    Definition Classes
    SCollectionSyntax
  10. implicit def bigQuerySCollectionTypedOps[T <: HasAnnotation](sc: SCollection[T]): SCollectionTypedOps[T]
    Definition Classes
    SCollectionSyntax
  11. implicit def bigQueryScioContextOps(sc: ScioContext): ScioContextOps
    Definition Classes
    ScioContextSyntax
  12. implicit def bigQueryTableRefOps(r: TableReference): TableReferenceOps
    Definition Classes
    TableReferenceSyntax
  13. implicit def bigQueryTableRowOps(tr: TableRow): TableRowOps
    Definition Classes
    TableRowSyntax
  14. object BigQueryIO
  15. object BigQuerySelect extends Serializable
  16. object BigQueryStorage extends Serializable
  17. object BigQuerySysProps
    Annotations
    @registerSysProps()
  18. object BigQueryTable extends Serializable
  19. object BigQueryTaps extends Serializable
  20. object BigQueryTyped
  21. object BigQueryTypedSelect extends Serializable
  22. object BigQueryTypedTable extends Serializable
  23. object BigQueryUtil

    Utility for BigQuery data types.

  24. object Date

    Utility for BigQuery DATE type.

  25. object DateTime

    Utility for BigQuery DATETIME type.

  26. object ExtendedErrorInfo
  27. object MockBigQuery

    Companion object for MockBigQuery.

  28. object Numeric
  29. object StorageUtil

    Utility for BigQuery Storage API.

  30. object Table
  31. object TableRow

    Create a TableRow with Map-like syntax.

    Create a TableRow with Map-like syntax. For example:

    val r = TableRow("name" -> "Alice", "score" -> 100)
  32. object TableRowJsonIO extends Serializable
  33. object Time

    Utility for BigQuery TIME type.

  34. object Timestamp

    Utility for BigQuery TIMESTAMP type.

Inherited from TableReferenceSyntax

Inherited from TableRowSyntax

Inherited from SCollectionSyntax

Inherited from ScioContextSyntax

Inherited from AnyRef

Inherited from Any

Ungrouped