final class ScioContextOps extends AnyVal
Enhanced version of ScioContext with BigQuery methods.
- Source
- ScioContextSyntax.scala
- Alphabetic
- By Inheritance
- ScioContextOps
- AnyVal
- Any
- Hide All
- Show All
- Public
- Protected
Instance Constructors
- new ScioContextOps(self: ScioContext)
Value Members
- final def !=(arg0: Any): Boolean
- Definition Classes
- Any
- final def ##: Int
- Definition Classes
- Any
- final def ==(arg0: Any): Boolean
- Definition Classes
- Any
- final def asInstanceOf[T0]: T0
- Definition Classes
- Any
- def bigQuerySelect(sqlQuery: Query): SCollection[TableRow]
Get an SCollection for a BigQuery SELECT query.
Get an SCollection for a BigQuery SELECT query. Both Legacy SQL and Standard SQL dialects are supported. By default the query dialect will be automatically detected. To override this behavior, start the query string with
#legacysql
or#standardsql
. - def bigQuerySelect(sqlQuery: Query, flattenResults: Boolean): SCollection[TableRow]
Get an SCollection for a BigQuery SELECT query.
Get an SCollection for a BigQuery SELECT query. Both Legacy SQL and Standard SQL dialects are supported. By default the query dialect will be automatically detected. To override this behavior, start the query string with
#legacysql
or#standardsql
. - def bigQueryStorage(query: Query): SCollection[TableRow]
Get an SCollection for a BigQuery SELECT query using the storage API.
Get an SCollection for a BigQuery SELECT query using the storage API.
- query
SQL query
- def bigQueryStorage(table: Table, selectedFields: List[String] = BigQueryStorage.ReadParam.DefaultSelectFields, rowRestriction: String = null): SCollection[TableRow]
Get an SCollection for a BigQuery table using the storage API.
Get an SCollection for a BigQuery table using the storage API.
- selectedFields
names of the fields in the table that should be read. If empty, all fields will be read. If the specified field is a nested field, all the sub-fields in the field will be selected. Fields will always appear in the generated class in the same order as they appear in the table, regardless of the order specified in selectedFields.
- rowRestriction
SQL text filtering statement, similar ti a WHERE clause in a query. Currently, we support combinations of predicates that are a comparison between a column and a constant value in SQL statement. Aggregates are not supported. For example:
"a > DATE '2014-09-27' AND (b > 5 AND c LIKE 'date')"
- def bigQueryTable[F](table: Table, format: Format[F])(implicit arg0: Coder[F]): SCollection[F]
Get an SCollection for a BigQuery table using the specified Format.
Get an SCollection for a BigQuery table using the specified Format.
Reading records as GenericRecord **should** offer better performance over TableRow records.
Note: When using
Format.GenericRecord
Bigquery types DATE, TIME and DATETIME are read as STRING. - def bigQueryTable(table: Table): SCollection[TableRow]
Get an SCollection for a BigQuery table.
- def getClass(): Class[_ <: AnyVal]
- Definition Classes
- AnyVal → Any
- final def isInstanceOf[T0]: Boolean
- Definition Classes
- Any
- def tableRowJsonFile(path: String, compression: Compression = TableRowJsonIO.ReadParam.DefaultCompression, emptyMatchTreatment: EmptyMatchTreatment = TableRowJsonIO.ReadParam.DefaultEmptyMatchTreatment, suffix: String = TableRowJsonIO.ReadParam.DefaultSuffix): SCollection[TableRow]
Get an SCollection for a BigQuery TableRow JSON file.
- def toString(): String
- Definition Classes
- Any
- def typedBigQuery[T <: HasAnnotation](newSource: Option[Source])(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
Get a typed SCollection for BigQuery Table or a SELECT query using the Storage API.
- def typedBigQuery[T <: HasAnnotation](newSource: Source)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
- def typedBigQuery[T <: HasAnnotation]()(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
- def typedBigQueryStorage[T <: HasAnnotation](table: Table, selectedFields: List[String], rowRestriction: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
- def typedBigQueryStorage[T <: HasAnnotation](table: Table, rowRestriction: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
- def typedBigQueryStorage[T <: HasAnnotation](rowRestriction: String)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
- def typedBigQueryStorage[T <: HasAnnotation](table: Table)(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
- def typedBigQueryStorage[T <: HasAnnotation]()(implicit arg0: scala.reflect.api.JavaUniverse.TypeTag[T], arg1: Coder[T]): SCollection[T]
Get a typed SCollection for a BigQuery storage API.
Get a typed SCollection for a BigQuery storage API.
Note that
T
must be annotated with BigQueryType.fromStorage or BigQueryType.fromQuery