Spanner

Scio supports reading and writing from Google Cloud Spanner.

Read from Spanner

Reads from Spanner occur via a query with spannerQuery or for an entire table with spannerTable. Both return an SCollection of Struct:

To read with a query:

import com.spotify.scio._
import com.spotify.scio.spanner._
import com.spotify.scio.values.SCollection
import org.apache.beam.sdk.io.gcp.spanner.SpannerConfig
import com.google.cloud.spanner.Struct

val config: SpannerConfig = SpannerConfig
    .create()
    .withProjectId("someProject")
    .withDatabaseId("someDatabase")
    .withInstanceId("someInstance")

val sc: ScioContext = ???
val queryStructs: SCollection[Struct] = sc.spannerQuery(config, "SELECT a, b FROM table WHERE c > 5")

To read an entire table:

import com.spotify.scio._
import com.spotify.scio.spanner._
import com.spotify.scio.values.SCollection
import org.apache.beam.sdk.io.gcp.spanner.SpannerConfig
import com.google.cloud.spanner.Struct

val config: SpannerConfig = SpannerConfig
  .create()
  .withProjectId("someProject")
  .withDatabaseId("someDatabase")
  .withInstanceId("someInstance")

val sc: ScioContext = ???
val tableStructs: SCollection[Struct] = sc.spannerTable(config, "table", columns=List("a", "b"))

Write to Spanner

An SCollection containing Mutation instances can be written to Spanner via saveAsSpanner:

import com.spotify.scio.spanner._
import com.spotify.scio.values.SCollection
import org.apache.beam.sdk.io.gcp.spanner.SpannerConfig
import com.google.cloud.spanner.Mutation

val config: SpannerConfig = SpannerConfig
  .create()
  .withProjectId("someProject")
  .withDatabaseId("someDatabase")
  .withInstanceId("someInstance")

val mutations: SCollection[Mutation] = ???
mutations.saveAsSpanner(config)