Datastore

Scio supports Google Datastore via Beam’s DatastoreIO.

Magnolify’s EntityType (available as part of the magnolify-datastore artifact) provides automatically-derived mappings between Datastore’s Entity and scala case classes. See full documentation here and an example usage here.

Reads

Read an SCollection of com.google.datastore.v1.Entity from Datastore with datastore:

import com.spotify.scio._
import com.spotify.scio.values.SCollection
import com.spotify.scio.datastore._
import com.google.datastore.v1.{Entity, Query}

val sc: ScioContext = ???

val projectId: String = ???
val query: Query = Query.getDefaultInstance
val entities: SCollection[Entity] = sc.datastore(projectId, query)

Writes

Write a collection of

saveAsDatastore

import com.spotify.scio._
import com.spotify.scio.values.SCollection
import com.spotify.scio.datastore._
import com.google.datastore.v1.{Entity, Query}

val projectId: String = ???
val entities: SCollection[Entity] = ???
entities.saveAsDatastore(projectId)