Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o] Verb: I can, know, understand, have knowledge.
Scio
Scio is a Scala API for Apache Beam and Google Cloud Dataflow inspired by Apache Spark and Scalding.
Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark.
Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam’s Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.
See Scio Scaladocs for current API documentation.
Getting help
Documentation
- Getting Started - current API documentation
- Scio, Scalding and Spark - comparison of these frameworks
- Runners - how Scio handles Beam runners and runner specific logic
- Scio data guideline - guideline for common problems
- Releases - Detailed release notes on new Scio releases
- FAQ - frequently asked questions
IO
- Avro - using Scio with Avro files
- BigQuery - using Scio with BigQuery in a type safe way
- Bigtable - using Scio with Bigtable
- Parquet - using Scio with Parquet files
- Protobuf - using Scio with Protobuf
Extras
Internals
- ScioIO - new IO system to simplify implementation and stubbing in
JobTest - OverrideTypeProvider - custom mappings for type-safe BigQuery
- Kryo - Kryo data serialization
- Coders - new Magnolia based Coders derivation
Further Readings
- Spotify Unwrapped: How We Brought You a Decade of Data
- Scio 0.7: a Deep Dive
- Big Data Processing at Spotify: The Road to Scio (Part 1)
- Big Data Processing at Spotify: The Road to Scio (Part 2)
- The world beyond batch: Streaming 101
- The world beyond batch: Streaming 102
- Dataflow/Beam & Spark: A Programming Model Comparison
- VLDB paper on the Dataflow Model
Presentations
- Scio in Depth - Apache Beam Summit, 2022
- Techbytes: Data Processing with Scio - Spotify Engineering Talk, 2019
- Techbytes: Handling Big Data at Spotify - Spotify Engineering Talk, 2019
- Scio - Big Data on Google Cloud with Scala and Scio - Apache Beam Summit London 2018 Talk
- Sorry - How Bieber broke Google Cloud at Spotify (slides) - Scala Up North 2017 Talk
- Scio - Moving to Google Cloud A Spotify Story (slides) - Philly ETE 2017 Talk
- Scio - A Scala API for Google Cloud Dataflow & Apache Beam (slides) - Scala by the Bay 2016 Talk
- From stream to recommendation with Cloud Pub/Sub and Cloud Dataflow - GCP NEXT 16 Talk
- Apache Beam Presentation Materials
Projects using or related to Scio
- Featran - A Scala feature transformation library for data science and machine learning
- Big Data Rosetta Code - Code snippets for solving common big data problems in various platforms. Inspired by Rosetta Code
- Ratatool - A tool for random data sampling and generation, which includes BigDiffy, a Scio library for pairwise field-level statistical diff of data sets (slides)
- Elitzur - Data validation for Scala and Scio
- Scio Koans - A collection of Scio exercises inspired by Ruby Koans and many others.
- scio-deep-dive - Building Scio from scratch step by step for an internal training session
- Klio - Large scale audio or binary file processing with Python and Apache Beam
- scala-flow - A lightweight Scala wrapper for Google Cloud Dataflow from Zendesk
- clj-headlights - Clojure API for Apache Beam, also from Zendesk
- datasplash - A Clojure API for Google Cloud Dataflow