Ecclesiastical Latin IPA: /ˈʃi.o/, [ˈʃiː.o], [ˈʃi.i̯o]
Verb: I can, know, understand, have knowledge.
0.3.0 and future versions depend on Apache Beam (
org.apache.beam) while earlier versions depend on Google Cloud Dataflow SDK (
com.google.cloud.dataflow). See this page for a list of breaking changes.
Getting Started is the best place to start with Scio. If you are new to Apache Beam and distributed data processing, check out the Beam Programming Guide first for a detailed explanation of the Beam programming model and concepts. If you have experience with other Scala data processing libraries, check out this comparison between Scio, Scalding and Spark. Finally check out this document about the relationship between Scio, Beam and Dataflow.
Example Scio pipelines and tests can be found under scio-examples. A lot of them are direct ports from Beam’s Java examples. See this page for some of them with side-by-side explanation. Also see Big Data Rosetta Code for common data processing code snippets in Scio, Scalding and Spark.
See Scio Scaladocs for current API documentation.
See scio-contrib for community-supported add-ons for Scio.
- Getting Started - current API documentation
- Scio REPL - tutorial for the interactive Scio REPL
- Scio, Beam and Dataflow - how Scio concepts map to Beam and Dataflow
- Scio, Scalding and Spark - comparison of these frameworks
- Runners - how Scio handles Beam runners and runner specific logic
- Scio data guideline - guideline for common problems
- Apache Beam - notes on Apache Beam compatibility
- Changelog - breaking changes in each release
- FAQ - frequently asked questions
- Powered By - see who is using Scio in production
- Type safe BigQuery - tutorial for the type safe BigQuery API
- HDFS - using Scio with HDFS files
- Bigtable - using Scio with Bigtable
- Avro - using Scio with Avro files
- Protobuf - using Scio with Protobuf
- Parquet - using Scio with Parquet files
- ScioIO - new IO system to simplify implementation and stubbing in
- OverrideTypeProvider - custom mappings for type-safe BigQuery
- Kryo - Kryo data serialization
- Coders - new Magnolia based Coders derivation
- Big Data Processing at Spotify: The Road to Scio (Part 1)
- Big Data Processing at Spotify: The Road to Scio (Part 2)
- The world beyond batch: Streaming 101
- The world beyond batch: Streaming 102
- Dataflow/Beam & Spark: A Programming Model Comparison
- VLDB paper on the Dataflow Model
- Scio - Big Data on Google Cloud with Scala and Scio - Apache Beam Summit London 2018 Talk
- Sorry - How Bieber broke Google Cloud at Spotify (slides) - Scala Up North 2017 Talk
- Scio - Moving to Google Cloud A Spotify Story (slides) - Philly ETE 2017 Talk
- Scio - A Scala API for Google Cloud Dataflow & Apache Beam (slides) - Scala by the Bay 2016 Talk
- From stream to recommendation with Cloud Pub/Sub and Cloud Dataflow - GCP NEXT 16 Talk
- Apache Beam Presentation Materials
- scio-contrib - Community-supported add-ons for Scio
- Featran - A Scala feature transformation library for data science and machine learning
- Big Data Rosetta Code - Code snippets for solving common big data problems in various platforms. Inspired by Rosetta Code
- Ratatool - A tool for random data sampling and generation, which includes BigDiffy, a Scio library for pairwise field-level statistical diff of data sets (slides)
- scio-deep-dive - Building Scio from scratch step by step for an internal training session
- scala-flow - A lightweight Scala wrapper for Google Cloud Dataflow from Zendesk
- clj-headlights - Clojure API for Apache Beam, also from Zendesk
- datasplash - A Clojure API for Google Cloud Dataflow