Packages

  • package root
    Definition Classes
    root
  • package com
    Definition Classes
    root
  • package spotify
    Definition Classes
    com
  • package scio
    Definition Classes
    spotify
  • package testing
    Definition Classes
    scio
  • object JobTest

    Set up a Scio job for end-to-end unit testing.

    Set up a Scio job for end-to-end unit testing. To be used in a PipelineSpec. For example:

    import com.spotify.scio.testing._

class WordCountTest extends PipelineSpec {

  // Mock input data, mock distributed cache and expected result
  val inData = Seq("a b c d e", "a b a b")
  val distCache = Map(1 -> "Jan", 2 -> "Feb", 3 -> "Mar")
  val expected = Seq("a: 3", "b: 3", "c: 1", "d: 1", "e: 1")

  // Test specification
  "WordCount" should "work" in {
    JobTest("com.spotify.scio.examples.WordCount")
    // Or the type safe version
    // JobTest[com.spotify.scio.examples.WordCount.type]

      // Command line arguments
      .args("--input=in.txt", "--output=out.txt")

      // Mock input data
      .input(TextIO("in.txt"), inData)

      // Mock distributed cache
      .distCache(DistCacheIO("gs://dataflow-samples/samples/misc/months.txt"), distCache)

      // Verify output
      .output(TextIO("out.txt")) { actual => actual should containInAnyOrder (expected) }

      // Run job test
      .run()
  }
}
    Definition Classes
    testing
  • BeamOptions
  • Builder

class Builder extends AnyRef

Source
JobTest.scala
Linear Supertypes
AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. Builder
  2. AnyRef
  3. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. Protected

Instance Constructors

  1. new Builder(state: BuilderState)

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##: Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def args(newArgs: String*): Builder

    Feed command line arguments to the pipeline being tested.

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.CloneNotSupportedException]) @native()
  7. def counter(counter: Counter)(assertion: (Long) => Any): Builder

    Evaluate a Counter in the pipeline being tested.

    Evaluate a Counter in the pipeline being tested.

    counter

    counter to be evaluated

    assertion

    assertion for the counter result's committed value

  8. def counters(assertion: (Map[MetricName, Long]) => Any): Builder

    Evaluate all Counters in the pipeline being tested.

    Evaluate all Counters in the pipeline being tested.

    assertion

    assertion on the collection of all job counters' committed values

  9. def distCache[T](key: DistCacheIO[T], value: T): Builder

    Feed an distributed cache to the pipeline being tested.

    Feed an distributed cache to the pipeline being tested. Note that DistCacheIO[T] must match the one used inside the pipeline, e.g. DistCacheIO[Set[String]]("dc.txt") with sc.distCache("dc.txt")(f => scala.io.Source.fromFile(f).getLines().toSet).

    value

    mock value, must be serializable.

  10. def distCacheFunc[T](key: DistCacheIO[T], initFn: () => T): Builder

    Feed an distributed cache to the pipeline being tested.

    Feed an distributed cache to the pipeline being tested. Note that DistCacheIO[T] must match the one used inside the pipeline, e.g. DistCacheIO[Set[String]]("dc.txt") with sc.distCache("dc.txt")(f => scala.io.Source.fromFile(f).getLines().toSet).

    initFn

    init function, must be serializable.

  11. def distribution(distribution: Distribution)(assertion: (DistributionResult) => Any): Builder

    Evaluate a Distribution in the pipeline being tested.

    Evaluate a Distribution in the pipeline being tested.

    distribution

    distribution to be evaluated

    assertion

    assertion for the distribution result's committed value

  12. def distributions(assertion: (Map[MetricName, DistributionResult]) => Any): Builder

    Evaluate all Distributions in the pipeline being tested.

    Evaluate all Distributions in the pipeline being tested.

    assertion

    assertion on the collection of all job distribution results' committed values

  13. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  14. def equals(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef → Any
  15. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.Throwable])
  16. def gauge(gauge: Gauge)(assertion: (GaugeResult) => Any): Builder

    Evaluate a Gauge in the pipeline being tested.

    Evaluate a Gauge in the pipeline being tested.

    gauge

    gauge to be evaluated

    assertion

    assertion for the gauge result's committed value

  17. def gauges(assertion: (Map[MetricName, GaugeResult]) => Any): Builder

    Evaluate all Gauges in the pipeline being tested.

    Evaluate all Gauges in the pipeline being tested.

    assertion

    assertion on the collection of all job gauge results' committed values

  18. final def getClass(): Class[_ <: AnyRef]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  19. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  20. def input[T](io: ScioIO[T], value: Iterable[T])(implicit arg0: Coder[T]): Builder

    Feed an input in the form of a raw Iterable[T] to the pipeline being tested.

    Feed an input in the form of a raw Iterable[T] to the pipeline being tested. Note that ScioIO[T] must match the one used inside the pipeline, e.g. AvroIO[MyRecord]("in.avro") with sc.avroFile[MyRecord]("in.avro").

  21. def inputStream[T](io: ScioIO[T], stream: TestStream[T])(implicit arg0: Coder[T]): Builder

    Feed an input in the form of a PTransform[PBegin, PCollection[T]] to the pipeline being tested.

    Feed an input in the form of a PTransform[PBegin, PCollection[T]] to the pipeline being tested. Note that PTransform inputs may not be supported for all ScioIO[T] types.

  22. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  23. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  24. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  25. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  26. def output[T](io: ScioIO[T])(assertion: (SCollection[T]) => Any): Builder

    Evaluate an output of the pipeline being tested.

    Evaluate an output of the pipeline being tested. Note that TestIO[T] must match the one used inside the pipeline, e.g. AvroIO[MyRecord]("out.avro") with data.saveAsAvroFile("out.avro") where data is of type SCollection[MyRecord].

    assertion

    assertion for output data. See SCollectionMatchers for available matchers on an SCollection.

  27. def run(): Unit

    Run the pipeline with test wiring.

  28. def setUp(): Unit

    Set up test wiring.

    Set up test wiring. Use this only if you have custom pipeline wiring and are bypassing run. Make sure tearDown is called afterwards.

  29. final def synchronized[T0](arg0: => T0): T0
    Definition Classes
    AnyRef
  30. def tearDown(): Unit

    Tear down test wiring.

    Tear down test wiring. Use this only if you have custom pipeline wiring and are bypassing run. Make sure setUp is called before.

  31. val testId: String

    Test ID for input and output wiring.

  32. def toString(): String
    Definition Classes
    Builder → AnyRef → Any
  33. def transformOverride(xformOverride: PTransformOverride): Builder

    Replace a PTransform in the pipeline being tested.

    Replace a PTransform in the pipeline being tested.

    xformOverride

    A PTransformOverride

  34. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  35. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException])
  36. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws(classOf[java.lang.InterruptedException]) @native()

Inherited from AnyRef

Inherited from Any

Ungrouped