object MDL extends SettingsBuilder with Serializable

Transform a column of continuous labelled features to n columns of binned categorical features. The optimum number of bins is computed using Minimum Description Length (MDL), which is an entropy measurement between the values and the targets.

The transformer expects an MDLRecord where the first field is a label and the second value is the scalar that will be transformed into buckets.

MDL is an iterative algorithm so all of the data needed to compute the buckets will be pulled into memory. If you run into memory issues the sampleRate parameter should be lowered.

References:

  • Fayyad, U., & Irani, K. (1993). "Multi-interval discretization of continuous-valued attributes for classification learning."
  • https://github.com/sramirez/spark-MDLP-discretization
Source
MDL.scala
Linear Supertypes
Serializable, Serializable, SettingsBuilder, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MDL
  2. Serializable
  3. Serializable
  4. SettingsBuilder
  5. AnyRef
  6. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. def apply[T](name: String, sampleRate: Double = 1.0, stoppingCriterion: Double = DefaultStoppingCriterion, minBinPercentage: Double = DefaultMinBinPercentage, maxBins: Int = DefaultMaxBins, seed: Int = Random.nextInt())(implicit arg0: ClassTag[T]): Transformer[MDLRecord[T], B[T], C]

    Create an MDL Instance.

    Create an MDL Instance.

    sampleRate

    percentage of records to keep to compute the buckets

    stoppingCriterion

    stopping criterion for MDL

    minBinPercentage

    minimum percent of total data allowed in a single bin

    maxBins

    maximum number of thresholds per feature

    seed

    seed for the sampler

  5. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  6. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  10. def fromSettings(setting: Settings): Transformer[MDLRecord[String], B[String], C]

    Create a new MDL from a settings object

    Create a new MDL from a settings object

    setting

    Settings object

    Definition Classes
    MDL → SettingsBuilder
  11. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  12. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  13. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  14. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  15. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  16. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  17. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  18. def toString(): String
    Definition Classes
    AnyRef → Any
  19. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()

Inherited from Serializable

Inherited from Serializable

Inherited from SettingsBuilder

Inherited from AnyRef

Inherited from Any

Ungrouped