org.apache.spark.mllib.feature

Class IDF

Object

org.apache.spark.mllib.feature.IDF

public class IDF
extends Object

:: Experimental :: Inverse document frequency (IDF). The standard formulation is used: idf = log((m + 1) / (d(t) + 1)), where m is the total number of documents and d(t) is the number of documents that contain term t.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	IDF.DocumentFrequencyAggregator Document frequency aggregator.

Constructor Summary

Constructors

Constructor and Description
IDF()

Method Summary

Methods

Modifier and Type	Method and Description
IDFModel	fit(JavaRDD<Vector> dataset) Computes the inverse document frequency.
IDFModel	fit(RDD<Vector> dataset) Computes the inverse document frequency.

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IDF

public IDF()

Method Detail

fit

public IDFModel fit(RDD<Vector> dataset)

Computes the inverse document frequency.

Parameters:

dataset - an RDD of term frequency vectors

fit

public IDFModel fit(JavaRDD<Vector> dataset)

Computes the inverse document frequency.

Parameters:

dataset - a JavaRDD of term frequency vectors