skoot.feature_selection.NearZeroVarianceFilter

class skoot.feature_selection.NearZeroVarianceFilter(cols=None, freq_cut=19.0, as_df=True)

Identify near zero variance predictors.

Diagnose and remove any features that have one unique value (i.e., are zero variance predictors) or that are have both of the following characteristics: they have very few unique values relative to the number of samples and the ratio of the frequency of the most common value to the frequency of the second most common value is large.

A note of caution: if you attempt to run this over large, continuous data, it might take a long time. Since for each column in cols it will compute value_counts, applying to continuous data could be a bad idea.

Parameters:

cols : array-like, shape=(n_features,), optional (default=None)

The names of the columns on which to apply the transformation. If no column names are provided, the transformer will be fit on the entire frame. Note that the transformation will also only apply to the specified columns, and any other non-specified columns will still be present after transformation.

freq_cut : float, optional (default=95/5)

The cutoff for the ratio of the most common value to the second most common value. That is, if the frequency of the most common value is >= freq_cut times the frequency of the second most, the feature will be dropped.

as_df : bool, optional (default=True)

Whether to return a Pandas DataFrame in the transform method. If False, will return a Numpy ndarray instead. Since most skutil transformers depend on explicitly-named DataFrame features, the as_df parameter is True by default.

Attributes

drop_	(array-like, shape=(n_features,)) Assigned after calling fit. These are the features that are designated as “bad” and will be dropped in the transform method.
ratios_	(array-like, shape=(n_features,)) The ratios of the counts of the most populous classes to the second most populated classes for each column in cols.

References

[R14]

Kuhn, M. & Johnson, K. “Applied Predictive Modeling” (2013). New York, NY: Springer.

[R15]

Caret (R package) nearZeroVariance R code https://bit.ly/2J0ozbM

Examples

An example of the near zero variance filter on a completely constant column:

>>> import pandas as pd
>>> import numpy as np
>>>
>>> X = pd.DataFrame.from_records(
...     data=np.array([[1,2,3], [4,5,3], [6,7,3], [8,9,3]]),
...     columns=['a','b','c'])
>>> flt = NearZeroVarianceFilter(freq_cut=25)
>>> flt.fit_transform(X)
   a  b
0  1  2
1  4  5
2  6  7
3  8  9

An example on a column with two unique values represented at 2:1. Also shows how we can extract the fitted ratios and drop names:

>>> X = pd.DataFrame.from_records(
...     data=np.array([[1,2,3], [4,5,3], [6,7,5]]),
...     columns=['a','b','c'])
>>> nzv = NearZeroVarianceFilter(freq_cut=2.)
>>> nzv.fit_transform(X)
   a  b
0  1  2
1  4  5
2  6  7
>>> nzv.ratios_
array([ 1.,  1.,  2.])
>>> nzv.drop_
['c']

Methods

fit(X[, y])	Fit the near-zero variance filter.
fit_transform(X[, y])	Fit to data, then transform it.
get_params([deep])	Get parameters for this estimator.
set_params(**params)	Set the parameters of this estimator.
transform(X)	Transform a test dataframe.

__init__(cols=None, freq_cut=19.0, as_df=True)

Initialize self. See help(type(self)) for accurate signature.

fit(X, y=None)

Fit the near-zero variance filter.

Parameters:

X : pd.DataFrame, shape=(n_samples, n_features)

The Pandas frame to fit. The frame will only be fit on the prescribed cols (see __init__) or all of them if cols is None. Furthermore, X will not be altered in the process of the fit.

y : array-like or None, shape=(n_samples,), optional (default=None)

Pass-through for sklearn.pipeline.Pipeline. Even if explicitly set, will not change behavior of fit.

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters:

X : numpy array of shape [n_samples, n_features]

Training set.

y : numpy array of shape [n_samples]

Target values.

Returns:

X_new : numpy array of shape [n_samples, n_features_new]

Transformed array.

get_params(deep=True)

Get parameters for this estimator.

Parameters:

deep : boolean, optional

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

params : mapping of string to any

Parameter names mapped to their values.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as pipelines). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Returns:self

transform(X)

Transform a test dataframe.

Parameters:

X : pd.DataFrame, shape=(n_samples, n_features)

The Pandas frame to transform. The operation will be applied to a copy of the input data, and the result will be returned.

Returns:

X_select : pd.DataFrame, shape=(n_samples, n_features)

The selected columns from X.