skoot.feature_selection.SparseFeatureFilter

class skoot.feature_selection.SparseFeatureFilter(cols=None, threshold=0.5, as_df=True)

Drop overly sparse features.

Retains features that are less sparse (NaN) than the provided threshold. Useful in situations where matrices are too sparse to impute reliably.

Parameters:

cols : array-like, shape=(n_features,), optional (default=None)

The names of the columns on which to apply the transformation. If no column names are provided, the transformer will be fit on the entire frame. Note that the transformation will also only apply to the specified columns, and any other non-specified columns will still be present after transformation.

threshold : float, optional (default=0.5)

The threshold of sparsity above which features will be deemed “too sparse” and will be dropped.

as_df : bool, optional (default=True)

Whether to return a Pandas DataFrame in the transform method. If False, will return a Numpy ndarray instead. Since most skutil transformers depend on explicitly-named DataFrame features, the as_df parameter is True by default.

Attributes

sparsity_	(array-like, shape=(n_features,)) The array of sparsity values
drop_	(array-like, shape=(n_features,)) Assigned after calling fit. These are the features that are designated as “bad” and will be dropped in the transform method.

Notes

Sometimes the presence of a value in an overly sparse column can be highly informative. If you’re using the sparse filter, consider creating a new (dummy) feature indicating whether there was a value present.

Examples

>>> import numpy as np
>>> import pandas as pd
>>>
>>> nan = np.nan
>>> X = np.array([
...     [1.0, 2.0, nan],
...     [2.0, 3.0, nan],
...     [3.0, nan, 1.0],
...     [4.0, 5.0, nan]
... ])
>>>
>>> X = pd.DataFrame.from_records(data=X, columns=['a','b','c'])
>>> dropper = SparseFeatureFilter(threshold=0.5)
>>> X_transform = dropper.fit_transform(X)
>>> assert X_transform.shape[1] == 2 # drop out last column

Methods

fit(X[, y])	Fit the transformer.
fit_transform(X[, y])	Fit to data, then transform it.
get_params([deep])	Get parameters for this estimator.
set_params(**params)	Set the parameters of this estimator.
transform(X)	Transform a test dataframe.

__init__(cols=None, threshold=0.5, as_df=True)

Initialize self. See help(type(self)) for accurate signature.

fit(X, y=None)

Fit the transformer.

Parameters:

X : pd.DataFrame, shape=(n_samples, n_features)

The Pandas frame to fit. The frame will only be fit on the prescribed cols (see __init__) or all of them if cols is None. Furthermore, X will not be altered in the process of the fit.

y : array-like or None, shape=(n_samples,), optional (default=None)

Pass-through for sklearn.pipeline.Pipeline. Even if explicitly set, will not change behavior of fit.

fit_transform(X, y=None, **fit_params)

Fit to data, then transform it.

Fits transformer to X and y with optional parameters fit_params and returns a transformed version of X.

Parameters:

X : numpy array of shape [n_samples, n_features]

Training set.

y : numpy array of shape [n_samples]

Target values.

Returns:

X_new : numpy array of shape [n_samples, n_features_new]

Transformed array.

get_params(deep=True)

Get parameters for this estimator.

Parameters:

deep : boolean, optional

If True, will return the parameters for this estimator and contained subobjects that are estimators.

Returns:

params : mapping of string to any

Parameter names mapped to their values.

set_params(**params)

Set the parameters of this estimator.

The method works on simple estimators as well as on nested objects (such as pipelines). The latter have parameters of the form <component>__<parameter> so that it’s possible to update each component of a nested object.

Returns:self

transform(X)

Transform a test dataframe.

Parameters:

X : pd.DataFrame, shape=(n_samples, n_features)

The Pandas frame to transform. The operation will be applied to a copy of the input data, and the result will be returned.

Returns:

X_select : pd.DataFrame, shape=(n_samples, n_features)

The selected columns from X.