SECTION 1: LEARNING OBJECTIVES

By the end of this lesson, you will be able to:

  1. Understand the importance of feature selection in financial modeling and its impact on model performance.

  2. Apply filter methods for feature selection (correlation, variance, mutual information).

  3. Use wrapper methods (forward selection, backward elimination, recursive feature elimination).

  4. Implement embedded methods (LASSO, Random Forest importance, XGBoost importance).

  5. Apply dimensionality reduction techniques (PCA, t-SNE) for financial data.

  6. Evaluate feature selection impact on model performance and interpretability.

  7. Document feature selection decisions for regulatory compliance.

  8. Build automated feature selection pipelines for banking data.

SECTION 2: WHY FEATURE SELECTION MATTERS

2.1 The Curse of Dimensionality

More features are not always better. In financial modeling, too many features can lead to:

 
 
Problem Impact Example
Overfitting Model works on training data, fails on new data Credit model overfits to noise
Multicollinearity Correlated features cause instability Income and savings both predict spending
Computational Cost Slower training and inference Large portfolios take hours to score
Interpretability Harder to explain to regulators Complex models fail SR 11-7
Data Requirements Need more data for stable estimates Small datasets with many features

2.2 Benefits of Feature Selection

 
 
Benefit Banking Example
Better Performance Reduced overfitting
Faster Training Fewer features to process
Improved Interpretability Easier to explain to regulators
Reduced Costs Less data storage and processing
Better Generalization Models work on new data

SECTION 3: FILTER METHODS

3.1 Variance Threshold

python
# ============= FILTER METHODS =============

import pandas as pd
import numpy as np
from sklearn.feature_selection import VarianceThreshold
from sklearn.preprocessing import StandardScaler
import matplotlib.pyplot as plt
import seaborn as sns

class FilterFeatureSelector:
    """Filter methods for feature selection."""
    
    def __init__(self, data, target=None):
        self.data = data.copy()
        self.target = target
        self.selected_features = []
        self.feature_scores = {}
    
    def variance_threshold(self, threshold=0.01):
        """Remove features with low variance."""
        
        # Standardize numeric features
        numeric_cols = self.data.select_dtypes(include=[np.number]).columns
        scaler = StandardScaler()
        scaled_data = scaler.fit_transform(self.data[numeric_cols])
        
        # Apply variance threshold
        selector = VarianceThreshold(threshold=threshold)
        selector.fit(scaled_data)
        
        # Get selected features
        selected_mask = selector.get_support()
        selected_features = numeric_cols[selected_mask].tolist()
        removed_features = numeric_cols[~selected_mask].tolist()
        
        self.selected_features = selected_features
        self.feature_scores['variance'] = {
            'selected': selected_features,
            'removed': removed_features,
            'n_selected': len(selected_features),
            'n_removed': len(removed_features)
        }
        
        print(f"\n📊 Variance Threshold (threshold={threshold}):")
        print(f"  Selected: {len(selected_features)} features")
        print(f"  Removed: {len(removed_features)} features")
        if removed_features:
            print(f"  Removed Features: {removed_features[:5]}...")
        
        return selected_features
    
    def correlation_threshold(self, threshold=0.8):
        """Remove features with high correlation."""
        
        numeric_cols = self.data.select_dtypes(include=[np.number]).columns
        corr_matrix = self.data[numeric_cols].corr()
        
        # Find high correlations
        high_corr_pairs = []
        features_to_remove = set()
        
        for i in range(len(corr_matrix.columns)):
            for j in range(i+1, len(corr_matrix.columns)):
                if abs(corr_matrix.iloc[i, j]) > threshold:
                    high_corr_pairs.append({
                        'feature1': corr_matrix.columns[i],
                        'feature2': corr_matrix.columns[j],
                        'correlation': corr_matrix.iloc[i, j]
                    })
                    # Remove the less important feature (in practice, use domain knowledge)
                    features_to_remove.add(corr_matrix.columns[j])
        
        selected_features = [col for col in numeric_cols if col not in features_to_remove]
        
        self.feature_scores['correlation'] = {
            'selected': selected_features,
            'removed': list(features_to_remove),
            'high_corr_pairs': high_corr_pairs,
            'n_selected': len(selected_features),
            'n_removed': len(features_to_remove)
        }
        
        print(f"\n📊 Correlation Threshold (threshold={threshold}):")
        print(f"  Selected: {len(selected_features)} features")
        print(f"  Removed: {len(features_to_remove)} features")
        if high_corr_pairs:
            print(f"  High Correlation Pairs:")
            for pair in high_corr_pairs[:5]:
                print(f"    • {pair['feature1']} ↔ {pair['feature2']}: {pair['correlation']:.3f}")
        
        return selected_features
    
    def mutual_information(self, threshold=None, n_features=None):
        """Select features using mutual information."""
        
        if self.target is None:
            print("Target variable required for mutual information")
            return []
        
        from sklearn.feature_selection import mutual_info_regression, mutual_info_classif
        
        numeric_cols = self.data.select_dtypes(include=[np.number]).columns
        X = self.data[numeric_cols].fillna(0)
        y = self.data[self.target]
        
        # Determine if classification or regression
        if y.dtype == 'object' or y.nunique() < 10:
            mi = mutual_info_classif(X, y, random_state=42)
        else:
            mi = mutual_info_regression(X, y, random_state=42)
        
        # Create scores
        scores = pd.DataFrame({
            'feature': numeric_cols,
            'score': mi
        }).sort_values('score', ascending=False)
        
        # Select features
        if threshold is not None:
            selected = scores[scores['score'] > threshold]['feature'].tolist()
        elif n_features is not None:
            selected = scores.head(n_features)['feature'].tolist()
        else:
            # Default: keep features with positive MI
            selected = scores[scores['score'] > 0]['feature'].tolist()
        
        self.feature_scores['mutual_information'] = {
            'selected': selected,
            'scores': scores,
            'n_selected': len(selected)
        }
        
        print(f"\n📊 Mutual Information:")
        print(f"  Selected: {len(selected)} features")
        print(f"  Top 5 Features:")
        for _, row in scores.head(5).iterrows():
            print(f"    • {row['feature']}: {row['score']:.3f}")
        
        return selected
    
    def select_features(self, methods=['variance', 'correlation', 'mutual_information']):
        """Run multiple filter methods."""
        
        print("\n" + "="*60)
        print("FILTER METHOD FEATURE SELECTION")
        print("="*60)
        
        results = {}
        
        if 'variance' in methods:
            results['variance'] = self.variance_threshold()
        
        if 'correlation' in methods:
            results['correlation'] = self.correlation_threshold()
        
        if 'mutual_information' in methods and self.target is not None:
            results['mutual_information'] = self.mutual_information(n_features=10)
        
        return results

# Create sample data for feature selection
np.random.seed(42)
n = 500

selection_data = pd.DataFrame({
    'income': np.random.normal(70000, 20000, n),
    'age': np.random.normal(45, 15, n),
    'credit_score': np.random.normal(700, 50, n),
    'debt': np.random.normal(50000, 30000, n),
    'savings': np.random.normal(30000, 20000, n),
    'num_transactions': np.random.poisson(20, n),
    'tenure_months': np.random.exponential(36, n),
    'late_payments': np.random.poisson(0.5, n),
    'dti_ratio': np.random.uniform(0.1, 0.5, n),
    'utilization': np.random.uniform(0.1, 0.9, n)
})

# Add target variable
selection_data['default'] = (selection_data['dti_ratio'] * 2 + 
                            selection_data['utilization'] * 1.5 + 
                            np.random.normal(0, 0.5, n) > 1).astype(int)

# Run filter selection
filter_selector = FilterFeatureSelector(selection_data, target='default')
filter_results = filter_selector.select_features()

# Visualize mutual information
if 'mutual_information' in filter_selector.feature_scores:
    scores = filter_selector.feature_scores['mutual_information']['scores']
    
    plt.figure(figsize=(10, 6))
    plt.barh(scores['feature'].head(10), scores['score'].head(10))
    plt.xlabel('Mutual Information Score')
    plt.title('Top 10 Features by Mutual Information')
    plt.tight_layout()
    plt.savefig('mutual_information.png', dpi=300)
    plt.show()

SECTION 4: WRAPPER METHODS

4.1 Recursive Feature Elimination

python
# ============= WRAPPER METHODS =============

from sklearn.feature_selection import RFE, RFECV
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

class WrapperFeatureSelector:
    """Wrapper methods for feature selection."""
    
    def __init__(self, data, target):
        self.data = data.copy()
        self.target = target
        self.X = None
        self.y = None
        self.selected_features = []
        self.feature_scores = {}
    
    def prepare_data(self):
        """Prepare data for modeling."""
        # Get numeric features
        numeric_cols = self.data.select_dtypes(include=[np.number]).columns
        self.X = self.data[numeric_cols].fillna(0)
        self.y = self.data[self.target]
        
        return self.X, self.y
    
    def rfe_forward(self, n_features=10):
        """Forward feature selection using RFE."""
        
        self.prepare_data()
        
        # Use logistic regression for classification
        estimator = LogisticRegression(max_iter=1000, random_state=42)
        
        # RFE with forward selection
        selector = RFE(estimator, n_features_to_select=n_features, step=1)
        selector.fit(self.X, self.y)
        
        # Get selected features
        selected = self.X.columns[selector.support_].tolist()
        rankings = pd.DataFrame({
            'feature': self.X.columns,
            'ranking': selector.ranking_
        }).sort_values('ranking')
        
        self.selected_features = selected
        self.feature_scores['rfe_forward'] = {
            'selected': selected,
            'rankings': rankings,
            'n_selected': len(selected),
            'n_features': len(self.X.columns)
        }
        
        print(f"\n📊 Forward RFE (n_features={n_features}):")
        print(f"  Selected: {len(selected)} features")
        print(f"  Top 5 Features:")
        for feature in selected[:5]:
            print(f"    • {feature}")
        
        return selected
    
    def rfe_backward(self, n_features=10):
        """Backward feature selection using RFECV."""
        
        self.prepare_data()
        
        # Use random forest for feature importance
        estimator = RandomForestClassifier(n_estimators=50, random_state=42)
        
        # RFE with cross-validation
        selector = RFECV(estimator, step=1, cv=5, scoring='accuracy', min_features_to_select=n_features)
        selector.fit(self.X, self.y)
        
        # Get selected features
        selected = self.X.columns[selector.support_].tolist()
        
        # Get feature rankings
        rankings = pd.DataFrame({
            'feature': self.X.columns,
            'ranking': selector.ranking_,
            'importance': selector.estimator_.feature_importances_
        }).sort_values('ranking')
        
        self.selected_features = selected
        self.feature_scores['rfe_backward'] = {
            'selected': selected,
            'rankings': rankings,
            'n_selected': len(selected),
            'optimal_features': selector.n_features_
        }
        
        print(f"\n📊 Backward RFECV:")
        print(f"  Optimal features: {selector.n_features_}")
        print(f"  Selected: {len(selected)} features")
        print(f"  Top 5 Features:")
        for feature in selected[:5]:
            print(f"    • {feature}")
        
        return selected
    
    def forward_selection(self, max_features=10):
        """Manual forward selection."""
        
        self.prepare_data()
        
        selected = []
        remaining = list(self.X.columns)
        scores = []
        
        while len(selected) < max_features and remaining:
            best_score = -float('inf')
            best_feature = None
            
            for feature in remaining:
                # Test adding this feature
                test_features = selected + [feature]
                X_test = self.X[test_features]
                
                # Train model and evaluate
                model = RandomForestClassifier(n_estimators=30, random_state=42)
                model.fit(X_test, self.y)
                score = model.score(X_test, self.y)
                
                if score > best_score:
                    best_score = score
                    best_feature = feature
            
            if best_feature:
                selected.append(best_feature)
                remaining.remove(best_feature)
                scores.append(best_score)
        
        self.selected_features = selected
        self.feature_scores['forward_selection'] = {
            'selected': selected,
            'scores': scores,
            'n_selected': len(selected)
        }
        
        print(f"\n📊 Forward Selection:")
        print(f"  Selected: {len(selected)} features")
        print(f"  Feature Selection Order:")
        for i, feature in enumerate(selected):
            print(f"    {i+1}. {feature} (Score: {scores[i]:.4f})")
        
        return selected
    
    def compare_methods(self):
        """Compare wrapper methods."""
        
        print("\n" + "="*60)
        print("WRAPPER METHODS COMPARISON")
        print("="*60)
        
        # Run all methods
        rfe_forward = self.rfe_forward(n_features=8)
        rfe_backward = self.rfe_backward(n_features=8)
        forward_sel = self.forward_selection(max_features=8)
        
        # Find common features
        common = set(rfe_forward) & set(rfe_backward) & set(forward_sel)
        
        print(f"\n📊 Common Features across all methods:")
        if common:
            for feature in common:
                print(f"  • {feature}")
        else:
            print("  No common features found")
        
        # Union of all features
        union = set(rfe_forward) | set(rfe_backward) | set(forward_sel)
        print(f"\n📊 Unique Features across all methods ({len(union)}):")
        for feature in union:
            print(f"  • {feature}")

# Run wrapper methods
wrapper_selector = WrapperFeatureSelector(selection_data, 'default')
wrapper_selector.compare_methods()

SECTION 5: EMBEDDED METHODS

5.1 Feature Importance from Tree-Based Models

python
# ============= EMBEDDED METHODS =============

from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor
from sklearn.linear_model import Lasso, Ridge
from xgboost import XGBClassifier, XGBRegressor

class EmbeddedFeatureSelector:
    """Embedded methods for feature selection."""
    
    def __init__(self, data, target):
        self.data = data.copy()
        self.target = target
        self.X = None
        self.y = None
        self.feature_importance = {}
    
    def prepare_data(self):
        """Prepare data for modeling."""
        numeric_cols = self.data.select_dtypes(include=[np.number]).columns
        self.X = self.data[numeric_cols].fillna(0)
        self.y = self.data[self.target]
        
        return self.X, self.y
    
    def random_forest_importance(self, n_estimators=100):
        """Feature importance from Random Forest."""
        
        self.prepare_data()
        
        # Determine if classification or regression
        if self.y.dtype == 'object' or self.y.nunique() < 10:
            model = RandomForestClassifier(n_estimators=n_estimators, random_state=42)
        else:
            model = RandomForestRegressor(n_estimators=n_estimators, random_state=42)
        
        model.fit(self.X, self.y)
        
        # Get feature importance
        importance = pd.DataFrame({
            'feature': self.X.columns,
            'importance': model.feature_importances_
        }).sort_values('importance', ascending=False)
        
        self.feature_importance['random_forest'] = {
            'importance': importance,
            'top_features': importance.head(10)['feature'].tolist(),
            'threshold': importance['importance'].mean()  # Keep features above average
        }
        
        print(f"\n📊 Random Forest Feature Importance:")
        print(importance.head(10))
        
        return importance
    
    def xgboost_importance(self):
        """Feature importance from XGBoost."""
        
        self.prepare_data()
        
        # Determine if classification or regression
        if self.y.dtype == 'object' or self.y.nunique() < 10:
            model = XGBClassifier(n_estimators=100, random_state=42)
        else:
            model = XGBRegressor(n_estimators=100, random_state=42)
        
        model.fit(self.X, self.y)
        
        # Get feature importance
        importance = pd.DataFrame({
            'feature': self.X.columns,
            'importance': model.feature_importances_
        }).sort_values('importance', ascending=False)
        
        self.feature_importance['xgboost'] = {
            'importance': importance,
            'top_features': importance.head(10)['feature'].tolist()
        }
        
        print(f"\n📊 XGBoost Feature Importance:")
        print(importance.head(10))
        
        return importance
    
    def lasso_importance(self, alpha=0.01):
        """Feature importance from Lasso (L1 regularization)."""
        
        self.prepare_data()
        
        # Use Lasso for regression, LogisticRegression with L1 for classification
        if self.y.dtype == 'object' or self.y.nunique() < 10:
            from sklearn.linear_model import LogisticRegression
            model = LogisticRegression(penalty='l1', solver='saga', C=1/alpha, max_iter=1000, random_state=42)
            model.fit(self.X, self.y)
            coefficients = model.coef_[0]
        else:
            model = Lasso(alpha=alpha, random_state=42)
            model.fit(self.X, self.y)
            coefficients = model.coef_
        
        # Get feature importance
        importance = pd.DataFrame({
            'feature': self.X.columns,
            'coefficient': coefficients,
            'importance': np.abs(coefficients)
        }).sort_values('importance', ascending=False)
        
        self.feature_importance['lasso'] = {
            'importance': importance,
            'top_features': importance[importance['importance'] > 0]['feature'].tolist(),
            'n_selected': (importance['importance'] > 0).sum()
        }
        
        print(f"\n📊 Lasso Feature Importance:")
        print(importance[importance['importance'] > 0].head(10))
        
        return importance
    
    def compare_methods(self):
        """Compare embedded methods."""
        
        print("\n" + "="*60)
        print("EMBEDDED METHODS COMPARISON")
        print("="*60)
        
        # Run all methods
        rf_importance = self.random_forest_importance()
        xgb_importance = self.xgboost_importance()
        lasso_importance = self.lasso_importance()
        
        # Get top features from each method
        rf_top = set(rf_importance.head(10)['feature'])
        xgb_top = set(xgb_importance.head(10)['feature'])
        lasso_top = set(lasso_importance[lasso_importance['importance'] > 0]['feature'])
        
        # Find common features
        common = rf_top & xgb_top & lasso_top
        
        print(f"\n📊 Common Features across all methods:")
        if common:
            for feature in common:
                print(f"  • {feature}")
        else:
            print("  No common features found")
        
        # Consensus features (appear in at least 2 methods)
        from collections import Counter
        all_features = list(rf_top) + list(xgb_top) + list(lasso_top)
        feature_counts = Counter(all_features)
        consensus = [feat for feat, count in feature_counts.items() if count >= 2]
        
        print(f"\n📊 Consensus Features (in at least 2 methods):")
        for feature in consensus:
            print(f"  • {feature}")
        
        return consensus

# Run embedded methods
embedded_selector = EmbeddedFeatureSelector(selection_data, 'default')
embedded_selector.compare_methods()

SECTION 6: DIMENSIONALITY REDUCTION

6.1 Principal Component Analysis (PCA)

python
# ============= PRINCIPAL COMPONENT ANALYSIS =============

from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

class DimensionalityReducer:
    """Dimensionality reduction techniques."""
    
    def __init__(self, data):
        self.data = data.copy()
        self.components = None
        self.explained_variance = None
    
    def pca(self, n_components=None, explained_variance_threshold=0.95):
        """Apply PCA for dimensionality reduction."""
        
        # Select numeric columns
        numeric_cols = self.data.select_dtypes(include=[np.number]).columns
        X = self.data[numeric_cols].fillna(0)
        
        # Scale data
        scaler = StandardScaler()
        X_scaled = scaler.fit_transform(X)
        
        # Determine number of components
        if n_components is None:
            # Use variance threshold
            pca_temp = PCA()
            pca_temp.fit(X_scaled)
            cumsum = np.cumsum(pca_temp.explained_variance_ratio_)
            n_components = np.argmax(cumsum >= explained_variance_threshold) + 1
        
        # Apply PCA
        pca = PCA(n_components=n_components)
        self.components = pca.fit_transform(X_scaled)
        self.explained_variance = pca.explained_variance_ratio_
        
        # Get feature loadings
        loadings = pd.DataFrame(
            pca.components_.T,
            columns=[f'PC{i+1}' for i in range(n_components)],
            index=numeric_cols
        )
        
        print(f"\n📊 PCA Results:")
        print(f"  Original features: {len(numeric_cols)}")
        print(f"  Components: {n_components}")
        print(f"  Total variance explained: {self.explained_variance.sum():.2%}")
        print(f"\n  Variance per component:")
        for i, var in enumerate(self.explained_variance):
            print(f"    PC{i+1}: {var:.2%}")
        
        print(f"\n  Top loadings for PC1:")
        top_loadings = loadings['PC1'].abs().sort_values(ascending=False).head(5)
        for feature, loading in top_loadings.items():
            print(f"    • {feature}: {loading:.3f}")
        
        return self.components, loadings
    
    def pca_visualization(self):
        """Visualize PCA results."""
        
        if self.components is None:
            self.pca(n_components=2)
        
        plt.figure(figsize=(12, 5))
        
        # 1. Scree plot
        plt.subplot(1, 2, 1)
        plt.bar(range(1, len(self.explained_variance) + 1), self.explained_variance, alpha=0.7)
        plt.plot(range(1, len(self.explained_variance) + 1), 
                np.cumsum(self.explained_variance), 'ro-')
        plt.xlabel('Principal Component')
        plt.ylabel('Explained Variance Ratio')
        plt.title('Scree Plot')
        plt.grid(True, alpha=0.3)
        
        # 2. 2D projection
        plt.subplot(1, 2, 2)
        plt.scatter(self.components[:, 0], self.components[:, 1], alpha=0.5)
        plt.xlabel(f'PC1 ({self.explained_variance[0]:.1%})')
        plt.ylabel(f'PC2 ({self.explained_variance[1] if len(self.explained_variance) > 1 else 0:.1%})')
        plt.title('PCA Projection (2D)')
        plt.grid(True, alpha=0.3)
        
        plt.tight_layout()
        plt.savefig('pca_visualization.png', dpi=300)
        plt.show()
    
    def t_sne(self, n_components=2, perplexity=30):
        """Apply t-SNE for visualization."""
        
        try:
            from sklearn.manifold import TSNE
            
            # Select numeric columns
            numeric_cols = self.data.select_dtypes(include=[np.number]).columns
            X = self.data[numeric_cols].fillna(0)
            
            # Scale data
            scaler = StandardScaler()
            X_scaled = scaler.fit_transform(X)
            
            # Apply t-SNE
            tsne = TSNE(n_components=n_components, perplexity=perplexity, random_state=42)
            tsne_result = tsne.fit_transform(X_scaled)
            
            print(f"\n📊 t-SNE Results:")
            print(f"  Perplexity: {perplexity}")
            print(f"  Components: {n_components}")
            
            # Visualize
            plt.figure(figsize=(8, 6))
            plt.scatter(tsne_result[:, 0], tsne_result[:, 1], alpha=0.5)
            plt.xlabel('t-SNE 1')
            plt.ylabel('t-SNE 2')
            plt.title('t-SNE Projection')
            plt.grid(True, alpha=0.3)
            plt.savefig('tsne_visualization.png', dpi=300)
            plt.show()
            
            return tsne_result
            
        except ImportError:
            print("t-SNE requires scikit-learn >= 0.17")
            return None

# Run dimensionality reduction
pca_reducer = DimensionalityReducer(selection_data)
pca_components, loadings = pca_reducer.pca(n_components=5)
pca_reducer.pca_visualization()

# Optional: t-SNE
# pca_reducer.t_sne()

SECTION 7: FEATURE SELECTION PIPELINE

7.1 Building an Automated Pipeline

python
# ============= AUTOMATED FEATURE SELECTION PIPELINE =============

class FeatureSelectionPipeline:
    """Complete feature selection pipeline."""
    
    def __init__(self, data, target, model_type='classification'):
        self.data = data.copy()
        self.target = target
        self.model_type = model_type
        self.selected_features = None
        self.pipeline_steps = []
    
    def run_pipeline(self, n_features=10):
        """Run complete feature selection pipeline."""
        
        print("\n" + "="*60)
        print("FEATURE SELECTION PIPELINE")
        print("="*60)
        
        # Step 1: Remove low variance features
        print("\n📊 Step 1: Variance Threshold")
        filter_selector = FilterFeatureSelector(self.data)
        filter_variance = filter_selector.variance_threshold(threshold=0.01)
        self.pipeline_steps.append({
            'step': 'variance_threshold',
            'selected': filter_variance,
            'n_selected': len(filter_variance)
        })
        
        # Step 2: Remove high correlation features
        print("\n📊 Step 2: Correlation Threshold")
        filter_corr = filter_selector.correlation_threshold(threshold=0.9)
        self.pipeline_steps.append({
            'step': 'correlation_threshold',
            'selected': filter_corr,
            'n_selected': len(filter_corr)
        })
        
        # Step 3: Mutual Information
        print("\n📊 Step 3: Mutual Information")
        filter_selector.target = self.target
        mi_features = filter_selector.mutual_information(n_features=20)
        self.pipeline_steps.append({
            'step': 'mutual_information',
            'selected': mi_features,
            'n_selected': len(mi_features)
        })
        
        # Step 4: Random Forest Importance
        print("\n📊 Step 4: Random Forest Importance")
        embedded_selector = EmbeddedFeatureSelector(self.data, self.target)
        rf_importance = embedded_selector.random_forest_importance()
        rf_selected = rf_importance.head(n_features)['feature'].tolist()
        self.pipeline_steps.append({
            'step': 'random_forest',
            'selected': rf_selected,
            'n_selected': len(rf_selected)
        })
        
        # Step 5: Final selection (intersection)
        print("\n📊 Step 5: Final Feature Selection")
        
        # Get all selected features
        all_selected = []
        for step in self.pipeline_steps:
            all_selected.extend(step['selected'])
        
        # Count frequency of each feature
        from collections import Counter
        feature_counts = Counter(all_selected)
        
        # Keep features that appear in at least 3 methods
        final_features = [feat for feat, count in feature_counts.items() if count >= 3]
        
        # If too few features, lower threshold
        if len(final_features) < n_features:
            final_features = [feat for feat, count in feature_counts.items() if count >= 2]
        
        self.selected_features = final_features
        
        print(f"\n✅ Final Selected Features ({len(final_features)}):")
        for feature in final_features:
            print(f"  • {feature} (appeared in {feature_counts[feature]} methods)")
        
        return final_features
    
    def get_selected_data(self):
        """Return data with selected features."""
        if self.selected_features is None:
            self.run_pipeline()
        
        feature_cols = [col for col in self.selected_features if col in self.data.columns]
        if self.target in self.data.columns and self.target not in feature_cols:
            return self.data[feature_cols + [self.target]]
        else:
            return self.data[feature_cols]

# Run feature selection pipeline
pipeline = FeatureSelectionPipeline(selection_data, 'default')
selected_features = pipeline.run_pipeline(n_features=10)
selected_data = pipeline.get_selected_data()

print("\n" + "="*60)
print("SELECTED DATA PREVIEW")
print("="*60)
print(selected_data.head())

SECTION 8: SUMMARY FOR THE DATA PRACTITIONER

8.1 The 1-Minute Elevator Pitch

“Feature selection identifies the most predictive features from your financial data. We use filter methods (variance, correlation, mutual information), wrapper methods (RFE, forward selection), and embedded methods (Random Forest, Lasso, XGBoost). Dimensionality reduction techniques like PCA help visualize high-dimensional data. Feature selection reduces overfitting, improves performance, and creates more interpretable models. In banking, feature selection is essential for regulatory compliance and building robust models.”

8.2 Key Takeaways

  1. Feature selection improves model performance and interpretability.

  2. Filter methods are fast and model-agnostic.

  3. Wrapper methods use model performance to select features.

  4. Embedded methods use regularization or feature importance.

  5. PCA reduces dimensionality while preserving information.

  6. t-SNE helps visualize high-dimensional data.

  7. Pipeline approach combines multiple selection methods.

  8. Domain knowledge should guide feature selection decisions.

  9. Validation ensures selected features work on new data.

  10. Documentation is required for regulatory compliance.

8.3 Recommended Next Steps

  1. Apply feature selection to your banking datasets

  2. Compare different feature selection methods

  3. Validate selected features on hold-out data

  4. Document feature selection decisions

  5. Build automated feature selection pipelines

[END OF LESSON 6]

 
 
PreviousNext