3. Logistic Regression

3.1 Initialize Logistic Regression Instance

The first binary classifier in this tutorial is the Logistic Regression classifier. We start building the classifier by initializing an instance of LogisticRegression() function:

from pyspark.ml.classification import LogisticRegression
# init log regression object
lr = LogisticRegression(featuresCol='features', labelCol='label', family='binomial',  maxIter=10)

As we explained in the previous Post of this tutorial, we add the names of independent and target variables to the classifier function.

3.2 Cross Validation

Cross validation is a common procedure taken within classification algorithms to avoid model overfitting against certain test data subsets. In order to apply cross validation in Apache Spark, we firstly need to build a parameter grid object. Parameter grids enables a given classifier to try out different input parameters’ settings and optimize the accuracy between each combination of them. The following code snippet builds a parameter grid that loop over a list of values of regression and elastic search parameters of logistic regression.

from pyspark.ml.tuning import ParamGridBuilder
# init parameter grid for log rgression params
paramGrid = ParamGridBuilder()
             .addGrid(lr.regParam, [0.1, 0.3]) # regularization parameter
             .addGrid(lr.elasticNetParam, [0.0]) # Elastic Net Parameter
             .build()

Secondly, we set the classifier evaluator, using BinaryClassificationEvaluator() function.

from pyspark.ml.evaluation import BinaryClassificationEvaluator
# create evaluator
evaluator = BinaryClassificationEvaluator(rawPredictionCol="label")

Thirdly, we create CrossValidator() object and pass model, parameter grid, and evaluation instances to it. In this step, we can set the number of folds we wish to use in our cross validation procedure. In the below code we use 10 folds.

from pyspark.ml.tuning import CrossValidator
# create cross validation object
crossval = CrossValidator(estimator=lr,
                          estimatorParamMaps=paramGrid,
                          evaluator=evaluator,
                          numFolds=10) 

3.3 Fit Log. Regression Model to Training Data Vector

After initializing the cross validation object, we use it to fit our log. regression model to the train vector.

# run cross-validation, and choose the best set of parameters.
cvModel = crossval.fit(train_vector)

3.4 Get Predictions

After fitting the model, we evaluate it using confusion matrix. This means we need to extract true & predicted labels out of the model. The following function achieves this goal.

def get_prediction_lists(prediction_df):
    '''
    this function returns true & predicted labels as Python lists, plus array of predictions, 
    using Spark prediction DF as input
    
    INPUTS:
    @prediction_df: Spark DF of classifier predictions 
    
    OUTPUTS:
    @l1: list of true labels
    @l2: list of predicted labels
    @m1: array of predictions
    '''
    
    l1 = map(int, get_col_as_list(prediction_df, "label"))
    l2 = map(int, get_col_as_list(prediction_df, "prediction"))
    m1 = get_col_as_nparray(prediction_df, "probability")
    
    return l1, l2, m1

3.5 Build Confusion Matrix

Once we obtain true and predicted data labels using the above function, we can build our confusion matrix using confusion_matrix() function.

from sklearn.metrics import confusion_matrix
# get confusion matrix
logreg_matrix = confusion_matrix(labels, preds)

Confusion matrix is a very effective tool for measuring the accuracy of binary and multi classifiers. It is used to calculate very common evaluation metrics such as precision, recall, and F1.