Rotation CIFAR Experiment¶
This experiment will use images from the CIFAR-100 database (https://www.cs.toronto.edu/~kriz/cifar.html) and showcase the backward transfer efficiency of algorithms in the ProgLearn project (https://github.com/neurodata/ProgLearn) as the images are rotated.
# Import necessary packages import numpy as np from tensorflow import keras from multiprocessing import Pool from functools import partial
# Create array to store errors errors_array = 
# Loads and reshapes data sets (X_train, y_train), (X_test, y_test) = keras.datasets.cifar100.load_data() # Joins the training and testing arrays into one data_x = np.concatenate([X_train, X_test]) data_y = np.concatenate([y_train, y_test]) data_y = data_y[:, 0]
Hyperparameters determine how the model will run.
granularity refers to the amount by which the angle will be increased each time. Setting this value at 1 will cause the algorithm to test every whole number rotation angle between 0 and 180 degrees.
reps refers to the number of repetitions tested for each angle of rotation. For each repetition, the data is randomly resampled.
max_depth refers to the maximum depth of each tree in the synergistic forest. If this value is not specified, it defaults to a max tree depth of 30.
### MAIN HYPERPARAMS ### granularity = 4 reps = 10 # use 100 reps for the main draft plot max_depth = 30 ########################
The progressive-learning repo contains two main algorithms, Synergistic Forest (SynF) and Synergistic Network (SynN), within
network.py, respectively. The main difference is that SynF uses random forests while SynN uses deep neural networks. Both algorithms, unlike LwF, EWC, Online_EWC, and SI, have been shown to achieve both forward and backward knowledge transfer.
For the purposes of this experiment, the SynF algorithm will be used.
If the chosen algorithm is trained on both straight up-and-down CIFAR images and rotated CIFAR images, rather than just straight up-and-down CIFAR images, will it perform better (achieve a higher backward transfer efficiency) when tested on straight up-and-down CIFAR images? How does the angle at which training images are rotated affect these results?
At a rotation angle of 0 degrees, the rotated images simply provide additional straight up-and-down CIFAR training data, so the backward transfer efficiency at this angle show whether or not the chosen algorithm can even achieve backward knowledge transfer. As the angle of rotation increases, the rotated images become less and less similar to the original dataset, so the backward transfer efficiency should logically decrease, while still being above 1.
# SynF from functions.rotation_cifar_functions import synf_experiment # Generate set of angles to test for BTE angles = np.arange(0, 181, granularity) # Parallel processing with Pool(48) as p: # Multiple sets of errors for each set of angles are appended to a larger array containing errors for all angles # Calling LF_experiment will run the experiment at a new angle of rotation errors_array.append( p.map( partial( synf_experiment, data_x=data_x, data_y=data_y, reps=reps, ntrees=10, acorn=1, ), angles, ) )
Rotation CIFAR Plot¶
This section takes the results of the experiment and plots the backward transfer efficiency against the angle of rotation for the images in CIFAR-100.
If done correctly, the plot should show that Backward Transfer Efficiency (BTE) is greater than 1 regardless of rotation, but the BTE should decrease as the angle of rotation is increased. The more the number of reps and the finer the granularity, the smoother this downward sloping curve should look.
# Calculate BTE for each angle of rotation bte =  for angle in angles: orig_error, transfer_error = errors_array[ int(angle / granularity) ] # (angle/granularity) gives the index of the errors for that angle bte.append( orig_error / transfer_error ) # (original error/transfer error) gives the BTE
# Plot angle of rotation vs. BTE from functions.rotation_cifar_functions import plot_bte plot_bte(bte, angles)
Why am I getting an “out of memory” error?¶
Pool(8) in the previous cell allows for parallel processing, so the number within the parenthesis should be, at max, the number of cores in the device on which this notebook is being run. Even if a warning is produced, the results of the experimented should not be affected.
Why is this taking so long to run? How can I speed it up to see if I am getting the expected outputs?¶
Decreasing the value of
reps, decreasing the value of
max_depth, or increasing the value of
granularity will all decrease runtime at the cost of noisier results.