{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# Import the packages for experiment\n", "import warnings\n", "\n", "warnings.simplefilter(\"ignore\")\n", "\n", "import pandas as pd\n", "import numpy as np\n", "\n", "import random\n", "from tensorflow import keras\n", "\n", "from itertools import product\n", "from joblib import Parallel, delayed" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Random Classification Experiment\n", "\n", "This experiment will use images from the **CIFAR 100** database (https://www.cs.toronto.edu/~kriz/cifar.html) and showcase the classification efficiency of the synergistic algorithms in the **ProgLearn** project (https://github.com/neurodata/ProgLearn)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Synergistic Learning\n", "\n", "The **ProgLearn** project aims to improve program performance on sequentially learned tasks, proposing a lifelong learning approach.\n", "\n", "It contains two different algorithms: **Synergistic Forests** (**SynF**) and **Synergistic Network** (**SynN**). **SynF** uses Uncertainy Forest as transformers, while **SynN** uses deep networks. These two algorithms achieve both forward knowledge transfer and backward knowledge transfer, and this experiment is designed to cover the **SynF** model." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Choosing hyperparameters\n", "\n", "The hyperparameters here are used for determining how the experiment will run." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "### MAIN HYPERPARAMS ###\n", "num_points_per_task = 500\n", "shift_num = 6\n", "task_num = 20\n", "tree_num = 10\n", "########################" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading datasets\n", "\n", "The CIFAR 100 database contains 100 classes of 600 images, each separating into 500 training images and 100 testing images." ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# load image datasets from the CIFAR-100 database\n", "(X_train, y_train), (X_test, y_test) = keras.datasets.cifar100.load_data()\n", "\n", "# modify data shapes for specific model\n", "data_x = np.concatenate([X_train, X_test])\n", "data_x = data_x.reshape(\n", " (data_x.shape[0], data_x.shape[1] * data_x.shape[2] * data_x.shape[3])\n", ")\n", "data_y = np.concatenate([y_train, y_test])\n", "data_y = data_y[:, 0]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Running experiment\n", "\n", "The following codes will run multiple experiments in parallel. For each experiment, we have task_num number of tasks. For each task, we randomly select 10 classes of the classes to train on. As we will observe below, each task increases Backwards Transfer Efficiency (BTE) with respect to Task 1 (Task 1 being the first task corresponding to 10 randomly selected classes)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "from functions.random_class_functions import synf_experiment\n", "\n", "slot_num = int(5000 / num_points_per_task)\n", "slot_fold = range(slot_num)\n", "shift_fold = range(1, shift_num + 1, 1)\n", "\n", "# run the synf model\n", "n_trees = [tree_num]\n", "iterable = product(n_trees, shift_fold, slot_fold)\n", "df_results = Parallel(n_jobs=-1, verbose=0)(\n", " delayed(synf_experiment)(\n", " data_x, data_y, ntree, shift, slot, num_points_per_task, acorn=12345\n", " )\n", " for ntree, shift, slot in iterable\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Plotting backward transfer efficiency\n", "\n", "Backward transfer efficiency (BTE) measures the relative effect of future task data on the performance on a certain task.\n", "\n", "$$BTE^t (f_n) := \\mathbb{E} [R^t (f_n^{" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from functions.random_class_functions import plot_bte\n", "\n", "plot_bte(bte, fontsize, ticksize)" ] } ], "metadata": { "interpreter": { "hash": "77d3befdf72f5c1a0d6b4996fdd6befdfb972b784410fca14e27e6ae1841315c" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" } }, "nbformat": 4, "nbformat_minor": 4 }