deepchem: Machine Learning models for Drug Discovery¶

Tutorial 1¶

Written by Evan Feinberg and Bharath Ramsundar

Copyright 2016, Stanford University

Welcome to the deepchem tutorial. In this iPython Notebook, one can follow along with the code below to learn how to fit machine learning models with rich predictive power on chemical datasets.¶

Overview:

In this tutorial, you will trace an arc from loading a raw dataset to fitting a cutting edge ML technique for predicting binding affinities. This will be accomplished by writing simple commands to access the deepchem Python API, encompassing the following broad steps:

  1. Loading a chemical dataset, consisting of a series of protein-ligand complexes.
  2. Featurizing each protein-ligand complexes with various featurization schemes.
  3. Fitting a series of models with these featurized protein-ligand complexes.
  4. Visualizing the results.

First, let's point to a "dataset" file. This can come in the format of a CSV file or Pandas DataFrame. Regardless of file format, it must be columnar data, where each row is a molecular system, and each column represents a different piece of information about that system. For instance, in this example, every row reflects a protein-ligand complex, and the following columns are present: a unique complex identifier; the SMILES string of the ligand; the binding affinity (Ki) of the ligand to the protein in the complex; a Python list of all lines in a PDB file for the protein alone; and a Python list of all lines in a ligand file for the ligand alone.

This should become clearer with the example.

In [25]:
import warnings
warnings.filterwarnings('ignore')
In [26]:
dataset_file= "../datasets/pdbbind_core_df.pkl.gz"
from deepchem.utils.save import load_from_disk
dataset = load_from_disk(dataset_file)

Let's see what dataset looks like:

In [2]:
print("Type of dataset is: %s" % str(type(dataset)))
print(dataset[:5])
print("Shape of dataset is: %s" % str(dataset.shape))
Type of dataset is: <class 'pandas.core.frame.DataFrame'>
  pdb_id                                             smiles  \
0   2d3u        CC1CCCCC1S(O)(O)NC1CC(C2CCC(CN)CC2)SC1C(O)O   
1   3cyx  CC(C)(C)NC(O)C1CC2CCCCC2C[NH+]1CC(O)C(CC1CCCCC...   
2   3uo4        OC(O)C1CCC(NC2NCCC(NC3CCCCC3C3CCCCC3)N2)CC1   
3   1p1q                         CC1ONC(O)C1CC([NH3+])C(O)O   
4   3ag9  NC(O)C(CCC[NH2+]C([NH3+])[NH3+])NC(O)C(CCC[NH2...   

                                          complex_id  \
0    2d3uCC1CCCCC1S(O)(O)NC1CC(C2CCC(CN)CC2)SC1C(O)O   
1  3cyxCC(C)(C)NC(O)C1CC2CCCCC2C[NH+]1CC(O)C(CC1C...   
2    3uo4OC(O)C1CCC(NC2NCCC(NC3CCCCC3C3CCCCC3)N2)CC1   
3                     1p1qCC1ONC(O)C1CC([NH3+])C(O)O   
4  3ag9NC(O)C(CCC[NH2+]C([NH3+])[NH3+])NC(O)C(CCC...   

                                         protein_pdb  \
0  [HEADER    2D3U PROTEIN\n, COMPND    2D3U PROT...   
1  [HEADER    3CYX PROTEIN\n, COMPND    3CYX PROT...   
2  [HEADER    3UO4 PROTEIN\n, COMPND    3UO4 PROT...   
3  [HEADER    1P1Q PROTEIN\n, COMPND    1P1Q PROT...   
4  [HEADER    3AG9 PROTEIN\n, COMPND    3AG9 PROT...   

                                          ligand_pdb  \
0  [COMPND    2d3u ligand \n, AUTHOR    GENERATED...   
1  [COMPND    3cyx ligand \n, AUTHOR    GENERATED...   
2  [COMPND    3uo4 ligand \n, AUTHOR    GENERATED...   
3  [COMPND    1p1q ligand \n, AUTHOR    GENERATED...   
4  [COMPND    3ag9 ligand \n, AUTHOR    GENERATED...   

                                         ligand_mol2 label  
0  [### \n, ### Created by X-TOOL on Thu Aug 28 2...  6.92  
1  [### \n, ### Created by X-TOOL on Thu Aug 28 2...  8.00  
2  [### \n, ### Created by X-TOOL on Fri Aug 29 0...  6.52  
3  [### \n, ### Created by X-TOOL on Thu Aug 28 2...  4.89  
4  [### \n, ### Created by X-TOOL on Thu Aug 28 2...  8.05  
Shape of dataset is: (193, 7)

One of the missions of deepchem is to form a synapse between the chemical and the algorithmic worlds: to be able to leverage the powerful and diverse array of tools available in Python to analyze molecules. This ethos applies to visual as much as quantitative examination:

In [3]:
import nglview
import tempfile
import os
import mdtraj as md
import numpy as np
import deepchem.utils.visualization
reload(deepchem.utils.visualization)
from deepchem.utils.visualization import combine_mdtraj, visualize_complex, convert_lines_to_mdtraj

first_protein, first_ligand = dataset.iloc[0]["protein_pdb"], dataset.iloc[0]["ligand_pdb"]

protein_mdtraj = convert_lines_to_mdtraj(first_protein)
ligand_mdtraj = convert_lines_to_mdtraj(first_ligand)
complex_mdtraj = combine_mdtraj(protein_mdtraj, ligand_mdtraj)
In [4]:
def visualize_complex(complex_mdtraj):
  ligand_atoms = [a.index for a in complex_mdtraj.topology.atoms if "LIG" in str(a.residue)]
  binding_pocket_atoms = md.compute_neighbors(complex_mdtraj, 0.5, ligand_atoms)[0]
  binding_pocket_residues = list(set([complex_mdtraj.topology.atom(a).residue.resSeq for a in binding_pocket_atoms]))
  binding_pocket_residues = [str(r) for r in binding_pocket_residues]
  binding_pocket_residues = " or ".join(binding_pocket_residues)

  traj = nglview.MDTrajTrajectory( complex_mdtraj ) # load file from RCSB PDB
  ngltraj = nglview.NGLWidget( traj )
  ngltraj.representations = [
  { "type": "cartoon", "params": {
  "sele": "protein", "color": "residueindex"
  } },
  { "type": "licorice", "params": {
  "sele": "(not hydrogen) and (resi (%s))" %  binding_pocket_residues
  } },
  { "type": "ball+stick", "params": {
  "sele": "resn LIG"
  } }
  ]
  return ngltraj
In [27]:
ngltraj = visualize_complex(complex_mdtraj)
ngltraj

Now that we're oriented, let's use ML to do some chemistry.

So, step (2) will entail featurizing the dataset.

The available featurizations that come standard with deepchem are ECFP4 fingerprints, RDKit descriptors, NNScore-style bdescriptors, and hybrid binding pocket descriptors. Details can be found on deepchem.io.

In [6]:
from deepchem.featurizers.fingerprints import CircularFingerprint
from deepchem.featurizers.basic import RDKitDescriptors
from deepchem.featurizers.nnscore import NNScoreComplexFeaturizer
from deepchem.featurizers.grid_featurizer import GridFeaturizer
grid_featurizer = GridFeaturizer(voxel_width=16.0, feature_types="voxel_combined", voxel_feature_types=["ecfp",
                                 "splif", "hbond", "pi_stack", "cation_pi", "salt_bridge"], ecfp_power=5, splif_power=5,
                                 parallel=True, flatten=True)
compound_featurizers = [CircularFingerprint(size=128)]
complex_featurizers = [grid_featurizer]

Note how we separate our featurizers into those that featurize individual chemical compounds, compound_featurizers, and those that featurize molecular complexes, complex_featurizers.

Now, let's perform the actual featurization. Calling featurizer.featurize() will return an instance of class FeaturizedSamples. Internally, featurizer.featurize() (a) computes the user-specified features on the data, (b) transforms the inputs into X and y NumPy arrays suitable for ML algorithms, and (c) constructs a FeaturizedSamples() instance that has useful methods, such as an iterator, over the featurized data.

In [7]:
#Make a directory in which to store the featurized complexes.
import tempfile, shutil
feature_dir = tempfile.mkdtemp()
samples_dir = tempfile.mkdtemp()
In [8]:
import deepchem.featurizers.featurize
reload(deepchem.featurizers.featurize)
from deepchem.featurizers.featurize import DataFeaturizer
In [ ]:
featurizers = compound_featurizers + complex_featurizers
featurizer = DataFeaturizer(tasks=["label"],
                            smiles_field="smiles",
                            protein_pdb_field="protein_pdb",
                            ligand_pdb_field="ligand_pdb",
                            compound_featurizers=compound_featurizers,
                            complex_featurizers=complex_featurizers,
                            id_field="complex_id",
                            verbose=False)
featurized_samples = featurizer.featurize(dataset_file, feature_dir, samples_dir,
                                          shard_size=32)

from deepchem.utils.save import save_to_disk, load_from_disk
featurized_samples_file = "examples/tutorial_samples.joblib"
save_to_disk(featurized_samples, featurized_samples_file)
In [9]:
featurized_samples_file = "tutorial_samples.joblib"
featurized_samples = load_from_disk(featurized_samples_file)

Now, we conduct a train-test split. If you'd like, you can choose splittype="scaffold" instead to perform a train-test split based on Bemis-Murcko scaffolds.

In [31]:
splittype = "random"
train_dir, test_dir = tempfile.mkdtemp(), tempfile.mkdtemp()

train_samples, test_samples = featurized_samples.train_test_split(
    splittype, train_dir, test_dir, seed=2016)

We generate separate instances of the Dataset() object to hermetically seal the train dataset from the test dataset. This style lends itself easily to validation-set type hyperparameter searches, which we will illustate in a separate section of this tutorial.

In [32]:
from deepchem.utils.dataset import Dataset
In [33]:
train_dataset = Dataset(data_dir=train_dir, samples=train_samples, 
                        featurizers=compound_featurizers, tasks=["label"])
test_dataset = Dataset(data_dir=test_dir, samples=test_samples, 
                       featurizers=compound_featurizers, tasks=["label"])

The performance of many ML algorithms hinges greatly on careful data preprocessing. Deepchem comes standard with a few options for such preprocessing.

In [34]:
input_transforms = ["normalize", "truncate"]
output_transforms = ["normalize"]
train_dataset.transform(input_transforms, output_transforms)
test_dataset.transform(input_transforms, output_transforms)

Now, we're ready to do some learning! To set up a model, we will need: (a) a dictionary task_types that maps a task, in this case label, i.e. the Ki, to the type of the task, in this case regression. For the multitask use case, one will have a series of keys, each of which is a different task (Ki, solubility, renal half-life, etc.) that maps to a different task type (regression or classification).

To fit a deepchem model, first we instantiate one of the provided (or user-written) model classes. In this case, we have a created a convenience class to wrap around any ML model available in Sci-Kit Learn that can in turn be used to interoperate with deepchem. To instantiate an SklearnModel, you will need (a) task_types, (b) model_params, another dict as illustrated below, and (c) a model_instance defining the type of model you would like to fit, in this case a RandomForestRegressor.

In [35]:
from sklearn.ensemble import RandomForestRegressor
from deepchem.models.standard import SklearnModel
In [36]:
task_types = {"label": "regression"}
model_params = {"data_shape": train_dataset.get_data_shape()}

model = SklearnModel(task_types, model_params, model_instance=RandomForestRegressor())
model.fit(train_dataset)
model_dir = tempfile.mkdtemp()
model.save(model_dir)
In [37]:
from deepchem.utils.evaluate import Evaluator
import pandas as pd
In [38]:
evaluator = Evaluator(model, train_dataset, verbose=True)
with tempfile.NamedTemporaryFile() as train_csv_out:
  with tempfile.NamedTemporaryFile() as train_stats_out:
    _, train_r2score = evaluator.compute_model_performance(
        train_csv_out, train_stats_out)

evaluator = Evaluator(model, test_dataset, verbose=True)
test_csv_out = tempfile.NamedTemporaryFile()
with tempfile.NamedTemporaryFile() as test_stats_out:
    _, test_r2score = evaluator.compute_model_performance(
        test_csv_out, test_stats_out)

print test_csv_out.name
train_test_performance = pd.concat([train_r2score, test_r2score])
train_test_performance["split"] = ["train", "test"]
train_test_performance
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6a50>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8ae0>
/local-scratch/enf/12500/tmp3x44sg
Out[38]:
task_name r2_score rms_error split
0 label 0.825151 0.963289 train
0 label 0.155815 1.757489 test

In this simple example, in few yet intuitive lines of code, we traced the machine learning arc from featurizing a raw dataset to fitting and evaluating a model.

Here, we featurized only the ligand. The signal we observed in R^2 reflects the ability of circular fingerprints and random forests to learn general features that make ligands "drug-like."

Let's take a quick look at what the algorithm determines to be high- and low-affinity drugs.

In [39]:
predictions = pd.read_csv(test_csv_out.name)
print(predictions)
predictions = predictions.sort(['label'], ascending=[0])
    Unnamed: 0                                                ids     label  \
0            0    2d3uCC1CCCCC1S(O)(O)NC1CC(C2CCC(CN)CC2)SC1C(O)O  0.577883   
1            1  3cyxCC(C)(C)NC(O)C1CC2CCCCC2C[NH+]1CC(O)C(CC1C...  1.142494   
2           24                      2zxdCC(C)C1[NH2+]CC(O)C(O)C1O -0.310858   
3           25                     3bfu[NH3+]C(CC1NSNC1O)C([O-])O  0.238070   
4           28                                 3u9qCCCCCCCCCC(O)O -0.750001   
5           44              3l7bNC1CCN(C2OC(CO)C(O)C(F)C2O)C(O)N1 -1.785122   
6           45  3oztOC(NCCCC1OC(N2CCC(O)CC2)C(O)C1O)C1CC(N(O)O... -0.880698   
7           49  3ivgCOC1CCC2C(CC(CNS(O)(O)C3CC4CCCCC4O3)N2CC(O... -0.791824   
8           54  1sqa[NH3+]CC1CCC(NC(O)C2CC3CCC(C([NH3+])[NH3+]...  1.775069   
9           56                     2xdlCCN(CC)C(O)C1CCC(O)C(OC)C1 -1.419170   
10          57                      3udhOC1NC2CCCCC2C12CC[NH2+]C2 -1.549867   
11          73         1w4oOCC1OC(N2CCC(O)NC2O)C(O)C1O[PH](O)(O)O -0.310858   
12          76                               3gy4NC(N)C1CCC(N)CC1 -0.373593   
13          78                          3b3wCC(C)CC([NH3+])C(=O)O -0.849330   
14          80       2zjwOC1CC2C(O)OC3C(O)C(O)CC4C(O)OC(C1O)C2C43  0.985658   
15          83  1os0[NH3+]C(CC1CCCCC1)[PH](O)(O)CC(CC1CCCCC1)C...  0.112601   
16          92          3acwOC1(C2CCC(C3CCCCC3)CC2)C[NH+]2CCC1CC2 -0.551341   
17          98           3mssCNC(O)C([NH3+])CC1CCC(OCC2CCCCC2)CC1 -0.603620   
18         107  2p4yCOC1CCC2C(C1)ONC2C1C(C)N(CC2CC(OC(C)C([O-]...  1.665283   
19         108                       3d4zOCC1C(O)C(O)C(O)C2NCCN21 -0.483378   
20         110                     3mfvNC1NCC(CCC([NH3+])C(O)O)N1 -1.722388   
21         112        1f8dCC(O)NC1C(O)CC(C(O)O)OC1C(O)C(O)C[NH3+] -1.262334   
22         117  3nw9CC1NCNC2C1NCN2C1OC(CCCNC(O)C2CC(C3CCC(F)CC...  1.665283   
23         131               3ehyCOC1CCC(S(O)(O)NC(C)C([O-])O)CC1  0.018499   
24         132  3ov1CC(O)NC(CC1CCC(O[PH](O)(O)O)CC1)C(O)NC1(C(... -0.321314   
25         133         4de1OC(NC1CCCC(C2[N-]NNN2)C1)C1CCC2NNCC2C1  0.076005   
26         136  2xnbCC1C(C2CCNC(NC3CCC(N4CC[NH2+]CC4)CC3)N2)SC...  0.530832   
27         139    2obfOCC1CC2CCC(S(O)(O)NC3CCC(Cl)CC3)CC2C[NH2+]1  1.586865   
28         154  3l3n[NH3+]CCCCC([NH2+]C(CCC1CCCCC1)C(O)O)C(O)N...  1.236596   
29         158                                3vh9OC1CCCC2CCCNC12  0.222386   
30         161                          2jdyCOC1OC(CO)C(O)C(O)C1O -0.755229   
31         174                        1n2vCCCCC1NC2C(N1)C(O)NNC2O -0.906837   
32         175           2votOCC1C(O)C(O)C(O)C2NC(CNC3CCCCC3)CN21  0.692896   
33         178                      3n7a[O-]C(O)C1(O)CCC(O)C(O)C1 -1.105497   

    label_pred  label_weight   y_means    y_stds  
0    -0.276078             1  5.814615  1.912819  
1     0.617702             1  5.814615  1.912819  
2    -0.122412             1  5.814615  1.912819  
3    -0.169293             1  5.814615  1.912819  
4    -0.804344             1  5.814615  1.912819  
5    -0.818249             1  5.814615  1.912819  
6     0.258280             1  5.814615  1.912819  
7    -0.275210             1  5.814615  1.912819  
8     0.015193             1  5.814615  1.912819  
9     0.054695             1  5.814615  1.912819  
10    0.007379             1  5.814615  1.912819  
11   -1.417286             1  5.814615  1.912819  
12   -0.110257             1  5.814615  1.912819  
13   -0.996477             1  5.814615  1.912819  
14   -0.187090             1  5.814615  1.912819  
15   -0.009116             1  5.814615  1.912819  
16   -0.614229             1  5.814615  1.912819  
17   -0.004775             1  5.814615  1.912819  
18    0.971047             1  5.814615  1.912819  
19   -0.298650             1  5.814615  1.912819  
20   -0.248730             1  5.814615  1.912819  
21   -0.250467             1  5.814615  1.912819  
22    0.432782             1  5.814615  1.912819  
23    0.060338             1  5.814615  1.912819  
24    1.404046             1  5.814615  1.912819  
25    0.669358             1  5.814615  1.912819  
26    0.748796             1  5.814615  1.912819  
27    0.014759             1  5.814615  1.912819  
28    0.317316             1  5.814615  1.912819  
29   -0.547380             1  5.814615  1.912819  
30   -0.352911             1  5.814615  1.912819  
31   -0.695403             1  5.814615  1.912819  
32   -0.112862             1  5.814615  1.912819  
33   -0.376264             1  5.814615  1.912819  
In [40]:
top_ligand = predictions.iloc[0]['ids']
ligand1 = convert_lines_to_mdtraj(dataset.loc[dataset['complex_id']==top_ligand]['ligand_pdb'].values[0])

def visualize_ligand(ligand_mdtraj):
  traj = nglview.MDTrajTrajectory( ligand_mdtraj ) # load file from RCSB PDB
  ngltraj = nglview.NGLWidget( traj )
  ngltraj.representations = [
  { "type": "ball+stick", "params": {
  "sele": "all"
  } }
  ]
  return ngltraj

ngltraj = visualize_ligand(ligand1)
ngltraj
In [41]:
worst_ligand = predictions.iloc[predictions.shape[0]-2]['ids']
ligand1 = convert_lines_to_mdtraj(dataset.loc[dataset['complex_id']==worst_ligand]['ligand_pdb'].values[0])
ngltraj = visualize_ligand(ligand1)
ngltraj

The protein-ligand complex view.¶

The preceding simple example, in few yet intuitive lines of code, traces the machine learning arc from featurizing a raw dataset to fitting and evaluating a model.

In this next section, we illustrate deepchem's modularity, and thereby the ease with which one can explore different featurization schemes, different models, and combinations thereof, to achieve the best performance on a given dataset. We will demonstrate this by examining protein-ligand interactions.

In the previous section, we featurized only the ligand. The signal we observed in R^2 reflects the ability of circular fingerprints and random forests to learn general features that make ligands "drug-like." However, the affinity of a drug for a target is determined not only by the drug itself, of course, but the way in which it interacts with a protein.

In [42]:
train_dir, validation_dir, test_dir = tempfile.mkdtemp(), tempfile.mkdtemp(), tempfile.mkdtemp()
splittype="random"
train_samples, validation_samples, test_samples = featurized_samples.train_valid_test_split(
    splittype, train_dir, validation_dir, test_dir, seed=2016)

task_types = {"label": "regression"}
performance = pd.DataFrame()
import deepchem.models.standard
from deepchem.models.standard import SklearnModel
from deepchem.utils.dataset import Dataset
from deepchem.utils.evaluate import Evaluator

n_trees_vals = [10, 20, 40, 80, 160]
for feature_type in (complex_featurizers + compound_featurizers):
    train_dataset = Dataset(data_dir=train_dir, samples=train_samples, 
                        featurizers=[feature_type], tasks=["label"])
    validation_dataset = Dataset(data_dir=validation_dir, samples=validation_samples, 
                       featurizers=[feature_type], tasks=["label"])

    input_transforms = ["normalize", "truncate"]
    output_transforms = ["normalize"]
    train_dataset.transform(input_transforms, output_transforms)
    validation_dataset.transform(input_transforms, output_transforms)
    
    for n_trees in n_trees_vals:
        model_params = {"data_shape": train_dataset.get_data_shape()}

        model = SklearnModel(task_types, model_params, model_instance=RandomForestRegressor(n_estimators=n_trees))
        model.fit(train_dataset)
        model_dir = tempfile.mkdtemp()
        model.save(model_dir)


        evaluator = Evaluator(model, train_dataset, verbose=True)
        with tempfile.NamedTemporaryFile() as train_csv_out:
          with tempfile.NamedTemporaryFile() as train_stats_out:
            _, train_r2score = evaluator.compute_model_performance(
                train_csv_out, train_stats_out)

        evaluator = Evaluator(model, validation_dataset, verbose=True)
        with tempfile.NamedTemporaryFile() as validation_csv_out:
          with tempfile.NamedTemporaryFile() as validation_stats_out:
            _, validation_r2score = evaluator.compute_model_performance(
                validation_csv_out, validation_stats_out)

        train_valid_performance = pd.concat([train_r2score, validation_r2score])
        train_valid_performance["split"] = ["train", "validation"]
        train_valid_performance["featurizer"] = [str(feature_type.__class__), str(feature_type.__class__)]
        train_valid_performance["n_trees"] = [n_trees, n_trees]
        print(train_valid_performance)
        performance = pd.concat([performance, train_valid_performance])
performance
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
  task_name  r2_score  rms_error       split  \
0     label  0.850802   0.890800       train   
0     label  0.380784   1.172148  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.grid_featurizer.G...       10  
0  <class 'deepchem.featurizers.grid_featurizer.G...       10  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
  task_name  r2_score  rms_error       split  \
0     label  0.877179    0.80823       train   
0     label  0.157616    1.36715  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.grid_featurizer.G...       20  
0  <class 'deepchem.featurizers.grid_featurizer.G...       20  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
  task_name  r2_score  rms_error       split  \
0     label  0.900207   0.728532       train   
0     label  0.279117   1.264717  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.grid_featurizer.G...       40  
0  <class 'deepchem.featurizers.grid_featurizer.G...       40  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
  task_name  r2_score  rms_error       split  \
0     label  0.895135   0.746817       train   
0     label  0.303473   1.243169  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.grid_featurizer.G...       80  
0  <class 'deepchem.featurizers.grid_featurizer.G...       80  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
  task_name  r2_score  rms_error       split  \
0     label  0.904690   0.711980       train   
0     label  0.275001   1.268323  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.grid_featurizer.G...      160  
0  <class 'deepchem.featurizers.grid_featurizer.G...      160  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
  task_name  r2_score  rms_error       split  \
0     label  0.818402   0.982774       train   
0     label  0.045733   1.455111  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.fingerprints.Circ...       10  
0  <class 'deepchem.featurizers.fingerprints.Circ...       10  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
  task_name  r2_score  rms_error       split  \
0     label  0.837979   0.928291       train   
0     label  0.257332   1.283685  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.fingerprints.Circ...       20  
0  <class 'deepchem.featurizers.fingerprints.Circ...       20  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
  task_name  r2_score  rms_error       split  \
0     label  0.865292   0.846439       train   
0     label  0.251678   1.288562  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.fingerprints.Circ...       40  
0  <class 'deepchem.featurizers.fingerprints.Circ...       40  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6e40>
  task_name  r2_score  rms_error       split  \
0     label  0.872805   0.822495       train   
0     label  0.279337   1.264525  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.fingerprints.Circ...       80  
0  <class 'deepchem.featurizers.fingerprints.Circ...       80  
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea97ca6f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8f60>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8b70>
  task_name  r2_score  rms_error       split  \
0     label  0.871897   0.825426       train   
0     label  0.278349   1.265391  validation   

                                          featurizer  n_trees  
0  <class 'deepchem.featurizers.fingerprints.Circ...      160  
0  <class 'deepchem.featurizers.fingerprints.Circ...      160  
Out[42]:
task_name r2_score rms_error split featurizer n_trees
0 label 0.850802 0.890800 train <class 'deepchem.featurizers.grid_featurizer.G... 10
0 label 0.380784 1.172148 validation <class 'deepchem.featurizers.grid_featurizer.G... 10
0 label 0.877179 0.808230 train <class 'deepchem.featurizers.grid_featurizer.G... 20
0 label 0.157616 1.367150 validation <class 'deepchem.featurizers.grid_featurizer.G... 20
0 label 0.900207 0.728532 train <class 'deepchem.featurizers.grid_featurizer.G... 40
0 label 0.279117 1.264717 validation <class 'deepchem.featurizers.grid_featurizer.G... 40
0 label 0.895135 0.746817 train <class 'deepchem.featurizers.grid_featurizer.G... 80
0 label 0.303473 1.243169 validation <class 'deepchem.featurizers.grid_featurizer.G... 80
0 label 0.904690 0.711980 train <class 'deepchem.featurizers.grid_featurizer.G... 160
0 label 0.275001 1.268323 validation <class 'deepchem.featurizers.grid_featurizer.G... 160
0 label 0.818402 0.982774 train <class 'deepchem.featurizers.fingerprints.Circ... 10
0 label 0.045733 1.455111 validation <class 'deepchem.featurizers.fingerprints.Circ... 10
0 label 0.837979 0.928291 train <class 'deepchem.featurizers.fingerprints.Circ... 20
0 label 0.257332 1.283685 validation <class 'deepchem.featurizers.fingerprints.Circ... 20
0 label 0.865292 0.846439 train <class 'deepchem.featurizers.fingerprints.Circ... 40
0 label 0.251678 1.288562 validation <class 'deepchem.featurizers.fingerprints.Circ... 40
0 label 0.872805 0.822495 train <class 'deepchem.featurizers.fingerprints.Circ... 80
0 label 0.279337 1.264525 validation <class 'deepchem.featurizers.fingerprints.Circ... 80
0 label 0.871897 0.825426 train <class 'deepchem.featurizers.fingerprints.Circ... 160
0 label 0.278349 1.265391 validation <class 'deepchem.featurizers.fingerprints.Circ... 160
In [44]:
%matplotlib inline

import matplotlib
import numpy as np
import matplotlib.pyplot as plt

df = pd.DataFrame(performance[['r2_score','split','featurizer']].values, index=performance['n_trees'].values, columns=['r2_score', 'split', 'featurizer'])
df = df.loc[df['split']=="validation"]
df = df.drop('split', 1)
fingerprint_df = df[df['featurizer'].str.contains('fingerprint')].drop('featurizer', 1)
print fingerprint_df
fingerprint_df.columns = ['ligand fingerprints']
grid_df = df[df['featurizer'].str.contains('grid')].drop('featurizer', 1)
grid_df.columns = ['complex features']
df = pd.concat([fingerprint_df, grid_df], axis=1)
print(df)

plt.clf()
df.plot()
plt.ylabel("$R^2$")
plt.xlabel("Number of trees")
      r2_score
10   0.0457328
20    0.257332
40    0.251678
80    0.279337
160   0.278349
    ligand fingerprints complex features
10            0.0457328         0.380784
20             0.257332         0.157616
40             0.251678         0.279117
80             0.279337         0.303473
160            0.278349         0.275001
Out[44]:
<matplotlib.text.Text at 0x7fea5ec33fd0>
<matplotlib.figure.Figure at 0x7fea97e9c210>
In [63]:
train_dir, validation_dir, test_dir = tempfile.mkdtemp(), tempfile.mkdtemp(), tempfile.mkdtemp()
splittype="random"
train_samples, validation_samples, test_samples = featurized_samples.train_valid_test_split(
    splittype, train_dir, validation_dir, test_dir, seed=2016)

feature_type = complex_featurizers
train_dataset = Dataset(data_dir=train_dir, samples=train_samples, 
                    featurizers=feature_type, tasks=["label"])
validation_dataset = Dataset(data_dir=validation_dir, samples=validation_samples, 
                   featurizers=feature_type, tasks=["label"])
test_dataset = Dataset(data_dir=test_dir, samples=test_samples, 
                   featurizers=feature_type, tasks=["label"])

input_transforms = ["normalize", "truncate"]
output_transforms = ["normalize"]
train_dataset.transform(input_transforms, output_transforms)
validation_dataset.transform(input_transforms, output_transforms)
test_dataset.transform(input_transforms, output_transforms)

model_params = {"data_shape": train_dataset.get_data_shape()}

rf_model = SklearnModel(task_types, model_params, model_instance=RandomForestRegressor(n_estimators=20))
rf_model.fit(train_dataset)
model_dir = tempfile.mkdtemp()
rf_model.save(model_dir)


evaluator = Evaluator(rf_model, train_dataset, verbose=True)
with tempfile.NamedTemporaryFile() as train_csv_out:
  with tempfile.NamedTemporaryFile() as train_stats_out:
    _, train_r2score = evaluator.compute_model_performance(
        train_csv_out, train_stats_out)

evaluator = Evaluator(rf_model, test_dataset, verbose=True)
test_csv_out = tempfile.NamedTemporaryFile()
with tempfile.NamedTemporaryFile() as test_stats_out:
    predictions, test_r2score = evaluator.compute_model_performance(
        test_csv_out, test_stats_out)

train_test_performance = pd.concat([train_r2score, test_r2score])
train_test_performance["split"] = ["train", "test"]
train_test_performance["featurizer"] = [str(feature_type.__class__), str(feature_type.__class__)]
train_test_performance["n_trees"] = [n_trees, n_trees]
print(train_test_performance)
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea72843d20>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7feacef65300>
Saving predictions to <open file '<fdopen>', mode 'w+b' at 0x7fea79578270>
Saving model performance scores to <open file '<fdopen>', mode 'w+b' at 0x7fea994b8810>
  task_name  r2_score  rms_error  split     featurizer  n_trees
0     label  0.862417   0.855422  train  <type 'list'>      160
0     label  0.381613   1.323630   test  <type 'list'>      160
In [60]:
import deepchem.models.deep
reload(deepchem.models.deep)
from deepchem.models.deep import SingleTaskDNN
import numpy.random
from operator import mul
import itertools

model_params = {"activation": "relu",
                "momentum": .9,
                "batch_size": 64,
                "nb_epoch": 30,
                "data_shape": train_dataset.get_data_shape()}

lr_list = np.power(10., np.random.uniform(-3, -1, size=4))
decay_list = np.power(10., np.random.uniform(-6, -2, size=4))
nb_hidden_list = [10, 100, 1000]
nb_epoch_list = [5]
nesterov_list = [False]
dropout_list = [0.05, .1]
nb_layers_list = [2]
init_list = ["glorot_uniform"]
batchnorm_list = [True, False]
hyperparameters = [lr_list, decay_list, nb_hidden_list, nb_epoch_list,
                   nesterov_list, dropout_list, nb_layers_list,
                   init_list, batchnorm_list]
num_combinations = reduce(mul, [len(l) for l in hyperparameters])
best_validation_score = -np.inf
best_hyperparams = None
best_model, best_model_dir = None, None
performance_df = pd.DataFrame()
for ind, hyperparameter_tuple in enumerate(itertools.product(*hyperparameters)):
    print("Testing %s" % str(hyperparameter_tuple))
    print("Combo %d/%d" % (ind, num_combinations))
    (lr, decay, nb_hidden, nb_epoch, nesterov, dropout,
     nb_layers, init, batchnorm) = hyperparameter_tuple
    model_params["nb_hidden"] = nb_hidden
    model_params["decay"] = decay
    model_params["learning_rate"] = lr
    model_params["nb_epoch"] = nb_epoch
    model_params["nesterov"] = nesterov
    model_params["dropout"] = dropout
    model_params["nb_layers"] = nb_layers
    model_params["init"] = init
    model_params["batchnorm"] = batchnorm
    model_dir = tempfile.mkdtemp()
    model = SingleTaskDNN(task_types, model_params)
    model.fit(train_dataset)
    model.save(model_dir)
    
    evaluator = Evaluator(model, validation_dataset)
    valid_csv_out = tempfile.NamedTemporaryFile()
    valid_stats_out = tempfile.NamedTemporaryFile()
    df, r2score = evaluator.compute_model_performance(
        valid_csv_out, valid_stats_out)
    r2score["hyperparameters"] = str(hyperparameters)
    performance_df = pd.concat([performance_df, r2score])
    valid_r2_score = r2score.iloc[0]["r2_score"]
    print("learning_rate %f, nb_hidden %d, nb_epoch %d, nesterov %s, dropout %f => Validation set R^2 %f" %
          (lr, nb_hidden, nb_epoch, str(nesterov), dropout, valid_r2_score))
    if valid_r2_score > best_validation_score:
        best_validation_score = valid_r2_score
        best_hyperparams = hyperparameter_tuple
        if best_model_dir is not None:
            shutil.rmtree(best_model_dir)
        best_model_dir = model_dir
        best_model = model
    else:
        shutil.rmtree(model_dir)
    print("Best hyperparameters so-far: %s" % str(best_hyperparams))
    print("best_validation_score so-far: %f" % best_validation_score)

print("Best hyperparameters: %s" % str(best_hyperparams))
print("best_validation_score: %f" % best_validation_score)
best_dnn = best_model
Testing (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 0/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.052310
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
best_validation_score so-far: 0.052310
Testing (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 1/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.236239
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.236239
Testing (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 2/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.032232
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.236239
Testing (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 3/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.124017
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.236239
Testing (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 4/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.043953
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.236239
Testing (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 5/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.463186
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 6/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.034678
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 7/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.148112
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 8/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.061816
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 9/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.312484
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 10/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.053784
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 11/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.230733
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 12/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.022895
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 13/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.029677
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 14/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.019486
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 15/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.071316
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 16/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.045613
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 17/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.174035
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 18/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.040136
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 19/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.109092
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 20/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.064414
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 21/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.255765
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 22/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.058625
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 23/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.205650
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 24/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.065106
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 25/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.111463
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 26/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.056052
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 27/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.195030
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 28/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.044930
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 29/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.069249
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 30/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.037436
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 31/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.341537
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 32/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.060559
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 33/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.229558
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 34/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.056056
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 35/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.265421
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 36/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.026662
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 37/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.170494
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 38/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.056345
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 39/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.156114
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 40/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.056831
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 41/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.247951
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 42/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.037699
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 43/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.074958
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 44/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.060160
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 45/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.121475
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 46/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.058707
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0018171038259935624, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 47/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.001817, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.240001
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 48/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.070213
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 49/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.160359
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 50/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.064826
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 51/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.256100
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 52/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.077248
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 53/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.295046
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 54/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.057201
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 55/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.292408
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 56/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.078842
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 57/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.181136
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 58/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.074854
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 59/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.104714
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 60/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.021680
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 61/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.224596
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 62/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.021744
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 63/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.173444
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 64/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.053579
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 65/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.134248
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 66/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.054936
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 67/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.127220
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 68/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.078482
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 69/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.210075
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 70/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.072222
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 71/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.212350
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 72/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.041186
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 73/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.154450
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 74/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.034845
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 75/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.189899
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 76/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.074428
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 77/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.006851
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 78/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.051135
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 79/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.049081
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 80/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.077782
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 81/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.196231
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 82/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.074711
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 83/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.149155
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 84/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.068293
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 85/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.382870
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 86/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.063284
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 87/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.206639
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 88/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.074274
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 89/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.018308
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 90/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.043826
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 91/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.102377
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 92/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.077323
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 93/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.157235
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 94/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.075745
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.0024041061631945851, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 95/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.002404, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.117638
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 96/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.122820
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 97/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.226547
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 98/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.377501
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 99/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.038925
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 100/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.067861
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 101/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.006282
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 102/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.049499
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 103/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.131996
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 104/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.096081
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 105/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 106/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.150608
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 107/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 108/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.110046
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 109/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.017482
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 110/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.004933
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 111/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.007658
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 112/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.068468
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 113/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 114/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.132118
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 115/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.003936
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 116/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.209971
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 117/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 118/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.092970
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 119/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 120/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.087274
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 121/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.059660
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 122/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.064765
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 123/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.139451
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 124/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.255557
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 125/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.331071
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 126/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.177373
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 127/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 128/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.097581
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 129/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 130/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.016843
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 131/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 132/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.158728
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 133/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.014144
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 134/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.007749
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 135/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.006227
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 136/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.000649
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 137/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.154036
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 138/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.156509
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 139/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.073605
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 140/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.064138
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 141/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 142/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.042885
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.087836491715029247, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 143/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.087836, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 144/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.165792
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 145/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.070515
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 146/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.159047
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 147/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.177436
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 148/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.072050
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 149/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.178082
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 150/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.161901
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 151/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.543556
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 152/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.191796
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 153/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 154/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.239052
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0049446750959222293, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 155/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 156/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.267473
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 157/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.224696
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 158/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.134377
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 159/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.160847
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 160/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.173279
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 161/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.069503
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 162/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.265625
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 163/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.097968
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 164/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.144767
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 165/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 166/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.262830
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 0.0046105219251146717, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 167/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 168/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.121178
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 169/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.191104
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 170/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.103315
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 171/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.009138
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 172/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.050506
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 173/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 174/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.148493
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 175/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.006741
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 176/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.116588
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 177/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 178/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.099925
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 1.1511283983866343e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 179/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 180/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 -0.118604
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 10, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 181/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.045078
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 182/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.057166
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 10, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 183/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 10, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 0.001017
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 184/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.176240
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 185/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.000722
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 186/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.003826
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 100, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 187/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 100, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -7744071744746713088.000000
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', True)
Combo 188/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 0.088296
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 1000, 5, False, 0.05, 2, 'glorot_uniform', False)
Combo 189/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.050000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', True)
Combo 190/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 -0.030295
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Testing (0.088157860097194923, 2.899543306854146e-06, 1000, 5, False, 0.1, 2, 'glorot_uniform', False)
Combo 191/192
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
learning_rate 0.088158, nb_hidden 1000, nb_epoch 5, nesterov False, dropout 0.100000 => Validation set R^2 nan
Best hyperparameters so-far: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score so-far: 0.463186
Best hyperparameters: (0.0018171038259935624, 0.0049446750959222293, 100, 5, False, 0.05, 2, 'glorot_uniform', False)
best_validation_score: 0.463186
In [66]:
 
ids             3gnwCC1CCCC(C(O)N2C3CCCC(O)C3NC3CC(C)(C)CS(O)(...
label                                                     1.31713
label_pred                                                1.20469
label_weight                                                    1
y_means                                                     6.883
y_stds                                                     1.6832
Name: 155, dtype: object
DNN Test set R^2 0.442633
In [99]:
dnn_test_csv_out = tempfile.NamedTemporaryFile()
dnn_test_stats_out = tempfile.NamedTemporaryFile()
dnn_test_evaluator = Evaluator(best_dnn, test_dataset)
dnn_test_df, dnn_test_r2score = dnn_test_evaluator.compute_model_performance(
    dnn_test_csv_out, dnn_test_stats_out)
dnn_test_r2_score = dnn_test_r2score.iloc[0]["r2_score"]
print("DNN Test set R^2 %f" % (dnn_test_r2_score))

task = "label"
dnn_predicted_test = np.array(dnn_test_df[task + "_pred"])
dnn_true_test = np.array(dnn_test_df[task])

plt.clf()
plt.scatter(dnn_true_test, dnn_predicted_test)
plt.xlabel('Predicted Ki')
plt.ylabel('True Ki')
plt.title(r'DNN predicted vs. true Ki')
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.plot([-3, 3], [-3, 3], marker=".", color='k')

rf_test_csv_out = tempfile.NamedTemporaryFile()
rf_test_stats_out = tempfile.NamedTemporaryFile()
rf_test_evaluator = Evaluator(rf_model, test_dataset)
rf_test_df, rf_test_r2score = rf_test_evaluator.compute_model_performance(
    rf_test_csv_out, rf_test_stats_out)
rf_test_r2_score = rf_test_r2score.iloc[0]["r2_score"]
print("RF Test set R^2 %f" % (rf_test_r2_score))
plt.show()

task = "label"
rf_predicted_test = np.array(rf_test_df[task + "_pred"])
rf_true_test = np.array(rf_test_df[task])
plt.scatter(rf_true_test, rf_predicted_test)
plt.xlabel('Predicted Ki')
plt.ylabel('True Ki')
plt.title(r'RF predicted vs. true Ki')
plt.xlim([-2, 2])
plt.ylim([-2, 2])
plt.plot([-3, 3], [-3, 3], marker=".", color='k')
plt.show()
DNN Test set R^2 0.442633
RF Test set R^2 0.381613
In [101]:
predictions = dnn_test_df.sort(['label'], ascending=[0])
In [102]:
top_complex = predictions.iloc[0]['ids']
best_complex = dataset.loc[dataset['complex_id']==top_complex]

protein_mdtraj = convert_lines_to_mdtraj(best_complex["protein_pdb"].values[0])
ligand_mdtraj = convert_lines_to_mdtraj(best_complex["ligand_pdb"].values[0])
complex_mdtraj = combine_mdtraj(protein_mdtraj, ligand_mdtraj)
ngltraj = visualize_complex(complex_mdtraj)
ngltraj
In [ ]:
 
In [103]:
top_complex = predictions.iloc[1]['ids']
best_complex = dataset.loc[dataset['complex_id']==top_complex]

protein_mdtraj = convert_lines_to_mdtraj(best_complex["protein_pdb"].values[0])
ligand_mdtraj = convert_lines_to_mdtraj(best_complex["ligand_pdb"].values[0])
complex_mdtraj = combine_mdtraj(protein_mdtraj, ligand_mdtraj)
ngltraj = visualize_complex(complex_mdtraj)
ngltraj
In [104]:
top_complex = predictions.iloc[predictions.shape[0]-1]['ids']
best_complex = dataset.loc[dataset['complex_id']==top_complex]

protein_mdtraj = convert_lines_to_mdtraj(best_complex["protein_pdb"].values[0])
ligand_mdtraj = convert_lines_to_mdtraj(best_complex["ligand_pdb"].values[0])
complex_mdtraj = combine_mdtraj(protein_mdtraj, ligand_mdtraj)
ngltraj = visualize_complex(complex_mdtraj)
ngltraj