Written by Bharath Ramsundar and Evan Feinberg

Copyright 2016, Stanford University

Computationally predicting molecular solubility through is useful for drug-discovery. In this tutorial, we will use the deepchem library to fit a simple statistical model that predicts the solubility of drug-like compounds. The process of fitting this model involves four steps:

  1. Loading a chemical dataset, consisting of a series of compounds along with aqueous solubility measurements.
  2. Transforming each compound into a feature vector $v \in \mathbb{R}^n$ comprehensible to statistical learning methods.
  3. Fitting a simple model that maps feature vectors to estimates of aqueous solubility.
  4. Visualizing the results.

We need to load a dataset of estimated aqueous solubility measurements [1] into deepchem. The data is in CSV format and contains SMILES strings, predicted aqueaous solubilities, and a number of extraneous (for our purposes) molecular properties. Here is an example line from the dataset:

Compound ID ESOL predicted log solubility in mols per litre Minimum Degree Molecular Weight Number of H-Bond Donors Number of Rings Number of Rotatable Bonds Polar Surface Area measured log solubility in mols per litre smiles
benzothiazole -2.733 2 135.191 0 2 0 12.89 -1.5 c2ccc1scnc1c2

Most of these fields are not useful for our purposes. The two fields that we will need are the "smiles" field and the "measured log solubility in mols per litre". The "smiles" field holds a SMILES string [2] that specifies the compound in question. Before we load this data into deepchem, we will load the data into python and do some simple preliminary analysis to gain some intuition for the dataset.

In [1]:
%load_ext autoreload
%autoreload 2
from deepchem.utils.save import load_from_disk

dataset_file= "../datasets/delaney-processed.csv"
dataset = load_from_disk(dataset_file)
print("Columns of dataset: %s" % str(dataset.columns.values))
print("Number of examples in dataset: %s" % str(dataset.shape[0]))
Columns of dataset: ['Compound ID' 'ESOL predicted log solubility in mols per litre'
 'Minimum Degree' 'Molecular Weight' 'Number of H-Bond Donors'
 'Number of Rings' 'Number of Rotatable Bonds' 'Polar Surface Area'
 'measured log solubility in mols per litre' 'smiles']
Number of examples in dataset: 1128

To gain a visual understanding of compounds in our dataset, let's draw them using rdkit. We define a couple of helper functions to get started.

In [2]:
import tempfile
from rdkit import Chem
from rdkit.Chem import Draw
from itertools import islice
from IPython.display import Image, HTML, display

def display_images(filenames):
    """Helper to pretty-print images."""
    imagesList=''.join(
        ["<img style='width: 140px; margin: 0px; float: left; border: 1px solid black;' src='%s' />"
         % str(s) for s in sorted(filenames)])
    display(HTML(imagesList))

def mols_to_pngs(mols, basename="test"):
    """Helper to write RDKit mols to png files."""
    filenames = []
    for i, mol in enumerate(mols):
        filename = "%s%d.png" % (basename, i)
        Draw.MolToFile(mol, filename)
        filenames.append(filename)
    return filenames

Now, we display some compounds from the dataset:

In [3]:
num_to_display = 12
molecules = []
for _, data in islice(dataset.iterrows(), num_to_display):
    molecules.append(Chem.MolFromSmiles(data["smiles"]))
display_images(mols_to_pngs(molecules))

Analyzing the distribution of solubilities shows us a nice spread of data.

In [4]:
%matplotlib inline
import seaborn
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

solubilities = np.array(dataset["measured log solubility in mols per litre"])
n, bins, patches = plt.hist(solubilities, 50, facecolor='green', alpha=0.75)
plt.xlabel('Measured log-solubility in mols/liter')
plt.ylabel('Number of compounds')
plt.title(r'Histogram of solubilities')
plt.grid(True)
plt.show()

With our preliminary analysis completed, we return to the original goal of constructing a predictive statistical model of molecular solubility using deepchem. The first step in creating such a molecule is translating each compound into a vectorial format that can be understood by statistical learning techniques. This process is commonly called featurization. deepchem packages a number of commonly used featurization for user convenience. In this tutorial, we will use ECPF4 fingeprints [3].

deepchem offers an object-oriented API for featurization. To get started with featurization, we first construct a Featurizer object. deepchem provides the CircularFingeprint class (a subclass of Featurizer that performs ECFP4 featurization).

In [5]:
from deepchem.featurizers.fingerprints import CircularFingerprint

featurizers = [CircularFingerprint(size=1024)]

Now, let's perform the actual featurization. deepchem provides the DataFeaturizer class for this purpose. The featurize() method for this class loads data from disk and uses provided Featurizerinstances to transform the provided data into feature vectors. The method constructs an instance of class FeaturizedSamples that has useful methods, such as an iterator, over the featurized data.

In [6]:
import tempfile, shutil
from deepchem.featurizers.featurize import DataFeaturizer

#Make directories to store the raw and featurized datasets.
feature_dir = tempfile.mkdtemp()
samples_dir = tempfile.mkdtemp()

featurizer = DataFeaturizer(tasks=["measured log solubility in mols per litre"],
                            smiles_field="smiles",
                            compound_featurizers=featurizers)
featurized_samples = featurizer.featurize(dataset_file, feature_dir, samples_dir)

When constructing statistical models, it's necessary to separate the provided data into train/test subsets. The train subset is used to learn the statistical model, while the test subset is used to evaluate the learned model. In practice, it's often useful to elaborate this split further and perform a train/validation/test split. The validation set is used to perform model selection. Proposed models are evaluated on the validation-set, and the best performed model is at the end tested on the test-set.

Choosing the proper method of performing a train/validation/test split can be challenging. Standard machine learning practice is to perform a random split of the data into train/validation/test, but random splits are not well suited for the purposes of chemical informatics. For our predictive models to be useful, we require them to have predictive power in portions of chemical space beyond the set of molecules in the training data. Consequently, our models should use splits of the data that separate compounds in the training set from those in the validation and test-sets. We use Bemis-Murcko scaffolds [5] to perform this separation (all compounds that share an underlying molecular scaffold will be placed into the same split in the train/test/validation split).

In [7]:
splittype = "scaffold"

train_dir = tempfile.mkdtemp()
valid_dir = tempfile.mkdtemp()
test_dir = tempfile.mkdtemp()
train_samples, valid_samples, test_samples = featurized_samples.train_valid_test_split(
    splittype, train_dir, valid_dir, test_dir)

Let's visually inspect some of the molecules in the separate splits to verify that they appear structurally dissimilar. The FeaturizedSamples class provides an itersamples method that lets us obtain the underlying compounds in each split.

In [8]:
train_mols = [Chem.MolFromSmiles(str(compound["smiles"]))
              for compound in islice(train_samples.itersamples(), num_to_display)]
display_images(mols_to_pngs(train_mols, basename="train"))
In [9]:
valid_mols = [Chem.MolFromSmiles(str(compound["smiles"]))
              for compound in islice(valid_samples.itersamples(), num_to_display)]
display_images(mols_to_pngs(valid_mols, basename="valid"))

Notice the visual distinction between the train/validation splits. The most-common scaffolds are reserved for the train split, with the rarer scaffolds allotted to validation/test.

To perform machine learning upon these datasets, we need to convert the samples into datasets suitable for machine-learning (that is, into data matrix $X \in \mathbb{R}^{n\times d}$ where $n$ is the number of samples and $d$ the dimensionality of the feature vector, and into label vector $y \in \mathbb{R}^n$). deepchem provides the Dataset class to facilitate this transformation. We simply need to instantiate separate instances of the Dataset() class, one corresponding to each split of the data. This style lends itself easily to validation-set hyperparameter searches, which we illustate below.

In [10]:
from deepchem.utils.dataset import Dataset
train_dataset = Dataset(data_dir=train_dir, samples=train_samples, 
                        featurizers=featurizers, tasks=["measured log solubility in mols per litre"])
valid_dataset = Dataset(data_dir=valid_dir, samples=valid_samples, 
                        featurizers=featurizers, tasks=["measured log solubility in mols per litre"])
test_dataset = Dataset(data_dir=test_dir, samples=test_samples, 
                       featurizers=featurizers, tasks=["measured log solubility in mols per litre"])

The performance of common machine-learning algorithms can be very sensitive to preprocessing of the data. One common transformation applied to data is to normalize it to have zero-mean and unit-standard-deviation. We will apply this transformation to the log-solubility (as seen above, the log-solubility ranges from -12 to 2).

In [11]:
input_transforms = []
output_transforms = ["normalize"]
train_dataset.transform(input_transforms, output_transforms)
valid_dataset.transform(input_transforms, output_transforms)
test_dataset.transform(input_transforms, output_transforms)

The next step after processing the data is to start fitting simple learning models to our data. deepchem provides a number of machine-learning model classes.

In particular, deepchem provides a convenience class, SklearnModel that wraps any machine-learning model available in scikit-learn [6]. Consequently, we will start by building a simple random-forest regressor that attempts to predict the log-solubility from our computed ECFP4 features. To train the model, we instantiate the SklearnModel object, then call the fit() method on the train_dataset we constructed above. We then save the model to disk.

In [12]:
from sklearn.ensemble import RandomForestRegressor
from deepchem.models.standard import SklearnModel

model_dir = tempfile.mkdtemp()
task_types = {"measured log solubility in mols per litre": "regression"}
model_params = {"data_shape": train_dataset.get_data_shape()}
model = SklearnModel(task_types, model_params, model_instance=RandomForestRegressor())
model.fit(train_dataset)
model.save(model_dir)
shutil.rmtree(model_dir)

We next evaluate the model on the validation set to see its predictive power. deepchem provides the Evaluator class to facilitate this process. To evaluate the constructed model object, create a new Evaluator instance and call the compute_model_performance() method.

In [13]:
from deepchem.utils.evaluate import Evaluator
valid_csv_out = tempfile.NamedTemporaryFile()
valid_stats_out = tempfile.NamedTemporaryFile()

evaluator = Evaluator(model, valid_dataset)
df, r2score = evaluator.compute_model_performance(
    valid_csv_out, valid_stats_out)
print(r2score)
                                   task_name  r2_score  rms_error
0  measured log solubility in mols per litre  0.266864   1.736323

The performance of this basic random-forest model isn't very strong. To construct stronger models, let's attempt to optimize the hyperparameters (choices made in the model-specification) to achieve better performance. For random forests, we can tweak n_estimators which controls the number of trees in the forest, and max_features which controls the number of features to consider when performing a split. We now build a series of SklearnModels with different choices for n_estimators and max_features and evaluate performance on the validation set.

In [14]:
import itertools

n_estimators_list = [10, 100]
max_features_list = ["auto", "sqrt", "log2", None]
warm_start_list = [True, False]
hyperparameters = [n_estimators_list, max_features_list, warm_start_list]
best_validation_score = -np.inf
best_hyperparams = None
best_model, best_model_dir = None, None
for hyperparameter_tuple in itertools.product(*hyperparameters):
    n_estimators, max_features, warm_start = hyperparameter_tuple
    
    model_dir = tempfile.mkdtemp()
    model = SklearnModel(
        task_types, model_params,
        model_instance=RandomForestRegressor(n_estimators=n_estimators,
                                             max_features=max_features,
                                             warm_start=warm_start))
    model.fit(train_dataset)
    model.save(model_dir)
    
    evaluator = Evaluator(model, valid_dataset)
    df, r2score = evaluator.compute_model_performance(
        valid_csv_out, valid_stats_out)
    valid_r2_score = r2score.iloc[0]["r2_score"]
    print("n_estimators %d, max_features %s, warm_start %s => Validation set R^2 %f" %
          (n_estimators, str(max_features), str(warm_start), valid_r2_score))
    if valid_r2_score > best_validation_score:
        best_validation_score = valid_r2_score
        best_hyperparams = hyperparameter_tuple
        if best_model_dir is not None:
            shutil.rmtree(best_model_dir)
        best_model_dir = model_dir
        best_model = model
    else:
        shutil.rmtree(model_dir)

print("Best hyperparameters: %s" % str(best_hyperparams))
print("best_validation_score: %f" % best_validation_score)

# Save best hyperparams
best_rf_hyperparams = best_hyperparams
best_rf = best_model
n_estimators 10, max_features auto, warm_start True => Validation set R^2 0.319809
n_estimators 10, max_features auto, warm_start False => Validation set R^2 0.193690
n_estimators 10, max_features sqrt, warm_start True => Validation set R^2 0.361448
n_estimators 10, max_features sqrt, warm_start False => Validation set R^2 0.416106
n_estimators 10, max_features log2, warm_start True => Validation set R^2 0.399511
n_estimators 10, max_features log2, warm_start False => Validation set R^2 0.424990
n_estimators 10, max_features None, warm_start True => Validation set R^2 0.262399
n_estimators 10, max_features None, warm_start False => Validation set R^2 0.202104
n_estimators 100, max_features auto, warm_start True => Validation set R^2 0.215860
n_estimators 100, max_features auto, warm_start False => Validation set R^2 0.257380
n_estimators 100, max_features sqrt, warm_start True => Validation set R^2 0.420479
n_estimators 100, max_features sqrt, warm_start False => Validation set R^2 0.446257
n_estimators 100, max_features log2, warm_start True => Validation set R^2 0.460630
n_estimators 100, max_features log2, warm_start False => Validation set R^2 0.443250
n_estimators 100, max_features None, warm_start True => Validation set R^2 0.245533
n_estimators 100, max_features None, warm_start False => Validation set R^2 0.254912
Best hyperparameters: (100, 'log2', True)
best_validation_score: 0.460630

The best model achieves significantly higher $R^2$ on the validation set than the first model we constructed. Now, let's perform the same sort of hyperparameter search, but with a simple deep-network instead.

In [16]:
from deepchem.models.deep import SingleTaskDNN
import numpy.random
from operator import mul

model_params = {"activation": "relu",
                "momentum": .9,
                "batch_size": 64,
                "nb_epoch": 30,
                "dropout": .5,
                "data_shape": train_dataset.get_data_shape()}

lr_list = np.power(10., np.random.uniform(-3, -1, size=4))
decay_list = np.power(10., np.random.uniform(-6, -2, size=4))
nb_hidden_list = [1000]
nb_epoch_list = [30]
nesterov_list = [False]
dropout_list = [.5]
nb_layers_list = [1]
init_list = ["glorot_uniform"]
batchnorm_list = [False]
hyperparameters = [lr_list, decay_list, nb_hidden_list, nb_epoch_list,
                   nesterov_list, dropout_list, nb_layers_list,
                   init_list, batchnorm_list]
num_combinations = reduce(mul, [len(l) for l in hyperparameters])
best_validation_score = -np.inf
best_hyperparams = None
best_model, best_model_dir = None, None
for ind, hyperparameter_tuple in enumerate(itertools.product(*hyperparameters)):
    print("Testing %s" % str(hyperparameter_tuple))
    print("Combo %d/%d" % (ind, num_combinations))
    (lr, decay, nb_hidden, nb_epoch, nesterov, dropout,
     nb_layers, init, batchnorm) = hyperparameter_tuple
    model_params["nb_hidden"] = nb_hidden
    model_params["decay"] = decay
    model_params["learning_rate"] = lr
    model_params["nb_epoch"] = nb_epoch
    model_params["nesterov"] = nesterov
    model_params["dropout"] = dropout
    model_params["nb_layers"] = nb_layers
    model_params["init"] = init
    model_params["batchnorm"] = batchnorm
    model_dir = tempfile.mkdtemp()
    model = SingleTaskDNN(task_types, model_params)
    model.fit(train_dataset)
    model.save(model_dir)
    
    evaluator = Evaluator(model, valid_dataset)
    df, r2score = evaluator.compute_model_performance(
        valid_csv_out, valid_stats_out)
    valid_r2_score = r2score.iloc[0]["r2_score"]
    print("learning_rate %f, nb_hidden %d, nb_epoch %d, nesterov %s, dropout %f => Validation set R^2 %f" %
          (lr, nb_hidden, nb_epoch, str(nesterov), dropout, valid_r2_score))
    if valid_r2_score > best_validation_score:
        best_validation_score = valid_r2_score
        best_hyperparams = hyperparameter_tuple
        if best_model_dir is not None:
            shutil.rmtree(best_model_dir)
        best_model_dir = model_dir
        best_model = model
    else:
        shutil.rmtree(model_dir)
    print("Best hyperparameters so-far: %s" % str(best_hyperparams))
    print("best_validation_score so-far: %f" % best_validation_score)

print("Best hyperparameters: %s" % str(best_hyperparams))
print("best_validation_score: %f" % best_validation_score)
best_dnn = best_model
Testing (0.04243796362196816, 5.4776565214720946e-05, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 0/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.042438, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.234038
Best hyperparameters so-far: (0.04243796362196816, 5.4776565214720946e-05, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.234038
Testing (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 1/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.042438, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.513632
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.04243796362196816, 0.0052872612271821079, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 2/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.042438, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.443992
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.04243796362196816, 9.6051767992826779e-06, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 3/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.042438, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.445577
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0082851719285325434, 5.4776565214720946e-05, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 4/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008285, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.331861
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0082851719285325434, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 5/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008285, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.384694
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0082851719285325434, 0.0052872612271821079, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 6/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008285, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.375547
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0082851719285325434, 9.6051767992826779e-06, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 7/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008285, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.386137
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0081738425145885529, 5.4776565214720946e-05, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 8/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008174, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.303353
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0081738425145885529, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 9/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008174, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.372184
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0081738425145885529, 0.0052872612271821079, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 10/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008174, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.360787
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.0081738425145885529, 9.6051767992826779e-06, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 11/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.008174, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.339308
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.030510643776668792, 5.4776565214720946e-05, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 12/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.030511, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.485997
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.030510643776668792, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 13/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.030511, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.479761
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.030510643776668792, 0.0052872612271821079, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 14/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.030511, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.416248
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Testing (0.030510643776668792, 9.6051767992826779e-06, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
Combo 15/16
Starting epoch 1
Starting epoch 2
Starting epoch 3
Starting epoch 4
Starting epoch 5
Starting epoch 6
Starting epoch 7
Starting epoch 8
Starting epoch 9
Starting epoch 10
Starting epoch 11
Starting epoch 12
Starting epoch 13
Starting epoch 14
Starting epoch 15
Starting epoch 16
Starting epoch 17
Starting epoch 18
Starting epoch 19
Starting epoch 20
Starting epoch 21
Starting epoch 22
Starting epoch 23
Starting epoch 24
Starting epoch 25
Starting epoch 26
Starting epoch 27
Starting epoch 28
Starting epoch 29
Starting epoch 30
learning_rate 0.030511, nb_hidden 1000, nb_epoch 30, nesterov False, dropout 0.500000 => Validation set R^2 0.339829
Best hyperparameters so-far: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score so-far: 0.513632
Best hyperparameters: (0.04243796362196816, 0.00035128687494915216, 1000, 30, False, 0.5, 1, 'glorot_uniform', False)
best_validation_score: 0.513632

Now that we have a reasonable choice of hyperparameters, let's evaluate the performance of our best models on the test-set.

In [17]:
rf_test_csv_out = tempfile.NamedTemporaryFile()
rf_test_stats_out = tempfile.NamedTemporaryFile()
rf_test_evaluator = Evaluator(best_rf, test_dataset)
rf_test_df, rf_test_r2score = rf_test_evaluator.compute_model_performance(
    rf_test_csv_out, rf_test_stats_out)
rf_test_r2_score = rf_test_r2score.iloc[0]["r2_score"]
print("RF Test set R^2 %f" % (rf_test_r2_score))
RF Test set R^2 0.456205
In [18]:
dnn_test_csv_out = tempfile.NamedTemporaryFile()
dnn_test_stats_out = tempfile.NamedTemporaryFile()
dnn_test_evaluator = Evaluator(best_dnn, test_dataset)
dnn_test_df, dnn_test_r2score = dnn_test_evaluator.compute_model_performance(
    dnn_test_csv_out, dnn_test_stats_out)
dnn_test_r2_score = dnn_test_r2score.iloc[0]["r2_score"]
print("DNN Test set R^2 %f" % (dnn_test_r2_score))
DNN Test set R^2 0.513083

Now, let's plot the predicted $R^2$ scores versus the true $R^2$ scores for the constructed model.

In [19]:
task = "measured log solubility in mols per litre"
rf_predicted_test = np.array(rf_test_df[task + "_pred"])
rf_true_test = np.array(rf_test_df[task])
plt.scatter(rf_predicted_test, rf_true_test)
plt.xlabel('Predicted log-solubility in mols/liter')
plt.ylabel('True log-solubility in mols/liter')
plt.title(r'RF- predicted vs. true log-solubilities')
plt.xlim([-3, 3])
plt.ylim([-3, 3])
plt.plot([-3, 3], [-3, 3], marker=".", color='k')
plt.show()
In [20]:
task = "measured log solubility in mols per litre"
dnn_predicted_test = np.array(dnn_test_df[task + "_pred"])
dnn_true_test = np.array(dnn_test_df[task])
plt.scatter(dnn_predicted_test, dnn_true_test)
plt.xlabel('Predicted log-solubility in mols/liter')
plt.ylabel('True log-solubility in mols/liter')
plt.title(r'DNN predicted vs. true log-solubilities')
plt.xlim([-3, 3])
plt.ylim([-3, 3])
plt.plot([-3, 3], [-3, 3], marker=".", color='k')
plt.show()

[1] John S. Delaney. ESOL: Estimating aqueous solubility directly from molecular structure. Journal of Chemical Information and Computer Sciences, 44(3):1000–1005, 2004.

[2] Anderson, Eric, Gilman D. Veith, and David Weininger. SMILES, a line notation and computerized interpreter for chemical structures. US Environmental Protection Agency, Environmental Research Laboratory, 1987.

[3] Rogers, David, and Mathew Hahn. "Extended-connectivity fingerprints." Journal of chemical information and modeling 50.5 (2010): 742-754.

[4] Van Der Walt, Stefan, S. Chris Colbert, and Gael Varoquaux. "The NumPy array:a structure for efficient numerical computation." Computing in Science & Engineering 13.2 (2011): 22-30.

[5] Bemis, Guy W., and Mark A. Murcko. "The properties of known drugs. 1. Molecular frameworks." Journal of medicinal chemistry 39.15 (1996): 2887-2893.

[6] Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python." The Journal of Machine Learning Research 12 (2011): 2825-2830.