Two zero cost tips for training neural nets (that I wish I had adopted earlier): - Checkpoint models, and use the one with the lowest val error at test time - Reduce the learning rate when validation loss plateaus. Maybe these are obvious, but in case you're not using them...