Training a neural network is a lot like traying to decide between having pancakes and waffles for breakfast...there's no right answer. Many different frameworks and training methods will give you adequate results. However, we might choose certain approaches because they are simpler, more robust, easier to understand, or give us nice uncertainty estimates. While exploring these different methodologies we often run into problems. I've hit a variety of different problems while trying to train neural nets ranging from simple numerical errors to much more subtle issues with custom loss functions. I've taken some of my favorite neural net troubleshooting guides along with my own advice and synthesized it below to help anyone who finds themselves in the same boat.