Deep Neural Networks

Tabular data with fastai

Gradient boosting solutions still clearly dominate tabular competitions (as well as real-world projects); however, sometimes neural networks can catch signals that gradient boosting models cannot get, and can be excellent single models or models that shine in an ensemble.

<aside> 💡 Ensemble Models As many Grandmasters of the present and the past often quote, mixing together diverse models (such as a neural network and a gradient boosting model) always produces better results than single models taken separately in a tabular data problem. Owen Zhang, previously number one on Kaggle, discusses at length in the following interview how neural networks and GBMs can be blended nicely for better results in a competition → video

</aside>

Detailed explanations how to build Deep Neural Networks with TensorFlow, see The Kaggle Book p.379 of 826.

The key things to take into account when building these solutions are:

• Use activations such as GeLU, SeLU, or Mish instead of ReLU; they are quoted in quite a few papers as being more suitable for modeling tabular data and our own experience confirms that they tend to perform better. • Experiment with batch size. Activation Functions
• Use augmentation with mixup (discussed in the section on autoencoders).
• Use quantile transformation on numeric features and force, as a result, uniform or Gaussian distributions.
• Leverage embedding layers, but also remember that embeddings do not model everything. In fact, they miss interactions between the embedded feature and all the others (so you have to force these interactions into the network with direct feature engineering).

If you don't want to build your own deep neural network in TensorFlow or Pytorch, you can rely on a few out-of-the-box architectural solutions. All these solutions come out of the box because they are packaged or because other Kagglers have written them based on the original papers. Based on their success in tabular competitions, here are the main ones you can try when taking on a tabular competition yourself (more details in The Kaggle Book p. 382 of 826):

• TabNet
• Neural Oblivious Decision Ensembles (NODE)

Also, as a piece of advice based on our experience, don't expect a neural network to be the best model in a tabular competition; this seldom happens. Instead, blend solutions from classical tabular data models, such as gradient boosting models and neural networks, because they tend to pick up different signals from the data that you can integrate together in Ensemble Models.