Checklist EDA

EDA Checklist

Kaggle Book EDA

• Missing values and, most importantly, missing value patterns correlated with the target.
• Skewed numeric variables and their possible transformations.
• Rare categories in categorical variables that can be grouped together.
• Potential Outlier, both univariate and multivariate.
• Highly correlated (or even duplicated) features. For categorical variables, focus on categories that overlap.
• The most predictive features for the problem.

You achieve this by several descriptive analyses, graphs, and charts, first examining each distinct feature (univariate analysis, in statistical terms), then matching a couple of variables (bivariate analysis, such as in a scatterplot), and finally considering more features together at once (a multivariate approach).

Automated strategies can help you (according to Kaggle Book):

AutoViz

SweetViz

Pandas Profiling

However, remember that EDA stops being a commodity and becomes an asset for the competition when it is highly specific to the problem at hand; this is something that you will never find from automated solutions and seldom in public Notebooks. You have to do your EDA by yourself and gather key, winning insights.

All things considered, our suggestion is to look into the automated tools a bit because they are really easy to learn and run. You will save a lot of time that you can instead spend looking at charts and reasoning about possible insights, and that will certainly help your competition performance. However, after doing that, you need to pick up Matplotlib and Seaborn and try something by yourself on not-so-standard plots that depend on the type of data provided and the problem.