Checkout LOFO!

With the Permutation Importance feature selection method, the performance of a model is tested after removing each individual feature and replacing that feature with random noise. In this way the importance of individual features can be directly compared, and a quantitative threshold can be used to determine feature inclusion.

PI doesn't require retraining. PI is still a good option, but LOFO is more accurate if you can afford the training time. PI gives your model a different set of inputs than it was trained with, which is a bit more unrealistic than LOFO, and IMO provides a worse estimate.

Umsetzung in Sebastian Raschka Blog.