There is an increasing awareness among practitioners that data drift poses a challenge to the robust deployment of machine learning models. But what precisely is meant by “drift” and how can we protect ourselves against it? 👇📽️ 🧵 https://t.co/Bru4FaCcGe

Check out Alex's (@oblibob) blog post on generative modelling using vector-quantized VAEs!

We have a paper accepted into the R2HCAI workshop titled, Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning. 🙌 Very excited to be a part of the conversation around the advances of responsible AI. 💪 Learn more: https://t.co/GiG9pjA6Hx #AAAI

For more details, check out our example benchmarking drift detectors with the KeOps backend here:

This drastically speeds and scales up the detectors to large dataset sizes, with dataset sizes in the order of 100,000’s easily achievable on a single consumer grade GPU.

Alibi Detect v0.11.0 introduces a new backend for the MMD and learned kernel MMD detectors. Internally, these detectors use the KeOps library, developed by @FeydyJean and @JoanGlaunes. This allows the kernel matrices to represented by symbolic tensors.

The sensitivity of a drift detector scales with dataset size. However, the memory and computational costs of a number of convenient and powerful kernel-based drift detectors, such as the MMD detector, do not scale favourably with increasing dataset size.

We are excited to announce the release of Alibi Detect v0.11.0, featuring widened serialisation support and a new backend that allows drift detection to be rapidly performed on large datasets.

Much more information on Permutation Importance and Partial Dependence Variance, including worked examples, can be found on our documentation pages: https://t.co/ZYgYz76vum https://t.co/d1NUeK8yQ6

Both of these insights are complementary as PI captures not only main feature effects but also interactions, and we recommend considering both, when possible, for a thorough analysis of model behaviour.

When to use PI vs PDV? The key lies in the interpretation of the importance values. Whilst PDV quantifies how much of the model's output variance is explained by each feature, PI measures how much model performance degrades when a feature is noised.

Furthermore, PDV can be extended to also quantify pairwise feature interaction strengths, allowing a deeper understanding which features interact with each other inside the model.

Partial Dependence Variance (PDV) derives from Partial Dependence (PD) plots. Intuitively, calculating PD for a feature, the resulting points on the plot will collectively have higher variance if the feature is more discriminative wrt the model. PDV formalizes this calculation.

The metric/loss can be customized. The plot below shows the feature importance wrt to accuracy and F1 metrics of a random forest predicting whether employees are likely to leave a company. The feature "satisfaction_level" is the most important one regardless of the metric.

Permutation Importance (PI) works by selecting a feature of interest, shuffling the values of that feature across the dataset and then measuring the effect on some metric or loss function on this new dataset with respect to the original.

Both Permutation Importance (PI) and Partial Dependence Variance (PDV) assign a scalar value to each feature to quantify their importance with respect to the model. Both methods are model-agnostic, but PI requires ground-truth labels, so will be more useful during development.

We are pleased to announce the release of Alibi Explain v0.9.0 with support for calculating global feature importance via Permutation Importance or Partial Dependence Variance.

For a more extensive discussion of the method, its usage and examples please visit our documentation page: https://t.co/ntscZBUGC5.

Our PD implementation in Alibi v0.8.0 has the following advantages over other implementations: - Applies to any black-box model - Full support for 1-way, 2-way and higher order PD for numerical and categorical variables - Flexible plotting functionality for 1-way and 2-way PD

There is an improvement upon PD plots called Accumulated Local Effects (ALE) which take feature correlations into account. This is implemented in Alibi, but only applies to numerical features:

Note that underlying PD computation is the assumption of feature independence (i.e. features are not correlated) which usually does not hold in practice and has to be taken into account when interpreting PD plots.