Seeing the smoke
before the fire

Janith Wanniarachchi
janith.wanniarachchi@monash.edu Department of Econometrics and Business Statistics, Monash University

---

Supervised by Prof. Dianne Cook, Dr. Kate Saunders, Dr. Patricia Menendez, Dr. Thiyanga Talagala
IASC-ARS 2023, Macquarie University, Sydney

Let’s be honest here

Building a good model is hard

Explaining how a good model work is even harder

All right I’ll be real with you, i have a hot take that everyone knows but no one wanted to address. Let’s be honest here, building models are hard, we spend enough years in grad school struggling making those but theres a bigger problem here, explaining these models is even harder.

Exhibit A: The good model

What if you could
poke around and find out
how this model works?

Introducing

Explainable AI (XAI) methods!

I bring to the court, exhibit A, the black box model which is performing good at the moment and smiling proudly for performing well, (must be nice to be happy like that) but we don’t know how it is making it’s decisions. What if we could poke around and find out how this model works? Well there’s a new field that’s being growing in pushing the boundaries of explaining black box models called explainable AI or XAI

XAI has a lot of facets

XAI can help you look at

How the model reacts to different features overall using Global Interpretability Methods

How the model gives a prediction to one single instance using Local Interpretability Methods

Much like us humans, XAI comes in different varieties, and the two main ones are global and local interpretability methods, since I’ll be talking about bushfires, let’s assume that we have this amazing model that can predict the probability of a bushfire given a future date and location and it’s scarily good. Global methods will tell us how let’s say the temperature affects the chances of bushfire, while local methods will be able to tell us which weather conditions it used to give the prediction for a specific location and time.

Explaining one prediction

There are several key local interpretability methods that are related to each other in how they approach the problem

1. LIME (Local Interpretable Model agnostic Explanations)
2. Anchors
3. Counterfactuals

From these two methods generally understanding on a local scale is more informative and there are several methods that are connected together called lime, counterfactuals and anchors.

How are these related

For a given data point

• LIME will tell you what features were important in giving that decision
• Counterfactuals will tell you how the data should change to sway the decision given the model.
• Anchors will tell you what is the decision process that the model followed for that instance and other similar instances
  • We will be talking about this today!

Lime can answer the question “what features were important in giving that decision?”, it achieves this by fitting a more understandable model that is as similar to the model as possible while also being explainable

while counterfactuals can answer the question “how the data should change to sway the decision given the model”, counterfactuals will try to find the closest instance to the given instance that has the prediction that is required while also being plausible

finally anchors can answer the question, what is the decision process that the model followed for that instance and other similar instances, and how it achieves this will be the talk for the day

What is an Anchor ?

Formally

a rule or a set of predicates that satisfy the given instance and is a sufficient condition for \(f(x)\) (i.e. the model output) with high probability

---

Simply put

We are going to find a big enough boundary box in the feature space containing other points that would have the same model prediction as the anchoring point.

let’s talk anchors, Formally its a set of predicates that satisfy the given instance and is sufficient for the model boundary, or simply put, an anchor is going to give you the decision rules that the model follows for a specific instance, how I hear you ask enthusiastically? We are going to build a boundary box in the feature space using predicates that can capture the instance and similar points with high precision.

What is a predicate ?

Predicates are simple logical statements. In this context a predicate is made up of

A feature

Age

A logical operator

>

A constant value

42

Simply put

A predicate is a boundary line that divides a feature into two subsets.

What are predicates? Basically it’s a boundary line that divides the feature space into two subsets.

How are Anchors made?

• This is achieved by formulating the problem as a Multi Armed Bandit problem to purely explore the feature space

• Multi-arm bandit problems are like playing different slot machines. You want to design a strategy to get the most rewards by choosing the best machines, even when you don’t know how likely they are to pay out.

Deciding on a box to make in high dimensional space is harder than folding a fitted sheet, so the authors of anchors formulated the problem as a multi armed bandit problem. The idea is that you are in a casino and want to get overall higher rewards by pulling the arms of the best slot machines, finally my gambling addiction from Genshin impact paid off

The Multi-arm Bandit problem

Simply put

Imagine you are the local point and you are trying to make a wall of similar friends like yourself by changing the walls of the room.

Your options are to either change the north, east, west, or south walls to put all of your friends inside the walls.

When you have lots of space you get rewarded and when you find like minded friends you get rewarded as well.

Now our arms and rewards, imagine this, you are in a love is blind pod trying to find friends that vibe with you by pushing the walls of your pod so that you have a big enough wall and have a majority of friends in your wall vibing with you.

Visualizing the way Anchors are built

• Watch as our multi armed bandit solution makes a boundary box in a 2D space, seeking similar points.
• Observe the agent’s journey through precision and coverage, trying to uncover the optimal solution.
• Compare the agent on different instances

Let’s see this in action i want you to look closely at the next slide you will see an animation of box growing and trying to capture the similar points and watch how the precision and coverage changes.

Instance 1 (Purple)

Instance 2 (Yellow)

On the right is the boundary in the data space On the left the precision (x-axis) and coverage (y-axis) of the boundary is shown in bright red while the pink dots are the previously explored boundaries.

What did we just observe

• The direction of the boundary box tells us the direction of the decision boundary
  • Golden class predictions → Lower right quadrant
  • Purple class predictions → Upper left quadrant

So what did we see? The algorithm was able to understand the overall direction of red and blue points and in a practical setting we can use multiple points which the model is uncertain about to get an idea about the models decision boundary

Tip

In a practical setting we can redo this for multiple instances and get an understanding of the model’s decision boundary and get an approximate understanding of the high and low feature values that affect a model prediction.

Applying XAI to bushfire modeling

So now to the piece de resistance Bushfires and xai.

Why bushfires?

• The Australian bushfires, notably during the 2019/2020 season, was devastating to animals and humans alike.

• Objective : Build a model that can predict the cause of a possible bushfire and use explainable AI to uncover the decision process of the model

Some of you may remember the black summer of 2019/20 where an area larger than spain got burned down because of bushfires.

And if we can build models that are close to approximating the causes of bushfires then we can xai to understand the different weather and human conditions that black box models understand are the reasons for different bushfire causes

Bushfire ignition data

• Data Period: 2000-2021
• Data Type: Historic records of Victoria bushfires
• Observation Content:
  • Location
  • Date
  • Descriptive label of fire cause

• Label Refinement to 6 distinct causes:
  • Accidents
  • Arson
  • Burning off
  • Relight
  • Lightning
  • Other

So let’s go out and collect data to build a model, we’re focusing on the state of Victoria in Australia and our predictor variable is going to be the causes of bushfire ignitions.

Gathering predictors

• Weather data has been collected from SILO database through the weatherOz package.
  • Temperature
  • Rainfall
  • Humidity
  • Sunlight
• The distance to the nearest powerline for each bushfire ignition was also collected

Now let’s gather some predictors, we can get a weather data for each past bushfire ignition and then we can also get some human interventions such as the distance to the nearest powerline.

Fitting the model

The problem is,

Given the location, the date, the weather and human activity data, predict the most likely cause for a bushfire.

As a baseline model, A Random forest model was fitted on a training set from 2000 to 2020 while the testing set contained data from 2021 to 2022 with an F1 score on the testing dataset of 0.83.

Now let’s fit the model, given the location and the date and the weather and human conditions we want to predict the cause of a bushfire in a specific place, and we build a model that is fairly okay for the moment.

How can XAI help in this scenario?

• Local interpretability methods are more useful in this setting
• LIME can tell you, the importance of the features for a predicted bushfire cause
• Counterfactuals can tell you, the feature value configuration that will give a different prediction
• Anchors can tell you, the decision rule for the predicted cause.

How can xai help? Remember the three musketeers of local methods well we can use them to get different perspectives, lime will tell you “Which of the weather and human features were important in making this decision” counterfactuals will tell you How should the weather and human factors be changed for this bushfire to be caused by another cause and anchors will tell you this What are the weather and human value ranges that the model considered for this instance and other instances with the same predicted cause

Visualizing Explanations

The easiest to explain is Anchors , which gives explanations that are high dimensional.

Challenges

• Data can be noisy
• Model boundaries can be hard to see clearly

Possibilities

• Use tours to explore the data globally and locally
• Use interactivity to link high dimensional visualizations together.

can we look at anchors in a visual way similar to the way we saw in the animation? Well yes but no. There are some challenges like data being noisy and the bounding box being hard to see but there are some possibilities like grand tours and interactivity helping us bring the model and users closer to each other

Visualizing anchors for bushfires

It’s hard to see a pattern in this, as the data is noisy!

Let’s take the fitted bushfire model, for simplicity we have divided the bushfire causes to lightning vs not lightning and use it to generate a likely decision boundary for a randomly selected lightning prediction. We are going to focus only on the weather data minimum and maximum temperature, sunlight radiation and rainfall.

Here the golden dots refer to the instances with the prediction lightning and the purple dots are the instances that do not have lightning as a prediction.

So here I’m using the tours method of looking at high dimensional data, to put it simply its as if you have a funny looking high dimensional shape and you put a light infront and see how the shadow changes as you move the funny shape around.

It’s quite noisy! Let’s focus on something a bit more simpler and familiar, like the palmerpenguins data but with just Adelie and Chinstrap penguin species.

Visualizing anchors with penguins

The discovered anchor of 4 dimensions in a two dimensional space.
Purple dots refer to the Chinstrap species while the golden dots refer to the Adelie species.

Here the purple dots refer to the instances with the prediction Chinstrap and the golden dots are the instances that are predicted as Adelie.

It’s a bit more clearer on the rough direction that the anchors is looking at.

Let’s zoom in to the box and inspect it’s contents.

So here I’m using the tours method of looking at high dimensional data, to put it simply its as if you have a funny looking high dimensional shape and you put a light infront and see how the shadow changes as you move the funny shape around.

So for a single point we can see the box is close to around the point, which is one of the issues of having an overall view because it’s limiting

Local View

Can you notice a clear separation and a majority of points being gold?

Maybe if we look closely and see how the points are changing inside box and how pairs of plots change we can see …

Contributions and Future work

• This talk showed how to simplify and explain how anchors works to bring XAI tools closer to data scientists
• This talk showed how to apply high dimensional visualization techniques for anchors and see the possible insights and limitations
• I’ll be working on exploring high dimensional visualizations of counterfactuals for model explanation
• I’ll be applying this work to bushfire modeling
• These are a few proposed ideas to visualize explanations in high dimensions but it needs a lot more work to make the visualization help us understand the explanations. So stay tuned!

Thank you!

Have any suggestions or ideas?

Janith Wanniarachchi

janith.wanniarachchi@monash.edu

@janithwanni

janith-wanniarachchi

janithwanni.netlify.app