It was the Maths! It was the People!

Image by Gerd Altmann from Pixabay

P: This A-level grade-prediction thing, am I right?

S: Yeah, it was nasty. Good thing they made that U-turn!

P: I know, right? Poor pupils. It was so unfair how the algorithm just lowered the grades of poorer students...

S: Well, not quite; it just didn't increase their grades as much as those of students from historically better schools.

P: What do you mean?

S: Well, the statistical model just built on grades from past years – so it kind of just standardised students' grades, you know?

P: Right, the maths was totally wrong!

S: Now wait, the maths was fine. However you put it, the algorithm did just as expected...

P: How is that possible? The algorithm just happened to disproportionately increase richer students' grades?

S: Well, the algorithm didn't do anything – it was the people behind the algorithm who made it do what they wanted: to standardise grades.

P: Right, but weren't those people data scientists?

S: Well, I mean, statisticians, sure–

P: Right, data scientists! It was the data and the statistics what were wrong!

S: Now hold on. The statistical model did as intended. What was wrong was what the people behind it intended.

P: Oh, so the whole racist facial recognition, unfair predictive policing and unjust justice algorithms have nothing to do with the actual data science either?

S: Those may very well be the results from applying data science tools, but they are also built by humans.

P: So it's all about the humans and not at all about the science?

S: Right.

P: ...

S: Fine. Science isn't exactly a holy grail. The thing is that human values seep into the scientific process. The values held by data scientists can build into their assumptions.

P: For example, the assumption that a school's past grades are relevant when predicting its pupils' future grades?

S: Right, which is a rather bizarre assumption in the first place. But others might find their way into the process. For example: more data mean more accurate predictions. Hence the decision to not use the algorithm for small classes.

P: Is that an assumption or just common practice? Something to do with some law of large numbers?

S: Right! Greater samples will more accurately result in a population average.

P: But that doesn't sound right – a student this year does not perform on the basis of some past "population average"...

S: That makes sense. It is strange to assign a grade to an individual given some population-level phenomenon...

P: But I'm still confused – why did they think it was fair to use an algorithm only for larger classes? Or, rather, why did they go ahead with the whole thing?

S: I think that's a much wider problem. Data science qua data science is deemed a holy grail of sorts. It's such a cool new set of tools and its applications can be so varied, that it just sounds like the right way to go. After all, it's the way we can work with the greatest amount of data.

P: ... I think this is why I very urgently want to blame the maths itself. Whilst I don't get it and will never be a data scientist, I see how data science is poorly applied to real-life issues.

S: And I grant that that is a sound analysis insofar that data science has made mistakes in the past and yet we still turn to it when we have difficult questions (like "what grade would UK students have got this year had they sat exams?").

P: There seems to be an emphasis on its predictive power.

S: Totally, and that's why it is seen as the holy grail.

P: But it isn't.

S: It isn't...

P: And yet, it is?

S: I need to say it is to keep my job, so yes!

P: But seriously now, do you think data science does have great potential?

S: Oh, absolutely! You can find great examples of how data science helps doctors offer more accurate diagnoses, for instance.

P: That sounds scary...

S: I guess there's a layer of "security," if you will, in this case. Ultimately, whatever an AI diagnoses, it is the doctor who makes the final call and actually communicates the good, not-so-good or more-nuanced news.

P: The human touch is still needed.

S: Exactly! And in other cases, policy-makers can use AI not as a decision-making tool, but as an extra source of information.

P: But it made the final decision for students' grades?

S: Correct.

P: So what is at fault is data science that is created by people with “values” and that data science being applied just like that, with no “human touch.”

S: That sounds about right. But values cannot be taken out of the process. If anything, we need more people to provide checks on each other’s values.

P: We need more diversity?

S: Right, more specialisations, more collaboration. An algorithm can’t be developed by statisticians who don’t understand the education system they’re affecting. They need other researchers to help them make decisions.

P: Like sociologists or economists?

S: That could help, yes.

P: But you wouldn’t reduce the values that “seep into” the science project.

S: And you can’t. You need to accept this and make sure that the contributions to the project are as varied as possible so the science includes as many ethical considerations as possible.

P: Huh...

S: Uhuh.

P: And hope for the best?

S: But only after doing the best you could and lots of tests.

P: Hm... Induction problem?

S: Induction problem.