How Big Data and machine learning hinder social mobility.

Poverty Safari — Darren McGarvey

Weapons of Math Destruction — Kathy O’Neill

The central theme of Darren McGarvey’s book is stress. The daily experience of walking to and from school on the Pollok estate in Glasgow; always being on your guard, on the constant look-out for threats. School was an aggressive environment with massive peer pressure to conform. At home, things were worse: his mother’s alcoholism and addiction were often the talk of the estate; her violence and emotional abuse left long-lasting scars. According to him, these stresses — often induced by poverty — affect a person’s whole formative period starting with foetal development. There is research showing they affect the interpretation and calculation of risk, induce people to prioritise short-term goals over longer-term planning. Already in his early teens McGarvey was struggling with depression and anxiety, in particular social anxiety. That directly led to substance abuse — he describes feeling normal and being able to interact normally only when under the influence of drink or drugs.

In the late 90’s and early 2000’s, under New Labour, McGarvey received a lot of help owing to his difficult childhood. He speaks highly of his psychologist, of the people who tried hard to set him straight, but at the same time says he was not pushed hard enough to battle his problems by himself. He was told he was a victim and used that as an excuse to continue the substance abuse. Both the Left and the Right, according to him, fail to understand the mental health issues around poverty and around growing up in a stressful environment. The Right leave people to sink or swim, the Left tell them nothing is their fault: neither do much for social mobility. Being routinely ignored by both sides, with people more keen to pursue their own agenda than listen[1], is a recurring theme.

Let us now suppose a Darren McGarvey entered the world of adulthood now and tried to build a ‘normal’ life. The main theme of ‘Weapons of Math Destruction’ is the extra stress that would be inflicted on him by decision-making computer algorithms. Developed over the last few decades, those automate many of the decision processes previously performed by humans — handling job applications, credit applications and so on. Typically, the more downmarket the organisation, the more badly-paid the job, the poorer the applicant, the more likely a ‘black box’ model is to be used. O’Neill’s examples come from the US, but the same models are being adopted here.

Originally, they were thought to benefit people from disadvantaged backgrounds and racial minorities. By removing the prejudices of the human being making the decision, it was felt, everyone could be judged on merit. This is all true as long as the model is conscientiously designed: all too often it is not, and consequences can be dire. For example, the same human prejudices can be encoded into the model by the programmers. Also, computers have no notion of fairness or common sense and don’t give you a chance to establish interpersonal contact: McGarvey mentions that one of his strengths is being able to intuit what the person he is talking to feels in order to build an understanding, and this would be taken out of the equation. What started out as a way to open doors to people like McGarvey turned out to do the opposite. Here are some of those new problems in more detail: they are problematic enough for the average person, but for someone from a stressed, disadvantaged environment they could well prove insurmountable:

(1) Lack of feedback. Hannah Arendt once said: ‘in a fully developed bureaucracy there is nobody left with whom one can argue, to whom one can represent grievances, on whom the pressures of power can be exerted. … the rule of Nobody … is what the political form known as bureaucracy truly is”. The black box models are the perfect form of “the rule of nobody”: all too often it is not possible to dispute their decision, ask for explanations and or have them tell you where you should improve to get a better result next time. Their inner workings are proprietary, a closely guarded secret, and their output is all too often treated as the absolute truth. There is no interaction, no person who made a decision and can explain it, no appeals process: people shrug their shoulders and say ‘nothing we can do, the computer decided’. What is more, the model may well go uncorrected if shown to be wrong. O’Neill cites the case of Sarah Wysocki, a Washington teacher highly thought of by colleagues and parents but fired on the word of a black box evaluation model. Even if she turns out to be a stellar teacher in her next job, no one will go back to the model and fired her and change the parameters: the model, like its victims, gets no feedback at all.

(2) ‘Soft’ racism. Racial and socio-economic profiling can be hidden inside black box models by using data — typically the postcode — as as a proxy. Here is how this works: the model trains by studying many (race, class, address) triples over some database; when the trained model is given a new address it uses this training to predict their race and class internally, and use that output to categorise[2]. The essence of racism is treating an individual not as an individual but as a representative of a particular ethnic group having made certain assumptions about the whole group: the decision model is free to do essentially the same: to attempt to guess your race and then treat you accordingly. Discrimination on racial and class grounds becomes discrimination for living among and having friends from people of a certain background. The algorithm predicts your behaviour based on that of ‘people like you’. With a human decision maker we can challenge their decision if we feel it to be racially biased. It is not possible to do the same of a black box model whose inner workings are proprietary and that offers inadequate explanations for its decisions.

(3) Employment law bypass. ‘Hidden personality tests’ are currently all the rage when it comes to job applications[3]. They are used on 60–70% of prospective workers. The computer gives no feedback: the applicant cannot call it up and ask where the application fell short, and indeed the ‘right’ criteria are deliberately hidden to stop people gaming the tests. By law, the employer is not allowed to ask about past mental health problems or people’s family situation but the algorithms resolve this using proxies: by finding volunteers and getting them to do the test one can match patterns of answers to their medical records. Of course, the correlation between those patterns and past mental illness/undesirable family situation might be highly inexact and not be very good at picking out the latter. The same analysis can be done using any information found about the applicant on the Internet. By crunching through vast amounts of data, algorithms find correlations between the information they are not allowed, by law, to use and the information out in the public domain. For instance, they could decide that people who pick option (A) for questions 1–5 and who post cat videos on their facebook page are prone to depression. Those correlations, too, are often wrong. In the end, it is highly likely that people will still end up being judged by their postcode — and possibly by the postcode of their friends. If you grew up on a sink estate, that sets you back even further.

(4) Targeted exploitation. Many nefarious organisations hunt for poor people’s money and the government cash/loans they might be entitled to. O’Neill gives the example of ‘for-profit’ US universities: they harvest data online to find vulnerable people who dream of social mobility. They are then bombarded with calls and talked into spending money and taking out governmental loans for a degree that most employers don’t think is worth anything: hardly a surprise when those organisations spend two and half times as much on marketing as on teaching. This is in the US, but ads for gambling and payday loans likewise attempt to target those who might be more easily swindled. Algorithms make the potential targets easier to locate.

(5) Model driven reality. The existence of a flawed black box algorithm may alter the very process it is attempting to model. Take for example a policing model that tracks the geographic occurrence of crime and attempts to predict where future crimes will occur. Violent crime may not be a dense enough dataset (machine learning methods are often quite data hungry), so it could also track petty crime and antisocial behaviour as well and count that as ‘crime’. This is done even though the geographical correlation between public order offenses and violent crime is questionable. The model suggests allocating policing resources to the areas where their past crime data is the most dense. As a result, they catch even more people and get even more data for those areas, which reinforces the biases in the model. An individual in a deprived area finds themselves at a much higher risk of a stop and search or of falling foul of a minor public order offense. The system is therefore rigged against them in yet another way.

(6) Automatic scheduling. With casualisation an increasingly common feature of low-paid employment, organisations have started using operations research packages to optimise their employees’ shifts in response to demand. In extreme cases, the same employee would have to shut up shop at night and open it in the morning. With jobs at such a premium, workers have no control over the shifts they are assigned, and because demand predictions might change (they are a function of the weather for example) they may have very little notice over shift changes. Understandably that wreaks havoc with people’s lives: not only are second jobs out of the question, but sleeping patterns are disrupted and children’s routines destroyed.

(7) Credit record as a proxy. It is hardly surprising that people from deprived backgrounds have poor credit records. Apart from being poor, many lack skills at managing money and are hunted by loan sharks. It is understandable that their credit record counts against them when applying for credit. It is far less understandable that it also gives them a disadvantage when applying for jobs and insurance. In 10% of cases in the US, the job applicant’s credit score was explicitly cited as a reason for rejection. In the most egregious example wealthy drivers with past drink driving convictions were shown to have been offered far lower premium than drivers from deprived backgrounds with a clean record. The motive was likely profit and not safety: the black box model guessed the poorer driver was less likely to shop around for better deals, was not so savvy. This is just one example of how it is expensive to be poor. And besides, the credit records themselves are often fraught with errors — an estimate suggests 5% of credit records have errors that boost costs of borrowing. But all too often, and especially when it comes to poor people, there is no one to go and fix those errors. People keep getting rejected because their credit record gets associated with criminals and fraudsters that have the same name — but there is often no feedback that says so.

(8) Keeping down the socially mobile. Stopping those organisations accessing credit records, however, once again leaves them to resort to proxies — called e-scores. Models will try to estimate credit records from contextual data — from shopping patterns to the addresses of friends on Facebook. Instead of your own credit record being misused, it will once again be estimated from those of ‘people like you’ and then misused. The word of the model becomes law, the definitive pronouncement on a person’s trustworthiness — and it is rarely subject to appeal. It will be wrong of course, but the organisations using those models do not care — they want to maximise profits and for that, they only have to be right in the average case.

This is a problem with social mobility that clearly needs more attention. People living in stressed communities and with a past history of problems find themselves having to navigate a world in which the power of supercomputers is used to filter out people with the ‘wrong’ addresses, the ‘wrong’ associations and the ‘wrong’ backgrounds. The drawbacks of machine learning algorithms — their capacity to generalise and to not be explainable — are almost perversely serving to achieve that goal.

Footnotes:

[1] It is the old story — the front-line staff were, according to him, excellent, but the layer of management immediately above was terrible and was more interested in ticking boxes than helping. In a very familiar adage, McGarvey talks of grassroots groups constantly having to justify their existence to faceless bureaucrats.

[2] Actually, there is a lot more flexibility here: you can also predict the probabilities that an individual belongs to each race.

[3] And ‘team interviews’, group exercises drawn up by HR consultancies must be equally daunting for candidates from disadvantaged backgrounds.