You Are Not a Persona

The thing your bank, your insurer, your government, and your favourite app are doing with your data that they would rather you didn't think about. And what to do about it.

Let me tell you what is probably happening to you right now, even as you read this, without you knowing.

When you applied for that loan last month, or made that insurance claim, or used the chat function on your bank's app, or clicked through the help menu on a government service, or messaged a customer service line, or opened a new account online, somewhere in the middle of those interactions an artificial intelligence system was involved in deciding what would happen next.

It might have been deciding whether you were a risk. Whether you got a faster line. Whether you got a human at all. Whether your claim went into the "probably fine" queue or the "look at this one closely" queue. Whether the words you typed sounded like a frustrated customer or a fraudster. Whether the photo you uploaded looked real. Whether you were the kind of person who pays late.

None of this was illegal. Most of it was disclosed somewhere in a 47-page document you didn't read because nobody reads them. Some of it might even have been good for you. Faster decisions. Fewer mistakes. Less time on hold.

That is not what I am writing to tell you.

I am writing to tell you the bit that is missing.

·  ·  ·

I am a risk professional. I have spent over two decades working inside the systems that decide how big organisations handle the data of people like you, like us. I read the regulations. I write the reports. I've both sold these tools and sat in the rooms where decisions get made about which AI tools get rolled out, how, and to whom.

I am writing this in plain English, on purpose, because the rooms I sit in tend to use language that keeps people like you, like us out of the conversation. That is not by accident. Complicated language protects the people inside the room from having to answer simple questions.

Here is a simple question.

Who in your bank, your insurer, your telco, your hospital, your government department, has the explicit job of standing up and saying "wait, what if this AI is wrong for the actual people we serve?"

The answer, at most organisations, is: nobody.

That is the thing I am writing to tell you.

·  ·  ·

In the last six weeks, three of the biggest rule-setters in this space have released new guidance: APRA (the prudential regulator for your financial institutions such as your superannuation fund), ASD (the cyber agency), and NIST (the American standards body, broadly used in Australia and globally). All three documents are serious. All three are competent. All three are also written for the inside of organisations. They tell businesses how to protect themselves.

The bit none of them really cover is what is happening on your side of the screen.

When the AI in the bank's loan model decides that your application looks like the kind that 89% of declined loans come from, that is a decision about you. You probably don't see how it was made. You probably can't ask the model why. You probably won't be told which data points tipped the verdict. If you call the bank, the person you speak to almost certainly doesn't know either, because they don't see the model's reasoning either. They see the output.

That is the gap. That is what I am writing about.

·  ·  ·

You hold the power of the purse.

Over the past two decades or more we've lost sight of who holds the power. For a very long time we've used terms such as "we live in a consumer society". But the power has been slowly and silently stripped away from the consumer and now rests with those holding our personal data, and perhaps more importantly our digital identity. Leaving us, the consumer, with less power but all the risk. Consumers need to take back the control we once had by informing ourselves before making purchasing decisions.

Four questions you can ask today of any organisation handling your data.

Did they tell you that an AI was involved?

Not buried in a privacy policy. In the moment. "By the way, the next step in this is automated. Here is what it is doing." If they didn't tell you, that is information. It tells you something about whether they were comfortable having you know.

Can a human review the decision?

Not "if you escalate," not "after three forms." Easily, quickly, on request. The right to talk to a person about a decision a computer made about you is not a luxury. It is the floor.

Do they know what data the AI is using about you?

This sounds obvious. It isn't. Many AI systems are built on data the organisation buys from somewhere else, that they may not fully understand, that may include information about you that you did not knowingly give them. If they cannot tell you, that is also information.

Does someone in their building have the actual authority to stop an AI from being used on you if it is wrong?

This is the big one. In aviation, any crew member can halt a flight if they see a safety problem. The most junior cabin attendant has the right to say "stop, listen to me." That is not because the captain says so on the day. It is built into how aviation works. Almost no organisation handling your data right now has anything like this for AI decisions. The ones that build it voluntarily, before they are made to, are the ones to trust.

·  ·  ·

The reason none of this is being talked about in the news much is that nobody is making a movie about it. The story of an AI quietly mis-categorising your loan, or your job application, or your insurance claim, is not a dramatic story. It is a slow, boring, individual story. Twenty thousand people quietly getting a slightly worse outcome each is harder to put on the front page than one person getting a dramatic one.

But twenty thousand quiet stories are what is happening. You are one of them.

·  ·  ·

What I would ask you to do.

When you next interact with a bank, an insurer, a government service, a telco, or any platform that handles your personal information, ask one question. Just one.

"Was any part of this decision made by an artificial intelligence?"

You might not get a clear answer. That, too, is information.

If you do get an answer, ask one more.

"Who in your organisation has the authority to stop an AI from being used if it is wrong?"

If the person you are talking to has to escalate the question, that is fine. The point of asking is not to get the answer today. The point is to put the question into the building. Enough customers asking enough buildings that question, often enough, will change what those buildings do. That is how every consumer-protection improvement in the last fifty years has happened. From food labelling to mortgage disclosure, to car safety ratings. People asked. Buildings listened.

You have more power in this than you have been told.

·  ·  ·

If you found this useful, share it with a friend. Tell them what is happening behind the apps they use. Tell them what to ask. Tell them they are not a data field. Tell them they are not a persona.

I will keep writing pieces like this, translating what happens in the back rooms into language that is useful to the people the back rooms are actually about.

The next one will be about the apps on your phone, and what they are doing with the information you have given them without quite knowing you have.