What is AI bias, and can it hurt your business?
Written lesson · about 3 minutes to read
The short answer
AI bias is when a tool repeats unfair patterns from the examples it learned from, so some people get worse results, in hiring, pricing or how well it understands their accent or name. It can hurt your business because the law against discrimination applies to a decision whether or not a tool helped make it.
Three things to remember
- 01AI learns from examples, including unfair ones, and can repeat those patterns without anyone noticing.
- 02The law against discrimination applies to a decision whether or not a tool helped make it.
- 03Keep a person on decisions about people, test results across groups, and write down how decisions are made.
What bias means in a tool
AI learns patterns from very large numbers of examples. If those examples contain unfair patterns, such as past hiring that favoured some kinds of people, the tool can learn them and repeat them. Nobody has to tell it to be unfair. It copies what it was shown, and it tends to handle less well the people its examples covered least.
Bias also shows up as uneven accuracy. A voice tool may mishear some accents more than others. A form reader may struggle with unfamiliar names. The result is that some customers get slower, worse service than others.
Where it can hurt a small business
The main risks are where a tool touches a decision about a person: sorting job applications, offering credit or payment terms, setting a price, or answering the phone. A screening tool that ranks candidates by how closely they resemble your past hires can quietly repeat whatever pattern those hires had.
The law against discrimination applies to a decision whether or not a tool helped make it. Saying the tool recommended it does not move the responsibility. Some places also have specific rules for automated hiring tools, and rules differ by state and country, so check with a lawyer for your own case.
What to do about it
Keep a person on decisions about people, including hiring, discipline, credit and refusals. The tool can prepare a shortlist, but a person decides, and looks at the applications it passed over as well as the ones it picked.
Test the outputs. Run the same sample cases through the tool with only the names or details changed, and see whether the results differ. Ask each vendor how they test for bias and whether they will show you the results. Write down how each decision is made: the criteria, who decided and why.
Where to start this week
List every place where a tool touches a decision about an applicant, a customer or a member of staff. Beside each one, write the name of the person who decides.
Set one rule: no tool rejects, refuses or ranks a person on its own. Then ask each vendor one question: how do you test for bias, and can I see the results? Note the date and outcome of each test, and repeat it whenever the tool changes. A vendor with a real answer will have a document to show you.
More in part 9
- 1How do you check an AI answer before you use it?Written
- 2Where should humans stay in the loop?
- 3What are guardrails?
- 4How do you keep AI on-brand in front of customers?Written
- 5What is AI bias, and can it hurt your business?WrittenYou are here
- 6What happens when an agent gets it wrong?
- 7What is an audit trail, and why will you need one?
- 8What happens when the model updates and your results change?
- 9What is a reward model?
- 10What are AI evals?
- 11What is a golden set?
- 12What is LLM-as-a-judge?
- 13What is calibration (and why confidence scores matter)?
- 14What is shadow testing?
- 15What is red teaming?
- 16What is AI observability?
Next step
Where does this fit your business?
Six quick questions about your business, then your AI plan: what to start with, what a person still checks, and the lessons that go with it. Or bring it straight to a free thirty-minute call.