What does facial-expression AI actually look at? How it works, its limits, and using it responsibly

Four different technologies hiding under "face recognition"

People often lump everything together as "face recognition", but several technologies with different jobs are involved. Knowing which one is at work makes it clearer how to take the results.

• Face detection: finding where faces are in an image, as rectangles. It doesn't know whose face it is.
• Landmark detection: marking points inside a detected face, such as eye corners, the tip of the nose, mouth corners and the jawline.
• Expression classification: estimating which of a preset list of expression categories (happy, surprised and so on) a face looks closest to.
• Identity recognition: working out who a face belongs to by comparing it with other face photos. Used for access control and phone unlocking.

ToolyStation's Live Emotion Detector, Photo Emotion Analyzer, AI First Impression and Animal Face Test use only the first three. None of them tries to work out who someone is or compares faces against other people's.

Step 1: finding the face

A face detection model is a program trained on large numbers of photos where faces have been marked. Given a new image, it finds rectangles likely to contain a face and gives each a confidence score. Rectangles scoring below a set threshold aren't accepted as faces.

If this step fails, nothing after it runs at all. So when expression results look strange, the first thing to suspect is whether the face was found properly. Faces that are too small, turned away or partly covered are the easiest to miss here.

Step 2: marking reference points on the face

A landmark model places a fixed number of points along the eyes, eyebrows, nose, mouth and jawline of a detected face. The model ToolyStation's Animal Face Test uses places 68 points per face.

With those points you can measure things like how many degrees the outer eye corners are raised, how far the mouth corners sit above the center of the mouth, or how pointed the jaw is, much as you would with a ruler. An animal-face result isn't the AI deciding "this person looks like a cat"; it's a few such ratios compared against reference values that people set for each animal.

Step 3: sorting expressions into categories

An expression model learns the relationship between face shapes and labels from training material in which each face photo carries a label like "happy" or "surprised". Given a new face, it computes a probability for each preset category (on this site, seven: happy, sad, angry, surprised, fearful, disgusted and neutral) and divides them so they total 100%.

The key point is that these seven are the only answers available. Show it a face that's bored, focused, embarrassed or tired, states outside the list, and it still has to spread its probabilities across the seven. That's how a tired face ends up reading as sad, or a concentrating face as angry.

An expression is not an emotion

Facial-expression AI sees the shape that facial muscles make. It can't see what someone actually feels. A polite smile, a face tense with nerves, and a face whose mouth naturally turns down may read as happy, angry and sad respectively, but that doesn't mean the person feels that way.

When people read emotions from faces, they also use the situation, tone of voice and what they know about the person. The AI sees only the shape of the face in one frame. The accurate way to read its output is as an estimate of which training photos this face most resembles.

Common reasons results go wrong

• Light and shadow: light from one side changes how the mouth and eyes look on the shadowed side.
• Angle: turning or tilting the head changes the proportions of eyes and mouth, so the same expression can be read as a different category.
• Covering: masks, hands, hair and thick glasses frames reduce what the model can see.
• Small faces: in group photos, small faces may be missed at detection, or found but too blurry in detail to classify well.
• Blur: in dim light the camera gathers light for longer, so even small movements blur the face. In blurry frames the shape of the mouth and eyes smears and results jump around.
• Subtle expressions: faint everyday expressions are harder than exaggerated ones. Try it and you'll see how differently a slight smile and a big grin are scored.

How to read the probabilities

Looking only at the top item misses a lot. What matters alongside it is the gap between first and second place. With something like "happy 85%, neutral 8%", the model sees the face fairly clearly as one category. With "neutral 40%, sad 35%", the model is torn, and a small change in light or angle could swap the top spot.

The same applies when comparing photos. If photo A scores 62% happy and photo B 58%, that gap could come from lighting or face size alone, so you can't conclude A has the bigger smile. Compare photos with similar face size and lighting, and only read meaning into clearly large differences.

Compare it with human judgment

There's a simple way to see how well the AI matches people's impressions. Pick about five photos of your own expressions, ask a few friends what expression each one shows, then run the same photos through the Photo Emotion Analyzer and compare.

Usually, for clear expressions like a big grin, people and the AI agree; for faint or mixed expressions, people disagree among themselves and the AI's answer wobbles too. If people can't agree on a photo, treat the AI's result as one possible reading rather than the right answer.

How the fun-test scores are made

Even convincing-sounding results like first-impression scores or animal types are built on the expression probabilities and landmark ratios described above. The AI isn't separately judging anyone's charm or personality.

• AI First Impression: applies formulas people wrote to the seven expression probabilities to produce scores like friendliness and trustworthiness. The happier the face reads, for example, the higher friendliness goes, which is why smiling and neutral photos score so differently.
• Animal Face Test: measures eye tilt, mouth corners, jawline and eye ratio, then compares how close they are to the preset values for each animal.

The formulas and reference values were chosen by people to make the results fun, not as scientifically validated standards. Each tool page spells out the calculation under "How it works and how to read the results", so you can compare if you're curious why you got a particular result.

What it shouldn't be used for

For the reasons above, expression results wobble easily and, above all, don't reveal what someone feels. That makes them unsuitable for important judgments about people.

• Hiring and assessment: rating people by expression scores from interview videos or photos
• Lie detection: judging truthfulness from changes in expression
• Covert analysis: pointing a camera at someone without their knowledge, or analyzing someone's photo without consent
• Children's photos: analyzing a child's face or sharing results without a guardian's consent

Even for fun, ask first if it's someone else's photo. The same goes for not using results to tease anyone.

Face photos and privacy

A face photo identifies a person by itself, so it deserves more care than other images. ToolyStation's expression tools process photos and camera video inside your browser and don't send them to a server. Only the model files are downloaded from outside; face images are never sent with them.

That said, the result cards from the first-impression and animal-face tests include a crop of the analyzed face. Once you share a card, that photo leaves your device, so double-check where you're posting it and who can see it.

Try some experiments yourself

The fastest way to understand how it works is to change one condition at a time and compare. Turn on the Live Emotion Detector and try these in order.

1. Keep a neutral face looking straight ahead while you switch the room light on and off, or turn toward a window. Watch how much the probabilities move while your expression stays the same.
2. Alternate between a slight smile and a big grin. See how much smiling it takes for "happy" to peak.
3. Slowly turn your head left and right. Noticing roughly where the face box disappears gives you a feel for detection's limits.
4. Cover your mouth with your hand, then your eyes. Compare which one shakes the results more.

These experiments make it tangible what expression AI is looking at, and how far to trust the numbers. Enjoy the results for fun, but remember that those numbers say less about a person than they appear to.