Checking sign-language illustrations automatically

See Me Please runs accessibility testing and inclusive user research with diverse and disabled testers, including Deaf and hard-of-hearing people who sign. Alongside that work we build the Inclusive Images Library, an open source set of illustrations co-designed with the communities they show.

We wanted the library to include fingerspelling: the hand alphabets used within sign languages to spell names, places and words that have no established sign. We wanted accurate, open A to Z sets for Auslan, British Sign Language (BSL), Irish Sign Language (ISL) and American Sign Language (ASL). We found very little that was open, consistent and safe to rely on.

In 2026 a team of University of Sydney School of Computer Science capstone students built the Fingerspelling Asset Generator with us. It produced draft A to Z libraries for ASL, Auslan and BSL. Making them showed us the next problem. Image models can draw hands that look right but are not, and the only dependable check was a fluent signer looking at every image.

The method on this page is our answer to that problem: a way to check each illustration automatically, in the terms that carry meaning in a signed language, while Deaf reviewers stay in charge of what counts as correct.

1. Start from the correct hand. The target handshape for each letter is posed on a 3D model of a pair of hands, and a sign-language consultant approves it, including which points of contact must happen and which must not.
2. Draw from the model. An image model draws the illustration, guided by renders of the 3D hands. It is never shown a photograph of a person.
3. Read the hand back. A hand-pose reader trained on synthetic line drawings, not photographs, recovers the hand from the finished illustration.
4. Compare what matters in sign. The shape of each finger, the thumb, which way the palm and fingers point, how many fingers show, which hand is which, and where the fingers touch.
5. Accept or fix. Each letter has its own tolerances, learned from expert reviewers and set so that wrongly accepted images stay below a chosen rate. A failure triggers a targeted fix, such as redrawing only the wrong finger.
6. Handle movement and privacy. Moving letters are checked frame by frame. Recordings of signers stay inside a controlled environment; only synthetic renders are sent to an image model.

This is a published design. It has not been built or measured, and we make no claims about how well it performs. When we run it, we will report what we find, and nothing will be released without sign-off from Deaf signers.

General-purpose image generators produce hands that look plausible but are articulated wrongly: an extra or missing digit, a finger curled that should be extended, a thumb on the wrong side of the palm, the wrong hand, or a fingertip touching the wrong finger of the other hand. In a signed language a different handshape, orientation or contact is a different letter. In the two-handed alphabets of the BANZSL family (Auslan and BSL), the vowels A, E, I, O and U are told apart mainly by which fingertip of the non-dominant hand the dominant index finger touches.

The obvious automated approaches fail for specific reasons:

1. Hand estimators trained on photographs rely on shading, skin texture and depth cues that clean illustrations lack, so their errors on line art are as large as the defects to be caught.
2. Raw joint error does not measure linguistic correctness. A 15-degree error may not matter for one letter and decide another, and a few millimetres can turn "contact" into "no contact".
3. A single global threshold either sends too many images to people or lets errors through, because letters differ in which features matter.
4. Regenerating blindly with a new random seed ignores what failed, and every rejected attempt is paid for.
5. Conditioning a remote generator on photographs of a signer sends their face and hand identity to a third party.
6. Pixel-space interpolation of moving letters can briefly create extra digits between keyframes, and generating every frame remotely is expensive.

The disclosure has six aspects, which can be used separately or together.

1. Verified generation. Render a target configuration on a rigged hand model, generate an illustration conditioned on the render, estimate the 3D hand from the illustration, map target and estimate into a phonological feature space, and accept only when every feature difference is within that sign’s tolerance. Otherwise regenerate, changing the conditioning according to what failed.
2. Style-domain verifier. Train the estimator on synthetic renders of the same rig, in the house illustration style, with exact pose, keypoint, visibility and contact labels. Lift its output to 3D by fitting the rig under joint limits and collision constraints.
3. Learned, calibrated tolerances. Learn per sign and per feature which differences make experts reject an image, and calibrate the auto-accept threshold to bound the false-acceptance rate at a chosen confidence.
4. Identity-free remote conditioning. Recover poses from recordings only inside a controlled environment, drive a canonical rig, and block any outbound request that is not a registered synthetic render.
5. Verified motion. Interpolate moving signs in joint space, verify every frame, and call the generator only for failing intervals, found by halving.
6. Bound verification record. Store with each accepted illustration a machine-readable record of how it was checked, bound by hash to the image and to the target pose.

• Capture store: consented recordings of signers, held in a controlled environment.
• Pose recovery: runs only inside that environment. An ensemble of estimators reads the peak frames, and the fused keypoints are fitted to a canonical rig. The signer’s own bone lengths are discarded.
• Target authoring: a sign-language consultant reviews the target configuration T and, more importantly, the target feature vector Φ*, including required and forbidden contacts. Φ* is the reference for verification.
• Rig and renderer: a two-hand rig with joint limits and a collision model, and a deterministic renderer for shaded, structure, depth and normal renders, per-digit masks and house-style line art.
• Synthetic data engine and line-art estimator: labelled training drawings from the same renderer, and the model trained on them.
• Feature extractor, decision engine and regeneration planner: compute features, apply gates and tolerances, and choose the next fix.
• Generator adapter and outbound guard: every request to an image generator passes the guard or is not sent.
• Review console and tolerance learner: expert reviewers accept or reject candidates without seeing the automated score, and their labels train the tolerances.
• Orchestrator, motion module and packager: run the loop within budgets, handle moving letters, and publish assets with their records.

The target configuration

For each hand, T records its side and role (dominant or non-dominant), a wrist pose in a body-centred frame, and a local rotation for every rig joint. It also records the rig, the source (a recording frame or direct authoring) and the consultant’s approval.

The phonological feature vector

Target and estimate are both described in features a signer would recognise, for the dominant hand and, in two-handed letters, the non-dominant hand.

Example default flexion thresholds, to be tuned with consultants: extended when MCP and PIP flexion are both below 25 degrees; bent when MCP is at least 45 and PIP below 30; curved when MCP is 20 to 60 and PIP 30 to 80; hooked when MCP is below 25 and PIP at least 60; closed when MCP is at least 60 and PIP at least 80. Example contact distances on a canonical adult rig: ε_t 6 mm and ε_n 15 mm, within a range of 3 to 25 mm.

The difference vector

Between the target Φ* and the estimate Φ̂, the difference vector δ holds:

• categorical features: 0 or 1, or an ordinal distance where classes are ordered (extended, then bent or curved, then hooked, then closed);
• angles: absolute difference in degrees;
• direction vectors: the angle between them;
• finger count: absolute difference; chirality: 0 or 1;
• contact graph: the share of required edges present, the number of forbidden edges present, and partial credit through a region hierarchy (for example 0.5 for touching the PIP joint instead of the tip of the right digit);
• occlusion: the number of inverted pairs.

Peak frames are chosen for each sign, such as the window of lowest joint velocity and contact events. Two or more independent hand-pose estimators read each frame; their keypoints are fused with confidence weighting and robust outlier rejection, then combined across frames with a temporal median. The rig is fitted by minimising a robust keypoint term (normalised by palm length, so the canonical rig fits articulation, not size), plus penalties on joint-limit violations, interpenetration and, once a consultant has marked them, required contacts further apart than ε_t, plus a weak prior towards a reference pose.

Example settings: a Huber threshold of 10 mm; weights of 10 for joint limits, 50 for collisions, 20 for contacts and 0.1 for the prior; a Levenberg-Marquardt solver over the quaternion tangent space, 200 iterations and three restarts. Residuals and penetration depth are recorded for audit.

A target can also be authored directly on the rig by a consultant, with no recording at all, or derived from a sign-notation description such as HamNoSys or SignWriting.

Pose sampling

For example: 40 per cent near the sign inventory (each target perturbed by 5 to 25 degrees per joint, within joint limits); 40 per cent uniform within the joint limits, rejecting collisions; and 20 per cent hard negatives, where one feature is deliberately changed: a digit’s class, a contact moved to the next finger, chirality swapped, or an extra digit added and labelled as an anomaly.

Camera and style sampling

Azimuth within ±40 degrees, elevation within ±25 and roll within ±10 of the house view; focal length 35 to 85 mm equivalent; crop 0.7 to 1.2 times the hand; line weight 1.5 to 4.5 pixels at 512 pixels; stroke jitter up to 1.5 pixels; varying crease density and optional fill. Optionally, a style-transfer pass trained only on accepted illustrations moves a share of renders towards the generator’s real output.

Labels and model

Every render carries exact 2D and 3D keypoints (21 per hand), visibility, per-digit segmentation, joint rotations, the full feature vector and the anomaly label. A top-down keypoint network is fine-tuned on about 200,000 synthetic pairs (range 50,000 to 1,000,000), with extra heads that predict each digit’s flexion class, contact probabilities and the number of visible digits. The digit-count head catches extra or missing fingers that keypoint models silently ignore. The 3D lift uses the same rig fit as pose recovery, without the contact term, so contact is measured rather than imposed.

Independence

The verifier shares no weights or conditioning with the generator. A second estimator of a different family runs as a sanity channel; if the two disagree on any digit by more than a threshold (for example 25 degrees), the image goes to a person.

Some mistakes are never allowed, however good the rest of the image is. These hard gates run before any learned score.

VERIFY(candidate, unit u, target Φ*, tolerance model M_u):
    estimate E and confidence c with the line-art estimator
    if mean(c) < c_min: return REVIEW (low confidence)
    Φ̂ ← features(E);  δ ← difference(Φ*, Φ̂)
    hard gates, never learned away:
        chirality differs                         → REJECT (chirality)
        visible finger count differs on any hand  → REJECT (finger count)
        anomaly head above a_max                  → REJECT (extra digit)
        unit is contact-critical and a required
        contact edge is missing                   → REJECT (contact)
    s ← M_u.reject_probability(δ)                 calibrated score
    if s ≤ t_accept(u):   ACCEPT
    elif s ≥ t_reject(u): REJECT (failing components of δ)
    else:                 REVIEW
    if the sanity channel disagrees: REVIEW

Example defaults: c_min 0.5 and a_max 0.5. The failing components are those whose per-feature tolerance is exceeded, ordered by how much each contributes to the score.

Each candidate has its difference vector and blind accept or reject decisions from expert reviewers (for example two Deaf reviewers, with a third settling disagreements), plus codes for which features failed.

A hierarchical logistic regression models the probability that experts reject an image given its differences, with a monotone spline per feature and sign-specific terms shrunk towards the language-level model, so signs with few labels borrow strength. Monotone gradient-boosted trees are an alternative.

The labels are split by recording session into training and calibration sets. The auto-accept threshold for each sign is the largest value for which the one-sided Clopper-Pearson upper bound on the false-acceptance rate stays at or below a chosen level: for example 2 per cent at 95 per cent confidence. With no false accepts, that needs at least 149 expert-rejected items, so calibration pools signs within a language (and, where it fits, across related languages such as Auslan and BSL, which share most of their manual alphabet), adding per-sign adjustments only where labels allow.

The reject threshold is set the same way to bound false rejection. Tolerance models are versioned and never edited in place. A new version is fitted when labels grow by a set share, or when a drift check on a blind-reviewed audit sample of auto-accepted images fires. Conformal risk control, a Bayesian bound, or tolerances set by hand are alternatives.

Each failing feature maps to actions, tried in order and never repeated for the same sign.

GENERATE_VERIFIED(unit u, target T, target features Φ*):
    R ← render(T, the unit's default view);  history ← []
    for round in 1 .. max_rounds:
        request ← build(R, style exemplar, prompt template for u, mask if any)
        GUARD(request)                       blocks and logs if any check fails
        image ← generator(request, seed derived from u and round)
        decision, δ, s ← VERIFY(image, u, Φ*, M_u)
        history.append(image, request hash, δ, s, decision)
        if decision = ACCEPT: return finalise(image, history)
        if decision = REVIEW: return to_review(image, history)
        R ← plan(R, failing components, history)    next untried action
    return to_review(best candidate by lowest s, history)

Example limits: at most 4 rounds (range 1 to 8) and 12 generation calls per sign. When a limit is reached, the best candidate goes to review with its differences, and every attempt is stored. The order of actions can itself be learned: a contextual bandit, such as Thompson sampling with the failing features as context, reorders them to reach acceptance in fewer calls. Prompt-only changes, latent adjustments, or generating several candidates in parallel and keeping the best one that passes every gate are alternatives.

Every request to an image generator runs through a guard in the same process as the generator adapter, with no bypass. For each image, it checks that:

• its SHA-256 hash is in the registry of images the renderer made for this job;
• a face detector finds no face (for example at a threshold of 0.3);
• its perceptual-hash distance to every frame of the job’s source recordings is above a minimum (for example 70 of 256 bits for PDQ);
• a classifier scores it as a render, not a photograph.

It also checks that the request text holds no recording identifiers, names or consent identifiers. If any check fails, the request is not sent, the job is blocked, an alarm is raised and nothing is retried until a person clears it. The guard is tested with a negative control (a source frame must be blocked), and every request and its results go to an append-only log.

Letters with movement, such as J and Z in ASL and moving letters in the British and Australian alphabets, are handled in joint space rather than pixel space.

MOTION(unit u, keyposes T_0 .. T_k with times):
    events ← contact begins or ends, a digit's flexion class changes,
             or palm orientation changes by more than 30 degrees
    keyframes ← poses at the events, plus the start and end
    between each pair of keyframes: interpolate per joint by spherical
        linear interpolation, project onto joint limits, and reject colliding
        frames, re-routing through a collision-free midpoint
    render every frame deterministically
    for each keyframe: GENERATE_VERIFIED       remote calls for keyframes only
    for each in-between frame: stylise locally from the frame's render and
        its neighbouring verified keyframes   no remote call
    SUBDIVIDE(first frame, last frame)

SUBDIVIDE(a, b):
    if every frame in [a, b] passes VERIFY: return
    if b − a ≤ min_interval: GENERATE_VERIFIED each failing frame; return
    m ← midpoint;  GENERATE_VERIFIED(m) as a new anchor
    SUBDIVIDE(a, m);  SUBDIVIDE(m, b)

Example settings: 24 frames per second and a minimum interval of 2 frames. Outputs include an animated vector graphic, a video with lead-in and lead-out, and a still version for people who turn off motion, built from verified keyframes with a motion cue such as a ghosted start pose and a trajectory.

Left-dominant variants

A left-dominant version mirrors the target in 3D, swaps the dominant and non-dominant roles where the sign is asymmetric, and is regenerated and re-verified. The image itself is never mirrored, because that would also mirror features such as the thumb’s position that must be checked independently.

Verification record

Each accepted illustration carries a machine-readable record: hashes of the image and the target, the sign, the estimator and dataset versions, the feature-space and tolerance-model versions with their target false-acceptance rate, the differences, the score, the decision, the rounds taken, the generator and seed, the guard checks passed, every human review with role and time, and any sign-off. The record’s hash goes into the asset manifest, optionally with a content-provenance manifest such as C2PA, and publication is refused if the hash does not match the asset.

Packaging

Accepted illustrations are vectorised, for example as centreline strokes in currentColor so they follow the page’s theme, written as vector graphics with a text title and description, and exported as raster images and video, each with its manifest and record.

Failing safe

Low estimator confidence, or disagreement between the two estimators, always goes to a person, never to acceptance. A sign with no tolerance model sends every candidate to review. Generation is refused for any target a consultant has not approved. If the generator is unavailable, requests wait and retry; they are never redirected to an unapproved provider or region.

• One-handed alphabets such as ASL and ISL, where the contact graph reduces to contacts within one hand, such as thumb to finger.
• Two-handed alphabets such as Auslan and BSL, with the full graph between hands and the vowels marked contact-critical.
• Other estimators: any model trained on the synthetic line art, a model that outputs rig parameters directly, or an ensemble of two.
• Other feature spaces: subsets for languages with few resources, or learned embeddings with the hard gates kept as a backstop.
• Other generators: a remote service that only receives guarded renders, a local model, a non-diffusion stylisation network, or plain deterministic line art.
• Verification only: checking illustrations drawn by people or third parties, for example for dictionary and education publishers.
• Other gesture vocabularies: therapy exercises, safety hand signals, instrument fingering, surgical and laboratory techniques, and tactile fingerspelling such as the Deafblind manual alphabet.
• Continuous signing, where the features extend to location on the body, movement and facial expression.
• Accessibility testing and user research materials: help content, study instructions, recruitment and consent information, and review items for Deaf signers, each with a record showing how it was checked.

The full disclosure, “Pose-verified generation of sign-language illustrations: closed-loop checking of hand articulation in a phonological feature space, with feature-targeted regeneration”, was published on November 5, 2026 under the Creative Commons Attribution 4.0 licence (CC BY 4.0). It includes every example parameter and all five figures.

Suggested classification (CPC): G09B 21/009, G06T 11/00, G06T 7/70, G06V 40/28, G06N 3/0475. The example values are design defaults; tuned values, training data and expert labels are not disclosed.

The verification method was invented by Katie McDermott, founder of See Me Please, and is published by See Me Please Innovations Pty Ltd.

The Fingerspelling Asset Generator, which produced the draft ASL, Auslan and BSL A to Z libraries this work builds on, was built in 2026 by University of Sydney School of Computer Science capstone students Kejia Cao, Kaiyi Cao, Yushu Lu, Donghai Yin, Junyang Chen and Yueyi Song, in collaboration with See Me Please. It is part of a collaboration between See Me Please and the School that has run every year since 2024.