Building Natural ASL Animation

English text
   |  gloss.ts        rule-based ASL grammar (compromise for POS/lemma)
   v
ASL gloss + non-manual markers
   |  synthesize.ts   ASL-LEX phonology -> articulatory trajectory
   v
keyframed body poses  (wrist target + palm orientation + handshape + face)
   |  player.ts       interpolation, blinks, breathing, sway
   v
rig.ts               analytic two-bone IK + calibrated finger FK + morphs
   v
the character, in three.js

interface Word {
  raw: string;
  norm: string;
  tags: string[];
  isProper: boolean;
  isTime: boolean;
  isWh: boolean;
}

- determiners and the copula are dropped (ASL has neither)
- time is established first  ("YESTERDAY ME GO STORE")
- topic-comment order, with the topic marked by raised brows
- wh-words move to the end of the clause and carry furrowed brows
- yes/no questions raise the brows over the whole clause
- negation becomes NOT plus a headshake spanning the predicate
- tense is lexical: FINISH for completed, WILL for future
- anything with no lexical sign is fingerspelled, names always

const DROP = new Set([
  'a', 'an', 'the', 'is', 'am', 'are', 'was', 'were', 'be', 'been', 'being',
  'do', 'does', 'did', 'of', 'to', 'there', 'that', 'as', 'at', 'by',
  'shall', 'would', 'could', 'may', 'might', 'must', 'ought',
]);

/**
 * Order a clause the way ASL does: TIME first, then the topic, then the
 * comment, with wh-words pushed to the end.
 */
function reorder(words: Word[]): { ordered: Word[]; topicCount: number } {
  const time = words.filter((w) => w.isTime);
  const wh = words.filter((w) => w.isWh);
  const rest = words.filter((w) => !w.isTime && !w.isWh);
  return { ordered: [...time, ...rest, ...wh], topicCount: time.length };
}

if (i < topicCount) tokenNmm.topic = true;

clause.push({ gloss: 'FINISH', signId: id, source: '(past)', nmm: {} });

/**
 * Third-person reference is spatial in ASL - the signer points at a locus - so
 * HE/SHE/THEY become an index point rather than a lexical sign.
 */
const THIRD_PERSON: Record<string, 'ix3' | 'ix3_plural'> = {
  he: 'ix3', him: 'ix3', his: 'ix3',
  she: 'ix3', her: 'ix3', hers: 'ix3',
  ...
};

// Resolve to a sign first, then decide about fingerspelling.
//
// This used to test `lex.index[mapped]`, which is keyed by English words,
// against a value that may already be a *sign id* - LEXICAL_MAP sends
// "thank" to `thank_you`, and no English word is spelled with an
// underscore.
const id = lookup(lex, [mapped, ...lemmas(w)]);

handshape  -> finger flexion / spread / thumb   (handshapes.ts)
location   -> wrist target + orientation        (locations.ts)
movement   -> the path between the two          (synthesize.ts)
sign type  -> what the non-dominant hand does   (synthesize.ts)

export interface Handshape {
  /** thumb chain: [CMC, MCP, IP] */
  thumb: Curl3;
  index: Curl3;
  middle: Curl3;
  ring: Curl3;
  pinky: Curl3;
  /** 0 = fingers touching, 1 = fully splayed */
  spread: number;
  /** 0 = thumb alongside the palm plane, 1 = rotated across the palm */
  thumbOpp: number;
  /** 0 = thumb tucked to the index, 1 = thumb swung away from the hand */
  thumbAbd: number;
}

path.length ? [start, ...path] : [start, { ...start }]

Unknown words - names above all - are fingerspelled rather than dropped.
The hand sits in the dominant-side "fingerspelling window" near the shoulder
with the palm toward the addressee, letters flow with a small forward drift,
doubled letters bounce sideways, and J and Z carry their citation movement.

export function spellWindow(anchors: Anchors, dom: Side): Vec3 {
  const s = dom === 'L' ? 1 : -1;
  const chin = anchors.chin;
  return [s * 0.82, chin[1] - 0.32, chin[2] + 0.72];
}

const doubled = i > 0 && letters[i - 1] === ch;
// letters drift very slightly forward across the word, doubles step sideways
const drift = (i / Math.max(1, letters.length - 1)) * 0.10;
const lateral = doubled ? -s * 0.16 : 0;

if (doubled) {
  // small re-articulation so the two letters read as two
  keys.push({ t: t - ctx.rate * 0.35, ease: 'easeOut', pose: ... });
}

if (MOVING_LETTERS.has(ch)) {
  const path = ch === 'j' ? jPath(base, s) : zPath(base, s);
  ...
}

// lead-in so the hand arrives in the window before the first letter
const lead = 0.16;

if (nmm.ynq) {
  morphs.browRaise = 0.9 * env;
  morphs.eyeWide = 0.3 * env;
  head = [0.09 * env, 0, 0];
}
if (nmm.whq) {
  morphs.browFurrow = 0.85 * env;
  morphs.squint = 0.25 * env;
  head = [0.07 * env, 0, 0.05 * env];
}
if (nmm.neg) {
  morphs.browFurrow = Math.max(morphs.browFurrow ?? 0, 0.5 * env);
  head = [head[0], Math.sin(phase * Math.PI * 4) * 0.16 * env, head[2]];
}

// ramp the marker in and out so it does not pop
const env = Math.min(1, Math.sin(Math.PI * Math.min(1, Math.max(0, phase))) * 1.8);

export const MOUTH_MORPHEMES: Record<string, Record<string, number>> = {
  oo: { mouthPucker: 0.85 },
  ee: { mouthWide: 0.8 },
  mm: { mouthPress: 0.7 },
  cha: { cheekPuff: 0.85, jawOpen: 0.3 },
  puff: { cheekPuff: 0.9, mouthPress: 0.45 },
  th: { mouthTH: 0.8 },
  ah: { jawOpen: 0.55 },
  pah: { jawOpen: 0.7, eyeWide: 0.5 },
  smile: { mouthSmile: 0.7 },
};

/**
 * An adverbial mouth morpheme owns the mouth for the length of its sign.
 *
 * The affect and effort passes below also reach for the mouth, and they use
 * Math.max, so without this the pursed `oo` of SMALL would be layered under a
 * warm sentence's smile - two antagonistic shapes driven at once, which on a
 * blendshape face reads as a grimace rather than as either expression.
 */
const mouthTaken = !!shape && nmm.mouth !== 'smile';

// The face carries the attitude of the utterance continuously.  A warm sign
// lifts the cheeks, brows and mouth together; a negative one draws the brows
// in, narrows the eyes and wrinkles the nose.  Half-envelope so the affect
// lingers across the sign rather than pulsing with it.

- spontaneous blink rate is roughly 15-20/min at rest and *drops* while
  concentrating, which for a signer means while actually signing
- a blink closes faster than it opens (~70 ms down, ~110 ms up)
- the two lids are not perfectly synchronous - a few milliseconds apart
- some blinks are partial: the lid drops ~40% and comes straight back
- saccades are ballistic: ~30-60 ms of movement between fixations that
  last a few hundred milliseconds, not a smooth drift

const BASE_INTERVAL_IDLE = 3.4;
const BASE_INTERVAL_SIGNING = 5.2;

Signers blink at prosodic boundaries the way speakers pause, and gaze
carries reference: looking at a locus in space is how a pronoun gets its
antecedent.

/** head rotation in radians: pitch (nod +down), yaw (+left), roll (+tilt left) */
head: Vec3;
/** torso rotation, same convention; used for role shift and body lean */
torso: Vec3;

const MIN_SIGN = 0.34;
const MAX_SIGN = 1.5;

const total = Math.min(MAX_SIGN, Math.max(MIN_SIGN, (sign.dur || 700) / 1000)) / speed;

/**
 * Fastest average wrist speed, in model units per second.  Easing doubles the
 * peak, so this corresponds to roughly 2 m/s at the fastest point - brisk
 * signing, but not a blur.  Transitions that would exceed it are stretched.
 */
const MAX_AVG_SPEED = 6.5;
/**
 * Fastest average hand rotation, in radians per second.  Easing roughly doubles
 * the peak, so 5.2 rad/s averages ~300 deg/s and peaks near 600 - inside what a
 * signer's hand actually does.  It used to be 7.3, which let peaks reach
 * ~920 deg/s; combined with a near-180-degree turn that reads as the hand
 * tumbling rather than travelling.
 */
const MAX_ANGULAR_SPEED = 5.2;

function hermite(
  p0: number, p1: number, p2: number, p3: number,
  dtPrev: number, dt: number, dtNext: number, s: number,
): number {
  // Secants either side of each knot.
  const d1 = (p1 - p0) / dtPrev;
  const d2 = (p2 - p1) / dt;
  const d3 = (p3 - p2) / dtNext;

  // Monotone (Fritsch-Carlson) tangents.  A plain finite difference overshoots
  // wherever the motion reverses, and a repeated sign reverses constantly -
  // SCHOOL claps down, up, down, up - so the hand would sail past the palm at
  // each extreme and come back.
  const limit = (a: number, b: number) => {
    if (a * b <= 0) return 0;                   // turning point: flat
    const m = (a + b) * 0.5;
    const cap = 3 * Math.min(Math.abs(a), Math.abs(b));
    return Math.sign(m) * Math.min(Math.abs(m), cap);
  };
  const m1 = limit(d1, d2);
  const m2 = limit(d2, d3);

  const s2 = s * s;
  const s3 = s2 * s;
  const h00 = 2 * s3 - 3 * s2 + 1;
  const h10 = s3 - 2 * s2 + s;
  const h01 = -2 * s3 + 3 * s2;
  const h11 = s3 - s2;
  return h00 * p1 + h10 * dt * m1 + h01 * p2 + h11 * dt * m2;
}

function easeInOut(t: number) { return t < 0.5 ? 2 * t * t : 1 - Math.pow(-2 * t + 2, 2) / 2; }
function easeOut(t: number) { return 1 - Math.pow(1 - t, 3); }

case 'hold': return t < HOLD_FRACTION
  ? 0
  : easeInOut((t - HOLD_FRACTION) / (1 - HOLD_FRACTION));

const ORIENT_CACHE = new WeakMap<HandTarget, Quaternion>();
export function handOrientation(h: HandTarget): Quaternion { ... }

// --- cleanup passes ------------------------------------------------------
// Order matters: slow the fast moves down first (which is what makes hands
// shoot through the body in the first place), then route whatever still
// crosses the body around it.
isolateHandTargets(keys);
if (opts.body?.valid) liftOutOfBody(keys, opts);
openCrowdedHands(keys);
const remap = enforceMaxSpeed(keys, opts.speed);
const routed = opts.body?.valid ? routeAroundBody(keys, opts) : keys;
// Routing inserts displaced waypoints, which can make an interval fast again,
// so the speed limit is re-applied afterwards.
const remap2 = enforceMaxSpeed(routed, opts.speed);

// The cleanup passes below mutate hand positions in place, and buildSign
// legitimately reuses one HandTarget across many keyframes (the idle hand of
// a one-handed sign is a single object).  Without this every pass would apply
// its correction once per referencing keyframe and the offsets would compound.

"Hi, my name is Muhammad"
   |
   |  compromise: POS tags, lemmas, proper-noun detection
   v
[hi][my][name][is][Muhammad]
   |
   |  DROP removes "is" (copula)
   |  LEXICAL_MAP: hi -> hello
   |  "Muhammad" is a proper noun -> fingerspell
   v
HELLO   MY   NAME   fs-MUHAMMAD
  |      |     |         |
  |      |     |         +- fingerspell.ts: window, drift,
  |      |     |            doubled M bounce, 0.16s lead-in
  |      |     |
  |      |     +- ASL-LEX: two-handed, contact, H handshape
  |      |        -> handsTouch = true
  |      |
  |      +- body-anchored possessive, overrides.ts orientation
  |
  +- WARM_SIGNS -> mouth: smile, affect 0.6
   |
   v  synthesize.ts: keyframes with wrist targets, orientations,
   |  handshapes, morphs, head/torso
   v
   |  cleanup: isolate -> lift -> open -> speed -> route -> speed
   v
   |  player.ts: monotone Hermite on position, slerp on orientation,
   |  blinks at sign boundaries, idle sway
   v
   |  rig.ts: two-bone analytic IK + swivel, calibrated finger FK,
   |  collision resolvers, morph application
   v
bone quaternions + morph influences -> three.js