Lessons Learned

This is the last article in the series. It is the retrospective: what held up, what did not, and what I would do differently.

I will start with the number that matters most, because it shaped everything else:

Four of five plausible fixes measured worse than the bug they were meant to fix.

Not four out of five bad ideas. Four out of five reasonable ideas: deadbands, low-pass filters, stateless solving, fixed-point convergence. Each is a technique you would defend in review. Each made the system worse, and I only know that because there was an instrument measuring it.

If you take one thing from this series, take that.

For grounding, the final state:

Corpus                48 sentences, ~31,000 hand-frames
Artifacts             4,062 unconstrained  ->  118
Zero-count kinds      handFlip, wristOverBend, teleport, jitter,
                      fingerCollision, bodyPenetration, headPenetration,
                      elbowLocked, elbowFolded, morphFlicker
Remaining             110 flicker, 6 hand collision, 2 hand-in-forearm
Reference sentences   clean at 30 / 60 / 120 fps
Test suite            ALL CHECKS PASSED (2,719 signs synthesise)
GLB                   10.6 MB
Bundle                381 kB First Load JS

Nothing pre-recorded. Every frame synthesised at runtime from ASL-LEX phonology.

The artifact detector (diagnostics.ts, 15 kinds) is the single highest-value thing in the project. It is the reason the four of five number exists rather than being a vague sense that some fixes helped. The full framework it sits inside is documented in Part 5 on loop engineering.

What made it work was not sophistication: it was existing before the fixes, running over a fixed corpus, and tracking multiple independent metrics so a change could not win by trading one for another.

That last property caught the most seductive failure in the project: a stateless solve-from-zero that cut flicker from 163 to 10 while multiplying total artifacts by eighteen. On a single-metric dashboard that is a triumph.

Bend axes, palm normals, segment lengths, and rest flexion are all measured from the rest pose at load time.

/**
 * How far each joint is already flexed in the bind pose.  The character is
 * modelled with relaxed, slightly curled fingers, so a flexion of 0 has to
 * *straighten* the finger rather than leave it alone.
 */
restFlex: number[];

This paid for itself repeatedly. The character was re-exported several times, twice for facial morph fixes, and the solver recalibrated on load each time with nothing to update by hand.

Choosing ASL-LEX over motion capture or a motion-generation model was the decision the whole project rests on.

Phonological annotation (handshape, location, movement, contact) is a parametric description of a sign. That is what a synthesiser can consume. Video is a recording of one production of one sign by one signer; you cannot interpolate between recordings and get grammar.

The research prompt named MediaPipe and Hugging Face models. Both were wrong: MediaPipe reads poses from video, and this project needed to generate them. Being allowed to reject the suggested technologies was what made the right answer reachable.

resolvers = { body: true, handHand: true, handLimb: true, swivel: true };

Being able to disable each subsystem at runtime turned something causes flicker into a bisection. One table settled it:

• All on: 115 flicker events.

• All off: 58 flicker events.

• handHand off: 71 flicker events.

58 events survive with everything off, so half the problem was never the collision system. That is fifteen minutes of work that would otherwise have been days of guessing.

/**
 * Slowing BOTH to 1.2 cut flicker (141 -> 112) but let collisions straight
 * back in (head penetration 0 -> 53).
 */
const MAX_ESCAPE_RATE = 14.0;

Comments at the point of temptation, not in a separate document. The next person to think 14 seems high reads why immediately.

Twice I described a residual wobble as sub-millimetre, invisible based on model-space numbers. The character is about 11 units tall in model space; those numbers meant nothing on their own.

Projected through the app's actual camera: median 3.6 px, worst 17.4 px. Clearly visible.

Both times, what corrected it was a human saying the animation still looked wrong while the metric said fine.

The lesson is not measure carefully. It is that a metric in convenient units is a proxy, and proxies drift from the thing you care about silently. For anything visual, measure in pixels through the real camera.

A change scored 76, the best total of the entire project, every metric down.

The clearance predicate returned max(0, penetration). It could never go negative. So release this correction only if there is clearance to spare was unsatisfiable, and the release silently never ran. Corrections became permanent. Hands stopped oscillating because they stopped returning to where they belonged.

What caught it: a parameter sweep with identical results at every value. margin 0.003 to 76, 0.006 to 76, 0.010 to 76, 0.022 to 76, 0.035 to 76.

A parameter that changes nothing is not well-tuned. It is not being read.

The honest score was 118.

Fingers hyperextending (inverted cross product), open handshapes rendering half-closed (rest pose treated as zero), spread working backwards (inverted bias table).

Three in a row is not bad luck. Any system with measured axes has a sign convention per axis, and each is a 50/50. The fix that scales is not care, it is writing the geometric argument at the point of definition:

// Axis whose positive rotation sweeps the fingertip toward the palm:
// closing a fist moves the tips along +palmNormal (out of the palm
// plane, ending against the palm), and (d x n) x d = +n.

The mouth morpheme table existed from early on. Nothing ever assigned nmm.mouth except a smile on warm signs, so every other entry was dead code for most of the project.

Worse, SCHOOL's stutter (which took three iterations to solve) was caused by the resolver fighting a contact the sign intends. ASL-LEX records contact = 1 for SCHOOL. The data was there from day one. I was not reading it at runtime.

The lesson: when you import a rich dataset, audit which fields you actually consume. I shipped a system using maybe half of what ASL-LEX offered while debugging problems the unused half described.

There is no .git in this project. The rejected-fix comments are the only record of what was tried; there is no bisect, no history.

Every other mistake here is recoverable. This one compounds.

• Initialise git first: Before the first script. The retrofit cost only grows.

• Build the detector before the animation system: It arrived partway through, and everything before it was evaluated by opinion.

• Audit the dataset's fields on day one: Write down every column, then mark which ones the code reads. The gap is your backlog.

• Split rig.ts: At 1,363 lines it holds rig binding, IK solving, and collision resolution. Cohesive, but past the size where finding things is quick.

• Name media directories differently from data directories: ASL data/ (3.8 GB of screen recordings) sits next to data/ (the actual ASL-LEX source). One is an input; the other is not. The names do not say so.

• Involve a Deaf signer from the start: More on this below: it is the real one.

The prompt I would write now for the detector:

Before writing any animation code, build the measuring instrument.
Measure in screen pixels, not model units - project through the camera
the app actually uses. Run it over a fixed corpus. Report the baseline
before any fixes exist.

Being specific rather than enthusiastic, because the specifics are more useful.

• Generating candidate causes: Given the hand flips during NAME, it enumerated eight plausible mechanisms across quaternion math, IK/FK switching, retargeting and blending. The actual cause was none of them (a swing-twist singularity) but the enumeration is what made the search systematic rather than random.

• Building instruments on demand: This is where the leverage really was. The 15-artifact detector, the posed-wrist trace probe, the filmstrip renderer, the pixel-space projection, the morph displacement profiler, each written in minutes when the question demanded it. Historically I would have estimated instead of measuring, because building the tool felt like a detour.

• Changing the economics of let me just check: When checking is cheap, you check.

• Mechanical thoroughness: Running a 48-sentence corpus across seven resolver configurations at three frame rates and tabulating it. Correct, tedious, and exactly the work humans skip.

• Holding context across sessions: Remembering that a low-pass filter had already measured 230 to 3286, so it did not get retried.

• Following a workflow when told to: Test to Observe to Identify Root Cause to Fix to Retest to Verify, plus do not apply fixes until you understand exactly why, reliably converted a proposed weight tweak into two genuine defects found in the morph generator.

• Noticing it looked wrong: The detector reported acceptable numbers for SCHOOL. A person watching said still flickering. The person was right, since the metric was in model-space units. This happened more than once, and it is the highest-value input in the whole system.

• Deciding what better means: Never releasing collision corrections scores better on every artifact metric. Deciding that hands permanently displaced by 6.6 hand-gaps is worse than measured flicker is a judgement about what the system is for. No metric contained that.

• Domain knowledge that is not in the data: That contact is phonemic in ASL, so hands must never intersect is wrong as a universal constraint. That names must be fingerspelled even when a sign exists. That blink rate drops under concentration.

• Distrusting a good result: The 76 was accepted for several minutes. What broke it was an instinct that the number was too clean.

• Setting the constraints: Do not recreate the character. No Docker or CI. Keep everything in this folder. Each removed a category of unwanted work. None would have been inferred.

The honest summary: the agent was faster at generating hypotheses and building tools to test them. The human contributions were noticing when a metric diverged from perception, and deciding what to optimise for. Those are not small.

The most important paragraph in the series.

No Deaf signer has reviewed this system. Not one.

I can tell you the animation is geometrically clean: zero body penetration, zero finger collisions, zero hand flips across 48 sentences. I can tell you the phonology comes from a peer-reviewed database and the grammar layer implements topic-comment order, time-first, copula dropping, wh-movement and non-manual markers.

I cannot tell you it is fluent, natural, or comfortable to read. Those are judgements only fluent signers can make, and none were involved.

That is worth naming plainly, because there is a pattern of hearing engineers building sign language technology without Deaf involvement, publishing accuracy numbers that measure the wrong thing, and producing tools the community finds unusable. The Deaf community has been clear about this for years.

My artifact counts measure engineering quality. They say nothing about linguistic quality. A system can score zero collisions and still be unreadable: palm orientation alone can turn a correct sign into a nonsense one, and ASL-LEX does not annotate palm orientation, so most of the 2,719 signs use a default.

So: treat this as an engineering demonstration of runtime sign synthesis, not as an ASL translation tool. It is a study of how to drive a rig from phonological data, and of how to work with an AI agent on a hard measurement problem. It is not a product, and it should not be used as one.

If this went further, Deaf collaboration would not be a validation step at the end. It would be the first thing.

Ordered by what I actually think matters.

Before any feature work. Specifically: are the reference sentences readable? Which of the tested signs are wrong? Is the timing comfortable? Are the non-manual markers doing what they should?

That review would very likely reorder everything below it, which is the point.

The weakest link in the linguistic chain. ASL-LEX does not annotate it, so it comes from per-location defaults plus overrides for eleven signs. Everything else is a guess that happens to look plausible.

Options: extend the override table with expert input, or derive orientation from video using pose estimation, which is, finally, a good use for MediaPipe. Not to generate animation, but to annotate the missing field.

The GLB is 10.6 MB. The entire JavaScript bundle is 381 kB, a factor of 28 smaller. Draco or meshopt is the obvious untaken win, followed by cutting clothing subdivision. Face and hand density stays; that is where the meaning is.

ASL uses classifier handshapes to depict shape, movement and spatial relationships. Not implemented at all. This is a large piece of real ASL that the current system simply cannot express.

Directional verbs (GIVE, ASK, HELP) inflect by moving between spatial loci. Third-person indexing works; spatial modulation of verbs does not.

Mechanical, and it should happen before any of the above.

1. 1

   Build the measuring instrument before the thing it measures: Four of five plausible fixes measured worse; without an instrument you ship one and believe it helped.

2. 2

   Track multiple metrics: A single-metric view would have accepted a change that cut flicker 163 to 10 while multiplying total artifacts eighteenfold.

3. 3

   Measure in units the user perceives: Model-space numbers called a 17-pixel wobble invisible.

4. 4

   Distrust suspiciously good results: The best score in the project came from a predicate that could never be satisfied.

5. 5

   Sweep every new parameter as a wiring test: Identical results at every value means it is not being read.

6. 6

   Calibrate from the rig, do not hard-code: Measuring at load time meant three re-exports cost nothing.

7. 7

   Audit which fields of your dataset you actually consume: SCHOOL's stutter was described by a column I was not reading.

8. 8

   Record rejected fixes with their measurements, in the code: At the point where someone will be tempted.

9. 9

   Name the constraint you do not want violated, including the negative space: It is one sentence and it removes a category of unwanted work.

10. 10

    The agent is fastest at hypotheses and tooling: The human contributions are noticing a metric diverging from perception, and deciding what better means.

11. 11

    Engineering metrics are not domain validation: Zero collisions says nothing about whether a Deaf signer can read it.

• ASL-LEX 2.0: Sevcikova Sehyr, Z., Caselli, N., Cohen-Goldberg, A. M., and Emmorey, K. (2021). JDSDE 26(2). CC BY-NC 4.0.

• Fritsch, F. N., and Carlson, R. E. (1980): Monotone Piecewise Cubic Interpolation. SIAM J. Numer. Anal. 17(2).

• Valli, C., and Lucas, C.: Linguistics of American Sign Language. ASL phonology and non-manual markers.

• Liddell, S. K. (2003): Grammar, Gesture, and Meaning in American Sign Language. Spatial reference, indexing, classifiers.

• Draco and meshopt: glTF geometry compression, the first optimisation this project should take.

• Claude Code documentation: docs.claude.com/en/docs/claude-code (https://docs.claude.com/en/docs/claude-code).

This is Part 10, the final article. The other parts:

1. 1

   How I Built an AI-Powered ASL Sign Character with Claude Code

2. 2

   How Blender MCP Works with Claude Code

3. 3

   Building a Production Ready Sign Language Character

4. 4

   Designing the Goal Prompt

5. 5

   Loop Engineering Explained

6. 6

   Building Natural ASL Animation

7. 7

   Project Folder Architecture

8. 8

   Every Prompt Used During Development

9. 9

   Common Problems We Solved

Building the artifact detector before attempting fixes, and running it over a corpus that never changed. It is the reason I can say four of five plausible fixes measured worse rather than guessing which ones helped.

Technically, measuring wobble in model-space units and concluding a visible artifact was invisible, twice. Structurally, not initialising version control, which is the one mistake here that compounds instead of staying fixed.

No. It is an engineering demonstration of runtime sign synthesis. No Deaf signer has reviewed the output, palm orientation is defaulted for most of the 2,719 signs because ASL-LEX does not annotate it, and classifier constructions and verb agreement are not implemented. The artifact numbers measure geometric cleanliness, not linguistic quality.

Nearly all of it. That is less interesting than where it helped most, which was building measurement tools on demand: the detector, the trace probe, the filmstrip renderer, the pixel projection. When checking a hypothesis becomes cheap, you check hypotheses instead of estimating, and that changes how the work goes.

Noticing the animation looked wrong while the metric said fine. Deciding that permanently displaced hands are worse than measured flicker, even though every artifact number improved. Knowing that contact is phonemic in ASL so hands must never intersect is wrong as a universal rule. And setting the constraints that prevented whole categories of unwanted work.

Not here. The rule-based layer is deterministic, runs in microseconds in the browser with no network dependency, and is debuggable: when it produces a wrong gloss you can find the rule and fix it. For a system meant to run entirely locally, those properties beat broader coverage.

Get a Deaf signer to review the five reference sentences. That review would very likely reorder the entire roadmap, which is exactly why it should come first rather than last.

That plausible fixes fail at a surprising rate, and you cannot tell which without an instrument that predates them. Four of five here. The corollary (distrust suspiciously good results, sweep parameters to check they are wired up, and count how often your conditional logic actually fires) applies to any system complex enough that you cannot hold it all in your head.

If you are working on hard, measurable engineering problems where an AI agent can compound the effort (real systems, real users, real feedback loops) that is exactly the kind of AI product engineering we ship at ETechViral. Thanks for reading the series.