Learnalist’s handwriting recogniser could read an A. It could read a 6. It could not read Ø, because Ø was not one of the classes it had been trained to recognise.

I could have expanded the dataset and trained another model. That would solve one character for everyone, but it would not solve the wider problem. People write the same character in different ways. A small model trained on a public dataset will eventually meet handwriting that does not look like its training data.

I wanted the app to learn from a handful of examples without uploading anyone’s handwriting or training a neural network on their phone.

The result uses two layers of recognition:

1. A small convolutional neural network recognises common letters, digits and an equals sign.
2. A personal profile compares the model’s internal representation of a new drawing with examples saved by the user.

The second layer is called Train in the app, but it does not change the neural network’s weights. It is closer to adding examples to a small local reference library.

This is the complete flow with Ø. The base model reads my first attempt as O. I add ten examples, switch back to Test, then write it twice more. Both new drawings are recognised as Ø.

Watch the 54-second walkthrough on YouTube. It opens with the failure and the result, then shows the complete test, training and retest flow.

The contact sheet makes the progression easier to scan: the initial failure, the sample count rising to ten, then two successful tests.

The project that made me try it

The idea started when I found InkWordle, a handwritten Wordle game for the reMarkable Paper Pro. You write each guess with the pen, and a small EMNIST model reads the letters on the device.

What caught my attention was the runtime. InkWordle uses tract to run its ONNX model as native Rust code on the tablet. There is no server or Python process involved in recognition. That made me wonder whether the same approach would fit Doodle Recall, where people already write answers from memory inside Learnalist.

InkWordle gave me the push and tract gave me a practical route, but I did not copy its recogniser or model. InkWordle is licensed under GPLv3. I trained Learnalist’s model independently and built a separate rasterisation, inference and personalisation pipeline for the app.

Training the base model

The base recogniser is a compact PyTorch model trained on EMNIST letters and digits. I also added synthetic equals signs, mostly to see whether I could extend a public handwriting dataset with a generated class.

The input is a single 28 by 28 greyscale image. Two convolution and pooling stages reduce it, then a 96-value hidden layer feeds the final classifier.

class DoodleCnn(nn.Module):
    def __init__(self) -> None:
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 16, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 32, 3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(2),
        )
        self.classifier = nn.Sequential(
            nn.Flatten(),
            nn.Linear(32 * 7 * 7, 96),
            nn.ReLU(),
            nn.Linear(96, 37),
        )

The 37 outputs cover A to Z, 0 to 9, and =. This is deliberately narrow. The model only needs to support the short answers people draw during Doodle Recall. A general handwriting system would be larger and would solve a different problem.

The training data combines EMNIST’s letter split, a balanced subset of its digits, and generated equals signs. The synthetic examples vary their width, spacing, angle, brightness and stroke thickness. The training script uses a fixed seed so I can reproduce a run and compare changes.

After training, I test against EMNIST’s held-out test split and a separate set of generated equals signs.

That test is useful, but it only tells me how the model performs on data shaped like its training data. It does not tell me how well it understands a finger drawing captured by the app. I need a separate test for that.

Exporting more than a prediction

A normal classifier returns its scores for the available classes. For personal recognition, I also export the 96 values from the hidden layer immediately before the classifier.

class ExportModel(nn.Module):
    def forward(self, value: torch.Tensor):
        embedding = self.model.embedding(value)
        logits = self.model.classifier[3](embedding)
        return logits, embedding

Those values form an embedding. Drawings that the model considers similar should sit near each other in this 96-dimensional space, even when the final classifier gives them the wrong label.

This makes the base model useful for more than its original 37 classes. It has already learnt features such as curves, crossings, loops and stroke arrangements. The personal recogniser can reuse those features for a character the classifier has never seen.

The exported ONNX file is bundled with the app. Recognition stays on the device and works without a network connection.

Running ONNX through tract

Learnalist is a Flutter app. A small Rust library loads the ONNX model with tract and exposes a C interface to Dart. Tract is a self-contained inference toolkit for TensorFlow and ONNX models, written in Rust.

The boundary is intentionally small. Dart supplies 784 floats and receives 37 logits plus the 96-value embedding.

#[unsafe(no_mangle)]
pub unsafe extern "C" fn doodle_recognition_predict_with_embedding(
    input: *const f32,
    input_len: usize,
    output: *mut f32,
    output_len: usize,
    embedding: *mut f32,
    embedding_len: usize,
) -> i32 {
    // Validate the buffers, run tract, then copy both outputs.
}

Rust gives me a portable inference boundary for iOS, Android and any future desktop version. Python is only needed by the developer who trains and exports the shared model. It does not need to be installed on the user’s device.

Before inference, Flutter converts the finger strokes into the same form the model saw during training. It finds the drawing bounds, preserves the aspect ratio, centres the glyph and rasterises it onto a 28 by 28 black canvas.

This preprocessing is part of the model contract. A change to centring, scale or stroke thickness can alter recognition even when the ONNX file stays the same.

What Train does in the app

The personal training screen asks the user to draw a character up to ten times. Each drawing passes through the normal rasteriser and base model. The app stores both the 28 by 28 raster and its normalised embedding.

These are the ten Ø examples at the moment I saved each one. The first frame shows the empty canvas before training began.

The app does not store those screenshots. It centres and scales each drawing onto the model’s 28 by 28 input canvas. Here are the same ten samples from the saved personal handwriting profile.

For each character, it calculates the mean embedding of the saved samples. This mean is the character’s centroid.

When a new drawing arrives, the app compares its embedding with every personal centroid using cosine similarity. The closest result is accepted only when it is strong enough and clearly better than the runner-up.

final scores = <(String, double)>[];

for (final entry in samples.entries) {
  final centroid = mean(entry.value.map((sample) => sample.embedding));
  scores.add((entry.key, dot(normalisedInput, centroid)));
}

scores.sort((a, b) => b.$2.compareTo(a.$2));

if (bestScore < threshold) return null;
if (bestScore - runnerUpScore < 0.04) return null;

return PersonalHandwritingMatch(
  character: bestCharacter,
  similarity: bestScore,
);

Rejecting an uncertain result matters. A personal profile should improve the base recogniser, not override every prediction because one saved example happens to be nearest.

The threshold becomes more useful as samples are added. With one example, the app requires a very close match. With several examples, it looks at how consistent the saved samples are and sets the threshold from that cluster, with a conservative lower bound.

If no personal result passes these checks, the app keeps the base model’s answer.

Why I keep the original rasters

The personal profile stores embeddings, but embeddings belong to a particular model and preprocessing pipeline. If I replace the base model or change how a glyph is rasterised, old embeddings may no longer be comparable with new ones.

Keeping the 28 by 28 rasters provides a migration path. The app versions the profile against the model and preprocessing code. When either version changes, it can run the saved rasters through the new model and rebuild the embeddings locally.

The profile also records which account owns it. It can be reset, individual samples can be removed, and custom characters remain visible even before their first sample is collected.

These details are less interesting than the similarity calculation, but they make the feature safe to ship rather than merely convincing in a prototype.

Testing my own handwriting

I recorded a small test with Ø, a useful Norwegian character that the base classifier cannot return.

Before training, the model read my drawing as O with 25.7% confidence. Its other leading candidates were B at 25.3% and 0 at 18.8%. There was no personal result because I had not saved any Ø samples.

I added ten examples, then made five new test drawings. The personal threshold was 0.9000.

Three of the five personal matches were accepted. The other two were close to Ø, but remained below the threshold. The app kept the base result instead of forcing a weak personal match.

The video ends after the first two successful tests. I kept it short because it demonstrates the feature clearly. The table comes from the complete device log and includes the three later attempts.

This is a test of one character written by one person. It is not a benchmark for handwriting recognition in general. It answers a narrower product question: can a few local examples correct a real failure while rejecting uncertain matches?

The app also has separate Train and Test views. Train saves a drawing. Test recognises it without saving it, then shows the result. Keeping those actions separate helps prevent an apparently successful test from quietly becoming another training example.

A useful middle ground

Full custom training is possible. I could train a new classification head on the device or adopt a framework designed for on-device learning. That brings more code, more failure modes and more decisions about when and how training runs.

Embedding comparison gave me a smaller first step. The shared model does the expensive work of learning useful handwriting features. Each user supplies a few labelled examples. A dot product and several conservative checks adapt the result to their handwriting.

This approach will not replace full training in every application. It works here because each personal class has few examples, inference needs to be quick, and the app can safely fall back to its bundled classifier.

The broader lesson is to inspect what a small model already knows before training another one. Its final prediction is only one output. The representation behind that prediction may be enough to build the custom behaviour your product needs.

Learnalist uses Doodle Recall to help people practise words, facts and short answers by writing them from memory.

References

• Learnalist on the App Store
• Learnalist mobile app
• Handwriting recognition walkthrough
• Convolutional Neural Networks, Stanford CS231n
• EMNIST dataset and downloads, NIST
• Embedding space, Google for Developers
• InkWordle
• tract
• ONNX is an open format built to represent machine learning models