Mojibake 0.1.1

Mojibake

Detects and repairs mojibake: text that was garbled by an encoding mistake.

café        ->  café
résumé     ->  résumé
El Niño     ->  El Niño
—          ->  —
30€        ->  30€
😊          ->  😊

(Every pair above is the actual input and output, verified against ftfy 6.3.1.)

using Mojibake;

string clean = Text.Fix("café");   // "café"

Mojibake is a focused .NET port of the core of Python's ftfy (fixes text for you) fix_encoding algorithm. It is ported verbatim from ftfy 6.3.1. On the test corpus, the repaired text and the step plan match ftfy 6.3.1 exactly: Text.Fix returns what ftfy returns, and Text.FixAndExplain reports the same step plan as ftfy.fix_encoding_and_explain. Text.Badness and Text.LooksLikeMojibake match ftfy exactly for BMP text; they can under-count when a non-BMP (astral-plane) letter or number sits immediately adjacent to a mojibake signature, because .NET regex matches on UTF-16 code units while Python matches on code points. This affects only the detection score, never repair. Full code-point-aware badness for astral input is planned for a future release. Mojibake adds one deliberate, safer deviation: a no-worsening guarantee (see below), so on the rare (theoretical) deeply multiply-encoded input where ftfy would emit a higher-badness partial result, Mojibake returns the input unchanged instead. Text.Fix is therefore parity-or-safer: it returns either exactly what ftfy returns, or, when that would be worse than the input, the input itself, never a third value. Zero external dependencies.

The gap this fills

ftfy is the gold standard for repairing mojibake, with millions of downloads a month, but it is a Python library. On .NET there has been nothing that does its job. UTF.Unknown (the Ude successor) only detects the charset of raw bytes; it cannot repair a string that has already been decoded incorrectly, which is exactly what mojibake is. The two compose cleanly:

• Use UTF.Unknown to detect and decode unknown bytes into a .NET string.
• Use Mojibake to repair an already-decoded string that came out garbled.

How mojibake happens

Text is stored as UTF-8 bytes. When a program reads those bytes but assumes they are Windows-1252 or Latin-1 (the most common single-byte encodings), every non-ASCII character comes apart into its individual UTF-8 bytes, each shown as its own Latin-1 character. The é in café (UTF-8 bytes C3 A9) becomes the two characters Ã and ©, so café reads as café. Smart quotes, dashes, and emoji break the same way, and text can be mangled two or three times over. Repair reverses the mistake: encode the broken string back to bytes with the wrong codec that produced it, then decode those bytes as UTF-8.

The safety contract: repair never makes text worse

The cardinal rule, inherited from ftfy, is that a repair only counts when it lowers the text's "badness" (a heuristic count of mojibake-signature character sequences). If no candidate repair strictly lowers the badness, the input is returned unchanged, and in fact the very same string reference is returned. That makes it safe to run Text.Fix over an entire dataset, including rows that are already clean:

string[] rows = { "café", "élan", "naïve", "menu", "über" };
foreach (string row in rows)
{
    string cleaned = Text.Fix(row);
    // "café" and "naïve" and "menu" come back untouched;
    // "élan" becomes "élan"; "über" becomes "über".
}

What it fixes

The default (FixOptions.Default) runs the safe, high-value pair:

The encoding repair itself is the full ftfy pipeline: the sloppy Windows-1252, 1251, 1250, 1253, 1254, and 1257 codecs, Latin-1, ISO-8859-2, MacRoman and CP437, plus restore_byte_a0, replace_lossy_sequences, decode_inconsistent_utf8, the utf-8-variants (CESU-8) decoder, and a fixed-point loop for multiply-encoded text.

string input = "café";

var options = new FixOptions { RemoveTerminalEscapes = true, NormalizeNfc = true };
string cleaned = Text.Fix(input, options);

// Or repair the encoding and nothing else:
string repaired = Text.Fix(input, FixOptions.EncodingOnly);

Auditing with FixAndExplain

FixAndExplain returns the repaired text plus the exact plan that produced it, mirroring ftfy's explain feature. Each step names a codec or a fixer.

FixResult result = Text.FixAndExplain("étude");

// result.Text            == "étude"
// result.Changed         == true
// result.OriginalBadness  > result.FixedBadness
// result.Steps           == [ (encode, latin-1), (decode, utf-8) ]

Detecting before fixing

Score text or flag suspicious rows without changing anything.

bool looksBroken = Text.LooksLikeMojibake("étude");   // true
double score     = Text.Badness("étude");             // greater than 0

string[] rows = { "café", "élan", "naïve" };
var suspicious = rows.Where(Text.LooksLikeMojibake).ToList();   // [ "élan" ]

Input handling

• null input throws ArgumentNullException.
• Empty input returns empty.
• No method ever throws on its text input, and repair never returns text worse than the input.
• Every method is stateless and thread-safe.

Performance

Repairing a typical short string takes on the order of ten microseconds (roughly 100,000 strings a second on a mixed corpus), and scoring badness a couple of microseconds. The heuristic regexes are linear in practice, with no catastrophic backtracking: a 200,000-character adversarial string of repeated mojibake repairs in about 0.2 seconds, and a 10 MB input in a few seconds. This is roughly four times slower than ftfy on small pathological inputs, which is not a concern at these speeds.

Scope

In scope for 0.1.0 is the encoding-repair core of ftfy plus the C1 control fixer by default, with a few adjacent fixers available as opt-in switches. Out of scope, and left for a later release, are the stylistic ftfy fixers (uncurl quotes, fix character width), first-class Unicode normalization beyond opt-in NFC, and the ftfy guess_bytes fallback for raw bytes of unknown encoding. Mojibake is not a charset detector for raw bytes; for that, reach for UTF.Unknown, then repair the result here.

Roadmap

• More of ftfy's fixers (uncurl quotes, fix character width).
• NFC and other normalization forms as first-class options.
• A guess-bytes helper for the raw-bytes case.

License

MIT. Copyright (c) 2026 Israel Iyonsi.