View source for Troubleshooting Encodings: Mojibake, Detection & Conversion Tools

= Character Encoding in Practice =
Character encoding defines how text is represented as bytes in memory or files.  
Understanding it is crucial for preventing data corruption, display errors, and security issues.

== Why encoding matters ==
Every text you see on a screen — whether a document, website, or code — is stored as bytes.  
When software reads text, it must interpret those bytes using the **correct encoding**.  
If it uses the wrong one, characters appear garbled or replaced with “�” symbols.

Example of mismatch:
```

Expected UTF-8: Zürich
Read as Latin-1: ZÃ¼rich

```
This happens because bytes representing “ü” in UTF-8 (0xC3 0xBC) are misread as two Latin-1 characters.

== Common encodings ==
{| class="wikitable"
! Encoding !! Description !! Typical Use
|-
| **ASCII** || 7-bit, 128 characters (A–Z, digits, punctuation) || English text, legacy systems
|-
| **ISO-8859-1 (Latin-1)** || 8-bit, supports Western European languages || Older web pages, documents
|-
| **CP1252 (Windows Latin-1)** || Microsoft’s extension of Latin-1 || Windows software and files
|-
| **UTF-8** || Variable-length (1–4 bytes), covers all Unicode || Modern web, Linux, programming
|-
| **UTF-16** || 16-bit (2 or 4 bytes), used internally by some runtimes || Windows API, Java, C#, Qt
|-
| **UTF-32** || Fixed-width, 4 bytes per character || Simple but space-inefficient
|}

== Detecting encodings ==
There is no reliable automatic way to detect a file’s encoding.  
Heuristics can help, but files should explicitly declare their encoding:

* HTML: `<meta charset="UTF-8">`
* XML: `<?xml version="1.0" encoding="UTF-8"?>`
* JSON: assumed UTF-8 by standard
* Programming source files: often declared via comments or file headers

== Common errors ==
* **Mojibake:** Garbled text from mismatched encoding/decoding.  
* **Hidden byte order mark (BOM):** Some editors add a BOM, which may break scripts.  
* **Mixed encoding:** Files containing UTF-8 mixed with legacy characters.  
* **Locale mismatch:** Programs assuming system locale encodings instead of UTF-8.

== Security implications ==
Encoding errors can introduce vulnerabilities:
* **Injection attacks:** Misinterpreted byte sequences may bypass sanitization.
* **Filename spoofing:** Homoglyph characters (like Cyrillic “а” vs Latin “a”) can disguise malicious files.
* **Protocol mismatch:** Data transmitted with the wrong encoding may cause corruption or code execution flaws.

== Best practices ==
* Always use **UTF-8** unless there’s a specific reason not to.  
* Avoid legacy code pages (like ISO-8859 or CP1252).  
* Declare the encoding in file headers or metadata.  
* Validate or sanitize user input before processing.  
* Use Unicode-aware libraries and APIs.

== Example: viewing encoding in files ==
On Unix-like systems:
```

file -i filename

```
Output example:
```

text/plain; charset=utf-8

```

To convert encodings safely:
```

iconv -f ISO-8859-1 -t UTF-8 input.txt -o output.txt

```

== Summary ==
* Encodings map text to binary and back.  
* Wrong encodings cause garbled output and potential data loss.  
* UTF-8 is the universal, backward-compatible standard.  
* Always specify, check, and validate text encodings in files and applications.

[[Category:Text]]