Common encoding problems in APIs

August 14, 2026 · 15 min read

Production APIs fail in predictable ways: text garbled as mojibake, binary fields double-Base64-encoded, clients sending Latin-1 while servers assume UTF-8, and gateways rewriting bodies. Most encoding bugs are integration issues, not crypto failures - and they show up in support tickets long after launch.

This article catalogs the patterns we see most often when teams ship JSON REST, webhooks, and multipart uploads, with fixes you can apply in specs, tests, and observability.

Double encoding

Double encoding happens when a client Base64-encodes data the server already stores as Base64, or when middleware URL-encodes an already-encoded query parameter. Symptoms include strings that decode to another Base64 string instead of JSON, and signatures that never match because the verifier hashes an extra layer.

Fix: document one canonical representation per field; reject inputs that match a Base64 regex when you expect plain JSON; add contract tests that round-trip samples from mobile, web, and partner SDKs.

Charset mismatches

HTTP defaults to ISO-8859-1 in older specs unless charset=utf-8 is set. Modern stacks assume UTF-8, but CSV imports, legacy banks, and mainframe bridges still ship other encodings. Always set Content-Type: application/json; charset=utf-8 on responses and validate incoming charset declarations.

POST /v1/notes HTTP/1.1
Content-Type: application/json; charset=utf-8

{"title":"Résumé","body":"…"}

Base64 in JSON

JSON has no binary type - teams embed Base64 strings for images, PDFs, and protobuf payloads. Problems arise when clients forget contentType, servers decode without size limits, or analytics pipelines treat Base64 blobs as human text. Prefer presigned object storage for large files; reserve inline Base64 for small attachments under a documented size cap.

{
  "fileName": "report.pdf",
  "contentType": "application/pdf",
  "encoding": "base64",
  "data": "JVBERi0xLjQKJeLjz9MK..."
}

Content-Type traps

Sending JSON with text/plain, or form data with wrong boundaries, causes frameworks to parse bodies differently than Postman tests. Multipart parts need per-part Content-Type; nested JSON as a form field is still a string - escape and charset rules apply.

Webhooks that verify HMAC over raw bodies break if proxies pretty-print JSON. Sign the exact bytes received; store raw body buffers before parsing when possible.

Prevention checklist

Publish OpenAPI with format: byte or explicit Base64 fields and examples.
Golden-file tests for non-ASCII, emoji, and empty string edge cases.
Log encoding failures with field names, not full payloads.
Provide local decode tools for partners to validate without hitting production.

When debugging live incidents, compare hex of server-received bytes vs client-sent bytes at the boundary - charset issues become obvious before debating business logic.

FAQ

Should APIs accept multiple encodings per field?: Avoid it. One canonical encoding per field reduces client bugs. Version the API if you must change representation.
Why does JSON look fine in Postman but break in Python?: Often missing response.encoding, assuming ASCII, or reading bytes without UTF-8. Set encoding explicitly on both sides.
Is gzip the same problem as Base64?: Different layer - compression vs binary-to-text. Confusion happens when gzip bytes are Base64-wrapped twice in JSON.
How do I test encoding in CI?: Fixture files with UTF-8, combining characters, and emoji; assert round-trip equality after HTTP client parse and server handler.

Related: UTF-8 vs Base64 explained

Browse all tools