RFC 4627 specifies a method to determine an encoding (one of UTF-8, UTF-16(BE|LE) or UTF-32(BE|LE)) of encoded JSON text. The proposed preliminary patch (it doesn't include the documentation yet) allows load() and loads() functions accept bytes data when it is encoded with standard Unicode encoding. Also accepted data with BOM (this doesn't specified in RFC 4627, but is widely used).

There is only one case where the method can give a misfire. Serialized string "\x00..." encoded in UTF-16LE may be erroneously detected as encoded in UTF-32LE. This case violates the two rules of RFC 4627: the string was serialized instead of a an object or an array, and the control character U+0000 was not escaped. The standard encoded JSON always detected correctly.

This patch requires "surrogatepass" error handler for utf-16/32 (see issue12892 and issue13916).

All dependencies for this issue are resolved now.

Here is updated patch, synchronized with tip.

You'll need to also update the "Character Encodings" subsection of the json docs.

Both json standard (ECMA-404) [1] and the new json rfc 7159 [2] do not mention
the encoding detection.

[1] http://www.ecma-international.org/publications/files/ECMA-ST/ECMA-404.pdf
[2] https://tools.ietf.org/html/rfc7159#section-8.1

From the rfc:

> JSON text SHALL be encoded in UTF-8, UTF-16, or UTF-32.  The default
  encoding is UTF-8, and JSON texts that are encoded in UTF-8 are
  interoperable in the sense that they will be read successfully by the
  maximum number of implementations; there are many implementations
  that cannot successfully read texts in other encodings (such as
  UTF-16 and UTF-32).

  Implementations MUST NOT add a byte order mark to the beginning of a
  JSON text.  In the interests of interoperability, implementations
  that parse JSON texts MAY ignore the presence of a byte order mark
  rather than treating it as an error.

I agree that the state of encoding detection in the new RFC seems unclear, given that the old RFC prefaced the part about the encoding detection with:
> Since the first two characters of a JSON text will always be ASCII
> characters

But in the new RFC:
> Appendix A.  Changes from RFC 4627
[...]
>    o  Changed the definition of "JSON text" so that it can be any JSON
>       value, removing the constraint that it be an object or array.

Thus,
> "ಠ_ಠ"
whose 2nd character is decidedly non-ASCII, is now a valid JSON text (i.e. standalone JSON document).

There seems to have been a thread about encoding detection in the RFC 7159 working group, but I don't have the time to read through it all:

> Re: [Json] JSON: remove gap between Ecma-404 and IETF draft
> http://www.ietf.org/mail-archive/web/json/current/msg01936.html

It eventually leads to a dedicated sub-thread:

> [Json] Encoding detection (Was: Re: JSON: remove gap between Ecma-404 and IETF draft)
> http://www.ietf.org/mail-archive/web/json/current/msg01959.html

If you adjusted the detect_encoding() logic according to Pete Cordell’s table at the bottom of <http://www.ietf.org/mail-archive/web/json/current/msg01959.html>, it might work for standalone strings.

However since the RFC encourages UTF-8 for best interoperability, I wonder if any of this autodetection is necessary. It might be simpler to just assume UTF-8, or use the “utf-8-sig” codec. Or are there real cases where detecting UTF-16 or -32 would be useful?

Hi Serhiy,

I have reviewed your patch, it seems to be ok.

Having hit the json.loads() problem recently when porting a project to Python 3, I'm keen to see this land for 3.6.

Accodingly, assigning to myself to review and merge Serhiy's patch - if it proves necessary, we can tweak the details of the encoding detection during beta.

New changeset e9e1bf9ec2ac by Nick Coghlan in branch 'default':
Issue #17909: Accept binary input in json.loads
https://hg.python.org/cpython/rev/e9e1bf9ec2ac

Thanks for tackling this Serhiy!

I removed issue 13916 from the dependency list, as while that's a reasonable suggestion, I don't think this fix is conditional on that change.