Strings, bytes and Unicode conversions#

Passing Python strings to C++#

When a Python str is passed from Python to a C++ function that accepts std::string or char * as arguments, pybind11 will encode the Python string to UTF-8. All Python str can be encoded in UTF-8, so this operation does not fail.

The C++ language is encoding agnostic. It is the responsibility of the programmer to track encodings. It’s often easiest to simply use UTF-8 everywhere.

m.def("utf8_test",
    [](const std::string &s) {
        cout << "utf-8 is icing on the cake.\n";
        cout << s;
    }
);
m.def("utf8_charptr",
    [](const char *s) {
        cout << "My favorite food is\n";
        cout << s;
    }
);

>>> utf8_test("🎂")
utf-8 is icing on the cake.
🎂

>>> utf8_charptr("🍕")
My favorite food is
🍕

Note

Some terminal emulators do not support UTF-8 or emoji fonts and may not display the example above correctly.

The results are the same whether the C++ function accepts arguments by value or reference, and whether or not const is used.

Passing bytes to C++#

A Python bytes object will be passed to C++ functions that accept std::string or char* without conversion. In order to make a function only accept bytes (and not str), declare it as taking a py::bytes argument.

Returning C++ strings to Python#

When a C++ function returns a std::string or char* to a Python caller, pybind11 will assume that the string is valid UTF-8 and will decode it to a native Python str, using the same API as Python uses to perform bytes.decode('utf-8'). If this implicit conversion fails, pybind11 will raise a UnicodeDecodeError.

m.def("std_string_return",
    []() {
        return std::string("This string needs to be UTF-8 encoded");
    }
);

>>> isinstance(example.std_string_return(), str)
True

Because UTF-8 is inclusive of pure ASCII, there is never any issue with returning a pure ASCII string to Python. If there is any possibility that the string is not pure ASCII, it is necessary to ensure the encoding is valid UTF-8.

Warning

Implicit conversion assumes that a returned char * is null-terminated. If there is no null terminator a buffer overrun will occur.

Explicit conversions#

If some C++ code constructs a std::string that is not a UTF-8 string, one can perform a explicit conversion and return a py::str object. Explicit conversion has the same overhead as implicit conversion.

// This uses the Python C API to convert Latin-1 to Unicode
m.def("str_output",
    []() {
        std::string s = "Send your r\xe9sum\xe9 to Alice in HR"; // Latin-1
        py::handle py_s = PyUnicode_DecodeLatin1(s.data(), s.length(), nullptr);
        if (!py_s) {
            throw py::error_already_set();
        }
        return py::reinterpret_steal<py::str>(py_s);
    }
);

>>> str_output()
'Send your résumé to Alice in HR'

The Python C API provides several built-in codecs. Note that these all return new references, so use reinterpret_steal() when converting them to a str.

One could also use a third party encoding library such as libiconv to transcode to UTF-8.

Return C++ strings without conversion#

If the data in a C++ std::string does not represent text and should be returned to Python as bytes, then one can return the data as a py::bytes object.

m.def("return_bytes",
    []() {
        std::string s("\xba\xd0\xba\xd0");  // Not valid UTF-8
        return py::bytes(s);  // Return the data without transcoding
    }
);

>>> example.return_bytes()
b'\xba\xd0\xba\xd0'

Note the asymmetry: pybind11 will convert bytes to std::string without encoding, but cannot convert std::string back to bytes implicitly.

m.def("asymmetry",
    [](std::string s) {  // Accepts str or bytes from Python
        return s;  // Looks harmless, but implicitly converts to str
    }
);

>>> isinstance(example.asymmetry(b"have some bytes"), str)
True

>>> example.asymmetry(b"\xba\xd0\xba\xd0")  # invalid utf-8 as bytes
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xba in position 0: invalid start byte

Wide character strings#

When a Python str is passed to a C++ function expecting std::wstring, wchar_t*, std::u16string or std::u32string, the str will be encoded to UTF-16 or UTF-32 depending on how the C++ compiler implements each type, in the platform’s native endianness. When strings of these types are returned, they are assumed to contain valid UTF-16 or UTF-32, and will be decoded to Python str.

#define UNICODE
#include <windows.h>

m.def("set_window_text",
    [](HWND hwnd, std::wstring s) {
        // Call SetWindowText with null-terminated UTF-16 string
        ::SetWindowText(hwnd, s.c_str());
    }
);
m.def("get_window_text",
    [](HWND hwnd) {
        const int buffer_size = ::GetWindowTextLength(hwnd) + 1;
        auto buffer = std::make_unique< wchar_t[] >(buffer_size);

        ::GetWindowText(hwnd, buffer.data(), buffer_size);

        std::wstring text(buffer.get());

        // wstring will be converted to Python str
        return text;
    }
);

Strings in multibyte encodings such as Shift-JIS must transcoded to a UTF-8/16/32 before being returned to Python.

Character literals#

C++ functions that accept character literals as input will receive the first character of a Python str as their input. If the string is longer than one Unicode character, trailing characters will be ignored.

When a character literal is returned from C++ (such as a char or a wchar_t), it will be converted to a str that represents the single character.

m.def("pass_char", [](char c) { return c; });
m.def("pass_wchar", [](wchar_t w) { return w; });

>>> example.pass_char("A")
'A'

While C++ will cast integers to character types (char c = 0x65;), pybind11 does not convert Python integers to characters implicitly. The Python function chr() can be used to convert integers to characters.

>>> example.pass_char(0x65)
TypeError

>>> example.pass_char(chr(0x65))
'A'

If the desire is to work with an 8-bit integer, use int8_t or uint8_t as the argument type.

Grapheme clusters#

A single grapheme may be represented by two or more Unicode characters. For example ‘é’ is usually represented as U+00E9 but can also be expressed as the combining character sequence U+0065 U+0301 (that is, the letter ‘e’ followed by a combining acute accent). The combining character will be lost if the two-character sequence is passed as an argument, even though it renders as a single grapheme.

>>> example.pass_wchar("é")
'é'

>>> combining_e_acute = "e" + "\u0301"

>>> combining_e_acute
'é'

>>> combining_e_acute == "é"
False

>>> example.pass_wchar(combining_e_acute)
'e'

Normalizing combining characters before passing the character literal to C++ may resolve some of these issues:

>>> example.pass_wchar(unicodedata.normalize("NFC", combining_e_acute))
'é'

In some languages (Thai for example), there are graphemes that cannot be expressed as a single Unicode code point, so there is no way to capture them in a C++ character type.

C++17 string views#

C++17 string views are automatically supported when compiling in C++17 mode. They follow the same rules for encoding and decoding as the corresponding STL string type (for example, a std::u16string_view argument will be passed UTF-16-encoded data, and a returned std::string_view will be decoded as UTF-8).

A string view does not own its character data. When a view is loaded as an argument to a pybind11-bound function, pybind11 keeps the Python object that provides the data alive until the function returns. This also applies to views nested in automatically converted STL containers. The C++ function must not retain any such view after it returns unless it separately guarantees that the backing storage remains alive and valid.

Lifetime support keeps the Python object alive, but does not prevent its storage from being invalidated. For example, if C++ releases the GIL or calls back into Python, resizing a backing bytearray while the view is in use can invalidate the view.

A direct Python-to-C++ py::cast() made when no bound-function call is active has no such lifetime support. When a cast to a non-owning view succeeds, the caller must keep the backing Python object alive, with its storage unchanged, for as long as the view is used. For a container of views, this requirement applies to every element: retain the elements directly or through an unmodified owning container, and do not cast an iterable that creates temporary elements.

Some view conversions require temporary backing storage, for example to encode text. Outside a bound-function call, such conversions raise cast_error instead of returning a dangling view.

References#