且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

协议缓冲区和UTF-8

更新时间:2022-10-18 08:05:36

这可能是过度,但 ICU库会做你需要的一切,你可以在Windows和Linux上使用它们。



但是,如果你只想要转换,那么在Windows下,对 MultiByteToWideChar WideCharToMultiByte 的简单调用可以进行转换UTF-8和



UTF-16之间。例如:

  // utf-8至utf-16 
MultiByteToWideChar(CP_UTF8,0,myUtf8String,-1 ,
myUtf16Buf,lengthOfUtf16Buf);

对于Linux,libidn可以做你需要的。它可以在UTF-8和UCS之间转换,我认为在某个级别等同于UTF-32。例如:

  // utf-8 to UCS 
ucsStr = stringprep_utf8_to_ucs4(asdf,4,&项目);但是,在Linux中,我认为你可能***只是使用UTF-8工作。除非你有一个现有的库为UTF-16,我不知道有一个令人信服的理由在Linux中使用它。

The history of Encoding Schemes / multiple Operating Systems and Endian-nes have led to a mess in terms of encoding all forms of string data (--i.e., all alphabets); for this reason protocol buffers only deals with ASCII or UTF-8 in its string types, and I can't see any polymorphic overloads that accept the C++ wstring. The question then is how is one expected to get a UTF-16 string into a protocol buffer ?

Presumably I need to keep the data as a wstring in my application code and then perform a UTF-8 conversion before I stuff it into (or extract from) the message. What is the simplest - Windows/Linux portable way to do this (A single function call from a well-supported library would make my day) ?

Data will originate from various web-servers (Linux and windows) and will eventually ends up in SQL Server (and possibly other end points).

-- edit 1--

Mark Wilkins suggestion seems to fit the bill, perhaps someone who has experience with the library can post a code snippet -- from wstring to UTF-8 -- so that I can gauge how easy it will be.

-- edit 2 --

sth's suggestion even more so. I will investigate boost serialization further.

It may be overkill, but the ICU libraries will do everything you need and you can use them on both Windows and Linux.

However, if you are only wanting conversion, then under Windows, a simple call to MultiByteToWideChar and WideCharToMultiByte can do the conversion between UTF-8 and

UTF-16. For example:

// utf-8 to utf-16
MultiByteToWideChar( CP_UTF8, 0, myUtf8String, -1,
                     myUtf16Buf, lengthOfUtf16Buf );

With Linux, libidn might do what you need. It can convert between UTF-8 and UCS, which I think is equivalent to UTF-32 at some level. For example:

// utf-8 to UCS
ucsStr = stringprep_utf8_to_ucs4( "asdf", 4, &items );

However, in Linux I think you might be best simply working with UTF-8. Unless you have an existing library for UTF-16, I am not sure there is a compelling reason to use it in Linux.