且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Perl Unicode内部-与utf8混淆

更新时间:2023-09-11 21:36:04

is_utf8返回有关使用哪种内部存储格式(句点)的信息.

is_utf8 returns information about which internal storage format was used, period.

  • 它与字符串的值无关(尽管某些字符串只能以两种格式之一存储).
  • 与字符串是否已解码无关.
  • 与字符串是否包含使用UTF-8编码的内容无关.
  • 这不是任何形式的有效性检查.

现在继续您的问题.

整个utf8编译指示对我来说还是个谜.
The whole utf8 pragma is a mystery for me.

use utf8;告诉perl您的源代码是使用UTF-8编码的.如果您不这样说,perl实际上会假定它是iso-8859-1(作为内部机制的副作用).

use utf8; tells perl your source code is encoded using UTF-8. If you don't tell it so, perl effectively assumes it's iso-8859-1 (as a side-effect of internal mechanisms).

utf8 ::名称空间中的功能与实用程序无关,它们具有多种用途.

The functions in the utf8:: namespace are unrelated to the pragma, and they serve a variety of purposes.

  • utf8::encodeutf8::decode:有用的编码和解码功能.与Encode的encode_utf8decode_utf8类似,但是它们就地工作.
  • utf8::upgradeutf8::downgrade:很少使用,但对于解决XS模块中的错误很有用.详情请见下文.
  • utf8::is_utf8:我不知道为什么有人会使用它.
  • utf8::encode and utf8::decode: Useful encoding and decoding functions. Similar to Encode's encode_utf8 and decode_utf8, but they work in-place.
  • utf8::upgrade and utf8::downgrade: Rarely used, but useful for working around bugs in XS modules. More on this below.
  • utf8::is_utf8: I don't know why someone would ever use that.
我怎么能保证(测试)比任何$ other_data都包含有效的unicode字符串?
HOW i can ensure (test it), than any $other_data contains valid unicode string?

有效Unicode字符串"对您意味着什么? Unicode在不同情况下具有不同的有效定义.

What does "valid Unicode string" mean to you? Unicode has different definitions of valid for different circumstances.

utf8 :: is_utf8($ data)有什么用途?
for what purpose is the utf8::is_utf8($data)?

调试.它会窥视Perl胆量.

Debugging. It peeks at Perl guts.

在上面的示例中utf8 :: is_utf8($ data)将显示OK-但不理解为什么.
In the above example utf8::is_utf8($data) will print OK - but don't understand WHY.

因为NFD恰好选择返回包含UTF8 = 1格式字符串的标量.

Because NFD happens to have chosen to return a scalar containing a string in the UTF8=1 format.

Perl有两种​​存储字符串的格式:

Perl has two formats for storing strings:

  • UTF8 = 0可以存储8位值的序列.
  • UTF8 = 1可以存储72位值的序列(尽管实际上限制为32位或64位).

第一种格式使用较少的内存,并且在访问字符串中的特定位置时速度更快,但是它所包含的内容受到限制. (例如,它不能存储Unicode代码点,因为它们需要21位.)Perl可以在两者之间***切换.

The first format uses less memory and is faster when it comes to access a specific position in the string, but it's limited in what it can contain. (For example, it can't store Unicode code points since they require 21 bits.) Perl can freely switch between the two.

use utf8;
use feature qw( say );

my $d = my $u = "abcdé";
utf8::downgrade($d);  # Switch to using the UTF8=0 format for $d.
utf8::upgrade($u);    # Switch to using the UTF8=1 format for $u.

say utf8::is_utf8($d) ?1:0;   # 0
say utf8::is_utf8($u) ?1:0;   # 1
say $d eq $u          ?1:0;   # 1

通常不必担心这一点,但是有错误的模块.尽管use feature qw( unicode_strings );,甚至还有Perl的错误角落.可以使用utf8::upgradeutf8::downgrade将标量的格式更改为XS函数期望的格式.

One normally doesn't have to worry about this, but there are buggy modules. There are even buggy corners of Perl remaining despite use feature qw( unicode_strings );. One can use utf8::upgrade and utf8::downgrade for changing the format of a scalar to that expected by the XS function.

或者它的名字不正确,该函数应命名为uni :: is_unicode($ data)???
Or it is miss-named and the function should be named as uni::is_unicode($data)???

那再好不过了. Perl无法知道字符串是否为Unicode字符串.如果您需要跟踪它,那么您需要自己对其进行跟踪.

That's no better. Perl has no way to know whether a string is a Unicode string or not. If you need to track that, you need to track it yourself.

UTF8 = 0格式的字符串可能包含Unicode代码点.

Strings in the UTF8=0 format may contain Unicode code points.

my $s = "abc";  # U+0041,0042,0043

UTF8 = 1格式的字符串可能包含不是Unicode代码点的值.

Strings in the UTF8=1 format may contain values that aren't Unicode code points.

my $s = pack('W*', @temperature_measurements);