更新时间:2023-02-18 16:59:32
如果您的文件位于 cp1252
和 utf-8
中,
import logging
def force_decode(string,codecs = ['utf8',' cp1252']):
用于编解码器中的i:
尝试:
返回string.decode(i)
,UnicodeDecodeError除外:
传递
logging.warn(无法解码url%s%([字符串]))
表示os.listdir(rootPath)中的项目:
#转换为Unicode
isinstance(item,str):
项目= force_decode(item)
打印项目
https://pypi.python.org/pypi/chardet
I've got about 1000 filenames read by os.listdir()
, some of them are encoded in UTF8 and some are CP1252.
I want to decode all of them to Unicode for further processing in my script. Is there a way to get the source encoding to correctly decode into Unicode?
Example:
for item in os.listdir(rootPath):
#Convert to Unicode
if isinstance(item, str):
item = item.decode('cp1252') # or item = item.decode('utf-8')
print item
if your files either in cp1252
and utf-8
, then there is an easy way.
import logging
def force_decode(string, codecs=['utf8', 'cp1252']):
for i in codecs:
try:
return string.decode(i)
except UnicodeDecodeError:
pass
logging.warn("cannot decode url %s" % ([string]))
for item in os.listdir(rootPath):
#Convert to Unicode
if isinstance(item, str):
item = force_decode(item)
print item
otherwise, there is a charset detect lib.