且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

python使用pyinstaller将scrapy转换为exe文件

更新时间:2023-01-18 14:34:56

我遇到了同样的情况.
与其试图让 pyinstaller 计算这个文件(我所有的尝试都失败了),我决定检查并更改部分 scrapy 代码以避免此错误.

I had the same situation.
Instead of trying to make pyinstaller count this file (I failed all my attempts to do it) I decided to check and change some part of scrapy code in order to avoid this error.

我注意到只有一个地方 \scrapy\VERSION 使用的文件-- \scrapy\__init__.py
我决定通过更改 scrapy__init__.py 从 scrapy\version 硬编码该值:

I noticed that there is only one place where \scrapy\VERSION file used-- \scrapy\__init__.py
I decided to hardcode that value from scrapy\version by changing scrapy__init__.py :

#import pkgutil
__version__ = "1.5.0" #pkgutil.get_data(__package__, 'VERSION').decode('ascii').strip()
version_info = tuple(int(v) if v.isdigit() else v
                     for v in __version__.split('.'))
#del pkgutil

此更改后,无需将版本存储在外部文件中.由于没有对 \scrapy\version 文件的引用 - 该错误不会发生.

After this change there is no need to store version in external file. As there is no reference to \scrapy\version file - that error will not occure.

在那之后,我遇到了与 FileNotFoundError: [Errno 2]rel="nofollow noreferrer">\scrapy\mime.types 文件.
\scrapy\mime.types 也有同样的情况——它只用于 \scrapy\responsetypes.py

After that I had the same FileNotFoundError: [Errno 2] with \scrapy\mime.types file.
There is the same situation with \scrapy\mime.types - it used only in \scrapy\responsetypes.py

...
#from pkgutil import get_data
...
    def __init__(self):
        self.classes = {}
        self.mimetypes = MimeTypes()
        #mimedata = get_data('scrapy', 'mime.types').decode('utf8')
        mimedata = """
        Copypaste all 750 lines of \scrapy\mime.types here
"""
        self.mimetypes.readfp(StringIO(mimedata))
        for mimetype, cls in six.iteritems(self.CLASSES):
            self.classes[mimetype] = load_object(cls)

此更改通过 \scrapy\mime.types 文件解决了 FileNotFoundError: [Errno 2].我同意将 750 行文本硬编码到 Python 代码中并不是***的决定.

This change resolved FileNotFoundError: [Errno 2] with \scrapy\mime.types file. I agree that hardcode 750 lines of text into python code is not the best decision.

之后我开始收到 ModuleNotFoundError: No module named scrapy.spiderloader .我将 "scrapy.spiderloader" 添加到 pyinstaller 的隐藏导入参数中.
下一个问题ModuleNotFoundError:没有名为scrapy.statscollectors的模块.
用于我的 scrapy 脚本的 pyinstaller 命令的最终版本包含 46 个隐藏的导入 - 之后我收到了可用的 .exe 文件.

After that I started to recieve ModuleNotFoundError: No module named scrapy.spiderloader . I added "scrapy.spiderloader" into hidden imports parameter of pyinstaller.
Next Issue ModuleNotFoundError: No module named scrapy.statscollectors.
Final version of pyinstaller command for my scrapy script consist of 46 hidden imports - after that I received working .exe file.