更新时间:2022-06-22 06:59:01
您可以将工作放入线程池中.您的 resolve_dns
连续执行3个请求,因此我创建了一个更通用的工作程序,该工作程序仅执行1个查询,并使用 collections.product
生成所有组合.在线程池中,我将chunksize设置为1以减少线程池批处理,如果某些查询花费很长时间,这会增加执行时间.
You can put the work into a thread pool. Your resolve_dns
does 3 requests serially so I created a slightly more generic worker that does only 1 query and used collections.product
to generate all combinations. In the thread pool I set chunksize to 1 to reduce thread pool batching, which can increase exec time if some queries take a long time.
import dns
from dns import resolver
import itertools
import collections
import multiprocessing.pool
def worker(arg):
"""query dns for (hostname, qname) and return (qname, [rdata,...])"""
try:
url, qname = arg
rdatalist = [rdata for rdata in resolver.query(url, qname)]
return qname, rdatalist
except dns.exception.DNSException, e:
return qname, []
def resolve_dns(url_list):
"""Given a list of hosts, return dict that maps qname to
returned rdata records.
"""
response_dict = collections.defaultdict(list)
# create pool for querys but cap max number of threads
pool = multiprocessing.pool.ThreadPool(processes=min(len(url_list)*3, 60))
# run for all combinations of hosts and qnames
for qname, rdatalist in pool.imap(
worker,
itertools.product(url_list, ('CNAME', 'MX', 'NS')),
chunksize=1):
response_dict[qname].extend(rdatalist)
pool.close()
return response_dict
url_list = ['example.com', '***.com']
result = resolve_dns(url_list)
for qname, rdatalist in result.items():
print qname
for rdata in rdatalist:
print ' ', rdata