更新时间:2023-02-02 23:29:20
我公司已经将一些结构化流量信息直接记录到MySQL日志数据库中。此数据库将下游复制到另一个数据库。所有分析都运行于最终数据库复制。我们的网站维持相当多的流量。到目前为止,它似乎没有任何大的问题。然而,我们的IT部门对当前设置的可扩展性有一些日益增长的关注,并建议我们将日志信息卸载到正确日志文件。然后,日志文件将重新插入到相同的下游数据库表中。这使我想到这个问题。 :)
My company have been logging some structured traffic info straight into a MySQL log database. This database is replicated downstream to another database. All analytics run off the final database replication. Our site sustain quite a bit of traffic. So far, it doesn't seem to have any major problems. However, our IT department have some growing concerns regarding to the scalability of the current setup and is suggesting that we offload the log info onto "proper" log-files. The log-files will then be reinserted back into the same downstream database tables. Which brings me to this question. :)
这里有一些关于日志文件vs log-db(关系)主题的优点和缺点:
Here are some of pros and cons that I see regarding to the subject of log-files vs log-db (relational):
我认为需要对日志数据库进行一些压力测试我的情况。这样至少我知道有多少余量。
I think some stress tests on the log database are needed in my situation. This way at least I know how much headroom I have.
最近,我一直在研究一些基于键值/文档的数据库,如Redis,Tokyo Cabinet和MongoDB。这些快速插入数据库可能是***点,因为它们提供持久性,高(写)吞吐量和不同程度的查询能力。他们可以使数据提取过程比通过日志文件的分析解析和减少地图更简单。
Recently, I've been looking into some key-value / document-based databases like Redis, Tokyo Cabinet, and MongoDB. These fast inserting databases can potentially be the sweet spot since they provide persistence, high (write) throughputs, and querying capabilities to varying degrees. They can make the data-extraction process much simpler than parsing and map-reducing through gigs of log files.
从长远来看,我相信拥有强大的分析数据仓库至关重要。从分析数据中释放应用程序数据,反之亦然可以是一个大赢家。
In the long run, I believe it is crucial to have a robust analytics data warehouse. Freeing application data from analytic data and vice versa can be a big WIN.
最后,我想指出,***上有很多类似/
Lastly, I would just like to point out there are many similar / closely related questions here on *** in case you want to broaden your discussion.
编辑:
rsyslog 看起来很有趣。它使您能够直接写入MySQL。如果你使用Ruby,你应该看看logging gem。它提供多目标日志记录功能。这太好了。
rsyslog looks very interesting. It gives you the ability to write directly to MySQL. If you are using Ruby, you should have a look at the logging gem. It provides multi-target logging capabilities. It's really nice.