且构网

分享程序员开发的那些事...
且构网 - 分享程序员编程开发的那些事

Hadoop Namenode元数据-fsimage和编辑日志

更新时间:2023-11-17 09:02:52

答案是通过查看编辑日志中的信息.如果编辑日志中没有可用信息,则当我们将新文件写入hdfs时,此问题在用例中正确.当您的namenode运行时,如果删除fsimage文件并尝试读取hdfs文件,则该文件可以读取.

Answer is by looking at information in the edit logs. If information is not available in the edit logs This question stands true for use-case when we write the new file to hdfs. While your namenode is running if you remove fsimage file and try to read the hdfs file it is able to read.

从正在运行的namenode中删除fsimage文件不会导致读/写操作出现问题.当我们重新启动namenode时,会出现错误,指出找不到图像文件.

Removing the fsimage file from the running namenode will not cause issue with the read / write operations. When we restart the namenode, there will be errors stating that image file is not found.

让我尝试提供更多解释以帮助您.

Let me try to give some more explanation to help you out.

仅在启动时hadoop会查找fsimage文件,如果该文件不存在,则不会出现namenode并记录格式化namenode的日志.

Only on start up hadoop looks fsimage file, in case if it is not there, namenode does not come up and log for formatting the namenode.

hadoop format -namenode命令创建fsimage文件(如果存在编辑日志).从编辑日志中获取namenode启动文件元数据之后(如果未找到,则通过fsimage文件搜索编辑日志中的信息).所以fsimage只是作为上次保存信息的检查点.这也是辅助节点与编辑日志保持同步(1小时/1百万次事务后)的原因之一,因此从最后一个检查点启动时,不需要太多同步.

hadoop format -namenode command creates fsimage file (if edit logs are present). After namenode startup file metadata is fetched from edit logs (and if not found information in edit logs searched thru fsimage file). so fsimage just works as checkpoint where inforamtion is saved last time. This is also one of the reason secondary node keeps on sync (after 1 hour / 1 milliion transactions) from edit logs so that on start up from last checkpoint not much needs to be synced.

如果您打开安全模式(命令:hdfs dfsadmin -safemode enter)并使用saveNamespace(命令:hdfs dfsadmin -saveNamespace),它将显示以下提到的日志消息.

2014-07-05 15:03:13,195 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file /data/hadoop-namenode-data-temp/current/fsimage.ckpt_0000000000000000169 using no compression
2014-07-05 15:03:13,205 INFO org.apache.hadoop.hdfs.server.namenode.FSImage: Image file /data/hadoop-namenode-data-temp/current/fsimage.ckpt_0000000000000000169 of size 288 bytes saved in 0 seconds.
2014-07-05 15:03:13,213 INFO org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager: Going to retain 2 images with txid >= 0
2014-07-05 15:03:13,237 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 170