Skip to content

Latest commit

 

History

History
13 lines (8 loc) · 830 Bytes

hdfs_optimization.md

File metadata and controls

13 lines (8 loc) · 830 Bytes

Some tips to improve HDFS performance

  1. set noatime and nodiratime attributes when mount disk.
  • noatime : do not update inode access time in this file system.
  • nodiratime : do not update directory access time in this file system.

  1. set the prefetched data-buffer size can significantly improve the performance of sequential data accessing. The default parameter is 256 sectors，we can increase it to 1024/2408 sectors. We have to write this setting to /etc/rc.local，otherwise will be invalid when rebooting.

     /sbin/blockdev --setra 2048 /dev/sda

  2. Decrease the preserved blocks in ext4 file system. By doing this we can give HDFS more storing space. generally，The default setting is 5%. It can be decreased to 1%.

     sudo tune2fs -m 1 /dev/sdb1