Managing the size of Hadoop DataNode using LVM
In this article, we will :
🔅Integrate LVM with Hadoop and
provide Elasticity to DataNode Storage
🔅Increase or Decrease the Size of Static
Partition in Linux.
🔅Automating LVM Partition using Python-Script.
Step -1: Launching 2 instances one for Hadoop master(Name node) and the other for Hadoop slave(Data node).
Step-2: Download and install JDK n Hadoop softwares in both the nodes:
Since I already have this softwares on my local hard disk I will copy them using WinSCP tool.
To install we will use the following commands:
yum install jdk-8u171-linux-x64.rpm -y
rpm -ivh hadoop-1.2.1–1.x86_64.rpm — force -y
Step-3: Configuring name node n data node by updating hdfs-site.xml and core-site.xml
Step-4: Format the namenode
Step-5: Create 2 EBS volumes and attach them to slave (data node)
Step-6: Create Physical volumes, logical volume, volume group for these 2 volumes so that we can use them.
If pvcreate cmd is not available then download it
Creating physical volumes
Create Volume group:
Creating Logical Volume:
Step-7: Format the partition
Step-8: Mount the partition to data node folder
Step-9: Start the name node n data node services
Checking the size of the data node
Step- 10: Extending Logical Volume size to increase the data node space
Step-11: Now again checking the report to see if the size is increased or not
The size is now increased from 3Gib to 6Gib
Step-12: Python script for LVM
Here is the link for LVM Script
Thankyou! keep learning! keep growing! keep sharing!