Managing the size of Hadoop DataNode using LVM

In this article, we will :
🔅Integrate LVM with Hadoop and
provide Elasticity to DataNode Storage
🔅Increase or Decrease the Size of Static
Partition in Linux.
🔅Automating LVM Partition using Python-Script.

Let’s start:

Step -1: Launching 2 instances one for Hadoop master(Name node) and the other for Hadoop slave(Data node).

Step-2: Download and install JDK n Hadoop softwares in both the nodes:

jdk-8u171-linux-x64.rpm

hadoop-1.2.1–1.x86_64.rpm

Since I already have this softwares on my local hard disk I will copy them using WinSCP tool.

To install we will use the following commands:

yum install jdk-8u171-linux-x64.rpm -y

rpm -ivh hadoop-1.2.1–1.x86_64.rpm — force -y

Step-3: Configuring name node n data node by updating hdfs-site.xml and core-site.xml

Name node hdfs-site.xml file

Name node core-site.xml file(IP: 0.0.0.0 n port 9001)

data node hdfs-site.xml

data node core_site.xml(ip: master IP(public IP of name node instance) n port 9001)

Step-4: Format the namenode

Step-5: Create 2 EBS volumes and attach them to slave (data node)

Step-6: Create Physical volumes, logical volume, volume group for these 2 volumes so that we can use them.

If pvcreate cmd is not available then download it

Creating physical volumes

Create Volume group:

Creating Logical Volume:

Step-7: Format the partition

Step-8: Mount the partition to data node folder

Step-9: Start the name node n data node services

Checking the size of the data node

Step- 10: Extending Logical Volume size to increase the data node space

Step-11: Now again checking the report to see if the size is increased or not

The size is now increased from 3Gib to 6Gib

Step-12: Python script for LVM

Here is the link for LVM Script

Task Done!!

Thankyou! keep learning! keep growing! keep sharing!

Krithika Sharma
If you enjoyed this, follow me on Medium for more
Let’s connect on LinkedIn