LVM Task : Exploring More about LVM…

Kaustubh Chaudhari

5 min readMar 15, 2021

Hello Guys,

Today we are going to Explore LVM in different ways and implement following tasks using LVM :

Setup Hadoop Cluster with One Master and One Slave Node.
Creating LVM steup for Datanode.
Increase and Decrease Size of LVM partition.
Automation Script in Python for LVM Automation.

Before starting with our Task let us get some Vocabulary related to Task.

Why we require LVM ?

We by default use Static Partition which is the Older way. Using Static Partition Our Volume size is fixed. If we have to change the size of Static Partition ,We can do it but, the process is quiet complicated. We have to unmount folder, format the Partition and then finally we can do it. There is a Concept called as LVM(Logical Volume Management) which enable you to change the size of Partition dynamically.

Terminologies related to LVM :

Physical Volume(PV): A physical volume is a collection of disk partitions used to store all server data.
Volume Group(VG): A volume group ( VG ) is the central unit of the Logical Volume Manager (LVM) architecture. It is what we create when we combine multiple physical volumes to create a single storage structure, equal to the storage capacity of the combined physical devices

3. Logical Volume(LV): Logical volumes are groups of information located on physical volumes. A hierarchy of structures is used to manage disk storage. Each individual disk drive, called a physical volume (PV) has a name, such as /dev/hdisk0. Every physical volume in use belongs to a volume group (VG)

4. Hadoop : Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.

Let’s Start…

Setup Hadoop Cluster with One Master and One Slave Node :

I am using AWS Cloud to Launch 2 VM’s, One for Hadoop Master and other for Hadoop Slave. Download Jdk and Hadoop in both the VM’s .

Next Steps to Configure Master :

Install Jdk : rpm -i <jdk software name>
Install Hadoop : rpm -i <hadoop software name> — force
Make one folder : mkdir /data1
Go to folder “/etc/hadoop” : cd /etc/hadoop
Make changes in “Hdfs-site.xml” and write : vim hdfs-site.xml

Make changes in “Core-site.xml” and write : vim core-site.xml

Format Master node : hadoop namenode -format

Start Namenode services : hadoop-daemon.sh start namenode
Run jps to check if Master is running.

Steps to setup Slave Node :

Install Jdk : rpm -i <jdk software name>
Install Hadoop : rpm -i <hadoop software name> — force
Make one folder : mkdir /slave
Go to folder “/etc/hadoop” : cd /etc/hadoop
Make changes in “Hdfs-site.xml” and write : vim hdfs-site.xml

Make changes in “Core-site.xml” and write : vim core-site.xml

Start Datanode services : hadoop-daemon.sh start datanode
Run jps to check if Datanode is running.

Check Cluster is connected : hadoop dfsadmin report

Our Hadoop Cluster is Set up Now…

2. Creating LVM steup for Datanode :

For this firstly we have to Create and Attach EBS Volumes to Datanode OS.

Now Install LVM in Datanode : yum install lvm2

Check how many disks are attached : fdisk -l

Choose two disks to create a physical Volume.(xvdf and xvdg) : pvcreate <Disk name>

Create Volume Group : vgcreate <vg name>

Create Logical Volume : lvcreate — size <size> — name <lv name> <vg name>

Format the Volume Group : mkfs.ext4 /dev/<vgname>/<lvname>

Mount logical Volume to /slave : mount /dev/<vgname>/<lvname> /slave
check it with df -h

Successfully attached Logical Volume to Datanode…

3. Increase and Decrease Size of LVM partition :

To increase size of LVM partition use following commands :

lvextend -L +<size> /dev/<vgname>/<lvname>
resize2fs /dev/<vgname>/<lvname>

Successfully increased size…

To increase size of LVM partition use following commands :

umount /slave
e2fsck -f /dev/mapper/<vgname>-<lvname>
resize2fs /dev/mapper/<vgname>-<lvname> <size to reduce>
lvreduce -L -<size> /dev/mapper/<vgname>-<lvname>
mount /dev/mapper/<vgname>-<lvname> /slave