HADOOP CLUSTER TASK— Steps to Contributing limited storage as Slave Node to the cluster

4 min readJan 7, 2021

In Hadoop Cluster, sometimes we don’t want that my data node(slave) contribute their whole storage to the master node.

To solving this challenge, we have some way available. One of the efficient way, I’m providing here…

Prerequisite:-

⏩ Configure one Datanode and one Masternode .

For implementing the given task we have to perform following steps :-

Create Virtual hard disk and attached it to the data node .
Create partition .

Decide how much volume you want.
Format that partition .
Mount it .

3. Again configure the data node .

4. Check storage shared by data node to name node.

Implementation:-

>> Step1:- Create virtual hard disk.

For creating the virtual hard disk , we have to go to the storage of datanode in settings . We have created a hard disk of 10GB.

We can see that the virtual hard disk :- “redhat_8_1.vdi “ of size 10GB is created successfully , but it’s not attched . So select that hard disk and attached it .

Virtual HardDisk is successfully attached.

We can check it through terminal .So open the terminal of the data node and run this command :- fdisk -l

Here we can see that the virtual hard disk is attached successfully to the data node.

* Step2:- Create partition.

Now we have to first create partion and then we can store data inside it . So for creating partition run this command fdisk /dev/xvdb

>> Here I am going to use some options .

1. Create new partition :- n

2. Primary Partition :- p (by default)

3. Partition number :- 1 (by default)

4. First sector :- 2048 (by default )

5. Last sector :- 1000000( it’s depend on you how much space you want in my case it’s 487.3 MiB)

6. Save this partition :- w

Now, Next step is we need to format this partition…

We want to format ( give the partition name in may case its /dev/xvdb1), and command to format the partition is given below cmd :- mkfs.ext4 /dev/sdb

Now we need to mount this partition, for this I’m creating one directory using mkdir command…

cmd to make directory:- mkdir /dn

cmd to mount in the directory:- mount /dev/xvdb1/dn

Now it’s done.

Now I need to set this directory in Hadoop hdfs.xml file…

Go to /etc/hadoop and then edit hdfs-site.xml

Finally, I am going to check that the limited amount of storage of DataNode has been distributed or not to the Hadoop cluster using the below command.

# hadoop dfsadmin -report

Now you can see the data node the limited 487.3 MB space of DataNode’s storage and the rest 10 GB Space of Storage of Datanode is free from the Hadoop Cluster.

Conclusion

So we can contribute a limited amount of storage of any DataNode in the Hadoop Cluster as much we want. Rather than giving the complete amount of storage of Datanode to the Hadoop Cluster.