Automation Using Python — AWS CLI, Docker, Linux Basics, Hadoop, Partitions…

Hello Guys !!! Welcome to new Topic…

Today we are Performing a task to Automate some Technical concepts using Python which include AWS CLI, Docker, Linux Basic Commands, Hadoop and Partitions…

This task is Done in a Team and we have distributed the task within ourselves and connected with each other via Google meet. The task is that we have to Automate above mentioned topics using python . So, instead of doing it seperately we distributed our part as…

Team members and the part they performed :=

1. Kaustubh Chaudhari (Team Leader) : Partitions and Blog.
2. Nitesh Thapliyal(Member) : AWS CLI and Docker.
3. Sri Vishnuvardhan (Member) : Linux Basic Commands and Hadoop.
4. Naitik Shah(Member) : Video making

This is the video of the task we have implemented …

KAUSTUBH CHAUDHARI on LinkedIn: #python #vimaldaga #righteducation

for more detail code you can visit our Github Repository also link for the same : click here

# Contents of the Task and the approach followed to do the same :=

1. AWS CLI
2. Docker
3. Linux Commands
4. Hadoop
5. Partitions

— Let’s Start…

# AWS CLI :=

• AWS CLI is the Command Line Interface to access the Amazon AWS Cloud . We can do many things which we cannot do using GUI console of AWS CLI. One of the thing is that we ca launch multiple instances at same time using AWS CLI but we cannot do it with GUI console, like this there are many use cases of AWS CLI.
• To use AWS CLI may be little difficult for many users so we have Automated this AWS CLI using Python. For doing this you first have AWS CLI installed in your system and also Python installed .
• Task you can do with this program are:=

1. Creating/Deleting Keypair
2. Launching /Starting / Stopping / Terminating Instances .
3. Creating / Deleting Security group
4. Creting and Attaching EBS volume to instance. and much more…

For more detail code visit or github repo : click here

# Docker :=

• Docker is the Tool which provides us Container Technolgy. Container Technology or Containerization is the concept which allows us to install multiple Os on top of our system without consumption of much space.
• One Docker OS have only upto 20 MB RAM, see how optimized it is
• But again using Docker for client must be a huge task , So understanding this use case we have Automated Docker with Python. Doing this we can do any Docker operation in only one click.
• For this you must have Docker and Python installed on your system.
• Concepts Implemented in program :=

1. Installing , Booting , Login Docker OS.
2. Check Docker Status
3. Check Docker Images
4. Check Docker Process
5. Download Os Image
6. Run OS
7. Delete Docker image

For more detail code visit or github repo : click here

# Linux Basic Commands :=

• Linux is one of the most used OS in industry and is capable of solving any big use case. But linux is famous for it’s Command Line Interface (CLI) which we called as Kernel .
• To do any task in CLI we need command which many of us don’t know or we just forget the command So, to solve this difficulty others face we have Automated Linux Basic Commands using Python.
• For this you need anyone Linux OS (We are using RHEL8) and Python installed.
• Some of Basic Linux Commands :=

1. date : shows date
2. cal : shows calender
3. free -m : Shows RAM usage
4. df -h : shows the file system
5. lscpu : shows CPU configuration of your system and much more…

For more detail code visit or github repo : click here

# Hadoop :=

• Hadoop is the Technical tool which helps us to solve the issue we face due to Big Data. Big Data is the huge amount of data which is generated via various sources and can be of any form like text, video, image, voice, etc.
• Advantages for using Hadoop for solving Big Data problems is that it is Open source, Scalability, Flexibility, Speed, High Throughput, Minimum Network Traffic, Fault Tolerance and low cost.
• Hadoop works on Master-Slave Architecture in which master node is the one which manages all the slaves and slave node provide storage to the master. This concept is known as HDFS cluster(Hadoop Distributed File Storage).
• Hadoop require lot of commands and file configurations which is tideous task and time consuming process So, understanding this use case we have Automated Hadoop using Python . In one click you can configure Master node and Slave node , Do various operations on Hadoop.
• For this we must have jdk , hadoop and python installed in our system.
• Task we can do with this program : =

1. View Hadoop status
2. Configuring HDFS cluster on Master node
3. Configuring HDFS cluster on Data node
4. Configuring HDFS cluster on Client node
5. View files stored in cluster
6. Change the block size lively

For more detail code visit or github repo : click here

# Linux Partitions :=

• Partition is the concept through which we can split our Hard Disk space into multiple sections called partitions, which are isolated from each other. In Linux Os, partition is one of the basic and important concept which many of us are not aware about.
• There are two type of Partitions in Linux :

1. Static Partition : This partition once made no changes can be made regarding the size of partition i.e we cannot chnage size of this partition dynamically . We have two type of Static partitions namely : Physical and Extended partitions. There can be only 4 physical partitions but we can create multiple Extended partitions .
2. LVM Partition : This partition can change it’s size dynamically without disturbing or deleting previous data and that to at Run time(Dynamically). For this We have to make our Hard disk as PV(physical volumes) and create VG(virtual group) a Logical Hard disk made by combining various PV’s . After creating VG we have to create partition in that VG called as LV(logical volume).

• To do this all thing we have type many commands manually which is tideous task and consumes lot of time So, to solve this use case we have Automated Linux partitions using Python .
• For this you must have Linux Os and Python installed on your system .
• Task done in the program : =

1. Creating/Displaying / Saving/Deleting Static Partition.
2. Creating LVM partition.
3. Displaying PV
4. Displaying VG
5. Displaying LV
6. Increasing size of partition dynamically…

For more detail code visit or github repo : click here

To see the details of code visit our Github repo : click here

Hope you all like the Content Stay tuned for more Automation and new Concepts about latest Technologies and Tools…

Thank you !!! Byee…