Apache Hadoop Prerequisites and Installation

-
Apache Hadoop Prerequisites and Installation
The following files are needed for any Hadoop project:
Please follow the steps given in the following link for seting up Hadoop on Windows 10 machines
How to set up Hadoop on Windows 10
  • hadoop-hdfs-3.3.4.jar (Java Archive File)
  • zookeeper-3.6.4.jar (Java Archive File)
  • log4j-1.2-api-2.19.0.jar (Java Archive File)
  • hadoop-mapreduce-client-core-3.3.4.jar (Java Archive File)
  • hbase-0.92.1.jar (Java Archive File)
  • hadoop-common-3.3.4.jar (Java Archive File)
  • AN IDE(Integrated DEvelopment Environment) like Eclipse
  • Click on the perspective menu in the menu bar and select Mapreduce as the perespective

After installing successfully Hadoop, we have a directory structure as given below in the hadoop-3.2.1 directory

Eclipse IDE Setup and initialization

Configuration

The binary executables to start the name node, data nodes, mapreduce sites, and HDFS filesystem are in the %HADOOP_HOME%\bin directory

Starting the server

The first step is to format the hdfs server namenode. This is achieved using hdfs namenode -format
Startiing HDFS daemons
Run the following command to start HDFS daemons. When you do so, there should be two new windows that open: one for datanode and the other for namenode:
$ %HADOOP_HOME%\sbin\start-dfs.cmd

Comments

Post a Comment

Popular posts from this blog

XPath for HTML markup

-
XPath Web Scraping Web Scraping is a technique to traverse the DOM (Document Object Model) of an HTML or an HTTP web page Web Scraping is achieved due to XPath XPath nodes: There are seven kinds of nodes Element - It represents any HTML 5 element. For example, <strong></strong> Attribute - It represents any one attribute of any HTML 5 element in the document object model Text - It represents the text between the opening HTML 5 tag and a closing HTML 5 tag Namespace - It represents the pseudo selector of an HTML 5 element Processing-Instructions Comment - It represents any HTML 5 comment It represents the topmost element of the tree is called the root element. For example, the root element for any HTNL 5 document is HTML The first XPath node is \ which Suppose, I have an HTML5 code snippet as follows: Gaurav Shirodkar Then to ...
Read more

Apache Hadoop | Running MapReduce Jobs

-
Image
Apache Hadoop | Running MapReduce Jobs After setting up your environment and running the HDFS and YARN daemons, we can start working on running MapReduce jobs on our local machine. We need to compile our code, produce a JAR file, move our inputs, and run a MapReduce job on Hadoop. Step 1 - Configure extra environment variables As a preface, it is best to setup some extra environment variables to make running jobs from the CLI quicker and easier. You can name these environment variables anything you want, but we will name them HADOOP_CP and HDFS_LOC to not potentially conlict with other environment variables. Open the Start Menu and type in 'environment' and press enter. A new window with System Properties should open up. Click the Environment Variables button near the bottom right. HADOOP_CP environment variable This is used to compile your Java files. The backticks (eg. `some command here`) do not work on Win...
Read more

Laravel | PHP | Basics | Part 2

-
Image
Learning Laravel through Screenshots Caching Laravel by default supports multiple cache stores like memcached , redis ,etc. To install redis as a cache store, you should use composer install predis/predis predis stands for PHP redis The default port number is 6379 Every Cache store requires a prefix which is stored in the ENV file or in the config/cache.php Following are the cache stores supported by laravel Blade Templates Blade is a templating engine for views in laravel. All the files which are of type blade.php are having the extension .blade.php They are stored in the resources/views directory YOu can create suub-folders in the reosurces/views directory like layouts for all the common UI components lke header , footer, sidebar , navbar, menubar, serach bar, etc. To include the parenet ...
Read more