Web Technology Tutorials

Apache Hadoop | Running MapReduce Jobs After setting up your environment and running the HDFS and YARN daemons, we can start working on running MapReduce jobs on our local machine. We need to compile our code, produce a JAR file, move our inputs, and run a MapReduce job on Hadoop. Step 1 - Configure extra environment variables As a preface, it is best to setup some extra environment variables to make running jobs from the CLI quicker and easier. You can name these environment variables anything you want, but we will name them HADOOP_CP and HDFS_LOC to not potentially conlict with other environment variables. Open the Start Menu and type in 'environment' and press enter. A new window with System Properties should open up. Click the Environment Variables button near the bottom right. HADOOP_CP environment variable This is used to compile your Java files. The backticks (eg. `some command here`) do not work on Win...

Apache Hadoop Prerequisites and Installation The following files are needed for any Hadoop project: Please follow the steps given in the following link for seting up Hadoop on Windows 10 machines How to set up Hadoop on Windows 10 hadoop-hdfs-3.3.4.jar (Java Archive File) zookeeper-3.6.4.jar (Java Archive File) log4j-1.2-api-2.19.0.jar (Java Archive File) hadoop-mapreduce-client-core-3.3.4.jar (Java Archive File) hbase-0.92.1.jar (Java Archive File) hadoop-common-3.3.4.jar (Java Archive File) AN IDE(Integrated DEvelopment Environment) like Eclipse Click on the perspective menu in the menu bar and select Mapreduce as the perespective After installing successfully Hadoop, we have a directory structure as given below in the hadoop-3.2.1 directory Eclipse IDE Setup and initialization Configuration The binary executables to start the name node, data nodes, mapreduce sites,...

Apache Hadoop | Use Case | Association Rule Mapping | Part 1 The classes required to run a Hadoop application: org.apache.zookeeper.*; org.apache.log4j.*; org.apache.hadoop.conf.Configuration; org.apache.hadoop.fs.Path; org.apache.hadoop.hbase.*; org.apache.hadoop.hbase.HBaseConfiguration; org.apache.hadoop.hbase.client.HTable; org.apache.hadoop.hbase.client.Put; org.apache.hadoop.hbase.client.Get; org.apache.hadoop.hbase.util.Bytes; org.apache.hadoop.hbase.client.Result; org.apache.hadoop.hbase.client.ResultScanner; org.apache.hadoop.hbase.client.Scan; org.apache.hadoop.io.IntWritable; org.apache.hadoop.io.LongWritable; org.apache.hadoop.io.ObjectWritable; org.apache.hadoop.io.Text; org.apache.hadoop.mapreduce.Job; org.apache.hadoop.mapreduce.Mapper; org.apache.hadoop.mapreduce.Reducer; org.apache.hadoop.mapreduce.lib.input.FileInputFormat; org.apache.hadoop.mapreduce.lib.input.TextInputFormat; org.apache.hadoop.mapreduce.lib.o...

Starting with Apache Hadoop System analysis and Design This article explains how the system is analyzed to carry out the work for the proposed system. System analysis is the process of gathering and interpreting facts, diagnosing problems, and using the facts to improve the system. System analysis does more than just solve the current problem especially when there is no such system exists that is going to be developed. The future needs of the business and the changes required to meet the needs are analyzed. Once the decision is made, the plan is developed to implement the recommendations. The plan includes all system design features, such as new data capture needs(storage system), operating systems, equipment, and personal needs. The system design is like a blueprint: it specifies all the features that are to be in the finished product. Class diagram The class diagram is static. It represents the static view of an application. It...

Linux commands | Miscellaneous Apache server does not support multiple threads (ornsub-proceesses) running in parallel lke nginx Instead it creates another process which has a different process id. NOrmally, two processes are initiated when you start a Apache Serrver. FOr example The configuration file for Apache web server conssists of a set of key-value pairs :-