Web Technology Tutorials

Apache Hadoop | Running MapReduce Jobs After setting up your environment and running the HDFS and YARN daemons, we can start working on running MapReduce jobs on our local machine. We need to compile our code, produce a JAR file, move our inputs, and run a MapReduce job on Hadoop. Step 1 - Configure extra environment variables As a preface, it is best to setup some extra environment variables to make running jobs from the CLI quicker and easier. You can name these environment variables anything you want, but we will name them HADOOP_CP and HDFS_LOC to not potentially conlict with other environment variables. Open the Start Menu and type in 'environment' and press enter. A new window with System Properties should open up. Click the Environment Variables button near the bottom right. HADOOP_CP environment variable This is used to compile your Java files. The backticks (eg. `some command here`) do not work on Win...

Apache Hadoop Prerequisites and Installation The following files are needed for any Hadoop project: Please follow the steps given in the following link for seting up Hadoop on Windows 10 machines How to set up Hadoop on Windows 10 hadoop-hdfs-3.3.4.jar (Java Archive File) zookeeper-3.6.4.jar (Java Archive File) log4j-1.2-api-2.19.0.jar (Java Archive File) hadoop-mapreduce-client-core-3.3.4.jar (Java Archive File) hbase-0.92.1.jar (Java Archive File) hadoop-common-3.3.4.jar (Java Archive File) AN IDE(Integrated DEvelopment Environment) like Eclipse Click on the perspective menu in the menu bar and select Mapreduce as the perespective After installing successfully Hadoop, we have a directory structure as given below in the hadoop-3.2.1 directory Eclipse IDE Setup and initialization Configuration The binary executables to start the name node, data nodes, mapreduce sites,...

Apache Hadoop | Use Case | Association Rule Mapping | Part 1 The classes required to run a Hadoop application: org.apache.zookeeper.*; org.apache.log4j.*; org.apache.hadoop.conf.Configuration; org.apache.hadoop.fs.Path; org.apache.hadoop.hbase.*; org.apache.hadoop.hbase.HBaseConfiguration; org.apache.hadoop.hbase.client.HTable; org.apache.hadoop.hbase.client.Put; org.apache.hadoop.hbase.client.Get; org.apache.hadoop.hbase.util.Bytes; org.apache.hadoop.hbase.client.Result; org.apache.hadoop.hbase.client.ResultScanner; org.apache.hadoop.hbase.client.Scan; org.apache.hadoop.io.IntWritable; org.apache.hadoop.io.LongWritable; org.apache.hadoop.io.ObjectWritable; org.apache.hadoop.io.Text; org.apache.hadoop.mapreduce.Job; org.apache.hadoop.mapreduce.Mapper; org.apache.hadoop.mapreduce.Reducer; org.apache.hadoop.mapreduce.lib.input.FileInputFormat; org.apache.hadoop.mapreduce.lib.input.TextInputFormat; org.apache.hadoop.mapreduce.lib.o...

AWS Cloud Computing in-depth Capacity planning is an iterative provcess with the follpwing steps: Determine the charcateristics of the prerseent system Measure of teh workload for the dofferent resources in the system (CPU,RAM,Disk,network,) and so forth Load the system until it is overloaded, determine when it breqaks , and specify what is required tomaintain acceptable performance Knowing when system's fail under load and what factor(s) is responsible for the failure if the critical step in capacity plamniing Predict the future based on historical trends and other factors Deploy or tear down resoureces to meet your predictions IterTate steps 1 5to 5 repeadtedly Amazon Virtual Private Cloud It provides a bridge between a company's existing network and the AWS cloud. VPC connects your network resources to a set of AWS systems over a Virtual Private Network (VPN) ...

Introduction to Amazon Web Services (AWS) Amazon Web Service Components Here, we will see what all features and services provided by AWS (Amazon Web Services). Elastic Compute Cloud(EC2): A Amazon server has some properties: The AWS key which is known only to the Amazon database and your server which is hosted on Amazon cloud. The AWS secret is also known to the Amazon database and your server which is hosted on Amazon cloud. A user can upload a file, download a file or delete a vfile on an AWS EC2 instance. The AWS key is a 20 character string having uppercase letters and digits. The AWS secret is a 40 character string having uppercase letters and digits The AWS key and secret are used for checking the identity and authenticating the client. They are generally stored in the config folder or bootstrap folder or .env file as it will be authenticated when the IoC...

Cloud Computing in detail Most large Infrastructure as a Service (IaaS) providers rely on virtual machine technology to deliver servers that can run applications. Virtual servers described in terms of a machine image or instances have charcateristics that often can be described in terms of real servers delivering a certain number of mivcroprocessor (CPU) cycles, memory access, and network bandwidth to customer.Virtual machines are containers that are assigned specific resources. The software that runs in the virtual machines is what defines the utility of the cloud coomputing system. Platforms: A platform in the cloud is a software layer that is used to craete higher levels of service. Many different Platform as a Service (PaaS) providers offer services to provide developers with different capabilities. These services offer all the hosted hardware and software needed to build and deploy web applications or services tha...

Cloud Computing A deployment model defines the purpose of the cloud, and the nature of how the cloud is located. The NIST(The U.S. National Institute of Standards and Technology) definition for the four deployment models is as follows: Public Cloud Private Cloud Hybrid Cloud Community Cloud Public Cloud The publkic cloud infrastructure is available for public use alternatively for a large industry group and ids owned by an organization selling cloud services. Private Cloud The private cloud mechanism is operated for the exclusive use of an organization. The cloud maay be managedby that organiation or a third party. Hybrid Cloud A Hybrid cloud combines multiple clouds(private, community or public) where those clouds retain their unique identities, but are bound together as a unit. Community cloud A Community cloud is one where the cloud has been organi...