Hadoop 2.4.0 – Use with a one node cluster

Created: 2014/04/26 ; Modified: 2015/05/23
Thumbnail

To try that tutorial, you must already have done the steps in the previous video.

  • Configure the filesystem (00:30)
    • vim etc/hadoop/core-site.xml
<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>
  • vim etc/hadoop/hdfs-site.xml
<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>
  • Prepare SSH password-less (01:50)
    • ssh-keygen -t dsa -P  » -f ~/.ssh/id_dsa
    • cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
    • ssh localhost
  • Prepare HDFS (03:43)
    • bin/hdfs namenode -format
    • sbin/start-dfs.sh
  • Create some directories on HDFS (05:27)
    • bin/hdfs dfs -mkdir /user
    • bin/hdfs dfs -mkdir /user/hadoop
    • bin/hdfs dfs -ls /
  • Test (06:47)
    • bin/hdfs dfs -put etc/hadoop input
    • bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.4.0.jar grep input output ‘dfs[a-z.]+’
    • bin/hdfs dfs -get output output
    • cat output/*
  • Stop HDFS (09:11)
    • sbin/stop-dfs.sh

Comments, questions and suggestions:

  • Marcelo Amaral dit :

    Thank you for this tutorial. It is very well explained.

    Are you planning to do a tutorial about how to setup more than one node?

    Thanks.

    • Thanks 🙂

      At this point, I am not planning anything since I have so many technologies I want to try and not enough time to put all in videos. I hope that will soon change

  • Caio dit :

    Hi, i’m having some troubles at port 50070, i dont got a connection with the browser, but all the commands at prompt seem ok.
    i’m new at all this and don’t know how this works, any tips would help

    • Hi,

      to connect to it, you need to make sure there is no firewall blocking you. For example, if your machine is on Amazon, by default, all the ports are blocked from the outside, so you will need to open them.
      In my demo, it was on a local virtual machine so I had full access.

      • Caio dit :

        Thanks, I will take a look at the port, but I have another question, I’m looking for hadoop tutorial that explains the configuration of the master and slaves, as would be the configuration of the xml and stuff, could show me some tutorial as good as your?

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *