How to Check If Hadoop Daemons Are Running

How to check if Hadoop daemons are running?

I did not find great solution to it, so I used

ps -ef | grep hadoop | grep -P  'namenode|datanode|tasktracker|jobtracker'

just to see if stuff is running

and

./hadoop dfsadmin -report

but last was not helpful until server was running.

How to check if hdfs is running?

Use any of the following approaches for to check your deamons status

1. JPS command would list all active deamons

2. the below is the most appropriate

      hadoop dfsadmin -report
   

   This would list down details of datanodes which is basically in a sense your HDFS

3. cat any file available in hdfs path.

hadoop daemons not starting

You have to write this directory in "core.site.xml" file not in hadoop-env.sh

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/Directory_hadoop_user_have_permission/temp/${user.name}</value>
  <description>A base for other temporary directories.</description>
</property>

<property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
</property>

</configuration>

Hadoop Daemons not stopping

This message is shown if the start/stop scripts cannot find a pid file got the deamon in the $HADOOP_PID_DIR folder (which defaults to /tmp).

If:

• these files have been deleted (by someone or something), or
• the env variable $HADOOP_PID_DIR has been changed since you started the deamons, or
• the user stopping the deamons is not the user that started them

then hadoop will show the error messages you are seeing.

Selected portions from the hadoop-daemon.sh file (for 1.0.0):

#   HADOOP_IDENT_STRING   A string representing this instance of hadoop. $USER by default

if [ "$HADOOP_IDENT_STRING" = "" ]; then
  export HADOOP_IDENT_STRING="$USER"
fi

# ....

if [ "$HADOOP_PID_DIR" = "" ]; then
  HADOOP_PID_DIR=/tmp
fi    

# ....

pid=$HADOOP_PID_DIR/hadoop-$HADOOP_IDENT_STRING-$command.pid

# ....

(stop)

  if [ -f $pid ]; then
    if kill -0 `cat $pid` > /dev/null 2>&1; then
      echo stopping $command
      kill `cat $pid`
    else
      echo no $command to stop
    fi
  else
    echo no $command to stop
  fi
  ;;

Cloudera Hadoop - Daemons not running

Finally, I found out how to start services on cloudera quickstart vm with some help from the community.

service hadoop-hdfs-namenode start

Now when i run JPS, I can see all the daemons running,

[root@quickstart cloudera]# jps
2374 JobHistoryServer
2070 NameNode
3294 RunJar
4445 Bootstrap
4803 
2947 -- process information unavailable
2196 SecondaryNameNode
1840 QuorumPeerMain
1908 DataNode
4836 
3094 RunJar
3777 Master
2865 RESTServer
2594 ResourceManager
2327 Bootstrap
3663 Bootstrap
2451 NodeManager
1999 JournalNode
3111 Jps
3684 HistoryServer
4784 Bootstrap

Thanks a lot for your attention.

how to know if TaskTracker is running through Java code?

One solution is to call the service's HTTP endpoint for the JMX metrics JSON dump URL. You can use any HTTP client library in your language of choice to do this. You mentioned TaskTracker, so it sounds like you are running Hadoop 1. Here is the configuration property that specifies the HTTP endpoint:

<property>
  <name>mapred.task.tracker.http.address</name>
  <value>0.0.0.0:50060</value>
  <description>
    The task tracker http server address and port.
    If the port is 0 then the server will start on a free port.
  </description>
</property>

Assuming you haven't overridden this, the URL to call with a GET request would be http://<host>:50060/jmx.

Many Hadoop daemons expose an HTTP server and respond to the /jmx URL. This solution is not limited to just the TaskTracker. The specific metrics payload in the JSON response will be different per daemon, but if you're just interested in whether or not the daemon is running, then simply checking for a successful HTTP response would be sufficient.

How to check if my hadoop is running in pseudo distributed mode?

To know if you are running hadoop in Standalone or Pseudo distributed mode, verify your configuration files. Below information might help.

What is best way to start and stop hadoop ecosystem, with command line?

start-all.sh & stop-all.sh : Used to start and stop hadoop daemons all at once. Issuing it on the master machine will start/stop the daemons on all the nodes of a cluster. Deprecated as you have already noticed.

start-dfs.sh, stop-dfs.sh and start-yarn.sh, stop-yarn.sh : Same as above but start/stop HDFS and YARN daemons separately on all the nodes from the master machine. It is advisable to use these commands now over start-all.sh & stop-all.sh

hadoop-daemon.sh namenode/datanode and yarn-deamon.sh resourcemanager : To start individual daemons on an individual machine manually. You need to go to a particular node and issue these commands.

Use case : Suppose you have added a new DN to your cluster and you need to start the DN daemon only on this machine,

bin/hadoop-daemon.sh start datanode

Note : You should have ssh enabled if you want to start all the daemons on all the nodes from one machine.

Hope this answers your query.

---

Related Topics