How to Find Hadoop Hdfs Directory on My System

How to find Hadoop hdfs directory on my system?

Your approach is wrong or may be understanding is wrong

dfs.datanode.data.dir, is where you want to store your data blocks

If you type hdfs dfs -ls / you will get list of directories in hdfs. Then you can transfer files from local to hdfs using -copyFromLocal or -put to a particular directory or using -mkdir you can create new directory

Refer below link for more information

http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html

How to navigate directories in Hadoop HDFS

There is no cd (change directory) command in hdfs file system. You can only list the directories and use them for reaching the next directory.

You have to navigate manually by providing the complete path using the ls command.

hdfs dfs -ls /user/username/app1/subdir/

how to show the hdfs root directory in hadoop?

If you can execute hadoop version command and it returns correct information, it means that Hadoop was installed good.

I think there might be a problem with HDFS configuration. Try this:

1. Locate core-site.xml file in your local file system. It should be located in /etc/hadoop/conf directory.

2. Open core-site.xml file and locate this property:

   <property>
        <name>fs.defaultFS</name>
        <value>hdfs://<name-of-your-host>:8020</value>
        <final>true</final>
   </property>
   

3. I suppose name parameter is wrong. You have to identify the address on which HDFS is running and update it in core-site.xml.

hadoop dfs -ls gives list of folders not present in the local file system

You can't see the hdfs directory structure in graphical view to view it you have to use your terminal only.
hdfs dfs -ls /
and to see local file directory structure in the terminal you should try
ls <path>
cd <path>
cd use to change the directory in terminal.

where can i find directory i have created using hadoop fs -mkdir in my ubuntu file system

You won't find that directory in your local (ubuntu) filesystem. You'll have to work with HDFS using its command line utilities (hdfs dfs ...).

Is there a hdfs command to list files in HDFS directory as per timestamp

No, there is no other option to sort the files based on datetime.

If you are using hadoop version < 2.7, you will have to use sort -k6,7 as you are doing:

hdfs dfs -ls /tmp | sort -k6,7

And for hadoop 2.7.x ls command , there are following options available :

Usage: hadoop fs -ls [-d] [-h] [-R] [-t] [-S] [-r] [-u] <args>

Options:
-d: Directories are listed as plain files.
-h: Format file sizes in a human-readable fashion (eg 64.0m instead of 67108864).
-R: Recursively list subdirectories encountered.
-t: Sort output by modification time (most recent first).
-S: Sort output by file size.
-r: Reverse the sort order.
-u: Use access time rather than modification time for display and sorting.

So you can easily sort the files:

hdfs dfs -ls -t -R (-r) /tmp 

find file in hadoop filesystem

If you are looking for equivalent of locate Linux command than such option does not exist in Hadoop. But if you are looking for the way of how to find specific file you can use name parameter of fs -find command for this:

hadoop fs -find /some_directory -name some_file_name

If you are looking for the actual location of hdfs file in your local file system you can use fsck command for this:

hdfs fsck /some_directory/some_file_name -files -blocks -locations

Where do hdfs directories reside in linux?

Actually we cannot see the hdfs directories directly so there are two methods to see the file in hdfs file system location

1st method is from terminal

hdfs dfs -ls /user/cloudera

so that you can see the files in hdfs filesystem in 'cloudera' named directory

2nd method is using the browser.

http://localhost:50070 in this you have to go to browse file system and see all files that are stored in hdfs file system.

Where is the physical path of HDFS system on disk?

You need to use the hdfs dfs / hadoop fs (or another hdfs client) to put the files into the dfs - the location of the HDFS blocks on disk will not help you to add files to the HDFS filesystem - eg

hdfs dfs -put /path/to/uid_details.txt /user/root/uid_details.txt

or

hdfs dfs -put /path/to/uid_details.txt hdfs://namenodeaddress/user/root/uid_details.txt 

---

Related Topics