Command to Store File on Hdfs

Command to store File on HDFS

If no path is provided, hadoop will try to copy the file in your hdfs home directory. In other words, if you're logged as utrecht, it will try to copy ubuntu-14.04-desktop-amd64.iso to /user/utrecht.

However, this folder doesn't exist from scratch (you can normally check the dfs via a web browser).
To make your command work, you have two choices :

  • copy it elsewhere (/ works, but putting everything there may lead to complications in the future)
  • create the directory you want with hdfs dfs -mkdir /yourFolderPath

How files or directories are getting stored in hadoop hdfs

HDFS file system is a distributed storage system wherein the storage location is virtual and created using the disk space from all the DataNodes. While installing hadoop, you must have specified paths for dfs.namenode.name.dir and dfs.datanode.data.dir. These are the locations at which all the HDFS related files are stored on individual nodes.

While storing the data onto HDFS, it is stored as blocks of a specified size (default 128MB in Hadoop 2.X). When you use hdfs dfs commands you will see the complete files but internally HDFS stores these files as blocks. If you check the above mentioned paths on your local file system, you will see a bunch of files which correcpond to files on your HDFS. But again, you will not see them as actual files as they are split into blocks.

Check below mentioned command's output to get more details on how much space from each DataNode is used to create the virtual HDFS storage.

hdfs dfsadmin -report #Or

sudo -u hdfs hdfs dfsadmin -report

HTH

Store file on hadoop

It seems like you need to provide local-src and HDFS-dst.
Can you try to add destination?
e.g. hadoop fs -put customer.tbl .

please also try execute "ls" on the HDFS:

hadoop fs -ls

please also try execute "ls" on the HDFS using hdfs command, 'hdfs' should be found under hadoop-version-number/bin/:

hdfs dfs -ls

How can I save the -ls output to hdfs directory

When you redirect, you are interacting with your LOCAL filesystem.
To save to HDFS you need to do a HDFS put.

$ hadoop fs -ls / > lsResult.txt
$ hadoop fs -put lsResult.txt /

How to store log files of a shell script in HDFS

The logs are not appending to the files. only the files are being
created.

Because tee is a linux command and does not work for files stored in HDFS.

Use -appendToFile

echo "`date +\"%Y-%m-%d %H:%M:%S\"` [ERROR] $message [Status] $status : failed" | hdfs dfs -appendToFile - ${failed_logs}

- in place of srcfile is to read the input from stdin.



Related Topics



Leave a reply



Submit