Command to store File on HDFS
If no path is provided, hadoop will try to copy the file in your hdfs home directory. In other words, if you're logged as utrecht, it will try to copy ubuntu-14.04-desktop-amd64.iso to /user/utrecht
.
However, this folder doesn't exist from scratch (you can normally check the dfs via a web browser).
To make your command work, you have two choices :
- copy it elsewhere (
/
works, but putting everything there may lead to complications in the future) - create the directory you want with
hdfs dfs -mkdir /yourFolderPath
How files or directories are getting stored in hadoop hdfs
HDFS file system is a distributed storage system wherein the storage location is virtual and created using the disk space from all the DataNodes. While installing hadoop, you must have specified paths for dfs.namenode.name.dir
and dfs.datanode.data.dir
. These are the locations at which all the HDFS related files are stored on individual nodes.
While storing the data onto HDFS, it is stored as blocks of a specified size (default 128MB in Hadoop 2.X). When you use hdfs dfs
commands you will see the complete files but internally HDFS stores these files as blocks. If you check the above mentioned paths on your local file system, you will see a bunch of files which correcpond to files on your HDFS. But again, you will not see them as actual files as they are split into blocks.
Check below mentioned command's output to get more details on how much space from each DataNode is used to create the virtual HDFS storage.
hdfs dfsadmin -report
#Or
sudo -u hdfs hdfs dfsadmin -report
HTH
Store file on hadoop
It seems like you need to provide local-src and HDFS-dst.
Can you try to add destination?
e.g. hadoop fs -put customer.tbl .
please also try execute "ls" on the HDFS:
hadoop fs -ls
please also try execute "ls" on the HDFS using hdfs command, 'hdfs' should be found under hadoop-version-number/bin/:
hdfs dfs -ls
How can I save the -ls output to hdfs directory
When you redirect, you are interacting with your LOCAL filesystem.
To save to HDFS you need to do a HDFS put.
$ hadoop fs -ls / > lsResult.txt
$ hadoop fs -put lsResult.txt /
How to store log files of a shell script in HDFS
The logs are not appending to the files. only the files are being
created.
Because tee
is a linux command and does not work for files stored in HDFS.
Use -appendToFile
echo "`date +\"%Y-%m-%d %H:%M:%S\"` [ERROR] $message [Status] $status : failed" | hdfs dfs -appendToFile - ${failed_logs}
-
in place of srcfile
is to read the input from stdin.
Related Topics
Using Gzip to Compress Files to Transfer with Aws Command
How to Ask Bash for the Current Options
How to Offload the 1Hz Tick in Dyntick Mode
How Come _Exit(0) (Exiting by Syscall) Prevents Me from Receiving Any Stdout Content
Old Logs Are Not Imported into Es by Logstash
Apache Proxypass Not Loading Resources
Libcurl Ssl Error After Fork()
How to Set the Socket Option So_Reuseport in Rust
Limit Output of All Linux Commands
X86_64 Assembly Execve *Char[] Syscall
Imagemagick: Generate Raw Image Data for PDF Flate Embedding
Sed Find and Replace Between Two Tags with Multi Line
Replace Parentheses and Spaces in Filenames with Underscore
How to Initialize Raw Socket for Vlan Sniffing
Preventing to Bash Script from Running in Parallel or Overlap Using Cron