How to Download Http Directory with All Files and Sub-Directories as They Appear on the Online Files/Folders List

How to download HTTP directory with all files and sub-directories as they appear on the online files/folders list?

Solution:

wget -r -np -nH --cut-dirs=3 -R index.html http://hostname/aaa/bbb/ccc/ddd/

Explanation:

• It will download all files and subfolders in ddd directory
• -r : recursively
• -np : not going to upper directories, like ccc/…
• -nH : not saving files to hostname folder
• --cut-dirs=3 : but saving it to ddd by omitting
  first 3 folders aaa, bbb, ccc
• -R index.html : excluding index.html
  files

Reference: http://bmwieczorek.wordpress.com/2008/10/01/wget-recursively-download-all-files-from-certain-directory-listed-by-apache/

How to download HTTP directory with all files and sub-directories as they appear on the online files/folders list using golang?

Here you can find the algorithm for the wget --recursive implementation: https://www.gnu.org/software/wget/manual/html_node/Recursive-Download.html

Basically, you access the page and then parse the HTML and follow each href link (and css link if necessary), which can be extracted like this: https://vorozhko.net/get-all-links-from-html-page-with-go-lang.

Once you have all the links just do a request on them and based on the Content-Type header you save it if it is not text/html or parse it for links if it is.

How to download an entire directory and subdirectories using wget?

You may use this in shell:

wget -r --no-parent http://abc.tamu.edu/projects/tzivi/repository/revisions/2/raw/tzivi/

The Parameters are:

-r     //recursive Download

and

--no-parent // Don´t download something from the parent directory

If you don't want to download the entire content, you may use:

-l1 just download the directory (tzivi in your case)

-l2 download the directory and all level 1 subfolders ('tzivi/something' but not 'tivizi/somthing/foo')  

And so on. If you insert no -l option, wget will use -l 5 automatically.

If you insert a -l 0 you´ll download the whole Internet, because wget will follow every link it finds.

How to create folder in root directory with Flutter?

Use a directory for the app to store files that only it can access. As a bonus the app doesn't need any additional permissions.

final directory = await getApplicationDocumentsDirectory();

See Read and write files for details.

---

Related Topics