Download Webpage and Dependencies, Including CSS Images

download webpage and dependencies, including css images

wget --page-requisites http://example.com/your/page.html

This option causes Wget to download all the files that are necessary to properly display a given html page. This includes such things as inlined images, sounds, and referenced stylesheets.

EDIT: meder is right: stock wget does not parse and download css images. There is, however, a patch that adds this feature: [1, 2]

UPDATE: The patch mentioned above has been merged into wget 1.12, released 22-Sep-2009:

** Added support for CSS. This includes:
 - Parsing links from CSS files, and from CSS content found in HTML
   style tags and attributes.
 - Supporting conversion of links found within CSS content, when
   --convert-links is specified.
 - Ensuring that CSS files end in the ".css" filename extension,
   when --convert-links is specified.

Download a working local copy of a webpage

wget is capable of doing what you are asking. Just try the following:

wget -p -k http://www.example.com/

The -p will get you all the required elements to view the site correctly (css, images, etc).
The -k will change all links (to include those for CSS & images) to allow you to view the page offline as it appeared online.

From the Wget docs:

‘-k’
‘--convert-links’
After the download is complete, convert the links in the document to make them
suitable for local viewing. This affects not only the visible hyperlinks, but
any part of the document that links to external content, such as embedded images,
links to style sheets, hyperlinks to non-html content, etc.

Each link will be changed in one of the two ways:

    The links to files that have been downloaded by Wget will be changed to refer
    to the file they point to as a relative link.

    Example: if the downloaded file /foo/doc.html links to /bar/img.gif, also
    downloaded, then the link in doc.html will be modified to point to
    ‘../bar/img.gif’. This kind of transformation works reliably for arbitrary
    combinations of directories.

    The links to files that have not been downloaded by Wget will be changed to
    include host name and absolute path of the location they point to.

    Example: if the downloaded file /foo/doc.html links to /bar/img.gif (or to
    ../bar/img.gif), then the link in doc.html will be modified to point to
    http://hostname/bar/img.gif. 

Because of this, local browsing works reliably: if a linked file was downloaded,
the link will refer to its local name; if it was not downloaded, the link will
refer to its full Internet address rather than presenting a broken link. The fact
that the former links are converted to relative links ensures that you can move
the downloaded hierarchy to another directory.

Note that only at the end of the download can Wget know which links have been
downloaded. Because of that, the work done by ‘-k’ will be performed at the end
of all the downloads. 

Save complete web page (incl css, images) using python/selenium

As you noted, Selenium cannot interact with the browser's context menu to use Save as..., so instead to do so, you could use an external automation library like pyautogui.

pyautogui.hotkey('ctrl', 's')
time.sleep(1)
pyautogui.typewrite(SEQUENCE + '.html')
pyautogui.hotkey('enter')

This code opens the Save as... window through its keyboard shortcut CTRL+S and then saves the webpage and its assets into the default downloads location by pressing enter. This code also names the file as the sequence in order to give it a unique name, though you could change this for your use case. If needed, you could additionally change the download location through some extra work with the tab and arrow keys.

Tested on Ubuntu 18.10; depending on your OS you may need to modify the key combination sent.

---

Full code, in which I also added conditional waits to improve speed:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.expected_conditions import visibility_of_element_located
from selenium.webdriver.support.ui import WebDriverWait
import pyautogui

URL = 'https://blast.ncbi.nlm.nih.gov/Blast.cgi?PROGRAM=blastx&PAGE_TYPE=BlastSearch&LINK_LOC=blasthome'
SEQUENCE = 'CCTAAACTATAGAAGGACAGCTCAAACACAAAGTTACCTAAACTATAGAAGGACAGCTCAAACACAAAGTTACCTAAACTATAGAAGGACAGCTCAAACACAAAGTTACCTAAACTATAGAAGGACAGCTCAAACACAAAGTTACCTAAACTATAGAAGGACA' #'GAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGAGAAGA'

# open page with selenium
# (first need to download Chrome webdriver, or a firefox webdriver, etc)
driver = webdriver.Chrome()
driver.get(URL)

# enter sequence into the query field and hit 'blast' button to search
seq_query_field = driver.find_element_by_id("seq")
seq_query_field.send_keys(SEQUENCE)

blast_button = driver.find_element_by_id("b1")
blast_button.click()

# wait until results are loaded
WebDriverWait(driver, 60).until(visibility_of_element_located((By.ID, 'grView')))

# open 'Save as...' to save html and assets
pyautogui.hotkey('ctrl', 's')
time.sleep(1)
pyautogui.typewrite(SEQUENCE + '.html')
pyautogui.hotkey('enter')

Download a working local copy of a webpage as a single html file

It certainly can be done. But you’ll have to do couple of simple things manually, since there are no available tools to automate some of the steps.

1. Download the web page using Wget with all dependencies.
2. Copy the contents of linked stylesheets and scripts to main HTML file.
3. Convert images to Base64 data URIs contained in HTML and CSS, then insert them to main HTML file.
4. Minify the edited HTML file.
5. Convert HTML file to Base64 data URI.

Here is an example of a single-page application encoded to Base64 data URI created to demonstrate the concept (copy and paste below code to web browser address bar):

data:text/html;charset=utf-8;base64,PCFkb2N0eXBlIGh0bWw+DQo8aHRtbCBsYW5nPSJlbiI+DQoJPG1ldGEgY2hhcnNldD0idXRmLTgiPg0KCTx0aXRsZT5TaW5nbGUtcGFnZSBBcHBsaWNhdGlvbiBFeGFtcGxlPC90aXRsZT4NCgk8c3R5bGU+DQoJCS8qIENvZGUgZnJvbSBDU1MgZmlsZXMgZ29lcyBoZXJlLiAqLw0KCQlib2R5IHsNCgkJCWZvbnQtZmFtaWx5OiBzYW5zLXNlcmlmOw0KCQl9DQoJCWJ1dHRvbiB7DQoJCQlkaXNwbGF5OiBibG9jaw0KCQl9DQoJPC9zdHlsZT4NCgk8c2NyaXB0Pg0KCQkvLyBDb2RlIGZyb20gLmpzIGZpbGVzIGdvZXMgaGVyZS4gDQoJCWZ1bmN0aW9uIGNoYW5nZVBhcmFncmFwaCgpIHsNCgkJICAgIGRvY3VtZW50LmdldEVsZW1lbnRzQnlUYWdOYW1lKCJwIilbMF0uaW5uZXJIVE1MID0gIkNvbnRlbnQgb2YgcGFyYWdyYXBoIGNoYW5nZWQuIjsNCgkJfQ0KCTwvc2NyaXB0Pg0KCTxib2R5Pg0KCQk8aW1nIHNyYz0iZGF0YTppbWFnZS9wbmc7YmFzZTY0LGlWQk9SdzBLR2dvQUFBQU5TVWhFVWdBQUFVQUFBQUR3QkFNQUFBQ0RBNkJZQUFBQU1GQk1WRVZVVmx1T2o1TC8vLzlrWm1xbXA2bUJnb1dhbTUyeHNyTnpkSGk4dkw3dDdlNzI5dmJHeDhqazVPVFEwZExhMnR2SHNtSDhBQUFDSjBsRVFWUjRBZXpCZ1FBQUFBQ0FvUDJwRjZrQ0FBQUFBQUFBQUFBQUFBQUFBQUFBWUExdElLU2twRERxUUdMQXFBTkhIY2dzSWd3a3d4SUJ6SllCaEJSaEdJYmZiWGZiMWUzcU5vRUU5NVN1bTJuM1Z1SndNSHNRa0FGUVpBVUF4bDA2UU9zRXVNaENDTWNRQVRFWEJhaURBOGdFSUpJQXNKYUFNdmsrVGdrQTVuL2cvN3p2NE9HYitZMmN4djdqVkVaMzRLZG5kNStrTlFudXd1b2NNbDJCOTVZZUZoRHZTVHFmRTAwdldhV3RBcUtrTnNHcndFWUw0S1BrSjNFcW5WanNndTBTWURTdVM5Qk1lQUN3WnFGenJBN0dyZ2x1NHl6cUVuUnlnSkdVdzlzU050ekt5YlNFelNXczF5VzR1WjhEcDY4QXRlR1dXaEJaTVp6TWdhd0J3M0d6SkI3WEpQaFoyN0N1aGd0VzFVSXFRVXY0WXFwa1BiZ21IVUJTazJDaUh0ejA3Y294T1JVdzlTbTdBQXVwRHkvcXVtYlVzY20xcEdkSHZ3RUVTRlpuNTNCZ0VZTGdJUTVOd0o4aHV4MlNZTHZBUVlFS1hvVG81YVQ4ZjhXZkJrWWFnT0FCTEh4U0RvbFVRcllDMytUVUwrZ3JWYk1BZlljM1Z2ZzFjeXoxcWlvTFEvQ0RuZ042QlBGcGVYWlJ6NXB6U0FJUVhBRytBcWlQVVVCbXhYQUprUUlRN0dEa1o5OXp2UFBQejhKYUNJSTZBYTc3ZEI5NDdlOWt0d1NJVjRNUWJPV01VcDkwci9veGRrRjFjb2oyRkFiZHdWaC9zUlZiZUhreVUyQThyYXBVV3NKVVliSUQ3MllQSVZhZzlNRzVvVUJwbGppSlFtVUw0NmZDNWM1UjlldFBlM0FnQUFBQWdBQm83UEZYR0tCcUFBQUFBQUFBQUFBQUFBQUFBQUFBQUxnTmtYVy9TUloxSldBQUFBQUFTVVZPUks1Q1lJST0iIGFsdD0iIj4NCgkJPGgxPlNpbmdsZS1wYWdlIEFwcGxpY2F0aW9uIEV4YW1wbGU8L2gxPg0KCQk8cD5UaGlzIGlzIGFuIGV4YW1wbGUgb2YgYSB3ZWIgYXBwIHRoYXQgaW50ZWdyYXRlcyBIVE1MLCBDU1MsIEphdmFTY3JpcHQsIGFuZCBhbiBpbWFnZSBpbnRvIG9uZSAuaHRtbCBmaWxlIHRoYXQgaXMgZW5jb2RlZCB0byBCYXNlNjQuPC9wPg0KCQk8YnV0dG9uIHR5cGU9ImJ1dHRvbiIgb25jbGljaz0iY2hhbmdlUGFyYWdyYXBoKCkiPkNoYW5nZSBQYXJhZ3JhcGg8L2J1dHRvbj4NCgk8L2JvZHk+DQo8L2h0bWw+

How to save a web page snapshot with all its elements (css, js, images, ...) into one file

HTTrack, -%M

---

Related Topics