Scrape Web Page Contents

How to scrape all the home page text content of a website?

A quick glance at the website that you're attempting to scrape from makes me suspect that not all content is loaded when sending a simple get request via the requests module. In other words, it seems likely that some components on the site, such as the footer you mentioned, are being loaded asynchronously with Javascript.

If that is the case, you'll probably want to use some sort of automation tool to navigate to the page, wait for it to load and then parse the fully loaded source code. For this, the most common tool would be Selenium. It can be a bit tricky to set up the first time since you'll also need to install a separate webdriver for whatever browser you'd like to use. That said, the last time I set this up it was pretty easy. Here's a rough example of what this might look like for you (once you've got Selenium properly set up):

from bs4 import BeautifulSoup
from selenium import webdriver

import time

driver = webdriver.Firefox(executable_path='/your/path/to/geckodriver')
driver.get('http://www.traiteurcheminfaisant.com')
time.sleep(2)

source = driver.page_source
soup = BeautifulSoup(source, 'html.parser')

full_text = soup.find_all()

print(full_text)

Scrape a webpage contents after ::before tag using Python

The page is dynamically loaded using javascript, so you have to use selenium:

from selenium import webdriver
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.get('https://knappily.com/')

time.sleep(3)

st = time.time()

while True:
    if time.time() - st <= 120: #Keeps loading more articles for 2 mins. You can increase the time if u want.
        try:
            WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[@id="buttons"]'))).click()
            time.sleep(1)
        except:
            break
    else:
        break

main_art = driver.find_element_by_class_name('impr-content').find_element_by_xpath('.//h1').text

print(f"{'-'*80}\nMain Article:\n{'-'*80}\n{main_art}\n{'-'*80}\nOther Articles:")

articles = driver.find_elements_by_class_name('article-content')

for article in articles:
    print('-'*80)
    print(article.find_element_by_xpath('.//h3').text)

Output:

--------------------------------------------------------------------------------
Main Article:
--------------------------------------------------------------------------------
Coronavirus has exposed India’s digital divide in education
--------------------------------------------------------------------------------
Other Articles:
--------------------------------------------------------------------------------
Live with no doubts
--------------------------------------------------------------------------------
October 30, 1917: British government approves Balfour Declaration
--------------------------------------------------------------------------------
Delhi is choking again
--------------------------------------------------------------------------------
US Election: the final countdown
--------------------------------------------------------------------------------
At least I tried.
--------------------------------------------------------------------------------
Scam 1992: The real Harshad Mehta story
--------------------------------------------------------------------------------
October 29, 1969: The Internet is born
--------------------------------------------------------------------------------
Terrorist attack in Nice as French cartoon row escalates
--------------------------------------------------------------------------------
Manual scavenging: India must now end the evil
--------------------------------------------------------------------------------
In solitude I find my answers.
--------------------------------------------------------------------------------
October 28, 1929: Black Monday begins the Great Depression
--------------------------------------------------------------------------------
Digital payments are skyrocketing in India
--------------------------------------------------------------------------------
COVID antibodies drop over time, but that’s alright
--------------------------------------------------------------------------------
Detach from what destroys you.
--------------------------------------------------------------------------------
US and India step up defence cooperation
--------------------------------------------------------------------------------
October 27, 1913: Woodrow Wilson swears by pacifism
--------------------------------------------------------------------------------
Cyber Crimes are rising in India
--------------------------------------------------------------------------------
France targets ‘radical Islam’, faces Muslim backlash
--------------------------------------------------------------------------------
You'll never know 'til you go.
--------------------------------------------------------------------------------
The never-ending crisis of IL&FS
--------------------------------------------------------------------------------
October 26, 1858: Theodore Roosevelt is born
--------------------------------------------------------------------------------
Reducing inequality: Chile shows how it’s done
--------------------------------------------------------------------------------
The Indus Valley Tech continues to amaze us
--------------------------------------------------------------------------------
And into the forest I go...
--------------------------------------------------------------------------------
NASA and asteroid Bennu: the problem of plenty
--------------------------------------------------------------------------------
October 25, 1760: George III becomes King
--------------------------------------------------------------------------------
Urban Floods: the deluge of Hyderabad
--------------------------------------------------------------------------------
I doubt, therefore I am.
--------------------------------------------------------------------------------
India: where pollution is deadlier than the pandemic
--------------------------------------------------------------------------------
October 24, 1945: The United Nations is founded
--------------------------------------------------------------------------------
Europe’s hospitals are filling up again
--------------------------------------------------------------------------------
Let go...or be dragged.
--------------------------------------------------------------------------------
Pakistan remains on the FATF grey list
--------------------------------------------------------------------------------
October 23, 1917: Lenin calls for October Revolution
--------------------------------------------------------------------------------
Coronavirus makes America sick again
--------------------------------------------------------------------------------
Courage, dear heart.
--------------------------------------------------------------------------------
Lagos is in total chaos
--------------------------------------------------------------------------------
October 22, 1859: Spain declares war on Morocco
--------------------------------------------------------------------------------
Mysterious deaths in South Korea raise flu vaccine safety concerns
--------------------------------------------------------------------------------
One life. Read more.
--------------------------------------------------------------------------------
The curious rise of ‘Robin Hood’ hackers
--------------------------------------------------------------------------------
October 21, 1854: Florence Nightingale leaves for the Crimea
--------------------------------------------------------------------------------
The United States sues Google
--------------------------------------------------------------------------------
Time flies, we don't.
--------------------------------------------------------------------------------
Neanderthals became ‘feminine’ by mating with humans
--------------------------------------------------------------------------------
October 20, 1911: Roald Amundsen sets out to South Pole
--------------------------------------------------------------------------------
Biden is winning, but Trump may still win
--------------------------------------------------------------------------------
Courage is a daily decision.
--------------------------------------------------------------------------------
Coronavirus cases cross 40 million
--------------------------------------------------------------------------------
October 19, 1910: Subrahmanyan Chandrasekhar is born
--------------------------------------------------------------------------------
Come grow old with me...
--------------------------------------------------------------------------------
Chinese economy continues to grow
--------------------------------------------------------------------------------
Antarctica’s Emperors have no future
--------------------------------------------------------------------------------
October 18, 1954: Transistor Radio is born
--------------------------------------------------------------------------------
The pandemic philosophy of Albert Camus
--------------------------------------------------------------------------------
One big adventure. Or nothing.
--------------------------------------------------------------------------------
India vs. Bangladesh, per capita basis
--------------------------------------------------------------------------------
October 17, 1973: The Oil Crisis begins
--------------------------------------------------------------------------------
French teacher beheaded for “insulting” Islam
--------------------------------------------------------------------------------
Collect memories, not things.
--------------------------------------------------------------------------------
The United Kingdom prepares for no-deal Brexit
--------------------------------------------------------------------------------
October 16, 1923: The Walt Disney Company is founded
--------------------------------------------------------------------------------
Three-finger salute: Thais protest for democracy
--------------------------------------------------------------------------------
But not every man really lives.
--------------------------------------------------------------------------------
Digital Inequality and the Coronavirus
--------------------------------------------------------------------------------
October 15, 1917: Mata Hari is executed
--------------------------------------------------------------------------------
Jacinda Ardern prefers to earn her victory
--------------------------------------------------------------------------------
One life. Complain less.
--------------------------------------------------------------------------------
Resignation Syndrome: And I hereby withdraw from the world
--------------------------------------------------------------------------------
October 14, 1933: Germany quits the League of Nations
--------------------------------------------------------------------------------
Europe leads the world in COVID cases, again
--------------------------------------------------------------------------------
Quiet men are deep & dangerous
--------------------------------------------------------------------------------
2020: the year MBA went online
--------------------------------------------------------------------------------
October 13, 1792: The construction of ‘The White House’ begins
--------------------------------------------------------------------------------
Donald Trump’s last great fight is on
--------------------------------------------------------------------------------
It is what it is.
--------------------------------------------------------------------------------
October 12, 1999: Pervez Musharraf scripts a coup in Pakistan
--------------------------------------------------------------------------------
Nobel Memorial Prize in Economics goes to auction theorists
--------------------------------------------------------------------------------
Coronavirus can survive for 28 days in cold and dark
--------------------------------------------------------------------------------
Be. Here. Now.
--------------------------------------------------------------------------------
Rafael Nadal wins French Open to tie Grand Slam record
--------------------------------------------------------------------------------
October 11, 1987: Operation Pawan begins in Sri Lanka
--------------------------------------------------------------------------------
North Korea flaunts monster missiles and zero COVID cases
--------------------------------------------------------------------------------
To seek nothing takes time
--------------------------------------------------------------------------------
Women political leaders don’t exist in China
--------------------------------------------------------------------------------
October 10, 1967: The Outer Space Treaty comes into force
--------------------------------------------------------------------------------
The speed limit of sound has been found
--------------------------------------------------------------------------------
Good travelers never arrive.
--------------------------------------------------------------------------------
October 9, 1941: The Manhattan Project begins
--------------------------------------------------------------------------------
World Food Program wins Nobel Peace Prize 2020
--------------------------------------------------------------------------------
Is China winning the vaccine race?
--------------------------------------------------------------------------------
Rise above the crowd.
--------------------------------------------------------------------------------
Louise Glück wins Nobel Prize in Literature
--------------------------------------------------------------------------------
October 8, 1912: First Balkan War begins
--------------------------------------------------------------------------------
Coronavirus: making the poor poorer
--------------------------------------------------------------------------------
Your library is your paradise.
--------------------------------------------------------------------------------
Gene editing pioneers win the Chemistry Nobel
--------------------------------------------------------------------------------
October 7, 1952: Vladimir Putin is born
--------------------------------------------------------------------------------
Facebook bans conspiracy group QAnon
--------------------------------------------------------------------------------
When I do good, I feel good
--------------------------------------------------------------------------------
Black Hole hunters win Nobel Prize in Physics
--------------------------------------------------------------------------------
October 6, 1973: Yom Kippur War begins
--------------------------------------------------------------------------------
Donald Trump is back in action
--------------------------------------------------------------------------------
Believing is Seeing.
--------------------------------------------------------------------------------
Hepatitis C virus hunters win Nobel Prize in Medicine
--------------------------------------------------------------------------------
October 5, 1910: Portugal becomes a Republic
--------------------------------------------------------------------------------
Europe is breaking coronavirus records again
--------------------------------------------------------------------------------
Keep it simply, silly.
--------------------------------------------------------------------------------
The President has been put on steroids
--------------------------------------------------------------------------------
October 4, 2006: WikiLeaks is founded
--------------------------------------------------------------------------------
Remdesivir: the President’s drug for COVID-19
--------------------------------------------------------------------------------
Write the first true word you know
--------------------------------------------------------------------------------
No place for nuclear waste in Germany
--------------------------------------------------------------------------------
October 3, 1952: Britain explodes an atomic bomb
--------------------------------------------------------------------------------
Donald Trump has been hospitalized
--------------------------------------------------------------------------------
Seek what's seeking you.
--------------------------------------------------------------------------------
Gandhi: because right is right and wrong is wrong
--------------------------------------------------------------------------------
October 2, 1904: Lal Bahadur Shastri is born
--------------------------------------------------------------------------------
Donald Trump tests positive for coronavirus
--------------------------------------------------------------------------------
On and on it goes...
--------------------------------------------------------------------------------
EU sues Britain over breaching Brexit deal
--------------------------------------------------------------------------------
October 1, 1988: Gorbachev is the head of Supreme Soviet
--------------------------------------------------------------------------------
Japan fears ‘suicide contagion’ after celebrity suicides
--------------------------------------------------------------------------------
Timothy Ray Brown: the ‘Berlin patient’ who conquered HIV
--------------------------------------------------------------------------------
What the hell, leap.
--------------------------------------------------------------------------------
September 30, 1938: Munich Agreement is signed to appease Hitler
--------------------------------------------------------------------------------
The never-ending Babri demolition case has delivered its first verdict
--------------------------------------------------------------------------------
First Presidential Debate: what a hot mess
--------------------------------------------------------------------------------
The absurd is the first truth.
--------------------------------------------------------------------------------
The BTS IPO is a smash hit
--------------------------------------------------------------------------------
September 29, 1913: Rudolf Diesel vanishes in thin air
--------------------------------------------------------------------------------
Coronavirus: one million deaths is not just a statistic
--------------------------------------------------------------------------------
Conviction needs Courage.
--------------------------------------------------------------------------------
Can gene therapy cure glaucoma?
--------------------------------------------------------------------------------
September 28, 1970: Gamal Abdel Nasser is no more
--------------------------------------------------------------------------------
Donald Trump’ tax returns have been outed
--------------------------------------------------------------------------------
Accept it. Endure it.
--------------------------------------------------------------------------------
September 27, 1940: Axis powers sign the Tripartite Pact
--------------------------------------------------------------------------------
Armenia and Azerbaijan are close to war
--------------------------------------------------------------------------------
India must prepare for a long coronavirus fight
--------------------------------------------------------------------------------
You got this.
--------------------------------------------------------------------------------
Coulrophobia: the irrational fear of clowns
--------------------------------------------------------------------------------
Children can fight coronavirus better
--------------------------------------------------------------------------------
September 26, 1820: Ishwar Chandra ‘Vidyasagar’ is born
--------------------------------------------------------------------------------
The battle for the soul of the Supreme Court
--------------------------------------------------------------------------------
Stop thinking. Start living.
--------------------------------------------------------------------------------
September 25, 1969: Organization of Islamic Cooperation is founded
--------------------------------------------------------------------------------
Vodafone wins international arbitration against India
--------------------------------------------------------------------------------
Divided nations clash at the United Nations
--------------------------------------------------------------------------------
When I do good, I feel good.
--------------------------------------------------------------------------------
September 24, 2014: Mangalyaan reaches the orbit of Mars
--------------------------------------------------------------------------------
Coronavirus may have a dominant mutant strain
--------------------------------------------------------------------------------
Will the silent majority rise again for Trump?
--------------------------------------------------------------------------------
Don't be a Maybe.
--------------------------------------------------------------------------------
September 23, 1908: Ramdhari Singh ‘Dinkar’ is born
--------------------------------------------------------------------------------
The unbearable tragedy of whale beaching
--------------------------------------------------------------------------------
US coronavirus death toll crosses 200,000
--------------------------------------------------------------------------------
Find comfort in the chaos
--------------------------------------------------------------------------------
Resilience: the obstacle is the path
--------------------------------------------------------------------------------
September 22, 1980: Iraq invades Iran
--------------------------------------------------------------------------------
Vaccine: arriving in November…or Never?
--------------------------------------------------------------------------------
Be a happy rebel.
--------------------------------------------------------------------------------
Who really owns TikTok now?
--------------------------------------------------------------------------------
September 21, 1743: Maharaja Jai Singh II is no more
--------------------------------------------------------------------------------
India debates agriculture marketing reforms
--------------------------------------------------------------------------------
It is what it is.
--------------------------------------------------------------------------------
We are overusing this planet
--------------------------------------------------------------------------------
September 20, 1933: Annie Besant is no more
--------------------------------------------------------------------------------
Tadej Pogacar: the impossible Tour de France winner
--------------------------------------------------------------------------------
The mountains are calling...
--------------------------------------------------------------------------------
India's contribution to the world: a big ZERO
--------------------------------------------------------------------------------
Europe prepares for a viral winter
--------------------------------------------------------------------------------
September 19, 1946: Winston Churchill proposes the United States of Europe
--------------------------------------------------------------------------------
Taiwan learns to play with fire
--------------------------------------------------------------------------------
Rise, little man.
--------------------------------------------------------------------------------
September 18, 2001: First Anthrax letters are mailed
--------------------------------------------------------------------------------
India’s weird relationship with weed
--------------------------------------------------------------------------------
Talk Less. Do More.
--------------------------------------------------------------------------------
September 17, 2011: Occupy Wall Street begins, the 99 percent rise
--------------------------------------------------------------------------------
Everyone is a prepper in 2020
--------------------------------------------------------------------------------
Be fearlessly you.
--------------------------------------------------------------------------------
Yoshihide Suga: the self-made Prime Minister of Japan
--------------------------------------------------------------------------------
September 16, 1987: Montreal Protocol is signed
--------------------------------------------------------------------------------
The emptiness of Trump’s Middle East ‘peace’ deal
--------------------------------------------------------------------------------
Let go...or be dragged.
--------------------------------------------------------------------------------
Do we need animals to make meat?
--------------------------------------------------------------------------------
September 15, 1861: M. Visvesvaraya is born
--------------------------------------------------------------------------------
Pakistan rises against ‘rape culture’
--------------------------------------------------------------------------------
It's not What, but How you do it
--------------------------------------------------------------------------------
Are they on Venus?
--------------------------------------------------------------------------------
September 14, 1960: OPEC is founded
--------------------------------------------------------------------------------
This virus just keeps breaking records
--------------------------------------------------------------------------------
He who is brave is free.
--------------------------------------------------------------------------------
Dominic Thiem wins US Open with a stunning comeback
--------------------------------------------------------------------------------
September 13, 1948: Indian Army enters Hyderabad
--------------------------------------------------------------------------------
Naomi Osaka wins US Open and rallies for social justice
--------------------------------------------------------------------------------
The fight is everything.
--------------------------------------------------------------------------------
September 12, 1990: Two Plus Four Agreement to reunify Germany is signed
--------------------------------------------------------------------------------
Facultative parthenogenesis: the strange world of virgin births
--------------------------------------------------------------------------------
Afghan peace talks begin to end the forever war
--------------------------------------------------------------------------------
If it's endurable, then endure it.
--------------------------------------------------------------------------------
The curious case of India’s low COVID-19 mortality rate
--------------------------------------------------------------------------------
September 11, 1965: Bashar al-Assad is born
--------------------------------------------------------------------------------
Shall we reopen the schools?
--------------------------------------------------------------------------------
One life. Live more.
--------------------------------------------------------------------------------
Brexit gets even more chaotic
--------------------------------------------------------------------------------
September 10, 2008: Acceleration begins at Large Hadron Collider
--------------------------------------------------------------------------------
Coronavirus did not stop Climate Change
--------------------------------------------------------------------------------
Let go...or be dragged.
--------------------------------------------------------------------------------
Caster Semenya: too fast to be a woman
--------------------------------------------------------------------------------
September 9, 1993: PLO recognizes Israel
--------------------------------------------------------------------------------
Oxford vaccine suffers a major setback
--------------------------------------------------------------------------------
Calm is a superpower.
--------------------------------------------------------------------------------
Are we inside a Matrioshka Brain?
--------------------------------------------------------------------------------
September 8, 1945: Hideki Tojo attempts suicide
--------------------------------------------------------------------------------
Remote workers of the world, unplug
--------------------------------------------------------------------------------
Let the dead leaves drop
--------------------------------------------------------------------------------
India: the economy can’t wait, the virus won’t stop
--------------------------------------------------------------------------------
September 7, 1923: The INTERPOL is founded
--------------------------------------------------------------------------------
The strange disqualification of Novak Djokovic
--------------------------------------------------------------------------------
Even anxieties have anxieties today
--------------------------------------------------------------------------------
Someone is mutilating horses in France
--------------------------------------------------------------------------------

...

--------------------------------------------------------------------------------
May 16, 1966: Cultural Revolution begins in China
--------------------------------------------------------------------------------
Coronavirus: 4.5 million cases, 300,000 deaths
--------------------------------------------------------------------------------
Above all, be kind.

>>> len(articles)
714

Web scraping - how to identify main content on a webpage

There's no way to do this that's guaranteed to work, but one strategy you might use is to try to find the element with the most visible text inside of it.

How to print and display all results of web scraper?

The .find_all you used in

events_elems = results.find_all('ul', class_='searchResults')

did not capture each individual rows from the website, i.e., you need to tighten the search criteria.

---

Related Topics