Automate Your Morning: Build a News Scraper with Python

Hello there, friend! I am so glad you are here. Today, we are going to dive into the world of Python Automation. If you have ever felt like you are doing the same thing over and over again on your computer, you are in the right place. We are going to build something that does the work for you!

The Real-World Scenario

Imagine you are a busy Market Researcher. Every single morning, you have to visit five different news websites to see what the top trending stories are. You copy the headlines and paste them into a spreadsheet. It takes 20 minutes of your time every day. That’s over 120 hours a year spent just copying and pasting!

The Analogy: Think of this script like a “Digital Personal Assistant.” Instead of you walking to the newsstand, picking up the paper, and highlighting the headlines with a pen, your assistant flies there in a split second, snaps a photo of the front page, and neatly types out the titles on your desk before you even finish your first sip of coffee.

The Solution

We are going to write a Python script that visits a popular news site (Hacker News), looks for the titles of the top articles, and saves them into a CSV file (which you can open in Excel). This is a classic example of “Web Scraping,” a powerful way to automate data collection.

Prerequisites

Don’t worry, setting this up is super easy! You just need to install two “libraries” (tools that help Python talk to websites). Open your terminal or command prompt and type:

pip install requests beautifulsoup4

The Code

Here is your copy-paste ready script. I have made sure every line is explained so you know exactly what is happening. Let’s look how cool this is!

"""
-----------------------------------------------------------------------
Authors: Sharanam & Vaishali Shah
-----------------------------------------------------------------------
Recipe: Automated Web Scraper for Daily Headlines
-----------------------------------------------------------------------
"""

import requests  # This is our 'Mailman'—it fetches the webpage for us.
from bs4 import BeautifulSoup  # This is our 'Organizer'—it makes sense of the HTML.
import csv  # This is our 'Secretary'—it writes the data into a file.

def scrape_top_news():
    # 1. The address of the website we want to visit
    url = 'https://news.ycombinator.com/'
    
    print("Starting the automation... please wait a second!")
    
    try:
        # 2. Ask the Mailman to get the page content
        response = requests.get(url)
        response.raise_for_status() # This checks if the website is actually working
        
        # 3. Hand the messy webpage code to the Organizer
        soup = BeautifulSoup(response.text, 'html.parser')
        
        # 4. Find all the story titles (on this site, they are in 'span' tags with class 'titleline')
        # We use a list to store our findings
        headlines_list = []
        stories = soup.find_all('span', class='titleline')
        
        for story in stories:
            title = story.text
            link = story.find('a')['href']
            headlines_list.append([title, link])
        
        # 5. Tell the Secretary to save this into a CSV file
        with open('daily_news_report.csv', 'w', newline='', encoding='utf-8') as file:
            writer = csv.writer(file)
            # Add a header row
            writer.writerow(['Headline', 'Link'])
            # Add all the stories we found
            writer.writerows(headlines_list)
            
        print("Success! Your 'daily_news_report.csv' is ready for you to view.")
        
    except Exception as e:
        print(f"Oops! Something went wrong: {e}")

# This tells Python to run the function we just made
if __name__ == "__main__":
    scrape_top_news()

Code Walkthrough

Let’s break it down together, line by line:

• import requests: We are bringing in a tool that allows Python to go out to the internet and “request” a website’s data.
• BeautifulSoup: Websites are written in a language called HTML, which looks like a mess to humans. This tool “cleans it up” so we can search for specific things, like titles.
• requests.get(url): This is like typing the URL into your browser and hitting Enter.
• soup.find_all: We are telling Python, “Go through the whole page and find every single title.” It’s like using ‘Find’ (Ctrl+F) on a document.
• csv.writer: This creates a file that Excel can read. We give it a header so you know which column is the headline and which is the link.

Sample Output

When you run this script, you will see this in your console:

Starting the automation... please wait a second!
Success! Your 'daily_news_report.csv' is ready for you to view.

And in your folder, you will find a new file named daily_news_report.csv. If you open it, you’ll see two neat columns with the latest tech news!

Conclusion

Look at that! You just built a working automation tool. You’ve moved from manually browsing to having a script do the “boring stuff” for you. Don’t worry if it felt like a lot at first—the more you play with it, the easier it gets. Try changing the URL to another site or changing the filename. Stay curious and keep coding!