Web Scraping with Python

What Is Web Scraping?

• Web scraping is an automated way to gather large amounts of data from websites. The Internet is one of the largest sources of information in the world. This data is usually in an unstructured HTML format. Web scraping tools convert this unstructured data into structured data

Why Is Web Scraping Used?

Web scraping is used for several important purposes, including:

• Data Collection: Gathering large amounts of data from various websites for analysis and research.

• Price Monitoring: Businesses use web scraping to monitor competitors’ prices and market trends. This helps them adjust their pricing strategies to stay competitive.

• News Aggregation: Collecting news articles from different sources to provide a comprehensive news feed.

• Content Aggregation: It allows for the collection of content from various websites to create comprehensive resources or databases. including aggregating reviews, listings, or other types of content.

• SEO and Marketing: Marketers use web scraping to gather information on keywords, backlinks, and website performance.This data helps in improving SEO strategies and making informed marketing decisions.

Is Web Scraping Legal?

The legality of web scraping depends on the website’s terms of service and the manner in which data is extracted.It is crucial to respect website policies and use the collected data ethically. Web scraping is often permissible for public data but scraping private or sensitive information without consent can be illegal.

Why Is Python Good For Web Scraping?

Python is a popular choice for web scraping due to several advantages:

• Ease of Use: Python’s simple and readable syntax makes it accessible for beginners.
• Large Collection of Libraries:  Python has a huge collection of libraries such as Numpy, Pandas, Matplotlib etc., which provides methods and services for various purposes. Hence, it is suitable for web scraping and for further manipulation of extracted data.
• Community Support: A large community means abundant resources, tutorials, and forums for troubleshooting.
• Versatility: Python is not only good for web scraping but also for data analysis, making it a one-stop-shop for handling scraped data.
• Dynamically typed: In Python, you don’t have to define datatypes for variables, you can directly use the variables wherever required. This saves time and makes your job faster.

Setting Up Your Environment

Before you start scraping the web with Python, you need to set up your development environment.

Install Python

Make sure Python is installed on your system. You can download it from python.org.

Set Up a Virtual Environment

Create a virtual environment with the following command.

Once you have created a virtual environment with the name ‘venv’. Activate it with following command

After activating the virtual environment, install dependencies with the following command. Upon successful installation, you should see these messages.

After installed everything, Now, go inside Project folder & Open the folder in vscode.

Libraries Used for Web Scraping with Python

Here are the main Python libraries used for web scraping,

• Requests: A simple and elegant HTTP library for making requests and fetching web pages.
• BeautifulSoup: A library for parsing HTML and XML documents to easily navigate and extract data.
• Scrapy: A powerful and versatile web scraping framework designed for complex scraping tasks.
• Selenium: A tool for automating web browsers, useful for scraping dynamic content rendered by JavaScript.
• lxml: A high-performance library for processing XML and HTML, known for its speed and ease of use.
• Pandas: A robust data manipulation and analysis library, ideal for cleaning and organizing scraped data.

Scraping Basics with BeautifulSoup

Beautiful Soup is a powerful Python library used for web scraping that extracts data from HTML and XML files by parsing these documents and generating a parse tree, making data extraction straightforward.BeautifulSoup is relatively easy to use and presents itself as a lightweight option for tackling simple scraping tasks with speed.

Fetching Content

The first step in web scraping is to fetch the content of the web page you want to scrape. You can use the requests library to send an HTTP request to the website and get the page content.

Example:

For this example, we are going to scrape Techify Solutions website to extract the data.

1. First Find the URL that you want to scrape. For eg. https://techifysolutions.com/what-we-do/
2. The data is usually nested in tags. So, we inspect the page to see, under which tag the data we want to scrape is nested.
3. Now let’s create the python script beautifulsoup_scraper.py inside web scrapping folder.
4. here is the code for basic example of scrapping data using  beautifulsoup.

Breakdown of the Example Code

• Import Libraries: Import the necessary libraries (requests, BeautifulSoup, and pandas).
• Fetch Content: Use requests.get() to fetch the content of the web page.
• Parse Content: Parse the web page content with BeautifulSoup using the ‘html.parser’.
• Find Relevant Elements: Use BeautifulSoup methods to find the HTML elements that contain the data you want to scrape. In this case, we look for div elements with the class content_main three-col and div elements with the class media-body.
• Extract Data: Extract the title and service details from the relevant elements and store them in a list of dictionaries.
• Save Data: Use pandas to save the extracted data to a CSV file.

Now, Run the code and extract the data

To run the code, use the below command

A file name beautifulsoup_services.csv is created and this file contains the extracted data.

Handling JavaScript-Rendered Content with Selenium

First of all lets understand, What is Selenium

Selenium is a powerful tool for controlling a web browser through a program. It allows you to interact with web pages just as a human user would: clicking buttons, filling out forms, and navigating between pages. This makes it an excellent choice for scraping JavaScript-rendered content, as it can wait for JavaScript to execute and manipulate the DOM.

Example: Using Selenium to Navigate a Website and Extract Information

Let’s walk through an example,

Just copy and paste the code and run that.

so for elaborate above code,

1. Import Necessary libraries
2. Set up the Chrome WebDriver. Ensure you have ChromeDriver installed and in your system’s PATH
3. Navigate to the desired website. In this case, we’re using a fictional website techifysolutions.com
4. Wait for JavaScript to Load the Content. Use time.sleep(10) for same. And then extract the content, and after extracting the data, close the WebDriver.
5. Finally, convert the extracted data into a pandas DataFrame and save it to a CSV file.

Now, Run the code and extract the data

To run the code, use the below command

Well done!!

Conclusion

To summarize,  web scraping with Python allows you to efficiently gather and use data from websites for various purposes, such as market analysis and content aggregation. Python’s easy-to-use libraries and strong community support make it an excellent choice for web scraping. By following ethical guidelines and legal considerations, web scraping can be a valuable tool for obtaining insightful data.