Puppeteer Tutorial #3 – Extract SEO Insights and Data Analysis

Web data extraction has evolved into a fundamental practice for gleaning valuable insights, aiding SEO analysis, and collecting crucial information.

Puppeteer, a robust Node.js library, empowers developers to automate web interactions and extract data effortlessly.

In this tutorial, we’ll delve into a practical example of using Puppeteer to extract SEO-related data, links, and images from a webpage, ultimately saving the results to a JSON file.

The final completed source code is given at the end of the article.

1. Introduction to Web Data Extraction

Web data extraction involves automatically retrieving information from websites, and it plays a crucial role in various fields, from search engine optimization to business intelligence. Puppeteer simplifies this process by providing an API to control headless or full browsers, making it an ideal choice for data extraction tasks.

2. Setting Up Puppeteer

Before we dive into the code, ensure you have Node.js installed on your system. Install Puppeteer using the following command:

npm install puppeteer

3. Navigating to a Webpage

Our journey begins by launching a browser instance and navigating to a target webpage.

In this example, we’ll navigate to “http://yahoo.com“.

const puppeteer = require("puppeteer");
const fs = require("fs");

async function run() {
    const browser = await puppeteer.launch({ headless: false });
    const page = await browser.newPage();

    await page.goto("http://yahoo.com");
    // Rest of the code...
}

run();

4. Extracting SEO-Related Data

SEO (Search Engine Optimization) insights are essential for improving a website’s visibility.

Let’s extract the page’s title, meta description, and meta keywords.

// ...
const title = await page.title();
const metaDescription = await page.$eval('meta[name="description"]', (element) => element.textContent);
const metaKeywords = await page.$eval('meta[name="keywords"]', (element) => element.textContent);

5. Extracting Links and Images

Links and images are essential components of web content.

We’ll extract both links and images from the webpage using the $$eval function.

// ...
const links = await page.$$eval("a", (elements) =>
    elements.map((element) => ({
        src: element.href,
        text: element.textContent,
    }))
);

const images = await page.$$eval("img", (elements) =>
    elements.map((element) => ({
        src: element.src,
        alt: element.alt,
    }))
);

6. Storing Data in JSON Format

To organize the extracted data, we’ll store it in a JSON format.

This will make it easily accessible for analysis and further processing.

// ...
const outputData = {
    title,
    metaDescription,
    metaKeywords,
    images,
    links,
    imageCount: images.length,
    linkCount: links.length,
};

const outputJSON = JSON.stringify(outputData);

fs.writeFileSync("output.json", outputJSON);

7. Final Completed Code

The final complete code is shared below for your reference.

const puppeteer = require("puppeteer");
const fs = require("fs");

async function run(){

    const browser = await puppeteer.launch({headless: false});
    const page = await browser.newPage();

    // Navigate to page 
    await page.goto("http://yahoo.com");

    // SEO Related data 
    const title = await page.title();
    const metaDescription = await page.$eval('meta[name="description"]', (element) => element.textContent);
    const metaKeywords = await page.$eval('meta[name="keywords"]', (element)=> element.textContent);

    // Extract Links 
    const links = await page.$$eval("a", (elements) => 
        elements.map((element) => ({
            src: element.href,
            text: element.textContent
        }))
    );

    // Extract Images 
    const images = await page.$$eval("img", (elements) => 
        elements.map((element) => ({
            src: element.src,
            alt: element.alt
        }))
    );

    // Take counts of the images and links
    const imageCount = images.length;
    const linkCount = links.length;

    // Prepare output format
    const outputData = {
        title,
        metaDescription,
        metaKeywords,
        images,
        links,
        imageCount,
        linkCount
    };

    // Convert JSON into a string 
    const outputJSON = JSON.stringify(outputData);

    // Write to file 
    fs.writeFileSync("output.json", outputJSON);

    await browser.close();

}

run();

Puppeteer empowers developers to automate web data extraction and analysis.

In this tutorial, we explored a practical example of using Puppeteer to extract SEO-related data, links, and images from a webpage, and then store the results in a structured JSON format.

With Puppeteer’s capabilities, you can streamline your data collection processes, analyze web content, and gain valuable insights that drive informed decision-making.

As you continue to explore Puppeteer’s functionalities, you’ll unlock endless possibilities for automating web interactions, extracting information, and enhancing your data-driven projects.

Happy scraping and analyzing!