Iterate Over Multiple Payloads And Take Multiple Screenshots With Puppeteer Aws Lambda

I am currently using the following Puppeteer AWS Lambda Layer to scrape 30 URLs and create and save screenshots in S3. At the moment, I send 30 individual payloads therefore runnin

Solution 1:

I tried to replicate the issue and modify the code to use loop.

While working on this issue, I found several things worth pointing out:

• the lambda requires a lot of RAM (at least 1GB in my test, but more better). Using small amount of RAM lead to failures.
• lambda timeout must be large to handle a number of URLs to screenshot.
• your img from the JSON payload is not used at all. I did not modify this behavior, as I don't know if this is by design or not.
• similar errors to yours were observed when running async for loop and/or not closing pages opened.
• modified return value to output an array of s3 urls.
• undefied URL

Modified code

Here is the modified code that worked in my tests using nodejs12.x runtime:

// src/capture.jsvarURL = require('url').URL;

// this module will be provided by the layerconst chromeLambda = require("chrome-aws-lambda");

// aws-sdk is always preinstalled in AWS Lambda in all Node.js runtimesconst S3Client = require("aws-sdk/clients/s3");

process.setMaxListeners(0) // <== Important line - Fix MaxListerners Error// create an S3 clientconst s3 = newS3Client({ region: process.env.S3_REGION });

// default browser viewport sizeconst defaultViewport = {
  width: 1920,
  height: 1080
};

// here starts our function!exports.handler = async event => {

  // launch a headless browserconst browser = await chromeLambda.puppeteer.launch({
    args: chromeLambda.args,
    executablePath: await chromeLambda.executablePath,
    defaultViewport
  });
  
  const s3_urls = [];

  for (const e of event) {
    console.log(e);

    console.log("Event URL string is ", e.url)

    const url = e.url;
    const domain = (newURL(url)).hostname.replace('www.', '');

    // open a new tabconst page = await browser.newPage();

    // navigate to the pageawait page.goto(e.url);

    // take a screenshotconst buffer = await page.screenshot()

    // upload the image using the current timestamp as filenameconst result = await s3
      .upload({
        Bucket: process.env.S3_BUCKET,
        Key: domain + `.png`,
        Body: buffer,
        ContentType: "image/png",
        ACL: "public-read"
      })
      .promise();
      
      await page.close();
      
      s3_urls.push({ url: result.Location });
      
  }
  
  await browser.close();

  // return the uploaded image urlreturn s3_urls;
};         

Example playload

[{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/gavurin.com.png","url":"https://gavurin.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/google.com.png","url":"https://google.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/amazon.com","url":"https://www.amazon.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/stackoverflow.com","url":"https://stackoverflow.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/duckduckgo.com","url":"https://duckduckgo.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/docs.aws.amazon.com","url":"https://docs.aws.amazon.com/lambda/latest/dg/gettingstarted-features.html"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/github.com","url":"https://github.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/github.com/shelfio/chrome-aws-lambda-layer","url":"https://github.com/shelfio/chrome-aws-lambda-layer"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/gwww.youtube.com","url":"https://www.youtube.com"},{"img":"https://s3screenshotbucket-useast1v5.s3.amazonaws.com/w3docs.com","url":"https://www.w3docs.com"}]

Example output in S3