Crawler: RenderJavaScript

Type: boolean | string[] | object

Default:

false

Parameter syntax

renderJavaScript: true|false|[
  'http://www.example.com',
  ...
] | {
  enabled: boolean,
  waitTime?: {
    min?: number,
    max?: number
  },
  adblock?: boolean,
  patterns: [
    'http://www.algolia.com/blog/**'
  ]
}

See code examples

About this parameter

When true, all web pages are rendered with a chrome headless browser. The crawler will use the rendered HTML.

Because rendering JavaScript-based web pages is slower than crawling regular HTML pages, you can apply this setting to a specified list of micromatch URL patterns. These patterns can include negations and wildcards.

With this setting enabled, JavaScript is executed on the web page by a headless Chrome browser. Because a lot of websites have infinite refreshes and updates, the browser has a timeout of 20 seconds by default. Change this value with renderJavascript.waitTime.

Enabling this feature can lead to inconsistent records across crawls, depending on the browser load and the website speed.

Make sure your JavaScript-enabled pages load fast enough for the crawler.

Examples

Copy
{
  renderJavaScript: true,
}

Copy
{
  renderJavaScript: [
    'http://www.mysite.com/dynamic-pages/**',
  ],
}

Copy
{
  renderJavaScript: {
    enabled: true,
    waitTime: {
      min: 7000,
      max: 15000,
    }
    adblock: true,
    patterns: [
      'http://www.mysite.com/dynamic-pages/**',
    ],
  }
}

Parameters

renderJavascript

`enabled`	type: boolean Optional Determines the activation of the JavaScript rendering feature.
`patterns`	type: string[] Optional Based on URL patterns, determines which web pages to render with JavaScript. If unspecified, and `enabled` is set to `true`, all web pages are rendered with JavaScript.
`waitTime`	type: object Optional Determines the shortest and longest wait time before considering a page as rendered.
`adblock`	type: boolean default: false Optional Whether to activate the custom ad blocker or not. It blocks most ads and tracking scripts, but can break some websites.

renderJavascript ➔ waitTime

`min`	type: number default: 0 Optional If the page runs faster than this minimum execution time, the browser remains open at least this long.
`max`	type: number default: 20000 Optional At this maximum execution time threshold, the execution stops and the HTML is processed as is.

saveBackup

initialIndexSettings