algolia crawler
Manage your Algolia crawlers.
algolia crawler crawl
algolia crawler crawl <crawler_id> --urls <url>... [flags]
Crawl specific URLs.
Examples
1
2
3
4
5
6
7
8
9
# Crawl the URLs "https://www.example.com" and "https://www.example2.com/" for the crawler with the ID "my-crawler"
$ algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/
# Crawl the URLs "https://www.example.com" and "https://www.example2.com/" for the crawler with the ID "my-crawler" and save them in the configuration
$ algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/ --save
# Crawl the URLs "https://www.example.com" and "https://www.example2.com/" for the crawler with the ID "my-crawler" and don't save them in the configuration
$ algolia crawler crawl my-crawler --urls https://www.example.com,https://www.example2.com/ --save=false
Flags
-
-s,--save -
When true, the URLs are added to your %[1]sextraUrls%[1]s (unless present in %[1]sstartUrls%[1]s or %[1]ssitemaps%[1]s).
When false, the URLs aren’t added.
When unspecified, the URLs are added to your %[1]sextraUrls%[1]s (unless present in %[1]sstartUrls%[1]s or %[1]ssitemaps%[1]s or they weren’t indexed during the preceding reindex). -
-u,--urls -
The URLs to crawl (maximum 50).
algolia crawler create
algolia crawler create <name> -F <file> [flags]
Create a crawler.
Examples
1
2
3
4
5
6
# Create a crawler named "my-crawler" with the configuration in the file "config.json"
$ algolia crawler create my-crawler -F config.json
# Create a crawler from another crawler's configuration
$ algolia crawler get another-crawler --config-only | algolia crawler create my-crawler -F -
Flags
-
-F,--file -
Path to the configuration file (use “-“ to read from standard input)
algolia crawler get
algolia crawler get <crawler_id> [flags]
Get a crawler.
Examples
1
2
3
4
5
6
# Get the crawler with the ID "my-crawler"
$ algolia crawler get my-crawler
# Get the crawler with the ID "my-crawler" and display only its configuration
$ algolia crawler get my-crawler --config-only
Flags
-
-c,--config-only -
Display only the crawler configuration
algolia crawler list
algolia crawler list [flags]
List crawlers.
Examples
1
2
3
4
5
6
7
8
9
# List all crawlers
$ algolia crawler list
# List crawlers with the name "my-crawler"
$ algolia crawler list --name my-crawler
# List crawlers with the appID "my-app-id"
$ algolia crawler list --app-id my-app-id
Flags
-
--app-id -
Filter by appID
-
--name -
Filter by name
Output formatting flags
-
--allow-missing-template-keys -
If true, ignore errors in templates due to missing fields or map keys. This only applies to golang and jsonpath output formats.
-
-o,--output -
Output format. One of: (json, jsonpath, jsonpath-as-json, jsonpath-file).
-
--template -
Template string or path to a template file to use when –output=jsonpath, –output=jsonpath-file.
algolia crawler pause
algolia crawler pause <crawler_id>... [flags]
Pause one or multiple crawlers.
Examples
1
2
3
4
5
6
# Pause the crawler with the ID "my-crawler"
$ algolia crawler pause my-crawler
# Pause the crawlers with the IDs "my-crawler-1" and "my-crawler-2"
$ algolia crawler pause my-crawler-1 my-crawler-2
algolia crawler reindex
algolia crawler reindex <crawler_id>... [flags]
Reindexes the specified crawlers.
Examples
1
2
3
4
5
6
# Reindex the crawler with the ID "my-crawler"
$ algolia crawler reindex my-crawler
# Reindex the crawlers with the IDs "my-crawler-1" and "my-crawler-2"
$ algolia crawler reindex my-crawler-1 my-crawler-2
algolia crawler run
algolia crawler run <crawler_id> [flags]
Start or resume a crawler.
Examples
1
2
3
# Run the crawler with the ID "my-crawler"
$ algolia crawler run my-crawler
algolia crawler stats
algolia crawler stats <crawler_id> [flags]
Get statistics about a crawler.
Examples
1
2
3
# Get statistics about the crawler with the ID "my-crawler"
$ algolia crawler stats my-crawler
Output formatting flags
-
--allow-missing-template-keys -
If true, ignore errors in templates due to missing fields or map keys. This only applies to golang and jsonpath output formats.
-
-o,--output -
Output format. One of: (json, jsonpath, jsonpath-as-json, jsonpath-file).
-
--template -
Template string or path to a template file to use when –output=jsonpath, –output=jsonpath-file.
algolia crawler test
algolia crawler test <crawler_id> --url <url> [-F <file>] [flags]
Tests a URL with the crawler’s configuration and shows the extracted records.
Examples
1
2
3
4
5
6
# Test the URL "https://www.example.com" against the crawler with the ID "my-crawler"
$ algolia crawler test my-crawler --url https://www.example.com
# Test the URL "https://www.example.com" against the crawler with the ID "my-crawler" and override the configuration with the file "config.json"
$ algolia crawler test my-crawler --url https://www.example.com -F config.json
Flags
-
-F,--config -
The configuration file to use to override the crawler’s configuration. (use “-“ to read from standard input)
-
-u,--url -
The URL to test.
algolia crawler unblock
algolia crawler unblock <crawler_id> [flags]
Unblock a crawler.
Examples
1
2
3
# Unblock the crawler with the ID "my-crawler"
$ algolia crawler unblock my-crawler
