jquery/test/runner/command.js

134 lines
4.4 KiB
JavaScript
Raw Normal View History

Tests: migrate testing infrastructure to minimal dependencies This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. **BrowserStack** - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - Browser workers are reused when running isolated modules in the same browser - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. **Selenium** - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close gh-5427
2024-03-05 18:53:39 +00:00
import yargs from "yargs/yargs";
import { browsers } from "./flags/browsers.js";
Tests: migrate testing infrastructure to minimal dependencies This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. **BrowserStack** - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - Browser workers are reused when running isolated modules in the same browser - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. **Selenium** - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close gh-5427
2024-03-05 18:53:39 +00:00
import { getPlan, listBrowsers, stopWorkers } from "./browserstack/api.js";
import { buildBrowserFromString } from "./browserstack/buildBrowserFromString.js";
import { modules } from "./flags/modules.js";
Tests: migrate testing infrastructure to minimal dependencies This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. **BrowserStack** - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - Browser workers are reused when running isolated modules in the same browser - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. **Selenium** - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close gh-5427
2024-03-05 18:53:39 +00:00
import { run } from "./run.js";
const argv = yargs( process.argv.slice( 2 ) )
.version( false )
.strict()
.command( {
command: "[options]",
describe: "Run jQuery tests in a browser"
} )
.option( "module", {
alias: "m",
type: "array",
choices: modules,
description:
"Run tests for a specific module. " +
"Pass multiple modules by repeating the option. " +
"Defaults to all modules."
} )
.option( "browser", {
alias: "b",
type: "array",
choices: browsers,
description:
"Run tests in a specific browser." +
"Pass multiple browsers by repeating the option." +
"If using BrowserStack, specify browsers using --browserstack." +
"Only the basic module is supported on jsdom.",
default: [ "chrome" ]
} )
.option( "headless", {
alias: "h",
type: "boolean",
description:
"Run tests in headless mode. Cannot be used with --debug or --browserstack.",
conflicts: [ "debug", "browserstack" ]
} )
.option( "amd", {
type: "boolean",
description: "Run tests using jQuery's source, which is written with AMD modules."
} )
.option( "concurrency", {
alias: "c",
type: "number",
description:
"Run tests in parallel in multiple browsers. " +
"Defaults to 8 in normal mode. In browserstack mode, " +
"defaults to the maximum available under your BrowserStack plan."
} )
.option( "debug", {
alias: "d",
type: "boolean",
description:
"Leave the browser open for debugging. Cannot be used with --headless.",
conflicts: [ "headless" ]
} )
.option( "retries", {
alias: "r",
type: "number",
description: "Number of times to retry failed tests by refreshing the URL."
} )
.option( "hard-retries", {
type: "number",
description:
"Number of times to retry failed tests by restarting the worker. " +
"This is in addition to the normal retries " +
"and are only used when the normal retries are exhausted."
} )
Tests: migrate testing infrastructure to minimal dependencies This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. **BrowserStack** - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - Browser workers are reused when running isolated modules in the same browser - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. **Selenium** - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close gh-5427
2024-03-05 18:53:39 +00:00
.option( "verbose", {
alias: "v",
type: "boolean",
description: "Log additional information."
} )
.option( "isolate", {
type: "boolean",
description: "Run each module by itself in the test page. This can extend testing time."
} )
.option( "browserstack", {
type: "array",
description:
"Run tests in BrowserStack.\n" +
"Requires BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY environment variables.\n" +
"The value can be empty for the default configuration, or a string in the format of\n" +
"\"browser_[browserVersion | :device]_os_osVersion\" (see --list-browsers).\n" +
"Pass multiple browsers by repeating the option.\n" +
"The --browser option is ignored when --browserstack has a value.\n" +
"Otherwise, the --browser option will be used, " +
"with the latest version/device for that browser, on a matching OS."
} )
.option( "run-id", {
type: "string",
description: "A unique identifier for the run in BrowserStack."
} )
Tests: migrate testing infrastructure to minimal dependencies This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. **BrowserStack** - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - Browser workers are reused when running isolated modules in the same browser - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. **Selenium** - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close gh-5427
2024-03-05 18:53:39 +00:00
.option( "list-browsers", {
type: "string",
description:
"List available BrowserStack browsers and exit.\n" +
"Leave blank to view all browsers or pass " +
"\"browser_[browserVersion | :device]_os_osVersion\" with each parameter " +
"separated by an underscore to filter the list (any can be omitted).\n" +
"\"latest\" can be used in place of \"browserVersion\" to find the latest version.\n" +
"\"latest-n\" can be used to find the nth latest browser version.\n" +
"Use a colon to indicate a device.\n" +
"Examples: \"chrome__windows_10\", \"safari_latest\", " +
"\"Mobile Safari\", \"Android Browser_:Google Pixel 8 Pro\".\n" +
"Use quotes if spaces are necessary."
} )
.option( "stop-workers", {
type: "boolean",
description:
"WARNING: This will stop all BrowserStack workers that may exist and exit," +
"including any workers running from other projects.\n" +
"This can be used as a failsafe when there are too many stray workers."
} )
.option( "browserstack-plan", {
type: "boolean",
description: "Show BrowserStack plan information and exit."
} )
.help().argv;
if ( typeof argv.listBrowsers === "string" ) {
listBrowsers( buildBrowserFromString( argv.listBrowsers ) );
} else if ( argv.stopWorkers ) {
stopWorkers();
} else if ( argv.browserstackPlan ) {
console.log( await getPlan() );
} else {
run( argv );
Tests: migrate testing infrastructure to minimal dependencies This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes: - `qunit` (our trusty unit testing library) - `selenium-webdriver` (for spinning up local browsers) - `express` (for starting a test server and adding middleware) - express middleware includes uses of `body-parser` and `raw-body` - `yargs` (for constructing a CLI with pretty help text) - BrowserStack (for running each of our QUnit modules separately in all of our supported browsers) - `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK) - We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly. **BrowserStack** - automatically retries individual modules in case of test failure(s) - automatically attempts to re-establish broken tunnels - automatically refreshes the page in case a test run has stalled - Browser workers are reused when running isolated modules in the same browser - runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none. - supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version. - cleans up after itself (closes the local tunnel, stops the test server, etc.) - Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables. **Selenium** - supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge - supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`) - Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit. - can run multiple browsers and multiple modules concurrently Other notes: - Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions: - `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x. - `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions. - `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions. - BrowserStack tests are set to run on each `main` branch commit - `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions. - This PR includes a workflow to dispatch BrowserStack runs on-demand - The Node version used for most workflow tests has been upgraded to 20.x - updated supportjQuery to 3.7.1 Run `npm run test:unit -- --help` for CLI documentation Close gh-5427
2024-03-05 18:53:39 +00:00
}