This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes:
- `qunit` (our trusty unit testing library)
- `selenium-webdriver` (for spinning up local browsers)
- `express` (for starting a test server and adding middleware)
- express middleware includes uses of `body-parser` and `raw-body`
- `yargs` (for constructing a CLI with pretty help text)
- BrowserStack (for running each of our QUnit modules separately in all of our supported browsers)
- `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK)
- We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly.
## BrowserStack Runner
- automatically retries individual modules in case of test failure(s)
- automatically attempts to re-establish broken tunnels
- automatically refreshes the page in case a test run has stalled
- runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none.
- supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified.
- cleans up after itself (closes the local tunnel, stops the test server, etc.)
- Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables.
## Selenium Runner
- supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge
- supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`)
- Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit.
- can run multiple browsers and multiple modules concurrently
Other notes:
- Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions:
- `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x.
- `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions.
- `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions.
- BrowserStack tests are set to run on each `main` branch commit
- `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions.
- This PR includes a workflow to dispatch BrowserStack runs on-demand
- The Node version used for most workflow tests has been upgraded to 20.x
- updated supportjQuery to 3.7.1
Run `npm run test:unit -- --help` for CLI documentation
Close gh-5418
Use Prettier 3.1.0 to reformat the Yaml files. This makes their format identical
to the one used on `3.x-stable`, making for much easier cherry-picks.
The main difference is the list under `steps:` was not indented while all other
lists were.
Closes gh-5364
Build was already happening in scripts like `test:browser` but those scripts
were missing `pretest`, meaning that running `npm install && npm test:browser`
may have failed if `pretest` wasn't run before or if its results were out of
date.
Even worse, with such stale data some tests may erroneously succeed.
This also removes a separate `pretest` step from GitHub Actions as it's no
longer needed.
Closes gh-5338
Updated tasks include:
- lint
- npmcopy
- build, minify, and process for distribution.
- new custom build command using yargs
- compare size of minified/gzip built files
- pretest scripts, including qunit-fixture, babel transpilation, and npmcopy
- node smoke tests
- promises aplus tests
- new watch task using `rollup.watch` directly
Also:
- upgraded husky and added the new lint command
- updated lint config to use new "flat" config format. See https://eslint.org/docs/latest/use/configure/configuration-files-new
- Temporarily disabled one lint rule until flat config is supported by eslint-plugin-import. See https://github.com/import-js/eslint-plugin-import/issues/2556
- committed package-lock.json
- updated all test scripts to use the new build
- added an express test server that uses middleware-mockserver (this can be used to run tests without karma)
- build-all-variants is now build:all
Close gh-5318
With this change, jQuery build no longer generates the `amd` directory with
AMD modules transpiled from source `src` ECMAScript Modules. To use individual
jQuery modules from source, ESM is now required.
Note that this DOES NOT affect the main `"jquery"` AMD module defined by built
jQuery files; those remain supported.
Closes gh-5276
PR gh-5190 added support for running tests on Playwright WebKit
in CI. For efficiency reasons, Playwright dependencies are only
installed for the `test:browser` npm script. However, that same
script is also used for Firefox ESR testing.
This change makes Playwright dependencies installed only for cases
where `WebKitHeadless` exists on the list of tested browsers.
Closes gh-5204
Ref gh-5190
So far, we've been running browser tests on GitHub Actions in Chrome
and Firefox. Regular Safari is not available in GitHub Actions but
Playwright WebKit comes close to a dev version of Safari.
With this change, our GitHub CI & local test runs will invoke tests on
all actively developed browser engines on all PRs.
Also, our GitHub Actions browser tests are now running on Node.js 18.
Detection of the Playwright WebKit browser in support unit tests is done
by checking if the `test_browser` query parameter is set to `"Playwright"`;
this is a `karma-webkit-launcher` feature. Detecting that browser via
user agent as we normally do is hard as the UA on Linux is very similar
to a real Safari one but it actually uses a newer version of the engine.
In addition, we now allow to pass custom browsers when one needs it;
e.g., to run the tests in all three engines on Linux/macOS, run:
```
grunt && BROWSERS=ChromeHeadless,FirefoxHeadless,WebkitHeadless grunt karma:main
```
Closes gh-5190
Re-introduce the `selector-native` similar to the one on the `3.x-stable`
branch. One difference is since the `main` branch inlined Sizzle, some
selector utils can be shared between the main `selector` module and
`selector-native`.
The main `selector` module can be disabled in favor of `selector-native`
via:
grunt custom:-selector
Other changes:
* Tests: Fix Safari detection - Chrome Headless has a different user
agent than Safari and a browser check in selector tests didn't take
that into account.
* Tests: Run selector-native tests in `npm test`
* Selector: Fix querying on document fragments
Ref gh-4395
Closes gh-5085
This adds testing on Node.js 17 in addition to the currently tested 10, 12, 14
and 16 versions.
Also, update Grunt & `karma-*` packages.
Testing in Karma on jsdom is broken in Node 17 at the moment; until we find
a fix, this change disables such testing on Node 17 or newer.
Node smoke tests & promises aplus tests are disabled on Node.js 10 as they
depend on jsdom and the latest jsdom version doesn't run properly on Node 10.
Closes gh-5023
