This is a complete rework of our testing infrastructure. The main goal is to modernize and drop deprecated or undermaintained dependencies (specifically, grunt, karma, and testswarm). We've achieved that by limiting our dependency list to ones that are unlikely to drop support any time soon. The new dependency list includes:
- `qunit` (our trusty unit testing library)
- `selenium-webdriver` (for spinning up local browsers)
- `express` (for starting a test server and adding middleware)
- express middleware includes uses of `body-parser` and `raw-body`
- `yargs` (for constructing a CLI with pretty help text)
- BrowserStack (for running each of our QUnit modules separately in all of our supported browsers)
- `browserstack-local` (for opening a local tunnel. This is the same package still currently used in the new Browserstack SDK)
- We are not using any other BrowserStack library. The newest BrowserStack SDK does not fit our needs (and isn't open source). Existing libraries, such as `node-browserstack` or `browserstack-runner`, either do not quite fit our needs, are under-maintained and out-of-date, or are not robust enough to meet all of our requirements. We instead call the [BrowserStack REST API](https://github.com/browserstack/api) directly.
**BrowserStack**
- automatically retries individual modules in case of test failure(s)
- automatically attempts to re-establish broken tunnels
- automatically refreshes the page in case a test run has stalled
- Browser workers are reused when running isolated modules in the same browser
- runs all browsers concurrently and uses as many sessions as are available under the BrowserStack plan. It will wait for available sessions if there are none.
- supports filtering the available list of browsers by browser name, browser version, device, OS, and OS version (see `npm run test:unit -- --list-browsers` for more info). It will retrieve the latest matching browser available if any of those parameters are not specified. Supports latest and latest-\d+ in place of browser version.
- cleans up after itself (closes the local tunnel, stops the test server, etc.)
- Requires `BROWSERSTACK_USERNAME` and `BROWSERSTACK_ACCESS_KEY` environment variables.
**Selenium**
- supports running any local browser as long as the driver is installed, including support for headless mode in Chrome, FF, and Edge
- supports running `basic` tests on the latest [jsdom](https://github.com/jsdom/jsdom#readme), which can be seen in action in this PR (see `test:browserless`)
- Node tests will run as before in PRs and all non-dependabot branches, but now includes tests on real Safari in a GH actions macos image instead of playwright-webkit.
- can run multiple browsers and multiple modules concurrently
Other notes:
- Stale dependencies have been removed and all remaining dependencies have been upgraded with a few exceptions:
- `sinon`: stopped supporting IE in version 10. But, `sinon` has been updated to 9.x.
- `husky`: latest does not support Node 10 and runs on `npm install`. Needed for now until git builds are migrated to GitHub Actions.
- `rollup`: latest does not support Node 10. Needed for now until git builds are migrated to GitHub Actions.
- BrowserStack tests are set to run on each `main` branch commit
- `debug` mode leaves Selenium browsers open whether they pass or fail and leaves browsers with test failures open on BrowserStack. The latter is to avoid leaving open too many sessions.
- This PR includes a workflow to dispatch BrowserStack runs on-demand
- The Node version used for most workflow tests has been upgraded to 20.x
- updated supportjQuery to 3.7.1
Run `npm run test:unit -- --help` for CLI documentation
Close gh-5427
Also, run such a suite in CI to make sure modules are working as expected
when used directly.
(partially cherry picked from 341c6d1b5a)
(partially cherry picked from 437f389a24)
Closes gh-4595
Ref gh-4550
Ref gh-4574
- Update QUnit to 1.23.1
- Remove unused dl#dl from test/index.html
- Remove unused map#imgmap from test/index.html
- Ensure all urls to data use baseURI
- Add the 'grunt karma:main' task
- customContextFile & customDebugFile
- Add 'npm run jenkins' script
Close gh-3744
Fixes gh-1999
All deps were updated except:
* jsdom - tests using a Symbol polyfill are hacky and break with newer jsdom;
we need to re-do them properly first
* qunitjs - versions 1.19.0 & 1.20.0 introduce race conditions to the tests,
making the fail randomly
Those two packages will be updated once issues related to them get resolved.
Fixes gh-2877
Tracked bower dependencies are located at "src/sizzle" and "test/libs".
The source-destination mapping is in the Gruntfile.
When updating a bower dependency, update the version in bower.json, run
`grunt bower`, and then commit the result. When adding a dependency,
update the bowercopy task accordingly.
Fixes#14615.
Closes gh-1452.
* Removed inline usage of QUnit.reset() because it is messing with the
expectation model as reset does .empty() which does a recursive cleanData
on everything in #qunit-fixture, so any expectJqData above .reset() would
fail negatively.
Instead of calling reset inline, either updated the following assertions to
take previous assertions' state into account, or broke the test() up into
2 tests at the point where it would call QUnit.reset.
* After introducing the new memory leak discovery a whole bunch of tests were
failing as they didn't clean up everything. However I didn't (yet) add
QUnit.expectJqData calls all over the place because in most if not all of
these cases it is valid data storage. For example in test "data()", there
will be an internal data key for "parsedAttrs". This particular test isn't
intending to test for memory leaks, so therefor I made the new discovery
system only push failures when the test contains at least 1 call to
QUnit.expectJqData.
When not, we'll assume that whatever data is being stored is acceptable
because the relevant elements still exist in the DOM anyway (QUnit.reset
will remove the elements and clean up the data automatically).
I did add a "Always check jQuery.data" mode in the test suite that will
trigger it everywhere. Maybe one day we'll include a call to everywhere,
but for now I'm keeping the status quo: Only consider data left in storage
to be a problem if the test says so ("opt-in").
* Had to move #fx-tests inside the fixture because ".remove()" test would
otherwise remove stuff permanently and cause random other tests to fail
as "#hide div" would yield an empty collection.
(Why wasn't this in the fixture in the first place?)
As a result moving fx-tests into the fixture a whole bunch of tests failed
that relied on arbitrary stuff about the document-wide or fixture-wide
state (e.g. number of divs etc.). So I had to adjust various tests to
limit their sample data to not be so variable and unlimited...
* Moved out tests for expando cleanup into a separate test.
* Fixed implied global variable 'pass' in effects.js that was causing
"TypeError: boolean is not a function" in *UNRELATED* dimensions.js that
uses a global variable "pass = function () {};" ...
* Removed spurious calls to _removeData. The new test exposed various failures
e.g. where div[0] isn't being assigned any data anyway.
(queue.js and attributes.js toggleClass).
* Removed spurious clean up at the bottom of test() functions that are
already covered by the teardown (calling QUnit.reset or removeClass to
supposedly undo any changes).
* Documented the parentheses-less magic line in toggleClass. It appeared that
it would always keep the current class name if there was any (since the
assignment started with "this.className || ...".
Adding parentheses + spacing is 8 bytes (though only 1 in gzip apparently).
Only added the comment for now, though I prefer clarity with logical
operators, I'd rather not face the yayMinPD[1] in this test-related commit.
* Updated QUnit urlConfig to the new format (raw string is deprecated).
* Clean up odd htmlentities in test titles, QUnit escapes this.
(^\s+test\(.*)(>\;) → $1>
(^\s+test\(.*)(<\;) → $1<
[1] jQuery MinJsGz Release Police Department (do the same, download less)
