Full Website Test

A full website test is often the quickest and easiest way to get up and running with Autonomous. To create the test, all you need to do is enter the URL of your website, and select which environments you want to test and Autonomous will do all the heavy lifting for you automatically.

A full website test has the following benefits:

  • Automatically creates a test that covers the entire website or web application, providing complete coverage easily.

  • Ensures the complete and consistent display of all screens across various environments.

  • Detects both visual disparities and unexpected alterations on any page throughout the entire website or web application.

  • Automatically detects technical issues in any of the pages, including HTTP errors and page timeouts.

  • Automatically detects pages that were added or removed from the website or web application every time the test runs.

A full website test automatically creates a list of all screens in your website based on the sitemap file and takes a screenshot of each page. These screenshots are used as a baseline in your first test. In future runs of the test, Autonomous compares pages to these baselines to identify pages that have changed, been added or removed, or did not load properly.

There is an option to create a custom flow test which should run when each screen is loaded. For example if you need to log into a website or click "Accept Cookies" before opening a screen.

When you create a full website test, the system automatically creates a plan which is scheduled to run daily and creates an application for the plan. If required, after you create the test, you can modify the schedule of the plan or add the test to a different plan or application.

To Create a Full Website Test

  1. In the Navigation bar, select Test management > Tests.

  2. Click Create a new test > Full website.

  3. In the URL field, enter the home page of the website. The URL must be in the format https://www.example.com/. You can also enter the full path to the sitemap, or to a sub-directory for example https://applitools.com/docs/.

    If you specify a sub-directory, Autonomous uses the sitemap in the root directory and identifies URLs in the sub-directory. If the sub-directory has a separate sitemap, enter the full path to the sitemap for the sub-directory.

  4. By default, the test uses the following environments; each environment defines a browser, an operating system, and a screen size:

    • Chromium, Linux, 1440 x 900

    • Safari, Mac, 1200 x 857

    • Microsoft Edge, Windows, 1080 x 1920

    If required, click on an environment to remove it from the test. After you create the test, you will be able to add or modify the environments in the test plan.

  5. To limit the number of URLs included in the first run of the test, click Optional settings and select Set a max. number of URLs to include in the text and enter a number of URLs. This is an easy way to quickly run the first test and get an indication of the results for a limited number of URLs. This option only applies the first time that the test runs, in future runs the test will include all URLs in the sitemap and additional URLs will be marked as New in the results.

  6. If you have a trial account, there is a limit to the number of tests you can include in a full website test.

  7. To enter basic authentication or run a custom flow before each URL opens, click Optional settings and choose Select a custom flow to run before this test and select one of the following:

    • Basic authentication and enter a username and password. Basic authentication adds an authentication encode string in each URL, so that the user name and password is included in the request to the browser.

    • An existing custom flow test, (see Custom Flow Test) for example a test which selects "Accept Cookies" which should run when each URL in the test opens.

  8. Click Next.

    Autonomous creates baselines for each screen in the website. A baseline is an initial image which will be used to compare to future runs of the test to identify differences. The number of baselines is based on the number of URLs in the test, multiplied by the number of environments.

    Once the baselines have been generated, Autonomous displays thumbnails of the first 10 baselines in the default environment.

  9. Once the baselines have been created, click Run Test.

    Autonomous runs the test to create a checkpoint for each screen which is compared to the baseline to identify issues and screens that have changed.

  10. The Test is Complete window gives a preview of the first ten screens in the test, click Review Results to view the complete results in the Results page.

Autonomous automatically creates a new plan for the test which is scheduled to run daily. Plan details are displayed in the lower right corner of the window. You can run the test again or modify the schedule in the Plans page. The plan is automatically assigned to a new application.

Modifying an Existing Test

To view or modify an existing full website test, in the Navigation bar, select Tests and click on the test that you want to view or modify.

Like many pages in Autonomous, at the top of the Test Details page there are breadcrumbs to show where you are in the system. Click on the word Tests next to the name of the test or Back to list to return to the Tests page.

Modifying Test Details

The top part of the Test Details page has a collapsible section which lists test details. If you modify this information the system prompts you to save your changes; changes are not automatically saved.

You can modify the following Test Details:

  • Test name – The name of test as it is displayed in the Tests page. By default this is the base URL of the test.

  • Start URL – This field is not relevant for a full website test and cannot be modified.

  • Description – Optional description of the test.

  • Application – The application that the test belongs to. Each test must belong to one application, if you select a different application you also need to update the plans running the test. For further information, see Applications.

  • Plans running this test – The plans associated with the test. The plan determines the test schedule, environments and parameters. A test can belong to multiple plans, the list of available plans is based on the application of the test. For further information see Plans.

  • To set other advanced settings for the test click . For details, see Advanced Test Settings.

Flow to Run Before Steps

You can enter basic authentication or run a custom test before each URL in a test. For example, if you need to close a popup or click "Accept Cookies" before each URL in the test.

To enter or modify basic authentication or run a custom flow, click Select a flow to run before the screen URLs and select one of the following:

  • Basic authentication and enter a username and password. Basic authentication adds an authentication encode string in each URL, so that the user name and password is included in the request to the browser. When using basic authentication, the authorization header includes the endcode in every request.

  • Existing customized test and select a Custom Flow Test, for example a test which selects "Accept Cookies", which should run when each URL in the test opens. The list of available custom flow tests is based on the plans that the test belongs to. If the custom flow does not run successfully, for example a popup does not appear, the test will ignore the custom flow test for that URL.

    By default, the start URL of a custom pre-flow test will be ignored. If you do not want to ignore the start URL, in the test's advanced settings, enable Use the custom flow's start URL for details, see Use Custom Flow's Start URL.

Excluding or Adding URLs

You can select URLs or folders to include or exclude from the test. For example if your sitemap or a sub-directory contains more than the maximum number of allowed URLs, no screens will be included in the test and you need to manually select which URLs or folders should be included.

When you run the test, if there are new URLs in the sitemap they will be added to the test unless they are located in a folder that has been excluded.

To Exclude URLs or Folders:

  1. In the Test Details page, select whether to view as URLs or Folders.

  2. Select URLs or folders that you want to exclude and click > Exclude screens.

Setting the Match Level

The match level determines the way that Autonomous compares the checkpoint image with the baseline image. You can change the match level for any screen in the test. For details see Match Level.

Setting the Screenshot Type

The screenshot type determines what will be included in the screenshot (Full screen, viewport, or region). For details, see Screenshot Type.

To set the screenshot type for a single URL, next to the URL click and select a screenshot type.

To select the screenshot type for multiple URLs, in the list of URLs, select the required URLs and click > Screenshot type and select the required type.

Modifying Advanced Settings

To set or modify advanced settings for the test or for specific URLs in the test, see Advanced Test Settings.

Troubleshooting

Following are some of the common problems that you may encounter when creating or running a full website test.

Problem:

Autonomous is unable to find the sitemap, or there is no sitemap file.

Solution:

By default, Autonomous looks for a sitemap file in the root directory, the file must be named sitemap.xml or listed in the robots.txt file. This error appears if Autonomous is unable to find a sitemap file. To resolve this problem try one of the following:

  • Add a sitemap file named sitemap.xml to the root directory.

  • If there is a sitemap file with a different name or in a different location, when creating the full website test, enter the full path to the sitemap file instead of the root directory.

  • If there is no sitemap file and you are unable to create one, create a URL list test instead of a Full Website Test. See URL List Test

Problem:

The domain test timed out

Solution:

This could be caused by a network issue, try again later.

Problem:

An error message appears that not all screens were included in the test or one or more directory was excluded from the test.

Solution:

There is a limit on the number of screens that can be included in a single test. The maximum number is based in your Autonomous plan. If your sitemap includes too many URLs, try one of the following:

  • Manually select URLs that should be excluded from the test. For details, see Excluding or Adding URLs.

  • Create a separate test for each sub-directory

  • Contact Support to increase the number of URLs allowed in your plan

This error is caused because the server timed out before it was able to process the complete sitemap, this is normally because the sitemap is extremely large.

If the sitemap includes multiple nested sitemaps, you should create a separate full website test for each nested sitemap. When you create the test, instead of entering the URL of your website, enter the full path to a nested sitemap.

Problem:

An "Unknown Error" occurred when creating a test for a sub-directory (for example http://www.mynewssite.com/sport)

Solution:

When creating a test for a sub-directory, Autonomous uses the sitemap in the root folder to identify screens in the sub-directory. If the sub-directory has a separate sitemap, when you create the test, enter the full path to the sitemap for the sub-directory.