@@ -0,0 +1,788 @@
---
name: windows-desktop-e2e
description: E2E testing for Windows native desktop apps (WPF, WinForms, Win32/MFC, Qt) using pywinauto and Windows UI Automation.
origin: ECC
---
# Windows Desktop E2E Testing
End-to-end testing for Windows native desktop applications using **pywinauto ** backed by Windows UI Automation (UIA). Covers WPF, WinForms, Win32/MFC, and Qt (5.x / 6.x) — with Qt-specific guidance as a dedicated section.
## When to Activate
- Writing or running E2E tests for a Windows native desktop application
- Setting up a desktop GUI test suite from scratch
- Diagnosing flaky or failing desktop automation tests
- Adding testability (AutomationId, accessible names) to an existing app
- Integrating desktop E2E into a CI/CD pipeline (GitHub Actions `windows-latest` )
### When NOT to Use
- Web applications → use `e2e-testing` skill (Playwright)
- Electron / CEF / WebView2 apps → the HTML layer needs browser automation, not UIA
- Mobile apps → use platform-specific tools (UIAutomator, XCUITest)
- Pure unit or integration tests that don't need a running GUI
## Core Concepts
All Windows desktop automation relies on **UI Automation (UIA) ** , a Windows-built-in accessibility API. Every supported framework exposes a tree of UIA elements with properties Claude can read and act on:
```
└── pywinauto (UIA backend)
└── Windows UI Automation API ← built into Windows, framework-agnostic
└── App's UIA provider ← each framework ships its own
└── Running .exe
```
**UIA quality by framework: **
| Framework | AutomationId | Reliability | Notes |
|-----------|-------------|-------------|-------|
| WPF | ★★★★★ | Excellent | `x:Name` maps directly to AutomationId |
| WinForms | ★★★★☆ | Good | `AccessibleName` = AutomationId |
| UWP / WinUI 3 | ★★★★★ | Excellent | Full Microsoft support |
| Qt 6.x | ★★★★★ | Excellent | Accessibility enabled by default; class names change to `Qt6*` |
| Qt 5.15+ | ★★★★☆ | Good | Improved Accessibility module |
| Qt 5.7– `QT_ACCESSIBILITY=1` ; objectName manual |
| Win32 / MFC | ★★★☆☆ | Fair | Control IDs accessible; text matching common |
## Setup & Prerequisites
``` bash
# Python 3.8+, Windows only
pip install pywinauto pytest pytest-html Pillow pytest-timeout
# Optional: screen recording
# Install ffmpeg and add to PATH: https://ffmpeg.org/download.html
```
Verify UIA is reachable:
``` python
from pywinauto import Desktop
Desktop ( backend = " uia " ) . windows ( ) # lists all top-level windows
```
Install **Accessibility Insights for Windows ** (free, from Microsoft) — your DevTools equivalent for inspecting the UIA element tree before writing any test.
## Testability Setup (by Framework)
The single most impactful thing you can do is **give every interactive control a stable AutomationId ** before writing tests.
### WPF
``` xml
<!-- XAML: x:Name becomes AutomationId automatically -->
<TextBox x:Name= "usernameInput" />
<PasswordBox x:Name= "passwordInput" />
<Button x:Name= "btnLogin" Content= "Login" />
<TextBlock x:Name= "lblError" />
```
### WinForms
``` csharp
// Set in designer or code
usernameInput . AccessibleName = "usernameInput" ;
passwordInput . AccessibleName = "passwordInput" ;
btnLogin . AccessibleName = "btnLogin" ;
lblError . AccessibleName = "lblError" ;
```
### Win32 / MFC
``` cpp
// Control resource IDs in .rc file are exposed as AutomationId strings
// IDC_EDIT_USERNAME -> AutomationId "1001"
// Prefer SetWindowText for Name; add IAccessible for richer support
```
### Qt — see dedicated section below
---
## Page Object Model
```
├── conftest.py # app launch fixture, failure screenshot
├── pytest.ini
├── config.py
├── pages/
│ ├── __init__.py # required for imports
│ ├── base_page.py # locators, wait, screenshot helpers
│ ├── login_page.py
│ └── main_page.py
├── tests/
│ ├── __init__.py
│ ├── test_login.py
│ └── test_main_flow.py
└── artifacts/ # screenshots, videos, logs
```
### base_page.py
``` python
import os , time
from pywinauto import Desktop
from config import ACTION_TIMEOUT , ARTIFACT_DIR
class BasePage :
def __init__ ( self , window ) :
self . window = window
# --- Locators (priority order) ---
def by_id ( self , auto_id , * * kw ) :
""" AutomationId — most stable. Use as first choice. """
return self . window . child_window ( auto_id = auto_id , * * kw )
def by_name ( self , name , * * kw ) :
""" Visible text / accessible name. """
return self . window . child_window ( title = name , * * kw )
def by_class ( self , cls , index = 0 , * * kw ) :
""" Control class + index — fragile, avoid if possible. """
return self . window . child_window ( class_name = cls , found_index = index , * * kw )
# --- Waits ---
def wait_visible ( self , spec , timeout = ACTION_TIMEOUT ) :
spec . wait ( " visible " , timeout = timeout )
return spec
def wait_gone ( self , spec , timeout = ACTION_TIMEOUT ) :
spec . wait_not ( " visible " , timeout = timeout )
return spec
def wait_window ( self , title , timeout = ACTION_TIMEOUT ) :
""" Wait for a new top-level window (dialogs, child windows). """
dlg = Desktop ( backend = " uia " ) . window ( title = title )
dlg . wait ( " visible " , timeout = timeout )
return dlg
def wait_until ( self , fn , timeout = ACTION_TIMEOUT , interval = 0.3 ) :
""" Poll an arbitrary condition — use when UIA events are unreliable. """
deadline = time . time ( ) + timeout
while time . time ( ) < deadline :
try :
if fn ( ) :
return True
except Exception :
pass
time . sleep ( interval )
raise TimeoutError ( f " Condition not met within { timeout } s " )
# --- Actions ---
def click ( self , spec ) :
self . wait_visible ( spec )
spec . click_input ( )
def type_text ( self , spec , text ) :
self . wait_visible ( spec )
ctrl = spec . wrapper_object ( )
try :
ctrl . set_edit_text ( text )
except Exception as e :
# Qt 5.x fallback: UIA Value Pattern may be incomplete
import sys , pywinauto . keyboard as kb
print ( f " [windows-desktop-e2e] set_edit_text failed ( { e } ), using keyboard fallback " , file = sys . stderr )
ctrl . click_input ( )
kb . send_keys ( " ^a " )
kb . send_keys ( text , with_spaces = True )
def get_text ( self , spec ) :
ctrl = spec . wrapper_object ( )
for attr in ( " window_text " , " get_value " ) :
try :
v = getattr ( ctrl , attr ) ( )
if v :
return v
except Exception :
pass
return " "
# --- Artifacts ---
def screenshot ( self , name ) :
os . makedirs ( ARTIFACT_DIR , exist_ok = True )
path = os . path . join ( ARTIFACT_DIR , f " { name } .png " )
self . window . capture_as_image ( ) . save ( path )
return path
```
### login_page.py
``` python
from pages . base_page import BasePage
class LoginPage ( BasePage ) :
@property
def username ( self ) : return self . by_id ( " usernameInput " )
@property
def password ( self ) : return self . by_id ( " passwordInput " )
@property
def btn_login ( self ) : return self . by_id ( " btnLogin " )
@property
def error_label ( self ) : return self . by_id ( " lblError " )
def login ( self , user , pwd ) :
self . type_text ( self . username , user )
self . type_text ( self . password , pwd )
self . click ( self . btn_login )
def login_ok ( self , user , pwd , main_title = " Main Window " ) :
self . login ( user , pwd )
return self . wait_window ( main_title )
def login_fail ( self , user , pwd ) :
self . login ( user , pwd )
self . wait_visible ( self . error_label )
return self . get_text ( self . error_label )
```
### conftest.py
> For new projects prefer the **Tier 1 sandbox fixture** (see below) — it adds filesystem isolation at zero extra cost. This basic fixture is for minimal/legacy setups only.
``` python
import os , pytest
os . environ [ " QT_ACCESSIBILITY " ] = " 1 " # Required for Qt 5.x UIA support
from pywinauto import Application
from config import APP_PATH , MAIN_WINDOW_TITLE , LAUNCH_TIMEOUT , ARTIFACT_DIR
@pytest.fixture
def app ( request ) :
if not APP_PATH :
pytest . exit ( " APP_PATH environment variable is not set " , returncode = 1 )
proc = Application ( backend = " uia " ) . start ( APP_PATH , timeout = LAUNCH_TIMEOUT )
win = proc . window ( title = MAIN_WINDOW_TITLE )
win . wait ( " visible " , timeout = LAUNCH_TIMEOUT )
yield win
# Screenshot on failure
if getattr ( getattr ( request . node , " rep_call " , None ) , " failed " , False ) :
os . makedirs ( ARTIFACT_DIR , exist_ok = True )
try :
win . capture_as_image ( ) . save (
os . path . join ( ARTIFACT_DIR , f " FAIL_ { request . node . name } .png " )
)
except Exception :
pass
# Graceful exit first, force-kill as fallback
# proc is a pywinauto Application — use wait_for_process_exit(), not wait_for_process()
try :
win . close ( )
proc . wait_for_process_exit ( timeout = 5 )
except Exception :
proc . kill ( )
@pytest.hookimpl ( tryfirst = True , hookwrapper = True )
def pytest_runtest_makereport ( item , call ) :
outcome = yield
setattr ( item , f " rep_ { outcome . get_result ( ) . when } " , outcome . get_result ( ) )
```
### config.py
``` python
import os
APP_PATH = os . environ . get ( " APP_PATH " , " " ) # set via env — no default path
MAIN_WINDOW_TITLE = os . environ . get ( " APP_TITLE " , " " )
LAUNCH_TIMEOUT = int ( os . environ . get ( " LAUNCH_TIMEOUT " , " 15 " ) )
ACTION_TIMEOUT = int ( os . environ . get ( " ACTION_TIMEOUT " , " 10 " ) )
ARTIFACT_DIR = os . path . join ( os . path . dirname ( __file__ ) , " artifacts " )
```
### pytest.ini
``` ini
[pytest]
testpaths = tests
markers =
smoke: fast smoke tests for critical paths
flaky: known-unstable tests
addopts = -v --tb=short --html=artifacts/report.html --self-contained-html
```
## Locator Strategy
```
(stable) (readable) (fragile) (last resort)
```
Inspect with Accessibility Insights → **Properties ** pane → look for `AutomationId` first.
``` python
# Inspect at runtime — paste into a REPL to explore the tree
win . print_control_identifiers ( )
# or narrow scope:
win . child_window ( auto_id = " groupBox1 " ) . print_control_identifiers ( )
```
## Wait Patterns
``` python
# Wait for control to appear
page . wait_visible ( page . by_id ( " statusLabel " ) )
# Wait for control to disappear (e.g. loading spinner)
page . wait_gone ( page . by_id ( " spinnerOverlay " ) )
# Wait for a dialog to pop up
dlg = page . wait_window ( " Confirm Delete " )
# Custom condition (e.g. text changes)
page . wait_until ( lambda : page . get_text ( page . by_id ( " lblStatus " ) ) == " Ready " )
```
**Never use `time.sleep()` as primary synchronization ** — use `wait()` or `wait_until()` .
## Artifact Management
``` python
# Screenshot on demand
page . screenshot ( " after_login " )
# Full-screen capture (when window is off-screen or minimised)
import pyautogui
pyautogui . screenshot ( " artifacts/fullscreen.png " )
# Screen recording with ffmpeg (start before test, stop after)
import subprocess
def start_recording ( name ) :
return subprocess . Popen ( [
" ffmpeg " , " -f " , " gdigrab " , " -framerate " , " 10 " ,
" -i " , " desktop " , " -y " , f " artifacts/videos/ { name } .mp4 "
] , stdin = subprocess . PIPE , stdout = subprocess . DEVNULL , stderr = subprocess . DEVNULL )
def stop_recording ( proc ) :
proc . stdin . write ( b " q " ) ; proc . stdin . flush ( ) ; proc . wait ( timeout = 10 )
```
## Flaky Test Handling
``` python
# Quarantine — equivalent to Playwright's test.fixme()
@pytest.mark.skip ( reason = " Flaky: animation race on slow CI. Issue #42 " )
def test_animated_transition ( self , app ) : . . .
# Skip in CI only
@pytest.mark.skipif ( os . environ . get ( " CI " ) == " true " , reason = " Flaky in CI #43 " )
def test_heavy_load ( self , app ) : . . .
```
Common causes and fixes:
| Cause | Fix |
|-------|-----|
| Control not ready | Replace `time.sleep` with `wait_visible` |
| Window not focused | Add `win.set_focus()` before interactions |
| Animation in progress | `wait_until(lambda: not loading_indicator.exists())` |
| Dialog timing | `wait_window(title, timeout=15)` |
| CI display not ready | Set `DISPLAY` or use virtual desktop in CI |
## Test Isolation & Sandbox
Three tiers of isolation — use the lightest tier that satisfies your needs.
### Tier 1 — Filesystem Isolation (default, always use)
Each test gets its own `APPDATA` / `LOCALAPPDATA` / `TEMP` via `subprocess.Popen` and `Application.connect()` . pytest's `tmp_path` fixture handles cleanup automatically.
``` python
# conftest.py — replace the basic `app` fixture with this
import os , subprocess , pytest
from pywinauto import Application
from config import APP_PATH , APP_ARGS , APP_TITLE , LAUNCH_TIMEOUT , ACTION_TIMEOUT , ARTIFACT_DIR
@pytest.fixture ( scope = " function " )
def app ( request , tmp_path ) :
""" Fresh process + isolated user-data dirs per test. """
if not APP_PATH :
pytest . exit ( " APP_PATH not set " , returncode = 1 )
# Redirect all per-user storage to an isolated tmp directory
sandbox_env = os . environ . copy ( )
sandbox_env [ " QT_ACCESSIBILITY " ] = " 1 "
sandbox_env [ " APPDATA " ] = str ( tmp_path / " AppData " / " Roaming " )
sandbox_env [ " LOCALAPPDATA " ] = str ( tmp_path / " AppData " / " Local " )
sandbox_env [ " TEMP " ] = sandbox_env [ " TMP " ] = str ( tmp_path / " Temp " )
for p in ( sandbox_env [ " APPDATA " ] , sandbox_env [ " LOCALAPPDATA " ] , sandbox_env [ " TEMP " ] ) :
os . makedirs ( p , exist_ok = True )
if not APP_TITLE :
pytest . exit ( " APP_TITLE environment variable is not set " , returncode = 1 )
# shlex.split handles quoted args with spaces; plain split() breaks on them
import shlex
# Launch via subprocess so we can pass env; connect pywinauto by PID
proc = subprocess . Popen (
[ APP_PATH ] + shlex . split ( APP_ARGS ) ,
env = sandbox_env ,
)
pw_app = Application ( backend = " uia " ) . connect ( process = proc . pid , timeout = LAUNCH_TIMEOUT )
win = pw_app . window ( title = APP_TITLE )
win . wait ( " visible " , timeout = LAUNCH_TIMEOUT )
yield win
if getattr ( getattr ( request . node , " rep_call " , None ) , " failed " , False ) :
os . makedirs ( ARTIFACT_DIR , exist_ok = True )
try :
win . capture_as_image ( ) . save (
os . path . join ( ARTIFACT_DIR , f " FAIL_ { request . node . name } .png " )
)
except Exception :
pass
try :
win . close ( )
proc . wait ( timeout = 5 )
except Exception :
proc . kill ( )
# tmp_path is cleaned up automatically by pytest
@pytest.hookimpl ( tryfirst = True , hookwrapper = True )
def pytest_runtest_makereport ( item , call ) :
outcome = yield
setattr ( item , f " rep_ { outcome . get_result ( ) . when } " , outcome . get_result ( ) )
```
### Tier 2 — Windows Job Object (optional: process-lifetime containment)
Attach the process to a Job Object so it is **automatically terminated ** when
the test fixture's job handle is GC'd. Also prevents the app from spawning
child processes that escape fixture cleanup.
> **Scope of isolation:** Job Objects do NOT virtualize filesystem access or
> block network traffic. File-write and network isolation require AppContainer,
> Windows Firewall rules, or Tier 3 (Windows Sandbox). Use Tier 2 only for
> process-lifetime and child-process containment.
Requires no extra dependencies.
``` python
import ctypes , ctypes . wintypes as wt
def restrict_process ( pid : int ) :
"""
Attach the process to a Job Object that prevents it from:
- spawning processes outside the job (LIMIT_KILL_ON_JOB_CLOSE)
Does NOT block network — use Windows Firewall rules for that.
"""
JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000
# Minimal rights: SET_QUOTA (0x0100) | TERMINATE (0x0001)
PROCESS_SET_QUOTA_AND_TERMINATE = 0x0101
kernel32 = ctypes . windll . kernel32
job = kernel32 . CreateJobObjectW ( None , None )
hproc = kernel32 . OpenProcess ( PROCESS_SET_QUOTA_AND_TERMINATE , False , pid )
# Correct struct layout — LimitFlags is at offset +16, not +44
class JOBOBJECT_BASIC_LIMIT_INFORMATION ( ctypes . Structure ) :
_fields_ = [
( " PerProcessUserTimeLimit " , wt . LARGE_INTEGER ) ,
( " PerJobUserTimeLimit " , wt . LARGE_INTEGER ) ,
( " LimitFlags " , wt . DWORD ) ,
( " MinimumWorkingSetSize " , ctypes . c_size_t ) ,
( " MaximumWorkingSetSize " , ctypes . c_size_t ) ,
( " ActiveProcessLimit " , wt . DWORD ) ,
( " Affinity " , ctypes . c_size_t ) ,
( " PriorityClass " , wt . DWORD ) ,
( " SchedulingClass " , wt . DWORD ) ,
]
info = JOBOBJECT_BASIC_LIMIT_INFORMATION ( )
info . LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE
ok = kernel32 . SetInformationJobObject ( job , 2 , ctypes . byref ( info ) , ctypes . sizeof ( info ) )
if not ok :
raise ctypes . WinError ( )
kernel32 . AssignProcessToJobObject ( job , hproc )
kernel32 . CloseHandle ( hproc )
return job # keep alive — job closes (kills proc) when GC'd
# After proc = subprocess.Popen(...): job = restrict_process(proc.pid)
```
### Tier 3 — Windows Sandbox (CI full-OS isolation)
When you need a clean Windows image per run (no leftover registry keys, no
shared GPU state, true isolation), run the **entire test suite ** inside
[Windows Sandbox ](https://learn.microsoft.com/windows/security/application-security/application-isolation/windows-sandbox/windows-sandbox-overview ).
**Requirement: ** Windows 10/11 Pro or Enterprise, Virtualization enabled.
Create `e2e-sandbox.wsb` in your project root:
``` xml
<Configuration >
<MappedFolders >
<!-- App binary (read - only) -->
<MappedFolder >
<HostFolder > C:\path\to\your\build\Release</HostFolder>
<SandboxFolder > C:\app</SandboxFolder>
<ReadOnly > true</ReadOnly>
</MappedFolder>
<!-- Test suite (read - write for artifacts) -->
<MappedFolder >
<HostFolder > C:\path\to\your\e2e_test</HostFolder>
<SandboxFolder > C:\e2e_test</SandboxFolder>
<ReadOnly > false</ReadOnly>
</MappedFolder>
</MappedFolders>
<LogonCommand >
<!--
Windows Sandbox starts with no Python. Install it silently first,
then install deps and run tests. Artifacts are written back to the
host via the MappedFolder above.
-->
<Command > powershell -Command "
winget install --id Python.Python.3.11 --silent --accept-package-agreements;
$env:PATH += ';' + $env:LOCALAPPDATA + '\Programs\Python\Python311\Scripts';
cd C:\e2e_test;
pip install -r requirements.txt;
pytest tests\ -v
"</Command>
</LogonCommand>
</Configuration>
```
Launch: `WindowsSandbox.exe e2e-sandbox.wsb`
> pywinauto and the app both run **inside** the sandbox (same session required).
> Artifacts are written back to the host via the mapped folder.
### Tier comparison
| Tier | Isolation | Setup cost | Works on CI | Use when |
|------|-----------|-----------|-------------|----------|
| 1 — `tmp_path` env redirect | Filesystem | Zero | Always | Default for all tests |
| 2 — Job Object | Process tree | Low | Always | Prevent child-process escape |
| 3 — Windows Sandbox | Full OS | Medium | Needs Pro/Enterprise image | Nightly clean-room runs |
### Prevent hanging tests
Add `pytest-timeout` to cap any single test. In `pytest.ini` set `timeout = 60` and `timeout_method = thread` . Note: `thread` method cannot kill Qt app subprocesses on Windows — add `atexit.register(lambda: [p.kill() for p in psutil.Process().children(recursive=True)])` in `conftest.py` to reap orphans.
## CI/CD Integration
``` yaml
# .github/workflows/e2e-desktop.yml
name : Desktop E2E
on : [ push, pull_request]
jobs :
e2e :
runs-on : windows-latest # real GUI environment, no Xvfb needed
steps :
- uses : actions/checkout@v4
- uses : actions/setup-python@v5
with : { python-version : "3.11" }
- name : Install deps
run : pip install pywinauto pytest pytest-html Pillow
- name : Build app
run : cmake --build build --config Release # adjust to your build system
- name : Run E2E
env :
APP_PATH : ${{ github.workspace }}\build\Release\MyApp.exe
APP_TITLE : "My Application"
CI : "true"
run : pytest tests/ --html=artifacts/report.html --self-contained-html --junitxml=artifacts/results.xml -v
- uses : actions/upload-artifact@v4
if : always()
with :
name : e2e-artifacts
path : artifacts/
retention-days : 14
```
## Qt Specific
### Enable UIA in Qt 5.x
Qt 5.x accessibility is disabled by default in some builds (especially 5.7– **before ** launching. Qt 6.x enables accessibility by default — skip this step for Qt 6.
``` python
# conftest.py — add at module top
import os
os . environ [ " QT_ACCESSIBILITY " ] = " 1 "
```
Or export it in CI:
``` yaml
env :
QT_ACCESSIBILITY : "1"
```
### Add Stable Identifiers to Qt Widgets
``` cpp
// Preferred: both objectName and accessibleName
void setTestId ( QWidget * w , const char * id ) {
w - > setObjectName ( id ) ;
w - > setAccessibleName ( id ) ; // becomes UIA Name property
}
// In your dialog constructor:
setTestId ( ui - > usernameEdit , " usernameInput " ) ;
setTestId ( ui - > passwordEdit , " passwordInput " ) ;
setTestId ( ui - > loginButton , " btnLogin " ) ;
setTestId ( ui - > errorLabel , " lblError " ) ;
```
Centralise all IDs in a header to avoid typos:
``` cpp
// test_ids.h
# define TID_USERNAME "usernameInput"
# define TID_PASSWORD "passwordInput"
# define TID_BTN_LOGIN "btnLogin"
# define TID_LBL_ERROR "lblError"
```
### Qt-Specific Quirks
**QComboBox ** — the dropdown is a separate top-level window:
``` python
from pywinauto import Desktop
def select_combo_item ( page , combo_spec , item_text ) :
page . click ( combo_spec )
# Dropdown appears as a new root-level window
# class_name varies by Qt version — verify with Accessibility Insights
# Qt 5.x: "Qt5QWindowIcon" | Qt 6.x: "Qt6QWindowIcon" — verify with Accessibility Insights
popup = Desktop ( backend = " uia " ) . window ( class_name_re = " Qt[56]QWindowIcon " )
popup . wait ( " visible " , timeout = 5 )
popup . child_window ( title = item_text ) . click_input ( )
```
**QMessageBox / QDialog ** — also separate top-level windows:
``` python
dlg = page . wait_window ( " Confirm " ) # wait for dialog title
dlg . child_window ( title = " OK " ) . click_input ( ) # click button inside it
```
**QTableWidget / QTableView ** — row/cell access:
``` python
table = page . by_id ( " tblUsers " ) . wrapper_object ( )
cell = table . cell ( row = 0 , column = 1 )
print ( cell . window_text ( ) )
```
**Self-drawn controls ** (`paintEvent` -only, `QGraphicsView` , `QOpenGLWidget` ) — UIA cannot see their internals. Use the Fallback section below.
## Fallback: Screenshot Mode
When a control is not reachable via UIA (self-drawn, third-party, game engine):
``` bash
```
``` python
import pyautogui , cv2 , numpy as np
from PIL import Image
def find_image_on_screen ( template_path , confidence = 0.85 ) :
""" Locate a template image on screen. Returns (x, y) center or None. """
screen = np . array ( pyautogui . screenshot ( ) )
template = np . array ( Image . open ( template_path ) )
result = cv2 . matchTemplate (
cv2 . cvtColor ( screen , cv2 . COLOR_RGB2BGR ) ,
cv2 . cvtColor ( template , cv2 . COLOR_RGB2BGR ) ,
cv2 . TM_CCOEFF_NORMED ,
)
_ , max_val , _ , max_loc = cv2 . minMaxLoc ( result )
if max_val > = confidence :
h , w = template . shape [ : 2 ]
return max_loc [ 0 ] + w / / 2 , max_loc [ 1 ] + h / / 2
return None
def click_image ( template_path , confidence = 0.85 ) :
pos = find_image_on_screen ( template_path , confidence )
if pos is None :
raise RuntimeError ( f " Image not found on screen: { template_path } " )
pyautogui . click ( * pos )
```
**Use sparingly ** — image matching breaks on DPI changes, theme switches, and partial occlusion.
Always try UIA first; fall back to screenshots only for genuinely unreachable controls.
## Anti-Patterns
``` python
# BAD: fixed sleep
time . sleep ( 3 )
page . click ( page . by_id ( " btnSubmit " ) )
# GOOD: condition wait
page . wait_visible ( page . by_id ( " btnSubmit " ) )
page . click ( page . by_id ( " btnSubmit " ) )
```
``` python
# BAD: brittle class+index locator as primary strategy
page . by_class ( " Edit " , index = 2 ) . type_keys ( " hello " )
# GOOD: AutomationId
page . by_id ( " usernameInput " ) . set_edit_text ( " hello " )
```
``` python
# BAD: assert on pixel coordinates
assert btn . rectangle ( ) . left == 120
# GOOD: assert on content / state
assert page . get_text ( page . by_id ( " lblStatus " ) ) == " Logged in "
assert page . by_id ( " btnLogout " ) . is_enabled ( )
```
``` python
# BAD: share app instance across all tests (state leaks)
@pytest.fixture ( scope = " session " )
def app ( ) : . . .
# GOOD: fresh process per test (or per class at most)
@pytest.fixture ( scope = " function " )
def app ( ) : . . .
```
## Running Tests
``` bash
# All tests
pytest tests/ -v
# Smoke only
pytest tests/ -m smoke -v
# Specific file
pytest tests/test_login.py -v
# With custom app path
APP_PATH = "C:\build\Release\MyApp.exe" APP_TITLE = "MyApp" pytest tests/ -v
# Detect flaky tests (repeat each 5 times)
pip install pytest-repeat
pytest tests/test_login.py --count= 5 -v
```
## Related Skills
- `e2e-testing` — Playwright E2E for web applications
- `cpp-testing` — C++ unit/integration testing with GoogleTest
- `cpp-coding-standards` — C++ code style and patterns
Affaan Mustafa