Playwright async wait_for_selector timing out on dynamic Shadow DOM elements

Root Cause Diagnosis

wait_for_selector times out because the selector either cannot cross the shadow boundary or the element never meets the default state="visible" condition. Playwright’s CSS and text engines pierce open shadow roots automatically, but they cannot reach closed roots, XPath never pierces shadow DOM, and the > child combinator does not cross a shadow boundary. Elements inside shadow trees are also often attached but fail the visibility check (zero-size box, display: contents, unassigned <slot>), so the wait runs to the 30 000 ms default timeout.

Quick Fix

Wait for attachment instead of visibility, and use a CSS selector with a descendant combinator:

await page.wait_for_selector("my-widget .inner-btn", state="attached", timeout=10_000)

If this still times out, the shadow root is probably closed. See Step 3.

Step-by-Step Resolution

Step 1: Confirm whether the shadow root is open or closed.

mode = await page.evaluate("""
() => {
  const host = document.querySelector('my-widget');
  if (!host) return 'HOST_NOT_FOUND';
  return host.shadowRoot ? 'open' : 'closed_or_none';
}
""")
print(mode)

• HOST_NOT_FOUND: the host element itself isn’t in the light DOM yet, or it is inside an iframe (Step 4).
• open: the selector or visibility state is the problem (Step 2).
• closed_or_none: the host exists but shadowRoot is null, which means the root is closed (Step 3).

Step 2: Fix the selector and wait state (open roots).

Use CSS with descendant combinators. Do not use XPath or > across the boundary.

# Fails: XPath does not pierce shadow roots
await page.wait_for_selector("xpath=//my-widget//button")

# Fails: ">" does not cross the shadow boundary
await page.wait_for_selector("my-widget > button")

# Works: descendant combinator pierces open shadow roots
await page.wait_for_selector("my-widget button.submit", state="attached")

Prefer locators, which auto-wait and also pierce open shadow roots:

btn = page.locator("my-widget").get_by_role("button", name="Submit")
await btn.wait_for(state="attached", timeout=10_000)
await btn.click()

To chain scoped selectors explicitly, use >>:

await page.wait_for_selector("my-widget >> .inner-btn", state="attached")

If the element must be interactable, wait for attached, then let the action’s built-in actionability checks handle visibility, enabled state and stability. Do not stack a visible wait on an element that never gets a layout box.

Step 3: Force closed shadow roots open (closed roots).

Playwright cannot pierce mode: 'closed'. Patch attachShadow with an init script that runs before any page script, i.e. before navigation:

await context.add_init_script("""
(() => {
  const orig = Element.prototype.attachShadow;
  Element.prototype.attachShadow = function (init) {
    return orig.call(this, { ...init, mode: 'open' });
  };
})();
""")
page = await context.new_page()
await page.goto("https://example.com")
await page.wait_for_selector("my-widget .inner-btn", state="attached")

This only affects shadow roots created after the script is installed. Call add_init_script on the context or page before goto, not after.

Step 4: Handle shadow hosts inside iframes.

Shadow piercing does not cross frame boundaries. Scope to the frame first:

frame = page.frame_locator("iframe#app")
await frame.locator("my-widget .inner-btn").wait_for(state="attached")

Step 5: Handle late custom-element upgrade and re-rendering.

If the host renders its shadow content asynchronously (Lit, Stencil, framework hydration), wait on the actual rendered node rather than the host. Where a selector is impractical, poll in the page:

await page.wait_for_function("""
() => {
  const root = document.querySelector('my-widget')?.shadowRoot;
  return !!root?.querySelector('.inner-btn');
}
""", timeout=15_000)

This works only for open roots. After a re-render the old element handle is stale, so re-query with a locator instead of reusing an ElementHandle.

Step 6: Inspect with the Playwright debugger.

PWDEBUG=1 python your_script.py

On Windows PowerShell:

$env:PWDEBUG=1; python your_script.py

In the Inspector, use Pick locator to see the selector Playwright generates for the shadow element, then copy it.

Why This Fails for Some Users

• Declarative Shadow DOM (<template shadowrootmode="closed">). The HTML parser creates these roots without calling attachShadow, so the init-script override in Step 3 does not apply. Open-mode declarative roots work normally. For closed ones, you have to interact through the light DOM or application APIs.
• Strict mode on locator.wait_for. Unlike page.wait_for_selector, locator(...).wait_for() raises a strict-mode violation if the selector resolves to multiple elements, which is common when several instances of the same custom element exist. Narrow it with .first, .nth(i), or a more specific scope.
• Explicit :light selectors. A selector written with the :light suffix or a css:light= engine prefix deliberately disables piercing. Remove it unless you intend light-DOM-only matching.
• Slotted content. Nodes projected into a <slot> live in the light DOM of the host. Query them from the host’s light DOM, not by looking inside the shadow root.
• Zero-size or display: contents hosts. The host or a wrapper has no layout box, so state="visible" never resolves even though the node exists. Use state="attached".
• Old Playwright versions. Upgrade to a current release and reinstall browsers. Check with pip show playwright, then run playwright install.
• Elements gated on interaction or viewport. Lazy-rendered shadow content may only appear after scroll, hover, or focus. The wait cannot succeed until that trigger fires, so perform it first (await page.mouse.wheel(0, 800) or await host.hover()).
• Bot detection or consent overlays. A blocking overlay or an anti-automation path can prevent the component from ever rendering. Check await page.screenshot(path="debug.png") to confirm the component is present before debugging selectors.