Is there a way to run a headless browser (with Playwright) from a background job?

Frappe Framework App Development
1

I have a very specific use case (Generating CAD previews) that requires using a headless browser.

I have succeeded to make it work by installing Playwright and their dependencies inside the Frappe environment:

bench pip install playwright
playwright install chromium
playwright install-deps chromium

# ...
dependencies = [
    # "frappe~=15.0.0" # Installed and managed by bench.
    "playwright~=1.54.0",
]
#...

Then writting functions like this:

from tempfile import NamedTemporaryFile
from pathlib import Path

import frappe
from frappe import _
from frappe.model.document import Document
from frappe.utils.file_manager import get_file, save_file
from playwright.sync_api import sync_playwright

SHARECAD_UNWANTED_ELEMENTS = [".checkcookie", ".popup", ".noprint", ".header"]


def get_sharecad_screenshot(path: str | Path):
	if not Path(path).exists():
		frappe.throw(_("File not found: {0}").format(path))
	screenshot_bytes = None
	with sync_playwright() as p:
		browser = p.chromium.launch(headless=True)
		page = browser.new_page(record_video_dir="./recordings")
		page.on("filechooser", lambda file_chooser: file_chooser.set_files(path))
		page.goto("https://sharecad.org/")
		page.locator(".text-file").click()
		page.wait_for_load_state("networkidle")
		page.add_style_tag(content=f"{', '.join(SHARECAD_UNWANTED_ELEMENTS)} {{display: none !important;}}")
		screenshot_bytes = page.screenshot()
		page.close()
	return screenshot_bytes

class ManufacturingItem(Document):
	# begin: auto-generated types
	# This code is auto-generated. Do not modify anything in this block.

	from typing import TYPE_CHECKING

	if TYPE_CHECKING:
		from frappe.types import DF

		from smris_management.smris_management.doctype.manufacturing_machine.manufacturing_machine import (
			ManufacturingMachine,
		)

		amended_from: DF.Link | None
		customer: DF.Link
		design_files: DF.Attach
		element: DF.Data
		manufacturing_machines: DF.TableMultiSelect[ManufacturingMachine]
		preview: DF.AttachImage | None
		product: DF.Autocomplete
		quantity: DF.Int
		raw_material: DF.Autocomplete
		thickness: DF.Float
	# end: auto-generated types
	pass

	def before_save(self):
		self.generate_preview()

	def generate_preview(self):
		"""Generate a preview image for the Manufacturing Item."""
		if not self.design_files:
			frappe.throw(_("No design file found for preview generation."))

		design_file_path = Path(self.design_files)
		with NamedTemporaryFile(
			delete=False,
			suffix=design_file_path.suffix,
			prefix=design_file_path.stem + "_",
		) as temp_file:
			temp_file.write(get_file(self.design_files)[-1])
			saved_file_path = Path(temp_file.name)

		screenshot_bytes = get_sharecad_screenshot(saved_file_path)
		file_doc = save_file(
			f"preview_{self.name}.png",
			screenshot_bytes,
			"Manufacturing Item",
			self.name,
			decode=False,  # Since we're passing bytes directly
			is_private=1,  # Set to 1 if you want the file to be private
		)
		self.preview = file_doc.file_url
		saved_file_path.unlink()

The previous code works, but get_sharecad_screenshot takes a few seconds to complete, blocking the UI and making the user experience in Frappe Desk bad.

I wanted to transform this into a background by changing before_save like this:

	def before_save(self):
		frappe.enqueue_doc("Manufacturing Item", self.name, "generate_preview", queue="short")

Once I did that, it broke completely and shown errors like this in logs/worker.error.log:

Traceback (most recent call last):
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/rq/worker.py", line 1428, in perform_job
    rv = job.perform()
         ^^^^^^^^^^^^^
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/rq/job.py", line 1278, in perform
    self._result = self._execute()
                   ^^^^^^^^^^^^^^^
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/rq/job.py", line 1315, in _execute
    result = self.func(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 225, in execute_job
    retval = method(**kwargs)
             ^^^^^^^^^^^^^^^^
  File "/workspace/frappe-bench/apps/frappe/frappe/utils/background_jobs.py", line 190, in run_doc_method
    getattr(frappe.get_doc(doctype, name), doc_method)(**kwargs)
  File "/workspace/frappe-bench/apps/smris_management/smris_management/smris_management/doctype/manufacturing_item/manufacturing_item.py", line 92, in generate_preview
    screenshot_bytes = get_sharecad_screenshot(saved_file_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/frappe-bench/apps/smris_management/smris_management/smris_management/browser.py", line 15, in get_sharecad_screenshot
    browser = p.chromium.launch(headless=True)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/playwright/sync_api/_generated.py", line 14494, in launch
    self._sync(
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/playwright/_impl/_sync_base.py", line 115, in _sync
    return task.result()
           ^^^^^^^^^^^^^
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/playwright/_impl/_browser_type.py", line 98, in launch
    await self._channel.send(
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 69, in send
    return await self._connection.wrap_api_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workspace/frappe-bench/env/lib/python3.12/site-packages/playwright/_impl/_connection.py", line 558, in wrap_api_call
    raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.TargetClosedError: BrowserType.launch: Target page, context or browser has been closed
Browser logs:

<launching> /workspace/.cache/ms-playwright/chromium_headless_shell-1181/chrome-linux/headless_shell --disable-field-trial-config --disable-background-networking --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=AcceptCHFrame,AutoExpandDetailsElement,AvoidUnnecessaryBeforeUnloadCheckSync,CertificateTransparencyComponentUpdater,DestroyProfileOnBrowserClose,DialMediaRouteProvider,ExtensionManifestV2Disabled,GlobalMediaControls,HttpsUpgrades,ImprovedCookieControls,LazyFrameLoading,LensOverlay,MediaRouter,PaintHolding,ThirdPartyStoragePartitioning,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --disable-search-engine-choice-screen --unsafely-disable-devtools-self-xss-warnings --edge-skip-compat-layer-relaunch --enable-automation --headless --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-zGWkET --remote-debugging-pipe --no-startup-window
<launched> pid=488
[pid=488][err] /workspace/.cache/ms-playwright/chromium_headless_shell-1181/chrome-linux/headless_shell: error while loading shared libraries: libnspr4.so: cannot open shared object file: No such file or directory
Call log:
  - <launching> /workspace/.cache/ms-playwright/chromium_headless_shell-1181/chrome-linux/headless_shell --disable-field-trial-config --disable-background-networking --disable-background-timer-throttling --disable-backgrounding-occluded-windows --disable-back-forward-cache --disable-breakpad --disable-client-side-phishing-detection --disable-component-extensions-with-background-pages --disable-component-update --no-default-browser-check --disable-default-apps --disable-dev-shm-usage --disable-extensions --disable-features=AcceptCHFrame,AutoExpandDetailsElement,AvoidUnnecessaryBeforeUnloadCheckSync,CertificateTransparencyComponentUpdater,DestroyProfileOnBrowserClose,DialMediaRouteProvider,ExtensionManifestV2Disabled,GlobalMediaControls,HttpsUpgrades,ImprovedCookieControls,LazyFrameLoading,LensOverlay,MediaRouter,PaintHolding,ThirdPartyStoragePartitioning,Translate --allow-pre-commit-input --disable-hang-monitor --disable-ipc-flooding-protection --disable-popup-blocking --disable-prompt-on-repost --disable-renderer-backgrounding --force-color-profile=srgb --metrics-recording-only --no-first-run --password-store=basic --use-mock-keychain --no-service-autorun --export-tagged-pdf --disable-search-engine-choice-screen --unsafely-disable-devtools-self-xss-warnings --edge-skip-compat-layer-relaunch --enable-automation --headless --hide-scrollbars --mute-audio --blink-settings=primaryHoverType=2,availableHoverTypes=2,primaryPointerType=4,availablePointerTypes=4 --no-sandbox --user-data-dir=/tmp/playwright_chromiumdev_profile-zGWkET --remote-debugging-pipe --no-startup-window
  - <launched> pid=488
  - [pid=488][err] /workspace/.cache/ms-playwright/chromium_headless_shell-1181/chrome-linux/headless_shell: error while loading shared libraries: libnspr4.so: cannot open shared object file: No such file or directory

It looks like Playwright can’t launch Chromium from the background job because error while loading shared libraries: libnspr4.so: cannot open shared object file: No such file or directory.

Is there any way to give full access to the environment to the background jobs so they can do the same thing that Frappe does from the document controllers?

1 Like
2

@Hamza5 Were you able to find a working solution here?
We have a similar situation where we have python apps that we’d like to bring into our frappe site, but doing it with playwright doesn’t seem very straightforward. The less acceptable alternative would be to keep our app outside of frappe and have the app do CRUD operations via the REST API, but the lack of a python SDK (frappe-client was discontinued) requires us to do python REST api calls like cavemen.

3

No, I didn’t find anything, and I ended up accepting the problem of hanging the UI just to launch Playwright in the foreground, because I found no way to do it in the background (which is something I used to do easily outside of Frappe).

That was my first time building a production app using Frappe, and most likely it will be the last time. I can’t accept these critical limitations of this framework. Now that LLMs have become very powerful at coding, enterprise CRUD apps can be coded in a few days, so no need to keep using a limited framework like this.