[First Timer] Developing, running and testing local frappe framework repo

codecakes · June 4, 2024, 7:46pm

How do you develop frappe framework and test changes locally?
Once you have the actual source of frappe framework, can bench leverage this to build apps locally?
In other words, Is there a way to modify and test frappe framework itself in local to generate an app with or w/o bench?

Is --clone_from the way to setup frappe app locally from the local frappe source?

What I understand about bench is that it installs frappe as the first app in the site project.

What I have explored till now:

This probably is for bootstrapped frappe apps and won’t resolve my issue. I am going through the bench commands and I’m assuming it expects something like this?:
bench init --clone_from=<frappe-bench_path>

And with bench init one can pass the parameter clone_from?

The other option is to pass

--frappe-path TEXT path to frappe repo
--frappe-branch TEXT Clone a particular branch of frappe

But that would defeat the purpose.

spryng · June 4, 2024, 8:59pm

You can start from here.
https://frappeframework.com/docs/user/en/tutorial/create-an-app
This answer most of your questions
Open your terminal cd into your bench folder and type bench --help.

Void_Moon · June 5, 2024, 2:43am

I don’t get why you’d not want to use the bench… It makes life so much easier.

revant_one · June 5, 2024, 5:35am

In case you are trying to contribute,

Setup development bench: bench init --frappe-branch develop frappe-bench. Change branch to version-15-hotfix if you are sending hotfix.
cd frappe-bench/apps/frappe
git fetch --unshallow, also add your frappe fork as git remote
make changes to new branch and push the branch to your remote.
make pull request
keep pushing requested changes to the new branch created in step 4.

If you wish to setup bench with your fork of frappe. Use --frappe-path and --frappe-branch as per git repo and branch.

codecakes · June 5, 2024, 6:30am

Thanks. Is this part of contribution guide somewhere in frappeframework.com docs? I could not find it in the src CONTRIBUTING.

But I’m keeping contribution out of scope of my query for now. As I see here changes to local development also require a git version on remote (a local or remote git server) with --frappe_path otherwise it takes the default frappe branch. So essentially, this should work too correct?:
bench init --frappe-branch=<BRANCH> --frappe-path=<GIT_PATH> frappe-bench

revant_one · June 5, 2024, 8:15am

try it!

Remote and branch should exist.

In case of private repo it will ask for credentials.

codecakes · June 13, 2024, 8:34pm

Let me expand on my original query before sharing the issue I am running into, so that the context is clear:

I am working on a variant of frappe setup using bench, sort of frappe cookie-cutter, not unlike frappe docker although with principles of isolation and not scripting my way through configurations as is done in frappe_docker.

The intent is to be able to fork and make changes locally to frappe repo and unfortunately for now, since the bench relies on an SVN, be able to test these changes from git branch and install apps to test out the features.

With my containerized-cookie-cutter approach, I don’t have to get into a container shell and script my way through configurations and setup. Some of the configuration is hardened, for now, as I am testing this setup. Once the bench installs frappe, second run onwards doesn’t need any time to setup and the container simply spins up for testing features out.

What has been done

Here is a snippet of services from docker-compose.yml I’ve drafted:

services:
    mariadb:
        image: mariadb:latest
        container_name: mariadb_frappe_container
        volumes:
            - type: bind
              source: ./infra/config/my.cnf
              target: /etc/mysql/my.cnf
            - type: volume
              source: maria-volume
              target: /var/lib/mysql
            - type: bind
              source: ./bench_apps/mariadb_config.sh
              target: /bin/mariadb_config.sh
        environment:
          - MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
          - MARIADB_DATABASE=${MARIADB_DATABASE}
          - MARIADB_USER=${MARIADB_USER}
          - MARIADB_PASSWORD=${MARIADB_PASSWORD}
        healthcheck:
          test: mariadb-admin ping -h localhost -u root --password=${MARIADB_ROOT_PASSWORD}
          interval: 10s
          retries: 5
        deploy:
          restart_policy:
            condition: on-failure
        ports:
            - target: 3306
              published: "3306"
              mode: ingress
        networks:
            - thebridge

    redis-cache:
        <<: *common-redis
        image: redis:latest
        container_name: redis-cache

    redis-rq:
        <<: *common-redis
        build:
            context: .
            dockerfile: Dockerfile.rq
            args:
              PYTHON_VERSION: "${PYTHON_VERSION}"
        environment:
          - PYTHON_VERSION=${PYTHON_VERSION}
        container_name: redis_frappe_rq_container
        ports:
          - target: 6380
            published: "6380"
            mode: ingress

    frappe:
      entrypoint: ["bash", "-c", "./bootstrap.sh"]
      working_dir: /app/bench_apps
      build:
        context: .
        dockerfile: Dockerfile
        args:
          PYTHON_VERSION: "${PYTHON_VERSION}"
          FRAPPE_BRANCH: "${FRAPPE_BRANCH}"
      environment:
        - PYTHON_VERSION=${PYTHON_VERSION}
        - FRAPPE_BRANCH=${FRAPPE_BRANCH}
        - SHELL=/bin/bash
        - MARIADB_ROOT_PASSWORD=${MARIADB_ROOT_PASSWORD}
        - MARIADB_PASSWORD=${MARIADB_PASSWORD}
        - MARIADB_DATABASE=${MARIADB_DATABASE}
        - MARIADB_USER=${MARIADB_USER}
      depends_on:
        - mariadb
        - redis-cache
        - redis-rq
      volumes:
        - workspace:/workspace:cached
        - type: bind
          source: bench_apps/
          target: /app/bench_apps/
        - type: bind
          source: infra/config/my.cnf
          target: /usr/local/etc/my.cnf
        - type: bind
          source: bench_apps/bootstrap.sh
          target: /app/bench_apps/bootstrap.sh
      ports:
        - 8000-8005:8000-8005
        - 9000-9005:9000-9005
      networks:
        - thebridge

In future iterations, I will add $FRAPPE_PATHas well, circling back to my original query on testing what one is developing on git. Some of these settings have been taken from docker-frappe, some are different. For instance:

mariadb configurations are mapped
frappe service also maps volumes for certain hardening configrations. This allows dynamic scripting for running the actual bench setup the way I would like.

For instance, bootstrap.sh is basically:
chmod +x wait-for-it.sh && ./wait-for-it.sh mariadb:3306 -- bash -c "sh bench_setup_scripts.sh";

where bench_setup_scripts.sh is what allows me to make changes while still decoupling the post-setup changes from the setup and still run the same container. This way i can deploy a swam in the future with the same cookie-cutter approach and make changes on many instances from same script. This is one possibility of what bench_setup_script could be:


#!/bin/bash

echo "running bench setup script";

if [ ! -d "frappe-bench" ]; then
  echo "creating frappe-bench";
  bench init --dev --clone-without-update --verbose --ignore-exist --no-backups --no-procfile --skip-redis-config-generation --frappe-branch=${FRAPPE_BRANCH} frappe-bench
else
  echo "frappe-bench already exists";
fi;

echo "switching directory to frappe-bench";
cd frappe-bench/

bench set-mariadb-host mariadb
bench set-redis-cache-host redis-cache:6379
bench set-redis-queue-host redis-cache:6379
bench set-redis-socketio-host redis-cache:6379

# setup dummy site
if [ ! -d "sites/lms.localhost" ]; then
  bench get-app lms https://github.com/frappe/lms.git;
  bench new-site lms.localhost --force --admin-password=admin --db-host=mariadb --db-port=3306 --db-password=${MARIADB_PASSWORD} --db-user=${MARIADB_USER} --mariadb-root-password=${MARIADB_ROOT_PASSWORD} --mariadb-user-host-login-scope="'${MARIADB_USER}'@'mariadb'" --verbose --install-app lms
fi;

bench reinstall --yes --admin-password=admin --mariadb-root-password=${MARIADB_ROOT_PASSWORD}

bench use lms.localhost

echo "starting frappe server..."
bench serve

The issue

I am able to run this in developer mode on http://127.0.0.1:8000/ just fine, however, with one caveat which still needs to be fixed and this is due to my lack of understanding of the tightly knit and complex set of commands and configurations around how frappe interacts with the database as I am still going through the docs.

I get this error probably due to my lack of understanding of how to invoke this command:

bench new-site lms.localhost --force --admin-password=admin --db-host=mariadb --db-port=3306 --db-password=${MARIADB_PASSWORD} --db-user=${MARIADB_USER} --mariadb-root-password=${MARIADB_ROOT_PASSWORD} --mariadb-user-host-login-scope="'${MARIADB_USER}'@'mariadb'" --verbose --install-app lms


frappe-dev-frappe-1        |   File "<frozen runpy>", line 198, in _run_module_as_main
frappe-dev-frappe-1        |   File "<frozen runpy>", line 88, in _run_code
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 128, in <module>
frappe-dev-frappe-1        |     main()
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 34, in main
frappe-dev-frappe-1        |     FrappeCommandGroup(commands=commands)(prog_name="bench")
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
frappe-dev-frappe-1        |     return self.main(*args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1078, in main
frappe-dev-frappe-1        |     rv = self.invoke(ctx)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
frappe-dev-frappe-1        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
frappe-dev-frappe-1        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
frappe-dev-frappe-1        |     return ctx.invoke(self.callback, **ctx.params)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 783, in invoke
frappe-dev-frappe-1        |     return __callback(*args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
frappe-dev-frappe-1        |     return f(get_current_context(), *args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 29, in _func
frappe-dev-frappe-1        |     ret = f(frappe._dict(ctx.obj), *args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/commands/site.py", line 433, in reinstall
frappe-dev-frappe-1        |     _reinstall(site, admin_password, db_root_username, db_root_password, yes, verbose=context.verbose)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/commands/site.py", line 451, in _reinstall
frappe-dev-frappe-1        |     frappe.clear_cache()
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/__init__.py", line 1053, in clear_cache
frappe-dev-frappe-1        |     for key in frappe.get_hooks("persistent_cache_keys"):
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/__init__.py", line 1681, in get_hooks
frappe-dev-frappe-1        |     hooks = _dict(_load_app_hooks())
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/utils/caching.py", line 59, in wrapper
frappe-dev-frappe-1        |     return_val = func(*args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/__init__.py", line 1646, in _load_app_hooks
frappe-dev-frappe-1        |     apps = [app_name] if app_name else get_installed_apps(_ensure_on_bench=True)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/utils/caching.py", line 59, in wrapper
frappe-dev-frappe-1        |     return_val = func(*args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/__init__.py", line 1615, in get_installed_apps
frappe-dev-frappe-1        |     installed = json.loads(db.get_global("installed_apps") or "[]")
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 1010, in get_global
frappe-dev-frappe-1        |     return self.get_default(key, user)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 1014, in get_default
frappe-dev-frappe-1        |     d = self.get_defaults(key, parent)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 1030, in get_defaults
frappe-dev-frappe-1        |     defaults = frappe.defaults.get_defaults_for(parent)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/defaults.py", line 240, in get_defaults_for
frappe-dev-frappe-1        |     .run(as_dict=True)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/query_builder/utils.py", line 84, in execute_query
frappe-dev-frappe-1        |     result = frappe.db.sql(query, params, *args, **kwargs)  # nosemgrep
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 264, in sql
frappe-dev-frappe-1        |     traceback.print_stack()
frappe-dev-frappe-1        | Error in query:
frappe-dev-frappe-1        | ('SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`=%(param1)s ORDER BY `creation`', {'param1': '__global'})
frappe-dev-frappe-1        |   File "<frozen runpy>", line 198, in _run_module_as_main
frappe-dev-frappe-1        |   File "<frozen runpy>", line 88, in _run_code
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 128, in <module>
frappe-dev-frappe-1        |     main()
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/utils/bench_helper.py", line 34, in main
frappe-dev-frappe-1        |     FrappeCommandGroup(commands=commands)(prog_name="bench")
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
frappe-dev-frappe-1        |     return self.main(*args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1078, in main
frappe-dev-frappe-1        |     rv = self.invoke(ctx)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
frappe-dev-frappe-1        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
frappe-dev-frappe-1        |     return _process_result(sub_ctx.command.invoke(sub_ctx))
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
frappe-dev-frappe-1        |     return ctx.invoke(self.callback, **ctx.params)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/core.py", line 783, in invoke
frappe-dev-frappe-1        |     return __callback(*args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/env/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
frappe-dev-frappe-1        |     return f(get_current_context(), *args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/commands/__init__.py", line 29, in _func
frappe-dev-frappe-1        |     ret = f(frappe._dict(ctx.obj), *args, **kwargs)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/commands/site.py", line 433, in reinstall
frappe-dev-frappe-1        |     _reinstall(site, admin_password, db_root_username, db_root_password, yes, verbose=context.verbose)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/commands/site.py", line 463, in _reinstall
frappe-dev-frappe-1        |     _new_site(
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/installer.py", line 74, in _new_site
frappe-dev-frappe-1        |     enable_scheduler = _is_scheduler_enabled(site)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/installer.py", line 30, in _is_scheduler_enabled
frappe-dev-frappe-1        |     enable_scheduler = cint(frappe.db.get_single_value("System Settings", "enable_scheduler"))
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 846, in get_single_value
frappe-dev-frappe-1        |     ).run()
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/query_builder/utils.py", line 84, in execute_query
frappe-dev-frappe-1        |     result = frappe.db.sql(query, params, *args, **kwargs)  # nosemgrep
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 264, in sql
frappe-dev-frappe-1        |     traceback.print_stack()
frappe-dev-frappe-1        | Error in query:
frappe-dev-frappe-1        | ('SELECT `value` FROM `tabSingles` WHERE `doctype`=%(param1)s AND `field`=%(param2)s', {'param1': 'System Settings', 'param2': 'enable_scheduler'})

In the url: http://127.0.0.1:8000/app/ takes me to http://127.0.0.1:8000/app/users

this is what i see:

I would like to understand if the command being run is wrongly executed and what exactly is going wrong here:

File "/app/bench_apps/frappe-bench/apps/frappe/frappe/defaults.py", line 240, in get_defaults_for
frappe-dev-frappe-1        |     .run(as_dict=True)
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/query_builder/utils.py", line 84, in execute_query
frappe-dev-frappe-1        |     result = frappe.db.sql(query, params, *args, **kwargs)  # nosemgrep
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 264, in sql
frappe-dev-frappe-1        |     traceback.print_stack()
frappe-dev-frappe-1        | Error in query:
frappe-dev-frappe-1        | ('SELECT `defkey`,`defvalue` FROM `tabDefaultValue` WHERE `parent`=%(param1)s ORDER BY `creation`', {'param1': '__global'})

 File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 846, in get_single_value
frappe-dev-frappe-1        |     ).run()
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/query_builder/utils.py", line 84, in execute_query
frappe-dev-frappe-1        |     result = frappe.db.sql(query, params, *args, **kwargs)  # nosemgrep
frappe-dev-frappe-1        |   File "/app/bench_apps/frappe-bench/apps/frappe/frappe/database/database.py", line 264, in sql
frappe-dev-frappe-1        |     traceback.print_stack()
frappe-dev-frappe-1        | Error in query:
frappe-dev-frappe-1        | ('SELECT `value` FROM `tabSingles` WHERE `doctype`=%(param1)s AND `field`=%(param2)s', {'param1': 'System Settings', 'param2': 'enable_scheduler'})

Fyi, this is how this container setup is run on my local:

MARIADB_ROOT_PASSWORD=<MARIADB_PASSWORD> MARIADB_DATABASE=<MARIADB_DATABASE> MARIADB_USER=<MARIADB_USER> MARIADB_PASSWORD=<MARIADB_PASSWORD> PYTHON_VERSION=<PYTHON3_VERSION> FRAPPE_BRANCH=develop  podman compose --project-name frappe-dev -f docker-compose.yml up --build

the env vars are picked up from .env file.

The common_site_config.json is:

{
 "background_workers": 1,
 "db_host": "mariadb",
 "default_site": "lms.localhost",
 "developer_mode": 1,
 "file_watcher_port": 6787,
 "frappe_user": "localuser",
 "gunicorn_workers": 5,
 "live_reload": true,
 "rebase_on_pull": false,
 "redis_cache": "redis://redis-cache:6379",
 "redis_queue": "redis://redis-cache:6379",
 "redis_socketio": "redis://redis-cache:6379",
 "restart_supervisor_on_update": false,
 "restart_systemd_on_update": false,
 "serve_default_site": true,
 "shallow_clone": true,
 "socketio_port": 9000,
 "use_redis_auth": false,
 "webserver_port": 8000
}

If you could point to the right steps of how to go about with bench new-site and if there are any other pre-requisites that I have missed which needs to be done before hitting bench start, that would be helpful.

revant_one · June 14, 2024, 12:59pm

codecakes:

  bench new-site lms.localhost --force --admin-password=admin --db-host=mariadb --db-port=3306 --db-password=${MARIADB_PASSWORD} --db-user=${MARIADB_USER} --mariadb-root-password=${MARIADB_ROOT_PASSWORD} --mariadb-user-host-login-scope="'${MARIADB_USER}'@'mariadb'" --verbose --install-app lms

add --no-mariadb-socket to this command and the site user will be created with wildcard host. That way even after container restarts and ip changes the access to user is not denied.

codecakes · June 14, 2024, 7:07pm

Yes, this is what solved it:

bench new-site lms.localhost --force --admin-password=admin --db-host=mariadb --db-port=3306 --db-password=${MARIADB_PASSWORD} --db-user=${MARIADB_USER} --mariadb-root-password=${MARIADB_ROOT_PASSWORD} --mariadb-user-host-login-scope="%" --verbose --install-app lms

I used --mariadb-user-host-login-scope since --no-mariadb-socket is deprecated and --mariadb-user-host-login-scope=“%”. both mariadb-user-host-login-scope and mariadb-socket are somewhat confusing.