[New feature proposal] Automated server side python script - pls community feedback?

In Odoo there is automated server action feature, this feature allows user to define server actions which is auto triggered when the relevant document inserted/updated/submitted/deleted, the following actions are supported

  • update the fields of the target document
  • create new document
  • execute python code
  • send mail
  • execute client action

This feature is more powerful than the following existing features

  • custom script: client side javascript only, not work for batch import and REST interface
  • web hook: developer oriented, need external HTTP server, it is asynchronous, can not be used for validation check

There is already Email Alert feature available in ERPNext, User can define trigger doctype, condition , message and recipient, also can assign a fixed value for one field after email alert sent, in other word, the current Email Alert feature already implemented the above mentioned two action types: update the fields of the target document and send mail, but it bundled this two actions together, and only allow fixed value assignment for 1 field!

Based on the above, I planned to base on existing Email Alert feature to implement the ERPNext’s automated server action, before diving into the program, I would like to get the community feedback concern the following
– any use cases?
– any further requirement?

Many thanks.

3 Likes

Have a look at hooks.py, that’s the way to define action based on triggers.

Follow this related issue

It doesn’t cover everything mentioned in opening post.

The most important thing is to enable non-developer to fully control/ heavily customize ERPNext per their own business requirement, without this proposed feature, the extent of customization can be done by key user / service provider is limited, especially for clients hosted on SaaS.
Till now I almost realized the above mentioned following features

  • update the fields of the target document
  • create new document
  • execute python code

I also plan to make the server action accessible from the front end ( via button) which currently can only be defined by developer in python file under server folder. in Odoo the server action can be made available via button under the More context menu in the front end form.

Before my local development become a new pull request, I would like to get the community’s feedback concern the real use cases, and further concerns and requirement, I will base on all the info to refine the design and documentation accordingly, I really like it to be part of v11!

1 Like

Today I managed to make it work for front end button to call the server action, with this feature SaaS user can do almost all heavy customizing as required, anyone interested in this feature? Any use case, or concern, should I setup a new bounty?

3 Likes

I think this could be very useful feature. Looking at issue, mentioned by @revant_one, I noticed it’s added to January 2018 milestone. Does it mean that somebody is already working on it?

Edit: apparently shreyashah115 is working on it as she is assignee.

Please follow the GitHub issue link posted by Revant. In the discussion thread of that GitHub issue, Maxwell Morais has a link to a youtube screencast of his solution. Kindly review and comment further.

1 Like

Thanks for the update. Here is the video: Improvements in Coreflow UI Engine - YouTube

strixaluco, thanks for the sharing, I just watched the video, it implemented a very user friendly way of generating the custom script, so it is a feature for enhancing the existing custom script (javascript), make it more easy to customize custom script.

but my proposed feature is on server side in Python, it works like below

  1. user create custom server action with python code to be called from custom button on the form, or auto called/triggered when user click the standard SAVE/Submit button
    small tips for python code
    no function definition allowed,
    input and return arguments to be passed via frappe.local.form_dict
    the return value to be passed via frappe.local.form_dict.update(‘custom_server_action_out’: data})
  2. if it is called via the custom button, in the callback function, point the method to be called to the custom server action
    like method: ‘custom_server_action.make_purchase_receipt’

custom_server_action_small1

take the Make->Receipt custom button in Purchase Order form as an example, following the steps below the user can implement the same standard feature without touching the python file which is not accessible on SaaS, so with this new feature either the internal IT or service provider can base on SaaS version to do heavy customization as needed!

  1. copy the below relevant code from purchase_order.py to the newly creation custom server action

def set_missing_values(source, target):
target.ignore_pricing_rule = 1
target.run_method(“set_missing_values”)
target.run_method(“calculate_taxes_and_totals”)

@frappe.whitelist()
def make_purchase_receipt(source_name, target_doc=None):
def update_item(obj, target, source_parent):
target.qty = flt(obj.qty) - flt(obj.received_qty)
target.stock_qty = (flt(obj.qty) - flt(obj.received_qty)) * flt(obj.conversion_factor)
target.amount = (flt(obj.qty) - flt(obj.received_qty)) * flt(obj.rate)
target.base_amount = (flt(obj.qty) - flt(obj.received_qty)) *
flt(obj.rate) * flt(source_parent.conversion_rate)

doc = get_mapped_doc("Purchase Order", source_name,	{
	"Purchase Order": {
		"doctype": "Purchase Receipt",
		"field_map": {
			"per_billed": "per_billed"
		},
		"validation": {
			"docstatus": ["=", 1],
		}
	},
	"Purchase Order Item": {
		"doctype": "Purchase Receipt Item",
		"field_map": {
			"name": "purchase_order_item",
			"parent": "purchase_order",
			"bom": "bom"
		},
		"postprocess": update_item,
		"condition": lambda doc: abs(doc.received_qty) < abs(doc.qty) and doc.delivered_by_supplier!=1
	},
	"Purchase Taxes and Charges": {
		"doctype": "Purchase Taxes and Charges",
		"add_if_empty": True
	}
}, target_doc, set_missing_values)

return doc
  1. adapt the code a little bit, updated version as below
    source_name=frappe.local.form_dict.get(‘source_name’)
    selected_children=frappe.local.form_dict.get(‘selected_children’)
    if selected_children:
    selected_children = json.loads(selected_children)
    frappe.flags.selected_children = selected_children or None

doc = frappe.model.mapper.get_mapped_doc(“Purchase Order”, source_name, {
“Purchase Order”: {
“doctype”: “Purchase Receipt”,
“field_map”: {
“per_billed”: “per_billed”
},
“validation”: {
“docstatus”: [“=”, 1],
}
},
“Purchase Order Item”: {
“doctype”: “Purchase Receipt Item”,
“field_map”: {
“name”: “purchase_order_item”,
“parent”: “purchase_order”,
“bom”: “bom”
},
“condition”: lambda doc: abs(doc.received_qty) < abs(doc.qty) and
doc.delivered_by_supplier!=1
},
“Purchase Taxes and Charges”: {
“doctype”: “Purchase Taxes and Charges”,
“add_if_empty”: True
}
})

source= frappe.get_doc(‘Purchase Order’, source_name)
flt = frappe.utils.flt
for (obj,target) in zip(source.items, doc.items):
target.qty = flt(obj.qty) - flt(obj.received_qty)
target.stock_qty = (flt(obj.qty) - flt(obj.received_qty)) * flt(obj.conversion_factor)
target.amount = (flt(obj.qty) - flt(obj.received_qty)) * flt(obj.rate)
target.base_amount = (flt(obj.qty) - flt(obj.received_qty)) *
flt(obj.rate) * flt(source.conversion_rate)

doc.ignore_pricing_rule = 1
doc.run_method(“set_missing_values”)
doc.run_method(“calculate_taxes_and_totals”)

frappe.local.form_dict.update({‘custom_server_action_out’:doc})

Explanation of the adaption
2.1 the python code to be compiled and eval(executed) dynamically does not support function definition and itself is not function, so the input argument and return value should be passed via passed in local context or global context, here frappe.local.form_dict is used
2.2 no import statement supported, frappe passed in as local context, the imported local variable should be replaced with full path, e.g get_mapped_doc should be frappe.model.mapper.get_mapped_doc, flt should be frappe.utils.flt
2.3 internal/embedded function definition and call it is not supported, implement the same for the postprocess, updateitem and set missing value after get_mapped_doc.

  1. refer the method call to the newly created custom server action like below in purchase_order.js
    make_purchase_receipt: function() {
    frappe.model.open_mapped_doc({
    // method: “erpnext.buying.doctype.purchase_order.purchase_order.make_purchase_receipt”,
    method: “custom_server_action.make_purchase_receipt”,
    frm: cur_frm
    })
    },

  2. go to purchase order form, click the Make->Receipt button, It will call your code in custom server action!

with this you can do almost anything you want!

@szufisher I already have a server version for the same behavior

In past, and when I mean past I’m telling 2012 and bofore, frappe aka wnframework already have into the :package: the support for Server Side Script.

The issue with that is that, you never will prevent the user to do bad things.

In a shared environment like frappe :cloud:, if you enable python scripts, do you will give to much power for the users, and the users will start to do crazy things into the server.

In python I don’t know any way to sandbox untrusted scripts, if you need make an pr to that behavior and get this approved you need to restrict the execution environment, what by my knowledgment , is really hard to implement in python.

2 Likes

Dear max_morais_dmm,
Yes you concern about the security is reasonable, need to be evaluated before this feature to be merged into core. at the moment, I just adapted the Odoo version of the safe_eval, which in turn is based on other project, here the code

Code

"""
safe_eval module - methods intended to provide more restricted alternatives to
                   evaluate simple and/or untrusted code.
Methods in this module are typically used as alternatives to eval() to parse
formula strings, conditions and expressions, mostly based on locals
condition/math builtins.
"""

# Module partially ripped from/inspired by several different sources:
#  - http://code.activestate.com/recipes/286134/
#  - safe_eval in lp:~xrg/openobject-server/optimize-5.0
#  - safe_eval in tryton http://hg.tryton.org/hgwebdir.cgi/trytond/rev/bbb5f73319ad
import dis
from opcode import HAVE_ARGUMENT, opmap, opname

import functools
from types import CodeType
import logging
import sys
import frappe
from six import text_type

unsafe_eval = eval

__all__ = ['test_expr', 'safe_eval', 'const_eval']

# The time module is usually already provided in the safe_eval environment
# but some code, e.g. datetime.datetime.now() (Windows/Python 2.5.2, bug
# lp:703841), does import time.
_ALLOWED_MODULES = ['_strptime', 'math', 'time']

_UNSAFE_ATTRIBUTES = ['f_builtins', 'f_globals', 'f_locals', 'gi_frame',
                      'co_code', 'func_globals']
_POSSIBLE_OPCODES_P3 = [
    # opcodes for `with` statement cleanup process
    'WITH_CLEANUP_START', 'WITH_CLEANUP_FINISH',
    # f-strings
    'FORMAT_VALUE', 'BUILD_STRING',
    # extended iterable unpacking: LHS has * e.g. `a, *b, c = thing()`
    'UNPACK_EX',
    # collection literals with unpacking e.g. [*a, *b]
    'BUILD_LIST_UNPACK', 'BUILD_TUPLE_UNPACK', 'BUILD_SET_UNPACK', 'BUILD_MAP_UNPACK',
    # packs args/kwargs for calls with multiple unpacks e.g. foo(*a, *b, *c)
    'BUILD_TUPLE_UNPACK_WITH_CALL', 'BUILD_MAP_UNPACK_WITH_CALL',
    # ???
    'GET_YIELD_FROM_ITER',
    # matrix operator
    'BINARY_MATRIX_MULTIPLY', 'INPLACE_MATRIX_MULTIPLY',
]

# opcodes necessary to build literal values
_CONST_OPCODES = set(opmap[x] for x in [
    # stack manipulations
    'POP_TOP', 'ROT_TWO', 'ROT_THREE', 'ROT_FOUR', 'DUP_TOP', 'DUP_TOPX',
    'DUP_TOP_TWO',  # replaces DUP_TOPX in P3
    'LOAD_CONST',
    'RETURN_VALUE', # return the result of the literal/expr evaluation
    # literal collections
    'BUILD_LIST', 'BUILD_MAP', 'BUILD_TUPLE', 'BUILD_SET',
    # 3.6: literal map with constant keys https://bugs.python.org/issue27140
    'BUILD_CONST_KEY_MAP',
    # until Python 3.5, literal maps are compiled to creating an empty map
    # (pre-sized) then filling it key by key
    'STORE_MAP',
] if x in opmap)

# operations on literal values
_EXPR_OPCODES = _CONST_OPCODES.union(set(opmap[x] for x in [
    'UNARY_POSITIVE', 'UNARY_NEGATIVE', 'UNARY_NOT',
    'UNARY_INVERT', 'BINARY_POWER', 'BINARY_MULTIPLY',
    'BINARY_DIVIDE', 'BINARY_FLOOR_DIVIDE', 'BINARY_TRUE_DIVIDE',
    'BINARY_MODULO', 'BINARY_ADD', 'BINARY_SUBTRACT', 'BINARY_SUBSCR',
    'BINARY_LSHIFT', 'BINARY_RSHIFT', 'BINARY_AND', 'BINARY_XOR',
    'BINARY_OR', 'INPLACE_ADD', 'INPLACE_SUBTRACT', 'INPLACE_MULTIPLY',
    'INPLACE_DIVIDE', 'INPLACE_REMAINDER', 'INPLACE_POWER',
    'INPLACE_LEFTSHIFT', 'INPLACE_RIGHTSHIFT', 'INPLACE_AND',
    'INPLACE_XOR','INPLACE_OR', 'STORE_SUBSCR',
    # slice operations (Python 3 only has BUILD_SLICE)
    'SLICE+0', 'SLICE+1', 'SLICE+2', 'SLICE+3', 'BUILD_SLICE',
    # comprehensions
    'LIST_APPEND', 'MAP_ADD', 'SET_ADD',
    'COMPARE_OP',
] if x in opmap))

_SAFE_OPCODES = _EXPR_OPCODES.union(set(opmap[x] for x in [
    'POP_BLOCK', 'POP_EXCEPT', # Seems to be a special-case of POP_BLOCK for P3
    'SETUP_LOOP', 'BREAK_LOOP', 'CONTINUE_LOOP',
    'MAKE_FUNCTION', 'CALL_FUNCTION',
    'EXTENDED_ARG',  # P3.6 for long jump offsets.
    # P3: https://bugs.python.org/issue27213
    'CALL_FUNCTION_EX',
    # Already in P2 but apparently the first one is used more aggressively in P3
    'CALL_FUNCTION_KW', 'CALL_FUNCTION_VAR', 'CALL_FUNCTION_VAR_KW',
    'GET_ITER', 'FOR_ITER', 'YIELD_VALUE',
    'JUMP_FORWARD', 'JUMP_IF_TRUE', 'JUMP_IF_FALSE', 'JUMP_ABSOLUTE',
    # New in Python 2.7 - http://bugs.python.org/issue4715 :
    'JUMP_IF_FALSE_OR_POP', 'JUMP_IF_TRUE_OR_POP', 'POP_JUMP_IF_FALSE',
    'POP_JUMP_IF_TRUE', 'SETUP_EXCEPT', 'END_FINALLY', 'RAISE_VARARGS',
     #add STORE_ATTR
    'LOAD_NAME', 'STORE_NAME', 'DELETE_NAME', 'LOAD_ATTR', 'STORE_ATTR',  
    'LOAD_FAST', 'STORE_FAST', 'DELETE_FAST', 'UNPACK_SEQUENCE',
    'LOAD_GLOBAL', # Only allows access to restricted globals
] if x in opmap))

_logger = logging.getLogger(__name__)

if hasattr(dis, 'get_instructions'):
    def _get_opcodes(codeobj):
        """_get_opcodes(codeobj) -> [opcodes]
        Extract the actual opcodes as an iterator from a code object
        >>> c = compile("[1 + 2, (1,2)]", "", "eval")
        >>> list(_get_opcodes(c))
        [100, 100, 23, 100, 100, 102, 103, 83]
        """
        return (i.opcode for i in dis.get_instructions(codeobj))
else:
    def _get_opcodes(codeobj):
        i = 0
        byte_codes = codeobj.co_code
        while i < len(byte_codes):
            code = ord(byte_codes[i:i+1])
            yield code

            if code >= HAVE_ARGUMENT:
                i += 3
            else:
                i += 1

def assert_no_dunder_name(code_obj, expr):
    """ assert_no_dunder_name(code_obj, expr) -> None
    Asserts that the code object does not refer to any "dunder name"
    (__$name__), so that safe_eval prevents access to any internal-ish Python
    attribute or method (both are loaded via LOAD_ATTR which uses a name, not a
    const or a var).
    Checks that no such name exists in the provided code object (co_names).
    :param code_obj: code object to name-validate
    :type code_obj: CodeType
    :param str expr: expression corresponding to the code object, for debugging
                     purposes
    :raises NameError: in case a forbidden name (containing two underscores)
                       is found in ``code_obj``
    .. note:: actually forbids every name containing 2 underscores
    """
    for name in code_obj.co_names:
        if "__" in name or name in _UNSAFE_ATTRIBUTES:
            raise NameError('Access to forbidden name %r (%r)' % (name, expr))

def assert_valid_codeobj(allowed_codes, code_obj, expr):
    """ Asserts that the provided code object validates against the bytecode
    and name constraints.
    Recursively validates the code objects stored in its co_consts in case
    lambdas are being created/used (lambdas generate their own separated code
    objects and don't live in the root one)
    :param allowed_codes: list of permissible bytecode instructions
    :type allowed_codes: set(int)
    :param code_obj: code object to name-validate
    :type code_obj: CodeType
    :param str expr: expression corresponding to the code object, for debugging
                     purposes
    :raises ValueError: in case of forbidden bytecode in ``code_obj``
    :raises NameError: in case a forbidden name (containing two underscores)
                       is found in ``code_obj``
    """
    assert_no_dunder_name(code_obj, expr)

    # almost twice as fast as a manual iteration + condition when loading
    # /web according to line_profiler
    codes = set(_get_opcodes(code_obj)) - allowed_codes
    if codes:
        raise ValueError("forbidden opcode(s) in %r: %s" % (expr, ', '.join(opname[x] for x in codes)))

    for const in code_obj.co_consts:
        if isinstance(const, CodeType):
            assert_valid_codeobj(allowed_codes, const, 'lambda')

def test_expr(expr, allowed_codes, mode="eval"):
    """test_expr(expression, allowed_codes[, mode]) -> code_object
    Test that the expression contains only the allowed opcodes.
    If the expression is valid and contains only allowed codes,
    return the compiled code object.
    Otherwise raise a ValueError, a Syntax Error or TypeError accordingly.
    """
    try:
        if mode == 'eval':
            # eval() does not like leading/trailing whitespace
            expr = expr.strip()
        code_obj = compile(expr, "", mode)
    except (SyntaxError, TypeError, ValueError):
        raise
    except Exception as e:
        exc_info = sys.exc_info()
        #pycompat.reraise(ValueError, ValueError('"%s" while compiling\n%r' % (ustr(e), expr)), exc_info[2])
    assert_valid_codeobj(allowed_codes, code_obj, expr)
    return code_obj


def const_eval(expr):
    """const_eval(expression) -> value
    Safe Python constant evaluation
    Evaluates a string that contains an expression describing
    a Python constant. Strings that are not valid Python expressions
    or that contain other code besides the constant raise ValueError.
    >>> const_eval("10")
    10
    >>> const_eval("[1,2, (3,4), {'foo':'bar'}]")
    [1, 2, (3, 4), {'foo': 'bar'}]
    >>> const_eval("1+2")
    Traceback (most recent call last):
    ...
    ValueError: opcode BINARY_ADD not allowed
    """
    c = test_expr(expr, _CONST_OPCODES)
    return unsafe_eval(c)

def expr_eval(expr):
    """expr_eval(expression) -> value
    Restricted Python expression evaluation
    Evaluates a string that contains an expression that only
    uses Python constants. This can be used to e.g. evaluate
    a numerical expression from an untrusted source.
    >>> expr_eval("1+2")
    3
    >>> expr_eval("[1,2]*2")
    [1, 2, 1, 2]
    >>> expr_eval("__import__('sys').modules")
    Traceback (most recent call last):
    ...
    ValueError: opcode LOAD_NAME not allowed
    """
    c = test_expr(expr, _EXPR_OPCODES)
    return unsafe_eval(c)

def _import(name, globals=None, locals=None, fromlist=None, level=-1):
    if globals is None:
        globals = {}
    if locals is None:
        locals = {}
    if fromlist is None:
        fromlist = []
    if name in _ALLOWED_MODULES:
        return __import__(name, globals, locals, level)
    raise ImportError(name)
_BUILTINS = {
    '__import__': _import,
    'True': True,
    'False': False,
    'None': None,
    'bytes': bytes,
    'str': str,
    'unicode': text_type,
    'bool': bool,
    'int': int,
    'float': float,
    'enumerate': enumerate,
    'dict': dict,
    'list': list,
    'tuple': tuple,
    'map': map,
    'abs': abs,
    'min': min,
    'max': max,
    'sum': sum,
    'reduce': functools.reduce,
    'filter': filter,
    'round': round,
    'len': len,
    'repr': repr,
    'set': set,
    'all': all,
    'any': any,
    'ord': ord,
    'chr': chr,
    'divmod': divmod,
    'isinstance': isinstance,
    'range': range,
    'xrange': range,
    'zip': zip,
    'Exception': Exception,
}
def safe_eval(expr, globals_dict=None, locals_dict=None, mode="eval", nocopy=False, locals_builtins=False):
    """safe_eval(expression[, globals[, locals[, mode[, nocopy]]]]) -> result
    System-restricted Python expression evaluation
    Evaluates a string that contains an expression that mostly
    uses Python constants, arithmetic expressions and the
    objects directly provided in context.
    This can be used to e.g. evaluate
    an OpenERP domain expression from an untrusted source.
    :throws TypeError: If the expression provided is a code object
    :throws SyntaxError: If the expression provided is not valid Python
    :throws NameError: If the expression provided accesses forbidden names
    :throws ValueError: If the expression provided uses forbidden bytecode
    """
    if type(expr) is CodeType:
        raise TypeError("safe_eval does not allow direct evaluation of code objects.")

    # prevent altering the globals/locals from within the sandbox
    # by taking a copy.
    if not nocopy:
        # isinstance() does not work below, we want *exactly* the dict class
        if (globals_dict is not None and type(globals_dict) is not dict) \
                or (locals_dict is not None and type(locals_dict) is not dict):
            _logger.warning(
                "Looks like you are trying to pass a dynamic environment, "
                "you should probably pass nocopy=True to safe_eval().")
        if globals_dict is not None:
            globals_dict = dict(globals_dict)
        if locals_dict is not None:
            locals_dict = dict(locals_dict)

    if globals_dict is None:
        globals_dict = {}

    globals_dict['__builtins__'] = _BUILTINS
    if locals_builtins:
        if locals_dict is None:
            locals_dict = {}
        locals_dict.update(_BUILTINS)
    c = test_expr(expr, _SAFE_OPCODES, mode=mode)
    try:
        return unsafe_eval(c, globals_dict, locals_dict)
   
    except Exception as e:
        exc_info = sys.exc_info()
        #pycompat.reraise(ValueError, ValueError('%s: "%s" while evaluating\n%r' % (ustr(type(e)), ustr(e), expr)), exc_info[2])
def test_python_expr(expr, mode="eval"):
    try:
        test_expr(expr, _SAFE_OPCODES, mode=mode)
    except (SyntaxError, TypeError, ValueError) as err:
        if len(err.args) >= 2 and len(err.args[1]) >= 4:
            error = {
                'message': err.args[0],
                'filename': err.args[1][0],
                'lineno': err.args[1][1],
                'offset': err.args[1][2],
                'error_line': err.args[1][3],
            }
            msg = "%s : %s at line %d\n%s" % (type(err).__name__, error['message'], error['lineno'], error['error_line'])
        else:
            msg = frappe.as_unicode(err)
        return msg
    return False

the code is checked at opcode level, it is a kind of sandbox?

1 Like

@JayRam, Any opinion from your side?

New pull request created, [New Feature] Custom Server Action: make customizing on server side on SaaS possible by szufisher · Pull Request #4941 · frappe/frappe · GitHub
request community help to do test and contribute documentation!

1 Like