Add support for user-defined Python preprocessor functions

Allow preprocessor functions to be defined in Python by putting a module
called 'kconfigfunctions' into sys.path. Internally, this simply adds
the functions to the predefined functions in Kconfig._functions.

User-defined Python functions make it simple to integrate information
from existing Python tools into Kconfig, e.g. to have Kconfig symbols
depend on hardware information stored in some other format. This might
be used to get device tree information into Kconfig in Zephyr.

Piggyback module docstring documentation for some extensions that were
previously only mentioned in the README.

1 files changed, 134 insertions, 25 deletions

diff --git a/kconfiglib.py b/kconfiglib.py
index b91381f..69f9e20 100644
--- a/kconfiglib.py
+++ b/kconfiglib.py
@@ -338,15 +338,14 @@ functions just avoid printing 'if y' conditions to give cleaner output.
 Kconfig extensions
 ==================
 
-Kconfiglib implements two Kconfig extensions related to 'source':
+Kconfiglib includes a couple of Kconfig extensions:
 
 'source' with relative path
 ---------------------------
 
-Kconfiglib supports a custom 'rsource' statement that sources Kconfig files
-with a path relative to directory of the Kconfig file containing the 'rsource'
-statement, instead of relative to the project root. This extension is not
-supported by Linux kernel tools as of writing.
+The 'rsource' statement sources Kconfig files with a path relative to directory
+of the Kconfig file containing the 'rsource' statement, instead of relative to
+the project root.
 
 Consider following directory tree:
 
@@ -365,11 +364,11 @@ Consider following directory tree:
 In this example, assume that src/SubSystem1/Kconfig wants to source
 src/SubSystem1/ModuleA/Kconfig.
 
-With 'source', the following statement would be used:
+With 'source', this statement would be used:
 
   source "src/SubSystem1/ModuleA/Kconfig"
 
-Using 'rsource', it can be rewritten as:
+With 'rsource', this turns into
 
   rsource "ModuleA/Kconfig"
 
@@ -379,8 +378,8 @@ If an absolute path is given to 'rsource', it acts the same as 'source'.
 be moved around freely.
 
 
-Globbed sourcing
-----------------
+Globbing 'source'
+-----------------
 
 'source' and 'rsource' accept glob patterns, sourcing all matching Kconfig
 files. They require at least one matching file, throwing a KconfigError
@@ -408,6 +407,102 @@ files matching "bar*" exist:
 'source' and 'osource' are analogous to 'include' and '-include' in Make.
 
 
+Generalized def_* keywords
+--------------------------
+
+def_int, def_hex, and def_string are available in addition to def_bool and
+def_tristate, allowing int, hex, and string symbols to be given a type and a
+default at the same time.
+
+
+Warnings for undefined symbols
+------------------------------
+
+Setting the environment variable KCONFIG_STRICT to "y" will cause warnings to
+be printed for all references to undefined Kconfig symbols within Kconfig
+files. The only gotcha is that all hex literals must be prefixed by "0x" or
+"0X", to make it possible to distuinguish them from symbol references.
+
+Some projects (e.g. the Linux kernel) use multiple Kconfig trees with many
+shared Kconfig files, leading to some safe undefined symbol references.
+KCONFIG_STRICT is useful in projects that only have a single Kconfig tree
+though.
+
+
+Preprocessor user functions defined in Python
+---------------------------------------------
+
+Preprocessor functions can be defined in Python, which makes it simple to
+integrate information from existing Python tools into Kconfig (e.g. to have
+Kconfig symbols depend on hardware information stored in some other format).
+
+Putting a Python module named kconfigfunctions(.py) anywhere in sys.path will
+cause it to be imported by Kconfiglib (in Kconfig.__init__()). Note that
+sys.path can be customized via PYTHONPATH, and includes the directory of the
+module being run by default, as well as installation directories.
+
+If the KCONFIG_FUNCTIONS environment variable is set, it gives a different
+module name to use instead of 'kconfigfunctions'.
+
+The imported module is expected to define a dictionary named 'functions', with
+the following format:
+
+  functions = {
+      "my-fn":       (my_fn,       <min.args>, <max.args>/None),
+      "my-other-fn": (my_other_fn, <min.args>, <max.args>/None),
+      ...
+  }
+
+  def my_fn(kconf, name, arg_1, arg_2, ...):
+      # kconf:
+      #   Kconfig instance
+      #
+      # name:
+      #   Name of the user-defined function ("my-fn"). Think argv[0].
+      #
+      # arg_1, arg_2, ...:
+      #   Arguments passed to the function from Kconfig (strings)
+      #
+      # Returns a string to be substituted as the result of calling the
+      # function
+      ...
+
+  def my_other_fn(kconf, name, arg_1, arg_2, ...):
+      ...
+
+  ...
+
+<min.args> and <max.args> are the minimum and maximum number of arguments
+expected by the function (excluding the implicit 'name' argument). If
+<max.args> is None, there is no upper limit to the number of arguments. Passing
+an invalid number of arguments will generate a KconfigError exception.
+
+Once defined, user functions can be called from Kconfig in the same way as
+other preprocessor functions:
+
+    config FOO
+        ...
+        depends on $(my-fn,arg1,arg2)
+
+If my_fn() returns "n", this will result in
+
+    config FOO
+        ...
+	depends on n
+
+Warning
+*******
+
+User-defined preprocessor functions are called as they're encountered at parse
+time, before all Kconfig files have been processed, and before the menu tree
+has been finalized. There are no guarantees that accessing Kconfig symbols or
+the menu tree via the 'kconf' parameter will work, and it could potentially
+lead to a crash. The 'kconf' parameter is provided for future extension (and
+because the predefined functions take it anyway).
+
+Preferably, user-defined functions should be stateless.
+
+
 Feedback
 ========
 
@@ -416,6 +511,7 @@ service, or open a ticket on the GitHub page.
 """
 import errno
 import glob
+import importlib
 import os
 import platform
 import re
@@ -799,6 +895,15 @@ class Kconfig(object):
             "warning-if": (_warning_if_fn, 2, 2),
         }
 
+        # Add any user-defined preprocessor functions
+        try:
+            self._functions.update(
+                importlib.import_module(
+                    os.environ.get("KCONFIG_FUNCTIONS", "kconfigfunctions")
+                ).functions)
+        except ImportError:
+            pass
+
 
         # This is used to determine whether previously unseen symbols should be
         # registered. They shouldn't be if we parse expressions after parsing,
@@ -2238,13 +2343,17 @@ class Kconfig(object):
             return res
 
         if fn in self._functions:
-            # Built-in function
+            # Built-in or user-defined function
 
             py_fn, min_arg, max_arg = self._functions[fn]
 
-            if not min_arg <= len(args) - 1 <= max_arg:
+            if len(args) - 1 < min_arg or \
+               (max_arg is not None and len(args) - 1 > max_arg):
+
                 if min_arg == max_arg:
                     expected_args = min_arg
+                elif max_arg is None:
+                    expected_args = "{} or more".format(min_arg)
                 else:
                     expected_args = "{}-{}".format(min_arg, max_arg)
 
@@ -2253,7 +2362,7 @@ class Kconfig(object):
                                    .format(self._filename, self._linenr, fn,
                                            expected_args, len(args) - 1))
 
-            return py_fn(self, args)
+            return py_fn(self, *args)
 
         # Environment variables are tried last
         if fn in os.environ:
@@ -5816,33 +5925,33 @@ def _warn_choice_select_imply(sym, expr, expr_type):
 
 # Predefined preprocessor functions
 
-def _filename_fn(kconf, args):
+def _filename_fn(kconf, _):
     return kconf._filename
 
-def _lineno_fn(kconf, args):
+def _lineno_fn(kconf, _):
     return str(kconf._linenr)
 
-def _info_fn(kconf, args):
-    print("{}:{}: {}".format(kconf._filename, kconf._linenr, args[1]))
+def _info_fn(kconf, _, msg):
+    print("{}:{}: {}".format(kconf._filename, kconf._linenr, msg))
 
     return ""
 
-def _warning_if_fn(kconf, args):
-    if args[1] == "y":
-        kconf._warn(args[2], kconf._filename, kconf._linenr)
+def _warning_if_fn(kconf, _, cond, msg):
+    if cond == "y":
+        kconf._warn(msg, kconf._filename, kconf._linenr)
 
     return ""
 
-def _error_if_fn(kconf, args):
-    if args[1] == "y":
+def _error_if_fn(kconf, _, cond, msg):
+    if cond == "y":
         raise KconfigError("{}:{}: {}".format(
-            kconf._filename, kconf._linenr, args[2]))
+            kconf._filename, kconf._linenr, msg))
 
     return ""
 
-def _shell_fn(kconf, args):
+def _shell_fn(kconf, _, command):
     stdout, stderr = subprocess.Popen(
-        args[1], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
+        command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
     ).communicate()
 
     if not _IS_PY2:
@@ -5854,7 +5963,7 @@ def _shell_fn(kconf, args):
 
     if stderr:
         kconf._warn("'{}' wrote to stderr: {}".format(
-                        args[1], "\n".join(stderr.splitlines())),
+                        command, "\n".join(stderr.splitlines())),
                     kconf._filename, kconf._linenr)
 
     # Manual universal newlines with splitlines() (to prevent e.g. stray \r's