Add Kconfig preprocessor

Implement the Kconfig preprocessor described in
https://github.com/torvalds/linux/blob/master/Documentation/kbuild/kconfig-macro-language.txt
(which is now in linux-next and will appear in Linux 4.18).

A new Kconfig.variables property holds all the preprocessor variables so
that they can be inspected programmatically. Preprocessor variables are
represented by a new Variable class.

With the preprocessor, environment variables are referenced with $(FOO)
instead of $FOO. For backwards compatibility, $FOO is accepted as well
for now (and leaves "$FOO" as-is if FOO doesn't exist). The $FOO syntax
might be dropped at some point in the future (together with a major
version increase). It should be supported for a few months at least.

Some internals were cleaned up too, mostly related to parsing. Some
outdated documentation was fixed as well.

5 files changed, 852 insertions, 299 deletions

diff --git a/README.rst b/README.rst
index 3884e7a..e118834 100644
--- a/README.rst
+++ b/README.rst
@@ -18,6 +18,14 @@ should be relatively easy, if needed.
 The `Zephyr <https://www.zephyrproject.org/>`_ project uses Kconfiglib
 exclusively, with lots of small helper scripts in other projects.
 
+Kconfiglib implements the recently added `Kconfig preprocessor
+<https://github.com/torvalds/linux/blob/master/Documentation/kbuild/kconfig-macro-language.txt>`_.
+For backwards compatibility, environment variables can be referenced both as
+``$(FOO)`` (the new syntax) and as ``$FOO`` (the old syntax). Support for the
+old syntax might be removed in the future (the major version would be increased
+at the same time). Using the old syntax with an undefined environment variable
+keeps the string as is.
+
 Installation
 ------------
 
@@ -193,10 +201,14 @@ The following Kconfig extensions are available:
 - Environment variables are expanded directly in e.g. ``source``
   and ``mainmenu`` statements, meaning ``option env`` symbols
   are redundant.
+
+  This is the standard behavior with the new `Kconfig preprocessor
+  <https://github.com/torvalds/linux/blob/master/Documentation/kbuild/kconfig-macro-language.txt>`_,
+  which Kconfiglib implements.
   
-  ``option env`` symbols are still supported for compatibility,
-  with the caveat that they must have the same name as the environment
-  variables they reference. A warning is printed if the names differ.
+  ``option env`` symbols are supported for backwards compatibility, with the
+  caveat that they must have the same name as the environment variables they
+  reference. A warning is printed if the names differ.
 
 Other features
 --------------
diff --git a/kconfiglib.py b/kconfiglib.py
index 333613f..5afe43e 100644
--- a/kconfiglib.py
+++ b/kconfiglib.py
@@ -79,14 +79,16 @@ Using Kconfiglib without the Makefile targets
 =============================================
 
 The make targets are only needed for a trivial reason: The Kbuild makefiles
-export environment variables which are referenced inside the Kconfig files
-(via e.g. 'source "arch/$SRCARCH/Kconfig").
+export environment variables which are referenced inside the Kconfig files and
+in scripts run from the Kconfig files (via e.g. 'source
+"arch/$(SRCARCH)/Kconfig" and '$(shell,...)').
 
-In practice, the only variables referenced (as of writing, and for many years)
-are ARCH, SRCARCH, and KERNELVERSION. To run Kconfiglib without the Makefile
-patch, do this:
+The environment variables referenced as of writing (Linux 4.2.18-rc4) are
+srctree, ARCH, SRCARCH, CC, and KERNELVERSION.
 
-  $ ARCH=x86 SRCARCH=x86 KERNELVERSION=`make kernelversion` python
+To run Kconfiglib without the Makefile patch, you can do this:
+
+  $ srctree=. ARCH=x86 SRCARCH=x86 CC=gcc KERNELVERSION=`make kernelversion` python(3)
   >>> import kconfiglib
   >>> kconf = kconfiglib.Kconfig()  # filename defaults to "Kconfig"
 
@@ -200,8 +202,8 @@ Intro to the menu tree
 The menu structure, as seen in e.g. menuconfig, is represented by a tree of
 MenuNode objects. The top node of the configuration corresponds to an implicit
 top-level menu, the title of which is shown at the top in the standard
-menuconfig interface. (The title with variables expanded is available in
-Kconfig.mainmenu_text in Kconfiglib.)
+menuconfig interface. (The title is also available in Kconfig.mainmenu_text in
+Kconfiglib.)
 
 The top node is found in Kconfig.top_node. From there, you can visit child menu
 nodes by following the 'list' pointer, and any following menu nodes by
@@ -371,6 +373,7 @@ import glob
 import os
 import platform
 import re
+import subprocess
 import sys
 import textwrap
 
@@ -457,9 +460,6 @@ class Kconfig(object):
       not found and $srctree was set when the Kconfig was created,
       $srctree/foo/defconfig is looked up as well.
 
-      References to Kconfig symbols ("$FOO") in the 'default' properties of the
-      defconfig_filename symbol are are expanded before the file is looked up.
-
       'defconfig_filename' is None if either no defconfig_list symbol exists,
       or if the defconfig_list symbol has no 'default' with a satisfied
       condition that specifies a file that exists.
@@ -474,10 +474,12 @@ class Kconfig(object):
       Acts as the root of the menu tree.
 
     mainmenu_text:
-      The prompt (title) of the top_node menu, with Kconfig variable references
-      ("$FOO") expanded. Defaults to "Linux Kernel Configuration" (like in the
-      C tools). Can be changed with the 'mainmenu' statement (see
-      kconfig-language.txt).
+      The prompt (title) of the top menu (top_node). Defaults to "Main menu".
+      Can be changed with the 'mainmenu' statement (see kconfig-language.txt).
+
+    variables:
+      A dictionary with all preprocessor variables, indexed by name. See the
+      Variable class.
 
     warnings:
       A list of strings containing all warnings that have been generated. This
@@ -514,8 +516,9 @@ class Kconfig(object):
     """
     __slots__ = (
         "_encoding",
-        "_set_re_match",
-        "_unset_re_match",
+        "_functions",
+        "_set_match",
+        "_unset_match",
         "_warn_for_no_prompt",
         "_warn_for_redun_assign",
         "_warn_for_undef_assign",
@@ -528,6 +531,7 @@ class Kconfig(object):
         "defconfig_list",
         "defined_syms",
         "m",
+        "mainmenu_text",
         "menus",
         "modules",
         "n",
@@ -535,6 +539,7 @@ class Kconfig(object):
         "srctree",
         "syms",
         "top_node",
+        "variables",
         "warnings",
         "y",
 
@@ -609,20 +614,15 @@ class Kconfig(object):
           Related PEP: https://www.python.org/dev/peps/pep-0538/
         """
         self.srctree = os.environ.get("srctree")
-
         self.config_prefix = os.environ.get("CONFIG_", "CONFIG_")
 
         # Regular expressions for parsing .config files, with the match()
         # method assigned directly as a small optimization (microscopic in this
         # case, but it's consistent with the other regexes)
 
-        self._set_re_match = \
-            re.compile(r"{}([^=]+)=(.*)".format(self.config_prefix),
-                       _RE_ASCII).match
-
-        self._unset_re_match = \
-            re.compile(r"# {}([^ ]+) is not set".format(self.config_prefix),
-                       _RE_ASCII).match
+        self._set_match = _re_match(self.config_prefix + r"([^=]+)=(.*)")
+        self._unset_match = \
+            _re_match(r"# {}([^ ]+) is not set".format(self.config_prefix))
 
 
         self.warnings = []
@@ -665,6 +665,21 @@ class Kconfig(object):
             sym = self.const_syms[nmy]
             sym.rev_dep = sym.weak_rev_dep = sym.direct_dep = self.n
 
+
+        # Maps preprocessor variables names to Variable instances
+        self.variables = {}
+
+        # Predefined preprocessor functions, with min/max number of arguments
+        self._functions = {
+            "info":       (_info_fn,       1, 1),
+            "error-if":   (_error_if_fn,   2, 2),
+            "filename":   (_filename_fn,   0, 0),
+            "lineno":     (_lineno_fn,     0, 0),
+            "shell":      (_shell_fn,      1, 1),
+            "warning-if": (_warning_if_fn, 2, 2),
+        }
+
+
         # This is used to determine whether previously unseen symbols should be
         # registered. They shouldn't be if we parse expressions after parsing,
         # as part of Kconfig.eval_string().
@@ -678,7 +693,7 @@ class Kconfig(object):
         self.top_node.item = MENU
         self.top_node.is_menuconfig = True
         self.top_node.visibility = self.y
-        self.top_node.prompt = ("Linux Kernel Configuration", self.y)
+        self.top_node.prompt = ("Main menu", self.y)
         self.top_node.parent = None
         self.top_node.dep = self.y
         self.top_node.filename = filename
@@ -698,9 +713,11 @@ class Kconfig(object):
         self._filename = filename
         self._linenr = 0
 
+        # Open the top-level Kconfig file
         self._file = self._open(filename)
 
         try:
+            # Parse everything
             self._parse_block(None, self.top_node, self.top_node)
         except UnicodeDecodeError as e:
             _decoding_error(e, self._filename)
@@ -735,14 +752,10 @@ class Kconfig(object):
         # awkward during dependency loop detection
         self._add_choice_deps()
 
+
         self._warn_for_no_prompt = True
 
-    @property
-    def mainmenu_text(self):
-        """
-        See the class documentation.
-        """
-        return _expand(self.top_node.prompt[0])
+        self.mainmenu_text = self.top_node.prompt[0]
 
     @property
     def defconfig_filename(self):
@@ -755,7 +768,7 @@ class Kconfig(object):
         for filename, cond in self.defconfig_list.defaults:
             if expr_value(cond):
                 try:
-                    with self._open(_expand(filename.str_value)) as f:
+                    with self._open(filename.str_value) as f:
                         return f.name
                 except IOError:
                     continue
@@ -811,17 +824,17 @@ class Kconfig(object):
                     choice._was_set = False
 
             # Small optimizations
-            set_re_match = self._set_re_match
-            unset_re_match = self._unset_re_match
+            set_match = self._set_match
+            unset_match = self._unset_match
             syms = self.syms
 
             for linenr, line in enumerate(f, 1):
                 # The C tools ignore trailing whitespace
                 line = line.rstrip()
 
-                set_match = set_re_match(line)
-                if set_match:
-                    name, val = set_match.groups()
+                match = set_match(line)
+                if match:
+                    name, val = match.groups()
                     if name not in syms:
                         self._warn_undef_assign_load(name, val, filename,
                                                      linenr)
@@ -866,22 +879,22 @@ class Kconfig(object):
                             sym.choice.set_value(val)
 
                     elif sym.orig_type == STRING:
-                        string_match = _conf_string_re_match(val)
-                        if not string_match:
+                        match = _conf_string_match(val)
+                        if not match:
                             self._warn("malformed string literal in "
                                        "assignment to {}. Assignment ignored."
                                        .format(_name_and_loc(sym)),
                                        filename, linenr)
                             continue
 
-                        val = unescape(string_match.group(1))
+                        val = unescape(match.group(1))
 
                 else:
-                    unset_match = unset_re_match(line)
-                    if not unset_match:
+                    match = unset_match(line)
+                    if not match:
                         # Print a warning for lines that match neither
-                        # set_re_match() nor unset_re_match() and that are not
-                        # blank lines or comments. 'line' has already been
+                        # set_match() nor unset_match() and that are not blank
+                        # lines or comments. 'line' has already been
                         # rstrip()'d, so blank lines show up as "" here.
                         if line and not line.lstrip().startswith("#"):
                             self._warn("ignoring malformed line '{}'"
@@ -890,7 +903,7 @@ class Kconfig(object):
 
                         continue
 
-                    name = unset_match.group(1)
+                    name = match.group(1)
                     if name not in syms:
                         self._warn_undef_assign_load(name, "n", filename,
                                                      linenr)
@@ -1273,21 +1286,21 @@ class Kconfig(object):
 
         with self._open_enc("auto.conf", _UNIVERSAL_NEWLINES_MODE) as f:
             for line in f:
-                set_match = self._set_re_match(line)
-                if not set_match:
+                match = self._set_match(line)
+                if not match:
                     # We only expect CONFIG_FOO=... (and possibly a header
                     # comment) in auto.conf
                     continue
 
-                name, val = set_match.groups()
+                name, val = match.groups()
                 if name in self.syms:
                     sym = self.syms[name]
 
                     if sym.orig_type == STRING:
-                        string_match = _conf_string_re_match(val)
-                        if not string_match:
+                        match = _conf_string_match(val)
+                        if not match:
                             continue
-                        val = unescape(string_match.group(1))
+                        val = unescape(match.group(1))
 
                     self.syms[name]._old_val = val
 
@@ -1480,14 +1493,11 @@ class Kconfig(object):
             # Extend the error message a bit in this case
             raise IOError(textwrap.fill(
                 "{}:{}: {} Also note that Kconfiglib expands references to "
-                "environment variables directly (via os.path.expandvars()), "
+                "environment variables directly, "
                 "meaning you do not need \"bounce\" symbols with "
                 "'option env=\"FOO\"'. For compatibility with the C tools, "
                 "name the bounce symbols the same as the environment variable "
-                "they reference (like the Linux kernel does). This change is "
-                "likely to soon appear in the C tools as well, and simplifies "
-                "the parsing implementation (symbols no longer need to be "
-                "evaluated during parsing)."
+                "they reference (like the Linux kernel does)."
                 .format(self._filename, self._linenr, e),
                 80))
 
@@ -1580,36 +1590,41 @@ class Kconfig(object):
         # hotspot during parsing.
 
         # Initial token on the line
-        command_match = _command_re_match(s)
-        if not command_match:
-            return (None,)
+        match = _command_match(s)
+        if not match:
+            if s.isspace() or s.lstrip().startswith("#"):
+                return (None,)
+            self._parse_error("unknown token at start of line")
 
         # Tricky implementation detail: While parsing a token, 'token' refers
         # to the previous token. See _STRING_LEX for why this is needed.
-        token = _get_keyword(command_match.group(1))
+        token = _get_keyword(match.group(1))
         if not token:
-            self._parse_error("expected keyword as first token")
+            # If the first token is not a keyword, we have a preprocessor
+            # variable assignment (or a bare macro on a line)
+            self._parse_assignment(s)
+            return (None,)
 
         tokens = [token]
         # The current index in the string being tokenized
-        i = command_match.end()
+        i = match.end()
 
         # Main tokenization loop (for tokens past the first one)
         while i < len(s):
             # Test for an identifier/keyword first. This is the most common
             # case.
-            id_keyword_match = _id_keyword_re_match(s, i)
-            if id_keyword_match:
+            match = _id_keyword_match(s, i)
+            if match:
                 # We have an identifier or keyword
 
                 # Jump past it
-                i = id_keyword_match.end()
+                i = match.end()
 
                 # Check what it is. lookup_sym() will take care of allocating
                 # new symbols for us the first time we see them. Note that
                 # 'token' still refers to the previous token.
 
-                name = id_keyword_match.group(1)
+                name = match.group(1)
                 keyword = _get_keyword(name)
                 if keyword:
                     # It's a keyword
@@ -1636,59 +1651,28 @@ class Kconfig(object):
                     token = name
 
             else:
-                # Not keyword/non-const symbol
-
-                # Note: _id_keyword_match and _initial_token_match strip
-                # trailing whitespace, making it safe to assume s[i] is the
-                # start of a token here. We manually strip trailing whitespace
-                # below as well.
-                #
-                # An old version stripped whitespace in this spot instead, but
-                # that leads to some redundancy and would cause
-                # _id_keyword_match to be tried against just "\n" fairly often
-                # (because file.readlines() keeps newlines).
+                # Neither a keyword nor a non-const symbol (except
+                # $()-expansion might still yield a non-const symbol).
 
+                # We always strip whitespace after tokens, so it is safe to
+                # assume that s[i] is the start of a token here.
                 c = s[i]
-                i += 1
 
                 if c in "\"'":
-                    # String literal/constant symbol
-                    if "\\" not in s:
-                        # Fast path: If the line contains no backslashes, we
-                        # can just find the matching quote.
+                    s, end_i = self._expand_str(s, i, c)
 
-                        end = s.find(c, i)
-                        if end == -1:
-                            self._parse_error("unterminated string")
-
-                        val = s[i:end]
-                        i = end + 1
-                    else:
-                        # Slow path for lines with backslashes (very rare,
-                        # performance irrelevant)
-
-                        quote = c
-                        val = ""
-
-                        while 1:
-                            if i >= len(s):
-                                self._parse_error("unterminated string")
-
-                            c = s[i]
-                            if c == quote:
-                                break
-
-                            if c == "\\":
-                                if i + 1 >= len(s):
-                                    self._parse_error("unterminated string")
-
-                                val += s[i + 1]
-                                i += 2
-                            else:
-                                val += c
-                                i += 1
+                    # os.path.expandvars() and the $UNAME_RELEASE replace() is
+                    # a backwards compatibility hack, which should be
+                    # reasonably safe as expandvars() leaves references to
+                    # undefined env. vars. as is.
+                    #
+                    # The preprocessor functionality changed how environment
+                    # variables are referenced, to $(FOO).
+                    val = os.path.expandvars(
+                        s[i + 1:end_i - 1].replace("$UNAME_RELEASE",
+                                                   platform.uname()[2]))
 
-                        i += 1
+                    i = end_i
 
                     # This is the only place where we don't survive with a
                     # single token of lookback: 'option env="FOO"' does not
@@ -1698,62 +1682,84 @@ class Kconfig(object):
                                 tokens[0] == _T_OPTION else \
                             self._lookup_const_sym(val)
 
-                elif c == "&":
-                    if i >= len(s) or s[i] != "&":
-                        self._parse_error("malformed operator")
-
+                elif s.startswith("&&", i):
                     token = _T_AND
-                    i += 1
-
-                elif c == "|":
-                    if i >= len(s) or s[i] != "|":
-                        self._parse_error("malformed operator")
+                    i += 2
 
+                elif s.startswith("||", i):
                     token = _T_OR
-                    i += 1
-
-                elif c == "!":
-                    if i < len(s) and s[i] == "=":
-                        token = _T_UNEQUAL
-                        i += 1
-                    else:
-                        token = _T_NOT
+                    i += 2
 
                 elif c == "=":
                     token = _T_EQUAL
+                    i += 1
+
+                elif s.startswith("!=", i):
+                    token = _T_UNEQUAL
+                    i += 2
+
+                elif c == "!":
+                    token = _T_NOT
+                    i += 1
 
                 elif c == "(":
                     token = _T_OPEN_PAREN
+                    i += 1
 
                 elif c == ")":
                     token = _T_CLOSE_PAREN
+                    i += 1
+
+                elif c == "$":
+                    s, end_i = self._expand_macro(s, i, ())
+                    val = s[i:end_i]
+                    # isspace() is False for empty strings
+                    if not val.strip():
+                        # Avoid creating a Kconfig symbol with a blank name.
+                        # It's almost guaranteed to be an error.
+                        self._parse_error("macro expanded to blank string")
+                    i = end_i
+
+                    # Compatibility with what the C implementation does. Might
+                    # be unexpected that you can reference non-constant symbols
+                    # this way though...
+                    token = self.const_syms[val] \
+                            if val in ("n", "m", "y") else \
+                            self._lookup_sym(val)
 
                 elif c == "#":
                     break
 
+
                 # Very rare
+
+                elif s.startswith("<=", i):
+                    token = _T_LESS_EQUAL
+                    i += 2
+
                 elif c == "<":
-                    if i < len(s) and s[i] == "=":
-                        token = _T_LESS_EQUAL
-                        i += 1
-                    else:
-                        token = _T_LESS
+                    token = _T_LESS
+                    i += 1
+
+                elif s.startswith(">=", i):
+                    token = _T_GREATER_EQUAL
+                    i += 2
 
-                # Very rare
                 elif c == ">":
-                    if i < len(s) and s[i] == "=":
-                        token = _T_GREATER_EQUAL
-                        i += 1
-                    else:
-                        token = _T_GREATER
+                    token = _T_GREATER
+                    i += 1
+
 
                 else:
-                    self._parse_error("invalid character in line")
+                    self._parse_error("unknown tokens in line")
+
 
                 # Skip trailing whitespace
                 while i < len(s) and s[i].isspace():
                     i += 1
 
+
+            # Add the token
             tokens.append(token)
 
         # None-terminating the token list makes the token fetching functions
@@ -1826,7 +1832,6 @@ class Kconfig(object):
 
         return token
 
-
     def _check_token(self, token):
         # If the next token is 'token', removes it and returns True
 
@@ -1837,6 +1842,223 @@ class Kconfig(object):
 
 
     #
+    # Preprocessor logic
+    #
+
+    def _parse_assignment(self, s):
+        # Parses a preprocessor variable assignment, registering the variable
+        # if it doesn't already exist. Also takes care of bare macros on lines
+        # (which are allowed, and can be useful for their side effects).
+
+        # Expand any macros in the left-hand side of the assignment (the
+        # variable name)
+        s = s.lstrip()
+        i = 0
+        while 1:
+            i = _assignment_lhs_fragment_match(s, i).end()
+            if s.startswith("$(", i):
+                s, i = self._expand_macro(s, i, ())
+            else:
+                break
+
+        if s.isspace():
+            # We also accept a bare macro on a line (e.g.
+            # $(warning-if,$(foo),ops)), provided it expands to a blank string
+            return
+
+        # Assigned variable
+        name = s[:i]
+
+
+        # Extract assignment operator (=, :=, or +=) and value
+        rhs_match = _assignment_rhs_match(s, i)
+        if not rhs_match:
+            self._parse_error("syntax error")
+
+        op, val = rhs_match.groups()
+
+
+        if name in self.variables:
+            # Already seen variable
+            var = self.variables[name]
+        else:
+            # New variable
+            var = Variable()
+            var.kconfig = self
+            var.name = name
+            var._n_expansions = 0
+            self.variables[name] = var
+
+            # += acts like = on undefined variables (defines a recursive
+            # variable)
+            if op == "+=":
+                op = "="
+
+        if op == "=":
+            var.is_recursive = True
+            var.value = val
+        elif op == ":=":
+            var.is_recursive = False
+            var.value = self._expand_whole(val, ())
+        else:  # op == "+="
+            # += does immediate expansion if the variable was last set
+            # with :=
+            var.value += " " + (val if var.is_recursive else \
+                                self._expand_whole(val, ()))
+
+    def _expand_whole(self, s, args):
+        # Expands preprocessor macros in all of 's'. Used whenever we don't
+        # have to worry about delimiters. See _expand_macro() re. the 'args'
+        # parameter.
+        #
+        # Returns the expanded string.
+
+        i = 0
+        while 1:
+            i = s.find("$(", i)
+            if i == -1:
+                break
+            s, i = self._expand_macro(s, i, args)
+        return s
+
+    def _expand_str(self, s, i, quote):
+        # Expands a quoted string starting at index 'i' in 's'. Handles both
+        # backslash escapes and macro expansion.
+        #
+        # Returns the expanded 's' (including the part before the string) and
+        # the index of the first character after the expanded string in 's'.
+
+        i += 1  # Skip over initial "/'
+        while 1:
+            match = _string_special_search(s, i)
+            if not match:
+                self._parse_error("unterminated string")
+
+
+            if match.group() == quote:
+                # Found the end of the string
+                return (s, match.end())
+
+            elif match.group() == "\\":
+                # Replace '\x' with 'x'. 'i' ends up pointing to the character
+                # after 'x', which allows macros to be canceled with '\$(foo)'.
+                i = match.end()
+                s = s[:match.start()] + s[i:]
+
+            elif match.group() == "$(":
+                # A macro call within the string
+                s, i = self._expand_macro(s, match.start(), ())
+
+            else:
+                # A ' quote within " quotes or vice versa
+                i += 1
+
+    def _expand_macro(self, s, i, args):
+        # Expands a macro starting at index 'i' in 's'. If this macro resulted
+        # from the expansion of another macro, 'args' holds the arguments
+        # passed to that macro.
+        #
+        # Returns the expanded 's' (including the part before the macro) and
+        # the index of the first character after the expanded macro in 's'.
+
+        start = i
+        i += 2  # Skip over "$("
+
+        # Start of current macro argument
+        arg_start = i
+
+        # Arguments of this macro call
+        new_args = []
+
+        while 1:
+            match = _macro_special_search(s, i)
+            if not match:
+                self._parse_error("missing end parenthesis in macro expansion")
+
+
+            if match.group() == ")":
+                # Found the end of the macro
+
+                new_args.append(s[arg_start:match.start()])
+
+                prefix = s[:start]
+
+                # $(1) is replaced by the first argument to the function, etc.,
+                # provided at least that many arguments were passed
+
+                try:
+                    # Does the macro look like an integer, with a corresponding
+                    # argument? If so, expand it to the value of the argument.
+                    prefix += args[int(new_args[0])]
+                except (ValueError, IndexError):
+                    # Regular variables are just functions without arguments,
+                    # and also go through the function value path
+                    prefix += self._fn_val(new_args)
+
+                return (prefix + s[match.end():],
+                        len(prefix))
+
+            elif match.group() == ",":
+                # Found the end of a macro argument
+                new_args.append(s[arg_start:match.start()])
+                arg_start = i = match.end()
+
+            else:  # match.group() == "$("
+                # A nested macro call within the macro
+                s, i = self._expand_macro(s, match.start(), args)
+
+    def _fn_val(self, args):
+        # Returns the result of calling the function args[0] with the arguments
+        # args[1..len(args)-1]. Plain variables are treated as functions
+        # without arguments.
+
+        fn = args[0]
+
+        if fn in self.variables:
+            var = self.variables[fn]
+
+            if len(args) == 1:
+                # Plain variable
+                if var._n_expansions:
+                    self._parse_error("Preprocessor variable {} recursively "
+                                      "references itself".format(var.name))
+            elif var._n_expansions > 100:
+                # Allow functions to call themselves, but guess that functions
+                # that are overly recursive are stuck
+                self._parse_error("Preprocessor function {} seems stuck "
+                                  "in infinite recursion".format(var.name))
+
+            var._n_expansions += 1
+            res = self._expand_whole(self.variables[fn].value, args)
+            var._n_expansions -= 1
+            return res
+
+        if fn in self._functions:
+            # Built-in function
+
+            py_fn, min_arg, max_arg = self._functions[fn]
+
+            if not min_arg <= len(args) - 1 <= max_arg:
+                if min_arg == max_arg:
+                    expected_args = min_arg
+                else:
+                    expected_args = "{}-{}".format(min_arg, max_arg)
+
+                raise KconfigError("{}:{}: bad number of arguments in call "
+                                   "to {}, expected {}, got {}"
+                                   .format(self._filename, self._linenr, fn,
+                                           expected_args, len(args) - 1))
+
+            return py_fn(self, args)
+
+        # Environment variables are tried last
+        if fn in os.environ:
+            return os.environ[fn]
+
+        return ""
+
+
+    #
     # Parsing
     #
 
@@ -1927,20 +2149,20 @@ class Kconfig(object):
                 prev.next = prev = node
 
             elif t0 == _T_SOURCE:
-                self._enter_file(_expand(self._expect_str_and_eol()))
+                self._enter_file(self._expect_str_and_eol())
                 prev = self._parse_block(None, parent, prev)
                 self._leave_file()
 
             elif t0 == _T_RSOURCE:
                 self._enter_file(os.path.join(
                     os.path.dirname(self._filename),
-                    _expand(self._expect_str_and_eol())
+                    self._expect_str_and_eol()
                 ))
                 prev = self._parse_block(None, parent, prev)
                 self._leave_file()
 
             elif t0 in (_T_GSOURCE, _T_GRSOURCE):
-                pattern = _expand(self._expect_str_and_eol())
+                pattern = self._expect_str_and_eol()
                 if t0 == _T_GRSOURCE:
                     # Relative gsource
                     pattern = os.path.join(os.path.dirname(self._filename),
@@ -2691,7 +2913,7 @@ class Kconfig(object):
             loc = "{}:{}: ".format(self._filename, self._linenr)
 
         raise KconfigError(
-            "{}Couldn't parse '{}': {}".format(loc, self._line.rstrip(), msg))
+            "{}couldn't parse '{}': {}".format(loc, self._line.rstrip(), msg))
 
     def _open_enc(self, filename, mode):
         # open() wrapper for forcing the encoding on Python 3. Forcing the
@@ -4423,6 +4645,41 @@ class MenuNode(object):
 
         return "\n".join(lines) + "\n"
 
+class Variable(object):
+    """
+    Represents a preprocessor variable/function.
+
+    The following attributes are available:
+
+    name:
+      The name of the variable.
+
+    value:
+      The unexpanded value of the variable.
+
+    expanded_value:
+      The expanded value of the variable. For simple variables (those defined
+      with :=), this will equal 'value'. Accessing this property will raise a
+      KconfigError if any variable in the expansion expands to itself.
+
+    is_recursive:
+      True if the variable is recursive (defined with =).
+    """
+    __slots__ = (
+        "_n_expansions",
+        "is_recursive",
+        "kconfig",
+        "name",
+        "value",
+    )
+
+    @property
+    def expanded_value(self):
+        """
+        See the class documentation.
+        """
+        return self.kconfig._expand_whole(self.value, ())
+
 class KconfigError(Exception):
     """
     Exception raised for Kconfig-related errors.
@@ -4616,14 +4873,14 @@ def escape(s):
     return s.replace("\\", r"\\").replace('"', r'\"')
 
 # unescape() helper
-_unescape_re_sub = re.compile(r"\\(.)").sub
+_unescape_sub = re.compile(r"\\(.)").sub
 
 def unescape(s):
     r"""
     Unescapes the string 's'. \ followed by any character is replaced with just
     that character. Used internally when reading .config files.
     """
-    return _unescape_re_sub(r"\1", s)
+    return _unescape_sub(r"\1", s)
 
 def standard_kconfig():
     """
@@ -4697,15 +4954,6 @@ def _make_depend_on(sc, expr):
         # Non-constant symbol, or choice
         expr._dependents.add(sc)
 
-def _expand(s):
-    # A predefined UNAME_RELEASE symbol is expanded in one of the 'default's of
-    # the DEFCONFIG_LIST symbol in the Linux kernel. This function maintains
-    # compatibility with it even though environment variables in strings are
-    # now expanded directly.
-
-    # platform.uname() has an internal cache, so this is speedy enough
-    return os.path.expandvars(s.replace("$UNAME_RELEASE", platform.uname()[2]))
-
 def _parenthesize(expr, type_):
     # expr_str() helper. Adds parentheses around expressions of type 'type_'.
 
@@ -4749,9 +4997,18 @@ def _internal_error(msg):
         "email service to tell me about this. Include the message above and "
         "the stack trace and describe what you were doing.")
 
-def _decoding_error(e, filename):
+def _decoding_error(e, filename, macro_linenr=None):
     # Gives the filename and context for UnicodeDecodeError's, which are a pain
     # to debug otherwise. 'e' is the UnicodeDecodeError object.
+    #
+    # If the decoding error is for the output of a $(shell,...) command,
+    # macro_linenr holds the line number where it was run (the exact line
+    # number isn't available for decoding errors in files).
+
+    if macro_linenr is None:
+        loc = filename
+    else:
+        loc = "output from macro at {}:{}".format(filename, macro_linenr)
 
     raise KconfigError(
         "\n"
@@ -4759,7 +5016,7 @@ def _decoding_error(e, filename):
         "Context: {}\n"
         "Problematic data: {}\n"
         "Reason: {}".format(
-            e.encoding, filename,
+            e.encoding, loc,
             e.object[max(e.start - 40, 0):e.end + 40],
             e.object[e.start:e.end],
             e.reason))
@@ -5189,6 +5446,51 @@ def _warn_choice_select_imply(sym, expr, expr_type):
 
     sym.kconfig._warn(msg)
 
+# Predefined preprocessor functions
+
+def _filename_fn(kconf, args):
+    return kconf._filename
+
+def _lineno_fn(kconf, args):
+    return str(kconf._linenr)
+
+def _info_fn(kconf, args):
+    print("{}:{}: {}".format(kconf._filename, kconf._linenr, args[1]))
+
+    return ""
+
+def _warning_if_fn(kconf, args):
+    if args[1] == "y":
+        kconf._warn(args[2], kconf._filename, kconf._linenr)
+
+    return ""
+
+def _error_if_fn(kconf, args):
+    if args[1] == "y":
+        raise KconfigError("{}:{}: {}".format(
+            kconf._filename, kconf._linenr, args[2]))
+
+    return ""
+
+def _shell_fn(kconf, args):
+    stdout, stderr = subprocess.Popen(
+        args[1], shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE
+    ).communicate()
+
+    if not _IS_PY2:
+        try:
+            stdout = stdout.decode(kconf._encoding)
+            stderr = stderr.decode(kconf._encoding)
+        except UnicodeDecodeError as e:
+            _decoding_error(e, kconf._filename, kconf._linenr)
+
+    if stderr:
+        kconf._warn(
+            "'{}' wrote to stderr: {}".format(args[1], stderr.rstrip("\n")),
+            kconf._filename, kconf._linenr)
+
+    return stdout.rstrip("\n").replace("\n", " ")
+
 #
 # Public global constants
 #
@@ -5382,22 +5684,53 @@ _TYPE_TOKENS = frozenset((
     _T_STRING,
 ))
 
+
+# Helper functions for getting compiled regular expressions, with the needed
+# matching function returned directly as a small optimization.
+#
 # Use ASCII regex matching on Python 3. It's already the default on Python 2.
-_RE_ASCII = 0 if _IS_PY2 else re.ASCII
+
+def _re_match(regex):
+    return re.compile(regex, 0 if _IS_PY2 else re.ASCII).match
+
+def _re_search(regex):
+    return re.compile(regex, 0 if _IS_PY2 else re.ASCII).search
+
+
+# Various regular expressions used during parsing
 
 # The initial token on a line. Also eats leading and trailing whitespace, so
 # that we can jump straight to the next token (or to the end of the line if
 # there is only one token).
 #
 # This regex will also fail to match for empty lines and comment lines.
-_command_re_match = re.compile(r"\s*([A-Za-z0-9_-]+)\s*", _RE_ASCII).match
+#
+# '$' is included to detect a variable assignment left-hand side with a $ in it
+# (which might be from a macro expansion).
+_command_match = _re_match(r"\s*([$A-Za-z0-9_-]+)\s*")
+
+# An identifier/keyword after the first token. Also eats trailing whitespace.
+_id_keyword_match = _re_match(r"([A-Za-z0-9_/.-]+)\s*")
+
+# A fragment in the left-hand side of a preprocessor variable assignment. These
+# are the portions between macro expansions ($(foo)). Macros are supported in
+# the LHS (variable name).
+_assignment_lhs_fragment_match = _re_match("[A-Za-z0-9_-]*")
+
+# The assignment operator and value (right-hand side) in a preprocessor
+# variable assignment
+_assignment_rhs_match = _re_match(r"\s*(=|:=|\+=)\s*(.*)")
+
+# Special characters/strings while expanding a macro (')', ',', and '$(')
+_macro_special_search = _re_search(r"\)|,|\$\(")
+
+# Special characters/strings while expanding a string (quotes, '\', and '$(')
+_string_special_search = _re_search(r'"|\'|\\|\$\(')
 
-# Matches an identifier/keyword, also eating trailing whitespace
-_id_keyword_re_match = re.compile(r"([A-Za-z0-9_/.-]+)\s*", _RE_ASCII).match
+# A valid right-hand side for an assignment to a string symbol in a .config
+# file, including escaped characters. Extracts the contents.
+_conf_string_match = _re_match(r'"((?:[^\\"]|\\.)*)"')
 
-# Matches a valid right-hand side for an assignment to a string symbol in a
-# .config file, including escaped characters. Extracts the contents.
-_conf_string_re_match = re.compile(r'"((?:[^\\"]|\\.)*)"', _RE_ASCII).match
 
 # Token to type mapping
 _TOKEN_TO_TYPE = {
diff --git a/tests/Kpreprocess b/tests/Kpreprocess
new file mode 100644
index 0000000..73053fe
--- /dev/null
+++ b/tests/Kpreprocess
@@ -0,0 +1,130 @@
+# Simple assignments (with bad formatting, as an additional test)
+
+simple-recursive=foo
+simple-immediate:=bar
+# Should become recursive
+simple-recursive-2+=baz
+
+	 whitespaced	 =	 foo
+
+
+# Simple += test. += should preserve the flavor of the variable (simple vs.
+# recursive).
+
+preserve-recursive = foo
+preserve-recursive += bar
+
+preserve-immediate := foo
+preserve-immediate += bar
+
+
+# Recursive substitution
+
+recursive = $(foo) $(bar) $($(b-char)a$(z-char))
+recursive += $(indir)
+
+foo = abc
+bar = def
+baz = ghi
+
+b-char = b
+z-char = z
+
+indir = jkl $(indir-2)
+indir-2 = mno
+
+
+# Immediate substitution
+
+def = foo
+immediate := $(undef)$(def)$(undef)$(def)
+def = bar
+undef = bar
+
+
+# Function calls
+
+# Chained function call
+quote = "$(1)" "$(2)"
+rev-quote = $(quote,$(2),$(1))
+surround-rev-quote = $(0) $(rev-quote,$(1),$(2)) $(0)
+surround-rev-quote-unused-arg = $(surround-rev-quote,$(1),$(2)) $(3)
+# No value is passed for $(3), so it expands to nothing
+fn-indir = surround-rev-quote
+messy-fn-res = $($(fn-indir)-unused-arg, a  b , c  d )
+
+# Special characters in function call
+comma = ,
+right-paren = )
+dollar = $
+left-paren = (
+fn = "$(1)"
+special-chars-fn-res = $(fn,$(comma)$(dollar)$(left-paren)foo$(right-paren))
+
+
+# Variable expansions in various locations (verified by checking how the symbol
+# prints)
+
+qaz = QAZ
+echo = $(1)
+
+config PRINT_ME
+	string "$(ENV_VAR)" if ($(echo,FOO) && $(echo,BAR)) || !$(echo,BAZ) || !(($(qaz)))
+	default "$(echo,"foo")" if "foo $(echo,"bar") baz" = "$(undefined)"
+
+
+# Recursive expansion (throws an exception)
+
+rec-1 = x $(rec-2) y
+rec-2 = x $(rec-3) y
+rec-3 = x $(rec-1) y
+
+# Functions are allowed to reference themselves, but an exception is thrown if
+# the function seems to be stuck (the recursion gets too deep)
+safe-fn-rec = $($(1))
+safe-fn-rec-2 = $(safe-fn-rec,safe-fn-rec-3)
+safe-fn-rec-3 = foo
+safe-fn-rec-res = $(safe-fn-rec,safe-fn-rec-2)
+
+unsafe-fn-rec = $(unsafe-fn-rec,$(1))
+
+
+# Expansion in the left-hand side of assignments
+
+dummy-arg-fn = bar
+lhs-indir-1 = lhs-indir-2
+lhs-indir-2 = -baz
+rhs = value
+# LHS expands to foo-bar-baz
+foo-$(dummy-arg-fn, ignored argument )$($(lhs-indir-1)) = $(rhs)
+# Expands to empty string, accepted
+  $(undefined)  
+
+# Variable with a space in its name
+empty =
+space = $(empty) $(empty)
+foo$(space)bar = value
+space-var-res = $(foo bar)
+
+
+# Built-in functions
+
+# Expands to "baz qaz"
+shell-res = $(shell,false && echo foo bar || echo baz qaz)
+
+# Warns about output on stderr, expands to nothing
+shell-stderr-res := $(shell,echo message on stderr >&2)
+
+# Expands to the current location
+location-res := $(filename):$(lineno)
+
+# Adds one warning, expands to nothing
+$(warning-if,,no warning)
+$(warning-if,n,no warning)
+warning-res := $(warning-if,y,a warning)
+
+# Does not cause an error, expands to nothing
+error-n-res := $(error-if,n,oops)
+
+# Causes an error when expanded
+error-y-res = $(error-if,y,oops)
diff --git a/tests/reltest b/tests/reltest
index cd31711..0916c7a 100755
--- a/tests/reltest
+++ b/tests/reltest
@@ -23,7 +23,7 @@ test_script() {
 for py in python2 python3; do
     echo -e "\n================= Test suite with $py =================\n"
 
-    if ! $py Kconfiglib/testsuite.py speedy; then
+    if ! $py Kconfiglib/testsuite.py; then
         echo "test suite failed for $py"
         exit 1
     fi
diff --git a/testsuite.py b/testsuite.py
index 536a98f..c18e69f 100644
--- a/testsuite.py
+++ b/testsuite.py
@@ -12,12 +12,6 @@
 # Some additional options can be turned on by passing them as arguments. They
 # default to off.
 #
-#  - speedy:
-#    Run scripts/kconfig/conf directly instead of using 'make' targets. Makes
-#    things a lot faster, but could break if Kconfig files start referencing
-#    additional environment variables beyond ARCH, SRCARCH, and KERNELVERSION.
-#    Safe as of Linux 4.14-rc3.
-#
 #  - obsessive:
 #    By default, only valid arch/defconfig pairs are tested. In obsessive mode,
 #    every arch will be tested with every defconfig. Increases the testing time
@@ -31,10 +25,10 @@
 #    Log timestamped defconfig test failures to the file test_defconfig_fails.
 #    Handy in obsessive mode.
 #
-# For example, this commands runs the test suite in speedy mode with logging
+# For example, this commands runs the test suite in obsessive mode with logging
 # enabled:
 #
-#   $ python(3) Kconfiglib/testsuite.py speedy log
+#   $ python(3) Kconfiglib/testsuite.py obsessive log
 #
 # pypy works too, and runs most tests much faster than CPython.
 #
@@ -79,30 +73,18 @@ def verify_equal(x, y):
     if x != y:
         fail("'{}' does not equal '{}'".format(x, y))
 
-# Referenced inside the kernel Kconfig files.
-#
-# The str() makes the type of the value 'str' on both Python 2 and Python 3,
-# which is nice for some later dictionary key sanity checks.
-os.environ["KERNELVERSION"] = str(
-    subprocess.check_output(("make", "kernelversion")).decode("utf-8").rstrip()
-)
-
 # Prevent accidental loading of configuration files by removing
 # KCONFIG_ALLCONFIG from the environment
 os.environ.pop("KCONFIG_ALLCONFIG", None)
 
-speedy = False
 obsessive = False
 obsessive_min_config = False
 log = False
 
 def run_tests():
-    global speedy, obsessive, log
+    global obsessive, log
     for s in sys.argv[1:]:
-        if s == "speedy":
-            speedy = True
-            print("Speedy mode enabled")
-        elif s == "obsessive":
+        if s == "obsessive":
             obsessive = True
             print("Obsessive mode enabled")
         elif s == "obsessive-min-config":
@@ -825,7 +807,7 @@ comment "advanced comment"
     print("Testing Kconfig.__repr__()")
 
     verify_repr(c, """
-<configuration with 14 symbols, main menu prompt "Linux Kernel Configuration", srctree not set, config symbol prefix "CONFIG_", warnings disabled, printing of warnings to stderr enabled, undef. symbol assignment warnings disabled, redundant symbol assignment warnings enabled>
+<configuration with 14 symbols, main menu prompt "Main menu", srctree not set, config symbol prefix "CONFIG_", warnings disabled, printing of warnings to stderr enabled, undef. symbol assignment warnings disabled, redundant symbol assignment warnings enabled>
 """)
 
     os.environ["srctree"] = "srctree value"
@@ -838,7 +820,7 @@ comment "advanced comment"
     c.enable_undef_warnings()
 
     verify_repr(c, """
-<configuration with 14 symbols, main menu prompt "Linux Kernel Configuration", srctree "srctree value", config symbol prefix "CONFIG_ value", warnings enabled, printing of warnings to stderr disabled, undef. symbol assignment warnings enabled, redundant symbol assignment warnings disabled>
+<configuration with 14 symbols, main menu prompt "Main menu", srctree "srctree value", config symbol prefix "CONFIG_ value", warnings enabled, printing of warnings to stderr disabled, undef. symbol assignment warnings enabled, redundant symbol assignment warnings disabled>
 """)
 
     os.environ.pop("srctree", None)
@@ -1474,7 +1456,7 @@ g
     print("Testing mainmenu_text")
 
     c = Kconfig("Kconfiglib/tests/empty")
-    verify(c.mainmenu_text == "Linux Kernel Configuration",
+    verify(c.mainmenu_text == "Main menu",
            "An empty Kconfig should get a default main menu prompt")
 
     # Expanded in the mainmenu text
@@ -2012,6 +1994,111 @@ g
             fail("dependency loop in {} not detected".format(filename))
 
 
+    print("Testing preprocessor")
+
+    os.environ["ENV_VAR"] = "env"
+    # We verify warnings manually
+    c = Kconfig("Kconfiglib/tests/Kpreprocess", warn_to_stderr=False)
+
+    def verify_variable(name, unexp_value, exp_value, recursive):
+        var = c.variables[name]
+
+        verify(var.value == unexp_value,
+               "expected variable '{}' to have the unexpanded value '{}', had "
+               "the value '{}'".format(name, unexp_value, var.value))
+
+        verify(var.expanded_value == exp_value,
+               "expected variable '{}' to have the expanded value '{}', had "
+               "the value '{}'".format(name, exp_value, var.expanded_value))
+
+        verify(var.is_recursive == recursive,
+               "{} was {}, shouldn't be"
+               .format(name, "recursive" if var.is_recursive else "simple"))
+
+    verify_variable("simple-recursive", "foo", "foo", True)
+    verify_variable("simple-immediate", "bar", "bar", False)
+    verify_variable("simple-recursive-2", "baz", "baz", True)
+
+    verify_variable("whitespaced", "foo", "foo", True)
+
+    verify_variable("preserve-recursive", "foo bar", "foo bar", True)
+    verify_variable("preserve-immediate", "foo bar", "foo bar", False)
+
+    verify_variable("recursive",
+                    "$(foo) $(bar) $($(b-char)a$(z-char)) $(indir)",
+                    "abc def ghi jkl mno",
+                    True)
+
+    verify_variable("immediate", "foofoo", "foofoo", False)
+
+    verify_variable("messy-fn-res",
+                    "$($(fn-indir)-unused-arg, a  b , c  d )",
+                    'surround-rev-quote " c  d " " a  b " surround-rev-quote ',
+                    True)
+
+    verify_variable("special-chars-fn-res",
+                    "$(fn,$(comma)$(dollar)$(left-paren)foo$(right-paren))",
+                    '",$(foo)"',
+                    True)
+
+    verify_str(c.syms["PRINT_ME"], r"""
+config PRINT_ME
+	string
+	prompt "env" if (FOO && BAR) || !BAZ || !QAZ
+	default "\"foo\"" if "foo \"bar\" baz" = ""
+""")
+
+    def verify_recursive(name):
+        try:
+            c.variables[name].expanded_value
+        except KconfigError:
+            pass
+        else:
+            fail("Expected '{}' expansion to flag recursive expansion, didn't"
+                 .format(name))
+
+    verify_recursive("rec-1")
+    # Indirectly verifies that it's not recursive
+    verify_variable("safe-fn-rec-res",
+                    "$(safe-fn-rec,safe-fn-rec-2)",
+                    "foo",
+                    True)
+    verify_recursive("unsafe-fn-rec")
+
+    verify_variable("foo-bar-baz", "$(rhs)", "value", True)
+
+    verify_variable("space-var-res", "$(foo bar)", "value", True)
+
+    verify_variable("shell-res",
+                    "$(shell,false && echo foo bar || echo baz qaz)",
+                    "baz qaz",
+                    True)
+
+    verify_variable("shell-stderr-res", "", "", False)
+
+    verify_variable("location-res",
+                    "Kconfiglib/tests/Kpreprocess:119",
+                    "Kconfiglib/tests/Kpreprocess:119",
+                    False)
+
+    verify_variable("warning-res", "", "", False)
+    verify_variable("error-n-res", "", "", False)
+
+    try:
+        c.variables["error-y-res"].expanded_value
+    except KconfigError:
+        pass
+    else:
+        fail("expanding error-y-res didn't raise an exception")
+
+    # Check that the expected warnings were generated
+    verify_equal(c.warnings, [
+        "Kconfiglib/tests/Kpreprocess:116: warning: 'echo message on stderr >&2' wrote to stderr: message on stderr",
+        "Kconfiglib/tests/Kpreprocess:124: warning: a warning"
+    ])
+
+
+
     print("\nAll selftests passed\n" if all_passed else
           "\nSome selftests failed\n")
 
@@ -2020,15 +2107,32 @@ def run_compatibility_tests():
     Runs tests on configurations from the kernel. Tests compability with the
     C implementation by comparing outputs.
     """
-    os.environ.pop("ARCH", None)
-    os.environ.pop("SRCARCH", None)
-    os.environ.pop("srctree", None)
 
-    if speedy and not os.path.exists("scripts/kconfig/conf"):
+    # Referenced inside the kernel Kconfig files.
+    #
+    # The str() makes the type of the value 'str' on both Python 2 and Python 3,
+    # which is nice for some later dictionary key sanity checks.
+
+    os.environ["KERNELVERSION"] = str(
+        subprocess.check_output("make kernelversion", shell=True)
+            .decode("utf-8").rstrip()
+    )
+
+    os.environ["CC_VERSION_TEXT"] = str(
+        subprocess.check_output("gcc --version | head -n1", shell=True)
+            .decode("utf-8").rstrip()
+    )
+
+    os.environ["srctree"] = "."
+    os.environ["CC"] = "gcc"
+
+
+    if not os.path.exists("scripts/kconfig/conf"):
         print("\nscripts/kconfig/conf does not exist -- running "
               "'make allnoconfig' to build it...")
         shell("make allnoconfig")
 
+
     print("Running compatibility tests...\n")
 
     test_fns = (test_defconfig,
@@ -2047,20 +2151,27 @@ def run_compatibility_tests():
         # function
         print(textwrap.dedent(test_fn.__doc__))
 
-        # Previously we used to load all the arches once and keep them around
-        # for the tests. That now uses a lot of memory (pypy helps a bit), so
-        # reload them for each test instead.
-        for kconf, arch, srcarch in all_arch_srcarch_kconfigs():
+        for arch, srcarch in all_arch_srcarch():
+            # Referenced inside the Kconfig files
+            os.environ["ARCH"] = arch
+            os.environ["SRCARCH"] = srcarch
+
             rm_configs()
-            test_fn(kconf, arch, srcarch)
+
+            test_fn(arch, srcarch)
 
     if all_passed:
         print("All selftests and compatibility tests passed")
     else:
         sys.exit("Some tests failed")
 
-def all_arch_srcarch_pairs():
+def all_arch_srcarch():
     for srcarch in os.listdir("arch"):
+        # These are currently broken with the C tools on linux-next as well.
+        # Perhaps they require cross-compilers to be installed.
+        if srcarch in ("arc", "h8300"):
+            continue
+
         if os.path.exists(os.path.join("arch", srcarch, "Kconfig")):
             yield (srcarch, srcarch)
 
@@ -2075,81 +2186,59 @@ def all_arch_srcarch_pairs():
 
     yield ("sh64", "sh")
 
-def all_arch_srcarch_kconfigs():
-    for arch, srcarch in all_arch_srcarch_pairs():
-        os.environ["ARCH"] = arch
-        os.environ["SRCARCH"] = srcarch
-        yield (Kconfig(), arch, srcarch)
-
-def test_allnoconfig(conf, arch, srcarch):
+def test_allnoconfig(arch, srcarch):
     """
     Verify that allnoconfig.py generates the same .config as
     'make allnoconfig', for each architecture. Runs the script via
-    'make scriptconfig', so kinda slow even in speedy mode.
+    'make scriptconfig'.
     """
-    # TODO: Support speedy mode for running the script
     shell("make scriptconfig SCRIPT=Kconfiglib/allnoconfig.py "
           "PYTHONCMD='{}'".format(sys.executable))
     shell("mv .config ._config")
-    if speedy:
-        shell("scripts/kconfig/conf --allnoconfig Kconfig")
-    else:
-        shell("make allnoconfig")
+    shell("scripts/kconfig/conf --allnoconfig Kconfig")
 
     compare_configs(arch)
 
-def test_allnoconfig_walk(conf, arch, srcarch):
+def test_allnoconfig_walk(arch, srcarch):
     """
     Verify that examples/allnoconfig_walk.py generates the same .config as
-    'make allnoconfig', for each architecture. Runs the script via 'make
-    scriptconfig', so kinda slow even in speedy mode.
+    'make allnoconfig', for each architecture. Runs the script via
+    'make scriptconfig'.
     """
-    # TODO: Support speedy mode for running the script
     shell("make scriptconfig SCRIPT=Kconfiglib/examples/allnoconfig_walk.py "
           "PYTHONCMD='{}'".format(sys.executable))
     shell("mv .config ._config")
-    if speedy:
-        shell("scripts/kconfig/conf --allnoconfig Kconfig")
-    else:
-        shell("make allnoconfig")
+    shell("scripts/kconfig/conf --allnoconfig Kconfig")
 
     compare_configs(arch)
 
-def test_allmodconfig(conf, arch, srcarch):
+def test_allmodconfig(arch, srcarch):
     """
     Verify that allmodconfig.py generates the same .config as
     'make allmodconfig', for each architecture. Runs the script via
-    'make scriptconfig', so kinda slow even in speedy mode.
+    'make scriptconfig'.
     """
-    # TODO: Support speedy mode for running the script
     shell("make scriptconfig SCRIPT=Kconfiglib/allmodconfig.py "
           "PYTHONCMD='{}'".format(sys.executable))
     shell("mv .config ._config")
-    if speedy:
-        shell("scripts/kconfig/conf --allmodconfig Kconfig")
-    else:
-        shell("make allmodconfig")
+    shell("scripts/kconfig/conf --allmodconfig Kconfig")
 
     compare_configs(arch)
 
-def test_allyesconfig(conf, arch, srcarch):
+def test_allyesconfig(arch, srcarch):
     """
     Verify that allyesconfig.py generates the same .config as
     'make allyesconfig', for each architecture. Runs the script via
-    'make scriptconfig', so kinda slow even in speedy mode.
+    'make scriptconfig'.
     """
-    # TODO: Support speedy mode for running the script
     shell("make scriptconfig SCRIPT=Kconfiglib/allyesconfig.py "
           "PYTHONCMD='{}'".format(sys.executable))
     shell("mv .config ._config")
-    if speedy:
-        shell("scripts/kconfig/conf --allyesconfig Kconfig")
-    else:
-        shell("make allyesconfig")
+    shell("scripts/kconfig/conf --allyesconfig Kconfig")
 
     compare_configs(arch)
 
-def test_sanity(conf, arch, srcarch):
+def test_sanity(arch, srcarch):
     """
     Do sanity checks on each configuration and call all public methods on all
     symbols, choices, and menu nodes for all architectures to make sure we
@@ -2157,35 +2246,37 @@ def test_sanity(conf, arch, srcarch):
     """
     print("For {}...".format(arch))
 
-    conf.modules
-    conf.defconfig_list
-    conf.defconfig_filename
-    conf.enable_redun_warnings()
-    conf.disable_redun_warnings()
-    conf.enable_undef_warnings()
-    conf.disable_undef_warnings()
-    conf.enable_warnings()
-    conf.disable_warnings()
-    conf.enable_stderr_warnings()
-    conf.disable_stderr_warnings()
-    conf.mainmenu_text
-    conf.unset_values()
-
-    conf.write_autoconf("/dev/null")
+    kconf = Kconfig()
+
+    kconf.modules
+    kconf.defconfig_list
+    kconf.defconfig_filename
+    kconf.enable_redun_warnings()
+    kconf.disable_redun_warnings()
+    kconf.enable_undef_warnings()
+    kconf.disable_undef_warnings()
+    kconf.enable_warnings()
+    kconf.disable_warnings()
+    kconf.enable_stderr_warnings()
+    kconf.disable_stderr_warnings()
+    kconf.mainmenu_text
+    kconf.unset_values()
+
+    kconf.write_autoconf("/dev/null")
 
     # No tempfile.TemporaryDirectory in Python 2
     tmpdir = tempfile.mkdtemp()
-    conf.sync_deps(os.path.join(tmpdir, "deps"))  # Create
-    conf.sync_deps(os.path.join(tmpdir, "deps"))  # Update
+    kconf.sync_deps(os.path.join(tmpdir, "deps"))  # Create
+    kconf.sync_deps(os.path.join(tmpdir, "deps"))  # Update
     shutil.rmtree(tmpdir)
 
     # Python 2/3 compatible
-    for key, sym in conf.syms.items():
+    for key, sym in kconf.syms.items():
         verify(isinstance(key, str), "weird key '{}' in syms dict".format(key))
 
         verify(not sym.is_constant, sym.name + " in 'syms' and constant")
 
-        verify(sym not in conf.const_syms,
+        verify(sym not in kconf.const_syms,
                sym.name + " in both 'syms' and 'const_syms'")
 
         for dep in sym._dependents:
@@ -2196,18 +2287,18 @@ def test_sanity(conf, arch, srcarch):
         sym.__repr__()
         sym.__str__()
         sym.assignable
-        conf.disable_warnings()
+        kconf.disable_warnings()
         sym.set_value(2)
         sym.set_value("foo")
         sym.unset_value()
-        conf.enable_warnings()
+        kconf.enable_warnings()
         sym.str_value
         sym.tri_value
         sym.type
         sym.user_value
         sym.visibility
 
-    for sym in conf.defined_syms:
+    for sym in kconf.defined_syms:
        verify(sym.nodes, sym.name + " is defined but lacks menu nodes")
 
        verify(not (sym.orig_type not in (BOOL, TRISTATE) and sym.choice),
@@ -2217,7 +2308,7 @@ def test_sanity(conf, arch, srcarch):
               "{} has broken dependency loop detection (_checked = {})"
               .format(sym.name, sym._checked))
 
-    for key, sym in conf.const_syms.items():
+    for key, sym in kconf.const_syms.items():
         verify(isinstance(key, str),
                "weird key '{}' in const_syms dict".format(key))
 
@@ -2238,17 +2329,17 @@ def test_sanity(conf, arch, srcarch):
         sym.__repr__()
         sym.__str__()
         sym.assignable
-        conf.disable_warnings()
+        kconf.disable_warnings()
         sym.set_value(2)
         sym.set_value("foo")
         sym.unset_value()
-        conf.enable_warnings()
+        kconf.enable_warnings()
         sym.str_value
         sym.tri_value
         sym.type
         sym.visibility
 
-    for choice in conf.choices:
+    for choice in kconf.choices:
         for sym in choice.syms:
             verify(sym.choice is choice,
                    "{0} is in choice.syms but 'sym.choice' is not the choice"
@@ -2270,7 +2361,7 @@ def test_sanity(conf, arch, srcarch):
 
     # Menu nodes
 
-    node = conf.top_node
+    node = kconf.top_node
 
     while 1:
         # Everything else should be well exercised elsewhere
@@ -2295,23 +2386,20 @@ def test_sanity(conf, arch, srcarch):
             else:
                 break
 
-def test_alldefconfig(conf, arch, srcarch):
+def test_alldefconfig(arch, srcarch):
     """
     Verify that alldefconfig.py generates the same .config as
     'make alldefconfig', for each architecture. Runs the script via
-    'make scriptconfig', so kinda slow even in speedy mode.
+    'make scriptconfig'.
     """
     shell("make scriptconfig SCRIPT=Kconfiglib/alldefconfig.py "
           "PYTHONCMD='{}'".format(sys.executable))
     shell("mv .config ._config")
-    if speedy:
-        shell("scripts/kconfig/conf --alldefconfig Kconfig")
-    else:
-        shell("make alldefconfig")
+    shell("scripts/kconfig/conf --alldefconfig Kconfig")
 
     compare_configs(arch)
 
-def test_defconfig(conf, arch, srcarch):
+def test_defconfig(arch, srcarch):
     """
     Verify that Kconfiglib generates the same .config as scripts/kconfig/conf,
     for each architecture/defconfig pair. In obsessive mode, this test includes
@@ -2322,6 +2410,8 @@ def test_defconfig(conf, arch, srcarch):
     With logging enabled, this test appends any failures to a file
     test_defconfig_fails in the root.
     """
+    kconf = Kconfig()
+
     if obsessive:
         defconfigs = []
 
@@ -2337,20 +2427,10 @@ def test_defconfig(conf, arch, srcarch):
     for defconfig in defconfigs:
         rm_configs()
 
-        conf.load_config(defconfig)
-        conf.write_config("._config")
-        if speedy:
-            shell("scripts/kconfig/conf --defconfig='{}' Kconfig".
-                  format(defconfig))
-        else:
-            shell("cp {} .config".format(defconfig))
-            # It would be a bit neater if we could use 'make *_defconfig'
-            # here (for example, 'make i386_defconfig' loads
-            # arch/x86/configs/i386_defconfig' if ARCH = x86/i386/x86_64),
-            # but that wouldn't let us test nonsensical combinations of
-            # arches and defconfigs, which is a nice way to find obscure
-            # bugs.
-            shell("make kconfiglibtestconfig")
+        kconf.load_config(defconfig)
+        kconf.write_config("._config")
+        shell("scripts/kconfig/conf --defconfig='{}' Kconfig".
+              format(defconfig))
 
         arch_defconfig_str = "  {:14}with {:60} ".format(arch, defconfig)
 
@@ -2364,11 +2444,13 @@ def test_defconfig(conf, arch, srcarch):
                     fail_log.write("{} with {} did not match\n"
                                    .format(arch, defconfig))
 
-def test_min_config(conf, arch, srcarch):
+def test_min_config(arch, srcarch):
     """
     Verify that Kconfiglib generates the same .config as 'make savedefconfig'
     for each architecture/defconfig pair.
     """
+    kconf = Kconfig()
+
     if obsessive_min_config:
         defconfigs = []
         for srcarch_ in os.listdir("arch"):
@@ -2377,16 +2459,12 @@ def test_min_config(conf, arch, srcarch):
         defconfigs = defconfig_files(srcarch)
 
     for defconfig in defconfigs:
-        conf.load_config(defconfig)
-        conf.write_min_config("._config")
+        kconf.load_config(defconfig)
+        kconf.write_min_config("._config")
 
         shell("cp {} .config".format(defconfig))
 
-        if speedy:
-            shell("scripts/kconfig/conf --savedefconfig=.config Kconfig")
-        else:
-            shell("make savedefconfig")
-            shell("mv defconfig .config")
+        shell("scripts/kconfig/conf --savedefconfig=.config Kconfig")
 
         arch_defconfig_str = "  {:14}with {:60} ".format(arch, defconfig)