Commit Graph

4235 Commits

Author SHA1 Message Date
Jim Mussared
671b35ceae py/builtinimport: Fix built-in imports when external import is disabled.
Follow-up to 24c02c4eb5 for when
MICROPY_ENABLE_EXTERNAL_IMPORT=0.  It now needs to try both extensible and
non-extensible modules.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-13 14:56:33 +10:00
Damien George
606ec9bfb1 py/compile: Fix async for's stack handling of iterator expression.
Prior to this fix, async for assumed the iterator expression was a simple
identifier, and used that identifier as a local to store the intermediate
iterator object.  This is incorrect behaviour.

This commit fixes the issue by keeping the iterator object on the stack as
an anonymous local variable.

Fixes issue #11511.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-13 13:50:50 +10:00
Jim Mussared
2fbc08c462 extmod/asyncio: Rename uasyncio to asyncio.
The asyncio module now has much better CPython compatibility and
deserves to be just called "asyncio".

This will avoid people having to write `from uasyncio import asyncio`.

Renames all files, and updates port manifests to use the new path. Also
renames the built-in _uasyncio to _asyncio.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-19 17:33:03 +10:00
Damien George
bf9ca0bb33 py/mkrules.mk: Allow $(AFLAGS) to set flags to $(AS).
Signed-off-by: Damien George <damien@micropython.org>
2023-06-15 11:02:15 +10:00
Jared Hancock
b3cd41dd4b py/lexer: Allow conversion specifiers in f-strings (e.g. !r).
PEP-498 allows for conversion specifiers like !r and !s to convert the
expression declared in braces to be passed through repr() and str()
respectively.

This updates the logic that detects the end of the expression to also stop
when it sees "![rs]" that is either at the end of the f-string or before
the ":" indicating the start of the format specifier. The "![rs]" is now
retained in the format string, whereas previously it stayed on the end
of the expression leading to a syntax error.

Previously: `f"{x!y:z}"` --> `"{:z}".format(x!y)`
Now: `f"{x!y:z}"` --> `"{!y:z}".format(x)`

Note that "!a" is not supported by `str.format` as MicroPython has no
`ascii()`, but now this will raise the correct error.

Updated cpydiff and added tests.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-14 19:11:04 +10:00
Damien George
5ce1a03a78 py/makemoduledefs.py: Automatically declare delegation attr functions.
So that the delegation functions don't need to be put somewhere global,
like in mpconfigport.h.  That would otherwise make it hard for extension
modules to use delegation.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-14 19:03:46 +10:00
Damien George
44295c9daa py/makemoduledefs.py: Fix declaring multiple module delegations.
Signed-off-by: Damien George <damien@micropython.org>
2023-06-14 19:03:29 +10:00
David Lechner
b02a5fa10a py/nlraarch64: Fix dangerous use of input register.
Starting with 2757acf6, the `top` variable in `nlr_jump()` in
`nlraarch64.c` was assigned to register `x19` by the compiler.  However,
the assembly code writes over that register with

    ldp x19, x20, [%0,  #32]

since `%0` is now `x19`. This causes the next line

    ldp lr,  x9,  [%0,  #16]

to load the wrong values.

To fix the issue, we move the value of the `top` variable from an unknown
register to a known register at the beginning of the asm code then only use
known/hard-coded registers after that.

Fixes issue #11754.

Signed-off-by: David Lechner <david@pybricks.com>
2023-06-14 17:43:44 +10:00
David Lechner
8cf9898dd3 py/parsenum: Fix typo in #endif comment.
This fixes a `#endif` comment to exactly match the `#if`.

Signed-off-by: David Lechner <david@pybricks.com>
2023-06-14 17:32:01 +10:00
Damien George
f01d5fb657 py/mkrules.mk: Automatically configure frozen options when manifest set.
Following how mkrules.cmake works.  This makes it easy for a port to enable
frozen code, by defining FROZEN_MANIFEST in its Makefile.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-08 23:12:56 +10:00
Jim Mussared
a1fbb1980c extmod/modtimeq: Remove timeq module.
This is a MicroPython-specific module that existed to support the old
version of uasyncio.  It's undocumented and not enabled on all ports and
takes up code size unnecessarily.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:28 +10:00
Jim Mussared
5e50975a6d py/modsys: Allow sys.path to be assigned to.
Previously sys.path could be modified by append/pop or slice assignment.

This allows `sys.path = [...]`, which can be simpler in many cases, but
also improves CPython compatibility.

It also allows sys.path to be set to a tuple which means that you can
clear sys.path (e.g. temporarily) with no allocations.

This also makes sys.path (and sys.argv for consistency) able to be disabled
via mpconfig. The unix port (and upytesthelper) require them, so they
explicitly verify that they're enabled.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
7d2ee8aed0 py/mpconfig: Enable module delegation if sys needs it.
Otherwise you can get into the confusing state where e.g. sys.ps1 is
enabled in config (via `MICROPY_PY_SYS_PS1_PS2`) but still doesn't actually
get enabled.

Also verify that the required delegation options are enabled in modsys.c.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
e6926d6021 py/objmodule: Workaround for MSVC with no module delegation.
When compiling mpy-cross, there is no `sys` module, and so there will
be no entries in the `mp_builtin_module_delegation_table`.

MSVC doesn't like this, so instead pretend as if the feature isn't
enabled at all.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
13c817e61c py/objmodule: Add a table of built-in modules with delegation.
This replaces the previous QSTR_null entry in the globals dict which could
leak out to Python (e.g. via iteration of mod.__dict__) and could lead to
crashes.

It results in smaller code size at the expense of turning a lookup into a
loop, but the list it is looping over likely only contains one or two
elements.

To allow a module to register its custom attr function it can use the new
`MP_REGISTER_MODULE_DELEGATION` macro.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
2eba98f1e0 all: Use MP_REGISTER_EXTENSIBLE_MODULE for overrideable built-ins.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:21 +10:00
Jim Mussared
24c02c4eb5 py/makemoduledefs.py: Add a way to register extensible built-in modules.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:21 +10:00
Jim Mussared
45ac651d1a all: Rename *umodule*.c to remove the "u" prefix.
Updates any includes, and references from Makefiles/CMake.

This essentially reverts what was done long ago in commit
136b5cbd76

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:17 +10:00
Jim Mussared
0ceccd4cf8 all: Rename *umodule*.h to remove the "u" prefix.
This work was funded through GitHub Sponsors.

Also updates #includes.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:14 +10:00
Jim Mussared
f5f9edf645 all: Rename UMODULE to MODULE in preprocessor/Makefile vars.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:11 +10:00
Jim Mussared
7f5d5c7271 all: Rename mod_umodule*, ^umodule* to remove the "u" prefix.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:07 +10:00
Jim Mussared
1bf2dcb15e all: Rename mp_umodule*, mp_module_umodule* to remove the "u" prefix.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:04 +10:00
Jim Mussared
dfe232d000 py/builtinimport: Remove weak links.
In order to keep "import umodule" working, the existing mechanism is
replaced with a simple fallback to drop the "u".

This makes importing of built-ins no longer touch the filesystem, which
makes a typical built-in import take ~0.15ms rather than 3-5ms.

(Weak links were added in c14a81662c)

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:04 +10:00
Jim Mussared
30628d1bb7 all: Rename MP_QSTR_umodule to MP_QSTR_module everywhere.
This renames the builtin-modules, such that help('modules') and printing
the module object will show "module" rather than "umodule".

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:53:57 +10:00
David Grayson
a79a6ab364 py/builtinimport: Remove partially-loaded modules from sys.modules.
Prior to this commit, importing a module that exists but has a syntax error
or some other problem that happens at import time would result in a
potentially-incomplete module object getting added to sys.modules.
Subsequent imports would use that object, resulting in confusing error
messages that hide the root cause of the problem.

This commit fixes that issue by removing the failed module from sys.modules
using the new NLR callback mechanism.

Note that it is still important to add the module to sys.modules while the
import is happening so that we can support circular imports just like
CPython does.

Fixes issue #967.

Signed-off-by: David Grayson <davidegrayson@gmail.com>
2023-06-05 23:21:52 +10:00
Damien George
ce31e5a2dc py: Use nlr jump callbacks to optimise compile/execute functions.
The changed functions now use less stack, and don't have any issues with
local variables needing to be declared volatile.

Testing on a PYBv1.0, imports (of .py, .mpy and frozen code) now use 64
less bytes of C stack per import depth.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-02 21:59:47 +10:00
Damien George
2757acf6ed py/nlr: Implement jump callbacks.
NLR buffers are usually quite large (use lots of C stack) and expensive to
push and pop.  Some of the time they are only needed to perform clean up if
an exception happens, and then they re-raise the exception.

This commit allows optimizing that scenario by introducing a linked-list of
NLR callbacks that are called automatically when an exception is raised.
They are essentially a light-weight NLR handler that can implement a
"finally" block, i.e. clean-up when an exception is raised, or (by passing
`true` to nlr_pop_jump_callback) when execution leaves the scope.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-02 21:50:57 +10:00
Damien George
f36ae5edcb py/nlr: Remove commented-out debugging code.
Also remove the unnecessary include of mpstate.h.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-02 21:47:34 +10:00
Damien George
24aa81e1da py/nlrsetjmp: Use MP_NLR_JUMP_HEAD macro to simplify code.
Signed-off-by: Damien George <damien@micropython.org>
2023-06-02 21:47:34 +10:00
Jim Mussared
6a8114eee8 py/objmodule: Don't use sys.modules to track a builtin __init__.
This can lead to duplicate initialisations if a module can be imported
via multiple names, so the module must track this itself anyway.

This reduces code size (diff is -40 bytes), and avoids special treatment of
builtin-modules-with-init with respect to sys.modules. No other builtin
modules get put into sys.modules.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-01 16:21:37 +10:00
Jim Mussared
ed90f30dd5 py/builtinimport: Allow builtin modules to be packages.
To use this:
 - Create a built-in module, and add the module object as a member of the
   parent module's globals dict.
 - The submodule can set its `__name__` to either `QSTR_foo_dot_bar` or
   `QSTR_bar`. The former requires using qstrdefs(port).h to make the qstr.

Because `bar` is a member of `foo`'s globals, it is possible to write
`import foo` and then immediately use `foo.bar` without importing it
explicitly. This means that if `bar` has an `__init__`, it will not be
called in this situation, and for that reason, sub-modules should not have
`__init__` methods. If this is required, then all initalisation for
sub-modules should be done by the top-level module's (i.e. `foo`'s)
`__init__` method.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-01 16:21:37 +10:00
Jim Mussared
525557738c py/builtinimport: Optimise sub-package loading.
This makes it so that sub-packages are resolved relative to their parent's
`__path__`, rather than re-resolving each parent's filesystem path.

The previous behavior was that `import foo.bar` would first re-search
`sys.path` for `foo`, then use the resulting path to find `bar`.

For already-loaded and u-prefixed modules, because we no longer need to
build the path from level to level, we no longer unnecessarily search
the filesystem. This should improve startup time.

Explicitly makes the resolving process clear:
 - Loaded modules are returned immediately without touching the filesystem.
 - Exact-match of builtins are also returned immediately.
 - Then the filesystem search happens.
 - If that fails, then the weak-link handling is applied.

This maintains the existing behavior: if a user writes `import time` they
will get time.py if it exits, otherwise the built-in utime. Whereas `import
utime` will always return the built-in.

This also fixes a regression from a7fa18c203
where we search the filesystem for built-ins. It is now only possible to
override u-prefixed builtins. This will remove a lot of filesystem stats
at startup, as micropython-specific modules (e.g. `pyb`) will no longer
attempt to look at the filesystem.

Added several improvements to the comments and some minor renaming and
refactoring to make it clearer how the import mechanism works. Overall
code size diff is +56 bytes on STM32.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-01 16:21:21 +10:00
Jim Mussared
42f3f66431 py/builtinimport: Handle empty sys.path correctly.
If sys.path is enabled, but empty, this will now no longer search the
filesystem. Previously an empty sys.path was equivalent to having
`sys.path=[""]`. This is a breaking change, but this behavior now matches
CPython.

This also provides an alternative mechanism to the u-prefix to force an
import of a builtin module:

```
import sys
_path = sys.path[:]
sys.path.clear()
import foo  # Forces the built-in foo.
sys.path.extend(_path)
del _path
```

Code size diff is -32 bytes on PYBV11.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-01 16:03:21 +10:00
Damien George
69dd013919 py/objint: Allow int() to parse anything with the buffer protocol.
This generalises and simplifies the code and follows CPython behaviour.

See similar change for floats in a07fc5b640.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-01 15:11:06 +10:00
Damien George
66dc1397c9 py/obj: Accept user types in mp_obj_get_int_maybe.
This is possible now that MP_UNARY_OP_INT_MAYBE exists.

As a consequence mp_obj_get_int now also supports user types, which was
previously possible with MP_UNARY_OP_INT but no tests existed for it.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-01 14:18:54 +10:00
Damien George
48ffd6596e py: Change MP_UNARY_OP_INT to MP_UNARY_OP_INT_MAYBE.
To be consistent with MP_UNARY_OP_INT_FLOAT and MP_UNARY_OP_INT_COMPLEX,
and allow int() to first check if a type supports __int__ before trying
other things (as per CPython).

Signed-off-by: Damien George <damien@micropython.org>
2023-06-01 13:01:07 +10:00
Damien George
ea7031faff py/runtime: If inplace binop fails then try corresponding normal binop.
The code that handles inplace-operator to normal-binary-operator fallback
is moved in this commit from py/objtype.c to py/runtime.c, making it apply
to all types, not just user classes.

Signed-off-by: Damien George <damien@micropython.org>
2023-05-19 13:44:00 +10:00
Damien George
4b57330465 py/objstr: Return unsupported binop instead of raising TypeError.
So that user types can implement reverse operators and have them work with
str on the left-hand-side, eg `"a" + UserType()`.

Signed-off-by: Damien George <damien@micropython.org>
2023-05-19 13:42:35 +10:00
Damien George
ca9068e0ef py/objarray: Disallow memoryview addition.
Following CPython.  This is important for subsequent commits to work
correctly.

Signed-off-by: Damien George <damien@micropython.org>
2023-05-19 13:33:54 +10:00
David Lechner
2fe6d4eb86 py/objdict: Fix __hash__ for dict_view types.
This adds a unary_op implementation for the dict_view type that makes
the implementation of `hash()` for these types compatible with CPython.

Signed-off-by: David Lechner <david@pybricks.com>
2023-05-19 12:06:17 +10:00
David Lechner
8491eb190f py/objslice: Ensure slice is not hashable.
As per https://bugs.python.org/issue408326, the slice object should not be
hashable.  Since MicroPython has an implicit fallback when the unary_op
slot is empty, we need to fill this slot.

Signed-off-by: David Lechner <david@pybricks.com>
2023-05-19 12:06:06 +10:00
David Lechner
eaccaa3677 py/obj: Remove mp_generic_unary_op().
Since converting to variable sized slots in mp_obj_type_t, we can now
reduce the code size a bit by removing mp_generic_unary_op() and the
corresponding slots where it is used. Instead we just implement the
generic `__hash__` operation in the runtime.

Signed-off-by: David Lechner <david@pybricks.com>
2023-05-19 12:04:44 +10:00
David Lechner
468ed218c9 py/gc: Make improvements to MICROPY_GC_HOOK_LOOP.
Changes in this commit:
- Add MICROPY_GC_HOOK_LOOP to gc_info() and gc_alloc().  Both of these can
  be long running (many milliseconds) which is too long to be blocking in
  some applications.
- Pass loop variable to MICROPY_GC_HOOK_LOOP(i) macro so that implementers
  can use it, e.g. to improve performance by only calling a function every
  X number of iterations.
- Drop outer call to MICROPY_GC_HOOK_LOOP in gc_mark_subtree().
2023-05-09 12:44:14 +10:00
Damien George
f1c6cb7725 py/stackctrl: Add gcc pragmas to ignore dangling-pointer warning.
This warning became apparent in gcc 13.

Signed-off-by: Damien George <damien@micropython.org>
2023-05-04 10:08:12 +10:00
Damien George
1b980c9dbe py/compile: Remove over-eager optimisation of tuples as if condition.
When a tuple is the condition of an if statement, it's only possible to
optimise that tuple away when it is a constant tuple (ie all its elements
are constants), because if it's not constant then the elements must be
evaluated in case they have side effects (even though the resulting tuple
will always be "true").

The code before this change handled the empty tuple OK (because it doesn't
need to be evaluated), but it discarded non-empty tuples without evaluating
them, which is incorrect behaviour (as show by the updated test).

This optimisation is anyway rarely applied because it's not common Python
coding practice to write things like `if (): ...` and `if (1, 2): ...`, so
removing this optimisation completely won't affect much code, if any.

Furthermore, when MICROPY_COMP_CONST_TUPLE is enabled, constant tuples are
already optimised by the parser, so expression with constant tuples like
`if (): ...` and `if (1, 2): ...` will continue to be optimised properly
(and so when this option is enabled the code that's deleted in this commit
is actually unreachable when the if condition is a constant tuple).

Signed-off-by: Damien George <damien@micropython.org>
2023-05-03 13:21:18 +10:00
Damien George
957bd51184 py/parse: Fix build when COMP_CONST_FOLDING=0 and COMP_MODULE_CONST=1.
Signed-off-by: Damien George <damien@micropython.org>
2023-05-03 13:12:54 +10:00
glenn20
9d735d1be7 py/ringbuf: Implement put_bytes/get_bytes functions. 2023-05-01 16:47:03 +10:00
Damien George
b1229efbd1 all: Fix spelling mistakes based on codespell check.
Signed-off-by: Damien George <damien@micropython.org>
2023-04-27 18:03:06 +10:00
Damien George
9955553001 extmod/modutime: Provide a generic time module.
Based on extmod/utime_mphal.c, with:
- a globals dict added
- time.localtime wrapper added
- time.time wrapper added
- time.time_ns function added

New configuration options are added for this module:
- MICROPY_PY_UTIME (enabled at basic features level)
- MICROPY_PY_UTIME_GMTIME_LOCALTIME_MKTIME
- MICROPY_PY_UTIME_TIME_TIME_NS

Signed-off-by: Damien George <damien@micropython.org>
2023-04-27 15:09:56 +10:00
Damien George
294baf52b3 all: Bump version to 1.20.0.
Signed-off-by: Damien George <damien@micropython.org>
2023-04-26 15:42:28 +10:00
Jim Mussared
256f47e2f8 extmod/btstack: Fix indicate/notify queuing.
This adds a mechanism to track a pending notify/indicate operation that
is deferred due to the send buffer being full. This uses a tracked alloc
that is passed as the content arg to the callback.

This replaces the previous mechanism that did this via the global pending
op queue, shared with client read/write ops.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-04-26 11:37:20 +10:00
Alex Riesen
a22136a732 py/makeqstrdefs.py: Fix handling GreenHills C/C++ preprocessor output.
The GreenHills preprocessor produces #line directives without a file name,
which the regular expression used to distiguish between
"# <number> file..." (GCC and similar) and "#line <number> file..."
(Microsoft C and similar) does not match, aborting processing.

Besides, the regular expression was unnecessarily wide, matching lines
containing a "#", followed by any  number of 'l','i','n', and 'e'
characters.

Signed-off-by: Alex Riesen <alexander.riesen@cetitec.com>
2023-04-11 16:41:10 +10:00
David Lechner
283c1ba07e py/obj: Fix spelling of staticmethod.
Signed-off-by: David Lechner <david@pybricks.com>
2023-03-28 10:02:05 -05:00
Damien George
d54208a2ff py/scheduler: Implement VM abort flag and mp_sched_vm_abort().
This is intended to be used by the very outer caller of the VM/runtime.  It
allows setting a top-level NLR handler that can be jumped to directly, in
order to forcefully abort the VM/runtime.

Enable using:

    #define MICROPY_ENABLE_VM_ABORT (1)

Set up the handler at the top level using:

    nlr_buf_t nlr;
    nlr.ret_val = NULL;
    if (nlr_push(&nlr) == 0) {
        nlr_set_abort(&nlr);
        // call into the VM/runtime
        ...
        nlr_pop();
    } else {
        if (nlr.ret_val == NULL) {
            // handle abort
            ...
        } else {
            // handle other exception that propagated to the top level
            ...
        }
    }
    nlr_set_abort(NULL);

Schedule an abort, eg from an interrupt handler, using:

    mp_sched_vm_abort();

Signed-off-by: Damien George <damien@micropython.org>
2023-03-21 18:08:57 +11:00
Damien George
5d4bfce034 py/mpstate: Add mp_thread_is_main_thread() helper macro.
Signed-off-by: Damien George <damien@micropython.org>
2023-03-21 18:01:01 +11:00
David Lechner
fa8ebb1390 py/obj: Add MP_NOINLINE to mp_obj_malloc_helper.
As the comment in py/obj.h says:

> Implementing this as a call rather than inline saves 8 bytes per usage.

So in order to get this savings, we need to tell the compiler to never
inline the function.

Signed-off-by: David Lechner <david@pybricks.com>
2023-03-17 13:16:06 -05:00
Damien George
668a7bd28a py/makeversionhdr.py: Always add micro to version string even if it's 0.
Moving forward, tags in this repository will always have three components.

Signed-off-by: Damien George <damien@micropython.org>
2023-03-10 18:39:14 +11:00
Laurens Valk
f3a596db7d py/builtinimport: Fix unix port build with external imports disabled.
Without this, building the unix port variants gives:
ports/unix/main.c:667: undefined reference to `mp_obj_is_package',
when MICROPY_ENABLE_EXTERNAL_IMPORT is 0.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2023-03-10 13:29:54 +11:00
Damien George
78dc2db2ba py/mpconfig: Provide config option for internal printf printer.
The C-level printf is usually used for internal debugging prints, and a
port/board may want to redirect this somewhere other than stdout.

Signed-off-by: Damien George <damien@micropython.org>
2023-03-10 11:19:15 +11:00
Damien George
b3c8ab37ec py/gc: Make gc_dump_info/gc_dump_alloc_table take a printer as argument.
So that callers can redirect the output if needed.

Signed-off-by: Damien George <damien@micropython.org>
2023-03-10 10:58:10 +11:00
Damien George
7c1584aef1 py/compile: Fix scope of assignment expression target in comprehensions.
When := is used in a comprehension the target variable is bound to the
parent scope, so it's either a global or a nonlocal.  Prior to this commit
that was handled by simply using the parent scope's id_info for the
target variable.  That's completely wrong because it uses the slot number
for the parent's Python stack to store the variable, rather than the slot
number for the comprehension.  This will in most cases lead to incorrect
behaviour or memory faults.

This commit fixes the scoping of the target variable by explicitly
declaring it a global or nonlocal, depending on whether the parent is the
global scope or not.  Then the id_info of the comprehension can be used to
access the target variable.  This fixes a lot of cases of using := in a
comprehension.

Code size change for this commit:

       bare-arm:    +0 +0.000%
    minimal x86:    +0 +0.000%
       unix x64:  +152 +0.019% standard
          stm32:   +96 +0.024% PYBV10
         cc3200:   +96 +0.052%
        esp8266:  +196 +0.028% GENERIC
          esp32:  +156 +0.010% GENERIC[incl +8(data)]
         mimxrt:   +96 +0.027% TEENSY40
     renesas-ra:   +88 +0.014% RA6M2_EK
            nrf:   +88 +0.048% pca10040
            rp2:  +104 +0.020% PICO
           samd:   +88 +0.033% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Fixes issue #10895.

Signed-off-by: Damien George <damien@micropython.org>
2023-03-09 12:13:12 +11:00
David Grayson
2bcd88d556 py/makeversionhdr.py: Optionally get git tag and git hash from env vars.
This is handy when you are doing builds outside of the Git repository but
still want to record that information.

Signed-off-by: David Grayson <davidegrayson@gmail.com>
2023-03-02 23:19:30 +11:00
Pepijn de Vos
72e9318325 py/emitnative: Explicitly compare comparison ops in binary_op emitter.
Without this it's possible to get a compiler error about the comparison
always being true, because MP_BINARY_OP_LESS is 0.  And it seems that gcc
optimises these 6 equality comparisons into the same size machine code as
before.
2023-02-27 10:25:39 +11:00
Damien George
2e4dda3c20 py/modmath: Fix two-argument math function domain check.
Prior to this fix, pow(1.5, inf) and pow(0.5, -inf) (among other things)
would incorrectly raise a ValueError, because the result is inf with the
first argument being finite.  This commit fixes this by allowing the result
to be infinite if the first or second (or both) argument is infinite.

This fix doesn't affect the other three math functions that have two
arguments:
- atan2 never returns inf, so always fails isinf(ans)
- copysign returns inf only if the first argument x is inf, so will never
  reach the isinf(y) check
- fmod never returns inf, so always fails isinf(ans)

Signed-off-by: Damien George <damien@micropython.org>
2023-02-24 15:55:12 +11:00
Jim Mussared
9848b0685f py/mkrules.cmake: Force build mpversion.h and frozen_content.c.
This ensures that all builds unconditionally run makeversionhdr.py and
makemanifest.py to generate mpversion.h and frozen_content.c respectively.
This now matches the Makefile behavior, and in particular this fixes the
issue on ESP32 builds that changes in code-to-be-frozen will cause the
build to update. Both these already tools know not to touch their output
if there is no change.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-02-08 12:24:15 +11:00
Jim Mussared
8b27482692 top: Update Python formatting to black "2023 stable style".
See https://black.readthedocs.io/en/stable/the_black_code_style/index.html

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-02-02 12:51:03 +11:00
Damien George
31139b437a py/mkrules: Support mpy-tool-flags in cmake frozen code generation.
Signed-off-by: Damien George <damien@micropython.org>
2023-01-24 16:55:23 +11:00
Damien George
d387ae3444 py/objint_mpz: Catch and reject @ and @= operating on big integers.
This will also catch / and /= when float support is disabled.

Fixes issue #10544.

Signed-off-by: Damien George <damien@micropython.org>
2023-01-23 13:03:51 +11:00
Alex Riesen
abaa4abd2d py: Add parenthesis to default impl of MP_OBJ_TO_PTR, MP_OBJ_FROM_PTR.
Unless MICROPY_OBJ_REPR == MICROPY_OBJ_REPR_D, these macros only work with
values and "->"/"." expressions as their sole argument.  In other words,
the macros are broken with expressions which contain operations of lower
precedence than the cast operator.

Depending on situation, the old code either results in compiler error:

 MP_OBJ_TO_PTR(flag ? o1 : o2) expands into "(void *)flag ? o1 : o2",
 which some compiler configurations will reject (e.g. GCC -Wint-conversion
 -Wint-to-pointer-cast -Werror)

Or in an incorrect address calculation:

 For ptr declared as "uint8_t *" the MP_OBJ_FROM_PTR(ptr + off)
 expands into ((mp_obj_t)ptr) + off, resulting in an obviously
 wrong address.

Signed-off-by: Alex Riesen <alexander.riesen@cetitec.com>
2023-01-20 21:59:49 +11:00
Jim Mussared
fb8792c095 py/lexer: Wrap in parenthesis all f-string arguments passed to format.
This is important for literal tuples, e.g.

    f"{a,b,}, {c}" --> "{}".format((a,b), (c),)

which would otherwise result in either a syntax error or the wrong result.

Fixes issue #9635.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-01-20 17:54:32 +11:00
Andrew Leech
5c4153ea37 py/objarray: Raise error on out-of-bound memoryview slice start.
32-bit platforms only support a slice offset start of 24 bit max due to the
limited size of the mp_obj_array_t.free member.  Similarly on 64-bit
platforms the limit is 56 bits.

This commit adds an OverflowError if the user attempts to slice a
memoryview beyond this limit.

Signed-off-by: Damien George <damien@micropython.org>
2023-01-20 16:31:37 +11:00
robert-hh
e78e0b7418 py/gc: Increase the address length in gc_dump_alloc_table().
Showing 8 digits instead of 5, supporting devices with more than 1 MByte of
RAM (which is common these days).  The masking was never needed, and the
related commented-out line can go.
2023-01-16 12:09:20 +11:00
Philip Peitsch
edc92d18db py/map: Clear value when re-using slot with ordered dictionaries.
To adhere to the contract of mp_map_lookup, namely:

    MP_MAP_LOOKUP_ADD_IF_NOT_FOUND behaviour:
    - returns slot, with key non-null and value=MP_OBJ_NULL if it was added
2023-01-13 14:21:32 +11:00
Damien George
910f579403 py/emitnative: Initialise locals as Python object type for native code.
In @micropython.native code the types of variables and expressions are
always Python objects, so they can be initialised as such.  This prevents
problems with compiling optimised code like while-loops where a local may
be referenced before it is assigned to.

Signed-off-by: Damien George <damien@micropython.org>
2022-12-16 11:44:10 +11:00
Jim Mussared
9e91764671 py/obj: Remove unused MP_DEFINE_CONST_OBJ_FULL_TYPE macro.
This was previously used for the definition of NIC types, but they have
been updated to use a protocol instead.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-12-15 17:40:36 +11:00
Damien George
ab0258fb1e py/gc: Fix debug printing of GC layout.
Signed-off-by: Damien George <damien@micropython.org>
2022-12-08 14:36:34 +11:00
Jeff Epler
d75ff42297 unix/coverage: Add extra GC coverage test for ATB gap byte.
The assertion that is added here (to gc.c) fails when running this new test
if ALLOC_TABLE_GAP_BYTE is set to 0.

Signed-off-by: Jeff Epler <jepler@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-12-08 14:35:08 +11:00
Jeff Epler
9f434dd8de py/gc: Ensure a gap of one byte after the ATB.
Prior to this fix the follow crash occurred.  With a GC layout of:

    GC layout:
      alloc table at 0x3fd80428, length 32001 bytes, 128004 blocks
      finaliser table at 0x3fd88129, length 16001 bytes, 128008 blocks
      pool at 0x3fd8bfc0, length 2048064 bytes, 128004 blocks

Block 128003 is an AT_HEAD and eventually is passed to gc_mark_subtree.
This causes gc_mark_subtree to call ATB_GET_KIND(128004).  When block 1 is
created with a finaliser, the first byte of the finaliser table becomes
0x2, but ATB_GET_KIND(128004) reads these bits as AT_TAIL, and then
gc_mark_subtree references past the end of the heap, which happened to be
past the end of PSRAM on the esp32-s2.

The fix in this commit is to ensure there is a one-byte gap after the ATB
filled permanently with AT_FREE.

Fixes issue #7116.

See also https://github.com/adafruit/circuitpython/issues/5021

Signed-off-by: Jeff Epler <jepler@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-12-08 14:29:58 +11:00
Jeff Epler
84071590b3 py/gc: Avoid valgrind false positives.
When you want to use the valgrind memory analysis tool on MicroPython, you
can arrange to define MICROPY_DEBUG_VALGRIND to enable use of special
valgrind macros.  For now, this only fixes `gc_get_ptr` so that it never
emits the diagnostic "Conditional jump or move depends on uninitialised
value(s)".

Signed-off-by: Jeff Epler <jepler@gmail.com>
2022-12-08 14:29:22 +11:00
Damien George
2283b6d68f py: Pass in address to compiled module instead of returning it.
This change makes it so the compiler and persistent code loader take a
mp_compiled_module_t* as their last argument, instead of returning this
struct.  This eliminates a duplicate context variable for all callers of
these functions (because the context is now stored in the
mp_compiled_module_t by the caller), and also eliminates any confusion
about which context to use after the mp_compile_to_raw_code or
mp_raw_code_load function returns (because there is now only one context,
that stored in mp_compiled_module_t.context).

Reduces code size by 16 bytes on ARM Cortex-based ports.

Signed-off-by: Damien George <damien@micropython.org>
2022-12-08 12:27:23 +11:00
Damien George
a2347433b0 py: Remove the word "yet" from exception messages.
These unimplemented features may never be implemented, and having the word
"yet" there takes up space.

Signed-off-by: Damien George <damien@micropython.org>
2022-12-06 13:34:52 +11:00
Laurens Valk
632d43ed44 py/mpconfig: Include micropython module in core features.
This excludes it from the minimal builds.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2022-12-06 13:34:17 +11:00
Laurens Valk
f724d90e6c py/modmicropython: Make module optional.
This module is useful, but it is not always needed. Disabling it saves
several kilobytes of build size, depending on other config options.

Signed-off-by: Laurens Valk <laurens@pybricks.com>
2022-12-06 13:34:17 +11:00
David Lechner
d5181034f2 py/bc: Fix checking for duplicate **kwargs.
The code was already checking for duplicate kwargs for named parameters but
if `**kwargs` was given as a parameter, it did not check for multiples of
the same argument name.

This fixes the issue by adding an addition test to catch duplicates and
adds a test to exercise the code.

Fixes issue #10083.

Signed-off-by: David Lechner <david@pybricks.com>
2022-11-28 11:23:22 +11:00
Rayane Chatrieux
f3e4c505d1 py/objdict: Implement dictionary union (PEP 584).
Implements dictionary union according to PEP 584's specifications, minus
the fact that dictionary entries are not guaranteed to be in insertion
order.  This feature is enabled with MICROPY_CPYTHON_COMPAT.

Includes a new test.

With the assistance of Fangrui Qin <qinf@purdue.edu>

Signed-off-by: Rayane Chatrieux <rayane.chatrieux@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-11-25 12:46:14 +11:00
Jim Mussared
6873a14b61 py/makeversionhdr.py: Allow running outside of repo.
If a CMake-build is run with `make BUILD=/outside/path` then
makeversionheader.py is run with the CWD set to the build directory, which
means the git version lookup will fail and silently fall back to the
mpconfig.h mode (giving the wrong result).

This commit:
 - Uses the location of makeversionheader.py to find the repo path.
 - Allows overriding this path via --repo-path.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-11-22 14:36:21 +11:00
Jan Willeke
40a3aa709c py/nlrmips: Add native NLR support for MIPS architecture.
This can be tested using ports/minimal and qemu:

    make CC=mips-linux-gnu-gcc-8

Then run with qemu-mips:

    stty raw opost -echo;
    QEMU_LD_PREFIX=/usr/mips-linux-gnu/ qemu-mips build/firmware.elf;
    sleep 1; reset

Signed-off-by: Jan Willeke <willeke@smartmote.de>
2022-11-15 17:09:49 +11:00
Damien George
0698dd72ea py/emitnative: Ensure load_subscr does not clobber existing REG_ARG_2.
Follow up from a similar fix in 426785a19e

Fixes issue #6314.

Signed-off-by: Damien George <damien@micropython.org>
2022-11-11 12:25:32 +11:00
Jim Mussared
2c8dab7ab4 py/objarray: Detect bytearray(str) without an encoding.
This prevents a very subtle bug caused by writing e.g. `bytearray('\xfd')`
which gives you `(0xc3, 0xbd)`.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-11-08 23:09:22 +11:00
Simon Arlott
f8b0ae32d3 py/builtinimport: Fix crash handling "weak link" module names.
There are two calls to mp_builtin___import__():
1. ports/unix/main.c:main_() which provides a str in args[0]
2. py/runtime.c:mp_import_name() which provides a qstr in args[0]

The default implementation of mp_builtin___import__() is
mp_builtin___import___default() which has a different implementation based
on MICROPY_ENABLE_EXTERNAL_IMPORT.

If MICROPY_ENABLE_EXTERNAL_IMPORT is disabled then the handling of weak
links assumes that args[0] is a `const char *`, when it is either a str or
qstr object.

Use the existing qstr of the module name instead, and also use a vstr
instead of strcpy() to ensure no overflow occurs.
2022-11-08 23:03:21 +11:00
Damien George
cd35b8a2a3 extmod/machine_timer: Move stm32's implementation of machine.Timer here.
So it can be reused by other ports.

Signed-off-by: Damien George <damien@micropython.org>
2022-10-27 14:32:43 +11:00
Damien George
c138e10fbb py/makeversionhdr: Fall back to py/mpconfig.h instead of docs/conf.py.
Commit 64af916c11 removed the version string
from docs/conf.py.  py/mpconfig.h is a better place to get the version
from, so use that (when there is no git repository).

Signed-off-by: Damien George <damien@micropython.org>
2022-10-27 13:00:48 +11:00
Jim Mussared
d75c7e822c py/obj: Add comments explaining the slot index scheme.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-26 00:08:41 +11:00
Jim Mussared
1ba0e8ff96 py/persistentcode: Only emit sub-version if generated code has native.
In order for v1.19.1 to load a .mpy, the formerly-feature-flags which are
now used for the sub-version must be zero.

The sub-version is only used to indicate a native version change, so it
should be zero when emitting bytecode-only .mpy files.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-25 14:57:04 +11:00
Jim Mussared
b161abc574 py/obj: Verify floating point type is correct for repr C.
Prevents double-precision floats being enabled on 32-bit architectures
where they will not fit into the mp_obj_t encoding.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-25 14:46:04 +11:00
David Lechner
ab317a0d66 py/misc: Remove use of bitfield from vstr_t.
Since there is only one flag, we don't need to use a bitfield in vstr_t.
Compilers emit extra instructions to access a bitfield, so this should
reduce the binary size a small amount.

Signed-off-by: David Lechner <david@pybricks.com>
2022-10-12 00:33:09 +11:00
Jim Mussared
d6d8722558 extmod: Make extmod.mk self-contained.
This makes it so that all a port needs to do is set the relevant variables
and "include extmod.mk" and doesn't need to worry about adding anything to
OBJ, CFLAGS, SRC_QSTR, etc.

Make all extmod variables (src, flags, etc) private to extmod.mk.

Also move common/shared, extmod-related fragments (e.g. wiznet, cyw43,
bluetooth) into extmod.mk.

Now that SRC_MOD, CFLAGS_MOD, CXXFLAGS_MOD are unused by both extmod.mk
(and user-C-modules in a previous commit), remove all uses of them from
port makefiles.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:31:49 +11:00
Jim Mussared
3cc6decfc4 py/py.mk: Make user-C-module handling self-contained in py.mk.
Removes the need for the port to add anything to OBJS or SRC_QSTR.

Also makes it possible for user-C-modules to differentiate between code
that should be processed for QSTR vs other files (e.g. helpers and
libraries).

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 23:23:18 +11:00
Jim Mussared
0e8dfaf538 py/modsys: Add support for sys.executable.
Only intended to be used on Unix and other "OS" ports.  Matches CPython.
This should give the absolute path to the executing binary.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
Signed-off-by: Damien George <damien@micropython.org>
2022-10-11 18:10:26 +11:00
Jim Mussared
c44b3927b8 py/objstr: Add a helper to set mp_obj_str_t data.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-10-11 17:50:19 +11:00