Commit Graph

4212 Commits

Author SHA1 Message Date
Jim Mussared
276bfa3146 py/lexer: Add missing initialisation for fstring_args_idx.
This was missed in 692d36d779. Probably
never noticed because everything enables `MICROPY_GC_CONSERVATIVE_CLEAR`,
but found via ASAN thanks to @gwangmu & @chibinz.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-09-29 13:58:26 +10:00
Ihor Nehrutsa
d83c1a43d4 py: Change ifdef DEBUG_PRINT to if DEBUG_PRINT.
Signed-off-by: Ihor Nehrutsa <Ihor.Nehrutsa@gmail.com>
2023-09-29 13:04:38 +10:00
Angus Gratton
2fcd28f713 py/mkrules.mk: Don't strip binary if STRIP variable is unset.
This provides a way to build a non-DEBUG host binary that still has symbols
and debug information.

Document this for the unix port, and update a comment in the unix port
Makefile.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-09-29 13:01:12 +10:00
Angus Gratton
fa68523968 py/nlrx64: Mark nlr_push() as naked function when possible.
Supported from GCC 8 and up, and Compiler Explorer suggests it works as
expected with Clang since 3.6 (2014).

- Fixes situation where building embedded MicroPython with -O0 and
  MICROPY_NLR_X64 crashes at runtime (due to nlr_push pushing the
  frame pointer register EBP). Closes #12421.

- Allows removing the macOS tweak to undo pushing EBP onto the stack
  in the generated function prelude.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-09-20 11:48:36 +10:00
Angus Gratton
3e8aed9fcc py/gc: Add "max new split" value in result of gc.mem_free().
Follow-up to 519c24dd48 when MICROPY_GC_SPLIT_HEAP_AUTO is enabled, based
on discussion at
https://github.com/orgs/micropython/discussions/12316#discussioncomment-6858007

gc.mem_free() is always a heuristic, but this makes it a more useful
heuristic for common use cases.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-09-15 12:19:13 +10:00
Damien George
397697aa83 py/persistentcode: Always close reader even if an exception is raised.
Fixes issue #3874.

Signed-off-by: Damien George <damien@micropython.org>
2023-09-14 13:03:31 +10:00
Damien George
5e122b11ea py/parse: Always free lexer even if an exception is raised.
Fixes issue #3843.

Signed-off-by: Damien George <damien@micropython.org>
2023-09-14 13:03:31 +10:00
Damien George
c9089e71a1 py/runtime: Add helpers to call a general function on nlr jump callback.
Signed-off-by: Damien George <damien@micropython.org>
2023-09-14 13:03:31 +10:00
IhorNehrutsa
6e79b55dc8 py/malloc: Fix DEBUG_print() args in m_realloc_maybe.
Signed-off-by: Ihor Nehrutsa <IhorNehrutsa@gmail.com>
2023-09-04 10:26:09 +10:00
David Lechner
ffb43b2dd3 py/modthread: Return thread id from start_new_thread().
In CPython, `_thread.start_new_thread()` returns an ID that is the same ID
that is returned by `_thread.get_ident()`.  The current MicroPython
implementation of `_thread.start_new_thread()` always returns `None`.

This modifies the required functions to return a value. The native thread
id is returned since this can be used for interop with other functions, for
example, `pthread_kill()` on *nix. `_thread.get_ident()` is also modified
to return the native thread id so that the values match and avoids the need
for a separate `native_id` attribute.

Fixes issue #12153.

Signed-off-by: David Lechner <david@pybricks.com>
2023-09-03 18:49:18 +10:00
Jim Mussared
a64f2fdca0 py/dynruntime.h: Implement MP_OBJ_NEW_QSTR.
Because mpy_ld.py doesn't know the target object representation, it emits
instances of `MP_OBJ_NEW_QSTR(MP_QSTR_Foo)` as const string objects, rather
than qstrs. However this doesn't work for map keys (e.g. for a locals dict)
because the map has all_keys_are_qstrs flag is set (and also auto-complete
requires the map keys to be qstrs).

Instead, emit them as regular qstrs, and make a functioning MP_OBJ_NEW_QSTR
function available (via `native_to_obj`, also used for e.g. making
integers).

Remove the code from mpy_ld.py to emit qstrs as constant strings, but leave
behind the scaffold to emit constant objects in case we want to do use this
in the future.

Strictly this should be a .mpy sub-version bump, even though the function
table isn't changing, it does lead to a change in behavior for a new .mpy
running against old MicroPython. `mp_native_to_obj` will incorrectly return
the qstr value directly as an `mp_obj_t`, leading to unexpected results.
But given that it's broken at the moment, it seems unlikely that anyone is
relying on this, so it's not work the other downsides of a sub-version bump
(i.e. breaking pure-Python modules that use @native). The opposite case of
running an old .mpy on new MicroPython is unchanged, and remains broken in
exactly the same way.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-09-01 16:14:22 +10:00
Daniël van de Giessen
d1f288c041 py/modstruct: Support pad bytes in struct format.
This adds support for the x format code in struct.pack and struct.unpack.

The primary use case for this is ignoring bytes while unpacking.  When
interfacing with existing systems, it may often happen that you either have
fields in a struct that aren't properly specified or you simply don't care
about them.  Being able to easily skip them is useful.

Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2023-09-01 14:50:49 +10:00
mcskatkat
e0a1480600 py/objstr: Fix str % {} edge case.
Eliminate `TypeError` when format string contains no named conversions.
This matches CPython behavior.

Signed-off-by: mcskatkat <mc_skatkat@hotmail.com>
2023-09-01 14:31:57 +10:00
Jim Mussared
f3eccb154c py/mpconfig: Enable SSL finalizers if finalizers are enabled.
The rp2 port was enabling SSL and had finalizers enabled via the "extra
features" level, but missed explicitly enabling `MICROPY_PY_SSL_FINALISER`
(like esp32, stm32, and mimxrt did).

This commit makes `MICROPY_PY_SSL_FINALISER` default to enabled if
finalizers are enabled, and removes the explicit setting of this for
esp32, stm32, mimxrt (because they all use the "extra features" level).

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-30 12:38:13 +10:00
Jim Mussared
b2b5bcce28 py/profile: Remove the requirement to disable MICROPY_COMP_CONST.
The only reason that const had to be disabled was to make the test output
match CPython when const was involved.  Instead, this commit fixes the test
to handle the lines where const is used.

Also:
- remove the special handling for MICROPY_PERSISTENT_CODE_SAVE in
  unix/mpconfigport.h, and make this automatic.
- move the check for MICROPY_PERSISTENT_CODE_SAVE to where it's used (like
  we do for other similar checks) and add a comment explaining it.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-30 10:58:04 +10:00
Jim Mussared
bc1af5ab7f cc3200/Makefile: Build firmware.zip.
This allows the cc3200 port to be build with the standard autobuild script
rather than the custom build-cc3200-latest.sh (which is now removed).

This also fixes the path inside the zip file (by using the `-j` flag to
zip).

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-24 15:44:20 +10:00
Jim Mussared
9573d31071 all: Remove query-variants make target.
This is difficult to implement on cmake-based ports, and having the list
of variants in mpconfigboard.{cmake,mk} duplicates information that's
already in board.json.

This removes the existing query-variants make target from stm32 & rp2
and the definition of BOARD_VARIANTS from the various board files.

Also renames the cmake variable to MICROPY_BOARD_VARIANT to match other
variables such as MICROPY_BOARD. The make variable stays as
BOARD_VARIANT.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-15 17:37:44 +10:00
Jim Mussared
ad123ed013 esp32/Makefile: Implement make submodules to match other ports.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-15 17:37:44 +10:00
Angus Gratton
519c24dd48 py/gc: Add new MICROPY_GC_SPLIT_HEAP_AUTO "auto grow heap" mode.
When set, the split heap is automatically extended with new areas on
demand, and shrunk if a heap area becomes empty during a GC pass or soft
reset.

To save code size the size allocation for a new heap block (including
metadata) is estimated at 103% of the failed allocation, rather than
working from the more complex algorithm in gc_try_add_heap(). This appears
to work well except in the extreme limit case when almost all RAM is
exhausted (~last few hundred bytes). However in this case some allocation
is likely to fail soon anyhow.

Currently there is no API to manually add a block of a given size to the
heap, although that could easily be added if necessary.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-08-15 10:48:02 +10:00
Angus Gratton
d325ee4509 py/gc: Apply some code formatting cleanup.
This commit:
- Breaks up some long lines for readability.
- Fixes a potential macro argument expansion issue.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-08-15 10:41:02 +10:00
Damien George
ef71028f77 extmod/modselect: Add optimisation to use system poll when possible.
A previous commit removed the unix-specific select module implementation
and made unix use the common one.

This commit adds an optimisation so that the system poll function is used
when polling objects that have a file descriptor.  With this optimisation
enabled, if code registers both file-descriptor-based objects, and non-
file-descriptor-based objects with select.poll() then the following occurs:

- the system poll is called for all file-descriptor-based objects with a
  timeout of 1ms

- then the bare-metal polling implementation is used for remaining objects,
  which calls into their ioctl method (which can be in C or Python)

In the case where all objects have file descriptors, the system poll is
called with the full timeout requested by the caller.  That makes it as
efficient as possible in the case everything has a file descriptor.

Benefits of this approach:

- all ports use the same select module implementation

- the unix port now supports polling of all objects and matches bare metal
  implementations

- it's still efficient for existing cases where only files and sockets are
  polled (on unix)

- the bare metal implementation does not change

- polling of SSL objects will now work on unix by calling in to the ioctl
  method on SSL objects (this is required for asyncio ssl support)

Note that extmod/vfs_posix_file.c has poll disable when the optimisation is
enabled, because the code is not reachable when the optimisation is used.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:11:40 +10:00
Damien Tournoud
2dcd745434 py/gc: Speed up incremental GC cycles by tracking the last used block.
In applications that use little memory and run GC regularly, the cost of
the sweep phase quickly becomes prohibitives as the amount of RAM
increases.

On an ESP32-S3 with 2 MB of external SPIRAM, for example, a trivial GC
cycle takes a minimum of 40ms, virtually all of it in the sweep phase.

Similarly, on the UNIX port with 1 GB of heap, a trivial GC takes 47 ms,
again virtually all of it in the sweep phase.

This commit speeds up the sweep phase in the case most of the heap is empty
by keeping track of the ID of the highest block we allocated in an area
since the last GC.

The performance benchmark run on PYBV10 shows between +0 and +2%
improvement across the existing performance tests.  These tests don't
really stress the GC, so they were also run with gc.threshold(30000) and
gc.threshold(10000).  For the 30000 case, performance improved by up to
+10% with this commit.  For the 10000 case, performance improved by at
least +10% on 6 tests, and up to +25%.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-04 17:25:16 +10:00
Jim Mussared
975a687447 py/mpconfig: Add MICROPY_PY_PLATFORM, enabled at extra features level.
Previously this was explicitly enabled on esp32/stm32/renesas/mimxrt/samd,
but didn't get a default feature level because it wasn't in py/mpconfig.h.

With this commit it's now enabled at the "extra features" level, which adds
rp2, unix-standard, windows, esp8266, webassembly, and some nrf boards.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-24 23:53:20 +10:00
Damien George
8ef5622b9b py/runtime: Always initialise sched_state in mp_init.
When MICROPY_SCHEDULER_STATIC_NODES is enabled, the logic is unchanged.

When MICROPY_SCHEDULER_STATIC_NODES is disable, sched_state is now always
initialised to MP_SCHED_IDLE when calling mp_init().  For example, the use
of mp_sched_vm_abort(), if it aborts a running scheduled function, can lead
to the scheduler starting off in a locked state when the runtime is
restarted, and then it stays locked.  This commit fixes that case by
resetting sched_state.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-24 15:04:27 +10:00
Jim Mussared
3533924c36 extmod/moddeflate: Add deflate module providing the DeflateIO class.
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.

This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.

Enable/disable this on all ports/boards that zlib was previously configured
for.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:32:40 +10:00
Jim Mussared
198311c780 py/stream: Add mp_stream___exit___obj that calls mp_stream_close.
There are enough places that implement __exit__ by forwarding directly to
mp_stream_close that this saves code size.

For the cases where __exit__ is a no-op, additionally make their
MP_STREAM_CLOSE ioctl handled as a no-op.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:49:03 +10:00
Jim Mussared
add1200343 all: Remove the zlib module.
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.

binascii.crc32 is temporarily disabled.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:48:29 +10:00
Jim Mussared
671b35ceae py/builtinimport: Fix built-in imports when external import is disabled.
Follow-up to 24c02c4eb5 for when
MICROPY_ENABLE_EXTERNAL_IMPORT=0.  It now needs to try both extensible and
non-extensible modules.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-13 14:56:33 +10:00
Damien George
606ec9bfb1 py/compile: Fix async for's stack handling of iterator expression.
Prior to this fix, async for assumed the iterator expression was a simple
identifier, and used that identifier as a local to store the intermediate
iterator object.  This is incorrect behaviour.

This commit fixes the issue by keeping the iterator object on the stack as
an anonymous local variable.

Fixes issue #11511.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-13 13:50:50 +10:00
Jim Mussared
2fbc08c462 extmod/asyncio: Rename uasyncio to asyncio.
The asyncio module now has much better CPython compatibility and
deserves to be just called "asyncio".

This will avoid people having to write `from uasyncio import asyncio`.

Renames all files, and updates port manifests to use the new path. Also
renames the built-in _uasyncio to _asyncio.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-19 17:33:03 +10:00
Damien George
bf9ca0bb33 py/mkrules.mk: Allow $(AFLAGS) to set flags to $(AS).
Signed-off-by: Damien George <damien@micropython.org>
2023-06-15 11:02:15 +10:00
Jared Hancock
b3cd41dd4b py/lexer: Allow conversion specifiers in f-strings (e.g. !r).
PEP-498 allows for conversion specifiers like !r and !s to convert the
expression declared in braces to be passed through repr() and str()
respectively.

This updates the logic that detects the end of the expression to also stop
when it sees "![rs]" that is either at the end of the f-string or before
the ":" indicating the start of the format specifier. The "![rs]" is now
retained in the format string, whereas previously it stayed on the end
of the expression leading to a syntax error.

Previously: `f"{x!y:z}"` --> `"{:z}".format(x!y)`
Now: `f"{x!y:z}"` --> `"{!y:z}".format(x)`

Note that "!a" is not supported by `str.format` as MicroPython has no
`ascii()`, but now this will raise the correct error.

Updated cpydiff and added tests.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-14 19:11:04 +10:00
Damien George
5ce1a03a78 py/makemoduledefs.py: Automatically declare delegation attr functions.
So that the delegation functions don't need to be put somewhere global,
like in mpconfigport.h.  That would otherwise make it hard for extension
modules to use delegation.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-14 19:03:46 +10:00
Damien George
44295c9daa py/makemoduledefs.py: Fix declaring multiple module delegations.
Signed-off-by: Damien George <damien@micropython.org>
2023-06-14 19:03:29 +10:00
David Lechner
b02a5fa10a py/nlraarch64: Fix dangerous use of input register.
Starting with 2757acf6, the `top` variable in `nlr_jump()` in
`nlraarch64.c` was assigned to register `x19` by the compiler.  However,
the assembly code writes over that register with

    ldp x19, x20, [%0,  #32]

since `%0` is now `x19`. This causes the next line

    ldp lr,  x9,  [%0,  #16]

to load the wrong values.

To fix the issue, we move the value of the `top` variable from an unknown
register to a known register at the beginning of the asm code then only use
known/hard-coded registers after that.

Fixes issue #11754.

Signed-off-by: David Lechner <david@pybricks.com>
2023-06-14 17:43:44 +10:00
David Lechner
8cf9898dd3 py/parsenum: Fix typo in #endif comment.
This fixes a `#endif` comment to exactly match the `#if`.

Signed-off-by: David Lechner <david@pybricks.com>
2023-06-14 17:32:01 +10:00
Damien George
f01d5fb657 py/mkrules.mk: Automatically configure frozen options when manifest set.
Following how mkrules.cmake works.  This makes it easy for a port to enable
frozen code, by defining FROZEN_MANIFEST in its Makefile.

Signed-off-by: Damien George <damien@micropython.org>
2023-06-08 23:12:56 +10:00
Jim Mussared
a1fbb1980c extmod/modtimeq: Remove timeq module.
This is a MicroPython-specific module that existed to support the old
version of uasyncio.  It's undocumented and not enabled on all ports and
takes up code size unnecessarily.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:28 +10:00
Jim Mussared
5e50975a6d py/modsys: Allow sys.path to be assigned to.
Previously sys.path could be modified by append/pop or slice assignment.

This allows `sys.path = [...]`, which can be simpler in many cases, but
also improves CPython compatibility.

It also allows sys.path to be set to a tuple which means that you can
clear sys.path (e.g. temporarily) with no allocations.

This also makes sys.path (and sys.argv for consistency) able to be disabled
via mpconfig. The unix port (and upytesthelper) require them, so they
explicitly verify that they're enabled.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
7d2ee8aed0 py/mpconfig: Enable module delegation if sys needs it.
Otherwise you can get into the confusing state where e.g. sys.ps1 is
enabled in config (via `MICROPY_PY_SYS_PS1_PS2`) but still doesn't actually
get enabled.

Also verify that the required delegation options are enabled in modsys.c.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
e6926d6021 py/objmodule: Workaround for MSVC with no module delegation.
When compiling mpy-cross, there is no `sys` module, and so there will
be no entries in the `mp_builtin_module_delegation_table`.

MSVC doesn't like this, so instead pretend as if the feature isn't
enabled at all.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
13c817e61c py/objmodule: Add a table of built-in modules with delegation.
This replaces the previous QSTR_null entry in the globals dict which could
leak out to Python (e.g. via iteration of mod.__dict__) and could lead to
crashes.

It results in smaller code size at the expense of turning a lookup into a
loop, but the list it is looping over likely only contains one or two
elements.

To allow a module to register its custom attr function it can use the new
`MP_REGISTER_MODULE_DELEGATION` macro.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:24 +10:00
Jim Mussared
2eba98f1e0 all: Use MP_REGISTER_EXTENSIBLE_MODULE for overrideable built-ins.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:21 +10:00
Jim Mussared
24c02c4eb5 py/makemoduledefs.py: Add a way to register extensible built-in modules.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:21 +10:00
Jim Mussared
45ac651d1a all: Rename *umodule*.c to remove the "u" prefix.
Updates any includes, and references from Makefiles/CMake.

This essentially reverts what was done long ago in commit
136b5cbd76

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:17 +10:00
Jim Mussared
0ceccd4cf8 all: Rename *umodule*.h to remove the "u" prefix.
This work was funded through GitHub Sponsors.

Also updates #includes.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:14 +10:00
Jim Mussared
f5f9edf645 all: Rename UMODULE to MODULE in preprocessor/Makefile vars.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:11 +10:00
Jim Mussared
7f5d5c7271 all: Rename mod_umodule*, ^umodule* to remove the "u" prefix.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:07 +10:00
Jim Mussared
1bf2dcb15e all: Rename mp_umodule*, mp_module_umodule* to remove the "u" prefix.
This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:04 +10:00
Jim Mussared
dfe232d000 py/builtinimport: Remove weak links.
In order to keep "import umodule" working, the existing mechanism is
replaced with a simple fallback to drop the "u".

This makes importing of built-ins no longer touch the filesystem, which
makes a typical built-in import take ~0.15ms rather than 3-5ms.

(Weak links were added in c14a81662c)

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-06-08 17:54:04 +10:00