Commit Graph

14999 Commits

Author SHA1 Message Date
Jim Mussared
4bbe879c26 tools/mpremote: Make soft-reset count as an action.
Otherwise `mpremote soft-reset` will implicitly run the repl command.

Fixes issue #10871.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-10 14:11:53 +10:00
Jos Verlinde
232c29c2ed tools/codeformat.py: Use pyproject.toml for black config.
Signed-off-by: Jos Verlinde <jos_verlinde@hotmail.com>
2023-08-10 11:54:36 +10:00
Daniël van de Giessen
8dbb29da3f tools/codeformat.py: Skip formatting ESP-IDF managed components.
Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2023-08-10 11:36:24 +10:00
Angus Gratton
597fcb4751 tools/mpy-tool.py: Use isinstance() for type checking.
Ruff version 283 expanded E721 to fail when making direct comparison
against a built-in type.  Change the code to use isinstance() as
suggested, these usages appear to have equivalent functionality.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2023-08-09 17:41:54 +10:00
Damien George
d529c20674 extmod/modssl_mbedtls: Fix ioctl of a socket in closed/error state.
Signed-off-by: Damien George <damien@micropython.org>
2023-08-09 13:22:57 +10:00
Damien George
20d3a6b196 extmod/modssl_mbedtls: Reject ioctls that are not supported.
An SSL stream can only handle CLOSE and POLL ioctls.  Other ones do not
make sense, or at least it doesn't make sense to pass the ioctl request
directly down to the underlying stream.

In particular MP_STREAM_GET_FILENO should not be passed to the underlying
stream because the SSL stream is not directly related to a file descriptor,
and the SSL stream must handle the polling itself.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-09 13:22:32 +10:00
Damien George
218242d1de tests/extmod: Skip select/socket tests if they can't create UDP socket.
Some targets (eg PYBV10) have the socket module but are unable to create
UDP sockets without a registered NIC.  So skip UDP tests on these targets.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:39:29 +10:00
Damien George
6b78a1bf00 tests/extmod: Add coverage tests for select module.
Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:37:48 +10:00
Damien George
3f417e8943 extmod/modselect: Remove undocumented support for flags arg to poll.
The signature of this method was poller.poll(timeout=-1, flags=0, /) but
the flags argument was not documented and is not CPython compatible.  So
it's removed in this commit.

(The optional flags remains for the ipoll() method, which is documented.)

Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:37:47 +10:00
Damien George
ef71028f77 extmod/modselect: Add optimisation to use system poll when possible.
A previous commit removed the unix-specific select module implementation
and made unix use the common one.

This commit adds an optimisation so that the system poll function is used
when polling objects that have a file descriptor.  With this optimisation
enabled, if code registers both file-descriptor-based objects, and non-
file-descriptor-based objects with select.poll() then the following occurs:

- the system poll is called for all file-descriptor-based objects with a
  timeout of 1ms

- then the bare-metal polling implementation is used for remaining objects,
  which calls into their ioctl method (which can be in C or Python)

In the case where all objects have file descriptors, the system poll is
called with the full timeout requested by the caller.  That makes it as
efficient as possible in the case everything has a file descriptor.

Benefits of this approach:

- all ports use the same select module implementation

- the unix port now supports polling of all objects and matches bare metal
  implementations

- it's still efficient for existing cases where only files and sockets are
  polled (on unix)

- the bare metal implementation does not change

- polling of SSL objects will now work on unix by calling in to the ioctl
  method on SSL objects (this is required for asyncio ssl support)

Note that extmod/vfs_posix_file.c has poll disable when the optimisation is
enabled, because the code is not reachable when the optimisation is used.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:11:40 +10:00
Damien George
ebc6556346 extmod/modselect: Factor low-level polling code into common function.
Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:11:40 +10:00
Damien George
7f2efb4144 extmod/modselect: Abstract out a poll_set_t struct and functions.
To make it easier to extend and modify this polling implementation.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:11:40 +10:00
Damien George
df08c38c28 unix/modselect: Remove unix-specific implementation of select module.
The unix port has a custom select module which only works with objects that
have a file descriptor, eg files and sockets.  On the other hand, bare
metal ports use the common extmod/modselect.c implementation of the select
module that supports polling of arbitrary objects, as long as those objects
provide a MP_STREAM_POLL in their ioctl implementation (which can be done
in C or Python).

This commit removes the unix-specific code and makes unix use the common
one provided by extmod/modselect.c instead.  All objects with file
descriptors implement MP_STREAM_POLL so they continue to work.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-07 12:09:38 +10:00
Damien George
22106bf2fa extmod/vfs_posix_file: Add poll support for missing ERR,HUP,NVAL values.
Signed-off-by: Damien George <damien@micropython.org>
2023-08-06 11:54:06 +10:00
Damien George
6a179019e8 unix/modsocket: Add poll support for missing ERR,HUP,NVAL poll values.
Signed-off-by: Damien George <damien@micropython.org>
2023-08-06 11:54:06 +10:00
Elecia White
b714f41812 docs/develop/gettingstarted: Update ARM package list.
Signed-off-by: Elecia White <elecia@logicalelegance.com>
2023-08-05 20:05:50 +10:00
Rene Straub
7fad499d1e docs/develop/gettingstarted: Clarify submodule initialization.
When building for a specific board this must be specified in make
submodules.  I.e. make BOARD=STM32F769DISC submodules.

Signed-off-by: Rene Straub <rene@see5.ch>
2023-08-04 21:45:31 +10:00
Damien Tournoud
2dcd745434 py/gc: Speed up incremental GC cycles by tracking the last used block.
In applications that use little memory and run GC regularly, the cost of
the sweep phase quickly becomes prohibitives as the amount of RAM
increases.

On an ESP32-S3 with 2 MB of external SPIRAM, for example, a trivial GC
cycle takes a minimum of 40ms, virtually all of it in the sweep phase.

Similarly, on the UNIX port with 1 GB of heap, a trivial GC takes 47 ms,
again virtually all of it in the sweep phase.

This commit speeds up the sweep phase in the case most of the heap is empty
by keeping track of the ID of the highest block we allocated in an area
since the last GC.

The performance benchmark run on PYBV10 shows between +0 and +2%
improvement across the existing performance tests.  These tests don't
really stress the GC, so they were also run with gc.threshold(30000) and
gc.threshold(10000).  For the 30000 case, performance improved by up to
+10% with this commit.  For the 10000 case, performance improved by at
least +10% on 6 tests, and up to +25%.

Signed-off-by: Damien George <damien@micropython.org>
2023-08-04 17:25:16 +10:00
Jim Mussared
70c564324c extmod/modssl_mbedtls: Reference SSLContext from SSLSocket.
Prevent the GC cleaning up (and finalising) the SSLContext while the
socket is still live.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-08-01 15:12:33 +10:00
Brett Cannon
01c758e26a unix/README: Fix Markdown link markup.
Signed-off-by: Brett Cannon <brett@python.org>
2023-07-27 21:40:41 +10:00
Damien George
d14ddcbdb5 tools/autobuild: Add support for application .bin files for esp32.
On esp32, the build output consists of:
- micropython.elf
- micropython.map
- micropython.bin -- application only
- micropython.uf2 -- application only
- firmware.bin -- bootloader, partition table and application

Currently everything is available at the download page except
micropython.bin.  This commit adds that file but with the extension changed
to .app-bin, to distinguish it from .bin (the full thing).

Signed-off-by: Damien George <damien@micropython.org>
2023-07-27 13:10:19 +10:00
Damien George
cfcce4b531 esp32/README: Specify that only IDF v5.0.2 is supported.
Signed-off-by: Damien George <damien@micropython.org>
2023-07-25 12:59:02 +10:00
Damien George
c9d2c5537b esp32/boards: Remove references to the IDF version in board.md files.
Listing the IDF version number in the board description is not as important
as it once was, when the IDF was still undergoing a lot of changes.  Now,
all builds use IDF 5.x and it's possible to query the exact version with
platform.platform().

Signed-off-by: Damien George <damien@micropython.org>
2023-07-25 12:35:21 +10:00
Damien George
b2adfc8077 esp32/machine_hw_spi: Check for valid SPI id in constructor, not init.
Otherwise constructing an invalid SPI instance (eg machine.SPI(3)) will
mess up machine.SPI(2)'s state before it's detected that it's an invalid
SPI id.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-25 11:33:51 +10:00
Damien George
162dd022b1 esp32/machine_hw_spi: Remove SPI host renaming for C3 and S3 variants.
On ESP32C3 it's not doing anything.  On ESP32S3 the original code prevented
prevented machine.SPI(1) from working.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-25 11:13:25 +10:00
Damien George
862944a71f esp32/machine_hw_spi: Remove unnecessary duplicate SPI pin defaults.
Signed-off-by: Damien George <damien@micropython.org>
2023-07-25 10:54:41 +10:00
Damien George
aab8061dce esp32/machine_hw_spi: Fix access of SPI(2).
SPI3_HOST is not a macro but rather an enum, so use SOC_SPI_PERIPH_NUM to
detect if it's defined.

Fixes issue #11919.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-25 10:54:29 +10:00
Jim Mussared
975a687447 py/mpconfig: Add MICROPY_PY_PLATFORM, enabled at extra features level.
Previously this was explicitly enabled on esp32/stm32/renesas/mimxrt/samd,
but didn't get a default feature level because it wasn't in py/mpconfig.h.

With this commit it's now enabled at the "extra features" level, which adds
rp2, unix-standard, windows, esp8266, webassembly, and some nrf boards.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-24 23:53:20 +10:00
Armin Brauns
14374850ce mpy-cross: Allow specifying stdin as input without --.
This way, a bare `-` is never interpreted as an option, even before
`--`. Filenames starting with `-` still need to be put after `--`.

Signed-off-by: Armin Brauns <armin.brauns@embedded-solutions.at>
2023-07-24 23:41:50 +10:00
Armin Brauns
3164749b3d mpy-cross: When reading from stdin, write output to stdout.
Unless -o is given, output defaults to stdout unless a source file is
given (in which case the source file name is used to derive an output
file name).

Signed-off-by: Armin Brauns <armin.brauns@embedded-solutions.at>
2023-07-24 23:41:50 +10:00
Armin Brauns
6a61e4ecd1 mpy-cross: Allow reading from stdin and writing to stdout.
Signed-off-by: Armin Brauns <armin.brauns@embedded-solutions.at>
2023-07-24 23:41:50 +10:00
Armin Brauns
625e03d2dc mpy-cross: Allow specifying source files starting with -.
Signed-off-by: Armin Brauns <armin.brauns@embedded-solutions.at>
2023-07-24 23:32:11 +10:00
Daniël van de Giessen
52dc48b2a0 esp32/machine_wdt: Allow feeding WDT from threads.
This changes the ESP32 WDT implementation to use a custom handle so that it
becomes possible to reset the WDT from a thread.

By default esp_task_wdt_add subscribes the task_id of the current task.
That means that if we're running in a different task we are unable to reset
the WDT, which prevents feeding the WDT from a thread directly, or even
from a timer (which may randomly run in a different task when there's
multiple threads).

As an added bonus, the name we set makes the error clearly specify that it
was the user-specified WDT that reset the chip.

Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2023-07-24 23:29:53 +10:00
Daniël van de Giessen
1bde5f3316 esp32/main: Remove unused mbedtls debug function.
Since commit beeb74 we already check in modussl_mbedtls whether this
function is provided by the ESP-IDF before calling it, thus we no longer
need to define it here in order to compile.

Removing it so that if CONFIG_MBEDTLS_DEBUG is defined we do not cause any
'multiple definition' compile errors.

Signed-off-by: Daniël van de Giessen <daniel@dvdgiessen.nl>
2023-07-24 23:28:16 +10:00
iabdalkader
c1acea0e73 esp32/boards/ARDUINO_NANO_ESP32: Fix deploy instructions.
Signed-off-by: iabdalkader <i.abdalkader@gmail.com>
2023-07-24 17:45:19 +10:00
Damien George
8ef5622b9b py/runtime: Always initialise sched_state in mp_init.
When MICROPY_SCHEDULER_STATIC_NODES is enabled, the logic is unchanged.

When MICROPY_SCHEDULER_STATIC_NODES is disable, sched_state is now always
initialised to MP_SCHED_IDLE when calling mp_init().  For example, the use
of mp_sched_vm_abort(), if it aborts a running scheduled function, can lead
to the scheduler starting off in a locked state when the runtime is
restarted, and then it stays locked.  This commit fixes that case by
resetting sched_state.

Signed-off-by: Damien George <damien@micropython.org>
2023-07-24 15:04:27 +10:00
Jim Mussared
ea1a5e43d0 examples/natmod/deflate: Add deflate as a dynamic native module.
This replaces the previous zlib version.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:32:42 +10:00
Jim Mussared
b804443cb3 docs/library/deflate: Add docs for deflate.DeflateIO.
Also update zlib & gzip docs to describe the micropython-lib modules.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:32:42 +10:00
Jim Mussared
8b315ef0d8 tests/extmod: Add deflate.DeflateIO tests.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:32:42 +10:00
Jim Mussared
3533924c36 extmod/moddeflate: Add deflate module providing the DeflateIO class.
This provides similar functionality to the former zlib.DecompIO and
especially CPython's gzip.GzipFile for both compression and decompression.

This class can be used directly, and also can be used from Python to
implement (via io.BytesIO) zlib.decompress and zlib.compress, as well as
gzip.GzipFile.

Enable/disable this on all ports/boards that zlib was previously configured
for.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:32:40 +10:00
Jim Mussared
e6c290c3d1 lib/uzlib: Add a source_read_data var to pass to source_read_cb.
For better abstraction for users of this API.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:29:34 +10:00
Jim Mussared
7f16bfca9f lib/uzlib/defl_static: Optimize zlib_start/finish_block.
Collapsing the two adjacent calls to outbits saves 32 bytes.

Bringing defl_static.c into lz77.c allows better inlining, saves 24 bytes.

Merge the Outbuf/uzlib_lz77_state_t structs, a minor simplification that
doesn't change code size.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:29:34 +10:00
Jim Mussared
ef5061fefd lib/uzlib/tinflate: Implement more compact lookup tables.
Saves 68 bytes on PYBV11.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:29:34 +10:00
Jim Mussared
d75a3cd861 lib/uzlib: Combine zlib/gzip header parsing to allow auto-detect.
This supports `wbits` values between +40 to +47.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:29:34 +10:00
Jim Mussared
c2b8e6e5d6 lib/uzlib: Clean up tinf -> uzlib rename.
This library used a mix of "tinf" and "uzlib" to refer to itself.  Remove
all use of "tinf" in the public API.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 19:29:24 +10:00
Jim Mussared
0900976384 lib/uzlib/defl_static: Implement some code size improvements.
This commit makes the following changes:
- Replace 256-byte reverse-bits-in-byte lookup table with computation.
- Replace length and distance code lookup tables with computation.
- Remove comp_disabled check (it's unused).
- Make the dest_write_cb take the data pointer directly, rather than the
  Outbuf.

Saves 500 bytes on PYBV11.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:58:33 +10:00
Jim Mussared
82db9926ed lib/uzlib/lz77: Always use separate history buffer.
Because we only use the streaming source, this is just extra code size.

Saves 64 bytes on PYBV11.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:57:49 +10:00
Damien George
c4feb806e0 lib/uzlib: Add memory-efficient, streaming LZ77 compression support.
The compression algorithm implemented in this commit uses much less memory
compared to the standard way of implementing it using a hash table and
large look-back window.  In particular the algorithm here doesn't allocate
hash table to store indices into the history of the previously seen text.
Instead it simply does a brute-force-search of the history text to find a
match for the compressor.  This is slower (linear search vs hash table
lookup) but with a small enough history (eg 512 bytes) it's not that slow.
And a small history does not impact the compression too much.

To give some more concrete numbers comparing memory use between the
approaches:

- Standard approach: inplace compression, all text to compress must be in
  RAM (or at least memory addressable), and then an additional 16k bytes
  RAM of hash table pointers, pointing into the text

- The approach in this commit: streaming compression, only a limited amount
  of previous text must be in RAM (user selectable, defaults to 512 bytes).

To compress, say, 1k of data, the standard approach requires all that data
to be in RAM, plus an additional 16k of RAM for the hash table pointers.
With this commit, you only need the 1k of data in RAM.  Or if it's
streaming from a file (or elsewhere), you could get away with only 256
bytes of RAM for the sliding history and still get very decent compression.

In summary: because compression takes such a large amount of RAM (in the
standard algorithm) and it's not really suitable for microcontrollers, the
approach taken in this commit is to minimise RAM usage as much as possible,
and still have acceptable performance (speed and compression ratio).

Signed-off-by: Damien George <damien@micropython.org>
2023-07-21 18:54:22 +10:00
Jim Mussared
198311c780 py/stream: Add mp_stream___exit___obj that calls mp_stream_close.
There are enough places that implement __exit__ by forwarding directly to
mp_stream_close that this saves code size.

For the cases where __exit__ is a no-op, additionally make their
MP_STREAM_CLOSE ioctl handled as a no-op.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:49:03 +10:00
Jim Mussared
add1200343 all: Remove the zlib module.
This will be replaced with a new deflate module providing the same
functionality, with an optional frozen Python wrapper providing a
replacement zlib module.

binascii.crc32 is temporarily disabled.

This work was funded through GitHub Sponsors.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2023-07-21 18:48:29 +10:00