Commit Graph

13154 Commits

Author SHA1 Message Date
Damien George
c4b8dae438 tests/unix: Add coverage test for freezing various objects.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-28 19:02:58 +11:00
robert-hh
5c46721a1c tools/mpy-tool.py: Fix frozen comment generation to escape chars.
That caused the compile of frozen_content.c to fail if characters like
backslash were in a short string.  Thanks to @hippy for identifying the
spot to change.
2022-02-28 18:47:24 +11:00
Damien George
0a217624e1 tools/upip.py: Remove unused op_basename() function.
It seems this was never used, at least not since its inclusion in this
repository.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-25 18:24:03 +11:00
Damien George
546e213265 esp32/boards: Add specific deploy instructions for S3 variant.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-25 16:52:58 +11:00
stijn
c2e8a5acd2 windows: Unify project file headers.
The xmlns attribute is required for older msbuild version (e.g. for
VS2015).  Add it where needed, and reorder the attributes so all
files look the same.
2022-02-25 16:41:11 +11:00
Damien George
d6564a3159 tools/mpremote: Add "umount" command.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-25 13:10:45 +11:00
Damien George
ad1f523e7e tools/mpremote: Add "resume" and "soft-reset" commands.
This makes the auto soft-reset behaviour of mpremote more logical, and now
configurable via these new commands.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-25 13:09:28 +11:00
Damien George
414b59d39c qemu-arm: Add tests for freezing viper and asm_thumb code.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
0a2895b099 tests/perf_bench: Skip bm_chaos test if random.randrange is unavailable.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
66fc0f45c1 tools/ci.sh: Run urandom test scripts as part of native module tests.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
73a1927fce github/workflows: Add new workflow to test .mpy file format and tools.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
c0f2af4e86 ports: Recompile bytecode tests now that .mpy format changed.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
Damien George
64bfaae7ab esp32/README.md: Update list of supported IDF versions.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 15:23:50 +11:00
Damien George
b1afbe3336 tools/ci.sh: Update IDF v4.4 build to use v4.4 tag.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 15:23:32 +11:00
stijn
5f50f4a130 unix: Show compiler info in REPL banner.
The unix port's main.c gets used by unix and windows ports, and with a
variety of compilers, so it's convenient to see which version is actually
being used immediately when starting micropython.  This is similar to what
CPython does.
2022-02-22 00:59:31 +11:00
stijn
49934fcf8b extmod/moduplatform: Move platform PP definitions into a header.
These are more generally useful than just for the module so make them
globally available, prefixed consistently with MICROPY_PLATFORM_.
2022-02-22 00:59:14 +11:00
Damien George
f30b32e084 esp32/modules: Create ffat partition object with block_size=512.
Because these are formatted with a 512 sector size.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-22 00:37:55 +11:00
Damien George
106a83de22 esp32/esp32_partition: Add support for specifying block_size.
To support filesystems that use a block size different from the native
erase-page size.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-22 00:37:25 +11:00
Damien George
5935fa229c esp32/mpconfigport.h: Enable remaining features to get to "extra" level.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 16:01:18 +11:00
Damien George
c8cd5a9960 esp32/mpconfigport.h: Use the "extra" feature level.
This commit is a no-op change to simplify existing config.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 15:50:01 +11:00
stijn
ff9c708507 tests/run-tests.py: Skip repl tests when running windows underneath.
Some versions of Python (for instance: the mingw-w64 version which can be
installed on MSYS2) do include a pty module and claim to be posix-like
(os.name == 'posix'), yet the select.select call used in run-tests.py hangs
forever.  To be on the safe side just exclude anything which might be
running on windows.
2022-02-18 15:14:47 +11:00
Damien George
5995fb5261 stm32/mboot: Allow a board more control over entry initialisation.
If MBOOT_BOARD_ENTRY_INIT is defined by a board then that function must now
make sure system clocks are configured, eg by calling mboot_entry_init().

Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 14:59:35 +11:00
Damien George
2b62f12103 stm32/mboot: Allow a board to fully configure system clocks.
If a board wants to customise the clocks it can define the following:

    MBOOT_CLK_PLLM
    MBOOT_CLK_PLLN
    MBOOT_CLK_PLLP
    MBOOT_CLK_PLLQ
    MBOOT_CLK_PLLR (only needed on STM32H7)
    MBOOT_FLASH_LATENCY
    MBOOT_CLK_AHB_DIV
    MBOOT_CLK_APB1_DIV
    MBOOT_CLK_APB2_DIV
    MBOOT_CLK_APB3_DIV (only needed on STM32H7)
    MBOOT_CLK_APB4_DIV (only needed on STM32H7)

Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 14:51:01 +11:00
Christophe Priouzeau
130f7db1fc stm32/boards/NUCLEO_WB55: Add Arduino pin alias definitions.
Fixes issue #8295.

Signed-off-by: Christophe Priouzeau <christophe.priouzeau@foss.st.com>
2022-02-18 14:45:49 +11:00
stijn
c14f47faa3 windows/Makefile: Specify CXXFLAGS in the Makefile.
Enables building user modules which use C++ code, like the unix port.
2022-02-18 14:40:36 +11:00
stijn
8bb50c6301 unix/Makefile: Remove explicit addition of -std=c++ flag.
This was added merely for building the C++ user module example, so it's a
better fit to add it in the corresponding micropython.mk.
2022-02-18 14:40:16 +11:00
robert-hh
9c05f3aa1d drivers/sdcard: Allow setting the final SPI baudrate.
This baudrate is supplied in the constructor. The default is 1320000 as
before.

Example:

    sd = sdcard.SDCard(spi, cs, baudrate=20_000_000)
2022-02-18 14:37:44 +11:00
iabdalkader
465b74e78d drivers/ninaw10: Add NIC-level ioctl function.
This commit adds support in the driver for irregular commands.  It
currently supports setting GPIO pin mode, and GPIO pin read/write value.
2022-02-18 14:35:26 +11:00
David Lechner
5d6408f8f9 tools/verifygitlog.py: Ignore line length in body if it's a URL.
This changes the git commit message line length check to ignore lines that
contain URLs, since these cannot be wrapped without breaking tools that
detect URLs and create a link.

Signed-off-by: David Lechner <david@pybricks.com>
2022-02-18 14:33:33 +11:00
David Lechner
28cb573b89 windows/appveyor: Fix printing of test failures.
In the `after_test` section, the current directory is `ports/windows` when
tests are run, so running `run-tests.py` without changing the directory or
specifying a path causes a file not found error.

This commit fixes the problem by changing the directory before calling
`run-tests.py`.

Signed-off-by: David Lechner <david@pybricks.com>
2022-02-18 14:31:52 +11:00
Bradley Wogsland
0dfd0447fa README: Update link for ARM embedded toolchain to developer.arm.com. 2022-02-18 14:28:29 +11:00
YoungJoon Chun
767058f328 rp2/Makefile: Add FROZEN_MANIFEST Makefile option, to override default. 2022-02-18 14:26:57 +11:00
Damien George
18acd0318f py/gc: Update debug code to compile with changes to qstr pool types.
Following on from 18b1ba086c and
f46a7140f5.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-17 11:17:21 +11:00
Damien George
2ea21abae0 tests/extmod/vfs_fat_finaliser.py: Make finalisation more robust.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-12 09:45:32 +11:00
Artyom Skrobov
f46a7140f5 py/qstr: Use const consistently to avoid a cast.
Originally at adafruit#4707

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2022-02-11 22:55:02 +11:00
Artyom Skrobov
18b1ba086c py/qstr: Separate hash and len from string data.
This allows the compiler to merge strings: e.g. "update",
"difference_update" and "symmetric_difference_update" will all point to the
same memory.

No functional change.

The size reduction depends on the number of qstrs in the build.  The change
this commit brings is:

   bare-arm:    -4 -0.007%
minimal x86:  +150 +0.092% [incl +48(data)]
   unix x64:  -608 -0.118%
unix nanbox:  -572 -0.126% [incl +32(data)]
      stm32: -1392 -0.352% PYBV10
     cc3200:  -448 -0.244%
    esp8266: -1208 -0.173% GENERIC
      esp32: -1028 -0.068% GENERIC[incl -1020(data)]
        nrf:  -440 -0.252% pca10040
        rp2: -1072 -0.217% PICO
       samd:  -368 -0.264% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Performance is also improved (on bare metal at least) for the
core_import_mpy_multi.py, core_import_mpy_single.py and core_qstr.py
performance benchmarks.

Originally at adafruit#4583

Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
2022-02-11 22:52:32 +11:00
Damien George
e8bc4a3a5b tests/run-perfbench.py: Use SKIP consistently, and increase print width.
A script will print "SKIP" if it wants to be skipped, so the test runner
must also use uppercase SKIP.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-11 22:19:38 +11:00
Damien George
a434705700 tests/perf_bench: Add perf test for yield-from execution.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-11 13:42:00 +11:00
Damien George
75da124cf8 tests/perf_bench: Add perf tests for qstr interning and importing .mpy.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-10 15:25:33 +11:00
Damien George
b33fdbe535 tests/run-perfbench.py: Allow a test to SKIP, and to have a .exp file.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-10 15:25:33 +11:00
YoungJoon Chun
d8a7bf83cc rp2/machine_uart: Fix UART RTS behaviour so RTS is deasserted.
The UART hardware flow control was not working correctly, the receive FIFO
was always fetched and RTS was never deasserted.  This is not a problem
when hardware flow control is not used: normally, if the receive FIFO is
full, the UART receiver won't receive data into the FIFO anymore, but the
current implementation fetches from the FIFO and discards it instead.
The problem is that data is discarded even when RTS is enabled.

This commit fixes the issue by only taking from the FIFO if there is room
in the ring buffer to put the character.

Signed-off-by: YoungJoon Chun <yjchun@mac.com>
2022-02-09 16:29:53 +11:00
Lars Kellogg-Stedman
8f6924c9fb docs/library/uasyncio.rst: Fix description of ThreadSafeFlag.wait.
When a task waits on a ThreadSafeFlag (and the wait method returns), the
flag is immediately reset.  This was not clear in the documentation, which
appeared to copy the description of the wait method from the Event class.

Signed-off-by: Lars Kellogg-Stedman <lars@oddbit.com>
2022-02-09 16:09:48 +11:00
Jos Verlinde
6653856b87 docs/library/machine.Pin.rst: Document defaults for Pin.init. 2022-02-09 15:55:10 +11:00
Jos Verlinde
7f67524031 docs/library/pyb.SPI.rst: Document default for prescaler argument.
To prevent "non-default argument follows default argument" errors.
2022-02-09 15:06:34 +11:00
Jos Verlinde
58cd2a8b0a docs/library/esp.rst: Document the osdebug function. 2022-02-09 15:06:19 +11:00
Jos Verlinde
f99b679916 docs/library/machine.WDT.rst: Use correct case for WDT.feed. 2022-02-09 15:05:49 +11:00
Jos Verlinde
7621b17636 docs/library/machine.SPI.rst: Add class constant SoftSPI.MSB and .LSB. 2022-02-09 15:05:38 +11:00
Jos Verlinde
8715ad9ccf docs/library/pyb.DAC.rst: Add DAC class constants. 2022-02-09 15:05:26 +11:00
Jos Verlinde
ba4f6f5fdf docs/library/pyb.rst: Add pyb.hid_mouse and pyb.hid_keyboard constants. 2022-02-09 15:04:57 +11:00