Commit Graph

2538 Commits

Author SHA1 Message Date
Damien George
414b59d39c qemu-arm: Add tests for freezing viper and asm_thumb code.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
c0f2af4e86 ports: Recompile bytecode tests now that .mpy format changed.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:29:02 +11:00
Damien George
f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
Damien George
64bfaae7ab esp32/README.md: Update list of supported IDF versions.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 15:23:50 +11:00
stijn
5f50f4a130 unix: Show compiler info in REPL banner.
The unix port's main.c gets used by unix and windows ports, and with a
variety of compilers, so it's convenient to see which version is actually
being used immediately when starting micropython.  This is similar to what
CPython does.
2022-02-22 00:59:31 +11:00
Damien George
f30b32e084 esp32/modules: Create ffat partition object with block_size=512.
Because these are formatted with a 512 sector size.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-22 00:37:55 +11:00
Damien George
106a83de22 esp32/esp32_partition: Add support for specifying block_size.
To support filesystems that use a block size different from the native
erase-page size.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-22 00:37:25 +11:00
Damien George
5935fa229c esp32/mpconfigport.h: Enable remaining features to get to "extra" level.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 16:01:18 +11:00
Damien George
c8cd5a9960 esp32/mpconfigport.h: Use the "extra" feature level.
This commit is a no-op change to simplify existing config.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 15:50:01 +11:00
Damien George
5995fb5261 stm32/mboot: Allow a board more control over entry initialisation.
If MBOOT_BOARD_ENTRY_INIT is defined by a board then that function must now
make sure system clocks are configured, eg by calling mboot_entry_init().

Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 14:59:35 +11:00
Damien George
2b62f12103 stm32/mboot: Allow a board to fully configure system clocks.
If a board wants to customise the clocks it can define the following:

    MBOOT_CLK_PLLM
    MBOOT_CLK_PLLN
    MBOOT_CLK_PLLP
    MBOOT_CLK_PLLQ
    MBOOT_CLK_PLLR (only needed on STM32H7)
    MBOOT_FLASH_LATENCY
    MBOOT_CLK_AHB_DIV
    MBOOT_CLK_APB1_DIV
    MBOOT_CLK_APB2_DIV
    MBOOT_CLK_APB3_DIV (only needed on STM32H7)
    MBOOT_CLK_APB4_DIV (only needed on STM32H7)

Signed-off-by: Damien George <damien@micropython.org>
2022-02-18 14:51:01 +11:00
Christophe Priouzeau
130f7db1fc stm32/boards/NUCLEO_WB55: Add Arduino pin alias definitions.
Fixes issue #8295.

Signed-off-by: Christophe Priouzeau <christophe.priouzeau@foss.st.com>
2022-02-18 14:45:49 +11:00
stijn
c14f47faa3 windows/Makefile: Specify CXXFLAGS in the Makefile.
Enables building user modules which use C++ code, like the unix port.
2022-02-18 14:40:36 +11:00
stijn
8bb50c6301 unix/Makefile: Remove explicit addition of -std=c++ flag.
This was added merely for building the C++ user module example, so it's a
better fit to add it in the corresponding micropython.mk.
2022-02-18 14:40:16 +11:00
David Lechner
28cb573b89 windows/appveyor: Fix printing of test failures.
In the `after_test` section, the current directory is `ports/windows` when
tests are run, so running `run-tests.py` without changing the directory or
specifying a path causes a file not found error.

This commit fixes the problem by changing the directory before calling
`run-tests.py`.

Signed-off-by: David Lechner <david@pybricks.com>
2022-02-18 14:31:52 +11:00
YoungJoon Chun
767058f328 rp2/Makefile: Add FROZEN_MANIFEST Makefile option, to override default. 2022-02-18 14:26:57 +11:00
YoungJoon Chun
d8a7bf83cc rp2/machine_uart: Fix UART RTS behaviour so RTS is deasserted.
The UART hardware flow control was not working correctly, the receive FIFO
was always fetched and RTS was never deasserted.  This is not a problem
when hardware flow control is not used: normally, if the receive FIFO is
full, the UART receiver won't receive data into the FIFO anymore, but the
current implementation fetches from the FIFO and discards it instead.
The problem is that data is discarded even when RTS is enabled.

This commit fixes the issue by only taking from the FIFO if there is room
in the ring buffer to put the character.

Signed-off-by: YoungJoon Chun <yjchun@mac.com>
2022-02-09 16:29:53 +11:00
Seon Rozenblum
aca40127bf esp32/boards: Add three UM ESP32-S3 based boards. 2022-02-09 00:13:10 +11:00
Damien George
468d1979ba esp32/machine_adc: Fix configuration of default ADC atten value.
Prior to this fix, if the ADC atten value was not explicitly given then
adc1_config_channel_atten() would never be called.

Fixes issue #8275.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-08 23:45:41 +11:00
Seon Rozenblum
ada836b834 esp32/machine_adcblock: Fix ADC bit width for ESP32-S3. 2022-02-08 12:35:47 +11:00
Damien George
6fe3856c4f esp32/partitions-16MiB.csv: Increase 14MiB filesystem to maximum size.
The original value was 14000000, it's now changed to 14 * 1024 * 1024.

Fixes issue #8266.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-08 12:25:10 +11:00
Andrew Leech
6f7d6c567f windows/uasyncio: Add support for uasyncio to windows dev variant. 2022-02-07 14:39:15 +11:00
Andrew Leech
c708262c12 windows/uselect: Enable micropython select in dev variant. 2022-02-07 14:39:15 +11:00
Andrew Leech
69c9a76786 windows/mingw: Include extmod/shared/lib sources properly. 2022-02-07 14:39:15 +11:00
Damien George
a7530cbc03 stm32/boards/NUCLEO_L432KC: Disable MICROPY_OPT_COMPUTED_GOTO.
To save space, after recent fixes to L4 ADC made it overflow flash.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-04 14:28:21 +11:00
Damien George
5679fe6aee rp2/modutime: Fix time.localtime day-of-week value.
The correct day-of-week is stored in the RTC (0=Monday, 6=Sunday) so there
is no need to adjust it for the return value of time.localtime().

Fixes issue #7889.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-04 10:48:59 +11:00
iabdalkader
31f2440388 stm32/adc: Remove obsolete FIRST/LAST GPIO channel macros. 2022-02-04 10:45:11 +11:00
iabdalkader
e5df4a96fa stm32/adc: Fix L4 ADC channel numbers.
Use HAL macro to map decimal numbers to channel numbers.  This is needed
since updating L4 HAL v1.17.0 in a0f5b3148a.

Fixes issue #8233.
2022-02-04 10:42:38 +11:00
Peter D. Gray
ae5f647a2d stm32/system_stm32: Make SystemClock_Config() a weak symbol.
This allows boards to override as needed.
2022-02-04 10:29:53 +11:00
Damien George
e306f2285b stm32/boards/NUCLEO_WL55: Add new board definition.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-04 09:43:43 +11:00
Damien George
e0a0719416 stm32: Add initial support for STM32WL MCUs.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-04 09:43:43 +11:00
Damien George
9127e63708 stm32/Makefile: Make stm32lib configurable.
The default stm32lib remains lib/stm32lib, but it can now be easily
overriden at build time by specifying STM32LIB_DIR, or STM32LIB_CMSIS_DIR
and STM32LIB_HAL_DIR.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-04 09:43:43 +11:00
Damien George
e4f59a0020 stm32/rtc: Use LL_RTC functions to simplify some MCU-specific code.
This also fixes a possible race condition when exiting initialisation mode:
reading then writing to ISR (via ISR &= ~RTC_ISR_INIT) will clear any flags
that were set by the hardware between the read and the write.  The correct
way to clear just the INIT bit is to just do a single write via ISR =
~RTC_ISR_INIT, which will not clear any other flags (they must be written
to 0 to clear), and that is exactly what LL_RTC_DisableInitMode does.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-04 08:35:14 +11:00
Damien George
26faf74d52 stm32/mboot: Add support for F469/479 MCUs in fwupdate.py.
And don't assert on the sector number in sector_erase, so it can support
erasing arbitrary sectors.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 13:21:57 +11:00
Damien George
4f918f4b26 stm32/mboot: Add MBOOT_BOARD_ENTRY_INIT for a board to add entry code.
Also change the signature of stm32_main to uint32_t, which is what it
should be.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 13:20:51 +11:00
Damien George
b8d55d4c52 stm32/mboot: Allow HSI to be used as the main clock source.
Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 13:20:22 +11:00
Damien George
fbd47fc46c ports: Consolidate inclusion of umachine module in built-ins.
The inclusion of `umachine` in the list of built-in modules is now done
centrally in py/objmodule.c.  Enabling MICROPY_PY_MACHINE will include this
module.

As part of this, all ports now have `umachine` as the core module name
(previously some had only `machine` as the name).

Signed-off-by: Damien George <damien@micropython.org>
2022-02-03 10:08:54 +11:00
Damien George
71b3ce3ace esp32: Create .uf2 binaries for S2 and S3 chips.
The name of the filesystem partition is updated to support "ffat", as used
by TinyUF2.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-02 23:47:49 +11:00
Damien George
872bab6b3c esp32: Remove unneeded modesp.h.
Made redundant by 71f4faac27

Signed-off-by: Damien George <damien@micropython.org>
2022-02-02 16:20:41 +11:00
Emil Kondayan
b18d4392b4 esp32/adc: Fix wrong mapping between ADC2 channel and GPIO number.
According to the IO_MUX table in the ESP32 datasheet, the ADC2 channels are
mapped to different GPIO numbers.
2022-02-02 15:29:02 +11:00
Michael O'Cleirigh
eae2e3516c esp32/main: Automatically size SPIRAM heap when allocated using malloc.
This change allows the same heap allocation rules to be used when using
malloc regardless if the board has SPRAM or normal RAM.

Integrating with the esp32-camera for example requires that ESP32 SPRAM be
allocatable using the esp-idf capabilities aware allocation functions.  In
the case of esp32-camera it's for the framebuffer.

Detect when CONFIG_SPIRAM_USE_MALLOC is in use and use the standard
automatic configuration of leaving 1/2 of the SPRAM available to other
FreeRTOS tasks.
2022-02-02 15:24:02 +11:00
Damien George
1f04a9a1fc esp32/esp32_rmt: Select correct last RMT channel on S2, S3, C3 variants.
For example the ESP32-C3 has 2 TX channels and 2 RX channels in total, and
in this case channel 1 must be the default for bitstream.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-01 16:58:56 +11:00
IhorNehrutsa
15e65b77eb esp32/machine_pwm: Clean up macro names and their use.
- Remove UI_RES_SHIFT macro.
- Rename PWFREQ to PWM_FREQ.
- Rename PWRES to PWM_RES_10_BIT.
- Use UI_RES_16_BIT flag instead of HIGHEST_PWM_RES.
2022-02-01 16:47:29 +11:00
IhorNehrutsa
a5e64c209f esp32/machine_pwm: Fix PWM not allowing frequencies < 611 Hz.
Fixes issue #8189.
2022-02-01 16:46:55 +11:00
Herwin Grobben
8f68e26f79 stm32: Add support for G4 MCUs, and add NUCLEO_G474RE board defn.
This commit adds support for the STM32G4 series of MCUs, and a board
definition for NUCLEO_G474RE.  This board has the REPL on LPUART1 which is
connected to the on-board ST-link USB-UART.
2022-02-01 16:21:01 +11:00
Damien George
60e05ae84e stm32/mboot: Compute and check CRC32 of dfu file in fwupdate.py.
Signed-off-by: Damien George <damien@micropython.org>
2022-01-31 18:49:08 +11:00
Damien George
ca1914fb47 stm32/mboot: Support H7 MCUs in fwupdate.py.
And optimise the speed of flash writing.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-31 18:26:55 +11:00
Damien George
29867a2439 stm32/mboot: Include hal_rcc_ex.c in source file list.
It's needed at least on F4 because this file overrides the weak function
HAL_RCC_DeInit() from hal_rcc.c.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-31 18:26:55 +11:00
Damien George
c8c229b96c stm32/mboot: Use PLL3 for USB clock source on H7 MCUs.
PLL3-Q is more reliable than PLL1-Q for the USB clock source when entering
mboot from various reset states (eg power on vs MCU reset).  (It was found
that if the main application used PLL3-Q then sometimes the USB clock
source would stay stuck on PLL3-Q and not switch to PLL1-Q after a reset.)

Other related changes:
- SystemCoreClockUpdate() should be called on H7 because the calculation
  can be involved in some cases.
- __set_PRIMASK(0) should be called because on H7 the built-in ST DFU
  bootloader exits with IRQs disabled.

Signed-off-by: Damien George <damien@micropython.org>
2022-01-31 18:26:55 +11:00
Damien George
c99ed8d6fa stm32/powerctrl: Write bootloader-state as 64-bit word to work on H7.
H7 MCUs have ECC and writes do not go through to SRAM until 64-bits have
been written (on another location is written).  So use 64-bit writes for
the bootloader-state variable so it is committed before the system reset.

As part of this change, the lower byte of the bootloader address in
BL_STATE must now be the magic number 0x5a5 for the state to be valid
(previously this was 0x000 which is not as robust).

Signed-off-by: Damien George <damien@micropython.org>
2022-01-31 18:26:55 +11:00