Commit Graph

19 Commits

Author SHA1 Message Date
Angus Gratton
2e6e53057b shared/tinyusb: Further refactor static USB device implementation.
App the mp_ prefix to usbd_ symbols and files which are defined here and
not in TinyUSB.

rp2 only for now. This includes some groundwork for dynamic USB devices
(defined in Python).

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
2022-11-11 16:47:36 +11:00
Simon Arlott
43dd3ea74d shared/runtime/gchelper_native: Fix pointer cast on x86_64.
gc_helper_collect_regs_and_stack() is casting a pointer to uint32_t; the
variables involved are always pointers so it should be using uintptr_t.
2022-11-08 19:15:37 +11:00
Jim Mussared
94beeabd2e py/obj: Convert make_new into a mp_obj_type_t slot.
Instead of being an explicit field, it's now a slot like all the other
methods.

This is a marginal code size improvement because most types have a make_new
(100/138 on PYBV11), however it improves consistency in how types are
declared, removing the special case for make_new.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:15 +10:00
Jim Mussared
6da41b5900 py/obj: Merge getiter and iternext mp_obj_type_t slots.
The goal here is to remove a slot (making way to turn make_new into a slot)
as well as reduce code size by the ~40 references to mp_identity_getiter
and mp_stream_unbuffered_iter.

This introduces two new type flags:
- MP_TYPE_FLAG_ITER_IS_ITERNEXT: This means that the "iter" slot in the
  type is "iternext", and should use the identity getiter.
- MP_TYPE_FLAG_ITER_IS_CUSTOM: This means that the "iter" slot is a pointer
  to a mp_getiter_iternext_custom_t instance, which then defines both
  getiter and iternext.

And a third flag that is the OR of both, MP_TYPE_FLAG_ITER_IS_STREAM: This
means that the type should use the identity getiter, and
mp_stream_unbuffered_iter as iternext.

Finally, MP_TYPE_FLAG_ITER_IS_GETITER is defined as a no-op flag to give
the default case where "iter" is "getiter".

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:13 +10:00
Jim Mussared
9dce82776d all: Remove unnecessary locals_dict cast.
Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Jim Mussared
662b9761b3 all: Make all mp_obj_type_t defs use MP_DEFINE_CONST_OBJ_TYPE.
In preparation for upcoming rework of mp_obj_type_t layout.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2022-09-19 19:06:01 +10:00
Damien George
7c8ec85fa3 shared/runtime/sys_stdio_mphal: Make func static and remove some TODOs.
stdio_obj_print is private to this file so can be made static.  The __del__
method does nothing so can be removed (it's only called by the GC if it
exists, so if it doesn't exist it won't be called).  And FileIO doesn't
support a constructor in MicroPython at this stage.

Signed-off-by: Damien George <damien@micropython.org>
2022-08-23 13:02:40 +10:00
robert-hh
93a17b9581 shared/runtime/softtimer: Remove obsolete #include statement. 2022-07-19 13:46:28 +10:00
robert-hh
2e2202993c shared/runtime/softtimer: Move softtimer.[ch] to shared/runtime.
And change the include lock to the naming scheme of that place.

This comes from ports/stm32/softtimer.[ch].
2022-07-19 12:28:26 +10:00
David Lechner
68f46342aa shared/runtime/pyexec: Use MP_REGISTER_ROOT_POINTER().
This uses MP_REGISTER_ROOT_POINTER() to register repl_line
instead of using a conditional inside of mp_state_vm_t.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:52:01 +10:00
iabdalkader
9f6f8b2fdd shared/runtime/tinyusb_helpers: Add TinyUSB helper functions.
Currently this file only includes a CDC jump-to-bootloader helper function.
2022-06-23 14:31:57 +10:00
Damien George
402df833fe py/modsys: Introduce sys.implementation._machine constant.
This contains a string useful for identifying the underlying machine.  This
string is kept consistent with the second part of the REPL banner via the
new config option MICROPY_BANNER_MACHINE.

This makes os.uname() more or less redundant, as all the information in
os.uname() is now available in the sys module.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 17:23:03 +10:00
Damien George
40047823bc py/modsys: Append MicroPython git version and build date to sys.version.
This commit adds the git hash and build date to sys.version.  This is
allowed according to CPython docs, and is what PyPy does.  The docs state:

    A string containing the version number of the Python interpreter plus
    additional information on the build number and compiler used.

Eg on CPython:

    Python 3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.version
    '3.10.4 (main, Mar 23 2022, 23:05:40) [GCC 11.2.0]'

and PyPy:

    Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)
    [PyPy 5.6.0 with GCC 6.2.0 20161109] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>> import sys
    >>>> sys.version
    '2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:43:30)\n[PyPy 5.6.0 with GCC ...

With this commit on MicroPython we now have:

    MicroPython v1.18-371-g9d08eb024 on 2022-04-28; linux [GCC 11.2.0] v...
    Use Ctrl-D to exit, Ctrl-E for paste mode
    >>> import sys
    >>> sys.version
    '3.4.0; MicroPython v1.18-371-g9d08eb024 on 2022-04-28'

Note that the start of the banner is the same as the end of sys.version.
This helps to keep code size under control because the string can be reused
by the compiler.

Signed-off-by: Damien George <damien@micropython.org>
2022-04-28 15:23:17 +10:00
Damien George
ac2293161e py/modsys: Add optional mutable attributes sys.ps1/ps2 and use them.
This allows customising the REPL prompt strings.

Signed-off-by: Damien George <damien@micropython.org>
2022-03-10 10:58:33 +11:00
Damien George
f2040bfc7e py: Rework bytecode and .mpy file format to be mostly static data.
Background: .mpy files are precompiled .py files, built using mpy-cross,
that contain compiled bytecode functions (and can also contain machine
code). The benefit of using an .mpy file over a .py file is that they are
faster to import and take less memory when importing.  They are also
smaller on disk.

But the real benefit of .mpy files comes when they are frozen into the
firmware.  This is done by loading the .mpy file during compilation of the
firmware and turning it into a set of big C data structures (the job of
mpy-tool.py), which are then compiled and downloaded into the ROM of a
device.  These C data structures can be executed in-place, ie directly from
ROM.  This makes importing even faster because there is very little to do,
and also means such frozen modules take up much less RAM (because their
bytecode stays in ROM).

The downside of frozen code is that it requires recompiling and reflashing
the entire firmware.  This can be a big barrier to entry, slows down
development time, and makes it harder to do OTA updates of frozen code
(because the whole firmware must be updated).

This commit attempts to solve this problem by providing a solution that
sits between loading .mpy files into RAM and freezing them into the
firmware.  The .mpy file format has been reworked so that it consists of
data and bytecode which is mostly static and ready to run in-place.  If
these new .mpy files are located in flash/ROM which is memory addressable,
the .mpy file can be executed (mostly) in-place.

With this approach there is still a small amount of unpacking and linking
of the .mpy file that needs to be done when it's imported, but it's still
much better than loading an .mpy from disk into RAM (although not as good
as freezing .mpy files into the firmware).

The main trick to make static .mpy files is to adjust the bytecode so any
qstrs that it references now go through a lookup table to convert from
local qstr number in the module to global qstr number in the firmware.
That means the bytecode does not need linking/rewriting of qstrs when it's
loaded.  Instead only a small qstr table needs to be built (and put in RAM)
at import time.  This means the bytecode itself is static/constant and can
be used directly if it's in addressable memory.  Also the qstr string data
in the .mpy file, and some constant object data, can be used directly.
Note that the qstr table is global to the module (ie not per function).

In more detail, in the VM what used to be (schematically):

    qst = DECODE_QSTR_VALUE;

is now (schematically):

    idx = DECODE_QSTR_INDEX;
    qst = qstr_table[idx];

That allows the bytecode to be fixed at compile time and not need
relinking/rewriting of the qstr values.  Only qstr_table needs to be linked
when the .mpy is loaded.

Incidentally, this helps to reduce the size of bytecode because what used
to be 2-byte qstr values in the bytecode are now (mostly) 1-byte indices.
If the module uses the same qstr more than two times then the bytecode is
smaller than before.

The following changes are measured for this commit compared to the
previous (the baseline):
- average 7%-9% reduction in size of .mpy files
- frozen code size is reduced by about 5%-7%
- importing .py files uses about 5% less RAM in total
- importing .mpy files uses about 4% less RAM in total
- importing .py and .mpy files takes about the same time as before

The qstr indirection in the bytecode has only a small impact on VM
performance.  For stm32 on PYBv1.0 the performance change of this commit
is:

diff of scores (higher is better)
N=100 M=100             baseline -> this-commit  diff      diff% (error%)
bm_chaos.py               371.07 ->  357.39 :  -13.68 =  -3.687% (+/-0.02%)
bm_fannkuch.py             78.72 ->   77.49 :   -1.23 =  -1.563% (+/-0.01%)
bm_fft.py                2591.73 -> 2539.28 :  -52.45 =  -2.024% (+/-0.00%)
bm_float.py              6034.93 -> 5908.30 : -126.63 =  -2.098% (+/-0.01%)
bm_hexiom.py               48.96 ->   47.93 :   -1.03 =  -2.104% (+/-0.00%)
bm_nqueens.py            4510.63 -> 4459.94 :  -50.69 =  -1.124% (+/-0.00%)
bm_pidigits.py            650.28 ->  644.96 :   -5.32 =  -0.818% (+/-0.23%)
core_import_mpy_multi.py  564.77 ->  581.49 :  +16.72 =  +2.960% (+/-0.01%)
core_import_mpy_single.py  68.67 ->   67.16 :   -1.51 =  -2.199% (+/-0.01%)
core_qstr.py               64.16 ->   64.12 :   -0.04 =  -0.062% (+/-0.00%)
core_yield_from.py        362.58 ->  354.50 :   -8.08 =  -2.228% (+/-0.00%)
misc_aes.py               429.69 ->  405.59 :  -24.10 =  -5.609% (+/-0.01%)
misc_mandel.py           3485.13 -> 3416.51 :  -68.62 =  -1.969% (+/-0.00%)
misc_pystone.py          2496.53 -> 2405.56 :  -90.97 =  -3.644% (+/-0.01%)
misc_raytrace.py          381.47 ->  374.01 :   -7.46 =  -1.956% (+/-0.01%)
viper_call0.py            576.73 ->  572.49 :   -4.24 =  -0.735% (+/-0.04%)
viper_call1a.py           550.37 ->  546.21 :   -4.16 =  -0.756% (+/-0.09%)
viper_call1b.py           438.23 ->  435.68 :   -2.55 =  -0.582% (+/-0.06%)
viper_call1c.py           442.84 ->  440.04 :   -2.80 =  -0.632% (+/-0.08%)
viper_call2a.py           536.31 ->  532.35 :   -3.96 =  -0.738% (+/-0.06%)
viper_call2b.py           382.34 ->  377.07 :   -5.27 =  -1.378% (+/-0.03%)

And for unix on x64:

diff of scores (higher is better)
N=2000 M=2000        baseline -> this-commit     diff      diff% (error%)
bm_chaos.py          13594.20 ->  13073.84 :  -520.36 =  -3.828% (+/-5.44%)
bm_fannkuch.py          60.63 ->     59.58 :    -1.05 =  -1.732% (+/-3.01%)
bm_fft.py           112009.15 -> 111603.32 :  -405.83 =  -0.362% (+/-4.03%)
bm_float.py         246202.55 -> 247923.81 : +1721.26 =  +0.699% (+/-2.79%)
bm_hexiom.py           615.65 ->    617.21 :    +1.56 =  +0.253% (+/-1.64%)
bm_nqueens.py       215807.95 -> 215600.96 :  -206.99 =  -0.096% (+/-3.52%)
bm_pidigits.py        8246.74 ->   8422.82 :  +176.08 =  +2.135% (+/-3.64%)
misc_aes.py          16133.00 ->  16452.74 :  +319.74 =  +1.982% (+/-1.50%)
misc_mandel.py      128146.69 -> 130796.43 : +2649.74 =  +2.068% (+/-3.18%)
misc_pystone.py      83811.49 ->  83124.85 :  -686.64 =  -0.819% (+/-1.03%)
misc_raytrace.py     21688.02 ->  21385.10 :  -302.92 =  -1.397% (+/-3.20%)

The code size change is (firmware with a lot of frozen code benefits the
most):

       bare-arm:  +396 +0.697%
    minimal x86: +1595 +0.979% [incl +32(data)]
       unix x64: +2408 +0.470% [incl +800(data)]
    unix nanbox: +1396 +0.309% [incl -96(data)]
          stm32: -1256 -0.318% PYBV10
         cc3200:  +288 +0.157%
        esp8266:  -260 -0.037% GENERIC
          esp32:  -216 -0.014% GENERIC[incl -1072(data)]
            nrf:  +116 +0.067% pca10040
            rp2:  -664 -0.135% PICO
           samd:  +844 +0.607% ADAFRUIT_ITSYBITSY_M4_EXPRESS

As part of this change the .mpy file format version is bumped to version 6.
And mpy-tool.py has been improved to provide a good visualisation of the
contents of .mpy files.

In summary: this commit changes the bytecode to use qstr indirection, and
reworks the .mpy file format to be simpler and allow .mpy files to be
executed in-place.  Performance is not impacted too much.  Eventually it
will be possible to store such .mpy files in a linear, read-only, memory-
mappable filesystem so they can be executed from flash/ROM.  This will
essentially be able to replace frozen code for most applications.

Signed-off-by: Damien George <damien@micropython.org>
2022-02-24 18:08:43 +11:00
iabdalkader
644f4dcc94 shared/runtime/pyexec: Cleanup EXEC_FLAG flag constants.
- Cleanup pyexec flags definitions so it's clear they are different.
- Use mp_uint_t for exec_flags because it should be unsigned.
2022-01-06 13:06:39 +11:00
Jim Mussared
e0bf4611c3 py: Only search frozen modules when '.frozen' is found in sys.path.
This changes makemanifest.py & mpy-tool.py to merge string and mpy names
into the same list (now mp_frozen_names).

The various paths for loading a frozen module (mp_find_frozen_module) and
checking existence of a frozen module (mp_frozen_stat) use a common
function that searches this list.

In addition, the frozen lookup will now only take place if the path starts
with ".frozen", which needs to be added to sys.path.

This fixes issues #1804, #2322, #3509, #6419.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-18 00:01:59 +11:00
Damien George
136369d72f all: Update to point to files in new shared/ directory.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-12 17:08:10 +10:00
Damien George
4d546713ec shared: Introduce new top-level dir and move 1st party lib code there.
This commit moves all first-party code developed for this project from lib/
to shared/, so that lib/ now only contains third-party code.

The following directories are moved as-is from lib to shared:

    lib/libc            -> shared/libc
    lib/memzip          -> shared/memzip
    lib/netutils        -> shared/netutils
    lib/timeutils       -> shared/timeutils
    lib/upytesthelper   -> shared/upytesthelper

All files in lib/embed/ have been moved to shared/libc/.

lib/mp-readline has been moved to shared/readline.

lib/utils has been moved to shared/runtime, with the exception of
lib/utils/printf.c which has been moved to shared/libc/printf.c.

Signed-off-by: Damien George <damien@micropython.org>
2021-07-12 17:07:36 +10:00