Only include .c and .cpp files explicitly in the list of files passed to
the preprocessor for QSTR extraction. All relevant .h files will be
included in this process by "#include" from the .c(pp) files. In
particular for moduledefs.h, this is included by py/objmodule.c (and
doesn't actually contain any extractable MP_QSTR_xxx, but rather defines
macros with MP_QSTR_xxx's in them which are then part of py/objmodule.c).
The main reason for this change is to simplify the preprocessing step on
the javascript port, which tries to compile .h files as C++ precompiled
headers if they are passed with -E to clang.
Signed-off-by: Damien George <damien@micropython.org>
Fixes the following (the line numbers match commit 0e87459e2b):
../../extmod/crypto-algorithms/sha256.c:49:19: runtime error: left shif...
../../extmod/moduasyncio.c:106:35: runtime error: member access within ...
../../py/binary.c:210:13: runtime error: left shift of negative value -...
../../py/mpz.c:744:16: runtime error: negation of -9223372036854775808 ...
../../py/objint.c:109:22: runtime error: left shift of 1 by 31 places c...
../../py/objint_mpz.c:374:9: runtime error: left shift of 4611686018427...
../../py/objint_mpz.c:374:9: runtime error: left shift of negative valu...
../../py/parsenum.c:106:14: runtime error: left shift of 46116860184273...
../../py/runtime.c:395:33: runtime error: left shift of negative value ...
../../py/showbc.c:177:28: runtime error: left shift of negative value -...
../../py/vm.c:321:36: runtime error: left shift of negative value -1```
Testing was done on an amd64 Debian Buster system using gcc-8.3 and these
settings:
CFLAGS += -g3 -Og -fsanitize=undefined
LDFLAGS += -fsanitize=undefined
The introduced TASK_PAIRHEAP macro's conditional (x ? &x->i : NULL)
assembles (under amd64 gcc 8.3 -Os) to the same as &x->i, since i is the
initial field of the struct. However, for the purposes of undefined
behavior analysis the conditional is needed.
Signed-off-by: Jeff Epler <jepler@gmail.com>
This introduces a new macro to get the main thread and uses it to ensure
that asynchronous exceptions such as KeyboardInterrupt (CTRL+C) are only
scheduled on the main thread. This is more deterministic than being
scheduled on a random thread and is more in line with CPython that only
allow signal handlers to run on the main thread.
Fixes issue #7026.
Signed-off-by: David Lechner <david@pybricks.com>
This moves mp_pending_exception from mp_state_vm_t to mp_state_thread_t.
This allows exceptions to be scheduled on a specific thread.
Signed-off-by: David Lechner <david@pybricks.com>
Prior to this commit, cache flushing for ARM native code was done only in
the assembler code asm_thumb_end_pass()/asm_arm_end_pass(), at the last
pass of the assembler. But this misses flushing the cache when loading
native code from an .mpy file, ie in persistentcode.c.
The change here makes sure the cache is always flushed/cleaned/invalidated
when assigning native code on ARM architectures.
This problem was found running tests/micropython/import_mpy_native_gc.py on
the mimxrt port.
Signed-off-by: Damien George <damien@micropython.org>
asan considers that memcmp(p, q, N) is permitted to access N bytes at each
of p and q, even for values of p and q that have a difference earlier.
Accessing additional values is frequently done in practice, reading 4 or
more bytes from each input at a time for efficiency, so when completing
"non_exist<TAB>" in the repl, this causes a diagnostic:
==16938==ERROR: AddressSanitizer: global-buffer-overflow on
address 0x555555cd8dc8 at pc 0x7ffff726457b bp 0x7fffffffda20 sp 0x7fff
READ of size 9 at 0x555555cd8dc8 thread T0
#0 0x7ffff726457a (/usr/lib/x86_64-linux-gnu/libasan.so.5+0xb857a)
#1 0x555555b0e82a in mp_repl_autocomplete ../../py/repl.c:301
#2 0x555555c89585 in readline_process_char ../../lib/mp-readline/re
#3 0x555555c8ac6e in readline ../../lib/mp-readline/readline.c:513
#4 0x555555b8dcbd in do_repl /home/jepler/src/micropython/ports/uni
#5 0x555555b90859 in main_ /home/jepler/src/micropython/ports/unix/
#6 0x555555b90a3a in main /home/jepler/src/micropython/ports/unix/m
#7 0x7ffff619a09a in __libc_start_main ../csu/libc-start.c:308
#8 0x55555595fd69 in _start (/home/jepler/src/micropython/ports/uni
0x555555cd8dc8 is located 0 bytes to the right of global variable
'import_str' defined in '../../py/repl.c:285:23' (0x555555cd8dc0) of
size 8
'import_str' is ascii string 'import '
Signed-off-by: Jeff Epler <jepler@gmail.com>
The proper way to do this is to test for __APPLE__ and __MACH__, where
__APPLE__ tests for an Apple OS and __MACH__ tests that it is based on CMU
Mach. Using both tests ensures that just Darwin (Apple's open source base
for MacOS, iOS, etc.) is recognized. __APPLE__ by itself will test for any
Apple OS, which can include older OS 7-9 and any future Apple OS. __MACH__
tests for any OS based on CMU Mach, including Darwin and GNU Hurd.
Fixes#7232.
Array equality is defined as each element being equal but to keep
code size down MicroPython implements a binary comparison. This
can only be used correctly for elements with the same binary layout
though so turn it into an NotImplementedError when comparing types
for which the binary comparison yielded incorrect results: types
with different sizes, and floating point numbers because nan != nan.
This commit makes gc_lock_depth have one counter per thread, instead of one
global counter. This makes threads properly independent with respect to
the GC, in particular threads can now independently lock the GC for
themselves without locking it for other threads. It also means a given
thread can run a hard IRQ without temporarily locking the GC for all other
threads and potentially making them have MemoryError exceptions at random
locations (this really only occurs on MCUs with multiple cores and no GIL,
eg on the rp2 port).
The commit also removes protection of the GC lock/unlock functions, which
is no longer needed when the counter is per thread (and this also fixes the
cas where a hard IRQ calling gc_lock() may stall waiting for the mutex).
It also puts the check for `gc_lock_depth > 0` outside the GC mutex in
gc_alloc, gc_realloc and gc_free, to potentially prevent a hard IRQ from
waiting on a mutex if it does attempt to allocate heap memory (and putting
the check outside the GC mutex is now safe now that there is a
gc_lock_depth per thread).
Signed-off-by: Damien George <damien@micropython.org>
Doing "import <tab>" will now complete/list built-in modules.
Originally at adafruit#4548 and adafruit#4608
Signed-off-by: Artyom Skrobov <tyomitch@gmail.com>
Anything beginning with "_" will now only be tab-completed if there is
already a partial match for such an entry. In other words, entering
foo.<tab> will no longer complete/list anything beginning with "_".
Originally at adafruit#1850
Signed-off-by: Kathryn Lingel <kathryn@lingel.net>
These commented-out lines of code have been unused for a long time, so
remove them to avoid confusion as to why they are there.
mp_obj_dict_free() never existed, this line was converted from
mp_map_deinit() and commented out as soon as it was added. The call to
mp_map_deinit(mp_loaded_modules_map) was commented in
1a1d11fa32.
Fixes issue #3507.
Signed-off-by: Damien George <damien@micropython.org>
This helper is added to properly set a pending exception, to mirror
mp_sched_schedule(), which schedules a function.
Signed-off-by: Damien George <damien@micropython.org>
If MICROPY_ENABLE_SCHEDULER is enabled then MP_STATE_VM(sched_state) must
be updated after handling the pending exception, which is done by the
mp_handle_pending() function.
Signed-off-by: Damien George <damien@micropython.org>
This introduces a new option, MICROPY_ERROR_REPORTING_NONE, which
completely disables all error messages. To be used in cases where
MicroPython needs to fit in very limited systems.
Signed-off-by: Damien George <damien@micropython.org>
This commit adds the errno attribute to exceptions, so code can retrieve
errno codes from an OSError using exc.errno.
The implementation here simply lets `errno` (and the existing `value`)
attributes work on any exception instance (they both alias args[0]). This
is for efficiency and to keep code size down. The pros and cons of this
are:
Pros:
- more compatible with CPython, less difference to document and learn
- OSError().errno will correctly return None, whereas the current way of
doing it via OSError().args[0] will raise an IndexError
- it reduces code size on most bare-metal ports (because they already have
the errno qstr)
- for Python code that uses exc.errno the generated bytecode is 2 bytes
smaller and more efficient to execute (compared with exc.args[0]); so
bytecode loaded to RAM saves 2 bytes RAM for each use of this attribute,
and bytecode that is frozen saves 2 bytes flash/ROM for each use
- it's easier/shorter to type, and saves 2 bytes of space in .py files that
use it (for each use)
Cons:
- increases code size by 4-8 bytes on minimal ports that don't already have
the `errno` qstr
- all exceptions now have .errno and .value attributes (a cpydiff test is
added to address this)
See also #2407.
Signed-off-by: Damien George <damien@micropython.org>
This allows configuring the pre-allocated size of sys.modules dict, in
order to prevent unwanted reallocations at run-time (3 sys-modules is
really not quite enough for a larger project).
When building with STATIC undefined (e.g., -DSTATIC=), there are two
instances of mp_type_code that collide at link time: in profile.c and in
builtinevex.c. This patch resolves the collision by renaming one of them.
The parts that are generic are added to py/ so they can be used by other
ports that use CMake.
py/usermod.cmake:
* Creates a usermod target to hang user C/CXX modules from.
* Gathers sources from user C/CXX modules and libs for QSTR scan.
ports/rp2/CMakeLists.txt:
* Includes py/usermod.cmake.
* Links the resulting usermod library to the MicroPython target.
py/mkrules.cmake:
Add cxxflags to qstr.i.last custom command for CXX modules:
* MICROPY_CPP_FLAGS so CXX modules will find includes.
* -DNO_QSTR to fix fatal error missing "genhdr/qstrdefs.generated.h".
Usage:
The rp2 port can be linked against user C modules by running:
make USER_C_MODULES=/path/to/module/micropython.cmake
CMake will print a list of included modules.
Co-authored-by: Graham Sanderson <graham.sanderson@raspberrypi.org>
Co-authored-by: Michael O'Cleirigh <michael.ocleirigh@rivulet.ca>
Signed-off-by: Phil Howard <phil@pimoroni.com>
mp_printf should be used to print the prefix because it's also used in
mp_bytecode_print2 (otherwise, depending on the system, different output
streams may be used).
Also print the current thread state when threading is enabled to easily see
which thread executes what opcode.
Signed-off-by: Damien George <damien@micropython.org>
