Commit Graph

215 Commits

Author SHA1 Message Date
David Lechner
fc3d7ae11b py/make_root_pointers: Add MP_REGISTER_ROOT_POINTER parser/generator.
This adds new compile-time infrastructure to parse source code files for
`MP_REGISTER_ROOT_POINTER()` and generates a new `root_pointers.h` header
file containing the collected declarations.  This works the same as the
existing `MP_REGISTER_MODULE()` feature.

Signed-off-by: David Lechner <david@pybricks.com>
2022-07-18 13:48:23 +10:00
David Lechner
c118b5d0e4 extmod/extmod.mk: Separate out extmod file list from py.mk to extmod.mk.
This separates extmod source files from `py.mk`.  Previously, `py.mk`
assumed that every consumer of the py/ directory also wanted to include
extmod/.  However, this is not the case.  For example, building mpy-cross
uses py/ but doesn't need extmod/.

This commit moves all extmod-specific items from `py.mk` to `extmod.mk` and
explicitly includes `extmod.mk` in ports that use it.

Signed-off-by: David Lechner <david@pybricks.com>
2022-06-21 00:14:34 +10:00
Damien George
f5769698e5 extmod/modlwip: Clean up inclusion of modlwip in build process.
The following changes are made:

- Guard entire file with MICROPY_PY_LWIP, so it can be included in the
  build while still being disabled (for consistency with other extmod
  modules).

- Add modlwip.c to list of all extmod source in py/py.mk and
  extmod/extmod.cmake so all ports can easily use it.

- Move generic modlwip GIT_SUBMODULES build configuration code from
  ports/rp2/CMakeLists.txt to extmod/extmod.cmake, so it can be reused by
  other ports.

- Remove now unnecessary inclusion of modlwip.c in EXTMOD_SRC_C in esp8266
  port, and in SRC_QSTR in mimxrt port.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-20 23:37:38 +10:00
Damien George
47f634300c py: Change makemoduledefs process so it uses output of qstr extraction.
This cleans up the parsing of MP_REGISTER_MODULE() and generation of
genhdr/moduledefs.h so that it uses the same process as compressed error
string messages, using the output of qstr extraction.

This makes sure all MP_REGISTER_MODULE()'s that are part of the build are
correctly picked up.  Previously the extraction would miss some (eg if you
had a mod.c file in the board directory for an stm32 board).

Build speed is more or less unchanged.

Thanks to @stinos for the ports/windows/msvc/genhdr.targets changes.

Signed-off-by: Damien George <damien@micropython.org>
2022-06-02 16:29:53 +10:00
Damien George
94077c6402 samd/moduos: Convert module to use extmod version.
Signed-off-by: Damien George <damien@micropython.org>
2022-03-09 10:03:23 +11:00
Jim Mussared
92353c2911 all: Remove support for FROZEN_DIR and FROZEN_MPY_DIR.
These have been deprecated for over two years in favour of FROZEN_MANIFEST
and manifest.py.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-12-17 23:54:05 +11:00
iabdalkader
2c5e9bbdfa extmod: Add platform module.
It contains the compiler version, and underlying system HAL/SDK version.
2021-09-19 23:35:10 +10:00
Damien George
af64c2ddbd extmod/machine_pwm: Factor out machine.PWM bindings to common code.
This commit refactors machine.PWM and creates extmod/machine_pwm.c.  The
esp8266, esp32 and rp2 ports all use this and provide implementations of
the required PWM functionality.  This helps to reduce code duplication and
keep the same Python API across ports.

This commit does not make any functional changes.

Signed-off-by: Damien George <damien@micropython.org>
2021-09-04 16:31:17 +10:00
Jim Mussared
870000f35b extmod: Add machine.bitstream.
This is a generic API for synchronously bit-banging data on a pin.

Initially this adds a single supported encoding, which supports controlling
WS2812 LEDs.

Signed-off-by: Jim Mussared <jim.mussared@gmail.com>
2021-08-19 22:50:11 +10:00
Damien George
136369d72f all: Update to point to files in new shared/ directory.
Signed-off-by: Damien George <damien@micropython.org>
2021-07-12 17:08:10 +10:00
Damien George
08e0e065f4 py/makeqstrdefs.py: Don't include .h files explicitly in preprocessing.
Only include .c and .cpp files explicitly in the list of files passed to
the preprocessor for QSTR extraction.  All relevant .h files will be
included in this process by "#include" from the .c(pp) files.  In
particular for moduledefs.h, this is included by py/objmodule.c (and
doesn't actually contain any extractable MP_QSTR_xxx, but rather defines
macros with MP_QSTR_xxx's in them which are then part of py/objmodule.c).

The main reason for this change is to simplify the preprocessing step on
the javascript port, which tries to compile .h files as C++ precompiled
headers if they are passed with -E to clang.

Signed-off-by: Damien George <damien@micropython.org>
2021-06-25 10:50:54 +10:00
Yonatan Goldschmidt
2d5cece5ac py/nlr: Implement NLR for AArch64. 2021-03-11 12:51:10 +11:00
stijn
0153148fd2 py/py.mk: Support C++ code for user C modules.
Support C++ code in .cpp files by providing CXX counterparts of the
_USERMOD_ flags we have for C already.  This merely enables the Makefile of
user C modules to use variables specific to C++ compilation, it is still up
to each port's main Makefile to also include these in the build.
2020-10-29 15:29:20 +11:00
Damien George
a902b69dd5 py/py.mk: Use additional CFLAGS to compile string0.c.
Otherwise functions like memset might get optimised to call themselves (eg
with gcc 10).  And provide CFLAGS_BUILTIN so these options can be changed
by a port if needed.

Fixes issue #6053.
2020-05-27 23:10:23 +10:00
Jim Mussared
154b4eb354 py: Implement "common word" compression scheme for error messages.
The idea here is that there's a moderate amount of ROM used up by exception
text.  Obviously we try to keep the messages short, and the code can enable
terse errors, but it still adds up.  Listed below is the total string data
size for various ports:

    bare-arm 2860
    minimal 2876
    stm32 8926  (PYBV11)
    cc3200 3751
    esp32 5721

This commit implements compression of these strings.  It takes advantage of
the fact that these strings are all 7-bit ascii and extracts the top 128
frequently used words from the messages and stores them packed (dropping
their null-terminator), then uses (0x80 | index) inside strings to refer to
these common words.  Spaces are automatically added around words, saving
more bytes.  This happens transparently in the build process, mirroring the
steps that are used to generate the QSTR data.  The MP_COMPRESSED_ROM_TEXT
macro wraps any literal string that should compressed, and it's
automatically decompressed in mp_decompress_rom_string.

There are many schemes that could be used for the compression, and some are
included in py/makecompresseddata.py for reference (space, Huffman, ngram,
common word).  Results showed that the common-word compression gets better
results.  This is before counting the increased cost of the Huffman
decoder.  This might be slightly counter-intuitive, but this data is
extremely repetitive at a word-level, and the byte-level entropy coder
can't quite exploit that as efficiently.  Ideally one would combine both
approaches, but for now the common-word approach is the one that is used.

For additional comparison, the size of the raw data compressed with gzip
and zlib is calculated, as a sort of proxy for a lower entropy bound.  With
this scheme we come within 15% on stm32, and 30% on bare-arm (i.e. we use
x% more bytes than the data compressed with gzip -- not counting the code
overhead of a decoder, and how this would be hypothetically implemented).

The feature is disabled by default and can be enabled by setting
MICROPY_ROM_TEXT_COMPRESSION at the Makefile-level.
2020-04-05 14:20:57 +10:00
Damien George
bc009fdd62 extmod/uasyncio: Add optional implementation of core uasyncio in C.
Implements Task and TaskQueue classes in C, using a pairing-heap data
structure.  Using this reduces RAM use of each Task, and improves overall
performance of the uasyncio scheduler.
2020-03-26 01:25:45 +11:00
Damien George
fe203bb3e2 py/pairheap: Add generic implementation of pairing heap data structure. 2020-01-22 17:31:18 +11:00
Jim Mussared
7d2ccd027f py/mkenv.mk: Move usage of 32-bit flags to py.mk.
This allows ports/variants to configure MICROPY_FORCE_32BIT after including
mkenv.mk, but before py.mk.
2020-01-12 10:34:10 +11:00
Damien George
a099505420 extmod: Add VFS littlefs bindings.
Both LFS1 and LFS2 are supported at the same time.
2019-10-29 14:17:29 +11:00
Damien George
e1c7b1cb43 extmod/vfs_blockdev: Factor out block device interface code. 2019-10-29 12:55:17 +11:00
Michael Neuling
079cc940a6 powerpc: Add initial port to bare metal PowerPC arch.
Runs in microwatt (GHDL and FPGA) and qemu.

Port done initially by Michael Neuling, with help from Anton Blanchard and
Jordan Niethe.
2019-10-22 22:45:33 +11:00
Damien George
e81f538e25 tools: Add mechanism to provide a manifest of frozen files.
This introduces a new build variable FROZEN_MANIFEST which can be set to a
manifest listing (written in Python) that describes the set of files to be
frozen in to the firmware.
2019-10-15 21:34:23 +11:00
Damien George
9adedce42e py: Add new Xtensa-Windowed arch for native emitter.
Enabled via the configuration MICROPY_EMIT_XTENSAWIN.
2019-10-05 13:44:53 +10:00
Jim Mussared
4ddd46e6cf docs/develop/qstr.rst: Add documentation for string interning. 2019-10-04 17:13:13 +10:00
Jim Mussared
16f8ceeaaa extmod/modbluetooth: Add low-level Python BLE API. 2019-10-01 09:51:02 +10:00
Jim Mussared
42e9bdf19b py/ringbuf: Add helpers for put16/get16. 2019-10-01 09:51:02 +10:00
Damien George
6e07fde895 py/mkrules.mk: Add QSTR_GLOBAL_REQUIREMENTS variable for qstr auto-gen. 2019-09-12 18:16:16 +10:00
Milan Rossa
310b3d1b81 py: Integrate sys.settrace feature into the VM and runtime.
This commit adds support for sys.settrace, allowing to install Python
handlers to trace execution of Python code.  The interface follows CPython
as closely as possible.  The feature is disabled by default and can be
enabled via MICROPY_PY_SYS_SETTRACE.
2019-08-30 16:44:12 +10:00
Damien George
3d7455a0bb py/py.mk: Remove trailing spaces at end of line. 2019-08-27 16:37:43 +10:00
Damien George
cd6b115815 extmod: Factor out makefile rules from py.mk to new extmod.mk file.
To logically separate extmod related rules out, and prevent py.mk from
growing too large.
2019-06-05 14:23:12 +10:00
Sébastien Rinsoz
c03f81c633 py: Update makefiles to use $(CAT) variable instead of hard coded "cat".
The variable $(CAT) is initialised with the "cat" value in mkenv.mk like
for the other command line tools (rm, echo, cp, mkdir etc).  With this,
for example, Windows users can specify the path of cat.exe.
2019-05-21 14:26:28 +10:00
Andrew Leech
89ff506513 py: Update and rework build system for including external C modules.
How to use this feature is documented in docs/develop/cmodules.rst.
2019-03-08 22:58:42 +11:00
Ayke van Laethem
2e516074da py: Implement a module system for external, user C modules.
This system makes it a lot easier to include external libraries as static,
native modules in MicroPython.  Simply pass USER_C_MODULES (like
FROZEN_MPY_DIR) as a make parameter.
2019-03-08 22:49:00 +11:00
Andrew Leech
cf22f4793c py: Allow registration of modules at their definition.
During make, makemoduledefs.py parses the current builds c files for
MP_REGISTER_MODULE(module_name, obj_module, enabled_define)

These are used to generate a header with the required entries for
"mp_rom_map_elem_t mp_builtin_module_table[]" in py/objmodule.c
2019-03-08 22:46:43 +11:00
Damien George
871954d75c py/py.mk: Update lwip build config to work with latest lwip version.
Also, to make it possible for ports to provide their own lwipopts.h, the
default include directory of extmod/lwip-include is no longer added and
instead a port should now make sure the correct include directory is
included in the list (can still use extmod/lwip-include).
2019-03-04 23:29:01 +11:00
Yonatan Goldschmidt
bc4f8b438b extmod/moduwebsocket: Refactor websocket to uwebsocket.
As mentioned in #4450, `websocket` was experimental with a single intended
user, `webrepl`. Therefore, we'll make this change without a weak
link `websocket` -> `uwebsocket`.
2019-02-14 00:35:45 +11:00
Craig Younkins
7f948a5645 py/py.mk: Fix broken Gmane URL. 2018-12-04 01:03:44 +11:00
Damien George
746dbf78d3 py/py.mk: When building axtls use -Wno-all to prevent all warnings.
Building axtls gives a lot of warnings with -Wall enabled, and explicitly
disabling all of them cannot be done in a way compatible with gcc and
clang, and likely other compilers.  So just use -Wno-all to prevent all of
the extra warnings (in addition to the necessary -Wno-unused-parameter,
-Wno-uninitialized, -Wno-sign-compare and -Wno-old-style-definition).

Fixes issue #4182.
2018-10-27 23:53:08 +11:00
Damien George
0be2ea50e9 py/py.mk: Build axtls library directly from its source files.
This removes the need for a separate axtls build stage, and builds all
axtls object files along with other code.  This simplifies and cleans up
the build process, automatically builds axtls when needed, and puts the
axtls object files in the correct $(BUILD) location.

The MicroPython axtls configuration file is provided in
extmod/axtls-include/config.h
2018-09-08 00:07:23 +10:00
Damien George
b8b2525576 extmod/modbtree: Update to work with new mp_stream_posix_XXX signatures. 2018-08-14 17:41:23 +10:00
Paul Sokolovsky
bb28fe7b7b py/py.mk: Don't hardcode path to libaxtls.a.
Use -L$(BUILD), not -Lbuild. Otherwise, builds for different archs/subarchs
using different values of BUILD may fail.
2018-08-14 15:10:52 +10:00
Paul Sokolovsky
567bc2d6ce extmod/moducryptolib: Add ucryptolib module with crypto functions.
The API follows guidelines of https://www.python.org/dev/peps/pep-0272/,
but is optimized for code size, with the idea that full PEP 0272
compatibility can be added with a simple Python wrapper mode.

The naming of the module follows (u)hashlib pattern.

At the bare minimum, this module is expected to provide:

* AES128, ECB (i.e. "null") mode, encrypt only

Implementation in this commit is based on axTLS routines, and implements
following:

* AES 128 and 256
* ECB and CBC modes
* encrypt and decrypt
2018-06-27 14:54:40 +10:00
Damien George
7ad04d17da py/mkrules.mk: Regenerate all qstrs when config files change.
A port can define QSTR_GLOBAL_DEPENDENCIES to add extra files.
2018-06-12 13:53:43 +10:00
Damien George
8d82b0edbd extmod: Add VfsPosix filesystem component.
This VFS component allows to mount a host POSIX filesystem within the uPy
VFS sub-system.  All traditional POSIX file access then goes through the
VFS, allowing to sandbox a uPy process to a certain sub-dir of the host
system, as well as mount other filesystem types alongside the host
filesystem.
2018-06-06 14:28:23 +10:00
Damien George
ef12a4bd05 py: Refactor how native emitter code is compiled with a file per arch.
Instead of emitnative.c having configuration code for each supported
architecture, and then compiling this file multiple times with different
macros defined, this patch adds a file per architecture with the necessary
code to configure the native emitter.  These files then #include the
emitnative.c file.

This simplifies emitnative.c (which is already very large), and simplifies
the build system because emitnative.c no longer needs special handling for
compilation and qstr extraction.
2018-04-10 15:06:47 +10:00
Damien George
638b860066 extmod/vfs_fat: Merge remaining vfs_fat_misc.c code into vfs_fat.c.
The only function left in vfs_fat_misc.c is fat_vfs_import_stat() which
can logically go into vfs_fat.c, allowing to remove vfs_fat_misc.c.
2018-02-23 17:24:57 +11:00
Damien George
65ef59a9b5 py/py.mk: Remove .. path component from list of extmod files.
This just makes it a bit cleaner in the output of the build process:
instead of "CC ../../py/../extmod/" there is now "CC ../../extmod/".
2018-02-22 12:48:51 +11:00
Damien George
8ca469cae2 py/py.mk: Split list of uPy sources into core and extmod files.
If a port only needs the core files then it can now use the $(PY_CORE_O)
variable instead of $(PY_O).  $(PY_EXTMOD_O) contains the list of extmod
files (including some files from lib/). $(PY_O) retains its original
definition as the list of all object file (including those for frozen code)
and is a convenience variable for ports that want everything.
2018-02-22 12:48:15 +11:00
Paul Sokolovsky
970eedce8f py/objdeque: Implement ucollections.deque type with fixed size.
So far, implements just append() and popleft() methods, required for
a normal queue. Constructor doesn't accept an arbitarry sequence to
initialize from (am empty deque is always created), so an empty tuple
must be passed as such. Only fixed-size deques are supported, so 2nd
argument (size) is required.

There's also an extension to CPython - if True is passed as 3rd argument,
append(), instead of silently overwriting the oldest item on queue
overflow, will throw IndexError. This behavior is desired in many
cases, where queues should store information reliably, instead of
silently losing some items.
2018-02-21 22:39:25 +11:00
Damien George
b25f92160b py/nlr: Factor out common NLR code to macro and generic funcs in nlr.c.
Each NLR implementation (Thumb, x86, x64, xtensa, setjmp) duplicates a lot
of the NLR code, specifically that dealing with pushing and popping the NLR
pointer to maintain the linked-list of NLR buffers.  This patch factors all
of that code out of the specific implementations into generic functions in
nlr.c, along with a helper macro in nlr.h.  This eliminates duplicated
code.
2017-12-28 16:46:30 +11:00