Commit Graph

284 Commits

Author SHA1 Message Date
Damien George
ee50a6effe py/mphal.h: Introduce mp_hal_time_ns and implement on various ports.
This should return a 64-bit value being the number of nanoseconds since
1970/1/1.

Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 16:13:44 +10:00
Damien George
92899354d9 unix/fatfs_port: Implement get_fattime.
Signed-off-by: Damien George <damien@micropython.org>
2020-08-22 14:45:57 +10:00
Jim Mussared
5d0be97bd9 unix: Make the MICROPY_xxx_ATOMIC_SECTION mutex recursive.
This mutex is used to make the unix port behave more like bare metal, i.e.
it allows "IRQ handlers" to run exclusively by making the mutex recursive.
2020-07-18 14:22:06 +10:00
Damien George
f5dd46b479 unix/variants: Enable VFS and all supported filesystems on dev variant.
So that micropython-dev can be used to test VFS code, and inspect and build
filesystem images that are compatible with bare-metal systems.

Signed-off-by: Damien George <damien@micropython.org>
2020-07-08 23:57:25 +10:00
Jim Mussared
4050281311 unix: Enable uasyncio on dev variant. 2020-07-01 22:44:41 +10:00
Jim Mussared
27abac95d8 unix: Make manifest selection match other ports.
Changes are:
- The default manifest.py is moved to the variants directory (it's in
  "boards" in other ports).
- The coverage variant now uses a custom manifest in its variant directory
  to add frzmpy/frzstr.
- The frzmpy/frzstr tests are moved to variants/coverage/.
2020-07-01 22:42:55 +10:00
Jim Mussared
3f77f2c60c unix/btstack_usb: Allow choosing adaptor via environment variable.
This allows running (for example):

    env MICROPYBTUSB=2-2 ./micropython-dev ../../examples/bluetooth/ble_temperature_central.py
2020-06-10 22:40:02 +10:00
Yu-Ming Chang
dd8db974d7 unix/main: Enter REPL when inspect active, even with stdin redirected.
This is how CPython behaves.
2020-05-16 14:13:15 +10:00
Damien George
5c8bf12acf all: Fix auto-enable of MICROPY_GCREGS_SETJMP to select GC behaviour.
Only enable it if MICROPY_GCREGS_SETJMP is not already defined, and no
supported architecture is defined.
2020-04-30 16:49:42 +10:00
Jim Mussared
710426024a all: Factor gchelper code to one place and use it for unix & ARM ports.
No functionality change is intended with this commit, it just consolidates
the separate implementations of GC helper code to the lib/utils/ directory
as a general set of helper functions useful for any port.  This reduces
duplication of code, and makes it easier for future ports or embedders to
get the GC implementation correct.

Ports should now link against gchelper_native.c and either gchelper_m0.s or
gchelper_m3.s (currently only Cortex-M is supported but other architectures
can follow), or use the fallback gchelper_generic.c which will work on
x86/x64/ARM.

The gc_helper_get_sp function from gchelper_m3.s is not really GC related
and was only used by cc3200, so it has been moved to that port and renamed
to cortex_m3_get_sp.
2020-04-29 23:45:19 +10:00
Jim Mussared
2e3c42775a unix: Add btstack to the unix submodules list.
But only when bluetooth is enabled, i.e. if building the dev or coverage
variants, and we have libusb available.

Update travis to match, i.e. specify the variant when doing
`make submodules`.
2020-04-29 16:54:12 +10:00
Jim Mussared
ebfd9ff2e6 extmod/modbluetooth: Fix sign compare and unused variable warnings. 2020-04-29 16:54:12 +10:00
Jim Mussared
af226199ea unix: Enable modbluetooth on the "dev" and "coverage" variants.
And MICROPY_PY_URANDOM_EXTRA_FUNCS is enabled on "dev" so tha the Bluetooth
examples all run.
2020-04-29 16:53:13 +10:00
Jim Mussared
7563d58210 unix: Add support for modbluetooth and BLE using btstack.
This commit adds full support to the unix port for Bluetooth using the
common extmod/modbluetooth Python bindings.  This uses the libusb HCI
transport, which supports many common USB BT adaptors.
2020-04-29 16:45:46 +10:00
Jim Mussared
cb5994d96e unix/modmachine: Add machine.idle(), implemented using sched_yield.
Also add a definition of MICROPY_EVENT_POLL_HOOK so the unix port can build
against modules that require this.
2020-04-29 16:37:46 +10:00
stijn
84fa3312cf all: Format code to add space after C++-style comment start.
Note: the uncrustify configuration is explicitly set to 'add' instead of
'force' in order not to alter the comments which use extra spaces after //
as a means of indenting text for clarity.
2020-04-23 11:24:25 +10:00
stijn
30840ebc99 all: Enable extra conversion warnings where applicable.
Add -Wdouble-promotion and -Wfloat-conversion for most ports to ban out
implicit floating point conversions, and add extra Travis builds using
MICROPY_FLOAT_IMPL_FLOAT to uncover warnings which weren't found
previously.  For the unix port -Wsign-comparison is added as well but only
there since only clang supports this but gcc doesn't.
2020-04-18 22:42:28 +10:00
stijn
70affd9ba2 all: Fix implicit floating point to integer conversions.
These are found when building with -Wfloat-conversion.
2020-04-18 22:42:24 +10:00
stijn
bcf01d1686 all: Fix implicit conversion from double to float.
These are found when building with -Wfloat-conversion.
2020-04-18 22:42:24 +10:00
stijn
0ba68f8a1d all: Fix implicit floating point promotion.
Initially some of these were found building the unix coverage variant on
MacOS because that build uses clang and has -Wdouble-promotion enabled, and
clang performs more vigorous promotion checks than gcc.  Additionally the
codebase has been compiled with clang and msvc (the latter with warning
level 3), and with MICROPY_FLOAT_IMPL_FLOAT to find the rest of the
conversions.

Fixes are implemented either as explicit casts, or by using the correct
type, or by using one of the utility functions to handle floating point
casting; these have been moved from nativeglue.c to the public API.
2020-04-18 22:36:14 +10:00
stijn
b909e8b2dd Revert "all: Fix implicit casts of float/double, and signed comparison."
This reverts commit a2110bd3fc.  There's
nothing inherently wrong with it, but upcoming commits will apply similar
fixes in a slightly different way.
2020-04-18 22:36:06 +10:00
Damien George
8e048d2548 all: Clean up error strings to use lowercase and change cannot to can't.
Now that error string compression is supported it's more important to have
consistent error string formatting (eg all lowercase English words,
consistent contractions).  This commit cleans up some of the strings to
make them more consistent.
2020-04-13 22:19:37 +10:00
David Lechner
1bbc15dd15 unix/Makefile: Fix regression using install on non-GNU systems.
This was fixed previously in 31fc81d3b8 but
regressed in 4af79e7694.

Fixes #5885.
2020-04-13 22:11:36 +10:00
Jim Mussared
c2cfbcc8d4 unix: Implement MICROPY_BEGIN/END_ATOMIC_SECTION protection macros.
This macro is used to implement global serialisation, typically by
disabling IRQs.  On the unix port, if threading is enabled, use the
existing thread mutex (that protects the thread list structure) for this
purpose.  Other places in the code (eg the scheduler) assume this macro
will provide serialisation.
2020-04-13 21:44:09 +10:00
Jim Mussared
45cf76465c unix: Fix behaviour of COPT/NDEBUG for unix variants.
Based on eg 1e6fd9f2b4, it's understood that
the intention for unix builds is that regular builds disable assert, but
the coverage build should set -O0 and enable asserts.

It looks like this didn't work (even before variants were introduced, eg at
v1.11) -- coverage always built with -Os and -DNDEBUG.

This commit makes it possible for variants to have finer-grained control
over COPT flags, and enables assert() and -O0 on coverage builds.

Other variants already match the defaults so they have been updated.
2020-04-13 21:20:32 +10:00
Jim Mussared
073b9a5eb8 ports: Enable error text compression for various ports, but not all.
Enabled on: bare-arm, minimal, unix coverage/dev/minimal, stm32, esp32,
esp8266, cc3200, teensy, qemu-arm, nrf.  Not enabled on others to be able
to test the code when the feature is disabled (the default case).

Code size change for this commit:

   bare-arm:  -600 -0.906%
minimal x86:  -308 -0.208%
   unix x64:    +0 +0.000%
unix nanbox:    +0 +0.000%
      stm32: -3368 -0.869% PYBV10
     cc3200: -1024 -0.558%
    esp8266: -2512 -0.368% GENERIC
      esp32: -2876 -0.205% GENERIC[incl -3168(data)]
        nrf: -1708 -1.173% pca10040
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2020-04-05 15:02:06 +10:00
Jim Mussared
def76fe4d9 all: Use MP_ERROR_TEXT for all error messages. 2020-04-05 15:02:06 +10:00
Damien George
8fff0b0acd unix/mpthreadport: Ensure enough thread stack to detect overflow.
Following up to 5e6cee07ab, some systems (eg
FreeBSD 12.0 64-bit) will crash if the stack-overflow margin is too small.
It seems the margin of 8192 bytes (or thereabouts) is always needed.  This
commit adds this much margin if the requested stack size is too small.

Fixes issue #5824.
2020-03-31 09:35:46 +11:00
David Lechner
a2110bd3fc all: Fix implicit casts of float/double, and signed comparison.
These were found by buiding the unix coverage variant on macOS (so clang
compiler).  Mostly, these are fixing implicit cast of float/double to
mp_float_t which is one of those two and one mp_int_t to size_t fix for
good measure.
2020-03-30 12:04:21 +11:00
Damien George
1a3e386c67 all: Remove spaces inside and around parenthesis.
Using new options enabled in the uncrustify configuration.
2020-03-28 23:36:44 +11:00
David Lechner
9418611c8a unix: Implement PEP 475 to retry syscalls failing with EINTR.
https://www.python.org/dev/peps/pep-0475/

This implements something similar to PEP 475 on the unix port, and for the
VfsPosix class.

There are a few differences from the CPython implementation:
- Since we call mp_handle_pending() between any ENITR's, additional
  functions could be called if MICROPY_ENABLE_SCHEDULER is enabled, not
  just signal handlers.
- CPython only handles signal on the main thread, so other threads will
  raise InterruptedError instead of retrying.  On MicroPython,
  mp_handle_pending() will currently raise exceptions on any thread.

A new macro MP_HAL_RETRY_SYSCALL is introduced to reduce duplicated code
and ensure that all instances behave the same.  This will also allow other
ports that use POSIX-like system calls (and use, eg, VfsPosix) to provide
their own implementation if needed.
2020-03-27 14:40:46 +11:00
David Lechner
5e6cee07ab unix/mpthreadport: Fix crash when thread stack size <= 8k.
The stack size adjustment for detecting stack overflow in threads was not
taking into account that the requested stack size could be <= 8k, in which
case the subtraction would overflow.  This is fixed in this commit by
ensuring that the adjustment can't be more than the available size.

This fixes the test tests/thread/thread_stacksize1.py which sometimes
crashes with a segmentation fault because of an uncaught NLR jump, which is
a "maximum recursion depth exceeded" exception.

Suggested-by: @dpgeorge
2020-03-27 13:59:18 +11:00
Damien George
91dd3948e8 unix: Enable uasyncio C helper module on coverage build. 2020-03-26 01:25:45 +11:00
Damien George
f9741d18f6 unix/coverage: Init all pairheap test nodes before using them. 2020-03-26 01:21:04 +11:00
David Lechner
3b07736b6d unix,windows: Use STDIN_FILENO, STDOUT_FILENO macros where appropriate.
This replaces 0 and 1 with STDIN_FILENO and STDOUT_FILENO to make the
intention of the code clearer.
2020-03-25 00:59:05 +11:00
David Lechner
b1066a9f96 unix: Remove custom definition of MP_PLAT_PRINT_STRN.
This removes the port-specific definition of MP_PLAT_PRINT_STRN on the unix
port.  Since fee7e5617f this is no longer a
single function call so we are not really optimising anything over using
the default definition of MP_PLAT_PRINT_STRN which calls
mp_hal_stdout_tx_strn_cooked().
2020-03-25 00:54:18 +11:00
stijn
f62cc41fac windows/msvc: Fix warnings regarding function declarations.
Fix missing mkdir and gettimeofday declarations, then silence msvc-specific
compiler warning C4996: 'The POSIX name for this item is deprecated'.
2020-03-25 00:38:11 +11:00
Damien George
2cdf1d25f5 unix: Remove custom file implementation to use extmod's VFS POSIX one.
The implementation in extmod/vfs_posix_file.c is now equivalent to that in
ports/unix/file.c, so remove the latter and use the former instead.
2020-03-18 21:01:07 +11:00
Damien George
359213fbe1 unix/Makefile: Detect and pass thru mpy-cross flags when running tests. 2020-03-11 20:20:18 +11:00
Andrew Leech
86bfabec11 py/modmicropython: Add heap_locked function to test state of heap.
This commit adds micropython.heap_locked() which returns the current
lock-depth of the heap, and can be used by Python code to check if the heap
is locked or not.  This new function is configured via
MICROPY_PY_MICROPYTHON_HEAP_LOCKED and is disabled by default.

This commit also changes the return value of micropython.heap_unlock() so
it returns the current lock-depth as well.
2020-03-11 16:54:16 +11:00
Damien George
3e0b46b9af unix/file: Don't raise OSError(EINVAL) on sys.stdin/out/err.flush().
sys.stdout.flush() is needed on CPython to flush the output, and the change
in this commit makes such an expression also work on MicroPython (although
MicroPython doesn't actual need to do any flushing).
2020-03-04 12:39:55 +11:00
Damien George
69661f3343 all: Reformat C and Python source code with tools/codeformat.py.
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-28 10:33:03 +11:00
Damien George
3f39d18c2b all: Add *FORMAT-OFF* in various places.
This string is recognised by uncrustify, to disable formatting in the
region marked by these comments.  This is necessary in the qstrdef*.h files
to prevent modification of the strings within the Q(...).  In other places
it is used to prevent excessive reformatting that would make the code less
readable.
2020-02-28 10:31:07 +11:00
Damien George
73670ef281 unix/unix_mphal: Adjust #if in mp_hal_stdin_rx_chr to improve format. 2020-02-28 10:30:28 +11:00
Damien George
410757f4f4 unix/mphalport.h: Fix build when MICROPY_USE_READLINE=0.
If the built-in input() is enabled (which it is by default) then it needs
some form of readline, so supply it with one when MICROPY_USE_READLINE=0.

Fixes issue #5658.
2020-02-20 00:45:58 +11:00
David Lechner
4adcaa4423 unix/mpthreadport: Fix Mac build by using SIGUSR1 if SIGRTMIN not avail.
Some platforms, like Apple, don't define SIGRTMIN, so fall back to SIGUSR1
in such a case.

Fixes #5659.
2020-02-20 00:30:08 +11:00
David Lechner
3bd2ae1a36 unix/mpthreadport: Use SIGRTMIN+5 instead of SIGUSR1 for thread-GC.
This changes the signal used to trigger garbage collection from SIGUSR1 to
SIGRTMIN + 5.  SIGUSR1 is quite common compared to SIGRTMIN (measured by
google search results) and is more likely to conflict with libraries that
may use the same signal.

POSIX specifies that there are at least 8 real-time signal so 5 was chosen
as a "random" number to further avoid potential conflict with libraries
that may use SIGRTMIN or SIGRTMAX.

Also, if we ever have a `usignal` module, it would be nice to leave SIGUSR1
and SIGUSR2 free for user programs.
2020-02-18 13:32:42 +11:00
David Lechner
4af79e7694 unix/Makefile: Allow to install all variants of the executable.
The install target is current broken when PROG is used to override the
default executable name.  This fixes it by removing the redundant TARGET
variable and uses PROG directly instead.

The install and uninstall targets are also moved to the common unix
Makefile so that all variants can be installed in the same way.
2020-02-16 23:37:40 +11:00
David Lechner
c5f4268c99 unix/variants/standard: Fix role of PREFIX when used to install.
Currently it is not possible to override PREFIX when installing micropython
using the makefile.  It is common practice to be able to run something like
this:

    $ make install PREFIX=/usr DESTDIR=/tmp/staging

This fixes such usage.
2020-02-16 23:35:52 +11:00
Damien George
baf11f237b unix/Makefile: Remove old variant targets that are no longer needed.
To eliminate confusion about what targets to use when building.
2020-02-16 00:15:57 +11:00
Damien George
ad7213d3c3 py: Add mp_raise_msg_varg helper and use it where appropriate.
This commit adds mp_raise_msg_varg(type, fmt, ...) as a helper for
nlr_raise(mp_obj_new_exception_msg_varg(type, fmt, ...)).  It makes the
C-level API for raising exceptions more consistent, and reduces code size
on most ports:

   bare-arm:   +28 +0.042%
minimal x86:  +100 +0.067%
   unix x64:   -56 -0.011%
unix nanbox:  -300 -0.068%
      stm32:  -204 -0.054% PYBV10
     cc3200:    +0 +0.000%
    esp8266:   -64 -0.010% GENERIC
      esp32:  -104 -0.007% GENERIC
        nrf:  -136 -0.094% pca10040
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
2020-02-13 11:52:40 +11:00
stijn
8b6e6008c7 unix/main: Use OS-dependent path separator when searching path. 2020-02-11 13:33:56 +11:00
Damien George
046ae80bdf unix, windows: Use mp_keyboard_interrupt instead of custom code.
The mp_keyboard_interrupt() function does exactly what is needed here, and
using it gets ctrl-C working when MICROPY_ENABLE_SCHEDULER is enabled on
these ports (and MICROPY_ASYNC_KBD_INTR is disabled).
2020-02-07 16:08:33 +11:00
Damien George
eaf30c516a tests/unix: Add coverage tests for kbd-intr and scheduler. 2020-02-07 16:08:29 +11:00
Damien George
5a91cd9ff3 lib/utils/pyexec: Handle pending exceptions after disabling kbd intrs.
Pending exceptions would otherwise be handled later on where there may not
be an NLR handler in place.

A similar fix is also made to the unix port's REPL handler.

Fixes issues #4921 and #5488.
2020-02-07 16:08:26 +11:00
Damien George
98a3911c43 py/scheduler: Add "raise_exc" argument to mp_handle_pending.
Previous behaviour is when this argument is set to "true", in which case
the function will raise any pending exception.  Setting it to "false" will
cancel any pending exception.
2020-02-07 16:08:20 +11:00
David Lechner
f1b6e6bb15 unix/modos: Implement putenv and unsetenv to complement getenv.
CPython also has os.environ, which should be used instead of os.getenv()
due to caching in the os.environ mapping.  But for MicroPython it makes
sense to only implement the basic underlying methods, ie getenv/putenv/
unsetenv.
2020-02-04 17:54:31 +11:00
David Lechner
83439e38fc unix/main: Add command-line -h option for printing help text.
This adds a -h option to print the usage help text and adds a new, shorter
error message that is printed when invalid arguments are given.  This
behaviour follows CPython (and other tools) more closely.
2020-02-04 17:54:31 +11:00
David Lechner
5a63bc5a44 unix/main: Add #if guard around -v option usage and document -i/-m opts.
This commit modifies the usage() function to only print the -v option help
text when MICROPY_DEBUG_PRINTERS is enabled.  The -v option requires this
build option to be enabled for it to have any effect.

The usage text is also modified to show the -i and -m options, and also
show that running a command, module or file are mutually exclusive.
2020-02-04 17:53:35 +11:00
David Lechner
122baa6787 unix/main: Add support for MICROPYINSPECT environment variable.
This adds support for a MICROPYINSPECT environment variable that works
exactly like PYTHONINSPECT; per CPython docs:

    If this is set to a non-empty string it is equivalent to specifying the
    -i option.

    This variable can also be modified by Python code using os.environ to
    force inspect mode on program termination.
2020-02-04 17:52:58 +11:00
David Lechner
4ab8bee82f unix/main: Print usage and NLR errors to stderr instead of stdout.
When stdout is redirected it is useful to have errors printed to stderr
instead of being redirected.

mp_stderr_print() can't be used in these two instances since the
MicroPython runtime is not running so we use fprintf(stderr) instead.
2020-02-01 22:44:08 +11:00
David Lechner
b72cb0ca1b py/mpthread.h: Use strong type for mp_thread_set_state() argument.
This modifies the signature of mp_thread_set_state() to use
mp_state_thread_t* instead of void*.  This matches the return type of
mp_thread_get_state(), which returns the same value.

`struct _mp_state_thread_t;` had to be moved before
`#include <mpthreadport.h>` since the stm32 port uses it in its
mpthreadport.h file.
2020-01-29 17:10:32 +11:00
David Lechner
d89ed3e62b unix/unix_mphal: Add compile check for incompatible GIL+ASYNC_KBD_INTR.
It is not safe to enable MICROPY_ASYNC_KBD_INTR and MICROPY_PY_THREAD_GIL
at the same time.  This will trigger a compiler error to ensure that it
is not possible to make this mistake.
2020-01-26 23:31:27 +11:00
David Lechner
fee7e5617f unix: Release GIL during all system calls.
Addition of GIL EXIT/ENTER pairs are:

- modos: release the GIL during system calls.  CPython does this as well.

- moduselect: release the GIL during the poll() syscall.  This call can be
  blocking, so it is important to allow other threads to run at this time.

- modusocket: release the GIL during system calls.  Many of these calls can
  be blocking, so it is important to allow other threads to run.

- unix_mphal: release the GIL during the read and write syscalls in
  mp_hal_stdin_rx_chr and mp_hal_stdout_tx_strn.  If we don't do this
  threads are blocked when the REPL or the builtin input function are used.

- file, main, mpconfigport.h: release GIL during syscalls in built-in
  functions that could block.
2020-01-26 23:21:29 +11:00
Damien George
96716b46e1 unix/Makefile: Reserve CFLAGS_EXTRA/LDFLAGS_EXTRA for external use.
When CFLAGS_EXTRA/LDFLAGS_EXTRA (or anything) is set on the command line of
a make invocation then it will completely override any setting or appending
of these variables in the makefile(s).  This means builds like the coverage
variant will have their mpconfigvariant.mk settings overridden.  Fix this
by using CFLAGS/LDFLAGS exclusively in the makefile(s), reserving the
CFLAGS_EXTRA/LDFLAGS_EXTRA variables for external command-line use only.
2020-01-24 11:51:21 +11:00
Yonatan Goldschmidt
35e664d779 tests/unix: Add coverage tests for mp_obj_is_type() and variants. 2020-01-24 10:57:17 +11:00
Damien George
27f41e624c tests/unix: Add coverage test for mp_obj_new_exception_args.
Because it's no longer called anywhere in the code.
2020-01-23 13:37:25 +11:00
Damien George
dccace6f3f tests/unix: Add coverage tests for pairheap data structure. 2020-01-22 17:31:18 +11:00
Damien George
3448e69c2d tests/unix: Add coverage test for new mp_obj_int_get_uint_checked func. 2020-01-14 23:45:56 +11:00
Yonatan Goldschmidt
853aaa06f2 lib/mp-readline: Add word-based move/delete EMACS key sequences.
This commit adds backward-word, backward-kill-word, forward-word,
forward-kill-word sequences for the REPL, with bindings to Alt+F, Alt+B,
Alt+D and Alt+Backspace respectively.  It is disabled by default and can be
enabled via MICROPY_REPL_EMACS_WORDS_MOVE.

Further enabling MICROPY_REPL_EMACS_EXTRA_WORDS_MOVE adds extra bindings
for these new sequences: Ctrl+Right, Ctrl+Left and Ctrl+W.

The features are enabled on unix micropython-coverage and micropython-dev.
2020-01-12 13:09:27 +11:00
Jim Mussared
977b532c8f unix: Rename unix binaries to micropython-variant (not _variant).
For consistency with mpy-cross, and other unix tools in general.
2020-01-12 10:37:40 +11:00
Jim Mussared
2357338e93 unix: Add placeholder DEV variant with settrace enabled.
This will eventually become the "full featured" unix binary with more
features enabled, specifically useful for development and testing.
2020-01-12 10:37:23 +11:00
Jim Mussared
bd2fff6687 unix: Add build variants, analogous to boards on bare-metal.
Invoking "make" will still build the standard "micropython" executable, but
other variants are now build using, eg, "make VARIANT=minimal".  This
follows how bare-metal ports specify a particular board, and allows running
any make target (eg clean, test) with any variant.

Convenience targets (eg "make coverage") are provided to retain the old
behaviour, at least for now.

See issue #3043.
2020-01-12 10:34:23 +11:00
Nicko van Someren
4c93955b7b py/objslice: Add support for indices() method on slice objects.
Instances of the slice class are passed to __getitem__() on objects when
the user indexes them with a slice.  In practice the majority of the time
(other than passing it on untouched) is to work out what the slice means in
the context of an array dimension of a particular length.  Since Python 2.3
there has been a method on the slice class, indices(), that takes a
dimension length and returns the real start, stop and step, accounting for
missing or negative values in the slice spec.  This commit implements such
a indices() method on the slice class.

It is configurable at compile-time via MICROPY_PY_BUILTINS_SLICE_INDICES,
disabled by default, enabled on unix, stm32 and esp32 ports.

This commit also adds new tests for slice indices and for slicing unicode
strings.
2019-12-28 23:55:15 +11:00
Damien George
269c9a08b6 unix/modos: Add uos.rename and uos.rmdir.
The existing uos.remove cannot be used to remove directories, instead
uos.rmdir is needed.  And also provide uos.rename to get a good set of
filesystem functionality without requiring additional Python-level os
functions (eg using ffi).
2019-12-28 11:54:49 +11:00
Andrew Leech
1b844e908c unix/modtime: Add utime.mktime function, to complement utime.localtime.
This also adds it to the windows port.
2019-12-28 11:11:54 +11:00
Damien George
09376f0e47 py: Introduce MP_ROM_NONE macro for ROM to refer to None object.
This helps to prevent mistakes, and allows easily changing the ROM value of
None if needed.
2019-12-27 22:51:17 +11:00
Laurens Valk
2679c9e116 unix/modtermios: Fix output speed setter in tcsetattr.
The input speed was being set twice and the output speed was not set.
2019-11-21 12:10:32 +11:00
Damien George
799b6d1e0c extmod: Consolidate FAT FS config to MICROPY_VFS_FAT across all ports.
This commit removes the Makefile-level MICROPY_FATFS config and moves the
MICROPY_VFS_FAT config to the Makefile level to replace it.  It also moves
the include of the oofatfs source files in the build from each port to a
central place in extmod/extmod.mk.

For a port to enabled VFS FAT support it should now set MICROPY_VFS_FAT=1
at the level of the Makefile.  This will include the relevant oofatfs files
in the build and set MICROPY_VFS_FAT=1 at the C (preprocessor) level.
2019-11-11 11:37:38 +11:00
Damien George
660a61a388 extmod/vfs_lfs: Allow compiling in VfsLfs1 and VfsLfs2 separately.
These classes are enabled via the config options MICROPY_VFS_LFS1 and
MICROPY_VFS_LFS2, which are disabled by default.
2019-10-30 12:08:58 +11:00
Damien George
4847460232 unix/mphalport.h: Define mp_hal_stdio_poll to dummy because it's unused.
And requires uintptr_t to declare the default version in py/mphal.h.
2019-10-29 22:22:37 +11:00
Damien George
62d5659cdd unix: Enable uos.VfsLfs1, uos.VfsLfs2 on coverage build. 2019-10-29 14:17:29 +11:00
Damien George
1582c7eeb0 unix,windows: Enable module weak links. 2019-10-22 16:23:43 +11:00
Jim Mussared
93bd61ca91 unix: Allow building without a manifest. 2019-10-21 23:21:18 +11:00
Jim Mussared
8ba963cfa3 tools/makemanifest.py: Eval relative paths w.r.t. current manifest file.
When loading a manifest file, e.g. by include(), it will chdir first to the
directory of that manifest.  This means that all file operations within a
manifest are relative to that manifest's location.

As a consequence of this, additional environment variables are needed to
find absolute paths, so the following are added: $(MPY_LIB_DIR),
$(PORT_DIR), $(BOARD_DIR).  And rename $(MPY) to $(MPY_DIR) to be
consistent.

Existing manifests are updated to match.
2019-10-21 23:01:41 +11:00
Damien George
b1c0355b93 unix: Convert to use FROZEN_MANIFEST to specify frozen code.
Removes symlinks in modules directory, all frozen code is now specified by
manifest.py.
2019-10-15 21:35:27 +11:00
Damien George
d7a9388fe0 ports: Add new make target "submodules" which inits required modules. 2019-10-15 17:14:41 +11:00
Damien George
4c5e1a0368 py/bc: Change mp_code_state_t.exc_sp to exc_sp_idx.
Change from a pointer to an index, to make space in mp_code_state_t.
2019-10-01 12:26:22 +10:00
Jim Mussared
f67fd95f8d unix/coverage: Add coverage tests for ringbuf. 2019-10-01 09:51:02 +10:00
Josh Lloyd
7d58a197cf py: Rename MP_QSTR_NULL to MP_QSTRnull to avoid intern collisions.
Fixes #5140.
2019-09-26 16:04:56 +10:00
Milan Rossa
310b3d1b81 py: Integrate sys.settrace feature into the VM and runtime.
This commit adds support for sys.settrace, allowing to install Python
handlers to trace execution of Python code.  The interface follows CPython
as closely as possible.  The feature is disabled by default and can be
enabled via MICROPY_PY_SYS_SETTRACE.
2019-08-30 16:44:12 +10:00
Damien George
af20c2ead3 py: Add global default_emit_opt variable to make emit kind persistent.
mp_compile no longer takes an emit_opt argument, rather this setting is now
provided by the global default_emit_opt variable.

Now, when -X emit=native is passed as a command-line option, the emitter
will be set for all compiled modules (included imports), not just the
top-level script.

In the future there could be a way to also set this variable from a script.

Fixes issue #4267.
2019-08-28 12:47:58 +10:00
Damien George
15b36aa0af unix/main: Only accept full emit cmd-line options if native enabled. 2019-08-28 12:47:58 +10:00
Damien George
7d851a27f1 extmod/modure: Make regex dump-code debugging feature optional.
Enabled via MICROPY_PY_URE_DEBUG, disabled by default (but enabled on unix
coverage build).  This is a rarely used feature that costs a lot of code
(500-800 bytes flash).  Debugging of regular expressions can be done
offline with other tools.
2019-08-19 16:43:00 +10:00
stijn
af5c998f37 py/modmath: Implement math.isclose() for non-complex numbers.
As per PEP 485, this function appeared in for Python 3.5.  Configured via
MICROPY_PY_MATH_ISCLOSE which is disabled by default, but enabled for the
ports which already have MICROPY_PY_MATH_SPECIAL_FUNCTIONS enabled.
2019-08-17 23:23:17 +10:00
Damien George
497683b315 gitignore: Put build-*/ pattern in top-level gitignore file. 2019-08-16 00:08:08 +10:00
Milan Rossa
6f0c6bd774 unix: Enable sys.atexit, triggered after the main script ends. 2019-08-15 17:31:04 +10:00
Paul m. p. P
42d30c5baf unix/unix_mphal: Include time.h for CLOCK_MONOTONIC. 2019-07-09 13:05:59 +10:00
Mikhail Zakharov
ced340d739 unix/unix_mphal: Use CLOCK_MONOTONIC for ticks_ms/us when available. 2019-06-26 11:07:45 +10:00
Damien George
9d72f07b6d unix/mpconfigport.mk: Update comment about TLS implementations.
As long as the submodule is checked out, mbedTLS is now fully integrated
into the unix build if MICROPY_SSL_MBEDTLS=1.
2019-06-05 15:28:30 +10:00
Damien George
8c9758ff2e unix/modusocket: Raise ETIMEDOUT when connect or accept has timeout. 2019-05-28 17:22:54 +10:00
Damien George
653e1756c0 various: Update early copyright years to match actual edit history. 2019-05-17 18:06:11 +10:00
Paul Sokolovsky
016d9a40fe various: Add and update my copyright line based on git history.
For modules I initially created or made substantial contributions to.
2019-05-17 18:04:15 +10:00
stijn
90fae9172a py/objarray: Add support for memoryview.itemsize attribute.
This allows figuring out the number of bytes in the memoryview object as
len(memview) * memview.itemsize.

The feature is enabled via MICROPY_PY_BUILTINS_MEMORYVIEW_ITEMSIZE and is
disabled by default.
2019-05-14 17:15:17 +10:00
Elad Namdar
3f54462add unix/modusocket: Fix use of setsockopt in usocket.settimeout impl.
The original code called setsockopt(SO_RCVTIMEO/SO_SNDTIMEO) with NULL
timeout structure argument, which is an illegal usage of that function.
The old code also didn't validate the return value of setsockopt, missing
the bug completely.
2019-05-08 13:12:30 +10:00
Yonatan Goldschmidt
ef9843653b extmod/moducryptolib: Add AES-CTR support.
Selectable at compile time via MICROPY_PY_UCRYPTOLIB_CTR.  Disabled by
default.
2019-05-06 18:09:48 +10:00
Damien George
906fb89fd7 unix/coverage: Add test for printing literal % character. 2019-05-03 23:21:28 +10:00
stijn
34a7d7ebeb unix/gcollect: Make sure stack/regs get captured properly for GC.
When building with link time optimization enabled it is possible both
gc_collect() and gc_collect_regs_and_stack() get inlined into gc_alloc()
which can result in the regs variable being pushed on the stack earlier
than some of the registers. Depending on the calling convention, those
registers might however contain pointers to blocks which have just been
allocated in the caller of gc_alloc(). Then those pointers end up higher on
the stack than regs, aren't marked by gc_collect_root() and hence get
sweeped, even though they're still in use.

As reported in #4652 this happened for in 32-bit msvc release builds:
mp_lexer_new() does two consecutive allocations and the latter triggered a
gc_collect() which would sweep the memory of the first allocation again.
2019-05-01 15:06:21 +10:00
Damiano Mazzella
0b86ba565c unix/mpthreadport: Use named semaphores on Mac OS X.
Unnamed semaphores (via sem_init) are not supported on this OS.  See #4465.
2019-03-27 10:50:01 +11:00
Damien George
c6a9bb23cd unix/Makefile: Update coverage tests to match those in Travis. 2019-03-08 16:51:09 +11:00
Damien George
b5f33ac2cb ports: Update to work with new oofatfs version. 2019-03-05 15:56:39 +11:00
Damien George
55ff562c70 unix/modffi: Eliminate unused-argument warning when debugging disabled. 2019-02-25 14:53:17 +11:00
Yonatan Goldschmidt
bc4f8b438b extmod/moduwebsocket: Refactor websocket to uwebsocket.
As mentioned in #4450, `websocket` was experimental with a single intended
user, `webrepl`. Therefore, we'll make this change without a weak
link `websocket` -> `uwebsocket`.
2019-02-14 00:35:45 +11:00
Yonatan Goldschmidt
66f0afc91d unix/modmachine: Handle repeated /dev/mem open errors.
If opening of /dev/mem has failed an `OSError` is appropriately raised, but
the next time `mem8/16/32` is accessed the invalid file descriptor is used
and the program gets a SIGSEGV.
2019-02-12 15:29:11 +11:00
Damien George
6e30f96b0b ports: Convert legacy uppercase macro names to lowercase. 2019-02-12 14:54:51 +11:00
Paul Sokolovsky
2f5d113fad py/warning: Support categories for warnings.
Python defines warnings as belonging to categories, where category is a
warning type (descending from exception type). This is useful, as e.g.
allows to disable warnings selectively and provide user-defined warning
types.  So, implement this in MicroPython, except that categories are
represented just with strings.  However, enough hooks are left to implement
categories differently per-port (e.g. as types), without need to patch each
and every usage.
2019-01-31 16:48:30 +11:00
stijn
42863830be py: Add optional support for 2-argument version of built-in next().
Configurable via MICROPY_PY_BUILTINS_NEXT2, disabled by default.
2019-01-27 13:01:28 +11:00
Mikhail Zakharov
18723e9889 unix/mpthreadport: Remove busy wait loop in thread garbage collection.
One can't use pthread calls in a signal handler because they are not
async-signal-safe (see man signal-safety).  Instead, sem_post can be used
to post from within a signal handler and this should be more efficient than
using a busy wait loop, waiting on a volatile variable.
2019-01-27 12:40:10 +11:00
Mikhail Zakharov
1e7b422226 unix/mpthreadport: Cleanup used memory on thread exit. 2019-01-27 12:39:45 +11:00
Mikhail Zakharov
f8c1be85d1 unix/mpthreadport: Add thread deinit code to stop threads on exit.
Free unused memory for threads and cancel any outstanding threads on
interpreter exit to avoid possible segmentaiton fault.
2019-01-27 12:38:23 +11:00
Paul Sokolovsky
80aca27a40 unix/modos: Rename unlink to remove to be consistent with other ports.
We standardized to provide uos.remove() as a more obvious and user-friendly
name.  That's what written in the docs.  The Unix port implementation
predates this convention, so update it now.
2018-11-26 23:27:04 +11:00
Paul m. p. P
454cca6016 py/objmodule: Implement PEP 562's __getattr__ for modules.
Configurable via MICROPY_MODULE_GETATTR, disabled by default.  Among other
things __getattr__ for modules can help to build lazy loading / code
unloading at runtime.
2018-10-23 11:22:50 +11:00
Paul Sokolovsky
5a91fce9f8 py/objstr: Make str.count() method configurable.
Configurable via MICROPY_PY_BUILTINS_STR_COUNT.  Default is enabled.
Disabled for bare-arm, minimal, unix-minimal and zephyr ports.  Disabling
it saves 408 bytes on x86.
2018-10-22 22:49:05 +11:00
Paul Sokolovsky
6ddcfe68b8 unix/Makefile: Allow to override/omit pthread lib name.
For example, on Android, pthread functions are part of libc, so LIBPTHREAD
should be empty.
2018-10-19 17:22:37 +11:00
Paul Sokolovsky
6c5b2bded2 unix/modffi: Add support for "q"/"Q" specs (int64_t/uint64_t). 2018-10-17 15:17:05 +11:00
Paul Sokolovsky
0c18633ea9 unix/modusocket: Finish socket.settimeout() implementation.
1. Return correct error code for non-blocking vs timed out socket
(POSIX returns EAGAIN for both, we want ETIMEDOUT in case of timed
out socket). To achieve this, blocking/non-blocking flag is added
to the mp_obj_socket_t, to avoid issuing fcntl() syscall each time
EAGAIN occurs. (mp_obj_socket_t used to be 8 bytes, having some room
in a standard 16-byte alloc block.)

2. Handle socket.settimeout(0) properly - in Python, that means
non-blocking mode, but SO_RCVTIMEO/SO_SNDTIMEO of 0 is infinite
timeout.

3. Overall, make sure that socket.settimeout() call switches blocking
state as expected.
2018-10-17 14:19:06 +11:00
Danielle Madeley
80a25810f9 unix/modusocket: Initial implementation of socket.settimeout(). 2018-10-17 14:19:06 +11:00
Paul Sokolovsky
b9bad7ff92 unix/moduselect: Raise OSError(ENOENT) if obj to modify is not in poller
Previously, the function silently succeeded. The new behavior is consistent
with both baremetal uselect implementation and CPython 3.
2018-10-05 16:56:43 +10:00
Paul Sokolovsky
af2030dec6 unix/mpconfigport.h: Enable MICROPY_PY_UHASHLIB_MD5 for uhashlib.md5.
This will allow to e.g. implement HTTP Digest authentication.

Adds 540 bytes for x86_32, 332 for arm_thumb2 (for Unix port, which already
includes axTLS library).
2018-09-26 15:12:28 +10:00
Christopher Swenson
8c656754aa py/modmath: Add math.factorial, optimised and non-opt implementations.
This commit adds the math.factorial function in two variants:
- squared difference, which is faster than the naive version, relatively
  compact, and non-recursive;
- a mildly optimised recursive version, faster than the above one.

There are some more optimisations that could be done, but they tend to take
more code, and more storage space.  The recursive version seems like a
sensible compromise.

The new function is disabled by default, and uses the non-optimised version
by default if it is enabled.  The options are MICROPY_PY_MATH_FACTORIAL
and MICROPY_OPT_MATH_FACTORIAL.
2018-09-26 15:03:04 +10:00
Damien George
6623d7a88c unix/modjni: Get building under coverage and nanbox builds.
Changes made:
- make use of MP_OBJ_TO_PTR and MP_OBJ_FROM_PTR where necessary
- fix shadowing of index variable i, renamed to j
- fix type of above variable to size_t to prevent comparison warning
- fix shadowing of res variable
- use "(void)" instead of "()" for functions that take no arguments
2018-09-20 16:04:08 +10:00
Paul Sokolovsky
11573fcabd unix/modjni: Update .getiter signature to include mp_obj_iter_buf_t* .
And thus be buildable again.
2018-09-20 16:03:53 +10:00
Paul Sokolovsky
2da5d41350 py/objstr: Make % (__mod__) formatting operator configurable.
Default is enabled, disabled for minimal builds. Saves 1296 bytes on x86,
976 bytes on ARM.
2018-09-20 14:41:08 +10:00
Paul Sokolovsky
064b8e0e8d unix/modos: Include extmod/vfs.h for MP_S_IFDIR, etc.
If DTTOIF() macro is not defined, the code refers to MP_S_IFDIR, etc.
symbols defined in extmod/vfs.h, so should include it.

This fixes build for Android.
2018-09-14 13:22:50 +10:00
Damien George
e6a6ded74e unix/mpconfigport_coverage.h: Enable uhashlib.md5. 2018-09-12 16:09:41 +10:00
Paul Sokolovsky
5615273bb0 unix/Makefile: Build libffi inside $BUILD.
Avoids polluting the source tree, allows to build for different (sub)archs
without intermediate cleaning.
2018-09-10 11:34:46 +03:00
Damien George
6ad5355e43 unix/Makefile: Remove building of libaxtls.a which is no longer needed. 2018-09-08 00:07:23 +10:00
Damien George
01ce2e1682 unix/Makefile: Enable ussl module with nanbox build. 2018-08-14 21:53:06 +10:00
Paul Sokolovsky
fe1ef507ef unix/Makefile: coverage: Explicitly build "axtls" too.
"coverage" build uses different BUILD directory comparing to the normal
build. Previously, any build picked up libaxtls.a from normal build's
directory, but that was fixed recently. So, for each build, we must
build axtls explicitly.

This fixes Travis build in particular.
2018-08-14 15:10:52 +10:00
Damien George
da2d2b6d88 py/mpconfig.h: Introduce MICROPY_DEBUG_PRINTER for debugging output.
This patch in effect renames MICROPY_DEBUG_PRINTER_DEST to
MICROPY_DEBUG_PRINTER, moving its default definition from
lib/utils/printf.c to py/mpconfig.h to make it official and documented, and
makes this macro a pointer rather than the actual mp_print_t struct.  This
is done to get consistency with MICROPY_ERROR_PRINTER, and provide this
macro for use outside just lib/utils/printf.c.

Ports are updated to use the new macro name.
2018-08-02 14:04:44 +10:00
Damien George
ef554ef9a2 unix: Use MP_STREAM_GET_FILENO to allow uselect to poll general objects.
This mechanism will scale to to an arbitrary number of pollable objects, so
long as they implement the MP_STREAM_GET_FILENO ioctl.  Since ussl objects
pass through ioctl requests transparently to the underlying socket object,
it will allow ussl sockets to be polled.  And a user object with uio.IOBase
as a base could support polling.
2018-07-20 13:09:49 +10:00
Damien George
3ab2f3fb2b unix/modos: Convert dir-type to stat-type for file type in ilistdir.
Fixes issue #3931.
2018-07-11 16:06:16 +10:00
Damien George
79d5e3abb3 unix/mpconfigport_coverage: Enable ure groups, span, start, end and sub. 2018-07-02 14:55:05 +10:00
Paul Sokolovsky
771911028c unix/mpconfigport.h: Enable MICROPY_PY_UCRYPTOLIB. 2018-06-27 14:56:59 +10:00
Paul Sokolovsky
567bc2d6ce extmod/moducryptolib: Add ucryptolib module with crypto functions.
The API follows guidelines of https://www.python.org/dev/peps/pep-0272/,
but is optimized for code size, with the idea that full PEP 0272
compatibility can be added with a simple Python wrapper mode.

The naming of the module follows (u)hashlib pattern.

At the bare minimum, this module is expected to provide:

* AES128, ECB (i.e. "null") mode, encrypt only

Implementation in this commit is based on axTLS routines, and implements
following:

* AES 128 and 256
* ECB and CBC modes
* encrypt and decrypt
2018-06-27 14:54:40 +10:00
Damien George
a5f5552a0a tests/unix/extra_coverage: Don't test stream objs with NULL write fun.
This behaviour of a NULL write C method on a stream that uses the write
adaptor objects is no longer supported.  It was only ever used by the
coverage build for testing the fail path of mp_get_stream_raise().
2018-06-18 12:35:56 +10:00
Damien George
565f590586 ports: Enable IOBase on unix, stm32, esp8266 and esp32.
It's a core feature, in particular required for user-streams with uasyncio.
2018-06-12 12:29:26 +10:00
Damien George
b2fa1b50ed ports: Call gc_sweep_all() when doing a soft reset.
This calls finalisers of things like files and sockets to cleanly close
them.
2018-06-12 11:56:25 +10:00
Damien George
fadd6bbe43 unix/moduos_vfs: Add missing uos functions from traditional uos module.
Now that the coverage build has fully switched to the VFS sub-system these
functions were no longer available, so add them to the uos_vfs module.

Also, vfs_open is no longer needed, it's available as the built-in open.
2018-06-06 14:28:23 +10:00
Damien George
1d40f12e44 unix: Support MICROPY_VFS_POSIX and enable it in coverage build.
The unix coverage build is now switched fully to the VFS implementation, ie
the uos module is the uos_vfs module.  For example, one can now sandbox uPy
to their home directory via:

    $ ./micropython_coverage

    >>> import uos
    >>> uos.umount('/') # unmount existing root VFS
    >>> vfs = uos.VfsPosix('/home/user') # create new POSIX VFS
    >>> uos.mount(vfs, '/') # mount new POSIX VFS at root

Some filesystem/OS features may no longer work with the coverage build due
to this change, and these need to be gradually fixed.

The standard unix port remains unchanged, it still uses the traditional uos
module which directly accesses the underlying host filesystem.
2018-06-06 14:28:23 +10:00