Commit Graph

9523 Commits

Author SHA1 Message Date
Damien George
d11fb09333 extmod/modussl_axtls: Fix __del__ to point to mp_stream_close_obj. 2018-06-13 13:16:21 +10:00
Damien George
035906419d extmod/uos_dupterm: Use native C stream methods on dupterm object.
This patch changes dupterm to call the native C stream methods on the
connected stream objects, instead of calling the Python readinto/write
methods.  This is much more efficient for native stream objects like UART
and webrepl and doesn't require allocating a special dupterm array.

This change is a minor breaking change from the user's perspective because
dupterm no longer accepts pure user stream objects to duplicate on.  But
with the recent addition of uio.IOBase it is possible to still create such
classes just by inheriting from uio.IOBase, for example:

    import uio, uos

    class MyStream(uio.IOBase):
        def write(self, buf):
            # existing write implementation
        def readinto(self, buf):
            # existing readinto implementation

    uos.dupterm(MyStream())
2018-06-12 15:06:11 +10:00
Damien George
5042d98514 stm32/Makefile: Rebuild all qstrs when any board configuration changes. 2018-06-12 13:53:43 +10:00
Damien George
7ad04d17da py/mkrules.mk: Regenerate all qstrs when config files change.
A port can define QSTR_GLOBAL_DEPENDENCIES to add extra files.
2018-06-12 13:53:43 +10:00
Damien George
0501427907 esp32: Remove port-specific uhashlib implementation and use common one.
Now that the common module has mbedtls support for both SHA1 and SHA256 it
can now be used on this port.
2018-06-12 13:50:11 +10:00
Yonatan Goldschmidt
c2fb725e72 extmod/moduhashlib: Make function objects STATIC.
These are not exported to anyone anyway.
2018-06-12 13:50:11 +10:00
Yonatan Goldschmidt
6963ee9075 extmod/moduhashlib: Allow using the sha256 implementation of mbedTLS. 2018-06-12 13:50:11 +10:00
Yonatan Goldschmidt
6630354ffe extmod/moduhashlib: Allow to disable the sha256 class.
Via the config value MICROPY_PY_UHASHLIB_SHA256.  Default to enabled to
keep backwards compatibility.

Also add default value for the sha1 class, to at least document its
existence.
2018-06-12 13:50:11 +10:00
Yonatan Goldschmidt
38682d4629 extmod/moduhashlib: Reorder funcs so that they are grouped by hash type.
Makes the code much more readable by reducing the number of #ifdefs and
keeping related functions close.
2018-06-12 13:47:20 +10:00
Yonatan Goldschmidt
b045ebd354 extmod/moduhashlib: Prefix all Python methods and objects with uhashlib.
For consistency with other modules, and to help avoid clashes with the
actual underlying functions that do the hashing (eg
crypto-algorithms/sha256.c:sha256_update).
2018-06-12 13:44:09 +10:00
Damien George
c901cc6862 tests/extmod: Add test for VFS and user-defined filesystem and files. 2018-06-12 12:29:26 +10:00
Damien George
9144b1f10c tests/io: Add simple IOBase test. 2018-06-12 12:29:26 +10:00
Damien George
565f590586 ports: Enable IOBase on unix, stm32, esp8266 and esp32.
It's a core feature, in particular required for user-streams with uasyncio.
2018-06-12 12:29:26 +10:00
Damien George
af0932a779 py/modio: Add uio.IOBase class to allow to define user streams.
A user class derived from IOBase and implementing readinto/write/ioctl can
now be used anywhere a native stream object is accepted.

The mapping from C to Python is:

    stream_p->read  --> readinto(buf)
    stream_p->write --> write(buf)
    stream_p->ioctl --> ioctl(request, arg)

Among other things it allows the user to:

- create an object which can be passed as the file argument to print:
  print(..., file=myobj), and then print will pass all the data to the
  object via the objects write method (same as CPython)
- pass a user object to uio.BufferedWriter to buffer the writes (same as
  CPython)
- use select.select on a user object
- register user objects with select.poll, in particular so user objects can
  be used with uasyncio
- create user files that can be returned from user filesystems, and import
  can import scripts from these user files

For example:

    class MyOut(io.IOBase):
        def write(self, buf):
            print('write', repr(buf))
            return len(buf)

    print('hello', file=MyOut())

The feature is enabled via MICROPY_PY_IO_IOBASE which is disabled by
default.
2018-06-12 12:29:26 +10:00
Damien George
6a445b60fa py/lexer: Add support for underscores in numeric literals.
This is a very convenient feature introduced in Python 3.6 by PEP 515.
2018-06-12 12:17:43 +10:00
Damien George
b2fa1b50ed ports: Call gc_sweep_all() when doing a soft reset.
This calls finalisers of things like files and sockets to cleanly close
them.
2018-06-12 11:56:25 +10:00
Damien George
522ea80f06 py/gc: Add gc_sweep_all() function to run all remaining finalisers.
This patch adds the gc_sweep_all() function which does a garbage collection
without tracing any root pointers, so frees all the memory, and most
importantly runs any remaining finalisers.

This helps primarily for soft reset: it will close any open files, any open
sockets, and help to get the system back to a clean state upon soft reset.
2018-06-12 11:55:29 +10:00
Damien George
8fb95d6520 tools/pydfu.py: Increase download packet size to full 2048 bytes.
The ST DFU bootloader supports a transfer size up to 2048 bytes, so send
that much data on each download (to device) packet.  This almost halves
total download time.
2018-06-08 15:32:49 +10:00
Damien George
24c416cc66 stm32/mboot: Increase USB rx_buf and DFU buf sizes to full 2048 bytes.
The DFU USB config descriptor returns 0x0800=2048 for the supported
transfer size, and this applies to both TX (IN) and RX (OUT).  So increase
the rx_buf to support this size without having a buffer overflow on
received data.

With this patch mboot in USB DFU mode now works with dfu-util.
2018-06-08 15:29:52 +10:00
Glenn Moloney
039f196c56 esp32/modnetwork: Fix isconnected() when using static IP config.
Currently <WLAN>.isconnected() always returns True if a static IP is set,
regardless of the state of the connection.

This patch introduces a new flag 'wifi_sta_connected' which is set in
event_handler() when GOT_IP event is received and reset when DISCONNECTED
event is received (unless re-connect is successful).  isconnected() now
simply returns the status of this flag (for STA_IF).

The pre-existing flag misleadingly named 'wifi_sta_connected" is also
renamed to 'wifi_sta_connect_requested'.

Fixes issue #3837
2018-06-08 13:13:21 +10:00
Damien George
a12d046c42 tests/pyb: Make i2c and pyb1 pyboard tests run again.
For i2c.py: the accelerometer now uses the new I2C driver so need to
explicitly init the legacy i2c object to get the test working.

For pyb1.py: the legacy pyb.hid() call will crash if the USB_HID object is
not initialised.
2018-06-08 13:00:27 +10:00
Damien George
190c7dba89 stm32/mpconfigport.h: Enable DELATTR_SETATTR and BUILTINS_NOTIMPLEMENTED
MICROPY_PY_DELATTR_SETATTR can now be enabled without a performance hit for
classes that don't use this feature.

MICROPY_PY_BUILTINS_NOTIMPLEMENTED is a minor addition that improves
compatibility with CPython.
2018-06-08 12:55:18 +10:00
Damien George
93150a0d40 ports: Enable descriptors on stm32, esp8266, esp32 ports.
They are now efficient (in runtime performance) and provide a useful
feature that's hard to obtain without them enabled.

See issue #3644 and PR #3826 for background.
2018-06-08 12:23:08 +10:00
Damien George
36c1052183 py/objtype: Optimise instance get/set/del by skipping special accessors.
This patch is a code optimisation, trading text bytes for speed.  On
pyboard it's an increase of 0.06% in code size for a gain (in pystone
performance) of roughly 6.5%.

The patch optimises load/store/delete of attributes in user defined classes
by not looking up special accessors (@property, __get__, __delete__,
__set__, __setattr__ and __getattr_) if they are guaranteed not to exist in
the class.

Currently, if you do my_obj.foo() then the runtime has to do a few checks
to see if foo is a property or has __get__, and if so delegate the call.
And for stores things like my_obj.foo = 1 has to first check if foo is a
property or has __set__ defined on it.

Doing all those checks each and every time the attribute is accessed has a
performance penalty.  This patch eliminates all those checks for cases when
it's guaranteed that the checks will always fail, ie no attributes are
properties nor have any special accessor methods defined on them.

To make this guarantee it checks all attributes of a user-defined class
when it is first created.  If any of the attributes of the user class are
properties or have special accessors, or any of the base classes of the
user class have them, then it sets a flag in the class to indicate that
special accessors must be checked for.  Then in the load/store/delete code
it checks this flag to see if it can take the shortcut and optimise the
lookup.

It's an optimisation that's pretty widely applicable because it improves
lookup performance for all methods of user defined classes, and stores of
attributes, at least for those that don't have special accessors.  And, it
allows to enable descriptors with minimal additional runtime overhead if
they are not used for a particular user class.

There is one restriction on dynamic class creation that has been introduced
by this patch: a user-defined class cannot go from zero special accessors
to one special accessor (or more) after that class has been subclassed.  If
the script attempts this an AttributeError is raised (see addition to
tests/misc/non_compliant.py for an example of this case).

The cost in code space bytes for the optimisation in this patch is:

   unix x64:  +528
unix nanbox:  +508
      stm32:  +192
     cc3200:  +200
    esp8266:  +332
      esp32:  +244

Performance tests that were done:

- on unix x86-64, pystone improved by about 5%
- on pyboard, pystone improved by about 6.5%, from 1683 up to 1794
- on pyboard, bm_chaos (from CPython benchmark suite) improved by about 5%
- on esp32, pystone improved by about 30% (but there are caching effects)
- on esp32, bm_chaos improved by about 11%
2018-06-08 12:12:08 +10:00
Damien George
bace1a16d0 py/objtype: Don't expose mp_obj_instance_attr().
mp_obj_is_instance_type() can be used instead to check for instance types.
2018-06-08 11:48:25 +10:00
Damien George
db5d8c97f1 py/obj.h: Introduce a "flags" entry in mp_obj_type_t. 2018-06-08 11:48:25 +10:00
Damien George
b789c640f7 travis: Install explicit version of urllib3 for coveralls.
Coveralls requires a "recent" version of urllib3, whereas requests requires
a "not so recent" version, less than 1.23.  So force urllib3 v1.22 to get
it all working.
2018-06-06 20:56:24 +10:00
Damien George
a8b9e71ac1 py/mpconfig.h: Add default MICROPY_VFS_FAT config value.
At least to document it's existence.
2018-06-06 14:33:42 +10:00
Damien George
c117effddd extmod/vfs: Introduce a C-level VFS protocol, with fast import_stat.
Following other C-level protocols, this VFS protocol is added to help
abstract away implementation details of the underlying VFS in an efficient
way.  As a starting point, the import_stat function is put into this
protocol so that the VFS sub-system does not need to know about every VFS
implementation in order to do an efficient stat for importing files.

In the future it might be worth adding other functions to this protocol.
2018-06-06 14:33:42 +10:00
Damien George
fadd6bbe43 unix/moduos_vfs: Add missing uos functions from traditional uos module.
Now that the coverage build has fully switched to the VFS sub-system these
functions were no longer available, so add them to the uos_vfs module.

Also, vfs_open is no longer needed, it's available as the built-in open.
2018-06-06 14:28:23 +10:00
Damien George
5ef0d2ab14 tests/extmod: Remove conditional import of uos_vfs, it no longer exists.
This conditional import was only used to get the tests working on the unix
coverage build, which has now switched to use VFS by default so the uos
module alone has the required functionality.
2018-06-06 14:28:23 +10:00
Damien George
6c02da2eec tests/extmod: Add test for importing a script from a user VFS. 2018-06-06 14:28:23 +10:00
Damien George
1d40f12e44 unix: Support MICROPY_VFS_POSIX and enable it in coverage build.
The unix coverage build is now switched fully to the VFS implementation, ie
the uos module is the uos_vfs module.  For example, one can now sandbox uPy
to their home directory via:

    $ ./micropython_coverage

    >>> import uos
    >>> uos.umount('/') # unmount existing root VFS
    >>> vfs = uos.VfsPosix('/home/user') # create new POSIX VFS
    >>> uos.mount(vfs, '/') # mount new POSIX VFS at root

Some filesystem/OS features may no longer work with the coverage build due
to this change, and these need to be gradually fixed.

The standard unix port remains unchanged, it still uses the traditional uos
module which directly accesses the underlying host filesystem.
2018-06-06 14:28:23 +10:00
Damien George
d4ce57e4e3 extmod/vfs: Add fast path for stating VfsPosix filesystem. 2018-06-06 14:28:23 +10:00
Damien George
a93144cb65 py/reader: Allow MICROPY_VFS_POSIX to work with MICROPY_READER_POSIX. 2018-06-06 14:28:23 +10:00
Damien George
8d82b0edbd extmod: Add VfsPosix filesystem component.
This VFS component allows to mount a host POSIX filesystem within the uPy
VFS sub-system.  All traditional POSIX file access then goes through the
VFS, allowing to sandbox a uPy process to a certain sub-dir of the host
system, as well as mount other filesystem types alongside the host
filesystem.
2018-06-06 14:28:23 +10:00
Damien George
f35aae366c extmod/vfs_fat: Rename FileIO/TextIO types to mp_type_vfs_fat_XXX.
So they don't clash with other VFS implementations.
2018-06-06 14:28:23 +10:00
Damien George
172c23fe5d extmod/vfs: Use u_rom_obj properly in argument structures. 2018-06-06 14:28:23 +10:00
Damien George
aace60a75e esp8266/modules/ntptime.py: Remove print of newly-set time.
It should be up to the user if they want to print the new time out or not.

Fixes issue #3766.
2018-06-05 14:30:35 +10:00
Damien George
a90124a9e2 esp32: Add support for building with external SPI RAM.
This patch adds support for building the firmware with external SPI RAM
enabled.  It is disabled by default because it adds overhead (due to
silicon workarounds) and reduces performance (because it's slower to have
bytecode and objects stored in external RAM).

To enable it, either use "make CONFIG_SPIRAM_SUPPORT=1", or add this line
to you custom makefile/GNUmakefile (before "include Makefile"):

    CONFIG_SPIRAM_SUPPORT = 1

When this option is enabled the MicroPython heap is automatically allocated
in external SPI RAM.

Thanks to Angus Gratton for help with the compiler and linker settings.
2018-06-05 13:57:59 +10:00
Angus Gratton
bc92206f89 esp32/Makefile: Extract common C & C++ flags for consistent compilation. 2018-06-05 13:05:12 +10:00
Damien George
df13ecde06 cc3200/mods: Include stream.h to get definition of mp_stream_p_t. 2018-06-04 16:58:45 +10:00
Damien George
1427f8f593 py/stream: Move definition of mp_stream_p_t from obj.h to stream.h.
Since a long time now, mp_obj_type_t no longer refers explicitly to
mp_stream_p_t but rather to an abstract "const void *protocol".  So there's
no longer any need to define mp_stream_p_t in obj.h and it can go with all
its associated definitions in stream.h.  Pretty much all users of this type
will already include the stream header.
2018-06-04 16:53:17 +10:00
Damien George
309fe39dbb stm32/modnetwork: Fix arg indexing in generic ifconfig method. 2018-06-03 21:50:49 +10:00
Damien George
7d86ac6c01 stm32: Add network driver for Wiznet5k using MACRAW mode and lwIP.
The Wiznet5k series of chips support a MACRAW mode which allows the host to
send and receive Ethernet frames directly.  This can be hooked into the
lwIP stack to provide a full "socket" implementation using this Wiznet
Ethernet device.  This patch adds support for this feature.

To enable the feature one must add the following to mpconfigboard.mk, or
mpconfigport.mk:

    MICROPY_PY_WIZNET5K = 5500

and the following to mpconfigboard.h, or mpconfigport.h:

    #define MICROPY_PY_LWIP (1)

After wiring up the module (X5=CS, X4=RST), usage on a pyboard is:

    import time, network
    nic = network.WIZNET5K(pyb.SPI(1), pyb.Pin.board.X5, pyb.Pin.board.X4)
    nic.active(1)
    while not nic.isconnected():
        time.sleep_ms(50) # needed to poll the NIC
    print(nic.ifconfig())

Then use the socket module as usual.

Compared to using the built-in TCP/IP stack on the Wiznet module, some
performance is lost in MACRAW mode: with a lot of memory allocated to lwIP
buffers, lwIP gives Around 750,000 bytes/sec max TCP download, compared
with 1M/sec when using the TCP/IP stack on the Wiznet module.
2018-06-01 14:21:38 +10:00
Damien George
5a5bc4a61f drivers/wiznet5k: Fix bug with MACRAW socket calculating packet size. 2018-06-01 13:44:09 +10:00
Damien George
d9f1ecece2 stm32/modnetwork: Provide generic implementation of ifconfig method.
All it needs is a lwIP netif to function.
2018-06-01 13:33:14 +10:00
Damien George
7437215ad7 stm32/modnetwork: Change base entry of NIC object from type to base.
mod_network_nic_type_t doesn't need to be an actual uPy type, it just needs
to be an object.
2018-06-01 13:31:28 +10:00
Damien George
6d87aa54d6 stm32/modnetwork: Don't take netif's down when network is deinited.
It should be up to the NIC itself to decide if the network interface is
removed upon soft reset.  Some NICs can keep the interface up over a soft
reset, which improves usability of the network.
2018-06-01 13:27:06 +10:00
Damien George
ea22406f76 extmod/modussl_mbedtls: Use mbedtls_entropy_func for CTR-DRBG entropy.
If mbedtls_ctr_drbg_seed() is available in the mbedtls bulid then so should
be mbedtls_entropy_func().  Then it's up to the port to configure a valid
entropy source, eg via MBEDTLS_ENTROPY_HARDWARE_ALT.
2018-05-31 21:52:29 +10:00