Commit Graph

55 Commits

Author SHA1 Message Date
Damien George
69661f3343 all: Reformat C and Python source code with tools/codeformat.py.
This is run with uncrustify 0.70.1, and black 19.10b0.
2020-02-28 10:33:03 +11:00
David Lechner
edbb73a411 py/qstr: Don't include or init qstr_mutex when GIL is enabled.
When threads and the GIL are enabled, then the qstr mutex is not needed.
The qstr_mutex field is never used in this case because of:

    #if MICROPY_PY_THREAD && !MICROPY_PY_THREAD_GIL
    #define QSTR_ENTER() mp_thread_mutex_lock(&MP_STATE_VM(qstr_mutex), 1)
    #define QSTR_EXIT() mp_thread_mutex_unlock(&MP_STATE_VM(qstr_mutex))
    #else
    #define QSTR_ENTER()
    #define QSTR_EXIT()
    #endif

So, we can completely remove qstr_mutex everywhere when MICROPY_PY_THREAD
&& !MICROPY_PY_THREAD_GIL.
2020-01-23 13:29:11 +11:00
Léa Saviot
bc129f1b84 py/qstr: Raise exception in qstr_from_strn if str to intern is too long.
The string length being longer than the allowed qstr length can happen in
many locations, for example in the parser with very long variable names.
Without an explicit check that the length is within range (as done in this
patch) the code would exhibit crashes and strange behaviour with truncated
strings.
2019-11-26 10:51:47 +11:00
Damien George
2b575418b6 py/qstr: Evaluate find_qstr only once then pass to Q_GET_HASH macro.
Q_GET_HASH may evaluate its argument more than once.
2019-02-19 23:44:01 +11:00
Damien George
0d165fec9c py/qstr: Put a lower bound on new qstr pool allocation. 2018-12-15 14:32:09 +11:00
Damien George
8e323b8fa8 py/qstr: Rewrite find_qstr to make manifest that it returns a valid ptr.
So long as the input qstr identifier is valid (below the maximum number of
qstrs) the function will always return a valid pointer.  This patch
eliminates the "return 0" dead-code.
2017-11-29 17:01:39 +11:00
Damien George
487dbdb267 py/compile: Use alloca instead of qstr_build when compiling import name.
The technique of using alloca is how dotted import names are composed in
mp_import_from and mp_builtin___import__, so use the same technique in the
compiler.  This puts less pressure on the heap (only the stack is used if
the qstr already exists, and if it doesn't exist then the standard qstr
block memory is used for the new qstr rather than a separate chunk of the
heap) and reduces overall code size.
2017-11-01 13:16:16 +11:00
Stefan Naumann
ace9fb5405 py: Add verbose debug compile-time flag MICROPY_DEBUG_VERBOSE.
It enables all the DEBUG_printf outputs in the py/ source code.
2017-08-15 11:53:36 +10:00
Alexander Steffen
55f33240f3 all: Use the name MicroPython consistently in comments
There were several different spellings of MicroPython present in comments,
when there should be only one.
2017-07-31 18:35:40 +10:00
Colin Hogben
f9b6b37cf6 py: Fix wrong assumption that m_renew will not move if shrinking
In both parse.c and qstr.c, an internal chunking allocator tidies up
by calling m_renew to shrink an allocated chunk to the size used, and
assumes that the chunk will not move.  However, when MICROPY_ENABLE_GC
is false, m_renew calls the system realloc, which does not guarantee
this behaviour.  Environments where realloc may return a different
pointer include:

(1) mbed-os with MBED_HEAP_STATS_ENABLED (which adds a wrapper around
malloc & friends; this is where I was hit by the bug);

(2) valgrind on linux (how I diagnosed it).

The fix is to call m_renew_maybe with allow_move=false.
2016-11-02 23:15:41 +11:00
Damien George
e60835bac5 py/qstr: Remove a comment.
qstrs are always null terminated so qstr_str will stay as part of the API.
2016-09-19 13:18:54 +10:00
Damien George
a1c93a62b1 py: Don't use gc or qstr mutex when the GIL is enabled.
There is no need since the GIL already makes gc and qstr operations
atomic.
2016-06-28 11:28:50 +01:00
Damien George
1f54ad2aed py: Make interning of qstrs thread safe. 2016-06-28 11:28:50 +01:00
Paul Sokolovsky
f469c76442 py: Rename __QSTR_EXTRACT flag to NO_QSTR.
It has more usages than just qstr extraction, for example, embedding (where
people don't care about efficient predefined qstrs).
2016-06-16 01:42:48 +03:00
Paul Sokolovsky
c618f91e22 py: Rework QSTR extraction to work in simple and obvious way.
When there're C files to be (re)compiled, they're all passed first to
preprocessor. QSTR references are extracted from preprocessed output and
split per original C file. Then all available qstr files (including those
generated previously) are catenated together. Only if the resulting content
has changed, the output file is written (causing almost global rebuild
to pick up potentially renumbered qstr's). Otherwise, it's not updated
to not cause spurious rebuilds. Related make rules are split to minimize
amount of commands executed in the interim case (when some C files were
updated, but no qstrs were changed).
2016-04-19 11:37:56 +03:00
Damien George
0a2e9650f5 py: Add ability to have frozen persistent bytecode from .mpy files.
The config variable MICROPY_MODULE_FROZEN is now made of two separate
parts: MICROPY_MODULE_FROZEN_STR and MICROPY_MODULE_FROZEN_MPY.  This
allows to have none, either or both of frozen strings and frozen mpy
files (aka frozen bytecode).
2016-04-13 16:07:47 +01:00
Damien George
257848587f py/qstr: Use size_t instead of mp_uint_t when counting allocated bytes. 2015-12-17 12:41:40 +00:00
Damien George
c3f64d9799 py: Change qstr_* functions to use size_t as the type for str len arg. 2015-11-29 14:25:04 +00:00
Damien George
59a41e8fcd py/qstr: Fix calc of qstr memory usage, due to new qstr chunk allocation. 2015-10-13 15:52:06 +01:00
Damien George
c3bd9415cc py: Make qstr hash size configurable, defaults to 2 bytes.
This patch makes configurable, via MICROPY_QSTR_BYTES_IN_HASH, the
number of bytes used for a qstr hash.  It was originally fixed at 2
bytes, and now defaults to 2 bytes.  Setting it to 1 byte will save
ROM and RAM at a small expense of hash collisions.
2015-07-20 11:03:13 +00:00
Damien George
ade9a05236 py: Improve allocation policy of qstr data.
Previous to this patch all interned strings lived in their own malloc'd
chunk.  On average this wastes N/2 bytes per interned string, where N is
the number-of-bytes for a quanta of the memory allocator (16 bytes on 32
bit archs).

With this patch interned strings are concatenated into the same malloc'd
chunk when possible.  Such chunks are enlarged inplace when possible,
and shrunk to fit when a new chunk is needed.

RAM savings with this patch are highly varied, but should always show an
improvement (unless only 3 or 4 strings are interned).  New version
typically uses about 70% of previous memory for the qstr data, and can
lead to savings of around 10% of total memory footprint of a running
script.

Costs about 120 bytes code size on Thumb2 archs (depends on how many
calls to gc_realloc are made).
2015-07-14 22:56:32 +01:00
Damien George
e72cda99fd py: Convert occurrences of non-debug printf to mp_printf. 2015-04-16 14:30:16 +00:00
Damien George
ea0461dcd3 py: Add option to micropython.qstr_info() to dump actual qstrs. 2015-02-10 11:02:28 +00:00
Damien George
963a5a3e82 py, unix: Allow to compile with -Wsign-compare.
See issue #699.
2015-01-16 17:47:07 +00:00
Damien George
4c81ba8015 py: Never intern data of large string/bytes object; add relevant tests.
Previously to this patch all constant string/bytes objects were
interned by the compiler, and this lead to crashes when the qstr was too
long (noticeable now that qstr length storage defaults to 1 byte).

With this patch, long string/bytes objects are never interned, and are
referenced directly as constant objects within generated code using
load_const_obj.
2015-01-13 16:21:23 +00:00
Damien George
95836f8439 py: Add MICROPY_QSTR_BYTES_IN_LEN config option, defaulting to 1.
This new config option sets how many fixed-number-of-bytes to use to
store the length of each qstr.  Previously this was hard coded to 2,
but, as per issue #1056, this is considered overkill since no-one
needs identifiers longer than 255 bytes.

With this patch the number of bytes for the length is configurable, and
defaults to 1 byte.  The configuration option filters through to the
makeqstrdata.py script.

Code size savings going from 2 to 1 byte:
- unix x64 down by 592 bytes
- stmhal down by 1148 bytes
- bare-arm down by 284 bytes

Also has RAM savings, and will be slightly more efficient in execution.
2015-01-11 22:27:30 +00:00
Damien George
6942f80a8f py: Add qstr cfg capability; generate QSTR_NULL and QSTR_ from script. 2015-01-11 22:06:53 +00:00
Damien George
01418e9690 py: Fix hard-coded hash for empty qstr (was 0x0000 now 0x1505). 2015-01-11 14:15:45 +00:00
Damien George
b4b10fd350 py: Put all global state together in state structures.
This patch consolidates all global variables in py/ core into one place,
in a global structure.  Root pointers are all located together to make
GC tracing easier and more efficient.
2015-01-07 20:33:00 +00:00
Damien George
51dfcb4bb7 py: Move to guarded includes, everywhere in py/ core.
Addresses issue #1022.
2015-01-01 20:32:09 +00:00
Damien George
109c1de015 py: Make gc.enable/disable just control auto-GC; alloc is still allowed.
gc.enable/disable are now the same as CPython: they just control whether
automatic garbage collection is enabled or not.  If disabled, you can
still allocate heap memory, and initiate a manual collection.
2014-10-31 21:30:46 +00:00
Damien George
0b13f3e026 py: Improve memory usage debugging; better GC AT dumping.
In unix port, mem_info(1) now prints pretty GC alloc table.
2014-10-24 23:12:25 +01:00
Damien George
39dc145478 py: Change [u]int to mp_[u]int_t in qstr.[ch], and some other places.
This should pretty much resolve issue #50.
2014-10-03 19:52:22 +01:00
Damien George
40f3c02682 Rename machine_(u)int_t to mp_(u)int_t.
See discussion in issue #50.
2014-07-03 13:25:24 +01:00
Paul Sokolovsky
59c675a64c py: Include mpconfig.h before all other includes.
It defines types used by all other headers.

Fixes #691.
2014-06-21 22:43:22 +03:00
Damien George
73c98d8709 py: Fix static defn in qstr; include mpconfigport.h with "" (not <>). 2014-06-11 19:18:03 +01:00
Damien George
2617eebf2f Change const byte* to const char* where sensible.
This removes need for some casts (at least, more than it adds need
for new casts!).
2014-05-25 22:27:57 +01:00
Damien George
04b9147e15 Add license header to (almost) all files.
Blanket wide to all .c and .h files.  Some files originating from ST are
difficult to deal with (license wise) so it was left out of those.

Also merged modpyb.h, modos.h, modstm.h and modtime.h in stmhal/.
2014-05-03 23:27:38 +01:00
Damien George
d553be5982 build: Simplify build directory layout by putting all headers in genhdr.
Any generated headers go in $(BUILD)/genhdr/, and are #included as
'genhdr/xxx.h'.
2014-04-17 18:03:27 +01:00
Andrew Scheller
70a7d7a943 build directory can now be renamed
The autogenerated header files have been moved about, and an extra
include dir has been added, which means you can give a custom
BUILD=newbuilddir option to make, and everything "just works"

Also tidied up the way the different Makefiles build their include-
directory flags
2014-04-16 22:16:28 +01:00
Paul Sokolovsky
59e269cfec qstr, objstr: Make sure that valid hash != 0, treat 0 as "not computed".
This feature was proposed with initial hashing RFC, and is prerequisite for
seamless static str object definition.
2014-04-14 01:43:01 +03:00
Damien George
6e628c49ca py: Replace naive and teribble hash function with djb2. 2014-03-25 15:27:15 +00:00
Damien George
1dc76af7bf py: Remove name of var arg from macros with var args. 2014-02-26 16:57:08 +00:00
Paul Sokolovsky
44739e280e Make DEBUG_printf() a proper function, implementation is port-dependent.
In particular, unix outputs to stderr, to allow to run testsuite against
micropython built with debug output (by redirecting stderr to /dev/null).
2014-02-16 18:20:49 +02:00
Paul Sokolovsky
520e2f58a5 Replace global "static" -> "STATIC", to allow "analysis builds". Part 2. 2014-02-12 18:31:30 +02:00
Damien George
4d5b28cd08 Add qstr_info() function and bindings for unix port. 2014-01-29 18:56:46 +00:00
Dave Hylands
c89c681a9f Rework makefiles. Add proper dependency checking. 2014-01-24 08:46:48 -08:00
Damien George
1e708fed18 py: Implement bool unary op; tidy up unary op dispatch. 2014-01-23 18:27:51 +00:00
Damien George
5fa93b6755 Second stage of qstr revamp: uPy str object can be qstr or not. 2014-01-22 14:35:10 +00:00
Damien George
55baff4c9b Revamp qstrs: they now include length and hash.
Can now have null bytes in strings.  Can define ROM qstrs per port using
qstrdefsport.h
2014-01-21 21:40:13 +00:00