PyPy Status Blog

Leysin Winter Sprint: 25/26th Feb. - 4th March 2017

2017-01-24T12:01:00.000+01:00

The next PyPy sprint will be in Leysin, Switzerland, for the twelveth time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome.

Goals and topics of the sprint

The list of topics is very open.

The main topic is Python 3.5 support in PyPy, as most py3.5 contributors should be present. It is also a good topic if you have no or limited experience with PyPy contribution: we can easily find something semi-independent that is not done in py3.5 so far, and do pair-programming with you.
Any other topic is fine too: JIT compiler optimizations, CFFI, the RevDB reverse debugger, improving to speed of your program on PyPy, etc.
And as usual, the main side goal is to have fun in winter sports :-) We can take a day off (for ski or anything else).

Exact times

Work days: starting 26th Feb (~noon), ending March 4th (~noon).

I have pre-booked the week from Saturday Feb 25th to Saturday March 4th. If it is possible for you to arrive Sunday before mid-afternoon, then you should get a booking from Sunday only. The break day should be around Wednesday.

It is fine to stay a few more days on either side, or conversely to book for a part of that time only.

Location & Accomodation

Leysin, Switzerland, "same place as before".

Let me refresh your memory: both the sprint venue and the lodging will be in a pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connection with wireless installed. You can also arrange your own lodging elsewhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue).

Please confirm that you are coming so that we can adjust the reservations as appropriate.

The options of rooms are a bit more limited than on previous years because the place for bed-and-breakfast is shrinking; but we should still have enough room for us. The price is around 60 CHF, breakfast included, in shared rooms (3 or 4 people). If there are people that would prefer a double or single room, please contact me and we'll see what choices you have. There are also a choice of hotels in Leysin.

Please register by Mercurial:

https://bitbucket.org/pypy/extradoc/
https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2017/

or on the pypy-dev mailing list if you do not yet have check-in rights:

http://mail.python.org/mailman/listinfo/pypy-dev

You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around, and at least one EU-format power strip.

PyPy2.7 v5.6 released - stdlib 2.7.12 support, C-API improvements, and more

2016-11-12T17:32:00.000+01:00

We have released PyPy2.7 v5.6 [0], about two months after PyPy2.7 v5.4. This new PyPy2.7 release includes the upstream stdlib version 2.7.12.

We continue to make incremental improvements to our C-API compatibility layer (cpyext). We pass all but 12 of the over-6000 tests in the upstream NumPy test suite, and have begun examining what it would take to support Pandas and PyQt.

Work proceeds at a good pace on the PyPy3.5 version due to a grant from the Mozilla Foundation, and some of those changes have been backported to PyPy2.7 where relevant.

The PowerPC and s390x backend have been enhanced with the capability to use SIMD instructions for micronumpy loops.

We changed timeit to now report average +/- standard deviation, which is better than the misleading minimum value reported in CPython.

We now support building PyPy with OpenSSL 1.1 in our built-in _ssl module, as well as maintaining support for previous versions.

CFFI has been updated to 1.9, improving an already great package for interfacing with C.

As always, this release fixed many issues and bugs raised by the growing community of PyPy users. We strongly recommend updating. You can download the PyPy2.7 v5.6 release here:

http://pypy.org/download.html

Downstream packagers have been hard at work. The Debian package is already available, and the portable PyPy versions are also ready, for those who wish to run PyPy on other Linux distributions like RHEL/Centos 5.

We would like to thank our donors for the continued support of the PyPy project.

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.

What is PyPy?

x86 machines on most common operating systems (Linux 32/64 bits, Mac OS X 64 bits, Windows 32 bits, OpenBSD, FreeBSD)

newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,

big- and little-endian variants of PPC64 running Linux,

s390x running Linux

What else is new?

(since the release of PyPy 5.4 in August, 2016)

There are many incremental improvements to RPython and PyPy, the complete listing is here.

Please update, and continue to help us make PyPy better.

Cheers, The PyPy team

[0] We skipped 5.5 since we share a code base with PyPy3, and PyPy3.3-v.5.5-alpha was released last month

Vectorization extended. PowerPC and s390x

2016-11-03T09:35:00.000+01:00

We are happy to announce that JIT support in both the PowerPC backend and the
s390x backend have been enhanced. Both can now vectorize loops via SIMD
instructions. Special thanks to IBM for funding this work.

If you are not familiar with this topic you can read more details here.

There are many more enhancements under the hood. Most notably, all pure operations are now delayed until the latest possible point. In some cases indices have been calculated more than once or they needed an additional register, because the old value is still used. Additionally it is now possible to load quadword-aligned memory in both PPC and s390x (x86 currently cannot do that).

NumPy & CPyExt

The community and core developers have been moving CPyExt towards a complete, but emulated, layer for CPython C extensions. This is great, because the one restriction preventing the wider deployment of PyPy in several scenarios will hopefully be removed. However, we advocate not to use CPyExt, but rather to not write C code at all (let PyPy speed up your Python code) or use cffi.

The work done here to support vectorization helps micronumpy (NumPyPy) to speed up operations for PPC and s390x. So why is PyPy supporting both NumPyPy and NumPy, do we actually need both? Yes, there are places where gcc can beat the JIT, and places where the tight integration between NumPyPy and PyPy is more performant. We do have plans to integrate both, hijacking the C-extension method calls to use NumPyPy where we know NumPyPy can be faster.

Just to give you an idea why this is a benefit:

NumPy arrays can carry custom dtypes and apply user defined python functions on the arrays. How could one optimize this kind of scenario? In a traditional setup, you cannot. But as soon as NumPyPy is turned on, you can suddenly JIT compile this code and vectorize it.

Another example is element access that occurs frequently, or any other calls that cross between Python and the C level frequently.

Benchmarks

Let's have a look at some benchmarks reusing mikefc's numpy benchmark suite (find the forked version here). I only ran a subset of microbenchmarks, showing that the core functionality is
functioning properly. Additionally it has been rewritten to use perf instead of the timeit stdlib module.

Setup

x86 runs on a Intel i7-2600 clocked at 3.40GHz using 4 cores. PowerPC runs on the Power 8 clocked at 3.425GHz providing 160 cores. Last but not least the mainframe machine clocked up to 4 GHz, but fully virtualized (as it is common for such machines). Note that PowerPC is a non private remote machine. It is used by many users and it is crowded with processes. It is hard to extract a stable benchmark there.

x86 ran on Fedora 24 (kernel version of 4.8.4), PPC ran on Fedora 21 (kernel version 3.17.4) and s390x ran on Redhat Linux 7.2 (kernel version 3.10.0). Respectivley, numpy on cpython had openblas available on x86, no blas implementation were present on s390x and PPC provided blas and lapack.

As you can see all machines run very different configurations. It does not make sense to compare across platforms, but rather implementations on the same platform.

Blue shows CPython 2.7.10+ available on that platform using the latest NumPy (1.11). Micro NumPy is used for PyPy. PyPy+ indicates that the vectorization optimization is turned on.
All bar charts show the median value of all runs (5 samples, 100 loops, 10 inner loops, for the operations on vectors (not matrices) the loops are set to 1000). PyPy additionally gets 3 extra executions to warmup the JIT.

The comparison is really comparing speed of machine code. It compares the PyPy's JIT output vs GCC's output. It has little to do with the speed of the interpreter.

Both new SIMD backends speedup the numeric kernels. Some times it is near to the speed of CPython, some times it is faster. The maximum parallelism very much depends on the extension emitted by the compiler. All three SIMD backends have the same vector register size (which is 128 bit). This means that all three behave similar but ppc and s390x gain more because they can load 128bit of memory from quadword aligned memory.

Future directions

Python is achieving rapid adoption in data science. This is currently a trend emerging in Europe, and Python is already heavily used for data science in the USA many other places around the world.

PyPy can make a valuable contribution for data scientists, helping them to rapidly write scientific programs in Python and run them at near native speed. If you happen to be in that situation, we are eager to hear you feedback or resolve your issues and also work together to improve the performance of your,
code. Just get in touch!

Richard Plangger (plan_rich) and the PyPy team

PyPy3 5.5.0 released

2016-10-12T10:37:00.000+02:00

We're pleased to announce the release of PyPy3 v5.5.0. Coming four months after PyPy3.3 v5.2, it improves compatibility with Python 3.3 (3.3.5). We strongly recommend updating from previous PyPy3 versions.

We would like to thank all of the people who donated to the py3k proposal for supporting the work that went into this release.

You can download the PyPy3.3 v5.5.0 release here: http://pypy.org/download.html

Improved Python 3.3.5 support.

os.get_terminal_size(), time.monotonic(), str.casefold()
faulthandler module
There are still some missing features such as a PEP 393-like space efficient string representation and including performance regressions (e.g. issue #2305). The focus for this release has been updating to 3.3 compatibility. Windows is also not yet supported.

ensurepip is also included (it's only included in CPython 3 >= 3.4).
Buffer interface improvements (numpy on top of cpyext)
Several JIT improvements (force-virtual-state, residual calls)
Search path for libpypy-c.so has changed (helps with cffi embedding on linux distributions)
Improve the error message when the user forgot the "self" argument of a method
Many more small improvements, please head over to our documentation for more information

Towards Python 3.5

We have started to work on Python 3.5, which is a version used by many software projects. It seems to get wide adoption. We are happy to be part of the Mozilla Open Source Support (MOSS) initiative.

Nevertheless we want to give our users the chance to use PyPy in their Python 3 projects, thus we have prepared this release.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7.10 and 3.3.5. It's fast due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This release supports:

x86 machines on most common operating systems except Windows
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux
big- and little-endian variants of PPC64 running Linux
s390x running Linux

Please try it out and let us know what you think. We welcome feedback, we know
you are using PyPy, please tell us about it!

Cheers

The PyPy Team

RevDB released, v5.4.1

2016-09-10T10:30:00.000+02:00

Hi all,

The first beta version of RevDB is out! Remember that RevDB is a reverse debugger for Python. The idea is that it is a debugger that can run forward and backward in time, letting you more easily understand your subtle bug in your big Python program.

RevDB should work on almost any Python program. Even if you are normally only using CPython, trying to reproduce the bug with RevDB is similar to trying to run the program on a regular PyPy---usually it just works, even if not quite always.

News from the alpha version in the previous blog post include notably support for:

Threads.
CPyExt, the compatibility layer of PyPy that can run CPython C extension modules.

as well as many other improvements.

You need to build it yourself for now. It is tested on 64-bit Linux. 32-bit Linux, OS/X, and other POSIX platforms should all either work out of the box or be just a few fixes away (contributions welcome). Win32 support is a lot more involved but not impossible.

See https://bitbucket.org/pypy/revdb/ for more information!

Armin

PyPy 5.4.1 bugfix released

2016-09-07T09:51:00.000+02:00

We have released a bugfix for PyPy2.7-v5.4.0, released last week, due to the following issues:

Update list of contributors in documentation and LICENSE file, this was unfortunately left out of 5.4.0. My apologies to the new contributors
Allow tests run with -A to find libm.so even if it is a script not a dynamically loadable file
Bump sys.setrecursionlimit() when translating PyPy, for translating with CPython
Tweak a float comparison with 0 in backendopt.inline to avoid rounding errors
Fix for an issue for translating the sandbox
Fix for and issue where unicode.decode('utf8', 'custom_replace') messed up the last byte of a unicode string sometimes
Update built-in cffi to version 1.8.1
Explicitly detect that we found as-yet-unsupported OpenSSL 1.1, and crash translation with a message asking for help porting it
Fix a regression where a PyBytesObject was forced (converted to a RPython object) when not required, reported as issue #2395

Thanks to those who reported the issues.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This release supports:

x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD),
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,
big- and little-endian variants of PPC64 running Linux,
s390x running Linux

Please update, and continue to help us make PyPy better.

Cheers

The PyPy Team

PyPy2 v5.4 released - incremental improvements and enhancements

2016-08-31T18:08:00.001+02:00

We have released PyPy2.7 v5.4, a little under two months after PyPy2.7 v5.3. This new PyPy2.7 release includes incremental improvements to our C-API compatibility layer (cpyext), enabling us to pass over 99% of the upstream numpy test suite.

We updated built-in cffi support to version 1.8, which now supports the “limited API” mode for c-extensions on CPython >=3.2.

We improved tooling for the PyPy JIT, and expanded VMProf support to OpenBSD and Dragon Fly BSD

As always, this release fixed many issues and bugs raised by the growing community of PyPy users. We strongly recommend updating.

You can download the PyPy2 v5.4 release here:

http://pypy.org/download.html

We would like to thank our donors for their continued support of the PyPy project. We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, testing and adapting popular modules to run on PyPy, or general help with making RPython’s JIT even better.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (PyPy and CPython 2.7 performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

This release supports:

x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD)
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux
big- and little-endian variants of PPC64 running Linux
s390x running Linux

What is New?

(since the release of PyPy 5.3 in June, 2016)

There are many incremental improvements to RPython and PyPy, the complete listing is here. Mozilla generously sponsored work toward python 3.5 compatibility, and we are beginning to see some cross-over improvements of RPython and PyPy2.7 as a result.

Please update, and continue to help us make PyPy better. Cheers

The PyPy Team

PyPy Tooling Upgrade: JitViewer and VMProf

2016-08-11T12:52:00.000+02:00

We are happy to announce a major JitViewer (JV) update.
JV allows you to inspect RPython's internal compiler representation (the language in which PyPy is implemented) including the generated machine code of your program. It can graphically show you details of the JIT compiled code and helps you pinpoint issues in your program.

VMProf is a statistical CPU profiler for python imposing very little overhead at runtime.

Both VMProf and JitViewer share a common goal: Present useful information for your python program.
The combination of both can reveal more information than either alone.
That is the reason why they are now both packaged together.
We also updated vmprof.com with various bug fixes and changes including an all new interface to JV.

This work was done with the goal of improving tooling and libraries around the Python/PyPy/RPython ecosystem.
Some of the tools we have developed:

CFFI - Foreign Function Interface that avoids CPyExt (CFFI docs)
RevDB - A reverse debugger for python (RevDB blog post)

and of course the tools we discuss here:

VMProf - A statistical CPU profiler (VMProf docs)
JitViewer - Visualization of the log file produced by RPython (JitLog docs)

A "brand new" JitViewer

JitViewer has two pieces: you create a log file when running your program, and then use a graphic tool to view what happened.

The old logging format was a hard-to-maintain, plain-text-logging facility. Frequent changes often broke internal tools.
Additionally, the logging output of a long running program required a lot of disk space.

Our new binary format encodes data densely, makes use of some compression (gzip), and tries to remove repetition where possible.
It also supports versioning for future proofing and can be extended easily.

And *drumroll* you no longer need to install a tool to view the log yourself
anymore! The whole system moved to vmprof.com and you can use it any time.

Sounds great. But what can you do with it? Here are two examples for a PyPy user:

PyPy crashed? Did you discover a bug?

For some hard to find bugs it is often necessary to look at the compiled code. The old
procedure often required you to upload a plain text file which was hard to parse and to look through.

A better way to share a crash report is to install the ``vmprof`` module from PyPi and execute either of the two commands:

# this program does not crash, but has some weird behaviour
$ pypy -m jitlog --web <your program args>
...
PyPy Jitlog: http://vmprof.com/#/<hash>/traces
# this program segfaults
$ pypy -m jitlog -o /tmp/log <your program args>
...
<Segfault>
$ pypy -m jitlog --upload /tmp/log
PyPy Jitlog: http://vmprof.com/#/<hash>/traces

Providing the link in the bug report allows PyPy developers to browse and identify potential issues.

Speed issues

VMProf is a great tool to find hot spots that consume a lot of time in your program. As soon as you have identified code that runs slowly, you can switch to jitlog and maybe pinpoint certain aspects that do not behave as expected. You will find an overview, and are able to browse the generated code. If you cannot make sense of all that, you can just share the link with us and we can have a look too.

Future direction

We hope that the new release will help both PyPy developers and PyPy users resolve potential issues and easily point them out.

Here are a few ideas what might come in the next few releases:

Combination of CPU profiles and the JITLOG (sadly did not make it into the current release).
Extend vmprof.com to be able to query vmprof/jitlog.
An example query for vmprof: 'methods.callsites() > 5' and
for the jitlog would be 'traces.contains('call_assembler').hasbridge('*my_func_name*')'.
Extend the jitlog to capture the information of the optimization stage.

Richard Plangger (plan_rich) and the PyPy team

PyPy gets funding from Mozilla for Python 3.5 support

2016-08-09T17:38:00.000+02:00

"Python 2.x versus Python 3.x": this is by now an old question. In the eyes of some people Python 2 is here to stay, and in the eyes of others Python has long been 3 only.

PyPy's own position is that PyPy will support Python 2.7 forever---the RPython language in which PyPy is written is a subset of 2.7, and we have no plan to upgrade that. But at the same time, we want to support 3.x. This is particularly true now: a relatively recent development is that Python 3.5 seems to attract more and more people. The "switch" to Python 3.x might be starting to happen.

Correspondingly, PyPy has been searching for a while for a way to support a larger-scale development effort. The goal is to support not just any old version of Python 3.x, but Python 3.5, as this seems to be the version that people are switching to. PyPy is close to supporting all of Python 3.3 now; but the list of what is new in Python 3.4 and 3.5 is far, far longer than anyone imagines. The long-term goal is also to get a version of "PyPy3" that is as good as "PyPy2" is, including its performance and its cpyext layer (CPython C API interoperability), for example.

So, the end result: Mozilla recently decided to award $200,000 to Baroque Software to work on PyPy as part of its Mozilla Open Source Support (MOSS) initiative. This money will be used to implement the Python 3.5 features in PyPy. Within the next year, we plan to use the money to pay four core PyPy developers half-time to work on the missing features and on some of the big performance and cpyext issues. This should speed up the progress of catching up with Python 3.x significantly. We are extremely thankful to Mozilla for supporting us in this way, and will keep you updated on the progress via this blog.

Reverse debugging for Python

2016-07-08T12:39:00.001+02:00

RevPDB

A "reverse debugger" is a debugger where you can go forward and backward in time. It is an uncommon feature, at least in the open source world, but I have no idea why. I have used undodb-gdb and rr, which are reverse debuggers for C code, and I can only say that they saved me many, many days of poking around blindly in gdb.

The PyPy team is pleased to give you "RevPDB", a reverse-debugger similar to rr but for Python.

An example is worth a thousand words. Let's say your big Python program has a bug that shows up inconsistently. You have nailed it down to something like:

start x.py, which does stuff (maybe involving processing files, answering some web requests that you simulate from another terminal, etc.);
sometimes, after a few minutes, your program's state becomes inconsistent and you get a failing assert or another exception.

This is the case where RevPDB is useful.

RevPDB is available only on 64-bit Linux and OS/X right now, but should not be too hard to port to other OSes. It is very much alpha-level! (It is a debugger full of bugs. Sorry about that.) I believe it is still useful---it helped me in one real use case already.

How to get RevPDB

The following demo was done with an alpha version for 64-bit Linux, compiled for Arch Linux. I won't provide the binary; it should be easy enough to retranslate (much faster than a regular PyPy because it contains neither a JIT nor a custom GC). Grab the PyPy sources from Mercurial, and then:

hg update reverse-debugger
# or "hg update ff376ccacb36" for exactly this demo
cd pypy/goal
../../rpython/bin/rpython -O2 --revdb targetpypystandalone.py  \
                  --withoutmod-cpyext --withoutmod-micronumpy

and possibly rename the final pypy-c to pypy-revdb to avoid confusion.

Other platforms than 64-bit Linux and OS/X need some fixes before they work.

Demo

For this demo, we're going to use this x.py as the "big program":

import os

class Foo(object):
    value = 5

lst1 = [Foo() for i in range(100)]
lst1[50].value += 1
for x in lst1:
    x.value += 1

for x in lst1:
    if x.value != 6:
        print 'oops!'
        os._exit(1)

Of course, it is clear what occurs in this small example: the check fails on item 50. For this demo, the check has been written with os._exit(1), because this exits immediately the program. If it was written with an assert, then its failure would execute things in the traceback module afterwards, to print the traceback; it would be a minor mess just to find the exact point of the failing assert. (This and other issues are supposed to be fixed in the future, but for now it is alpha-level.)

Anyway, with a regular assert and a regular post-mortem pdb, we could observe that x.value is indeed 7 instead of 6 when the assert fails. Imagine that the program is much bigger: how would we find the exact chain of events that caused this value 7 to show up on this particular Foo object? This is what RevPDB is for.

First, we need for now to disable Address Space Layout Randomization (ASLR), otherwise replaying will not work. This is done once with the following command line, which changes the state until the next reboot:

echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

UPDATE: the above is no longer necessary from revision ff376ccacb36.

Run x.py with RevPDB's version of PyPy instead of the regular interpreter (CPython or PyPy):

PYPYRDB=log.rdb ./pypy-revdb x.py

This pypy-revdb executable is like a slow PyPy executable, running (for now) without a JIT. This produces a file log.rdb which contains a complete log of this execution. (If the bug we are tracking occurs rarely, we need to re-run it several times until we get the failure. But once we got the failure, then we're done with this step.)

Start:

rpython/translator/revdb/revdb.py log.rdb

We get a pdb-style debugger. This revdb.py is a normal Python program, which you run with an unmodified Python; internally, it looks inside the log for the path to pypy-revdb and run it as needed (as one forking subprocess, in a special mode).

Initially, we are at the start of the program---not at the end, like we'd get in a regular debugger:

File "<builtin>/app_main.py", line 787 in setup_bootstrap_path:
(1)$

The list of commands is available with help.

Go to the end with continue (or c):

(1)$ continue
File "/tmp/x.py", line 14 in <module>:
...
  lst1 = [Foo() for i in range(100)]
  lst1[50].value += 1
  for x in lst1:
      x.value += 1

  for x in lst1:
      if x.value != 6:
          print 'oops!'
>         os._exit(1)
(19727)$

We are now at the beginning of the last executed line. The number 19727 is the "time", measured in number of lines executed. We can go backward with the bstep command (backward step, or bs), line by line, and forward again with the step command. There are also commands bnext, bcontinue and bfinish and their forward equivalents. There is also "go TIME" to jump directly to the specified time. (Right now the debugger only stops at "line start" events, not at function entry or exit, which makes some cases a bit surprising: for example, a step from the return statement of function foo() will jump directly to the caller's caller, if the caller's current line was return foo() + 2, because no "line start" event occurs in the caller after foo() returns to it.)

We can print Python expressions and statements using the p command:

(19727)$ p x
$0 = <__main__.Foo object at 0xfffffffffffeab3e>
(19727)$ p x.value
$1 = 7
(19727)$ p x.value + 1
8

The "$NUM =" prefix is only shown when we print an object that really exists in the debugged program; that's why the last line does not contain it. Once a $NUM has been printed, then we can use it in further expressions---even at a different point time. It becomes an anchor that always refers to the same object:

(19727)$ bstep

File "/tmp/x.py", line 13 in <module>:
...

  lst1 = [Foo() for i in range(100)]
  lst1[50].value += 1
  for x in lst1:
      x.value += 1

  for x in lst1:
      if x.value != 6:
>         print 'oops!'
          os._exit(1)
(19726)$ p $0.value
$1 = 7

In this case, we want to know when this value 7 was put in this attribute. This is the job of a watchpoint:

(19726)$ watch $0.value
Watchpoint 1 added
updating watchpoint value: $0.value => 7

This watchpoint means that $0.value will be evaluated at each line. When the repr() of this expression changes, the watchpoint activates and execution stops:

(19726)$ bcontinue
[searching 19629..19726]
[searching 19338..19629]

updating watchpoint value: $0.value => 6
Reverse-hit watchpoint 1: $0.value
File "/tmp/x.py", line 9 in <module>:
  import os

  class Foo(object):
      value = 5

  lst1 = [Foo() for i in range(100)]
  lst1[50].value += 1
  for x in lst1:
>     x.value += 1

  for x in lst1:
      if x.value != 6:
          print 'oops!'
          os._exit(1)
(19524)$

Note that using the $NUM syntax is essential in watchpoints. You can't say "watch x.value", because the variable x will go out of scope very soon when we move forward or backward in time. In fact the watchpoint expression is always evaluated inside an environment that contains the builtins but not the current locals and globals. But it also contains all the $NUM, which can be used to refer to known objects. It is thus common to watch $0.attribute if $0 is an object, or to watch len($1) if $1 is some list. The watch expression can also be a simple boolean: for example, "watch $2 in $3" where $3 is some dict and $2 is some object that you find now in the dict; you would use this to find out the time when $2 was put inside $3, or removed from it.

Use "info watchpoints" and "delete <watchpointnum>" to manage watchpoints.

There are also regular breakpoints, which you set with "b FUNCNAME". It breaks whenever there is a call to a function that happens to have the given name. (It might be annoying to use for a function like __init__() which has many homonyms. There is no support for breaking on a fully-qualified name or at a given line number for now.)

In our demo, we stop at the line x.value += 1, which is where the value was changed from 6 to 7. Use bcontinue again to stop at the line lst1[50].value += 1, which is where the value was changed from 5 to 6. Now we know how this value attribute ends up being 7.

(19524)$ bcontinue
[searching 19427..19524]
[searching 19136..19427]

updating watchpoint value: $0.value => 5
Reverse-hit watchpoint 1: $0.value
File "/tmp/x.py", line 7 in <module>:
  import os

  class Foo(object):
      value = 5

  lst1 = [Foo() for i in range(100)]
> lst1[50].value += 1
  for x in lst1:
      x.value += 1

  for x in lst1:
      if x.value != 6:
...
(19422)$

Try to use bcontinue yet another time. It will stop now just before $0 is created. At that point in time, $0 refers to an object that does not exist yet, so the watchpoint now evaluates to an error message (but it continues to work as before, with that error message as the string it currently evaluates to).

(19422)$ bcontinue
[searching 19325..19422]

updating watchpoint value: $0.value => RuntimeError:
               '$0' refers to an object created later in time
Reverse-hit watchpoint 1: $0.value
File "/tmp/x.py", line 6 in <module>:
  import os

  class Foo(object):
      value = 5

> lst1 = [Foo() for i in range(100)]
  lst1[50].value += 1
  for x in lst1:
      x.value += 1

  for x in lst1:
...
(19371)$

In big programs, the workflow is similar, just more complex. Usually it works this way: we find interesting points in time with some combination of watchpoints and some direct commands to move around. We write down on a piece of (real or virtual) paper these points in history, including most importantly their time, so that we can construct an ordered understanding of what is going on.

The current revdb can be annoying and sometimes even crash; but the history you reconstruct can be kept. All the times and expressions printed are still valid when you restart revdb. The only thing "lost" is the $NUM objects, which you need to print again. (Maybe instead of $0, $1, ... we should use $<big number>, where the big number identifies uniquely the object by its creation time. These numbers would continue to be valid even after revdb is restarted. They are more annoying to use than just $0 though.)

Screencast: Here's a (slightly typo-y) screencast of cfbolz using the reverse debugger:

Current issues

General issues:

If you are using revdb on a log that took more than a few minutes to record, then it can be painfully slow. This is because revdb needs to replay again big parts of the log for some operations.
The pypy-revdb is currently missing the following modules:
- thread (implementing multithreading is possible, but not done yet);
- cpyext (the CPython C API compatibility layer);
- micronumpy (minor issue only);
- _continuation (for greenlets).
Does not contain a JIT, and does not use our fast garbage collectors. You can expect pypy-revdb to be maybe 3 times slower than CPython.
Only works on Linux and OS/X. There is no fundamental reason for this restriction, but it is some work to fix.
Replaying a program uses a lot more memory; maybe 15x as much than during the recording. This is because it creates many forks. If you have a program that consumes 10% of your RAM or more, you will need to reduce MAX_SUBPROCESSES in process.py.

Replaying also comes with a bunch of user interface issues:

Attempted to do I/O or access raw memory: we get this whenever trying to print some expression that cannot be evaluated with only the GC memory---or which can, but then the __repr__() method of the result cannot. We need to reset the state with bstep + step before we can print anything else. However, if only the __repr__() crashes, you still see the $NUM = prefix, and you can use that $NUM afterwards.
id() is globally unique, returning a reproducible 64-bit number, so sometimes using id(x) is a workaround for when using x doesn't work because of Attempted to do I/O issues (e.g. p [id(x) for x in somelist]).
as explained in the demo, next/bnext/finish/bfinish might jump around a bit non-predictably.
similarly, breaks on watchpoints can stop at apparently unexpected places (when going backward, try to do "step" once). The issue is that it can only stop at the beginning of every line. In the extreme example, if a line is foo(somelist.pop(getindex())), then somelist is modified in the middle. Immediately before this modification occurs, we are in getindex(), and immediately afterwards we are in foo(). The watchpoint will stop the program at the end of getindex() if running backward, and at the start of foo() if running forward, but never actually on the line doing the change.
watchpoint expressions must not have any side-effect at all. If they do, the replaying will get out of sync and revdb.py will complain about that. Regular p expressions and statements can have side-effects; these effects are discarded as soon as you move in time again.
sometimes even "p import foo" will fail with Attempted to do I/O. Use instead "p import sys; foo = sys.modules['foo']".
use help to see all commands. backtrace can be useful. There is no up command; you have to move in time instead, e.g. using bfinish to go back to the point where the current function was called.

How RevPDB is done

If I had to pick the main advantage of PyPy over CPython, it is that we have got with the RPython translation toolchain a real place for experimentation. Every now and then, we build inside RPython some feature that gives us an optionally tweaked version of the PyPy interpreter---tweaked in a way that would be hard to do with CPython, because it would require systematic changes everywhere. The most obvious and successful examples are the GC and the JIT. But there have been many other experiments along the same lines, from the so-called stackless transformation in the early days, to the STM version of PyPy.

RevPDB works in a similar way. It is a version of PyPy in which some operations are systematically replaced with other operations.

To keep the log file at a reasonable size, we duplicate the content of all GC objects during replaying---by repeating the same actions on them, without writing anything in the log file. So that means that in the pypy-revdb binary, the operations that do arithmetic or read/write GC-managed memory are not modified. Most operations are like that. However, the other operations, the ones that involve either non-GC memory or calls to external C functions, are tweaked. Each of these operations is replaced with code that works in two modes, based on a global flag:

in "recording" mode, we log the result of the operation (but not the arguments);
in "replaying" mode, we don't really do the operation at all, but instead just fetch the result from the log.

Hopefully, all remaining unmodified operations (arithmetic and GC load/store) are completely deterministic. So during replaying, every integer or non-GC pointer variable will have exactly the same value as it had during recording. Interestingly, it means that if the recording process had a big array in non-GC memory, then in the replaying process, the array is not allocated at all; it is just represented by the same address, but there is nothing there. When we record "read item 123 from the array", we record the result of the read (but not the "123"). When we replay, we're seeing again the same "read item 123 from the array" operation. At that point, we don't read anything; we just return the result from the log. Similarly, when recording a "write" to the array, we record nothing (this write operation has no result); so that when replaying, we redo nothing.

Note how that differs from anything managed by GC memory: GC objects (including GC arrays) are really allocated, writes really occur, and reads are redone. We don't touch the log in this case.

Other reverse debuggers for Python

There are already some Python experiments about reverse debugging. This is also known as "omniscient debugging". However, I claim that the result they get to is not very useful (for the purpose presented here). How they work is typically by recording changes to some objects, like lists and dictionaries, in addition to recording the history of where your program passed through. However, the problem of Python is that lists and dictionaries are not the end of the story. There are many, many, many types of objects written in C which are mutable---in fact, the immutable ones are the exception. You can try to systematically record all changes, but it is a huge task and easy to forget a detail.

In other words it is a typical use case for tweaking the RPython translation toolchain, rather than tweaking the CPython (or PyPy) interpreter directly. The result that we get here with RevPDB is more similar to rr anyway, in that only a relatively small number of external events are recorded---not every single change to every single list and dictionary.

Some links:

epdb: https://github.com/native-human/epdb
pode: https://github.com/rodsenra/pode

For C:

rr: http://rr-project.org/
undodb-gdb: http://undo.io/

Future work

As mentioned above, it is alpha-level, and only works on Linux and OS/X. So the plans for the immediate future are to fix the various issues described above, and port to more operating systems. The core of the system is in the C file and headers in rpython/translator/revdb/src-revdb.

For interested people, there is also the Duhton interpreter and its reverse-debugger branch, which is where I prototyped the RPython concept before moving to PyPy. The basics should work for any interpreter written in RPython, but they require some specific code to interface with the language; in the case of PyPy, it is in pypy/interpreter/reverse_debugging.py.

In parallel, there are various user interface improvements that people could be interested in, like a more "pdb++" experience. (And the script at rpython/translator/revdb/revdb.py should be moved out into some more "official" place, and the reverse-debugger branch should be merged back to default.)

I would certainly welcome any help!

-+- Armin

PyPy2 v5.3 released - major C-extension support improvements

2016-06-08T19:45:00.002+02:00

We have released PyPy2.7 v5.3, about six weeks after PyPy 5.1 and a week after PyPy3.3 v5.2 alpha 1, the first PyPy release targeting 3.3 compatibility. This new PyPy2.7 release includes major improvements for the C-API compatibility layer. In addition to complete support for lxml, we now pass most (more than 95%) of the upstream numpy test suite. We can build and run scipy and matplotlib as well. Most of the failures have to do with (ab) use of the C-API, for instance writing to a read-only pointer obtained from PyString_AsString().

Note that the C-API compatibility layer is significantly slower than CPython, as explained in the blog post about the new strategy for reflection of C objects into the PyPy interpreter.

We updated cffi to version 1.7 (incremental changes which provide a nicer developer experience, documented here). We would encourage developers to move their C-extension modules to cffi, but are willing to help you work through issues with existing code; come to #pypy on IRC and let us know how we can help you help us do better.

You can download the PyPy2 v5.3 release here:

http://pypy.org/download.html

We would like to thank our donors for their continued support of the PyPy project. We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on PyPy, or general help with making RPython’s JIT even better.

What is PyPy?

x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD)
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux
big- and little-endian variants of PPC64 running Linux
s390x running Linux

Other Highlights

(since the release of PyPy 5.1 in April, 2016)

New features:
- Merge a major expansion of the C-API support in cpyext, also expand cpyext tests to allow running them after translation as well as untranslated
- Instead of “GIL not held when a CPython C extension module calls PyXxx”, we now silently acquire/release the GIL. Helps with C extension modules that call some PyXxx() functions without holding the GIL (arguably, they are theoretically buggy).
- Support command line -v to trace import statements
- Revive traceviewer, a tool to use pygame to view traces
Numpy via our internal _numpypy module:
- Implement ufunc.outer
- Move PyPy-specific numpypy headers to a subdirectory (also changed the repo accordingly)
Performance improvements:
- Use bitstrings to compress lists of descriptors that are attached to an EffectInfo
- Remove most of the _ovf, _zer and _val operations from RPython. Kills quite some code internally, and allows the JIT to do better optimizations: for example, app-level code like x / 2 or x % 2 can now be turned into x >> 1 or x & 1, even if x is possibly negative.
- Rework the way registers are moved/spilled in before_call()
Internal refactorings:
- Refactor code to better support Python3-compatible syntax
- Reduce the size of generated C sources during translation by eliminating many many unused struct declarations (Issue #2281)
- Reduce the size of generated code by using the same function objects in all generated subclasses
- Share cpyext Py* function wrappers according to the signature, shrinking the translated libpypy.so by about 10% (without the JIT)

Please update, and continue to help us make PyPy better. Cheers
The PyPy Team

PyPy3.3 v5.2 alpha 1 released

2016-05-30T23:53:00.001+02:00

We're pleased to announce the first alpha release of PyPy3.3 v5.2. This is the
first release of PyPy which targets Python 3.3 (3.3.5) compatibility.

We would like to thank all of the people who donated to the py3k proposal
for supporting the work that went into this and future releases.

You can download the PyPy3.3 v5.2 alpha 1 release here:

http://pypy.org/download.html#python-3-3-5-compatible-pypy3-3-v5-2

Highlights

Python 3.3.5 support!
- Being an early alpha release, there are some missing features such as a
  PEP 393-like space efficient string representation and known issues
  including performance issues (e.g. issue #2305). The focus for this
  release has been updating to 3.3 compatibility. Windows is also not yet
  supported.
ensurepip is also included (it's only included in CPython 3 >= 3.4).

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for
CPython 2.7.10 and one day 3.3.5. It's fast due to its integrated tracing JIT
compiler.

We also welcome developers of other dynamic languages to see what RPython
can do for them.

This release supports:

x86 machines on most common operating systems except Windows
(Linux 32/64, Mac OS X 64, OpenBSD, FreeBSD),

newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,

big- and little-endian variants of PPC64 running Linux,

s390x running Linux

Please try it out and let us know what you think. We welcome feedback, we know
you are using PyPy, please tell us about it!

We'd especially like to thank these people for their contributions to this
release:

Manuel Jacob, Ronan Lamy, Mark Young, Amaury Forgeot d'Arc, Philip Jenvey,
Martin Matusiak, Vasily Kuznetsov, Matti Picus, Armin Rigo and many others.

Cheers

The PyPy Team

PyPy 5.1.1 bugfix released

2016-05-03T18:18:00.000+02:00

We have released a bugfix for PyPy 5.1, due to a regression in installing third-party packages depending on numpy (using our numpy fork available at https://bitbucket.org/pypy/numpy ).

Thanks to those who reported the issue. We also fixed a regression in translating PyPy which increased the memory required to translate. Improvement will be noticed by downstream packagers and those who translate rather than
download pre-built binaries.

What is PyPy?

x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD),
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,
big- and little-endian variants of PPC64 running Linux,
s390x running Linux

Please update, and continue to help us make PyPy better.

Cheers

The PyPy Team

PyPy 5.1 released

2016-04-20T22:04:00.000+02:00

We have released PyPy 5.1, about a month after PyPy 5.0.

This release includes more improvement to warmup time and memory requirements, extending the work done on PyPy 5.0. We have seen an additional reduction of about 20% in memory requirements, and up to 30% warmup time improvement, more detail in the blog post.

We also now have full support for the IBM s390x. Since this support is in RPython, any dynamic language written using RPython, like PyPy, will automagically be supported on that architecture.

We updated cffi to 1.6 (cffi 1.6 itself will be released shortly), and continue to improve support for the wider python ecosystem using the PyPy interpreter.

You can download the PyPy 5.1 release here:

http://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project.
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on pypy, or general help with making RPython’s JIT even better.

What is PyPy?

x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD),
newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux,
big- and little-endian variants of PPC64 running Linux,
s390x running Linux

Other Highlights

(since the release of PyPy 5.0 in March, 2016

New features:
- A new jit backend for the IBM s390x, which was a large effort over the past few months.
- Add better support for PyUnicodeObject in the C-API compatibility layer
- Support GNU/kFreeBSD Debian ports in vmprof
- Add __pypy__._promote
- Make attrgetter a single type for CPython compatibility
Bug Fixes
- Catch exceptions raised in an exit function
- Fix a corner case in the JIT
- Fix edge cases in the cpyext refcounting-compatible semantics (more work on cpyext compatibility is coming in the cpyext-ext branch, but isn’t ready yet)
- Try harder to not emit NEON instructions on ARM processors without NEON support
- Improve the rpython posix module system interaction function calls
- Detect a missing class function implementation instead of calling a random function
- Check that PyTupleObjects do not contain any NULLs at the point of conversion to W_TupleObjects
- In ctypes, fix _anonymous_ fields of instances
- Fix JIT issue with unpack() on a Trace which contains half-written operations
- Fix sandbox startup (a regression in 5.0)
- Fix possible segfault for classes with mangled mro or __metaclass__
- Fix isinstance(deque(), Hashable) on the pure python deque
- Fix an issue with forkpty()
- Issues reported with our previous release were resolved after reports from users on our issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy
Numpy:
- Implemented numpy.where for a single argument
- Indexing by a numpy scalar now returns a scalar
- Fix transpose(arg) when arg is a sequence
- Refactor include file handling, now all numpy ndarray, ufunc, and umath functions exported from libpypy.so are declared in pypy_numpy.h, which is included only when building our fork of numpy
- Add broadcast
Performance improvements:
- Improve str.endswith([tuple]) and str.startswith([tuple]) to allow JITting
- Merge another round of improvements to the warmup performance
- Cleanup history rewriting in pyjitpl
- Remove the forced minor collection that occurs when rewriting the assembler at the start of the JIT backend
- Port the resource module to cffi
Internal refactorings:
- Use a simpler logger to speed up translation
- Drop vestiges of Python 2.5 support in testing
- Update rpython functions with ones needed for py3k

Please update, and continue to help us make PyPy better.
Cheers
The PyPy Team

PyPy Enterprise Edition

2016-04-18T11:13:00.001+02:00

With the latest additions, PyPy's JIT now supports the Z architecture on Linux. The newest architecture revision (also known as s390x, or colloquially referred to as "big iron") is the 64-bit extension for IBM mainframes. Currently only Linux 64 bit is supported (not z/OS nor TPF).
This is the fourth assembler backend supported by PyPy in addition to x86 (32 and 64), ARM (32-bit only) and PPC64 (both little- and big-endian). It might seem that we kind of get a hang of new architectures. Thanks to IBM for funding this work!

History

When I went to university one lecture covered the prediction of Thomas Watson in 1943. His famous quote "I think there is a world market for maybe five computers ...", turned out not to be true.

However, even 70 years later, mainframes are used more often than you think. They back critical tasks requiring a high level of stability/security and offer high hardware and computational utilization rates by virtualization.

With the new PyPy JIT backend we are happy to present a fast Python virtual machine for mainframes and contribute more free software running on s390x.

Meta tracing

Even though the JIT backend has been tested on PyPy, it is not restricted to the Python programming language. Do you have a great idea for a DSL, or another language that should run on mainframes? Go ahead and just implement your interpreter using RPython.

How do I get a copy?

PyPy can be built using the usual instructions found here. As soon as the next PyPy version has been released we will provide binaries. Until then you can just grab a nightly here.We are currently busy to get the next version of PyPy ready, so an official release will be rolled out soon.

Comparing s390x to x86

The goal of this comparison is not to scientifically evaluate the benefits/disadvantages on s390x, but rather to see that PyPy's architecture delivers the same benefits as it does on other platforms. Similar to the comparison done for PPC I ran the benchmarks using the same setup. The first column is the speedup of the PyPy JIT VM compared to the speedup of a pure PyPy interpreter 1). Note that the s390x's OS was virtualized.

Label               x86     s390x      s390x (run 2)

ai                 13.7      12.4       11.9
bm_chameleon        8.5       6.3        6.8
bm_dulwich_log      5.1       5.0        5.1
bm_krakatau         5.5       2.0        2.0
bm_mako             8.4       5.8        5.9
bm_mdp              2.0       3.8        3.8
chaos              56.9      52.6       53.4
crypto_pyaes       62.5      64.2       64.2
deltablue           3.3       3.9        3.6
django             28.8      22.6       21.7
eparse              2.3       2.5        2.6
fannkuch            9.1       9.9       10.1
float              13.8      12.8       13.8
genshi_text        16.4      10.5       10.9
genshi_xml          8.2       7.9        8.2
go                  6.7       6.2       11.2
hexiom2            24.3      23.8       23.5
html5lib            5.4       5.8        5.7
json_bench         28.8      27.8       28.1
meteor-contest      5.1       4.2        4.4
nbody_modified     20.6      19.3       19.4
pidigits            1.0      -1.1       -1.0
pyflate-fast        9.0       8.7        8.5
pypy_interp         3.3     4.2        4.4
raytrace-simple    69.0     100.9       93.4
richards           94.1      96.6       84.3
rietveld            3.2       2.5        2.7
slowspitfire        2.8       3.3        4.2
spambayes           5.0       4.8        4.8
spectral-norm      41.9      39.8       42.6
spitfire            3.8       3.9        4.3
spitfire_cstringio 7.6       7.9        8.2
sympy_expand        2.9       1.8        1.8
sympy_integrate     4.3       3.9        4.0
sympy_str           1.5       1.3        1.3
sympy_sum           6.2       5.8        5.9
telco              61.2      48.5       54.8
twisted_iteration 55.5      41.9       43.8
twisted_names       8.2       9.3        9.7
twisted_pb         12.1      10.4       10.2
twisted_tcp         4.9       4.8        5.2

Geometric mean:    9.31      9.10       9.43

As you can see the benefits are comparable on both platforms.
Of course this is scientifically not good enough, but it shows a tendency. s390x can achieve the same results as you can get on x86.

Are you running your business application on a mainframe? We would love to get some feedback. Join us in IRC tell us if PyPy made your application faster!

plan_rich & the PyPy Team

1) PyPy revision for the benchmarks: 4b386bcfee54

Warmup improvements: more efficient trace representation

2016-04-07T10:56:00.002+02:00

Hello everyone.

I'm pleased to inform that we've finished another round of improvements to the warmup performance of PyPy. Before I go into details, I'll recap the achievements that we've done since we've started working on the warmup performance. I picked a random PyPy from November 2014 (which is definitely before we started the warmup work) and compared it with a recent one, after 5.0. The exact revisions are respectively ffce4c795283 and cfbb442ae368. First let's compare pure warmup benchmarks that can be found in our benchmarking suite. Out of those, pypy-graph-alloc-removal numbers should be taken with a grain of salt, since other work could have influenced the results. The rest of the benchmarks mentioned is bottlenecked purely by warmup times.

You can see how much your program spends in warmup running PYPYLOG=jit-summary:- pypy your-program.py under "tracing" and "backend" fields (in the first three lines). An example looks like that:

[e00c145a41] {jit-summary
Tracing:        71      0.053645 <- time spent tracing & optimizing
Backend:        71      0.028659 <- time spent compiling to assembler
TOTAL:                  0.252217 <- total run time of the program

The results of the benchmarks

benchmark	time - old	time - new	speedup	JIT time - old	JIT time - new
function_call	1.86	1.42	1.3x	1.12s	0.57s
function_call2	5.17s	2.73s	1.9x	4.2s	1.6s
bridges	2.77s	2.07s	1.3x	1.5s	0.8s
pypy-graph-alloc-removal	2.06s	1.65s	1.25x	1.25s	0.79s

As we can see, the overall warmup benchmarks got up to 90% faster with JIT time dropping by up to 2.5x. We have more optimizations in the pipeline, with an idea how to transfer some of the JIT gains into more of a total program runtime by jitting earlier and more eagerly.

Details of the last round of optimizations

Now the nitty gritty details - what did we actually do? I covered a lot of warmup improvements in the past blog posts so I'm going to focus on the last change, the jit-leaner-frontend branch. This last change is simple, instead of using pointers to store the "operations" objects created during tracing, we use a compact list of 16-bit integers (with 16bit pointers in between). On 64bit machine the memory wins are tremendous - the new representation is 4x more efficient to use 16bit pointers than full 64bit pointers. Additionally, the smaller representation has much better cache behavior and much less pointer chasing in memory. It also has a better defined lifespan, so we don't need to bother tracking them by the GC, which also saves quite a bit of time.

The change sounds simple, but the details in the underlaying data mean that everything in the JIT had to be changed which took quite a bit of effort :-)

Going into the future on the JIT front, we have an exciting set of optimizations, ranging from faster loops through faster warmup to using better code generation techniques and broadening the kind of program that PyPy speeds up. Stay tuned for the updates.

We would like to thank our commercial partners for making all of this possible. The work has been performed by baroquesoftware and would not be possible without support from people using PyPy in production. If your company uses PyPy and want it to do more or does not use PyPy but has performance problems with the Python installation, feel free to get in touch with me, trust me using PyPy ends up being a lot cheaper than rewriting everything in go :-)

Best regards,
Maciej Fijalkowski

PyPy 5.0.1 bugfix released

2016-03-19T17:44:00.000+01:00

PyPy 5.0.1

We have released a bugfix for PyPy 5.0, after reports that the newly released lxml 3.6.0, which now supports PyPy 5.0 +, can crash on large files. Thanks to those who reported the crash. Please update, downloads are available at

pypy.org/download.html

The changes between PyPy 5.0 and 5.0.1 are only two bug fixes: one in cpyext, which fixes notably (but not only) lxml; and another for a corner case of the JIT.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (PyPy and CPython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, FreeBSD), newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and the big- and little-endian variants of PPC64 running Linux.

Please update, and continue to help us make PyPy better.

Cheers
The PyPy Team

PyPy 5.0 released

2016-03-10T18:03:00.000+01:00

PyPy 5.0

We have released PyPy 5.0, about three months after PyPy 4.0.1. We encourage all users of PyPy to update to this version.

You can download the PyPy 5.0 release here:

http://pypy.org/download.html

Faster and Leaner

We continue to improve the warmup time and memory usage of JIT-related metadata. The exact effects depend vastly on the program you’re running and can range from insignificant to warmup being up to 30% faster and memory dropping by about 30%.

C-API Upgrade

We also merged a major upgrade to our C-API layer (cpyext), simplifying the interaction between c-level objects and PyPy interpreter level objects. As a result, lxml (prerelease) with its cython compiled component passes all tests on PyPy. The new cpyext is also much faster. This major refactoring will soon be followed by an expansion of our C-API compatibility.

Profiling with vmprof supported on more platforms

vmprof has been a go-to profiler for PyPy on linux for a few releases and we’re happy to announce that thanks to the cooperation with jetbrains, vmprof now works on Linux, OS X and Windows on both PyPy and CPython.

CFFI

While not applicable only to PyPy, cffi is arguably our most significant contribution to the python ecosystem. PyPy 5.0 ships with cffi-1.5.2 which now allows embedding PyPy (or CPython) in a C program.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (pypy and cpython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, freebsd), newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and 64 bit PowerPC hardware, specifically Linux running the big- and little-endian variants of ppc64.

Other Highlights (since 4.0.1 released in November 2015)

New features:
- Support embedding PyPy in a C-program via cffi and static callbacks in cffi.
  This deprecates the old method of embedding PyPy
- Refactor vmprof to work cross-operating-system, deprecate using buggy
  libunwind on Linux platforms. Vmprof even works on Windows now.
- Support more of the C-API type slots, like tp_getattro, and fix C-API
  macros, functions, and structs such as _PyLong_FromByteArray(),
  PyString_GET_SIZE, f_locals in PyFrameObject, Py_NAN, co_filename in
  PyCodeObject
- Use a more stable approach for allocating PyObjects in cpyext. (see
  blog post). Once the PyObject corresponding to a PyPy object is created,
  it stays around at the same location until the death of the PyPy object.
  Done with a little bit of custom GC support. It allows us to kill the
  notion of “borrowing” inside cpyext, reduces 4 dictionaries down to 1, and
  significantly simplifies the whole approach (which is why it is a new
  feature while technically a refactoring) and allows PyPy to support the
  populart lxml module (as of the next release) with no PyPy specific
  patches needed
- Make the default filesystem encoding ASCII, like CPython
- Use hypothesis in test creation, which is great for randomizing tests
Bug Fixes
- Backport always using os.urandom for uuid4 from cpython and fix the JIT as well
  (issue #2202)
- More completely support datetime, optimize timedelta creation
- Fix for issue #2185 which caused an inconsistent list of operations to be
  generated by the unroller, appeared in a complicated DJango app
- Fix an elusive issue with stacklets on shadowstack which showed up when
  forgetting stacklets without resuming them
- Fix entrypoint() which now acquires the GIL
- Fix direct_ffi_call() so failure does not bail out before setting CALL_MAY_FORCE
- Fix (de)pickling long values by simplifying the implementation
- Fix RPython rthread so that objects stored as threadlocal do not force minor
  GC collection and are kept alive automatically. This improves perfomance of
  short-running Python callbacks and prevents resetting such object between
  calls
- Support floats as parameters to itertools.isslice()
- Check for the existence of CODESET, ignoring it should have prevented PyPy
  from working on FreeBSD
- Fix for corner case (likely shown by Krakatau) for consecutive guards with
  interdependencies
- Fix applevel bare class method comparisons which should fix pretty printing
  in IPython
- Issues reported with our previous release were resolved after reports from users on our issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy
Numpy:
- Updates to numpy 1.10.2 (incompatibilities and not-implemented features
  still exist)
- Support dtype=((‘O’, spec)) union while disallowing record arrays with
  mixed object, non-object values
- Remove all traces of micronumpy from cpyext if –withoutmod-micronumpy option used
- Support indexing filtering with a boolean ndarray
- Support partition() as an app-level function, together with a cffi wrapper
  in pypy/numpy, this now provides partial support for partition()
Performance improvements:
- Optimize global lookups
- Improve the memory signature of numbering instances in the JIT. This should
  massively decrease the amount of memory consumed by the JIT, which is
  significant for most programs. Also compress the numberings using variable-
  size encoding
- Optimize string concatenation
- Use INT_LSHIFT instead of INT_MUL when possible
- Improve struct.unpack by casting directly from the underlying buffer.
  Unpacking floats and doubles is about 15 times faster, and integer types
  about 50% faster (on 64 bit integers). This was then subsequently
  improved further in optimizeopt.py.
- Optimize two-tuple lookups in mapdict, which improves warmup of instance
  variable access somewhat
- Reduce all guards from int_floordiv_ovf if one of the arguments is constant
- Identify permutations of attributes at instance creation, reducing the
  number of bridges created
- Greatly improve re.sub() performance
Internal refactorings:
- Refactor and improve exception analysis in the annotator
- Remove unnecessary special handling of space.wrap().
- Support list-resizing setslice operations in RPython
- Tweak the trace-too-long heuristic for multiple jit drivers
- Refactor bookkeeping (such a cool word - three double letters) in the
  annotater
- Refactor wrappers for OS functions from rtyper to rlib and simplify them
- Simplify backend loading instructions to only use four variants
- Simplify GIL handling in non-jitted code
- Refactor naming in optimizeopt
- Change GraphAnalyzer to use a more precise way to recognize external
  functions and fix null pointer handling, generally clean up external
  function handling
- Remove pure variants of getfield_gc_* operations from the JIT by
  determining purity while tracing
- Refactor databasing
- Simplify bootstrapping in cpyext
- Refactor rtyper debug code into python.rtyper.debug
- Seperate structmember.h from Python.h Also enhance creating api functions
  to specify which header file they appear in (previously only pypy_decl.h)
- Fix tokenizer to enforce universal newlines, needed for Python 3 support

Please try it out and let us know what you think. We welcome feedback, we know you are using PyPy, please tell us about it!
Cheers
The PyPy Team

C-API Support update

2016-02-25T16:54:00.001+01:00

As you know, PyPy can emulate the CPython C API to some extent. In this post I will describe an important optimization that we merged to improve the performance and stability of the C-API emulation layer.

The C-API is implemented by passing around PyObject * pointers in the C code. The problem with providing the same interface with PyPy is that objects don't natively have the same PyObject * structure at all; and additionally their memory address can change. PyPy handles the difference by maintaining two sets of objects. More precisely, starting from a PyPy object, it can allocate on demand a PyObject structure and fill it with information that points back to the original PyPy objects; and conversely, starting from a C-level object, it can allocate a PyPy-level object and fill it with information in the opposite direction.

I have merged a rewrite of the interaction between C-API C-level objects and PyPy's interpreter level objects. This is mostly a simplification based on a small hack in our garbage collector. This hack makes the garbage collector aware of the reference-counted PyObject structures. When it considers a pair consisting of a PyPy object and a PyObject, it will always free either none or both of them at the same time. They both stay alive if either there is a regular GC reference to the PyPy object, or the reference counter in the PyObject is bigger than zero.

This gives a more stable result. Previously, a PyPy object might grow a corresponding PyObject, loose it (when its reference counter goes to zero), and later have another corresponding PyObject re-created at a different address. Now, once a link is created, it remains alive until both objects die.

The rewrite significantly simplifies our previous code (which used to be based on at least 4 different dictionaries), and should make using the C-API somewhat faster (though it is still slower than using pure python or cffi).

A side effect of this work is that now PyPy actually supports the upstream lxml package---which is is one of the most popular packages on PyPI. (Specifically, you need version 3.5.0 with this pull request to remove old PyPy-specific hacks that were not really working. See details.) At this point, we no longer recommend using the cffi-lxml alternative: although it may still be faster, it might be incomplete and old.

We are actively working on extending our C-API support, and hope to soon merge a branch to support more of the C-API functions (some numpy news coming!). Please try it out and let us know how it works for you.

Armin Rigo and the PyPy team

Using CFFI for embedding

2016-01-06T12:17:00.000+01:00

Introduction

CFFI has been a great success so far to call C libraries in your Python programs, in a way that is both simple and that works across CPython 2.x and 3.x and PyPy.

This post assumes that you know what CFFI is and how to use it in API mode (ffi.cdef(), ffi.set_source(), ffi.compile()). A quick overview can be found in this paragraph.

The major news of CFFI 1.4, released last december, was that you can now declare C functions with extern "Python" in the cdef(). These magic keywords make the function callable from C (where it is defined automatically), but calling it will call some Python code (which you attach with the @ffi.def_extern() decorator). This is useful because it gives a more straightforward, faster and libffi-independent way to write callbacks. For more details, see the documentation.

You are, in effect, declaring a static family of C functions which call Python code. The idea is to take pointers to them, and pass them around to other C functions, as callbacks. However, the idea of a set of C functions which call Python code opens another path: embedding Python code inside non-Python programs.

Embedding

Embedding is traditionally done using the CPython C API: from C code, you call Py_Initialize() and then some other functions like PyRun_SimpleString(). In the simple cases it is, indeed, simple enough; but it can become a complicated story if you throw in supporting application-dependent object types; and a messy story if you add correctly running on multiple threads, for example.

Moreover, this approach is specific to CPython (2.x or 3.x). It does not work at all on PyPy, which has its own very different, minimal embedding API.

The new-and-coming thing about CFFI 1.5, meant as replacement of the above solutions, is direct embedding support---with no fixed API at all. The idea is to write some Python script with a cdef() which declares a number of extern "Python" functions. When running the script, it creates the C source code and compiles it to a dynamically-linked library (.so on Linux). This is the same as in the regular API-mode usage. What is new is that these extern "Python" can now also be exported from the .so, in the C sense. You also give a bit of initialization-time Python code directly in the script, which will be compiled into the .so too.

This library can now be used directly from any C program (and it is still importable in Python). It exposes the C API of your choice, which you specified with the extern "Python" declarations. You can use it to make whatever custom API makes sense in your particular case. You can even directly make a "plug-in" for any program that supports them, just by exporting the API expected for such plugins.

Trying it out on CPython

This is still being finalized, but please try it out. You can see embedding.py directly online for a quick glance. Or see below the instructions on Linux with CPython 2.7 (CPython 3.x and non-Linux platforms are still a work in progress right now, but this should be quickly fixed):

get the branch static-callback-embedding of CFFI:

hg clone https://bitbucket.org/cffi/cffi
hg up static-callback-embedding

make the _cffi_backend.so:
```
python setup_base.py build_ext -f -i
```

run embedding.py in the demo directory:

cd demo
PYTHONPATH=.. python embedding.py

this produces _embedding_cffi.c. Run gcc to build it. On Linux:

gcc -shared -fPIC _embedding_cffi.c -o _embedding_cffi.so  \
    -lpython2.7 -I/usr/include/python2.7

try out the demo C program in embedding_test.c:

gcc embedding_test.c _embedding_cffi.so
PYTHONPATH=.. LD_LIBRARY_PATH=. ./a.out

Note that if you get ImportError: cffi extension module '_embedding_cffi' has unknown version 0x2701, it means that the _cffi_backend module loaded is a pre-installed one instead of the more recent one in "..". Be sure to use PYTHONPATH=.. for now. (Some installations manage to be confused enough to load the system-wide cffi even if another version is in the PYTHONPATH. I think a virtualenv can be used to work around this issue.)

Try it out on PyPy

Very similar steps can be followed on PyPy, but it requires the cffi-static-callback-embedding branch of PyPy, which you must first translate from sources. The difference is then that you need to adapt the first gcc command line: replace -lpython2.7 with -lpypy-c and to fix the -I path (and possibly add a -L path).

More details

How it works, more precisely, is by automatically initializing CPython/PyPy the first time any of the extern "Python" functions is called from the C program. This is done using locks in case of multi-threading, so several threads can concurrently do this "first call". This should work even if two different threads call the first time a function from two different embedded CFFI extensions that happen to be linked with the same program. Explicit initialization is never needed.

The custom initialization-time Python code you put in ffi.embedding_init_code() is executed at that time. If this code starts to be big, you can move it to independent modules or packages. Then the initialization-time Python code only needs to import them. In that case, you have to carefully set up sys.path if the modules are not installed in the usual Python way.

If the Python code is big and full of dependencies, a better alternative would be to use virtualenv. How to do that is not fully fleshed out so far. You can certainly run the whole program with the environment variables set up by the virtualenv's activate script first. There are probably other solutions that involve using gcc's -Wl,-rpath=\$ORIGIN/ or -Wl,-rpath=/fixed/path/ options to load a specific libpython or libypypy-c library. If you try it out and it doesn't work the way you would like, please complain :-)

Another point: right now this does not support CPython's notion of multiple subinterpreters. The logic creates a single global Python interpreter, and runs everything in that context. Maybe a future version would have an explicit API to do that — or maybe it should be the job of a 3rd-party extension module to provide a Python interface over the notion of subinterpreters...

More generally, any feedback is appreciated.

Have fun,

Armin

Leysin Winter Sprint (20-27th February 2016)

2016-01-04T13:24:00.001+01:00

The next PyPy sprint will be in Leysin, Switzerland, for the eleventh time. This is a fully public sprint: newcomers and topics other than those proposed below are welcome.

Goals and topics of the sprint

The details depend on who is here and ready to work. The list of topics is mostly the same as last year (did PyPy became a mature project with only long-term goals?):

cpyext (CPython C API emulation layer): various speed and completeness topics
cleaning up the optimization step in the JIT, change the register allocation done by the JIT's backend, or more improvements to the warm-up time
finish vmprof - a statistical profiler for CPython and PyPy
Py3k (Python 3.x support), NumPyPy (the numpy module)
STM (Software Transaction Memory), notably: try to come up with benchmarks, and measure them carefully in order to test and improve the conflict reporting tools, and more generally to figure out how practical it is in large projects to avoid conflicts
And as usual, the main side goal is to have fun in winter sports :-) We can take a day off for ski.

Exact times

I have booked the week from Saturday 20 to Saturday 27. It is fine to leave either the 27 or the 28, or even stay a few more days on either side. The plan is to work full days between the 21 and the 27. You are of course allowed to show up for a part of that time only, too.

Location & Accomodation

Leysin, Switzerland, "same place as before". Let me refresh your memory: both the sprint venue and the lodging will be in a pair of chalets built specifically for bed & breakfast: http://www.ermina.ch/. The place has a good ADSL Internet connection with wireless installed. You can also arrange your own lodging elsewhere (as long as you are in Leysin, you cannot be more than a 15 minutes walk away from the sprint venue).

Please confirm that you are coming so that we can adjust the reservations as appropriate.

The options of rooms are a bit more limited than on previous years because the place for bed-and-breakfast is shrinking: what is guaranteed is only one double-bed room and a bigger room with 5-6 individual beds (the latter at 50-60 CHF per night, breakfast included). If there are more people that would prefer a single room, please contact me and we'll see what choices you have. There are a choice of hotels, many of them reasonably priced for Switzerland.

Please register by Mercurial:

https://bitbucket.org/pypy/extradoc/
https://bitbucket.org/pypy/extradoc/raw/extradoc/sprintinfo/leysin-winter-2016

or on the pypy-dev mailing list if you do not yet have check-in rights:

http://mail.python.org/mailman/listinfo/pypy-dev

You need a Swiss-to-(insert country here) power adapter. There will be some Swiss-to-EU adapters around, and at least one EU-format power strip.

PyPy 4.0.1 released please update

2015-11-19T22:01:00.000+01:00

PyPy 4.0.1

We have released PyPy 4.0.1, three weeks after PyPy 4.0.0. We have fixed a few critical bugs in the JIT compiled code, reported by users. We therefore encourage all users of PyPy to update to this version. There are a few minor enhancements in this version as well.

You can download the PyPy 4.0.1 release here:

http://pypy.org/download.html

CFFI update

While not applicable only to PyPy, cffi is arguably our most significant contribution to the python ecosystem. PyPy 4.0.1 ships with cffi-1.3.1 with the improvements it brings.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (pypy and cpython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, freebsd), newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux, and the big- and little-endian variants of ppc64 running Linux.

Other Highlights (since 4.0.0 released three weeks ago)

Bug Fixes
- Fix a bug when unrolling double loops in JITted code
- Fix multiple memory leaks in the ssl module, one of which affected CPython as well (thanks to Alex Gaynor for pointing those out)
- Use pkg-config to find ssl headers on OS-X
- Issues reported with our previous release were resolved after reports from users on our issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy
New features
- Internal cleanup of RPython class handling
- Support stackless and greenlets on PPC machines
- Improve debug logging in subprocesses: use PYPYLOG=jit:log.%d for example to have all subprocesses write the JIT log to a file called ‘log.%d’, with ‘%d’ replaced with the subprocess’ PID.
- Support PyOS_double_to_string in our cpyext capi compatibility layer
Numpy
- Improve support for __array_interface__
- Propagate most NAN mantissas through float16-float32-float64 conversions
Performance improvements and refactorings
- Improvements in slicing byte arrays
- Improvements in enumerate()
- Silence some warnings while translating

Please update, and continue to help us make PyPy better.

Cheers
The PyPy Team

PyPy 4.0.0 Released - A Jit with SIMD Vectorization and More

2015-10-29T13:17:00.001+01:00

PyPy 4.0.0

We’re pleased and proud to unleash PyPy 4.0.0, a major update of the PyPy python 2.7.10 compatible interpreter with a Just In Time compiler. We have improved warmup time and memory overhead used for tracing, added vectorization for numpy and general loops where possible on x86 hardware (disabled by default), refactored rough edges in rpython, and increased functionality of numpy.
You can download the PyPy 4.0.0 release here:

http://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project.
We would also like to thank our contributors (7 new ones since PyPy 2.6.0) and encourage new people to join the project. PyPy has many layers and we need help with all of them: PyPy and RPython documentation improvements, tweaking popular modules to run on PyPy, or general help with making RPython’s JIT even better.

New Version Numbering

Since the past release, PyPy 2.6.1, we decided to update the PyPy 2.x.x versioning directly to PyPy 4.x.x, to avoid confusion with CPython 2.7 and 3.5. Note that this version of PyPy uses the stdlib and implements the syntax of CPython 2.7.10.

Vectorization

Richard Plangger began work in March and continued over a Google Summer of Code to add a vectorization step to the trace optimizer. The step recognizes common constructs and emits SIMD code where possible, much as any modern compiler does. This vectorization happens while tracing running code, so it is actually easier at run-time to determine the availability of possible vectorization than it is for ahead-of-time compilers.
Availability of SIMD hardware is detected at run time, without needing to precompile various code paths into the executable.
The first version of the vectorization has been merged in this release, since it is so new it is off by default. To enable the vectorization in built-in JIT drivers (like numpy ufuncs), add –jit vec=1, to enable all implemented vectorization add –jit vec_all=1
Benchmarks and a summary of this work appear here

Internal Refactoring: Warmup Time Improvement and Reduced Memory Usage

Maciej Fijalkowski and Armin Rigo refactored internals of Rpython that now allow PyPy to more efficiently use guards in jitted code. They also rewrote unrolling, leading to a warmup time improvement of 20% or so. The reduction in guards also means a reduction in the use of memory, also a savings of around 20%.

Numpy

Our implementation of numpy continues to improve. ndarray and the numeric dtypes are very close to feature-complete; record, string and unicode dtypes are mostly supported. We have reimplemented numpy linalg, random and fft as cffi-1.0 modules that call out to the same underlying libraries that upstream numpy uses. Please try it out, especially using the new vectorization (via –jit vec=1 on the command line) and let us know what is missing for your code.

CFFI

While not applicable only to PyPy, cffi is arguably our most significant contribution to the python ecosystem. Armin Rigo continued improving it, and PyPy reaps the benefits of cffi-1.3: improved manangement of object lifetimes, __stdcall on Win32, ffi.memmove(), and percolate const, restrict keywords from cdef to C code.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It’s fast (pypy and cpython 2.7.x performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
This release supports x86 machines on most common operating systems (Linux 32/64, Mac OS X 64, Windows 32, OpenBSD, freebsd), as well as newer ARM hardware (ARMv6 or ARMv7, with VFPv3) running Linux.
We also introduce support for the 64 bit PowerPC hardware, specifically Linux running the big- and little-endian variants of ppc64.

Other Highlights (since 2.6.1 release two months ago)

Bug Fixes
- Applied OPENBSD downstream fixes
- Fix a crash on non-linux when running more than 20 threads
- In cffi, ffi.new_handle() is more cpython compliant
- Accept unicode in functions inside the _curses cffi backend exactly like cpython
- Fix a segfault in itertools.islice()
- Use gcrootfinder=shadowstack by default, asmgcc on linux only
- Fix ndarray.copy() for upstream compatability when copying non-contiguous arrays
- Fix assumption that lltype.UniChar is unsigned
- Fix a subtle bug with stacklets on shadowstack
- Improve support for the cpython capi in cpyext (our capi compatibility layer). Fixing these issues inspired some thought about cpyext in general, stay tuned for more improvements
- When loading dynamic libraries, in case of a certain loading error, retry loading the library assuming it is actually a linker script, like on Arch and Gentoo
- Issues reported with our previous release were resolved after reports from users on our issue tracker at https://bitbucket.org/pypy/pypy/issues or on IRC at #pypy
New features:
- Add an optimization pass to vectorize loops using x86 SIMD intrinsics.
- Support __stdcall on Windows in CFFI
- Improve debug logging when using PYPYLOG=???
- Deal with platforms with no RAND_egd() in OpenSSL
Numpy:
- Add support for ndarray.ctypes
- Fast path for mixing numpy scalars and floats
- Add support for creating Fortran-ordered ndarrays
- Fix casting failures in linalg (by extending ufunc casting)
- Recognize and disallow (for now) pickling of ndarrays with objects embedded in them
Performance improvements and refactorings:
- Reuse hashed keys across dictionaries and sets
- Refactor JIT interals to improve warmup time by 20% or so at the cost of a minor regression in JIT speed
- Recognize patterns of common sequences in the JIT backends and optimize them
- Make the garbage collecter more incremental over external_malloc() calls
- Share guard resume data where possible which reduces memory usage
- Fast path for zip(list, list)
- Reduce the number of checks in the JIT for lst[a:]
- Move the non-optimizable part of callbacks outside the JIT
- Factor in field immutability when invalidating heap information
- Unroll itertools.izip_longest() with two sequences
- Minor optimizations after analyzing output from vmprof and trace logs
- Remove many class attributes in rpython classes
- Handle getfield_gc_pure* and getfield_gc_* uniformly in heap.py
- Improve simple trace function performance by lazily calling fast2locals and locals2fast only if truly necessary

Please try it out and let us know what you think. We welcome feedback, we know you are using PyPy, please tell us about it!
Cheers
The PyPy Team

Automatic SIMD vectorization support in PyPy

2015-10-20T15:38:00.000+02:00

Hi everyone,

it took some time to catch up with the JIT refacrtorings merged in this summer. But, (drums) we are happy to announce that:

The next release of PyPy, "PyPy 4.0.0", will ship the new auto vectorizer

The goal of this project was to increase the speed of numerical applications in both the NumPyPy library and for arbitrary Python programs. In PyPy we have focused a lot on improvements in the 'typical python workload', which usually involves object and string manipulations, mostly for web development. We're hoping with this work that we'll continue improving the other very important Python use case - numerics.

What it can do!

It targets numerics only. It will not execute object manipulations faster, but it is capable of enhancing common vector and matrix operations.
Good news is that it is not specifically targeted for the NumPy library and the PyPy virtual machine. Any interpreter (written in RPython) is able make use of the vectorization. For more information about that take a look here, or consult the documentation. For the time being it is not turn on by default, so be sure to enable it by specifying --jit vec=1 before running your program.

If your language (written in RPython) contains many array/matrix operations, you can easily integrate the optimization by adding the parameter 'vec=1' to the JitDriver.

NumPyPy Improvements

Let's take a look at the core functions of the NumPyPy library (*).
The following tests tests show the speedup of the core functions commonly used in Python code interfacing with NumPy, on CPython with NumPy, on the PyPy 2.6.1 relased several weeks ago, and on PyPy 15.11 to be released soon. Timeit was used to test the time needed to run the operation in the plot title on various vector (lower case) and square matrix (upper case) sizes displayed on the X axis. The Y axis shows the speedup compared to CPython 2.7.10. This means that higher is better.

In comparison to PyPy 2.6.1, the speedup greatly improved. The hardware support really strips down the runtime of the vector and matrix operations. There is another operation we would like to highlight: the dot product.
It is a very common operation in numerics and PyPy now (given a moderate sized matrix and vector) decreases the time spent in that operation. See for yourself:

These are nice improvements in the NumPyPy library and we got to a competitive level only making use of SSE4.1.

Future work

This is not the end of the road. The GSoC project showed that it is possible to implement this optimization in PyPy. There might be other improvements we can make to carry this further:

Check alignment at runtime to increase the memory throughput of the CPU
Support the AVX vector extension which (at least) doubles the size of the vector register
Handle each and every corner case in Python traces to enable it globally
Do not rely only on loading operations to trigger the analysis, there might be cases where combination of floating point values could be done in parallel

Cheers,
The PyPy Team

(*) The benchmark code can be found here it was run using this configuration: i7-2600 CPU @ 3.40GHz (4 cores).

PowerPC backend for the JIT

2015-10-16T18:08:00.000+02:00

Hi all,

PyPy's JIT now supports the 64-bit PowerPC architecture! This is the third architecture supported, in addition to x86 (32 and 64) and ARM (32-bit only). More precisely, we support Linux running the big- and the little-endian variants of ppc64. Thanks to IBM for funding this work!

The new JIT backend has been merged into "default". You should be able to translate PPC versions as usual directly on the machines. For the foreseeable future, I will compile and distribute binary versions corresponding to the official releases (for Fedora), but of course I'd welcome it if someone else could step in and do it. Also, it is unclear yet if we will run a buildbot.

To check that the result performs well, I logged in a ppc64le machine and ran the usual benchmark suite of PyPy (minus sqlitesynth: sqlite was not installed on that machine). I ran it twice at a difference of 12 hours, as an attempt to reduce risks caused by other users suddenly using the machine. The machine was overall relatively quiet. Of course, this is scientifically not good enough; it is what I could come up with given the limited resources.

Here are the results, where the numbers are speed-up factors between the non-jit and the jit version of PyPy. The first column is x86-64, for reference. The second and third columns are the two ppc64le runs. All are Linux. A few benchmarks are not reported here because the runner doesn't execute them on non-jit (however, apart from sqlitesynth, they all worked).

    ai                        13.7342        16.1659     14.9091
    bm_chameleon               8.5944         8.5858        8.66
    bm_dulwich_log             5.1256         5.4368      5.5928
    bm_krakatau                5.5201         2.3915      2.3452
    bm_mako                    8.4802         6.8937      6.9335
    bm_mdp                     2.0315         1.7162      1.9131
    chaos                     56.9705        57.2608     56.2374
    sphinx
    crypto_pyaes               62.505         80.149     79.7801
    deltablue                  3.3403         5.1199      4.7872
    django                    28.9829         23.206       23.47
    eparse                     2.3164         2.6281       2.589
    fannkuch                   9.1242        15.1768     11.3906
    float                     13.8145        17.2582     17.2451
    genshi_text               16.4608        13.9398     13.7998
    genshi_xml                 8.2782         8.0879      9.2315
    go                         6.7458        11.8226     15.4183
    hexiom2                   24.3612        34.7991     33.4734
    html5lib                   5.4515         5.5186       5.365
    json_bench                28.8774        29.5022     28.8897
    meteor-contest             5.1518         5.6567      5.7514
    nbody_modified            20.6138        22.5466     21.3992
    pidigits                   1.0118          1.022      1.0829
    pyflate-fast               9.0684        10.0168     10.3119
    pypy_interp                3.3977         3.9307      3.8798
    raytrace-simple           69.0114       108.8875    127.1518
    richards                  94.1863       118.1257    102.1906
    rietveld                   3.2421         3.0126      3.1592
    scimark_fft
    scimark_lu
    scimark_montecarlo
    scimark_sor
    scimark_sparsematmul
    slowspitfire               2.8539         3.3924      3.5541
    spambayes                  5.0646         6.3446       6.237
    spectral-norm             41.9148        42.1831     43.2913
    spitfire                   3.8788         4.8214       4.701
    spitfire_cstringio          7.606         9.1809      9.1691
    sqlitesynth
    sympy_expand               2.9537         2.0705      1.9299
    sympy_integrate            4.3805         4.3467      4.7052
    sympy_str                  1.5431         1.6248      1.5825
    sympy_sum                  6.2519          6.096      5.6643
    telco                     61.2416        54.7187     55.1705
    trans2_annotate
    trans2_rtype
    trans2_backendopt
    trans2_database
    trans2_source
    twisted_iteration         55.5019        51.5127     63.0592
    twisted_names              8.2262         9.0062      10.306
    twisted_pb                12.1134         13.644     12.1177
    twisted_tcp                4.9778          1.934      5.4931

    GEOMETRIC MEAN               9.31           9.70       10.01

The last line reports the geometric mean of each column. We see that the goal was reached: PyPy's JIT actually improves performance by a factor of around 9.7 to 10 times on ppc64le. By comparison, it "only" improves performance by a factor 9.3 on Intel x86-64. I don't know why, but I'd guess it mostly means that a non-jitted PyPy performs slightly better on Intel than it does on PowerPC.

Why is that? Actually, if we do the same comparison with an ARM column too, we also get higher numbers there than on Intel. When we discovered that a few years ago, we guessed that on ARM running the whole interpreter in PyPy takes up a lot of resources, e.g. of instruction cache, which the JIT's assembler doesn't need any more after the process is warmed up. And caches are much bigger on Intel. However, PowerPC is much closer to Intel, so this argument doesn't work for PowerPC. But there are other more subtle variants of it. Notably, Intel is doing crazy things about branch prediction, which likely helps a big interpreter---both the non-JITted PyPy and CPython, and both for the interpreter's main loop itself and for the numerous indirect branches that depend on the types of the objects. Maybe the PowerPC is as good as Intel, and so this argument doesn't work either. Another one would be: on PowerPC I did notice that gcc itself is not perfect at optimization. During development of this backend, I often looked at assembler produced by gcc, and there are a number of small inefficiencies there. All these are factors that slow down the non-JITted version of PyPy, but don't influence the speed of the assembler produced just-in-time.

Anyway, this is just guessing. The fact remains that PyPy can now be used on PowerPC machines. Have fun!

A bientôt,

Armin.

PyPy Status Blog

Leysin Winter Sprint: 25/26th Feb. - 4th March 2017

Goals and topics of the sprint

Exact times

Location & Accomodation

PyPy2.7 v5.6 released - stdlib 2.7.12 support, C-API improvements, and more

What is PyPy?

What else is new?

Vectorization extended. PowerPC and s390x

NumPy & CPyExt

Benchmarks

Setup

Future directions

PyPy3 5.5.0 released

Towards Python 3.5

What is PyPy?

RevDB released, v5.4.1

PyPy 5.4.1 bugfix released

What is PyPy?

PyPy2 v5.4 released - incremental improvements and enhancements

What is PyPy?

What is New?

(since the release of PyPy 5.3 in June, 2016)

PyPy Tooling Upgrade: JitViewer and VMProf

A "brand new" JitViewer

PyPy crashed? Did you discover a bug?

Speed issues

Future direction

PyPy gets funding from Mozilla for Python 3.5 support

Reverse debugging for Python

RevPDB

How to get RevPDB

Demo

Current issues

How RevPDB is done

Other reverse debuggers for Python

Future work

PyPy2 v5.3 released - major C-extension support improvements

What is PyPy?

Other Highlights

(since the release of PyPy 5.1 in April, 2016)

PyPy3.3 v5.2 alpha 1 released

Highlights

What is PyPy?

PyPy 5.1.1 bugfix released

What is PyPy?

PyPy 5.1 released

What is PyPy?

Other Highlights

(since the release of PyPy 5.0 in March, 2016

New features:

Bug Fixes

Numpy:

Performance improvements:

Internal refactorings:

PyPy Enterprise Edition

History

Meta tracing

How do I get a copy?

Comparing s390x to x86

Warmup improvements: more efficient trace representation

Details of the last round of optimizations

PyPy 5.0.1 bugfix released

PyPy 5.0.1

What is PyPy?

PyPy 5.0 released

PyPy 5.0

Faster and Leaner

C-API Upgrade

Profiling with vmprof supported on more platforms

CFFI

What is PyPy?

Other Highlights (since 4.0.1 released in November 2015)

C-API Support update

Using CFFI for embedding

Introduction

Embedding

Trying it out on CPython

Try it out on PyPy

More details