A few months ago, we quietly released Jsonnet: a simple yet rich configuration language (i.e., a programming language for specifying data). Many systems can be configured with JSON, but writing it by hand is troublesome. Jsonnet is packed with useful data-specification features that expand into JSON for other systems to act upon. Below is a trivial example of such expansion:

// Jsonnet Example {    person1: {        name: "Alice",        welcome: "Hello " + self.name + "!",    },    person2: self.person1 { name: "Bob" }, }

➡

{   "person1": {      "name": "Alice",      "welcome": "Hello Alice!"   },   "person2": {      "name": "Bob",      "welcome": "Hello Bob!"   } }

Jsonnet doesn’t just generate JSON: Jsonnet is also an extension of JSON. By adding new constructs between the gaps of existing JSON syntax, Jsonnet adds useful features without breaking backwards compatibility. Any valid JSON is also a valid Jsonnet program that simply emits that JSON unchanged, and existing systems that consume JSON (or its cousin YAML) can be easily modified to accept data in the full Jsonnet language. As such, Jsonnet is an example of a templating language, but one specifically designed for JSON data and less error-prone than other techniques.

“Jsonnet” is a portmanteau of JSON and sonnet. We chose that name to convey that data expressed in Jsonnet is easier to write and maintain because it is more elegant and concise, like a poem. This is not just due to syntactic niceties like comments and permissive quotes/commas, but because Jsonnet has all the modern multi-paradigm programming language conveniences needed to manage complexity. One key benefit is the ability to use Jsonnet's mixin and import features to write modular configuration template libraries, allowing the creation of domain-specific configuration languages for particular applications.

Most configuration languages are created ad-hoc for the needs of a given application, accruing features over time and becoming unwieldy. From day one, Jsonnet was designed as a coherent programming language, benefitting from both academic techniques and our experience implementing production languages. Unlike most configuration languages, Jsonnet has a full operational semantics, ensuring matching behavior from third party implementations as well as mathematical analysis. It is a very small and carefully chosen extension to JSON that can express both object-oriented and declarative styles. More importantly, unlike regular programming languages, Jsonnet is hermetic:  Its evaluation is independent of any implicit environmental factors, ensuring that high level configuration will resolve to the same thing every time.

Jsonnet is open source. It’s currently available as a library with C and Python bindings, and also as a command line utility. A real-world example configuration can be found on the website, where 217 lines (9.7kB) of Jsonnet expand into 740 lines (25kB) of configuration for other tools. Learn more about Jsonnet by reading the tutorial and experimenting with our javascript demo!

by Dave Cunningham, New York Technical Infrastructure team

Years of writing and maintaining Python code have taught us the value of automated tools for code formatting, but the existing ones didn’t quite do what we wanted. In the best traditions of the open source community, it was time to write yet another Python formatter.

YAPF takes a different approach to formatting Python code: it reformats the entire program, not just individual lines or constructs that violate a style guide rule. The ultimate goal is to let  engineers focus on the bigger picture and not worry about the formatting. The end result should look the same as if an engineer had worried about the formatting.

You can run YAPF on the entire program or just a part of the program. It’s also possible to flag certain parts of a program which YAPF shouldn’t alter, which is useful for generated files or sections with large literals.

Consider this horribly-formatted code:

x = {  'a':37,'b':42,

'c':927}

y = 'hello ''world'

z = 'hello '+'world'

a = 'hello {}'.format('world')

class foo  (     object  ):

 def f    (self   ):

   return       \

37*-+2

 def g(self, x,y=42):

     return y

def f  (   a ) :

 return      37+-+a[42-x :  y**3]

YAPF reformats this into something much more consistent and readable:

x = {'a': 37, 'b': 42, 'c': 927}

y = 'hello ' 'world'

z = 'hello ' + 'world'

a = 'hello {}'.format('world')

class foo(object):

   def f(self):

       return 37 * -+2

   def g(self, x, y=42):

       return y

def f(a):

   return 37 + -+a[42 - x:y ** 3]

Head to YAPF's GitHub page for more information on how to use it, and take a look at YAPF’s own source code to see a much larger example of the output it produces.

by Bill Wendling, YouTube Code Health Team

Today on the Open Source blog we have guest writer Sebastien Goasguen, an avid open source contributor and member of the Apache Software Foundation. Below, Sebastien highlights the significant contributions that two Google Summer of Code students have made to Apache CloudStack.

In December 2013, Google announced the General Availability (GA) of the public cloud, Google Compute Engine (GCE).  Apache CloudStack now has a brand new GCE compatible interface (Gstack) which allows users to take advantage of the GCE clients (i.e gcloud and gcutil) to access their CloudStack cloud. This interface was made possible through the Google Summer of Code (GSoC) program.

In the summer of 2013, Ian Duffy, a student from Dublin City University, participated in GSoC through the Apache Software Foundation and worked on a LDAP plugin to CloudStack. He did such a great job that he finished early and was made an Apache CloudStack committer. Since he finished his primary GSoC project so early, I encouraged him to take on another! He brought in a friend for the ride — Darren Brogan, another student at Dublin City University. Together they worked on the GCE interface to CloudStack and even learned Python in doing so.

Both Ian and Darren remained engaged with the CloudStack community and as their third year project in University, they successfully developed an Amazon EC2 interface to CloudStack. Since he enjoyed his experience so much, Darren also applied to the GSoC 2014 program and proposed to revisit Gstack, improve it, extend the unit tests, and make it compatible with the GCE v1 API. He is making excellent progress so far and we are all excited to see the results.

Technically, Gstack is a Python Flask application that provides a REST API compatible with the GCE API and forwards the requests to the corresponding CloudStack API. The source is available on GitHub and the binary is downloadable via PyPi.

Installation and Configuration of Gstack

Are you interested in using Gstack? Check out the full documentation. To get a taste for things, you can grab the binary package from Pypi using pip in one single command.

        pip install gstack

Or if you plan to explore the source and work on it, you can clone the repository and install it by hand. Pull requests are of course welcome.

       git clone https://github.com/NOPping/gstack.git
   
sudo python./setup.py install

Both of these installation methods will install a gstack and a gstack-configure binary in your path. Before running Gstack you must configure it. To do so run:

   gstack-configure

And enter your configuration information when prompted. You will need to specify the host and port where you want gstack to run on, as well as the CloudStack endpoint that you want gstack to forward the requests to. In the example below we use the exoscale cloud:

   $ gstack-configure

   gstack bind address [0.0.0.0]: localhost

   gstack bind port [5000]:
   
Cloudstack host [localhost]: api.exoscale.ch
   
Cloudstack port [8080]: 443
   
Cloudstack protocol [http]: https

   Cloudstack path [/client/api]: /compute

The information will be stored in a configuration file available at ~/.gstack/gstack.conf:

   $ cat ~/.gstack/gstack.conf 

   PATH = 'compute/v1/projects/'

   GSTACK_BIND_ADDRESS = 'localhost'
   
GSTACK_PORT = '5000'
   
CLOUDSTACK_HOST = 'api.exoscale.ch'
   
CLOUDSTACK_PORT = '443'
   
CLOUDSTACK_PROTOCOL = 'https'
 
   CLOUDSTACK_PATH = '/compute'

You are now ready to start Gstack in the foreground with:

   gstack

That's all there is to running Gstack. You can then use gcutil to send requests to gstack which will forward them to a CloudStack endpoint.  Although it is still a work in progress, it is now compatible with GCE GA v1.0 API. It provides a solid base to start working on hybrid solutions between GCE public cloud and a CloudStack based private cloud.

GSoC has been a terrific opportunity for all of us at Apache. Darren and Ian both learned how to work with an open source community and ultimately became an integral part of it. They learned tools like JIRA, git, and Review Board and gained confidence working publicly on mailing lists. Their work on Gstack and EC2stack is certainly of high value to CloudStack and could eventually become the base for interesting products that will use hybrid clouds.

By Sebastien Goasguen, Senior Open Source Architect, Citrix and Apache Software Foundation member

Engineers at Google deal with encoded data on a daily basis. It’s very common to handle files encoded in a variety of different formats. For example, email attachments are Base64 encoded and web requests are URL encoded. Custom encodings bring another level of complication especially when different codings are chained together. Over time this constant need to encode / decode data left me with a large, unmanageable collection of scripts. This collection was simply not scaling, so I set off to create a better solution. We needed something easy to use and extensible enough to serve our future needs.

Today, I’m happy to introduce Babbage, an open source tool for manipulating data in many different formats. With Babbage you can easily decode or encode data with just a click. Paste in “SGVsbG8h”, select base 64 decode and you get “Hello!”. You can paste in text to process with plugins (which are an easy way to transform data). Babbage comes with a basic set of plugins to cover simple encodings and obfuscation techniques such as Base64, URL encoding, XOR and others. If you have something a bit more complicated, you can chain multiple plugins together. Babbage is open source and written so that anyone can create their own collection of plugins with libraries already in use.

Babbage was written in Python and JavaScript with Google Closure on top of Google App Engine. The full source code is available on GitHub. Develop something cool and share it with the world! We are always looking for new contributions — feel free to contact us on our developers discussion group.

By Tom Fitzgerald, Google Engineering

The Google Open Source team recently sponsored the PyTennessee conference in Nashville. Adam Fletcher, an Engineer at Google and today's guest blogger, volunteered at the conference and helped introduce Python to an enthusiastic group of students. 

On February 23rd & 24th the first PyTennessee took place in Nashville, Tennessee, and brought hundreds of pythonistas from all over the nation to learn about a diverse set of Python-related topics. On Saturday the 24th, PyTennessee ran a Young Coders event, based on a similar event that took place at the 2013 US PyCon. Google was proud to sponsor this event, providing funding for the Raspberry Pi computers the coders used throughout the day.

 Mayor of Nashville, Karl Dean, with the students

The Young Coders event introduced 25 new programmers, aged 12-18, to the world of Python by providing each student with a Raspberry Pi running Linux and a day of instruction in the Python programming language. Students were taught about the basic data types and control flow in Python in the morning and then spent the afternoon making and modifying games. When the event wrapped up the students got to take home their Raspberry Pi computers to continue their programming exploration at home. Additionally, the students each got a copy of Python For Kids, an excellent introductory book.

Raspberry Pi, the compact computer the students used to learn Python

Earlier in the day the Mayor of Nashville, Karl Dean, stopped by to learn about the Young Coders event and to talk to the students. Mayor Dean was excited about Nashville as a technology center; Nashville is one of the cities being evaluated for Google Fiber, and Google has selected Nashville as one of the Google for Entrepreneurs Tech Hub Network cities.

Later, the students used their newfound Python knowledge to modify various games. Students altered the startup screen, changed the frame rates, modified the fundamental rules, and made other fun changes to games written in the PyGame framework.

Two students hard at work

Katie Cunningham (right) with two Young Coders

The Young Coders event would not have been successful without its excellent instructor, Katie Cunningham. Big thanks to her and to the entire PyTennessee team for for organizing such a wonderful event, and for providing the space to help train the next generation of computer scientists!

By Adam Fletcher, Google Site Reliability Engineer

"I hear and I forget; I see and I remember; I do and I understand." — Confucius

Lots of online education is delivered using video and text. However, opportunities for learners to do things and get feedback on their work are also important — after all, one does not learn to play the piano by watching videos of many virtuoso performances.

We're excited to announce Oppia, a project that aims to make it easy for anyone to create online interactive activities, called 'explorations', that others can learn from. Oppia does this by modeling a mentor who poses questions for the learner to answer. Based on the learner's responses, the mentor decides what question to ask next, what feedback to give, whether to delve deeper, or whether to proceed to something new. You can think of this as a smart feedback system that tries to “teach a person to fish”, instead of simply revealing the correct answer or marking the submitted answer as wrong. If you’d like to get an idea of what these explorations are like, you can try out some examples at www.oppia.org.

The Oppia learning interface. 

  The Oppia editing interface.

                               

A unique feature of Oppia is that it allows multiple people from around the world to create and collaborate on explorations. They can do this through a web interface — no programming required.

Oppia gathers data on how learners interact with it, making it easy for exploration authors to spot and fix shortcomings in an exploration. They would do this by logging in, finding an answer that many learners are giving but which the system is not responding to adequately, and creating a new learning path for it, based on what they would actually say if they were interacting in-person with the learner. Oppia can then give this feedback to future learners.

A video by Yana Malysheva, one of the developers, explaining how Oppia works.

                       
Oppia knows how to deal with numeric, text, and multiple choice inputs, as well as some more specialized types such as a clickable map and a code evaluator. We've also built an extensible framework that lets developers extend the range of input types that Oppia can understand.

The explorations created on an Oppia server can be embedded in any web page. These embeddings can refer to a particular version, so that further changes to the canonical version of the exploration do not automatically appear in the embedded one. This feature allows learning experiences that have been created using Oppia explorations to retain their integrity over time.

Oppia is built using Python and AngularJS on top of Google App Engine. You can download the source code; we hope you find it useful! Please feel free to contribute suggestions through our issue tracker, or contact us at our developers discussion group. We actively welcome new contributors, so if you would like to help out, please don't hesitate to get in touch.

By Sean Lip, Software Engineer, Google Research

SymPy is a computer algebra system (CAS) written in pure Python. The core allows basic manipulation of expressions (like differentiation or expansion) and it contains many modules for common tasks (limits, integrals, differential equations, series, matrices, quantum physics, geometry, plotting, and code generation).

SymPy has participated in the Google Summer of Code program in previous years under the umbrellas of Python Software Foundation, Portland State University, and the Space Telescope Science Institute, where we were very successful. In fact, several of our core developers, including four of the mentors from this year, started working with SymPy as Google Summer of Code students. This was our first year participating as a standalone organization, and we would like to share our experience.

As part of the application process we required each student to submit a patch (as a GitHub pull request) that had to be reviewed and accepted. This allowed us to see that each applicant knew how to use git as well as communicate effectively during the review process.This also encouraged only serious applicants to apply. We had over 10 mentors available and we ended up with 9 students, all of whom were successful at final evaluations.

Tom Bachmann - Definite Integration using Meijer G-functions, mentored by Aaron Meurer

Tom implemented an algorithm for computing symbolic definite integrals that uses so-called Meijer G-functions. This is the state-of-the-art algorithm for computing definite integrals, and indeed the results of his project are very impressive. This project has pushed SymPy forward a long way to becoming the strongest open source computer algebra system with respect to symbolic definite integration.

Vladimir Peric - Porting to Python 3, mentored by Ronan Lamy

Vladimir ported SymPy to work on Python 3 and ported all testing infrastructure so that SymPy gets regularly tested in Python 2.x, 3.2 and PyPy. Thanks to Vladimir’s work, the next version of SymPy, 0.7.2, which will hopefully be released later this year, will work in both Python 2 and Python 3, and it may support PyPy as well.

Gilbert Gede - PyDy, mentored by Luke Peterson

Gilbert implemented a physics module to assist in generating symbolic equations of motion for complex multibody systems using Kane's Method. He expanded on the code written by his mentor, Luke, in 2009, and the module can now generate equations of motion for a bicycle. Gilbert also wrote very thorough documentation both for the Kane’s Method and the module in SymPy.

Tomo Lazovich - Position and Momentum Bases for Quantum Mechanics, mentored by Brian Granger

Tomo has greatly improved the quantum mechanics module by implementing position/momentum representations for operators and eigenstates in various coordinate systems (including cartesian, cylindrical, and spherical) that allows you to easily represent many of the "textbook" quantum mechanics systems, including particle in a box, simple harmonic oscillator, hydrogen atom, etc.

Saptarshi Mandal - Combinatorics package for Sympy, mentored by Christian Muise

Saptarshi’s project was to mimic the various capabilities of Combinatorica, a Mathematica package for combinatorics. Most of the functionality involving elementary combinatorial objects such as Permutations, Partitions, Subsets, Gray codes and Prufer codes are complete.

Sherjil Ozair - Symbolic Linear Algebra, mentored by Vinzent Steinberg

Sherjil improved the speed of the linear algebra module by using efficient coefficient types for values of entries of matrices. Previously, SymPy used generic expressions in this place, which slowed down computations considerably and caused trouble with solving of the zero equivalence problem. He also implemented sparse matrix representation and unified the API with dense matrices. In addition, Sherjil also added a few linear algebra related algorithms (e.g. Cholesky decomposition).

Matthew Rocklin - SymPy Stats: Random Variables, mentored by Andy Terrel

Matthew improved the statistics module to use symbolics and introduced a Random Variable type, with support for finite, continuous, and multivariable normal random variables. With these you can symbolically compute things like probabilities of a given condition, conditional spaces, and expectation values. As a side consequence of this project, he also improved some of our Sets classes and implemented a MatrixExpr class, which allows you to compute with matrices symbolically, including computing with block matrices.

Sean Vig - Symbolic Clebsch-Gordon coefficients/Wigner symbols and Implementing Addition of Spin Angular Momenta, mentored by Ondřej Čertík

Sean was working on the quantum mechanics module and has implemented symbolic Clebsch-Gordan coefficients, Wigner D function, and related mathematical concepts that are used very often in quantum physics when dealing with angular momentum and then the necessary classes to support coupled spin algebra.

Jeremias Yehdegho - Implementing F5, mentored by Mateusz Paprocki

Jeremias worked on implementing algorithms related to Groebner bases. Groebner bases are a useful tool in many areas of computer algebra. He implemented the F5B algorithm, which is an improved version of the classical Buchberger’s algorithm that was previously implemented in SymPy, and an algorithm for converting Groebner bases between different orders of monomials and worked on applications of Groebner bases. This allowed for handling problems of much larger size in SymPy.

The full report can be found here, where each student wrote a wiki page about their experience during the summer and you can also find their blogs and links to applications. Each student was required to blog about their progress each week and all blogs were synchronized at planet.sympy.org.

In previous years, there was usually one student from each summer who became a regular contributor and also a mentor for the next year. It has been a rewarding experience for the whole SymPy community.

By Ondřej Čertík, Aaron Meurer and Mateusz Paprocki, SymPy Google Summer of Code Mentors

June and July are especially packed with conferences this year and Googlers are hitting the roads to organize, speak at and participate in conferences all over the world.

Earlier in the month over the June 17-19 weekend, Google was pleased to host a gathering of GCC developers in our London office. The informal working meeting was organized by Diego Novillo and Ian Taylor with assistance from Cat Allman to discuss current/future work and coordinate efforts.

Members of the Open Source Programs Office were in Portland, Oregon last week at Open Source Bridge. Open Source Bridge is a conference for developers working with open source technologies and for anyone interested in learning the open source way. During the conference, Google hosted a BoF session one evening for all students, mentors and anyone interested in learning more about Google Summer of Code.

Also last week, a gaggle of Googlers including Alex Martelli and Wesley Chun descended on Florence, Italy for EuroPython, the official European conference about the Python programming language. The conference completely sold out this year.

This week, Googlers will be heading to Porto Alegre, Brazil to speak at FISL (International Free Software Forum), the largest free software event in Latin America from June 29 - July 2.

Jeremy Allison, OSPO team member and co-founder of Samba, will be speaking on “The State of Samba” Wednesday, June 29th at 10am.

Carol Smith, OSPO team member and Program Manager for Google Summer of Code, will be talking about the Google Summer of Code program on Wednesday, June 29th at 4pm. She'll include statistics and interesting tidbits from the previous 6 years and some interesting changes we've made to the program for this year.

Andrew Gerrand, Google Developer Advocate for the Go Programming language, will be presenting a talk Thursday, June 30th at 9am on “Writing Simpler Code: Programming in Go.” His talk will demonstrate how you can use Go to write elegant, reusable, and correct programs. Later that day at 3pm, Andrew will host a tutorial titled “Get Started with Go”.

Next month many Googlers and open source enthusiasts will be converging on Portland, Oregon for the largest open source conference, OSCON, held from July 25-29 at the Oregon Convention Center. We will have another post in the next couple of weeks with a comprehensive list of all of the Googlers speaking at OSCON. We hope to see you at an event soon!

By Stephanie Taylor, Open Source Programs

As many of you may know, Python is one of the official languages here at Google. Guido van Rossum, the creator of Python, is a Googler too—so naturally we’re thrilled to be supporting PyCon 2011 USA, the largest annual gathering for the community using and developing the open-source Python programming language. The PyCon conference days will be March 11th to the 13th, preceded by two tutorial days, March 9th and 10th. For those of you with coding in mind, the Sprints run afterwards from March 14th-17th. All-in-all that’s nine days of Python nirvana!!

In addition to having many Googlers in attendance, some of us will be presenting as well.

• On Wednesday, March 9th at 2 PM, I will be leading a Google App Engine tutorial with fellow teammate Ikai Lan. Tutorials have gotten so popular at PyCon, they’ve now been expanded into a two-day affair!

• On Friday the 11th, the very first day of sessions, App Engine engineer Brett Slatkin will kick things off with his talk, “Creating Complex Data Pipelines in the Cloud” using the new App Engine Pipeline API at 10:25 AM.

• After lunch on Friday, I’ll take my Google hat off momentarily to discuss Python 3 in my talk subtitled “The Next Generation is Here Already” at 1:35 PM. It is mostly a repeat of the well-received talk I gave last year but with updates. The main point is to introduce folks to the next version of the language and discuss how its backwards-incompatibility will affect users, when users should port their apps to Python 3, what the differences from Python 2 are, etc. My job is to calm and soothe, dispelling any FUD (fear, uncertainty, doubt) about Python 3.

• On Saturday morning at 9:25 AM, Python creator, BDFL, and App Engine engineer Guido van Rossum will do his annual Q&A session for all conference attendees in a fireside chat session.

• Later Saturday morning at 11:05 AM, I’m looking forward to speaking about “Running Django Apps on Google App Engine.” This is exciting for me, not only because it’s a relatively new topic, but it represents a major change for Django developers: being able to write Django apps that run on NoSQL or non-relational databases -- it’s been only RDBMSs all this time. Furthermore, with Django-nonrel, you can move Django projects/apps between traditional hosting and App Engine, helping to break that “vendor lock-in” issue that many have had concerns about when hosting apps in the cloud. A good part of my talk does focus on porting apps from App Engine to Django however.

• Right after my talk, at 11:45 AM comes another famous Googler, author of Python in a Nutshell, co-editor of the Python Cookbook, and a long-time member of the Python community, Alex Martelli. Alex’s invited talk on “API Design anti-patterns” will be insightful and cerebral, sure to cause many future hallway discussions.

• Late Saturday afternoon at 4:15 PM, Google engineer Augie Fackler will deliver his talk entitled, “HTTP in Python: which library for what task?” There are many libraries that do HTTP. Which ones should you use and when? What are the benefits and tradeoffs?

• Finally, several members of the Google App Engine team, App Engine forum gurus, and experienced App Engine users are attending PyCon this year. I’m hoping to establish an OpenSpace session one of the conference evenings where we can meet other users, chat about best practices, and do some informal Q&A letting people ask anything they want (except “When will you support newer versions of Python?”). :-)

You can find the entire PyCon schedule online. It’s interactive if you log-in, allowing you to bookmark sessions you’re interested in attending. This will be PyCon’s biggest year yet, so hopefully you can join us in Atlanta next week! Keep an eye out on the PyCon blog to get the latest news, and be sure to follow the Twitter hashtag (#pycon).

We invite you to join Google team members at all our talks, plus stop by our booth to meet our technical staff as they demo select developer tools and APIs. We’ll have handouts there and also encourage you to try a short coding puzzle for a prize!

By Wesley Chun (@wescpy), Google Developer Relations team

PyPy is a reimplementation of Python in Python, using advanced techniques to try to attain better performance than CPython. Many years of hard work have finally paid off. Our speed results often beat CPython, ranging from being slightly slower, to speedups of up to 2x on real application code, to speedups of up to 10x on small benchmarks. This post describes what we did on PyPy during the last year, leading up to those results.

In the spring of 2009 we completed an update of PyPy to support Python version 2.5 with much appreciated financial support from Google. Most of the work on the update was done by Maciej Fijalkowski and Samuele Pedroni. While this work was in progress, Armin Rigo and Carl Friedrich Bolz were hard at work rebuilding the framework of the Just In Time compiler (JIT). The old framework, using techniques based on Partial Evaluation, only gave good results in constrained cases, but it would usually generate far too much code for Python. It was time to do more research from scratch. What we discovered was that the techniques typical of Tracing JIT compilers better suit the optimization of dynamic languages than techniques based on Partial Evaluation. However, we still follow our original meta-programming approach and remain convinced that writing a JIT compiler generator is more promising than the typical hand-coding of a JIT compiler for a particular interpreter. In other words, like in the original attempt, we get a JIT compiler that is not tied to Python but is generated from the source code of any interpreter for any dynamic language. If you are interested in the details, there is a very approachable paper about it, along with various blog posts.

During the autumn we applied and refined the JIT framework, added more optimisations and wrote a code generator for the x86 CPU family. In the early stages we could get good speed only at the price of huge memory consumption, but much work was spent addressing this problem. We have now reached a point where memory consumption is usually reasonable, but it is the nature of JIT compilers to trade some memory for speed. We began work using the benchmarks from The Great Computer Language Benchmarks Game to identify some problem areas. We have another blog post about our work in this area for the curious. Thanks go to Andrew Mahone who ported many of the Alioth benchmarks for our use. We also did some work on benchmarks that behave more like actual applications. The Django templating engine is twice as fast, and Twisted benchmarks are up to 2.85 as fast. For details, see our progress reports from the blog. While the results on a substantial number of benchmarks are really good, there is a lot more to do. We still have spots where performance is fairly bad, for instance our regular expression engine and the handling of generators. We have ideas about how to improve this and we have a list of further optimisations that could be performed.

The largest issue preventing users from adopting PyPy is the lack of extension modules. In addition to his constant efforts in making sure that PyPy runs on the Windows platform, Amaury Forgeot d'Arc has managed to port Oracle bindings to PyPy, while Alexander Schremmer has worked out a way to use the Remote Procedure Call library RPyC to use CPython extension modules with PyPy. Alexanders' goal was to get PyQt to run with PyPy and he was quite successful (apart from some bugs with PyQt itself), which you can read about on our blog. We also had Benjamin Peterson single-handledly rewrite our previously slow and problem-ridden Python parser, which now is much leaner and meaner. It is beta software, in the sense that it may speed up your applications significantly or not at all. We will need your help finding the odd quirks that prevent your Python programs from running and help to spot the places where performance can be improved. More information can be found on the PyPy blog, at our website or on the #pypy IRC channel at freenode.net.

By Bea Düring, PyPy Team

The leaves are turning here in Mountain View, but they are not the only ones blazing away. It's a busy time of year for open source for Google, with lots of talks and events going on.

Recently:
- Ben Collins-Sussman and Brian (Fitz) Fitzpatrick gave their "Myth of the Genius Programmer" talk as part of the Opening sessions at "Reflections / Projections", the 15th ACM@UIUC Student Computing Conference at the University of Illinois Urbana-Champaign.

- They were joined by Googler and Python maintainer Alex Martelli, who spoke on "Python and the Programmer".

- Chris DiBona, head of the Open Source Programs Office at Google gave a keynote at AstriCon in Glendale, Arizona.

Currently:
- Earlier this week Leslie Hawthorn, manager of the Google Summer of Code program, was part of the amazing team that completed a new "Manual on GSoC Mentoring" in 2, count them, 2 DAYS, finishing up late last night. You will hear more about this feat in a later post after the...

- Google Summer of Code Mentor Summit 2009, being held in Mountain View this weekend, October 24th and 25th. This invitation-only gathering of mentors from each of the participating mentoring organizations in this year's GSoC gives the projects a chance to come together to compare notes on the mentoring process and cross-pollinate their projects. A good time promises to be had by all, and a full report will be forthcoming.

Coming up:
- Jonathan Blocksom will be speaking on Google App Engine and the All For Good project at the DC edition of Stack Overflow Dev Days, October 26th.

- On October 4th the LISA Conference in Baltimore, Maryland will feature a talk by Daniel Berlin and Joe Gregorio on the Google Wave Federation Protocol, the underlying open network protocol for sharing waves between wave providers. Interested attendees of LISA will be able to sign up for a developers Wave Sandbox Account. They will also have a chance to win Googley prizes at the Google Birds of a Feather session the next evening, hosted by Cat Allman and Tom Limoncelli.

By Cat Allman, Open Source Programs

On Saturday, October 10, the Front Range Pythoneers had a Python "unconference" at the Google facilities in Boulder, Colorado, USA. An "Unconference" is a conference organized around the principles ofopen space technologies, which tries to provide many of the benefits of traditional conferences without the associated ceremony. We still got to enjoy some delicious pizza, though.

Introducing the Pycon Boulder Attendees to Principles of Open Space

Photo Credit: Matt Boersma

It was unseasonably snowy and cold Saturday morning, but in spite of the weather, almost everybody that signed up in advance was there, along with a few last-minute registrants. We had nearly 40 attendees join us for 15+ sessions, plus the always loved "hallway track." Many thanks to the three Googlers who came out to shepherd our group and facilitate the meeting.

You can find more information about the event and sessions on our wiki, Tweets about the event and this great post-conference write up. You can also check out some more photos of the participants and our scheduling process. We discussed the following topics, among others:

• Intro to Python
• ctypes tips & tricks
• Python extensions
• matplotlib
• Web development frameworks, including Django
• Database access
• Concurrency & pre-emptive multithreading
• Event-based/actor development (e.g., Twisted & Kamaelia)
• Cooperative multithreading (stackless/greenlets)
• pyprocessing
• Game development with Pyglet/OpenGL
• Jython
• RPC using Twisted/Foolscap
• Desktop Application Development (PyQT/QT, wxWidgets, GTK)
• IPython
• Google's office decoration contest, "Pimp My Cube"

The best surprise of the event? Bruce Eckel, the author of Thinking in Java, was among the participants. Thanks again to Google for hosting the unconference; it worked really well for our purposes. The Google Boulder facility is gorgeous.

By Greg Holling, Front Range Pythoneers