Py3k

5 December 2008
18:35

Python 3.0 has been been released with some fanfare, here's my take, as someone who writes a lot of Python code, but who has admittedly never actively participated in the evolution of the language or the standard library. For some background, I've been one of the early developers of Trac, and I've also written and continue to maintain a coupe of open-source Python libraries such as Genshi, Babel, and CouchDB-Python.

So, Python 3.0. Technically, it's a great thing. It does away with old warts in the language design, improves consistency and simplicity. The standard library with all of its inconsistently (and sometimes awkwardly) named modules has been reorganized. Strings are unicode by default. Integers have unlimited precision. And quite a bit more.

But…

Backwards Incompatible

Such deep changes come at the cost of backwards compatibility. Python 3.0 (formerly Python 3000) was always intended to be the milestone that broke backwards compatibility, shedding the old skin to enable a brighter future. So this certainly doesn't come unexpected.

But what this really means is that there's hardly anyone who will want to actually upgrade to Python 3.0 for the time being. Pretty much none of all those convenient third-party applications and libraries out there are going to work under Python 3.0. Bummer.

But surely there's a migration plan of some sort? There is, here. To quote:

  1. (Prerequisite:) Start with excellent test coverage.
  2. Port to Python 2.6. This should be no more work than the average port from Python 2.x to Python 2.(x+1). Make sure all your tests pass.
  3. (Still using 2.6:) Turn on the -3 command line switch. This enables warnings about features that will be removed (or change) in 3.0. Run your test suite again, and fix code that you get warnings about until there are no warnings left, and all your tests still pass.
  4. Run the 2to3 source-to-source translator over your source code tree. (See 2to3 - Automated Python 2 to 3 code translation for more on this tool.) Run the result of the translation under Python 3.0. Manually fix up any remaining issues, fixing problems until all tests pass again.

Setting aside whether the prerequisite of “excellent test coverage” is realistic or not (it's not), the key here is “Port to Python 2.6”. Python 2.6 was released on October 1st, 2008, a bit over 2 months ago. No major Linux distribution ships with Python 2.6 at this point, as far as I know.

Most of the libraries I write and/or use have the baseline at Python 2.3 or 2.4, as you can't expect all your potential users to be running the very latest version (I still run 2.4 on this very server, for example.) Many libraries are just now dropping support for 2.3, which is now over five years old.

So for most libraries out there, porting to 2.6 is going to mean a separate development branch, because we can't just stop supporting versions 2.4 and 2.5, the most widespread Python releases at this point. But even with a separate version ported to 2.6, we still don't necessarily have a version that'll run under Python 3.0, we still need to run the 2to3 translator, and manually fix any issues it couldn't find or translate correctly.

I think this is the major flaw in the migration plan: assuming that developers can stop supporting Python 2.4 and 2.5, and just move on and port to 2.6. It will take a couple of years until we can realistically specify 2.6 as the minimum required version for Python applications and libraries. If we still want to support Python 3.0 in the meantime, we'll have to deal with three development branches targeting three different versions of Python.

Very few application and library developers are going to go through that kind of trouble. And that's going to cause a lot of friction in the Python community.

Forward

Still, I don't think that Python 3.0 is a bad thing. But that it's displayed so prominently on the Python web site, without any kind of warning that it's not going to work with 99% of the Python code out there, scares the hell out of me. People are going to download and install 3.0 by default, and nothing's going to work. They're going to complain, and many are going to simply walk away.

In my humble opinion, Python 3.0 should be put into a kind of stealth “technology preview” mode for the next couple of years, until 2.6 (or 2.7 or 2.8) is everywhere. That includes toning down (or hiding, if you will) the 3.x line on the web site, not assuming general availability of 3.x in new Python books, and so on.

At the point when 2.6 is established enough to be the new baseline for applications and libraries, there's a real chance that most of them will also provide a version compatible with Python 3.0, and then users will actually be able to use the next iteration of the language.

There's a third-party ecosystem that'll need to come along with Py3k, and it's going to take some time.

Reactions

  1. Paul Bonser says:

    6 December 2008
    09:01

    I'll definitely agree that adoption of Python 3.0 will most likely be quite slow.

    It'll be another year or two before you even get Python 2.5 by default on every system (web hosts are notoriously bad about having up-to-date versions of languages like Python and PHP).

    This must just be me being crazy, but what would be really helpful would be a compiler written to be compatible with Python 2.4 which takes Python 2.6 code and compiles it into 2.4 compatible bytecode.

    It would add support for any new syntax and provide Python-only versions of any builtins whose behavior has changed with 2.6 (including the optionally-imported ones)

    The important thing would be to have it be written in Python and compile code down into .pyc files like Python normally would. You would just wrap your code with a small stub that invokes this compiler rather than loading it directly.

    Basically, if you could take the compiler module from 2.6 and get it to run on 2.4 (you'd have to write a parser to generate the CST, since it uses the built-in one by default) you'd get this pretty easily.

    By the time all the systems actually have 2.6 on them, you could have all of your code running with no warnings in -3 mode, but still be running on 2.4 systems.

  2. Alan says:

    6 December 2008
    10:57

    While I think the whole migration thing, as you point out, won't be straightforward, it's something we'll have to live with. It's a major revision: if you just want most of py3k features while retaining backwards compatibility, you'll be perfectly happy with python 2.6. But py3k final was a necessary step - nobody would have began tinkering with it as long as it was beta or rc.

    But hey, running existing code on python 2.6 (it's not yet in distros, but new releases take a while before being packaged, that's normal) with the -3 switch will let you write backwards compatibile code by just doing small changes - it'll run on 2.3, 2.4, 2.5 and 2.6 with no issue.

    The whole idea is to retain just two branches: one for the 2.x, and the other for the 3.x branch of development.

  3. Richard Jones says:

    7 December 2008
    00:49

    It should be mentioned that "port to python 2.6" in most cases means "port to python 2.0, 2.1, 2.2 or 2.3" which is when most of the compatibility features were introduced. Things like new-style classes, using "in" instead of "has_key" etc.

  4. Matt Good says:

    7 December 2008
    04:44

    Porting to 2.6 shouldn't require dropping support for older Python versions. It didn't take long for Trac to support 2.5 while maintaining 2.3 compatibility and I don't think 2.6 will be much different. I guess the question is whether the warnings from the -3 switch are fixable without breaking 2.3 compatibility. I'll have to try this soon to see what's required.

    Alan, actually you still only maintain 1 branch. You continue writing code for 2.x, fixing any -3 warnings, and the running the 2to3 converter. Like all generated code, the converted 3.0 code should remain unmodified. Or that's the theory anyway. We'll see how it works in practice, but I'm hopeful that we'll see libraries offering 2.x/3.x compatible releases soon.

  5. Christopher Lenz says:

    10 December 2008
    23:56

    It's quite possible that I've overestimated how much work it'll be to port code over while retaining compatibility with older Python releases.

    The “Port to Python 2.6” part of the porting instructions threw me off here, I thought that meant you'd want to make use of the new __future__ imports and such, thus dropping support for older versions. Apparently that isn't what it meant, so considering that 2.6 is a backwards-compatible release, I'm not sure what “porting to Python 2.6” is supposed to mean, anyway :P

    The overall point of this post still stands though, the migration to 3.0 will take a long time, and I personally think it'd be better for the community if the Python 3.x stream was toned down on python.org, thereby making 2.x the default download for the time being. But I can also understand the other point of view on this: that going into stealth mode would not put sufficient pressure on library authors to port their code.

    As the author of libraries that depend on (almost) nothing but the standard library, I can definitely feel that pressure, even though I've not seen anyone actually request 3.0 support on the mailing lists so far. So maybe it's a good thing. Only time will tell whether this works out for the Python community, I guess.