PythonPackaging == hot_action

Ever since my friend Adrian posted up a slide at PyCon telling us that in Django Trunk == hot_action, I’ve been getting lots of e-mails from various folks telling me about the hot_action that I could get from various python packages. But today, the hot action isn’t the contents of the packages, so much as the packaging systems.

As a starter there’s a very, very good background post byKevin Teague that provides great overview that provides the then current Python Packaging ecosystem.

But even though that post is very recent it’s not at all complete, because last week spawned two interesting new projects pyinstall from Ian Bicking, and distribute a potential setuptools fork.

The distribute project didn’t last long, as it’s already been declared officially dead.

But I still think it’s interesting, and it’s definitely spawned a lot of good discussion on the distutils-sig.

When I talked to Tarek’s last week his motivation behind starting distribute was always to help move setuptools forward, and his hope was that anything that happened there would be folded back into setuptools proper as time went on.

There were lots of people at PyCon this year who were both vocal about what was wrong with easy_install and motivated to help improve it. Beyond that some of them were people who do packaging all the time, and some of them had some previous experience writing packaging systems in python.

On of the most interesting tidbits from that discussion is this suggestion from Tarek and Guido’s response:

> My own inclination is that a scalable future for distutils means an improved
> sdist format, the end of setup.py as an command-line interface, and
> community-maintained platform-specific installation tools that process
> source or binary distributions. Most complaints about distutils (and
> setuptools, for that matter) are focused on installation policy&preference
> issues. Making it possible and practical for a variety of tools to flourish
> around a standardized format (ala WSGI) seems like the way to go.

Given the success of WSGI (which I use every day) this sounds like a
very good plan!

And like Guido I’m pretty enthusiastic about extending and standardizing the format while allowing a variety of tools to grow up around that new standard.

But I’m a bit skeptical about focusing only on sdist, since I think we do need some kind of way to distribute binary packages to Windows and OS X users, and while eggs work pretty well there, I don’t what that stuff to be left behind.

Which brings me back to pyinstall for pure python packages I really like pyinstall, because it has all of the key features that I think that a python package system should have (repeatable installs, single file installs, etc), but it”s focus on only source-level distribution makes penetration in the windows market harder. And I guess the same could be said of OS X where getting a compiler can be require a 1gig download.

Where do we go from here?

It’s fair to say that easy_install is not perfect, but it is much better than nothing, and last week’s news had made me excited about the future of python packaging again. My thanks to Philip, Ian, Tarek, and all the other people who have put in effort to make this part of the python better, and hopes are with you.

If you are interested in getting involved, and helping to make all this stuff better, there’s also going to be a sprint this weekend.

4 Responses to “PythonPackaging == hot_action”


  1. This is great news, and thanks Mark for wrapping it up with a nice bow for all of us. I am especially excited to hear that setup.py will go away as an interface for setuptools. I also agree that better support for binaries needs to be addressed.

    One of the things that I really want is an easy_uninstall that actually works! I’m guessing this has to be on the hit-list.

  2. Chris,

    You mean Zed’s easy_@(#$* uninstall isn’t enough for you. You want something that actually cleans up after itself!

    Now you’re just dreaming ;)

    But in all seriousness pyinstall keeps track of all the files that get installed so it would be a lot easier to extend it to do uninstalls.

  3. I’d rather see binary installation handled by some other tool that takes a source distribution and rewrites it as a binary installation. pyinstall works pretty well, but if it took on binary installation (that actually works, because eggs alone don’t work all that well) then it’d be much more challenging. (Though to be fair, eggs just on Windows work okay.)

  4. 4Tushar

    I’m a long time user of python and recently have managed to lead my company towards using python for major development work. We use PyInstaller (note: not ‘pyinstall’) to create .exe files on Windows and distribute clean, fully encapsulated binaries for Linux and Windows platforms. PyInstaller is *great* software and makes production-class development in python _possible_.

    Personally, I would *really* like to see PyInstaller integrated into the standard library along with a whole (modern) set of packaging and distribution tools.

    I’ve used distutils and setuptools in the past, and I’m continually frustrated that they don’t do what I want in a sufficiently obvious way (tearing my hair out is getting tiresome!). Python programming is normally the opposite: a joy and a pleasure: let’s make packaging and distribution the same.

    It really is time now for a new start. AMK hit the nail on the head when he said “what would you want if you were thinking afresh?”.

    So, here’s “What I Want”:
    * BINARY installations (this is the real world folks – I want to sell & distibute products in binary form!)
    * pyc-only distributions (cf java class files)
    * source distributions
    * standard python packages (which can work on multiple platforms)
    * integration with platform installers where possible (think of Windows MSI, debian apt, RPM, autopackage, etc)
    * simple command line tools to do all of the above (so that I can integrate into a continuous build environment)

    (P.S. If anyone has any good ideas on the pyc front, please let me know as that’s what I’m looking into at the moment that brought me here in the first place.)

Comments are currently closed.