Compound Thinking

For some (very small) values of done…

What does done mean? Somebody was telling me that Ruby 1.9 was “pretty much done” last night.

What is Ruby 1.9? Do we have a spec? A test suite? Anything? If the answer is no, it’s not ready.

–Charels Nutter (in a comment here)

Partly that’s because Charels wants 1.9 as a reference implementation — not just usable interpreter.

But, that highlights one of the reasons I like the way python is developed. There is a test suite, there is a reasnably complete set of specifications. So, from here, it looks like Python 3 is a lot “more done” than Ruby 1.9. But that’s OK, they are both way more done that Perl 6. ;)

I’m still looking forward to Ruby 1.9, and python 3.0.

So many revolutions, so little time.

Tim Bray is blogging about “inflection points” in the uptake of various technologies.

Python get’s a very positive review:

Today you’d be nuts not to look seriously at PHP, Python, and Ruby.

So, the rise of the so-called scripting languages is one of the inflection points, but it’s not the only one.

He singles out web-framework development as one place where there’s a lot of stuff happening, and a lot of new “rails-like” frameworks are cropping up all the time. TurboGears will live or die in the context of a much larger web-development revolution, and we need to be prepared to make our way forward in the midst of that.

What comes after rails will not be a rails clone. It will learn the right lessons from rails, avoid the pitfalls of rails, but it will also need to carve out something new and better than rails. For RDBMS users, I think the key difference between TG and Rails is the power and flexibility of SQLAlchemy. We need to “sell” this better.

There are a lot of other revolutions coming according to Tim. And I do think we’re looking at big changes in terms of everything from programming language choice, to web-development tools, to end-user desktops, and data persistence mechanisms. We’re also just beginning to see what the world of high-end javascript and other “rich” internet applications is going to do to our view of end-user software.

He doesn’t even mention the rise of EC2 and the Google App Engine as sea-changes in the way we buy computational resources, and I think that’s going to have a huge impact.

In the end my prediction is that the way we develop applications will change more in the next 5 years than it did in the last 5, and it’s time to start getting our heads wrapped around these issues, or we’ll be left behind.

JVM as platform for dynamic languages?

Martin Fowler recently blogged about how to choose between the emerging JRuby and Groovy scripting languages on the JVM. He pretty much ignores both Rhino, and Jython, which is strange since Jython is the grand-daddy of dynamic languages on the JVM, and Rhino seems to have the most internal support from Sun, and has been getting some serious press in the blogosphere. And I think his failure to address Jython, and Rhino reduces the value of the article.

But he does, finally, at the end of his article, he get around to a quick mention of Jython in the context of asking a bigger question:

Will either matter to Java? After all Jython’s been around for a long time without making a huge impact on the JVM. Tool support is frankly pathetic for any of these languages when you compare it to what you have for Java at the moment.

This question is skating right around the critical issue, without ever touching it directly.

Why is there so little uptake for dynamic languages on the JVM? And what will it take to change that?

Another way to ask the same question would be: We have a lot of deep experience with C integration in the Perl/Python/Ruby communities, but there is no such deep experience with the JVS as a platform for scripting languages. Why?

To sharpen the question a bit further, we should note that the Jython/java integration story is in many waysbetter than the Python/C story. In Python 2.5, c-types makes using C libraries from python easier than ever — but you don’t have to do anything special at all to use Java libraries from jython. In spite of this, Jython and all the other new “dynamic languages” on the JVM have not become a popular way to work with Java libraries.

Why?

I’m going to address this from the perspective of Jython user, because that’s the JVM/dynamic language experience that I have. But, I’ve played with Jruby enough to know that the issues are similar.

One problem that seems to stop python programmers from flowing over into Jython easily is the impedance mismatch between the way things are done in Python and the way they would be done in Java. As a python developer you find yourself constantly switching mental gears to write anything complex in Jython. You’re thinking about the python code you’d write, and things sort of flow until you run up to something in a java library which requires you to look at the world in a fundamentally different way.

Of course, the same is could be said to be true of Python and C. Sure, you could argue that C programmers and python have more similar pragmatic approach to library development, but I don’t think that’s the main difference. The main difference is that developers tend to “wrap” C libraries and provide “pythonic” bindings which reduce the impedance mismatch. Nobody seems to do this in jython, partly because it’s just too easy to ignore the need for good pythonic API’s and just get something done. It seems that easy is sometimes the enemy of good — which is definitely not news to any experienced programmer.

But, perhaps there is another possible reason, is that the JVM was written explicitly for java, and is not particularly friendly to dynamic languages. Reflection is slow, and in general features used by dynamic languages haven’t been a priority for the JVM. Fortunately there’s some work happening already to make the JVM a much, much friendlier place for dynamic languages. But I don’t expect that Jython is going to be faster than C Python any time soon. So that, means that the JVM as a platform needs to offer something more than just a good way to implement python. I’ve heard some rumblings from the Jython team that this may not be a significant factor in the future, but it’s certainly colored people’s perceptions in the past.

This brings us back to the fundamental nature of the JVM as a viable platform. It’s one thing to have languages that run on the JVM, and call that the platform, but the language and the managed runtime environment are only part of the picture — in my experience the platform is defined as much by it’s libraries as anything else.

I think this is the main problem that the JVM will face as it evolves into a multi-language platform over the next few years — the java standard library is complex in unexpected ways (how do you open a file and process it line by line — I always have to look it up?) and that makes it difficult to use as the foundation for a vibrant scripting platform.

If however people aren’t working with java libraries, you’re just running Jython on the JVM, and all you get is a slower slightly more out of date version of Python, with fewer libraries, and that’s not a very compelling case. Perhaps you need to run on the JVM for political reasons, or you need access to one of the java libraries that don’t have an obvious python equivalent, like i-text or lucene. But except for those relatively narrow use cases, why wouldn’t you stick with CPython?

The Microsoft threat

Microsoft intentionally has worked to create libraries that make sense for C# users, but which are accessible from languages like VB.net, and IronPython. Thus they have created an alternative Python platform which is much more interesting than Jython. And to be successful in turning the JVM into a multi-language platform I think something similar needs to happen to Java.

At least right now, the situation the way that Groovy, Jython, Scala and JRuby have been implemented makes it unclear how well you’ll be able to integrate libraries written in one of these languages with libraries from another. Effectively this places Java libraries at the center, and relegates dynamic languages to “second class” status. At this moment, writing libraries in Jython that would be in an attempt to make them usable to Jruby and Groovy folks seems like a fools errand.

So, having a common core of reusable libraries written in Java seems like a key ingredient in making the JVM a better platform. These libraries would have to be built:

* with multi-language reuse in mind
* in ways that don’t require lots of boilerplate code
* to work well with dynamically typed languages

That kind of standard library would benefit allof these new players on the JVM. And that’s what I think Sun ought to invest in if they want the JVM to become a platform that JRuby, Jython, Scala, Groovy, and whatever else comes next, can thrive.

Another Microsoft Threat

At the same time, Microsoft has been widening the gap in another way, by introducing the DLR — a new set of tools for dynamic language implementors.

Much of the thinking behind the DLR comes from Jim Hugunin (the original implementor of Jython), and if it works the DLR will help to make dynamic languages first class citizens on the Microsoft CLR since you will be able to use them to write libraries that are cross-compatible, making IronPython libraries usable in IronRuby, etc.

The aforementioned project to make the JVM more friendly to dynamic languages is a good place for language implementors to work together. And it would be very interesting to see the jython, jruby, and other language implementors get together and provide a good solid answer to the DLR. I’ve been talking to some of the Jython guys this week, and they tell me that this is happening, and that a lot of the work is already done.

Given the politics of language communities, and the necessities of the Open Source development model, I’m sure this has been more difficult than it could have been, so I’m very excited to hear that the people involved have taken the time to make that happen. I haven’t seen a lot of talk about this on the internets, but it looks like people are quietly doing the right thing without running the hype machine in overdrive which is refreshing.

After talking to the Jython guys this week, I’m actually pretty optimistic about the JVM. It’s growing in a direction that provides a solid memory-managed platform for the dynamic languages of the present and future, and it could become a platform for collaboration between language developers.

The CLR could do the same thing, but I want to see it happen on the JVM, because that would make it free in all the important senses of the word.

TurboGears 2 sprint

I’ll be hosting a TG2 sprint here in Ann Arbor Michigan on October 27th, hopefully we’ll also have a large virtual presence from folks around the world.

I’ve created a page on the Wiki for Sprint Organization tasks: http://docs.turbogears.org/SprintOrganization

If the sprint goes well it could get us very, very close to the point where we could reasonably do a TurboGears 2 technology preview release.

The main things we’ll need to do is sync up with the latest pylons, improve our tests, and do some basic doc organization work (migrate some information from the doc strings to the docs wiki, and create some pages which link to external docs). There’s lots of other things I want to do like improve out user authentication/authorization/registration support, or create a toolbox tool for helping create SQLAlchemy models. So, there’s tasks for anybody who can lend a hand.

If you’re available and want to help out (either attending physically or virtually), feel free to add your name to the wiki.

Likewise, if you can host a in-person sprint in another location, please feel free to edit the wiki with that information.

Relational Databases as an “implementaiton detail?”

I found this little tidbit in amidst the hugely overraught comment treads on Derik’s post describing his return to PHP from the wonderful world of Ruby on Rails:

I loved the part about the database as an “implementation detail.” That sings to me.

–Daniel Waite

And, in spite of the poetic hyperbole of folks like Daniel, I think relational databases exist for good reasons and knowing about how they work is a good thing. Active record treats the database kinda like a big hash table in the sky, which is OK for some applications, but not so OK for others.

I find that large, complex apps have a way of growing in unexpected ways over time — and relational algebra is much better designed to handle unexpected requests for complex datasets than hash tables or even the most well designed objects. SQL, and relational algebra are implementation details in the same way that using a hammer is an implementation detail in building a tree house. Sure you can do without, but you will have a harder time, and will definitely end up building a different treehouse!

Don’t misunderstand me, Object’s are good things, and I sure as hell hope people understand them, use them, and make great software with them. But relational algebra is a good thing too. And it’s become clear that any really good database toolkit can’t forsake one in favor of the other, and that’s why SQLAlchemy, Storm, DejaVu, and Hibernate are all better tools than Rails’ Active Record.

As for Derik’s original article. In spite of all the brouhaha, Derek’s post basically says: “hey I liked Rails, and I learned a lot about good programming from it. And ultimately it was easier to get my site rewrite done in PHP because I learned rails first.”

I think it’s obvious that Rails could have done the job for Derek, but two things got in the way:

• everything was already in PHP so Rails wasn’t and incremental update, it was a rewrite.
• Active Record got in the way of using SQL as a relational algebra engine, rather than a glorified hash persistence mechanism.

TurboGears users should pay attention to both points. If you have a large complex app, you’ll be better off with an ORM that treats SQL as a full partner, which is why we’re migrating the defaults to SQLALchemy which does a fantastic job of making both relatinal algebra, and object oriented programming part of it’s core.

The second point is much harder to deal with, but if you can wrap existing application functionality in a larger TurboGears app, and migrate it piece by piece you’ll definitely have more success. I’ve been doing a lot of Web Service architecture stuff this year, and that’s proven to be a great way to connect existing resources to TurboGears applications.