Compound Thinking

Suburbia

Prolonged adolecence is not a new problem, it’s just new to the masses:

Children of kings and great magnates were the first to grow up out of touch with the world. Suburbia means half the population can live like kings….

Paul Graham

You can’t shelter people from everything bad or scary, and expect them to live in the real world.

Project managers, System Administrators, and parents should take note of this.

People can only step up and take responsibility when they actually know what’s going on. Seems to me that there are lesions for how we talk to people about project risks, how we handle e-mail spam problems, and how we think about IT services. It’ll be a while before I figure out what exactly all of those lesions are….

What is data?

Ocean asks on his blog is data an asset?

Data is certainly not like many other assets, it doesn’t depreciate, you can copy it endlessly, and it’s next to impossible to imagine a commodities market for data. Heck copying the data can either increase it’s value (think “The DaVinchi Code”) or decrease it (think passwords). People don’t pay for data as much as they pay for human attention. You can use data to get attention or you can use attention to collate, assimilate, and otherwise transform raw data into useful information, but either way data needs people to understand and interpret it to become valuable.

So at best:

data + human_understanding == value

Bruce Schenier takes it one step further, calling data the pollution of the the information age.

Data Pollution

Data sucks up space, time and human attention. But more than that, data can be parsed, manipulated, and transformed to fit various agendas. And in a world where data about all of us is “owned” by various large corporations, from Amazon, to Google, to Enron, it’s not always clear how that data will be used. Besides which millions of credit card numbers are stolen from various companies who store our data “in good faith.” Data costs money in terms of maintenance, in terms of storage, and in terms of liability. Heck, I know people who work for companies who have an e-mail retention policy — which is really more of a mandatory e-mail deletion policy.

Polluted Data

And that assumes that all that data is verifiable true, and that’s definitely not the case. I sold a car once and the new owner didn’t take it to the DMV to get it registered before his friend drove it without a license and got it impounded. And that showed up on my credit report for years. I have a friend who somehow ended up “deceased” even though she’s still very much alive and well.

All of this is to say that as software developers, IT Mangers, and companies in general need to think a lot more about data, and to invest in some better terms for the various different things we call data.

We need to differentiate between raw data, information, and knowledge. We need to help our customers think about the life cycle of the data they want us to capture. We need to educate people about the costs and benefits associated with keeping data, and ultimately we need to follow the mantra:

Think before you store

And if you’re concerned about privacy, and individual liberty, please take a few min and read Bruce’s article.

Threads, Processes, Rails, TurboGears, and Scalability

Threads may not be be best way, or the only way, to scale out your code. Multi-process solutions seem more and more attractive to me.

Unfortunately multi-process and the JVM are currently two tastes that don’t taste great together. You can do it, but it’s not the kind of thing you want to do too much. So, the Jruby guys had a problem — Rail’s scalability story is only multi-process (rails core is NOT thread safe), and Java’s not so good that that….

Solution: Running “multiple isolated execution environments” in a single java process.

I think that’s a neat hack. The JRuby team is to be congratulated in making this work. It lets Rails mix multi-process concurrency with multi-threaded concurrency, if only on the JVM. But it’s likely to incur some memory bloat, so it’s probably not as good as it would be if Rails itself were to become threadsafe.

I’m not sure that the Jython folks have done anything like this. And I’m not sure they should. It’s a solution python folks don’t really have. Django used to have some thread-safety issues, but those have been worked out on some level. While the Django people aren’t promising anything about thread safety, it seems that there are enough people using it in a multi-threaded environment to notice if anything’s not working right.

At the same time, TurboGears has been threadsafe, from the beginning, as has Pylons, Zope, and many other python web dev tools. The point is, you have good web-framework options, without resorting to multiple python environments in one JVM.

Why you actually want multi-threaded execution…

In TurboGears we’ve found that the combination of both multi-threaded and multi-process concurrency works significantly better than either one would alone. This allows us to use threads to maximize the throughput of one process up to the point where python’s interpreter lock becomes the bottleneck, and use multi-processing to scale beyond that point, and to provide additional system redundancy.

A multi threaded system is particularly important for people who use Windows, which makes multi-process computing much more memory intensive than it needs to be. As my Grandma always said Windows “can’t fork worth a damn.” ;)

But, given how hard multi-threaded computing can be to get right TurboGears and related projects work hard to keep our threads isolated and not manipulate any shared resources across threads. So, really it’s kinda like shared-memory optimized micro-processes running inside larger OS level processes, and that makes multi-threaded applications a lot more reasonable to wrap your brain around. Once you start down the path of lock managment the non-deterministic character of the system can quickly overwhelm your brain.

As far as i can see, the same would be true for a Ruby web server in Ruby 1.9, where there is both OS level thread support and an interpreter lock.

I’m well aware of the fact that stackless, twisted, and Nginx have proved that there are other (asynchronous) methods that can easily outperform the multi-threaded+multi-process model throughput/concurrency per unit of server hardware. The async model requires thinking about the problem space pretty differently, so it’s not a drop in replacement, but for some problems async is definitely the way to go.

Anyway, hats off to the Jruby team, and here’s hoping that Rails itself becomes threadsafe at some point in the future.

The motivational meeting…

Last week, I ranted a little bit about motivational meetings. Today I’ll make the opposite case.

Why have motivational meetings?

The right way to use motivational meetings is to reaffirm the purposes of the group, and help people to connect the dots between their individual efforts and the collective goals of the group, and to connect those goals with their own individual aspirations.

Basically, motivating people is easy:

• Give them work that is meaningful to them and to the organization
• Treat them with respect

Treating people with respect includes paying them a fair wage, and not doing any of these things.

Among other things it also means not letting people who aren’t contributing to the common goals of the organization hold back the group by not doing their job.

Research has shown that one of the survey questions most highly correlated with motivation and performance is:

Does my supervisor, or someone at work, seem to care about me as a person?

Which is another way of saying does your boss respect you. At the same time the single highest correlation for any question was:

I get to do what I do best everyday at work.

So, it’s really important to line people’s intrisic skills and internal long-term motivational drivers with the work you ask them to do.

If you’re not doing those two things, motivational meetings are a loss. If you are doing them you can use a meeting to remind people of how their deeper motivations are connected to what they are doing now.

P.S. My info on the top questions and their correlation to performance comes from Gallop research via the very interesting book First Break All the Rules, which is one of the best, and most evidence based, books on managing for exceptional performance I’ve read.

Motivate me when I’m dead…

It think the very idea that motivation can be “imparted” in a morning meeting, or a half day seminar is kind of demeaning.

Motivation is a complex network of hopes, dreams, fears, needs, frustrations, incentives, and personal morality. Motivating people is as much about connecting their individual aspirations to the goals of the organization as anything. If people can do what’s right, become who they want to be, and get paid to do it, that’s a far more powerful motivator than you can get from any meeting.

If those things are true, you can skip the motivational meeting. Everybody would rather sit down and get some work done. And if they aren’t true it’s not likely that another meeting will help.