I was developing on a vps with only 256MB of RAM, so I wanted to avoid memory intensive operations. I’m using ruby, so I started looking into how to ‘stream’ the xml with ruby in such a way that I could read if it was the type of record I wanted to keep, somehow manipulate and store the data and move to the next record. The more I looked into that strategy, the more complex and ominous it seemed. I just really wanted to find a simpler way that wouldn’t require all the apparent tediousness of streaming the xml file (it may seem simple enough, but there are dependencies you have to install, api’s you have to read and learn, etc). There were just too many moving pieces and it made me feel eery about the outcome. Another aspect of it was just pure laziness. I just don’t care how I get that data into the db – I just want to get it in there.

Fortunately, the glorious wonderfulness of linux utilities saved me from a long tedious solution. Behold:

  csplit -q catalog.xml '/<title_index_item>/' '{*}'

This splits the large xml file into sub-xml files that start with ‘xx’ by default, followed by an incremental number, xx10004 for example. It splits the file based on the <title_index_item> tag – which is the tag for the items I want. See `man csplit` for more info…

  find xx* | xargs grep -L 'label="instant"' | xargs rm -f

The ‘find’ lists all the sub-xml files, then we grep for the filenames that do not contain what I’m looking for, then delete those files. I’m left with a directory of 15,000+ xml files with just the type of data I’m interested in.

  find . | wc -l

This command is just so I can track the progress of the operations. Obviously `ls` would return too much data, so we pipe the file list to `wc -l` which gives us the number of lines – which in this case is the number of files in the dir.

So, two lines of code on the command-line instead of a far-more complex ruby/streaming-xml solution. Now I can have a ruby script process each individual file and add the data to the db – a much simpler problem to deal with.

My point is that you can accumulate code debt by doing the seemingly ‘right’ solution sometimes. There are probably coders that will cringe that I just used a couple of shell commands to do this rather than write up a long, well-documented, properly OOP, TDD, etc “right” solution. However, the “right” solution in that case would have accumulated code debt – more code to maintain, more moving pieces, more things that can go wrong. I’d trade maintaining two lines of shell code over 10′s or 100′s of lines of ruby code and their dependencies any day of the week.

Semco purchased several million dollars of computer equipment in the 80′s to ‘improve’ their systems. However, this is what happened:

=====

Everything has become excessively complicated and confused, which is exactly the condition computers were supposed to remedy. I can shamelessly say that Semco was swept along on the electronic tide, going from the Stone Age to the Space Age in just a few years. But the more I learned about microchip management, the more I came to fear it.

I once visited a small unit of our three days before the end of the month.

“What do you think this month’s billings will be?” I innocently asked the resident computer wonk.

“We already know. We have the number here in the terminal.”

“How can you? There are still three days left in the month.”

“Oh, no. We stop billing four working days before the end of the month.”

I was shocked. It meant inventory was tied up longer than it had to be, increasing interest costs, and orders would sit for days before leaving the plant, increasing customer frustration. Just picture Federal Express stopping for a few days each month to feed paperwork to its computers.

“Well, we have to fill out the basic invoice, send it to the head office, wait for batch processing on the central computer…”

“Wait,” I interrupted, “why can’t you just issue the invoices on your own terminal?”

“Because then they wouldn’t be automatically recorded in the accounts receivable program, which cross-references all the invoices.”

“I see,” I said, although I didn’t. “Go on.”

“The lot is batch processed after all invoices from all our units are gathered – we run them in numerical order, for tax purposes – and then our invoices are returned to us. And then we can ship the products.”

“How did you do all this in the past?”

“Oh, that was very primitive. We would wait until the last minute, then type out the invoice. Sometimes we would be here in the middle of the night, getting invoicing out of the way, to make the month’s sales larger.”

“How many invoices did you issue then?”

“About 150.”

“And now?”

“About 120.”

Two days later the unit was off the computer and back on the primitive manual system. And soon invoicing rose 15 percent, as employees got back to making last-minute shipments in all-out efforts to move finished products out the door. And within a month all the other computer terminals at our business units had been returned to headquarters, and our mainframe there was disconnected.

This much seemed clear: either you can adopt sophisticated, complex systems to try to manage the complications, or you can simplify everything. [emphasis mine]

Finally, we chose the other path.

=====

I’ve seen this happen to several companies. There will be a huge push to get the whole company using some big new complex ‘Enterprise’ software package that promises to be a huge improvement over the existing system. These projects often end up taking *way* more time, money, and man-power to implement than expected. I’ve seen it be so expensive that the company went out of business or had to lay off a large percentage of it’s employees because of the cost.

Alternatively, if a company focuses on simplicity by developing incrementally with continual small improvements to their system they can much more effectively manage the risk and cost of the project and make course corrections as necessary.

=====
Without rules all answers are suggested by common sense. No, I can’t define what common sense is, but I know it when I hear it. Some of our people stay in four-star hotels and others, sometimes with much higher salaries, choose lesser digs. Some people spend $200 a day on meals; others get by on half as much. The point is if we can’t trust a manager to use good judgment about such things, we sure as hell shouldn’t be sending him off to do business in our name.

A company makes, sells, bills, and, God willing, collects. It doesn’t need to know if the taxi ride being claimed by a manager was for business. Or if another manager couldn’t have stayed in a hotel with three rather than four stars. With few exceptions, rules and regulations only serve to:

1. Divert attention from a company’s objectives.
2. Provide a false sense of security for executives.
3. Create work for bean counters.
4. Teach men to stone dinosaurs and start fires with sticks.

The desire for rules and the need for innovation are, I believe, incompatible. (Remember, Order or Progress.) Rules freeze companies inside a glacier; innovation lets them ride sleighs over it.
=====

While this may seem counter-intuitive to some, I’ve often seen this principle in action. For example, when I was working for a startup that was acquired by a multi-billion dollar software company everything seemed to slow down by a factor of about 10X after the acquisition. All of the sudden there were all these rules that needed to be followed – many of them rather pointless. At the startup, we always had free drinks available. The acquiring company decided that we now needed to pay for the drinks – a quarter each. So programmers that were making about $50 an hour were now asked to go find quarters throughout the day for drinks. That means, a programmer making almost $1 per minute was spending several minutes every day worrying about quarters for drinks. I would estimate about $10 lost per day per programmer just to make some bean counter at headquarters feel good about ‘saving’ money. That is just one of many examples where the company decided on Order over Progress.

Be careful whenever you have the idea of creating a ‘money saving’ rule, because if you don’t think through the non-obvious costs, you could lose a lot more than what it seems you’re saving! Semco and Google are great examples of focusing on the real business priorities.

I recorded this in Austin at the South by Southwest Interactive festival where Kathy Sierra gave the opening remarks (totally amazing as usual). Be sure to see her visuals at Brian Fitzgerald’s blog – they are pretty important in understanding the talk.

Why Zebras Don’t Get Ulcers

Yeah. We’ve gotten quite a lot of VC calls. But one of the things we’re seeing that we really don’t care too much for is that way too many companies are taking money when they don’t need it. And the whole idea we had was that having too little money is a great way of getting great product, because it’s a way to get focused. So we have definitely said to ourselves, “We don’t want any outside money. We actually don’t even want to grow our team.” We’re trying to design our products in a way that they can scale with more users without us having to scale as a company. So, through Signal vs. Noise, we’re trying hard to deliver a pushback to companies that feel like they have to hire a bunch of people as early as possible and to take money to realize their vision by saying, “If your vision of your product costs a million bucks to make, try rescoping that idea in your head so it fits in $100K and get it out there earlier. Instead of having a 1-year product cycle, what could you do in 1 month?” And sure, that doesn’t work for every company, but in the web age, it works for way more companies than are trying to.

- From David Heinemeier Hansson’s interview in Founders at Work by Jessica Livingston