one big library

I cut my teeth as a legit full-time programmer at my previous job. I'd written webapps before but nothing particularly complicated. I didn't know any one language much better than others. I hadn't built something substantial from scratch and seen it through from requirements to long-term (i.e. more than a year or two) maintenance.

All of this is to say - I still maintain two apps from that period. The thing about those apps is that they reflect how I didn't know what I was doing when I started, and I only knew enough about what I was doing when I finished regular coding on them to know what was wrong but not necessarily how to fix it. So I'm left with two apps people still care about. One's been broken for a year, and I made the mistake of rewriting it from scratch, so, natch, I haven't finished yet. Don't do that.

The other one broke last night. Badly. I just fixed it, finally, but I can see nothing but bad decisions and do-repeat-yourselfness and things that need Cleaning Up in there. Even so, and despite there being one horrific showstopper bug in there that needed a fix to get it running again, this other app kept chugging along, however pokily, until a perfect storm of long-term-maintenance-mode and indifferent-and-remote-and-slapdash-sysadminnery threatened to bring down the whole thing.

So, here are some pearls of idiot wisdom for you. By which I mean to say: don't make my mistakes. They weren't original mistakes when I made 'em, but if you go and do them now, they'll just be embarrassing *and* tired.

• If you're leaving an app running indefinitely, e.g. for N years until somebody might say "okay, fine, you can shut it down," set up a regular log rotation scheme. If you don't, your logs will eat up your hard drives eventually. Maybe not overnight, maybe not in a month, but in 2.8 years since you've paid regular attention to it, yeah, maybe. Learn logrotate or whatever the kids use these days, and use it.
• Before you run an upgrade, check your available disk space.
• Whenever you check on the server, check your available disk space. (Get the point yet?)
• If your old app fails suddenly after an upgrade, and you can't tell why, it might not be all the dependencies in your old app. It might, in fact, be that your logfiles filled up your disk, ya dope.
• If you clean up that mess and find that your app still doesn't work, remember: if you filled your drive, things broke unpredictably. Your data and other subsystems might not be in a stable state. If you have a database, verify that your tables aren't corrupt using something like myisamchk and fix them if they are corrupt.
• If you're still stuck, go onto your friendly nearest IRC channel or list and say "hey, I'm an idiot, this old app of mine failed and I see X, Y, and Z but not what's going wrong still after I cleaned all that up." Your friendly chatmates might say lots of useful things to help you rule out the myriad things that might have gone wrong. They will listen to you vent a bit and help you lighten up a bit. They will help you focus on the problem you have to fix instead of your emotional baggage of the half-baked app you wrote and never cleaned up real nice and the ridiculous non-rotated logs that filled up the disk and broke everything. They will remind you to pour yourself a drink and chill out. And if you listen well and stay patient with yourself, they'll share your joy when you fix your problem.

And finally: always keep a good bottle of bourbon at the ready for such times.

[Much thanks to robcaSSon, edsu, rangi, erikhatcher, and wickr for the patient support!]