tl;dr: Keep forgiving yourself and keep working.

I am having the same problem, and only recently it got better. I have it only for open-ended work (scientific projects, other personal projects - everything which is of type "I should have it done" and the same time it is not closed; even worse when others are waiting for results). It seems to be very different from "normal" work (when someone gives me a particular task) and work with an expiry date.

The wisest (and most successful) piece of advice I found is this one (from Smart Guy Productivity Pitfalls - Book of Hook, which has more good points and is definitely worth reading):

6. Do not overpromise to make up for poor productivity. There's a tendency when we're falling behind to try to overcompensate with future promises. "When I'm done, it'll be AWESOME" or "I know I'm late, but I'm positive I'll be done by Monday". By doing those things we just build more debt we can't pay off, and that will eventually lead to a catastrophic melt down when the super final absolutely last deadline date shows up. Just get shit done, don't talk about how you're going to get shit done.

Also, somewhat related is forgiving yourself for being not productive enough (constantly feeling guilty does not help; not only for me, but it seems it does not work for most of people):

The key finding was that students who'd forgiven themselves for their initial bout of procrastination subsequently showed less negative affect in the intermediate period between exams and were less likely to procrastinate before the second round of exams. Crucially, self-forgiveness wasn't related to performance in the first set of exams but it did predict better performance in the second set.

And from a bit different angle, from Elizabeth Gibert's TED talk on genius (it's about treating inspiration, but it is similar for everything - no matter how good you are, you won't do everything; so why should you be bothered by missing a few things?):

And [Tom Waits]'s speeding along, and all of a sudden he hears this little fragment of melody, that comes into his head as inspiration often comes, elusive and tantalizing, and he wants it, you know, it's gorgeous, and he longs for it, but he has no way to get it. He doesn't have a piece of paper, he doesn't have a pencil, he doesn't have a tape recorder.

So he starts to feel all of that old anxiety start to rise in him like, "I'm going to lose this thing, and then I'm going to be haunted by this song forever. I'm not good enough, and I can't do it." And instead of panicking, he just stopped. He just stopped that whole mental process and he did something completely novel. He just looked up at the sky, and he said, "Excuse me, can you not see that I'm driving?" (Laughter) "Do I look like I can write down a song right now? If you really want to exist, come back at a more opportune moment when I can take care of you. Otherwise, go bother somebody else today. Go bother Leonard Cohen."

And from my personal stuff (I mean things that I found helpful):

• using to-do list only for task (i.e. things I know I can do in a few hour max), not projects (it's depressing to have "finish this paper" on the same list for long months, cf. relevant PhD Comics strip),
• underpromise and overdeliver to oneself; i.e. committing to do each day less task than expected (this way, with the same results, it's "wow, I did things from the list plus 2 extra" instead of "I only made almost half of the first point out of 7"; extrapolating one's maximal efficiency does not work...).

There are two threads in answers here that I'd like to respond to:

Research is just like any other job. The tools needed to manage guilt are no different.

Yes. and no.

The nitty-gritty of work - deadlines, working in groups, answering to a 'boss' - are the same. That is indeed true. What's different about research work that I think ff524 is alluding to is the "freedom trap". Because research work involves more freedom and more unstructured effort, and there's a direct correlation between output and success (not effort and success of course), the anxiety is not external ("my boss needs this done", or "I can't let my team down") but extremely internal ("I am an inferior researcher if I'm not working all the time" or "someone else is getting ahead in their career while I'm slacking off").

And this is incessant. Every minute spent not working is tied up in internal accusations. And it's exhausting. And that's what we'd like to be free of.

Do I have an answer ? Not really. It's a slow process of realizing that

• feeling guilty about work is a meta-worry that doesn't lead anywhere constructive (this realization only works in flashes :))
• all the other people racing ahead will also need to rest at some point.
• a guilt-free mind is clear and prepared for research (whether it's leisure time or not: as ff524 says, this is not necessarily about partitioning work and free time so much as not feeling guilty when not thinking about work. Indeed, one of the pleasures of being a researcher is that I can think about my work whenever I like, even when day dreaming on a bus to work (ahem).

In that respect, Piotr Migdal's answer about

forgiving yourself and working

is spot on. Guilt is rarely a constructive force, and it can lead you to make bad decisions to compensate. Blowing off that paper deadline ? it's ok. Dropping a fascinating research project because you're overcommitted ? that's ok too. Not spending enough time with students ? Hard to wave off, but it's ok.

But forgiving yourself only works if you trust yourself,

and again the Tom Waits analogy is brilliant. You have to trust that blowing off one paper deadline won't make you a lazy git who doesn't write any papers. That missing one student meeting doesn't make you an abusive advisor. That ignoring a collaboration doesn't make you a toxic personality. That if you can learn to trust in your own research instincts and drive that you'll be able to pick up and go full steam ahead, but this time with less guilt than before.

This is not a time-management answer, and you didn't want one ! So all that I can say is that reducing guilt is a slow process (I haven't figured it out yet), and you have to keep reminding yourself to forgive and trust.

To answer your specific question:

How do you avoid feeling guilty about all the unfinished (and unfinishable) work in academia?

You try to come to the understanding that there is always more work to be done, and that this is the way it is, not just in academia, but also in almost every other walk of professional life.

Disentangle the feelings of guilt and anxiety. There is work that you should have done/be doing (e.g. to test an idea fully, rather than assume the result; meeting deadlines) and there is work you could have done/or be doing (e.g. new ideas/extensions).

Concentrate on completing all the work that you know must be done. Set yourself practical goals and list them, marking them off when achieved.

Set out time for the other tasks you know need to be completed. e.g. 2 hours a week for meeting student A, 1 hour for student B, 2 hours for grant writing. Stick to those arrangements. Now add in time for "fun" work stuff - perhaps not directly related to your main goals, but perhaps which interest you at the moment.

Keeping a track of how much time you are spending on different types of task, and seeing how you are progressing in each activity, will allow you to fine-tune your time-management.

Having time set aside for each activity type - and sticking to your timetable - allows you to feel less anxious about the work you should be doing, because you know that you've boxed off time in your schedule to set to work on them. It gives you the confidence to say, okay, I'm not doing mission-critical stuff right now, but it's the time of the week for reading/meeting people/setting up webpage and I know that I'll be back on that task when I've the time allocated for it.