We're at a point in release beyond saturation where it's not just impossible for one human being to keep up with all the T on the airwaves, it's no longer possible to watch all the "Must Watch" shows out there.
Unless you're a professional TV writer, TV is a way to supplement your life and naturally it will be balanced with other time demands of your life. You can adjust your answer accordingly if you fall under the category of aspiring professional, semi-professional or serious hobbyist, but unless you're veering into being irresponsible, we all have to abide by some sort of TV diet.
To compound the problem, we're living in a Golden Age of TV. As a result, there are going to be a great number of programs more worthy of your attention than programs you have time to watch.
But this isn't a curse. It's a blessing. In contrast to people who say that TV is a waste of time and bad for your brain hi mom while praising other forms of art like cinema or live theater, I maintain that the capacity of TV to enrich you culturally, socially, and intellectually is greater than ever before. There's no greater evidence of that than the fact that the amount of enriching programs on the air exceeds our available time to watch them all.
So if you have to give up on some critically acclaimed program like Mad Men or Dexter to be able to keep up with Justified or Enlightened, it's still a win-win situation. The only danger you have of "losing" as a TV watcher is if you don't use your TV diet to challenge yourself. If you watch a soap opera, reality TV show, a standard procedural, or a sitcom that doesn't push the boundaries (i.e. Two and a Half Men, According to Jim), a rerun of something you've already seen, then you are just using the medium as comfort food and guilty of eroding your brain like your parents (if they were anything like mine when I was younger) accused you of. Of course, the degree to which something like Grimm, Raising the Bar, Royal Pains, or Southland transcends the procedural or whether a certain reality TV show has merit, is up to you the viewer to justify. But that's part of the fun. I've never bought the argument that Happy Endings has merit beyond the standard sitcom, but I did enjoy the process of my fellow film critics slowly discovering that Happy Endings wasn't a typical sitcom (Cougar Town also falls into this category).
It's not just a blessing, but a challenge. Sure, it is really easy to fall back on TV as comfort food. It takes a little effort for me to explore something new than to fall back on a rerun of Futurama, Archer or Newsradio which are instant gratification for me. In fact, since the era of YouTube, my attention span has significantly shortened to the degree to which an hour-long drama can feel like something of a chore. A show like Homeland is so suspenseful that I have no trouble jumping on board, but I've also challenged myself with shows that might not be immediately as rewarding like Scandal, Revenge, or (the now defunct) Terra Nova to develop myself intellectually [edit: What was I thinking when I wrote of Scandal as challenging? Perhaps 12 Monkeys or Humans would be better recent examples]. With a show like Hell on Wheels, it paid off heavily [Sense8 and The Bridge are a couple other examples of shows paying off heavily if you get past the slow burn].
As for discarding shows, I've never seen an episode of Dexter, Friday Night Lights, and have missed large swaths of Mad Men and Breaking Bad but I don't consider myself the lesser for it as long as whatever I'm watching grows me as a TV watcher. Sure, it might hurt my ambitions as a professional TV reviewer [edit: I am now slightly semi-professional as a TV reviewer, but the same still applies as I have larger ambitions], but there's a wealth of material I'm already exploring.