Properly understanding the use of date and time within a system is one of those areas that looks simple initially, then looks quite complex, but if you approach it in the right way can get downright simple again.
What is a date? Sounds like a simple question, but its something that's worth spending some time on.
One well known date is the Fourth of July. Notice that there's no year attached to it. Sure, you could say 1776, but most people use that representation to mean any instance of 4 July rather than a specific one. You could define a repeating series of specific annual date objects, but that's just hiding the fact that an anniversary isn't the same as a time_t date field.
Maybe we're talking more specifically about a particular date such as the original 4 July 1776. It would be tempting to define that as the period of time from midnight to midnight on 4 July, EST, but that's also not quite right - people in CST don't think it as running from 1AM on the 4th to 1AM on the third, after all.
Next you have actually defined points in time. These are used when recording events, for example, such as a credit card transaction or a password reset request. These are the only items that actually map to a traditional java.util.Date or a time_t field.
Then you have relative dates - "yesterday" or "EOM". The user-facing definition of those doesn't change, but the actual point-in-time referred to will change as time passes. These can get confusing as well if they're being communicated across timezones. Do you mean "your yesterday" or "my yesterday?"
Finally you have ranges of the above. These can be colloquial (the Thanksgiving holiday in 2012 covered 22 Nov and 23 Nov) or absolute (the system was down for maintenance between 2012-11-22T21:00:00 and 2012-11-22T23:30:00). You can have relative ranges too, such as "last week".
Storage and representation
There's really only one simple rule. When it comes to actual specific point-in-time dates, everywhere except during user-facing input and output you should be using UTC exclusively. I do mean everywhere, by the way - your server timezones should be UTC, your log files should be written in UTC, your database and appservers should believe that their native timezone is UTC - you get the idea.
I strongly recommend that you also pick up a physical clock or two to have for the office and anywhere else you do a lot of server-side work.
This approach buys you several things. First, daylight savings time doesn't exist in UTC. Think about the implications of processes that might want to fire at 1:30 AM for a minute. Once a year they'll fire twice, and once they won't fire at all. Second, nobody in their right mind actually prefers to look at UTC dates natively, so you basically have to get timezone conversion handled properly in your GUIs and reports. These days almost every serious system will end up with users in more than one timezone, and trying to bless one of them as "more correct" will almost always backfire eventually.
For the other types its less important how you store them than it is that you recognize them as different and distinct and treat them as such. For example, you could choose to store the original independence day as the native Date version of 1776-07-04T00:00:00 in the database, but you shouldn't be passing it around either in code or in an API as such, because its not an actually specific point-in-time. Your technology choices should probably influence storage and handling - do whatever's easiest, as long as you always know what you're really looking at.
Whatever your client is - web application, mobile app, etc - I recommend using the “parent” timezone if such a thing exists (for example, if you’re describing event registrations, use the timezone of the event). If there’s no such natural timezone, use the user’s own. It often makes sense to allow people to “dictate” a timezone - for B2C software this may be set at the organization level if nowhere else - but for personal use, deferring to the device just makes sense most of the time.
Ideally, wherever there’s a mismatch between the displayed timezone and the device timezone, you should include the timezone abbreviation on screen somewhere (next to a field, in a report header, etc). This can get a bit silly after a while so you’ll need to balance style and substance.
Whatever you end up doing, timezones are real and - when handled correctly - effectively invisible. If you notice them, something’s probably not quite right.