Why was DateTime removed from OData v4

As explained in this post, it is NOT human nature to resist change, but we do like to follow the path of least resistance, and a change that forces us to do extra work for percievably little or no benefit is going to rub anyone the wrong way.

As Software Developers we often get frustrated by the release of new development tools or frameworks that are not fully backward compatible with our current codebase or hardware, we want the new features without having to modify and re-test the current code.

There is a paradox of sorts here, we like the idea of backward-compatibility, but often the legacy support prevents the product from actually evolving and improving and can ultimately result in a product that we no longer see the value in.

Dates and Time offer an intriguing paradox in the development world, in many cases we think we do not care about the time zone associated with a value, and yet in this age of global communication and cloud computing, the time zone is vital to determine the Date even without the time component. There are 3 basic techniques developers use to work around this:

1. Separate Time and Date into their own fields
2. Sanitize all inputs and convert to and from UTC
3. Embrace DateTimeOffset and retain the original time zone from data entry

DateTimeOffset was introduced in .Net Framework 2.0, at the time with this guidance:

Although the DateTimeOffset type includes most of the functionality of the DateTime type, it is not intended to replace the DateTime type in application development

and by 3.5 it was considered the default with this advice, an edit to the same article:

Note
These uses for DateTimeOffset values are much more common than those for DateTime values. As a result, consider DateTimeOffset as the default date and time type for application development.

Originally DateTimeOffset was created to simplify application logic, just like Date and Time were combined into DateTime to simplify transferring these two related concepts between logic blocks and models without losing their relationship to each other, DateTimeOffset was created as we started to require time zone information for more logic and our applications started to become more global. What wasn't originally appreciated at the time was the fundamental shift that was to occur, that before our user base started to commonly span multiple time zones, it would become more common for our applications or databases to be hosted on infrastructure in another time zone. This facet of Web 2.0 was addressed in the release of .Net 3.5, a time when suddenly even applications that traditionally had no need for it, needed to be aware of what the local time zone might be to operate as expected.

The first rule of .Net is that we don't talk about .Net v3...

The development and release of .Net 3.5 also coincided with the release of SQL Server 2008 which introduced support for DateTimeOffset with similar advice to use DateTimeOffset as the default storage for date and time. If you only need a date or time then this release included support for both Date and Time data types as these are usually more appropriate for scenarios where the timezone is not relevant the he sotred value.

The argument for DateTimeOffset in .Net was that with the proliferation of distributed and globally hosted web applications and Web API, the exact point in time, or even the specific Day of week is ambiguous to the executing code. Especially if the code is executing in a different Time Zone to that of the user, no longer can we use the local time zone of the executing code to resolve the local time (or Date) from the internal UTC. Instead, we need to store another element of information, the Time Zone that the Date and Time value is relative to and then we need to access that information for all boundaries and comparisons.

The argument for SQL Server was similar, it was becoming common for databases to be hosted externally or "in the cloud", and the concept of geo-redundancy or replication was also being adopted for on-premises databases. Again how to resolve the expected time zone and how to reliably convert values into that time zone become problematic without specifically storing or providing that value. But the SQL problem was compounded by the ambiguity of where the database was actually hosted, the application runtime might be in the same PC as the user and the database could be the same PC, the same building or on the other side of the world, the application cannot detect or know this level of information without a fair amount of effort. The main problem with this was that you could not determine if the conversion to or from UTC had already been performed, there is no reliable indicator in the data to tell you this, instead you have to rely on convention alone and keep track of the logic you were writing to determine if the underlying value was expected to represent UTC or Local or some other Time Zone.

On the balance of effort to implement the different workarounds for the concepts of Date, Time and Time zone, DateTimeOffset requires the least amount of custom code and logic overall, not just in your application code but in the database as well. This is due to the fact that internally the value is stored and compared in UTC so there are no conversions needed in our comparison or processing logic, but the original Offset is retained so the value can always be easily represented to the user in the original native time zone that the value is meant to represent.

The transition to DateTimeOffset may require us to reduce the code we currently have to work around the limitations of DateTime. DateTimeOffset was the path of least resistance that we should have adopted long ago, it is a path you will be forced to walk at some point, if you have not already.

Daylight Savings

What is often missed in this conversation is that the DateTimeOffset structure also solves many issues presented by Daylight Savings. In a legacy system that uses the UTC convention for internal storage, running a report before or after the daylight savings boundary can result in values that represent a time on the other side of the boundary being out by the variance of the local daylight savings offset. In most visual cases, an event at 10pm on a given day before or after the boundary should still be considered to have occurred at 10pm on that day, it is not 9pm for instance if my local time zone has changed to +11 from +10.

The convention to split Date and Time solves the daylight savings problem better than the UTC convention but does not help us solve the internationalization issues or even data that spans a local time zone region difference.

DateTimeOffset is an encapsulation of the convention where we want to store all Date and Time values internally as UTC but still view the values in their original time zone, this supports data that covers regional boundaries as well as data in regions that observe daylight savings.

What is wrong with DateTime anyway?

Before the release of .Net many languages and database designs would store date, time, (and if needed the time zone) as separate values, the only correlation between them being the name of the value store or the object that it is contained within. This code pattern has bled into many early .Net application designs.

So it would seem that the resistance to the change over to DateTimeOffset was predated by the previous resistance to a change over to DateTime in the first place.

The concept of combining Date and Time together is not unique to .Net this is just a commentary on why you might find some older code examples that still use this legacy design pattern.

The separation of Date and Time was a structural and storage issue. In the past bytes came at a premium that affected memory for processing, as well as storage. There was a time when your entire application needed to run in memory measured in kilobytes, where unused bytes taking up space on the stack meant you had to compromise on the length of your compiled code. But also we tend to read and manipulate these concepts separately, at many times mutually exclusive of the other. No longer bound by the physical constraints, we can now store these values as a single unit and still perform operations on either concept without altering the other if we need to.

The fundamental idea behind DateTime was a good one, lots of programming logic needs both Date and Time together to complete an operation, so lets keep the data together to reduce the round trips and lookups that might need to be performed at runtime. But both the thought and the implementation was incomplete.

When we introduce issues like Daylight Savings and Time Zones, our code becomes less deterministic and a LOT of assumptions need to be made. For instance originally there was no way in the code (from the value alone) to determine if the value was supposed to represent UTC or the Local time zone. This could easily result in double conversions or no conversion at all. Time zone became one of those silent runtime errors that were obvious when you finally detect them, but hard to track down and maintain. The problem with value conventions like the UTC storage is that they are hard to detect or enforce. But it is imperative that you stick to the convention to avoid errors.

In .Net 2.0 (the same version that introduced DateTimeOffset) the DateTime structure was extended to include a Kind property to help you identify if the DateTime was supposed to represent UTC or the local time zone. There is no way to modify what Local means, it is a global reference to the time zone of the executing operating system, you can see how and when Local is resolved in the original source code, while this potentially helps implementations of the UTC convention, it doesn't assist with comparing across different time zones. That was primarily the reason for DateTimeOffset, to allow for direct comparisons between different time zones.

There were three fundamental flaws with utilizing the DateTypeKind

1. The Kind is not preserved during serialization, so the kind can be lost in the transition between data at rest (in storage) and data in memory, but perhaps more important, the Kind is not transmitted to any externally integrated systems, not by default. It is expected that at the boundary between memory, storage or other systems, that the DateTime be resolved into an expected pre-declared time zone that is either UTC, or the same timezone as that of the underlying operating system.

2. Related to the first, The Kind information is not preserved then we need to manually re-inject this information during deserialization, so when resolving the data from storage into memory. This can help to reduce the double conversion scenarios, but only if we include this information at the time that any conversion is made. Most .Net developers would be aware of DateTimeKind but would probably never have directly coded it in their database or web api logic.

3. Local is hardcoded in the framework to be the time zone of the executing Operating System, in cloud hosted infrastructure this might be UTC or an entirely different time zone to that of the user or the data storage. Especially in modern times when we can deploy code to ServerLess, PaaS or IaaS resources that might be in multiple regions, the concept of Local in general is not only ambiguous but out of our control.

DateTimeOffset persists the time zone information across the entire stack, from the user input, through processing, then storage, through serialization back into the final output where the value will be rendered. There is no abiguity about the value at any point in time, if you need to enforce visibility of the value in a specific time zone then the value can be easily translated without having to second guess if a conversion had previously been processed.

Why specifically deal with this in UtcNow, in OData 4.0?

OData v4 is ALL about standard conventions, with a goal of reducing the specification down to a minimal set of conventions needed to universally query and transmit data between systems.

When it comes to Date and Time, you can see from this article alone there are a number of closely related concepts, so OData uses Date and TimeOfDay to express these elements when you need them exclusive of each other and DateTimeOffset when you need to maintain the relative association of the values.

DateTimeOffset covers all possible uses of DateTime but in an explicitly declarative fasion, there is no kind to interpret or to misinterpret and it can be serialized directly into and from the javascript date type that also includes Date, Time and Time zone component values, without losing any information. Given the ubiquity of javascript across html browsers and that javascript makes up the majority of OData consumers it makes sense that it was adopted as a standard into the OData protocol as well.

Removing DateTime from the .Net implementation of OData in OData v4 is not a decision made exclusively by Microsoft or the .Net team, it is a deliberate adherence to the OData v4 protocol specification. To introduce another primitive type to the system would require compliance on all implementations which would require all of us to agree if the value represented UTC, your Local or my Local or some other arbitrary time zone.

Removing DateTime removes a whole layer or management and ambiguity from everyone's codebase and allows a more collaborative environment. Your time zone sensitive implementation can ingest data from my application that operates in a single local time zone of UTC+10 without making any assumptions or asking for any additional metadata.

DateTime was a useful, yet failed experiment from a time when Time Zone was not an important aspect for processing specific points in time. Since 2005 .Net developers have been actively encouraged to use the DateTimeOffset which is a simple and obvious evolution from DateTime. All the original arguments for transitioning to a combined Date and Time value type apply equally to using DateTimeOffset except that we now recognise the importance of the Time Zone when interpreting the value.