MATLAB: Help correcting a messy time series data

missing datamixed datatime series

Hi,
I have multiple files of "daily" minimum temperature. The time series is non-continuos. Files start and end in different dates, some days in the middle are missing (the rows are missing), and some days have more than one measurement.
This is an example of what I have:
1999 01 01 5.2
1999 01 02 4.3
1999 01 02 5.0
1999 01 02 4.1
1999 01 03 3.8
1999 01 05 3.2
...
So day 02-jan has 3 different meassurements and day 04-jan is missing. Say that I need all the files to begin at 31-dec-1998 and end at 07-jan-1999, my files should end up looking like this:
1999 12 31 6.6
1999 01 01 5.2
1999 01 02 4.1
1999 01 03 3.8
1999 01 04 NaN
1999 01 05 3.2
1999 01 06 NaN
1999 01 07 NaN
So far I managed, using this thread , to fill the missing dates with NaNs.
I still need to:
1) take the minimum value of the days with more than one measurement. I have absolutely no idea how to do this…
2) complete the period with NaN or crop the data between the starting and the ending date that I need. So far I managed with a long and messy script to crop and/or add the missing data at the end. But I couldn't make that work for the beggining, and I'm sure there must be an easier, more effective way of doing so.
I would appreciate any help you can give me!

Best Answer

  • YMD_data = YourData(:,1:3);
    temperature_data = YourData(:,4); %must be column vector
    first_wanted = datenum('31-dec-1998');
    last_wanted = datenum('07-jan-1999');
    dayspan = last_wanted - first_wanted + 1;
    dnum = datenum(YMD_data);
    in_range = first_wanted <= dnum & dnum <= last_wanted;
    useful_dnum = dnum(in_range);
    useful_temperature = temperature_data(in_range);
    relday = useful_dnum - min(first_wanted) + 1; %so first day is 1, next is 2, etc
    mintemp = accumarray(relday, useful_temperature, [dayspan, 1], @min, NaN);
    wanted_dates_vec = datevec(first_wanted : last_wanted);
    results = wanted_dates_vec(:,1:3, mintemp);
    That's it. All of the real work is being done by the accumarray() call, which is going to construct an entry for each consecutive day, and the entry is going to be the min() of all of the data entries with the same date relative date number. The entries for which there is no information will be filled with NaN.
    The bit after that constructs the output table. datevec() produces an N x 6 array in which the first three entries are Y M D (then H Min S). Extract those three, paste on the column output from accumarray and the task is done. The entries will appear in sequence, the minimums taken, the missing data NaN filled. No subselection of the output is needed because we selected what we wanted before passing it into accumarray.