Depending on the total number of trials you have, another strategy is instead of saving your full struct after each trial, save it after a given number of trials. For example, we're doing classification image experiments where there is a rest period (e.g. blank screen with `Take a break and press a key to continue') implemented every 200 trials. During the rest period the trial information struct is saved, hence if the computer crashes after 2199 trials, you lose 199 trials instead of 2199 trials, yet the overhead of partial backups is minimal. Also at the end of all trials the struct is saved. Hence I get a series of files such as:
BD-4-05.07.09-19.36.15-tmp-200.mat
BD-4-05.07.09-19.38.49-tmp-400.mat
BD-4-05.07.09-19.41.31-tmp-600.mat
BD-4-05.07.09-19.44.59-tmp-800.mat
BD-4-05.07.09-19.48.02-tmp-1000.mat
BD-4-05.07.09-19.50.54-tmp-1200.mat
BD-4-05.07.09-19.54.20-tmp-1400.mat
BD-4-05.07.09-19.57.35-tmp-1600.mat
BD-4-05.07.09-19.57.37-1600.mat
Which includes a trial condition string (`BD-4`), a timestamp, and the number of trials included in the matrix. This is done in the code by:
%nn is the trial number, data is the struct containing the trial information
if(mod(nn,200) == 0)
%Also save current data in a temporary file
filename = [data.title '-' datestr(now,'dd.mm.yy-HH.MM.SS') '-tmp-' int2str(nn) '.mat'];
save(filename,'data');
end
data is not cleared after 200 trials. However I do have a function to stitch together two structs (say, from two different sessions), which is:
function [data] = mergeDataSets(names)
load(names{1});
tmp = data;
for ii = 2:length(names)
load(names{ii});
fn = fieldnames(data);
for jj = 1:length(fn)
val = data.(fn{jj});
if isnumeric(val) && length(val) > 1
tmp.(fn{jj}) = cat(1,tmp.(fn{jj}),data.(fn{jj}));
end
end
end
data = tmp;
end
This runs through all the keys of the structs and merges the vectors/matrices making them up along the first dimension (e.g it works for column vectors but not row vectors) assuming they contain numeric data.
Patrick Mineault
On Mon, Jul 6, 2009 at 5:38 AM, e_flister
<e_flister@...> wrote:
> > What I do is to use a struct array and just save it every trial. There's no need to append, because there's no significant cost to saving a 1x100 struct vs a 1x1 struct. You just save the whole thing every time.
just a word of caution - using this strategy makes save time linear in the number of trials and can become quite noticeable (and contributes to incomparability of trials early vs. late in a session). we have an automated system for training rats in their home cages, and they self-pace their trials, peaking around 1 trial/sec. our trial records are large and our sessions are typically many hundreds of trials long. by the end of sessions, save times were dominating trials, until we switched to an -append strategy. this meant generating a unique variable name for each trial -- lame!
even more lame is that loading struct arrays (or other dynamic structures like cells) from files is extremely slow -- apparently .mat's don't cache the memory layout of variables, so matlab sequentially allocates each member of a struct array as it is read. in our case, analyzing several months of daily records stored as struct arrays is intractable, so we have to maintain an intermediate file compiled into simple vectors. ugh!
-e