[time-nuts] Archiving Timing Data

Chris Albertson albertson.chris at gmail.com
Mon Jan 10 20:42:03 UTC 2011


We have mountains of data here too.  The best why to store it is in a
"real" database of some kind.  There are several that are free, open
source and multi-platform.  The best for this use is "Postgres".   As
this is free and open source there is no reason not to use it.

In the past I've kept snapshots for simulations that have run for
hours/days/weeks and we got many hundreds of millions of data points.
 Then we are able to query for almost any conditions and expression,
for example "Give me a A, B where A-B less than 4 from July 5th 1998"

I can tell you first hand that having a billion lines of tab separated
data is worse than useless.  You need itcataloged such that you can
very quickly (seconds) find useful subsets of the data and you can
NEVER know in advance what subset you might need.




On Mon, Jan 10, 2011 at 12:22 PM, Peter Vince <pvince at theiet.org> wrote:
> Would a TSB (Tab Separated Value) format be preferable?  Full-stops
> and commas are used in numbers as decimal and thousands separators (or
> vice versa), so using tab character would avoid any problems with
> commas in the actual data (and make it is a bit easier to quickly
> eyeball when viewed in a text editor).
>
> Peter  (G8ZZR, London, England)
>
>
> On 9 January 2011 17:15, Bob Camp <lists at rtty.us> wrote:
> ...
>> I doubt very much I'm the only one taking a mountain of timing data and not properly cataloging it. My guess is that maybe > 90% of the list members are in the same boat. How about:
>>
>> 1) A set of not to restrictive data format standards (CSV with a few restrictions ...)
> ...
>
> _______________________________________________
> time-nuts mailing list -- time-nuts at febo.com
> To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
> and follow the instructions there.
>



-- 
=====
Chris Albertson
Redondo Beach, California



More information about the time-nuts mailing list