<div dir="ltr">Oh ... I would also need the size of each stream's sample (including the size of vectors if there are any) ...<div><br></div><div>About index loading: the way the index was marshalled needed to be changed (but was not) after the change you made to indexing (i.e. making indexes dense). A 3-line patch improves performance quite a lot already. Alignment is already pretty good on my test file (~4s).</div>
<div><br></div><div>I've generated a 6G log file with 3 streams, ~650k samples in 1 minute of logical time, reading the streams using the Ruby stream aligner yields ~115x if I don't read the data sample, ~8x if I do. To me, this definitely means that the performance problem is not pocolog's low-level handling but both I/O (I have "images" of 1M, so I/O is actually a big performance problem), typelib demarshalling (there is one thing in this code path that sucks big time currently in the typelib-ruby binding related to demarshalling, did not measure the actual impact of it, though) and the fact that I hit the ruby GC pretty hard by creating 650k samples and 650k intermediate buffers.</div>
<div><br></div><div>The point is: as soon as you'll get ruby bindings, you'll have the same issues (GC, typelib demarshalling and I/O will not go away). The only thing that goes away is the creation of 650k intermediate buffers, but that can be removed with very little work already.</div>
<div><br></div><div>Now, I do get 8x replay speed when demarshalling the data. That's very far from 0.6. When you talk about bad performance, were you using the replay UI ? </div><div><div><br></div></div><div>For reference:</div>
<div><br></div><div>pocolog test.0.log < this is a 5.9G file</div><div><div><br></div><div>Stream images [/images] < these are 1M samples</div><div> 5994 samples from Sun 08/06/2014 22:36:42 to Sun 08/06/2014 22:46:41 [0:09:59.299]</div>
<div>Stream motors [/motors] < these are 100 bytes samples</div><div> 599708 samples from Sun 08/06/2014 22:36:42 to Sun 08/06/2014 22:46:41 [0:09:59.706]</div><div>Stream pose_samples [/pose_samples] < these are 512 bytes samples</div>
<div> 59970 samples from Sun 08/06/2014 22:36:42 to Sun 08/06/2014 22:46:41 [0:09:59.690]</div></div><div><br></div><div>Sylvain</div></div>