BitMagic: overview of succinct and compression algorithms for data science. Bit-vectors, Elias-Gamma, BInary Interpolative Coding, Rank-Select compression, Bit-Planes compression, XOR compression.

BitMagic v.8.0.1 implements new serialized compression (improves on top of binary interpolative encoding). Current improvement is incremental, but translates in significant disk/traffic savings on large collections of bit-vectors and succinct arrays.

Generate videos in just a few seconds. Try Grok Imagine, free for a limited time.

Bioinformatics twitter. Igor Tolstoy, my long time colleague and contributor to BitMagic is looking for an engineering job. (Bioinformatics, Math, ML). In short (tweat) : Igor is very passioned about everything he is doing. (and I can answer PM).

Tech.notes on performance of sorting algorithms on compressive memory (case with 10x+ memory savings). Performance is slower than non-compressive, but still useful for low-memory (or big-data) situations, where it would otherwise spill to disk.

BitMagic v.7.13.4. Bug fixes (!), new methods for bit swapping and swapping elements in succinct vectors: allows efficient sorting in compressive memory. Optimizations, improvements in examples.

RT @i_bogosavljevic: I am looking for work, preferably in the domain of software performance. If you know someone who would profit from the….

Note on Apple MacBook Air M2. Great performance, but easily overheats (even in ST app like BitMagic performance test). (keyboard without PgUp/Home/etc. duh. ).

BM v7.12.3: optimizations across the code base: SIMD, parallel memory reads, reduction in number of branches, batter indexing algos. API changes, all results in improved speed of binary search in succinct vector of strings. (fast and compact dicts).

NCBI released a beta version of a new Comparative Genome Viewer. BM is used for memory compact genome wide alignments in WASM and network compression of data models.

BM Library has just been recognized with a Community Choice award by SourceForge. BM reached significant milestones in terms of downloads and user engagement from the SourceForge community.

BM v.7.11.2 released. Bug fixes and optimizations for bit-vector and succinct vectors immutable mode to reduce systems memory consumption and improve heap (de-)fragmentation.

Bioinformatics twitter: NCBI SRA released v3.0.0 with improved fasterq-dump (unsorted fasta mode(!)). It has potential to improve the data download experience.

BitMagic v7.10.3. Immutable vectors with memory savings via heap defrag and lower internal overheads (as a bonus some ops are also up to 5% faster) (screenshot shows synthetic example). Better Rank-Select index. New RS succinct vector gather API. Bug fxs.

C/C++ twitter: any tar reading library good for "fractal" .tar when archived file is inside tar: (file.gz inside tar.gz/bz2), tar inside tar.gz, etc. other possible recursives?.

BitMagic v.7.9.3. If succinct vectors are not fast it is not useful! .New version has new optimizations, new API methods and bug fixes. A few: AND masked search, improved random access to vector elements, faster Rank-Select index, better compare().

Filling the example gap. New example showcases the nomenclature of bit-vector traversal / decoding methods. C++ iterator concept was a brilliant invention, but not fastest when iterator FSM becomes complex. for_each/visitor design pattern is faster.

Taking a chance to bring this to attention:. This is a drop-in allocator replacement. When used with a BitMagic heavy app improves both process memory footprint and speed (sic!).

New BitMagic example how to use std::sort() with compressed memory vector. Useful for cases when application is under memory pressure and if not succinct data structures would have to use disk. Contributed by @shke_da.

RT @MisterTechBlog: Apple M1: Crowd-Sourced Reverse-Engineering Doc Published.

BitMagic v.7.8.0 released. New index free search algorithms on integer bit-sliced succinct vectors: fast without data decompression. New GT, GE, LT, LE and RANGE searches. New optimizations for SSE2 SIMD build. Plus this version now supports ARM Neon.