Lollo the Bichon Friseé in christmas outfit.

Release v0.4.4 · DanEngelbrecht/golongtail FIXED fix(s3): use HeadObject for checking if blob exists bergemalm ADDED New option to control the number of worker thread in remote stores to avoid overflowing the network connection. --remote-w...

New maintenance releases of longtail and golongtail

golongtail 0.4.4
github.com/DanEngelbrec...

longtail 0.4.3 github.com/DanEngelbrec...

If you are using longtail as a lib you need to switch from `Longtail_ChangeVersion()` to `Longtail_ChangeVersion2()` and use `Longtail_CreateConcurrentChunkWriteAPI`. It is that you drop the LRU block store layer is becomes redundant and only adds memory overhead.
Very happy with how this turned out

Turns out the new strategy is significantly simpler, uses less memory, reduces file I/O, cuts down on CPU usage and drastically reduces wall time to write a new version when it contains files larger than a single block.

The implementation was focused sequential writes of files that caused huge amount of extra I/O, memory and CPU work to be done.
Now, realizing that I don't *need* to write big files from start to finish, but instead I can write the parts in any order, I got to work implementing a new strategy.

Just before the holiday break at work I noticed that writing a version from an archive containing files larger than a single block was **very** slow - like, slow as in "this can't be right".
But it was.

Image showing memory usage, and wall time of speed ups of the new longtail version. 2.7 -> 3.25 x less memory, 2.2 -> 14 x faster

Happy new year all.
I finally got some time and energy to poke around with longtail.
New release includes `Longtail_ChangeVersion2()` used by default used by the cmdline tools.
Massive speed-ups for assets larger than one block.
Details in thread
github.com/DanEngelbrec...
github.com/DanEngelbrec...

Just release 0.4.0 of both longtail and golongtail.
Major point for both is MacOS Arm64 builds.

github.com/DanEngelbrec...

github.com/DanEngelbrec...