Breaking data.table in #rstats. This was noticed by our brilliant data analyst Hamza Mir, when analyzing a 10 #gigarow #EMR dataset. #fread will assign a name to each row, i.e. the row number. If the number of lines exceeds 2^31, then a long vector of names has to be generated and R does not support those apparently (yet). Reproducible example (#Perl code to generate a very large dataset on left, and #rstats output with failure). ALT-text for code. See also https://stackoverflow.com/questions/76456801/data-tablefread-fails-for-larger-file-long-vectors-not-supported-yet

Breaking #fread in #rstats with #gigarow datasets

mstdn.science/@ChristosArgyrop/1143563...

(or alternatively here: mast.hpc.social/@ChristosArgyrop/1143563... )

Christos Argyropoulos MD, PhD (@ChristosArgyrop@mstdn.science) Attached: 2 images Breaking data.table in #rstats. This was noticed by our brilliant data analyst Hamza Mir, when analyzing a 10 #gigarow #EMR dataset. #fread will assign a name to each row, i.e. the…

Breaking #fread in #rstats with #gigarow datasets

mstdn.science/@ChristosArg...

(or alternatively here: mast.hpc.social/@ChristosArg... )