{"id":18,"date":"2020-06-03T13:06:18","date_gmt":"2020-06-03T11:06:18","guid":{"rendered":"http:\/\/visnow.org\/?page_id=18"},"modified":"2020-10-22T16:43:40","modified_gmt":"2020-10-22T14:43:40","slug":"jlargearrays","status":"publish","type":"page","link":"https:\/\/visnow.org\/index.php\/projects\/jlargearrays\/","title":{"rendered":"JLargeArrays"},"content":{"rendered":"\n<p><strong>JLargeArrays<\/strong> is a Java library of one-dimensional arrays that can store up to 2<sup>63<\/sup> elements.<\/p>\n\n\n\n<p>All current implementations of Java Virtual Machines allow the creation of one-dimensional arrays of length smaller than 2<sup>31<\/sup> elements. In addition, since Java lacks true multidimensional arrays, most of the numerical libraries use one-dimensional arrays to store multidimensional data. With the current limitation, it is not possible to store volumes of a size larger than 1290<sup>3<\/sup>. On the other hand, the data from scientific simulations or medical scanners continuously grow in size and it is not uncommon to go beyond that limit. JLargeArrays addresses the problem of the maximal size of one-dimensional Java arrays providing ones that can store up to 2<sup>63<\/sup> elements. Performance comparison with native Java arrays and Fastutil library shows that JLargeArrays is the fastest solution overall. Possible applications in Java collections as well as numerical and visualization frameworks are also discussed.<\/p>\n\n\n\n<p>In 1999, David Flanagan in his book \u201dJava in a Nutshell\u201d wrote: <\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p><em>Array index values are integers [..]. Although long is an integer data type, long values cannot be used as array indexes. This may seem surprising at first, but consider that an int index supports arrays with over two billion elements. An int[] with this many elements would require eight gigabytes of memory. When you think of it this way, it is not surprising that long values are not allowed as array indexes.<\/em><\/p><\/blockquote>\n\n\n\n<p>Even though that statement may sound rather amusing today, it is still valid for all current JVM implementations. One may argue that datasets larger than 2<sup>31<\/sup> elements should be processed in an out-of-core or distributed fashion and not loaded at once into the memory. Albeit, that approach usually requires reimplementation of existing algorithms to fit a particular framework, such as <em>MapReduce<\/em>. On the other hand, servers with more than 500GB of RAM memory are not uncommon and the array length limitation prevents Java programs to fully utilize their capabilities. This motivated the development of JLargeArrays. <\/p>\n\n\n\n<p>JLargeArrays uses low-level memory operations available in <em>sun.misc.Unsafe<\/em>. Even though that class is marked as an internal proprietary API that may be removed in a future release, it is widely used by JDK itself in several packages such as <em>java.nio<\/em> or <em>java.util.concurrent<\/em>. In addition, against common belief, it is available in all major JDK distributions including Oracle, IBM, and OpenJDK. Therefore, we claim that JLargeArrays is a portable library that might be used in many different applications.<\/p>\n\n\n\n<p>See the following paper for more details:<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\"><p>P. Wendykier, B. Borucki and K. S. Nowinski, &#8220;Large Java arrays and their applications,&#8221;\u00a0<em>2015 International Conference on High Performance Computing &amp; Simulation (HPCS)<\/em>, Amsterdam, 2015, pp. 460-467, doi: 10.1109\/HPCSim.2015.7237077. [<a href=\"https:\/\/ieeexplore.ieee.org\/document\/7237077?arnumber=7237077\">link<\/a>]<\/p><\/blockquote>\n\n\n\n<p><\/p>\n ","protected":false},"excerpt":{"rendered":"<p>JLargeArrays is a Java library of one-dimensional arrays that can store up to 263 elements. All current implementations of Java Virtual Machines allow the creation of one-dimensional arrays of length smaller than 231 elements. In addition, since Java lacks true multidimensional arrays, most of the numerical libraries use one-dimensional arrays to store multidimensional data. With the current limitation, it is not possible to store volumes of a size larger than&hellip;<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":16,"menu_order":4,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-18","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/pages\/18","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/comments?post=18"}],"version-history":[{"count":8,"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/pages\/18\/revisions"}],"predecessor-version":[{"id":488,"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/pages\/18\/revisions\/488"}],"up":[{"embeddable":true,"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/pages\/16"}],"wp:attachment":[{"href":"https:\/\/visnow.org\/index.php\/wp-json\/wp\/v2\/media?parent=18"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}