{ "ml": { "v": "4", "fs": [ { "id": 0, "prob": 0.8383554220199585, "class": 0 } ], "prob": 0.8383554, "class": 0, "models": [ { "m": "az", "prob": 0.8383554, "class": 0 }, { "m": "az/scripts", "prob": 0.36127168, "class": 0 }, { "m": "az/javascript", "prob": 0.62352943, "class": 0 } ], "version": "v17.17", "thresholds": [ 0.999503, 0.9998882 ], "analyzed_at": "2026-05-26T03:51:55Z" }, "path": "pm-gawk.texi", "raw": { "v": "5", "fs": [ { "f": "O(P)H(Po)", "x": 38, "ff": { "i": [ [ "", "check" ], [ "", "gawk" ], [ "", "being" ], [ "", "''" ], [ "", "pmg" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "s.N" ], [ "", "inp" ], [ "", "wc" ], [ "", "wc" ], [ "", "wc" ], [ "", "wc" ], [ "", "wc" ], [ "", "wc" ], [ "", "wc" ], [ "", "wc" ], [ "", "components" ], [ "", "monoliths" ] ], "m": { "ast": { "max_depth": 45.0, "call_count": 66.0, "node_count": 2062.0, "member_count": 12.0, "identifier_count": 390.0, "member_depth_max": 3.0, "string_literal_count": 63.0 }, "file": { "size": 68046.0, "entropy": 4.79, "size_bytes": 68046.0 }, "text": { "digit_ratio": 0.02, "space_count": 14343.0, "total_lines": 1403.0, "char_entropy": 4.79, "unique_chars": 96.0, "escape_density": 0.0, "string_density": 0.04, "ascii_art_lines": 22.0, "avg_line_length": 47.5, "max_line_length": 121.0, "empty_line_ratio": 0.11, "last_line_length": 69.0, "long_token_count": 4.0, "whitespace_ratio": 0.23, "most_common_ratio": 0.08, "identifier_density": 0.52, "line_length_stddev": 29.06, "unicode_escape_count": 2.0, "normalized_string_count": 1.68, "repeated_char_sequences": 150.0, "suspicious_string_ratio": 0.48, "most_common_char_is_null": 0.0, "suspicious_comment_ratio": 0.2, "max_inline_whitespace_run": 83.0, "trailing_whitespace_lines": 1.0, "most_common_char_codepoint": 101.0, "suspicious_identifier_ratio": 0.04, "normalized_unique_identifiers": 37.31 }, "calls": { "count": 66.0 }, "members": { "count": 12.0 }, "strings": { "total": 63.0, "avg_length": 985.17, "max_length": 8238.0, "path_count": 3.0, "avg_entropy": 3.85, "email_count": 3.0, "empty_count": 4.0, "sql_strings": 21.0, "total_bytes": 62066.0, "entropy_stddev": 1.14, "very_long_strings": 18.0, "shell_command_strings": 6.0, "embedded_code_candidates": 3.0 }, "comments": { "chars": 237.0, "lines": 5.0, "total": 5.0, "to_code_ratio": 0.0, "high_entropy_comments": 1.0 }, "identifiers": { "count": 390.0, "total": 733.0, "avg_length": 5.88, "max_length": 18.0, "min_length": 1.0, "avg_entropy": 2.14, "reuse_ratio": 0.53, "unique_count": 390.0, "length_stddev": 3.02, "has_digit_ratio": 0.0, "sequential_names": 14.0, "single_char_count": 13.0, "single_char_ratio": 0.03, "all_lowercase_ratio": 0.82, "all_uppercase_ratio": 0.04, "numeric_suffix_count": 1.0 } }, "s": [ [ 182, "10", "2" ], [ 186, "10", "15" ], [ 193, "10", "2022" ], [ 221, "10", "47" ], [ 231, "10", "75" ], [ 252, "10", "58" ], [ 257, "10", "90" ], [ 579, "10", "11" ], [ 983, "10", "2022" ], [ 1725, "10", "16" ], [ 1735, "10", "2022" ], [ 1825, "10", "7" ], [ 2079, "10", "0" ], [ 2088, "10", "1" ], [ 2334, "u8", "remember'' script-defined variables and functions across executions;\npass variables between unrelated scripts without serializin" ], [ 2335, "u8", "remember'' script-defined variables and functions across executions;\npass variables between unrelated scripts without serializin" ], [ 3207, "u8", "@pmg{},''\ncan " ], [ 3208, "u8", "@pmg{},''\ncan " ], [ 3223, "u8", "remember'' script-defined variables and functions across\nexecutions and can pass variables and functions between unrelated\nscrip" ], [ 3224, "u8", "remember'' script-defined variables and functions across\nexecutions and can pass variables and functions between unrelated\nscrip" ], [ 4786, "u8", "director's cut,'' at the web site for the persistent memory\nallocator used in @pmg{}: @*\n@center @url{http://web.eecs.umich.edu/" ], [ 4787, "u8", "director's cut,'' at the web site for the persistent memory\nallocator used in @pmg{}: @*\n@center @url{http://web.eecs.umich.edu/" ], [ 6891, "u8", "remembered'' @code{myvar} across executions of unrelated\nscripts. (If the @gwk{} executable in your search @env{$PATH} lacks\nth" ], [ 6892, "u8", "remembered'' @code{myvar} across executions of unrelated\nscripts. (If the @gwk{} executable in your search @env{$PATH} lacks\nth" ], [ 7283, "u8", "forget'' script variables, delete the heap file.\n\n@sp 2\n\nToggling persistence by @command{export}-ing and @command{unset}-ing\n" ], [ 7284, "u8", "forget'' script variables, delete the heap file.\n\n@sp 2\n\nToggling persistence by @command{export}-ing and @command{unset}-ing\n" ], [ 7411, "u8", "ambient'' envars requires care: Forgetting to @command{unset} when\nyou no longer want persistence can cause confusing bugs. For" ], [ 7412, "u8", "ambient'' envars requires care: Forgetting to @command{unset} when\nyou no longer want persistence can cause confusing bugs. For" ], [ 9294, "u8", " \u003c 74-0.txt | tr A-Z a-z \u003e sawyer.txt\n $ tr -c a-zA-Z " ], [ 9359, "u8", " \u003c 76-0.txt | tr A-Z a-z \u003e finn.txt\n@end verbatim\n\nIt's easy to count word frequencies with AWK's associative arrays.\n@pmg{} mak" ], [ 9594, "u8", "read once, analyze\nhappily ever after''):\n@verbatim\n $ truncate -s 100M twain.pma\n $ export GAWK_PERSIST_FILE=twai" ], [ 9595, "u8", "read once, analyze\nhappily ever after''):\n@verbatim\n $ truncate -s 100M twain.pma\n $ export GAWK_PERSIST_FILE=twai" ], [ 11168, "u8", "flapdoodle,'' " ], [ 11169, "u8", "flapdoodle,'' " ], [ 11184, "u8", "yellocution,'' and\n" ], [ 11185, "u8", "yellocution,'' and\n" ], [ 11205, "u8", "sockdolager'') are easy to find:\n@verbatim\n $ gawk 'BEGIN{for(w in hf) if (!(w in ts)) print w}'\n@end verbatim\n@c also: d" ], [ 11206, "u8", "sockdolager'') are easy to find:\n@verbatim\n $ gawk 'BEGIN{for(w in hf) if (!(w in ts)) print w}'\n@end verbatim\n@c also: d" ], [ 12130, "u8", "Markov'' AWK script on page 79 of Kernighan \u0026 Pike's\n@cite{The Practice of Programming} generates random text reminiscent\nof a g" ], [ 12131, "u8", "Markov'' AWK script on page 79 of Kernighan \u0026 Pike's\n@cite{The Practice of Programming} generates random text reminiscent\nof a g" ], [ 12346, "u8", "learning'' or " ], [ 12347, "u8", "learning'' or " ], [ 12362, "u8", "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical mo" ], [ 12363, "u8", "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical mo" ], [ 12817, "u8", "2" ], [ 12820, "u8", "3" ], [ 12823, "u8", "4" ], [ 12826, "u8", "5" ], [ 12829, "u8", " \u003e A.dat\n $ printf '5" ], [ 12860, "u8", "6" ], [ 12863, "u8", "7" ], [ 12866, "u8", "8" ], [ 12869, "u8", "9" ], [ 12872, "u8", " \u003e B.dat\n@end verbatim\n@noindent\nA conventional @emph{non}-persistent AWK script can compute basic\nsummary statistics:\n@verbatim" ], [ 20602, "u8", "@i{O(N)}\nexpected time'' in the lingo of asymptotic analysis. If @i{W} is the\nnumber of unique words in the input files, the si" ], [ 20603, "u8", "@i{O(N)}\nexpected time'' in the lingo of asymptotic analysis. If @i{W} is the\nnumber of unique words in the input files, the si" ], [ 23073, "u8", "paged in'' to the memory\nof the @pmg{} process. In the worst case, the heap file is not in the\nfile system's in-memory cache, s" ], [ 23074, "u8", "paged in'' to the memory\nof the @pmg{} process. In the worst case, the heap file is not in the\nfile system's in-memory cache, s" ], [ 27645, "u8", "dirty'') memory backed by files. For example, the OS may write\ndirty memory pages to the heap file periodically and/or when the" ], [ 27646, "u8", "dirty'') memory backed by files. For example, the OS may write\ndirty memory pages to the heap file periodically and/or when the" ], [ 27793, "u8", "too much'' memory is dirty. Such " ], [ 27794, "u8", "too much'' memory is dirty. Such " ], [ 27829, "u8", "eager'' writeback\ncan degrade performance noticeably and brings no benefit to @pmg{}.\nFortunately some OSes allow paging default" ], [ 27830, "u8", "eager'' writeback\ncan degrade performance noticeably and brings no benefit to @pmg{}.\nFortunately some OSes allow paging default" ], [ 28000, "u8", "lazy'' rather than eager. For Linux see the discussion\nof the @code{dirty_*} parameters at\n@url{https://www.kernel.org/doc/html" ], [ 28001, "u8", "lazy'' rather than eager. For Linux see the discussion\nof the @code{dirty_*} parameters at\n@url{https://www.kernel.org/doc/html" ], [ 29751, "u8", "hi'' plus one for the newline). The\n@command{du} utility reports that this file consumes 4@tie{}KiB of\nstorage---one block of d" ], [ 29752, "u8", "hi'' plus one for the newline). The\n@command{du} utility reports that this file consumes 4@tie{}KiB of\nstorage---one block of d" ], [ 30584, "u8", "pay as you go'' storage cost of sparse files offers both\nconvenience and control for @pmg{} users. If your file system\nsupports" ], [ 30585, "u8", "pay as you go'' storage cost of sparse files offers both\nconvenience and control for @pmg{} users. If your file system\nsupports" ], [ 32397, "u8", "pay only for what you\nneed.''@footnote{Remarkably, this guideline is widely ignored in\nsurprising ways. Certain well-known text" ], [ 32398, "u8", "pay only for what you\nneed.''@footnote{Remarkably, this guideline is widely ignored in\nsurprising ways. Certain well-known text" ], [ 32847, "u8", "persistent memory'' has sometimes been used to\ndenote novel byte-addressable non-volatile memory hardware---an\nunfortunate pract" ], [ 32848, "u8", "persistent memory'' has sometimes been used to\ndenote novel byte-addressable non-volatile memory hardware---an\nunfortunate pract" ], [ 35352, "u8", " \u003e random.dat\n@end verbatim\n@noindent\nVarying the size of random inputs can, for example, find where\nperformance " ], [ 35467, "u8", "falls off the cliff'' as @pmg{}'s memory footprint\nexceeds the capacity of DRAM and paging begins.\n\n@c TODO:\n@c virtual *machine" ], [ 35468, "u8", "falls off the cliff'' as @pmg{}'s memory footprint\nexceeds the capacity of DRAM and paging begins.\n\n@c TODO:\n@c virtual *machine" ], [ 36043, "u8", "cold start'' condition resembling the\nimmediate aftermath of a machine reboot. Accesses to ordinary files\non durable storage wi" ], [ 36044, "u8", "cold start'' condition resembling the\nimmediate aftermath of a machine reboot. Accesses to ordinary files\non durable storage wi" ], [ 39155, "u8", " " ], [ 39156, "u8", " " ], [ 39166, "u8", " # 6\nunsetenv GAWK_PERSIST_FILE # disable persistence until wanted " ], [ 39167, "u8", " # 6\nunsetenv GAWK_PERSIST_FILE # disable persistence until wanted " ], [ 39793, "u8", "' ; alias tock 'set t2 = " ], [ 39794, "u8", " ; alias tock 'set t2 = " ], [ 39831, "u8", "' # 13\nalias tyme '$PMG -v t1=$t1 -v t2=$t2 \"BEGIN{print t2-t1}\"' # 14\nalias tsync 'tick ; " ], [ 39832, "u8", " # 13\nalias tyme '$PMG -v t1=$t1 -v t2=$t2 \"BEGIN{print t2-t1}\"' # 14\nalias tsync 'tick ; s" ], [ 39999, "u8", "' # 15\nalias drop_caches 'echo 3 | sudo tee /proc/sys/vm/drop_caches' # 16\nalias sn" ], [ 40000, "u8", " # 15\nalias drop_caches 'echo 3 | sudo tee /proc/sys/vm/drop_caches' # 16\nalias snd" ], [ 40268, "u8", " # 18\necho \"run dir: $dir\" ; echo 'pwd: ' " ], [ 40269, "u8", " # 18\necho \"run dir: $dir\" ; echo 'pwd: ' " ], [ 40351, "u8", " # 19\necho 'dir content:' ; ls -l $dir |\u0026 $PMG '{print \" \" $0}' # 20\n" ], [ 40352, "u8", " # 19\necho 'dir content:' ; ls -l $dir |\u0026 $PMG '{print \" \" $0}' # 20\n" ], [ 40788, "u8", "_background_ratio\" # restore these # 23\nsudo sh -c \"echo 100 \u003e " ], [ 40878, "u8", "_ratio\" # paging params # 24\nsudo sh -c \"echo 1080000 \u003e " ], [ 40968, "u8", "_expire_centisecs\" # to defaults # 25\nsudo sh -c \"echo 1080000 \u003e " ], [ 41058, "u8", "_writeback_centisecs\" # if necessary # 26\nforeach d ( " ], [ 41133, "u8", "_background_ratio " ], [ 41157, "u8", "_ratio " ], [ 41166, "u8", " # 27\n " ], [ 41223, "u8", "_expire_centisecs " ], [ 41247, "u8", "_writeback_centisecs ) # 28\n printf \" %-38s %7d" ], [ 41332, "u8", " $d " ], [ 41344, "u8", " # 29\nend # " ], [ 41345, "u8", " # 29\nend # " ], [ 41515, "u8", " 's (around 2ms for TK)' # 31\ntick ; $PMG 'BEGIN{print \"pm-gawk? yes\"}' " ], [ 41516, "u8", " 's (around 2ms for TK)' # 31\ntick ; $PMG 'BEGIN{print \"pm-gawk? yes\"}' " ], [ 41688, "u8", " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' " ], [ 41689, "u8", " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' " ], [ 43314, "u8", ", # 51\n argv[0]); return 1; } " ], [ 43759, "u8", ", N); return 2; } # 56\n PICKCH; " ], [ 44045, "u8", "); break; } # 59\n if ('@' == c) { putchar(" ], [ 44117, "u8", "); PICKCH; } # 60\n else { putchar( c ); PICK; } " ], [ 44873, "u8", " # 68\necho \"input file: $inp\" # 69\necho 'input wc: " ], [ 44874, "u8", " # 68\necho \"input file: $inp\" # 69\necho 'input wc: " ], [ 45016, "u8", " ; echo 'input uniq: ' " ], [ 45017, "u8", " ; echo 'input uniq: ' " ], [ 45058, "u8", " # 70\n@end verbatim\n@page @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@verbatim\nsnd ###########" ], [ 45059, "u8", " # 70\n@end verbatim\n@page @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@verbatim\nsnd ###########" ], [ 45395, "u8", " ; echo '' # 73\nrm -f rwa " ], [ 45396, "u8", " ; echo '' # 73\nrm -f rwa " ], [ 45845, "u8", " ; echo '' # 78\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # " ], [ 45846, "u8", " ; echo '' # 78\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # " ], [ 46049, "u8", " # 80\n print \"output: \" n[\"foo\"]}' # 81\ntock ; echo 'T " ], [ 46206, "u8", " ; echo '' # 82\nrm -f ft " ], [ 46207, "u8", " ; echo '' # 82\nrm -f ft " ], [ 46565, "u8", " ; echo '' # 86\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # " ], [ 46566, "u8", " ; echo '' # 86\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # " ], [ 46836, "u8", " ; echo '' # 89\nrm -f heap.pma " ], [ 46837, "u8", " ; echo '' # 89\nrm -f heap.pma " ], [ 47375, "u8", " ; echo '' # 95\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # " ], [ 47376, "u8", " ; echo '' # 95\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # " ], [ 47647, "u8", " ; echo '' # 98\nunsetenv GAWK_PERSIST_FILE " ], [ 47648, "u8", " ; echo '' # 98\nunsetenv GAWK_PERSIST_FILE " ], [ 48266, "u8", " # " ], [ 48267, "u8", " # " ], [ 48298, "u8", " 'bytes xz' # 105\nend # 106\necho '' ; echo " ], [ 48299, "u8", " 'bytes xz' # 105\nend # 106\necho '' ; echo " ], [ 48447, "u8", " " ], [ 48448, "u8", " " ], [ 48458, "u8", " ; echo '' # 107\n@end verbatim\n@end smallformat\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = =" ], [ 48459, "u8", " ; echo '' # 107\n@end verbatim\n@end smallformat\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = =" ], [ 52808, "u8", "\n $ cp --reflink=always heap.pma $backup_base.pma\n $ chmod a-w $backup_base.pma\n $ sync\n $ touch $ba" ], [ 52809, "u8", "\n $ cp --reflink=always heap.pma $backup_base.pma\n $ chmod a-w $backup_base.pma\n $ sync\n $ touch $ba" ], [ 53868, "u8", "pay as you go'' storage footprints, reflink\ncopying offers " ], [ 53869, "u8", "pay as you go'' storage footprints, reflink\ncopying offers " ], [ 53929, "u8", "pay as you @emph{change}'' storage\ncosts.@footnote{The system call that implements reflink copying is\ndescribed in @command{man " ], [ 53930, "u8", "pay as you @emph{change}'' storage\ncosts.@footnote{The system call that implements reflink copying is\ndescribed in @command{man " ], [ 55536, "u8", "success indicator'' file with\nextension @file{.done} to address a nasty corner case: Power may fail\n@emph{while} a backup is bei" ], [ 55537, "u8", "success indicator'' file with\nextension @file{.done} to address a nasty corner case: Power may fail\n@emph{while} a backup is bei" ], [ 56094, "u8", "train\nas you would fight'' by testing your hardware/software stack against\nrealistic failures. For realistic power-failure test" ], [ 56095, "u8", "train\nas you would fight'' by testing your hardware/software stack against\nrealistic failures. For realistic power-failure test" ], [ 57374, "u8", "Markov'' script persistent. Volos provided and tested the advice on\ntuning OS parameters in @ref{Virtual Memory and Big Data}. " ], [ 57375, "u8", "Markov'' script persistent. Volos provided and tested the advice on\ntuning OS parameters in @ref{Virtual Memory and Big Data}. " ], [ 62510, "u8", "Ken,'' which contained a\npersistent memory allocator that resembles a simplified @code{pma}: It\npresented a @code{malloc()}-like" ], [ 62511, "u8", "Ken,'' which contained a\npersistent memory allocator that resembles a simplified @code{pma}: It\npresented a @code{malloc()}-like" ], [ 63108, "u8", "buy in'' to a larger and more\ncomplicated value proposition. Whatever its other virtues might be,\nKen isn't ideal for showcasin" ], [ 63109, "u8", "buy in'' to a larger and more\ncomplicated value proposition. Whatever its other virtues might be,\nKen isn't ideal for showcasin" ], [ 63594, "u8", "failure-atomic @code{msync()}''\n(FAMS). FAMS strengthens the semantics of ordinary standard\n@code{msync()} by guaranteeing that" ], [ 63595, "u8", "failure-atomic @code{msync()}''\n(FAMS). FAMS strengthens the semantics of ordinary standard\n@code{msync()} by guaranteeing that" ], [ 65084, "10", "2022" ], [ 65302, "u8", "s allocator, @code{pma} is not coupled to a particular" ], [ 66121, "10", "2022" ], [ 66993, "10", "2" ], [ 67035, "u8", "s unobtrusive; as you gain" ], [ 67142, "u8", "s simple in both concept and implementation, and more" ], [ 67354, "u8", "s all that I needed for my dissertation research" ], [ 67531, "u8", "s" ], [ 67565, "u8", "m curious to see what new purposes" ], [ 7943, "xor", "`AUTHOE*`EMAIL[TTK\"-;2" ], [ 8708, "xor", "`AUTHZG*`EMAIL[TTK" ], [ 9271, "xor", "`AUTHKRT2E@" ], [ 9336, "xor", "`AUTHKRT2E@" ], [ 16852, "xor", "`AUTHOA*`EMAIL[TTK" ], [ 16945, "xor", "`AUTHO@*`EMAIL[TTK" ], [ 17033, "xor", "`AUTHOB*`EMAIL[TTK" ], [ 18354, "xor", "`AUTHORR-E@" ], [ 19382, "xor", "`AUTHORR-E@" ], [ 29514, "xor", "`AUTHKRL3E@" ], [ 30094, "xor", "`AUTHKRL3E@" ], [ 31901, "xor", "`AUTHKRL3E@" ], [ 35344, "xor", ": X.HH.NgESA" ], [ 38858, "xor", "rkVTHORT/E" ], [ 40410, "xor", "`AUTHTRL3E@" ], [ 40524, "xor", "`FUP)89l\t'= =$[TPKELLY``ACM" ], [ 43274, "xor", "pkUTHO4phG" ], [ 44274, "xor", "pAKIHB_niE" ], [ 44439, "xor", "`BUT^\\x]`EMAIL[TPKELLY``ACM" ], [ 49639, "xor", "iAUTHORF/\n" ], [ 49691, "xor", "`AUTHORoh+DAILI@BET_^Y``ACM" ], [ 52813, "xor", "`AUTHKRC0E@L" ], [ 53008, "xor", "`AUTHKRL3E@" ], [ 59463, "xor", "`AUTHO)I\u0026EXO[L" ], [ 64008, "xor", "rQDGFORm9o" ] ], "v": { "source.language": "javascript", "source.imports[0]": "''", "source.imports[1]": "being", "source.imports[2]": "check", "source.imports[3]": "components", "source.imports[4]": "gawk", "source.imports[5]": "inp", "source.imports[6]": "monoliths", "source.imports[7]": "pmg", "source.imports[8]": "s.N", "source.imports[9]": "wc", "source.strings[0]": "\n $ cp --reflink=always heap.pma $backup_base.pma\n $ chmod a-w $backup_base.pma\n $ sync\n $ touch $backup_base.done\n $ chmod a-w $backup_base.done\n $ sync\n $ ls -l heap*\n -rw-rw-r--. 1 me me 4096000 Aug 6 15:53 heap.pma\n -r--r--r--. 1 me me 0 Aug 6 16:16 heap_bk_1659827771.done\n -r--r--r--. 1 me me 4096000 Aug 6 16:16 heap_bk_1659827771.pma\n@end verbatim\n@noindent\nTimestamps in backup filenames make it easy to find the most recent\ncopy if the heap file is damaged, even if last-mod metadata are\ninadvertently altered.\n\n@c TODO: sync individual files above instead of globally (?)\n@c First carefully check what sync does in both cases\n@c using strace, verify that \"sync [file]\" is correct.\n@c Also check whether non-GNU/Linux offers fine-grained\n@c sync command. Cygwin? Solaris?\n\nThe @command{cp} command's @command{--reflink} option reduces both the\nstorage footprint of the copy and the time required to make it. Just\nas sparse files provide ", "source.strings[1]": " # 6\nunsetenv GAWK_PERSIST_FILE # disable persistence until wanted # 7\nset dir = '/dev/shm' # where heap file et al. will live # 8\nset tmr = '/usr/bin/time' # can also use shell built-in \"time\" # 9\nset isz = 1073741824 # input size; 1 GiB # 10\n# set isz = 100000000 # small input for quick testing # 11\ncd $dir # tick/tock/tyme below are homebrew timer, good within ~2ms # 12\nalias tick 'set t1 = ", "source.strings[2]": " # 29\nend # 30\ntick ; tock ; echo 'timr ovrhd: ' ", "source.strings[3]": " # 27\n ", "source.strings[4]": " # 19\necho 'dir content:' ; ls -l $dir |\u0026 $PMG '{print \" \" $0}' # 20\necho 'timer: ' $tmr ; echo 'AWKLIBPATH: ' $AWKLIBPATH # 21\n@end verbatim\n@page @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@verbatim\necho 'OS params:' ; set vm = '/proc/sys/vm/dirty' # 22\nsudo sh -c \"echo 100 \u003e ", "source.strings[5]": " # 19\necho 'dir content:' ; ls -l $dir |\u0026 $PMG '{print \" \" $0}' # 20\necho 'timer: ' $tmr ; echo 'AWKLIBPATH: ' $AWKLIBPATH # 21\n@end verbatim\n@page @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@verbatim\necho 'OS params:' ; set vm = '/proc/sys/vm/dirty' # 22\nsudo sh -c \"echo 100 \u003e ${vm}_background_ratio\" # restore these # 23\nsudo sh -c \"echo 100 \u003e ${vm}_ratio\" # paging params # 24\nsudo sh -c \"echo 1080000 \u003e ${vm}_expire_centisecs\" # to defaults # 25\nsudo sh -c \"echo 1080000 \u003e ${vm}_writeback_centisecs\" # if necessary # 26\nforeach d ( ${vm}_background_ratio ${vm}_ratio \\ # 27\n ${vm}_expire_centisecs ${vm}_writeback_centisecs ) # 28\n printf \" %-38s %7d\\n\" $d ", "source.strings[6]": " # 18\necho \"run dir: $dir\" ; echo 'pwd: ' ", "source.strings[7]": " # 15\nalias drop_caches 'echo 3 | sudo tee /proc/sys/vm/drop_caches' # 16\nalias snd 'tsync; drop_caches' # 17\necho \"pm-gawk: $PMG\" ; echo 'std gawk: ' ", "source.strings[8]": " # 80\n print \"output: \" n[\"foo\"]}' # 81\ntock ; echo 'T rwarray query O(W): ' ", "source.strings[9]": " # 68\necho \"input file: $inp\" # 69\necho 'input wc: ' ", "source.has_imports": true, "source.strings[10]": " # 13\nalias tyme '$PMG -v t1=$t1 -v t2=$t2 \"BEGIN{print t2-t1}\"' # 14\nalias tsync 'tick ; sync ; tock ; echo \"sync time: \" ", "source.strings[11]": " # 70\n@end verbatim\n@page @c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@verbatim\nsnd ############################################################################ # 71\ntick ; $tmr $PMG '{n[$1]++}END{print \"output: \" n[\"foo\"]}' $inp # 72\ntock ; echo 'T naive O(N): ' ", "source.strings[12]": " 'bytes xz' # 105\nend # 106\necho '' ; echo 'end: ' ", "source.strings[13]": " 's (around 2ms for TK)' # 31\ntick ; $PMG 'BEGIN{print \"pm-gawk? yes\"}' # 32\ntock ; echo 'pmg ovrhd: ' ", "source.strings[14]": " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' # 34\necho 'input size ' $isz # 35\necho \"input file: $inp\" # 36\nset rg = rgen # spit out and compile C program to generate random inputs # 37\nrm -f $inp $rg.c $rg # 38\ncat \u003c\u003cEOF \u003e $rg.c # 39\n// generate N random words, one per line, no blank lines # 40\n// charset is e.g. 'abcdefg@' where '@' becomes newline # 41\n#include \u003cstdio.h\u003e # 42\n#include \u003cstdlib.h\u003e # 43\n#include \u003cstring.h\u003e # 44\n#define RCH c = a[rand() % L]; # 45\n#define PICK do { RCH } while (0) # 46\n#define PICKCH do { RCH } while (c == '@') # 47\n#define FP(...) fprintf(stderr, __VA_ARGS__) # 48\nint main(int argc, char *argv[]) { # 49\n if (4 != argc) { # 50\n FP(\"usage: %s charset N seed", "source.strings[15]": " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' # 34\necho 'input size ' $isz # 35\necho \"input file: $inp\" # 36\nset rg = rgen # spit out and compile C program to generate random inputs # 37\nrm -f $inp $rg.c $rg # 38\ncat \u003c\u003cEOF \u003e $rg.c # 39\n// generate N random words, one per line, no blank lines # 40\n// charset is e.g. 'abcdefg@' where '@' becomes newline # 41\n#include \u003cstdio.h\u003e # 42\n#include \u003cstdlib.h\u003e # 43\n#include \u003cstring.h\u003e # 44\n#define RCH c = a[rand() % L]; # 45\n#define PICK do { RCH } while (0) # 46\n#define PICKCH do { RCH } while (c == '@') # 47\n#define FP(...) fprintf(stderr, __VA_ARGS__) # 48\nint main(int argc, char *argv[]) { # 49\n if (4 != argc) { # 50\n FP(\"usage: %s charset N seed\\n\", # 51\n argv[0]); return 1; } # 52\n char c, *a = argv[1]; size_t L = strlen(a); # 53\n long int N = atol(argv[2]); # 54\n srand( atol(argv[3])); # 55\n if (2 \u003e N) { FP(\"N == %ld \u003c 2\\n\", N); return 2; } # 56\n PICKCH; # 57\n for (;;) { # 58\n if (2 == N) { PICKCH; putchar(c); putchar('\\n'); break; } # 59\n if ('@' == c) { putchar('\\n'); PICKCH; } # 60\n else { putchar( c ); PICK; } # 61\n if (0 \u003e= --N) break; # 62\n } # 63\n} # 64\nEOF # 65\ngcc -std=c11 -Wall -Wextra -O3 -o $rg $rg.c # 66\nset t = '@@@@@@@' ; set c = \"abcdefghijklmnopqrstuvwxyz$t$t$t$t$t$t\" # 67\ntick ; ./$rg \"$c\" $isz 47 \u003e $inp ; tock ; echo 'gen time: ' ", "source.strings[16]": " ; alias tock 'set t2 = ", "source.strings[17]": " ; echo '' # 73\nrm -f rwa # 74\nsnd ############################################################################ # 75\necho '' # 76\ntick ; $tmr $PMG -l rwarray '{n[$1]++}END{print \"writea\",writea(\"rwa\",n)}' $inp # 77\ntock ; echo 'T rwarray build O(N): ' ", "source.strings[18]": " ; echo '' # 78\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 79\ntick ; $tmr $PMG -l rwarray 'BEGIN{print \"reada\",reada(\"rwa\",n); ", "source.strings[19]": " ; echo '' # 78\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 79\ntick ; $tmr $PMG -l rwarray 'BEGIN{print \"reada\",reada(\"rwa\",n); \\ # 80\n print \"output: \" n[\"foo\"]}' # 81\ntock ; echo 'T rwarray query O(W): ' ", "source.strings[20]": " ; echo '' # 86\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 87\ntick ; $tmr $PMG '{n[$2] = $1}END{print \"output: \" n[\"foo\"]}' ft # 88\ntock ; echo 'T freqtbl query O(W): ' ", "source.strings[21]": " ; echo '' # 95\nsnd # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # 96\ntick ; $tmr $PMG 'BEGIN{print \"output: \" n[\"foo\"]}' # 97\ntock ; echo 'T pm-gawk query O(1): ' ", "source.strings[22]": " ; echo '' # 82\nrm -f ft # 83\nsnd ############################################################################ # 84\ntick ; $tmr $PMG '{n[$1]++}END{for(w in n)print n[w], w}' $inp \u003e ft # 85\ntock ; echo 'T freqtbl build O(N): ' ", "source.strings[23]": " ; echo '' # 89\nrm -f heap.pma # 90\nsnd ############################################################################ # 91\ntruncate -s 3G heap.pma # enlarge if needed # 92\nsetenv GAWK_PERSIST_FILE heap.pma # 93\ntick ; $tmr $PMG '{n[$1]++}' $inp # 94\ntock ; echo 'T pm-gawk build O(N): ' ", "source.strings[24]": " ; echo '' # 98\nunsetenv GAWK_PERSIST_FILE # 99\nsnd ############################################################################ # 100\necho 'Note: all output lines above should be identical' ; echo '' # 101\necho 'dir content:' ; ls -l $dir |\u0026 $PMG '{print \" \" $0}' # 102\necho '' ; echo 'storage footprints:' # 103\nforeach f ( rwa ft heap.pma ) # compression is very slow, so we comment it out # 104\n echo \" $f \" ", "source.strings[25]": " ; echo '' # 107\n@end verbatim\n@end smallformat\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Results\n@section Results\n\nRunning the script of @ref{Experiments} with default parameters on an\naging laptop yielded the results summarized in the table below. More\nextensive experiments, not reported here, yield qualitatively similar\nresults. Keep in mind that performance measurements are often\nsensitive to seemingly irrelevant factors. For example, the program\nthat runs first may have the advantage of a cooler CPU; later\ncontestants may start with a hot CPU and consequent clock throttling\nby a modern processor's thermal regulation apparatus. Very generally,\nperformance measurement is a notoriously tricky business. For\nscripting, whose main motive is convenience rather than speed, the\nproper role for performance measurements is to qualitatively test\nhypotheses such as those that follow from asymptotic analyses and to\nprovide a rough idea of when various approaches are practical.\n\n@sp 1\n\n@verbatim\n run time peak memory intermediate\n AWK script (sec) footprint (K) storage (K)\n\n naive O(N) 242.132 2,081,360 n/a\n rwarray build O(N) 250.288 2,846,868 156,832\n rwarray query O(W) 11.653 2,081,444\n freqtbl build O(N) 288.408 2,400,120 69,112\n freqtbl query O(W) 11.624 2,336,616\n pm-gawk build O(N) 251.946 2,079,520 2,076,608\n pm-gawk query O(1) 0.026 3,252\n@end verbatim\n\n@sp 1\n\nThe results are consistent with the asymptotic analysis of\n@ref{Constant-Time Array Access}. All four approaches require roughly\nfour minutes to read the synthetic input data. The\nna@\"{@dotless{i}}ve approach must do this every time it performs a\nquery, but the other three build an associative array to support\nqueries and separately serve such queries. The @code{freqtbl} and\n@code{rwarray} approaches build an associative array of word\nfrequencies, serialize it to an intermediate file, and then read the\nentire intermediate file prior to serving queries; the former does\nthis manually and the latter uses a @gwk{} extension. Both can serve\nqueries in roughly ten seconds, not four minutes. As we'd expect from\nthe asymptotic analysis, performing work proportional to the number of\nwords is preferable to work proportional to the size of the raw input\ncorpus: @i{O(W)} time is faster than @i{O(N)}. And as we'd expect,\n@pmg{}'s constant-time queries are faster still, by roughly two orders\nof magnitude. For the computations considered here, @pmg{} makes the\ndifference between blink-of-an-eye interactive queries and response\ntimes long enough for the user's mind to wander.\n\nWhereas @code{freqtbl} and @code{rwarray} reconstruct an associative\narray prior to accessing an individual element, @pmg{} stores a\nready-made associative array in persistent memory. That's why its\nintermediate file (the heap file) is much larger than the other two\nintermediate files, why the heap file is nearly as large as @pmg{}'s\npeak memory footprint while building the persistent array, and why its\nmemory footprint is very small while serving a query that accesses a\nsingle array element. The upside of the large heap file is @i{O(1)}\naccess instead of @i{O(W)}---a classic time-space tradeoff. If\nstorage is a scarce resource, all three intermediate files can be\ncompressed, @code{freqtbl} by a factor of roughly 2.7, @code{rwarray}\nby roughly 5.6x, and @pmg{} by roughly 11x using @command{xz}.\nCompression is CPU-intensive and slow, another time-space tradeoff.\n\n@page\n@c ==================================================================\n@node Data Integrity\n@chapter Data Integrity\n\nMishaps including power outages, OS kernel panics, scripting bugs, and\ncommand-line typos can harm your data, but precautions can mitigate\nthese risks. In scripting scenarios it usually suffices to create\nsafe backups of important files at appropriate times. As simple as\nthis sounds, care is needed to achieve genuine protection and to\nreduce the costs of backups. Here's a prudent yet frugal way to back\nup a heap file between uses:\n@verbatim\n $ backup_base=heap_bk_", "source.strings[26]": " ; echo 'input uniq: ' ", "source.strings[27]": " \u003c 74-0.txt | tr A-Z a-z \u003e sawyer.txt\n $ tr -c a-zA-Z ", "source.strings[28]": " \u003c 76-0.txt | tr A-Z a-z \u003e finn.txt\n@end verbatim\n\nIt's easy to count word frequencies with AWK's associative arrays.\n@pmg{} makes these arrays persistent, so we need not re-ingest the\nentire corpus every time we ask a new question (", "source.strings[29]": " \u003e A.dat\n $ printf '5", "source.strings[30]": " \u003e B.dat\n@end verbatim\n@noindent\nA conventional @emph{non}-persistent AWK script can compute basic\nsummary statistics:\n@verbatim\n $ cat summary_conventional.awk\n 1 == NR { min = max = $1 }\n min \u003e $1 { min = $1 }\n max \u003c $1 { max = $1 }\n { sum += $1 }\n END { print \"min: \" min \" max: \" max \" mean: \" sum/NR }\n\n $ gawk -f summary_conventional.awk A.dat B.dat\n min: 1 max: 9 mean: 5\n@end verbatim\n\nTo use @pmg{} for the same purpose, we first create a heap file for\nour AWK script variables and tell @pmg{} where to find it via the\nusual environment variable:\n@verbatim\n $ truncate -s 10M stats.pma\n $ export GAWK_PERSIST_FILE=stats.pma\n@end verbatim\n@noindent\n@pmg{} requires changing the above script to ensure that @code{min}\nand @code{max} are initialized exactly once, when the heap file is\nfirst used, and @emph{not} every time the script runs. Furthermore,\nwhereas script-defined variables such as @code{min} retain their\nvalues across @pmg{} executions, built-in AWK variables such as\n@code{NR} are reset to zero every time @pmg{} runs, so we can't use\nthem in the same way. Here's a modified script for @pmg{}:\n@verbatim\n $ cat summary_persistent.awk\n ! init { min = max = $1; init = 1 }\n min \u003e $1 { min = $1 }\n max \u003c $1 { max = $1 }\n { sum += $1; ++n }\n END { print \"min: \" min \" max: \" max \" mean: \" sum/n }\n@end verbatim\n@noindent\nNote the different pattern on the first line and the introduction of\n@code{n} to supplant @code{NR}. When used with @pmg{}, this new\ninitialization logic supports the same kind of cumulative processing\nthat we saw in the text-analysis scenario. For example, we can ingest\nour input files separately:\n@verbatim\n $ gawk -f summary_persistent.awk A.dat\n min: 1 max: 5 mean: 3\n\n $ gawk -f summary_persistent.awk B.dat\n min: 1 max: 9 mean: 5\n@end verbatim\n@noindent\nAs expected, after the second @pmg{} invocation consumes the\nsecond input file, the output matches that of the non-persistent\nscript that read both files at once.\n\nExercise: Amend the AWK scripts above to compute the median and\nmode(s) using both conventional @gwk{} and @pmg{}. (The median is the\nnumber in the middle of a sorted list; if the length of the list is\neven, average the two numbers at the middle. The modes are the values\nthat occur most frequently.)\n\n@c heaps not portable across machines, use only with same gawk executable (?)\n@c refer to gawk docs for portability constraints on heaps\n@c can use only one heap at a time\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n\nOur third and final set of examples shows that @pmg{} allows us to\nbundle both script-defined data and also user-defined @emph{functions}\nin a persistent heap that may be passed freely between unrelated AWK\nscripts.\n\n@c ADR doesn't like return in count() below\n@c TK: it was put there for a reason:\n@c $ truncate -s 10M funcs.pma\n@c $ export GAWK_PERSIST_FILE=funcs.pma\n@c $ gawk 'function count(A,t) {for(i in A)t++; return t}'\n@c $ gawk 'BEGIN { a[\"x\"] = 4; a[\"y\"] = 5; a[\"z\"] = 6 }'\n@c $ gawk 'BEGIN { print count(a) }'\n@c 3\n@c $ gawk 'BEGIN { delete a }'\n@c $ gawk 'BEGIN { print count(a) }'\n@c [!!blank line, not zero!!]\n@c $\n\nThe following shell transcript repeatedly invokes @pmg{} to create and\nthen employ a user-defined function. These separate invocations\ninvolve several different AWK scripts that communicate via the heap\nfile. Each invocation can add user-defined functions and add or\nremove data from the heap that subsequent invocations will access.\n@smallformat\n@verbatim\n $ truncate -s 10M funcs.pma\n $ export GAWK_PERSIST_FILE=funcs.pma\n $ gawk 'function count(A,t) {for(i in A)t++; return \"\"==t?0:t}'\n $ gawk 'BEGIN { a[\"x\"] = 4; a[\"y\"] = 5; a[\"z\"] = 6 }'\n $ gawk 'BEGIN { print count(a) }'\n 3\n $ gawk 'BEGIN { delete a[\"x\"] }'\n $ gawk 'BEGIN { print count(a) }'\n 2\n $ gawk 'BEGIN { delete a }'\n $ gawk 'BEGIN { print count(a) }'\n 0\n $ gawk 'BEGIN { for (i=0; i\u003c47; i++) a[i]=i }'\n $ gawk 'BEGIN { print count(a) }'\n 47\n@end verbatim\n@end smallformat\n@noindent\nThe first @pmg{} command creates user-defined function @code{count()},\nwhich returns the number of entries in a given associative array; note\nthat variable @code{t} is local to @code{count()}, not global. The\nnext @pmg{} command populates a persistent associative array with\nthree entries; not surprisingly, the @code{count()} call in the\nfollowing @pmg{} command finds these three entries. The next two\n@pmg{} commands respectively delete an array entry and print the\nreduced count, 2. The two commands after that delete the entire array\nand print a count of zero. Finally, the last two @pmg{} commands\npopulate the array with 47 entries and count them.\n\n@c I could be persuaded to leave the polynomial example as an\n@c exercise, offering to send my answer to readers upon request.\n\nThe following shell script invokes @pmg{} repeatedly to create a\ncollection of user-defined functions that perform basic operations on\nquadratic polynomials: evaluation at a given point, computing the\ndiscriminant, and using the quadratic formula to find the roots. It\nthen factorizes @math{x^2 + x - 12} into @math{(x - 3)(x + 4)}.\n@smallformat\n@verbatim\n #!/bin/sh\n rm -f poly.pma\n truncate -s 10M poly.pma\n export GAWK_PERSIST_FILE=poly.pma\n gawk 'function q(x) { return a*x^2 + b*x + c }'\n gawk 'function p(x) { return \"q(\" x \") = \" q(x) }'\n gawk 'BEGIN { print p(2) }' # evaluate \u0026 print\n gawk 'BEGIN{ a = 1; b = 1; c = -12 }' # new coefficients\n gawk 'BEGIN { print p(2) }' # eval/print again\n gawk 'function d(s) { return s * sqrt(b^2 - 4*a*c)}'\n gawk 'BEGIN{ print \"discriminant (must be \u003e=0): \" d(1)}'\n gawk 'function r(s) { return (-b + d(s))/(2*a)}'\n gawk 'BEGIN{ print \"root: \" r( 1) \" \" p(r( 1)) }'\n gawk 'BEGIN{ print \"root: \" r(-1) \" \" p(r(-1)) }'\n gawk 'function abs(n) { return n \u003e= 0 ? n : -n }'\n gawk 'function sgn(x) { return x \u003e= 0 ? \"- \" : \"+ \" } '\n gawk 'function f(s) { return \"(x \" sgn(r(s)) abs(r(s))}'\n gawk 'BEGIN{ print \"factor: \" f( 1) \")\" }'\n gawk 'BEGIN{ print \"factor: \" f(-1) \")\" }'\n rm -f poly.pma\n@end verbatim\n@end smallformat\n@noindent\n\n@page\n@c ==================================================================\n@node Performance\n@chapter Performance\n\nThis chapter explains several performance advantages that result from\nthe implementation of persistent memory in @pmg{}, shows how tuning\nthe underlying operating system sometimes improves performance, and\npresents experimental performance measurements. To make the\ndiscussion concrete, we use examples from a GNU/Linux system---GNU\nutilities atop the Linux OS---but the principles apply to other modern\noperating systems.\n\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Constant-Time Array Access\n@section Constant-Time Array Access\n\n@pmg{} preserves the efficiency of data access when data structures\nare created by one process and later re-used by a different process.\n\nConsider the associative arrays used to analyze Mark Twain's books in\n@ref{Examples}. We created arrays @code{ts[]} and @code{hf[]} by\nreading files @file{sawyer.txt} and @file{finn.txt}. If @i{N} denotes\nthe total volume of data in these files, building the associative\narrays typically requires time proportional to @i{N}, or ", "source.strings[31]": " \u003e random.dat\n@end verbatim\n@noindent\nVarying the size of random inputs can, for example, find where\nperformance ", "source.strings[32]": "' # 15\nalias drop_caches 'echo 3 | sudo tee /proc/sys/vm/drop_caches' # 16\nalias snd 'tsync; drop_caches' # 17\necho \"pm-gawk: $PMG\" ; echo 'std gawk: ' ", "source.strings[33]": "' # 13\nalias tyme '$PMG -v t1=$t1 -v t2=$t2 \"BEGIN{print t2-t1}\"' # 14\nalias tsync 'tick ; sync ; tock ; echo \"sync time: \" ", "source.strings[34]": "' ; alias tock 'set t2 = ", "source.strings[35]": "); PICKCH; } # 60\n else { putchar( c ); PICK; } # 61\n if (0 \u003e= --N) break; # 62\n } # 63\n} # 64\nEOF # 65\ngcc -std=c11 -Wall -Wextra -O3 -o $rg $rg.c # 66\nset t = '@@@@@@@' ; set c = \"abcdefghijklmnopqrstuvwxyz$t$t$t$t$t$t\" # 67\ntick ; ./$rg \"$c\" $isz 47 \u003e $inp ; tock ; echo 'gen time: ' ", "source.strings[36]": "); break; } # 59\n if ('@' == c) { putchar(", "source.strings[37]": ", # 51\n argv[0]); return 1; } # 52\n char c, *a = argv[1]; size_t L = strlen(a); # 53\n long int N = atol(argv[2]); # 54\n srand( atol(argv[3])); # 55\n if (2 \u003e N) { FP(\"N == %ld \u003c 2", "source.strings[38]": ", N); return 2; } # 56\n PICKCH; # 57\n for (;;) { # 58\n if (2 == N) { PICKCH; putchar(c); putchar(", "source.strings[39]": ": X.HH.NgESA", "source.strings[40]": "@i{O(N)}\nexpected time'' in the lingo of asymptotic analysis. If @i{W} is the\nnumber of unique words in the input files, the size of the associative\narrays will be proportional to @i{W}, or @i{O(W)}. Accessing\nindividual array elements requires only @emph{constant} or @i{O(1)}\nexpected time, not @i{O(N)} or @i{O(W)} time, because @gwk{}\nimplements arrays as hash tables.\n\n@c how much larger is N than W for the Twain texts?\n@c % wc -w sawyer.txt finn.txt\n@c 77523 sawyer.txt\n@c 120864 finn.txt\n@c 198387 total\n@c % cat sawyer.txt finn.txt | sort | uniq | wc -w\n@c 10447\n@c\n@c #words is 19x larger than #uniquewords\n@c\n@c Note that the total number of English words in existence is fixed,\n@c so as the size of a corpus increases without bound, the ratio of\n@c vocabulary size to corpus size tends toward zero.\n\nThe performance advantage of @pmg{} arises when different processes\ncreate and access associative arrays. Accessing an element of a\npersistent array created by a previous @pmg{} process, as we did\nearlier in\n@c\n@verb{|BEGIN{print ts[\"river\"], hf[\"river\"]}|},\n@c\nstill requires only @i{O(1)} time, which is asymptotically far\nsuperior to the alternatives. Na@\"{@dotless{i}}vely reconstructing\narrays by re-ingesting all raw inputs in every @gwk{} process that\naccesses the arrays would of course require @i{O(N)} time---a\nprofligate cost if the text corpus is large. Dumping arrays to files\nand re-loading them as needed would reduce the preparation time for\naccess to @i{O(W)}. That can be a substantial improvement in\npractice; @i{N} is roughly 19 times larger than @i{W} in our Twain\ncorpus. Nonetheless @i{O(W)} remains far slower than @pmg{}'s\n@i{O(1)}. As we'll see in @ref{Results}, the difference is not merely\ntheoretical.\n\nThe persistent memory implementation beneath @pmg{} enables it to\navoid work proportional to @i{N} or @i{W} when accessing an element of\na persistent associative array. Under the hood, @pmg{} stores\nscript-defined AWK variables such as associative arrays in a\npersistent heap laid out in a memory-mapped file (the heap file).\nWhen an AWK script accesses an element of an associative array, @pmg{}\nperforms a lookup on the corresponding hash table, which in turn\naccesses memory on the persistent heap. Modern operating systems\nimplement memory-mapped files in such a way that these memory accesses\ntrigger the bare minimum of data movement required: Only those parts\nof the heap file containing needed data are ", "source.strings[41]": "@pmg{},''\ncan ", "source.strings[42]": "Ken,'' which contained a\npersistent memory allocator that resembles a simplified @code{pma}: It\npresented a @code{malloc()}-like C interface and it allocated memory\nfrom a file-backed memory mapping. Experience with Ken convinced me\nthat the software abstraction of persistent memory offers important\nattractions compared with the alternatives for managing persistent\ndata (e.g., relational databases and key-value stores).\nUnfortunately, Ken's allocator is so deeply intertwined with the rest\nof Ken that it's essentially inseparable; to enjoy the benefits of\nKen's persistent memory, one must ", "source.strings[43]": "Markov'' AWK script on page 79 of Kernighan \u0026 Pike's\n@cite{The Practice of Programming} generates random text reminiscent\nof a given corpus using a simple statistical modeling technique. This\nscript consists of a ", "source.strings[44]": "Markov'' script persistent. Volos provided and tested the advice on\ntuning OS parameters in @ref{Virtual Memory and Big Data}. Stan Park\nprovided insights about virtual memory, file systems, and utilities.\n\n@c ==================================================================\n@c ==================================================================\n@c ==================================================================\n\n@node Installation\n@appendix Installation\n\n@c UPDATE below or remove this section if it's obsolete\n\n@gwk{} 5.2 featuring persistent memory is expected to be released in\nSeptember 2022; look for it at @url{http://ftp.gnu.org/gnu/gawk/}. If\n5.2 is not released yet, the master git branch is available at\n@c\n@url{http://git.savannah.gnu.org/cgit/gawk.git/snapshot/gawk-master.tar.gz}.\n@c\nUnpack the tarball, run @command{./bootstrap.sh},\n@command{./configure}, @command{make}, and @command{make check}, then\ntry some of the examples presented earlier. In the normal course of\nevents, 5.2 and later @gwk{} releases featuring @pmg{} will appear in\nthe software package management systems of major GNU/Linux distros.\nEventually @pmg{} will be available in the default @gwk{} on such\nsystems.\n\n@c ADR comments on above, \"run ./bootstrap.sh, ./configure ...\"\n@c TK replies: I haven't been doing this. Neither the README nor the\n@c INSTALL in the gawk tarball mention bootstrap.sh. If it's\n@c important, shouldn't they? The top of bootstrap.sh says its\n@c purpose is \"to avoid out-of-date issues in Git sandboxes.\"\n@c When a neurodivergent source code control system requires us to\n@c write shell scripts to work around the problems that it creates\n@c gratuitously, the universe is trying to tell us something about\n@c it.\n\n@c official gawk:\n@c http://ftp.gnu.org/gnu/gawk/ [where to look for 5.2 after release]\n@c https://www.skeeve.com/gawk/gawk-5.1.62.tar.gz [doesn't support persistent functions]\n@c http://git.savannah.gnu.org/cgit/gawk.git/snapshot/gawk-master.tar.gz [if 5.2 isn't released yet]\n@c http://git.savannah.gnu.org/cgit/gawk.git [ongoing development]\n\n@c ==================================================================\n@node Debugging\n@appendix Debugging\n\n@c TODO: ADR: @cite -\u003e @ref to info file below\n\nFor bugs unrelated to persistence, see the @gwk{} documentation,\ne.g., @cite{GAWK: Effective AWK Programming},\navailable at @url{https://www.gnu.org/software/gawk/manual/}.\n\nIf @pmg{} doesn't behave as you expect, first consider whether you're\nusing the heap file that you intend; using the wrong heap file is a\ncommon mistake. Other fertile sources of bugs for newcomers are the\nfact that a @code{BEGIN} block is executed every time @pmg{} runs,\nwhich isn't always what you really want, and the fact that built-in\nAWK variables such as @code{NR} are always reset to zero every time\nthe interpreter runs. See the discussion of initialization\nsurrounding the min/max/mean script in @ref{Examples}.\n\nIf you suspect a persistence-related bug in @pmg{}, you can set\nan environment variable that will cause its persistent heap module,\n@code{pma}, to emit more verbose error messages; for details see the\nmain @gwk{} documentation.\n@c or the @code{pma} documentation at\n@c @url{http://web.eecs.umich.edu/~tpkelly/pma/}.\n\nProgrammers: You can re-compile @gwk{} with assertions enabled, which\nwill trigger extensive integrity checks within @code{pma}. Ensure\nthat @file{pma.c} is compiled @emph{without} the @code{-DNDEBUG} flag\nwhen @command{make} builds @gwk{}. Run the resulting executable on small\ninputs, because the integrity checks can be very slow. If assertions\nfail, that likely indicates bugs somewhere in @pmg{}. Report such\nbugs to me (Terence Kelly) and also following the procedures in the\nmain @gwk{} documentation. Specify what version of @gwk{} you're\nusing, and try to provide a small and simple script that reliably\nreproduces the bug.\n\n@page\n@c ==================================================================\n@node History\n@appendix History\n\nThe @pmg{} persistence feature is based on a new persistent memory\nallocator, @code{pma}, whose design is described in\n@url{https://queue.acm.org/detail.cfm?id=3534855}. It is instructive\nto trace the evolutionary paths that led to @code{pma} and @pmg{}.\n\nI wrote many AWK scripts during my dissertation research on Web\ncaching twenty years ago, most of which processed log files from Web\nservers and Web caches. Persistent @gwk{} would have made these\nscripts smaller, faster, and easier to write, but at the time I was\nunable even to imagine that @pmg{} is possible. So I wrote a lot of\nbothersome, inefficient code that manually dumped and re-loaded AWK\nscript variables to and from text files. A decade would pass before\nmy colleagues and I began to connect the dots that make persistent\nscripting possible, and a further decade would pass before @pmg{} came\ntogether.\n\nCirca 2011 while working at HP Labs I developed a fault-tolerant\ndistributed computing platform called ", "source.strings[45]": "_background_ratio ", "source.strings[46]": "_background_ratio\" # restore these # 23\nsudo sh -c \"echo 100 \u003e ", "source.strings[47]": "_expire_centisecs ", "source.strings[48]": "_expire_centisecs\" # to defaults # 25\nsudo sh -c \"echo 1080000 \u003e ", "source.strings[49]": "_ratio ", "source.strings[50]": "_ratio\" # paging params # 24\nsudo sh -c \"echo 1080000 \u003e ", "source.strings[51]": "_writeback_centisecs ) # 28\n printf \" %-38s %7d", "source.strings[52]": "_writeback_centisecs\" # if necessary # 26\nforeach d ( ", "source.strings[53]": "`AUTHKRC0E@L", "source.strings[54]": "`AUTHKRL3E@", "source.strings[55]": "`AUTHKRT2E@", "source.strings[56]": "`AUTHO)I\u0026EXO[L", "source.strings[57]": "`AUTHO@*`EMAIL[TTK", "source.strings[58]": "`AUTHOA*`EMAIL[TTK", "source.strings[59]": "`AUTHOB*`EMAIL[TTK", "source.strings[60]": "`AUTHOE*`EMAIL[TTK\"-;2", "source.strings[61]": "`AUTHORR-E@", "source.strings[62]": "`AUTHORoh+DAILI@BET_^Y``ACM", "source.strings[63]": "`AUTHTRL3E@", "source.strings[64]": "`AUTHZG*`EMAIL[TTK", "source.strings[65]": "`BUT^\\x]`EMAIL[TPKELLY``ACM", "source.strings[66]": "`FUP)89l\t'= =$[TPKELLY``ACM", "source.strings[67]": "ambient'' envars requires care: Forgetting to @command{unset} when\nyou no longer want persistence can cause confusing bugs. Fortunately,\n@command{bash} allows you to pass envars more deliberately, on a\nper-command basis:\n@verbatim\n $ rm heap.pma # start fresh\n $ unset GAWK_PERSIST_FILE # eliminate ambient envar\n $ truncate -s 4096000 heap.pma # create new heap file\n\n $ GAWK_PERSIST_FILE=heap.pma gawk 'BEGIN{myvar = 47}'\n $ gawk 'BEGIN{myvar += 7; print myvar}'\n 7\n $ GAWK_PERSIST_FILE=heap.pma gawk 'BEGIN{myvar += 7; print myvar}'\n 54\n@end verbatim\n@noindent\nThe first @gwk{} invocation sees the special envar prepended on the\ncommand line, so it activates @pmg{}. The second @gwk{} invocation,\nhowever, does @emph{not} see the envar and therefore does not access\nthe script variable stored in the heap file. The third @gwk{}\ninvocation does see the special envar and therefore uses the script\nvariable from the heap file.\n\nWhile sometimes less error prone than ambient envars, per-command\nenvar passing as shown above is verbose and shouty. A shell alias\nsaves keystrokes and reduces visual clutter:\n@verbatim\n $ alias pm='GAWK_PERSIST_FILE=heap.pma'\n $ pm gawk 'BEGIN{print ++myvar}'\n 55\n $ pm gawk 'BEGIN{print ++myvar}'\n 56\n@end verbatim\n\n@page\n@c ==================================================================\n@node Examples\n@chapter Examples\n\nOur first example uses @pmg{} to streamline analysis of a prose\ncorpus, Mark Twain's @cite{Tom Sawyer} and @cite{Huckleberry Finn}\nfrom\n@c\n@url{https://gutenberg.org/files/74/74-0.txt}\n@c\nand\n@c\n@url{https://gutenberg.org/files/76/76-0.txt}.\n@c\nWe first convert non-alphabetic characters to newlines (so each line\nhas at most one word) and convert to lowercase:\n@verbatim\n $ tr -c a-zA-Z ", "source.strings[68]": "ambient'' envars requires care: Forgetting to @command{unset} when\nyou no longer want persistence can cause confusing bugs. Fortunately,\n@command{bash} allows you to pass envars more deliberately, on a\nper-command basis:\n@verbatim\n $ rm heap.pma # start fresh\n $ unset GAWK_PERSIST_FILE # eliminate ambient envar\n $ truncate -s 4096000 heap.pma # create new heap file\n\n $ GAWK_PERSIST_FILE=heap.pma gawk 'BEGIN{myvar = 47}'\n $ gawk 'BEGIN{myvar += 7; print myvar}'\n 7\n $ GAWK_PERSIST_FILE=heap.pma gawk 'BEGIN{myvar += 7; print myvar}'\n 54\n@end verbatim\n@noindent\nThe first @gwk{} invocation sees the special envar prepended on the\ncommand line, so it activates @pmg{}. The second @gwk{} invocation,\nhowever, does @emph{not} see the envar and therefore does not access\nthe script variable stored in the heap file. The third @gwk{}\ninvocation does see the special envar and therefore uses the script\nvariable from the heap file.\n\nWhile sometimes less error prone than ambient envars, per-command\nenvar passing as shown above is verbose and shouty. A shell alias\nsaves keystrokes and reduces visual clutter:\n@verbatim\n $ alias pm='GAWK_PERSIST_FILE=heap.pma'\n $ pm gawk 'BEGIN{print ++myvar}'\n 55\n $ pm gawk 'BEGIN{print ++myvar}'\n 56\n@end verbatim\n\n@page\n@c ==================================================================\n@node Examples\n@chapter Examples\n\nOur first example uses @pmg{} to streamline analysis of a prose\ncorpus, Mark Twain's @cite{Tom Sawyer} and @cite{Huckleberry Finn}\nfrom\n@c\n@url{https://gutenberg.org/files/74/74-0.txt}\n@c\nand\n@c\n@url{https://gutenberg.org/files/76/76-0.txt}.\n@c\nWe first convert non-alphabetic characters to newlines (so each line\nhas at most one word) and convert to lowercase:\n@verbatim\n $ tr -c a-zA-Z '\\n' \u003c 74-0.txt | tr A-Z a-z \u003e sawyer.txt\n $ tr -c a-zA-Z '\\n' \u003c 76-0.txt | tr A-Z a-z \u003e finn.txt\n@end verbatim\n\nIt's easy to count word frequencies with AWK's associative arrays.\n@pmg{} makes these arrays persistent, so we need not re-ingest the\nentire corpus every time we ask a new question (", "source.strings[69]": "buy in'' to a larger and more\ncomplicated value proposition. Whatever its other virtues might be,\nKen isn't ideal for showcasing the benefits of persistent memory in\nisolation.\n\nAnother entangled aspect of Ken was a crash-tolerance mechanism that,\nin retrospect, can be viewed as a user-space implementation of\nfailure-atomic @code{msync()}. The first post-Ken disentanglement\neffort isolated the crash-tolerance mechanism and implemented it in\nthe Linux kernel, calling the result ", "source.strings[70]": "cold start'' condition resembling the\nimmediate aftermath of a machine reboot. Accesses to ordinary files\non durable storage will then be served from the storage devices, not\nfrom cache. Read about @command{sync} and\n@file{/proc/sys/vm/drop_caches} at @*\n@c\n@url{https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html}.\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Experiments\n@section Experiments\n\nThe C-shell (@command{csh}) script listed below illustrates concepts\nand implements tips presented in this chapter. It produced the\nresults discussed in @ref{Results} in roughly 20 minutes on an aging\nlaptop. You can cut and paste the code listing below into a file, or\ndownload it from @url{http://web.eecs.umich.edu/~tpkelly/pma/}.\n\n@c TODO: post script to Web site when finalized \n\nThe script measures the performance of four different ways to support\nword frequency queries over a text corpus: The na@\"{@dotless{i}}ve\napproach of reading the corpus into an associative array for every\nquery; manually dumping a text representation of the word-frequency\ntable and manually loading it prior to a query; using @gwk{}'s\n@code{rwarray} extension to dump and load an associative array; and\nusing @pmg{} to maintain a persistent associative array.\n\nComments at the top explain prerequisites. Lines 8--10 set input\nparameters: the directory where tests are run and where files\nincluding the heap file are held, the off-the-shelf timer used to\nmeasure run times and other performance characteristics such as peak\nmemory usage, and the size of the input. The default input size\nresults in @pmg{} memory footprints under 3 GiB, which is large enough\nfor interesting results and small enough to fit in DRAM and avoid\npaging on today's computers. Lines 13--14 define a homebrew timer.\n\nTwo sections of the script are relevant if the default run directory\nis changed from @file{/dev/shm/} to a directory in a conventional\nstorage-backed file system: Lines 15--17 define the mechanism for\nclearing file data cached in DRAM; lines 23--30 set Linux kernel\nparameters to discourage eager paging.\n\nLines 37--70 spit out, compile, and run a little C program to generate\na random text corpus. This program is fast, flexible, and\ndeterministic, generating the same random output given the same\nparameters.\n\nLines 71--100 run the four different AWK approaches on the same random\ninput, reporting separately the time to build and to query the\nassociative array containing word frequencies.\n\n@c ADR suggests making shell script available on web site\n@c TK will do eventually\n\n@sp 1\n\n@c first line of C-shell script can't contain line-number comment\n\n@smallformat\n@verbatim\n#!/bin/csh -f\n# Set PMG envar to path of pm-gawk executable and AWKLIBPATH # 2\n# to find rwarray.so # 3\n# Requires \"sudo\" to work; consider this for /etc/sudoers file: # 4\n# Defaults:youruserid !authenticate # 5\necho 'begin: ' ", "source.strings[71]": "director's cut,'' at the web site for the persistent memory\nallocator used in @pmg{}: @*\n@center @url{http://web.eecs.umich.edu/~tpkelly/pma/}\n\n@sp 1\n\n@noindent\nTwo publications describe the persistent memory allocator and early\nexperiences with a @pmg{} prototype based on a fork of the official\n@gwk{} sources:\n@itemize\n@item @url{https://queue.acm.org/detail.cfm?id=3534855}\n@item @url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}\n@end itemize\n\n@sp 1\n\n@noindent\nFeel free to send me questions, suggestions, and experiences: @*\n\n@noindent\n@w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @email{tpkelly@@eecs.umich.edu} @w{ } (preferred) @*\n@w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @email{tpkelly@@cs.princeton.edu} @*\n@w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @w{ } @email{tpkelly@@acm.org}\n\n@page\n@c ==================================================================\n@node Quick Start\n@chapter Quick Start\n\n@c example heaps are larger than strictly necessary so that readers\n@c who use them more extensively are less likely to exhaust memory\n\nHere's @pmg{} in action at the @command{bash} shell prompt (@samp{$}):\n@verbatim\n $ truncate -s 4096000 heap.pma\n $ export GAWK_PERSIST_FILE=heap.pma\n $ gawk 'BEGIN{myvar = 47}'\n $ gawk 'BEGIN{myvar += 7; print myvar}'\n 54\n@end verbatim\n@noindent\nFirst, @command{truncate} creates an empty (all-zero-bytes) @dfn{heap\nfile} where @pmg{} will store script variables; its size is a multiple\nof the system page size (4@tie{}KiB). Next, @command{export} sets an\nenvironment variable that enables @pmg{} to find the heap file; if\n@gwk{} does @emph{not} see this envar, persistence is not activated.\nThe third command runs a one-line AWK script that initializes variable\n@code{myvar}, which will reside in the heap file and thereby outlive\nthe interpreter process that initialized it. Finally, the fourth\ncommand invokes @pmg{} on a @emph{different} one-line script that\nincrements and prints @code{myvar}. The output shows that @pmg{} has\nindeed ", "source.strings[72]": "dirty'') memory backed by files. For example, the OS may write\ndirty memory pages to the heap file periodically and/or when the OS\nbelieves that ", "source.strings[73]": "eager'' writeback\ncan degrade performance noticeably and brings no benefit to @pmg{}.\nFortunately some OSes allow paging defaults to be over-ridden so that\nwriteback is ", "source.strings[74]": "failure-atomic @code{msync()}''\n(FAMS). FAMS strengthens the semantics of ordinary standard\n@code{msync()} by guaranteeing that the durable state of a\nmemory-mapped file always reflects the most recent successful\n@code{msync()} call, even in the presence of failures such as power\noutages and OS or application crashes. The original Linux kernel FAMS\nprototype is described in a paper by Park et al. in EuroSys 2013. My\ncolleagues and I subsequently implemented FAMS in several different\nways including in file systems (FAST 2015) and user-space libraries.\nMy most recent FAMS implementation, which leverages the reflink\ncopying feature described elsewhere in this manual, is now the\nfoundation of a new crash-tolerance feature in the venerable and\nubiquitous GNU @command{dbm} (@command{gdbm}) database\n(@url{https://queue.acm.org/detail.cfm?id=3487353}).\n\nIn recent years my attention has returned to the advantages of\npersistent memory programming, lately a hot topic thanks to the\ncommercial availability of byte-addressable non-volatile memory\nhardware (which, confusingly, is nowadays marketed as ", "source.strings[75]": "falls off the cliff'' as @pmg{}'s memory footprint\nexceeds the capacity of DRAM and paging begins.\n\n@c TODO:\n@c virtual *machines* / cloud machines can make performance hard to measure repeatably\n@c here we assume good old fashioned OS install directly on \"bare metal\"\n\nExperiments require careful methodology, especially when the heap file\nis in a storage-backed file system. Overlooking the file system's\nDRAM cache can easily misguide interpretation of results and foil\nrepeatability. Fortunately Linux allows us to invalidate the file\nsystem cache and thus mimic a ", "source.strings[76]": "flapdoodle,'' ", "source.strings[77]": "forget'' script variables, delete the heap file.\n\n@sp 2\n\nToggling persistence by @command{export}-ing and @command{unset}-ing\n", "source.strings[78]": "hi'' plus one for the newline). The\n@command{du} utility reports that this file consumes 4@tie{}KiB of\nstorage---one block of disk, as small as a non-sparse file's storage\nfootprint can be. Now let's use @command{truncate} to create a\nlogically enormous sparse file and check its physical size:\n@verbatim\n $ truncate -s 1T sparse\n $ ls -l sparse\n -rw-rw-r--. 1 me me 1099511627776 Aug 5 22:33 sparse\n $ du -h sparse\n 0 sparse\n@end verbatim\n@noindent\nWhereas @command{ls} reports the logical file size that we expect (one\nTiB or 2 raised to the power 40 bytes), @command{du} reveals that the\nfile occupies no storage whatsoever. The file system will allocate\nphysical storage resources beneath this file as data is written to it;\nreading unwritten regions of the file yields zeros.\n\nThe ", "source.strings[79]": "iAUTHORF/\n", "source.strings[80]": "lazy'' rather than eager. For Linux see the discussion\nof the @code{dirty_*} parameters at\n@url{https://www.kernel.org/doc/html/latest/admin-guide/sysctl/vm.html}.\nChanging these parameters can prevent wasteful eager\npaging:@footnote{The @command{tee} rigmarole is explained at\n@url{https://askubuntu.com/questions/1098059/which-is-the-right-way-to-drop-caches-in-lubuntu}.}\n@verbatim\n $ echo 100 | sudo tee /proc/sys/vm/dirty_background_ratio\n $ echo 100 | sudo tee /proc/sys/vm/dirty_ratio\n $ echo 300000 | sudo tee /proc/sys/vm/dirty_expire_centisecs\n $ echo 50000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs\n@end verbatim\n@noindent\nTuning paging parameters can help non-persistent @gwk{} as well as\n@pmg{}. [Disclaimer: OS tuning is an occult art, and your mileage may\nvary.]\n\n@c sudo sh -c 'echo 3 \u003e /proc/sys/vm/drop_caches'\n@c\n@c sudo doesn't convey root privileges to the redirection '\u003e' when calling from cmd line\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Sparse Heap Files\n@section Sparse Heap Files\n\nTo be frugal with storage resources, @pmg{}'s heap file should be\ncreated as a @dfn{sparse file}: a file whose logical size is larger\nthan its storage resource footprint. Modern file systems support\nsparse files, which are easy to create using the @command{truncate}\ntool shown in our examples.\n\nLet's first create a conventional @emph{non}-sparse file using\n@command{echo}:\n@verbatim\n $ echo hi \u003e dense\n $ ls -l dense\n -rw-rw-r--. 1 me me 3 Aug 5 23:08 dense\n $ du -h dense\n 4.0K dense\n@end verbatim\n@noindent\nThe @command{ls} utility reports that file @file{dense} is three bytes\nlong (two for the letters in ", "source.strings[81]": "learning'' or ", "source.strings[82]": "m curious to see what new purposes", "source.strings[83]": "pAKIHB_niE", "source.strings[84]": "paged in'' to the memory\nof the @pmg{} process. In the worst case, the heap file is not in the\nfile system's in-memory cache, so the required pages must be faulted\ninto memory from storage. Our asymptotic analysis of efficiency\napplies regardless of whether the heap file is cached or not. The\nentire heap file is @emph{not} accessed merely to access an element of\na persistent associative array.\n\nPersistent memory thus enables @pmg{} to offer the flexibility of\nde-coupling data ingestion from analytic queries without the fuss and\noverhead of serializing and loading data structures and without\nsacrificing constant-time access to the associative arrays that make\nAWK scripting convenient and productive.\n\n@c Further details on @pmg{}'s persistent heap are available in\n@c @url{https://queue.acm.org/detail.cfm?id=3534855}\n@c\n@c and [excessively long NVMW URL below]\n@c\n@c @url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}.\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Virtual Memory and Big Data\n@section Virtual Memory and Big Data\n\nSmall data sets seldom spoil the delights of AWK by causing\nperformance troubles, with or without persistence. As the size of the\n@gwk{} interpreter's internal data structures approaches the capacity\nof physical memory, however, acceptable performance requires\nunderstanding modern operating systems and sometimes tuning them.\nFortunately @pmg{} offers new degrees of control for\nperformance-conscious users tackling large data sets. A terse\nmnemonic captures the basic principle: Precluding paging promotes peak\nperformance and prevents perplexity.\n\nModern operating systems feature @dfn{virtual memory} that strives to\nappear both larger than installed DRAM (which is small) and faster\nthan installed storage devices (which are slow). As a program's\nmemory footprint approaches the capacity of DRAM, the virtual memory\nsystem transparently @dfn{pages} (moves) the program's data between\nDRAM and a @dfn{swap area} on a storage device. Paging can degrade\nperformance mildly or severely, depending on the program's memory\naccess patterns. Random accesses to large data structures can trigger\nexcessive paging and dramatic slowdown. Unfortunately, the hash\ntables beneath AWK's signature associative arrays inherently require\nrandom memory accesses, so large associative arrays can be\nproblematic.\n\n@c ADR comments regarding below, \"SSDs alleviate much of the\n@c performance problem of hard disks vs. RAM disks.\"\n@c\n@c TK replies: When a significant amount of paging to *any*\n@c conventional block storage device starts, speed plummets by orders\n@c of magnitude. I'd wager that the difference between paging vs. not\n@c is larger than the difference between paging to SSD vs. HDD. So\n@c while SSDs are faster than HDDs, when paging begins they won't\n@c usually make the difference between acceptable vs. unacceptable\n@c performance.\n@c\n@c If you decide to try to find out for yourself, note that on many\n@c Linux systems the notorious OOM killer terminates a process well\n@c before its anonymous-memory footprint reaches the capacity of DRAM,\n@c so it's difficult even to provoke the phenomenon of interest. And\n@c paging anonymous memory to swap can be managed differently than\n@c paging involving a file-backed memory mapping.\n@c\n@c Bottom line: For the large majority of purposes, gawk users\n@c would be wise to avoid paging entirely, regardless of whether\n@c the pm-gawk feature is used.\n\nPersistence changes the rules in our favor: The OS pages data to\n@pmg{}'s @emph{heap file} instead of the swap area. This won't help\nperformance much if the heap file resides in a conventional\nstorage-backed file system. On Unix-like systems, however, we may\nplace the heap file in a DRAM-backed file system such as\n@file{/dev/shm/}, which entirely prevents paging to slow storage\ndevices. Temporarily placing the heap file in such a file system is a\nreasonable expedient, with two caveats: First, keep in mind that\nDRAM-backed file systems perish when the machine reboots or crashes,\nso you must copy the heap file to a conventional storage-backed file\nsystem when your computation is done. Second, @pmg{}'s memory\nfootprint can't exceed available DRAM if you place the heap file in a\nDRAM-backed file system.\n\nTuning OS paging parameters may improve performance if you decide to\nrun @pmg{} with a heap file in a conventional storage-backed file\nsystem. Some OSes have unhelpful default habits regarding modified\n(", "source.strings[85]": "pay as you @emph{change}'' storage\ncosts.@footnote{The system call that implements reflink copying is\ndescribed in @command{man ioctl_ficlone}.} A reflink copy shares\nstorage with the original file. The file system ensures that\nsubsequent changes to either file don't affect the other. Reflink\ncopying is not available on all file systems; XFS, BtrFS, and OCFS2\ncurrently support it.@footnote{The @command{--reflink} option creates\ncopies as sparse as the original. If reflink copying is not\navailable, @command{--sparse=always} should be used.} Fortunately you\ncan install, say, an XFS file system @emph{inside an ordinary file} on\nsome other file system, such as @code{ext4}.@footnote{See\n@url{https://www.usenix.org/system/files/login/articles/login_winter19_08_kelly.pdf}.}\n\n@c The @command{filefrag} utility reveals how the storage allocated to\n@c the two files changes if they diverge.\n\nAfter creating a backup copy of the heap file we use @command{sync} to\nforce it down to durable media. Otherwise the copy may reside only in\nvolatile DRAM memory---the file system's cache---where an OS crash or\npower failure could corrupt it.@footnote{On some OSes @command{sync}\nprovides very weak guarantees, but on Linux @command{sync} returns\nonly after all file system data are flushed down to durable storage.\nIf your @command{sync} is unreliable, write a little C program that\ncalls @code{fsync()} to flush a file. To be safe, also call\n@code{fsync()} on every enclosing directory on the file's\n@code{realpath()} up to the root.} After @command{sync}-ing the\nbackup we create and @command{sync} a ", "source.strings[86]": "pay as you go'' storage cost of sparse files offers both\nconvenience and control for @pmg{} users. If your file system\nsupports sparse files, go ahead and create lavishly capacious heap\nfiles for @pmg{}. Their logical size costs nothing and persistent\nmemory allocation within @pmg{} won't fail until physical storage\nresources beneath the file system are exhausted. But if instead you\nwant to @emph{prevent} a heap file from consuming too much storage,\nsimply set its initial size to whatever bound you wish to enforce; it\nwon't eat more disk than that. Copying sparse files with GNU\n@command{cp} creates sparse copies by default.\n\nFile-system encryption can preclude sparse files: If the cleartext of\na byte offset range within a file is all zero bytes, the corresponding\nciphertext probably shouldn't be all zeros! Encrypting at the storage\nlayer instead of the file system layer may offer acceptable security\nwhile still permitting file systems to implement sparse files.\n\nSometimes you might prefer a dense heap file backed by pre-allocated\nstorage resources, for example to increase the likelihood that\n@pmg{}'s internal memory allocation will succeed until the persistent\nheap occupies the entire heap file. The @command{fallocate} utility\nwill do the trick:\n@verbatim\n $ fallocate -l 1M mibi\n $ ls -l mibi\n -rw-rw-r--. 1 me me 1048576 Aug 5 23:18 mibi\n $ du -h mibi\n 1.0M mibi\n@end verbatim\n@noindent\nWe get the MiB we asked for, both logically and physically.\n\n@c UPDATE: search for username in \"ls\" examples\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Persistence versus Durability\n@section Persistence versus Durability\n\nArguably the most important general guideline for good performance in\ncomputer systems is, ", "source.strings[87]": "pay as you go'' storage footprints, reflink\ncopying offers ", "source.strings[88]": "pay only for what you\nneed.''@footnote{Remarkably, this guideline is widely ignored in\nsurprising ways. Certain well-known textbook algorithms continue to\ngrind away fruitlessly long after having computed all of their\noutput. @*\n@c\nSee @url{https://queue.acm.org/detail.cfm?id=3424304}.}\n@c\nTo apply this maxim to @pmg{} we must distinguish two concepts that\nare frequently conflated: persistence and durability.@footnote{In\nrecent years the term ", "source.strings[89]": "persistent memory'' has sometimes been used to\ndenote novel byte-addressable non-volatile memory hardware---an\nunfortunate practice that contradicts sensible long-standing\nconvention and causes needless confusion. NVM provides durability.\nPersistent memory is a software abstraction that doesn't require NVM.\nSee @url{https://queue.acm.org/detail.cfm?id=3358957}.} (A third\nlogically distinct concept is the subject of @ref{Data Integrity}.)\n\n@dfn{Persistent} data outlive the processes that access them, but\ndon't necessarily last forever. For example, as explained in\n@command{man mq_overview}, message queues are persistent because they\nexist until the system shuts down. @dfn{Durable} data reside on a\nphysical medium that retains its contents even without continuously\nsupplied power. For example, hard disk drives and solid state drives\nstore durable data. Confusion arises because persistence and\ndurability are often correlated: Data in ordinary file systems backed\nby HDDs or SSDs are typically both persistent and durable.\nFamiliarity with @code{fsync()} and @code{msync()} might lead us to\nbelieve that durability is a subset of persistence, but in fact the\ntwo characteristics are orthogonal: Data in the swap area are durable\nbut not persistent; data in DRAM-backed file systems such as\n@file{/dev/shm/} are persistent but not durable.\n\nDurability often costs more than persistence, so performance-conscious\n@pmg{} users pay the added premium for durability only when\npersistence alone is not sufficient. Two ways to avoid unwanted\ndurability overheads were discussed in @ref{Virtual Memory and Big\nData}: Place @pmg{}'s heap file in a DRAM-backed file system, or\ndisable eager writeback to the heap file. Expedients such as these\nenable you to remove durability overheads from the critical path of\nmulti-stage data analyses even when you want heap files to eventually\nbe durable: Allow @pmg{} to run at full speed with persistence alone;\nforce the heap file to durability (using the @command{cp} and\n@command{sync} utilities as necessary) after output has been emitted\nto the next stage of the analysis and the @pmg{} process using the\nheap has terminated.\n\nExperimenting with synthetic data builds intuition for how persistence\nand durability affect performance. You can write a little AWK or C\nprogram to generate a stream of random text, or just cobble together a\nquick and dirty generator on the command line:\n@verbatim\n $ openssl rand --base64 1000000 | tr -c a-zA-Z ", "source.strings[90]": "persistent memory'' has sometimes been used to\ndenote novel byte-addressable non-volatile memory hardware---an\nunfortunate practice that contradicts sensible long-standing\nconvention and causes needless confusion. NVM provides durability.\nPersistent memory is a software abstraction that doesn't require NVM.\nSee @url{https://queue.acm.org/detail.cfm?id=3358957}.} (A third\nlogically distinct concept is the subject of @ref{Data Integrity}.)\n\n@dfn{Persistent} data outlive the processes that access them, but\ndon't necessarily last forever. For example, as explained in\n@command{man mq_overview}, message queues are persistent because they\nexist until the system shuts down. @dfn{Durable} data reside on a\nphysical medium that retains its contents even without continuously\nsupplied power. For example, hard disk drives and solid state drives\nstore durable data. Confusion arises because persistence and\ndurability are often correlated: Data in ordinary file systems backed\nby HDDs or SSDs are typically both persistent and durable.\nFamiliarity with @code{fsync()} and @code{msync()} might lead us to\nbelieve that durability is a subset of persistence, but in fact the\ntwo characteristics are orthogonal: Data in the swap area are durable\nbut not persistent; data in DRAM-backed file systems such as\n@file{/dev/shm/} are persistent but not durable.\n\nDurability often costs more than persistence, so performance-conscious\n@pmg{} users pay the added premium for durability only when\npersistence alone is not sufficient. Two ways to avoid unwanted\ndurability overheads were discussed in @ref{Virtual Memory and Big\nData}: Place @pmg{}'s heap file in a DRAM-backed file system, or\ndisable eager writeback to the heap file. Expedients such as these\nenable you to remove durability overheads from the critical path of\nmulti-stage data analyses even when you want heap files to eventually\nbe durable: Allow @pmg{} to run at full speed with persistence alone;\nforce the heap file to durability (using the @command{cp} and\n@command{sync} utilities as necessary) after output has been emitted\nto the next stage of the analysis and the @pmg{} process using the\nheap has terminated.\n\nExperimenting with synthetic data builds intuition for how persistence\nand durability affect performance. You can write a little AWK or C\nprogram to generate a stream of random text, or just cobble together a\nquick and dirty generator on the command line:\n@verbatim\n $ openssl rand --base64 1000000 | tr -c a-zA-Z '\\n' \u003e random.dat\n@end verbatim\n@noindent\nVarying the size of random inputs can, for example, find where\nperformance ", "source.strings[91]": "pkUTHO4phG", "source.strings[92]": "rQDGFORm9o", "source.strings[93]": "read once, analyze\nhappily ever after''):\n@verbatim\n $ truncate -s 100M twain.pma\n $ export GAWK_PERSIST_FILE=twain.pma\n $ gawk '{ts[$1]++}' sawyer.txt # ingest\n $ gawk 'BEGIN{print ts[\"work\"], ts[\"play\"]}' # query\n 92 11\n $ gawk 'BEGIN{print ts[\"necktie\"], ts[\"knife\"]}' # query\n 2 27\n@end verbatim\n@noindent\nThe @command{truncate} command above creates a heap file large enough\nto store all of the data it must eventually contain, with plenty of\nroom to spare. (As we'll see in @ref{Sparse Heap Files}, this isn't\nwasteful.) The @command{export} command ensures that subsequent\n@gwk{} invocations activate @pmg{}. The first @pmg{} command stores\n@cite{Tom Sawyer}'s word frequencies in associative array @code{ts[]}.\nBecause this array is persistent, subsequent @pmg{} commands can\naccess it without having to parse the input file again.\n\nExpanding our analysis to encompass a second book is easy. Let's\npopulate a new associative array @code{hf[]} with the word frequencies\nin @cite{Huckleberry Finn}:\n@verbatim\n $ gawk '{hf[$1]++}' finn.txt\n@end verbatim\n@noindent\nNow we can freely intermix accesses to both books' data conveniently\nand efficiently, without the overhead and coding fuss of repeated\ninput parsing:\n@verbatim\n $ gawk 'BEGIN{print ts[\"river\"], hf[\"river\"]}'\n 26 142\n@end verbatim\n\nBy making AWK more interactive, @pmg{} invites casual conversations\nwith data. If we're curious what words in @cite{Finn} are absent from\n@cite{Sawyer}, answers (including ", "source.strings[94]": "remember'' script-defined variables and functions across\nexecutions and can pass variables and functions between unrelated\nscripts without serializing/parsing text files---all with near-zero\nfuss. @pmg{} does @emph{not} require non-volatile memory hardware nor\nany other exotic infrastructure; it runs on the ordinary conventional\ncomputers and operating systems that most of us have been using for\ndecades.\n\n@sp 1\n\n@c TODO: ADR: hyperlinks to info page below\n\n@noindent\nThe main @gwk{} documentation@footnote{See\n@url{https://www.gnu.org/software/gawk/manual/} @w{ } and @w{ }\n@code{man gawk} @w{ } and @w{ } @code{info gawk}.} covers the basics\nof the new persistence feature. This supplementary manual provides\nadditional detail, tutorial examples, and a peek under the hood of\n@pmg{}. If you're familiar with @gwk{} and Unix-like environments,\ndive straight in: @*\n\n@itemize @c @w{}\n@item @ref{Quick Start} hits the ground running with a few keystrokes.\n@item @ref{Examples} shows how @pmg{} streamlines typical AWK scripting.\n@item @ref{Performance} covers asymptotic efficiency, OS tuning, and more.\n@item @ref{Data Integrity} explains how to protect data from mishaps.\n@item @ref{Acknowledgments} thanks those who made @pmg{} happen.\n@item @ref{Installation} shows where obtain @pmg{}.\n@item @ref{Debugging} explains how to handle suspected bugs.\n@item @ref{History} traces @pmg{}'s persistence technology.\n@end itemize\n\n@c UPDATE: revise above when content finalized\n\n@sp 1\n\n@noindent\nYou can find the latest version of this manual, and also the\n", "source.strings[95]": "remember'' script-defined variables and functions across executions;\npass variables between unrelated scripts without serializing/parsing\ntext files; and handle data sets larger than available memory plus\nswap. This supplementary manual provides an in-depth look at\npersistent-memory @gwk{}.\n\n@insertcopying\n@end ifnotdocbook\n@end ifnotxml\n@end ifnottex\n\n@menu\n* Introduction::\n* Quick Start::\n* Examples::\n* Performance::\n* Data Integrity::\n* Acknowledgments::\n* Installation::\n* Debugging::\n* History::\n@end menu\n\n@c ==================================================================\n@node Introduction\n@chapter Introduction\n\n@sp 1\n\n@c UPDATE below after official release\nGNU AWK (@gwk{}) 5.2, expected in September 2022, introduces a new\n@emph{persistent memory} feature that makes AWK scripting easier and\nsometimes improves performance. The new feature, called ", "source.strings[96]": "remembered'' @code{myvar} across executions of unrelated\nscripts. (If the @gwk{} executable in your search @env{$PATH} lacks\nthe persistence feature, the output in the above example will be\n@samp{7} instead of @samp{54}. @xref{Installation}.) To disable\npersistence until you want it again, prevent @gwk{} from finding the\nheap file via @command{unset GAWK_PERSIST_FILE}. To permanently\n", "source.strings[97]": "rkVTHORT/E", "source.strings[98]": "s all that I needed for my dissertation research", "source.strings[99]": "s allocator, @code{pma} is not coupled to a particular", "source.strings[100]": "s simple in both concept and implementation, and more", "source.strings[101]": "s unobtrusive; as you gain", "source.strings[102]": "sockdolager'') are easy to find:\n@verbatim\n $ gawk 'BEGIN{for(w in hf) if (!(w in ts)) print w}'\n@end verbatim\n@c also: doxolojer meedyevil ridicklous dingnation gumption cavortings\n@c phrenology [words about slavery] shakespeare camelopard ope\n@c mesmerism sapheads disremember consekens prevarication\n@c missionaryin cannibal nebokoodneezer sentimentering palavering\n\nRumors of Twain's death may be exaggerated. If he publishes new books\nin the future, it will be easy to incorporate them into our analysis\nincrementally. The performance benefits of incremental processing for\ncommon AWK chores such as log file analysis are discussed in\n@url{https://queue.acm.org/detail.cfm?id=3534855} and the companion\npaper cited therein, and below in @ref{Performance}.\n@c @url{http://nvmw.ucsd.edu/nvmw2022-program/nvmw2022-data/nvmw2022-paper35-final_version_your_extended_abstract.pdf}.\n\nExercise: The ", "source.strings[103]": "success indicator'' file with\nextension @file{.done} to address a nasty corner case: Power may fail\n@emph{while} a backup is being copied from the primary heap file,\nleaving either file, or both, corrupt on storage---a particularly\nworrisome possibility for jobs that run unattended. Upon reboot, each\n@file{.done} file attests that the corresponding backup succeeded,\nmaking it easy to identify the most recent successful backup.\n\n@c TODO: \".done\" -\u003e \".ready\" so ls alphabetizes nicely (?)\n\nFinally, if you're serious about tolerating failures you must ", "source.strings[104]": "too much'' memory is dirty. Such ", "source.strings[105]": "train\nas you would fight'' by testing your hardware/software stack against\nrealistic failures. For realistic power-failure testing, see\n@c @url{https://cacm.acm.org/magazines/2020/9/246938-is-persistent-memory-persistent/fulltext}\n@c\n@c and\n@url{https://queue.acm.org/detail.cfm?id=3400902}.\n\n@page\n@c ==================================================================\n@node Acknowledgments\n@chapter Acknowledgments\n\n@c UPDATE: make sure nobody is overlooked\n\nHaris Volos, Zi Fan Tan, and Jianan Li developed a persistent @gwk{}\nprototype based on a fork of the @gwk{} source. Advice from @gwk{}\nmaintainer Arnold Robbins to me, which I forwarded to them, proved\nvery helpful. Robbins moreover implemented, documented, and tested\n@pmg{} for the official version of @gwk{}; along the way he suggested\nnumerous improvements for the @code{pma} memory allocator beneath\n@pmg{}. Corinna Vinschen suggested other improvements to @code{pma}\nand tested @pmg{} on Cygwin. Nelson H.@: F.@: Beebe provided access\nto Solaris machines for testing. Robbins, Volos, Li, Tan, Jon\nBentley, and Hans Boehm reviewed drafts of this user manual and\nprovided useful feedback. Bentley suggested the min/max/mean example\nin @ref{Examples}, and also the exercise of making Kernighan \u0026 Pike's\n", "source.strings[106]": "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical model to incrementally ingest additions to\nthe input corpus.\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n\nOur second example considers another domain that plays to AWK's\nstrengths, data analysis. For simplicity we'll create two small input\nfiles of numeric data.\n@verbatim\n $ printf '1", "source.strings[107]": "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical model to incrementally ingest additions to\nthe input corpus.\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n\nOur second example considers another domain that plays to AWK's\nstrengths, data analysis. For simplicity we'll create two small input\nfiles of numeric data.\n@verbatim\n $ printf '1\\n2\\n3\\n4\\n5\\n' \u003e A.dat\n $ printf '5\\n6\\n7\\n8\\n9\\n' \u003e B.dat\n@end verbatim\n@noindent\nA conventional @emph{non}-persistent AWK script can compute basic\nsummary statistics:\n@verbatim\n $ cat summary_conventional.awk\n 1 == NR { min = max = $1 }\n min \u003e $1 { min = $1 }\n max \u003c $1 { max = $1 }\n { sum += $1 }\n END { print \"min: \" min \" max: \" max \" mean: \" sum/NR }\n\n $ gawk -f summary_conventional.awk A.dat B.dat\n min: 1 max: 9 mean: 5\n@end verbatim\n\nTo use @pmg{} for the same purpose, we first create a heap file for\nour AWK script variables and tell @pmg{} where to find it via the\nusual environment variable:\n@verbatim\n $ truncate -s 10M stats.pma\n $ export GAWK_PERSIST_FILE=stats.pma\n@end verbatim\n@noindent\n@pmg{} requires changing the above script to ensure that @code{min}\nand @code{max} are initialized exactly once, when the heap file is\nfirst used, and @emph{not} every time the script runs. Furthermore,\nwhereas script-defined variables such as @code{min} retain their\nvalues across @pmg{} executions, built-in AWK variables such as\n@code{NR} are reset to zero every time @pmg{} runs, so we can't use\nthem in the same way. Here's a modified script for @pmg{}:\n@verbatim\n $ cat summary_persistent.awk\n ! init { min = max = $1; init = 1 }\n min \u003e $1 { min = $1 }\n max \u003c $1 { max = $1 }\n { sum += $1; ++n }\n END { print \"min: \" min \" max: \" max \" mean: \" sum/n }\n@end verbatim\n@noindent\nNote the different pattern on the first line and the introduction of\n@code{n} to supplant @code{NR}. When used with @pmg{}, this new\ninitialization logic supports the same kind of cumulative processing\nthat we saw in the text-analysis scenario. For example, we can ingest\nour input files separately:\n@verbatim\n $ gawk -f summary_persistent.awk A.dat\n min: 1 max: 5 mean: 3\n\n $ gawk -f summary_persistent.awk B.dat\n min: 1 max: 9 mean: 5\n@end verbatim\n@noindent\nAs expected, after the second @pmg{} invocation consumes the\nsecond input file, the output matches that of the non-persistent\nscript that read both files at once.\n\nExercise: Amend the AWK scripts above to compute the median and\nmode(s) using both conventional @gwk{} and @pmg{}. (The median is the\nnumber in the middle of a sorted list; if the length of the list is\neven, average the two numbers at the middle. The modes are the values\nthat occur most frequently.)\n\n@c heaps not portable across machines, use only with same gawk executable (?)\n@c refer to gawk docs for portability constraints on heaps\n@c can use only one heap at a time\n\n@page\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n\nOur third and final set of examples shows that @pmg{} allows us to\nbundle both script-defined data and also user-defined @emph{functions}\nin a persistent heap that may be passed freely between unrelated AWK\nscripts.\n\n@c ADR doesn't like return in count() below\n@c TK: it was put there for a reason:\n@c $ truncate -s 10M funcs.pma\n@c $ export GAWK_PERSIST_FILE=funcs.pma\n@c $ gawk 'function count(A,t) {for(i in A)t++; return t}'\n@c $ gawk 'BEGIN { a[\"x\"] = 4; a[\"y\"] = 5; a[\"z\"] = 6 }'\n@c $ gawk 'BEGIN { print count(a) }'\n@c 3\n@c $ gawk 'BEGIN { delete a }'\n@c $ gawk 'BEGIN { print count(a) }'\n@c [!!blank line, not zero!!]\n@c $\n\nThe following shell transcript repeatedly invokes @pmg{} to create and\nthen employ a user-defined function. These separate invocations\ninvolve several different AWK scripts that communicate via the heap\nfile. Each invocation can add user-defined functions and add or\nremove data from the heap that subsequent invocations will access.\n@smallformat\n@verbatim\n $ truncate -s 10M funcs.pma\n $ export GAWK_PERSIST_FILE=funcs.pma\n $ gawk 'function count(A,t) {for(i in A)t++; return \"\"==t?0:t}'\n $ gawk 'BEGIN { a[\"x\"] = 4; a[\"y\"] = 5; a[\"z\"] = 6 }'\n $ gawk 'BEGIN { print count(a) }'\n 3\n $ gawk 'BEGIN { delete a[\"x\"] }'\n $ gawk 'BEGIN { print count(a) }'\n 2\n $ gawk 'BEGIN { delete a }'\n $ gawk 'BEGIN { print count(a) }'\n 0\n $ gawk 'BEGIN { for (i=0; i\u003c47; i++) a[i]=i }'\n $ gawk 'BEGIN { print count(a) }'\n 47\n@end verbatim\n@end smallformat\n@noindent\nThe first @pmg{} command creates user-defined function @code{count()},\nwhich returns the number of entries in a given associative array; note\nthat variable @code{t} is local to @code{count()}, not global. The\nnext @pmg{} command populates a persistent associative array with\nthree entries; not surprisingly, the @code{count()} call in the\nfollowing @pmg{} command finds these three entries. The next two\n@pmg{} commands respectively delete an array entry and print the\nreduced count, 2. The two commands after that delete the entire array\nand print a count of zero. Finally, the last two @pmg{} commands\npopulate the array with 47 entries and count them.\n\n@c I could be persuaded to leave the polynomial example as an\n@c exercise, offering to send my answer to readers upon request.\n\nThe following shell script invokes @pmg{} repeatedly to create a\ncollection of user-defined functions that perform basic operations on\nquadratic polynomials: evaluation at a given point, computing the\ndiscriminant, and using the quadratic formula to find the roots. It\nthen factorizes @math{x^2 + x - 12} into @math{(x - 3)(x + 4)}.\n@smallformat\n@verbatim\n #!/bin/sh\n rm -f poly.pma\n truncate -s 10M poly.pma\n export GAWK_PERSIST_FILE=poly.pma\n gawk 'function q(x) { return a*x^2 + b*x + c }'\n gawk 'function p(x) { return \"q(\" x \") = \" q(x) }'\n gawk 'BEGIN { print p(2) }' # evaluate \u0026 print\n gawk 'BEGIN{ a = 1; b = 1; c = -12 }' # new coefficients\n gawk 'BEGIN { print p(2) }' # eval/print again\n gawk 'function d(s) { return s * sqrt(b^2 - 4*a*c)}'\n gawk 'BEGIN{ print \"discriminant (must be \u003e=0): \" d(1)}'\n gawk 'function r(s) { return (-b + d(s))/(2*a)}'\n gawk 'BEGIN{ print \"root: \" r( 1) \" \" p(r( 1)) }'\n gawk 'BEGIN{ print \"root: \" r(-1) \" \" p(r(-1)) }'\n gawk 'function abs(n) { return n \u003e= 0 ? n : -n }'\n gawk 'function sgn(x) { return x \u003e= 0 ? \"- \" : \"+ \" } '\n gawk 'function f(s) { return \"(x \" sgn(r(s)) abs(r(s))}'\n gawk 'BEGIN{ print \"factor: \" f( 1) \")\" }'\n gawk 'BEGIN{ print \"factor: \" f(-1) \")\" }'\n rm -f poly.pma\n@end verbatim\n@end smallformat\n@noindent\n\n@page\n@c ==================================================================\n@node Performance\n@chapter Performance\n\nThis chapter explains several performance advantages that result from\nthe implementation of persistent memory in @pmg{}, shows how tuning\nthe underlying operating system sometimes improves performance, and\npresents experimental performance measurements. To make the\ndiscussion concrete, we use examples from a GNU/Linux system---GNU\nutilities atop the Linux OS---but the principles apply to other modern\noperating systems.\n\n@c = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =\n@node Constant-Time Array Access\n@section Constant-Time Array Access\n\n@pmg{} preserves the efficiency of data access when data structures\nare created by one process and later re-used by a different process.\n\nConsider the associative arrays used to analyze Mark Twain's books in\n@ref{Examples}. We created arrays @code{ts[]} and @code{hf[]} by\nreading files @file{sawyer.txt} and @file{finn.txt}. If @i{N} denotes\nthe total volume of data in these files, building the associative\narrays typically requires time proportional to @i{N}, or ", "source.strings[108]": "yellocution,'' and\n" }, "ct": [ "$f", "$inp", "being", "c", "c()", "check", "components", "gawk", "monoliths", "pmg", "s", "s()", "s()()", "s()()()", "s()()()()", "s()()()()()", "s()()()()()()", "s()()()()()()()", "s()()()()()()()()", "s.N", "s.N()", "s.N()()", "s.N()()()", "s.N()()()()", "s.N()()()()()", "s.N()()()()()()", "s.N()()()()()()()", "stand", "wc", "wc()", "wc()()", "wc()()()", "wc()()()()", "wc()()()()()", "wc()()()()()()", "wc()()()()()()()" ], "id": "javascript", "mc": [ "Texts.A", "acm.org", "anticipated.I", "cs.princeton", "cs.princeton.edu", "eecs.umich", "eecs.umich.edu", "gawk().Persistent", "gawk().Persistent.end", "gawk.info", "gwk.The", "s.N" ] }, "id": 0, "sz": 68046, "ts": [ { "c": 0.800000011920929, "d": "Sequential source identifier names", "e": [ "identifiers.sequential_names = 6.00" ], "i": "metadata/file/text/profile::sequential-identifiers", "l": 2 }, { "a": "T1071.001", "c": 0.8999999761581421, "d": "Command/result field word", "e": [ "command" ], "i": "objectives/command-and-control/beacon/network/json::result-command-word", "l": 1, "m": "B0030" }, { "c": 0.20000000298023224, "d": "command keyword", "e": [ "command" ], "i": "micro-behaviors/data/text/keywords/lexicon::command-dup", "l": 1 }, { "c": 0.8999999761581421, "d": "File has 30 or more lines", "e": [ "text.total_lines = 59.00", "text.total_lines = 53.00" ], "i": "metadata/file/text/metrics::file-has-30-plus-lines", "l": 1 }, { "c": 0.699999988079071, "d": "bash shell keyword (beyond shebang)", "e": [ "bash" ], "i": "micro-behaviors/process/create/shell/invoke::bash-keyword", "l": 3 }, { "c": 1.0, "d": "shell code embedded in string", "e": [ "Detected shell at offset 0x9c3f", "Detected shell at offset 0x1cf3", "Detected shell at offset 0x9c40", "Detected shell at offset 0x9b98", "Detected shell at offset 0xa008", "Detected shell at offset 0x1cf4", "Detected shell at offset 0x9f54", "Detected shell at offset 0x9fae", "Detected shell at offset 0x9b97" ], "i": "metadata/lang/embedded::shell", "l": 2 }, { "c": 0.75, "d": "Tiny multi-line text script", "e": [ "text.total_lines = 53.00" ], "i": "metadata/file/text/profile::tiny-text-script", "l": 2 }, { "a": "T1548.003", "c": 0.699999988079071, "d": "sudo command keyword", "e": [ "sudo" ], "i": "objectives/privilege-escalation/elevation-control/sudo::sudo-keyword", "l": 1 }, { "c": 0.800000011920929, "d": "shell execution with -c flag", "e": [ "sh -c" ], "i": "micro-behaviors/process/create/shell/invoke::sh-c-exec", "l": 3 }, { "a": "T1110.001", "c": 0.8500000238418579, "d": "Service account ubuntu", "e": [ "ubuntu" ], "i": "objectives/lateral-movement/brute-force/password::service-account-ubuntu", "l": 1, "m": "E1110" }, { "c": 1.0, "d": "cmd command name", "e": [ "cmd" ], "i": "micro-behaviors/process/create/shell/invoke::cmd-name", "l": 2 }, { "a": "T1110.001", "c": 0.8999999761581421, "d": "Weak password master", "e": [ "master" ], "i": "objectives/lateral-movement/brute-force/password::weak-pass-master", "l": 1, "m": "E1110" }, { "c": 0.699999988079071, "d": "Package references COPYING", "e": [ "copying" ], "i": "metadata/package/documentation::references-license-2", "l": 2 }, { "a": "T1572", "d": "Regex component marker", "e": [ "persistent memory'' has sometimes been used to\ndenote novel byte-addressable non-volatile memory hardware---an\nunfortunate pract", "persistent memory'' has sometimes been used to\ndenote novel byte-addressable non-volatile memory hardware---an\nunfortunate pract", " # 6\nunsetenv GAWK_PERSIST_FILE # disable persistence until wanted " ], "i": "objectives/command-and-control/channel/tunnel/proxy::gost-scheme--rx-4", "l": 1 }, { "a": "T1195.002", "c": 1.0, "d": "Array index [i] access", "e": [ " \u003e B.dat\n@end verbatim\n@noindent\nA conventional @emph{non}-persistent AWK script can compute basic\nsummary statistics:\n@verbatim", "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical mo" ], "i": "objectives/supply-chain/hidden-payload/staging::array-index-i", "l": 1 }, { "c": 0.7200000286102295, "d": "LLM instruction context", "e": [ "prompt" ], "i": "micro-behaviors/data/text/llm/override::llm-instruction-context", "l": 1 }, { "a": "T1552.001,T1082,T1041", "c": 0.9399999976158142, "d": "Regex component marker", "e": [ "error" ], "i": "objectives/supply-chain/recon-exfil/npm::console-output-silenced--rx-3", "l": 1 }, { "c": 1.0, "d": "Regex component marker", "e": [ "needle" ], "i": "well-known/malware/stealer/needle::needle-panel-marker--rx-8", "l": 1 }, { "a": "T1552.001, T1041", "c": 0.9399999976158142, "d": "\"search\" verb inside a string literal", "e": [ "search" ], "i": "objectives/supply-chain/recon-exfil/ai-cli::prompt-search-word", "l": 1, "m": "E1552" }, { "c": 0.30000001192092896, "d": "for loop (content)", "e": [ " for release:\n" ], "i": "micro-behaviors/data/control-flow/analysis::for-loop-content", "l": 2 }, { "c": 0.800000011920929, "d": "Do-while block marker", "e": [ "do {" ], "i": "micro-behaviors/process/create/script/wsh::wsh-do-block-marker", "l": 1 }, { "c": 1.0, "d": "Infinite loop structure (e.g. for(;;) or while(1))", "e": [ "for (;;)" ], "i": "micro-behaviors/data/control-flow/loop::infinite-loop-header", "l": 2 }, { "a": "T1195.002", "c": 1.0, "d": "Loop construct (for/while)", "e": [ "while (", "for(", "for (" ], "i": "objectives/supply-chain/hidden-payload/staging::loop-construct", "l": 1 }, { "c": 0.9200000166893005, "d": "Regex component marker", "e": [ "echo" ], "i": "objectives/anti-static/obfuscation/syntax::js-eval-word-parts--rx-4", "l": 1 }, { "a": "T1528", "c": 0.9200000166893005, "d": "Regex component marker", "e": [ "modify" ], "i": "objectives/credential-access/cloud/token::google-gmail-full-scope--rx-4", "l": 1 }, { "a": "T1548", "c": 0.8500000238418579, "d": "/etc/sudoers sudo configuration", "e": [ "/etc/sudoers" ], "i": "micro-behaviors/fs/path/config/privesc::etc-sudoers", "l": 2 }, { "c": 0.8999999761581421, "d": "Low string density in source", "e": [ "text.string_density = 0.04" ], "i": "metadata/file/text/metrics::low-string-density-large-file", "l": 1 }, { "a": "T1105", "c": 0.800000011920929, "d": "Shell print command", "e": [ "printf" ], "i": "objectives/command-and-control/dropper/execution/browser::shell-print-command", "l": 1, "m": "B0024" }, { "c": 0.20000000298023224, "d": "pwd keyword", "e": [ "pwd" ], "i": "micro-behaviors/data/text/keywords/lexicon::pwd", "l": 1 }, { "a": "T1110.001", "c": 0.20000000298023224, "d": "admin keyword", "e": [ "admin" ], "i": "micro-behaviors/data/text/keywords/username::admin-keyword", "l": 2, "m": "B0028" }, { "a": "T1566.002", "c": 0.8999999761581421, "d": "Regex component marker", "e": [ "fixed" ], "i": "objectives/credential-access/phishing/tactics::full-window-iframe-shell--rx-5", "l": 1, "m": "B0030" }, { "a": "T1204.001", "c": 0.8600000143051147, "d": "Regex component marker", "e": [ "permission" ], "i": "objectives/execution/lure/video-meeting::meeting-device-permission-lure--rx-2", "l": 1 }, { "c": 0.30000001192092896, "d": "xz compression reference", "e": [ "xz" ], "i": "micro-behaviors/dylib/library/family::xz-ref", "l": 2 }, { "a": "T1528", "c": 0.8600000143051147, "d": "Regex component marker", "e": [ "html" ], "i": "objectives/credential-access/cloud/token::oauth-redirect-with-target-email--rx-10", "l": 2 }, { "a": "T1041", "c": 1.0, "d": "Regex component marker", "e": [ "processes" ], "i": "well-known/malware/rat/overlord::overlord-features--rx-4", "l": 1 }, { "a": "T1552.001,T1082,T1041", "c": 0.9399999976158142, "d": "Regex component marker", "e": [ "info" ], "i": "objectives/supply-chain/recon-exfil/npm::console-output-silenced--rx-4", "l": 1 }, { "c": 1.0, "d": "Loop structure", "e": [ "for (", "for(", "while (" ], "i": "micro-behaviors/data/encode/xor::js-loop", "l": 2 }, { "a": "T1005", "c": 0.75, "d": "Regex component marker", "e": [ "master" ], "i": "objectives/credential-access/financial/credit-card::mastercard-bin--rx-2", "l": 1, "m": "B0030" }, { "a": "T1105", "c": 0.8500000238418579, "d": "macOS shell interpreter", "e": [ "bash" ], "i": "objectives/command-and-control/dropper/execution/browser::macos-shell-interpreter", "l": 1, "m": "B0024" }, { "a": "T1005", "c": 0.9200000166893005, "d": "File extension list variable marker", "e": [ "exts" ], "i": "objectives/collection/stealer/tactics::office-secret-extension-list--rx-4", "l": 1, "m": "B0028" }, { "a": "T1528", "c": 0.9200000166893005, "d": "Regex component marker", "e": [ "metadata" ], "i": "objectives/credential-access/cloud/token::google-gmail-full-scope--rx-2", "l": 1 }, { "a": "T1059.007", "c": 0.8999999761581421, "d": "Regex component marker", "e": [ "send" ], "i": "objectives/command-and-control/remote-command/protocol::websocket-host-env-send--rx-12", "l": 1 }, { "c": 0.9200000166893005, "d": "Regex component marker", "e": [ "alpha" ], "i": "objectives/anti-static/obfuscation/syntax::js-eval-word-parts--rx-3", "l": 1 }, { "c": 1.0, "d": ".tar.gz extension", "e": [ "Markov'' script persistent. Volos provided and tested the advice on\ntuning OS parameters in @ref{Virtual Memory and Big Data}. " ], "i": "objectives/command-and-control/dropper/standalone::tar-gz-ext", "l": 1 }, { "c": 1.0, "d": "HISTORY documentation reference", "e": [ "History" ], "i": "metadata/package/documentation::references-changelog-history", "l": 2 }, { "c": 0.30000001192092896, "d": "stderr string reference", "e": [ " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' ", " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' " ], "i": "micro-behaviors/process/fd/stdio::stderr-string", "l": 2 }, { "c": 1.0, "d": "Regex component marker", "e": [ "gate" ], "i": "well-known/malware/stealer/needle::needle-panel-marker--rx-10", "l": 1 }, { "a": "T1048.003", "c": 0.949999988079071, "d": "Regex component marker", "e": [ "live" ], "i": "objectives/supply-chain/recon-exfil/dns::dns-lookup-with-canary-domain--rx-6", "l": 1, "m": "E1041" }, { "c": 0.6000000238418579, "d": "Copyright notice", "e": [ "Copyright" ], "i": "metadata/package/license::copyright-word", "l": 2 }, { "a": "T1204.001", "c": 0.8600000143051147, "d": "Regex component marker", "e": [ "allow" ], "i": "objectives/execution/lure/video-meeting::meeting-device-permission-lure--rx-6", "l": 1 }, { "c": 0.8999999761581421, "d": "Interactive process monitors (htop/atop)", "e": [ "atop" ], "i": "micro-behaviors/os/sysinfo/performance/monitoring::htop-atop", "l": 2 }, { "c": 0.7200000286102295, "d": "LLM role context", "e": [ "system" ], "i": "micro-behaviors/data/text/llm/override::llm-role-context", "l": 1 }, { "c": 0.699999988079071, "d": "base64 encoding keyword", "e": [ "base64" ], "i": "micro-behaviors/data/text/encoded::base64-token", "l": 2 }, { "c": 0.8500000238418579, "d": "Code uses generator functions", "e": [ "yield " ], "i": "metadata/lang/javascript-features::has-generators", "l": 2 }, { "c": 1.0, "d": "Uses awk for text processing", "e": [ "awk" ], "i": "well-known/tool/sysadmin/repo-maintenance::awk-processing", "l": 2 }, { "c": 1.0, "d": "XOR bitwise operator in expression", "e": [ "x^2" ], "i": "micro-behaviors/data/source/syntax/keyword::js-keyword-xor", "l": 2 }, { "a": "T1041", "c": 0.75, "d": "HTTP upload request call token", "e": [ "request" ], "i": "objectives/supply-chain/recon-exfil/npm-install-targeting::upload-request-call", "l": 1 }, { "c": 0.9399999976158142, "d": "Regex component marker", "e": [ "assert" ], "i": "metadata/package/testing/harness/v8::v8-mjsunit-flags-assert-file--rx-10", "l": 1 }, { "a": "T1005", "c": 0.4000000059604645, "d": "Generic data extension", "e": [ ".dat" ], "i": "objectives/collection/file-targeting/filter::dat-extension", "l": 1, "m": "B0024" }, { "a": "T1071", "c": 0.800000011920929, "d": "Random number generation function call", "e": [ "rand()" ], "i": "micro-behaviors/communications/ip/spoof::bitwise-ip-rand-source", "l": 2, "m": "C0001" }, { "c": 0.800000011920929, "d": "Do-while close marker", "e": [ "} while" ], "i": "micro-behaviors/process/create/script/wsh::wsh-do-while-close", "l": 1 }, { "c": 1.0, "d": "Package references CHANGES", "e": [ "changes" ], "i": "metadata/package/documentation::references-changelog-changes", "l": 2 }, { "a": "T1528", "c": 0.9200000166893005, "d": "Regex component marker", "e": [ "auth" ], "i": "objectives/credential-access/cloud/token::google-gmail-full-scope--rx-6", "l": 1 }, { "c": 1.0, "d": "JavaScript length property", "e": [ "length" ], "i": "metadata/lang/scripted::length-prop", "l": 1 }, { "c": 0.20000000298023224, "d": "output keyword", "e": [ "output" ], "i": "micro-behaviors/data/text/keywords/lexicon::output", "l": 1 }, { "c": 1.0, "d": "\"Environment\" keyword", "e": [ "environment" ], "i": "micro-behaviors/data/text/keywords/env::environment-keyword", "l": 1 }, { "a": "T1070.004", "c": 0.800000011920929, "d": "rm -f shell command (force, non-recursive)", "e": [ " ; echo '' # 89\nrm -f heap.pma ", " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' ", " 's (around 4-5 ms for TK)' # 33\nset inp = 'input.dat' ", "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical mo", " ; echo '' # 73\nrm -f rwa ", " ; echo '' # 82\nrm -f ft ", " \u003e B.dat\n@end verbatim\n@noindent\nA conventional @emph{non}-persistent AWK script can compute basic\nsummary statistics:\n@verbatim" ], "i": "micro-behaviors/fs/directory/delete::rm-f", "l": 1, "m": "C0047" }, { "c": 0.8199999928474426, "d": "JavaScript static key string assignment", "e": [ "='GAWK_PERSIST_FILE=heap.pma'" ], "i": "micro-behaviors/crypto/symmetric/aes/library::js-static-key-string-assignment", "l": 1 }, { "c": 0.9599999785423279, "d": "Regex component marker", "e": [ "assert" ], "i": "metadata/package/testing/harness/v8::v8-mjsunit-native-test-file--rx-5", "l": 1 }, { "c": 0.8199999928474426, "d": "Regex component marker", "e": [ "Speed" ], "i": "objectives/discovery/system/ics-environment/register::ics-register-flow-rate--rx-77", "l": 1 }, { "a": "T1027", "c": 0.949999988079071, "d": "Regex component marker", "e": [ "base64" ], "i": "objectives/anti-static/obfuscation/code-metrics/string-blob::js-multi-line-blob-benign-text--rx-8", "l": 1, "m": "B0032" }, { "c": 0.6000000238418579, "d": "Linux platform string", "e": [ "training'' phase followed by an\noutput-generation phase. Use @pmg{} to de-couple the two phases and\nto allow the statistical mo", "paged in'' to the memory\nof the @pmg{} process. In the worst case, the heap file is not in the\nfile system's in-memory cache, s", "lazy'' rather than eager. For Linux see the discussion\nof the @code{dirty_*} parameters at\n@url{https://www.kernel.org/doc/html", "falls off the cliff'' as @pmg{}'s memory footprint\nexceeds the capacity of DRAM and paging begins.\n\n@c TODO:\n@c virtual *machine", "cold start'' condition resembling the\nimmediate aftermath of a machine reboot. Accesses to ordinary files\non durable storage wi", "pay as you @emph{change}'' storage\ncosts.@footnote{The system call that implements reflink copying is\ndescribed in @command{man ", "\n $ cp --reflink=always heap.pma $backup_base.pma\n $ chmod a-w $backup_base.pma\n $ sync\n $ touch $ba", "Markov'' script persistent. Volos provided and tested the advice on\ntuning OS parameters in @ref{Virtual Memory and Big Data}. ", " \u003e B.dat\n@end verbatim\n@noindent\nA conventional @emph{non}-persistent AWK script can compute basic\nsummary statistics:\n@verbatim" ], "i": "micro-behaviors/os/sysinfo/platform/string::os-string-linux", "l": 2 }, { "a": "T1195.002", "c": 0.699999988079071, "d": "References shell script file (.sh)", "e": [ " ./bootstrap.sh," ], "i": "objectives/supply-chain/install-hook/scripts/hook-file::shell-script-ref", "l": 1 }, { "c": 0.8999999761581421, "d": "Prototype guard keyword", "e": [ "prototype" ], "i": "micro-behaviors/data/source/prototype::js-proto-guard-prototype", "l": 2 }, { "a": "T1548.003", "c": 0.8600000143051147, "d": "sudoers policy persistence path", "e": [ " /etc/sudoers" ], "i": "objectives/persistence/system/surface/paths::sudoers-policy-path", "l": 4 }, { "d": "Mnemonic keyword", "e": [ "mnemonic" ], "i": "micro-behaviors/data/text/crypto::mnemonic-word", "l": 1 }, { "a": "T1555", "c": 0.8999999761581421, "d": "Regex component marker", "e": [ "base64" ], "i": "objectives/credential-access/wallet/solana::solana-secretkey-serialization--rx-6", "l": 1 } ], "sha": "37aa8e2269ec42fff38a8a9c39a6310f760ce20fb0f4ebe8d586c230de2f3048", "path": "pm-gawk.texi", "type": "javascript" } ], "tv": "60d49" } }