| Benchmark Suites | 14 |
| Sessions | 475 |
| Unique Puzzles | 2,939 (1,498 Test ✓ / 51.0%) |
| Total Task Runs | 8,514 |
| Runs Train ✓ | 3,315 (38.9%) |
| Runs Test ✓ | 2,707 (31.8%) |
| Total Duration | 589:51:12 |
| Total Errors | 2,931 |
| Total Tokens | 296,725,259 |
| In Tokens | 205,150,173 |
| Out Tokens | 88,563,177 |
| Total Retries | 12,189 |
| Suite Directory | Sessions | Unique Tasks | Task Runs | Duration | Avg / Sess | Err | Train ✓ | Train% | Test ✓ | Test% | Score | Tokens | Retries |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sessions | 29 | 950 | 2,232 | 216:51:54 | 07:28:41 | 1,164 | 926 | 41.5% | 743 | 33.3% | 0.00 | 68,309,006 | 4,356 |
| sessions_001 | 156 | 400 | 1,047 | 53:04:21 | 00:20:24 | 0 | 148 | 14.1% | 135 | 12.9% | 0.00 | 68,522,396 | 0 |
| sessions_002 | 56 | 20 | 113 | 01:23:06 | 00:01:29 | 8 | 43 | 38.1% | 42 | 37.2% | 0.00 | 1,051,550 | 0 |
| sessions_003 | 39 | 300 | 447 | 14:28:16 | 00:22:15 | 0 | 142 | 31.8% | 137 | 30.6% | 0.00 | 11,473,907 | 0 |
| sessions_005 | 30 | 524 | 1,007 | 43:41:16 | 01:27:22 | 75 | 399 | 39.6% | 340 | 33.8% | 0.00 | 25,602,234 | 10 |
| sessions_1D | 18 | 901 | 901 | 31:08:00 | 01:43:46 | 127 | 690 | 76.6% | 667 | 74.0% | 0.00 | 20,722,063 | 221 |
| sessions_ARCv2_eval | 22 | 120 | 686 | 130:50:19 | 05:56:49 | 569 | 40 | 5.8% | 8 | 1.2% | 0.00 | 38,724,924 | 925 |
| sessions_ARCv2_train | 26 | 100 | 276 | 08:56:24 | 00:20:37 | 88 | 194 | 70.3% | 153 | 55.4% | 0.00 | 5,377,594 | 331 |
| sessions_ARCv2_train_ | 9 | 10 | 76 | 01:32:58 | 00:10:19 | 20 | 61 | 80.3% | 57 | 75.0% | 0.00 | 903,154 | 228 |
| sessions_ARCv2_train_100_200 | 6 | 100 | 223 | 11:02:33 | 01:50:25 | 130 | 101 | 45.3% | 69 | 30.9% | 0.00 | 6,464,883 | 244 |
| sessions_ARCv2_train_200_300 | 4 | 200 | 310 | 17:57:40 | 04:29:25 | 245 | 70 | 22.6% | 46 | 14.8% | 0.00 | 9,254,062 | 479 |
| sessions_ConceptARC | 21 | 176 | 481 | 25:11:41 | 01:11:59 | 205 | 281 | 58.4% | 90 | 18.7% | 0.00 | 16,811,453 | 1,496 |
| sessions_mini_arc | 1 | 149 | 149 | 05:27:43 | 05:27:43 | 76 | 84 | 56.4% | 75 | 50.3% | 0.00 | 2,002,729 | 399 |
| sessions_optorex_1D | 58 | 566 | 566 | 28:14:54 | 00:29:13 | 224 | 136 | 24.0% | 145 | 25.6% | 0.00 | 21,505,304 | 3,500 |