ACCRE R9 Cluster Quick and Dirty Status
Report generated at Tue Oct 13 03:23:01 AM CDT 2026
Problem Nodes
HOSTNAMES STATE TIMESTAMP REASON COMMENT
cn1209 drain 2026-10-13T00:20:01 auto - na - update-r auto - na - update-done
cn1213 drain 2026-10-12T22:15:01 auto - na - update-r auto - na - update-done
cn1221 drain 2026-10-13T00:45:02 auto - na - update-r auto - na - update-done
cn1222 drain 2026-10-13T01:10:01 auto - na - update-r auto - na - update-done
cn1223 drain 2026-10-13T01:30:01 auto - na - update-r auto - na - update-done
cn1236 drain 2026-10-13T03:22:01 na - na - update-pen na - na - update-pending
cn1238 drain 2026-10-13T01:50:01 auto - na - update-r auto - na - update-done
cn1324 drain 2026-10-12T21:12:52 auto - na - update-r auto - na - update-running AUTO-FAILED at upgrade
cn1442 drain 2026-10-08T17:29:33 Troy - RT101677 - fo Troy - RT101677 - for cn1443 replace breakout fiber
cn1443 drain* 2026-10-08T09:48:23 Troy - RT101677 - fo Troy - RT101677 - for cn1443 replace breakout fiber
cn1445 drain 2026-10-08T17:29:33 Troy - RT101677 - fo Troy - RT101677 - for cn1443 replace breakout fiber
cn1482 drain 2026-10-06T16:57:15 Troy - RT101522 - fo Troy - RT101522 - for cn1484 TDX replace fiber breakout
cn1483 drain 2026-10-06T16:57:15 Troy - RT101522 - fo Troy - RT101522 - for cn1484 TDX replace fiber breakout
cn1484 drain* 2026-10-06T15:03:24 Troy - RT101522 - fo Troy - RT101522 - for cn1484 TDX replace fiber breakout
cn1485 drain 2026-10-06T16:57:15 Troy - RT101522 - fo Troy - RT101522 - for cn1484 TDX replace fiber breakout
cn1492 drain 2026-10-13T02:40:01 auto - na - update-r auto - na - update-done
cn1497 drain 2026-10-13T02:10:01 auto - na - update-r auto - na - update-done
cn1595 drain* 2026-07-27T16:58:32 Troy - RT100528 - Th Troy - RT100528 - Thermal issues
cn1600 drng 2026-09-23T10:48:07 goffta1 - RT101604 - goffta1 - RT101604 - extension, then upgrade
cn1700 drain 2026-09-09T09:49:01 Alex - RT100934 - Po Alex - RT100934 - Power issues
cn1817 drain 2026-09-08T11:04:20 Nobody - RT101264 - Nobody - RT101264 - Drain for CGW file recovery
gpu0302 drain 2026-10-12T22:46:59 gres/gpu count repor auto - na - update-done
Upgrade Campaign Status
NODE UPGRADE - target 5.14.0-687.52.1
========================================================================
FLEET [###-------------------------] 10% 48/467 on target
(1 partition fully on target, not listed: sam)
batch 11/387 [#-------------------] 3%
batch_gpu 11/39 [######--------------] 28%
interactive 21/32 [#############-------] 66%
interactive_gpu 3/7 [#########-----------] 43%
------------------------------------------------------------------------
IN THE WORKFLOW 10 nodes out of service
Done, safe to resume (on target, healthy): 4
batch auto 2 cn[1209,1213]
batch_gpu auto 1 gpu0302
interactive auto 1 cn1324
On target, health not yet confirmed (in downtime / no nagios data): 6
batch auto 6 cn[1221-1223,1238,1492,1497]
------------------------------------------------------------------------
LAST 24 HOURS 16 back in service, 0 handed back untouched
Completed: 16
auto 5 cn[1323,1546,1626],gpu[0064,0068]
goffta1 11 gpu[0058,0060-0061,0075-0077,0079,0085,0308-0310]
Queue Summary (Batch)
GROUP USER ACTIVE_JOBS ACTIVE_CORES PENDING_JOBS PENDING_CORES
-----------------------------------------------------------------------------------------
accre_guests 0 0 1 100
haojz 0 0 1 100
-----------------------------------------------------------------------------------------
anderson_mri 16 61 70 296
liy109 16 61 70 296
-----------------------------------------------------------------------------------------
biostat_faculty 442 442 38 38
nianh1 442 442 38 38
-----------------------------------------------------------------------------------------
booth_lab 2 6 0 0
chenh55 1 4 0 0
muesm 1 2 0 0
-----------------------------------------------------------------------------------------
booth_lab_yanjun 1 4 0 0
weiy8 1 4 0 0
-----------------------------------------------------------------------------------------
cem_lab 0 0 7 56
holtcm 0 0 7 56
-----------------------------------------------------------------------------------------
cgg 1 9 151 868
lib13 1 9 150 804
liy110 0 0 1 64
-----------------------------------------------------------------------------------------
cms 753 4232 426 1275
cmslocal 233 1603 164 489
cmspilot 520 2629 262 786
-----------------------------------------------------------------------------------------
coxlab 1 2 0 0
blostf1 1 2 0 0
-----------------------------------------------------------------------------------------
das_lab 1 1 0 0
shiltmh1 1 1 0 0
-----------------------------------------------------------------------------------------
davis_lab 0 0 1 16
bluejor 0 0 1 16
-----------------------------------------------------------------------------------------
g_gamazon_lab 1 3 0 0
evanspd1 1 3 0 0
-----------------------------------------------------------------------------------------
gravity 13 40 20 56
ligo 13 40 20 56
-----------------------------------------------------------------------------------------
h_biostat_student 34 34 138 336
gaix1 0 0 2 200
liuk20 34 34 136 136
-----------------------------------------------------------------------------------------
h_cqs 30 77 0 0
yanb1 29 29 0 0
yangj24 1 48 0 0
-----------------------------------------------------------------------------------------
h_oguz_lab 0 0 4 16
lih30 0 0 4 16
-----------------------------------------------------------------------------------------
h_vmac_imaging 1 8 0 0
workmea 1 8 0 0
-----------------------------------------------------------------------------------------
isde-rer 0 0 1 8
maybergm 0 0 1 8
-----------------------------------------------------------------------------------------
kaczkurkin_lab 1 15 0 0
abbasia 1 15 0 0
-----------------------------------------------------------------------------------------
l3_ferrell_lab 1 12 0 0
graciet 1 12 0 0
-----------------------------------------------------------------------------------------
l3_humphreys_lab 4 16 56 240
niuy3 0 0 4 32
zhanl43 4 16 52 208
-----------------------------------------------------------------------------------------
l3_runnoe_group 9 63 60 240
becerraa 1 4 0 0
kaldorme 8 59 60 240
-----------------------------------------------------------------------------------------
l3_vuiis_cci 1 1 0 0
vuiis_daily_s 1 1 0 0
-----------------------------------------------------------------------------------------
lea_lab 82 162 691 1382
arneram 2 2 0 0
brassl1 80 160 691 1382
-----------------------------------------------------------------------------------------
maiziezhou_lab 138 327 1429 2519
chenp12 131 131 524 524
maow 1 4 898 1802
wangh67 6 192 7 193
-----------------------------------------------------------------------------------------
maiziezhou_lab_phd 7 248 34 646
chowx 0 0 1 2
yuanw2 6 120 29 580
zhuy45 1 128 4 64
-----------------------------------------------------------------------------------------
mass_spec 2 48 0 0
masapps 2 48 0 0
-----------------------------------------------------------------------------------------
mchaourab 128 2048 171 2736
butskhl 128 2048 171 2736
-----------------------------------------------------------------------------------------
mchaourab-csb 13 208 0 0
butskhl 13 208 0 0
-----------------------------------------------------------------------------------------
mltf 0 0 4 4
mltfpilot 0 0 4 4
-----------------------------------------------------------------------------------------
moro_lab 22 68 521 1081
kitchc2 22 68 481 1041
salams2 0 0 40 40
-----------------------------------------------------------------------------------------
nbody 100 100 374 374
blakeghg 100 100 374 374
-----------------------------------------------------------------------------------------
ng_lab 0 0 1 16
kimj119 0 0 1 16
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 366 366
goffta1 0 0 366 366
-----------------------------------------------------------------------------------------
palmeri_lab 3 180 9 540
jeongj6 3 180 9 540
-----------------------------------------------------------------------------------------
p_cortez_lab 4 8 176 356
paulchs 4 8 176 356
-----------------------------------------------------------------------------------------
p_csb_meiler 0 0 10741 20868
agarwm5 0 0 10263 20390
browkl12 0 0 1 1
yange8 0 0 477 477
-----------------------------------------------------------------------------------------
p_dsi 0 0 6 9
yangi1 0 0 6 9
-----------------------------------------------------------------------------------------
p_englot_group 8 192 0 0
redaa1 8 192 0 0
-----------------------------------------------------------------------------------------
p_matheny_lab 85 423 44 220
koolajd1 85 423 44 220
-----------------------------------------------------------------------------------------
p_meiler 0 0 1 3
yange8 0 0 1 3
-----------------------------------------------------------------------------------------
rke_group 0 0 6 6
kennedkk 0 0 6 6
-----------------------------------------------------------------------------------------
rokaslab 275 562 63183 98422
copea1 66 66 4777 4777
danist 191 380 56913 87673
riedlio 1 48 0 0
sautet1 17 68 1493 5972
-----------------------------------------------------------------------------------------
rubinov_lab 0 0 2 448
wuw11 0 0 2 448
-----------------------------------------------------------------------------------------
ruderferlab 1 16 2 8
colemapl 1 16 2 8
-----------------------------------------------------------------------------------------
sbcs 19 190 13 130
shij19 19 190 13 130
-----------------------------------------------------------------------------------------
taylor_group 66 653 35 350
petrop3 66 653 35 350
-----------------------------------------------------------------------------------------
tk_lab 5 200 0 0
yoonh15 5 200 0 0
-----------------------------------------------------------------------------------------
vgi 68 345 221 1331
actkinkv 1 10 0 0
parkj71 67 335 221 1331
-----------------------------------------------------------------------------------------
womelsdorf_lab 1 16 0 0
gerritcg 1 16 0 0
-----------------------------------------------------------------------------------------
yang_lab 16 16 0 0
larkinsd 16 16 0 0
-----------------------------------------------------------------------------------------
yang_lab_csb 16 246 6 108
ranx 3 12 0 0
zhengm9 13 234 6 108
-----------------------------------------------------------------------------------------
Totals: 2371 11282 79009 135468
Queue Summary (Batch GPU)
GROUP USER ACTIVE_JOBS ACTIVE_GPUS PENDING_JOBS PENDING_GPUS
-----------------------------------------------------------------------------------------
accre_guests_acc 4 4 0 0
liy110 4 4 0 0
-----------------------------------------------------------------------------------------
cgg_acc 1 2 0 0
sandk13 1 2 0 0
-----------------------------------------------------------------------------------------
csb_gpu_acc 19 22 5236 5242
gonzc11 0 0 5206 5206
larkinsd 2 2 14 14
ranx 1 4 1 4
yangz24 0 0 1 4
zhengm9 16 16 14 14
-----------------------------------------------------------------------------------------
h_oguz_lab_acc 0 0 2 2
lih30 0 0 2 2
-----------------------------------------------------------------------------------------
maple_lab_acc 0 0 1 1
changx3 0 0 1 1
-----------------------------------------------------------------------------------------
mchaourab_acc 57 57 15 15
lindsr1 57 57 15 15
-----------------------------------------------------------------------------------------
mltf_acc 0 0 1 1
mltfpilot 0 0 1 1
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 23 120
goffta1 0 0 23 120
-----------------------------------------------------------------------------------------
p_dsi_acc 0 0 12 12
yangi1 0 0 12 12
-----------------------------------------------------------------------------------------
p_meiler_acc 37 37 7 7
agarwm5 0 0 1 1
browkl12 0 0 6 6
holtcm 7 7 0 0
tydingcw 30 30 0 0
-----------------------------------------------------------------------------------------
taylor_group_acc 2 7 0 0
laaln1 1 6 0 0
schultls 1 1 0 0
-----------------------------------------------------------------------------------------
Totals: 120 129 5297 5400
Queue Summary (interactive)
GROUP USER ACTIVE_JOBS ACTIVE_CORES PENDING_JOBS PENDING_CORES
-----------------------------------------------------------------------------------------
booth_lab_int 3 3 0 0
comptoab 1 1 0 0
mathura 1 1 0 0
vessa 1 1 0 0
-----------------------------------------------------------------------------------------
chbe_4830_int 1 2 0 0
henrylw 1 2 0 0
-----------------------------------------------------------------------------------------
g_giri_group_int 1 4 0 0
breyem3 1 4 0 0
-----------------------------------------------------------------------------------------
h_vmac_imaging_int 1 6 0 0
jackb13 1 6 0 0
-----------------------------------------------------------------------------------------
maiziezhou_lab_int 16 64 8 32
tangk10 16 64 8 32
-----------------------------------------------------------------------------------------
maiziezhou_lab_phd_int 7 125 6 16
chowx 0 0 6 16
yuanw2 7 125 0 0
-----------------------------------------------------------------------------------------
nbody_int 2 26 0 0
pater32 2 26 0 0
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 10 10
goffta1 0 0 10 10
-----------------------------------------------------------------------------------------
rubinov_lab_int 1 10 0 0
sardarn 1 10 0 0
-----------------------------------------------------------------------------------------
vgi_int 2 8 0 0
elderm 1 4 0 0
nitinr 1 4 0 0
-----------------------------------------------------------------------------------------
yang_lab_int 1 8 0 0
shaoq1 1 8 0 0
-----------------------------------------------------------------------------------------
Totals: 35 256 24 58
Queue Summary (interactive_gpu)
GROUP USER ACTIVE_JOBS ACTIVE_GPUS PENDING_JOBS PENDING_GPUS
-----------------------------------------------------------------------------------------
accre_dgx_iacc 1 4 0 0
rubinom 1 4 0 0
-----------------------------------------------------------------------------------------
dsi_dgx_iacc 3 3 3 24
chattec 1 1 0 0
jenkm22 0 0 3 24
schultls 1 1 0 0
wut18 1 1 0 0
-----------------------------------------------------------------------------------------
nodeupgrade 0 0 1 2
goffta1 0 0 1 2
-----------------------------------------------------------------------------------------
p_matheny_lab_iacc 1 2 0 0
koolajd1 1 2 0 0
-----------------------------------------------------------------------------------------
Totals: 5 9 4 26
Partition Summary
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
interactive up 14-00:00:0 2 mix- cn[1804,1812]
interactive up 14-00:00:0 2 drain cn[1324,1817]
interactive up 14-00:00:0 10 mix cn[1287,1301,1325,1329-1330,1805,1813-1814,1818-1819]
interactive up 14-00:00:0 2 alloc cn[1302,1328]
interactive up 14-00:00:0 16 idle cn[1322-1323,1326,1707,1800-1803,1806-1811,1815-1816]
batch* up 14-00:00:0 16 mix- cn[1203,1205,1208,1216,1224,1234-1235,1390,1398,1488,1502,1509,1532,1631-1632,1705]
batch* up 14-00:00:0 3 drain* cn[1443,1484,1595]
batch* up 14-00:00:0 1 drng cn1600
batch* up 14-00:00:0 15 drain cn[1209,1213,1221-1223,1236,1238,1442,1445,1482-1483,1485,1492,1497,1700]
batch* up 14-00:00:0 151 mix cn[1227,1281-1283,1285,1290-1292,1295,1299,1312,1314-1315,1332-1334,1337-1338,1342-1346,1348,1353-1354,1359,1361,1363-1364,1367,1373-1377,1384,1388,1393-1397,1399-1400,1402,1406-1407,1409,1411,1416,1420-1421,1423,1425-1426,1430-1431,1434-1435,1437-1438,1441,1446-1449,1453-1456,1458,1461,1463,1466,1470,1472,1476-1477,1479-1481,1499,1507,1520,1523,1533,1536-1537,1544-1546,1548-1550,1552,1554-1555,1558-1559,1563-1565,1567-1571,1573-1574,1576-1577,1580,1583-1584,1587-1589,1592-1594,1597,1599,1601-1604,1606-1608,1613-1614,1616-1617,1619-1621,1624,1627-1630,1633,1701-1704,1706,1708-1709,2000]
batch* up 14-00:00:0 201 alloc cn[1202,1204,1206-1207,1210-1212,1215,1217-1220,1226,1228-1233,1237,1239-1242,1257-1262,1264-1275,1277-1280,1284,1286,1288-1289,1293-1294,1296-1298,1303-1311,1313,1316-1318,1320-1321,1327,1331,1335-1336,1339-1341,1347,1349-1352,1357-1358,1360,1362,1365-1366,1368-1369,1371-1372,1378-1383,1385,1387,1389,1391-1392,1401,1403-1405,1408,1410,1414-1415,1417-1419,1422,1424,1427,1432,1436,1439,1450,1452,1457,1460,1462,1464,1467-1469,1471,1473-1475,1478,1486-1487,1489-1491,1493-1496,1498,1500-1501,1503-1506,1508,1510-1519,1522,1524-1525,1527-1531,1534-1535,1538,1543,1547,1551,1553,1556-1557,1561-1562,1566,1575,1578-1579,1581-1582,1585-1586,1596,1598,1605,1609-1610,1612,1615,1618,1622-1623,1625-1626,1710]
batch_gpu up 14-00:00:0 15 mix- gpu[0059,0062-0063,0065-0066,0069-0070,0072,0078,0081-0082,0300-0301,0303-0304]
batch_gpu up 14-00:00:0 1 drain gpu0302
batch_gpu up 14-00:00:0 10 mix gpu[0064,0068,0077,0079,0305-0307,0309-0310],hgx03
batch_gpu up 14-00:00:0 8 alloc gpu[0067,0071,0073-0076,0080,0208]
batch_gpu up 14-00:00:0 5 idle gpu[0085,0308],gracehopper[01-02],hgx02
interactive_gpu up 14-00:00:0 2 mix dgx03,gpu0207
interactive_gpu up 14-00:00:0 5 idle dgx[01,04],gpu[0058,0060-0061]
sam up 2-02:00:00 1 alloc cms-sam-01
sam up 2-02:00:00 1 idle cms-sam-02
reserved inact infinite 0 n/a