TTIC SLURM Cluster: Process Statistics
Fri Dec 2 07:35:12 CST 2022 [Docs] | [Job Process Info] | [Job Queue] | [Cluster Usage & Load]

Node User Command CPU % MEM % MEM (kb)
cpu2 NCPU: 8 load average: 0.06, 0.04, 0.05
cpu3 NCPU: 8 load average: 0.00, 0.01, 0.05
cpu4 NCPU: 8 load average: 0.00, 0.01, 0.05
cpu5 NCPU: 8 load average: 0.00, 0.01, 0.05
cpu6 NCPU: 8 load average: 0.02, 0.04, 0.05
cpu7 NCPU: 8 load average: 0.00, 0.02, 0.05
cpu8 NCPU: 8 load average: 0.14, 0.06, 0.06
cpu9 NCPU: 8 load average: 0.00, 0.01, 0.05
cpu10 NCPU: 12 load average: 0.00, 0.01, 0.05
cpu11 NCPU: 12 load average: 0.00, 0.01, 0.05
cpu12 NCPU: 12 load average: 0.11, 0.04, 0.05
cpu13 NCPU: 12 load average: 0.00, 0.01, 0.05
cpu14 NCPU: 12 load average: 0.22, 0.14, 0.08
cpu15 NCPU: 12 load average: 0.00, 0.01, 0.05
cpu16 NCPU: 12 load average: 0.00, 0.01, 0.05
cpu17 NCPU: 12 load average: 0.06, 0.03, 0.05
cpu18 NCPU: 20 load average: 0.00, 0.01, 0.05
cpu19 NCPU: 8 load average: 0.00, 0.01, 0.05
cpu20 NCPU: 8 load average: 0.01, 0.02, 0.05
cpu21 NCPU: 8 load average: 0.05, 0.04, 0.05
cpu22 NCPU: 8 load average: 0.00, 0.01, 0.05
cpu23 NCPU: 64 load average: 0.00, 0.01, 0.05
gpu1 NCPU: 20 load average: 0.01, 0.08, 0.09
gpu2 NCPU: 16 load average: 0.00, 0.01, 0.05
gpu3 NCPU: 28 load average: 0.00, 0.01, 0.05
gpu4 NCPU: 20 load average: 0.07, 0.03, 0.05
gpu5 NCPU: 48 load average: 0.00, 0.01, 0.05
hzhao bash 0.0 0.0 2284
hzhao bash 0.0 0.0 2260
hzhao bash 0.0 0.0 2260
hzhao bash 0.0 0.0 2256
hzhao bash 0.0 0.0 2256
hzhao bash 0.0 0.0 2248
hzhao bash 0.0 0.0 2260
hzhao bash 0.0 0.0 2252
gpu6 NCPU: 20 load average: 0.00, 0.01, 0.05
jiahao bash 0.0 0.0 4688
xdu bash 0.0 0.0 2560
xdu bash 0.0 0.0 1456
xdu accelerate 0.0 0.0 281320
xdu python 0.0 1.0 4176744
jiahao bash 0.0 0.0 5100
xdu bash 0.0 0.0 2504
xdu bash 0.0 0.0 2424
gpu7 NCPU: 20 load average: 0.05, 0.07, 0.05
gpu8 NCPU: 20 load average: 1.15, 1.03, 0.93
kangrui python 100 0.7 2863820
kangrui slurm_script 0.0 0.0 1544
gpu10 NCPU: 6 load average: 0.00, 0.01, 0.05
gpu11 NCPU: 20 load average: 0.00, 0.01, 0.05
tenzorok python 395 10.5 20660752
tenzorok python 81.9 3.0 5901256
tenzorok python 45.7 2.6 5294664
tenzorok python 32.3 4.4 8664232
tenzorok jupyter-noteboo 0.5 0.1 211908
tenzorok bash 0.0 0.0 2332
tenzorok start_jupyter 0.0 0.0 1480
gpu12 NCPU: 16 load average: 0.00, 0.01, 0.05
gpu13 NCPU: 16 load average: 0.00, 0.01, 0.05
gpu14 NCPU: 16 load average: 0.00, 0.01, 0.05
gpu15 NCPU: 16 load average: 0.00, 0.01, 0.05
gpu16 NCPU: 16 load average: 2.56, 2.72, 2.83
sudarshan python 95.1 2.7 7363352
sudarshan python 94.3 2.7 7355548
sudarshan python 94.2 2.7 7349144
sudarshan wandb-service(2 3.6 0.0 125948
sudarshan wandb-service(2 3.6 0.0 126064
sudarshan wandb-service(2 3.4 0.0 125848
sudarshan slurm_script 0.0 0.0 1648
sudarshan slurm_script 0.0 0.0 1644
sudarshan slurm_script 0.0 0.0 1648
gpu17 NCPU: 16 load average: 0.34, 0.38, 0.40
tenzorok python 38.1 1.5 4073000
tenzorok python 5.8 1.0 2666416
tenzorok jupyter-noteboo 1.8 0.0 220036
tenzorok python 1.2 1.0 2812524
tenzorok bash 0.0 0.0 2344
tenzorok start_jupyter 0.0 0.0 1420
tenzorok python 0.0 0.0 60372
tenzorok python 0.0 0.0 58540
tenzorok python 0.0 0.0 58600
gpu19 NCPU: 8 load average: 0.23, 0.34, 0.39
xdu watch 0.9 0.0 2760
xdu bash 0.0 0.0 2388
xdu bash 0.0 0.0 2672
xdu watch 0.0 0.0 1076
xdu nvidia-smi 0.0 0.0 14180
gpu20 NCPU: 4 load average: 0.04, 0.06, 0.10
whc python 0.3 5.7 3752668
whc bash 0.0 0.0 2308
jjahn bash 0.0 0.0 2544
whc bash 0.0 0.0 2352
gpu21 NCPU: 4 load average: 0.08, 0.15, 0.13
marcelo python 1.4 1.5 1016600
marcelo slurm_script 0.0 0.0 1524
marcelo jupyter-lab 0.0 0.1 131056
marcelo bash 0.0 0.0 2124
marcelo python 0.0 0.0 60756
marcelo python 0.0 0.0 61172
gpu22 NCPU: 4 load average: 0.19, 0.20, 0.17
gpu23 NCPU: 4 load average: 0.13, 0.14, 0.14
jiahao jupyter-noteboo 0.0 0.1 74100
jiahao bash 0.0 0.0 1700
jiahao bash 0.0 0.0 2344
jiahao bash 0.0 0.0 3364
jiahao bash 0.0 0.0 368
jiahao bash 0.0 0.0 2196
jiahao bash 0.0 0.0 2456
jiahao nvim 0.0 0.0 20772
jiahao node 0.0 0.1 69076
jiahao node 0.0 0.1 89164
jiahao bash 0.0 0.0 2028
jiahao bash 0.0 0.0 1492
gpu24 NCPU: 8 load average: 0.01, 0.04, 0.05
gpu25 NCPU: 4 load average: 0.00, 0.01, 0.05
gpu28 NCPU: 20 load average: 0.03, 0.04, 0.05
jiahao bash 0.0 0.0 5132
jiahao bash 0.0 0.0 4288
gpu29 NCPU: 20 load average: 0.00, 0.01, 0.05
gpu30 NCPU: 20 load average: 0.00, 0.01, 0.05
gpu31 NCPU: 24 load average: 0.00, 0.03, 0.05
gpu32 NCPU: 24 load average: 0.00, 0.01, 0.05
gpu33 NCPU: 20 load average: 0.00, 0.01, 0.05
gpu34 NCPU: 8 load average: 0.00, 0.01, 0.05
gpu35 NCPU: 16 load average: 0.00, 0.01, 0.05
gpu36 NCPU: 20 load average: 0.00, 0.01, 0.05
gpu37 NCPU: 20 load average: 0.00, 0.01, 0.05
gpu38 NCPU: 20 load average: 0.02, 0.02, 0.05
gpu39 NCPU: 8 load average: 0.03, 0.09, 0.13

TTIC SLURM Cluster: GPU Statistics
Fri Dec 2 07:35:12 CST 2022

Node Fan SpeedTemp Perf State PowerMax MemoryMax GPU Usage User
gpu185%28CP814W140W19MiB16376MiB0%
gpu185%29CP813W140W19MiB16376MiB0%
gpu185%29CP814W140W19MiB16376MiB0%
gpu185%29CP817W140W19MiB16376MiB0%
gpu185%28CP812W140W19MiB16376MiB0%
gpu185%28CP815W140W19MiB16376MiB0%
gpu185%29CP813W140W19MiB16376MiB0%
gpu185%29CP813W140W19MiB16376MiB0%
gpu185%29CP815W140W19MiB16376MiB0%
gpu185%29CP814W140W28MiB16376MiB0%
gpu285%28CP814W140W19MiB16376MiB0%
gpu285%30CP817W140W19MiB16376MiB0%
gpu285%30CP817W140W19MiB16376MiB0%
gpu285%29CP814W140W27MiB16376MiB0%
gpu385%33CP816W250W10MiB11264MiB0%
gpu385%30CP827W250W10MiB11264MiB0%
gpu385%31CP829W250W10MiB11264MiB0%
gpu385%30CP824W250W10MiB11264MiB0%
gpu485%33CP818W140W19MiB16376MiB0%
gpu485%32CP817W140W19MiB16376MiB0%
gpu485%30CP816W140W19MiB16376MiB0%
gpu485%29CP816W140W27MiB16376MiB0%
gpu585%31CP822W300W19MiB49140MiB0%
gpu585%30CP819W300W19MiB49140MiB0%
gpu585%30CP819W300W19MiB49140MiB0%
gpu585%28CP831W300W19MiB49140MiB0%
gpu585%30CP824W300W28MiB49140MiB0%
gpu585%28CP830W300W19MiB49140MiB0%
gpu585%30CP822W300W19MiB49140MiB0%
gpu585%28CP824W300W19MiB49140MiB0%
gpu685%32CP820W300W38516MiB49140MiB0%xdu
gpu685%29CP819W300W21MiB49140MiB0%
gpu685%32CP819W300W21MiB49140MiB0%
gpu685%32CP821W300W21MiB49140MiB0%
gpu685%29CP819W300W21MiB49140MiB0%
gpu685%32CP821W300W21MiB49140MiB0%
gpu685%31CP817W300W21MiB49140MiB0%
gpu685%33CP821W300W30MiB49140MiB0%
gpu785%32CP822W300W19MiB49140MiB0%
gpu785%30CP819W300W19MiB49140MiB0%
gpu785%33CP819W300W19MiB49140MiB0%
gpu785%33CP815W300W19MiB49140MiB0%
gpu785%30CP818W300W19MiB49140MiB0%
gpu785%31CP818W300W19MiB49140MiB0%
gpu785%31CP823W300W19MiB49140MiB0%
gpu785%31CP829W300W28MiB49140MiB0%
gpu885%31CP818W300W19MiB49140MiB0%
gpu885%67CP2265W300W7972MiB49140MiB92%kangrui
gpu885%32CP818W300W27MiB49140MiB0%
gpu1085%31CP820W250W11MiB12288MiB0%
gpu1085%26CP812W250W11MiB12288MiB0%
gpu1185%33CP821W250W10MiB11264MiB0%
gpu1185%33CP817W250W10MiB11264MiB0%
gpu1185%33CP822W250W10MiB11264MiB0%
gpu1185%31CP820W250W10MiB11264MiB0%
gpu1285%29CP815W140W19MiB16376MiB0%
gpu1285%29CP814W140W19MiB16376MiB0%
gpu1285%30CP815W140W19MiB16376MiB0%
gpu1285%29CP819W140W27MiB16376MiB0%
gpu1385%36CP816W250W10MiB11264MiB0%
gpu1385%31CP825W250W10MiB11264MiB0%
gpu1385%33CP838W250W10MiB11264MiB0%
gpu1385%33CP833W250W10MiB11264MiB0%
gpu1485%29CP815W140W19MiB16376MiB0%
gpu1485%30CP814W140W19MiB16376MiB0%
gpu1485%29CP814W140W19MiB16376MiB0%
gpu1485%30CP814W140W27MiB16376MiB0%
gpu1585%32CP826W250W10MiB11264MiB0%
gpu1585%31CP818W250W10MiB11264MiB0%
gpu1585%30CP817W250W10MiB11264MiB0%
gpu1585%31CP824W250W10MiB11264MiB0%
gpu1685%73CP2218W300W18030MiB49140MiB46%sudarshan
gpu1685%82CP2224W300W20540MiB49140MiB25%sudarshan
gpu1685%64CP2148W300W17198MiB49140MiB12%sudarshan
gpu1685%35CP826W300W27MiB49140MiB0%
gpu1785%36CP827W250W1247MiB12288MiB0%tenzorok
gpu1785%36CP825W250W10MiB12288MiB0%
gpu1785%35CP827W250W2930MiB12288MiB0%tenzorok
gpu1785%36CP826W250W10MiB12288MiB0%
gpu1985%31CP824W260W18MiB24576MiB0%
gpu1985%28CP817W260W27MiB24576MiB0%
gpu2030%35CP820W230W16501MiB24564MiB0%whc
gpu2030%57CP066W230W0MiB24564MiB2%
gpu2133%43CP054W260W0MiB49152MiB0%
gpu2132%44CP053W260W0MiB49152MiB0%
gpu2230%58CP072W230W0MiB24564MiB0%
gpu2230%57CP069W230W0MiB24564MiB1%
gpu2333%42CP056W260W0MiB24576MiB0%
gpu2334%42CP055W260W0MiB24576MiB0%
gpu2485%23CP820W250W11MiB11264MiB0%
gpu2485%21CP811W250W11MiB11264MiB0%
gpu2521%38CP053W250W0MiB11264MiB0%
gpu2513%44CP053W250W0MiB11264MiB0%
gpu2885%29CP822W300W19MiB49140MiB0%
gpu2885%28CP824W300W19MiB49140MiB0%
gpu2885%30CP826W300W19MiB49140MiB0%
gpu2885%30CP823W300W19MiB49140MiB0%
gpu2885%28CP827W300W19MiB49140MiB0%
gpu2885%29CP815W300W19MiB49140MiB0%
gpu2885%29CP819W300W19MiB49140MiB0%
gpu2885%29CP817W300W28MiB49140MiB0%
gpu2985%32CP815W280W10MiB24576MiB0%
gpu2985%31CP87W280W10MiB24576MiB0%
gpu2985%30CP816W280W10MiB24576MiB0%
gpu2985%30CP87W280W10MiB24576MiB0%
gpu3085%31CP833W250W10MiB11264MiB0%
gpu3030%26CP811W250W1MiB11264MiB0%
gpu3027%29CP821W250W1MiB11264MiB0%
gpu3085%28CP810W250W10MiB11264MiB0%
gpu3085%27CP85W250W10MiB11264MiB0%
gpu3085%27CP819W250W10MiB11264MiB0%
gpu3085%28CP818W250W10MiB11264MiB0%
gpu3085%27CP821W250W10MiB11264MiB0%
gpu3185%34CP833W250W10MiB11264MiB0%
gpu3185%34CP846W250W10MiB11264MiB0%
gpu3185%32CP834W250W10MiB11264MiB0%
gpu3185%31CP841W250W10MiB11264MiB0%
gpu3285%32CP840W250W10MiB11264MiB0%
gpu3285%32CP828W250W10MiB11264MiB0%
gpu3285%32CP821W250W10MiB11264MiB0%
gpu3285%30CP834W250W10MiB11264MiB0%
gpu3385%32CP829W250W10MiB11264MiB0%
gpu3385%27CP820W250W10MiB11264MiB0%
gpu3385%29CP88W250W10MiB11264MiB0%
gpu3385%30CP83W250W10MiB11264MiB0%
gpu3385%26CP86W250W10MiB11264MiB0%
gpu3385%27CP84W250W10MiB11264MiB0%
gpu3385%30CP814W250W10MiB11264MiB0%
gpu3385%28CP825W250W10MiB11264MiB0%
gpu3427%34CP814W250W1MiB11264MiB0%
gpu3427%28CP84W250W1MiB11264MiB0%
gpu3585%37CP823W250W10MiB11264MiB0%
gpu3585%35CP836W250W10MiB11264MiB0%
gpu3585%32CP823W250W10MiB11264MiB0%
gpu3585%33CP826W250W10MiB11264MiB0%
gpu3685%33CP835W250W10MiB11264MiB0%
gpu3685%26CP817W250W10MiB11264MiB0%
gpu3685%31CP821W250W10MiB11264MiB0%
gpu3685%29CP86W250W10MiB11264MiB0%
gpu3685%27CP86W250W10MiB11264MiB0%
gpu3685%28CP819W250W10MiB11264MiB0%
gpu3685%28CP821W250W10MiB11264MiB0%
gpu3685%29CP818W250W10MiB11264MiB0%
gpu3785%33CP826W250W10MiB11264MiB0%
gpu3785%31CP826W250W10MiB11264MiB0%
gpu3785%38CP825W260W10MiB11264MiB0%
gpu3785%31CP833W250W10MiB11264MiB0%
gpu3885%31CP828W300W19MiB49140MiB0%
gpu3885%27CP825W300W19MiB49140MiB0%
gpu3885%32CP817W300W19MiB49140MiB0%
gpu3885%30CP816W300W19MiB49140MiB0%
gpu3885%29CP822W300W19MiB49140MiB0%
gpu3885%30CP818W300W19MiB49140MiB0%
gpu3885%28CP830W300W19MiB49140MiB0%
gpu3885%31CP824W300W28MiB49140MiB0%
gpu3928%60CP070W250W0MiB11264MiB0%
gpu3915%60CP064W250W0MiB11264MiB0%