Disk Performance
Overview: Shows the performance of the storage LUNs, aggregated into the Glade filesystems.
Ignore the negative values, this is due to plotting two data streams within a single graph.
Graph Descriptions
GLADE disks
| Item | Description |
|---|---|
| Horizontal Axis | time stamp |
| Vertical Positive Axis | Bytes read / second (units will auto-adjust) |
| Vertical Negative Axis | Bytes written / second (units will auto-adjust) |
| Positive Data | data above zero represents reads. Green: glade_scratch, Dark Blue: glade_p Light Blue: glade_u |
| Negative Data | data below zero represents writes. Green: glade_scratch, Dark Blue: glade_p Light Blue: glade_u |
| Black Line | Average rate of reads/write for the displayed time period. (need to look at more than 1 hour) |
| Reads Legend | Current read rate, total amount of reads in time period, average reads per sec in time period |
| Writes Legend | Current writes rate, total amount of writes in time period, average writes per sec in time period |
Scratch disks, /glade/u disks, /glade/p disks, EREBUS disks
| Item | Description |
|---|---|
| Horizontal Axis | time stamp |
| Vertical Positive Axis | Bytes read (units will auto-adjust) |
| Vertical Negative Axis | Bytes written (units will auto-adjust) |
| Positive Data | data above zero represents reads |
| Negative Data | data below zero represents writes |
| Black Line | Average rate of reads/write for the displayed time period. (need to look at more than 1 hour) |
| Reads Legend | total reads, reads in last hour, ave reads per sec |
| Writes Legend | total writes, writes in last hour, ave writes per sec |
glade100ib1 NSDs
| Item | Description |
|---|---|
| Horizontal Axis | time stamp |
| Vertical Axis | Bytes per second (units will auto-adjust) |
| Green | data coming into the NSD ports on the glade IB switch (reads from Glade) |
| Blue | data going out of the NSD ports on the Glade IB switch (writes to Glade) |
| In Legend | Current rate, total amount transferred in time period, average reads per sec in time period |
| Out Legend | Current rate, total amount transferred in time period, average reads per sec in time period |
CFDS NSD Cluster Network
| Item | Description |
|---|---|
| Horizontal Axis | time stamp |
| Vertical Axis | Bytes per second (units will auto-adjust) |
| Green | 10G Ethernet and IPoIB traffic coming into NSDs (writes to Glade) |
| Blue | 10G Ethernet and IPoIB traffic going out of NSDs (read from Glade) |
| In Legend | total amount transferred in time period |
| Out Legend | total amount transferred in time period |
Data Files
Data Source
Disk performance data is collected by a script running "iostat" running on each NSD. It reports reads/writes in KB every 15 seconds for the whole 15 second interval. It's split into the different filesystems (depending on the disk being measured) and sent to ganglia.
"glade100ib1 NSDs" data is collected by a script running "perfquery" running on glademgt1. It runs against the NSD ports on the Glade IB switch. So "in" shows traffic received by the switch ports sent from the NSDs, for example. NSD writes = switch port reads
"CFDS NSD Cluster Network" data is collected by ganglia's gmond running on each NSD. It reports IP-based network traffic (10G and IB) every 15 seconds. "In" shows traffic received by the NSD nic.
Ganglia Processing
Disk performance data is multiplied by 1024 (to convert KB to bytes) and writes are multiplied by -1 (to flip the plot for better readability).
Uses rrdtool's "last", "total", and "average" functions to create the legends on the graphs.