As told before basic motivation to change server hardware was the "lagging behind" problem of data import and computation when server gets heavy load doing backup and normal operation in parallel. As you can see from the "server status" section on the wiki the situation has now completely changed, "data lagging behind" is history
Below you see the server stats of the old server. You see a first peek in CPU load about midnight. This is caused by various jobs starting for maintenance (lvmsnapshot, doing a compressed tar of 700GB auf weather data, scrubbing old data from the database). The measure for speed of data processing is given by the "Import Age" value, which shows how long it takes for new data coming in to be scheduled for processing. As you can see this rose up to 300 minutes. So when when you looked at recent weather graphs during that period you saw data 5 hours old. Next day the server recovers slowly from that (import age going down again). All in all not a good situation which needed to be resolved.
Old server:
- alt.png (67.62 KiB) Viewed 8312 times
As explained in the previous thread I did build a specialized server that can handle much more IO in parallel by using fast SSDs. The database resides now on 1.5 TB of SSD space, which gives an enormous IO boost especially when it comes to random IO, which is basically the challange the server has to face each night. When you look at the new results, the improvement is staggering. When maintenance jobs do start at midnight, CPU load goes up close to 800% which is exactly the full load a 4 core double threaded core can handle fine. Doing the CPU intensive backup compression is finished after about 3 hours. Druing that time the "Import Age" remains at 0-1 minutes, so there is absolutely no lagging behind. You can also see that the length of the queue of data sets from Meteoplug clients to be imported remains below 5 (compared to about 700 on the old system). To say it in just a few words, this new machine eats all your data with ease and likes to get much more
New server:
- neu.png (54.43 KiB) Viewed 8312 times
Nevertheless, I have to say thanks to the old hardware as it ran well for years and did never gave me a headache because of hardware faults. The HDDs took a tremendous load, got never time to sleep and performed bravely at their limits. Hopefully, the new server will be as reliable. From the data monitor regarding system health, CPU and disk temperatures are well inside specs, even when going at full throttle. 70°C is a fine peak core temp for a core i7 and far away from physical limits. Wearing of SSDs also looks fine. The used SSDs from Samsung (model 830) are known to take 20.000 wear level counts. Being below 20 wear level counts after 2-3 weeks of testing with full data load (by mirroring operation of the old server for testing the new setup) looks also very fine. You can watch this data ongoing when you scroll down on "Server Status" page on Meteoplug Wiki.
New server - health:
- health.png (33.92 KiB) Viewed 8311 times