New Server - Up and running

General Discussion about Meteoplug server
Post Reply
User avatar
admin
Site Admin
Posts: 332
Joined: Sun Feb 07, 2010 9:22 pm

New Server - Up and running

Post by admin » Wed Oct 10, 2012 8:19 am

Migration to new server went fine. As far as I can see there are no data losses even for time of migration. If you experience unexpected problems with graphing please drop a mail to "info(at)meteoplug.com".

As told before basic motivation to change server hardware was the "lagging behind" problem of data import and computation when server gets heavy load doing backup and normal operation in parallel. As you can see from the "server status" section on the wiki the situation has now completely changed, "data lagging behind" is history :mrgreen:

Below you see the server stats of the old server. You see a first peek in CPU load about midnight. This is caused by various jobs starting for maintenance (lvmsnapshot, doing a compressed tar of 700GB auf weather data, scrubbing old data from the database). The measure for speed of data processing is given by the "Import Age" value, which shows how long it takes for new data coming in to be scheduled for processing. As you can see this rose up to 300 minutes. So when when you looked at recent weather graphs during that period you saw data 5 hours old. Next day the server recovers slowly from that (import age going down again). All in all not a good situation which needed to be resolved.

Old server:
alt.png
alt.png (67.62 KiB) Viewed 8312 times

As explained in the previous thread I did build a specialized server that can handle much more IO in parallel by using fast SSDs. The database resides now on 1.5 TB of SSD space, which gives an enormous IO boost especially when it comes to random IO, which is basically the challange the server has to face each night. When you look at the new results, the improvement is staggering. When maintenance jobs do start at midnight, CPU load goes up close to 800% which is exactly the full load a 4 core double threaded core can handle fine. Doing the CPU intensive backup compression is finished after about 3 hours. Druing that time the "Import Age" remains at 0-1 minutes, so there is absolutely no lagging behind. You can also see that the length of the queue of data sets from Meteoplug clients to be imported remains below 5 (compared to about 700 on the old system). To say it in just a few words, this new machine eats all your data with ease and likes to get much more :D

New server:
neu.png
neu.png (54.43 KiB) Viewed 8312 times

Nevertheless, I have to say thanks to the old hardware as it ran well for years and did never gave me a headache because of hardware faults. The HDDs took a tremendous load, got never time to sleep and performed bravely at their limits. Hopefully, the new server will be as reliable. From the data monitor regarding system health, CPU and disk temperatures are well inside specs, even when going at full throttle. 70°C is a fine peak core temp for a core i7 and far away from physical limits. Wearing of SSDs also looks fine. The used SSDs from Samsung (model 830) are known to take 20.000 wear level counts. Being below 20 wear level counts after 2-3 weeks of testing with full data load (by mirroring operation of the old server for testing the new setup) looks also very fine. You can watch this data ongoing when you scroll down on "Server Status" page on Meteoplug Wiki.

New server - health:
health.png
health.png (33.92 KiB) Viewed 8311 times

Stormedy
Posts: 15
Joined: Fri Dec 30, 2011 12:52 am

Re: New Server - Up and running

Post by Stormedy » Wed Oct 17, 2012 10:32 pm

....the new server really works fine - congratulations!
I wonder since quite a while I have the following notes on my meteoplug-station server log:

warning: Oct 17 23:44:53 psi METEONET 00:B3:F6:00:5E:9F: awekas-push: no response from awekas
warning: Oct 17 23:44:53 psi METEONET 00:B3:F6:00:5E:9F: regiowetter: no response from regiowetter
info: Oct 17 23:44:53 psi METEONET 00:B3:F6:00:5E:9F: Feeded networks: awekas-pull, meteonews

but Awekas and Regiowetter get my weather informations every 10 minutes - where is the bug for this message ?

Ed

Post Reply