Lag: Experimental Results and A Few Pokes with a Stick at the Black Box | Support | Forum

A A A

Please consider registering
guest

sp_LogInOut Log In sp_Registration Register

Register | Lost password?
Advanced Search

— Forum Scope —




— Match —





— Forum Options —





Minimum search word length is 4 characters - maximum search word length is 84 characters

sp_Feed Topic RSS dirt
Lag: Experimental Results and A Few Pokes with a Stick at the Black Box
Topic Rating: +1 (1 votes) 
June 30, 2013
5:25 pm
robplusplus
Member
Members
Forum Posts: 94
Member Since:
March 28, 2013
sp_UserOfflineSmall Offline
1sp_Permalink sp_Print
+1

I've carried out a few experiments to narrow down the cause of the lag. Here are the results:

 

1. Although the worst lag spikes seem to correlate with players logging in, this fails to explain other large lag spikes or the more pervasive low-level lag we've been experiencing (which is worse on the weekend).

2. Login lag spikes seem to be longer when donators log in, though I have yet to determine whether this is true on a consistent basis.

3. The 'complexity' of the chunks loaded during a login lag spike have no correlation to the lag spike's length or intensity. When I log in, I frequently cause lag spikes so severe that a server-wide lag-out occurs although my entity count (E-value) is 0/1 and I have no redstone circuitry running. One possible exception to this is Residence; the res for my island consists of 18 connected areas. Granted, Highmoor has a significantly more complex res than I do and LWK's login lag spikes aren't as severe as mine tend to be.

4. One factor that I recently uncovered is Dynmap use during login. I logged in 4 times last night, leaving more than 30 seconds between each logout and the following login to allow the chunks to be unloaded from memory. Emulated observed the duration of each lag spike.

  • Trial 1: I logged in with Dynmap running in Firefox 21.0. Two lag spikes occurred almost sequentially, each ~5 seconds in duration.
  • Trial 2: I logged in without Dynmap running. No lag spikes occurred.
  • Trial 3: I logged in with Dynmap running. One lag spike occurred, ~5 seconds in duration.
  • Trial 4: I logged in without Dynmap running. No lag spikes occurred.

5. Certain mcMMO events have long been known to cause lag spikes and client/server discrepancies (e.g., invisible blocks) to occur. These include Tree Feller (even on only 4 giant jungle trees) and Giga Drill Breaker (likely due to the high rate of blocks being excavated and dropped).

 

There are a number of potential bottlenecks to server performance. The most likely suspects are processing power, available memory, and network bandwidth. Although I've never seen the specs for our server's dedicated hardware, I am confident that the demands for processing power and memory are being well met. This leaves bandwidth, which can be further dissected into web bandwidth, server hosting bandwidth, and internal bandwidth.

I was surprised to discover recently (thanks to a comment from Meat) that VeteranCraft's web services are hosted by GoDaddy.com. Low-end web hosting may have met our needs in the early days but we are growing both in number and sophistication of player. Frequently, I have found that the main page and forums load very slowly or not at all. I had used domain.com for my old server's needs until the server hosting company I was using commented on the slow page loads and agreed to assume our web hosting at the same low price as domain.com. The results were immediate and excellent.

This brings to the matter of our server hosting company, with which I am admittedly unfamiliar. Granted, it is nearly impossible to block a DDOS attack; I wouldn't expect what I assume to be a company that hosts game servers to have that level of protection. My suggestion is that we migrate the server away from a gaming-based hosting company to a more business-oriented one. The low-level pervasive lag we have been experiencing mainly on the weekend may be due to our current provider's customers collectively using up all of the provider's bandwidth; a business-oriented provider would likely have its peak bandwidth demands during business hours on weekdays, making an ideal fit with our own need for more bandwidth at night and on the weekend. As for the DDOS attacks that have indirectly harmed us recently, I believe that we deserve better than to be "collateral damage" because some kid sucks at Call of Duty.

I use the term "internal bandwidth" to refer to the movement of data within our server's hardware. Specifically, I suspect that permissions handling and SQL may be responsible for some of the issues we have. Permissions plugins (e.g., the excellent PEX) must not only function in real-time (to prevent hacking during the first few seconds after restart) but also be loaded so that plugins that require permissions-related information (e.g., permission groups to determine username color) can run properly. SQL is a database language that is very popular with plugin authors and is essential for certain combinations of plugins (e.g., iConomy requires SQL to avoid flatfile data corruption if used with Votifier due to Votifier's use of multi-threading), so I assume that VeteranCraft uses it to some degree. Despite being fairly powerful and moderately user-friendly, it is notoriously slow, especially if the SQL implementation is hosted on separate hardware from the actual server (which is entirely possible with some server hosting services). If a plugin is waiting for a response from an SQL server, this could easily choke the main thread of execution and cause significant lag without throwing a single exception in the console. There is a plugin called NoLagg which has, among many clever functions, the ability to throw an exception if the main thread of execution is hung up for a significant amount of time or if a plugin calls for an object from a thread other than the main one. When the exception is thrown, the offending plugin is named as part of the exception's stacktrace. Imagine my surprise when I found out that the normally efficient WorldGuard plugin was killing my old server!

 

I understand that the elimination of lag is the Holy Grail of server administration and I understand the difficulties that the Tech Team is currently facing. I hope that this post will help in some small way.

July 1, 2013
5:24 pm
frelling
Senior Tech
Forum Posts: 3264
Member Since:
August 18, 2011
sp_UserOfflineSmall Offline

Rob, thanks for the in-depth info.The short and sweet of it is that there is a race/deadlock condition that rears its ugly head more frequently now. Our usual suspects - Residence, mcMMO, and DynMap - are clearly involved but none of them are the singular source. Even monitoring and digging into the bowels of the JVM with VisualVM and Profiler are leaving us baffled as to the exact source. So far we've limited ourselves to non-invasive debugging so as not to impact the availability of our server, but that is coming to an end.

lol, no we are not hosting at GoDaddy, only Meat's website for VC is located there. We've intentionally steered away from game hosting companies and are using HiVelocity. They do host a few larger Minecraft servers, but its not their mainstay business. As far as server its a Xeon E2-1230v2 with 16GB, 2x1TB SATA III HDD, 120 GB SATA III SDD.

As far as network speed, we've got full duplex 1Gbps to the local network; however throughput is capped by the router at 100 Mbps both ways. So far we haven't even come close to maxing in either direction. On a busy day outgoing network rates are 3-4 Mbps. SQL and other management traffic is handled via a virtual switch and never leaves the machine.

Human beings, who are almost unique in having the ability to learn from the experiences of others, are also remarkable for their disinclination to do so. - D. Adams
Forum Timezone: America/New_York

Most Users Ever Online: 442

Currently Online:
17 Guest(s)

Currently Browsing this Page:
1 Guest(s)

Top Posters:

Emulated: 3206

ryanpitts: 1300

Dalferes: 747

Pherian: 660

Okarim: 594

Member Stats:

Guest Posters: 10

Members: 13091

Moderators: 3

Admins: 2

Forum Stats:

Groups: 8

Forums: 45

Topics: 6229

Posts: 27411

Newest Members: RainoldsEn, BarbaracoN, frequency-drive.ru LYI53, frequency-drive.ru YYU10, BundlesBek, frequency-drive.ru SCD50, Huongokwicle, frequency-drive.ru CCP03, frequency-drive.ru ISB51, AlfredoAcide, Jamessax, DonaldPsymn, frequency-drive.ru VHX75, riobetgammerRB, frequency-drive.ru IEC60, Josephnor, JerryNof, Apefautt, Warrentourn, parimatch777

Moderators: terrorisly: 424, mudwog: 127, LightWarriorK: 2154

Administrators: meatbawllz: 2475, frelling: 3264