Topic: The network issues thread (client lag and disconnects)

My name is DEV Karon, and from now on I will be working on improving the network and server infrastructure of Perpetuum. As our game world gets bigger, with more and more people joining every day from the farthest reaches of the planet, it has become an important priority to provide everyone with a high level of service and be able to understand and solve any possible problems that may arise. With a 15 year background in web and client-server development, operation and hosting, not to mention being an avid MMO player, I’m hopefully up to the challenge.


The lag detective
To further enhance the gameplay experience of Perpetuum for everyone, we have decided to launch an investigation into any sustained network problems our users might experience. The results we get out of this will influence a lot of technical and financial decisions of our company, including a possible relocation of our servers and the design of our future network infrastructure.

But before we dive into the details, it’s important to clear up a couple of facts that define our current situation.


The what’s up
1. The absolute majority of our users are fine - they don’t experience unusual lag or an excessive number of disconnected sessions. A little lag or a single disconnect happens from time to time for every internet based service on planet Earth, this is just a fact of life on the 'net.

2. Our servers, network equipment, the data center we are located in and the network service we buy are all world class. Internet connectivity and data centers are like LEGO: the building blocks of hardware (routers, switches and rackmount servers) and service levels (bandwidth, traffic limits and uptime guarantees) are just about the same anywhere in the world, be it London, UK or Budapest, Hungary. Rest assured we only buy from the best on the market.

3. Packet routing problems (suboptimal routing decisions, packet loss and black hole routers) affect every global internet based service (regardless of it being located in London, Budapest or Hong Kong), but are more easily detected when using latency-sensitive applications like realtime gaming.

4. Latency is king, and we know it. The difference (e.g. for a North American user) between our servers being in Budapest or London basically boils down to distance (number of hops) and latency (ping times). Budapest is a little bit further down the pipe (an extra 30-40 ms) which should not be perceived during normal gameplay. Of course with distance traveled and routers used, packet round-trip times gradually increase, so does the tiny chance of error somewhere along the path, but apart from that there is nothing inherently wrong in being hosted here. Budapest is of course much more economical for us while we are growing, so planning any change in this matter needs to be well thought through.

5. The single Hungarian peering point has absolutely nothing to do with how our international users access our services.The BIX peering point in Hungary connecting all the national ISP’s is created and maintained by the providers themselves, and it’s sole purpose is to exchange inland traffic - all the big service providers have their own, separate international connections with backup links.


What we plan to do
All that said we take network problems very seriously, being fully aware of the fact that playing an MMO with real-time combat needs to run smooth for every active user. We are planning to deploy extensive monitoring and analysis technology to understand and solve the specific problems better, and to make future decisions that will improve the level of service to our users. Most of it is going to happen behind the curtains, via automated processes gathering statistics during regular gameplay, but we are also open to receiving your input on this matter. On a case-by-case basis we may even contact you directly and ask for your help checking out a particular issue.


When is this relevant to you
You should reply to this thread if you experience a serious network issue with Perpetuum that is persistent (happening often, e.g. on a daily basis), and still ongoing (not something from last month), like:
- unnecessarily high lag (huge delays and rubberbanding), or
- frequent client disconnects with no relation to the quality of your internet service, or maybe
- some particularly evil combination of the above.


How can you help
In your reply first give us an accurate description of the problem you are having, for example:
“I am experiencing lag and frequent disconnects for 3 days now, one every 20-25 minutes”

In addition to that please share with us the following:
- your username
- your current IP address (use whatismyip.com)
- name of ISP and country
- the bandwidth (e.g. 15 Mbit down / 1 Mbit up) and
- the kind of service you are using (e.g. DSL, Cable, Optical or 3G Mobile)
- the results of a recent speedtest.net run

We also need you to:
- download and install Pingplotter from pingplotter.com and launch two (2) instances of it
- set  “# of times to trace” to Unlimited, “Trace interval” at 3 seconds and “Samples to include” to ALL,
- start tracing test1.perpetuum-online.com (at the game server location) and test2.perpetuum-online.com (a fairly fast host at a different ISP for comparison purposes) in the two respective Pingplotter instances,
- please pause all uploads and downloads during the course of this test,
- now start the Perpetuum client and play for at least 30 minutes,
- if you notice any disconnects, write down your local time, and resume playing,
- when you are done playing for 30 minutes, go to File menu -> Save Sample set in both Pingplotter instances,
- attach the two .pp2 files to your reply.

If you prefer to keep your results private, you can send your problem description, ISP information and Pingplotter files to the netsupport@perpetuum-online.com email address.

Re: The network issues thread (client lag and disconnects)

Edit: Please keep this thread focused on the topic. The 'When is this relevant' section of the post clearly indicates whether it is suggested to post here. - DEV Calvin

3 (edited by The Ghost of Guns nButter 2011-05-25 09:30:54)

Re: The network issues thread (client lag and disconnects)

2 members of my corp have indicated that they were experiencing ~50% packet loss on the frankfurt hop. They don't feel like posting in this thread so I figured I'd just put a stickynote on the table for them smile

edit: I myself am sitting around a pleasant 150 ms ping. On rare occasions my client crashes but that does not appear to be caused by the internet.

Re: The network issues thread (client lag and disconnects)

All in all my conectivity is usually fine.  There have been a few bad times but that coud be my ISP.  Once issue that I have on a regular basis is with auto run.  I play almopst every day.  It seems one or 2 days of the week auto run enters periods where it stops every 20 - 50 feet.  Start it again, runs 50 feet or so and stops, repeat, repeat.  When this occurs the game is unplayable except for mining (assuming you have the patients to get to a mining spot).  Can you do somthgin on the coding side so it is not so sensitive to the "blips"?

Re: The network issues thread (client lag and disconnects)

Everdead i wrote:

All in all my conectivity is usually fine.  There have been a few bad times but that coud be my ISP.  Once issue that I have on a regular basis is with auto run.  I play almopst every day.  It seems one or 2 days of the week auto run enters periods where it stops every 20 - 50 feet.  Start it again, runs 50 feet or so and stops, repeat, repeat.  When this occurs the game is unplayable except for mining (assuming you have the patients to get to a mining spot).  Can you do somthgin on the coding side so it is not so sensitive to the "blips"?

you're definitely not the only one and they're aware of it .. it's because the movement is server side to prevent hacked clients from teleporting or speed running everywhere so any little hiccup in your connection with the server stops you cold. 

it is absolutely infuriating.  there are several hops between where i am in the US and the servers in Hungary that are the issue and i know they're basically the same for everyone if you look at some of the other threads about lag and disconnects.

Population graphs

<GM Synapse> please don't abuse our fresh players before blowing them up. And for god sakes, don't do that after it!

Re: The network issues thread (client lag and disconnects)

We added a diagnostic feature in the client that can help us find out where the network congestions are:

1. Open the console (`/~ key by default, can be reassigned in the keyboard settings)
2. Enter openprofiler to open the debug window.
3. Select the tab "traceroute".
4. Press the TRACE button and wait until the process finishes. (It will say "Statistics sent." once it finished.)

What this will do is perform a standard ICMP traceroute and send the data to our server, after which the server will do the same internally to get statistic data from both routes.

Hopefully this will highlight where the failing routers are.

[14:15:15] <Freya Sabbat> ...Dear god, the Devs are as bad as us

Re: The network issues thread (client lag and disconnects)

DEV Gargaj wrote:

We added a diagnostic feature in the client that can help us find out where the network congestions are:

1. Open the console (`/~ key by default, can be reassigned in the keyboard settings)
2. Enter openprofiler to open the debug window.
3. Select the tab "traceroute".
4. Press the TRACE button and wait until the process finishes. (It will say "Statistics sent." once it finished.)

What this will do is perform a standard ICMP traceroute and send the data to our server, after which the server will do the same internally to get statistic data from both routes.

Hopefully this will highlight where the failing routers are.

Will do, thanks for adding it.

Population graphs

<GM Synapse> please don't abuse our fresh players before blowing them up. And for god sakes, don't do that after it!

Re: The network issues thread (client lag and disconnects)

I think you should add the trace as a right click menu item directly onto the lag meter in the lower corner, using the console seems easy enough to me as a cisco engineer, but not everyone is used to typing in commands while playing games.

Re: The network issues thread (client lag and disconnects)

sometimes i wished i could change the opacity of the profiling window and "embed" it like an external overlay - to give me data but not interfere with mouse imput.

I have tried that tracing route while i had lag issues, but with a steady ping of 60 (while moving around)

1. xxx.xxx.xxx.xxx - 0 ms - 11013 (That was my home network IP)
2. 87.186.224.190 - 42 ms - 11013 (Hop in germany)
3. 87.186.253.114 - 48 ms - 11013 (hop in germany)
4. 62.154.15.93 - 54 ms - 11013    (hop in hungary)
5. 195.228.152.152 - 2097172 ms - 11010 (perpetuum server - crazy time)
6. 81.183.0.82 - 58 ms - 11013     (hop in hungary)
7. 81.183.0.35 - 59 ms - 11013     (hop in hungary)
8. 81.183.2.225 - 60 ms - 11013   (hop in hungary)
9. 84.2.225.113 - 59 ms - 11013   (hop in hungary)
10. 195.228.152.152 - 59 ms - 0   (Perpetuum Server again)

can someone explain me this?

*Disclaimer: This post can contain strong sarcasm or cynical remarks. keep that in mind!
Whining - It's amazing how fast your trivial concerns will disappear

Re: The network issues thread (client lag and disconnects)

Annihilator wrote:

sometimes i wished i could change the opacity of the profiling window and "embed" it like an external overlay - to give me data but not interfere with mouse imput.

I have tried that tracing route while i had lag issues, but with a steady ping of 60 (while moving around)

1. xxx.xxx.xxx.xxx - 0 ms - 11013 (That was my home network IP)
2. 87.186.224.190 - 42 ms - 11013 (Hop in germany)
3. 87.186.253.114 - 48 ms - 11013 (hop in germany)
4. 62.154.15.93 - 54 ms - 11013    (hop in hungary)
5. 195.228.152.152 - 2097172 ms - 11010 (perpetuum server - crazy time)
6. 81.183.0.82 - 58 ms - 11013     (hop in hungary)
7. 81.183.0.35 - 59 ms - 11013     (hop in hungary)
8. 81.183.2.225 - 60 ms - 11013   (hop in hungary)
9. 84.2.225.113 - 59 ms - 11013   (hop in hungary)
10. 195.228.152.152 - 59 ms - 0   (Perpetuum Server again)

can someone explain me this?

im telling you anyone from the US gonna run into problems in germany. Seems alot is in frankfurt, huge packet losses. As far as what happened to you, could just be random. I will post a screenshot of my average ms with one account, around 250+ normal, good day it is 220. Location is twin Falls, id  30mb down/15mb upload is my current connection.

11 (edited by Malpherius 2011-06-16 19:31:25)

Re: The network issues thread (client lag and disconnects)

Okay, I had another thread open in Q&A and still have problems. After speaking to DEV CRM last night, I agreed to submit data as requested.

I live in the UK and my ISP is British Telecom (BT). I have a traditional router through a phone line setup. My internet connection remains reliable and fast during a PO connection/lag/disconnect issues. During the test, I was the only one in the house using the internet to the best of my knowledge. For the purposes of the test results, I chose not to use a VPN to show a true through route to the server and the issues on the hops.

Ping plotter detected a lot of packet loss as I approached hops near the PO server. After perhaps 18 minutes of playtime, I lost connection. When attempting to reconnect, I was immeadiatly kicked (after selecting agent). The client reconnected, then I selected agent and the client paused with a full loading bar - ping plotter showing full packet loss. I waited out 30mins of "play time" (okay so 12mins of watching a loading bar). I will email the ping plotter files as requested.

Speedtest.net results:
PING 44ms
DL Speed: 13.80 Mbps
UL Speed: 0.80 Mbps

Let me know if you want more data.

(Ninja edit)
18.57 - GMT+0 (UK) - 16/06/11 - Client disconnected.
18.58 - GMT+0 (UK) - 16/06/11 - Failed trying to log in after selecting agent.

Re: The network issues thread (client lag and disconnects)

There have been many cases of disconnects and lag associated with being on the island of Tellesis.  I have had a few of those disconnects but others have had them much more frequently to the point they were afraid of engaging in combat without sufficient backup.

This issue also seems to be coupled with an inability to reconnect on your first attempt thereafter.  You will be removed from the game at the time of the failed reconnect, but its likely you will not get back into the game until the second try.  This seems to indicate that your session is not expiring and the server doesn't know what to do when you try to connect another time and boots you automatically.

I have not experienced such problems on any other islans, so it seems this is an island specific problem.

->You just lost The Game<-

Re: The network issues thread (client lag and disconnects)

Neoxx wrote:

There have been many cases of disconnects and lag associated with being on the island of Tellesis.  I have had a few of those disconnects but others have had them much more frequently to the point they were afraid of engaging in combat without sufficient backup.

This issue also seems to be coupled with an inability to reconnect on your first attempt thereafter.  You will be removed from the game at the time of the failed reconnect, but its likely you will not get back into the game until the second try.  This seems to indicate that your session is not expiring and the server doesn't know what to do when you try to connect another time and boots you automatically.

I have not experienced such problems on any other islans, so it seems this is an island specific problem.

I am on the ASINTEC Alpha terminal island, so its not island specific, although you are experiencing a similiar issue.

Re: The network issues thread (client lag and disconnects)

I just noticed youre on British Telecom; we have 3 people in the corp who start suffering from random disconnects and Lag N/A as soon as its peak-time in UK (18:00 GMT 0+ onwards).

[18:20:30] <GLiMPSE> Chairman Of My Heart o/
CIR Complaint Form

The Imperial Grand Wizard of Justice

15 (edited by Malpherius 2011-06-17 13:50:48)

Re: The network issues thread (client lag and disconnects)

Syndic wrote:

I just noticed youre on British Telecom; we have 3 people in the corp who start suffering from random disconnects and Lag N/A as soon as its peak-time in UK (18:00 GMT 0+ onwards).

I happen to be leaving work early to head to town, so I will be home around 13:20 GMT +0... I'll have a quick look to see what the lag/connection situation is before I head out.

UPDATE: Tried logging in and running two instances of Ping plotter... Hops and packet loses are the same, Lag N/A when logged in, kicked after a few minutes at 13:49 GMT +0. Guess its nothing to do with peak time in the UK for BT.

Re: The network issues thread (client lag and disconnects)

Did the problem go away or is it persistent since you first reported it?

17 (edited by Malpherius 2011-06-22 15:50:01)

Re: The network issues thread (client lag and disconnects)

Exactly the same issues. Can connect more often and stay on for a little longer, but still not safe to play without risking loosing my bot or not being able to login at all after a disconnect.

Ping Plotter indicates that there is a lot of packet loss all the way to Hungary, with most hops being over BT kit. (Although total loss accurs in Hungary mostly). My ISP subscription is up soon and my house mates have agreed to change ISP, so I'll keep my fingers crossed.

If there are any UK players reading this thread - Could you tell me what ISP you use error/lag free? Muchas Gracias!

Malph.

18 (edited by Gharl Incognito 2011-06-25 13:32:06)

Re: The network issues thread (client lag and disconnects)

I can't tell you any of my IP info because it's all written in Chinese and I can't read it.

The issue I've been having is that my lag skyrockets, everything freezes except my robot which can walk around for 2 mins and then, bam, disconnect.  I reconnect without a problem minus one robot. 

This has happend a few times, two in particular in which I lost a bot, will stay in my mind.  I spoke to one of the Devs in game about it and was lucky enough to get my wreck back.

I've send crash and traceroute reports.  Can someone tell me if there is any point in my trying to play this game or will the disconnects just keep flowing. 

Tried to run those programs you linked, the download was blocked, ask China why.

EDIT: removed constand in game death rage

Sociorum, inimicos, omnes

-:does speak for NSA on the forums:-

19 (edited by Eitan Cadmus 2011-06-28 10:12:34)

Re: The network issues thread (client lag and disconnects)

Disregard, apparently the server had a heartattack earlier.

Re: The network issues thread (client lag and disconnects)

Question Karon:

With most routers intentionally designed to answer packets destined for themselves at a considerably lower priority (lower than lowest? smile) than packets going through them... how does pingplotter help with that?

This is often the reason why, when you graphically plot pings, you will find that places in the middle can take longer than getting to the destination. This, of course, should make you doubt the results in the middle, since logically the sum of the whole will always be greater than the sum of a half. As a result, fluctuation of the RTT in the middle should be dismissed as misleading information.

Re: The network issues thread (client lag and disconnects)

Don Knotts wrote:

Question Karon:

With most routers intentionally designed to answer packets destined for themselves at a considerably lower priority (lower than lowest? smile) than packets going through them... how does pingplotter help with that?

Don, we are aware of the limitations in ICMP. Still, running Pingplotter over a longer period of time (with data averaged out) provides some useful insight about particularly bad links and routers, not to mention overlapping data from different user sessions, which is going to help identifying the real issues by providing statistical significance.

We are also very interested in seeing the route itself, as there are cases where packets reach us over a less optimal route (more hops and/or unreliable links) compared to how packets from our end get may routed to that particular ISP.

We need more people sending us the pingplotter files (along with their network info, as outlined in my OP), but we are also gathering data through other means to complement that.

Re: The network issues thread (client lag and disconnects)

While i was reading your forums i can across a post where people in Australia were saying they hve problems, I myself live in Australia in the Southern Highlands about 1h 30 mins drive from Sydney. With ADSL2 testing the way you prescribed on both test 1 and test2 without using any there party apps my pings were respectively  an average of 379 on test serv1 and 376 on test serv2.

Just thought i'd slip this in to give you a rough idea, generally the game plays well for me with the odd dropout, hope it is usefull in some way cheers.

By the way love the game keep up your great efforts cheers

Re: The network issues thread (client lag and disconnects)

I have already emailed my ping plotter results to you guys, however to post here i have been having strange results after the last patch where i get a ping in game from 350-400ms jump to 1500 or more at random intervals causing major rubber banding.
Before patch i had a stable 400-450ms and great game play sad maybe the extra players have done it not sure.
Ping plotter shows a stable ping of around 377ms however on the second address i get a 35% packet loss at one connection or two before you IP.

24 (edited by adriaans 2011-07-02 06:30:02)

Re: The network issues thread (client lag and disconnects)

I'm in the UK (stafford to be ecact), stable ping at 40, however in a several hour long playsession I will get random massive ping spikes (3000 to 7000) (packet loss?) where nothing in the game happens, then suddenly the game 'jumps' back to where it should be as if nothing happened.  no disconnects.
This has usually happened when looting while shooting, though this may just have been a coincidence. (update: seems it was coincidence).
I'm on a 50MB fiber connection (50 down, 5-10 up). Virgin media being the provider.

My brother in Norway has the same issue, though Norway ISP's suck and he only has 1/1 MB (and gets less than what is paid for). Agder bredband is the provider.

Do you wish for me to collect any data? It only happens every few hours...

edit: used the ingame traceroute right after such a spike:

1. 86.28.124.1 - 8 ms - 11013
2. 62.255.32.177 - 10 ms - 11013
3. 213.105.103.97 - 9 ms - 11013
4. 62.253.185.85 - 15 ms - 11013
5. 213.105.159.34 - 13 ms - 11013
6. 62.253.184.2 - 15 ms - 11013
7. 130.117.14.141 - 75 ms - 11013
8. 130.117.50.117 - 195 ms - 11013
9. 130.117.50.157 - 24 ms - 11013
10. 154.54.39.198 - 30 ms - 11013
11. 130.117.50.246 - 34 ms - 11013
12. 154.54.39.6 - 48 ms - 11013
13. 130.117.51.174 - 83 ms - 11013
14. 154.54.37.133 - 51 ms - 11013
15. 149.6.182.42 - 53 ms - 11013
16. 81.183.0.96 - 46 ms - 11013
17. 81.183.0.45 - 49 ms - 11013
18. 81.183.2.225 - 49 ms - 11013
19. 84.2.225.113 - 49 ms - 11013
20. 195.228.152.152 - 49 ms - 0

Statistics sent.

Re: The network issues thread (client lag and disconnects)

I am an Australian player with an adsl2+ connection with adam internet ,  I have run pingplotter over a number of days and have found there to be a couple of commone ip's causing issues, but my main m/s drop occurs through germany. I will send the pingplotter results to the email listed above, but the main  ip ive noticed is 194.25.210.178