Co-op out-of-sync issues, trying to find best solution

Have a question, suggestion, or comment about Aleph One's features and functionality (Lua, MML, the engine itself, etc)? Post such topics here.
Post Reply
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

I have recently started playing Co-op with my friend (he is in Finland, I'm in America) and we play over Hamachi. We've been able to play Infinity a few times getting up to the final chapter, but in doing so I am finding a lot of networking issues that are making playing a bit less pleasant. I wanted to ask about them and see if there is a solution to reduce their likelihood of occurring.

First, we use Irons Co-Op script, which by the way is bloody awesome. However, sometimes we go out of sync. We have never had a 'freeze' or a 'hang', just a sync issue. One time when we played (when I was messing with the script and disabled the backpacking so items still dropped) we had 0 sync issues. Today, I put it back and we had several, almost always occurring in those dream levels (which created the admittedly funny effect of causing the levels representing discontinuity to actually cause a discontinuity). If we go out of sync, we can't rehost because then the one joining (me) loses all her weapons :(, probably because the save game file on the host got the wrong data. Out of sync normally occurs during transport between levels, and when using teleporters within levels. This made getting from Hang Brain, through Electric Sheep Three, to the first save term on Eat the Path very, very difficult, since we kept having to use the older save.

We tried without the script, and the sync issues vanished (we were able to get past the dream levels). I'm also going to modify the script yet again to not save inventory and see if that caused the issues.

I would prefer not to scratch start each level since I like the idea of having more ammo because we found more secrets. We end up having to start from scratch anyway from the rebellion level now... but in general I prefer keeping ammo between levels as it makes finding secrets more rewarding.

I was wondering if maybe there are any settings I can change, or any requirements on network up/down speed, that the community knows of, to ensure there are no/few issues? We are otherwise having a blast with Marathon. We never have sync issues in competitive games, by the way.

In the mean-time I am probably going to keep messing with the script until we have a mostly flawless gameplay that we find fun.
User avatar
irons
Vidmaster
Posts: 2651
Joined: Mar 1st '06, 20:44
Location: (.Y.)
Contact:

It's probably just a problem with the script. What features of the script do you use?
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

irons wrote:It's probably just a problem with the script. What features of the script do you use?
The features that seemed to cause the out-of-sync were the inventory persistence on death. Specifically, I commented out the lines in where the player inventory is affected (effectively removing the functionality of fill_beackpeack, empty_inventory, and empty_beackpeack), and the out-of-sync appeared to go away. The game we will play tomorrow, I will completely comment out the calls to those functions.

Beacons don't appear to cause any issues. All other features of the script I left intact when we had that no-sync-issue run. I'll be playing again tomorrow with him and I'll once again remove the inventory persistence, and update this topic with the results. Hopefully there are no confounding variables and that feature was indeed the smoking gun so at least we will know exactly what the issue was.
User avatar
irons
Vidmaster
Posts: 2651
Joined: Mar 1st '06, 20:44
Location: (.Y.)
Contact:

Yikes. That's the best feature. Let me know and if I have time I'll look into it.
User avatar
treellama
Vidmaster
Posts: 6110
Joined: Jun 2nd '06, 02:05
Location: Pittsburgh
Contact:

I can confirm that coop script causes OOS, and the beackpeack functionality is most suspicious. I meant to rewrite that script but I left for Mars before I could
User avatar
irons
Vidmaster
Posts: 2651
Joined: Mar 1st '06, 20:44
Location: (.Y.)
Contact:

oh doctor manhattan welcome beack
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

Okay, me and him tried without inventory persistence. No sync issues at all. In hindsight, I did notice the out-of-sync occurred around death times. We can still play by dropping a beacon and recovering the stuff (sometimes some ammo seemed to not drop... but the sharing rules meant that death in a sense multiplied ammo so it all worked out).

I guess this means the culprit is the inventory persistence with a high degree of confidence. I unfortuantely do not know enough about the internals of Aleph One to know why... it seems to happen when the inventory is directly adjusted, because I remember not having sync issues when I only partially commented it out and it merely read, not wrote, to the player inventory.

I hope this information helps. We'll be playing through Marathon 2 some time in the future and I'll keep using the no inventory-persist script and will update this if I see any other issues.

Edit: I have an idea but I cannot test it until next weekend: The out-of-sync does not occur on item sharing, only on backpacking. Backpacking loops over every item type in the game when filling the backpack and emptying the inventory. Could it be possible that attempting to read/write an item type that is irrelevant to co-op, such as the ball, even when it is just setting it to zero, is triggering something in the engine that is messing with the synchronisation? Next time we play, I'll have it loop over a constant array of mnemonics of only those items relevant to a co-op game and see if that helps.
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

Nope :(

I modified the script to add a list of items, and iterate over them in empty_inventory and fill_beackpeack instead of iterating over every ItemType in the game:

Code: Select all

BEACKPEACK_ITEMS = { 
  "pistol", "pistol ammo", 
  "fusion pistol", "fusion pistol ammo",
  "assault rifle", "assault rifle ammo", "assault rifle grenades",
  "missile launcher", "missile launcher ammo",
  "alien weapon", "alien weapon ammo",
  "flamethrower", "flamethrower ammo",
  "shotgun", "shotgun ammo",
  "smg", "smg ammo",
  "key", "uplink chip" 
}
It wasn't good enough. The out-of-sync still occurred. I commented out the functions again and we resumed play with no issues.

This doesn't make sense: sharing works perfectly. Is there something about setting the weapon amount in death/revive triggers?

I'm afraid I am out of ideas. I hope y'all can figure it out. Let me know if you want me try anything else and I'll do so next time me and my friend play.
User avatar
irons
Vidmaster
Posts: 2651
Joined: Mar 1st '06, 20:44
Location: (.Y.)
Contact:

Treellama and I were just talking about this. Sadly, I can't get Aleph One to start without segfaulting, so even if I'm up to the task I can't do much about it right now.
User avatar
Wrkncacnter
Vidmaster
Posts: 1953
Joined: Jan 29th '06, 03:51
Contact:

irons wrote:Treellama and I were just talking about this. Sadly, I can't get Aleph One to start without segfaulting, so even if I'm up to the task I can't do much about it right now.
Sounds like you got some bad ram.
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

irons wrote:Treellama and I were just talking about this. Sadly, I can't get Aleph One to start without segfaulting, so even if I'm up to the task I can't do much about it right now.
Oh, sorry to hear that... In the meantime I am going over the script again trying to think of maybe an alternate way of implementing preserving items, simply in the hope that whatever this alternate way is, it would avoid the bug of the current way. But at the moment I can't think of one. I thought of maybe moving the beackpack functions (empty_inventory, fill_beackpeack, and empty_beackpeack) to the idle trigger, and setting some global variable on death/revive just so the code executes on the idle tick, in case there is something wrong with executing it on the death/revive events... but one idle tick later is too much as the player will have probably already dropped all their items and died before the next tick when the fill_beackpeack would actually be executed... so, that idea is likely not going to work.
User avatar
irons
Vidmaster
Posts: 2651
Joined: Mar 1st '06, 20:44
Location: (.Y.)
Contact:

fuck, you wrote:Sounds like you got some bad ram.
fuck, you
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

I'll stay out of the intimate moment you two are having.

Having said that, I had no sync issues setting weapon amounts in the revive trigger. I had no issues reading weapons into the beackpack value. Next test I do will be keeping inventory without emptying it first (resulting in dropped ammo and a LOT of item duplication) and if THAT succeeds with no issues, then I will declare empty_inventory to be the culprit.

I can't really test except once a weekend... so this is going rather slow. Is what I am doing in vain or is it helping in any way?
User avatar
irons
Vidmaster
Posts: 2651
Joined: Mar 1st '06, 20:44
Location: (.Y.)
Contact:

It is. Thanks for your help. Again, I'd be doing a lot more if Aleph One weren't segfaulting. I'm not actually this lazy normally.
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

We didn't get a chance to playtest as much due to a completely unrelated error (apparently getting blown up by a Bob in God Will Sort The Dead causes an assertion error; I'll make a separate report for that bug as it really doesn't seem like it is caused by the lua script). As a result, I can't confirm if keeping the inventory without deleting causes sync issues, as while we had none, we didn't play for long enough to really find out.
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

I've attached the co-op script we used this Sunday, the one that both causes the assertion error (possibly due to way too bloody many items: https://github.com/Aleph-One-Marathon/a ... /issues/20) and that may or may not have sync issues. If there are no sync issues with this script, and they start to appear after commenting back in empty_inventory... then that's the answer.

If that is the case, part of me was thinking of setting everything to 1 for empty_inventory if greater than 0, in case Aleph One doesn't like setting things to zero... unlikely, but grasping at straws are starting to become my only options.

I would upload the file as an attachment here, but the extension .lua was not allowed, nor .txt ... so I'll just post the first part of the script that I modded in case anyone else wants to try and test. I cut out all the functions that I didn't touch, so you should only need to add the globals I added, and modify the functions as I did.

Code: Select all

-- Co-op 2.2
-- By Jon Irons
-- jonirons@gmail.com

-- user variables

-- beacon as it appears physically
BEACON = "alien short light"

-- beacon placement sound
BEACON_SOUND = "computer login"

-- "teleporting" out leaves this behind:
OUT_EFFECT = "major defender detonation"

-- same effect for teleport in by default
IN_EFFECT = OUT_EFFECT

-- sounds for teleport in and out
OUT_SOUND = "teleport out"
IN_SOUND = "teleport in"

-- this is the fader effect when you teleport
FADER = "negative"

-- items that are 'beackpeacked' on death. These items will
-- not be dropped by the player on death and will instead be
-- returned to them on respawn.
BEACKPEACK_ITEMS = {
  "pistol", "pistol ammo",
  "fusion pistol", "fusion pistol ammo",
  "assault rifle", "assault rifle ammo", "assault rifle grenades",
  "missile launcher", "missile launcher ammo",
  "alien weapon", "alien weapon ammo",
  "flamethrower", "flamethrower ammo",
  "shotgun", "shotgun ammo",
  "smg", "smg ammo",
  "key", "uplink chip"
}

-- TODO is setting SMG a problem in Marathon 2? Could that be why it still
-- went out of sync?

t = {
   ["print function"] = print, -- string key, function value
   ["foo"]            = 123,   -- string key, number value
}

-- Key is the name of the item as the mnemonic
-- Value is the numerical amount
-- TODO temporary: intended to give better starting inventory when
-- it is dropped before it can be recovered again until inventory
-- saving works.
STARTING_ITEMS = {
  ["pistol"] = 2,
  ["pistol ammo"] = 8,
  ["fusion pistol"] = 1,
  ["fusion pistol ammo"] = 5,
  ["assault rifle"] = 1,
  ["assault rifle ammo"] = 2,
  ["assault rifle grenades"] = 1,
  ["shotgun"] = 1,
  ["shotgun ammo"] = 5
}

-- modify this list if your scenario needs to change items
-- that shouldn't be shared among players for gameplay
-- balance reasons.
NON_SHARE_ITEMS = {
   "invisibility",
   "invincibility",
   "infravision",
   "key",
   "uplink chip"
}

-- DO NOT MODIFY BELOW THIS LINE. ESPECIALLY YOU, TIM.

-- hacker sections: "constants," "main," and "icons"

CollectionsUsed = {4,26}
Triggers = {}

-- forward compatibility
if not math.mod then
  math.mod = function (a, b) return a % b end
end

-- constants
INVINC = 2 * 30
MAX_OVERLAY_STRING = 12
INFO_TIME = 20
BEACON_DELAY = 60
MP = "%-%-[Mm]" -- monster info pattern
AGE_X = 0.5
AGE_Y = 0.5 -- y offset of beacon age
SOLO_DEBUG = true
MONSTERS = 0

-- main
function Triggers.init()
  if Game.type ~= "cooperative play" and not SOLO_DEBUG then return end
  for p in Players() do
    p._beacon = {}
    p._beacon.owner = p
    set_player_beacon(p, p.monster.x, p.monster.y, p.monster.z,
      p.monster.polygon, p.direction, p.elevation)
    p._overhead_active = false
    p._beacon.age = 0
    p._beacon.delay = 0
    p._info_time = 0
    p._last_target = ""
    p._beackpeack = {}
    p._share = true
    p._receive = true
    p._kills = 0
    --prevent save game restore from sapping inventory?
    --if not p.dead then fill_beackpeack(p) end
    fill_beackpeack(p)
    p._monster_info = true
    if string.match(p.name, MP) then p._monster_info = false end
  end
end

function Triggers.idle()
   MONSTERS = 0
   for m in Monsters() do
      if m.valid and not
	 m.type.friends[Players[0].monster.type.class.mnemonic]
      then
	 MONSTERS = MONSTERS + 1
      end
   end

   for p in Players() do
      p._beacon.age = p._beacon.age + 1
      detect_overhead_map(p)
      detect_overhead_catchup(p)
      detect_beacon_set(p)
      detect_beacon_cycle(p)
      hud(p)
      beacon_info(p)
   end
end

function Triggers.player_killed(player, killer, action, projectile)
   fill_beackpeack(player)
   empty_inventory(player)
   player._overhead_active = false
   player.weapons.active = false
end

function Triggers.player_revived(player)
   empty_beackpeack(player)
   --starting_weapons(player)
   if Level.rebellion then player.life = 150 end
   player.weapons.active = true
end

function fill_beackpeack(player)
  for key, item in pairs(BEACKPEACK_ITEMS) do
    player._beackpeack[item] = player.items[item];
  end
end

-- Empties inventory of a player. Intended to prevent weapons from dropping
-- on the ground. The inventory should be saved to the beackpeack first.
function empty_inventory(player)
  -- for key, item in pairs(BEACKPEACK_ITEMS) do
	  -- player._share, player._receive = false, false
	  -- player.items[item] = 0
	  -- player._share, player._receive = true, true
  -- end
end

-- Empties the beackpeack into the player inventory.
function empty_beackpeack(player)
  player._share, player._receive = false, false
  for k,v in pairs(player._beackpeack) do

    player.items[k] = v
    player._beackpeack[k] = 0
  end
  player._share, player._receive = true, true
end

function starting_weapons(player)
  player._share, player._receive = false, false
  for k, v in pairs(STARTING_ITEMS) do
    if player.items[k] < v then
      player.items[k] = v
    end
  end
  player._share, player._receive = true, true
end
User avatar
Hopper
Mjolnir Mark IV
Posts: 585
Joined: May 10th '09, 17:02
Contact:

Sparklo and I did some testing this week, and can confirm OOS under EMFH, without Aliens or Dead Players Drop Items. (We commented out the game-type check on line 62.)

We tracked our OOS to a player's current weapon becoming mismatched between the two instances. Sometimes a player would respawn with a weapon other than the pistol initially selected, and the gatherer and joiner wouldn't always agree on the weapon.

This suggests that inventory filling is the culprit, and I suspect the use of Lua's "pairs" function to iterate over the beackpeack. The order of pairs in Lua is not guaranteed; if it differed between the two computers, that would account for the behavior we saw.

During testing, we also noticed the script behavior doesn't seem right while one player is dead. Since inventory is saved on death and restored on revive, any weapons or ammo picked up by another player in between will not be shared. Adding dead players' shared items directly to the backpack would prevent a player from losing a key weapon by dying at an inopportune moment.

The attached script restores the inventory in a consistent order, and partially addresses the inventory loss during death. (It won't work if a player is dead during a save or level transition, but neither will the current script AFAICT.) We haven't tested it much, so I can't say for sure it fixes or reduces OOS, but I'm not sure when we'll get back to it.
Attachments
Co-Op-2.2 testing.lua.zip
(5.21 KiB) Downloaded 330 times
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

I think I see; arrays in lua are associative and effectively maps, which gives them the same lack of iteration guarantee as maps in other languages? Come to think of it, I remember on one OOS issue, my friend said I spawned holding the flamethrower, despite me spawning holding a shotgun. In hindsight, this seems to make perfect sense.

I'm torn between my enjoyment of being able to experience true co-op and my gratitude for investigating this issue and figuring it out, and my egotistical need to be the hero in all the things. After careful consideration, soul searching, and punching Fluttershy, I will choose the first option.

Thank you for the testing. Me and my friend cannot play because he is staying somewhere with bad internet, so we will resume in a few weeks and report any further issues or experiences here. Might have to redo all of Marathon properly this time :)
User avatar
Hopper
Mjolnir Mark IV
Posts: 585
Joined: May 10th '09, 17:02
Contact:

If you hadn't narrowed down the problem and documented your process, I wouldn't have found anything. Thank you, Erika!
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

I am really sorry I haven't been able to test this in a while; personal things came up... and then I got The Phantom Pain finally and that is eating away all my 'just want to have fun' time... in my defense it's TPP.

Anyway, has anyone else tested Hopper's modification and can confirm it fixes co-op completely? Me and my friend are probably going to finish Durandal at least some time this month and I'll as always let you know what happens when we finally get back to that.
User avatar
Crater Creator
Vidmaster
Posts: 943
Joined: Feb 29th '08, 03:54
Contact:

Erika Redmark wrote:I am really sorry I haven't been able to test this in a while; personal things came up... and then I got The Phantom Pain finally and that is eating away all my 'just want to have fun' time... in my defense it's TPP.
The name of that game is unfortunate... it sounds like you have some horrible health ailment you've nicknamed The Phantom Pain, which is keeping you from Aleph One co-op. [MGross]
Erika Redmark
Born on Board
Posts: 12
Joined: Nov 27th '15, 23:04

We finally finished Durandal with the new script.

Perfect.

No sync issues at all.

I did notice that the monster counter/who you are slaved to for teleport information was missing, but it wasn't really all that relevant to our gameplay.

Looking back, that was quite an insidious bug with the non-deterministic iteration order. Really glad that is fixed, and just all around awesome, thank you. Maybe we can redo co-op properly, or start Tempus Irae proper with the new script.

I did have a horrible health ailment. And the only prescription was more Marathon.
Post Reply