I figured out how Marathon's visibility tests works.
It is 2D and is done with ray line intersection.
Two rays on the edges of the viewable area are left and right.
The rays are used to determine if the line is visible.
Four ways the line could be detected.
The line vertices are both in view area.
The line has one vertex in view area and one out.
The line has both vertices outside the view area.
The line has both view area rays in the line segment.
If both vertices are in view area then it is visible.
If one vertex is in the view area and one out then one of the rays edges intersect the line making a new vertex, then the new line is visible.
If both vertices are outside the view area then it is not visible.
If both view area rays are in the line segment then the rays intersect the line and new vertices are made.
In Marathon the lines are portals.
Lines hold reference to a polygon, the polygon that should be visible or not.
When the line is visible, new rays are made with the origin is the camera and the direction is the lines vertices.
The new rays determine if any lines are visible in the next polygon.
Each polygon has lists of line portals.
Line portals are a list of two vertices going clockwise or counter clockwise.
Now I figured it out
I found a video that explains line segment intersection and 2d cross product.
https://www.youtube.com/watch?v=wLlfmQUO88I
Use 2d cross product to check if the line segment vertices are on the side of the rays in the view area.
Use ray line intersection when one vertex is in the view area and the ray is intersecting the line or when both view area rays intersect the line segment.
Gather line vertices and put in a list then cast rays in the direction to the vertices to check visible polygons for more lines.
https://www.youtube.com/watch?v=wLlfmQUO88I
Use 2d cross product to check if the line segment vertices are on the side of the rays in the view area.
Use ray line intersection when one vertex is in the view area and the ray is intersecting the line or when both view area rays intersect the line segment.
Gather line vertices and put in a list then cast rays in the direction to the vertices to check visible polygons for more lines.