Have brainstormed this problem for a side-project. Rather than testing a ray-segment intersection for each edge of the triangle I believe the following works and should be cheaper, though I haven't coded it up yet.
Given Triangle t = { p1, p2, p3 }, Point p within t, Vector v.
Construct Ray r from p with direction v. Find Edge e of t
intersected by r.
/\
/ \
/ \
/ p*-> \
/ v \
/_....---''
Solution:
For each pair (pa, pb):
Let q be any point in the triangle (e.g. its centroid)
Compute c = Clockness(pa - q, pb - q)
Construct va = (p, pa), vb = (p, pb)
If Clockness(va, v) == c && Clockness(v, vb) == c:
Return Edge e = (pa, pb), which r intersects.
Define clockness(a : Vector, b : Vector) as sign(cross(a, b)) - intuitively it represents whether the cross between two 2D vectors goes "into" the plane or "out of" the plane on which they sit.
In most computational geometry libraries, triangles are represented in some order (e.g. clockwise) by convention, so computing c should be free. Intuitively the clockness between (p, pa), v should be the same as the clockness from v, (p, pb), which should be the same as the clockness between (p, pa), (p, pb). That is, cross (p, pa), v should point in the same direction as cross v, (p, pb) which should point in the same direction as cross (p, pa), (p, pb).
Edit:
This seems to work, here's my code. I had to add an additional case for when you're sitting on the triangle edge:
public static int IntersectRayWithContainedOriginForVertexIndexOpposingEdge(DoubleVector2 origin, DoubleVector2 direction, ref Triangle triangle) {
// See my explanation on http://math.stackexchange.com/questions/2139740/fast-3d-algorithm-to-find-a-ray-triangle-edge-intersection/2197942#2197942
// Note: Triangle points (A = p1, B = p2, C = p3) are CCW, origin is p, direction is v.
// Results are undefined if ray origin is not in triangle (though you can probably math out what it means).
// If a point is on the edge of the triangle, there will be neither-neither for clockness on the correct edge.
for (int i = 0; i < 3; i++) {
var va = triangle.Points[i] - origin;
var vb = triangle.Points[(i + 1) % 3] - origin;
var cvad = Clockness(va, direction);
var cdvb = Clockness(direction, vb);
// In-triangle case
if (cvad == Geometry.Clockness.CounterClockwise &&
cdvb == Geometry.Clockness.CounterClockwise) {
return (i + 2) % 3;
}
// On-edge case
if (cvad == Geometry.Clockness.Neither &&
cdvb == Geometry.Clockness.Neither) {
return (i + 2) % 3;
}
}
throw new ArgumentException("Presumably origin wasn't in triangle (is this case reachable even with malformed input?)");
}