Here's a script that can be used to animate the individual vertices of a mask path:Rototracking palette for AE (6.5, 7, CS3)
Although it's intended for rototracking, you can use it to control mask vertices by other means. Here are some steps you can try:
1. Download the RotAE1.1.jsx script from the above link.
2. Select File > Scripts > Run Script File and choose the RotAE1.1.jsx script to launch the floating palette.
3. Create a new comp, with a solid layer of the same size.
4. Select the mask pen tool and draw a 3 point mask on the solid layer.
5. Twirl open the timeline properties and select Mask 1.
6. Click the palette's "Apply Tracker" button.
7. With the layer selected, press the U key twice quickly (the UU shortcut reveals all altered properies). You'll see a motion tracker with 3 points has been added to the layer.
8. Double click the layer in the timeline to open its layer window. Select to view Motion Tracker Points and you'll see each point is at the position of one of the mask vertices.
If you were doing this on a footage layer, you could now use the tracker controls to create keyframes. But in this example, we're doing it with nulls and expressions.
9. Select Layer > New > Null Object. Create three nulls, called Null 1, Null 2, Null 3.
10. Select all three nulls and press P to view the Position properties.
11. Select the solid's Tracker 1 > Track Point 1 > Attach Point property and do Edit > Copy.
12. Select the Null 1 Position property and do Edit > Paste.
13. Repeat with the other two track points.
Return to looking at the composition panel. There are now 3 nulls, one positioned at each of the mask vertices.
14. Animate the Null Position properties.
15. Select the Track Point 1 Attach Point property and choose Animation > Add Expression.
16. Either use the expression pickwhip to link to the Null 1 Position property, or add this expression:
17. Repeat for the other two track points, linking them to Null 2 and 3.
18. Select all three Attach Point properties and choose Animation > Keyframe Assistant > Convert Expression To Keyframes.
19. Select the solid layer's Tracker 1 and Mask 1 property group names.
20. Finally, click the palette's "Apply Mask" button.
The mask vertices will now be animated to match the Nulls. If you change the Null animations, repeat steps 18-20 to update the mask.
Does that help?