After some messing around I've switched to a different simulation method, with precomputing the neighboring vertices into a texture. I've made a tool that converts unity cloth's into shader based cloths. Here as a demo I'm trying a 30k vertex mesh. Still lots of work ahead tho