I don't think that there is a way such that hundreds or thousands of people could collaborate on a problem without subdivision into smaller groups with certain synchronization points. Large software projects with thousands of developers are usually broken down to groups with 2-20 members. Imagine that you'd have to talk to each member in your group for 30 minutes per week, that would have to talking for 10 hours in the latter case, every week, just to synchronize with what the others are doing (this is sometimes called the combinatorial explosion of communication).
In this sense I'd say that Wikipedia is very successful in collecting and explaining a lot of mathematical knowledge and pointing out open problems. If you supplement this wiki ansatz with links to more specialized wikis where subgroups document their work about certain open problems, you'd come close to a large scale collaboration. The nLab is an example of this, but it has not attracted a large crowd yet. The nLab has an accompanying forum, where people talk about their thoughts before there is an agreement about what should be written on the wiki.
You can write about your work and where you are stuck on the wiki, explaining all the background and ask on the forum for help. Others can chime in either by supplementing what you wrote on the wiki, or by joining the discussion on the forum. I don't see why this could not work with a lot more participants, as long as they form subgroups and avoid a situation where every participant has to talk and listen to everyone else.