Your question sounds a bit like a setup for a joke ;-)
"Approximately how long will it take for 3 17 yr old Girl Scouts & 1 leader to build the castle?"
The gingerbread castle is a BIG project with lots of pieces, depending on which variations you choose. The big slow-down with gingerbread baking though isn't usually the number of pieces. Waiting for the icing "glue" to dry before decorating is the time-consuming part. If you could split your gingerbread-building party into 2 or 3 separate days or chunks of time, that would be best:
1) Make the gingerbread, cut out the template pieces, and bake.
2) Once the pieces have fully cooled, mix up your royal icing and glue everything together.
3) Once the icing has fully set and the castle feels sturdy, then decorate.
OR
alternate 2 and 3) Many professional bakers decorate their pieces BEFORE assembling. It depends on how elaborately you'll decorate and how soft/breakable your decorations are. Most "regular" folks (who plan to actually eat their creations) stick to the more common approach of assembly and THEN decorating.