Flower Girls don't have to match the adults perfectly, but they should certainly fit in with the formality of the event.
Be sure to dress your flower girl or ring bearer appropriately for their age. Strapless or halter dresses may be too sophisticated for the little ones (even with a cardigan), while overly froufrou tulle dresses might not look right on, say, a 10-year-old.