Preheat oven to 350°F (175°C). Grease two or three 9-inch round cake pans, line with parchment paper circles, and grease the parchment.
Wash and dry oranges, then zest to get 2 tablespoons for cake batter and 1 tablespoon for frosting. Juice oranges to get ⅓ cup for batter and 2 tablespoons for frosting, straining out seeds.
In a large bowl, sift together flour, sugar, baking powder, baking soda, and salt. Whisk thoroughly to combine and set aside.
In another bowl, whisk together eggs, buttermilk, oil, orange juice, orange zest, and vanilla until smooth. Add 4-6 drops of orange gel food coloring if desired and whisk until evenly distributed.
Pour wet ingredients into dry ingredients and gently fold until just combined. Mix white vinegar with baking soda (it will fizz) and quickly fold into batter with a few strokes.
Divide batter evenly between prepared pans and tap gently on counter to release air bubbles. Bake for 25-28 minutes (two layers) or 22-25 minutes (three layers) until toothpick comes out with moist crumbs.
Cool cakes in pans for 10 minutes, then run knife around edges and invert onto cooling racks. Peel off parchment and let cool completely to room temperature, about 1-2 hours.
Beat cream cheese and butter together for 2-3 minutes until fluffy. Add powdered sugar gradually, beating until smooth, then add orange zest, juice, vanilla, and salt. Beat 2-3 minutes until frosting holds stiff peaks.
Place first cake layer on serving plate and spread 1 cup frosting to edges. Add second layer (and third if using), then apply thin crumb coat over entire cake. Refrigerate 20-30 minutes to set.
Apply final layer of frosting generously over chilled cake using smooth strokes. Garnish with orange zest, fresh orange slices, or candied orange peel. Refrigerate 30 minutes before slicing.