I'm adding another answer because this answer is significantly different than my previous one.
I agree that understanding the function you suggested is a first step towards understanding the full problem ; if we can understand a simple piecewise-constant function, then we can approach functions using piecewise constant functions (in a similar fashion than the one used to construct the Lebesgue integral) and perhaps use a continuity theorem to have a formula for the general $A_{\alpha}(f)$-average.
I am taking $f(x) = \mathbb I_{[0,1/2]} + m \mathbb I_{]1/2,1]}$ as you suggested. Let $S_n$ denote the group of all permutations of $\{1,\dots, n\}$. Since $n/2$ will appear at some point and I don't want to write $\Gamma$-functions to preserve combinatorial intuition, I will suppose $n$ even.
$ a_k(Y(f)) = \frac 1{\binom nk^{\frac 1k}} \left( \sum_{1 \le i_1 < \dots < i_k \le n} f(x_k) \right)^{1/k} \\ = \frac 1{\binom nk^{\frac 1k}} \left( \frac 1{k!(n-k)!} \sum_{\sigma \in S_n} \prod_{j=1}^k f(x_{\sigma(j)}) \right)^{1/k} \\ = \left( \frac 1{n!} \sum_{\sigma \in S_n} m^{\#\{ 1 \le j \le k \, | \, \sigma(j) > n/2 \}} \right)^{1/k}. $ Let $\psi_{k,n}(\sigma) = \#\{ 1 \le j \le k \, | \, \sigma(j) > n/2 \}$ and $\phi_{k,n}(\ell) = \# \{ \sigma \in S_n \, | \, \psi_{k,n} (\sigma) = \ell \}$ for $0 \le \ell \le k$.
By considering the probability space $(S_n, \mathcal P(S_n), \mu)$ where $\mu(\sigma) = \frac 1{n!}$, the above sum can be seen as the $1/k$'th power of the moment generating function of $\psi_{k,n}(\sigma)$ evaluated at $t = \log m$. This is probably of interest but I don't see if it helps right now.
By regrouping the terms, we get $ a_k(Y(f)) = \left( \sum_{\ell = \max \{0, k - n/2 \}}^{\min\{k,n/2\}} \frac{\phi_{k,n}(\ell)}{n!} m^{\ell} \right)^{1/k}. $ Now we need to understand the asymptotic behavior of $\phi_{k,n}(\ell)$. Asking $\psi_{k,n}(\sigma) = \ell$ is essentially the same as sampling $n$ balls labelled from $1$ to $n$, putting them in an order (i.e. constructing $\sigma$) and asking that amongst the $k$ first balls sampled, precisely $\ell$ of them have an index $> n/2$. Therefore, if we consider $\psi_{k,n}(\sigma)$ as a random variable of the random permutation $\sigma$, $\psi_{k,n}$ follows an hypergeometric law : $ \frac{\phi_{k,n}(\ell)}{n!} = \mathbb P(\psi_{k,n}(\sigma) = \ell) = \frac{\binom{n/2}{\ell} \binom{n/2}{k-\ell}}{\binom nk}, \quad \max\{0, k-n/2\} \le \ell \le \min \{ k, n/2 \}. $ One must not see this probability as a "random" thing. Since we are considering all permutations $\sigma$, this probability is more like the proportion of permutations that satisfy $\phi_{k,n}(\sigma) = \ell$ and thus can be used to study asymptotic behavior. We obtain $ a_k(Y(f)) = \left( \sum_{l = \max \{0, k - n/2 \}}^{\min\{k,n/2\}} \mathbb P(\phi_{k,n}(\sigma) = \ell) m^{\ell} \right)^{1/k} \\ = \left( \sum_{l = \max \{0, k - n/2 \}}^{\min\{k,n/2\}} \frac{\binom{n/2}{\ell} \binom{n/2}{k-\ell}}{\binom nk} m^{\ell} \right)^{1/k} \\ = \left(\sum_{l = \max \{0, k - n/2 \}}^{\min\{k,n/2\}} \frac{ \binom{n-k}{n/2 - \ell} \binom k{\ell} }{ \binom n{n/2} } m^{\ell} \right)^{1/k} \\ $ Using Stirling's formula, one computes that
$ \binom{n}{n/2}^{1/k} \sim \left(\frac{2^{n+1}}{\sqrt{2\pi n}}\right)^{1/\alpha n} \longrightarrow 2^{1/\alpha}, $ so that $ a_k(Y(f)) \sim 2^{-1/\alpha} \left(\sum_{\ell = \max \{0, k - n/2 \}}^{\min\{k,n/2\}} \binom{n-k}{n/2 - \ell} \binom k{\ell} m^{\ell} \right)^{1/k} \\ $ Now understanding the last limit would be key to this problem, but unfortunately I haven't made much progress there. What's cool now though is that we can approximate this limit and plot the graph. Here it is :

In the plot, $m = 500$, $n=200$ and $1 \le k \le 200$. As $k$ increases, the approximation of $A_{\alpha}(f)$ by $a_k^n(f)$ is shown. The plot used the last formula to obtain the value of the points. The plot suggests that the function $A_{\alpha}(f)$ has an inflexion point at $\alpha = 1/2$ (when $A_{\alpha}(f)$ is seen as a function of $\alpha$).