Open Sesame: Menu UI (Part III)

Updated: Aug 22, 2019

The Unity animator is probably the most confusing thing I've delved into so far, so I'm going to try and go slow while I explain it. When you add an animator component to any gameObject in your scene, Unity creates a state machine for that object, which allows you to define different behavioral states and the transitions between them.

Let's start with the menus themselves. A menu has two states: open and closed. Wait. That's already not quite accurate...what about when you open a new menu on top of another one, what state is the hidden one in? What about before a menu is opened for the first time, what state is it in then? And while we're at it, let's throw in one more confusing little tidbit: you do not have direct control over the transitions between states. Oh, you can set parameters like the amount of time the transition takes, but if I truly have just a "MenuOpen" and a "MenuClosed" state, Unity will be the one deciding how to transition from one to the other. Usually it will do so linearly, but what if that's not what you want?

Okay, so what did I want? Well, I wanted each new menu to expand from the button that was clicked, first growing to fill the width of the screen while staying the same height, then expanding to fill the entire height. Next, the buttons and other content should fade in. This should include the "translation" effect I described a few posts ago, and which will be the second big thing to figure out a little later here. Once the translation is finished - and only once the translation is finished - the player should be able to navigate on the new menu. At this point, I would consider the menu "open", but the problem is that everything I just described needs to be accounted for somehow. I suppose, technically, we could define four states: "MenuOpening", "MenuOpen", "MenuClosing", and "MenuClosed" fact, this would probably be more intuitive than what I ended up with. Instead, I added the transition stuff at the beginning of the MenuOpen state, and set the looping parameter to false. Looping would make sense for something like the idle in a set of character animations, but in my game it wouldn't make sense to loop a menu state that literally nothing is happening in. So I got into the mindset of using animations rather than states; my MenuOpen is the animation that plays when the menu opens, and when it finishes, it sits there, still technically in the MenuOpen state but not doing anything. In fact, it's waiting for a trigger to transition to a different state, and I only have one trigger set up: the trigger to MenuClose. Notice the name change: MenuClose rather than MenuClosing or MenuClosed, because again, we're treating the state as the single, un-looping animation. MenuClose actually just plays the MenuOpen animation in reverse, with one addition: when it's over, it deactivates the menu completely.

Those are the "basics", and they work because A) each menu gameObject starts deactivated, and with its animation entry state set to MenuOpen. This means that when the player clicks a button onscreen, the relevant menu activates itself and immediately starts playing the MenuOpen animation. So far, so good. B) Nothing else happens to the menu with the button they clicked; it's just sitting there, hidden behind the new active one. So if the player closes the one they just opened (by clicking the back button), MenuClose plays and turns off the new menu, leaving the old one visible and active again. Then, if they re-open that menu they just closed, since it's been completely deactivated, it's like it's opening for the first time and goes into MenuOpen, correctly. There's just one more component needed to make this work, and it's the idea of a "currently selected" menu item. The currently selected item is a flag for Unity to know how menu navigation should take place when the different directional buttons and select button are pressed. I needed to make sure that I updated "currently selected" as I changed menus, so that navigation would always happen on the current menu, and only happen on the current menu. Luckily, Unity allows animation events, which is lingo for "calling code functions at a certain animation keyframe". I can set "currently selected" to null as soon as any animation starts (so that player input is completely ignored during said animation, eliminating the chance for unexpected behavior), and reset it whenever the animation is finished. Perfect!

Actually defining the animations should be easy, in theory. You can control exactly how every single menu component behaves at each keyframe, for as long as you want the animation to be. So I can set my menu to expand from the button exactly the way I want. The only thing is, since each menu grows out of a different button, it needs to have different anchor positions set, which means a unique set of animations has to be coded for each. This is something that I managed to get around with the buttons...but more on that later. Still, this is a lot of extra work, but not really a problem. I actually found a case where I needed additional behavior during a single one of the menus' animations, so I just added that part where it was needed. The real problem I had was due to a known bug with a parameter called "Write Defaults". I'm not going to say a lot about this, because even scouring forums and reading official dev responses, I'm still not sure I completely understand it. The basic idea is that, it's common for an animation state to change some parameters that other states do not. Write Defaults should define how those parameters behave in the unaffected states, but it seems to be that the default behavior is completely unintuitive, and setting it opposite...doesn't work. From what I could gather from the years-long forum discussions, the devs who coded it weirdly originally have no plans to fix it, so the best solution is to just manually set every single parameter that changes in any state in every state. Yeah, ouch.

Okay, sorry for the giant wall of text. It looks like I'm going to have to save button animations for next time, but here's a gif of walking through the final MenuOpen: