Have you ever had to sort a view of nodes by a referenced taxonomy term? In our system, each staff member is entered as a separate node, containing the staff member's name, position and email address, and also containing the department the staff member works in. The department is a taxonomy reference field. We then have a view that pulls in each staff member's node, organized by department, and the departments are listed alphabetically.
The problem we found was that even though this view listed the appropriate staff under the appropriate department, the departments were not listed in alphabetical order, but rather seemed to be ordered randomly. I looked at the view and saw that the primary sort criterion was "Staff Department"--the taxonomy term being referenced by the node. Perhaps you can already see what the problem was!
When I looked at the displayed view, I saw that the departments were actually ordered by the term id of the Staff Department taxonomy term. This is a non-intuitive fact about how Drupal orders a view based on a taxonomy term--it doesn't sort by the vocabulary terms themselves, such as Accounting, IT or Marketing. It sorts by the numerical term id. This is non-intuitive, because why would you ever want to sort by arbitrary numerical ids?
But of course there is a way to sort by the actual terms. When you go to the view edit page and click on "Add" next to "SORT CRITERIA," instead of selecting the name of the field that references that taxonomy (e.g., "Staff Department"), select "Taxonomy term: Name" (or whatever field on the taxonomy that makes the most sense to sort by).