Personally, I had the most trouble with world 2. Because where you enter a "door" and where you come out after that just seems not connected in any logical pattern. Your direction is turned, the height level changed... For a long time, I thought the area you start in and meet Trowzer is another area than the one where the palace stands. After some time I finally realized the main purpose of that entrance is to help you get up fast. Man, those could've been stairs.
The first level looks all the same just like the second, but it has the aqueduct to help you. I never had big problems with the later levels. Maybe I got used, maybe the design was better. I guess the Casino is easier to remember because it's more diverse, while the remaining two levels have paths you can follow.
In DK64, the one level I found the most disorientating probably was the ice cave. Again, there just isn't a lot of different things to see or remember positions on in an ice world