Every language has an annoyance.
In English, it’s the definite vs. indefinite articles. I have been told that this is almost impossible for a foreign person to get right, though I don’t see what’s so hard about it. In Spanish, it’s the conjugations – I like to joke that there’s a special conjugation for something that happens on Tuesday under a full moon. But I’m only partly joking.
There are two things in Japanese. Wa vs. Ga, and counters.
The Japanese counter system is insane. In case you didn’t know, there’s a counter for many different things, and it’s a suffix for the number words (ichi, ni…). There’s ko （1個）, which is the counter for objects. There’s ban (番), which is the counter for order (first, second, third – this is where “ichiban” comes from). There’s sai (歳), which is the counter for years (juu sai desu means ten years old). There’s kai (回), which is the counter for “times” (watashi wa juu kai deshita means I did it ten times). Each one has its own kanji, and you have to use the correct one. There are over fifty of these. One for books. One for long, cylindrical objects. Using the wrong one is a grammatical error, and in most cases, you can’t use a more generic one as a substitute if you don’t remember what the correct one is.
Not only that, but sometimes the actual words change based upon the counter. One object is “ichi ko”, but one person is “hitori”. Why? Because they’re Japanese, that’s why!
It’s actually not insanely hard once you learn the different counters, honestly. Compared to wa vs. ga it’s pretty straightforward if you put a little time into memorization. But for me, it is, hands down, the most freaking annoying part of the Japanese language that I know.