Unsatisfying ending

They were thinking "Sequel"

They thought attaching James Cameron was enough.

I agree, but you make it sound like this mess was a masterpiece that got ruined by a weak ending for no reason.
The ending was written by the same no talent writers that penned the rest of the script. It's its byproduct.
Having a traditionally reasonable ending (let's say it lasts another half hour, where she manages to get to Zalem, find Nova and deal with him) would make the rest better?
It was just unbalanced and all over the place from start to finish, so such ending fits the movie.

They got some things right, it could and should have been way better, but it's not.
Including the stupid plan of making this an introductory chapter of a series, that ends with her becoming the motorball champion...who the f cares about that, right?
That's like ending Robocop after his first few arrests, or Terminator after they escape L.A. we want to see the hero dealing with the real bad guy!

What was wrong with the ending?

That was a problem with the animated OVAs as well, and I imagine that's how those chapters in the manga ended. However, the animated OVAs didn't have that character up in Zarem, so considering the movie just teased him at the end, I think his inclusion was a mistake.

The thing is, most manga series are like television series that exist to perpetuate themselves, rather than telling a story with a beginning, middle, and end. As a result, the adaptations of the manga often lack closure as they're only telling you a few chapters of a story, that in all likelihood, is still ongoing and may never have a conclusion.

I remember waiting years for them to continue the cliffhanger ending of the Berserk animated series, only for them to reboot it 20 years later and start telling the story again from the start. When they finally got into the new material, I lost interest as the story line went downhill. The manga has been going for 30 years and still seems no closer to actually ending the story.