Each pair of characters in your party has a friendship level (shown as heart meters in each character's status screen). These can range from zero (five empty hearts) to forty (five full hearts). Each pair has a predefined level they will start at which will generally be low but above zero. For a pair of characters to quality for an ending, they need to have at least 28 points (three and a half hearts) when you beat the final boss.
Each character can only participate in one ending (aside from Gather Up). If they qualify for multiple pair endings, the pair with the higher friendship value will be chosen. Remaining characters then check against the other remaining characters, and any who don't qualify for a pair ending with a not-yet-paired character will instead get their solo ending.
In case of tie, pairings are prioritized in this order:
For example, suppose your party is Claude, Rena, and Celine. Rena has 28 friendship points with both Claude and Celine while Claude has 40 with Celine. You'll get the Claude and Celine ending and Rena's solo ending.
You can see which endings your party is currently slated for by checking each party member's status screen. If they qualify for a pair ending, little hearts will be displayed around the face of the character they are paired with.
A few pairings have two endings - a normal one for if their friendship is non-maxed (28 to 39, or three and a half hearts to just below five full hearts) and a different one if their friendship is maxed (40, or five full hearts). They are:
Claude and Rena actually have three endings. Claude and Rena 1 if Rena is the main character, Claude and Rena 2 if Claude is the main character and their friendship is not maxed, and Claude and Rena 3 if their friendship is maxed.
Three characters have special endings that become available under certain circumstances. These endings have high friendship levels associated (32, or four full hearts), making it impossible to get the character's normal solo ending once they are available. It's still possible to get pair endings with these characters if you get higher friendship levels. These special endings are:
Note: I'm finding incomplete and contradictory information online about the relationship level of these endings in this version of the game, so I'm not completely certain these endings all come in at 32 points. This has implications for whether other pair endings are still possible, and I won't know until I confirm on a future playthrough. Please let me know if you know for sure.
There's one special whole-party ending. Unlike other endings, it's shown after the credits and doesn't rule out any other endings. If you qualify for it, it will be shown in addition to any other endings you qualify for.
To get this ending, you need a full party of eight characters and every single pair of party members needs to have a friendship level of at least 28 (three and a half hearts). Thankfully, you can ensure this by using items to set all pairs to 28 points (see below).
As mentioned, each pair starts at a predefined level which is below the threshold required for an ending. As you play through the game, several things will change these levels: Private Actions, boss fights, usage of Pickpocketing or Contraband, and more. But the easiest way to manipulate friendship levels is through items.
The Publication Super Specialty allows you to create two books per character: one at Publication level five and another at Publication level ten. The books can be used on each other character to set their friendship level with the author to 16 (two full hearts) for the level five book or 28 (three and a half hearts) for the level ten book. (You can also just make one of each book and sell it to the publishing shop in Lacuer or Centropolis; then you can buy however many copies you need from the publishing shop in Lacuer.) There are also two potions that can be used on a party member to set their friendship levels with all other party members at once: Potion of Trust (which can be created through Compounding using Rose Hip and Artemis Leaf and can also be duplicated through Replication) sets them to 16 while Aphrodisiac (available from a PA, from a Unique Spot, from an overworld chest, and repeatably as a Bunny Race reward) sets them to 28.
Since 16 is below the ending threshold and 28 is above it, this allows you to fine-tune your friendship levels to get your desired set of endings. In general, you'll want to maximize friendship levels until you save at the final save point; then after beating the final boss you can reload and change levels as desired to get a different set of endings.
For example, suppose (as before) your party is Claude, Rena, and Celine. Rena has 28 friendship points with both Claude and Celine while Claude has 40 with Celine. You're slated for the Claude and Celine ending and Rena's solo ending. If you have Claude read either of Celine's books, have Celine read either of Claude's books, have either of them use an Aphrodisiac, or have Celine use a Potion of Trust, you'll instead get a Claude and Rena ending (either 1 or 2 depending on who your main character is) and a Celine solo ending (either 1 or 2 depending on whether the Clyde sidequest has been completed). If you instead have Claude use a Potion of Trust, you'll get the Rena and Celine ending and Claude's solo ending.
Note that this isn't sufficient to get any of the seven endings that require the level to be maxed-out or to get any normal pair ending for Celine, Ashton, or Opera when their special solo ending is available. If you're targeting these endings, you'll need to use other means. To be safe, use items to set the level to 28 as soon as possible (and put it back if it ever drops below that). Keep the characters in the active party together for boss fights, be careful with their Private Actions, and don't use Pickpocket or Contraband without using Muscat Jelly or Konjac Jelly to prevent reducing the friendship level. After saving at the final save point, you can then use items to go down to 28 for the corresponding non-maxed endings or special solo endings.
In earlier versions of this game (Star Ocean: The Second Story and Star Ocean: Second Evolution), relationship levels were more complicated and harder to manipulate. Each pair of characters had four levels between them (friendship from character A to character B, romance from A to B, friendship from B to A, and romance from B to A). Streamlining this to a single value mostly just makes things easier to keep track of, though since the Claude and Rena endings previously depended on which character liked the other one more, those are now divided between the Claude and Rena routes instead.
Additionally, the items that changed levels did not set any values high enough to meet the threshold for an ending, making it much harder to get all the possible endings for a given party in a single playthrough. The elusive Reunion/Gather Up special ending was especially difficult to get.
Also, this is the first version of the game to actually show you your current friendship levels and which pairings you currently qualify for.