CSOM to read type-specific fields for the RoleAssignment.Member property

I am trying to optimize some code that retrieves the RoleAssignment.Member property in some queries. The current code looks something like this:

context.Load(webCollection,
    webs => webs.Include(
        web => web.RoleAssignments.Include(
            ra => ra.Member)));

The problem is that I only need some of the properties of Member, and the above query returns all of them. These queries that include RoleAssignments may return thousands of items, so I'd like to minimize the amount of data returned as much as possible.

Unfortunately, the type of the Member property can be either User or Group, and one of the properties I need is specific to User: IsSiteAdmin.

I tried casting the Member property in the query like this:

context.Load(webCollection,
    webs => webs.Include(
        web => web.RoleAssignments.Include(
            ra => ((User)ra.Member).IsSiteAdmin)));

This compiled, ran, and sent the request that I expected, but the server returned the following error:

ErrorMessage=Field or property "IsSiteAdmin" does not exist.
ErrorTypeName=Microsoft.SharePoint.Client.InvalidClientQueryException

Is there any way to query specific fields from the derived classes in the RoleAssignment.Member property?