Here are some ideas regarding OAuth support for custom REST APIs in Domino.
There is a central configuration database on the server where the admin can register OAuth apps.
For each app, he can define one or more custom scopes with a technical id and a label that is displayed in the browser when the user is asked to grant access.
The database should be customizable just like the domcfg.nsf, so that we can change the layout of the web form the user sees to grant access.
For each custom scope, we can define URL patterns for the OAuth DSAPI filter. The DSAPI filter could then automatically switch the user context when a HTTP request is received with a valid OAuth token and pass through the request to the Domino HTTP server, e.g. to server classic Notes UI or an XPages application.
The DSAPI filter writes a custom HTTP header that the classic or XPages application can read to find out for which scopes the user has granted access (of course setting this HTTP header in the browser should be blocked).
That way every kind of web application on Domino could benefit from OAuth support (classic UI, servlets, XPages). The whitelist is used to restrict access, so that only specific REST APIs or web resources can be requested in user context.
Securing the config database could be done by signing each document with the current user and making sure that the user has unrestricted access (similar to the Update Site NSF).