In my last post I described many of the REST endpoints available in SharePoint to manage role assignments. In this post, I will provide a concrete example of using these endpoints in a provisioning-like scenario. I say provision-like because real provisioning scenarios tend to be very specific and one-offs (i.e. I need 7 sites, each with 5 lists and 3 groups, based on a naming convention by organization, and these permissions, and blah and blah and blah). Such specific requirements can’t be written into a one size fits all solution, so I’m just going to mimic them by creating a whole bunch of role assignments, and then deal with some of the issues of initiating a bunch of ajax calls in a short period of time.
What We’re Building
I want to build a page that demonstrates what a provisioning scenario might look like. Namely, this is a whole lot of individual REST service calls, which presents some unique challenges when implementing in JavaScript in the browser.
In the interface below, you can choose a SharePoint group, one or more SharePoint lists in the current site, and one or more role definitions, and with one click you can assign each of those role definitions to that SharePoint group for each of those SharePoint lists. As filled out below, there are 6 lists and 4 role definitions to be applied to the group “CSRDemos Members”.
So the first thing I need to do is check each list to see if role inheritance isn’t broken (that’s 6 lists, so 6 REST calls and 6 network connections). Then if role inheritance isn’t broken, I need to break it (6 more REST service calls), and if it is already broken I need to delete each list’s role assignments for the SharePoint group (also 6 REST service calls, so stage 2 is 6 calls either way). And finally, I need to make 4 role assignments for each of 6 lists (i.e. 4×6 = 24 more REST service calls). Hmmm…that seems like a lot:
(6 hasuniquepermissions) + (6 breakroleinheritance/roleassignments) + (6 lists * 4 role definitions) = 36 REST calls
Piece of cake…right?
And while you’re waiting, I pop up this nifty CSS-only spinner! What more could you ask for? I’ve said in previous REST posts that I could probably use some sort of busy indicator here or there, but this time I really need it.
Actually, it’s just not as bad as you might imagine. Below I show the console output from a run with these inputs. I log a bunch of console messages with things like how many connections I currently have open, the results from each service call, and finally the elapsed time. In the screenshot below, the elapsed time is 5.6 seconds. 36 REST calls in 5.6 seconds, I can live with that. And in fact, it could be much faster. I implement some throttling in my code, which is fancy tech talk for I slow it down some. Without that throttling, it takes roughly .7 to 1.5 seconds.
Don’t believe me? It’s pretty clearly shown in the screenshot below. This is IE’s network tab in developer tools. It is currently showing the output from one button click with the above parameters, in other words the 36 REST service calls I described above. Notice how the network connections are bunched together in groups of 5. And as time progresses, a gap is clearly noticeable between those bunches.
That’s because I only open 5 network connections at a time. When I need more, I pause in one-second intervals and wait for old connections to be released before initiating new requests. This actually slows down the client quite a bit, but also reduces the load on the server quite a bit. It’s a trade-off, and you need to be aware of it if you’re going to implement anything that does bulk operations with SharePoint (REST or not). You can’t always pound the server full-throttle without eventually getting nasty calls from your farm administrators (or if you are a farm admin, you’re customers).
Now one of the knocks on REST vs CSOM is that CSOM has support for batch operations. In truth, so does SharePoint REST, but it’s very cumbersome and not even available until 2016. And anyway, realize that batching is a trade-off too. By opening multiple connections you can perform many operations in parallel, but at a cost of more resources used on both the client and the server, and you can tweak that with throttling. By doing batch operations I use a lot fewer resources on both the client and the server, but quite possibly the result appears slower to the end user because now all operations are executed on the server in serial, and the only way to tweak that is with more and better hardware. Or at least changesets are executed in serial (see the second Andrew Connell post referenced below). Batch operations within a changeset may be serial or may be parallel, but either way it’s not up to you.
Now let’s say I had a more realistic scenario like say I had the same 6 lists in 10 sub-sites, and I needed to add role assignments to 3 different groups. But as long as we’re shooting for realism, let’s say I only need one role definition for each group, because there is really no reason to ever need more than one. If I need the union of permissions of 2 permission levels I can create a custom level with those permissions. And most permission levels have a privilege/subordinate relationship anyway, meaning Designer is Contributor+. So there’s no reason to assign a group Contributor and Designer, just Designer will do. I just did it above to generate a lot of connections and see how it performed. So anyway, here’s what this scenario looks like:
10 sites * 6 lists * 2 service calls (hasuniquerollassignments and breakinheritance/removeperms) = 120 service calls
And:
10 sites * 6 lists * 3 groups * 1 role assignment = 180 service calls
So if I can do 36 REST calls in 5.6 seconds with throttling, I can extrapolate that 300 similar REST calls would take approximately 46 seconds with the same throttling. And I can adjust the throttling to reduce the server load or increase the speed as needed (to a degree). 46 seconds might seems like a lot of time, but at least you get to look at my beautiful spinner while you’re waiting, and that would be roughly on the order of magnitude of 10 seconds without any throttling. And keep in mind that this is provisioning code. It probably only needs to be run once, or maybe once each time a new sub-site is added, can probably be run on off-peak hours, and therefore maybe neither performance nor server load is that much of a consideration. And if you want to do a lot of work, it’s going to take a bit of time. Even if you do it all manually, the server’s still going to end up doing the same work, albeit not all at once.
Either way, this is starting to look doable, even from JavaScript in the browser. There does come a point where it starts to look more like a job for PowerShell, but since many SCAs have neither the skills nor the option to run PowerShell against their site, anything in the range of 30-60 seconds is reasonably doable in the browser. Even longer really, but ideally you should provide a bit more than a pretty spinner to show that something is happening if it’s going to go much longer. Like maybe show the most recent console log message above the spinner. Some quick changing text like that can be very reassuring to the user that they’re not just locked up, even if it’s too fast to read.
Provisioning Role Assignments with Code
The sample code for this post is pretty big, so I’ll make no effort to show it all here. Much of it is just the same stuff I showed in my last post with the $.ajax calls converted to fetch. I’m just going to show some of the more interesting bits. Most of the code is encapsulated in an object literal called roleDefinitionManager (or rdm for short). The action starts with the rmd.init method:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | init: function() { rdm.initGroupSelect(); rdm.initListSelect(); rdm.initRoleDefinitionSelect(); var doIt = document.getElementById("doItNow"); doIt.addEventListener("click", function(e) { // construct an object that describes all the things we need to do rdm.todo = {}; rdm.todo.principalId = document.getElementById("siteGroup").value; rdm.todo.lists = rdm.getValues(document.getElementById("listsSelected"), true); rdm.todo.roleDefIds = rdm.getValues( document.getElementById("roleDefinitionsSelected"), false); console.log(JSON.stringify(rdm.todo, null, 4)); // validate required inputs var errors = rdm.validateRequired(rdm.todo.principalId.length < 1, "groupError"); errors += rdm.validateRequired(rdm.todo.lists.length < 1, "listsError"); errors += rdm.validateRequired(rdm.todo.roleDefIds.length < 1, "roleDefinitionsError"); // reset connection statistics rdm.connectionCount = 0; rdm.totalReleased = 0; rdm.totalExpected = rdm.todo.lists.length + (rdm.todo.lists.length * rdm.todo.roleDefIds.length); // if nothing failed validation if(errors === 0) { // start setting role assignments rdm.setRoleAssignments(); } }); }, |
First it initializes the options for the 3 select elements for group, lists, and role definitions. Then it sets up a click handler for the terribly named “Do it now” button. When that button is clicked, it summarizes what is to be done in a member property called todo, and then calls setRoleAssignments shown below. It also initializes some member properties used for connection management, but that will be easier to explain in a later code block.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 | /** * Set all role assignments on all lists. * @return undefined */ setRoleAssignments: function() { rdm.startTime = new Date(); try { for (var i = 0; i & lt; rdm.todo.lists.length; i++) { var list = rdm.todo.lists[i]; rdm.setRoleAssignmentsOnList(list); } } catch (e) { document.getElementById("spinnerOverlay").style.display = "none"; } }, /** * Set role assignments on a specific list. * @param list {string} the title of the list. * @return undefined */ setRoleAssignmentsOnList: function(list) { var doListInternal = function () { url = _spPageContextInfo.webAbsoluteUrl + "/_api/web/lists/getbytitle('" + list + "')/hasuniqueroleassignments"; fetch(url, { method: 'GET', credentials: 'include', headers: { 'accept': 'application/json;odata=nometadata' } }).then(function (response) { if (response.status !== 200) { throw new Error(response.status + " " + response.statusText); } return response.json(); }).then(function (json) { console.log("hasuniqueassignments(" + list + ")" + " returned " + JSON.stringify(json, null, 4)); if (json.value) { rdm.removeGroupPermissions(list); } else { rdm.breakRoleInheritance(list); } }).catch(function (error) { alert(JSON.stringify(error, null, 4)); }); }; // execute immediately if {connectionMax} is not exceeded, otherwise // try again every {interval} milliseconds until {connectionMax} is // not exceeded, and then execute rdm.executeOrDelayUntilConnectionAvailable( doListInternal, rdm.interval); }, |
setRoleAssignments just loops through the lists and calls setRollAssignmentsOnList. It just mostly does a call to the REST endpoint hasuniquerollassignments on the list. It does this to see if it should then break role inheritance, or just delete any role assignments from the list for the principal id. The code for these methods is omitted as uninteresting, but either of them ultimately calls a method that creates the new role assignments on the list. The more interesting thing in the above code is that the web service call is encapsulated in the nested method doListInternal, and the setRollAssignmentsOnList method ends with a call to executeOrDelayUntilConnectionAvailable.
If you’ve been doing any SharePoint JavaScript code, you must have heard about ExecuteOrDelayUntilScriptLoaded. executeOrDelayUntilConnectionAvailable (shown below) does basically the same thing but with different criteria for delaying. reserveConnection just increments a counter of connections and returns the current counter if a connection is available (i.e. max connections would not be exceeded). If no connection is available, it returns 0. releaseConnection just decrements that counter. So executeOrDelayUntilConnectionAvailable tries to reserve a connection and call the callback. If it cannot, it calls setInterval with the interval passed in and upon waking up tries again, over and over every interval until it successfully reserves a connection and calls the callback. Obviously, this is where the throttling can be tweaked, mostly by adjusting the maximum connections and/or the interval.
Note that each browser has a maximum number of concurrent connections and a lower maximum number of concurrent connections per host. When I started this, a fundamental misunderstanding led me to believe I had to do something like this reserve connection scheme because of these limits. But if you ask for more connections than the browser allows it doesn’t fail (at least not until a very high number in modern browsers). The browser just queues up your request and makes the connection when one is available much as I am doing. Still, doing it myself does allow me to throttle my requests which is quite useful. So if I set the maximum to 100, I’ll always reserve a connection as soon as I ask so I’ll plow ahead as soon as the browser will let me. If you look at the network tab of the developer tools, you’ll see that it still doesn’t open up 36 connections at once, there are just no appreciable gaps between close and open. I wouldn’t be surprised if there is still some high limit beyond which the browser will barf, but I haven’t hit it, and it’s probably browser specific.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 | connectionCount: 0, totalExpected: 0, totalRelease: 0, startTime: null, endTime: null, // change these to throttle, lowering connectionMax and increasing // interval will slow down time to complete, but also reduce load on the // server connectionMax: 5, interval: 1000, /** * Reserve a connection for a callback to use if it would not exceed {connectionMax} connections. * Othewise, set and interval for {interval} milliseconds and try again when called back. * @param callback {function} the function to be called when a connection is available * @param interval {interval} the number of milliseconds to wait before trying again * @return undefined */ executeOrDelayUntilConnectionAvailable: function(callback, interval) { if(rdm.reserveConnection()) { callback(); } else { // check every 250ms if there is a connection available to us var i = setInterval(function() { console.log("checking for connection again"); // returns false if we have too many connections if(rdm.reserveConnection()) { console.log("got connection after waiting"); // not too many connections, do our work callback(); // release the timer clearInterval(i); } // otherwise, try again in 250ms }, interval); } }, /** * Reserve a connection if one is available. * @return {int} the 1-based id of the connection, or 0 if none available */ reserveConnection: function() { console.log("reserving connection, count = " + rdm.connectionCount); rdm.connectionCount++; if(rdm.connectionCount > rdm.connectionMax) { console.log("waiting for connection, count = " + rdm.connectionCount); rdm.connectionCount--; return 0; } document.getElementById("spinnerOverlay").style.display = "block"; return rdm.connectionCount; }, /** * Release a connection reservation. * @return {int} the number of connections still reserved */ releaseConnection: function() { console.log("releasing connection, count = " + rdm.connectionCount); rdm.connectionCount--; rdm.totalReleased++; if(rdm.totalReleased === rdm.totalExpected) { rdm.endTime = new Date(); var elapsed = Math.round((rdm.endTime - rdm.startTime) / 100); console.log("elapsed time: " + elapsed/10); document.getElementById("spinnerOverlay").style.display = "none"; } return rdm.connectionCount; }, |
And that’s really all the code I’m going to dump on you here. The rest of it is more of the same, I described the more interesting REST-related bits in my last post, and as usually I’ll include a download to a complete working page at the end of the post.
Sum Up Role Assignments Sample
The code for this sample was really quite the PITA, which is why I split this post into two parts. At some point it would be interesting to redo this using batch operations à la the Andrew Connell articles referenced below. But I wouldn’t do that without writing some kind of wrapper around the painfully tedious multi-part mime parsing described in those articles, and I’ll make no promises about when I might get around to that.
Reference
- Browserscope – information about connection limits in many modern browsers
- Part 1 – SharePoint REST API Batching – Understanding Batching Requests – Andrew Connell
- Part 2 – SharePoint REST API Batching – Exploring Batch Requests, Responses and Changesets – Andrew Connell