Scheduler Module
- class resources.scheduler.ResourceScheduler(user_resources=None, sched_opts={})
Calculates and returns resource set ids from a dictionary of resource sets by group. The available resource sets are read initially from the resources module or from a resources object passed in.
Resource sets are locally provisioned to work items by a call to the
assign_resources
function, and a cache of available resource sets is maintained for the life of the object (usually corresponding to one call of the allocation function). Note that work item resources are formally assigned to workers only when a work item is sent to the worker.- assign_resources(rsets_req, use_gpus=None, user_params=[])
Schedule resource sets to a work item if possible.
If the resources required are less than one node, they will be allocated to the smallest available sufficient slot.
If the resources required are more than one node, then the scheduler will attempt to find an even split. If no even split is possible, then enough additional resource sets will be assigned to enable an even split.
Returns a list of resource set IDs or raises an exception (either InsufficientResourcesError or InsufficientFreeResources).
- find_rsets_any_slots(valid_rsets_by_group, max_grpsize, rsets_req, ngroups, rsets_per_group)
Find optimal non-matching slots across groups
- find_candidate(rsets_by_group, group_list, rsets_per_group, max_upper_bound)
Find a candidate slot in a group
- get_avail_rsets_by_group()
Return a dictionary of resource set IDs for each group (e.g. node)
If groups are not set they will all be in one group (group 0)
E.g: Say 8 resource sets / 2 nodes GROUP 1: [1,2,3,4] GROUP 2: [5,6,7,8]
- filter_for_rset_type(avail_rsets_by_group, use_gpus)
Return avail_rsets_by_group filtered by rset type (gpus/non-gpus/all)
- filter_out_rset_team(avail_rsets_by_group, rset_team)
Return avail_rsets_by_group filtered by rset type (gpus/non-gpus/all)
- static get_slots_of_len(d, n)
Filter dictionary to values >= n
- get_avail_slots_by_group(rsets_by_group)
Return a dictionary of free slot IDS for each group (e.g. node)
- calc_rsets_even_grps(rsets_req, max_grpsize, max_groups, extend)
Calculate an even breakdown to best fit rsets_req input
- check_params(user_params, ngroups)
Return True if all user params divide by number of groups, else False
- calc_even_split_uneven_groups(rsets_per_grp, ngroups, rsets_req, sorted_lens, max_grps, user_params)
Calculate an even breakdown to best fit rsets_req with uneven groups
- assign_team_from_slots(slots_avail_by_group, cand_groups, cand_slots, rsets_per_group)
Assign resource set team from slots
- static get_sorted_lens(avail_rsets)
Get max length of a list value in a dictionary
- get_matching_slots(slots_avail_by_group, num_groups_req, rsets_per_group)
Get first N matching slots across groups
Assumes num_groups_req > 0.
- check_total_rsets(rsets_req, use_gpus)
Raise exceptions if rsets requested is more than total that exist or available