Scheduler Module

class resources.scheduler.ResourceScheduler(user_resources=None, sched_opts={})

Calculates and returns resource set ids from a dictionary of resource sets by group. The available resource sets are read initially from the resources module or from a resources object passed in.

Resource sets are locally provisioned to work items by a call to the assign_resources function, and a cache of available resource sets is maintained for the life of the object (usually corresponding to one call of the allocation function). Note that work item resources are formally assigned to workers only when a work item is sent to the worker.

assign_resources(rsets_req, use_gpus=None, user_params=[])

Schedule resource sets to a work item if possible.

If the resources required are less than one node, they will be allocated to the smallest available sufficient slot.

If the resources required are more than one node, then the scheduler will attempt to find an even split. If no even split is possible, then enough additional resource sets will be assigned to enable an even split.

Returns a list of resource set IDs or raises an exception (either InsufficientResourcesError or InsufficientFreeResources).

find_rsets_any_slots(valid_rsets_by_group, max_grpsize, rsets_req, ngroups, rsets_per_group)

Find optimal non-matching slots across groups

find_candidate(rsets_by_group, group_list, rsets_per_group, max_upper_bound)

Find a candidate slot in a group

get_avail_rsets_by_group()

Return a dictionary of resource set IDs for each group (e.g. node)

If groups are not set they will all be in one group (group 0)

E.g: Say 8 resource sets / 2 nodes GROUP 1: [1,2,3,4] GROUP 2: [5,6,7,8]

filter_for_rset_type(avail_rsets_by_group, use_gpus)

Return avail_rsets_by_group filtered by rset type (gpus/non-gpus/all)

filter_out_rset_team(avail_rsets_by_group, rset_team)

Return avail_rsets_by_group filtered by rset type (gpus/non-gpus/all)

static get_slots_of_len(d, n)

Filter dictionary to values >= n

get_avail_slots_by_group(rsets_by_group)

Return a dictionary of free slot IDS for each group (e.g. node)

calc_rsets_even_grps(rsets_req, max_grpsize, max_groups, extend)

Calculate an even breakdown to best fit rsets_req input

check_params(user_params, ngroups)

Return True if all user params divide by number of groups, else False

calc_even_split_uneven_groups(rsets_per_grp, ngroups, rsets_req, sorted_lens, max_grps, user_params)

Calculate an even breakdown to best fit rsets_req with uneven groups

assign_team_from_slots(slots_avail_by_group, cand_groups, cand_slots, rsets_per_group)

Assign resource set team from slots

static get_sorted_lens(avail_rsets)

Get max length of a list value in a dictionary

get_matching_slots(slots_avail_by_group, num_groups_req, rsets_per_group)

Get first N matching slots across groups

Assumes num_groups_req > 0.

check_total_rsets(rsets_req, use_gpus)

Raise exceptions if rsets requested is more than total that exist or available