3 ms·

But how is the length of autoregression be determined?

• onesphere 3y ago
  If each token leads to another, it eventually leads to what commonly comes out as an ‘end token’. So each token has a certain chance of leading to a next token which might, possibly, say to end. current_token = <prompt> while current_token == true: next_token = statistically[current_token][choose_random_from(common_order)] current_token = next_token s = inf for i in common_order: p = 1 x = i while true: x = statistically[x][common_order[x]] if x == false: break p *= x if p < s: s = p shortest_to_end_start_token = i
  • onesphere 3y ago
    Or: s = - inf ... if p > s: s = p longest_to_end_start_token = i And then maybe you could sort all tokens by their length (if you wanted to find a length of sub-x).
    • onesphere 3y ago
      The dimension of variable 'p' (or its number of bits) could depend on the length of common_order (size of training data).
      • onesphere 3y ago
        Or: x = statistically[x][common_order[x][-1]] # least probable
        • onesphere 3y ago
          The eency weency ... got washed out.