1 Introduction
I had some success in classifying the TMRC2 samples by strain via ML and want to try something more difficult. Thus I will use the normalized gene expression data and try classifying it by cure/fail.
ref_col <- "finaloutcome"
outcome_factor <- as.factor(as.character(pData(tc_clinical)[[ref_col]]))2 Starter data
In the strain classifier I used normalized variants. I am thinking to use normalized expression here and therefore explicitly limit myself to ~ 20k variables (significantly down from the 1.6M).
In addition, caret expects the data as (rows == samples) and (columns == variables) where each element is one observation. Thus we will need to transpose the expression matrix.
tc_norm <- normalize_expt(tc_clinical, transform = "log2", convert = "cpm")## transform_counts: Found 1005383 values equal to 0, adding 1 to the matrix.texprs <- t(exprs(tc_norm))3 Filtering
The ML text I am reading provide some neat examples for how one might filter the data to make it more suitable for model creation.
3.1 Near zero variance, or genefilter’s cv
The first filter I was introduced to is quite familiar from our sequencing data, the removal of features with near-zero-variance. Indeed, I am pretty certain that normalize_expt() could do this equivalently and significantly faster than caret::preProcess().
system.time({
equivalent <- normalize_expt(tc_norm, filter = "cv", cv_min = 0.1)
})## Removing 3174 low-count genes (16749 remaining).## user system elapsed
## 4.712 0.565 5.309dim(exprs(equivalent))## [1] 16749 184## Given this large amount of data, this step is slow, taking > 10 minutes.
## Yeah seriously, the following three lines get 16,723 genes in 10 minutes while
## the normalize_expt() call above gets 16,749 genes in 2.4 seconds.
#system.time({
# nzv <- preProcess(texprs, method="nzv", uniqueCut=15)
# nzv_texprs <- predict(nzv, texprs)
# dim(nzv_texprs)
#}
nzv_texprs <- t(exprs(equivalent))3.2 Filtering to the highest standard deviation variables
I think I am a bit confused by this filter, one would think that the nzv filter above, if applied correctly, should give you exactly this.
For the moment, I am excluding the following block in order to see how much time/memory keeping these variables costs. If I recall properly, the model of the top-2k variant positions cost ~ 1-4G of memory. I hope that this scales linearly, but I am thinking it might not.
standard_devs <- apply(nzv_texprs, 2, sd)
top_predictors <- order(standard_devs, decreasing = TRUE)[1:3000]
nzv_texprs <- nzv_texprs[, top_predictors]3.3 Center the data
I think centering may not be needed for this data, but here is how:
nzv_center <- preProcess(nzv_texprs, method = "center")
nzv_texprs <- predict(nzv_center, nzv_texprs)4 Merge the appropriate metadata
There are a few metadata factors which might prove of interest for classification. The most obvious are of course outcome, clinic, donor, visit, celltype. I am, for the moment, only likely to focus on outcome. AFAICT I can only include one of these at a time in the data, which is a shame.
interesting_meta <- pData(tc_clinical)[, c("finaloutcome", "donor", "persistence",
"visitnumber", "selectionmethod",
"typeofcells", "time", "clinic")]
ml_df <- as.data.frame(cbind(outcome_factor, as.data.frame(nzv_uncorr)))
ml_df[["outcome_factor"]] <- as.factor(ml_df[["outcome_factor"]])
dim(ml_df)## [1] 184 120815 Split the data into training/testing
caret provides nice functionality for splitting up the data. I suspect there are many more fun knobs I can play with for instances where I need to exclude some levels of a factor and such. In this case I just want to split by outcome.
5.1 Via data splitting
## The variable outcome_factor was created at the top of this document,
## which is a little awkward here.
train_idx <- createDataPartition(outcome_factor, p = 0.6,
list = FALSE,
times = 1)
summary(ml_df[train_idx, "outcome_factor"])## cure failure
## 74 38## Training set has 112 samples.
train_rownames <- rownames(ml_df)[train_idx]
not_train_idx <- ! rownames(ml_df) %in% train_rownames
summary(ml_df[not_train_idx, "outcome_factor"])## cure failure
## 48 24not_train_rownames <- rownames(ml_df)[not_train_idx]
train_df <- as.data.frame(ml_df[train_rownames, ])
dim(train_df)## [1] 112 12081test_df <- as.data.frame(ml_df[not_train_rownames, ])
dim(test_df)## [1] 72 12081## Remove the outcome factor from the test data, just in case.
test_outcome <- test_df[["outcome_factor"]]
test_df[["outcome_factor"]] <- NULL5.2 Via sampling
There are a few likely sampling methods: cross-validation, bootstrapping, and jackknifing. I will try those out later.
6 Try out training and prediction methods
My goals from here on will be to get the beginnings of a sense of the various methods I can use to create the models from the training data and predict the outcome on the test data. I am hoping also to pick up some idea of what the various arguments mean while I am at it.
6.1 Try out KNN
k-nearest neighbors is somewhat similar to a kmeans estimate. Thus the primary argument is ‘k’
6.1.1 Model creation and performance
knn_fit <- knn3(x = train_df[, -1],
y = train_df[["outcome_factor"]],
k = 3)
knn_predict_trained <- predict(knn_fit, train_df[, -1], type = "class")
confusionMatrix(data = train_df[[1]], reference = knn_predict_trained)## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 69 5
## failure 4 34
##
## Accuracy : 0.92
## 95% CI : (0.853, 0.963)
## No Information Rate : 0.652
## P-Value [Acc > NIR] : 3.5e-11
##
## Kappa : 0.822
##
## Mcnemar's Test P-Value : 1
##
## Sensitivity : 0.945
## Specificity : 0.872
## Pos Pred Value : 0.932
## Neg Pred Value : 0.895
## Prevalence : 0.652
## Detection Rate : 0.616
## Detection Prevalence : 0.661
## Balanced Accuracy : 0.909
##
## 'Positive' Class : cure
## As the confusion matrix shows, this failed for a few samples. Perhaps let us change k and see if it improves.
Here is a table of fase positives/negatives for a few values of ‘k’, in this context a false positive is calling a known cure as a failure and false negative is calling a known failure as a cure.
|—|—|—| |k |fp |fn | |2 |0 |8 | |3 |5 |5 | |4 |8 |9 | |5 |11 |7 | |6 |15 |8 |
Note: this depends on the luck of rand(), so the above numbers shift moderately from one run to the next. Thus I think I will just use 2 or 3.
knn_fit <- knn3(x = train_df[, -1],
y = train_df[["outcome_factor"]],
k = 3)
knn_predict_trained <- predict(knn_fit, train_df[, -1], type = "class")
confusionMatrix(data = train_df[[1]], reference = knn_predict_trained)## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 69 5
## failure 4 34
##
## Accuracy : 0.92
## 95% CI : (0.853, 0.963)
## No Information Rate : 0.652
## P-Value [Acc > NIR] : 3.5e-11
##
## Kappa : 0.822
##
## Mcnemar's Test P-Value : 1
##
## Sensitivity : 0.945
## Specificity : 0.872
## Pos Pred Value : 0.932
## Neg Pred Value : 0.895
## Prevalence : 0.652
## Detection Rate : 0.616
## Detection Prevalence : 0.661
## Balanced Accuracy : 0.909
##
## 'Positive' Class : cure
## 6.1.2 Visualize the accuracy with a ROC curve
names(knn_predict_trained) <- rownames(train_df)
summary(knn_predict_trained == pData(tc_norm)[train_idx, "finaloutcome"])## Mode FALSE TRUE
## logical 9 103tt <- predict(knn_fit, train_df[, -1])
curve <- pROC::roc(response = train_df[, 1], predictor = tt[, 1],
levels = rev(levels(outcome_factor)))## Setting direction: controls < casesplot(curve)
pROC::auc(curve)## Area under the curve: 0.9676.1.3 Predict the rest of the data with this model.
## Return the classifications
knn_predict_test <- predict(knn_fit, test_df, type = "class")
## and the values, which will be used for our ROC curve.
knn_predict_test_values <- predict(knn_fit, test_df)
names(knn_predict_test) <- rownames(test_df)
knn_agreement <- knn_predict_test == test_outcome
names(knn_agreement) <- rownames(test_df)
summary(knn_agreement)## Mode FALSE TRUE
## logical 16 56names(knn_agreement)[knn_agreement == FALSE]## [1] "TMRC30018" "TMRC30210" "TMRC30105" "TMRC30164" "TMRC30119" "TMRC30103" "TMRC30026" "TMRC30165" "TMRC30044" "TMRC30166" "TMRC30200"
## [12] "TMRC30177" "TMRC30208" "TMRC30218" "TMRC30220" "TMRC30264"curve <- pROC::roc(response = test_outcome,
predictor = knn_predict_test_values[, 1])## Setting levels: control = cure, case = failure## Setting direction: controls > casesplot(curve)
pROC::auc(curve)## Area under the curve: 0.803## Find the samples which (dis)agree
ref_col <- "finaloutcome"
ref_result <- pData(tc_clinical)[[ref_col]]
names(ref_result) <- rownames(pData(tc_clinical))
ref_xref <- names(ref_result) %in% names(knn_predict_test)
ref_comp <- ref_result[ref_xref]
ref_comp_idx <- ref_comp != "unknown"
ref_comp <- ref_comp[ref_comp_idx]
ref_comp_idx <- !is.na(ref_comp)
ref_comp <- ref_comp[ref_comp_idx]
test_comp_idx <- names(knn_predict_test) %in% names(ref_comp)
test_comp <- knn_predict_test[test_comp_idx]
agreement <- as.character(ref_comp) == as.character(test_comp)
agree_num <- sum(agreement)
agree_prop <- agree_num / length(ref_comp)
agree_prop## [1] 0.7778disagree <- as.character(ref_comp) != as.character(test_comp)
test_comp[disagree]## TMRC30018 TMRC30210 TMRC30105 TMRC30164 TMRC30119 TMRC30103 TMRC30026 TMRC30165 TMRC30044 TMRC30166 TMRC30200 TMRC30177 TMRC30208 TMRC30218
## failure failure cure failure cure failure cure failure failure failure cure cure cure cure
## TMRC30220 TMRC30264
## cure cure
## Levels: cure failure6.2 Perform cross-validation to estimate k
The cross validation method of repeated sampling the data is all done within the train() function. With that in mind, here it is operating with the knn method.
6.2.1 CV with knn
When train() is called with the trControl and tuneGrid, we can control how the knn training is repeated, in this case it will iterate over k from 1 to 10.
cv_control <- trainControl(method = "cv", number = 10)
knn_train_fit <- train(outcome_factor ~ ., data = train_df,
method = "knn",
trControl = cv_control,
tuneGrid = data.frame(k = 1:10))
knn_train_fit[["bestTune"]]## k
## 1 1plot(x = 1:10, 1 - knn_train_fit$results[, 2], pch = 19,
ylab = "prediction error", xlab = "k")
lines(loess.smooth(x = 1:10, 1 - knn_train_fit$results[, 2],degree = 2),
col = "#CC0000")
6.2.2 Bootstrap with knn
boot_control <- trainControl(method = "boot", number = 20,
returnResamp = "all")
knn_train_fit <- train(outcome_factor ~ ., data = train_df,
method = "knn",
trControl = boot_control,
tuneGrid = data.frame(k = 1:10))
knn_train_fit[["bestTune"]]## k
## 1 1plot(x = 1:10, 1 - knn_train_fit$results[, 2], pch = 19,
ylab = "prediction error", xlab = "k")
lines(loess.smooth(x = 1:10, 1 - knn_train_fit$results[, 2],degree = 2),
col = "#CC0000")
6.2.3 Explain the important variables
In this instance we will search for genes which were important for the model’s creation.
The DALEX package provides a function: feature_importance() which seeks to use a series of other methods to extract (in this case, genes) features which do a good job of explaining the result produced by the model. In the case of this dataset, which has thousands of features, this does not appear to end well.
explainer_knn <- explain(knn_fit, label = "knn",
data = train_df[, -1],
y = as.numeric(train_df[, 1]))## Preparation of a new explainer is initiated
## -> model label : knn
## -> data : 112 rows 12080 cols
## -> target variable : 112 values
## -> predict function : yhat.default will be used ( default )
## -> predicted values : No value for predict function target column. ( default )
## -> model_info : package Model of class: knn3 package unrecognized , ver. Unknown , task regression ( default )
## -> predicted values : numerical, min = 0 , mean = 0.3988 , max = 1
## -> residual function : difference between y and yhat ( default )
## -> residuals : numerical, min = 0.3333 , mean = 0.9405 , max = 1.667
## A new explainer has been created!## AFAICT the following will take forever unless we drastically reduce the complexity of the model.
## features <- feature_importance(explainer_knn, n_sample = 50, type = "difference")6.3 Random Forest
The parameter ‘mtry’ is often important, if I read the text correctly it controls how many variables to sample in each split of the tree. Thus higher numbers should presumably make it more specific at the risk of overfitting.
Setting min.node.size sets the minimume node size of terminal nodes in each tree. Each increment up speeds the algorithm.
I am going to use my boot control trainer from above and see how it goes.
library(ranger)
##rf_method <- trainControl(method = "ranger", number = 10, verbose = TRUE)
rf_train_fit <- train(outcome_factor ~ ., data = train_df,
method = "ranger", trControl = boot_control,
importance = "permutation",
tuneGrid = data.frame(
mtry = 200,
min.node.size = 1,
splitrule = "gini"),
verbose = TRUE)
rf_train_fit[["finalModel"]][["prediction.error"]]## [1] 0.1964variable_importance <- varImp(rf_train_fit)
plot(variable_importance, top = 15)
rf_predict_trained <- predict(rf_train_fit, train_df)
names(rf_predict_trained) <- rownames(train_df)
summary(rf_predict_trained == pData(tc_norm)[train_idx, "finaloutcome"])## Mode TRUE
## logical 1126.3.0.1 Digression do the genes provide by varImp mean anything?
Let us take a moment and see if the top-n genes returned by varImp() have some meaning which jumps out. One might assume, given our extant Differential Expression results, that the interleukin response will be a likely candidate.
test_importance <- varImp(rf_train_fit, scale = TRUE)
test_idx <- order(test_importance[["importance"]][["Overall"]], decreasing = TRUE)
test_imp <- test_importance[["importance"]][test_idx, ]
names(test_imp) <- rownames(test_importance[["importance"]])[test_idx]
head(test_imp, n = 200)## ENSG00000173702 ENSG00000103642 ENSG00000183741 ENSG00000248405 ENSG00000164303 ENSG00000108774 ENSG00000181404 ENSG00000172426 ENSG00000126709
## 100.00 84.95 83.39 76.51 68.64 66.18 62.33 62.17 61.17
## ENSG00000137965 ENSG00000134758 ENSG00000185304 ENSG00000180720 ENSG00000187608 ENSG00000160201 ENSG00000187950 ENSG00000081479 ENSG00000129295
## 60.16 59.37 54.80 54.13 53.95 50.57 50.49 48.42 47.02
## ENSG00000256870 ENSG00000237440 ENSG00000168280 ENSG00000106009 ENSG00000160932 ENSG00000172613 ENSG00000197826 ENSG00000174898 ENSG00000059378
## 46.33 45.91 44.97 44.19 43.11 42.47 42.25 41.59 41.24
## ENSG00000204632 ENSG00000182253 ENSG00000142396 ENSG00000126003 ENSG00000167608 ENSG00000073734 ENSG00000103226 ENSG00000140104 ENSG00000115155
## 39.88 39.29 38.36 38.30 37.33 37.10 35.40 34.21 33.74
## ENSG00000163435 ENSG00000138642 ENSG00000138035 ENSG00000167098 ENSG00000173193 ENSG00000136206 ENSG00000090339 ENSG00000166920 ENSG00000226288
## 33.73 33.60 33.11 33.09 32.59 32.42 32.13 31.51 31.12
## ENSG00000101605 ENSG00000196345 ENSG00000236444 ENSG00000284776 ENSG00000181315 ENSG00000187626 ENSG00000184979 ENSG00000184508 ENSG00000176945
## 31.12 31.05 30.78 30.68 30.40 30.30 30.22 29.79 29.57
## ENSG00000155530 ENSG00000130813 ENSG00000260272 ENSG00000204428 ENSG00000167962 ENSG00000168255 ENSG00000108830 ENSG00000174132 ENSG00000198870
## 29.30 29.25 28.72 28.68 28.36 27.99 27.67 27.57 27.48
## ENSG00000284194 ENSG00000104691 ENSG00000179933 ENSG00000286169 ENSG00000186654 ENSG00000151005 ENSG00000135365 ENSG00000130876 ENSG00000146809
## 27.45 27.38 27.12 26.98 26.93 26.90 26.86 26.79 26.71
## ENSG00000095015 ENSG00000096060 ENSG00000144560 ENSG00000196369 ENSG00000106436 ENSG00000185347 ENSG00000197822 ENSG00000148572 ENSG00000125821
## 26.55 26.55 26.47 26.33 26.29 26.28 26.11 26.07 25.98
## ENSG00000178084 ENSG00000144744 ENSG00000087303 ENSG00000176222 ENSG00000173698 ENSG00000176390 ENSG00000127314 ENSG00000166295 ENSG00000177875
## 25.98 25.86 25.79 25.79 25.77 25.64 25.49 25.49 25.29
## ENSG00000100523 ENSG00000161640 ENSG00000253626 ENSG00000054179 ENSG00000214360 ENSG00000179455 ENSG00000165164 ENSG00000198189 ENSG00000150455
## 25.18 25.11 24.94 24.91 24.91 24.69 24.58 24.45 24.42
## ENSG00000121058 ENSG00000163116 ENSG00000110768 ENSG00000136574 ENSG00000134326 ENSG00000105248 ENSG00000188157 ENSG00000148339 ENSG00000179407
## 24.42 24.41 24.41 24.39 24.39 24.27 24.22 24.22 24.05
## ENSG00000136244 ENSG00000205857 ENSG00000162627 ENSG00000183454 ENSG00000260001 ENSG00000269028 ENSG00000206560 ENSG00000204519 ENSG00000137628
## 24.00 23.78 23.73 23.68 23.65 23.50 23.47 23.43 23.42
## ENSG00000232629 ENSG00000084734 ENSG00000158477 ENSG00000185361 ENSG00000154654 ENSG00000179008 ENSG00000196839 ENSG00000168958 ENSG00000107736
## 23.32 23.21 23.20 23.01 22.99 22.97 22.96 22.95 22.93
## ENSG00000284505 ENSG00000172456 ENSG00000188981 ENSG00000137331 ENSG00000188039 ENSG00000104856 ENSG00000164007 ENSG00000006634 ENSG00000110906
## 22.93 22.91 22.91 22.84 22.80 22.80 22.72 22.61 22.61
## ENSG00000119522 ENSG00000168671 ENSG00000128872 ENSG00000169427 ENSG00000186714 ENSG00000172986 ENSG00000103375 ENSG00000134690 ENSG00000163645
## 22.55 22.52 22.50 22.43 22.37 22.29 22.29 22.29 22.29
## ENSG00000185038 ENSG00000153498 ENSG00000009790 ENSG00000160867 ENSG00000121653 ENSG00000133106 ENSG00000174145 ENSG00000053770 ENSG00000146007
## 22.29 22.25 22.25 22.25 22.18 22.18 22.09 22.09 22.09
## ENSG00000146205 ENSG00000155158 ENSG00000162972 ENSG00000165688 ENSG00000179344 ENSG00000241149 ENSG00000178150 ENSG00000096006 ENSG00000111450
## 22.09 22.09 22.09 22.09 22.09 22.09 22.02 22.01 21.92
## ENSG00000183793 ENSG00000166971 ENSG00000170807 ENSG00000161594 ENSG00000232268 ENSG00000241553 ENSG00000133169 ENSG00000120217 ENSG00000260097
## 21.90 21.90 21.86 21.85 21.83 21.82 21.82 21.82 21.76
## ENSG00000055332 ENSG00000158865 ENSG00000162714 ENSG00000166133 ENSG00000143156 ENSG00000137204 ENSG00000104267 ENSG00000105662 ENSG00000174243
## 21.75 21.71 21.71 21.71 21.71 21.63 21.54 21.54 21.54
## ENSG00000070761 ENSG00000085871 ENSG00000162775 ENSG00000137802 ENSG00000152767 ENSG00000158806 ENSG00000204001 ENSG00000125900 ENSG00000010818
## 21.54 21.54 21.53 21.42 21.42 21.42 21.39 21.38 21.38
## ENSG00000125355 ENSG00000149124 ENSG00000182348 ENSG00000115419 ENSG00000102271 ENSG00000241399 ENSG00000112419 ENSG00000152779 ENSG00000169994
## 21.38 21.38 21.33 21.31 21.22 21.22 21.22 21.22 21.22
## ENSG00000004948 ENSG00000145920
## 21.09 21.09most_important_ids <- names(head(test_imp, n = 200))
convert_ids(most_important_ids, from = "ENSEMBL", to = "SYMBOL")## Before conversion: 200, after conversion: 197.## ENSEMBL SYMBOL
## 1 ENSG00000173702 MUC13
## 2 ENSG00000103642 LACTB
## 3 ENSG00000183741 CBX6
## 4 ENSG00000248405 PRR5-ARHGAP8
## 5 ENSG00000164303 ENPP6
## 6 ENSG00000108774 RAB5C
## 7 ENSG00000181404 WASHC1
## 8 ENSG00000172426 RSPH9
## 9 ENSG00000126709 IFI6
## 10 ENSG00000137965 IFI44
## 11 ENSG00000134758 RNF138
## 12 ENSG00000185304 RGPD2
## 13 ENSG00000180720 CHRM4
## 14 ENSG00000187608 ISG15
## 15 ENSG00000160201 U2AF1
## 16 ENSG00000160201 LOC102724594
## 17 ENSG00000187950 OVCH1
## 18 ENSG00000081479 LRP2
## 19 ENSG00000129295 DNAAF11
## 20 ENSG00000256870 SLC5A8
## 21 ENSG00000237440 ZNF737
## 22 ENSG00000168280 KIF5C
## 23 ENSG00000106009 BRAT1
## 24 ENSG00000160932 LY6E
## 25 ENSG00000172613 RAD9A
## 26 ENSG00000197826 CFAP299
## 27 ENSG00000174898 CATSPERD
## 28 ENSG00000059378 PARP12
## 29 ENSG00000204632 HLA-G
## 30 ENSG00000182253 SYNM
## 31 ENSG00000142396 ERVK3-1
## 32 ENSG00000126003 PLAGL2
## 33 ENSG00000167608 TMC4
## 34 ENSG00000073734 ABCB11
## 35 ENSG00000103226 NOMO3
## 36 ENSG00000140104 CLBA1
## 37 ENSG00000115155 OTOF
## 38 ENSG00000163435 ELF3
## 39 ENSG00000138642 HERC6
## 40 ENSG00000138035 PNPT1
## 41 ENSG00000167098 SUN5
## 42 ENSG00000173193 PARP14
## 43 ENSG00000136206 SPDYE1
## 44 ENSG00000090339 ICAM1
## 45 ENSG00000166920 C15orf48
## 46 ENSG00000226288 OR52I2
## 47 ENSG00000101605 MYOM1
## 48 ENSG00000196345 ZKSCAN7
## 49 ENSG00000236444 UBE2L5
## 51 ENSG00000181315 ZNF322
## 52 ENSG00000187626 ZKSCAN4
## 53 ENSG00000184979 USP18
## 54 ENSG00000184508 HDDC3
## 55 ENSG00000176945 MUC20
## 56 ENSG00000155530 LRGUK
## 57 ENSG00000130813 SHFL
## 59 ENSG00000204428 LY6G5C
## 60 ENSG00000167962 ZNF598
## 61 ENSG00000168255 POLR2J3-UPK3BL2
## 62 ENSG00000108830 RND2
## 63 ENSG00000174132 FAM174A
## 64 ENSG00000198870 STKLD1
## 65 ENSG00000284194 SCO2
## 66 ENSG00000104691 UBXN8
## 67 ENSG00000179933 C14orf119
## 69 ENSG00000186654 PRR5
## 70 ENSG00000151005 TKTL2
## 71 ENSG00000135365 PHF21A
## 72 ENSG00000130876 SLC7A10
## 73 ENSG00000146809 ASB15
## 74 ENSG00000095015 MAP3K1
## 75 ENSG00000096060 FKBP5
## 76 ENSG00000144560 VGLL4
## 77 ENSG00000196369 SRGAP2B
## 78 ENSG00000106436 MYL10
## 79 ENSG00000185347 TEDC1
## 80 ENSG00000197822 OCLN
## 81 ENSG00000148572 NRBF2
## 82 ENSG00000125821 DTD1
## 83 ENSG00000178084 HTR3C
## 84 ENSG00000144744 UBA3
## 85 ENSG00000087303 NID2
## 86 ENSG00000176222 ZNF404
## 87 ENSG00000173698 ADGRG2
## 88 ENSG00000176390 CRLF3
## 89 ENSG00000127314 RAP1B
## 90 ENSG00000166295 ANAPC16
## 91 ENSG00000177875 CCDC184
## 92 ENSG00000100523 DDHD1
## 93 ENSG00000161640 SIGLEC11
## 94 ENSG00000253626 EIF5AL1
## 95 ENSG00000054179 ENTPD2
## 96 ENSG00000214360 EFCAB9
## 97 ENSG00000179455 MKRN3
## 98 ENSG00000165164 CFAP47
## 99 ENSG00000198189 HSD17B11
## 100 ENSG00000150455 TIRAP
## 101 ENSG00000121058 COIL
## 102 ENSG00000163116 STPG2
## 103 ENSG00000110768 GTF2H1
## 104 ENSG00000136574 GATA4
## 105 ENSG00000134326 CMPK2
## 106 ENSG00000105248 YJU2
## 107 ENSG00000188157 AGRN
## 108 ENSG00000148339 SLC25A25
## 109 ENSG00000179407 DNAJB8
## 110 ENSG00000136244 IL6
## 111 ENSG00000205857 NANOGNB
## 112 ENSG00000162627 SNX7
## 113 ENSG00000183454 GRIN2A
## 114 ENSG00000260001 TGFBR3L
## 116 ENSG00000206560 ANKRD28
## 117 ENSG00000204519 ZNF551
## 118 ENSG00000137628 DDX60
## 119 ENSG00000232629 HLA-DQB2
## 120 ENSG00000084734 GCKR
## 121 ENSG00000158477 CD1A
## 122 ENSG00000185361 TNFAIP8L1
## 123 ENSG00000154654 NCAM2
## 124 ENSG00000179008 C14orf39
## 125 ENSG00000196839 ADA
## 126 ENSG00000168958 MFF
## 127 ENSG00000107736 CDH23
## 128 ENSG00000284505 LYNX1-SLURP2
## 129 ENSG00000172456 FGGY
## 130 ENSG00000188981 MSANTD1
## 131 ENSG00000137331 IER3
## 132 ENSG00000188039 NWD1
## 133 ENSG00000104856 RELB
## 134 ENSG00000164007 CLDN19
## 135 ENSG00000006634 DBF4
## 136 ENSG00000110906 KCTD10
## 137 ENSG00000119522 DENND1A
## 138 ENSG00000168671 UGT3A2
## 139 ENSG00000128872 TMOD2
## 140 ENSG00000169427 KCNK9
## 141 ENSG00000186714 CCDC73
## 142 ENSG00000172986 GXYLT2
## 143 ENSG00000103375 AQP8
## 144 ENSG00000134690 CDCA8
## 145 ENSG00000163645 ERICH6
## 146 ENSG00000185038 MROH2A
## 147 ENSG00000153498 SPACA7
## 148 ENSG00000009790 TRAF3IP3
## 149 ENSG00000160867 FGFR4
## 150 ENSG00000121653 MAPK8IP1
## 151 ENSG00000133106 EPSTI1
## 152 ENSG00000174145 NWD2
## 153 ENSG00000053770 AP5M1
## 154 ENSG00000146007 ZMAT2
## 155 ENSG00000146205 ANO7
## 156 ENSG00000155158 TTC39B
## 157 ENSG00000162972 MAIP1
## 158 ENSG00000165688 PMPCA
## 159 ENSG00000179344 HLA-DQB1
## 160 ENSG00000241149 ZNF722
## 161 ENSG00000178150 ZNF114
## 162 ENSG00000096006 CRISP3
## 163 ENSG00000111450 STX2
## 164 ENSG00000183793 NPIPA5
## 165 ENSG00000166971 AKTIP
## 166 ENSG00000170807 LMOD2
## 167 ENSG00000161594 KLHL10
## 168 ENSG00000232268 OR52I1
## 169 ENSG00000241553 ARPC4
## 170 ENSG00000133169 BEX1
## 171 ENSG00000120217 CD274
## 172 ENSG00000260097 SPDYE6
## 173 ENSG00000055332 EIF2AK2
## 174 ENSG00000158865 SLC5A11
## 175 ENSG00000162714 ZNF496
## 176 ENSG00000166133 RPUSD2
## 177 ENSG00000143156 NME7
## 178 ENSG00000137204 SLC22A7
## 179 ENSG00000104267 CA2
## 180 ENSG00000105662 CRTC1
## 181 ENSG00000174243 DDX23
## 182 ENSG00000070761 CFAP20
## 183 ENSG00000085871 MGST2
## 184 ENSG00000162775 RBM15
## 185 ENSG00000137802 MAPKBP1
## 186 ENSG00000152767 FARP1
## 187 ENSG00000158806 NPM2
## 188 ENSG00000204001 LCN8
## 189 ENSG00000125900 SIRPD
## 190 ENSG00000010818 HIVEP2
## 191 ENSG00000125355 TMEM255A
## 192 ENSG00000149124 GLYAT
## 193 ENSG00000182348 ZNF804B
## 194 ENSG00000115419 GLS
## 195 ENSG00000102271 KLHL4
## 196 ENSG00000241399 CD302
## 197 ENSG00000112419 PHACTR2
## 198 ENSG00000152779 SLC16A12
## 199 ENSG00000169994 MYO7B
## 200 ENSG00000004948 CALCR
## 201 ENSG00000145920 CPLX2importance_gp <- simple_gprofiler(names(head(test_imp, n = 60)))## No results to show
## Please make sure that the organism is correct or set significant = FALSE## No results to show
## Please make sure that the organism is correct or set significant = FALSE
## No results to show
## Please make sure that the organism is correct or set significant = FALSE
## No results to show
## Please make sure that the organism is correct or set significant = FALSE
## No results to show
## Please make sure that the organism is correct or set significant = FALSE
## No results to show
## Please make sure that the organism is correct or set significant = FALSE
## No results to show
## Please make sure that the organism is correct or set significant = FALSE
## No results to show
## Please make sure that the organism is correct or set significant = FALSEimportance_gp$pvalue_plots$BP## NULL6.3.1 Now the random forest testers!
rf_predict_test <- predict(rf_train_fit, test_df)
names(rf_predict_test) <- rownames(test_df)
rf_agreement <- rf_predict_test == pData(tc_norm)[not_train_idx, "finaloutcome"]
names(rf_agreement) <- rownames(test_df)
summary(rf_agreement)## Mode FALSE TRUE
## logical 17 55names(rf_agreement)[rf_agreement == FALSE]## [1] "TMRC30018" "TMRC30056" "TMRC30080" "TMRC30119" "TMRC30103" "TMRC30122" "TMRC30026" "TMRC30166" "TMRC30048" "TMRC30200" "TMRC30177"
## [12] "TMRC30238" "TMRC30208" "TMRC30218" "TMRC30220" "TMRC30264" "TMRC30265"6.4 GLM, or Logistic regression and regularization
Logistic regression is a statistical method for binary responses. However, it is able to work with multiple classes as well. The general idea of this method is to find parameters which increase the likelihood that the observed data is sampled from a statistical distribution of interest. The transformations and linear regression-esque tasks performed are confusing, but once those are performed, the task becomes setting the model’s (fitting) parameters to values which increase the probability that the statistical model looks like the actual dataset given the training data, and that when samples, will return values which are similar. The most likely statistical distributions one will want to fit are the Gaussian, in which case we want to transform/normalize the mean/variance of our variables so they look whatever normal distribution we are using. Conversely, logistic regression uses a binnomial distribution (like our raw sequencing data!) but which is from 0-1.
6.4.1 Using a single gene
Let us take the most important gene observed in one of our previous training sets: ENSG00000248405 PRR5-ARHGAP8
gene_id <- "ENSG00000248405"
single_fit <- train(
outcome_factor ~ ENSG00000248405, data = train_df,
method = "glm", family = "binomial", trControl = trainControl("none"))
tt <- data.frame("ENSG00000248405" = seq(min(train_df[[gene_id]]),
max(train_df[[gene_id]]),len=100))
## predict probabilities for the simulated data
tt$subtype = predict(single_fit, newdata=tt, type="prob")[, 1]
## plot the sigmoid curve and the training data
plot(ifelse(outcome_factor == "cure", 1, 0) ~ ENSG00000248405,
data=train_df, col="red4",
ylab="CF as 0 or 1", xlab="favorite gene expression")
lines(subtype ~ ENSG00000248405, tt, col="green4", lwd=2)
Having tried with 1 gene, let us extend this to all genes. In my first try of this, it took a long time.
glm_fit <- train(outcome_factor ~ ., data = train_df,
trControl = boot_control,
method = "glm", family = "binomial")## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not converge## Warning in predict.lm(object, newdata, se.fit, scale = 1, type = if (type == : prediction from a rank-deficient fit may be misleading## Warning: glm.fit: algorithm did not convergelibrary(glmnet)## Loading required package: Matrix##
## Attaching package: 'Matrix'## The following object is masked from 'package:S4Vectors':
##
## expand## Loaded glmnet 4.1-7##rf_method <- trainControl(method = "ranger", number = 10, verbose = TRUE)
## train_method <- trainControl(method = "cv", number = 10)
glm_fit <- train(outcome_factor ~ ., data = train_df, method = "glmnet",
trControl = boot_control, importance = "permutation",
tuneGrid = data.frame(
alpha = 0.5,
lambda = seq(0.1, 0.7, 0.05)),
verbose = TRUE)
glm_predict_trained <- predict(glm_fit, train_df)
names(glm_predict_trained) <- rownames(train_df)
summary(glm_predict_trained == pData(tc_norm)[train_idx, "finaloutcome"])## Mode TRUE
## logical 112confusionMatrix(data = train_df[, 1], reference = glm_predict_trained)## Confusion Matrix and Statistics
##
## Reference
## Prediction cure failure
## cure 74 0
## failure 0 38
##
## Accuracy : 1
## 95% CI : (0.968, 1)
## No Information Rate : 0.661
## P-Value [Acc > NIR] : <2e-16
##
## Kappa : 1
##
## Mcnemar's Test P-Value : NA
##
## Sensitivity : 1.000
## Specificity : 1.000
## Pos Pred Value : 1.000
## Neg Pred Value : 1.000
## Prevalence : 0.661
## Detection Rate : 0.661
## Detection Prevalence : 0.661
## Balanced Accuracy : 1.000
##
## 'Positive' Class : cure
## 6.4.2 Now the GLM testers!
glm_predict_test <- predict(glm_fit, test_df)
names(glm_predict_test) <- rownames(test_df)
glm_predict_test## TMRC30186 TMRC30253 TMRC30140 TMRC30176 TMRC30226 TMRC30227 TMRC30016 TMRC30231 TMRC30233 TMRC30018 TMRC30210 TMRC30213 TMRC30216 TMRC30214
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30274 TMRC30254 TMRC30277 TMRC30240 TMRC30280 TMRC30281 TMRC30284 TMRC30282 TMRC30285 TMRC30056 TMRC30105 TMRC30164 TMRC30080 TMRC30094
## cure cure cure cure cure cure cure cure cure failure failure cure failure failure
## TMRC30119 TMRC30082 TMRC30103 TMRC30122 TMRC30022 TMRC30029 TMRC30083 TMRC30028 TMRC30026 TMRC30165 TMRC30044 TMRC30166 TMRC30195 TMRC30048
## failure cure cure failure cure cure failure cure failure cure cure cure cure failure
## TMRC30045 TMRC30055 TMRC30041 TMRC30139 TMRC30160 TMRC30183 TMRC30167 TMRC30078 TMRC30076 TMRC30159 TMRC30129 TMRC30088 TMRC30142 TMRC30168
## cure failure cure cure cure cure cure failure failure cure cure failure cure cure
## TMRC30146 TMRC30199 TMRC30200 TMRC30204 TMRC30177 TMRC30155 TMRC30206 TMRC30136 TMRC30238 TMRC30217 TMRC30208 TMRC30218 TMRC30220 TMRC30264
## cure failure cure failure failure cure failure cure cure failure failure failure cure cure
## TMRC30144 TMRC30265
## cure cure
## Levels: cure failureglm_agreement <- glm_predict_test == pData(tc_norm)[not_train_idx, "finaloutcome"]
names(glm_agreement) <- rownames(test_df)
summary(glm_agreement)## Mode FALSE TRUE
## logical 6 66names(glm_agreement)[glm_agreement == FALSE]## [1] "TMRC30080" "TMRC30200" "TMRC30238" "TMRC30220" "TMRC30264" "TMRC30265"6.5 Gradient Booster
library(xgboost)##
## Attaching package: 'xgboost'## The following object is masked from 'package:IRanges':
##
## slice##rf_method <- trainControl(method = "ranger", number = 10, verbose = TRUE)
train_method <- trainControl(method = "cv", number = 10)
gb_fit <- train(outcome_factor ~ ., data = train_df,
method = "xgbTree", trControl = train_method,
tuneGrid = data.frame(
nrounds = 200,
eta = c(0.05, 0.1, 0.3),
max_depth = 4,
gamma = 0,
colsample_bytree = 1,
subsample = 0.5,
min_child_weight = 1),
verbose = TRUE)
gb_predict_trained <- predict(gb_fit, train_df)
names(gb_predict_trained) <- rownames(train_df)
summary(gb_predict_trained == pData(tc_norm)[train_idx, "finaloutcome"])## Mode TRUE
## logical 1126.5.1 Now the GB testers!
gb_predict_test <- predict(gb_fit, test_df)
names(gb_predict_test) <- rownames(test_df)
gb_predict_test## TMRC30186 TMRC30253 TMRC30140 TMRC30176 TMRC30226 TMRC30227 TMRC30016 TMRC30231 TMRC30233 TMRC30018 TMRC30210 TMRC30213 TMRC30216 TMRC30214
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30274 TMRC30254 TMRC30277 TMRC30240 TMRC30280 TMRC30281 TMRC30284 TMRC30282 TMRC30285 TMRC30056 TMRC30105 TMRC30164 TMRC30080 TMRC30094
## cure cure cure cure cure cure cure cure cure failure failure cure failure failure
## TMRC30119 TMRC30082 TMRC30103 TMRC30122 TMRC30022 TMRC30029 TMRC30083 TMRC30028 TMRC30026 TMRC30165 TMRC30044 TMRC30166 TMRC30195 TMRC30048
## failure cure cure failure cure cure failure cure failure cure cure failure cure failure
## TMRC30045 TMRC30055 TMRC30041 TMRC30139 TMRC30160 TMRC30183 TMRC30167 TMRC30078 TMRC30076 TMRC30159 TMRC30129 TMRC30088 TMRC30142 TMRC30168
## cure failure cure cure cure cure cure failure failure cure cure failure cure cure
## TMRC30146 TMRC30199 TMRC30200 TMRC30204 TMRC30177 TMRC30155 TMRC30206 TMRC30136 TMRC30238 TMRC30217 TMRC30208 TMRC30218 TMRC30220 TMRC30264
## cure failure failure failure failure cure failure cure cure failure failure failure cure cure
## TMRC30144 TMRC30265
## cure cure
## Levels: cure failuregb_agreement <- gb_predict_test == pData(tc_norm)[not_train_idx, "finaloutcome"]
names(gb_agreement) <- rownames(test_df)
summary(gb_agreement)## Mode FALSE TRUE
## logical 6 66names(gb_agreement)[gb_agreement == FALSE]## [1] "TMRC30080" "TMRC30166" "TMRC30238" "TMRC30220" "TMRC30264" "TMRC30265"6.6 SVM
library(kernlab)##
## Attaching package: 'kernlab'## The following object is masked from 'package:ggplot2':
##
## alpha## The following object is masked from 'package:BiocGenerics':
##
## type##rf_method <- trainControl(method = "ranger", number = 10, verbose = TRUE)
train_method <- trainControl(method = "cv", number = 10)
svm_fit <- train(outcome_factor ~ ., data = train_df,
method = "svmRadial", trControl = train_method,
tuneGrid = data.frame(
C = c(0.25, 0.5, 1),
sigma = 1),
verbose = TRUE)## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.
## Warning in .local(x, ...): Variable(s) `' constant. Cannot scale data.svm_predict_trained <- predict(svm_fit, train_df)
names(svm_predict_trained) <- rownames(train_df)
svm_predict_trained## TMRC30156 TMRC30185 TMRC30178 TMRC30179 TMRC30221 TMRC30222 TMRC30223 TMRC30224 TMRC30269 TMRC30148 TMRC30149 TMRC30150 TMRC30138 TMRC30153
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30151 TMRC30234 TMRC30235 TMRC30270 TMRC30225 TMRC30228 TMRC30229 TMRC30230 TMRC30017 TMRC30232 TMRC30209 TMRC30211 TMRC30212 TMRC30215
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30271 TMRC30273 TMRC30275 TMRC30272 TMRC30276 TMRC30255 TMRC30256 TMRC30239 TMRC30278 TMRC30279 TMRC30257 TMRC30019 TMRC30258 TMRC30283
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30071 TMRC30020 TMRC30113 TMRC30058 TMRC30169 TMRC30093 TMRC30107 TMRC30170 TMRC30032 TMRC30096 TMRC30115 TMRC30118 TMRC30180 TMRC30014
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30121 TMRC30196 TMRC30030 TMRC30021 TMRC30037 TMRC30031 TMRC30027 TMRC30194 TMRC30054 TMRC30046 TMRC30070 TMRC30049 TMRC30047 TMRC30191
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30053 TMRC30068 TMRC30171 TMRC30192 TMRC30042 TMRC30158 TMRC30132 TMRC30157 TMRC30123 TMRC30181 TMRC30072 TMRC30133 TMRC30043 TMRC30116
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30184 TMRC30172 TMRC30134 TMRC30174 TMRC30137 TMRC30161 TMRC30175 TMRC30145 TMRC30143 TMRC30197 TMRC30182 TMRC30198 TMRC30201 TMRC30203
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30202 TMRC30205 TMRC30152 TMRC30154 TMRC30241 TMRC30237 TMRC30207 TMRC30074 TMRC30077 TMRC30219 TMRC30079 TMRC30135 TMRC30173 TMRC30147
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## Levels: cure failuresummary(svm_predict_trained == pData(tc_norm)[train_idx, "finaloutcome"])## Mode FALSE TRUE
## logical 38 746.6.1 Now the SVM testers!
svm_predict_test <- predict(svm_fit, test_df)
names(svm_predict_test) <- rownames(test_df)
svm_predict_test## TMRC30186 TMRC30253 TMRC30140 TMRC30176 TMRC30226 TMRC30227 TMRC30016 TMRC30231 TMRC30233 TMRC30018 TMRC30210 TMRC30213 TMRC30216 TMRC30214
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30274 TMRC30254 TMRC30277 TMRC30240 TMRC30280 TMRC30281 TMRC30284 TMRC30282 TMRC30285 TMRC30056 TMRC30105 TMRC30164 TMRC30080 TMRC30094
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30119 TMRC30082 TMRC30103 TMRC30122 TMRC30022 TMRC30029 TMRC30083 TMRC30028 TMRC30026 TMRC30165 TMRC30044 TMRC30166 TMRC30195 TMRC30048
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30045 TMRC30055 TMRC30041 TMRC30139 TMRC30160 TMRC30183 TMRC30167 TMRC30078 TMRC30076 TMRC30159 TMRC30129 TMRC30088 TMRC30142 TMRC30168
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30146 TMRC30199 TMRC30200 TMRC30204 TMRC30177 TMRC30155 TMRC30206 TMRC30136 TMRC30238 TMRC30217 TMRC30208 TMRC30218 TMRC30220 TMRC30264
## cure cure cure cure cure cure cure cure cure cure cure cure cure cure
## TMRC30144 TMRC30265
## cure cure
## Levels: cure failuresvm_agreement <- svm_predict_test == pData(tc_norm)[not_train_idx, "finaloutcome"]
names(svm_agreement) <- rownames(test_df)
summary(svm_agreement)## Mode FALSE TRUE
## logical 24 48names(svm_agreement)[svm_agreement == FALSE]## [1] "TMRC30056" "TMRC30105" "TMRC30094" "TMRC30119" "TMRC30122" "TMRC30083" "TMRC30026" "TMRC30048" "TMRC30055" "TMRC30078" "TMRC30076"
## [12] "TMRC30088" "TMRC30199" "TMRC30200" "TMRC30204" "TMRC30177" "TMRC30206" "TMRC30238" "TMRC30217" "TMRC30208" "TMRC30218" "TMRC30220"
## [23] "TMRC30264" "TMRC30265"7 Reapply to persistence
ref_col <- "persistence"
outcome_factor <- as.factor(as.character(pData(tc_clinical)[[ref_col]]))
ml_df <- as.data.frame(cbind(outcome_factor, as.data.frame(nzv_uncorr)))
ml_df[["outcome_factor"]] <- as.factor(ml_df[["outcome_factor"]])
only_yn <- ml_df[["outcome_factor"]] == "Y" | ml_df[["outcome_factor"]] == "N"
outcome_factor <- as.factor(as.character(outcome_factor[only_yn]))
ml_df <- ml_df[only_yn, ]
ml_df[["outcome_factor"]] <- as.factor(as.character(ml_df[["outcome_factor"]]))
summary(ml_df[["outcome_factor"]])## N Y
## 55 101train_idx <- createDataPartition(outcome_factor, p = 0.5,
list = FALSE,
times = 1)
summary(ml_df[train_idx, "outcome_factor"])## N Y
## 28 51## Training set has 112 samples.
train_rownames <- rownames(ml_df)[train_idx]
not_train_idx <- ! rownames(ml_df) %in% train_rownames
summary(ml_df[not_train_idx, "outcome_factor"])## N Y
## 27 50not_train_rownames <- rownames(ml_df)[not_train_idx]
train_df <- as.data.frame(ml_df[train_rownames, ])
dim(train_df)## [1] 79 12081test_df <- as.data.frame(ml_df[not_train_rownames, ])
dim(test_df)## [1] 77 12081## Remove the outcome factor from the test data, just in case.
test_outcome <- test_df[["outcome_factor"]]
test_df[["outcome_factor"]] <- NULL
train_fit <- train(outcome_factor ~ ., data = train_df,
method = "ranger", trControl = boot_control,
importance = "permutation",
tuneGrid = data.frame(
mtry = 200,
min.node.size = 1,
splitrule = "gini"),
verbose = TRUE)
train_fit[["finalModel"]][["prediction.error"]]## [1] 0.2532variable_importance <- varImp(train_fit)
plot(variable_importance, top = 15)
rf_predict_trained <- predict(rf_train_fit, train_df)
confusionMatrix(data = train_df[[1]], reference = rf_predict_trained)## Error in confusionMatrix.default(data = train_df[[1]], reference = rf_predict_trained): The data must contain some levels that overlap the reference.rf_predict_test <- predict(rf_train_fit, test_df)
names(rf_predict_test) <- rownames(test_df)
used_names <- names(rf_predict_test)
used_pdata <- pData(tc_norm)[used_names, ]
rf_agreement <- as.character(rf_predict_test) == used_pdata[[ref_col]]
names(rf_agreement) <- rownames(test_df)
summary(rf_agreement)## Mode FALSE
## logical 77names(rf_agreement)[rf_agreement == FALSE]## [1] "TMRC30156" "TMRC30178" "TMRC30223" "TMRC30138" "TMRC30153" "TMRC30151" "TMRC30234" "TMRC30235" "TMRC30227" "TMRC30231" "TMRC30232"
## [12] "TMRC30233" "TMRC30209" "TMRC30216" "TMRC30214" "TMRC30273" "TMRC30275" "TMRC30276" "TMRC30255" "TMRC30256" "TMRC30240" "TMRC30279"
## [23] "TMRC30280" "TMRC30283" "TMRC30285" "TMRC30071" "TMRC30056" "TMRC30105" "TMRC30058" "TMRC30103" "TMRC30122" "TMRC30107" "TMRC30096"
## [34] "TMRC30115" "TMRC30121" "TMRC30166" "TMRC30048" "TMRC30054" "TMRC30070" "TMRC30049" "TMRC30055" "TMRC30053" "TMRC30068" "TMRC30158"
## [45] "TMRC30132" "TMRC30157" "TMRC30123" "TMRC30181" "TMRC30133" "TMRC30184" "TMRC30159" "TMRC30129" "TMRC30172" "TMRC30137" "TMRC30142"
## [56] "TMRC30143" "TMRC30197" "TMRC30182" "TMRC30203" "TMRC30202" "TMRC30205" "TMRC30177" "TMRC30154" "TMRC30241" "TMRC30237" "TMRC30206"
## [67] "TMRC30238" "TMRC30074" "TMRC30217" "TMRC30208" "TMRC30219" "TMRC30218" "TMRC30220" "TMRC30173" "TMRC30264" "TMRC30147" "TMRC30265"8 Try something a little different
I redid the mappings with a combined host/parasite transcriptome in the hopes that it might provide some interesting variance for these classifiers. Lets us poke at it and see if anything is relevant or if it was a bad idea.
wanted_idx <- grepl(x = rownames(exprs(hslp_gene_expt)), pattern = "^LP")## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'grepl': error in evaluating the argument 'x' in selecting a method for function 'rownames': error in evaluating the argument 'object' in selecting a method for function 'exprs': object 'hslp_gene_expt' not foundwanted_ids <- rownames(exprs(hslp_gene_expt))[wanted_idx]## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'rownames': error in evaluating the argument 'object' in selecting a method for function 'exprs': object 'hslp_gene_expt' not foundtest_lp <- exclude_genes_expt(hslp_gene_expt, ids = wanted_ids, method = "keep")## Note, I renamed this to subset_genes().## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'fData': object 'hslp_gene_expt' not foundtt <- plot_libsize(test_lp)## Error in h(simpleError(msg, call)): error in evaluating the argument 'data' in selecting a method for function 'plot_libsize': object 'test_lp' not foundtt$plot## NULLI am going to mostly copy paste the code from up above here but change the inputs to fit my new combined data structure of human and parasite data.
ref_col <- "finaloutcome"
outcome_factor <- as.factor(as.character(pData(hslp_gene_expt)[[ref_col]]))## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'as.factor': error in evaluating the argument 'object' in selecting a method for function 'pData': object 'hslp_gene_expt' not foundhslp_norm <- normalize_expt(hslp_gene_expt, transform = "log2", convert = "cpm")## Error in h(simpleError(msg, call)): error in evaluating the argument 'expt' in selecting a method for function 'state': object 'hslp_gene_expt' not foundhslp_norm <- normalize_expt(hslp_norm, filter = "cv", cv_min = 0.1)## Error in h(simpleError(msg, call)): error in evaluating the argument 'expt' in selecting a method for function 'state': object 'hslp_norm' not foundretained_lp <- grepl(x = rownames(exprs(hslp_norm)), pattern = "^LP")## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'grepl': error in evaluating the argument 'x' in selecting a method for function 'rownames': error in evaluating the argument 'object' in selecting a method for function 'exprs': object 'hslp_norm' not foundsum(retained_lp)## Error in eval(expr, envir, enclos): object 'retained_lp' not foundnzv_thslp <- t(exprs(hslp_norm))## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 't': error in evaluating the argument 'object' in selecting a method for function 'exprs': object 'hslp_norm' not foundnzv_hslp_center <- preProcess(nzv_thslp, method = "center")## Error in preProcess(nzv_thslp, method = "center"): object 'nzv_thslp' not foundnzv_thslp <- predict(nzv_hslp_center, nzv_thslp)## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'predict': object 'nzv_hslp_center' not foundnzv_thslp_correlated <- preProcess(nzv_thslp, method = "corr", cutoff = 0.95)## Error in preProcess(nzv_thslp, method = "corr", cutoff = 0.95): object 'nzv_thslp' not foundnzv_thslp_uncorr <- predict(nzv_thslp_correlated, nzv_thslp)## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'predict': object 'nzv_thslp_correlated' not foundinteresting_meta <- pData(hslp_norm)[, c("finaloutcome", "donor",
"visitnumber", "selectionmethod",
"typeofcells", "time", "clinic")]## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'pData': object 'hslp_norm' not foundhslp_ml_df <- as.data.frame(cbind(outcome_factor,
as.data.frame(nzv_thslp_uncorr)))## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'as.data.frame': error in evaluating the argument 'x' in selecting a method for function 'as.data.frame': object 'nzv_thslp_uncorr' not foundhslp_ml_df[["outcome_factor"]] <- as.factor(hslp_ml_df[["outcome_factor"]])## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'as.factor': object 'hslp_ml_df' not founddim(hslp_ml_df)## Error in eval(expr, envir, enclos): object 'hslp_ml_df' not found9 Split the data for training/testing
## The variable outcome_factor was created at the top of this document,
## which is a little awkward here.
train_idx <- createDataPartition(outcome_factor, p = 0.6,
list = FALSE,
times = 1)
summary(hslp_ml_df[train_idx, "outcome_factor"])## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'summary': object 'hslp_ml_df' not found## Training set has 112 samples.
train_rownames <- rownames(hslp_ml_df)[train_idx]## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'rownames': object 'hslp_ml_df' not foundnot_train_idx <- ! rownames(hslp_ml_df) %in% train_rownames## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function '%in%': error in evaluating the argument 'x' in selecting a method for function 'rownames': object 'hslp_ml_df' not foundsummary(hslp_ml_df[not_train_idx, "outcome_factor"])## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'summary': object 'hslp_ml_df' not foundnot_train_rownames <- rownames(hslp_ml_df)[not_train_idx]## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'rownames': object 'hslp_ml_df' not foundtrain_df <- as.data.frame(hslp_ml_df[train_rownames, ])## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'as.data.frame': object 'hslp_ml_df' not founddim(train_df)## [1] 79 12081test_df <- as.data.frame(hslp_ml_df[not_train_rownames, ])## Error in h(simpleError(msg, call)): error in evaluating the argument 'x' in selecting a method for function 'as.data.frame': object 'hslp_ml_df' not founddim(test_df)## [1] 77 12080## Remove the outcome factor from the test data, just in case.
test_outcome <- test_df[["outcome_factor"]]
test_df[["outcome_factor"]] <- NULLboot_control <- trainControl(method = "boot", number = 20,
returnResamp = "all")
knn_fit <- knn3(x = train_df[, -1],
y = train_df[["outcome_factor"]],
k = 3)
knn_predict_trained <- predict(knn_fit, train_df[, -1], type = "class")
confusionMatrix(data = train_df[[1]], reference = knn_predict_trained)## Confusion Matrix and Statistics
##
## Reference
## Prediction N Y
## N 24 4
## Y 4 47
##
## Accuracy : 0.899
## 95% CI : (0.81, 0.955)
## No Information Rate : 0.646
## P-Value [Acc > NIR] : 2.59e-07
##
## Kappa : 0.779
##
## Mcnemar's Test P-Value : 1
##
## Sensitivity : 0.857
## Specificity : 0.922
## Pos Pred Value : 0.857
## Neg Pred Value : 0.922
## Prevalence : 0.354
## Detection Rate : 0.304
## Detection Prevalence : 0.354
## Balanced Accuracy : 0.889
##
## 'Positive' Class : N
## names(knn_predict_trained) <- rownames(train_df)
summary(knn_predict_trained == pData(hslp_gene_expt)[train_idx, "finaloutcome"])## Error in h(simpleError(msg, call)): error in evaluating the argument 'object' in selecting a method for function 'summary': error in evaluating the argument 'object' in selecting a method for function 'pData': object 'hslp_gene_expt' not foundknn_predict_test <- predict(knn_fit, test_df, type = "class")
## and the values, which will be used for our ROC curve.
knn_predict_test_values <- predict(knn_fit, test_df)
names(knn_predict_test) <- rownames(test_df)
knn_agreement <- knn_predict_test == test_outcome
names(knn_agreement) <- rownames(test_df)## Error in names(knn_agreement) <- rownames(test_df): 'names' attribute [77] must be the same length as the vector [0]summary(knn_agreement)## Mode
## logicalnames(knn_agreement)[knn_agreement == FALSE]## NULLcv_control <- trainControl(method = "cv", number = 10)
knn_train_fit <- train(outcome_factor ~ ., data = train_df,
method = "knn",
trControl = cv_control,
tuneGrid = data.frame(k = 1:10))
knn_train_fit[["bestTune"]]## k
## 3 3plot(x = 1:10, 1 - knn_train_fit$results[, 2], pch = 19,
ylab = "prediction error", xlab = "k")
lines(loess.smooth(x = 1:10, 1 - knn_train_fit$results[, 2],degree = 2),
col = "#CC0000")
LS0tCnRpdGxlOiAiVE1SQzMgTUwgQ2xhc3NpZmljYXRpb24gb2Ygb3V0Y29tZTogMjAyMzAyIgphdXRob3I6ICJhdGIgYWJlbGV3QGdtYWlsLmNvbSIKZGF0ZTogImByIFN5cy5EYXRlKClgIgpvdXRwdXQ6CiBodG1sX2RvY3VtZW50OgogIGNvZGVfZG93bmxvYWQ6IHRydWUKICBjb2RlX2ZvbGRpbmc6IHNob3cKICBmaWdfY2FwdGlvbjogdHJ1ZQogIGZpZ19oZWlnaHQ6IDcKICBmaWdfd2lkdGg6IDcKICBoaWdobGlnaHQ6IHRhbmdvCiAga2VlcF9tZDogZmFsc2UKICBtb2RlOiBzZWxmY29udGFpbmVkCiAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlCiAgc2VsZl9jb250YWluZWQ6IHRydWUKICB0aGVtZTogcmVhZGFibGUKICB0b2M6IHRydWUKICB0b2NfZmxvYXQ6CiAgIGNvbGxhcHNlZDogZmFsc2UKICAgc21vb3RoX3Njcm9sbDogZmFsc2UKLS0tCgo8c3R5bGU+CiAgYm9keSAubWFpbi1jb250YWluZXIgewogICAgbWF4LXdpZHRoOiAxNjAwcHg7CiAgfQo8L3N0eWxlPgoKYGBge3Igb3B0aW9ucywgaW5jbHVkZSA9IEZBTFNFfQpsaWJyYXJ5KGhwZ2x0b29scykKdHQgPC0gZGV2dG9vbHM6OmxvYWRfYWxsKCJ+L2hwZ2x0b29scyIpCmtuaXRyOjpvcHRzX2tuaXQkc2V0KHByb2dyZXNzID0gVFJVRSwKICAgICAgICAgICAgICAgICAgICAgdmVyYm9zZSA9IFRSVUUsCiAgICAgICAgICAgICAgICAgICAgIHdpZHRoID0gOTAsCiAgICAgICAgICAgICAgICAgICAgIGVjaG8gPSBUUlVFKQprbml0cjo6b3B0c19jaHVuayRzZXQoZXJyb3IgPSBUUlVFLAogICAgICAgICAgICAgICAgICAgICAgZmlnLndpZHRoID0gOCwKICAgICAgICAgICAgICAgICAgICAgIGZpZy5oZWlnaHQgPSA4LAogICAgICAgICAgICAgICAgICAgICAgZHBpID0gOTYpCm9sZF9vcHRpb25zIDwtIG9wdGlvbnMoZGlnaXRzID0gNCwKICAgICAgICAgICAgICAgICAgICAgICBzdHJpbmdzQXNGYWN0b3JzID0gRkFMU0UsCiAgICAgICAgICAgICAgICAgICAgICAga25pdHIuZHVwbGljYXRlLmxhYmVsID0gImFsbG93IikKZ2dwbG90Mjo6dGhlbWVfc2V0KGdncGxvdDI6OnRoZW1lX2J3KGJhc2Vfc2l6ZSA9IDEyKSkKdmVyIDwtICIyMDIzMDEiCnJ1bmRhdGUgPC0gZm9ybWF0KFN5cy5EYXRlKCksIGZvcm1hdCA9ICIlWSVtJWQiKQoKIyMgdG1wIDwtIHRyeShzbShsb2FkbWUoZmlsZW5hbWUgPSBnc3ViKHBhdHRlcm4gPSAiXFwuUm1kIiwgcmVwbGFjZSA9ICJcXC5yZGFcXC54eiIsIHggPSBwcmV2aW91c19maWxlKSkpKQpybWRfZmlsZSA8LSBnbHVlOjpnbHVlKCJ0bXJjM19jbGFzc2lmaWVyX3t2ZXJ9LlJtZCIpCnNhdmVmaWxlIDwtIGdzdWIocGF0dGVybiA9ICJcXC5SbWQiLCByZXBsYWNlID0gIlxcLnJkYVxcLnh6IiwgeCA9IHJtZF9maWxlKQpsb2FkZWQgPC0gbG9hZChmaWxlID0gZ2x1ZTo6Z2x1ZSgicmRhL3RtcmMzX2RhdGFfc3RydWN0dXJlcy12e3Zlcn0ucmRhIikpCmxpYnJhcnkoY2FyZXQpCmxpYnJhcnkocFJPQykKbGlicmFyeShEQUxFWCkKYGBgCgojIEludHJvZHVjdGlvbgoKSSBoYWQgc29tZSBzdWNjZXNzIGluIGNsYXNzaWZ5aW5nIHRoZSBUTVJDMiBzYW1wbGVzIGJ5IHN0cmFpbiB2aWEgTUwKYW5kIHdhbnQgdG8gdHJ5IHNvbWV0aGluZyBtb3JlIGRpZmZpY3VsdC4gIFRodXMgSSB3aWxsIHVzZSB0aGUKbm9ybWFsaXplZCBnZW5lIGV4cHJlc3Npb24gZGF0YSBhbmQgdHJ5IGNsYXNzaWZ5aW5nIGl0IGJ5IGN1cmUvZmFpbC4KCmBgYHtyIGNsYXNzaWZ5X2NmfQpyZWZfY29sIDwtICJmaW5hbG91dGNvbWUiCm91dGNvbWVfZmFjdG9yIDwtIGFzLmZhY3Rvcihhcy5jaGFyYWN0ZXIocERhdGEodGNfY2xpbmljYWwpW1tyZWZfY29sXV0pKQpgYGAKCiMgU3RhcnRlciBkYXRhCgpJbiB0aGUgc3RyYWluIGNsYXNzaWZpZXIgSSB1c2VkIG5vcm1hbGl6ZWQgdmFyaWFudHMuICBJIGFtIHRoaW5raW5nIHRvCnVzZSBub3JtYWxpemVkIGV4cHJlc3Npb24gaGVyZSBhbmQgdGhlcmVmb3JlIGV4cGxpY2l0bHkgbGltaXQgbXlzZWxmCnRvIH4gMjBrIHZhcmlhYmxlcyAoc2lnbmlmaWNhbnRseSBkb3duIGZyb20gdGhlIDEuNk0pLgoKSW4gYWRkaXRpb24sIGNhcmV0IGV4cGVjdHMgdGhlIGRhdGEgYXMgKHJvd3MgPT0gc2FtcGxlcykgYW5kCihjb2x1bW5zID09IHZhcmlhYmxlcykgd2hlcmUgZWFjaCBlbGVtZW50IGlzIG9uZSBvYnNlcnZhdGlvbi4KVGh1cyB3ZSB3aWxsIG5lZWQgdG8gdHJhbnNwb3NlIHRoZSBleHByZXNzaW9uIG1hdHJpeC4KCmBgYHtyIHN0YXJ0ZXJ9CnRjX25vcm0gPC0gbm9ybWFsaXplX2V4cHQodGNfY2xpbmljYWwsIHRyYW5zZm9ybSA9ICJsb2cyIiwgY29udmVydCA9ICJjcG0iKQp0ZXhwcnMgPC0gdChleHBycyh0Y19ub3JtKSkKYGBgCgojIEZpbHRlcmluZwoKVGhlIE1MIHRleHQgSSBhbSByZWFkaW5nIHByb3ZpZGUgc29tZSBuZWF0IGV4YW1wbGVzIGZvciBob3cgb25lIG1pZ2h0CmZpbHRlciB0aGUgZGF0YSB0byBtYWtlIGl0IG1vcmUgc3VpdGFibGUgZm9yIG1vZGVsIGNyZWF0aW9uLgoKIyMgTmVhciB6ZXJvIHZhcmlhbmNlLCBvciBnZW5lZmlsdGVyJ3MgY3YKClRoZSBmaXJzdCBmaWx0ZXIgSSB3YXMgaW50cm9kdWNlZCB0byBpcyBxdWl0ZSBmYW1pbGlhciBmcm9tIG91cgpzZXF1ZW5jaW5nIGRhdGEsIHRoZSByZW1vdmFsIG9mIGZlYXR1cmVzIHdpdGggbmVhci16ZXJvLXZhcmlhbmNlLgpJbmRlZWQsIEkgYW0gcHJldHR5IGNlcnRhaW4gdGhhdCBub3JtYWxpemVfZXhwdCgpIGNvdWxkIGRvIHRoaXMKZXF1aXZhbGVudGx5IGFuZCBzaWduaWZpY2FudGx5IGZhc3RlciB0aGFuIGNhcmV0OjpwcmVQcm9jZXNzKCkuCgpgYGB7ciBuenZ9CnN5c3RlbS50aW1lKHsKICBlcXVpdmFsZW50IDwtIG5vcm1hbGl6ZV9leHB0KHRjX25vcm0sIGZpbHRlciA9ICJjdiIsIGN2X21pbiA9IDAuMSkKfSkKZGltKGV4cHJzKGVxdWl2YWxlbnQpKQoKIyMgR2l2ZW4gdGhpcyBsYXJnZSBhbW91bnQgb2YgZGF0YSwgdGhpcyBzdGVwIGlzIHNsb3csIHRha2luZyA+IDEwIG1pbnV0ZXMuCiMjIFllYWggc2VyaW91c2x5LCB0aGUgZm9sbG93aW5nIHRocmVlIGxpbmVzIGdldCAxNiw3MjMgZ2VuZXMgaW4gMTAgbWludXRlcyB3aGlsZQojIyB0aGUgbm9ybWFsaXplX2V4cHQoKSBjYWxsIGFib3ZlIGdldHMgMTYsNzQ5IGdlbmVzIGluIDIuNCBzZWNvbmRzLgojc3lzdGVtLnRpbWUoewojICBuenYgPC0gcHJlUHJvY2Vzcyh0ZXhwcnMsIG1ldGhvZD0ibnp2IiwgdW5pcXVlQ3V0PTE1KQojICBuenZfdGV4cHJzIDwtIHByZWRpY3Qobnp2LCB0ZXhwcnMpCiMgIGRpbShuenZfdGV4cHJzKQojfQpuenZfdGV4cHJzIDwtIHQoZXhwcnMoZXF1aXZhbGVudCkpCmBgYAoKIyMgRmlsdGVyaW5nIHRvIHRoZSBoaWdoZXN0IHN0YW5kYXJkIGRldmlhdGlvbiB2YXJpYWJsZXMKCkkgdGhpbmsgSSBhbSBhIGJpdCBjb25mdXNlZCBieSB0aGlzIGZpbHRlciwgb25lIHdvdWxkIHRoaW5rIHRoYXQgdGhlCm56diBmaWx0ZXIgYWJvdmUsIGlmIGFwcGxpZWQgY29ycmVjdGx5LCBzaG91bGQgZ2l2ZSB5b3UgZXhhY3RseSB0aGlzLgoKRm9yIHRoZSBtb21lbnQsIEkgYW0gZXhjbHVkaW5nIHRoZSBmb2xsb3dpbmcgYmxvY2sgaW4gb3JkZXIgdG8gc2VlIGhvdwptdWNoIHRpbWUvbWVtb3J5IGtlZXBpbmcgdGhlc2UgdmFyaWFibGVzIGNvc3RzLiAgSWYgSSByZWNhbGwgcHJvcGVybHksCnRoZSBtb2RlbCBvZiB0aGUgdG9wLTJrIHZhcmlhbnQgcG9zaXRpb25zIGNvc3QgfiAxLTRHIG9mIG1lbW9yeS4gIEkKaG9wZSB0aGF0IHRoaXMgc2NhbGVzIGxpbmVhcmx5LCBidXQgSSBhbSB0aGlua2luZyBpdCBtaWdodCBub3QuCgpgYGB7ciBleGNsdWRlX3NkcywgZXZhbD1GQUxTRX0Kc3RhbmRhcmRfZGV2cyA8LSBhcHBseShuenZfdGV4cHJzLCAyLCBzZCkKdG9wX3ByZWRpY3RvcnMgPC0gb3JkZXIoc3RhbmRhcmRfZGV2cywgZGVjcmVhc2luZyA9IFRSVUUpWzE6MzAwMF0Kbnp2X3RleHBycyA8LSBuenZfdGV4cHJzWywgdG9wX3ByZWRpY3RvcnNdCmBgYAoKIyMgQ2VudGVyIHRoZSBkYXRhCgpJIHRoaW5rIGNlbnRlcmluZyBtYXkgbm90IGJlIG5lZWRlZCBmb3IgdGhpcyBkYXRhLCBidXQgaGVyZSBpcyBob3c6CgpgYGB7ciBjZW50ZXJ9Cm56dl9jZW50ZXIgPC0gcHJlUHJvY2VzcyhuenZfdGV4cHJzLCBtZXRob2QgPSAiY2VudGVyIikKbnp2X3RleHBycyA8LSBwcmVkaWN0KG56dl9jZW50ZXIsIG56dl90ZXhwcnMpCmBgYAoKIyMgRHJvcCBjb3JyZWxhdGVkCgpUaGlzIGlzIGEgZmlsdGVyIHdoaWNoIGRvZXMgbm90IGNvcnJlc3BvbmQgdG8gYW55IG9mIHRob3NlIHdlIHVzZSBpbgpzZXF1ZW5jaW5nIGRhdGEgYmVjYXVzZSBnZW5lcyB3aGljaCBhcmUgaGlnaGx5IGNvcnJlbGF0ZWQgYXJlCmxpa2VseSB0byBiZSBvZiBpbW1lZGlhdGUgaW50ZXJlc3QuCgpJbiB0aGUgc2FtZSBmYXNoaW9uLCBJIHdhbnQgdG8gbGVhdmUgdGhpcyBvZmYgYmVjYXVzZSBsYXRlcgphcHBsaWNhdGlvbnMgb2YgdGhpcyBtb2RlbCB3aWxsIGluY2x1ZGUgbG93IGNvdmVyYWdlIHNhbXBsZXMgd2hpY2ggbWF5Cm5vdCBoYXZlIGV2ZXJ5IHZhcmlhbnQgcmVwcmVzZW50ZWQuCgpgYGB7ciBjb3JyZWxhdGVkfQojIyBUaGlzIHN0ZXAgdGFrZXMgYSB3aGlsZS4uLgpzeXN0ZW0udGltZSh7CiAgbnp2X2NvcnJlbGF0ZWQgPC0gcHJlUHJvY2VzcyhuenZfdGV4cHJzLCBtZXRob2QgPSAiY29yciIsIGN1dG9mZiA9IDAuOTUpCiAgbnp2X3VuY29yciA8LSBwcmVkaWN0KG56dl9jb3JyZWxhdGVkLCBuenZfdGV4cHJzKQp9KQpkaW0obnp2X3VuY29ycikKYGBgCgojIE1lcmdlIHRoZSBhcHByb3ByaWF0ZSBtZXRhZGF0YQoKVGhlcmUgYXJlIGEgZmV3IG1ldGFkYXRhIGZhY3RvcnMgd2hpY2ggbWlnaHQgcHJvdmUgb2YgaW50ZXJlc3QgZm9yCmNsYXNzaWZpY2F0aW9uLiAgVGhlIG1vc3Qgb2J2aW91cyBhcmUgb2YgY291cnNlIG91dGNvbWUsIGNsaW5pYywKZG9ub3IsIHZpc2l0LCBjZWxsdHlwZS4gIEkgYW0sIGZvciB0aGUgbW9tZW50LCBvbmx5IGxpa2VseSB0byBmb2N1cyBvbgpvdXRjb21lLiAgQUZBSUNUIEkgY2FuIG9ubHkgaW5jbHVkZSBvbmUgb2YgdGhlc2UgYXQgYSB0aW1lIGluIHRoZQpkYXRhLCB3aGljaCBpcyBhIHNoYW1lLgoKYGBge3IgbWVyZ2VfcGllY2VzfQppbnRlcmVzdGluZ19tZXRhIDwtIHBEYXRhKHRjX2NsaW5pY2FsKVssIGMoImZpbmFsb3V0Y29tZSIsICJkb25vciIsICJwZXJzaXN0ZW5jZSIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAidmlzaXRudW1iZXIiLCAic2VsZWN0aW9ubWV0aG9kIiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJ0eXBlb2ZjZWxscyIsICJ0aW1lIiwgImNsaW5pYyIpXQoKbWxfZGYgPC0gYXMuZGF0YS5mcmFtZShjYmluZChvdXRjb21lX2ZhY3RvciwgYXMuZGF0YS5mcmFtZShuenZfdW5jb3JyKSkpCm1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0gPC0gYXMuZmFjdG9yKG1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0pCmRpbShtbF9kZikKYGBgCgojIFNwbGl0IHRoZSBkYXRhIGludG8gdHJhaW5pbmcvdGVzdGluZwoKY2FyZXQgcHJvdmlkZXMgbmljZSBmdW5jdGlvbmFsaXR5IGZvciBzcGxpdHRpbmcgdXAgdGhlIGRhdGEuICBJCnN1c3BlY3QgdGhlcmUgYXJlIG1hbnkgbW9yZSBmdW4ga25vYnMgSSBjYW4gcGxheSB3aXRoIGZvciBpbnN0YW5jZXMKd2hlcmUgSSBuZWVkIHRvIGV4Y2x1ZGUgc29tZSBsZXZlbHMgb2YgYSBmYWN0b3IgYW5kIHN1Y2guICBJbiB0aGlzCmNhc2UgSSBqdXN0IHdhbnQgdG8gc3BsaXQgYnkgb3V0Y29tZS4KCiMjIFZpYSBkYXRhIHNwbGl0dGluZwoKYGBge3Igc3BsaXRfdHJhaW5fdGVzdH0KIyMgVGhlIHZhcmlhYmxlIG91dGNvbWVfZmFjdG9yIHdhcyBjcmVhdGVkIGF0IHRoZSB0b3Agb2YgdGhpcyBkb2N1bWVudCwKIyMgd2hpY2ggaXMgYSBsaXR0bGUgYXdrd2FyZCBoZXJlLgp0cmFpbl9pZHggPC0gY3JlYXRlRGF0YVBhcnRpdGlvbihvdXRjb21lX2ZhY3RvciwgcCA9IDAuNiwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgbGlzdCA9IEZBTFNFLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICB0aW1lcyA9IDEpCnN1bW1hcnkobWxfZGZbdHJhaW5faWR4LCAib3V0Y29tZV9mYWN0b3IiXSkKIyMgVHJhaW5pbmcgc2V0IGhhcyAxMTIgc2FtcGxlcy4KCnRyYWluX3Jvd25hbWVzIDwtIHJvd25hbWVzKG1sX2RmKVt0cmFpbl9pZHhdCm5vdF90cmFpbl9pZHggPC0gISByb3duYW1lcyhtbF9kZikgJWluJSB0cmFpbl9yb3duYW1lcwpzdW1tYXJ5KG1sX2RmW25vdF90cmFpbl9pZHgsICJvdXRjb21lX2ZhY3RvciJdKQpub3RfdHJhaW5fcm93bmFtZXMgPC0gcm93bmFtZXMobWxfZGYpW25vdF90cmFpbl9pZHhdCgp0cmFpbl9kZiA8LSBhcy5kYXRhLmZyYW1lKG1sX2RmW3RyYWluX3Jvd25hbWVzLCBdKQpkaW0odHJhaW5fZGYpCnRlc3RfZGYgPC0gYXMuZGF0YS5mcmFtZShtbF9kZltub3RfdHJhaW5fcm93bmFtZXMsIF0pCmRpbSh0ZXN0X2RmKQojIyBSZW1vdmUgdGhlIG91dGNvbWUgZmFjdG9yIGZyb20gdGhlIHRlc3QgZGF0YSwganVzdCBpbiBjYXNlLgp0ZXN0X291dGNvbWUgPC0gdGVzdF9kZltbIm91dGNvbWVfZmFjdG9yIl1dCnRlc3RfZGZbWyJvdXRjb21lX2ZhY3RvciJdXSA8LSBOVUxMCmBgYAoKIyMgVmlhIHNhbXBsaW5nCgpUaGVyZSBhcmUgYSBmZXcgbGlrZWx5IHNhbXBsaW5nIG1ldGhvZHM6IGNyb3NzLXZhbGlkYXRpb24sCmJvb3RzdHJhcHBpbmcsIGFuZCBqYWNra25pZmluZy4gIEkgd2lsbCB0cnkgdGhvc2Ugb3V0IGxhdGVyLgoKIyBUcnkgb3V0IHRyYWluaW5nIGFuZCBwcmVkaWN0aW9uIG1ldGhvZHMKCk15IGdvYWxzIGZyb20gaGVyZSBvbiB3aWxsIGJlIHRvIGdldCB0aGUgYmVnaW5uaW5ncyBvZiBhIHNlbnNlIG9mIHRoZQp2YXJpb3VzIG1ldGhvZHMgSSBjYW4gdXNlIHRvIGNyZWF0ZSB0aGUgbW9kZWxzIGZyb20gdGhlIHRyYWluaW5nIGRhdGEKYW5kIHByZWRpY3QgdGhlIG91dGNvbWUgb24gdGhlIHRlc3QgZGF0YS4gIEkgYW0gaG9waW5nIGFsc28gdG8gcGljayB1cApzb21lIGlkZWEgb2Ygd2hhdCB0aGUgdmFyaW91cyBhcmd1bWVudHMgbWVhbiB3aGlsZSBJIGFtIGF0IGl0LgoKIyMgVHJ5IG91dCBLTk4KCmstbmVhcmVzdCBuZWlnaGJvcnMgaXMgc29tZXdoYXQgc2ltaWxhciB0byBhIGttZWFucyBlc3RpbWF0ZS4gIFRodXMKdGhlIHByaW1hcnkgYXJndW1lbnQgaXMgJ2snCgojIyMgTW9kZWwgY3JlYXRpb24gYW5kIHBlcmZvcm1hbmNlCgpgYGB7ciBrbm59Cmtubl9maXQgPC0ga25uMyh4ID0gdHJhaW5fZGZbLCAtMV0sCiAgICAgICAgICAgICAgICB5ID0gdHJhaW5fZGZbWyJvdXRjb21lX2ZhY3RvciJdXSwKICAgICAgICAgICAgICAgIGsgPSAzKQprbm5fcHJlZGljdF90cmFpbmVkIDwtIHByZWRpY3Qoa25uX2ZpdCwgdHJhaW5fZGZbLCAtMV0sIHR5cGUgPSAiY2xhc3MiKQpjb25mdXNpb25NYXRyaXgoZGF0YSA9IHRyYWluX2RmW1sxXV0sIHJlZmVyZW5jZSA9IGtubl9wcmVkaWN0X3RyYWluZWQpCmBgYAoKQXMgdGhlIGNvbmZ1c2lvbiBtYXRyaXggc2hvd3MsIHRoaXMgZmFpbGVkIGZvciBhIGZldyBzYW1wbGVzLiAgUGVyaGFwcwpsZXQgdXMgY2hhbmdlIGsgYW5kIHNlZSBpZiBpdCBpbXByb3Zlcy4KCkhlcmUgaXMgYSB0YWJsZSBvZiBmYXNlIHBvc2l0aXZlcy9uZWdhdGl2ZXMgZm9yIGEgZmV3IHZhbHVlcyBvZiAnaycsCmluIHRoaXMgY29udGV4dCBhIGZhbHNlIHBvc2l0aXZlIGlzIGNhbGxpbmcgYSBrbm93biBjdXJlIGFzIGEgZmFpbHVyZQphbmQgZmFsc2UgbmVnYXRpdmUgaXMgY2FsbGluZyBhIGtub3duIGZhaWx1cmUgYXMgYSBjdXJlLgoKfC0tLXwtLS18LS0tfAp8ayAgfGZwIHxmbiB8CnwyICB8MCAgfDggIHwKfDMgIHw1ICB8NSAgfAp8NCAgfDggIHw5ICB8Cnw1ICB8MTEgfDcgIHwKfDYgIHwxNSB8OCAgfAoKTm90ZTogdGhpcyBkZXBlbmRzIG9uIHRoZSBsdWNrIG9mIHJhbmQoKSwgc28gdGhlIGFib3ZlIG51bWJlcnMgc2hpZnQKbW9kZXJhdGVseSBmcm9tIG9uZSBydW4gdG8gdGhlIG5leHQuICBUaHVzIEkgdGhpbmsgSSB3aWxsIGp1c3QgdXNlIDIKb3IgMy4KCmBgYHtyIGtubl9maXQyfQprbm5fZml0IDwtIGtubjMoeCA9IHRyYWluX2RmWywgLTFdLAogICAgICAgICAgICAgICAgeSA9IHRyYWluX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0sCiAgICAgICAgICAgICAgICBrID0gMykKa25uX3ByZWRpY3RfdHJhaW5lZCA8LSBwcmVkaWN0KGtubl9maXQsIHRyYWluX2RmWywgLTFdLCB0eXBlID0gImNsYXNzIikKY29uZnVzaW9uTWF0cml4KGRhdGEgPSB0cmFpbl9kZltbMV1dLCByZWZlcmVuY2UgPSBrbm5fcHJlZGljdF90cmFpbmVkKQpgYGAKCiMjIyBWaXN1YWxpemUgdGhlIGFjY3VyYWN5IHdpdGggYSBST0MgY3VydmUKCmBgYHtyIGtubl90cmFpbmVkfQpuYW1lcyhrbm5fcHJlZGljdF90cmFpbmVkKSA8LSByb3duYW1lcyh0cmFpbl9kZikKc3VtbWFyeShrbm5fcHJlZGljdF90cmFpbmVkID09IHBEYXRhKHRjX25vcm0pW3RyYWluX2lkeCwgImZpbmFsb3V0Y29tZSJdKQoKdHQgPC0gcHJlZGljdChrbm5fZml0LCB0cmFpbl9kZlssIC0xXSkKY3VydmUgPC0gcFJPQzo6cm9jKHJlc3BvbnNlID0gdHJhaW5fZGZbLCAxXSwgcHJlZGljdG9yID0gdHRbLCAxXSwKICAgICAgICAgICAgICAgICAgIGxldmVscyA9IHJldihsZXZlbHMob3V0Y29tZV9mYWN0b3IpKSkKcGxvdChjdXJ2ZSkKcFJPQzo6YXVjKGN1cnZlKQpgYGAKCiMjIyBQcmVkaWN0IHRoZSByZXN0IG9mIHRoZSBkYXRhIHdpdGggdGhpcyBtb2RlbC4KCmBgYHtyIGtubl90ZXN0fQojIyBSZXR1cm4gdGhlIGNsYXNzaWZpY2F0aW9ucwprbm5fcHJlZGljdF90ZXN0IDwtIHByZWRpY3Qoa25uX2ZpdCwgdGVzdF9kZiwgdHlwZSA9ICJjbGFzcyIpCiMjIGFuZCB0aGUgdmFsdWVzLCB3aGljaCB3aWxsIGJlIHVzZWQgZm9yIG91ciBST0MgY3VydmUuCmtubl9wcmVkaWN0X3Rlc3RfdmFsdWVzIDwtIHByZWRpY3Qoa25uX2ZpdCwgdGVzdF9kZikKCm5hbWVzKGtubl9wcmVkaWN0X3Rlc3QpIDwtIHJvd25hbWVzKHRlc3RfZGYpCmtubl9hZ3JlZW1lbnQgPC0ga25uX3ByZWRpY3RfdGVzdCA9PSB0ZXN0X291dGNvbWUKbmFtZXMoa25uX2FncmVlbWVudCkgPC0gcm93bmFtZXModGVzdF9kZikKc3VtbWFyeShrbm5fYWdyZWVtZW50KQpuYW1lcyhrbm5fYWdyZWVtZW50KVtrbm5fYWdyZWVtZW50ID09IEZBTFNFXQoKY3VydmUgPC0gcFJPQzo6cm9jKHJlc3BvbnNlID0gdGVzdF9vdXRjb21lLAogICAgICAgICAgICAgICAgICAgcHJlZGljdG9yID0ga25uX3ByZWRpY3RfdGVzdF92YWx1ZXNbLCAxXSkKcGxvdChjdXJ2ZSkKcFJPQzo6YXVjKGN1cnZlKQoKIyMgRmluZCB0aGUgc2FtcGxlcyB3aGljaCAoZGlzKWFncmVlCnJlZl9jb2wgPC0gImZpbmFsb3V0Y29tZSIKcmVmX3Jlc3VsdCA8LSBwRGF0YSh0Y19jbGluaWNhbClbW3JlZl9jb2xdXQpuYW1lcyhyZWZfcmVzdWx0KSA8LSByb3duYW1lcyhwRGF0YSh0Y19jbGluaWNhbCkpCnJlZl94cmVmIDwtIG5hbWVzKHJlZl9yZXN1bHQpICVpbiUgbmFtZXMoa25uX3ByZWRpY3RfdGVzdCkKcmVmX2NvbXAgPC0gcmVmX3Jlc3VsdFtyZWZfeHJlZl0KcmVmX2NvbXBfaWR4IDwtIHJlZl9jb21wICE9ICJ1bmtub3duIgpyZWZfY29tcCA8LSByZWZfY29tcFtyZWZfY29tcF9pZHhdCnJlZl9jb21wX2lkeCA8LSAhaXMubmEocmVmX2NvbXApCnJlZl9jb21wIDwtIHJlZl9jb21wW3JlZl9jb21wX2lkeF0KCnRlc3RfY29tcF9pZHggPC0gbmFtZXMoa25uX3ByZWRpY3RfdGVzdCkgJWluJSBuYW1lcyhyZWZfY29tcCkKdGVzdF9jb21wIDwtIGtubl9wcmVkaWN0X3Rlc3RbdGVzdF9jb21wX2lkeF0KCmFncmVlbWVudCA8LSBhcy5jaGFyYWN0ZXIocmVmX2NvbXApID09IGFzLmNoYXJhY3Rlcih0ZXN0X2NvbXApCmFncmVlX251bSA8LSBzdW0oYWdyZWVtZW50KQphZ3JlZV9wcm9wIDwtIGFncmVlX251bSAvIGxlbmd0aChyZWZfY29tcCkKYWdyZWVfcHJvcAoKZGlzYWdyZWUgPC0gYXMuY2hhcmFjdGVyKHJlZl9jb21wKSAhPSBhcy5jaGFyYWN0ZXIodGVzdF9jb21wKQp0ZXN0X2NvbXBbZGlzYWdyZWVdCmBgYAoKIyMgUGVyZm9ybSBjcm9zcy12YWxpZGF0aW9uIHRvIGVzdGltYXRlIGsKClRoZSBjcm9zcyB2YWxpZGF0aW9uIG1ldGhvZCBvZiByZXBlYXRlZCBzYW1wbGluZyB0aGUgZGF0YSBpcyBhbGwgZG9uZQp3aXRoaW4gdGhlIHRyYWluKCkgZnVuY3Rpb24uICBXaXRoIHRoYXQgaW4gbWluZCwgaGVyZSBpdCBpcyBvcGVyYXRpbmcKd2l0aCB0aGUga25uIG1ldGhvZC4KCiMjIyBDViB3aXRoIGtubgoKV2hlbiB0cmFpbigpIGlzIGNhbGxlZCB3aXRoIHRoZSB0ckNvbnRyb2wgYW5kIHR1bmVHcmlkLCB3ZSBjYW4gY29udHJvbApob3cgdGhlIGtubiB0cmFpbmluZyBpcyByZXBlYXRlZCwgaW4gdGhpcyBjYXNlIGl0IHdpbGwgaXRlcmF0ZSBvdmVyIGsKZnJvbSAxIHRvIDEwLgoKYGBge3IgY3Zfa25ufQpjdl9jb250cm9sIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCkKa25uX3RyYWluX2ZpdCA8LSB0cmFpbihvdXRjb21lX2ZhY3RvciB+IC4sIGRhdGEgPSB0cmFpbl9kZiwKICAgICAgICAgICAgICAgICAgICAgICBtZXRob2QgPSAia25uIiwKICAgICAgICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSBjdl9jb250cm9sLAogICAgICAgICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZShrID0gMToxMCkpCmtubl90cmFpbl9maXRbWyJiZXN0VHVuZSJdXQoKcGxvdCh4ID0gMToxMCwgMSAtIGtubl90cmFpbl9maXQkcmVzdWx0c1ssIDJdLCBwY2ggPSAxOSwKICAgICB5bGFiID0gInByZWRpY3Rpb24gZXJyb3IiLCB4bGFiID0gImsiKQpsaW5lcyhsb2Vzcy5zbW9vdGgoeCA9IDE6MTAsIDEgLSBrbm5fdHJhaW5fZml0JHJlc3VsdHNbLCAyXSxkZWdyZWUgPSAyKSwKICAgICAgY29sID0gIiNDQzAwMDAiKQpgYGAKCiMjIyBCb290c3RyYXAgd2l0aCBrbm4KCmBgYHtyIGJvb3Rfa25ufQpib290X2NvbnRyb2wgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJib290IiwgbnVtYmVyID0gMjAsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcmV0dXJuUmVzYW1wID0gImFsbCIpCgprbm5fdHJhaW5fZml0IDwtIHRyYWluKG91dGNvbWVfZmFjdG9yIH4gLiwgZGF0YSA9IHRyYWluX2RmLAogICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJrbm4iLAogICAgICAgICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IGJvb3RfY29udHJvbCwKICAgICAgICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoayA9IDE6MTApKQprbm5fdHJhaW5fZml0W1siYmVzdFR1bmUiXV0KCnBsb3QoeCA9IDE6MTAsIDEgLSBrbm5fdHJhaW5fZml0JHJlc3VsdHNbLCAyXSwgcGNoID0gMTksCiAgICAgeWxhYiA9ICJwcmVkaWN0aW9uIGVycm9yIiwgeGxhYiA9ICJrIikKbGluZXMobG9lc3Muc21vb3RoKHggPSAxOjEwLCAxIC0ga25uX3RyYWluX2ZpdCRyZXN1bHRzWywgMl0sZGVncmVlID0gMiksCiAgICAgIGNvbCA9ICIjQ0MwMDAwIikKYGBgCgojIyMgRXhwbGFpbiB0aGUgaW1wb3J0YW50IHZhcmlhYmxlcwoKSW4gdGhpcyBpbnN0YW5jZSB3ZSB3aWxsIHNlYXJjaCBmb3IgZ2VuZXMgd2hpY2ggd2VyZSBpbXBvcnRhbnQgZm9yIHRoZQptb2RlbCdzIGNyZWF0aW9uLgoKVGhlIERBTEVYIHBhY2thZ2UgcHJvdmlkZXMgYSBmdW5jdGlvbjogZmVhdHVyZV9pbXBvcnRhbmNlKCkgd2hpY2gKc2Vla3MgdG8gdXNlIGEgc2VyaWVzIG9mIG90aGVyIG1ldGhvZHMgdG8gZXh0cmFjdCAoaW4gdGhpcyBjYXNlLApnZW5lcykgZmVhdHVyZXMgd2hpY2ggZG8gYSBnb29kIGpvYiBvZiBleHBsYWluaW5nIHRoZSByZXN1bHQgcHJvZHVjZWQKYnkgdGhlIG1vZGVsLiAgSW4gdGhlIGNhc2Ugb2YgdGhpcyBkYXRhc2V0LCB3aGljaCBoYXMgdGhvdXNhbmRzIG9mCmZlYXR1cmVzLCB0aGlzIGRvZXMgbm90IGFwcGVhciB0byBlbmQgd2VsbC4KCmBgYHtyIGV4cGxhaW5fa25ufQpleHBsYWluZXJfa25uIDwtIGV4cGxhaW4oa25uX2ZpdCwgbGFiZWwgPSAia25uIiwKICAgICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSB0cmFpbl9kZlssIC0xXSwKICAgICAgICAgICAgICAgICAgICAgICAgIHkgPSBhcy5udW1lcmljKHRyYWluX2RmWywgMV0pKQoKIyMgQUZBSUNUIHRoZSBmb2xsb3dpbmcgd2lsbCB0YWtlIGZvcmV2ZXIgdW5sZXNzIHdlIGRyYXN0aWNhbGx5IHJlZHVjZSB0aGUgY29tcGxleGl0eSBvZiB0aGUgbW9kZWwuCiMjIGZlYXR1cmVzIDwtIGZlYXR1cmVfaW1wb3J0YW5jZShleHBsYWluZXJfa25uLCBuX3NhbXBsZSA9IDUwLCB0eXBlID0gImRpZmZlcmVuY2UiKQpgYGAKCiMjIFJhbmRvbSBGb3Jlc3QKClRoZSBwYXJhbWV0ZXIgJ210cnknIGlzIG9mdGVuIGltcG9ydGFudCwgaWYgSSByZWFkIHRoZSB0ZXh0IGNvcnJlY3RseQppdCBjb250cm9scyBob3cgbWFueSB2YXJpYWJsZXMgdG8gc2FtcGxlIGluIGVhY2ggc3BsaXQgb2YgdGhlIHRyZWUuClRodXMgaGlnaGVyIG51bWJlcnMgc2hvdWxkIHByZXN1bWFibHkgbWFrZSBpdCBtb3JlIHNwZWNpZmljIGF0IHRoZQpyaXNrIG9mIG92ZXJmaXR0aW5nLgoKU2V0dGluZyBtaW4ubm9kZS5zaXplIHNldHMgdGhlIG1pbmltdW1lIG5vZGUgc2l6ZSBvZiB0ZXJtaW5hbCBub2RlcyBpbgplYWNoIHRyZWUuICBFYWNoIGluY3JlbWVudCB1cCBzcGVlZHMgdGhlIGFsZ29yaXRobS4KCkkgYW0gZ29pbmcgdG8gdXNlIG15IGJvb3QgY29udHJvbCB0cmFpbmVyIGZyb20gYWJvdmUgYW5kIHNlZSBob3cgaXQgZ29lcy4KCmBgYHtyIHJmX3Rlc3R9CmxpYnJhcnkocmFuZ2VyKQojI3JmX21ldGhvZCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kID0gInJhbmdlciIsIG51bWJlciA9IDEwLCB2ZXJib3NlID0gVFJVRSkKCnJmX3RyYWluX2ZpdCA8LSB0cmFpbihvdXRjb21lX2ZhY3RvciB+IC4sIGRhdGEgPSB0cmFpbl9kZiwKICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJyYW5nZXIiLCB0ckNvbnRyb2wgPSBib290X2NvbnRyb2wsCiAgICAgICAgICAgICAgICBpbXBvcnRhbmNlID0gInBlcm11dGF0aW9uIiwKICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZSgKICAgICAgICAgICAgICAgICAgICBtdHJ5ID0gMjAwLAogICAgICAgICAgICAgICAgICAgIG1pbi5ub2RlLnNpemUgPSAxLAogICAgICAgICAgICAgICAgICAgIHNwbGl0cnVsZSA9ICJnaW5pIiksCiAgICAgICAgICAgICAgICB2ZXJib3NlID0gVFJVRSkKcmZfdHJhaW5fZml0W1siZmluYWxNb2RlbCJdXVtbInByZWRpY3Rpb24uZXJyb3IiXV0KdmFyaWFibGVfaW1wb3J0YW5jZSA8LSB2YXJJbXAocmZfdHJhaW5fZml0KQpwbG90KHZhcmlhYmxlX2ltcG9ydGFuY2UsIHRvcCA9IDE1KQoKcmZfcHJlZGljdF90cmFpbmVkIDwtIHByZWRpY3QocmZfdHJhaW5fZml0LCB0cmFpbl9kZikKbmFtZXMocmZfcHJlZGljdF90cmFpbmVkKSA8LSByb3duYW1lcyh0cmFpbl9kZikKc3VtbWFyeShyZl9wcmVkaWN0X3RyYWluZWQgPT0gcERhdGEodGNfbm9ybSlbdHJhaW5faWR4LCAiZmluYWxvdXRjb21lIl0pCmBgYAoKIyMjIyBEaWdyZXNzaW9uIGRvIHRoZSBnZW5lcyBwcm92aWRlIGJ5IHZhckltcCBtZWFuIGFueXRoaW5nPwoKTGV0IHVzIHRha2UgYSBtb21lbnQgYW5kIHNlZSBpZiB0aGUgdG9wLW4gZ2VuZXMgcmV0dXJuZWQgYnkgdmFySW1wKCkKaGF2ZSBzb21lIG1lYW5pbmcgd2hpY2gganVtcHMgb3V0LiAgT25lIG1pZ2h0IGFzc3VtZSwgZ2l2ZW4gb3VyIGV4dGFudApEaWZmZXJlbnRpYWwgRXhwcmVzc2lvbiByZXN1bHRzLCB0aGF0IHRoZSBpbnRlcmxldWtpbiByZXNwb25zZSB3aWxsIGJlCmEgbGlrZWx5IGNhbmRpZGF0ZS4KCmBgYHtyIHZhcmltcF9ncH0KdGVzdF9pbXBvcnRhbmNlIDwtIHZhckltcChyZl90cmFpbl9maXQsIHNjYWxlID0gVFJVRSkKdGVzdF9pZHggPC0gb3JkZXIodGVzdF9pbXBvcnRhbmNlW1siaW1wb3J0YW5jZSJdXVtbIk92ZXJhbGwiXV0sIGRlY3JlYXNpbmcgPSBUUlVFKQp0ZXN0X2ltcCA8LSB0ZXN0X2ltcG9ydGFuY2VbWyJpbXBvcnRhbmNlIl1dW3Rlc3RfaWR4LCBdCm5hbWVzKHRlc3RfaW1wKSA8LSByb3duYW1lcyh0ZXN0X2ltcG9ydGFuY2VbWyJpbXBvcnRhbmNlIl1dKVt0ZXN0X2lkeF0KaGVhZCh0ZXN0X2ltcCwgbiA9IDIwMCkKbW9zdF9pbXBvcnRhbnRfaWRzIDwtIG5hbWVzKGhlYWQodGVzdF9pbXAsIG4gPSAyMDApKQpjb252ZXJ0X2lkcyhtb3N0X2ltcG9ydGFudF9pZHMsIGZyb20gPSAiRU5TRU1CTCIsIHRvID0gIlNZTUJPTCIpCgppbXBvcnRhbmNlX2dwIDwtIHNpbXBsZV9ncHJvZmlsZXIobmFtZXMoaGVhZCh0ZXN0X2ltcCwgbiA9IDYwKSkpCmltcG9ydGFuY2VfZ3AkcHZhbHVlX3Bsb3RzJEJQCmBgYAoKIyMjIE5vdyB0aGUgcmFuZG9tIGZvcmVzdCB0ZXN0ZXJzIQoKYGBge3IgcmZfcHJlZGljdF90ZXN0fQpyZl9wcmVkaWN0X3Rlc3QgPC0gcHJlZGljdChyZl90cmFpbl9maXQsIHRlc3RfZGYpCm5hbWVzKHJmX3ByZWRpY3RfdGVzdCkgPC0gcm93bmFtZXModGVzdF9kZikKcmZfYWdyZWVtZW50IDwtIHJmX3ByZWRpY3RfdGVzdCA9PSBwRGF0YSh0Y19ub3JtKVtub3RfdHJhaW5faWR4LCAiZmluYWxvdXRjb21lIl0KbmFtZXMocmZfYWdyZWVtZW50KSA8LSByb3duYW1lcyh0ZXN0X2RmKQpzdW1tYXJ5KHJmX2FncmVlbWVudCkKbmFtZXMocmZfYWdyZWVtZW50KVtyZl9hZ3JlZW1lbnQgPT0gRkFMU0VdCmBgYAoKIyMgR0xNLCBvciBMb2dpc3RpYyByZWdyZXNzaW9uIGFuZCByZWd1bGFyaXphdGlvbgoKTG9naXN0aWMgcmVncmVzc2lvbiBpcyBhIHN0YXRpc3RpY2FsIG1ldGhvZCBmb3IgYmluYXJ5IHJlc3BvbnNlcy4KSG93ZXZlciwgaXQgaXMgYWJsZSB0byB3b3JrIHdpdGggbXVsdGlwbGUgY2xhc3NlcyBhcyB3ZWxsLiAgVGhlCmdlbmVyYWwgaWRlYSBvZiB0aGlzIG1ldGhvZCBpcyB0byBmaW5kIHBhcmFtZXRlcnMgd2hpY2ggaW5jcmVhc2UgdGhlCmxpa2VsaWhvb2QgdGhhdCB0aGUgb2JzZXJ2ZWQgZGF0YSBpcyBzYW1wbGVkIGZyb20gYSBzdGF0aXN0aWNhbApkaXN0cmlidXRpb24gb2YgaW50ZXJlc3QuICBUaGUgdHJhbnNmb3JtYXRpb25zIGFuZCBsaW5lYXIKcmVncmVzc2lvbi1lc3F1ZSB0YXNrcyBwZXJmb3JtZWQgYXJlIGNvbmZ1c2luZywgYnV0IG9uY2UgdGhvc2UgYXJlCnBlcmZvcm1lZCwgdGhlIHRhc2sgYmVjb21lcyBzZXR0aW5nIHRoZSBtb2RlbCdzIChmaXR0aW5nKSBwYXJhbWV0ZXJzCnRvIHZhbHVlcyB3aGljaCBpbmNyZWFzZSB0aGUgcHJvYmFiaWxpdHkgdGhhdCB0aGUgc3RhdGlzdGljYWwgbW9kZWwKbG9va3MgbGlrZSB0aGUgYWN0dWFsIGRhdGFzZXQgZ2l2ZW4gdGhlIHRyYWluaW5nIGRhdGEsIGFuZCB0aGF0IHdoZW4Kc2FtcGxlcywgd2lsbCByZXR1cm4gdmFsdWVzIHdoaWNoIGFyZSBzaW1pbGFyLiAgVGhlIG1vc3QgbGlrZWx5CnN0YXRpc3RpY2FsIGRpc3RyaWJ1dGlvbnMgb25lIHdpbGwgd2FudCB0byBmaXQgYXJlIHRoZSBHYXVzc2lhbiwgaW4Kd2hpY2ggY2FzZSB3ZSB3YW50IHRvIHRyYW5zZm9ybS9ub3JtYWxpemUgdGhlIG1lYW4vdmFyaWFuY2Ugb2Ygb3VyCnZhcmlhYmxlcyBzbyB0aGV5IGxvb2sgd2hhdGV2ZXIgbm9ybWFsIGRpc3RyaWJ1dGlvbiB3ZSBhcmUgdXNpbmcuCkNvbnZlcnNlbHksIGxvZ2lzdGljIHJlZ3Jlc3Npb24gdXNlcyBhIGJpbm5vbWlhbCBkaXN0cmlidXRpb24gKGxpa2UKb3VyIHJhdyBzZXF1ZW5jaW5nIGRhdGEhKSBidXQgd2hpY2ggaXMgZnJvbSAwLTEuCgojIyMgVXNpbmcgYSBzaW5nbGUgZ2VuZQoKTGV0IHVzIHRha2UgdGhlIG1vc3QgaW1wb3J0YW50IGdlbmUgb2JzZXJ2ZWQgaW4gb25lIG9mIG91ciBwcmV2aW91cwp0cmFpbmluZyBzZXRzOiBFTlNHMDAwMDAyNDg0MDUgUFJSNS1BUkhHQVA4CgpgYGB7ciBnbG1fcmVncmVzc2lvbn0KZ2VuZV9pZCA8LSAiRU5TRzAwMDAwMjQ4NDA1IgpzaW5nbGVfZml0IDwtIHRyYWluKAogICAgb3V0Y29tZV9mYWN0b3IgfiBFTlNHMDAwMDAyNDg0MDUsIGRhdGEgPSB0cmFpbl9kZiwKICAgIG1ldGhvZCA9ICJnbG0iLCBmYW1pbHkgPSAiYmlub21pYWwiLCB0ckNvbnRyb2wgPSB0cmFpbkNvbnRyb2woIm5vbmUiKSkKCnR0IDwtIGRhdGEuZnJhbWUoIkVOU0cwMDAwMDI0ODQwNSIgPSBzZXEobWluKHRyYWluX2RmW1tnZW5lX2lkXV0pLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIG1heCh0cmFpbl9kZltbZ2VuZV9pZF1dKSxsZW49MTAwKSkKIyMgcHJlZGljdCBwcm9iYWJpbGl0aWVzIGZvciB0aGUgc2ltdWxhdGVkIGRhdGEKdHQkc3VidHlwZSA9IHByZWRpY3Qoc2luZ2xlX2ZpdCwgbmV3ZGF0YT10dCwgdHlwZT0icHJvYiIpWywgMV0KIyMgcGxvdCB0aGUgc2lnbW9pZCBjdXJ2ZSBhbmQgdGhlIHRyYWluaW5nIGRhdGEKcGxvdChpZmVsc2Uob3V0Y29tZV9mYWN0b3IgPT0gImN1cmUiLCAxLCAwKSB+IEVOU0cwMDAwMDI0ODQwNSwKICAgICBkYXRhPXRyYWluX2RmLCBjb2w9InJlZDQiLAogICAgIHlsYWI9IkNGIGFzIDAgb3IgMSIsIHhsYWI9ImZhdm9yaXRlIGdlbmUgZXhwcmVzc2lvbiIpCmxpbmVzKHN1YnR5cGUgfiBFTlNHMDAwMDAyNDg0MDUsIHR0LCBjb2w9ImdyZWVuNCIsIGx3ZD0yKQpgYGAKCkhhdmluZyB0cmllZCB3aXRoIDEgZ2VuZSwgbGV0IHVzIGV4dGVuZCB0aGlzIHRvIGFsbCBnZW5lcy4gIEluIG15CmZpcnN0IHRyeSBvZiB0aGlzLCBpdCB0b29rIGEgbG9uZyB0aW1lLgoKYGBge3IgZ2xtX2FsbH0KZ2xtX2ZpdCA8LSB0cmFpbihvdXRjb21lX2ZhY3RvciB+IC4sIGRhdGEgPSB0cmFpbl9kZiwKICAgICAgICAgICAgICAgICB0ckNvbnRyb2wgPSBib290X2NvbnRyb2wsCiAgICAgICAgICAgICAgICAgbWV0aG9kID0gImdsbSIsIGZhbWlseSA9ICJiaW5vbWlhbCIpCmBgYAoKYGBge3IgZ2xtX3Rlc3R9CmxpYnJhcnkoZ2xtbmV0KQojI3JmX21ldGhvZCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kID0gInJhbmdlciIsIG51bWJlciA9IDEwLCB2ZXJib3NlID0gVFJVRSkKIyMgdHJhaW5fbWV0aG9kIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiY3YiLCBudW1iZXIgPSAxMCkKZ2xtX2ZpdCA8LSB0cmFpbihvdXRjb21lX2ZhY3RvciB+IC4sIGRhdGEgPSB0cmFpbl9kZiwgbWV0aG9kID0gImdsbW5ldCIsCiAgICAgICAgICAgICAgICAgdHJDb250cm9sID0gYm9vdF9jb250cm9sLCBpbXBvcnRhbmNlID0gInBlcm11dGF0aW9uIiwKICAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoCiAgICAgICAgICAgICAgICAgICBhbHBoYSA9IDAuNSwKICAgICAgICAgICAgICAgICAgIGxhbWJkYSA9IHNlcSgwLjEsIDAuNywgMC4wNSkpLAogICAgICAgICAgICAgICAgIHZlcmJvc2UgPSBUUlVFKQoKZ2xtX3ByZWRpY3RfdHJhaW5lZCA8LSBwcmVkaWN0KGdsbV9maXQsIHRyYWluX2RmKQpuYW1lcyhnbG1fcHJlZGljdF90cmFpbmVkKSA8LSByb3duYW1lcyh0cmFpbl9kZikKc3VtbWFyeShnbG1fcHJlZGljdF90cmFpbmVkID09IHBEYXRhKHRjX25vcm0pW3RyYWluX2lkeCwgImZpbmFsb3V0Y29tZSJdKQpjb25mdXNpb25NYXRyaXgoZGF0YSA9IHRyYWluX2RmWywgMV0sIHJlZmVyZW5jZSA9IGdsbV9wcmVkaWN0X3RyYWluZWQpCmBgYAoKIyMjIE5vdyB0aGUgR0xNIHRlc3RlcnMhCgpgYGB7ciBnbG1fcHJlZGljdF90ZXN0fQpnbG1fcHJlZGljdF90ZXN0IDwtIHByZWRpY3QoZ2xtX2ZpdCwgdGVzdF9kZikKbmFtZXMoZ2xtX3ByZWRpY3RfdGVzdCkgPC0gcm93bmFtZXModGVzdF9kZikKZ2xtX3ByZWRpY3RfdGVzdApnbG1fYWdyZWVtZW50IDwtIGdsbV9wcmVkaWN0X3Rlc3QgPT0gcERhdGEodGNfbm9ybSlbbm90X3RyYWluX2lkeCwgImZpbmFsb3V0Y29tZSJdCm5hbWVzKGdsbV9hZ3JlZW1lbnQpIDwtIHJvd25hbWVzKHRlc3RfZGYpCnN1bW1hcnkoZ2xtX2FncmVlbWVudCkKbmFtZXMoZ2xtX2FncmVlbWVudClbZ2xtX2FncmVlbWVudCA9PSBGQUxTRV0KYGBgCgojIyBHcmFkaWVudCBCb29zdGVyCgpgYGB7ciBnYl90ZXN0fQpsaWJyYXJ5KHhnYm9vc3QpCiMjcmZfbWV0aG9kIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAicmFuZ2VyIiwgbnVtYmVyID0gMTAsIHZlcmJvc2UgPSBUUlVFKQp0cmFpbl9tZXRob2QgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwKQoKZ2JfZml0IDwtIHRyYWluKG91dGNvbWVfZmFjdG9yIH4gLiwgZGF0YSA9IHRyYWluX2RmLAogICAgICAgICAgICAgICAgbWV0aG9kID0gInhnYlRyZWUiLCB0ckNvbnRyb2wgPSB0cmFpbl9tZXRob2QsCiAgICAgICAgICAgICAgICB0dW5lR3JpZCA9IGRhdGEuZnJhbWUoCiAgICAgICAgICAgICAgICAgICAgbnJvdW5kcyA9IDIwMCwKICAgICAgICAgICAgICAgICAgICBldGEgPSBjKDAuMDUsIDAuMSwgMC4zKSwKICAgICAgICAgICAgICAgICAgICBtYXhfZGVwdGggPSA0LAogICAgICAgICAgICAgICAgICAgIGdhbW1hID0gMCwKICAgICAgICAgICAgICAgICAgICBjb2xzYW1wbGVfYnl0cmVlID0gMSwKICAgICAgICAgICAgICAgICAgICBzdWJzYW1wbGUgPSAwLjUsCiAgICAgICAgICAgICAgICAgICAgbWluX2NoaWxkX3dlaWdodCA9IDEpLAogICAgICAgICAgICAgICAgdmVyYm9zZSA9IFRSVUUpCgpnYl9wcmVkaWN0X3RyYWluZWQgPC0gcHJlZGljdChnYl9maXQsIHRyYWluX2RmKQpuYW1lcyhnYl9wcmVkaWN0X3RyYWluZWQpIDwtIHJvd25hbWVzKHRyYWluX2RmKQpzdW1tYXJ5KGdiX3ByZWRpY3RfdHJhaW5lZCA9PSBwRGF0YSh0Y19ub3JtKVt0cmFpbl9pZHgsICJmaW5hbG91dGNvbWUiXSkKYGBgCgojIyMgTm93IHRoZSBHQiB0ZXN0ZXJzIQoKYGBge3IgZ2JfcHJlZGljdF90ZXN0fQpnYl9wcmVkaWN0X3Rlc3QgPC0gcHJlZGljdChnYl9maXQsIHRlc3RfZGYpCm5hbWVzKGdiX3ByZWRpY3RfdGVzdCkgPC0gcm93bmFtZXModGVzdF9kZikKZ2JfcHJlZGljdF90ZXN0CmdiX2FncmVlbWVudCA8LSBnYl9wcmVkaWN0X3Rlc3QgPT0gcERhdGEodGNfbm9ybSlbbm90X3RyYWluX2lkeCwgImZpbmFsb3V0Y29tZSJdCm5hbWVzKGdiX2FncmVlbWVudCkgPC0gcm93bmFtZXModGVzdF9kZikKc3VtbWFyeShnYl9hZ3JlZW1lbnQpCm5hbWVzKGdiX2FncmVlbWVudClbZ2JfYWdyZWVtZW50ID09IEZBTFNFXQpgYGAKCiMjIFNWTQoKYGBge3Igc3ZtX3Rlc3R9CmxpYnJhcnkoa2VybmxhYikKIyNyZl9tZXRob2QgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJyYW5nZXIiLCBudW1iZXIgPSAxMCwgdmVyYm9zZSA9IFRSVUUpCnRyYWluX21ldGhvZCA8LSB0cmFpbkNvbnRyb2wobWV0aG9kID0gImN2IiwgbnVtYmVyID0gMTApCgpzdm1fZml0IDwtIHRyYWluKG91dGNvbWVfZmFjdG9yIH4gLiwgZGF0YSA9IHRyYWluX2RmLAogICAgICAgICAgICAgICAgbWV0aG9kID0gInN2bVJhZGlhbCIsIHRyQ29udHJvbCA9IHRyYWluX21ldGhvZCwKICAgICAgICAgICAgICAgIHR1bmVHcmlkID0gZGF0YS5mcmFtZSgKICAgICAgICAgICAgICAgICAgICBDID0gYygwLjI1LCAwLjUsIDEpLAogICAgICAgICAgICAgICAgICAgIHNpZ21hID0gMSksCiAgICAgICAgICAgICAgICB2ZXJib3NlID0gVFJVRSkKCnN2bV9wcmVkaWN0X3RyYWluZWQgPC0gcHJlZGljdChzdm1fZml0LCB0cmFpbl9kZikKbmFtZXMoc3ZtX3ByZWRpY3RfdHJhaW5lZCkgPC0gcm93bmFtZXModHJhaW5fZGYpCnN2bV9wcmVkaWN0X3RyYWluZWQKc3VtbWFyeShzdm1fcHJlZGljdF90cmFpbmVkID09IHBEYXRhKHRjX25vcm0pW3RyYWluX2lkeCwgImZpbmFsb3V0Y29tZSJdKQpgYGAKCiMjIyBOb3cgdGhlIFNWTSB0ZXN0ZXJzIQoKYGBge3Igc3ZtX3ByZWRpY3RfdGVzdH0Kc3ZtX3ByZWRpY3RfdGVzdCA8LSBwcmVkaWN0KHN2bV9maXQsIHRlc3RfZGYpCm5hbWVzKHN2bV9wcmVkaWN0X3Rlc3QpIDwtIHJvd25hbWVzKHRlc3RfZGYpCnN2bV9wcmVkaWN0X3Rlc3QKc3ZtX2FncmVlbWVudCA8LSBzdm1fcHJlZGljdF90ZXN0ID09IHBEYXRhKHRjX25vcm0pW25vdF90cmFpbl9pZHgsICJmaW5hbG91dGNvbWUiXQpuYW1lcyhzdm1fYWdyZWVtZW50KSA8LSByb3duYW1lcyh0ZXN0X2RmKQpzdW1tYXJ5KHN2bV9hZ3JlZW1lbnQpCm5hbWVzKHN2bV9hZ3JlZW1lbnQpW3N2bV9hZ3JlZW1lbnQgPT0gRkFMU0VdCmBgYAoKIyBSZWFwcGx5IHRvIHBlcnNpc3RlbmNlCgpgYGB7ciBwZXJzaXN0ZW5jZX0KcmVmX2NvbCA8LSAicGVyc2lzdGVuY2UiCm91dGNvbWVfZmFjdG9yIDwtIGFzLmZhY3Rvcihhcy5jaGFyYWN0ZXIocERhdGEodGNfY2xpbmljYWwpW1tyZWZfY29sXV0pKQoKbWxfZGYgPC0gYXMuZGF0YS5mcmFtZShjYmluZChvdXRjb21lX2ZhY3RvciwgYXMuZGF0YS5mcmFtZShuenZfdW5jb3JyKSkpCm1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0gPC0gYXMuZmFjdG9yKG1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0pCm9ubHlfeW4gPC0gbWxfZGZbWyJvdXRjb21lX2ZhY3RvciJdXSA9PSAiWSIgfCBtbF9kZltbIm91dGNvbWVfZmFjdG9yIl1dID09ICJOIgpvdXRjb21lX2ZhY3RvciA8LSBhcy5mYWN0b3IoYXMuY2hhcmFjdGVyKG91dGNvbWVfZmFjdG9yW29ubHlfeW5dKSkKCgptbF9kZiA8LSBtbF9kZltvbmx5X3luLCBdCm1sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0gPC0gYXMuZmFjdG9yKGFzLmNoYXJhY3RlcihtbF9kZltbIm91dGNvbWVfZmFjdG9yIl1dKSkKc3VtbWFyeShtbF9kZltbIm91dGNvbWVfZmFjdG9yIl1dKQoKdHJhaW5faWR4IDwtIGNyZWF0ZURhdGFQYXJ0aXRpb24ob3V0Y29tZV9mYWN0b3IsIHAgPSAwLjUsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxpc3QgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGltZXMgPSAxKQpzdW1tYXJ5KG1sX2RmW3RyYWluX2lkeCwgIm91dGNvbWVfZmFjdG9yIl0pCiMjIFRyYWluaW5nIHNldCBoYXMgMTEyIHNhbXBsZXMuCgp0cmFpbl9yb3duYW1lcyA8LSByb3duYW1lcyhtbF9kZilbdHJhaW5faWR4XQpub3RfdHJhaW5faWR4IDwtICEgcm93bmFtZXMobWxfZGYpICVpbiUgdHJhaW5fcm93bmFtZXMKc3VtbWFyeShtbF9kZltub3RfdHJhaW5faWR4LCAib3V0Y29tZV9mYWN0b3IiXSkKbm90X3RyYWluX3Jvd25hbWVzIDwtIHJvd25hbWVzKG1sX2RmKVtub3RfdHJhaW5faWR4XQoKdHJhaW5fZGYgPC0gYXMuZGF0YS5mcmFtZShtbF9kZlt0cmFpbl9yb3duYW1lcywgXSkKZGltKHRyYWluX2RmKQp0ZXN0X2RmIDwtIGFzLmRhdGEuZnJhbWUobWxfZGZbbm90X3RyYWluX3Jvd25hbWVzLCBdKQpkaW0odGVzdF9kZikKIyMgUmVtb3ZlIHRoZSBvdXRjb21lIGZhY3RvciBmcm9tIHRoZSB0ZXN0IGRhdGEsIGp1c3QgaW4gY2FzZS4KdGVzdF9vdXRjb21lIDwtIHRlc3RfZGZbWyJvdXRjb21lX2ZhY3RvciJdXQp0ZXN0X2RmW1sib3V0Y29tZV9mYWN0b3IiXV0gPC0gTlVMTAoKdHJhaW5fZml0IDwtIHRyYWluKG91dGNvbWVfZmFjdG9yIH4gLiwgZGF0YSA9IHRyYWluX2RmLAogICAgICAgICAgICAgICAgbWV0aG9kID0gInJhbmdlciIsIHRyQ29udHJvbCA9IGJvb3RfY29udHJvbCwKICAgICAgICAgICAgICAgIGltcG9ydGFuY2UgPSAicGVybXV0YXRpb24iLAogICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKAogICAgICAgICAgICAgICAgICAgIG10cnkgPSAyMDAsCiAgICAgICAgICAgICAgICAgICAgbWluLm5vZGUuc2l6ZSA9IDEsCiAgICAgICAgICAgICAgICAgICAgc3BsaXRydWxlID0gImdpbmkiKSwKICAgICAgICAgICAgICAgIHZlcmJvc2UgPSBUUlVFKQp0cmFpbl9maXRbWyJmaW5hbE1vZGVsIl1dW1sicHJlZGljdGlvbi5lcnJvciJdXQp2YXJpYWJsZV9pbXBvcnRhbmNlIDwtIHZhckltcCh0cmFpbl9maXQpCnBsb3QodmFyaWFibGVfaW1wb3J0YW5jZSwgdG9wID0gMTUpCgpyZl9wcmVkaWN0X3RyYWluZWQgPC0gcHJlZGljdChyZl90cmFpbl9maXQsIHRyYWluX2RmKQpjb25mdXNpb25NYXRyaXgoZGF0YSA9IHRyYWluX2RmW1sxXV0sIHJlZmVyZW5jZSA9IHJmX3ByZWRpY3RfdHJhaW5lZCkKCnJmX3ByZWRpY3RfdGVzdCA8LSBwcmVkaWN0KHJmX3RyYWluX2ZpdCwgdGVzdF9kZikKbmFtZXMocmZfcHJlZGljdF90ZXN0KSA8LSByb3duYW1lcyh0ZXN0X2RmKQp1c2VkX25hbWVzIDwtIG5hbWVzKHJmX3ByZWRpY3RfdGVzdCkKdXNlZF9wZGF0YSA8LSBwRGF0YSh0Y19ub3JtKVt1c2VkX25hbWVzLCBdCgpyZl9hZ3JlZW1lbnQgPC0gYXMuY2hhcmFjdGVyKHJmX3ByZWRpY3RfdGVzdCkgPT0gdXNlZF9wZGF0YVtbcmVmX2NvbF1dCm5hbWVzKHJmX2FncmVlbWVudCkgPC0gcm93bmFtZXModGVzdF9kZikKc3VtbWFyeShyZl9hZ3JlZW1lbnQpCm5hbWVzKHJmX2FncmVlbWVudClbcmZfYWdyZWVtZW50ID09IEZBTFNFXQpgYGAKCiMgVHJ5IHNvbWV0aGluZyBhIGxpdHRsZSBkaWZmZXJlbnQKCkkgcmVkaWQgdGhlIG1hcHBpbmdzIHdpdGggYSBjb21iaW5lZCBob3N0L3BhcmFzaXRlIHRyYW5zY3JpcHRvbWUgaW4KdGhlIGhvcGVzIHRoYXQgaXQgbWlnaHQgcHJvdmlkZSBzb21lIGludGVyZXN0aW5nIHZhcmlhbmNlIGZvciB0aGVzZQpjbGFzc2lmaWVycy4gIExldHMgdXMgcG9rZSBhdCBpdCBhbmQgc2VlIGlmIGFueXRoaW5nIGlzIHJlbGV2YW50IG9yIGlmCml0IHdhcyBhIGJhZCBpZGVhLgoKYGBge3IgcG9rZV9jb21iaW5lZH0Kd2FudGVkX2lkeCA8LSBncmVwbCh4ID0gcm93bmFtZXMoZXhwcnMoaHNscF9nZW5lX2V4cHQpKSwgcGF0dGVybiA9ICJeTFAiKQp3YW50ZWRfaWRzIDwtIHJvd25hbWVzKGV4cHJzKGhzbHBfZ2VuZV9leHB0KSlbd2FudGVkX2lkeF0KdGVzdF9scCA8LSBleGNsdWRlX2dlbmVzX2V4cHQoaHNscF9nZW5lX2V4cHQsIGlkcyA9IHdhbnRlZF9pZHMsIG1ldGhvZCA9ICJrZWVwIikKCnR0IDwtIHBsb3RfbGlic2l6ZSh0ZXN0X2xwKQp0dCRwbG90CmBgYAoKSSBhbSBnb2luZyB0byBtb3N0bHkgY29weSBwYXN0ZSB0aGUgY29kZSBmcm9tIHVwIGFib3ZlIGhlcmUgYnV0IGNoYW5nZQp0aGUgaW5wdXRzIHRvIGZpdCBteSBuZXcgY29tYmluZWQgZGF0YSBzdHJ1Y3R1cmUgb2YgaHVtYW4gYW5kIHBhcmFzaXRlIGRhdGEuCgpgYGB7ciBjbGFzc2lmeV9ocGNmfQpyZWZfY29sIDwtICJmaW5hbG91dGNvbWUiCm91dGNvbWVfZmFjdG9yIDwtIGFzLmZhY3Rvcihhcy5jaGFyYWN0ZXIocERhdGEoaHNscF9nZW5lX2V4cHQpW1tyZWZfY29sXV0pKQoKaHNscF9ub3JtIDwtIG5vcm1hbGl6ZV9leHB0KGhzbHBfZ2VuZV9leHB0LCB0cmFuc2Zvcm0gPSAibG9nMiIsIGNvbnZlcnQgPSAiY3BtIikKaHNscF9ub3JtIDwtIG5vcm1hbGl6ZV9leHB0KGhzbHBfbm9ybSwgZmlsdGVyID0gImN2IiwgY3ZfbWluID0gMC4xKQpyZXRhaW5lZF9scCA8LSBncmVwbCh4ID0gcm93bmFtZXMoZXhwcnMoaHNscF9ub3JtKSksIHBhdHRlcm4gPSAiXkxQIikKc3VtKHJldGFpbmVkX2xwKQoKbnp2X3Roc2xwIDwtIHQoZXhwcnMoaHNscF9ub3JtKSkKbnp2X2hzbHBfY2VudGVyIDwtIHByZVByb2Nlc3Mobnp2X3Roc2xwLCBtZXRob2QgPSAiY2VudGVyIikKbnp2X3Roc2xwIDwtIHByZWRpY3Qobnp2X2hzbHBfY2VudGVyLCBuenZfdGhzbHApCgpuenZfdGhzbHBfY29ycmVsYXRlZCA8LSBwcmVQcm9jZXNzKG56dl90aHNscCwgbWV0aG9kID0gImNvcnIiLCBjdXRvZmYgPSAwLjk1KQpuenZfdGhzbHBfdW5jb3JyIDwtIHByZWRpY3Qobnp2X3Roc2xwX2NvcnJlbGF0ZWQsIG56dl90aHNscCkKCmludGVyZXN0aW5nX21ldGEgPC0gcERhdGEoaHNscF9ub3JtKVssIGMoImZpbmFsb3V0Y29tZSIsICJkb25vciIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInZpc2l0bnVtYmVyIiwgInNlbGVjdGlvbm1ldGhvZCIsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgInR5cGVvZmNlbGxzIiwgInRpbWUiLCAiY2xpbmljIildCgpoc2xwX21sX2RmIDwtIGFzLmRhdGEuZnJhbWUoY2JpbmQob3V0Y29tZV9mYWN0b3IsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICBhcy5kYXRhLmZyYW1lKG56dl90aHNscF91bmNvcnIpKSkKaHNscF9tbF9kZltbIm91dGNvbWVfZmFjdG9yIl1dIDwtIGFzLmZhY3Rvcihoc2xwX21sX2RmW1sib3V0Y29tZV9mYWN0b3IiXV0pCmRpbShoc2xwX21sX2RmKQpgYGAKCiMgU3BsaXQgdGhlIGRhdGEgZm9yIHRyYWluaW5nL3Rlc3RpbmcKCmBgYHtyIHNwbGl0X3RyYWluX3Rlc3R9CiMjIFRoZSB2YXJpYWJsZSBvdXRjb21lX2ZhY3RvciB3YXMgY3JlYXRlZCBhdCB0aGUgdG9wIG9mIHRoaXMgZG9jdW1lbnQsCiMjIHdoaWNoIGlzIGEgbGl0dGxlIGF3a3dhcmQgaGVyZS4KdHJhaW5faWR4IDwtIGNyZWF0ZURhdGFQYXJ0aXRpb24ob3V0Y29tZV9mYWN0b3IsIHAgPSAwLjYsCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIGxpc3QgPSBGQUxTRSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgdGltZXMgPSAxKQpzdW1tYXJ5KGhzbHBfbWxfZGZbdHJhaW5faWR4LCAib3V0Y29tZV9mYWN0b3IiXSkKIyMgVHJhaW5pbmcgc2V0IGhhcyAxMTIgc2FtcGxlcy4KCnRyYWluX3Jvd25hbWVzIDwtIHJvd25hbWVzKGhzbHBfbWxfZGYpW3RyYWluX2lkeF0Kbm90X3RyYWluX2lkeCA8LSAhIHJvd25hbWVzKGhzbHBfbWxfZGYpICVpbiUgdHJhaW5fcm93bmFtZXMKc3VtbWFyeShoc2xwX21sX2RmW25vdF90cmFpbl9pZHgsICJvdXRjb21lX2ZhY3RvciJdKQpub3RfdHJhaW5fcm93bmFtZXMgPC0gcm93bmFtZXMoaHNscF9tbF9kZilbbm90X3RyYWluX2lkeF0KCnRyYWluX2RmIDwtIGFzLmRhdGEuZnJhbWUoaHNscF9tbF9kZlt0cmFpbl9yb3duYW1lcywgXSkKZGltKHRyYWluX2RmKQp0ZXN0X2RmIDwtIGFzLmRhdGEuZnJhbWUoaHNscF9tbF9kZltub3RfdHJhaW5fcm93bmFtZXMsIF0pCmRpbSh0ZXN0X2RmKQojIyBSZW1vdmUgdGhlIG91dGNvbWUgZmFjdG9yIGZyb20gdGhlIHRlc3QgZGF0YSwganVzdCBpbiBjYXNlLgp0ZXN0X291dGNvbWUgPC0gdGVzdF9kZltbIm91dGNvbWVfZmFjdG9yIl1dCnRlc3RfZGZbWyJvdXRjb21lX2ZhY3RvciJdXSA8LSBOVUxMCmBgYAoKYGBge3IgcmFuZG9tX2ZvcmVzdH0KYm9vdF9jb250cm9sIDwtIHRyYWluQ29udHJvbChtZXRob2QgPSAiYm9vdCIsIG51bWJlciA9IDIwLAogICAgICAgICAgICAgICAgICAgICAgICAgICAgIHJldHVyblJlc2FtcCA9ICJhbGwiKQoKa25uX2ZpdCA8LSBrbm4zKHggPSB0cmFpbl9kZlssIC0xXSwKICAgICAgICAgICAgICAgIHkgPSB0cmFpbl9kZltbIm91dGNvbWVfZmFjdG9yIl1dLAogICAgICAgICAgICAgICAgayA9IDMpCmtubl9wcmVkaWN0X3RyYWluZWQgPC0gcHJlZGljdChrbm5fZml0LCB0cmFpbl9kZlssIC0xXSwgdHlwZSA9ICJjbGFzcyIpCmNvbmZ1c2lvbk1hdHJpeChkYXRhID0gdHJhaW5fZGZbWzFdXSwgcmVmZXJlbmNlID0ga25uX3ByZWRpY3RfdHJhaW5lZCkKCm5hbWVzKGtubl9wcmVkaWN0X3RyYWluZWQpIDwtIHJvd25hbWVzKHRyYWluX2RmKQpzdW1tYXJ5KGtubl9wcmVkaWN0X3RyYWluZWQgPT0gcERhdGEoaHNscF9nZW5lX2V4cHQpW3RyYWluX2lkeCwgImZpbmFsb3V0Y29tZSJdKQoKa25uX3ByZWRpY3RfdGVzdCA8LSBwcmVkaWN0KGtubl9maXQsIHRlc3RfZGYsIHR5cGUgPSAiY2xhc3MiKQojIyBhbmQgdGhlIHZhbHVlcywgd2hpY2ggd2lsbCBiZSB1c2VkIGZvciBvdXIgUk9DIGN1cnZlLgprbm5fcHJlZGljdF90ZXN0X3ZhbHVlcyA8LSBwcmVkaWN0KGtubl9maXQsIHRlc3RfZGYpCgpuYW1lcyhrbm5fcHJlZGljdF90ZXN0KSA8LSByb3duYW1lcyh0ZXN0X2RmKQprbm5fYWdyZWVtZW50IDwtIGtubl9wcmVkaWN0X3Rlc3QgPT0gdGVzdF9vdXRjb21lCm5hbWVzKGtubl9hZ3JlZW1lbnQpIDwtIHJvd25hbWVzKHRlc3RfZGYpCnN1bW1hcnkoa25uX2FncmVlbWVudCkKbmFtZXMoa25uX2FncmVlbWVudClba25uX2FncmVlbWVudCA9PSBGQUxTRV0KCmN2X2NvbnRyb2wgPC0gdHJhaW5Db250cm9sKG1ldGhvZCA9ICJjdiIsIG51bWJlciA9IDEwKQprbm5fdHJhaW5fZml0IDwtIHRyYWluKG91dGNvbWVfZmFjdG9yIH4gLiwgZGF0YSA9IHRyYWluX2RmLAogICAgICAgICAgICAgICAgICAgICAgIG1ldGhvZCA9ICJrbm4iLAogICAgICAgICAgICAgICAgICAgICAgIHRyQ29udHJvbCA9IGN2X2NvbnRyb2wsCiAgICAgICAgICAgICAgICAgICAgICAgdHVuZUdyaWQgPSBkYXRhLmZyYW1lKGsgPSAxOjEwKSkKa25uX3RyYWluX2ZpdFtbImJlc3RUdW5lIl1dCgpwbG90KHggPSAxOjEwLCAxIC0ga25uX3RyYWluX2ZpdCRyZXN1bHRzWywgMl0sIHBjaCA9IDE5LAogICAgIHlsYWIgPSAicHJlZGljdGlvbiBlcnJvciIsIHhsYWIgPSAiayIpCmxpbmVzKGxvZXNzLnNtb290aCh4ID0gMToxMCwgMSAtIGtubl90cmFpbl9maXQkcmVzdWx0c1ssIDJdLGRlZ3JlZSA9IDIpLAogICAgICBjb2wgPSAiI0NDMDAwMCIpCgpgYGAK