问题描述
我试图执行自动编码器以进行异常检测.我使用 H2O R 包使用 h2o.anomaly
函数为样本数据生成重建 MSE.但是,我也尝试根据以下文档链接中的 MSE 公式自己手动计算它:http://docs.h2o.ai/h2o/latest-stable/h2o-docs/performance-and-prediction.html#mse-mean-squared-error
I was trying to perform Autoencoder for anomaly detection. I used H2O R package to generate reconstruction MSE for a sample data using h2o.anomaly
function. However, I have also tried to manually calculate it by myself according the the MSE formula from the documentation link below:http://docs.h2o.ai/h2o/latest-stable/h2o-docs/performance-and-prediction.html#mse-mean-squared-error
我用来构建模型的由三个特征和5行组成的训练数据如下:
The training data consisting of three features and 5 rows that I used to build the model is as below:
head(train_dat)
Feature1 Feature2 Feature3
1 68.18 0.1806535 3.871201
2 71.51 0.3987761 2.484907
3 67.77 0.4285304 3.332205
4 69.58 0.1823216 2.890372
5 70.98 0.4134333 1.791759
我用于预测的由三个特征和5行组成的测试数据如下:
The test data consisting of three features and 5 rows that I used for prediction is as below:
head(test_dat)
Feature1 Feature2 Feature3
1 68.33000 0.4350239 2.708050
2 73.98000 0.5550339 3.044522
3 67.11000 0.7323679 2.639057
4 69.90395 0.9999787 4.499810
5 71.28867 0.4882539 3.091042
运行训练和预测后,重建的特征如下:
After running training and prediction, the reconstructed features are as below:
head(mod.out)
reconstr_Feature1 reconstr_Feature2 reconstr_Feature3
1 69.66297 0.4239244 2.346250
2 69.88329 0.3963843 2.381598
3 69.46544 0.4610502 2.233164
4 68.96117 0.4229165 2.676295
5 69.63208 0.3895452 2.530025
当我使用 h2o.anomaly
函数进行 MSE 计算时,我收到如下 MSE 输出:
When I used the h2o.anomaly
function for MSE calculation, I received MSE output as below:
head(mse.list)
Reconstruction.MSE
1 0.05310159
2 0.57037600
3 0.54427385
4 2.08407248
5 0.14251951
然而,当我尝试通过应用下面的函数来计算 MSE 时,我得到了不同的 MSE 输出:
However, when I tried to calculate the MSE by applying the function below, I obtained different MSE output:
mod.anon.validate <- apply((test_dat - mod.out)^2, 1, mean)
mse.list.validate <- as.data.frame(mod.anon.validate)
head(mse.list.validate)
mod.anon.validate
1 0.6359438
2 5.7492281
3 1.9288268
4 1.5156829
5 1.0229217
我想知道我在手动 MSE 计算中做错了什么?当它被称为重建MSE"时,它与一般的MSE有什么不同?完整的 R 脚本如下:
I was wondering what I have done wrong in my manual MSE calculation? When it is called "Reconstruction MSE", is it different from the general MSE? The full R script is as below:
### H2O Autoencoder test run ###
#Load test and training data.
test_dat <- read.table("sample.test.dat", header=TRUE)
train_dat <- read.table("sample.train.dat", header=TRUE)
#Start H2O
library(h2o)
localH2O <- h2o.init(port =54321)
#Training and deep learning
feature_names <- names(train_dat[1:3])
unmod.hex <- as.h2o(train_dat, destination_frame="train.hex") ; mod.hex=as.h2o(test_dat, destination_frame="test.hex")
unmod.dl <- h2o.deeplearning(x=feature_names,
training_frame=unmod.hex,
autoencoder = TRUE,
reproducible = T,
hidden = c(3,2,3), epochs = 50,
activation = "Tanh")
#Output result
mod.out <- as.data.frame(h2o.predict(unmod.dl,mod.hex,type=response))
mod.anon <- h2o.anomaly(unmod.dl, mod.hex, per_feature=FALSE)
mse.list <- as.data.frame(mod.anon)
mod.anon.validate <- apply((test_dat - mod.out)^2, 1, mean)
mse.list.validate <- as.data.frame(mod.anon.validate)
感谢您的帮助.
推荐答案
计算不匹配,因为 MSE 是在规范化空间中计算的.如果您在 h2o.deeplearning()
中设置 standardize=FALSE
参数,它将匹配:
The calculations don't match because MSE is calculated in the normalised space. If you set standardize=FALSE
param in h2o.deeplearning()
it will match:
unmod.dl <- h2o.deeplearning(x=feature_names, standardize = FALSE,
training_frame=unmod.hex,
autoencoder = TRUE,
reproducible = T,
hidden = c(3,2,3), epochs = 50,
activation = "Tanh")
mod.out <- as.data.frame(h2o.predict(unmod.dl, mod.hex, type=response))
mod.anon <- h2o.anomaly(unmod.dl, mod.hex, per_feature=FALSE)
mse.list <- as.data.frame(mod.anon)
mse.list
> mse.list
Reconstruction.MSE
1 1512.740
2 1777.491
3 1458.438
4 1587.593
5 1648.999
> mod.anon.validate <- apply((test_dat - mod.out)^2, 1, mean)
> mse.list.validate <- as.data.frame(mod.anon.validate)
> mse.list.validate
mod.anon.validate
1 1512.740
2 1777.491
3 1458.438
4 1587.593
5 1648.999
这篇关于使用 H2O R 包中的 h2o.anomaly 函数重建 MSE 计算的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持!