【Spark机器学习速成宝典】模型篇04朴素贝叶斯【Naive Bayes】（Python版）

目录

　　朴素贝叶斯原理

　　朴素贝叶斯代码(Spark Python)

---

朴素贝叶斯原理

　　详见博文：http://www.cnblogs.com/itmorn/p/7905975.html

返回目录

朴素贝叶斯代码(Spark Python)

　　

　　代码里数据：https://pan.baidu.com/s/1jHWKG4I 密码：acq1

# -*-coding=utf-8 -*-

from pyspark import SparkConf, SparkContext

sc = SparkContext('local')

from pyspark.mllib.classification import NaiveBayes, NaiveBayesModel

from pyspark.mllib.util import MLUtils

# Load and parse the data file.

data = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt")

'''

每一行使用以下格式表示一个标记的稀疏特征向量

label index1:value1 index2:value2 ...

tempFile.write(b"+1 1:1.0 3:2.0 5:3.0\\n-1\\n-1 2:4.0 4:5.0 6:6.0")

>>> tempFile.flush()

>>> examples = MLUtils.loadLibSVMFile(sc, tempFile.name).collect()

>>> tempFile.close()

>>> examples[0]

LabeledPoint(1.0, (6,[0,2,4],[1.0,2.0,3.0]))

>>> examples[1]

LabeledPoint(-1.0, (6,[],[]))

>>> examples[2]

LabeledPoint(-1.0, (6,[1,3,5],[4.0,5.0,6.0]))

'''

# Split data approximately into training (60%) and test (40%) 将数据集按照6:4的比例分成训练集和测试集

training, test = data.randomSplit([0.6, 0.4])

# Train a naive Bayes model. 训练朴素贝叶斯模型

model = NaiveBayes.train(training, 1.0)

# Make prediction and test accuracy. 预测和测试准确率

predictionAndLabel = test.map(lambda p: (model.predict(p.features), p.label))

accuracy = 1.0 * predictionAndLabel.filter(lambda pl: pl[0] == pl[1]).count() / test.count()

print('model accuracy {}'.format(accuracy)) #

# Save and load model 保存和加载模型

output_dir = 'myNaiveBayesModel'

model.save(sc, output_dir)

sameModel = NaiveBayesModel.load(sc, output_dir)

predictionAndLabel = test.map(lambda p: (sameModel.predict(p.features), p.label))

accuracy = 1.0 * predictionAndLabel.filter(lambda pl: pl[0] == pl[1]).count() / test.count()

print('sameModel accuracy {}'.format(accuracy)) #

返回目录