项目总述
主要任务是从之前同项目的组员建的关系型数据库里提取出我们需要的GitHub的数据，并把结果保存到文件，以便之后插入到数据库。
从已经建立好的关系型数据库上多线程地读取GitHub的项目信息。主要信息包括项目的名称，用户名，被Star的数量，是否被fork，以及该项目用到的编程语言。
在统计之前的neo4j数据库长这样，其中紫色的是项目信息，蓝色的是用户信息，灰色的是commit信息，这些大量的信息中我们需要提取出我们需要的数据:
结果呈现
实验室的分项目，运行需要内网。最后的结果保存在ssh [email protected]:usr/zy里的30个out点txt文件。
最后的输出文件大致如下:
插入到数据库之后的数据(该数据放在192.168.7.118的mysql上)
一些注意点
user和starNum这两个信息不是project里所属的字段，而是通过有边连接来查询相应的数据。例如：
MATCH (m:User)-[r:Star]->(n:Project) where n.projectId='1' RETURN count(r) as fx
project里面有的字段信息如图:
关于多线程
对于起多少线程比较合适的问题，看到网上一般的回答是：
查看一下服务器的cpu嘻嘻
看到16个cpu(包括8个物理8个逻辑)，于是34个线程走起。实测好像还好，带宽还行，基本没有出现被几个线程独占资源的情况。
踩过的一些坑
neo4j的连接的关闭，网上的代码说的都是session.close(); driver.close(); 我一开始也是按照这么来的，但最初我是查询一条连接一次，就发现始终会报Too Many Open Files的异常，也就是打开了太多文件，文件句柄超过了服务器的限制（一般Linux机器的限制是1024，可以通过ulimt -a来查看当前系统最大的文件打开数）这样的问题我通过一步步排查发现每次连接数据库会打开两个文件，但关闭只会关闭一个就导致打开的文件越堆越多。
后来的解决方法是，只开一个连接，一直不关。因为同一类型的查询用的是同一个数据库，因此我们可以一直连着这个连接，直至用完再关闭（好像也可以用连接池来解决这个问题）这样直接在构造函数连接就行了。
以及保存的时候注意一下服务器的空间是不是够存。有的时候代码卡在那儿只是因为服务器放不下了owo.
*:first-child {
margin-top: 0 !important;
}
.markdown-body>*:last-child {
margin-bottom: 0 !important;
}
.markdown-body .anchor {
position: absolute;
top: 0;
bottom: 0;
left: 0;
display: block;
padding-right: 6px;
padding-left: 30px;
margin-left: -30px;
}
.markdown-body .anchor:focus {
outline: none;
}
.markdown-body h1,
.markdown-body h2,
.markdown-body h3,
.markdown-body h4,
.markdown-body h5,
.markdown-body h6 {
position: relative;
margin-top: 1em;
margin-bottom: 16px;
font-weight: bold;
line-height: 1.4;
}
.markdown-body h1 .octicon-link,
.markdown-body h2 .octicon-link,
.markdown-body h3 .octicon-link,
.markdown-body h4 .octicon-link,
.markdown-body h5 .octicon-link,
.markdown-body h6 .octicon-link {
display: none;
color: #000;
vertical-align: middle;
}
.markdown-body h1:hover .anchor,
.markdown-body h2:hover .anchor,
.markdown-body h3:hover .anchor,
.markdown-body h4:hover .anchor,
.markdown-body h5:hover .anchor,
.markdown-body h6:hover .anchor {
height: 1em;
padding-left: 8px;
margin-left: -30px;
line-height: 1;
text-decoration: none;
}
.markdown-body h1:hover .anchor .octicon-link,
.markdown-body h2:hover .anchor .octicon-link,
.markdown-body h3:hover .anchor .octicon-link,
.markdown-body h4:hover .anchor .octicon-link,
.markdown-body h5:hover .anchor .octicon-link,
.markdown-body h6:hover .anchor .octicon-link {
display: inline-block;
}
.markdown-body h1 {
padding-bottom: 0.3em;
font-size: 2.25em;
line-height: 1.2;
border-bottom: 1px solid #eee;
}
.markdown-body h2 {
padding-bottom: 0.3em;
font-size: 1.75em;
line-height: 1.225;
border-bottom: 1px solid #eee;
}
.markdown-body h3 {
font-size: 1.5em;
line-height: 1.43;
}
.markdown-body h4 {
font-size: 1.25em;
}
.markdown-body h5 {
font-size: 1em;
}
.markdown-body h6 {
font-size: 1em;
color: #777;
}
.markdown-body p,
.markdown-body blockquote,
.markdown-body ul,
.markdown-body ol,
.markdown-body dl,
.markdown-body table,
.markdown-body pre {
margin-top: 0;
margin-bottom: 16px;
}
.markdown-body hr {
height: 4px;
padding: 0;
margin: 16px 0;
background-color: #e7e7e7;
border: 0 none;
}
.markdown-body ul,
.markdown-body ol {
padding-left: 2em;
}
.markdown-body ul ul,
.markdown-body ul ol,
.markdown-body ol ol,
.markdown-body ol ul {
margin-top: 0;
margin-bottom: 0;
}
.markdown-body li>p {
margin-top: 16px;
}
.markdown-body dl {
padding: 0;
}
.markdown-body dl dt {
padding: 0;
margin-top: 16px;
font-size: 1em;
font-style: italic;
font-weight: bold;
}
.markdown-body dl dd {
padding: 0 16px;
margin-bottom: 16px;
}
.markdown-body blockquote {
padding: 0 15px;
color: #777;
border-left: 4px solid #ddd;
}
.markdown-body blockquote>:first-child {
margin-top: 0;
}
.markdown-body blockquote>:last-child {
margin-bottom: 0;
}
.markdown-body table {
display: block;
width: 100%;
overflow: auto;
word-break: normal;
word-break: keep-all;
}
.markdown-body table th {
font-weight: bold;
}
.markdown-body table th,
.markdown-body table td {
padding: 6px 13px;
border: 1px solid #ddd;
}
.markdown-body table tr {
background-color: #fff;
border-top: 1px solid #ccc;
}
.markdown-body table tr:nth-child(2n) {
background-color: #f8f8f8;
}
.markdown-body img {
max-width: 100%;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
.markdown-body code {
padding: 0;
padding-top: 0.2em;
padding-bottom: 0.2em;
margin: 0;
font-size: 85%;
background-color: rgba(0,0,0,0.04);
border-radius: 3px;
}
.markdown-body code:before,
.markdown-body code:after {
letter-spacing: -0.2em;
content: "\00a0";
}
.markdown-body pre>code {
padding: 0;
margin: 0;
font-size: 100%;
word-break: normal;
white-space: pre;
background: transparent;
border: 0;
}
.markdown-body .highlight {
margin-bottom: 16px;
}
.markdown-body .highlight pre,
.markdown-body pre {
padding: 16px;
overflow: auto;
font-size: 85%;
line-height: 1.45;
background-color: #f7f7f7;
border-radius: 3px;
}
.markdown-body .highlight pre {
margin-bottom: 0;
word-break: normal;
}
.markdown-body pre {
word-wrap: normal;
}
.markdown-body pre code {
display: inline;
max-width: initial;
padding: 0;
margin: 0;
overflow: initial;
line-height: inherit;
word-wrap: normal;
background-color: transparent;
border: 0;
}
.markdown-body pre code:before,
.markdown-body pre code:after {
content: normal;
}
.markdown-body .highlight {
background: #fff;
}
.markdown-body .highlight .mf,
.markdown-body .highlight .mh,
.markdown-body .highlight .mi,
.markdown-body .highlight .mo,
.markdown-body .highlight .il,
.markdown-body .highlight .m {
color: #945277;
}
.markdown-body .highlight .s,
.markdown-body .highlight .sb,
.markdown-body .highlight .sc,
.markdown-body .highlight .sd,
.markdown-body .highlight .s2,
.markdown-body .highlight .se,
.markdown-body .highlight .sh,
.markdown-body .highlight .si,
.markdown-body .highlight .sx,
.markdown-body .highlight .s1 {
color: #df5000;
}
.markdown-body .highlight .kc,
.markdown-body .highlight .kd,
.markdown-body .highlight .kn,
.markdown-body .highlight .kp,
.markdown-body .highlight .kr,
.markdown-body .highlight .kt,
.markdown-body .highlight .k,
.markdown-body .highlight .o {
font-weight: bold;
}
.markdown-body .highlight .kt {
color: #458;
}
.markdown-body .highlight .c,
.markdown-body .highlight .cm,
.markdown-body .highlight .c1 {
color: #998;
font-style: italic;
}
.markdown-body .highlight .cp,
.markdown-body .highlight .cs {
color: #999;
font-weight: bold;
}
.markdown-body .highlight .cs {
font-style: italic;
}
.markdown-body .highlight .n {
color: #333;
}
.markdown-body .highlight .na,
.markdown-body .highlight .nv,
.markdown-body .highlight .vc,
.markdown-body .highlight .vg,
.markdown-body .highlight .vi {
color: #008080;
}
.markdown-body .highlight .nb {
color: #0086B3;
}
.markdown-body .highlight .nc {
color: #458;
font-weight: bold;
}
.markdown-body .highlight .no {
color: #094e99;
}
.markdown-body .highlight .ni {
color: #800080;
}
.markdown-body .highlight .ne {
color: #990000;
font-weight: bold;
}
.markdown-body .highlight .nf {
color: #945277;
font-weight: bold;
}
.markdown-body .highlight .nn {
color: #555;
}
.markdown-body .highlight .nt {
color: #000080;
}
.markdown-body .highlight .err {
color: #a61717;
background-color: #e3d2d2;
}
.markdown-body .highlight .gd {
color: #000;
background-color: #fdd;
}
.markdown-body .highlight .gd .x {
color: #000;
background-color: #faa;
}
.markdown-body .highlight .ge {
font-style: italic;
}
.markdown-body .highlight .gr {
color: #aa0000;
}
.markdown-body .highlight .gh {
color: #999;
}
.markdown-body .highlight .gi {
color: #000;
background-color: #dfd;
}
.markdown-body .highlight .gi .x {
color: #000;
background-color: #afa;
}
.markdown-body .highlight .go {
color: #888;
}
.markdown-body .highlight .gp {
color: #555;
}
.markdown-body .highlight .gs {
font-weight: bold;
}
.markdown-body .highlight .gu {
color: #800080;
font-weight: bold;
}
.markdown-body .highlight .gt {
color: #aa0000;
}
.markdown-body .highlight .ow {
font-weight: bold;
}
.markdown-body .highlight .w {
color: #bbb;
}
.markdown-body .highlight .sr {
color: #017936;
}
.markdown-body .highlight .ss {
color: #8b467f;
}
.markdown-body .highlight .bp {
color: #999;
}
.markdown-body .highlight .gc {
color: #999;
background-color: #EAF2F5;
}
.markdown-body .octicon {
font: normal normal 16px octicons-anchor;
line-height: 1;
display: inline-block;
text-decoration: none;
-webkit-font-smoothing: antialiased;
-moz-osx-font-smoothing: grayscale;
-webkit-user-select: none;
-moz-user-select: none;
-ms-user-select: none;
user-select: none;
}
.markdown-body .octicon-link:before {
content: '\f05c';
}
.markdown-body .task-list-item {
list-style-type: none;
}
.markdown-body .task-list-item+.task-list-item {
margin-top: 3px;
}
.markdown-body .task-list-item input {
float: left;
margin: 0.3em 0 0.25em -1.6em;
vertical-align: middle;
}
/*
github.com style (c) Vasily Polovnyov
*/
.hljs {
display: block;
overflow-x: auto;
padding: 0.5em;
color: #333;
background: #f8f8f8;
-webkit-text-size-adjust: none;
}
.hljs-comment,
.diff .hljs-header {
color: #998;
font-style: italic;
}
.hljs-keyword,
.css .rule .hljs-keyword,
.hljs-winutils,
.nginx .hljs-title,
.hljs-subst,
.hljs-request,
.hljs-status {
color: #333;
font-weight: bold;
}
.hljs-number,
.hljs-hexcolor,
.ruby .hljs-constant {
color: #008080;
}
.hljs-string,
.hljs-tag .hljs-value,
.hljs-doctag,
.tex .hljs-formula {
color: #d14;
}
.hljs-title,
.hljs-id,
.scss .hljs-preprocessor {
color: #900;
font-weight: bold;
}
.hljs-list .hljs-keyword,
.hljs-subst {
font-weight: normal;
}
.hljs-class .hljs-title,
.hljs-type,
.vhdl .hljs-literal,
.tex .hljs-command {
color: #458;
font-weight: bold;
}
.hljs-tag,
.hljs-tag .hljs-title,
.hljs-rule .hljs-property,
.django .hljs-tag .hljs-keyword {
color: #000080;
font-weight: normal;
}
.hljs-attribute,
.hljs-variable,
.lisp .hljs-body,
.hljs-name {
color: #008080;
}
.hljs-regexp {
color: #009926;
}
.hljs-symbol,
.ruby .hljs-symbol .hljs-string,
.lisp .hljs-keyword,
.clojure .hljs-keyword,
.scheme .hljs-keyword,
.tex .hljs-special,
.hljs-prompt {
color: #990073;
}
.hljs-built_in {
color: #0086b3;
}
.hljs-preprocessor,
.hljs-pragma,
.hljs-pi,
.hljs-doctype,
.hljs-shebang,
.hljs-cdata {
color: #999;
font-weight: bold;
}
.hljs-deletion {
background: #fdd;
}
.hljs-addition {
background: #dfd;
}
.diff .hljs-change {
background: #0086b3;
}
.hljs-chunk {
color: #aaa;
}
-->