GoldenGate实时投递数据到大数据平台（1）-MongoDB

mongodb安装

安装

linux下可使用apt-get install mongodb-server 或 yum install mongodb-server 进行安装。

也可以在windows上安装mongodb，只要网络连通，数据就可以从关系型数据库投递到mongodb。详细安装文档可网络搜索。

启动mongodb

先创建一个数据目录

mkdir /opt/mongodb-data

然后启动DB服务, --rest是启动restful web

mongod --dbpath /opt/mongodb-data -–reset

可以通过restful admin web访问mongodb，如下

http://192.168.89.129:28017/

基本使用

插入示例数据并查询

简单统计test db.

OGG安装测试

以下使用ogg12.3 for big data介质进行数据投递，下载地址：

http://www.oracle.com/technetwork/middleware/goldengate/downloads/index.html

ogg for bigdata需要JDK 1.8及以上，安装完成之后，需要设置LD_LIBRARY_PATH环境变量

export LD_LIBRARY_PATH=/u01/jdk1.8.0_111/jre/lib/amd64/server

OGG for bigdata解压即可安装，安装完成之后，拷贝安装目录下自带的示例到dirprm目录。

[oracle@ol73 mongodb]$ pwd

/u01/ogg4bd_12.3/AdapterExamples/big-data/mongodb

[oracle@ol73 mongodb]$ cp * /u01/ogg4bd_12.3/dirprm/

修改dirprm/mongo.props文件，主要调整mongodb jdbc jar的路径和client URI

gg.handlerlist=mongodb

gg.handler.mongodb.type=mongodb

#The following handler properties are optional.

#Please refer to the Oracle GoldenGate for BigData documentation

#for details about the configuration.

gg.handler.mongodb.clientURI=mongodb://192.168.89.129:27017/

goldengate.userexit.timestamp=utc

goldengate.userexit.writers=javawriter

javawriter.stats.display=TRUE

javawriter.stats.full=TRUE

gg.log=log4j

gg.log.level=INFO

gg.report.time=30sec

#Path to MongoDB Java driver.

gg.classpath=/u01/ogg4bd_12.3/mongo-java-driver-3.4.0.jar

javawriter.bootoptions=-Xmx512m -Xms32m -Djava.class.path=.:ggjava/ggjava.jar:./dirprm

OGG投递进程参数，可直接使用示例自带的参数文件

GGSCI (ol73) 1> view param rmongo

REPLICAT rmongo

-- Trail file for this example is located in "AdapterExamples/trail" directory

-- Command to add REPLICAT

-- add replicat rmongo, exttrail AdapterExamples/trail/tr

TARGETDB LIBFILE libggjava.so SET property=dirprm/mongo.props

REPORTCOUNT EVERY 1 MINUTES, RATE

GROUPTRANSOPS 1000

MAP QASOURCE.*, TARGET QASOURCE.*;

使用OGG自带的增量数据创建投递进程

GGSCI>add replicat rmongo, exttrail AdapterExamples/trail/tr

启动投递进程

GGSCI (ol73) 3> start rmongo

Sending START request to MANAGER ...

REPLICAT RMONGO starting

统计变更数据

GGSCI (ol73) 9> stats rmongo, total

Sending STATS request to REPLICAT RMONGO ...

Start of Statistics at 2017-12-20 17:38:34.

Replicating from QASOURCE.TCUSTMER to QASOURCE.TCUSTMER:

*** Total statistics since 2017-12-20 17:38:18 ***

Total inserts 5.00

Total updates 1.00

Total deletes 0.00

Total discards 0.00

Total operations 6.00

Replicating from QASOURCE.TCUSTORD to QASOURCE.TCUSTORD:

*** Total statistics since 2017-12-20 17:38:18 ***

Total inserts 5.00

Total updates 3.00

Total deletes 2.00

Total discards 0.00

Total operations 10.00

End of Statistics.

可以看到变更数据中有2张表，包括有insert/update/delete等操作。

在数据库端验证结果

$mongo

查看数据库，已经看到有新插入的QASOURCE

> show dbs

QASOURCE 0.078GB

admin (empty)

local 0.078GB

test 0.078GB

> use QASOURCE

switched to db QASOURCE

简单查看一下数据库的基本信息

> db.stats()

{

"db" : "QASOURCE",

"collections" : 4,

"objects" : 15,

"avgObjSize" : 116.26666666666667,

"dataSize" : 1744,

"storageSize" : 32768,

"numExtents" : 4,

"indexes" : 2,

"indexSize" : 16352,

"fileSize" : 67108864,

"nsSizeMB" : 16,

"dataFileVersion" : {

"major" : 4,

"minor" : 5

"extentFreeList" : {

"num" : 0,

"totalSize" : 0

"ok" : 1

}

查看一下数据库中的集合

> db.getCollectionInfos()

[

{

"name" : "TCUSTMER"

{

"name" : "TCUSTORD"

{

"name" : "system.indexes"

}

]

> db.getCollectionNames()

[ "TCUSTMER", "TCUSTORD", "system.indexes" ]

> db.getCollectionNames()

[ "TCUSTMER", "TCUSTORD", "system.indexes" ]

查看TCUSTMER中的数据

> db.TCUSTMER.find()

{ "_id" : "WILL", "CITY" : "SEATTLE", "STATE" : "WA", "CUST_CODE" : "WILL", "NAME" : "BG SOFTWARE CO." }

{ "_id" : "JANE", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "JANE", "NAME" : "ROCKY FLYER INC." }

{ "_id" : "DAVE", "CITY" : "TALLAHASSEE", "STATE" : "FL", "CUST_CODE" : "DAVE", "NAME" : "DAVE'S PLANES INC." }

{ "_id" : "BILL", "CITY" : "DENVER", "STATE" : "CO", "CUST_CODE" : "BILL", "NAME" : "BILL'S USED CARS" }

{ "_id" : "ANN", "CITY" : "NEW YORK", "STATE" : "NY", "CUST_CODE" : "ANN", "NAME" : "ANN'S BOATS" }

查找CITY=’DENVER’的记录

> db.TCUSTMER.find({"CITY":"DENVER"})